Index: projects/clang350-import/bin/ps/ps.1 =================================================================== --- projects/clang350-import/bin/ps/ps.1 (revision 275748) +++ projects/clang350-import/bin/ps/ps.1 (revision 275749) @@ -1,765 +1,766 @@ .\"- .\" Copyright (c) 1980, 1990, 1991, 1993, 1994 .\" The Regents of the University of California. All rights reserved. .\" .\" Redistribution and use in source and binary forms, with or without .\" modification, are permitted provided that the following conditions .\" are met: .\" 1. Redistributions of source code must retain the above copyright .\" notice, this list of conditions and the following disclaimer. .\" 2. Redistributions in binary form must reproduce the above copyright .\" notice, this list of conditions and the following disclaimer in the .\" documentation and/or other materials provided with the distribution. .\" 4. Neither the name of the University nor the names of its contributors .\" may be used to endorse or promote products derived from this software .\" without specific prior written permission. .\" .\" THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND .\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE .\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE .\" ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE .\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL .\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS .\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) .\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT .\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY .\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF .\" SUCH DAMAGE. .\" .\" @(#)ps.1 8.3 (Berkeley) 4/18/94 .\" $FreeBSD$ .\" -.Dd August 27, 2014 +.Dd December 9, 2014 .Dt PS 1 .Os .Sh NAME .Nm ps .Nd process status .Sh SYNOPSIS .Nm .Op Fl aCcdefHhjlmrSTuvwXxZ .Op Fl O Ar fmt | Fl o Ar fmt .Op Fl G Ar gid Ns Op , Ns Ar gid Ns Ar ... .Op Fl J Ar jid Ns Op , Ns Ar jid Ns Ar ... .Op Fl M Ar core .Op Fl N Ar system .Op Fl p Ar pid Ns Op , Ns Ar pid Ns Ar ... .Op Fl t Ar tty Ns Op , Ns Ar tty Ns Ar ... .Op Fl U Ar user Ns Op , Ns Ar user Ns Ar ... .Nm .Op Fl L .Sh DESCRIPTION The .Nm utility displays a header line, followed by lines containing information about all of your processes that have controlling terminals. If the .Fl x options is specified, .Nm will also display processes that do not have controlling terminals. .Pp A different set of processes can be selected for display by using any combination of the .Fl a , G , J , p , T , t , and .Fl U options. If more than one of these options are given, then .Nm will select all processes which are matched by at least one of the given options. .Pp For the processes which have been selected for display, .Nm will usually display one line per process. The .Fl H option may result in multiple output lines (one line per thread) for some processes. By default all of these output lines are sorted first by controlling terminal, then by process ID. The .Fl m , r , u , and .Fl v options will change the sort order. If more than one sorting option was given, then the selected processes will be sorted by the last sorting option which was specified. .Pp For the processes which have been selected for display, the information to display is selected based on a set of keywords (see the .Fl L , O , and .Fl o options). The default output format includes, for each process, the process' ID, controlling terminal, state, CPU time (including both user and system time) and associated command. .Pp The options are as follows: .Bl -tag -width indent .It Fl a Display information about other users' processes as well as your own. If the .Va security.bsd.see_other_uids sysctl is set to zero, this option is honored only if the UID of the user is 0. .It Fl c Change the .Dq command column output to just contain the executable name, rather than the full command line. .It Fl C Change the way the CPU percentage is calculated by using a .Dq raw CPU calculation that ignores .Dq resident time (this normally has no effect). .It Fl d Arrange processes into descendancy order and prefix each command with indentation text showing sibling and parent/child relationships. If either of the .Fl m and .Fl r options are also used, they control how sibling processes are sorted relative to each other. Note that this option has no effect if the .Dq command column is not the last column displayed. .It Fl e Display the environment as well. .It Fl f Show commandline and environment information about swapped out processes. This option is honored only if the UID of the user is 0. .It Fl G Display information about processes which are running with the specified real group IDs. .It Fl H Show all of the .Em kernel visible threads associated with each process. Depending on the threading package that is in use, this may show only the process, only the kernel scheduled entities, or all of the process threads. .It Fl h Repeat the information header as often as necessary to guarantee one header per page of information. .It Fl j Print information associated with the following keywords: .Cm user , pid , ppid , pgid , sid , jobc , state , tt , time , and .Cm command . .It Fl J Display information about processes which match the specified jail IDs. This may be either the .Cm jid or .Cm name of the jail. Use .Fl J .Sy 0 to display only host processes. This flag implies .Fl x by default. .It Fl L List the set of keywords available for the .Fl O and .Fl o options. .It Fl l Display information associated with the following keywords: .Cm uid , pid , ppid , cpu , pri , nice , vsz , rss , mwchan , state , .Cm tt , time , and .Cm command . .It Fl M Extract values associated with the name list from the specified core instead of the currently running system. .It Fl m Sort by memory usage, instead of the combination of controlling terminal and process ID. .It Fl N Extract the name list from the specified system instead of the default, which is the kernel image the system has booted from. .It Fl O Add the information associated with the space or comma separated list of keywords specified, after the process ID, in the default information display. Keywords may be appended with an equals .Pq Ql = sign and a string. This causes the printed header to use the specified string instead of the standard header. .It Fl o Display information associated with the space or comma separated list of keywords specified. The last keyword in the list may be appended with an equals .Pq Ql = sign and a string that spans the rest of the argument, and can contain space and comma characters. This causes the printed header to use the specified string instead of the standard header. Multiple keywords may also be given in the form of more than one .Fl o option. So the header texts for multiple keywords can be changed. If all keywords have empty header texts, no header line is written. .It Fl p Display information about processes which match the specified process IDs. .It Fl r Sort by current CPU usage, instead of the combination of controlling terminal and process ID. .It Fl S Change the way the process times, namely cputime, systime, and usertime, are calculated by summing all exited children to their parent process. .It Fl T Display information about processes attached to the device associated with the standard input. .It Fl t Display information about processes attached to the specified terminal devices. Full pathnames, as well as abbreviations (see explanation of the .Cm tt keyword) can be specified. .It Fl U Display the processes belonging to the specified usernames. .It Fl u Display information associated with the following keywords: .Cm user , pid , %cpu , %mem , vsz , rss , tt , state , start , time , and .Cm command . The .Fl u option implies the .Fl r option. .It Fl v Display information associated with the following keywords: .Cm pid , state , time , sl , re , pagein , vsz , rss , lim , tsiz , .Cm %cpu , %mem , and .Cm command . The .Fl v option implies the .Fl m option. .It Fl w Use 132 columns to display information, instead of the default which is your window size. If the .Fl w option is specified more than once, .Nm will use as many columns as necessary without regard for your window size. Note that this option has no effect if the .Dq command column is not the last column displayed. .It Fl X When displaying processes matched by other options, skip any processes which do not have a controlling terminal. This is the default behaviour. .It Fl x When displaying processes matched by other options, include processes which do not have a controlling terminal. This is the opposite of the .Fl X option. If both .Fl X and .Fl x are specified in the same command, then .Nm will use the one which was specified last. .It Fl Z Add .Xr mac 4 label to the list of keywords for which .Nm will display information. .El .Pp A complete list of the available keywords are listed below. Some of these keywords are further specified as follows: .Bl -tag -width lockname .It Cm %cpu The CPU utilization of the process; this is a decaying average over up to a minute of previous (real) time. Since the time base over which this is computed varies (since processes may be very young) it is possible for the sum of all .Cm %cpu fields to exceed 100%. .It Cm %mem The percentage of real memory used by this process. .It Cm class Login class associated with the process. .It Cm flags The flags associated with the process as in the include file .In sys/proc.h : .Bl -column P_SINGLE_BOUNDARY 0x40000000 .It Dv "P_ADVLOCK" Ta No "0x00001" Ta "Process may hold a POSIX advisory lock" .It Dv "P_CONTROLT" Ta No "0x00002" Ta "Has a controlling terminal" .It Dv "P_KTHREAD" Ta No "0x00004" Ta "Kernel thread" .It Dv "P_FOLLOWFORK" Ta No "0x00008" Ta "Attach debugger to new children" .It Dv "P_PPWAIT" Ta No "0x00010" Ta "Parent is waiting for child to exec/exit" .It Dv "P_PROFIL" Ta No "0x00020" Ta "Has started profiling" .It Dv "P_STOPPROF" Ta No "0x00040" Ta "Has thread in requesting to stop prof" .It Dv "P_HADTHREADS" Ta No "0x00080" Ta "Has had threads (no cleanup shortcuts)" .It Dv "P_SUGID" Ta No "0x00100" Ta "Had set id privileges since last exec" .It Dv "P_SYSTEM" Ta No "0x00200" Ta "System proc: no sigs, stats or swapping" .It Dv "P_SINGLE_EXIT" Ta No "0x00400" Ta "Threads suspending should exit, not wait" .It Dv "P_TRACED" Ta No "0x00800" Ta "Debugged process being traced" .It Dv "P_WAITED" Ta No "0x01000" Ta "Someone is waiting for us" .It Dv "P_WEXIT" Ta No "0x02000" Ta "Working on exiting" .It Dv "P_EXEC" Ta No "0x04000" Ta "Process called exec" .It Dv "P_WKILLED" Ta No "0x08000" Ta "Killed, shall go to kernel/user boundary ASAP" .It Dv "P_CONTINUED" Ta No "0x10000" Ta "Proc has continued from a stopped state" .It Dv "P_STOPPED_SIG" Ta No "0x20000" Ta "Stopped due to SIGSTOP/SIGTSTP" .It Dv "P_STOPPED_TRACE" Ta No "0x40000" Ta "Stopped because of tracing" .It Dv "P_STOPPED_SINGLE" Ta No "0x80000" Ta "Only one thread can continue" .It Dv "P_PROTECTED" Ta No "0x100000" Ta "Do not kill on memory overcommit" .It Dv "P_SIGEVENT" Ta No "0x200000" Ta "Process pending signals changed" .It Dv "P_SINGLE_BOUNDARY" Ta No "0x400000" Ta "Threads should suspend at user boundary" .It Dv "P_HWPMC" Ta No "0x800000" Ta "Process is using HWPMCs" .It Dv "P_JAILED" Ta No "0x1000000" Ta "Process is in jail" +.It Dv "P_TOTAL_STOP" Ta No "0x2000000" Ta "Stopped for system suspend" .It Dv "P_INEXEC" Ta No "0x4000000" Ta "Process is in execve()" .It Dv "P_STATCHILD" Ta No "0x8000000" Ta "Child process stopped or exited" .It Dv "P_INMEM" Ta No "0x10000000" Ta "Loaded into memory" .It Dv "P_SWAPPINGOUT" Ta No "0x20000000" Ta "Process is being swapped out" .It Dv "P_SWAPPINGIN" Ta No "0x40000000" Ta "Process is being swapped in" .It Dv "P_PPTRACE" Ta No "0x80000000" Ta "Vforked child issued ptrace(PT_TRACEME)" .El .It Cm flags2 The flags kept in .Va p_flag2 associated with the process as in the include file .In sys/proc.h : .Bl -column P2_INHERIT_PROTECTED 0x00000001 .It Dv "P2_INHERIT_PROTECTED" Ta No "0x00000001" Ta "New children get P_PROTECTED" .El .It Cm label The MAC label of the process. .It Cm lim The soft limit on memory used, specified via a call to .Xr setrlimit 2 . .It Cm lstart The exact time the command started, using the .Ql %c format described in .Xr strftime 3 . .It Cm lockname The name of the lock that the process is currently blocked on. If the name is invalid or unknown, then .Dq ???\& is displayed. .It Cm logname The login name associated with the session the process is in (see .Xr getlogin 2 ) . .It Cm mwchan The event name if the process is blocked normally, or the lock name if the process is blocked on a lock. See the wchan and lockname keywords for details. .It Cm nice The process scheduling increment (see .Xr setpriority 2 ) . .It Cm rss the real memory (resident set) size of the process (in 1024 byte units). .It Cm start The time the command started. If the command started less than 24 hours ago, the start time is displayed using the .Dq Li %l:ps.1p format described in .Xr strftime 3 . If the command started less than 7 days ago, the start time is displayed using the .Dq Li %a6.15p format. Otherwise, the start time is displayed using the .Dq Li %e%b%y format. .It Cm state The state is given by a sequence of characters, for example, .Dq Li RWNA . The first character indicates the run state of the process: .Pp .Bl -tag -width indent -compact .It Li D Marks a process in disk (or other short term, uninterruptible) wait. .It Li I Marks a process that is idle (sleeping for longer than about 20 seconds). .It Li L Marks a process that is waiting to acquire a lock. .It Li R Marks a runnable process. .It Li S Marks a process that is sleeping for less than about 20 seconds. .It Li T Marks a stopped process. .It Li W Marks an idle interrupt thread. .It Li Z Marks a dead process (a .Dq zombie ) . .El .Pp Additional characters after these, if any, indicate additional state information: .Pp .Bl -tag -width indent -compact .It Li + The process is in the foreground process group of its control terminal. .It Li < The process has raised CPU scheduling priority. .It Li E The process is trying to exit. .It Li J Marks a process which is in .Xr jail 2 . The hostname of the prison can be found in .Pa /proc/ Ns Ao Ar pid Ac Ns Pa /status . .It Li L The process has pages locked in core (for example, for raw .Tn I/O ) . .It Li N The process has reduced CPU scheduling priority (see .Xr setpriority 2 ) . .It Li s The process is a session leader. .It Li V The process' parent is suspended during a .Xr vfork 2 , waiting for the process to exec or exit. .It Li W The process is swapped out. .It Li X The process is being traced or debugged. .El .It Cm tt An abbreviation for the pathname of the controlling terminal, if any. The abbreviation consists of the three letters following .Pa /dev/tty , or, for pseudo-terminals, the corresponding entry in .Pa /dev/pts . This is followed by a .Ql - if the process can no longer reach that controlling terminal (i.e., it has been revoked). A .Ql - without a preceding two letter abbreviation or pseudo-terminal device number indicates a process which never had a controlling terminal. The full pathname of the controlling terminal is available via the .Cm tty keyword. .It Cm wchan The event (an address in the system) on which a process waits. When printed numerically, the initial part of the address is trimmed off and the result is printed in hex, for example, 0x80324000 prints as 324000. .El .Pp When printing using the command keyword, a process that has exited and has a parent that has not yet waited for the process (in other words, a zombie) is listed as .Dq Li , and a process which is blocked while trying to exit is listed as .Dq Li . If the arguments cannot be located (usually because it has not been set, as is the case of system processes and/or kernel threads) the command name is printed within square brackets. The .Nm utility first tries to obtain the arguments cached by the kernel (if they were shorter than the value of the .Va kern.ps_arg_cache_limit sysctl). The process can change the arguments shown with .Xr setproctitle 3 . Otherwise, .Nm makes an educated guess as to the file name and arguments given when the process was created by examining memory or the swap area. The method is inherently somewhat unreliable and in any event a process is entitled to destroy this information. The ucomm (accounting) keyword can, however, be depended on. If the arguments are unavailable or do not agree with the ucomm keyword, the value for the ucomm keyword is appended to the arguments in parentheses. .Sh KEYWORDS The following is a complete list of the available keywords and their meanings. Several of them have aliases (keywords which are synonyms). .Pp .Bl -tag -width ".Cm sigignore" -compact .It Cm %cpu percentage CPU usage (alias .Cm pcpu ) .It Cm %mem percentage memory usage (alias .Cm pmem ) .It Cm acflag accounting flag (alias .Cm acflg ) .It Cm args command and arguments .It Cm class login class .It Cm comm command .It Cm command command and arguments .It Cm cow number of copy-on-write faults .It Cm cpu short-term CPU usage factor (for scheduling) .It Cm dsiz data size (in Kbytes) .It Cm emul system-call emulation environment .It Cm etime elapsed running time, format .Op days- Ns .Op hours: Ns minutes:seconds. .It Cm etimes elapsed running time, in decimal integer seconds .It Cm fib default FIB number, see .Xr setfib 1 .It Cm flags the process flags, in hexadecimal (alias .Cm f ) .It Cm flags2 the additional set of process flags, in hexadecimal (alias .Cm f2 ) .It Cm gid effective group ID (alias .Cm egid ) .It Cm group group name (from egid) (alias .Cm egroup ) .It Cm inblk total blocks read (alias .Cm inblock ) .It Cm jid jail ID .It Cm jobc job control count .It Cm ktrace tracing flags .It Cm label MAC label .It Cm lim memoryuse limit .It Cm lockname lock currently blocked on (as a symbolic name) .It Cm logname login name of user who started the session .It Cm lstart time started .It Cm lwp process thread-id .It Cm majflt total page faults .It Cm minflt total page reclaims .It Cm msgrcv total messages received (reads from pipes/sockets) .It Cm msgsnd total messages sent (writes on pipes/sockets) .It Cm mwchan wait channel or lock currently blocked on .It Cm nice nice value (alias .Cm ni ) .It Cm nivcsw total involuntary context switches .It Cm nlwp number of threads tied to a process .It Cm nsigs total signals taken (alias .Cm nsignals ) .It Cm nswap total swaps in/out .It Cm nvcsw total voluntary context switches .It Cm nwchan wait channel (as an address) .It Cm oublk total blocks written (alias .Cm oublock ) .It Cm paddr process pointer .It Cm pagein pageins (same as majflt) .It Cm pgid process group number .It Cm pid process ID .It Cm ppid parent process ID .It Cm pri scheduling priority .It Cm re core residency time (in seconds; 127 = infinity) .It Cm rgid real group ID .It Cm rgroup group name (from rgid) .It Cm rss resident set size .It Cm rtprio realtime priority (101 = not a realtime process) .It Cm ruid real user ID .It Cm ruser user name (from ruid) .It Cm sid session ID .It Cm sig pending signals (alias .Cm pending ) .It Cm sigcatch caught signals (alias .Cm caught ) .It Cm sigignore ignored signals (alias .Cm ignored ) .It Cm sigmask blocked signals (alias .Cm blocked ) .It Cm sl sleep time (in seconds; 127 = infinity) .It Cm ssiz stack size (in Kbytes) .It Cm start time started .It Cm state symbolic process state (alias .Cm stat ) .It Cm svgid saved gid from a setgid executable .It Cm svuid saved UID from a setuid executable .It Cm systime accumulated system CPU time .It Cm tdaddr thread address .It Cm tdev control terminal device number .It Cm time accumulated CPU time, user + system (alias .Cm cputime ) .It Cm tpgid control terminal process group ID .It Cm tracer tracer process ID .\".It Cm trss .\"text resident set size (in Kbytes) .It Cm tsid control terminal session ID .It Cm tsiz text size (in Kbytes) .It Cm tt control terminal name (two letter abbreviation) .It Cm tty full name of control terminal .It Cm ucomm name to be used for accounting .It Cm uid effective user ID (alias .Cm euid ) .It Cm upr scheduling priority on return from system call (alias .Cm usrpri ) .It Cm uprocp process pointer .It Cm user user name (from UID) .It Cm usertime accumulated user CPU time .It Cm vsz virtual size in Kbytes (alias .Cm vsize ) .It Cm wchan wait channel (as a symbolic name) .It Cm xstat exit or stop status (valid only for stopped or zombie process) .El .Pp Note that the .Cm pending column displays bitmask of signals pending in the process queue when .Fl H option is not specified, otherwise the per-thread queue of pending signals is shown. .Sh ENVIRONMENT The following environment variables affect the execution of .Nm : .Bl -tag -width ".Ev COLUMNS" .It Ev COLUMNS If set, specifies the user's preferred output width in column positions. By default, .Nm attempts to automatically determine the terminal width. .El .Sh FILES .Bl -tag -width ".Pa /boot/kernel/kernel" -compact .It Pa /boot/kernel/kernel default system namelist .El .Sh EXAMPLES Display information on all system processes: .Pp .Dl $ ps -auxw .Sh SEE ALSO .Xr kill 1 , .Xr pgrep 1 , .Xr pkill 1 , .Xr procstat 1 , .Xr w 1 , .Xr kvm 3 , .Xr strftime 3 , .Xr mac 4 , .Xr procfs 5 , .Xr pstat 8 , .Xr sysctl 8 , .Xr mutex 9 .Sh STANDARDS For historical reasons, the .Nm utility under .Fx supports a different set of options from what is described by .St -p1003.2 , and what is supported on .No non- Ns Bx operating systems. .Sh HISTORY The .Nm command appeared in .At v4 . .Sh BUGS Since .Nm cannot run faster than the system and is run as any other scheduled process, the information it displays can never be exact. .Pp The .Nm utility does not correctly display argument lists containing multibyte characters. Index: projects/clang350-import/cddl/contrib/opensolaris/cmd/zfs/zfs.8 =================================================================== --- projects/clang350-import/cddl/contrib/opensolaris/cmd/zfs/zfs.8 (revision 275748) +++ projects/clang350-import/cddl/contrib/opensolaris/cmd/zfs/zfs.8 (revision 275749) @@ -1,3643 +1,3634 @@ '\" te .\" Copyright (c) 2013, Martin Matuska . .\" All Rights Reserved. .\" .\" The contents of this file are subject to the terms of the .\" Common Development and Distribution License (the "License"). .\" You may not use this file except in compliance with the License. .\" .\" You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE .\" or http://www.opensolaris.org/os/licensing. .\" See the License for the specific language governing permissions .\" and limitations under the License. .\" .\" When distributing Covered Code, include this CDDL HEADER in each .\" file and include the License file at usr/src/OPENSOLARIS.LICENSE. .\" If applicable, add the following below this CDDL HEADER, with the .\" fields enclosed by brackets "[]" replaced with your own identifying .\" information: Portions Copyright [yyyy] [name of copyright owner] .\" .\" Copyright (c) 2010, Sun Microsystems, Inc. All Rights Reserved. .\" Copyright (c) 2014 by Delphix. All rights reserved. .\" Copyright (c) 2011, Pawel Jakub Dawidek .\" Copyright (c) 2012, Glen Barber .\" Copyright (c) 2012, Bryan Drewery .\" Copyright (c) 2013 by Saso Kiselkov. All rights reserved. -.\" Copyright (c) 2013 Nexenta Systems, Inc. All Rights Reserved. .\" Copyright (c) 2014, Joyent, Inc. All rights reserved. .\" Copyright (c) 2013, Steven Hartland +.\" Copyright (c) 2014 Nexenta Systems, Inc. All Rights Reserved. .\" Copyright (c) 2014, Xin LI .\" Copyright (c) 2014, The FreeBSD Foundation, All Rights Reserved. .\" .\" $FreeBSD$ .\" -.Dd November 12, 2014 +.Dd December 12, 2014 .Dt ZFS 8 .Os .Sh NAME .Nm zfs .Nd configures ZFS file systems .Sh SYNOPSIS .Nm .Op Fl \&? .Nm .Cm create .Op Fl pu .Oo Fl o Ar property Ns = Ns Ar value Oc Ns ... Ar filesystem .Nm .Cm create .Op Fl ps .Op Fl b Ar blocksize .Oo Fl o Ar property Ns = Ns Ar value Oc Ns ... .Fl V .Ar size volume .Nm .Cm destroy .Op Fl fnpRrv .Ar filesystem Ns | Ns Ar volume .Nm .Cm destroy .Op Fl dnpRrv .Sm off .Ar filesystem Ns | Ns volume .Ns @snap .Op % Ns Ar snap .Op , Ns Ar snap Op % Ns Ar snap .Op , Ns ... .Sm on .Nm .Cm destroy .Ar filesystem Ns | Ns Ar volume Ns # Ns Ar bookmark .Nm .Cm snapshot Ns | Ns Cm snap .Op Fl r .Oo Fl o Ar property Ns = Ns Ar value Oc Ns ... .Ar filesystem@snapname Ns | Ns Ar volume@snapname .Ar filesystem@snapname Ns | Ns Ar volume@snapname Ns ... .Nm .Cm rollback .Op Fl rRf .Ar snapshot .Nm .Cm clone .Op Fl p .Oo Fl o Ar property Ns = Ns Ar value Oc Ns ... .Ar snapshot filesystem Ns | Ns Ar volume .Nm .Cm promote .Ar clone-filesystem .Nm .Cm rename .Op Fl f .Ar filesystem Ns | Ns Ar volume Ns | Ns Ar snapshot .Ar filesystem Ns | Ns Ar volume Ns | Ns Ar snapshot .Nm .Cm rename .Op Fl f .Fl p .Ar filesystem Ns | Ns Ar volume .Ar filesystem Ns | Ns Ar volume .Nm .Cm rename .Fl r .Ar snapshot snapshot .Nm .Cm rename .Fl u .Op Fl p .Ar filesystem filesystem .Nm .Cm list .Op Fl r Ns | Ns Fl d Ar depth .Op Fl Hp .Op Fl o Ar property Ns Oo , Ns property Ns Oc Ns ... .Op Fl t Ar type Ns Oo , Ns type Ns Oc Ns ... .Oo Fl s Ar property Oc Ns ... .Oo Fl S Ar property Oc Ns ... .Ar filesystem Ns | Ns Ar volume Ns | Ns Ar snapshot .Nm .Cm set .Ar property Ns = Ns Ar value .Ar filesystem Ns | Ns Ar volume Ns | Ns Ar snapshot Ns ... .Nm .Cm get .Op Fl r Ns | Ns Fl d Ar depth .Op Fl Hp .Op Fl o Ar all | field Ns Oo , Ns Ar field Oc Ns ... .Op Fl t Ar type Ns Oo Ns , Ar type Oc Ns ... .Op Fl s Ar source Ns Oo Ns , Ns Ar source Oc Ns ... .Ar all | property Ns Oo Ns , Ns Ar property Oc Ns ... .Ar filesystem Ns | Ns Ar volume Ns | Ns Ar snapshot Ns ... .Nm .Cm inherit .Op Fl rS .Ar property .Ar filesystem Ns | Ns Ar volume Ns | Ns Ar snapshot Ns ... .Nm .Cm upgrade .Op Fl v .Nm .Cm upgrade .Op Fl r .Op Fl V Ar version .Fl a | Ar filesystem .Nm .Cm userspace .Op Fl Hinp .Op Fl o Ar field Ns Oo , Ns Ar field Oc Ns ... .Oo Fl s Ar field Oc Ns ... .Oo Fl S Ar field Oc Ns ... .Op Fl t Ar type Ns Oo Ns , Ns Ar type Oc Ns ... .Ar filesystem Ns | Ns Ar snapshot .Nm .Cm groupspace .Op Fl Hinp .Op Fl o Ar field Ns Oo , Ns field Oc Ns ... .Oo Fl s Ar field Oc Ns ... .Oo Fl S Ar field Oc Ns ... .Op Fl t Ar type Ns Oo Ns , Ns Ar type Oc Ns ... .Ar filesystem Ns | Ns Ar snapshot .Nm .Cm mount .Nm .Cm mount .Op Fl vO .Op Fl o Ar property Ns Oo , Ns Ar property Oc Ns ... .Fl a | Ar filesystem .Nm .Cm unmount Ns | Ns Cm umount .Op Fl f .Fl a | Ar filesystem Ns | Ns Ar mountpoint .Nm .Cm share .Fl a | Ar filesystem .Nm .Cm unshare .Fl a | Ar filesystem Ns | Ns Ar mountpoint .Nm .Cm bookmark .Ar snapshot .Ar bookmark .Nm .Cm send .Op Fl DnPpRveL .Op Fl i Ar snapshot | Fl I Ar snapshot .Ar snapshot .Nm .Cm send .Op Fl eL .Op Fl i Ar snapshot Ns | Ns bookmark .Ar filesystem Ns | Ns Ar volume Ns | Ns Ar snapshot .Nm .Cm receive Ns | Ns Cm recv .Op Fl vnFu .Ar filesystem Ns | Ns Ar volume Ns | Ns Ar snapshot .Nm .Cm receive Ns | Ns Cm recv .Op Fl vnFu .Op Fl d | e .Ar filesystem .Nm .Cm allow .Ar filesystem Ns | Ns Ar volume .Nm .Cm allow .Op Fl ldug .Ar user Ns | Ns Ar group Ns Oo Ns , Ns Ar user Ns | Ns Ar group Oc Ns ... .Ar perm Ns | Ns Ar @setname Ns .Oo Ns , Ns Ar perm Ns | Ns Ar @setname Oc Ns ... .Ar filesystem Ns | Ns Ar volume .Nm .Cm allow .Op Fl ld .Fl e Ns | Ns Cm everyone .Ar perm Ns | Ns Ar @setname Ns Op Ns , Ns Ar perm Ns | Ns Ar @setname Ns .Ns ... .Ar filesystem Ns | Ns Ar volume .Nm .Cm allow .Fl c .Ar perm Ns | Ns Ar @setname Ns Op Ns , Ns Ar perm Ns | Ns Ar @setname Ns .Ns ... .Ar filesystem Ns | Ns Ar volume .Nm .Cm allow .Fl s .Ar @setname .Ar perm Ns | Ns Ar @setname Ns Op Ns , Ns Ar perm Ns | Ns Ar @setname Ns .Ns ... .Ar filesystem Ns | Ns Ar volume .Nm .Cm unallow .Op Fl rldug .Ar user Ns | Ns Ar group Ns Oo Ns , Ns Ar user Ns | Ns Ar group Oc Ns ... .Oo Ar perm Ns | Ns Ar @setname Ns Op , Ns Ar perm Ns | Ns Ar @setname Ns .Ns ... Oc .Ar filesystem Ns | Ns Ar volume .Nm .Cm unallow .Op Fl rld .Fl e Ns | Ns Cm everyone .Oo Ar perm Ns | Ns Ar @setname Ns Op , Ns Ar perm Ns | Ns Ar @setname Ns .Ns ... Oc .Ar filesystem Ns | Ns Ar volume .Nm .Cm unallow .Op Fl r .Fl c .Oo Ar perm Ns | Ns Ar @setname Ns Op , Ns Ar perm Ns | Ns Ar @setname Ns .Ns ... Oc .Ar filesystem Ns | Ns Ar volume .Nm .Cm unallow .Op Fl r .Fl s .Ar @setname .Oo Ar perm Ns | Ns Ar @setname Ns Op , Ns Ar perm Ns | Ns Ar @setname Ns .Ns ... Oc .Ar filesystem Ns | Ns Ar volume .Nm .Cm hold .Op Fl r .Ar tag snapshot Ns ... .Nm .Cm holds .Op Fl r .Ar snapshot Ns ... .Nm .Cm release .Op Fl r .Ar tag snapshot Ns ... .Nm .Cm diff .Op Fl FHt .Ar snapshot .Op Ar snapshot Ns | Ns Ar filesystem .Nm .Cm jail .Ar jailid Ns | Ns Ar jailname filesystem .Nm .Cm unjail .Ar jailid Ns | Ns Ar jailname filesystem .Sh DESCRIPTION The .Nm command configures .Tn ZFS datasets within a .Tn ZFS storage pool, as described in .Xr zpool 8 . A dataset is identified by a unique path within the .Tn ZFS namespace. For example: .Bd -ragged -offset 4n .No pool/ Ns Brq filesystem,volume,snapshot .Ed .Pp where the maximum length of a dataset name is .Dv MAXNAMELEN (256 bytes). .Pp A dataset can be one of the following: .Bl -hang -width 12n .It Sy file system A .Tn ZFS dataset of type .Em filesystem can be mounted within the standard system namespace and behaves like other file systems. While .Tn ZFS file systems are designed to be .Tn POSIX compliant, known issues exist that prevent compliance in some cases. Applications that depend on standards conformance might fail due to nonstandard behavior when checking file system free space. .It Sy volume A logical volume exported as a raw or block device. This type of dataset should only be used under special circumstances. File systems are typically used in most environments. .It Sy snapshot A read-only version of a file system or volume at a given point in time. It is specified as .Em filesystem@name or .Em volume@name . .El .Ss ZFS File System Hierarchy A .Tn ZFS storage pool is a logical collection of devices that provide space for datasets. A storage pool is also the root of the .Tn ZFS file system hierarchy. .Pp The root of the pool can be accessed as a file system, such as mounting and unmounting, taking snapshots, and setting properties. The physical storage characteristics, however, are managed by the .Xr zpool 8 command. .Pp See .Xr zpool 8 for more information on creating and administering pools. .Ss Snapshots A snapshot is a read-only copy of a file system or volume. Snapshots can be created extremely quickly, and initially consume no additional space within the pool. As data within the active dataset changes, the snapshot consumes more data than would otherwise be shared with the active dataset. .Pp Snapshots can have arbitrary names. Snapshots of volumes can be cloned or rolled back, but cannot be accessed independently. .Pp File system snapshots can be accessed under the .Pa \&.zfs/snapshot directory in the root of the file system. Snapshots are automatically mounted on demand and may be unmounted at regular intervals. The visibility of the .Pa \&.zfs directory can be controlled by the .Sy snapdir property. .Ss Clones A clone is a writable volume or file system whose initial contents are the same as another dataset. As with snapshots, creating a clone is nearly instantaneous, and initially consumes no additional space. .Pp Clones can only be created from a snapshot. When a snapshot is cloned, it creates an implicit dependency between the parent and child. Even though the clone is created somewhere else in the dataset hierarchy, the original snapshot cannot be destroyed as long as a clone exists. The .Sy origin property exposes this dependency, and the .Cm destroy command lists any such dependencies, if they exist. .Pp The clone parent-child dependency relationship can be reversed by using the .Cm promote subcommand. This causes the "origin" file system to become a clone of the specified file system, which makes it possible to destroy the file system that the clone was created from. .Ss Mount Points Creating a .Tn ZFS file system is a simple operation, so the number of file systems per system is likely to be numerous. To cope with this, .Tn ZFS automatically manages mounting and unmounting file systems without the need to edit the .Pa /etc/fstab file. All automatically managed file systems are mounted by .Tn ZFS at boot time. .Pp By default, file systems are mounted under .Pa /path , where .Ar path is the name of the file system in the .Tn ZFS namespace. Directories are created and destroyed as needed. .Pp A file system can also have a mount point set in the .Sy mountpoint property. This directory is created as needed, and .Tn ZFS automatically mounts the file system when the .Qq Nm Cm mount Fl a command is invoked (without editing .Pa /etc/fstab ) . The .Sy mountpoint property can be inherited, so if .Em pool/home has a mount point of .Pa /home , then .Em pool/home/user automatically inherits a mount point of .Pa /home/user . .Pp A file system .Sy mountpoint property of .Cm none prevents the file system from being mounted. .Pp If needed, .Tn ZFS file systems can also be managed with traditional tools .Pq Xr mount 8 , Xr umount 8 , Xr fstab 5 . If a file system's mount point is set to .Cm legacy , .Tn ZFS makes no attempt to manage the file system, and the administrator is responsible for mounting and unmounting the file system. .Ss Jails .No A Tn ZFS dataset can be attached to a jail by using the .Qq Nm Cm jail subcommand. You cannot attach a dataset to one jail and the children of the same dataset to another jails. To allow management of the dataset from within a jail, the .Sy jailed property has to be set and the jail needs access to the .Pa /dev/zfs device. The .Sy quota property cannot be changed from within a jail. See .Xr jail 8 for information on how to allow mounting .Tn ZFS datasets from within a jail. .Pp .No A Tn ZFS dataset can be detached from a jail using the .Qq Nm Cm unjail subcommand. .Pp After a dataset is attached to a jail and the jailed property is set, a jailed file system cannot be mounted outside the jail, since the jail administrator might have set the mount point to an unacceptable value. .Ss Deduplication Deduplication is the process for removing redundant data at the block-level, reducing the total amount of data stored. If a file system has the .Cm dedup property enabled, duplicate data blocks are removed synchronously. The result is that only unique data is stored and common components are shared among files. .Ss Native Properties Properties are divided into two types, native properties and user-defined (or "user") properties. Native properties either export internal statistics or control .Tn ZFS behavior. In addition, native properties are either editable or read-only. User properties have no effect on .Tn ZFS behavior, but you can use them to annotate datasets in a way that is meaningful in your environment. For more information about user properties, see the .Qq Sx User Properties section, below. .Pp Every dataset has a set of properties that export statistics about the dataset as well as control various behaviors. Properties are inherited from the parent unless overridden by the child. Some properties apply only to certain types of datasets (file systems, volumes, or snapshots). .Pp The values of numeric properties can be specified using human-readable suffixes (for example, .Sy k , KB , M , Gb , and so forth, up to .Sy Z for zettabyte). The following are all valid (and equal) specifications: .Bd -ragged -offset 4n 1536M, 1.5g, 1.50GB .Ed .Pp The values of non-numeric properties are case sensitive and must be lowercase, except for .Sy mountpoint , sharenfs , No and Sy sharesmb . .Pp The following native properties consist of read-only statistics about the dataset. These properties can be neither set, nor inherited. Native properties apply to all dataset types unless otherwise noted. .Bl -tag -width 2n .It Sy available The amount of space available to the dataset and all its children, assuming that there is no other activity in the pool. Because space is shared within a pool, availability can be limited by any number of factors, including physical pool size, quotas, reservations, or other datasets within the pool. .Pp This property can also be referred to by its shortened column name, .Sy avail . .It Sy compressratio For non-snapshots, the compression ratio achieved for the .Sy used space of this dataset, expressed as a multiplier. The .Sy used property includes descendant datasets, and, for clones, does not include the space shared with the origin snapshot. For snapshots, the .Sy compressratio is the same as the .Sy refcompressratio property. Compression can be turned on by running: .Qq Nm Cm set compression=on Ar dataset The default value is .Cm off . .It Sy creation The time this dataset was created. .It Sy clones For snapshots, this property is a comma-separated list of filesystems or volumes which are clones of this snapshot. The clones' .Sy origin property is this snapshot. If the .Sy clones property is not empty, then this snapshot can not be destroyed (even with the .Fl r or .Fl f options). .It Sy defer_destroy This property is .Cm on if the snapshot has been marked for deferred destroy by using the .Qq Nm Cm destroy -d command. Otherwise, the property is .Cm off . .It Sy filesystem_count The total number of filesystems and volumes that exist under this location in the dataset tree. This value is only available when a .Sy filesystem_limit has been set somewhere in the tree under which the dataset resides. .It Sy logicalreferenced The amount of space that is .Qq logically accessible by this dataset. See the .Sy referenced property. The logical space ignores the effect of the .Sy compression and .Sy copies properties, giving a quantity closer to the amount of data that applications see. However, it does include space consumed by metadata. .Pp This property can also be referred to by its shortened column name, .Sy lrefer . .It Sy logicalused The amount of space that is .Qq logically consumed by this dataset and all its descendents. See the .Sy used property. The logical space ignores the effect of the .Sy compression and .Sy copies properties, giving a quantity closer to the amount of data that applications see. .Pp This property can also be referred to by its shortened column name, .Sy lused . .It Sy mounted For file systems, indicates whether the file system is currently mounted. This property can be either .Cm yes or .Cm no . .It Sy origin For cloned file systems or volumes, the snapshot from which the clone was created. See also the .Sy clones property. .It Sy referenced The amount of data that is accessible by this dataset, which may or may not be shared with other datasets in the pool. When a snapshot or clone is created, it initially references the same amount of space as the file system or snapshot it was created from, since its contents are identical. .Pp This property can also be referred to by its shortened column name, .Sy refer . .It Sy refcompressratio The compression ratio achieved for the .Sy referenced space of this dataset, expressed as a multiplier. See also the .Sy compressratio property. .It Sy snapshot_count The total number of snapshots that exist under this location in the dataset tree. This value is only available when a .Sy snapshot_limit has been set somewhere in the tree under which the dataset resides. .It Sy type The type of dataset: .Sy filesystem , volume , No or Sy snapshot . .It Sy used The amount of space consumed by this dataset and all its descendents. This is the value that is checked against this dataset's quota and reservation. The space used does not include this dataset's reservation, but does take into account the reservations of any descendent datasets. The amount of space that a dataset consumes from its parent, as well as the amount of space that are freed if this dataset is recursively destroyed, is the greater of its space used and its reservation. .Pp When snapshots (see the .Qq Sx Snapshots section) are created, their space is initially shared between the snapshot and the file system, and possibly with previous snapshots. As the file system changes, space that was previously shared becomes unique to the snapshot, and counted in the snapshot's space used. Additionally, deleting snapshots can increase the amount of space unique to (and used by) other snapshots. .Pp The amount of space used, available, or referenced does not take into account pending changes. Pending changes are generally accounted for within a few seconds. Committing a change to a disk using .Xr fsync 2 or .Sy O_SYNC does not necessarily guarantee that the space usage information is updated immediately. .It Sy usedby* The .Sy usedby* properties decompose the .Sy used properties into the various reasons that space is used. Specifically, .Sy used No = .Sy usedbysnapshots + usedbydataset + usedbychildren + usedbyrefreservation . These properties are only available for datasets created with .Tn ZFS pool version 13 pools and higher. .It Sy usedbysnapshots The amount of space consumed by snapshots of this dataset. In particular, it is the amount of space that would be freed if all of this dataset's snapshots were destroyed. Note that this is not simply the sum of the snapshots' .Sy used properties because space can be shared by multiple snapshots. .It Sy usedbydataset The amount of space used by this dataset itself, which would be freed if the dataset were destroyed (after first removing any .Sy refreservation and destroying any necessary snapshots or descendents). .It Sy usedbychildren The amount of space used by children of this dataset, which would be freed if all the dataset's children were destroyed. .It Sy usedbyrefreservation The amount of space used by a .Sy refreservation set on this dataset, which would be freed if the .Sy refreservation was removed. .It Sy userused@ Ns Ar user The amount of space consumed by the specified user in this dataset. Space is charged to the owner of each file, as displayed by .Qq Nm ls Fl l . The amount of space charged is displayed by .Qq Nm du and .Qq Nm ls Fl s . See the .Qq Nm Cm userspace subcommand for more information. .Pp Unprivileged users can access only their own space usage. The root user, or a user who has been granted the .Sy userused privilege with .Qq Nm Cm allow , can access everyone's usage. .Pp The .Sy userused@ Ns ... properties are not displayed by .Qq Nm Cm get all . The user's name must be appended after the .Sy @ symbol, using one of the following forms: .Bl -bullet -offset 2n .It POSIX name (for example, .Em joe ) .It POSIX numeric ID (for example, .Em 1001 ) .El .It Sy userrefs This property is set to the number of user holds on this snapshot. User holds are set by using the .Qq Nm Cm hold command. .It Sy groupused@ Ns Ar group The amount of space consumed by the specified group in this dataset. Space is charged to the group of each file, as displayed by .Nm ls Fl l . See the .Sy userused@ Ns Ar user property for more information. .Pp Unprivileged users can only access their own groups' space usage. The root user, or a user who has been granted the .Sy groupused privilege with .Qq Nm Cm allow , can access all groups' usage. .It Sy volblocksize Ns = Ns Ar blocksize For volumes, specifies the block size of the volume. The .Ar blocksize cannot be changed once the volume has been written, so it should be set at volume creation time. The default .Ar blocksize for volumes is 8 Kbytes. Any power of 2 from 512 bytes to 128 Kbytes is valid. .Pp This property can also be referred to by its shortened column name, .Sy volblock . .It Sy written The amount of .Sy referenced space written to this dataset since the previous snapshot. .It Sy written@ Ns Ar snapshot The amount of .Sy referenced space written to this dataset since the specified snapshot. This is the space that is referenced by this dataset but was not referenced by the specified snapshot. .Pp The .Ar snapshot may be specified as a short snapshot name (just the part after the .Sy @ ) , in which case it will be interpreted as a snapshot in the same filesystem as this dataset. The .Ar snapshot may be a full snapshot name .Pq Em filesystem@snapshot , which for clones may be a snapshot in the origin's filesystem (or the origin of the origin's filesystem, etc). .El .Pp The following native properties can be used to change the behavior of a .Tn ZFS dataset. .Bl -tag -width 2n .It Xo .Sy aclinherit Ns = Ns Cm discard | .Cm noallow | .Cm restricted | .Cm passthrough | .Cm passthrough-x .Xc Controls how .Tn ACL entries are inherited when files and directories are created. A file system with an .Sy aclinherit property of .Cm discard does not inherit any .Tn ACL entries. A file system with an .Sy aclinherit property value of .Cm noallow only inherits inheritable .Tn ACL entries that specify "deny" permissions. The property value .Cm restricted (the default) removes the .Em write_acl and .Em write_owner permissions when the .Tn ACL entry is inherited. A file system with an .Sy aclinherit property value of .Cm passthrough inherits all inheritable .Tn ACL entries without any modifications made to the .Tn ACL entries when they are inherited. A file system with an .Sy aclinherit property value of .Cm passthrough-x has the same meaning as .Cm passthrough , except that the .Em owner@ , group@ , No and Em everyone@ Tn ACE Ns s inherit the execute permission only if the file creation mode also requests the execute bit. .Pp When the property value is set to .Cm passthrough , files are created with a mode determined by the inheritable .Tn ACE Ns s. If no inheritable .Tn ACE Ns s exist that affect the mode, then the mode is set in accordance to the requested mode from the application. .It Sy aclmode Ns = Ns Cm discard | groupmask | passthrough | restricted Controls how an .Tn ACL is modified during .Xr chmod 2 . A file system with an .Sy aclmode property of .Cm discard (the default) deletes all .Tn ACL entries that do not represent the mode of the file. An .Sy aclmode property of .Cm groupmask reduces permissions granted in all .Em ALLOW entries found in the .Tn ACL such that they are no greater than the group permissions specified by .Xr chmod 2 . A file system with an .Sy aclmode property of .Cm passthrough indicates that no changes are made to the .Tn ACL other than creating or updating the necessary .Tn ACL entries to represent the new mode of the file or directory. An .Sy aclmode property of .Cm restricted will cause the .Xr chmod 2 operation to return an error when used on any file or directory which has a non-trivial .Tn ACL whose entries can not be represented by a mode. .Xr chmod 2 is required to change the set user ID, set group ID, or sticky bits on a file or directory, as they do not have equivalent .Tn ACL entries. In order to use .Xr chmod 2 on a file or directory with a non-trivial .Tn ACL when .Sy aclmode is set to .Cm restricted , you must first remove all .Tn ACL entries which do not represent the current mode. .It Sy atime Ns = Ns Cm on | off Controls whether the access time for files is updated when they are read. Turning this property off avoids producing write traffic when reading files and can result in significant performance gains, though it might confuse mailers and other similar utilities. The default value is .Cm on . .It Sy canmount Ns = Ns Cm on | off | noauto If this property is set to .Cm off , the file system cannot be mounted, and is ignored by .Qq Nm Cm mount Fl a . Setting this property to .Cm off is similar to setting the .Sy mountpoint property to .Cm none , except that the dataset still has a normal .Sy mountpoint property, which can be inherited. Setting this property to .Cm off allows datasets to be used solely as a mechanism to inherit properties. One example of setting .Sy canmount Ns = Ns Cm off is to have two datasets with the same .Sy mountpoint , so that the children of both datasets appear in the same directory, but might have different inherited characteristics. .Pp When the .Cm noauto value is set, a dataset can only be mounted and unmounted explicitly. The dataset is not mounted automatically when the dataset is created or imported, nor is it mounted by the .Qq Nm Cm mount Fl a command or unmounted by the .Qq Nm Cm umount Fl a command. .Pp This property is not inherited. .It Sy checksum Ns = Ns Cm on | off | fletcher2 | fletcher4 | sha256 | noparity Controls the checksum used to verify data integrity. The default value is .Cm on , which automatically selects an appropriate algorithm (currently, .Cm fletcher4 , but this may change in future releases). The value .Cm off disables integrity checking on user data. The value .Cm noparity not only disables integrity but also disables maintaining parity for user data. This setting is used internally by a dump device residing on a RAID-Z pool and should not be used by any other dataset. Disabling checksums is .Em NOT a recommended practice. .It Sy compression Ns = Ns Cm on | off | lzjb | gzip | gzip- Ns Ar N | zle | Cm lz4 Controls the compression algorithm used for this dataset. The .Cm lzjb compression algorithm is optimized for performance while providing decent data compression. Setting compression to .Cm on uses the .Cm lzjb compression algorithm. The .Cm gzip compression algorithm uses the same compression as the .Xr gzip 1 command. You can specify the .Cm gzip level by using the value .Cm gzip- Ns Ar N where .Ar N is an integer from 1 (fastest) to 9 (best compression ratio). Currently, .Cm gzip is equivalent to .Cm gzip-6 (which is also the default for .Xr gzip 1 ) . The .Cm zle compression algorithm compresses runs of zeros. .Pp The .Sy lz4 compression algorithm is a high-performance replacement for the .Sy lzjb algorithm. It features significantly faster compression and decompression, as well as a moderately higher compression ratio than .Sy lzjb , but can only be used on pools with the .Sy lz4_compress feature set to .Sy enabled . See .Xr zpool-features 7 for details on ZFS feature flags and the .Sy lz4_compress feature. .Pp This property can also be referred to by its shortened column name .Cm compress . Changing this property affects only newly-written data. .It Sy copies Ns = Ns Cm 1 | 2 | 3 Controls the number of copies of data stored for this dataset. These copies are in addition to any redundancy provided by the pool, for example, mirroring or RAID-Z. The copies are stored on different disks, if possible. The space used by multiple copies is charged to the associated file and dataset, changing the .Sy used property and counting against quotas and reservations. .Pp Changing this property only affects newly-written data. Therefore, set this property at file system creation time by using the .Fl o Cm copies= Ns Ar N option. .It Sy dedup Ns = Ns Cm on | off | verify | sha256 Ns Op Cm ,verify Configures deduplication for a dataset. The default value is .Cm off . The default deduplication checksum is .Cm sha256 (this may change in the future). When .Sy dedup is enabled, the checksum defined here overrides the .Sy checksum property. Setting the value to .Cm verify has the same effect as the setting .Cm sha256,verify . .Pp If set to .Cm verify , .Tn ZFS will do a byte-to-byte comparsion in case of two blocks having the same signature to make sure the block contents are identical. .It Sy devices Ns = Ns Cm on | off The .Sy devices property is currently not supported on .Fx . .It Sy exec Ns = Ns Cm on | off Controls whether processes can be executed from within this file system. The default value is .Cm on . .It Sy mlslabel Ns = Ns Ar label | Cm none The .Sy mlslabel property is currently not supported on .Fx . .It Sy filesystem_limit Ns = Ns Ar count | Cm none Limits the number of filesystems and volumes that can exist under this point in the dataset tree. The limit is not enforced if the user is allowed to change the limit. Setting a .Sy filesystem_limit on a descendent of a filesystem that already has a .Sy filesystem_limit does not override the ancestor's .Sy filesystem_limit , but rather imposes an additional limit. This feature must be enabled to be used .Po see .Xr zpool-features 7 .Pc . .It Sy mountpoint Ns = Ns Ar path | Cm none | legacy Controls the mount point used for this file system. See the .Qq Sx Mount Points section for more information on how this property is used. .Pp When the .Sy mountpoint property is changed for a file system, the file system and any children that inherit the mount point are unmounted. If the new value is .Cm legacy , then they remain unmounted. Otherwise, they are automatically remounted in the new location if the property was previously .Cm legacy or .Cm none , or if they were mounted before the property was changed. In addition, any shared file systems are unshared and shared in the new location. .It Sy nbmand Ns = Ns Cm on | off The .Sy nbmand property is currently not supported on .Fx . .It Sy primarycache Ns = Ns Cm all | none | metadata Controls what is cached in the primary cache (ARC). If this property is set to .Cm all , then both user data and metadata is cached. If this property is set to .Cm none , then neither user data nor metadata is cached. If this property is set to .Cm metadata , then only metadata is cached. The default value is .Cm all . .It Sy quota Ns = Ns Ar size | Cm none Limits the amount of space a dataset and its descendents can consume. This property enforces a hard limit on the amount of space used. This includes all space consumed by descendents, including file systems and snapshots. Setting a quota on a descendent of a dataset that already has a quota does not override the ancestor's quota, but rather imposes an additional limit. .Pp Quotas cannot be set on volumes, as the .Sy volsize property acts as an implicit quota. .It Sy snapshot_limit Ns = Ns Ar count | Cm none Limits the number of snapshots that can be created on a dataset and its descendents. Setting a .Sy snapshot_limit on a descendent of a dataset that already has a .Sy snapshot_limit does not override the ancestor's .Sy snapshot_limit , but rather imposes an additional limit. The limit is not enforced if the user is allowed to change the limit. For example, this means that recursive snapshots taken from the global zone are counted against each delegated dataset within a jail. This feature must be enabled to be used .Po see .Xr zpool-features 7 .Pc . .It Sy userquota@ Ns Ar user Ns = Ns Ar size | Cm none Limits the amount of space consumed by the specified user. Similar to the .Sy refquota property, the .Sy userquota space calculation does not include space that is used by descendent datasets, such as snapshots and clones. User space consumption is identified by the .Sy userspace@ Ns Ar user property. .Pp Enforcement of user quotas may be delayed by several seconds. This delay means that a user might exceed their quota before the system notices that they are over quota and begins to refuse additional writes with the .Em EDQUOT error message. See the .Cm userspace subcommand for more information. .Pp Unprivileged users can only access their own groups' space usage. The root user, or a user who has been granted the .Sy userquota privilege with .Qq Nm Cm allow , can get and set everyone's quota. .Pp This property is not available on volumes, on file systems before version 4, or on pools before version 15. The .Sy userquota@ Ns ... properties are not displayed by .Qq Nm Cm get all . The user's name must be appended after the .Sy @ symbol, using one of the following forms: .Bl -bullet -offset 2n .It POSIX name (for example, .Em joe ) .It POSIX numeric ID (for example, .Em 1001 ) .El .It Sy groupquota@ Ns Ar group Ns = Ns Ar size | Cm none Limits the amount of space consumed by the specified group. Group space consumption is identified by the .Sy userquota@ Ns Ar user property. .Pp Unprivileged users can access only their own groups' space usage. The root user, or a user who has been granted the .Sy groupquota privilege with .Qq Nm Cm allow , can get and set all groups' quotas. .It Sy readonly Ns = Ns Cm on | off Controls whether this dataset can be modified. The default value is .Cm off . .It Sy recordsize Ns = Ns Ar size Specifies a suggested block size for files in the file system. This property is designed solely for use with database workloads that access files in fixed-size records. .Tn ZFS automatically tunes block sizes according to internal algorithms optimized for typical access patterns. .Pp For databases that create very large files but access them in small random chunks, these algorithms may be suboptimal. Specifying a .Sy recordsize greater than or equal to the record size of the database can result in significant performance gains. Use of this property for general purpose file systems is strongly discouraged, and may adversely affect performance. .Pp The size specified must be a power of two greater than or equal to 512 and less than or equal to 128 Kbytes. If the .Sy large_blocks feature is enabled on the pool, the size may be up to 1 Mbyte. See .Xr zpool-features 7 for details on ZFS feature flags. .Pp Changing the file system's .Sy recordsize affects only files created afterward; existing files are unaffected. .Pp This property can also be referred to by its shortened column name, .Sy recsize . .It Sy redundant_metadata Ns = Ns Cm all | most Controls what types of metadata are stored redundantly. ZFS stores an extra copy of metadata, so that if a single block is corrupted, the amount of user data lost is limited. This extra copy is in addition to any redundancy provided at the pool level .Pq e.g. by mirroring or RAID-Z , and is in addition to an extra copy specified by the .Sy copies property .Pq up to a total of 3 copies . For example if the pool is mirrored, .Cm copies Ns = Ns Ar 2 , and .Cm redundant_metadata Ns = Ns Ar most , then ZFS stores 6 copies of most metadata, and 4 copies of data and some metadata. .Pp When set to .Cm all , ZFS stores an extra copy of all metadata. If a single on-disk block is corrupt, at worst a single block of user data .Po which is .Cm recordsize bytes long can be lost. .Pc .Pp When set to .Cm most , ZFS stores an extra copy of most types of metadata. This can improve performance of random writes, because less metadata must be written. In practice, at worst about 100 blocks .Po of .Cm recordsize bytes each .Pc of user data can be lost if a single on-disk block is corrupt. The exact behavior of which metadata blocks are stored redundantly may change in future releases. .Pp The default value is .Cm all . .It Sy refquota Ns = Ns Ar size | Cm none Limits the amount of space a dataset can consume. This property enforces a hard limit on the amount of space used. This hard limit does not include space used by descendents, including file systems and snapshots. .It Sy refreservation Ns = Ns Ar size | Cm none The minimum amount of space guaranteed to a dataset, not including its descendents. When the amount of space used is below this value, the dataset is treated as if it were taking up the amount of space specified by .Sy refreservation . The .Sy refreservation reservation is accounted for in the parent datasets' space used, and counts against the parent datasets' quotas and reservations. .Pp If .Sy refreservation is set, a snapshot is only allowed if there is enough free pool space outside of this reservation to accommodate the current number of "referenced" bytes in the dataset. .Pp This property can also be referred to by its shortened column name, .Sy refreserv . .It Sy reservation Ns = Ns Ar size | Cm none The minimum amount of space guaranteed to a dataset and its descendents. When the amount of space used is below this value, the dataset is treated as if it were taking up the amount of space specified by its reservation. Reservations are accounted for in the parent datasets' space used, and count against the parent datasets' quotas and reservations. .Pp This property can also be referred to by its shortened column name, .Sy reserv . .It Sy secondarycache Ns = Ns Cm all | none | metadata Controls what is cached in the secondary cache (L2ARC). If this property is set to .Cm all , then both user data and metadata is cached. If this property is set to .Cm none , then neither user data nor metadata is cached. If this property is set to .Cm metadata , then only metadata is cached. The default value is .Cm all . .It Sy setuid Ns = Ns Cm on | off Controls whether the .No set- Ns Tn UID bit is respected for the file system. The default value is .Cm on . .It Sy sharesmb Ns = Ns Cm on | off | Ar opts The .Sy sharesmb property currently has no effect on .Fx . .It Sy sharenfs Ns = Ns Cm on | off | Ar opts Controls whether the file system is shared via .Tn NFS , and what options are used. A file system with a .Sy sharenfs property of .Cm off is managed the traditional way via .Xr exports 5 . Otherwise, the file system is automatically shared and unshared with the .Qq Nm Cm share and .Qq Nm Cm unshare commands. If the property is set to .Cm on no .Tn NFS export options are used. Otherwise, .Tn NFS export options are equivalent to the contents of this property. The export options may be comma-separated. See .Xr exports 5 for a list of valid options. .Pp When the .Sy sharenfs property is changed for a dataset, the .Xr mountd 8 daemon is reloaded. .It Sy logbias Ns = Ns Cm latency | throughput Provide a hint to .Tn ZFS about handling of synchronous requests in this dataset. If .Sy logbias is set to .Cm latency (the default), .Tn ZFS will use pool log devices (if configured) to handle the requests at low latency. If .Sy logbias is set to .Cm throughput , .Tn ZFS will not use configured pool log devices. .Tn ZFS will instead optimize synchronous operations for global pool throughput and efficient use of resources. .It Sy snapdir Ns = Ns Cm hidden | visible Controls whether the .Pa \&.zfs directory is hidden or visible in the root of the file system as discussed in the .Qq Sx Snapshots section. The default value is .Cm hidden . .It Sy sync Ns = Ns Cm standard | always | disabled Controls the behavior of synchronous requests (e.g. .Xr fsync 2 , O_DSYNC). This property accepts the following values: .Bl -tag -offset 4n -width 8n .It Sy standard This is the POSIX specified behavior of ensuring all synchronous requests are written to stable storage and all devices are flushed to ensure data is not cached by device controllers (this is the default). .It Sy always All file system transactions are written and flushed before their system calls return. This has a large performance penalty. .It Sy disabled Disables synchronous requests. File system transactions are only committed to stable storage periodically. This option will give the highest performance. However, it is very dangerous as .Tn ZFS would be ignoring the synchronous transaction demands of applications such as databases or .Tn NFS . Administrators should only use this option when the risks are understood. .El .It Sy volsize Ns = Ns Ar size For volumes, specifies the logical size of the volume. By default, creating a volume establishes a reservation of equal size. For storage pools with a version number of 9 or higher, a .Sy refreservation is set instead. Any changes to .Sy volsize are reflected in an equivalent change to the reservation (or .Sy refreservation ) . The .Sy volsize can only be set to a multiple of .Cm volblocksize , and cannot be zero. .Pp The reservation is kept equal to the volume's logical size to prevent unexpected behavior for consumers. Without the reservation, the volume could run out of space, resulting in undefined behavior or data corruption, depending on how the volume is used. These effects can also occur when the volume size is changed while it is in use (particularly when shrinking the size). Extreme care should be used when adjusting the volume size. .Pp Though not recommended, a "sparse volume" (also known as "thin provisioning") can be created by specifying the .Fl s option to the .Qq Nm Cm create Fl V command, or by changing the reservation after the volume has been created. A "sparse volume" is a volume where the reservation is less then the volume size. Consequently, writes to a sparse volume can fail with .Sy ENOSPC when the pool is low on space. For a sparse volume, changes to .Sy volsize are not reflected in the reservation. .It Sy volmode Ns = Ns Cm default | geom | dev | none This property specifies how volumes should be exposed to the OS. Setting it to .Sy geom exposes volumes as .Xr geom 4 providers, providing maximal functionality. Setting it to .Sy dev exposes volumes only as cdev device in devfs. Such volumes can be accessed only as raw disk device files, i.e. they can not be partitioned, mounted, participate in RAIDs, etc, but they are faster, and in some use scenarios with untrusted consumer, such as NAS or VM storage, can be more safe. Volumes with property set to .Sy none are not exposed outside ZFS, but can be snapshoted, cloned, replicated, etc, that can be suitable for backup purposes. Value .Sy default means that volumes exposition is controlled by system-wide sysctl/tunable .Va vfs.zfs.vol.mode , where .Sy geom , .Sy dev and .Sy none are encoded as 1, 2 and 3 respectively. The default values is .Sy geom . This property can be changed any time, but so far it is processed only during volume creation and pool import. .It Sy vscan Ns = Ns Cm off | on The .Sy vscan property is currently not supported on .Fx . .It Sy xattr Ns = Ns Cm off | on The .Sy xattr property is currently not supported on .Fx . .It Sy jailed Ns = Ns Cm off | on Controls whether the dataset is managed from a jail. See the .Qq Sx Jails section for more information. The default value is .Cm off . .El .Pp The following three properties cannot be changed after the file system is created, and therefore, should be set when the file system is created. If the properties are not set with the .Qq Nm Cm create or .Nm zpool Cm create commands, these properties are inherited from the parent dataset. If the parent dataset lacks these properties due to having been created prior to these features being supported, the new file system will have the default values for these properties. .Bl -tag -width 4n .It Sy casesensitivity Ns = Ns Cm sensitive | insensitive | mixed Indicates whether the file name matching algorithm used by the file system should be case-sensitive, case-insensitive, or allow a combination of both styles of matching. The default value for the .Sy casesensitivity property is .Cm sensitive . Traditionally, UNIX and POSIX file systems have case-sensitive file names. .Pp The .Cm mixed value for the .Sy casesensitivity property indicates that the file system can support requests for both case-sensitive and case-insensitive matching behavior. .It Sy normalization Ns = Ns Cm none | formC | formD | formKC | formKD Indicates whether the file system should perform a .Sy unicode normalization of file names whenever two file names are compared, and which normalization algorithm should be used. File names are always stored unmodified, names are normalized as part of any comparison process. If this property is set to a legal value other than .Cm none , and the .Sy utf8only property was left unspecified, the .Sy utf8only property is automatically set to .Cm on . The default value of the .Sy normalization property is .Cm none . This property cannot be changed after the file system is created. .It Sy utf8only Ns = Ns Cm on | off Indicates whether the file system should reject file names that include characters that are not present in the .Sy UTF-8 character code set. If this property is explicitly set to .Cm off , the normalization property must either not be explicitly set or be set to .Cm none . The default value for the .Sy utf8only property is .Cm off . This property cannot be changed after the file system is created. .El .Pp The .Sy casesensitivity , normalization , No and Sy utf8only properties are also new permissions that can be assigned to non-privileged users by using the .Tn ZFS delegated administration feature. .Ss Temporary Mount Point Properties When a file system is mounted, either through .Xr mount 8 for legacy mounts or the .Qq Nm Cm mount command for normal file systems, its mount options are set according to its properties. The correlation between properties and mount options is as follows: .Bl -column -offset 4n "PROPERTY" "MOUNT OPTION" .It "PROPERTY MOUNT OPTION" .It "atime atime/noatime" .It "exec exec/noexec" .It "readonly ro/rw" .It "setuid suid/nosuid" .El .Pp In addition, these options can be set on a per-mount basis using the .Fl o option, without affecting the property that is stored on disk. The values specified on the command line override the values stored in the dataset. These properties are reported as "temporary" by the .Qq Nm Cm get command. If the properties are changed while the dataset is mounted, the new setting overrides any temporary settings. .Ss User Properties In addition to the standard native properties, .Tn ZFS supports arbitrary user properties. User properties have no effect on .Tn ZFS behavior, but applications or administrators can use them to annotate datasets (file systems, volumes, and snapshots). .Pp User property names must contain a colon .Pq Sy \&: character to distinguish them from native properties. They may contain lowercase letters, numbers, and the following punctuation characters: colon .Pq Sy \&: , dash .Pq Sy \&- , period .Pq Sy \&. and underscore .Pq Sy \&_ . The expected convention is that the property name is divided into two portions such as .Em module Ns Sy \&: Ns Em property , but this namespace is not enforced by .Tn ZFS . User property names can be at most 256 characters, and cannot begin with a dash .Pq Sy \&- . .Pp When making programmatic use of user properties, it is strongly suggested to use a reversed .Tn DNS domain name for the .Ar module component of property names to reduce the chance that two independently-developed packages use the same property name for different purposes. Property names beginning with .Em com.sun are reserved for use by Sun Microsystems. .Pp The values of user properties are arbitrary strings, are always inherited, and are never validated. All of the commands that operate on properties .Po .Qq Nm Cm list , .Qq Nm Cm get , .Qq Nm Cm set and so forth .Pc can be used to manipulate both native properties and user properties. Use the .Qq Nm Cm inherit command to clear a user property. If the property is not defined in any parent dataset, it is removed entirely. Property values are limited to 1024 characters. .Sh SUBCOMMANDS All subcommands that modify state are logged persistently to the pool in their original form. .Bl -tag -width 2n .It Xo .Nm .Op Fl \&? .Xc .Pp Displays a help message. .It Xo .Nm .Cm create .Op Fl pu .Oo Fl o Ar property Ns = Ns Ar value Oc Ns ... .Ar filesystem .Xc .Pp Creates a new .Tn ZFS file system. The file system is automatically mounted according to the .Sy mountpoint property inherited from the parent. .Bl -tag -width indent .It Fl p Creates all the non-existing parent datasets. Datasets created in this manner are automatically mounted according to the .Sy mountpoint property inherited from their parent. Any property specified on the command line using the .Fl o option is ignored. If the target filesystem already exists, the operation completes successfully. .It Fl u Newly created file system is not mounted. .It Fl o Ar property Ns = Ns Ar value Sets the specified property as if the command .Qq Nm Cm set Ar property Ns = Ns Ar value was invoked at the same time the dataset was created. Any editable .Tn ZFS property can also be set at creation time. Multiple .Fl o options can be specified. An error results if the same property is specified in multiple .Fl o options. .El .It Xo .Nm .Cm create .Op Fl ps .Op Fl b Ar blocksize .Oo Fl o Ar property Ns = Ns Ar value Oc Ns ... .Fl V .Ar size volume .Xc .Pp Creates a volume of the given size. The volume is exported as a block device in .Pa /dev/zvol/path , where .Ar path is the name of the volume in the .Tn ZFS namespace. The size represents the logical size as exported by the device. By default, a reservation of equal size is created. .Pp .Ar size is automatically rounded up to the nearest 128 Kbytes to ensure that the volume has an integral number of blocks regardless of .Ar blocksize . .Bl -tag -width indent .It Fl p Creates all the non-existing parent datasets. Datasets created in this manner are automatically mounted according to the .Sy mountpoint property inherited from their parent. Any property specified on the command line using the .Fl o option is ignored. If the target filesystem already exists, the operation completes successfully. .It Fl s Creates a sparse volume with no reservation. See .Sy volsize in the .Qq Sx Native Properties section for more information about sparse volumes. .It Fl b Ar blocksize Equivalent to .Fl o Cm volblocksize Ns = Ns Ar blocksize . If this option is specified in conjunction with .Fl o Cm volblocksize , the resulting behavior is undefined. .It Fl o Ar property Ns = Ns Ar value Sets the specified property as if the .Qq Nm Cm set Ar property Ns = Ns Ar value command was invoked at the same time the dataset was created. Any editable .Tn ZFS property can also be set at creation time. Multiple .Fl o options can be specified. An error results if the same property is specified in multiple .Fl o options. .El .It Xo .Nm .Cm destroy .Op Fl fnpRrv .Ar filesystem Ns | Ns Ar volume .Xc .Pp Destroys the given dataset. By default, the command unshares any file systems that are currently shared, unmounts any file systems that are currently mounted, and refuses to destroy a dataset that has active dependents (children or clones). .Bl -tag -width indent .It Fl r Recursively destroy all children. .It Fl R Recursively destroy all dependents, including cloned file systems outside the target hierarchy. .It Fl f Force an unmount of any file systems using the .Qq Nm Cm unmount Fl f command. This option has no effect on non-file systems or unmounted file systems. .It Fl n Do a dry-run ("No-op") deletion. No data will be deleted. This is useful in conjunction with the .Fl v or .Fl p flags to determine what data would be deleted. .It Fl p Print machine-parsable verbose information about the deleted data. .It Fl v Print verbose information about the deleted data. .El .Pp Extreme care should be taken when applying either the .Fl r or the .Fl R options, as they can destroy large portions of a pool and cause unexpected behavior for mounted file systems in use. .It Xo .Nm .Cm destroy .Op Fl dnpRrv .Sm off .Ar snapshot .Op % Ns Ar snapname .Op , Ns ... .Sm on .Xc .Pp The given snapshots are destroyed immediately if and only if the .Qq Nm Cm destroy command without the .Fl d option would have destroyed it. Such immediate destruction would occur, for example, if the snapshot had no clones and the user-initiated reference count were zero. .Pp If a snapshot does not qualify for immediate destruction, it is marked for deferred deletion. In this state, it exists as a usable, visible snapshot until both of the preconditions listed above are met, at which point it is destroyed. .Pp An inclusive range of snapshots may be specified by separating the first and last snapshots with a percent sign .Pq Sy % . The first and/or last snapshots may be left blank, in which case the filesystem's oldest or newest snapshot will be implied. .Pp Multiple snapshots (or ranges of snapshots) of the same filesystem or volume may be specified in a comma-separated list of snapshots. Only the snapshot's short name (the part after the .Sy @ ) should be specified when using a range or comma-separated list to identify multiple snapshots. .Bl -tag -width indent .It Fl r Destroy (or mark for deferred deletion) all snapshots with this name in descendent file systems. .It Fl R Recursively destroy all clones of these snapshots, including the clones, snapshots, and children. If this flag is specified, the .Fl d flag will have no effect. .It Fl n Do a dry-run ("No-op") deletion. No data will be deleted. This is useful in conjunction with the .Fl v or .Fl p flags to determine what data would be deleted. .It Fl p Print machine-parsable verbose information about the deleted data. .It Fl v Print verbose information about the deleted data. .It Fl d Defer snapshot deletion. .El .Pp Extreme care should be taken when applying either the .Fl r or the .Fl R options, as they can destroy large portions of a pool and cause unexpected behavior for mounted file systems in use. .It Xo .Nm .Cm destroy .Ar filesystem Ns | Ns Ar volume Ns # Ns Ar bookmark .Xc .Pp The given bookmark is destroyed. .It Xo .Nm .Cm snapshot Ns | Ns Cm snap .Op Fl r .Oo Fl o Ar property Ns = Ns Ar value Oc Ns ... .Ar filesystem@snapname Ns | Ns volume@snapname .Ar filesystem@snapname Ns | Ns volume@snapname Ns ... .Xc .Pp Creates snapshots with the given names. All previous modifications by successful system calls to the file system are part of the snapshots. Snapshots are taken atomically, so that all snapshots correspond to the same moment in time. See the .Qq Sx Snapshots section for details. .Bl -tag -width indent .It Fl r Recursively create snapshots of all descendent datasets .It Fl o Ar property Ns = Ns Ar value Sets the specified property; see .Qq Nm Cm create for details. .El .It Xo .Nm .Cm rollback .Op Fl rRf .Ar snapshot .Xc .Pp Roll back the given dataset to a previous snapshot. When a dataset is rolled back, all data that has changed since the snapshot is discarded, and the dataset reverts to the state at the time of the snapshot. By default, the command refuses to roll back to a snapshot other than the most recent one. In order to do so, all intermediate snapshots and bookmarks must be destroyed by specifying the .Fl r option. .Pp The .Fl rR options do not recursively destroy the child snapshots of a recursive snapshot. Only direct snapshots of the specified filesystem are destroyed by either of these options. To completely roll back a recursive snapshot, you must rollback the individual child snapshots. .Bl -tag -width indent .It Fl r Destroy any snapshots and bookmarks more recent than the one specified. .It Fl R Destroy any more recent snapshots and bookmarks, as well as any clones of those snapshots. .It Fl f Used with the .Fl R option to force an unmount of any clone file systems that are to be destroyed. .El .It Xo .Nm .Cm clone .Op Fl p .Oo Fl o Ar property Ns = Ns Ar value Oc Ns ... .Ar snapshot filesystem Ns | Ns Ar volume .Xc .Pp Creates a clone of the given snapshot. See the .Qq Sx Clones section for details. The target dataset can be located anywhere in the .Tn ZFS hierarchy, and is created as the same type as the original. .Bl -tag -width indent .It Fl p Creates all the non-existing parent datasets. Datasets created in this manner are automatically mounted according to the .Sy mountpoint property inherited from their parent. If the target filesystem or volume already exists, the operation completes successfully. .It Fl o Ar property Ns = Ns Ar value Sets the specified property; see .Qq Nm Cm create for details. .El .It Xo .Nm .Cm promote .Ar clone-filesystem .Xc .Pp Promotes a clone file system to no longer be dependent on its "origin" snapshot. This makes it possible to destroy the file system that the clone was created from. The clone parent-child dependency relationship is reversed, so that the origin file system becomes a clone of the specified file system. .Pp The snapshot that was cloned, and any snapshots previous to this snapshot, are now owned by the promoted clone. The space they use moves from the origin file system to the promoted clone, so enough space must be available to accommodate these snapshots. No new space is consumed by this operation, but the space accounting is adjusted. The promoted clone must not have any conflicting snapshot names of its own. The .Cm rename subcommand can be used to rename any conflicting snapshots. .It Xo .Nm .Cm rename .Op Fl f .Ar filesystem Ns | Ns Ar volume Ns | Ns Ar snapshot .Ar filesystem Ns | Ns Ar volume Ns | Ns Ar snapshot .Xc .It Xo .Nm .Cm rename .Op Fl f .Fl p .Ar filesystem Ns | Ns Ar volume .Ar filesystem Ns | Ns Ar volume .Xc .It Xo .Nm .Cm rename .Fl u .Op Fl p .Ar filesystem filesystem .Xc .Pp Renames the given dataset. The new target can be located anywhere in the .Tn ZFS hierarchy, with the exception of snapshots. Snapshots can only be renamed within the parent file system or volume. When renaming a snapshot, the parent file system of the snapshot does not need to be specified as part of the second argument. Renamed file systems can inherit new mount points, in which case they are unmounted and remounted at the new mount point. .Bl -tag -width indent .It Fl p Creates all the nonexistent parent datasets. Datasets created in this manner are automatically mounted according to the .Sy mountpoint property inherited from their parent. .It Fl u Do not remount file systems during rename. If a file system's .Sy mountpoint property is set to .Cm legacy or .Cm none , file system is not unmounted even if this option is not given. .It Fl f Force unmount any filesystems that need to be unmounted in the process. This flag has no effect if used together with the .Fl u flag. .El .It Xo .Nm .Cm rename .Fl r .Ar snapshot snapshot .Xc .Pp Recursively rename the snapshots of all descendent datasets. Snapshots are the only dataset that can be renamed recursively. .It Xo .Nm .Cm list .Op Fl r Ns | Ns Fl d Ar depth .Op Fl Hp .Op Fl o Ar property Ns Oo , Ns Ar property Oc Ns ... .Op Fl t Ar type Ns Oo , Ns Ar type Oc Ns ... .Oo Fl s Ar property Oc Ns ... .Oo Fl S Ar property Oc Ns ... .Ar filesystem Ns | Ns Ar volume Ns | Ns Ar snapshot Ns ... .Xc .Pp Lists the property information for the given datasets in tabular form. If specified, you can list property information by the absolute pathname or the relative pathname. By default, all file systems and volumes are displayed. Snapshots are displayed if the .Sy listsnaps property is .Cm on (the default is .Cm off ) . The following fields are displayed, .Sy name , used , available , referenced , mountpoint . .Bl -tag -width indent .It Fl r Recursively display any children of the dataset on the command line. .It Fl d Ar depth Recursively display any children of the dataset, limiting the recursion to .Ar depth . A depth of .Sy 1 will display only the dataset and its direct children. .It Fl H Used for scripting mode. Do not print headers and separate fields by a single tab instead of arbitrary white space. .It Fl p Display numbers in parsable (exact) values. .It Fl o Ar property Ns Oo , Ns Ar property Oc Ns ... A comma-separated list of properties to display. The property must be: .Bl -bullet -offset 2n .It One of the properties described in the .Qq Sx Native Properties section .It A user property .It The value .Cm name to display the dataset name .It The value .Cm space to display space usage properties on file systems and volumes. This is a shortcut for specifying .Fl o .Sy name,avail,used,usedsnap,usedds,usedrefreserv,usedchild .Fl t .Sy filesystem,volume syntax. .El .It Fl t Ar type Ns Oo , Ns Ar type Oc Ns ... A comma-separated list of types to display, where .Ar type is one of .Sy filesystem , snapshot , snap , volume , bookmark , No or Sy all . For example, specifying .Fl t Cm snapshot displays only snapshots. .It Fl s Ar property A property for sorting the output by column in ascending order based on the value of the property. The property must be one of the properties described in the .Qq Sx Properties section, or the special value .Cm name to sort by the dataset name. Multiple properties can be specified at one time using multiple .Fl s property options. Multiple .Fl s options are evaluated from left to right in decreasing order of importance. .Pp The following is a list of sorting criteria: .Bl -bullet -offset 2n .It Numeric types sort in numeric order. .It String types sort in alphabetical order. .It Types inappropriate for a row sort that row to the literal bottom, regardless of the specified ordering. .It If no sorting options are specified the existing behavior of .Qq Nm Cm list is preserved. .El .It Fl S Ar property Same as the .Fl s option, but sorts by property in descending order. .El .It Xo .Nm .Cm set .Ar property Ns = Ns Ar value .Ar filesystem Ns | Ns Ar volume Ns | Ns Ar snapshot .Xc .Pp Sets the property to the given value for each dataset. Only some properties can be edited. See the "Properties" section for more information on what properties can be set and acceptable values. Numeric values can be specified as exact values, or in a human-readable form with a suffix of .Sy B , K , M , G , T , P , E , Z (for bytes, kilobytes, megabytes, gigabytes, terabytes, petabytes, exabytes, or zettabytes, respectively). User properties can be set on snapshots. For more information, see the .Qq Sx User Properties section. .It Xo .Nm .Cm get .Op Fl r Ns | Ns Fl d Ar depth .Op Fl Hp .Op Fl o Ar all | field Ns Oo , Ns Ar field Oc Ns ... .Op Fl t Ar type Ns Oo , Ns Ar type Oc Ns ... .Op Fl s Ar source Ns Oo , Ns Ar source Oc Ns ... .Ar all | property Ns Oo , Ns Ar property Oc Ns ... .Ar filesystem Ns | Ns Ar volume Ns | Ns Ar snapshot Ns ... .Xc .Pp Displays properties for the given datasets. If no datasets are specified, then the command displays properties for all datasets on the system. For each property, the following columns are displayed: .Pp .Bl -hang -width "property" -offset indent -compact .It name Dataset name .It property Property name .It value Property value .It source Property source. Can either be local, default, temporary, inherited, or none (\&-). .El .Pp All columns except the .Sy RECEIVED column are displayed by default. The columns to display can be specified by using the .Fl o option. This command takes a comma-separated list of properties as described in the .Qq Sx Native Properties and .Qq Sx User Properties sections. .Pp The special value .Cm all can be used to display all properties that apply to the given dataset's type (filesystem, volume, snapshot, or bookmark). .Bl -tag -width indent .It Fl r Recursively display properties for any children. .It Fl d Ar depth Recursively display any children of the dataset, limiting the recursion to .Ar depth . A depth of .Sy 1 will display only the dataset and its direct children. .It Fl H Display output in a form more easily parsed by scripts. Any headers are omitted, and fields are explicitly separated by a single tab instead of an arbitrary amount of space. .It Fl p Display numbers in parsable (exact) values. .It Fl o Cm all | Ar field Ns Oo , Ns Ar field Oc Ns ... A comma-separated list of columns to display. Supported values are .Sy name,property,value,received,source . Default values are .Sy name,property,value,source . The keyword .Cm all specifies all columns. .It Fl t Ar type Ns Oo , Ns Ar type Oc Ns ... A comma-separated list of types to display, where .Ar type is one of .Sy filesystem , snapshot , volume , No or Sy all . For example, specifying .Fl t Cm snapshot displays only snapshots. .It Fl s Ar source Ns Oo , Ns Ar source Oc Ns ... A comma-separated list of sources to display. Those properties coming from a source other than those in this list are ignored. Each source must be one of the following: .Sy local,default,inherited,temporary,received,none . The default value is all sources. .El .It Xo .Nm .Cm inherit .Op Fl rS .Ar property .Ar filesystem Ns | Ns Ar volume Ns | Ns Ar snapshot Ns ... .Xc .Pp Clears the specified property, causing it to be inherited from an ancestor. If no ancestor has the property set, then the default value is used. See the .Qq Sx Properties section for a listing of default values, and details on which properties can be inherited. .Bl -tag -width indent .It Fl r Recursively inherit the given property for all children. .It Fl S For properties with a received value, revert to this value. This flag has no effect on properties that do not have a received value. .El .It Xo .Nm .Cm upgrade .Op Fl v .Xc .Pp Displays a list of file systems that are not the most recent version. .Bl -tag -width indent .It Fl v Displays .Tn ZFS filesystem versions supported by the current software. The current .Tn ZFS filesystem version and all previous supported versions are displayed, along with an explanation of the features provided with each version. .El .It Xo .Nm .Cm upgrade .Op Fl r .Op Fl V Ar version .Fl a | Ar filesystem .Xc .Pp Upgrades file systems to a new on-disk version. Once this is done, the file systems will no longer be accessible on systems running older versions of the software. .Qq Nm Cm send streams generated from new snapshots of these file systems cannot be accessed on systems running older versions of the software. .Pp In general, the file system version is independent of the pool version. See .Xr zpool 8 for information on the .Nm zpool Cm upgrade command. .Pp In some cases, the file system version and the pool version are interrelated and the pool version must be upgraded before the file system version can be upgraded. .Bl -tag -width indent .It Fl r Upgrade the specified file system and all descendent file systems. .It Fl V Ar version Upgrade to the specified .Ar version . If the .Fl V flag is not specified, this command upgrades to the most recent version. This option can only be used to increase the version number, and only up to the most recent version supported by this software. .It Fl a Upgrade all file systems on all imported pools. .It Ar filesystem Upgrade the specified file system. .El .It Xo .Nm .Cm userspace .Op Fl Hinp .Op Fl o Ar field Ns Oo , Ns Ar field Oc Ns ... .Oo Fl s Ar field Oc Ns ... .Oo Fl S Ar field Oc Ns ... .Op Fl t Ar type Ns Oo , Ns Ar type Oc Ns ... .Ar filesystem Ns | Ns Ar snapshot .Xc .Pp Displays space consumed by, and quotas on, each user in the specified filesystem or snapshot. This corresponds to the .Sy userused@ Ns Ar user and .Sy userquota@ Ns Ar user properties. .Bl -tag -width indent .It Fl n Print numeric ID instead of user/group name. .It Fl H Do not print headers, use tab-delimited output. .It Fl p Use exact (parsable) numeric output. .It Fl o Ar field Ns Oo , Ns Ar field Oc Ns ... Display only the specified fields from the following set: .Sy type,name,used,quota . The default is to display all fields. .It Fl s Ar field Sort output by this field. The .Fl s and .Fl S flags may be specified multiple times to sort first by one field, then by another. The default is .Fl s Cm type Fl s Cm name . .It Fl S Ar field Sort by this field in reverse order. See .Fl s . .It Fl t Ar type Ns Oo , Ns Ar type Oc Ns ... Print only the specified types from the following set: .Sy all,posixuser,smbuser,posixgroup,smbgroup . .Pp The default is .Fl t Cm posixuser,smbuser . .Pp The default can be changed to include group types. .It Fl i Translate SID to POSIX ID. This flag currently has no effect on .Fx . .El .It Xo .Nm .Cm groupspace .Op Fl Hinp .Op Fl o Ar field Ns Oo , Ns Ar field Oc Ns ... .Oo Fl s Ar field Oc Ns ... .Oo Fl S Ar field Oc Ns ... .Op Fl t Ar type Ns Oo , Ns Ar type Oc Ns ... .Ar filesystem Ns | Ns Ar snapshot .Xc .Pp Displays space consumed by, and quotas on, each group in the specified filesystem or snapshot. This subcommand is identical to .Qq Nm Cm userspace , except that the default types to display are .Fl t Sy posixgroup,smbgroup . .It Xo .Nm .Cm mount .Xc .Pp Displays all .Tn ZFS file systems currently mounted. .Bl -tag -width indent .It Fl f .El .It Xo .Nm .Cm mount .Op Fl vO .Op Fl o Ar property Ns Oo , Ns Ar property Oc Ns ... .Fl a | Ar filesystem .Xc .Pp Mounts .Tn ZFS file systems. .Bl -tag -width indent .It Fl v Report mount progress. .It Fl O Perform an overlay mount. Overlay mounts are not supported on .Fx . .It Fl o Ar property Ns Oo , Ns Ar property Oc Ns ... An optional, comma-separated list of mount options to use temporarily for the duration of the mount. See the .Qq Sx Temporary Mount Point Properties section for details. .It Fl a Mount all available .Tn ZFS file systems. This command may be executed on .Fx system startup by .Pa /etc/rc.d/zfs . For more information, see variable .Va zfs_enable in .Xr rc.conf 5 . .It Ar filesystem Mount the specified filesystem. .El .It Xo .Nm .Cm unmount Ns | Ns Cm umount .Op Fl f .Fl a | Ar filesystem Ns | Ns Ar mountpoint .Xc .Pp Unmounts currently mounted .Tn ZFS file systems. .Bl -tag -width indent .It Fl f Forcefully unmount the file system, even if it is currently in use. .It Fl a Unmount all available .Tn ZFS file systems. .It Ar filesystem | mountpoint Unmount the specified filesystem. The command can also be given a path to a .Tn ZFS file system mount point on the system. .El .It Xo .Nm .Cm share .Fl a | Ar filesystem .Xc .Pp Shares .Tn ZFS file systems that have the .Sy sharenfs property set. .Bl -tag -width indent .It Fl a Share all .Tn ZFS file systems that have the .Sy sharenfs property set. This command may be executed on .Fx system startup by .Pa /etc/rc.d/zfs . For more information, see variable .Va zfs_enable in .Xr rc.conf 5 . .It Ar filesystem Share the specified filesystem according to the .Tn sharenfs property. File systems are shared when the .Tn sharenfs property is set. .El .It Xo .Nm .Cm unshare .Fl a | Ar filesystem Ns | Ns Ar mountpoint .Xc .Pp Unshares .Tn ZFS file systems that have the .Tn sharenfs property set. .Bl -tag -width indent .It Fl a Unshares .Tn ZFS file systems that have the .Sy sharenfs property set. This command may be executed on .Fx system shutdown by .Pa /etc/rc.d/zfs . For more information, see variable .Va zfs_enable in .Xr rc.conf 5 . .It Ar filesystem | mountpoint Unshare the specified filesystem. The command can also be given a path to a .Tn ZFS file system shared on the system. .El .It Xo .Nm .Cm bookmark .Ar snapshot .Ar bookmark .Xc .Pp Creates a bookmark of the given snapshot. Bookmarks mark the point in time when the snapshot was created, and can be used as the incremental source for a .Qq Nm Cm send command. .Pp This feature must be enabled to be used. See .Xr zpool-features 7 for details on ZFS feature flags and the .Sy bookmark feature. .It Xo .Nm .Cm send .Op Fl DnPpRveL .Op Fl i Ar snapshot | Fl I Ar snapshot .Ar snapshot .Xc .Pp Creates a stream representation of the last .Ar snapshot argument (not part of .Fl i or .Fl I ) which is written to standard output. The output can be redirected to a file or to a different system (for example, using .Xr ssh 1 ) . By default, a full stream is generated. .Bl -tag -width indent .It Fl i Ar snapshot Generate an incremental stream from the first .Ar snapshot Pq the incremental source to the second .Ar snapshot Pq the incremental target . The incremental source can be specified as the last component of the snapshot name .Pq the Em @ No character and following and it is assumed to be from the same file system as the incremental target. .Pp If the destination is a clone, the source may be the origin snapshot, which must be fully specified (for example, .Cm pool/fs@origin , not just .Cm @origin ) . .It Fl I Ar snapshot Generate a stream package that sends all intermediary snapshots from the first .Ar snapshot to the second .Ar snapshot . For example, .Ic -I @a fs@d is similar to .Ic -i @a fs@b; -i @b fs@c; -i @c fs@d . The incremental source may be specified as with the .Fl i option. .It Fl R Generate a replication stream package, which will replicate the specified filesystem, and all descendent file systems, up to the named snapshot. When received, all properties, snapshots, descendent file systems, and clones are preserved. .Pp If the .Fl i or .Fl I flags are used in conjunction with the .Fl R flag, an incremental replication stream is generated. The current values of properties, and current snapshot and file system names are set when the stream is received. If the .Fl F flag is specified when this stream is received, snapshots and file systems that do not exist on the sending side are destroyed. .It Fl D Generate a deduplicated stream. Blocks which would have been sent multiple times in the send stream will only be sent once. The receiving system must also support this feature to receive a deduplicated stream. This flag can be used regardless of the dataset's .Sy dedup property, but performance will be much better if the filesystem uses a dedup-capable checksum (eg. .Sy sha256 ) . .It Fl L Generate a stream which may contain blocks larger than 128KB. This flag has no effect if the .Sy large_blocks pool feature is disabled, or if the .Sy recordsize property of this filesystem has never been set above 128KB. The receiving system must have the .Sy large_blocks pool feature enabled as well. See .Xr zpool-features 7 for details on ZFS feature flags and the .Sy large_blocks feature. .It Fl e Generate a more compact stream by using WRITE_EMBEDDED records for blocks which are stored more compactly on disk by the .Sy embedded_data pool feature. This flag has no effect if the .Sy embedded_data feature is disabled. The receiving system must have the .Sy embedded_data feature enabled. If the .Sy lz4_compress feature is active on the sending system, then the receiving system must have that feature enabled as well. See .Xr zpool-features 7 for details on ZFS feature flags and the .Sy embedded_data feature. .It Fl p Include the dataset's properties in the stream. This flag is implicit when .Fl R is specified. The receiving system must also support this feature. .It Fl n Do a dry-run ("No-op") send. Do not generate any actual send data. This is useful in conjunction with the .Fl v or .Fl P flags to determine what data will be sent. .It Fl P Print machine-parsable verbose information about the stream package generated. .It Fl v Print verbose information about the stream package generated. This information includes a per-second report of how much data has been sent. .El .Pp The format of the stream is committed. You will be able to receive your streams on future versions of .Tn ZFS . .It Xo .Nm .Cm send .Op Fl eL .Op Fl i Ar snapshot Ns | Ns Ar bookmark .Ar filesystem Ns | Ns Ar volume Ns | Ns Ar snapshot .Xc .Pp Generate a send stream, which may be of a filesystem, and may be incremental from a bookmark. If the destination is a filesystem or volume, the pool must be read-only, or the filesystem must not be mounted. When the stream generated from a filesystem or volume is received, the default snapshot name will be .Pq --head-- . .Bl -tag -width indent .It Fl i Ar snapshot Ns | Ns bookmark Generate an incremental send stream. The incremental source must be an earlier snapshot in the destination's history. It will commonly be an earlier snapshot in the destination's filesystem, in which case it can be specified as the last component of the name .Pq the Em # No or Em @ No character and following . .Pp If the incremental target is a clone, the incremental source can be the origin snapshot, or an earlier snapshot in the origin's filesystem, or the origin's origin, etc. .It Fl L Generate a stream which may contain blocks larger than 128KB. This flag has no effect if the .Sy large_blocks pool feature is disabled, or if the .Sy recordsize property of this filesystem has never been set above 128KB. The receiving system must have the .Sy large_blocks pool feature enabled as well. See .Xr zpool-features 7 for details on ZFS feature flags and the .Sy large_blocks feature. .It Fl e Generate a more compact stream by using WRITE_EMBEDDED records for blocks which are stored more compactly on disk by the .Sy embedded_data pool feature. This flag has no effect if the .Sy embedded_data feature is disabled. The receiving system must have the .Sy embedded_data feature enabled. If the .Sy lz4_compress feature is active on the sending system, then the receiving system must have that feature enabled as well. See .Xr zpool-features 7 for details on ZFS feature flags and the .Sy embedded_data feature. .El .It Xo .Nm .Cm receive Ns | Ns Cm recv .Op Fl vnFu .Ar filesystem Ns | Ns Ar volume Ns | Ns Ar snapshot .Xc .It Xo .Nm .Cm receive Ns | Ns Cm recv .Op Fl vnFu .Op Fl d | e .Ar filesystem .Xc .Pp Creates a snapshot whose contents are as specified in the stream provided on standard input. If a full stream is received, then a new file system is created as well. Streams are created using the .Qq Nm Cm send subcommand, which by default creates a full stream. .Qq Nm Cm recv can be used as an alias for .Qq Nm Cm receive . .Pp If an incremental stream is received, then the destination file system must already exist, and its most recent snapshot must match the incremental stream's source. For .Sy zvol Ns s, the destination device link is destroyed and recreated, which means the .Sy zvol cannot be accessed during the .Sy receive operation. .Pp When a snapshot replication package stream that is generated by using the .Qq Nm Cm send Fl R command is received, any snapshots that do not exist on the sending location are destroyed by using the .Qq Nm Cm destroy Fl d command. .Pp The name of the snapshot (and file system, if a full stream is received) that this subcommand creates depends on the argument type and the .Fl d or .Fl e option. .Pp If the argument is a snapshot name, the specified .Ar snapshot is created. If the argument is a file system or volume name, a snapshot with the same name as the sent snapshot is created within the specified .Ar filesystem or .Ar volume . If the .Fl d or .Fl e option is specified, the snapshot name is determined by appending the sent snapshot's name to the specified .Ar filesystem . If the .Fl d option is specified, all but the pool name of the sent snapshot path is appended (for example, .Sy b/c@1 appended from sent snapshot .Sy a/b/c@1 ) , and if the .Fl e option is specified, only the tail of the sent snapshot path is appended (for example, .Sy c@1 appended from sent snapshot .Sy a/b/c@1 ) . In the case of .Fl d , any file systems needed to replicate the path of the sent snapshot are created within the specified file system. .Bl -tag -width indent .It Fl d Use the full sent snapshot path without the first element (without pool name) to determine the name of the new snapshot as described in the paragraph above. .It Fl e Use only the last element of the sent snapshot path to determine the name of the new snapshot as described in the paragraph above. .It Fl u File system that is associated with the received stream is not mounted. .It Fl v Print verbose information about the stream and the time required to perform the receive operation. .It Fl n Do not actually receive the stream. This can be useful in conjunction with the .Fl v option to verify the name the receive operation would use. .It Fl F Force a rollback of the file system to the most recent snapshot before performing the receive operation. If receiving an incremental replication stream (for example, one generated by .Qq Nm Cm send Fl R Bro Fl i | Fl I Brc ) , destroy snapshots and file systems that do not exist on the sending side. .El .It Xo .Nm .Cm allow .Ar filesystem Ns | Ns Ar volume .Xc .Pp Displays permissions that have been delegated on the specified filesystem or volume. See the other forms of .Qq Nm Cm allow for more information. .It Xo .Nm .Cm allow .Op Fl ldug .Ar user Ns | Ns Ar group Ns Oo Ns , Ns Ar user Ns | Ns Ar group Oc Ns ... .Ar perm Ns | Ns Ar @setname Ns .Oo Ns , Ns Ar perm Ns | Ns Ar @setname Oc Ns ... .Ar filesystem Ns | Ns Ar volume .Xc .It Xo .Nm .Cm allow .Op Fl ld .Fl e Ns | Ns Cm everyone .Ar perm Ns | Ns Ar @setname Ns Op Ns , Ns Ar perm Ns | Ns Ar @setname Ns .Ns ... .Ar filesystem Ns | Ns Ar volume .Xc .Pp Delegates .Tn ZFS administration permission for the file systems to non-privileged users. .Bl -tag -width indent .It Xo .Op Fl ug .Ar user Ns | Ns Ar group Ns Oo , Ar user Ns | Ns Ar group Oc Ns ... .Xc Specifies to whom the permissions are delegated. Multiple entities can be specified as a comma-separated list. If neither of the .Fl ug options are specified, then the argument is interpreted preferentially as the keyword .Cm everyone , then as a user name, and lastly as a group name. To specify a user or group named .Qq everyone , use the .Fl u or .Fl g options. To specify a group with the same name as a user, use the .Fl g option. .It Op Fl e Ns | Ns Cm everyone Specifies that the permissions be delegated to .Qq everyone . .It Xo .Ar perm Ns | Ns Ar @setname Ns Oo , Ns Ar perm Ns | Ns Ar @setname Oc Ns ... .Xc The permissions to delegate. Multiple permissions may be specified as a comma-separated list. Permission names are the same as .Tn ZFS subcommand and property names. See the property list below. Property set names, which begin with an at sign .Pq Sy @ , may be specified. See the .Fl s form below for details. .It Xo .Op Fl ld .Ar filesystem Ns | Ns Ar volume .Xc Specifies where the permissions are delegated. If neither of the .Fl ld options are specified, or both are, then the permissions are allowed for the file system or volume, and all of its descendents. If only the .Fl l option is used, then is allowed "locally" only for the specified file system. If only the .Fl d option is used, then is allowed only for the descendent file systems. .El .Pp Permissions are generally the ability to use a .Tn ZFS subcommand or change a .Tn ZFS property. The following permissions are available: .Bl -column -offset 4n "secondarycache" "subcommand" .It NAME Ta TYPE Ta NOTES .It allow Ta subcommand Ta Must Xo also have the permission that is being allowed .Xc .It clone Ta subcommand Ta Must Xo also have the 'create' ability and 'mount' ability in the origin file system .Xc .It create Ta subcommand Ta Must also have the 'mount' ability .It destroy Ta subcommand Ta Must also have the 'mount' ability .It diff Ta subcommand Ta Allows lookup of paths within a dataset given an object number, and the ability to create snapshots necessary to 'zfs diff' .It hold Ta subcommand Ta Allows adding a user hold to a snapshot .It mount Ta subcommand Ta Allows mount/umount of Tn ZFS No datasets .It promote Ta subcommand Ta Must Xo also have the 'mount' and 'promote' ability in the origin file system .Xc .It receive Ta subcommand Ta Must also have the 'mount' and 'create' ability .It release Ta subcommand Ta Allows Xo releasing a user hold which might destroy the snapshot .Xc .It rename Ta subcommand Ta Must Xo also have the 'mount' and 'create' ability in the new parent .Xc .It rollback Ta subcommand Ta Must also have the 'mount' ability .It send Ta subcommand .It share Ta subcommand Ta Allows Xo sharing file systems over the .Tn NFS protocol .Xc .It snapshot Ta subcommand Ta Must also have the 'mount' ability .It groupquota Ta other Ta Allows accessing any groupquota@... property .It groupused Ta other Ta Allows reading any groupused@... property .It userprop Ta other Ta Allows changing any user property .It userquota Ta other Ta Allows accessing any userquota@... property .It userused Ta other Ta Allows reading any userused@... property .It aclinherit Ta property .It aclmode Ta property .It atime Ta property .It canmount Ta property .It casesensitivity Ta property .It checksum Ta property .It compression Ta property .It copies Ta property .It dedup Ta property .It devices Ta property .It exec Ta property .It filesystem_limit Ta property .It logbias Ta property .It jailed Ta property .It mlslabel Ta property .It mountpoint Ta property .It nbmand Ta property .It normalization Ta property .It primarycache Ta property .It quota Ta property .It readonly Ta property .It recordsize Ta property .It refquota Ta property .It refreservation Ta property .It reservation Ta property .It secondarycache Ta property .It setuid Ta property .It sharenfs Ta property .It sharesmb Ta property .It snapdir Ta property .It snapshot_limit Ta property .It sync Ta property .It utf8only Ta property .It version Ta property .It volblocksize Ta property .It volsize Ta property .It vscan Ta property .It xattr Ta property .El .It Xo .Nm .Cm allow .Fl c .Ar perm Ns | Ns Ar @setname Ns Op Ns , Ns Ar perm Ns | Ns Ar @setname Ns .Ns ... .Ar filesystem Ns | Ns Ar volume .Xc .Pp Sets "create time" permissions. These permissions are granted (locally) to the creator of any newly-created descendent file system. .It Xo .Nm .Cm allow .Fl s .Ar @setname .Ar perm Ns | Ns Ar @setname Ns Op Ns , Ns Ar perm Ns | Ns Ar @setname Ns .Ns ... .Ar filesystem Ns | Ns Ar volume .Xc .Pp Defines or adds permissions to a permission set. The set can be used by other .Qq Nm Cm allow commands for the specified file system and its descendents. Sets are evaluated dynamically, so changes to a set are immediately reflected. Permission sets follow the same naming restrictions as ZFS file systems, but the name must begin with an "at sign" .Pq Sy @ , and can be no more than 64 characters long. .It Xo .Nm .Cm unallow .Op Fl rldug .Ar user Ns | Ns Ar group Ns Oo Ns , Ns Ar user Ns | Ns Ar group Oc Ns ... .Oo Ar perm Ns | Ns Ar @setname Ns Op , Ns Ar perm Ns | Ns Ar @setname Ns .Ns ... Oc .Ar filesystem Ns | Ns Ar volume .Xc .It Xo .Nm .Cm unallow .Op Fl rld .Fl e Ns | Ns Cm everyone .Oo Ar perm Ns | Ns Ar @setname Ns Op , Ns Ar perm Ns | Ns Ar @setname Ns .Ns ... Oc .Ar filesystem Ns | Ns Ar volume .Xc .It Xo .Nm .Cm unallow .Op Fl r .Fl c .Oo Ar perm Ns | Ns Ar @setname Ns Op , Ns Ar perm Ns | Ns Ar @setname Ns .Ns ... Oc .Ar filesystem Ns | Ns Ar volume .Xc .Pp Removes permissions that were granted with the .Qq Nm Cm allow command. No permissions are explicitly denied, so other permissions granted are still in effect. For example, if the permission is granted by an ancestor. If no permissions are specified, then all permissions for the specified .Ar user , group , No or everyone are removed. Specifying .Cm everyone .Po or using the Fl e option .Pc only removes the permissions that were granted to everyone , not all permissions for every user and group. See the .Qq Nm Cm allow command for a description of the .Fl ldugec options. .Bl -tag -width indent .It Fl r Recursively remove the permissions from this file system and all descendents. .El .It Xo .Nm .Cm unallow .Op Fl r .Fl s .Ar @setname .Oo Ar perm Ns | Ns Ar @setname Ns Op , Ns Ar perm Ns | Ns Ar @setname Ns .Ns ... Oc .Ar filesystem Ns | Ns Ar volume .Xc .Pp Removes permissions from a permission set. If no permissions are specified, then all permissions are removed, thus removing the set entirely. .It Xo .Nm .Cm hold .Op Fl r .Ar tag snapshot Ns ... .Xc .Pp Adds a single reference, named with the .Ar tag argument, to the specified snapshot or snapshots. Each snapshot has its own tag namespace, and tags must be unique within that space. .Pp If a hold exists on a snapshot, attempts to destroy that snapshot by using the .Qq Nm Cm destroy command returns .Em EBUSY . .Bl -tag -width indent .It Fl r Specifies that a hold with the given tag is applied recursively to the snapshots of all descendent file systems. .El .It Xo .Nm .Cm holds .Op Fl r .Ar snapshot Ns ... .Xc .Pp Lists all existing user references for the given snapshot or snapshots. .Bl -tag -width indent .It Fl r Lists the holds that are set on the named descendent snapshots, in addition to listing the holds on the named snapshot. .El .It Xo .Nm .Cm release .Op Fl r .Ar tag snapshot Ns ... .Xc .Pp Removes a single reference, named with the .Ar tag argument, from the specified snapshot or snapshots. The tag must already exist for each snapshot. .Bl -tag -width indent .It Fl r Recursively releases a hold with the given tag on the snapshots of all descendent file systems. .El .It Xo .Nm .Cm diff .Op Fl FHt .Ar snapshot .Op Ar snapshot Ns | Ns Ar filesystem .Xc .Pp Display the difference between a snapshot of a given filesystem and another snapshot of that filesystem from a later time or the current contents of the filesystem. The first column is a character indicating the type of change, the other columns indicate pathname, new pathname .Pq in case of rename , change in link count, and optionally file type and/or change time. .Pp The types of change are: .Bl -column -offset 2n indent .It \&- Ta path was removed .It \&+ Ta path was added .It \&M Ta path was modified .It \&R Ta path was renamed .El .Bl -tag -width indent .It Fl F Display an indication of the type of file, in a manner similar to the .Fl F option of .Xr ls 1 . .Bl -column -offset 2n indent .It \&B Ta block device .It \&C Ta character device .It \&F Ta regular file .It \&/ Ta directory .It \&@ Ta symbolic link .It \&= Ta socket .It \&> Ta door (not supported on Fx ) .It \&| Ta named pipe (not supported on Fx ) .It \&P Ta event port (not supported on Fx ) .El .It Fl H Give more parsable tab-separated output, without header lines and without arrows. .It Fl t Display the path's inode change time as the first column of output. .El .It Xo .Nm .Cm jail .Ar jailid filesystem .Xc .Pp Attaches the specified .Ar filesystem to the jail identified by JID .Ar jailid . From now on this file system tree can be managed from within a jail if the .Sy jailed property has been set. To use this functionality, the jail needs the .Va allow.mount and .Va allow.mount.zfs parameters set to 1 and the .Va enforce_statfs parameter set to a value lower than 2. .Pp See .Xr jail 8 for more information on managing jails and configuring the parameters above. .It Xo .Nm .Cm unjail .Ar jailid filesystem .Xc .Pp Detaches the specified .Ar filesystem from the jail identified by JID .Ar jailid . .El .Sh EXIT STATUS The following exit values are returned: .Bl -tag -offset 2n -width 2n .It 0 Successful completion. .It 1 An error occurred. .It 2 Invalid command line options were specified. .El .Sh EXAMPLES .Bl -tag -width 0n .It Sy Example 1 No Creating a Tn ZFS No File System Hierarchy .Pp The following commands create a file system named .Em pool/home and a file system named .Em pool/home/bob . The mount point .Pa /home is set for the parent file system, and is automatically inherited by the child file system. .Bd -literal -offset 2n .Li # Ic zfs create pool/home .Li # Ic zfs set mountpoint=/home pool/home .Li # Ic zfs create pool/home/bob .Ed .It Sy Example 2 No Creating a Tn ZFS No Snapshot .Pp The following command creates a snapshot named .Sy yesterday . This snapshot is mounted on demand in the .Pa \&.zfs/snapshot directory at the root of the .Em pool/home/bob file system. .Bd -literal -offset 2n .Li # Ic zfs snapshot pool/home/bob@yesterday .Ed .It Sy Example 3 No Creating and Destroying Multiple Snapshots .Pp The following command creates snapshots named .Em yesterday of .Em pool/home and all of its descendent file systems. Each snapshot is mounted on demand in the .Pa \&.zfs/snapshot directory at the root of its file system. The second command destroys the newly created snapshots. .Bd -literal -offset 2n .Li # Ic zfs snapshot -r pool/home@yesterday .Li # Ic zfs destroy -r pool/home@yesterday .Ed .It Sy Example 4 No Disabling and Enabling File System Compression .Pp The following command disables the .Sy compression property for all file systems under .Em pool/home . The next command explicitly enables .Sy compression for .Em pool/home/anne . .Bd -literal -offset 2n .Li # Ic zfs set compression=off pool/home .Li # Ic zfs set compression=on pool/home/anne .Ed .It Sy Example 5 No Listing Tn ZFS No Datasets .Pp The following command lists all active file systems and volumes in the system. Snapshots are displayed if the .Sy listsnaps property is .Cm on . The default is .Cm off . See .Xr zpool 8 for more information on pool properties. .Bd -literal -offset 2n .Li # Ic zfs list NAME USED AVAIL REFER MOUNTPOINT pool 450K 457G 18K /pool pool/home 315K 457G 21K /home pool/home/anne 18K 457G 18K /home/anne pool/home/bob 276K 457G 276K /home/bob .Ed .It Sy Example 6 No Setting a Quota on a Tn ZFS No File System .Pp The following command sets a quota of 50 Gbytes for .Em pool/home/bob . .Bd -literal -offset 2n .Li # Ic zfs set quota=50G pool/home/bob .Ed .It Sy Example 7 No Listing Tn ZFS No Properties .Pp The following command lists all properties for .Em pool/home/bob . .Bd -literal -offset 2n .Li # Ic zfs get all pool/home/bob NAME PROPERTY VALUE SOURCE pool/home/bob type filesystem - pool/home/bob creation Tue Jul 21 15:53 2009 - pool/home/bob used 21K - pool/home/bob available 20.0G - pool/home/bob referenced 21K - pool/home/bob compressratio 1.00x - pool/home/bob mounted yes - pool/home/bob quota 20G local pool/home/bob reservation none default pool/home/bob recordsize 128K default pool/home/bob mountpoint /home/bob default pool/home/bob sharenfs off default pool/home/bob checksum on default pool/home/bob compression on local pool/home/bob atime on default pool/home/bob devices on default pool/home/bob exec on default pool/home/bob filesystem_limit none default pool/home/bob setuid on default pool/home/bob readonly off default pool/home/bob jailed off default pool/home/bob snapdir hidden default pool/home/bob snapshot_limit none default pool/home/bob aclmode discard default pool/home/bob aclinherit restricted default pool/home/bob canmount on default pool/home/bob xattr on default pool/home/bob copies 1 default pool/home/bob version 5 - pool/home/bob utf8only off - pool/home/bob normalization none - pool/home/bob casesensitivity sensitive - pool/home/bob vscan off default pool/home/bob nbmand off default pool/home/bob sharesmb off default pool/home/bob refquota none default pool/home/bob refreservation none default pool/home/bob primarycache all default pool/home/bob secondarycache all default pool/home/bob usedbysnapshots 0 - pool/home/bob usedbydataset 21K - pool/home/bob usedbychildren 0 - pool/home/bob usedbyrefreservation 0 - pool/home/bob logbias latency default pool/home/bob dedup off default pool/home/bob mlslabel - pool/home/bob sync standard default pool/home/bob refcompressratio 1.00x - .Ed .Pp The following command gets a single property value. .Bd -literal -offset 2n .Li # Ic zfs get -H -o value compression pool/home/bob on .Ed .Pp The following command lists all properties with local settings for .Em pool/home/bob . .Bd -literal -offset 2n .Li # Ic zfs get -s local -o name,property,value all pool/home/bob NAME PROPERTY VALUE pool/home/bob quota 20G pool/home/bob compression on .Ed .It Sy Example 8 No Rolling Back a Tn ZFS No File System .Pp The following command reverts the contents of .Em pool/home/anne to the snapshot named .Em yesterday , deleting all intermediate snapshots. .Bd -literal -offset 2n .Li # Ic zfs rollback -r pool/home/anne@yesterday .Ed .It Sy Example 9 No Creating a Tn ZFS No Clone .Pp The following command creates a writable file system whose initial contents are the same as .Em pool/home/bob@yesterday . .Bd -literal -offset 2n .Li # Ic zfs clone pool/home/bob@yesterday pool/clone .Ed .It Sy Example 10 No Promoting a Tn ZFS No Clone .Pp The following commands illustrate how to test out changes to a file system, and then replace the original file system with the changed one, using clones, clone promotion, and renaming: .Bd -literal -offset 2n .Li # Ic zfs create pool/project/production .Ed .Pp Populate .Pa /pool/project/production with data and continue with the following commands: .Bd -literal -offset 2n .Li # Ic zfs snapshot pool/project/production@today .Li # Ic zfs clone pool/project/production@today pool/project/beta .Ed .Pp Now make changes to .Pa /pool/project/beta and continue with the following commands: .Bd -literal -offset 2n .Li # Ic zfs promote pool/project/beta .Li # Ic zfs rename pool/project/production pool/project/legacy .Li # Ic zfs rename pool/project/beta pool/project/production .Ed .Pp Once the legacy version is no longer needed, it can be destroyed. .Bd -literal -offset 2n .Li # Ic zfs destroy pool/project/legacy .Ed .It Sy Example 11 No Inheriting Tn ZFS No Properties .Pp The following command causes .Em pool/home/bob and .Em pool/home/anne to inherit the .Sy checksum property from their parent. .Bd -literal -offset 2n .Li # Ic zfs inherit checksum pool/home/bob pool/home/anne .Ed .It Sy Example 12 No Remotely Replicating Tn ZFS No Data .Pp The following commands send a full stream and then an incremental stream to a remote machine, restoring them into .Sy poolB/received/fs@a and .Sy poolB/received/fs@b , respectively. .Sy poolB must contain the file system .Sy poolB/received , and must not initially contain .Sy poolB/received/fs . .Bd -literal -offset 2n .Li # Ic zfs send pool/fs@a | ssh host zfs receive poolB/received/fs@a .Li # Ic zfs send -i a pool/fs@b | ssh host zfs receive poolB/received/fs .Ed .It Xo .Sy Example 13 Using the .Qq zfs receive -d Option .Xc .Pp The following command sends a full stream of .Sy poolA/fsA/fsB@snap to a remote machine, receiving it into .Sy poolB/received/fsA/fsB@snap . The .Sy fsA/fsB@snap portion of the received snapshot's name is determined from the name of the sent snapshot. .Sy poolB must contain the file system .Sy poolB/received . If .Sy poolB/received/fsA does not exist, it is created as an empty file system. .Bd -literal -offset 2n .Li # Ic zfs send poolA/fsA/fsB@snap | ssh host zfs receive -d poolB/received .Ed .It Sy Example 14 No Setting User Properties .Pp The following example sets the user-defined .Sy com.example:department property for a dataset. .Bd -literal -offset 2n .Li # Ic zfs set com.example:department=12345 tank/accounting .Ed .It Sy Example 15 No Performing a Rolling Snapshot .Pp The following example shows how to maintain a history of snapshots with a consistent naming scheme. To keep a week's worth of snapshots, the user destroys the oldest snapshot, renames the remaining snapshots, and then creates a new snapshot, as follows: .Bd -literal -offset 2n .Li # Ic zfs destroy -r pool/users@7daysago .Li # Ic zfs rename -r pool/users@6daysago @7daysago .Li # Ic zfs rename -r pool/users@5daysago @6daysago .Li # Ic zfs rename -r pool/users@4daysago @5daysago .Li # Ic zfs rename -r pool/users@3daysago @4daysago .Li # Ic zfs rename -r pool/users@2daysago @3daysago .Li # Ic zfs rename -r pool/users@yesterday @2daysago .Li # Ic zfs rename -r pool/users@today @yesterday .Li # Ic zfs snapshot -r pool/users@today .Ed .It Xo .Sy Example 16 Setting .Qq sharenfs Property Options on a ZFS File System .Xc .Pp The following command shows how to set .Sy sharenfs property options to enable root access for a specific network on the .Em tank/home file system. The contents of the .Sy sharenfs property are valid .Xr exports 5 options. .Bd -literal -offset 2n .Li # Ic zfs set sharenfs="maproot=root,network 192.168.0.0/24" tank/home .Ed .Pp Another way to write this command with the same result is: .Bd -literal -offset 2n .Li # Ic set zfs sharenfs="-maproot=root -network 192.168.0.0/24" tank/home .Ed .It Xo .Sy Example 17 Delegating .Tn ZFS Administration Permissions on a .Tn ZFS Dataset .Xc .Pp The following example shows how to set permissions so that user .Em cindys can create, destroy, mount, and take snapshots on .Em tank/cindys . The permissions on .Em tank/cindys are also displayed. .Bd -literal -offset 2n .Li # Ic zfs allow cindys create,destroy,mount,snapshot tank/cindys .Li # Ic zfs allow tank/cindys -------------------------------------------------------------- -Local+Descendent permissions on (tank/cindys) - user cindys create,destroy,mount,snapshot -------------------------------------------------------------- +---- Permissions on tank/cindys -------------------------------------- +Local+Descendent permissions: + user cindys create,destroy,mount,snapshot .Ed .It Sy Example 18 No Delegating Create Time Permissions on a Tn ZFS No Dataset .Pp The following example shows how to grant anyone in the group .Em staff to create file systems in .Em tank/users . This syntax also allows staff members to destroy their own file systems, but not destroy anyone else's file system. The permissions on .Em tank/users are also displayed. .Bd -literal -offset 2n .Li # Ic zfs allow staff create,mount tank/users .Li # Ic zfs allow -c destroy tank/users .Li # Ic zfs allow tank/users -------------------------------------------------------------- -Create time permissions on (tank/users) - create,destroy -Local+Descendent permissions on (tank/users) - group staff create,mount -------------------------------------------------------------- +---- Permissions on tank/users --------------------------------------- +Permission sets: + destroy +Local+Descendent permissions: + group staff create,mount .Ed .It Xo .Sy Example 19 Defining and Granting a Permission Set on a .Tn ZFS Dataset .Xc .Pp The following example shows how to define and grant a permission set on the .Em tank/users file system. The permissions on .Em tank/users are also displayed. .Bd -literal -offset 2n .Li # Ic zfs allow -s @pset create,destroy,snapshot,mount tank/users .Li # Ic zfs allow staff @pset tank/users .Li # Ic zfs allow tank/users -------------------------------------------------------------- -Permission sets on (tank/users) +---- Permissions on tank/users --------------------------------------- +Permission sets: @pset create,destroy,mount,snapshot -Create time permissions on (tank/users) - create,destroy -Local+Descendent permissions on (tank/users) - group staff @pset,create,mount -------------------------------------------------------------- +Local+Descendent permissions: + group staff @pset .Ed .It Sy Example 20 No Delegating Property Permissions on a Tn ZFS No Dataset .Pp The following example shows to grant the ability to set quotas and reservations on the .Sy users/home file system. The permissions on .Sy users/home are also displayed. .Bd -literal -offset 2n .Li # Ic zfs allow cindys quota,reservation users/home -.Li # Ic zfs allow cindys -------------------------------------------------------------- -Local+Descendent permissions on (users/home) +.Li # Ic zfs allow users/home +---- Permissions on users/home --------------------------------------- +Local+Descendent permissions: user cindys quota,reservation -------------------------------------------------------------- .Li # Ic su - cindys .Li cindys% Ic zfs set quota=10G users/home/marks .Li cindys% Ic zfs get quota users/home/marks -NAME PROPERTY VALUE SOURCE -users/home/marks quota 10G local +NAME PROPERTY VALUE SOURCE +users/home/marks quota 10G local .Ed .It Sy Example 21 No Removing ZFS Delegated Permissions on a Tn ZFS No Dataset .Pp The following example shows how to remove the snapshot permission from the .Em staff group on the .Em tank/users file system. The permissions on .Em tank/users are also displayed. .Bd -literal -offset 2n .Li # Ic zfs unallow staff snapshot tank/users .Li # Ic zfs allow tank/users -------------------------------------------------------------- -Permission sets on (tank/users) +---- Permissions on tank/users --------------------------------------- +Permission sets: @pset create,destroy,mount,snapshot -Create time permissions on (tank/users) - create,destroy -Local+Descendent permissions on (tank/users) - group staff @pset,create,mount -------------------------------------------------------------- +Local+Descendent permissions: + group staff @pset .Ed .It Sy Example 22 Showing the differences between a snapshot and a ZFS Dataset .Pp The following example shows how to see what has changed between a prior snapshot of a ZFS Dataset and its current state. The .Fl F option is used to indicate type information for the files affected. .Bd -literal -offset 2n .Li # Ic zfs diff tank/test@before tank/test M / /tank/test/ M F /tank/test/linked (+1) R F /tank/test/oldname -> /tank/test/newname - F /tank/test/deleted + F /tank/test/created M F /tank/test/modified .Ed .El .Sh SEE ALSO .Xr chmod 2 , .Xr fsync 2 , .Xr exports 5 , .Xr fstab 5 , .Xr rc.conf 5 , .Xr jail 8 , .Xr mount 8 , .Xr umount 8 , .Xr zpool 8 .Sh AUTHORS This manual page is a .Xr mdoc 7 reimplementation of the .Tn OpenSolaris manual page .Em zfs(1M) , modified and customized for .Fx and licensed under the Common Development and Distribution License .Pq Tn CDDL . .Pp The .Xr mdoc 7 implementation of this manual page was initially written by .An Martin Matuska Aq mm@FreeBSD.org . Index: projects/clang350-import/cddl/contrib/opensolaris/cmd/zfs =================================================================== --- projects/clang350-import/cddl/contrib/opensolaris/cmd/zfs (revision 275748) +++ projects/clang350-import/cddl/contrib/opensolaris/cmd/zfs (revision 275749) Property changes on: projects/clang350-import/cddl/contrib/opensolaris/cmd/zfs ___________________________________________________________________ Modified: svn:mergeinfo ## -0,0 +0,1 ## Merged /head/cddl/contrib/opensolaris/cmd/zfs:r275623-275748 Index: projects/clang350-import/cddl/contrib/opensolaris =================================================================== --- projects/clang350-import/cddl/contrib/opensolaris (revision 275748) +++ projects/clang350-import/cddl/contrib/opensolaris (revision 275749) Property changes on: projects/clang350-import/cddl/contrib/opensolaris ___________________________________________________________________ Modified: svn:mergeinfo ## -0,0 +0,2 ## Merged /vendor/illumos/dist:r275547 Merged /head/cddl/contrib/opensolaris:r275623-275748 Index: projects/clang350-import/cddl =================================================================== --- projects/clang350-import/cddl (revision 275748) +++ projects/clang350-import/cddl (revision 275749) Property changes on: projects/clang350-import/cddl ___________________________________________________________________ Modified: svn:mergeinfo ## -0,0 +0,1 ## Merged /head/cddl:r275623-275748 Index: projects/clang350-import/contrib/binutils/bfd/ChangeLog =================================================================== --- projects/clang350-import/contrib/binutils/bfd/ChangeLog (revision 275748) +++ projects/clang350-import/contrib/binutils/bfd/ChangeLog (revision 275749) @@ -1,1792 +1,1838 @@ +2009-02-15 Alan Modra + + * elf64-ppc.c (struct _ppc64_elf_section_data): Delete t_symndx, + add toc.symndx and toc.add. + (ppc64_elf_check_relocs): Don't set htab->tls_get_addr here. + Set up toc.add. + (get_tls_mask): Add toc_addend param, set from toc.add. Adjust all + callers. + (ppc64_elf_tls_setup): Set htab->tls_get_addr and tls_get_addr_fd. + (branch_reloc_hash_match): New function, extracted from.. + (ppc64_elf_tls_optimize): ..here. + (ppc64_elf_relocate_section): Properly set addends when optimizing + tls sequences. Avoid unnecessary reading and writing of insns. + Only redo reloc when symbol changed. Bypass symbol checks when + using tlsld_got. + * elf32-ppc.c (ppc_elf_tls_setup): Correct comment. + (branch_reloc_hash_match): New function, extracted from.. + (ppc_elf_tls_optimize): ..here. + (ppc_elf_relocate_section): Avoid unnecessary reading of insns. + Don't clear addend on zapped __tls_get_addr reloc. + +2008-08-11 Alan Modra + + * elf64-ppc.c (toc_adjusting_stub_needed): Any call via the plt + needs r2 valid, not just those to external syms. + +2007-11-06 Alan Modra + + * elf32-ppc.c (ppc_elf_check_relocs): Don't refcount tlsld_got here.. + (ppc_elf_gc_sweep_hook): ..or here.. + (ppc_elf_tls_optimize): ..or here. Make two passes through the + relocs, ensuring that tls_get_addr calls follow gd and ld relocs. + (allocate_dynrelocs): Refcount tlsld_got here. + (ppc_elf_size_dynamic_sections): Call allocate_dynrelocs before + allocating tlsld_got. + (ppc_elf_relocate_section): Remove check that a tls_get_addr + call follows gd and ld relocs. + +2007-08-13 Alan Modra + + * elf64-ppc.c (ADDI_R12_R12, LD_R11_0R2, LD_R2_0R2): Define. + Update stub comments. + (build_plt_stub): Build two variants, one without "addis". + (ppc_build_one_stub): Build stubs without "addis" if possible. + (ppc_size_one_stub): Size new stubs. + 2007-07-02 Joseph Myers * elfxx-mips.c (mips_elf_calculate_relocation): Handle R_MIPS_TLS_DTPREL32 and R_MIPS_TLS_DTPREL64. * elf64-mips.c (mips_elf64_howto_table_rela): Support R_MIPS_TLS_DTPREL64. 2007-07-02 Alan Modra * Makefile.am: Run "make dep-am". * Makefile.in: Regenerate. * aclocal.m4: Regenerate. * config.in: Regenerate. * po/SRC-POTFILES.in: Regenerate. * po/bfd.pot: Regenerate. 2007-07-02 Alan Modra * elf-eh-frame.c (_bfd_elf_discard_section_eh_frame): Tidy symsec != NULL tests. 2007-07-02 Alan Modra PR 4712 * elf.c (assign_file_positions_for_load_sections): Adjust lma by p_vaddr_offset. 2007-06-30 H.J. Lu * aclocal.m4: Regenerated. * Makefile.in: Likewise. 2007-06-29 Joseph Myers * elf32-ppc.c (ppc_elf_merge_obj_attributes): New. (ppc_elf_merge_private_bfd_data): Call it. 2007-06-29 Joseph Myers * elfxx-mips.c (mips_elf_merge_obj_attributes): New. (_bfd_mips_elf_merge_private_bfd_data): Call it. 2007-06-29 Joseph Myers * elf-attrs.c: New. * Makefile.am (BFD32_BACKENDS): Add elf-attrs.lo. (BFD32_BACKENDS_CFILES): Add elf-attrs.c. (elf-attrs.lo): Generate dependencies. * Makefile.in: Regenerate. * configure.in (elf): Add elf-attrs.lo. * configure: Regenerate. * elf-bfd.h (struct elf_backend_data): Add entries for object attributes. (NUM_KNOWN_OBJ_ATTRIBUTES, obj_attribute, obj_attribute_list, OBJ_ATTR_PROC, OBJ_ATTR_GNU, OBJ_ATTR_FIRST, OBJ_ATTR_LAST, Tag_NULL, Tag_File, Tag_Section, Tag_Symbol, Tag_compatibility): New. (struct elf_obj_tdata): Add entries for object attributes. (elf_known_obj_attributes, elf_other_obj_attributes, elf_known_obj_attributes_proc, elf_other_obj_attributes_proc): New. (bfd_elf_obj_attr_size, bfd_elf_set_obj_attr_contents, bfd_elf_get_obj_attr_int, bfd_elf_add_obj_attr_int, bfd_elf_add_proc_attr_int, bfd_elf_add_obj_attr_string, bfd_elf_add_proc_attr_string, bfd_elf_add_obj_attr_compat, bfd_elf_add_proc_attr_compat, _bfd_elf_attr_strdup, _bfd_elf_copy_obj_attributes, _bfd_elf_obj_attrs_arg_type, _bfd_elf_parse_attributes, _bfd_elf_merge_object_attributes): New. * elf.c (_bfd_elf_copy_private_bfd_data): Copy object attributes. (bfd_section_from_shdr): Handle attributes sections. * elflink.c (bfd_elf_final_link): Handle attributes sections. * elfxx-target.h (elf_backend_obj_attrs_vendor, elf_backend_obj_attrs_section, elf_backend_obj_attrs_arg_type, elf_backend_obj_attrs_section_type): New. (elfNN_bed): Update. * elf32-arm.c (NUM_KNOWN_ATTRIBUTES, aeabi_attribute, aeabi_attribute_list): Remove. (struct elf32_arm_obj_tdata): Remove object attributes fields. (check_use_blx, bfd_elf32_arm_set_vfp11_fix, using_thumb2, elf32_arm_copy_private_bfd_data, elf32_arm_merge_eabi_attributes): Update for new object attributes interfaces. (uleb128_size, is_default_attr, eabi_attr_size, elf32_arm_eabi_attr_size, write_uleb128, write_eabi_attribute, elf32_arm_set_eabi_attr_contents, elf32_arm_bfd_final_link, elf32_arm_new_eabi_attr, elf32_arm_get_eabi_attr_int, elf32_arm_add_eabi_attr_int, attr_strdup, elf32_arm_add_eabi_attr_string, elf32_arm_add_eabi_attr_compat, copy_eabi_attributes, elf32_arm_parse_attributes): Remove. Moved to generic code in elf-attrs.c. (elf32_arm_obj_attrs_arg_type): New. (elf32_arm_fake_sections): Do not handle .ARM.attributes. (elf32_arm_section_from_shdr): Do not handle SHT_ARM_ATTRIBUTES. (bfd_elf32_bfd_final_link): Remove. (elf_backend_obj_attrs_vendor, elf_backend_obj_attrs_section, elf_backend_obj_attrs_arg_type, elf_backend_obj_attrs_section_type): New. * elf32-bfin.c (bfin_elf_copy_private_bfd_data): Copy object attributes. * elf32-frv.c (frv_elf_copy_private_bfd_data): Likewise. * elf32-iq2000.c (iq2000_elf_copy_private_bfd_data): Likewise. * elf32-mep.c (mep_elf_copy_private_bfd_data): Likewise. * elf32-mt.c (mt_elf_copy_private_bfd_data): Likewise. * elf32-sh.c (sh_elf_copy_private_data): Likewise. * elf64-sh64.c (sh_elf64_copy_private_data_internal): Likewise. 2007-06-29 Paul Brook * elf32-arm.c (bfd_elf32_arm_process_before_allocation): Suppress call veneers for call relocations against undefined symbols. (elf32_arm_final_link_relocate): Turn call to undefined symbol into a jump to the next instruction. 2007-06-29 Michael Snyder * bfd.c (bfd_demangle): Plug memory leak (Coverity). 2007-06-29 M R Swami Reddy * Makefile.am: Add cr16 related entry * Makefile.in: Regenerate * archures.c: Add bfd_cr16_arch * bfd-in2.h: Regenerate * config.bfd: Add cr16-elf * configure.in: Add bfd_elf32_cr16_vec * configure: Regenerate. * targets.c: Added cr16 related information * cpu-cr16.c: New file. * elf32-cr16.c: New file. * reloc.c: Added cr16 relocs. 2007-06-29 Alan Modra * elflink.c (_bfd_elf_link_assign_sym_version): Improve error message for undefined version nodes. (elf_gc_sweep): Don't warn when zero size sections are removed. 2007-06-29 Nathan Froyd * elf32-ppc.c (ppc_elf_info_to_howto): Check for invalid relocation types. 2007-06-29 Alan Modra * elf32-spu.c (spu_elf_fake_sections): New function. (elf_backend_fake_sections): Define. 2007-06-29 Alan Modra * elf.c (assign_file_positions_for_load_sections): Use elf section header sh_size rather than bfd section size, simplifying .tbss handling. 2007-06-29 Alan Modra PR ld/4701 * elf.c (assign_file_positions_for_load_sections): Ensure bss segments meet gABI alignment requirements. Don't allocate file space for bss sections in a segment also containing file or program headers. 2007-06-27 Alan Modra * bfd.c (struct bfd): Rename "next" to "archive_next". * archive.c: Rename uses throughout file. * archive64.c: Likewise. * coff-rs6000.c: Likewise. * ecoff.c: Likewise. * som.c: Likewise. * bfd-in2.h: Regenerate. * elf32-ppc.c (ppc_elf_select_plt_layout): Properly iterate over input bfds. * elf32-spu.c (spu_elf_create_sections): Likewise. 2007-06-26 H.J. Lu * dwarf2.c (find_line): New. Contains the duplicated code from: (_bfd_dwarf2_find_nearest_line): Use it. (_bfd_dwarf2_find_line): Use it. 2007-06-26 Joseph Myers * elf32-arm.c (copy_eabi_attributes): Copy type of attributes. 2007-06-25 Richard Sandiford * elfxx-mips.c (mips_elf_calculate_relocation): Allow local stubs to be used for calls from MIPS16 code. 2007-06-23 Andreas Schwab * configure.in (--with-separate-debug-dir): New option. * configure: Regenerate. * Makefile.am (dwarf2.lo): Add rule to pass DEBUGDIR. * Makefile.in: Regenerate. * dwarf2.c (_bfd_dwarf2_find_nearest_line): Pass DEBUGDIR to bfd_follow_gnu_debuglink. (_bfd_dwarf2_find_line): Likewise. 2007-06-22 Nick Clifton * dwarf2.c: Add support for reading in debug information via a .gnu_debuglink section: (struct dwarf2_debug): Add bfd field to record the bfd containing the debug info. (parse_comp_unit): Remove ABFD parameter. Instead use the bfd field in the dwarf2_debug structure. (_bfd_dwarf2_find_nearest_line): If a debug info section could not be found in the current bfd call bfd_follow_gnu_debuglink to see if another file contains the debug information. If it does, open it and continue. (_bfd_dwarf2_find_line): Likewise. 2007-06-19 H.J. Lu PR ld/4590 * elfxx-ia64.c (sort_dyn_sym_info): Keep the valid got_offset when removing duplicated entries. (get_dyn_sym_info): Initialize the got_offset field to -1. Update call to sort_dyn_sym_info. (elfNN_ia64_relocate_section): Call sort_dyn_sym_info to sort array of addend and remove duplicates. 2007-06-18 H.J. Lu * libpei.h (_bfd_XXi_final_link_postscript): Remove duplication. (bfd_target_pei_p): New. (bfd_target_pei_arch): New (bfd_target_efi_p): Likewise. (bfd_target_efi_arch): New (bfd_pe_executable_p): Use bfd_target_pei_p and bfd_target_efi_p. * peicode.h (arch_type): New enum. (pe_arch): New function. (pe_bfd_object_p): Don't match PE/EFI target with EFI/PE file if there is an EFI/PE target. 2007-06-14 H.J. Lu * Makefile.am (ACLOCAL_AMFLAGS): Add -I . -I ../config. * acinclude.m4: Don't include m4 files. Remove libtool kludge. * Makefile.in: Regenerated. * aclocal.m4: Likewise. * configure: Likewise. 2007-06-11 Sterling Augustine Bob Wilson * elf32-xtensa.c (extend_ebb_bounds_forward): Use renamed XTENSA_PROP_NO_TRANSFORM flag instead of XTENSA_PROP_INSN_NO_TRANSFORM. (extend_ebb_bounds_backward, compute_text_actions): Likewise. (compute_ebb_proposed_actions, coalesce_shared_literal): Likewise. (xtensa_get_property_predef_flags): Likewise. (compute_removed_literals): Pass new arguments to is_removable_literal. (is_removable_literal): Add sec, prop_table and ptblsize arguments. Do not remove literal if the NO_TRANSFORM property flag is set. 2007-05-31 Richard Sandiford * elfxx-mips.c (mips_elf_initialize_tls_index): When processing a type (3) single-GOT entry, read tls_type from the hash table entry rather than the GOT entry. 2007-06-01 Alan Modra * simple.c (bfd_simple_get_relocated_section_contents): Init input_bfds_tail. 2007-05-30 Alan Modra * elf.c (elf_fake_sections): Adjust test for SHT_NOBITS sections created by objcopy --only-keep-debug. (_bfd_elf_init_private_section_data): Only change elf_section_type if it is SHT_NULL. * elf.c (assign_file_positions_for_load_sections): Correct sh_type to SHT_NOBITS earlier. Base actions in rest of function on sh_type and sh_flags instead of bfd section flags. Delete voff and code keeping nobits segments aligned. 2007-05-25 Eric Christopher * elf-eh-frame.c (_bfd_elf_discard_section_eh_frame): Check that sym_sec isn't NULL before accessing. 2007-05-24 Steve Ellcey * Makefile.in: Regnerate. * configure: Regenerate. * aclocal.m4: Regenerate. * doc/Makefile.in: Regenerate. 2007-05-22 Paul Brook * elf32-arm.c (output_arch_syminfo): Replace plt_shndx and plt_offset with sec and sec_shndx. (elf32_arm_ouput_plt_map_sym): Use them. (elf32_arm_output_arch_local_syms): Output mapping symbols for interworking glue. 2007-05-18 Paul Brook * elf32-arm.c (ARM2THUMB_V5_STATIC_GLUE_SIZE): Define. (a2t1v5_ldr_insn, a2t1v5_ldr_insn): New. (record_arm_to_thumb_glue): Add v5t non-pic glue. (elf32_arm_create_thumb_stub): Ditto. 2007-05-16 H.J. Lu Alan Modra * elflink.c (_bfd_elf_adjust_dynamic_copy): Align dynamic bss section to the minimum alignment. 2007-05-15 H.J. Lu Alan Modra PR ld/4504 * elf-bfd.h (_bfd_elf_adjust_dynamic_copy): New. * elflink.c (_bfd_elf_adjust_dynamic_copy): New. * elf-m10300.c (_bfd_mn10300_elf_adjust_dynamic_symbol): Call _bfd_elf_adjust_dynamic_copy to adjust for the copy in dynamic bss section. * elf32-arm.c (elf32_arm_adjust_dynamic_symbol): Likewise. * elf32-cris.c (elf_cris_adjust_dynamic_symbol): Likewise. * elf32-hppa.c (elf32_hppa_adjust_dynamic_symbol): Likewise. * elf32-i370.c (i370_elf_adjust_dynamic_symbol): Likewise. * elf32-i386.c (elf_i386_adjust_dynamic_symbol): Likewise. * elf32-m32r.c (m32r_elf_adjust_dynamic_symbol): Likewise. * elf32-m68k.c (elf_m68k_adjust_dynamic_symbol): Likewise. * elf32-ppc.c (ppc_elf_adjust_dynamic_symbol): Likewise. * elf32-s390.c (elf_s390_adjust_dynamic_symbol): Likewise. * elf32-sh.c (sh_elf_adjust_dynamic_symbol): Likewise. * elf32-vax.c (elf_vax_adjust_dynamic_symbol): Likewise. * elf64-ppc.c (ppc64_elf_adjust_dynamic_symbol): Likewise. * elf64-s390.c (elf_s390_adjust_dynamic_symbol): Likewise. * elf64-sh64.c (sh64_elf64_adjust_dynamic_symbol): Likewise. * elf64-x86-64.c (elf64_x86_64_adjust_dynamic_symbol): Likewise. * elfxx-mips.c (_bfd_mips_vxworks_adjust_dynamic_symbol): Likewise. * elfxx-sparc.c (_bfd_sparc_elf_adjust_dynamic_symbol): Likewise. 2007-05-15 Richard Sandiford * elfxx-mips.c (mips_elf_create_local_got_entry): Remove input_section argument. Create .rela.dyn relocations against symbol 0 rather than the section symbol. (mips_elf_local_got_index): Remove input_section argument. Update call to mips_elf_create_local_got_entry. (mips_elf_got_page, mips_elf_got16_entry): Likewise. (mips_elf_calculate_relocation): Update calls to mips_elf_local_got_index, mips_elf_got16_entry and mips_elf_got_page. 2007-05-15 Mark Shinwell * elf32-arm.c (elf32_arm_final_link_relocate): Correctly handle the Thumb-2 JUMP19 relocation. 2007-05-15 Alan Modra PR 4479 * elf.c (elf_fake_sections): Don't allow backend to change SHT_NOBITS if called for strip/objcopy --only-keep-debug. * elfxx-mips.c (_bfd_mips_elf_fake_sections): Remove similar fix from here. 2007-05-14 Alan Modra * elf.c (bfd_elf_string_from_elf_section): Return NULL on invalid shstrndx. (bfd_elf_get_str_section): Likewise. 2007-05-12 Alan Modra PR 4497 * elf-eh-frame.c (struct cie): Add "local_personality". Make "personality" a union. (cie_eq): Compare local_personality too. Adjust personality comparison. (_bfd_elf_discard_section_eh_frame): Check binding on personality reloc sym to allow for bad symtab. Use stashed local syms rather than reading personality local sym. Handle discarded sections. 2007-05-12 Alan Modra * elf32-spu.c (elf_howto_table): Add howto for R_SPU_ADDR16X. 2007-05-11 Alan Modra * elf32-ppc.h (ppc_elf_select_plt_layout): Update prototype. (enum ppc_elf_plt_type): Move from.. * elf32-ppc.c: ..here. (struct ppc_elf_obj_tdata): Add makes_plt_call and has_rel16. (struct ppc_elf_link_hash_table): Reorder. Add old_bfd. Delete can_use_new_plt. Make is_vxworks a bitfield. (ppc_elf_link_hash_table_create): Don't clear is_vxworks (again). (ppc_elf_check_relocs): Update setting of reloc flags. Set old_bfd. (ppc_elf_select_plt_layout): Modify parameters. Use bfd reloc flags to better detect object files needing old bss-style plt. Allow secure plt to be used without rel16 relocs being detected. Warn if secure plt request cannot be allowed. 2007-05-11 Alan Modra * reloc.c (BFD_RELOC_SPU_PPU32, BFD_RELOC_SPU_PPU64): Define. * elf-bfd.h (struct elf_backend_data): Change return type of elf_backend_relocate_section to int. * elf32-spu.c (elf_howto_table): Add howtos for R_SPU_PPU32 and R_SPU_PPU64. (spu_elf_bfd_to_reloc_type): Convert new relocs. (spu_elf_count_relocs): New function. (elf_backend_count_relocs): Define. (spu_elf_relocate_section): Arrange to emit R_SPU_PPU32 and R_SPU_PPU64 relocs. * elflink.c (elf_link_input_bfd): Emit relocs if relocate_section returns 2. * bfd-in2.h: Regenerate. * libbfd.h: Regenerate. 2007-05-10 Richard Sandiford * elf32-arm.c (elf32_arm_check_relocs): Don't create PLT entries for R_ARM_ABS12 relocs. (elf32_arm_finish_dynamic_symbol): Fix the loop that creates non-shared VxWorks PLT entries. 2007-05-11 Alan Modra PR 4454 * elf-eh-frame.c (struct cie): Make "personality" a bfd_vma. (_bfd_elf_discard_section_eh_frame): Handle local syms on personality relocation. 2007-05-10 Richard Sandiford * elf.c (assign_file_positions_for_load_sections): Use p_memsz rather than p_filesz to calculate the LMA of the end of a segment. 2007-05-10 Jakub Jelinek * elf64-ppc.c (ppc64_elf_adjust_dynamic_symbol): Don't do copyreloc processing if symbol is defined in the executable. 2007-05-10 Alexandre Oliva * elf32-frv.c (_frvfdpic_check_discarded_relocs): New. (frvfdpic_elf_discard_info): New. (elf_backend_discard_info): Define for FDPIC. 2007-05-09 Mark Shinwell * elf32-arm.c (bfd_elf32_arm_vfp11_erratum_scan): Don't attempt to scan if the bfd doesn't correspond to an ELF image. (bfd_elf32_arm_vfp11_fix_veneer_locations): Likewise. 2007-05-08 Alexandre Oliva * elf32-frv.c (elf32_frv_relocate_section): Discard dynamic relocs for which _bfd_elf_section_offset returns -1. 2007-05-08 Alan Modra * elf32-spu.c (spu_elf_size_stubs): Use "void *" for psyms. (mark_functions_via_relocs): Likewise. 2007-05-07 Alan Modra * elflink.c (bfd_elf_final_link): Correct reloc handling for elf_backend_count_relocs. (bfd_elf_discard_info): Print an error if we can't read syms. 2007-05-07 Alan Modra * elf32-spu.c (spu_elf_reloc_type_lookup): Return NULL on invalid reloc code. (spu_elf_gc_mark_hook, spu_elf_section_processing): Delete functions. (elf_backend_gc_mark_hook, elf_backend_section_processing): Don't define. 2007-05-07 Alan Modra * elf.c (assign_file_positions_for_load_sections): Don't check core segment. 2007-05-04 H.J. Lu * elflink.c (elf_link_sort_relocs): Return if both .rela.dyn and .rel.dyn aren't present. 2007-05-04 Nick Clifton * elflink.c (elf_link_sort_relocs): If both .rela.dyn and .rel.dyn sections are present examine the indirect sections in an attempt to compute the correct relocation size. If there is any ambiguity, produce an error and refuse to sort. 2007-05-03 Sandra Loosemore * elf32-arm.c (allocate_dynrelocs): Fix typo in comment. 2007-05-03 Vincent Riviere Nick Clifton PR gas/3041 * aoutx.h (swap_std_reloc_out): Treat relocs against weak symbols in the same way as relocs against external symbols. 2007-05-02 Alan Modra * elf.c (assign_file_positions_for_load_sections): Set sh_offset here. Set sh_type to SHT_NOBITS if we won't be allocating file space. Don't bump p_memsz for non-alloc sections. Adjust section-in-segment check. (assign_file_positions_for_non_load_sections): Don't set sh_offset here for sections that have already been handled above. 2007-04-30 Alan Modra * elf32-spu.c (struct spu_link_hash_table): Add stack_analysis and emit_stack_syms bitfields. (get_sym_h): Read all symbols if stack analysis will be done. (spu_elf_create_sections): Add stack_analysis and emit_stack_syms params, and stash in hash table. (is_hint): Split off from.. (is_branch): ..here. Adjust callers. (spu_elf_size_stubs): Add stack_analysis param. Arrange to read and keep all syms. (write_one_stub): Fix mem leak. (find_function_stack_adjust): New function. (sort_syms_syms, sort_syms_psecs): New vars. (sort_syms): New function. (struct call_info, struct function_info): New. (struct spu_elf_stack_info): New. (alloc_stack_info, maybe_insert_function, func_name): New functions. (is_nop, insns_at_end, check_function_ranges): Likewise. (find_function, insert_callee, mark_functions_via_relocs): Likewise. (pasted_function, interesting_section, discover_functions): Likewise. (mark_non_root, call_graph_traverse, build_call_tree): Likewise. (sum_stack, spu_elf_stack_analysis, spu_elf_final_link): Likewise. (bfd_elf32_bfd_final_link): Define. * elf32-spu.h (struct _spu_elf_section_data): Add stack_info field. (spu_elf_create_sections, spu_elf_size_stubs): Update prototypes. 2007-04-28 Sergey Rogozhkin * elfxx-mips.c (mips_elf_create_dynamic_relocation): Don't access memory which we might not own. 2007-04-27 Bob Wilson * elf32-xtensa.c (elf_xtensa_make_sym_local): Restore deleted function. (elf_xtensa_hide_symbol, elf_backend_hide_symbol): Likewise. (elf_xtensa_allocate_dynrelocs): Use elf_xtensa_make_sym_local. 2007-04-27 Bob Wilson * elf32-xtensa.c (xtensa_read_table_entries): Step through table contents and relocs in parallel. 2007-04-27 Bob Wilson * elf32-xtensa.c (relax_property_section): Remove extra irel increment. 2007-04-27 Alan Modra * cpu-rs6000.c: Write Mimi's name in ASCII. * coff-rs6000.c: Likewise. * rs6000-core.c: Likewise. 2007-04-27 Alan Modra * sysdep.h: Include config.h first. Many files: Include sysdep.h before bfd.h. * Makefile.am: Run "make dep-am". * Makefile.in: Regenerate. 2007-04-25 Alan Modra * sysdep.h: Revert last change. 2007-04-24 Nick Clifton * coffcode.h (coff_slurp_reloc_table): Initialise dst.r_offset. * coff-m68k.c (m68kcoff_rtype_to_howto): Initialize relent.howto. 2007-04-24 Alan Modra * elf-eh-frame.c (_bfd_elf_discard_section_eh_frame): Warn if eh_frame_hdr table won't be created. 2007-04-24 Alan Modra * acinclude.m4: Include config/stdint.m4. * configure.in: Invoke GCC_HEADER_STDINT. * sysdep.h: Don't include ansidecl.h here. * configure: Regenerate. * config.in: Regenerate. * Makefile.in: Regenerate. 2007-04-23 Nathan Sidwell * archures.c (bfd_mach_mcf_isa_c, bfd_mach_mcf_isa_c_mac, bfd_mach_mcf_isa_c_emac): New. * elf32-m68k.c (ISAC_PLT_ENTRY_SIZE, elf_isac_plt0_entry, elf_isac_plt_entry, elf_isac_plt_info): New. (elf32_m68k_object_p): Add ISA_C. (elf32_m68k_print_private_bfd_data): Print ISA_C. (elf32_m68k_get_plt_info): Detect ISA_C. * cpu-m68k.c (arch_info): Add ISAC. (m68k_arch_features): Likewise, (bfd_m68k_compatible): ISAs B & C are not compatible. 2007-04-21 Nick Clifton * ecoff.c (_bfd_ecoff_write_armap): Initialise rehash. (ecoff_link_add_archive_symbols): Likewise. * coff-m68k.c (m68kcoff_common_addend_rtype_to_howto): Initialise relent.howto. * ieee.c (parse_int): Initialise x. (must_parse_int): Initialise result. (ieee_slurp_external_symbols): Initialise value. 2007-04-21 Alan Modra * config.bfd (spu-*-elf): Delete targ_selvecs. 2007-04-19 Nick Clifton * coffcode.h (coff_rtype_to_howto): Initialise genrel.howto. 2007-04-19 Alan Modra * bfd.c (bfd_demangle): New function. * Makefile.am: Run "make dep-am". * Makefile.in: Regenerate. * bfd-in2.h: Regenerate. 2007-04-18 Matthias Klose * Makefile.am (libbfd_la_LDFLAGS): Use bfd soversion. (bfdver.h): Use the date in non-release builds for the soversion. * Makefile.in: Regenerate. 2007-04-17 Paul Brook * elf.c (_bfd_elf_is_function_type): New function. * elflink.c (_bfd_elf_merge_symbol): Use bed->is_function_type. (_bfd_elf_dynamic_symbol_p, _bfd_elf_symbol_refs_local_p, is_global_data_symbol_definition, elf_link_add_object_symbols): Ditto. * elf-bfd.h (elf_backend_data): Add is_function_type. (_bfd_elf_is_function_type): Add prototype. * elfxx-target.h (elf_backend_is_function_type): Add default definition. (elfNN_bed): Add elf_backend_is_function_type. * elf32-arm.c (elf32_arm_is_function_type): New function. (elf_backend_is_function_type): Define. 2007-04-17 Daniel Jacobowitz * elfxx-mips.c (_bfd_mips_elf_size_dynamic_sections): Put DT_MIPS_RLD_MAP before DT_DEBUG again. 2007-04-14 Steve Ellcey * Makefile.am: Add ACLOCAL_AMFLAGS. * configure.in: Change macro call order. * Makefile.in: Regnerate. * doc/Makefile.in: Regenerate. * configure: Regenerate. 2007-04-14 Jakub Jelinek * elflink.c (bfd_elf_final_link): Don't free symbuf for non-elf input bfds. (bfd_elf_size_dynamic_sections): Don't access elf_section_data for non-elf input bfds. 2007-04-12 Richard Sandiford * elfxx-mips.c (_bfd_mips_elf_size_dynamic_sections): Don't add DT_MIPS_RTLD_MAP for PIEs. 2007-04-12 Richard Sandiford * elfxx-mips.c (mips_elf_calculate_relocation): Set DT_TEXTREL when creating a __GOTT_BASE__ or __GOTT_INDEX__ relocation in a read-only section. (_bfd_mips_elf_check_relocs): Likewise. 2007-04-12 Bob Wilson * elf32-xtensa.c (xtensa_is_insntable_section): New. (xtensa_is_proptable_section): New. (elf_xtensa_discard_info_for_section): Handle "full" .xt.prop property tables with 12-byte entries, as well as tables with 8-byte entries. Sort the relocations before examining them. (relax_property_section): Use xtensa_is_proptable_section and xtensa_is_littable_section. Rewrite code for combining table entries to be more robust in case of unexpected relocations. Do not set offset of unused relocations to less than zero. (xtensa_is_property_section): Use other functions instead of duplicating section name comparisons. (xtensa_is_littable_section): Use CONST_STRNEQ for ".gnu.linkonce.p.". (xtensa_get_property_predef_flags): Use xtensa_is_insntable_section. 2007-04-12 Bob Wilson * elf32-xtensa.c (elf_xtensa_gc_mark_hook): Don't follow references from Xtensa property sections. 2007-04-12 Alan Modra * elf32-spu.c (needs_ovl_stub): Test that spu_elf_section_data is non-NULL before dereferencing. 2007-04-11 Nathan Sidwell * elf-vxworks.c (elf_vxworks_emit_relocs): Remap weakdef PLT slot relocs too. 2007-04-10 Richard Henderson * elf64-alpha.c (struct alpha_elf_link_hash_table): Add relax_trip. (elf64_alpha_size_got_sections): Remove unused something_changed local. (elf64_alpha_size_plt_section): Return void. (elf64_alpha_size_rela_got_section): Likewise. (elf64_alpha_relax_section): Only regenerate got+plt if the relax_trip counter has changed. 2007-04-09 Daniel Jacobowitz * Makefile.am (bfdver.h): Do not generate doc/bfdver.texi. * doc/Makefile.am (DOCFILES): Add bfdver.texi. (bfdver.texi): New rule. * Makefile.in, doc/Makefile.in: Regenerated. 2007-04-03 Matt Thomas * elf32-vax.c (elf_vax_relocate_section): Do not emit a PCREL reloc in a shared object if it is not in a CODE section or if it is against a section symbol. This allows DWARF2 to use pcrel format. 2007-04-05 H.J. Lu PR ld/4304 * elflink.c (bfd_elf_final_link): Call einfo callback in bfd_link_info instead of _bfd_error_handler for DT_TEXTREL warning. 2007-04-05 Alan Modra * elf32-spu.c (spu_elf_output_symbol_hook): New function. (elf_backend_link_output_symbol_hook): Define. 2007-04-02 Nick Clifton PR binutils/4292 * bfd.c (bfd_fprintf_vma): Do not print addresses of 32-bit targets as 64-bit values, even if running on a 64-bit host. * coffgen.c (coff_print_symbol): Likewise. 2007-03-29 Nick Clifton PR binutils/4110 * elf.c (IS_VALID_GROUP_SECTION_HEADER): New macro. (setup_group): Use it. Report corrupt group section headers. (bfd_section_from_shdr): Use new macro. Replace constant 4 with GRP_ENTRY_SIZE. Cope with NULLs in the group section table. (elf_fake_section): Replace constant 4 with GRP_ENTRY_SIZE. 2007-03-29 Alan Modra PR ld/4267 * elflink.c (evaluate_complex_relocation_symbols): Use bfd_vma for rel->r_info values. (bfd_elf_perform_complex_relocation): Likewise. * elf32-ppc.c (allocate_dynrelocs): Set plt.offset to -1 for unused entries. Don't clear plt.plist in loop. 2007-03-28 Richard Sandiford * elfxx-sparc.h (_bfd_sparc_elf_link_hash_table): Remove append_rela. * elfxx-sparc.c (sparc_elf_append_rela_64, sparc_elf_append_rela_32): Merge into... (sparc_elf_append_rela): ...this new function. (SPARC_ELF_APPEND_RELA): Delete. (_bfd_sparc_elf_link_hash_table_create): Don't initialize the deleted append_rela field. (_bfd_sparc_elf_relocate_section): Use sparc_elf_append_rela instead of SPARC_ELF_APPEND_RELA. (_bfd_sparc_elf_finish_dynamic_symbol): Likewise. Use the elf_size_info structure to find the size of a RELA entry and the associated swap-out function. (sparc64_finish_dyn, sparc64_finish_dyn): Merge into... (sparc_finish_dyn): ...this new function. (_bfd_sparc_elf_finish_dynamic_sections): Update calls accordingly. 2007-03-28 Richard Sandiford Phil Edwards * doc/bfd.texinfo: Put the contents after the title page rather than at the end of the document. 2007-03-27 Andreas Schwab * elfxx-ia64.c (elf_backend_default_execstack): Define to 0. 2007-03-26 H.J. Lu * configure: Regenerated. 2007-03-26 Alan Modra * elf32-spu.c (struct stubarr): Add stub_hash_table and err fields. (allocate_spuear_stubs): New function. (spu_elf_size_stubs): Call allocate_spuear_stubs. 2007-03-26 Alan Modra * aout-adobe.c (aout_32_bfd_reloc_name_lookup): Define. * aout-arm.c (MY_bfd_reloc_name_lookup): Define. (MY (bfd_reloc_name_lookup)): New function. * aout-ns32k.c (MY (bfd_reloc_name_lookup)): New function. * aout-target.h (NAME (aout, reloc_name_lookup)): Declare. (MY_bfd_reloc_name_lookup): Define. * aout-tic30.c (tic30_aout_reloc_name_lookup): New function. (MY_bfd_reloc_name_lookup): Define. * aoutx.h (NAME (aout, reloc_type_lookup)): Don't declare. (NAME (aout, reloc_name_lookup)): New function. * bout.c (b_out_bfd_reloc_name_lookup): New function. * coff-alpha.c (alpha_bfd_reloc_name_lookup): New function. (_bfd_ecoff_bfd_reloc_name_lookup): Define. * coff-arm.c (coff_arm_reloc_name_lookup): New function. (coff_bfd_reloc_name_lookup): Define. * coff-i386.c (coff_bfd_reloc_name_lookup): Define. (coff_i386_reloc_name_lookup): New function. * coff-i860.c (coff_i860_reloc_name_lookup): New function. (coff_bfd_reloc_name_lookup): Define. * coff-i960.c (coff_i960_reloc_name_lookup): New function. (coff_bfd_reloc_name_lookup): Define. * coff-m68k.c (m68k_reloc_name_lookup): New function. (coff_bfd_reloc_name_lookup): Define. * coff-maxq.c (maxq_reloc_name_lookup): New function. (coff_bfd_reloc_name_lookup): Define. * coff-mcore.c (mcore_coff_reloc_name_lookup): New function. (coff_bfd_reloc_name_lookup): Define. * coff-mips.c (mips_bfd_reloc_name_lookup): New function. (_bfd_ecoff_bfd_reloc_name_lookup): Define. * coff-ppc.c (ppc_coff_reloc_name_lookup): New function. (coff_bfd_reloc_name_lookup): Define. * coff-rs6000.c (coff_bfd_reloc_name_lookup): Define. (_bfd_xcoff_reloc_name_lookup): New function. (rs6000coff_vec, pmac_xcoff_vec): Init new field. * coff-sh.c (coff_bfd_reloc_name_lookup): Define. (sh_coff_reloc_name_lookup): New function. * coff-sparc.c (coff_sparc_reloc_name_lookup): New function. (coff_bfd_reloc_name_lookup): Define. * coff-tic30.c (coff_bfd_reloc_name_lookup): Define. (tic30_coff_reloc_name_lookup): New function. * coff-tic4x.c (coff_bfd_reloc_name_lookup): Define. (tic4x_coff_reloc_name_lookup): New function. * coff-tic54x.c (coff_bfd_reloc_name_lookup): Define. (tic54x_coff_reloc_name_lookup): New function. * coff-x86_64.c (coff_bfd_reloc_name_lookup): Define. (coff_amd64_reloc_name_lookup): New function. * coff-z80.c (coff_z80_reloc_name_lookup): New function. (coff_bfd_reloc_name_lookup): Define. * coff-z8k.c (coff_z8k_reloc_name_lookup): New function. (coff_bfd_reloc_name_lookup): Define. * coff64-rs6000.c (coff_bfd_reloc_name_lookup): Define. (xcoff64_reloc_name_lookup): New function. (rs6000coff64_vec, aix5coff64_vec): Init new field. * coffcode.h (coff_bfd_reloc_name_lookup): Define. * elf-hppa.h (elf_hppa_reloc_name_lookup): New function. * elf-m10200.c (bfd_elf32_bfd_reloc_name_lookup): New function. * elf-m10300.c (bfd_elf32_bfd_reloc_name_lookup): New function. * elf32-arc.c (bfd_elf32_bfd_reloc_name_lookup): New function. * elf32-arm.c (elf32_arm_reloc_name_lookup): New function. (bfd_elf32_bfd_reloc_name_lookup): Define. * elf32-avr.c (bfd_elf32_bfd_reloc_name_lookup): New function. * elf32-bfin.c (bfin_bfd_reloc_name_lookup): New function. (bfd_elf32_bfd_reloc_name_lookup): Define. * elf32-cr16c.c (elf_cr16c_reloc_name_lookup): New function. (bfd_elf32_bfd_reloc_name_lookup): Define. * elf32-cris.c (cris_reloc_name_lookup): New function. (bfd_elf32_bfd_reloc_name_lookup): Define. * elf32-crx.c (elf_crx_reloc_name_lookup): New function. (bfd_elf32_bfd_reloc_name_lookup): Define. * elf32-d10v.c (bfd_elf32_bfd_reloc_name_lookup): New function. * elf32-d30v.c (bfd_elf32_bfd_reloc_name_lookup): New function. * elf32-dlx.c (elf32_dlx_reloc_name_lookup): New function. (bfd_elf32_bfd_reloc_name_lookup): Define. * elf32-fr30.c (fr30_reloc_name_lookup): New function. (bfd_elf32_bfd_reloc_name_lookup): Define. * elf32-frv.c (frv_reloc_name_lookup): New function. (bfd_elf32_bfd_reloc_name_lookup): Define. * elf32-gen.c (bfd_elf32_bfd_reloc_name_lookup): Define. * elf32-h8300.c (elf32_h8_reloc_name_lookup): New function. (bfd_elf32_bfd_reloc_name_lookup): Define. * elf32-hppa.c (bfd_elf32_bfd_reloc_name_lookup): Define. * elf32-i370.c (i370_elf_reloc_name_lookup): New function. (bfd_elf32_bfd_reloc_name_lookup): Define. * elf32-i386.c (elf_i386_reloc_name_lookup): New function. (bfd_elf32_bfd_reloc_name_lookup): Define. * elf32-i860.c (elf32_i860_reloc_name_lookup): New function. (bfd_elf32_bfd_reloc_name_lookup): Define. * elf32-i960.c (elf32_i960_reloc_name_lookup): New function. (bfd_elf32_bfd_reloc_name_lookup): Define. * elf32-ip2k.c (ip2k_reloc_name_lookup): New function. (bfd_elf32_bfd_reloc_name_lookup): Define. * elf32-iq2000.c (iq2000_reloc_name_lookup): New function. (bfd_elf32_bfd_reloc_name_lookup): Define. * elf32-m32c.c (m32c_reloc_name_lookup): New function. (bfd_elf32_bfd_reloc_name_lookup): Define. * elf32-m32r.c (bfd_elf32_bfd_reloc_name_lookup): New function. * elf32-m68hc11.c (bfd_elf32_bfd_reloc_name_lookup): New function. * elf32-m68hc12.c (bfd_elf32_bfd_reloc_name_lookup): New function. * elf32-m68k.c (reloc_name_lookup): New function. (bfd_elf32_bfd_reloc_name_lookup): Define. * elf32-m88k.c (bfd_elf32_bfd_reloc_name_lookup): Define. * elf32-mcore.c (mcore_elf_reloc_name_lookup): New function. (bfd_elf32_bfd_reloc_name_lookup): Define. * elf32-mep.c (mep_reloc_name_lookup): New function. (bfd_elf32_bfd_reloc_name_lookup): Define. * elf32-mips.c (bfd_elf32_bfd_reloc_name_lookup): New function. (mips_vxworks_bfd_reloc_name_lookup): Likewise. (bfd_elf32_bfd_reloc_name_lookup): Define. * elf32-msp430.c (bfd_elf32_bfd_reloc_name_lookup): New function. * elf32-mt.c (mt_reloc_name_lookup): New function. (bfd_elf32_bfd_reloc_name_lookup): Define. * elf32-openrisc.c (openrisc_reloc_name_lookup): New function. (bfd_elf32_bfd_reloc_name_lookup): Define. * elf32-or32.c (bfd_elf32_bfd_reloc_name_lookup): New function. * elf32-pj.c (pj_elf_reloc_name_lookup): New function. (bfd_elf32_bfd_reloc_name_lookup): Define. * elf32-ppc.c (ppc_elf_reloc_name_lookup): New function. (bfd_elf32_bfd_reloc_name_lookup): Define. * elf32-s390.c (elf_s390_reloc_name_lookup): New function. (bfd_elf32_bfd_reloc_name_lookup): Define. * elf32-score.c (elf32_score_reloc_name_lookup): New function. (bfd_elf32_bfd_reloc_name_lookup): Define. * elf32-sh.c (sh_elf_reloc_name_lookup): New function. (bfd_elf32_bfd_reloc_name_lookup): Define. * elf32-sparc.c (bfd_elf32_bfd_reloc_name_lookup): Define. * elf32-spu.c (spu_elf_reloc_name_lookup): New function. (bfd_elf32_bfd_reloc_name_lookup): Define. * elf32-v850.c (v850_elf_reloc_name_lookup): New function. (bfd_elf32_bfd_reloc_name_lookup): Define. * elf32-vax.c (reloc_name_lookup): New function. (bfd_elf32_bfd_reloc_name_lookup): Define. * elf32-xc16x.c (xc16x_reloc_name_lookup): New function. (bfd_elf32_bfd_reloc_name_lookup): Define. * elf32-xstormy16.c (xstormy16_reloc_name_lookup): New function. (bfd_elf32_bfd_reloc_name_lookup): Define. * elf32-xtensa.c (elf_xtensa_reloc_name_lookup): New function. (bfd_elf32_bfd_reloc_name_lookup): Define. * elf64-alpha.c (elf64_alpha_bfd_reloc_name_lookup): New function. (bfd_elf64_bfd_reloc_name_lookup): Define. * elf64-gen.c (bfd_elf64_bfd_reloc_name_lookup): Define. * elf64-hppa.c (bfd_elf64_bfd_reloc_name_lookup): Define. * elf64-mips.c (bfd_elf64_bfd_reloc_name_lookup): New function. * elf64-mmix.c (bfd_elf64_bfd_reloc_name_lookup): New function. * elf64-ppc.c (ppc64_elf_reloc_name_lookup): New function. (bfd_elf64_bfd_reloc_name_lookup): Define. * elf64-s390.c (elf_s390_reloc_name_lookup): New function. (bfd_elf64_bfd_reloc_name_lookup): Define. * elf64-sh64.c (sh_elf64_reloc_name_lookup): New function. (bfd_elf64_bfd_reloc_name_lookup): Define. * elf64-sparc.c (bfd_elf64_bfd_reloc_name_lookup): Define. * elf64-x86-64.c (elf64_x86_64_reloc_name_lookup): New function. (bfd_elf64_bfd_reloc_name_lookup): Define. * elfn32-mips.c (bfd_elf32_bfd_reloc_name_lookup): New function. * elfxx-ia64.c (elfNN_ia64_reloc_name_lookup): New function. (bfd_elfNN_bfd_reloc_name_lookup): Define. * elfxx-sparc.c (_bfd_sparc_elf_reloc_name_lookup): New function. * elfxx-sparc.h (_bfd_sparc_elf_reloc_name_lookup): Declare. * i386msdos.c (msdos_bfd_reloc_name_lookup): Define. * i386os9k.c (aout_32_bfd_reloc_name_lookup): Define. * ieee.c (ieee_bfd_reloc_name_lookup): Define. * libaout.h (NAME (aout, reloc_name_lookup)): Declare. * libbfd-in.h (_bfd_norelocs_bfd_reloc_name_lookup): Declare. * mipsbsd.c (MY_bfd_reloc_name_lookup): Define. (MY(reloc_type_lookup)): Rename from MY(reloc_howto_type_lookup). (MY(reloc_name_lookup)): New function. * nlm-target.h (nlm_bfd_reloc_name_lookup): Define. * oasys.c (oasys_bfd_reloc_name_lookup): Define. * pdp11.c (NAME (aout, reloc_name_lookup)): New function. * pe-mips.c (coff_mips_reloc_name_lookup): New function. (coff_bfd_reloc_name_lookup): Define. * reloc.c (bfd_reloc_name_lookup): New function. * riscix.c (riscix_reloc_name_lookup): New function. (MY_bfd_reloc_name_lookup): Define. * som.c (som_bfd_reloc_name_lookup): New function. * targets.c (struct bfd_target): Add reloc_name_lookup. (BFD_JUMP_TABLE_RELOCS): Add NAME##_bfd_reloc_name_lookup. * versados.c (versados_bfd_reloc_name_lookup): Define. * vms.c (vms_bfd_reloc_name_lookup): New function. * bfd-in2.h: Regenerate. * libbfd.h: Regenerate. 2007-03-26 Thiemo Seufer PR ld/4208 * elfxx-mips.c (mips_elf_next_relocation): Don't signal an error if no matching relocation is found. (_bfd_mips_elf_relocate_section): Only warn about missing relocations. 2007-03-23 H.J. Lu * configure: Regenerated. 2007-03-23 Joseph Myers * configure.in: Use ACX_PKGVERSION and ACX_BUGURL. * configure: Regenerate. 2007-03-23 Alan Modra * elf32-spu.c (spu_stub_name): Don't put input section in stub name. Remove input_sec param. Adjust all calls. (write_one_stub): Adjust stub symbol. (needs_ovl_stub): New function, extracted from.. (spu_elf_size_stubs): ..here. (spu_elf_relocate_section): Use needs_ovl_stub. 2007-03-22 Joseph Myers * Makefile.am (bfdver.h): Only set VERSION_PACKAGE in bfdver.texi if nonempty. * configure.in (REPORT_BUGS_TO): Do not use "URL:" prefix. * Makefile.in, configure, doc/Makefile.in: Regenerate. 2007-03-22 H.J. Lu PR ld/4210 * peXXigen.c (_bfd_XXi_swap_aouthdr_out): Use the virtual address and virtual size of the last section for the image size. 2007-03-22 H.J. Lu PR ld/4007 * elf.c (assign_file_positions_for_load_sections): Check if all sections are in the segment. 2007-03-20 H.J. Lu PR ld/4208 * elfxx-mips.c (_bfd_mips_elf_relocate_section): Report missing matching LO16 relocation for HI16 relocation. 2007-03-20 Paul Brook * bfd-in.h (bfd_elf32_arm_set_target_relocs): Update prototype. * bfd-in2.h: Regenerate. * elf32-arm.c (elf32_arm_link_hash_table): Add pic_veneer. (record_arm_to_thumb_glue): Use globals->pic_veneer. (elf32_arm_create_thumb_stub): Ditto. (bfd_elf32_arm_set_target_relocs): Set globals->pic_veneer. 2007-03-18 Mark Shinwell * bfd-in.h (bfd_elf32_arm_set_target_relocs): Add "bfd *" argument and extra last argument. * bfd-in2.h: Regenerate. * elf32-arm.c (elf32_arm_obj_tdata): Add no_enum_size_warning member. (bfd_elf32_arm_set_target_relocs): Add "bfd *" argument and extra last argument. Set no_enum_size_warning appropriately. (elf32_arm_merge_eabi_attributes): Improve enum sizes diagnostic, suppressing it when no_enum_size_warning dictates. 2007-03-20 Nick Clifton PR binutils/3535 * elf.c (copy_private_bfd_data): Widen the scope of Solaris specific conditions that need the program headers to be rewritten. 2007-03-19 H.J. Lu * peXXigen.c (_bfd_XXi_swap_aouthdr_in): Store Magic, MajorLinkerVersion, MinorLinkerVersion, SizeOfCode, SizeOfInitializedData, SizeOfUninitializedData, AddressOfEntryPoint, BaseOfCode and BaseOfData in internal extra PE a.out header. (IMAGE_NT_OPTIONAL_HDR_MAGIC): Defined as 0x10b if not defined. (IMAGE_NT_OPTIONAL_HDR64_MAGIC): Defined as 0x20b if not defined. (IMAGE_NT_OPTIONAL_HDRROM_MAGIC): Defined as 0x107 if not defined. (_bfd_XX_print_private_bfd_data_common): Also print Magic, MajorLinkerVersion, MinorLinkerVersion, SizeOfCode, SizeOfInitializedData, SizeOfUninitializedData, AddressOfEntryPoint, BaseOfCode and BaseOfData from internal extra PE a.out header. 2007-03-18 H.J. Lu * targets.c (_bfd_target_vector): Add bfd_efi_app_x86_64_vec only if BFD64 is defined. 2007-03-17 Alan Modra * elf32-spu.c (spu_elf_size_stubs): Always use an overlay stub on setjmp calls. 2007-03-15 H.J. Lu * doc/Makefile.in: Regenerated. 2007-03-15 H.J. Lu * Makefile.am (bfdver.h): Substitute report_bugs_to. Also create doc/bfdver.texi. * Makefile.in: Regenerated. * configure.in (--with-bugurl): New option. * configure: Regenerated. * version.h (REPORT_BUGS_TO): New. 2007-03-13 H.J. Lu PR binutils/3826 * elf-bfd.h (elf_backend_data): Add elf_osabi. (_bfd_elf_set_osabi): New. * elf.c (_bfd_elf_set_osabi): New. * elf32-hppa.c (elf32_hppa_post_process_headers): Removed. (elf_backend_post_process_headers): Defined with _bfd_elf_set_osabi. (ELF_OSABI): Properly defined for each target. * elf32-i370.c (i370_elf_post_process_headers): Removed. (ELF_OSABI): Defined. (elf_backend_post_process_headers): Defined with _bfd_elf_set_osabi. * elf32-i386.c (ELF_OSABI): Defined to ELFOSABI_FREEBSD for freebsd. (elf_i386_post_process_headers): Set EI_OSABI with elf_osabi. * elf32-msp430.c (elf32_msp430_post_process_headers): Removed. (ELF_OSABI): Defined. (elf_backend_post_process_headers): Defined with _bfd_elf_set_osabi. * elf64-alpha.c (ELF_OSABI): Defined to ELFOSABI_FREEBSD for freebsd. (elf64_alpha_fbsd_post_process_headers): Set EI_OSABI with elf_osabi. * elf64-hppa.c (elf64_hppa_post_process_headers): Set EI_OSABI with elf_osabi. (ELF_OSABI): Properly defined for each target. (elf_backend_post_process_headers): Defined with _bfd_elf_set_osabi for Linux. * elf64-sparc.c (elf64_sparc_fbsd_post_process_headers): Removed. (ELF_OSABI): Defined to ELFOSABI_FREEBSD for freebsd. (elf_backend_post_process_headers): Defined with _bfd_elf_set_osabi. * elf64-x86-64.c (elf64_x86_64_fbsd_post_process_headers): Removed. (ELF_OSABI): Defined to ELFOSABI_FREEBSD for freebsd. (elf_backend_post_process_headers): Defined with _bfd_elf_set_osabi. * elfcode.h (elf_object_p): Match the ELFOSABI_NONE ELF target with any ELF target of the compatible machine for which we do not have a specific backend. * elfxx-ia64.c (elfNN_hpux_post_process_headers): Set EI_OSABI with elf_osabi. * elfxx-target.h (ELF_OSABI): Default to ELFOSABI_NONE. (elfNN_bed): Initialize elf_osabi with ELF_OSABI. 2007-03-08 Alan Modra * elf32-v850.c (v850_elf_link_output_symbol_hook): Clean out V850_OTHER_* bits. 2007-03-08 Alan Modra * Makefile.am (BFD64_BACKENDS): Add efi-app-x86_64.lo. (BFD64_BACKENDS_CFILES): Add efi-app-x86_64.c. Run "make dep-am". * Makefile.in: Regenerate. * po/SRC-POTFILES.in: Regenerate. 2007-03-07 H.J. Lu * config.bfd (x86_64-*-freebsd*): Add bfd_efi_app_x86_64_vec. (x86_64-*-kfreebsd*-gnu): Likewise. (x86_64-*-netbsd*): Likewise. (x86_64-*-openbsd*): Likewise. (x86_64-*-linux-*): Likewise. * configure.in: Support bfd_efi_app_x86_64_vec. * configure: Regenerated. * efi-app-x86_64.c: New file. * targets.c (bfd_efi_app_x86_64_vec): New. (_bfd_target_vector): Add bfd_efi_app_x86_64_vec. 2007-03-07 Kai Tietz * reloc.c (bfd_generic_get_relocated_section_contents): Remove const for type reloc_howto_type. 2007-03-07 Alan Modra PR 3958 * elf-bfd.h (RELOC_FOR_GLOBAL_SYMBOL): No error on relocatable link. (elf_discarded_section): Move.. * bfd-in.h: ..to here. * bfd-in2.h: Regenerate. * elflink.c (elf_link_input_bfd): Don't zap relocs against symbols from discarded sections before relocate_section has done its job. * reloc.c (bfd_generic_get_relocated_section_contents): Handle relocs against symbols from discarded sections. * elf-hppa.h (elf_hppa_howto_table): Set size. Set dst_mask on SECREL32. (elf_hppa_relocate_section): Handle relocatable link after setting sec, sym, h etc. for final link. Squash error messages for relocatable link. Clear section contents for relocs against symbols in discarded sections, and zero reloc. Remove existing zero r_symndx code. * elf-m10200.c (mn10200_elf_relocate_section): Likewise. * elf-m10300.c (mn10300_elf_relocate_section): Likewise. * elf32-arm.c (elf32_arm_relocate_section): Likewise. * elf32-avr.c (elf32_avr_relocate_section): Likewise. * elf32-bfin.c (bfinfdpic_relocate_section): Likewise. (bfin_relocate_section): Likewise. * elf32-cr16c.c (elf32_cr16c_relocate_section): Likewise. * elf32-cris.c (cris_elf_relocate_section): Likewise. * elf32-crx.c (elf32_crx_relocate_section): Likewise. * elf32-d10v.c (elf32_d10v_relocate_section): Likewise. * elf32-fr30.c (fr30_elf_relocate_section): Likewise. * elf32-frv.c (elf32_frv_relocate_section): Likewise. * elf32-h8300.c (elf32_h8_relocate_section): Likewise. * elf32-hppa.c (elf32_hppa_relocate_section): Likewise. * elf32-i370.c (i370_elf_relocate_section): Likewise. * elf32-i386.c (elf_i386_relocate_section): Likewise. * elf32-i860.c (elf32_i860_relocate_section): Likewise. * elf32-ip2k.c (ip2k_elf_relocate_section): Likewise. * elf32-iq2000.c (iq2000_elf_relocate_section): Likewise. * elf32-m32c.c (m32c_elf_relocate_section): Likewise. * elf32-m32r.c (m32r_elf_relocate_section): Likewise. * elf32-m68hc1x.c (elf32_m68hc11_check_relocs): Likewise. * elf32-m68k.c (elf_m68k_relocate_section): Likewise. * elf32-mcore.c (mcore_elf_relocate_section): Likewise. * elf32-mep.c (mep_elf_relocate_section): Likewise. * elf32-msp430.c (elf32_msp430_relocate_section): Likewise. * elf32-mt.c (mt_elf_relocate_section): Likewise. * elf32-openrisc.c (openrisc_elf_relocate_section): Likewise. * elf32-ppc.c (ppc_elf_relocate_section): Likewise. * elf32-s390.c (elf_s390_relocate_section): Likewise. * elf32-score.c (_bfd_score_elf_relocate_section): Likewise. * elf32-sh.c (sh_elf_relocate_section): Likewise. * elf32-spu.c (spu_elf_relocate_section): Likewise. * elf32-v850.c (v850_elf_relocate_section): Likewise. * elf32-vax.c (elf_vax_relocate_section): Likewise. * elf32-xc16x.c (elf32_xc16x_relocate_section): Likewise. * elf32-xstormy16.c (xstormy16_elf_relocate_section): Likewise. * elf32-xtensa.c (elf_xtensa_relocate_section): Likewise. * elf64-alpha.c (elf64_alpha_relocate_section_r): Likewise. (elf64_alpha_relocate_section): Likewise. * elf64-mmix.c (mmix_elf_relocate_section): Likewise. * elf64-ppc.c (ppc64_elf_relocate_section): Likewise. * elf64-s390.c (elf_s390_relocate_section): Likewise. * elf64-sh64.c (sh_elf64_relocate_section): Likewise. * elf64-x86-64.c (elf64_x86_64_relocate_section): Likewise. * elfxx-ia64.c (elfNN_ia64_relocate_section): Likewise. * elfxx-mips.c (_bfd_mips_elf_relocate_section): Likewise. * elfxx-sparc.c (_bfd_sparc_elf_relocate_section): Likewise. * elf32-arm.c (elf32_arm_relocate_section): Always adjust section symbols for relocatable link. Don't use always-zero st_value. (elf_backend_rela_normal): Don't define. * elf32-bfin.c (bfinfdpic_relocate_section): Use RELOC_FOR_GLOBAL_SYMBOL. * elf32-frv.c (elf32_frv_relocate_section): Likewise. * elf32-d10v.c (elf32_d10v_relocate_section): Combine SEC_MERGE section symbol adjustments with same for relocatable link. * elf32-i386.c (elf_i386_relocate_section): Likewise. * elf32-m68hc1x.c (m68hc11_get_relocation_value): Move.. (elf32_m68hc11_check_relocs): ..to here. * elf32-score.c (score_elf_final_link_relocate): Remove zero r_symndx code. * elfxx-mips.c (mips_elf_calculate_relocation): Likewise. 2007-03-07 Alan Modra PR 4144 * elf.c (assign_file_positions_for_load_sections): Don't adjust p_memsz for !SEC_LOAD section vma modulo page size. Instead, use the same lma based adjustment for SEC_LOAD sections. 2007-03-01 Bob Wilson * elf32-xtensa.c (analyze_relocations): Zero src_count if not relaxing. (find_relaxable_sections): Do not increment src_count for unknown opcodes. Decode only once instead of calling is_l32r_relocation. (compute_text_actions): Remove unused no_insn_move flag. Assert that src_next matches src_count in relax_info. 2007-03-01 Joseph Myers * Makefile.am (bfdver.h): Use "." not " " between version number and date. * Makefile.in: Regenerate. * configure.in (PKGVERSION): Default to "(GNU Binutils) ". * configure: Regenerate. 2007-02-28 Nathan Sidwell * config.bfd (sh-*-uclinux, sh[12]-*-uclinux): New stanza. 2007-02-27 Alan Modra * elf32-spu.h (struct _ovl_stream): Make "start" and "end" const. * elf32-spu.c (ovl_mgr_pread): Add const to casts. 2007-02-23 Carlos O'Donell * dwarf2.c (_bfd_dwarf2_find_nearest_line): Assume 32-bit DWARF even with 64-bit addresses. (_bfd_dwarf2_find_nearest_line): Likewise. 2007-02-23 Nick Clifton PR binutils/3535 * elf.c (copy_private_bfd_data): Always rewrite the program headers when a Solaris interpreter segment is involved. 2007-02-22 Paul Brook * elflink.c (gc_mark_hook_fn): Remove. (_bfd_elf_gc_mark): Rename gc_mark_hook_fn to elf_gc_mark_hook_fn. (bfd_elf_gc_sections): Ditto. Call gc_mark_extra_sections. * elf-bfd.h (elf_gc_mark_hook_fn): Define. (elf_backend_data): Add gc_mark_extra_sections. * elfxx-target.h (elf_backend_gc_mark_extra_sections): Provide default definition. (elfNN_bed): Add elf_backend_gc_mark_extra_sections. * elf32-arm.c (elf32_arm_gc_mark_extra_sections): New function. (elf_backend_gc_mark_extra_sections): Define. 2007-02-21 Nick Clifton * elf.c (_bfd_elf_map_sections_to_segments): If the override_segment_assignment callback is defined then call it. 2007-02-21 Alan Modra * elf32-spu.c (spu_elf_size_stubs): Correct order of warning args. 2007-02-19 Thiemo Seufer * elfxx-mips.c (_bfd_mips_elf_print_private_bfd_data): Remove translation marker from untranslatable strings. 2007-02-19 Alan Modra * elf-bfd.h (struct elf_backend_data): Add default_execstack. * elflink.c (bfd_elf_size_dynamic_sections): Heed default_execstack. * elfxx-target.h (elf_backend_default_execstack): Define to 1. (elfNN_bed): Init new field. * elf64-ppc.c (elf_backend_default_execstack): Define to 0. 2007-02-17 Mark Mitchell Nathan Sidwell Vladimir Prus * configure.in (--with-pkgversion): New option. * configure: Regenerate. * Makefile.am (bfdver.h): Substitute for @bfd_version_package@. * Makefile.in: Regenerate. * version.h (BFD_VERSION_STRING): Define using @bfd_version_package@. 2007-02-16 Carlos O'Donell * elfxx-mips.c (_bfd_mips_elf_print_private_bfd_data): Print EF_MIPS_NOREORDER, EF_MIPS_PIC, EF_MIPS_CPIC, EF_MIPS_XGOT and EF_MIPS_UCODE. 2007-02-15 Alan Modra * libbfd-in.h (_bfd_norelocs_get_reloc_upper_bound): Don't define, declare. (_bfd_norelocs_canonicalize_reloc): Likewise. * libbfd.h: Regenerate. * libbfd.c (_bfd_norelocs_get_reloc_upper_bound): New function. (_bfd_norelocs_canonicalize_reloc): Likewise. * binary.c (binary_bfd_reloc_type_lookup): Don't define. (binary_get_reloc_upper_bound, binary_canonicalize_reloc): Likewise. (binary_vec): Use _bfd_norelocs in BFD_JUMP_TABLE_RELOCS. * ihex.c: Similarly. * mach-o-target.c: Similarly. * mach-o.c: Similarly. * mmo.c: Similarly. * pef.c: Similarly. * ppcboot.c: Similarly. * srec.c: Similarly. * xsym.c: Similarly. 2007-02-14 H.J. Lu PR ld/3953 * elflink.c (_bfd_elf_add_default_symbol): Check warning symbol when adding default symbol. 2007-02-13 Alan Modra * elf64-ppc.c (create_linkage_sections): Use section ".branch_lt" for branch lookup table. 2007-02-12 Alan Modra * elf64-ppc.c (create_linkage_sections): Don't create .rela.rodata.brlt for --emit-relocs. (ppc_build_one_stub): Create relocs for brlt --emit-relocs here. (ppc_size_one_stub): Count them. Simplify test of stub type when counting stub relocs. Set SEC_RELOC too. (ppc64_elf_size_stubs): Clear reloc_count and SEC_RELOC. (ppc64_elf_finish_dynamic_sections): Output brlt relocs. 2007-02-12 Alan Modra * elflink.c (evaluate_complex_relocation_symbols): Ignore relocs with a zero symbol index. 2007-02-12 Alan Modra * elflink.c (bfd_elf_discard_info): Tidy setting of "eh". 2007-02-05 Dave Brolley Stan Cox PR ld/3972 * elflink.c (elf_link_input_bfd): Always setup finfo->internal_syms. 2007-02-05 Dave Brolley Richard Sandiford Stan Cox Nick Clifton DJ Delorie Frank Ch. Eigler Ben Elliston Richard Henderson * Makefile.am (ALL_MACHINES): Add cpu-mep.lo. (ALL_MACHINES_CFILES): Add CPU_MEP.c. (BFD32_BACKENDS): Add elf32-mep.lo. (BFD32_BACKENDS_CFILES): Add elf32-mep.c. (cpu-mep.lo,elf32-mep.lo): New targets. * archures.c (bfd_arch_mep): New enumerator. (bfd_mach_mep, bfd_mach_mep_h1): New macros. (bfd_mep_arch): New external variable. (bfd_archures_list): Add bfd_mep_arch. * config.bfd: Support mep-*-elf. * configure.in: Support bfd_elf32_mep_vec and bfd_elf32_mep_little_vec. * reloc.c (BFD_RELOC_MEP_*): New relocation numbers. * targets.c (bfd_elf32_mep_vec,bfd_elf32_mep_little_vec): New extern declarations. (_bfd_target_vector): Add bfd_elf32_mep_vec and bfd_elf32_mep_little_vec. * mep-relocs.pl: New file. * cpu-mep.c: New file. * elf32-mep.c: New file. * bfd-in2.h: Regenerate. * po/POTFILES.in: Regenerate. * libbfd.h: Regenerate. * Makefile.in: Regenerate. * configure: Regenerate. 2007-02-05 Dave Brolley * elflink.c (evaluate_complex_relocation_symbols): Check for STT_SRELC for global symbols. (perform_complex_relocations): Renamed to perform_complex_relocation and now examines only one relocation. (elf_link_input_bfd): Don't call perform_complex_relocations. * elf-bfd.h (bfd_elf_perform_complex_relocation_: New prototype. * elf32-mep.c (mep_elf_howto_table): Add R_RELC. (OD,OS,OU): #undefs corrected to N, S and U repectively. (mep_reloc_type_lookup): Handle BFD_RELOC_RELC. (complex_reloc_installation_howto): Removed. (mep_info_to_howto_rela): Remove special case for r_type==0xff. (mep_elf_relocate_section): Call bfd_elf_perform_complex_relocation. 2007-02-05 Dave Brolley Richard Sandiford DJ Delorie Graydon Hoare Nick Clifton Jeff Holcomb Frank Ch. Eigler * elf-bfd.h (bfd_elf_perform_complex_relocations): New prototype. * elf.c (swap_out_syms): Handle BSF_RELC and BSF_SRELC. * elfcode.h (elf_slurp_symbol_table): Handle STT_RELC and STT_SRELC. * elflink.c (set_symbolValue): New static function. (resolve_symbol): Likewise. (resolve_section): Likewise. (undefined_reference): Likewise. (eval_symbol): Likewise. (evaluate_complex_relocation_symbols): Likewise. (put_value): Likewise. (get_value): Likewise. (decode_complex_addend): (bfd_elf_perform_complex_relocation): New function. (elf_link_input_bfd): Call evaluate_complex_relocation_symbols. * reloc.c (BFD_RELOC_RELC): New relocation number. * syms.c (BSF_RELC,BSF_SRELC): New macros. 2007-02-05 Bob Wilson * elf32-xtensa.c (elf_xtensa_make_sym_local): Delete. (elf_xtensa_hide_symbol, elf_backend_hide_symbol): Delete. (elf_xtensa_fix_refcounts, elf_xtensa_allocate_plt_size) (elf_xtensa_allocate_got_size): Replace these with... (elf_xtensa_allocate_dynrelocs): ...this new function. (elf_xtensa_size_dynamic_sections): Use it. 2007-02-05 Bob Wilson * elf32-xtensa.c (elf_howto_table) : Set src_mask to zero. : Likewise. Also fix dst_mask. : Set pcrel_offset to TRUE. 2007-02-02 Bob Wilson * elf32-xtensa.c (xtensa_elf_dynamic_symbol_p): Renamed to... (elf_xtensa_dynamic_symbol_p): ...this. 2007-02-02 Bob Wilson * elf32-xtensa.c (plt_reloc_count): Move into link hash table. (struct elf_xtensa_link_hash_table): New. (elf_xtensa_hash_table): New. (elf_xtensa_link_hash_table_create): New. (elf_xtensa_check_relocs): Update plt_reloc_count references. Update arguments to add_extra_plt_sections. (elf_xtensa_create_dynamic_sections): Record new sections in the hash table. Update for plt_reloc_count and add_extra_plt_sections. (add_extra_plt_sections, elf_xtensa_create_plt_entry): Replace dynobj argument with link info. Update calls to elf_xtensa_get_plt_section and elf_xtensa_get_gotplt_section. (elf_xtensa_allocate_local_got_size, elf_xtensa_size_dynamic_sections) (elf_xtensa_relocate_section, elf_xtensa_finish_dynamic_sections) (elf_xtensa_discard_info_for_section, shrink_dynamic_reloc_sections) (relax_property_section): Get sections from the hash table and update function calls. (elf_xtensa_get_plt_section, elf_xtensa_get_gotplt_section): Replace dynobj argument with link info. Get sections for first plt chunk from the hash table. (bfd_elf32_bfd_link_hash_table_create): Define. 2007-02-02 Jakub Jelinek * elf-bfd.h (struct elf_obj_tdata): Change symbuf type to void *. * elf.c (struct elf_symbuf_symbol, struct elf_symbuf_head): New types. (struct elf_symbol): Change first member into union. (elf_sort_elf_symbol): Compare pointers to internal syms rather than internal syms. Only compare st_shndx fields. (elf_create_symbuf): New function. (bfd_elf_match_symbols_in_sections): Use it. If symbufs are available for bfds, use a binary search, otherwise don't qsort symbols unnecessarily only to select which symbols are for the particular shndx. 2007-02-01 Nick Clifton PR ld/3852 * elf.c (_bfd_elf_link_hash_table_init): Initialize all the fields in the elf_link_hash_table structure. 2007-02-01 Alan Modra * elf-bfd.h (struct elf_backend_data): Add elf_backend_write_core_note. * elfxx-target.h (elf_backend_write_core_note): Define and use. * elf.c (elfcore_write_prpsinfo): Call the above. Add support for 32-bit core note on 64-bit target. (elfcore_write_prstatus): Likewise. (elfcore_write_lwpstatus): Make note_name const. (elfcore_write_prfpreg): Likewise. (elfcore_write_pstatus): Add support for 32-bit core note on 64-bit target. * elf32-ppc.c (ppc_elf_write_core_note): New function. (elf_backend_write_core_note): Define. * elf64-ppc.c (ppc64_elf_write_core_note): New function. (elf_backend_write_core_note): Define. 2007-01-31 H.J. Lu * elf32-cris.c (INCLUDED_TARGET_FILE): Removed. (elf32_bed): Defined for elf32-us-cris. * elf64-sh64.c (elf64_bed): Defined for Linux. (INCLUDED_TARGET_FILE): Removed. * elfxx-target.h (elfNN_bed): Always define. Don't check INCLUDED_TARGET_FILE. 2007-01-31 DJ Delorie * elf-m10300.c (mn10300_elf_relocate_section): Clarify the warning message for dangerous relocs, special case the common user error. 2007-01-30 H.J. Lu * elf.c (copy_elf_program_header): Start from the first section in a segment and stop when all sections in a segment are accounted for. 2007-01-29 Julian Brown * bfd-in2.h: Regenerate. * bfd-in.h (bfd_arm_vfp11_fix): New enum. Specify how VFP11 instruction scanning should be done. (bfd_elf32_arm_init_maps, bfd_elf32_arm_vfp11_erratum_scan) (bfd_elf32_arm_vfp11_fix_veneer_locations): Add prototypes. (bfd_elf32_arm_set_target_relocs): Add vfp11 fix type argument to prototype. * elf-bfd.h (elf_backend_write_section): Add struct bfd_link_info argument. * elf32-arm.c (VFP11_ERRATUM_VENEER_SECTION_NAME) (VFP11_ERRATUM_VENEER_ENTRY_NAME): Define macros. (elf32_vfp11_erratum_type): New enum. (elf32_vfp11_erratum_list): New struct. List of veneers or jumps to veneers. (_arm_elf_section_data): Add mapsize, erratumcount, erratumlist. (elf32_arm_link_hash_table): Add vfp11_erratum_glue_size, vfp11_fix and num_vfp11_fixes fields. (elf32_arm_link_hash_table_create): Initialise vfp11_fix, vfp11_erratum_glue_size, num_vfp11_fixes fields. (VFP11_ERRATUM_VENEER_SIZE): Define. Size of an (ARM) veneer. (bfd_elf32_arm_allocate_interworking_sections): Initialise erratum glue section. (elf32_arm_section_map_add): Add an code/data mapping symbol entry to a section's map. (record_vfp11_erratum_veneer): Create a single veneer, and its associated symbols. (bfd_elf32_arm_add_glue_sections_to_bfd): Add vfp11 erratum glue. (bfd_elf32_arm_init_maps): Initialise mapping symbol table for input BFDs. (bfd_elf32_arm_set_vfp11_fix): Set the type of erratum workaround required. (bfd_arm_vfp11_pipe): Define VFP11 instruction pipes. (bfd_arm_vfp11_regno): Recode a register number from a VFP11 insn. (bfd_arm_vfp11_write_mask): Update write mask according to coded register number. (bfd_arm_vfp11_antidependency): New function. (bfd_arm_vfp11_insn_decode): Decode a VFP11 insn. (elf32_arm_compare_mapping): Declare. (bfd_elf32_arm_vfp11_erratum_scan): Scan the sections of an input BFD for potential erratum-triggering insns. Record results. (bfd_elf32_arm_vfp11_fix_veneer_locations): Find out where veneers and branches to veneers have been placed in virtual memory after layout. (bfd_elf32_arm_set_target_relocs): Set vfp11_fix field in global hash table. (elf32_arm_output_symbol_hook): Remove. (elf32_arm_write_section): Output veneers, and branches to veneers. Use maps from input sections, not output sections, for code byte-swapping. * elf32-ppc.c (ppc_elf_write_section): Add dummy link_info argument. * elf32-score.c (_bfd_score_elf_write_section): Likewise. * elfxx-mips.c (_bfd_mips_elf_write_section): Likewise. * elfxx-mips.h (_bfd_mips_elf_write_section): Likewise. 2007-01-27 H.J. Lu * elf64-hppa.c (elf64_bed): Defined for HPUX and Linux. (INCLUDED_TARGET_FILE): Removed. 2007-01-27 Mike Frysinger * elf32-hppa.c (elf32_bed): Define for hpux, linux and netbsd. (INCLUDED_TARGET_FILE): Remove. 2007-01-25 DJ Delorie * elf32-m32c.c (m32c_elf_howto_table): Don't complain about R_M32C_16 or R_M32C_24 relocs. 2007-01-25 Nick Clifton PR binutils/3874 * elf32-avr.c (avr_link_hash_table): Check to make sure that the hash table was created by elf32_avr_link_hash_table_create before using it. (elf32_avr_link_hash_newfunc): New function. Just pass the call through to _bfd_elf_link_hash_newfunc. (elf32_avr_link_hash_table_create): Use elf32_avr_link_hash_newfunc instead of _bfd_elf_link_hash_newfunc. (elf32_avr_relocate_section): Check for the hash table pointer being NULL. (elf32_avr_relax_section, avr_build_one_stub, elf32_avr_setup_params, get_local_syms, elf32_avr_size_stubs, elf32_avr_build_stubs): Likewise. 2007-01-16 H.J. Lu PR ld/3831 * elf-bfd.h (bfd_elf_link_mark_dynamic_symbol): Add an argument, Elf_Internal_Sym *. * elflink.c (bfd_elf_link_mark_dynamic_symbol): Mark a data symbol dynamic if info->dynamic_data is TRUE. (bfd_elf_record_link_assignment): Updated call to bfd_elf_record_link_assignment. (_bfd_elf_merge_symbol): Likewise. Always call bfd_elf_link_mark_dynamic_symbol. 2007-01-12 H.J. Lu * Makefile.am (BFD_LIBS): Removed. * Makefile.in: Regenerated. 2007-01-11 H.J. Lu PR binutils/3631 * Makefile.am (OFILES): Add @bfd64_libs@. (libbfd_la_SOURCES): Remove $(BFD64_LIBS_CFILES). * Makefile.in: Regenerated. * configure.in (bfd_libs): Replaced by ... (bfd64_libs): This. * configure: Regenerated. 2007-01-11 Nathan Sidwell * elf.c (assign_file_positions_for_load_sections): We can require fewer phdrs than expected. 2007-01-08 Kazu Hirata * archures.c (bfd_mach_cpu32_fido): Rename to bfd_mach_fido. * bfd-in2.h: Regenerate. * cpu-m68k.c (arch_info_struct): Use bfd_mach_fido instead of bfd_mach_cpu32_fido. (m68k_arch_features): Use fido_a instead of cpu32. (bfd_m68k_compatible): Reject the combination of Fido and ColdFire. Accept the combination of CPU32 and Fido with a warning. * elf32-m68k.c (elf32_m68k_object_p, elf32_m68k_merge_private_bfd_data, elf32_m68k_print_private_bfd_data): Treat Fido as an architecture by itself. 2007-01-08 Kai Tietz * config.bfd: Renamed target x86_64-*-mingw64 to x86_64-*-mingw*. 2007-01-05 Jakub Jelinek * texhex.c (first_phase): Don't fall through into the default case. (pass_over): Replace abort () calls with return FALSE. Fix buffer overflow. 2007-01-04 Jie Zhang * elf-eh-frame.c (_bfd_elf_discard_section_eh_frame): Don't set SEC_EXCLUDE on zero size .eh_frame. For older changes see ChangeLog-2006 Local Variables: mode: change-log left-margin: 8 fill-column: 74 version-control: never End: Index: projects/clang350-import/contrib/binutils/bfd/bfd-in2.h =================================================================== --- projects/clang350-import/contrib/binutils/bfd/bfd-in2.h (revision 275748) +++ projects/clang350-import/contrib/binutils/bfd/bfd-in2.h (revision 275749) @@ -1,5398 +1,5403 @@ /* DO NOT EDIT! -*- buffer-read-only: t -*- This file is automatically generated from "bfd-in.h", "init.c", "opncls.c", "libbfd.c", "bfdio.c", "bfdwin.c", "section.c", "archures.c", "reloc.c", "syms.c", "bfd.c", "archive.c", "corefile.c", "targets.c", "format.c", "linker.c" and "simple.c". Run "make headers" in your build bfd/ to regenerate. */ /* Main header file for the bfd library -- portable access to object files. Copyright 1990, 1991, 1992, 1993, 1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007 Free Software Foundation, Inc. Contributed by Cygnus Support. This file is part of BFD, the Binary File Descriptor library. This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin Street - Fifth Floor, Boston, MA 02110-1301, USA. */ #ifndef __BFD_H_SEEN__ #define __BFD_H_SEEN__ #ifdef __cplusplus extern "C" { #endif #include "ansidecl.h" #include "symcat.h" #if defined (__STDC__) || defined (ALMOST_STDC) || defined (HAVE_STRINGIZE) #ifndef SABER /* This hack is to avoid a problem with some strict ANSI C preprocessors. The problem is, "32_" is not a valid preprocessing token, and we don't want extra underscores (e.g., "nlm_32_"). The XCONCAT2 macro will cause the inner CONCAT2 macros to be evaluated first, producing still-valid pp-tokens. Then the final concatenation can be done. */ #undef CONCAT4 #define CONCAT4(a,b,c,d) XCONCAT2(CONCAT2(a,b),CONCAT2(c,d)) #endif #endif /* This is a utility macro to handle the situation where the code wants to place a constant string into the code, followed by a comma and then the length of the string. Doing this by hand is error prone, so using this macro is safer. The macro will also safely handle the case where a NULL is passed as the arg. */ #define STRING_COMMA_LEN(STR) (STR), ((STR) ? sizeof (STR) - 1 : 0) /* Unfortunately it is not possible to use the STRING_COMMA_LEN macro to create the arguments to another macro, since the preprocessor will mis-count the number of arguments to the outer macro (by not evaluating STRING_COMMA_LEN and so missing the comma). This is a problem for example when trying to use STRING_COMMA_LEN to build the arguments to the strncmp() macro. Hence this alternative definition of strncmp is provided here. Note - these macros do NOT work if STR2 is not a constant string. */ #define CONST_STRNEQ(STR1,STR2) (strncmp ((STR1), (STR2), sizeof (STR2) - 1) == 0) /* strcpy() can have a similar problem, but since we know we are copying a constant string, we can use memcpy which will be faster since there is no need to check for a NUL byte inside STR. We can also save time if we do not need to copy the terminating NUL. */ #define LITMEMCPY(DEST,STR2) memcpy ((DEST), (STR2), sizeof (STR2) - 1) #define LITSTRCPY(DEST,STR2) memcpy ((DEST), (STR2), sizeof (STR2)) /* The word size used by BFD on the host. This may be 64 with a 32 bit target if the host is 64 bit, or if other 64 bit targets have been selected with --enable-targets, or if --enable-64-bit-bfd. */ #define BFD_ARCH_SIZE @wordsize@ /* The word size of the default bfd target. */ #define BFD_DEFAULT_TARGET_SIZE @bfd_default_target_size@ #define BFD_HOST_64BIT_LONG @BFD_HOST_64BIT_LONG@ #define BFD_HOST_LONG_LONG @BFD_HOST_LONG_LONG@ #if @BFD_HOST_64_BIT_DEFINED@ #define BFD_HOST_64_BIT @BFD_HOST_64_BIT@ #define BFD_HOST_U_64_BIT @BFD_HOST_U_64_BIT@ typedef BFD_HOST_64_BIT bfd_int64_t; typedef BFD_HOST_U_64_BIT bfd_uint64_t; #endif #if BFD_ARCH_SIZE >= 64 #define BFD64 #endif #ifndef INLINE #if __GNUC__ >= 2 #define INLINE __inline__ #else #define INLINE #endif #endif /* Forward declaration. */ typedef struct bfd bfd; /* Boolean type used in bfd. Too many systems define their own versions of "boolean" for us to safely typedef a "boolean" of our own. Using an enum for "bfd_boolean" has its own set of problems, with strange looking casts required to avoid warnings on some older compilers. Thus we just use an int. General rule: Functions which are bfd_boolean return TRUE on success and FALSE on failure (unless they're a predicate). */ typedef int bfd_boolean; #undef FALSE #undef TRUE #define FALSE 0 #define TRUE 1 #ifdef BFD64 #ifndef BFD_HOST_64_BIT #error No 64 bit integer type available #endif /* ! defined (BFD_HOST_64_BIT) */ typedef BFD_HOST_U_64_BIT bfd_vma; typedef BFD_HOST_64_BIT bfd_signed_vma; typedef BFD_HOST_U_64_BIT bfd_size_type; typedef BFD_HOST_U_64_BIT symvalue; #ifndef fprintf_vma #if BFD_HOST_64BIT_LONG #define sprintf_vma(s,x) sprintf (s, "%016lx", x) #define fprintf_vma(f,x) fprintf (f, "%016lx", x) #else #define _bfd_int64_low(x) ((unsigned long) (((x) & 0xffffffff))) #define _bfd_int64_high(x) ((unsigned long) (((x) >> 32) & 0xffffffff)) #define fprintf_vma(s,x) \ fprintf ((s), "%08lx%08lx", _bfd_int64_high (x), _bfd_int64_low (x)) #define sprintf_vma(s,x) \ sprintf ((s), "%08lx%08lx", _bfd_int64_high (x), _bfd_int64_low (x)) #endif #endif #else /* not BFD64 */ /* Represent a target address. Also used as a generic unsigned type which is guaranteed to be big enough to hold any arithmetic types we need to deal with. */ typedef unsigned long bfd_vma; /* A generic signed type which is guaranteed to be big enough to hold any arithmetic types we need to deal with. Can be assumed to be compatible with bfd_vma in the same way that signed and unsigned ints are compatible (as parameters, in assignment, etc). */ typedef long bfd_signed_vma; typedef unsigned long symvalue; typedef unsigned long bfd_size_type; /* Print a bfd_vma x on stream s. */ #define fprintf_vma(s,x) fprintf (s, "%08lx", x) #define sprintf_vma(s,x) sprintf (s, "%08lx", x) #endif /* not BFD64 */ #define HALF_BFD_SIZE_TYPE \ (((bfd_size_type) 1) << (8 * sizeof (bfd_size_type) / 2)) #ifndef BFD_HOST_64_BIT /* Fall back on a 32 bit type. The idea is to make these types always available for function return types, but in the case that BFD_HOST_64_BIT is undefined such a function should abort or otherwise signal an error. */ typedef bfd_signed_vma bfd_int64_t; typedef bfd_vma bfd_uint64_t; #endif /* An offset into a file. BFD always uses the largest possible offset based on the build time availability of fseek, fseeko, or fseeko64. */ typedef @bfd_file_ptr@ file_ptr; typedef unsigned @bfd_file_ptr@ ufile_ptr; extern void bfd_sprintf_vma (bfd *, char *, bfd_vma); extern void bfd_fprintf_vma (bfd *, void *, bfd_vma); #define printf_vma(x) fprintf_vma(stdout,x) #define bfd_printf_vma(abfd,x) bfd_fprintf_vma (abfd,stdout,x) typedef unsigned int flagword; /* 32 bits of flags */ typedef unsigned char bfd_byte; /* File formats. */ typedef enum bfd_format { bfd_unknown = 0, /* File format is unknown. */ bfd_object, /* Linker/assembler/compiler output. */ bfd_archive, /* Object archive file. */ bfd_core, /* Core dump. */ bfd_type_end /* Marks the end; don't use it! */ } bfd_format; /* Values that may appear in the flags field of a BFD. These also appear in the object_flags field of the bfd_target structure, where they indicate the set of flags used by that backend (not all flags are meaningful for all object file formats) (FIXME: at the moment, the object_flags values have mostly just been copied from backend to another, and are not necessarily correct). */ /* No flags. */ #define BFD_NO_FLAGS 0x00 /* BFD contains relocation entries. */ #define HAS_RELOC 0x01 /* BFD is directly executable. */ #define EXEC_P 0x02 /* BFD has line number information (basically used for F_LNNO in a COFF header). */ #define HAS_LINENO 0x04 /* BFD has debugging information. */ #define HAS_DEBUG 0x08 /* BFD has symbols. */ #define HAS_SYMS 0x10 /* BFD has local symbols (basically used for F_LSYMS in a COFF header). */ #define HAS_LOCALS 0x20 /* BFD is a dynamic object. */ #define DYNAMIC 0x40 /* Text section is write protected (if D_PAGED is not set, this is like an a.out NMAGIC file) (the linker sets this by default, but clears it for -r or -N). */ #define WP_TEXT 0x80 /* BFD is dynamically paged (this is like an a.out ZMAGIC file) (the linker sets this by default, but clears it for -r or -n or -N). */ #define D_PAGED 0x100 /* BFD is relaxable (this means that bfd_relax_section may be able to do something) (sometimes bfd_relax_section can do something even if this is not set). */ #define BFD_IS_RELAXABLE 0x200 /* This may be set before writing out a BFD to request using a traditional format. For example, this is used to request that when writing out an a.out object the symbols not be hashed to eliminate duplicates. */ #define BFD_TRADITIONAL_FORMAT 0x400 /* This flag indicates that the BFD contents are actually cached in memory. If this is set, iostream points to a bfd_in_memory struct. */ #define BFD_IN_MEMORY 0x800 /* The sections in this BFD specify a memory page. */ #define HAS_LOAD_PAGE 0x1000 /* This BFD has been created by the linker and doesn't correspond to any input file. */ #define BFD_LINKER_CREATED 0x2000 /* Symbols and relocation. */ /* A count of carsyms (canonical archive symbols). */ typedef unsigned long symindex; /* How to perform a relocation. */ typedef const struct reloc_howto_struct reloc_howto_type; #define BFD_NO_MORE_SYMBOLS ((symindex) ~0) /* General purpose part of a symbol X; target specific parts are in libcoff.h, libaout.h, etc. */ #define bfd_get_section(x) ((x)->section) #define bfd_get_output_section(x) ((x)->section->output_section) #define bfd_set_section(x,y) ((x)->section) = (y) #define bfd_asymbol_base(x) ((x)->section->vma) #define bfd_asymbol_value(x) (bfd_asymbol_base(x) + (x)->value) #define bfd_asymbol_name(x) ((x)->name) /*Perhaps future: #define bfd_asymbol_bfd(x) ((x)->section->owner)*/ #define bfd_asymbol_bfd(x) ((x)->the_bfd) #define bfd_asymbol_flavour(x) (bfd_asymbol_bfd(x)->xvec->flavour) /* A canonical archive symbol. */ /* This is a type pun with struct ranlib on purpose! */ typedef struct carsym { char *name; file_ptr file_offset; /* Look here to find the file. */ } carsym; /* To make these you call a carsymogen. */ /* Used in generating armaps (archive tables of contents). Perhaps just a forward definition would do? */ struct orl /* Output ranlib. */ { char **name; /* Symbol name. */ union { file_ptr pos; bfd *abfd; } u; /* bfd* or file position. */ int namidx; /* Index into string table. */ }; /* Linenumber stuff. */ typedef struct lineno_cache_entry { unsigned int line_number; /* Linenumber from start of function. */ union { struct bfd_symbol *sym; /* Function name. */ bfd_vma offset; /* Offset into section. */ } u; } alent; /* Object and core file sections. */ #define align_power(addr, align) \ (((addr) + ((bfd_vma) 1 << (align)) - 1) & ((bfd_vma) -1 << (align))) typedef struct bfd_section *sec_ptr; #define bfd_get_section_name(bfd, ptr) ((ptr)->name + 0) #define bfd_get_section_vma(bfd, ptr) ((ptr)->vma + 0) #define bfd_get_section_lma(bfd, ptr) ((ptr)->lma + 0) #define bfd_get_section_alignment(bfd, ptr) ((ptr)->alignment_power + 0) #define bfd_section_name(bfd, ptr) ((ptr)->name) #define bfd_section_size(bfd, ptr) ((ptr)->size) #define bfd_get_section_size(ptr) ((ptr)->size) #define bfd_section_vma(bfd, ptr) ((ptr)->vma) #define bfd_section_lma(bfd, ptr) ((ptr)->lma) #define bfd_section_alignment(bfd, ptr) ((ptr)->alignment_power) #define bfd_get_section_flags(bfd, ptr) ((ptr)->flags + 0) #define bfd_get_section_userdata(bfd, ptr) ((ptr)->userdata) #define bfd_is_com_section(ptr) (((ptr)->flags & SEC_IS_COMMON) != 0) #define bfd_set_section_vma(bfd, ptr, val) (((ptr)->vma = (ptr)->lma = (val)), ((ptr)->user_set_vma = TRUE), TRUE) #define bfd_set_section_alignment(bfd, ptr, val) (((ptr)->alignment_power = (val)),TRUE) #define bfd_set_section_userdata(bfd, ptr, val) (((ptr)->userdata = (val)),TRUE) /* Find the address one past the end of SEC. */ #define bfd_get_section_limit(bfd, sec) \ (((sec)->rawsize ? (sec)->rawsize : (sec)->size) \ / bfd_octets_per_byte (bfd)) /* Return TRUE if section has been discarded. */ #define elf_discarded_section(sec) \ (!bfd_is_abs_section (sec) \ && bfd_is_abs_section ((sec)->output_section) \ && (sec)->sec_info_type != ELF_INFO_TYPE_MERGE \ && (sec)->sec_info_type != ELF_INFO_TYPE_JUST_SYMS) /* Forward define. */ struct stat; typedef enum bfd_print_symbol { bfd_print_symbol_name, bfd_print_symbol_more, bfd_print_symbol_all } bfd_print_symbol_type; /* Information about a symbol that nm needs. */ typedef struct _symbol_info { symvalue value; char type; const char *name; /* Symbol name. */ unsigned char stab_type; /* Stab type. */ char stab_other; /* Stab other. */ short stab_desc; /* Stab desc. */ const char *stab_name; /* String for stab type. */ } symbol_info; /* Get the name of a stabs type code. */ extern const char *bfd_get_stab_name (int); /* Hash table routines. There is no way to free up a hash table. */ /* An element in the hash table. Most uses will actually use a larger structure, and an instance of this will be the first field. */ struct bfd_hash_entry { /* Next entry for this hash code. */ struct bfd_hash_entry *next; /* String being hashed. */ const char *string; /* Hash code. This is the full hash code, not the index into the table. */ unsigned long hash; }; /* A hash table. */ struct bfd_hash_table { /* The hash array. */ struct bfd_hash_entry **table; /* A function used to create new elements in the hash table. The first entry is itself a pointer to an element. When this function is first invoked, this pointer will be NULL. However, having the pointer permits a hierarchy of method functions to be built each of which calls the function in the superclass. Thus each function should be written to allocate a new block of memory only if the argument is NULL. */ struct bfd_hash_entry *(*newfunc) (struct bfd_hash_entry *, struct bfd_hash_table *, const char *); /* An objalloc for this hash table. This is a struct objalloc *, but we use void * to avoid requiring the inclusion of objalloc.h. */ void *memory; /* The number of slots in the hash table. */ unsigned int size; /* The number of entries in the hash table. */ unsigned int count; /* The size of elements. */ unsigned int entsize; /* If non-zero, don't grow the hash table. */ unsigned int frozen:1; }; /* Initialize a hash table. */ extern bfd_boolean bfd_hash_table_init (struct bfd_hash_table *, struct bfd_hash_entry *(*) (struct bfd_hash_entry *, struct bfd_hash_table *, const char *), unsigned int); /* Initialize a hash table specifying a size. */ extern bfd_boolean bfd_hash_table_init_n (struct bfd_hash_table *, struct bfd_hash_entry *(*) (struct bfd_hash_entry *, struct bfd_hash_table *, const char *), unsigned int, unsigned int); /* Free up a hash table. */ extern void bfd_hash_table_free (struct bfd_hash_table *); /* Look up a string in a hash table. If CREATE is TRUE, a new entry will be created for this string if one does not already exist. The COPY argument must be TRUE if this routine should copy the string into newly allocated memory when adding an entry. */ extern struct bfd_hash_entry *bfd_hash_lookup (struct bfd_hash_table *, const char *, bfd_boolean create, bfd_boolean copy); /* Replace an entry in a hash table. */ extern void bfd_hash_replace (struct bfd_hash_table *, struct bfd_hash_entry *old, struct bfd_hash_entry *nw); /* Base method for creating a hash table entry. */ extern struct bfd_hash_entry *bfd_hash_newfunc (struct bfd_hash_entry *, struct bfd_hash_table *, const char *); /* Grab some space for a hash table entry. */ extern void *bfd_hash_allocate (struct bfd_hash_table *, unsigned int); /* Traverse a hash table in a random order, calling a function on each element. If the function returns FALSE, the traversal stops. The INFO argument is passed to the function. */ extern void bfd_hash_traverse (struct bfd_hash_table *, bfd_boolean (*) (struct bfd_hash_entry *, void *), void *info); /* Allows the default size of a hash table to be configured. New hash tables allocated using bfd_hash_table_init will be created with this size. */ extern void bfd_hash_set_default_size (bfd_size_type); /* This structure is used to keep track of stabs in sections information while linking. */ struct stab_info { /* A hash table used to hold stabs strings. */ struct bfd_strtab_hash *strings; /* The header file hash table. */ struct bfd_hash_table includes; /* The first .stabstr section. */ struct bfd_section *stabstr; }; #define COFF_SWAP_TABLE (void *) &bfd_coff_std_swap_table /* User program access to BFD facilities. */ /* Direct I/O routines, for programs which know more about the object file than BFD does. Use higher level routines if possible. */ extern bfd_size_type bfd_bread (void *, bfd_size_type, bfd *); extern bfd_size_type bfd_bwrite (const void *, bfd_size_type, bfd *); extern int bfd_seek (bfd *, file_ptr, int); extern file_ptr bfd_tell (bfd *); extern int bfd_flush (bfd *); extern int bfd_stat (bfd *, struct stat *); /* Deprecated old routines. */ #if __GNUC__ #define bfd_read(BUF, ELTSIZE, NITEMS, ABFD) \ (warn_deprecated ("bfd_read", __FILE__, __LINE__, __FUNCTION__), \ bfd_bread ((BUF), (ELTSIZE) * (NITEMS), (ABFD))) #define bfd_write(BUF, ELTSIZE, NITEMS, ABFD) \ (warn_deprecated ("bfd_write", __FILE__, __LINE__, __FUNCTION__), \ bfd_bwrite ((BUF), (ELTSIZE) * (NITEMS), (ABFD))) #else #define bfd_read(BUF, ELTSIZE, NITEMS, ABFD) \ (warn_deprecated ("bfd_read", (const char *) 0, 0, (const char *) 0), \ bfd_bread ((BUF), (ELTSIZE) * (NITEMS), (ABFD))) #define bfd_write(BUF, ELTSIZE, NITEMS, ABFD) \ (warn_deprecated ("bfd_write", (const char *) 0, 0, (const char *) 0),\ bfd_bwrite ((BUF), (ELTSIZE) * (NITEMS), (ABFD))) #endif extern void warn_deprecated (const char *, const char *, int, const char *); /* Cast from const char * to char * so that caller can assign to a char * without a warning. */ #define bfd_get_filename(abfd) ((char *) (abfd)->filename) #define bfd_get_cacheable(abfd) ((abfd)->cacheable) #define bfd_get_format(abfd) ((abfd)->format) #define bfd_get_target(abfd) ((abfd)->xvec->name) #define bfd_get_flavour(abfd) ((abfd)->xvec->flavour) #define bfd_family_coff(abfd) \ (bfd_get_flavour (abfd) == bfd_target_coff_flavour || \ bfd_get_flavour (abfd) == bfd_target_xcoff_flavour) #define bfd_big_endian(abfd) ((abfd)->xvec->byteorder == BFD_ENDIAN_BIG) #define bfd_little_endian(abfd) ((abfd)->xvec->byteorder == BFD_ENDIAN_LITTLE) #define bfd_header_big_endian(abfd) \ ((abfd)->xvec->header_byteorder == BFD_ENDIAN_BIG) #define bfd_header_little_endian(abfd) \ ((abfd)->xvec->header_byteorder == BFD_ENDIAN_LITTLE) #define bfd_get_file_flags(abfd) ((abfd)->flags) #define bfd_applicable_file_flags(abfd) ((abfd)->xvec->object_flags) #define bfd_applicable_section_flags(abfd) ((abfd)->xvec->section_flags) #define bfd_my_archive(abfd) ((abfd)->my_archive) #define bfd_has_map(abfd) ((abfd)->has_armap) #define bfd_valid_reloc_types(abfd) ((abfd)->xvec->valid_reloc_types) #define bfd_usrdata(abfd) ((abfd)->usrdata) #define bfd_get_start_address(abfd) ((abfd)->start_address) #define bfd_get_symcount(abfd) ((abfd)->symcount) #define bfd_get_outsymbols(abfd) ((abfd)->outsymbols) #define bfd_count_sections(abfd) ((abfd)->section_count) #define bfd_get_dynamic_symcount(abfd) ((abfd)->dynsymcount) #define bfd_get_symbol_leading_char(abfd) ((abfd)->xvec->symbol_leading_char) #define bfd_set_cacheable(abfd,bool) (((abfd)->cacheable = bool), TRUE) extern bfd_boolean bfd_cache_close (bfd *abfd); /* NB: This declaration should match the autogenerated one in libbfd.h. */ extern bfd_boolean bfd_cache_close_all (void); extern bfd_boolean bfd_record_phdr (bfd *, unsigned long, bfd_boolean, flagword, bfd_boolean, bfd_vma, bfd_boolean, bfd_boolean, unsigned int, struct bfd_section **); /* Byte swapping routines. */ bfd_uint64_t bfd_getb64 (const void *); bfd_uint64_t bfd_getl64 (const void *); bfd_int64_t bfd_getb_signed_64 (const void *); bfd_int64_t bfd_getl_signed_64 (const void *); bfd_vma bfd_getb32 (const void *); bfd_vma bfd_getl32 (const void *); bfd_signed_vma bfd_getb_signed_32 (const void *); bfd_signed_vma bfd_getl_signed_32 (const void *); bfd_vma bfd_getb16 (const void *); bfd_vma bfd_getl16 (const void *); bfd_signed_vma bfd_getb_signed_16 (const void *); bfd_signed_vma bfd_getl_signed_16 (const void *); void bfd_putb64 (bfd_uint64_t, void *); void bfd_putl64 (bfd_uint64_t, void *); void bfd_putb32 (bfd_vma, void *); void bfd_putl32 (bfd_vma, void *); void bfd_putb16 (bfd_vma, void *); void bfd_putl16 (bfd_vma, void *); /* Byte swapping routines which take size and endiannes as arguments. */ bfd_uint64_t bfd_get_bits (const void *, int, bfd_boolean); void bfd_put_bits (bfd_uint64_t, void *, int, bfd_boolean); extern bfd_boolean bfd_section_already_linked_table_init (void); extern void bfd_section_already_linked_table_free (void); /* Externally visible ECOFF routines. */ #if defined(__STDC__) || defined(ALMOST_STDC) struct ecoff_debug_info; struct ecoff_debug_swap; struct ecoff_extr; struct bfd_symbol; struct bfd_link_info; struct bfd_link_hash_entry; struct bfd_elf_version_tree; #endif extern bfd_vma bfd_ecoff_get_gp_value (bfd * abfd); extern bfd_boolean bfd_ecoff_set_gp_value (bfd *abfd, bfd_vma gp_value); extern bfd_boolean bfd_ecoff_set_regmasks (bfd *abfd, unsigned long gprmask, unsigned long fprmask, unsigned long *cprmask); extern void *bfd_ecoff_debug_init (bfd *output_bfd, struct ecoff_debug_info *output_debug, const struct ecoff_debug_swap *output_swap, struct bfd_link_info *); extern void bfd_ecoff_debug_free (void *handle, bfd *output_bfd, struct ecoff_debug_info *output_debug, const struct ecoff_debug_swap *output_swap, struct bfd_link_info *); extern bfd_boolean bfd_ecoff_debug_accumulate (void *handle, bfd *output_bfd, struct ecoff_debug_info *output_debug, const struct ecoff_debug_swap *output_swap, bfd *input_bfd, struct ecoff_debug_info *input_debug, const struct ecoff_debug_swap *input_swap, struct bfd_link_info *); extern bfd_boolean bfd_ecoff_debug_accumulate_other (void *handle, bfd *output_bfd, struct ecoff_debug_info *output_debug, const struct ecoff_debug_swap *output_swap, bfd *input_bfd, struct bfd_link_info *); extern bfd_boolean bfd_ecoff_debug_externals (bfd *abfd, struct ecoff_debug_info *debug, const struct ecoff_debug_swap *swap, bfd_boolean relocatable, bfd_boolean (*get_extr) (struct bfd_symbol *, struct ecoff_extr *), void (*set_index) (struct bfd_symbol *, bfd_size_type)); extern bfd_boolean bfd_ecoff_debug_one_external (bfd *abfd, struct ecoff_debug_info *debug, const struct ecoff_debug_swap *swap, const char *name, struct ecoff_extr *esym); extern bfd_size_type bfd_ecoff_debug_size (bfd *abfd, struct ecoff_debug_info *debug, const struct ecoff_debug_swap *swap); extern bfd_boolean bfd_ecoff_write_debug (bfd *abfd, struct ecoff_debug_info *debug, const struct ecoff_debug_swap *swap, file_ptr where); extern bfd_boolean bfd_ecoff_write_accumulated_debug (void *handle, bfd *abfd, struct ecoff_debug_info *debug, const struct ecoff_debug_swap *swap, struct bfd_link_info *info, file_ptr where); /* Externally visible ELF routines. */ struct bfd_link_needed_list { struct bfd_link_needed_list *next; bfd *by; const char *name; }; enum dynamic_lib_link_class { DYN_NORMAL = 0, DYN_AS_NEEDED = 1, DYN_DT_NEEDED = 2, DYN_NO_ADD_NEEDED = 4, DYN_NO_NEEDED = 8 }; enum notice_asneeded_action { notice_as_needed, notice_not_needed, notice_needed }; extern bfd_boolean bfd_elf_record_link_assignment (bfd *, struct bfd_link_info *, const char *, bfd_boolean, bfd_boolean); extern struct bfd_link_needed_list *bfd_elf_get_needed_list (bfd *, struct bfd_link_info *); extern bfd_boolean bfd_elf_get_bfd_needed_list (bfd *, struct bfd_link_needed_list **); extern bfd_boolean bfd_elf_size_dynamic_sections (bfd *, const char *, const char *, const char *, const char * const *, struct bfd_link_info *, struct bfd_section **, struct bfd_elf_version_tree *); extern bfd_boolean bfd_elf_size_dynsym_hash_dynstr (bfd *, struct bfd_link_info *); extern void bfd_elf_set_dt_needed_name (bfd *, const char *); extern const char *bfd_elf_get_dt_soname (bfd *); extern void bfd_elf_set_dyn_lib_class (bfd *, enum dynamic_lib_link_class); extern int bfd_elf_get_dyn_lib_class (bfd *); extern struct bfd_link_needed_list *bfd_elf_get_runpath_list (bfd *, struct bfd_link_info *); extern bfd_boolean bfd_elf_discard_info (bfd *, struct bfd_link_info *); extern unsigned int _bfd_elf_default_action_discarded (struct bfd_section *); /* Return an upper bound on the number of bytes required to store a copy of ABFD's program header table entries. Return -1 if an error occurs; bfd_get_error will return an appropriate code. */ extern long bfd_get_elf_phdr_upper_bound (bfd *abfd); /* Copy ABFD's program header table entries to *PHDRS. The entries will be stored as an array of Elf_Internal_Phdr structures, as defined in include/elf/internal.h. To find out how large the buffer needs to be, call bfd_get_elf_phdr_upper_bound. Return the number of program header table entries read, or -1 if an error occurs; bfd_get_error will return an appropriate code. */ extern int bfd_get_elf_phdrs (bfd *abfd, void *phdrs); /* Create a new BFD as if by bfd_openr. Rather than opening a file, reconstruct an ELF file by reading the segments out of remote memory based on the ELF file header at EHDR_VMA and the ELF program headers it points to. If not null, *LOADBASEP is filled in with the difference between the VMAs from which the segments were read, and the VMAs the file headers (and hence BFD's idea of each section's VMA) put them at. The function TARGET_READ_MEMORY is called to copy LEN bytes from the remote memory at target address VMA into the local buffer at MYADDR; it should return zero on success or an `errno' code on failure. TEMPL must be a BFD for an ELF target with the word size and byte order found in the remote memory. */ extern bfd *bfd_elf_bfd_from_remote_memory (bfd *templ, bfd_vma ehdr_vma, bfd_vma *loadbasep, int (*target_read_memory) (bfd_vma vma, bfd_byte *myaddr, int len)); /* Return the arch_size field of an elf bfd, or -1 if not elf. */ extern int bfd_get_arch_size (bfd *); /* Return TRUE if address "naturally" sign extends, or -1 if not elf. */ extern int bfd_get_sign_extend_vma (bfd *); extern struct bfd_section *_bfd_elf_tls_setup (bfd *, struct bfd_link_info *); extern void _bfd_fix_excluded_sec_syms (bfd *, struct bfd_link_info *); extern unsigned bfd_m68k_mach_to_features (int); extern int bfd_m68k_features_to_mach (unsigned); extern bfd_boolean bfd_m68k_elf32_create_embedded_relocs (bfd *, struct bfd_link_info *, struct bfd_section *, struct bfd_section *, char **); extern bfd_boolean bfd_bfin_elf32_create_embedded_relocs (bfd *, struct bfd_link_info *, struct bfd_section *, struct bfd_section *, char **); /* SunOS shared library support routines for the linker. */ extern struct bfd_link_needed_list *bfd_sunos_get_needed_list (bfd *, struct bfd_link_info *); extern bfd_boolean bfd_sunos_record_link_assignment (bfd *, struct bfd_link_info *, const char *); extern bfd_boolean bfd_sunos_size_dynamic_sections (bfd *, struct bfd_link_info *, struct bfd_section **, struct bfd_section **, struct bfd_section **); /* Linux shared library support routines for the linker. */ extern bfd_boolean bfd_i386linux_size_dynamic_sections (bfd *, struct bfd_link_info *); extern bfd_boolean bfd_m68klinux_size_dynamic_sections (bfd *, struct bfd_link_info *); extern bfd_boolean bfd_sparclinux_size_dynamic_sections (bfd *, struct bfd_link_info *); /* mmap hacks */ struct _bfd_window_internal; typedef struct _bfd_window_internal bfd_window_internal; typedef struct _bfd_window { /* What the user asked for. */ void *data; bfd_size_type size; /* The actual window used by BFD. Small user-requested read-only regions sharing a page may share a single window into the object file. Read-write versions shouldn't until I've fixed things to keep track of which portions have been claimed by the application; don't want to give the same region back when the application wants two writable copies! */ struct _bfd_window_internal *i; } bfd_window; extern void bfd_init_window (bfd_window *); extern void bfd_free_window (bfd_window *); extern bfd_boolean bfd_get_file_window (bfd *, file_ptr, bfd_size_type, bfd_window *, bfd_boolean); /* XCOFF support routines for the linker. */ extern bfd_boolean bfd_xcoff_link_record_set (bfd *, struct bfd_link_info *, struct bfd_link_hash_entry *, bfd_size_type); extern bfd_boolean bfd_xcoff_import_symbol (bfd *, struct bfd_link_info *, struct bfd_link_hash_entry *, bfd_vma, const char *, const char *, const char *, unsigned int); extern bfd_boolean bfd_xcoff_export_symbol (bfd *, struct bfd_link_info *, struct bfd_link_hash_entry *); extern bfd_boolean bfd_xcoff_link_count_reloc (bfd *, struct bfd_link_info *, const char *); extern bfd_boolean bfd_xcoff_record_link_assignment (bfd *, struct bfd_link_info *, const char *); extern bfd_boolean bfd_xcoff_size_dynamic_sections (bfd *, struct bfd_link_info *, const char *, const char *, unsigned long, unsigned long, unsigned long, bfd_boolean, int, bfd_boolean, bfd_boolean, struct bfd_section **, bfd_boolean); extern bfd_boolean bfd_xcoff_link_generate_rtinit (bfd *, const char *, const char *, bfd_boolean); /* XCOFF support routines for ar. */ extern bfd_boolean bfd_xcoff_ar_archive_set_magic (bfd *, char *); /* Externally visible COFF routines. */ #if defined(__STDC__) || defined(ALMOST_STDC) struct internal_syment; union internal_auxent; #endif extern bfd_boolean bfd_coff_get_syment (bfd *, struct bfd_symbol *, struct internal_syment *); extern bfd_boolean bfd_coff_get_auxent (bfd *, struct bfd_symbol *, int, union internal_auxent *); extern bfd_boolean bfd_coff_set_symbol_class (bfd *, struct bfd_symbol *, unsigned int); extern bfd_boolean bfd_m68k_coff_create_embedded_relocs (bfd *, struct bfd_link_info *, struct bfd_section *, struct bfd_section *, char **); /* ARM VFP11 erratum workaround support. */ typedef enum { BFD_ARM_VFP11_FIX_DEFAULT, BFD_ARM_VFP11_FIX_NONE, BFD_ARM_VFP11_FIX_SCALAR, BFD_ARM_VFP11_FIX_VECTOR } bfd_arm_vfp11_fix; extern void bfd_elf32_arm_init_maps (bfd *); extern void bfd_elf32_arm_set_vfp11_fix (bfd *, struct bfd_link_info *); extern bfd_boolean bfd_elf32_arm_vfp11_erratum_scan (bfd *, struct bfd_link_info *); extern void bfd_elf32_arm_vfp11_fix_veneer_locations (bfd *, struct bfd_link_info *); /* ARM Interworking support. Called from linker. */ extern bfd_boolean bfd_arm_allocate_interworking_sections (struct bfd_link_info *); extern bfd_boolean bfd_arm_process_before_allocation (bfd *, struct bfd_link_info *, int); extern bfd_boolean bfd_arm_get_bfd_for_interworking (bfd *, struct bfd_link_info *); /* PE ARM Interworking support. Called from linker. */ extern bfd_boolean bfd_arm_pe_allocate_interworking_sections (struct bfd_link_info *); extern bfd_boolean bfd_arm_pe_process_before_allocation (bfd *, struct bfd_link_info *, int); extern bfd_boolean bfd_arm_pe_get_bfd_for_interworking (bfd *, struct bfd_link_info *); /* ELF ARM Interworking support. Called from linker. */ extern bfd_boolean bfd_elf32_arm_allocate_interworking_sections (struct bfd_link_info *); extern bfd_boolean bfd_elf32_arm_process_before_allocation (bfd *, struct bfd_link_info *); void bfd_elf32_arm_set_target_relocs (bfd *, struct bfd_link_info *, int, char *, int, int, bfd_arm_vfp11_fix, int, int); extern bfd_boolean bfd_elf32_arm_get_bfd_for_interworking (bfd *, struct bfd_link_info *); extern bfd_boolean bfd_elf32_arm_add_glue_sections_to_bfd (bfd *, struct bfd_link_info *); /* ELF ARM mapping symbol support */ #define BFD_ARM_SPECIAL_SYM_TYPE_MAP (1 << 0) #define BFD_ARM_SPECIAL_SYM_TYPE_TAG (1 << 1) #define BFD_ARM_SPECIAL_SYM_TYPE_OTHER (1 << 2) #define BFD_ARM_SPECIAL_SYM_TYPE_ANY (~0) extern bfd_boolean bfd_is_arm_special_symbol_name (const char * name, int type); extern void bfd_elf32_arm_set_byteswap_code (struct bfd_link_info *, int); /* ARM Note section processing. */ extern bfd_boolean bfd_arm_merge_machines (bfd *, bfd *); extern bfd_boolean bfd_arm_update_notes (bfd *, const char *); extern unsigned int bfd_arm_get_mach_from_notes (bfd *, const char *); /* TI COFF load page support. */ extern void bfd_ticoff_set_section_load_page (struct bfd_section *, int); extern int bfd_ticoff_get_section_load_page (struct bfd_section *); /* H8/300 functions. */ extern bfd_vma bfd_h8300_pad_address (bfd *, bfd_vma); /* IA64 Itanium code generation. Called from linker. */ extern void bfd_elf32_ia64_after_parse (int); extern void bfd_elf64_ia64_after_parse (int); /* This structure is used for a comdat section, as in PE. A comdat section is associated with a particular symbol. When the linker sees a comdat section, it keeps only one of the sections with a given name and associated with a given symbol. */ struct coff_comdat_info { /* The name of the symbol associated with a comdat section. */ const char *name; /* The local symbol table index of the symbol associated with a comdat section. This is only meaningful to the object file format specific code; it is not an index into the list returned by bfd_canonicalize_symtab. */ long symbol; }; extern struct coff_comdat_info *bfd_coff_get_comdat_section (bfd *, struct bfd_section *); /* Extracted from init.c. */ void bfd_init (void); /* Extracted from opncls.c. */ bfd *bfd_fopen (const char *filename, const char *target, const char *mode, int fd); bfd *bfd_openr (const char *filename, const char *target); bfd *bfd_fdopenr (const char *filename, const char *target, int fd); bfd *bfd_openstreamr (const char *, const char *, void *); bfd *bfd_openr_iovec (const char *filename, const char *target, void *(*open) (struct bfd *nbfd, void *open_closure), void *open_closure, file_ptr (*pread) (struct bfd *nbfd, void *stream, void *buf, file_ptr nbytes, file_ptr offset), int (*close) (struct bfd *nbfd, void *stream), int (*stat) (struct bfd *abfd, void *stream, struct stat *sb)); bfd *bfd_openw (const char *filename, const char *target); bfd_boolean bfd_close (bfd *abfd); bfd_boolean bfd_close_all_done (bfd *); bfd *bfd_create (const char *filename, bfd *templ); bfd_boolean bfd_make_writable (bfd *abfd); bfd_boolean bfd_make_readable (bfd *abfd); unsigned long bfd_calc_gnu_debuglink_crc32 (unsigned long crc, const unsigned char *buf, bfd_size_type len); char *bfd_follow_gnu_debuglink (bfd *abfd, const char *dir); struct bfd_section *bfd_create_gnu_debuglink_section (bfd *abfd, const char *filename); bfd_boolean bfd_fill_in_gnu_debuglink_section (bfd *abfd, struct bfd_section *sect, const char *filename); /* Extracted from libbfd.c. */ /* Byte swapping macros for user section data. */ #define bfd_put_8(abfd, val, ptr) \ ((void) (*((unsigned char *) (ptr)) = (val) & 0xff)) #define bfd_put_signed_8 \ bfd_put_8 #define bfd_get_8(abfd, ptr) \ (*(unsigned char *) (ptr) & 0xff) #define bfd_get_signed_8(abfd, ptr) \ (((*(unsigned char *) (ptr) & 0xff) ^ 0x80) - 0x80) #define bfd_put_16(abfd, val, ptr) \ BFD_SEND (abfd, bfd_putx16, ((val),(ptr))) #define bfd_put_signed_16 \ bfd_put_16 #define bfd_get_16(abfd, ptr) \ BFD_SEND (abfd, bfd_getx16, (ptr)) #define bfd_get_signed_16(abfd, ptr) \ BFD_SEND (abfd, bfd_getx_signed_16, (ptr)) #define bfd_put_32(abfd, val, ptr) \ BFD_SEND (abfd, bfd_putx32, ((val),(ptr))) #define bfd_put_signed_32 \ bfd_put_32 #define bfd_get_32(abfd, ptr) \ BFD_SEND (abfd, bfd_getx32, (ptr)) #define bfd_get_signed_32(abfd, ptr) \ BFD_SEND (abfd, bfd_getx_signed_32, (ptr)) #define bfd_put_64(abfd, val, ptr) \ BFD_SEND (abfd, bfd_putx64, ((val), (ptr))) #define bfd_put_signed_64 \ bfd_put_64 #define bfd_get_64(abfd, ptr) \ BFD_SEND (abfd, bfd_getx64, (ptr)) #define bfd_get_signed_64(abfd, ptr) \ BFD_SEND (abfd, bfd_getx_signed_64, (ptr)) #define bfd_get(bits, abfd, ptr) \ ((bits) == 8 ? (bfd_vma) bfd_get_8 (abfd, ptr) \ : (bits) == 16 ? bfd_get_16 (abfd, ptr) \ : (bits) == 32 ? bfd_get_32 (abfd, ptr) \ : (bits) == 64 ? bfd_get_64 (abfd, ptr) \ : (abort (), (bfd_vma) - 1)) #define bfd_put(bits, abfd, val, ptr) \ ((bits) == 8 ? bfd_put_8 (abfd, val, ptr) \ : (bits) == 16 ? bfd_put_16 (abfd, val, ptr) \ : (bits) == 32 ? bfd_put_32 (abfd, val, ptr) \ : (bits) == 64 ? bfd_put_64 (abfd, val, ptr) \ : (abort (), (void) 0)) /* Byte swapping macros for file header data. */ #define bfd_h_put_8(abfd, val, ptr) \ bfd_put_8 (abfd, val, ptr) #define bfd_h_put_signed_8(abfd, val, ptr) \ bfd_put_8 (abfd, val, ptr) #define bfd_h_get_8(abfd, ptr) \ bfd_get_8 (abfd, ptr) #define bfd_h_get_signed_8(abfd, ptr) \ bfd_get_signed_8 (abfd, ptr) #define bfd_h_put_16(abfd, val, ptr) \ BFD_SEND (abfd, bfd_h_putx16, (val, ptr)) #define bfd_h_put_signed_16 \ bfd_h_put_16 #define bfd_h_get_16(abfd, ptr) \ BFD_SEND (abfd, bfd_h_getx16, (ptr)) #define bfd_h_get_signed_16(abfd, ptr) \ BFD_SEND (abfd, bfd_h_getx_signed_16, (ptr)) #define bfd_h_put_32(abfd, val, ptr) \ BFD_SEND (abfd, bfd_h_putx32, (val, ptr)) #define bfd_h_put_signed_32 \ bfd_h_put_32 #define bfd_h_get_32(abfd, ptr) \ BFD_SEND (abfd, bfd_h_getx32, (ptr)) #define bfd_h_get_signed_32(abfd, ptr) \ BFD_SEND (abfd, bfd_h_getx_signed_32, (ptr)) #define bfd_h_put_64(abfd, val, ptr) \ BFD_SEND (abfd, bfd_h_putx64, (val, ptr)) #define bfd_h_put_signed_64 \ bfd_h_put_64 #define bfd_h_get_64(abfd, ptr) \ BFD_SEND (abfd, bfd_h_getx64, (ptr)) #define bfd_h_get_signed_64(abfd, ptr) \ BFD_SEND (abfd, bfd_h_getx_signed_64, (ptr)) /* Aliases for the above, which should eventually go away. */ #define H_PUT_64 bfd_h_put_64 #define H_PUT_32 bfd_h_put_32 #define H_PUT_16 bfd_h_put_16 #define H_PUT_8 bfd_h_put_8 #define H_PUT_S64 bfd_h_put_signed_64 #define H_PUT_S32 bfd_h_put_signed_32 #define H_PUT_S16 bfd_h_put_signed_16 #define H_PUT_S8 bfd_h_put_signed_8 #define H_GET_64 bfd_h_get_64 #define H_GET_32 bfd_h_get_32 #define H_GET_16 bfd_h_get_16 #define H_GET_8 bfd_h_get_8 #define H_GET_S64 bfd_h_get_signed_64 #define H_GET_S32 bfd_h_get_signed_32 #define H_GET_S16 bfd_h_get_signed_16 #define H_GET_S8 bfd_h_get_signed_8 /* Extracted from bfdio.c. */ long bfd_get_mtime (bfd *abfd); file_ptr bfd_get_size (bfd *abfd); /* Extracted from bfdwin.c. */ /* Extracted from section.c. */ typedef struct bfd_section { /* The name of the section; the name isn't a copy, the pointer is the same as that passed to bfd_make_section. */ const char *name; /* A unique sequence number. */ int id; /* Which section in the bfd; 0..n-1 as sections are created in a bfd. */ int index; /* The next section in the list belonging to the BFD, or NULL. */ struct bfd_section *next; /* The previous section in the list belonging to the BFD, or NULL. */ struct bfd_section *prev; /* The field flags contains attributes of the section. Some flags are read in from the object file, and some are synthesized from other information. */ flagword flags; #define SEC_NO_FLAGS 0x000 /* Tells the OS to allocate space for this section when loading. This is clear for a section containing debug information only. */ #define SEC_ALLOC 0x001 /* Tells the OS to load the section from the file when loading. This is clear for a .bss section. */ #define SEC_LOAD 0x002 /* The section contains data still to be relocated, so there is some relocation information too. */ #define SEC_RELOC 0x004 /* A signal to the OS that the section contains read only data. */ #define SEC_READONLY 0x008 /* The section contains code only. */ #define SEC_CODE 0x010 /* The section contains data only. */ #define SEC_DATA 0x020 /* The section will reside in ROM. */ #define SEC_ROM 0x040 /* The section contains constructor information. This section type is used by the linker to create lists of constructors and destructors used by <>. When a back end sees a symbol which should be used in a constructor list, it creates a new section for the type of name (e.g., <<__CTOR_LIST__>>), attaches the symbol to it, and builds a relocation. To build the lists of constructors, all the linker has to do is catenate all the sections called <<__CTOR_LIST__>> and relocate the data contained within - exactly the operations it would peform on standard data. */ #define SEC_CONSTRUCTOR 0x080 /* The section has contents - a data section could be <> | <>; a debug section could be <> */ #define SEC_HAS_CONTENTS 0x100 /* An instruction to the linker to not output the section even if it has information which would normally be written. */ #define SEC_NEVER_LOAD 0x200 /* The section contains thread local data. */ #define SEC_THREAD_LOCAL 0x400 /* The section has GOT references. This flag is only for the linker, and is currently only used by the elf32-hppa back end. It will be set if global offset table references were detected in this section, which indicate to the linker that the section contains PIC code, and must be handled specially when doing a static link. */ #define SEC_HAS_GOT_REF 0x800 /* The section contains common symbols (symbols may be defined multiple times, the value of a symbol is the amount of space it requires, and the largest symbol value is the one used). Most targets have exactly one of these (which we translate to bfd_com_section_ptr), but ECOFF has two. */ #define SEC_IS_COMMON 0x1000 /* The section contains only debugging information. For example, this is set for ELF .debug and .stab sections. strip tests this flag to see if a section can be discarded. */ #define SEC_DEBUGGING 0x2000 /* The contents of this section are held in memory pointed to by the contents field. This is checked by bfd_get_section_contents, and the data is retrieved from memory if appropriate. */ #define SEC_IN_MEMORY 0x4000 /* The contents of this section are to be excluded by the linker for executable and shared objects unless those objects are to be further relocated. */ #define SEC_EXCLUDE 0x8000 /* The contents of this section are to be sorted based on the sum of the symbol and addend values specified by the associated relocation entries. Entries without associated relocation entries will be appended to the end of the section in an unspecified order. */ #define SEC_SORT_ENTRIES 0x10000 /* When linking, duplicate sections of the same name should be discarded, rather than being combined into a single section as is usually done. This is similar to how common symbols are handled. See SEC_LINK_DUPLICATES below. */ #define SEC_LINK_ONCE 0x20000 /* If SEC_LINK_ONCE is set, this bitfield describes how the linker should handle duplicate sections. */ #define SEC_LINK_DUPLICATES 0x40000 /* This value for SEC_LINK_DUPLICATES means that duplicate sections with the same name should simply be discarded. */ #define SEC_LINK_DUPLICATES_DISCARD 0x0 /* This value for SEC_LINK_DUPLICATES means that the linker should warn if there are any duplicate sections, although it should still only link one copy. */ #define SEC_LINK_DUPLICATES_ONE_ONLY 0x80000 /* This value for SEC_LINK_DUPLICATES means that the linker should warn if any duplicate sections are a different size. */ #define SEC_LINK_DUPLICATES_SAME_SIZE 0x100000 /* This value for SEC_LINK_DUPLICATES means that the linker should warn if any duplicate sections contain different contents. */ #define SEC_LINK_DUPLICATES_SAME_CONTENTS \ (SEC_LINK_DUPLICATES_ONE_ONLY | SEC_LINK_DUPLICATES_SAME_SIZE) /* This section was created by the linker as part of dynamic relocation or other arcane processing. It is skipped when going through the first-pass output, trusting that someone else up the line will take care of it later. */ #define SEC_LINKER_CREATED 0x200000 /* This section should not be subject to garbage collection. Also set to inform the linker that this section should not be listed in the link map as discarded. */ #define SEC_KEEP 0x400000 /* This section contains "short" data, and should be placed "near" the GP. */ #define SEC_SMALL_DATA 0x800000 /* Attempt to merge identical entities in the section. Entity size is given in the entsize field. */ #define SEC_MERGE 0x1000000 /* If given with SEC_MERGE, entities to merge are zero terminated strings where entsize specifies character size instead of fixed size entries. */ #define SEC_STRINGS 0x2000000 /* This section contains data about section groups. */ #define SEC_GROUP 0x4000000 /* The section is a COFF shared library section. This flag is only for the linker. If this type of section appears in the input file, the linker must copy it to the output file without changing the vma or size. FIXME: Although this was originally intended to be general, it really is COFF specific (and the flag was renamed to indicate this). It might be cleaner to have some more general mechanism to allow the back end to control what the linker does with sections. */ #define SEC_COFF_SHARED_LIBRARY 0x10000000 /* This section contains data which may be shared with other executables or shared objects. This is for COFF only. */ #define SEC_COFF_SHARED 0x20000000 /* When a section with this flag is being linked, then if the size of the input section is less than a page, it should not cross a page boundary. If the size of the input section is one page or more, it should be aligned on a page boundary. This is for TI TMS320C54X only. */ #define SEC_TIC54X_BLOCK 0x40000000 /* Conditionally link this section; do not link if there are no references found to any symbol in the section. This is for TI TMS320C54X only. */ #define SEC_TIC54X_CLINK 0x80000000 /* End of section flags. */ /* Some internal packed boolean fields. */ /* See the vma field. */ unsigned int user_set_vma : 1; /* A mark flag used by some of the linker backends. */ unsigned int linker_mark : 1; /* Another mark flag used by some of the linker backends. Set for output sections that have an input section. */ unsigned int linker_has_input : 1; /* Mark flags used by some linker backends for garbage collection. */ unsigned int gc_mark : 1; unsigned int gc_mark_from_eh : 1; /* The following flags are used by the ELF linker. */ /* Mark sections which have been allocated to segments. */ unsigned int segment_mark : 1; /* Type of sec_info information. */ unsigned int sec_info_type:3; #define ELF_INFO_TYPE_NONE 0 #define ELF_INFO_TYPE_STABS 1 #define ELF_INFO_TYPE_MERGE 2 #define ELF_INFO_TYPE_EH_FRAME 3 #define ELF_INFO_TYPE_JUST_SYMS 4 /* Nonzero if this section uses RELA relocations, rather than REL. */ unsigned int use_rela_p:1; /* Bits used by various backends. The generic code doesn't touch these fields. */ /* Nonzero if this section has TLS related relocations. */ unsigned int has_tls_reloc:1; + /* Nonzero if this section has a call to __tls_get_addr. */ + unsigned int has_tls_get_addr_call:1; + /* Nonzero if this section has a gp reloc. */ unsigned int has_gp_reloc:1; /* Nonzero if this section needs the relax finalize pass. */ unsigned int need_finalize_relax:1; /* Whether relocations have been processed. */ unsigned int reloc_done : 1; /* End of internal packed boolean fields. */ /* The virtual memory address of the section - where it will be at run time. The symbols are relocated against this. The user_set_vma flag is maintained by bfd; if it's not set, the backend can assign addresses (for example, in <>, where the default address for <<.data>> is dependent on the specific target and various flags). */ bfd_vma vma; /* The load address of the section - where it would be in a rom image; really only used for writing section header information. */ bfd_vma lma; /* The size of the section in octets, as it will be output. Contains a value even if the section has no contents (e.g., the size of <<.bss>>). */ bfd_size_type size; /* For input sections, the original size on disk of the section, in octets. This field is used by the linker relaxation code. It is currently only set for sections where the linker relaxation scheme doesn't cache altered section and reloc contents (stabs, eh_frame, SEC_MERGE, some coff relaxing targets), and thus the original size needs to be kept to read the section multiple times. For output sections, rawsize holds the section size calculated on a previous linker relaxation pass. */ bfd_size_type rawsize; /* If this section is going to be output, then this value is the offset in *bytes* into the output section of the first byte in the input section (byte ==> smallest addressable unit on the target). In most cases, if this was going to start at the 100th octet (8-bit quantity) in the output section, this value would be 100. However, if the target byte size is 16 bits (bfd_octets_per_byte is "2"), this value would be 50. */ bfd_vma output_offset; /* The output section through which to map on output. */ struct bfd_section *output_section; /* The alignment requirement of the section, as an exponent of 2 - e.g., 3 aligns to 2^3 (or 8). */ unsigned int alignment_power; /* If an input section, a pointer to a vector of relocation records for the data in this section. */ struct reloc_cache_entry *relocation; /* If an output section, a pointer to a vector of pointers to relocation records for the data in this section. */ struct reloc_cache_entry **orelocation; /* The number of relocation records in one of the above. */ unsigned reloc_count; /* Information below is back end specific - and not always used or updated. */ /* File position of section data. */ file_ptr filepos; /* File position of relocation info. */ file_ptr rel_filepos; /* File position of line data. */ file_ptr line_filepos; /* Pointer to data for applications. */ void *userdata; /* If the SEC_IN_MEMORY flag is set, this points to the actual contents. */ unsigned char *contents; /* Attached line number information. */ alent *lineno; /* Number of line number records. */ unsigned int lineno_count; /* Entity size for merging purposes. */ unsigned int entsize; /* Points to the kept section if this section is a link-once section, and is discarded. */ struct bfd_section *kept_section; /* When a section is being output, this value changes as more linenumbers are written out. */ file_ptr moving_line_filepos; /* What the section number is in the target world. */ int target_index; void *used_by_bfd; /* If this is a constructor section then here is a list of the relocations created to relocate items within it. */ struct relent_chain *constructor_chain; /* The BFD which owns the section. */ bfd *owner; /* A symbol which points at this section only. */ struct bfd_symbol *symbol; struct bfd_symbol **symbol_ptr_ptr; /* Early in the link process, map_head and map_tail are used to build a list of input sections attached to an output section. Later, output sections use these fields for a list of bfd_link_order structs. */ union { struct bfd_link_order *link_order; struct bfd_section *s; } map_head, map_tail; } asection; /* These sections are global, and are managed by BFD. The application and target back end are not permitted to change the values in these sections. New code should use the section_ptr macros rather than referring directly to the const sections. The const sections may eventually vanish. */ #define BFD_ABS_SECTION_NAME "*ABS*" #define BFD_UND_SECTION_NAME "*UND*" #define BFD_COM_SECTION_NAME "*COM*" #define BFD_IND_SECTION_NAME "*IND*" /* The absolute section. */ extern asection bfd_abs_section; #define bfd_abs_section_ptr ((asection *) &bfd_abs_section) #define bfd_is_abs_section(sec) ((sec) == bfd_abs_section_ptr) /* Pointer to the undefined section. */ extern asection bfd_und_section; #define bfd_und_section_ptr ((asection *) &bfd_und_section) #define bfd_is_und_section(sec) ((sec) == bfd_und_section_ptr) /* Pointer to the common section. */ extern asection bfd_com_section; #define bfd_com_section_ptr ((asection *) &bfd_com_section) /* Pointer to the indirect section. */ extern asection bfd_ind_section; #define bfd_ind_section_ptr ((asection *) &bfd_ind_section) #define bfd_is_ind_section(sec) ((sec) == bfd_ind_section_ptr) #define bfd_is_const_section(SEC) \ ( ((SEC) == bfd_abs_section_ptr) \ || ((SEC) == bfd_und_section_ptr) \ || ((SEC) == bfd_com_section_ptr) \ || ((SEC) == bfd_ind_section_ptr)) /* Macros to handle insertion and deletion of a bfd's sections. These only handle the list pointers, ie. do not adjust section_count, target_index etc. */ #define bfd_section_list_remove(ABFD, S) \ do \ { \ asection *_s = S; \ asection *_next = _s->next; \ asection *_prev = _s->prev; \ if (_prev) \ _prev->next = _next; \ else \ (ABFD)->sections = _next; \ if (_next) \ _next->prev = _prev; \ else \ (ABFD)->section_last = _prev; \ } \ while (0) #define bfd_section_list_append(ABFD, S) \ do \ { \ asection *_s = S; \ bfd *_abfd = ABFD; \ _s->next = NULL; \ if (_abfd->section_last) \ { \ _s->prev = _abfd->section_last; \ _abfd->section_last->next = _s; \ } \ else \ { \ _s->prev = NULL; \ _abfd->sections = _s; \ } \ _abfd->section_last = _s; \ } \ while (0) #define bfd_section_list_prepend(ABFD, S) \ do \ { \ asection *_s = S; \ bfd *_abfd = ABFD; \ _s->prev = NULL; \ if (_abfd->sections) \ { \ _s->next = _abfd->sections; \ _abfd->sections->prev = _s; \ } \ else \ { \ _s->next = NULL; \ _abfd->section_last = _s; \ } \ _abfd->sections = _s; \ } \ while (0) #define bfd_section_list_insert_after(ABFD, A, S) \ do \ { \ asection *_a = A; \ asection *_s = S; \ asection *_next = _a->next; \ _s->next = _next; \ _s->prev = _a; \ _a->next = _s; \ if (_next) \ _next->prev = _s; \ else \ (ABFD)->section_last = _s; \ } \ while (0) #define bfd_section_list_insert_before(ABFD, B, S) \ do \ { \ asection *_b = B; \ asection *_s = S; \ asection *_prev = _b->prev; \ _s->prev = _prev; \ _s->next = _b; \ _b->prev = _s; \ if (_prev) \ _prev->next = _s; \ else \ (ABFD)->sections = _s; \ } \ while (0) #define bfd_section_removed_from_list(ABFD, S) \ ((S)->next == NULL ? (ABFD)->section_last != (S) : (S)->next->prev != (S)) #define BFD_FAKE_SECTION(SEC, FLAGS, SYM, NAME, IDX) \ /* name, id, index, next, prev, flags, user_set_vma, */ \ { NAME, IDX, 0, NULL, NULL, FLAGS, 0, \ \ /* linker_mark, linker_has_input, gc_mark, gc_mark_from_eh, */ \ 0, 0, 1, 0, \ \ /* segment_mark, sec_info_type, use_rela_p, has_tls_reloc, */ \ 0, 0, 0, 0, \ \ - /* has_gp_reloc, need_finalize_relax, reloc_done, */ \ - 0, 0, 0, \ + /* has_tls_get_addr_call, has_gp_reloc, need_finalize_relax, */ \ + 0, 0, 0, \ \ - /* vma, lma, size, rawsize */ \ - 0, 0, 0, 0, \ + /* reloc_done, vma, lma, size, rawsize */ \ + 0, 0, 0, 0, 0, \ \ /* output_offset, output_section, alignment_power, */ \ 0, (struct bfd_section *) &SEC, 0, \ \ /* relocation, orelocation, reloc_count, filepos, rel_filepos, */ \ NULL, NULL, 0, 0, 0, \ \ /* line_filepos, userdata, contents, lineno, lineno_count, */ \ 0, NULL, NULL, NULL, 0, \ \ /* entsize, kept_section, moving_line_filepos, */ \ 0, NULL, 0, \ \ /* target_index, used_by_bfd, constructor_chain, owner, */ \ 0, NULL, NULL, NULL, \ \ /* symbol, symbol_ptr_ptr, */ \ (struct bfd_symbol *) SYM, &SEC.symbol, \ \ /* map_head, map_tail */ \ { NULL }, { NULL } \ } void bfd_section_list_clear (bfd *); asection *bfd_get_section_by_name (bfd *abfd, const char *name); asection *bfd_get_section_by_name_if (bfd *abfd, const char *name, bfd_boolean (*func) (bfd *abfd, asection *sect, void *obj), void *obj); char *bfd_get_unique_section_name (bfd *abfd, const char *templat, int *count); asection *bfd_make_section_old_way (bfd *abfd, const char *name); asection *bfd_make_section_anyway_with_flags (bfd *abfd, const char *name, flagword flags); asection *bfd_make_section_anyway (bfd *abfd, const char *name); asection *bfd_make_section_with_flags (bfd *, const char *name, flagword flags); asection *bfd_make_section (bfd *, const char *name); bfd_boolean bfd_set_section_flags (bfd *abfd, asection *sec, flagword flags); void bfd_map_over_sections (bfd *abfd, void (*func) (bfd *abfd, asection *sect, void *obj), void *obj); asection *bfd_sections_find_if (bfd *abfd, bfd_boolean (*operation) (bfd *abfd, asection *sect, void *obj), void *obj); bfd_boolean bfd_set_section_size (bfd *abfd, asection *sec, bfd_size_type val); bfd_boolean bfd_set_section_contents (bfd *abfd, asection *section, const void *data, file_ptr offset, bfd_size_type count); bfd_boolean bfd_get_section_contents (bfd *abfd, asection *section, void *location, file_ptr offset, bfd_size_type count); bfd_boolean bfd_malloc_and_get_section (bfd *abfd, asection *section, bfd_byte **buf); bfd_boolean bfd_copy_private_section_data (bfd *ibfd, asection *isec, bfd *obfd, asection *osec); #define bfd_copy_private_section_data(ibfd, isection, obfd, osection) \ BFD_SEND (obfd, _bfd_copy_private_section_data, \ (ibfd, isection, obfd, osection)) bfd_boolean bfd_generic_is_group_section (bfd *, const asection *sec); bfd_boolean bfd_generic_discard_group (bfd *abfd, asection *group); /* Extracted from archures.c. */ enum bfd_architecture { bfd_arch_unknown, /* File arch not known. */ bfd_arch_obscure, /* Arch known, not one of these. */ bfd_arch_m68k, /* Motorola 68xxx */ #define bfd_mach_m68000 1 #define bfd_mach_m68008 2 #define bfd_mach_m68010 3 #define bfd_mach_m68020 4 #define bfd_mach_m68030 5 #define bfd_mach_m68040 6 #define bfd_mach_m68060 7 #define bfd_mach_cpu32 8 #define bfd_mach_fido 9 #define bfd_mach_mcf_isa_a_nodiv 10 #define bfd_mach_mcf_isa_a 11 #define bfd_mach_mcf_isa_a_mac 12 #define bfd_mach_mcf_isa_a_emac 13 #define bfd_mach_mcf_isa_aplus 14 #define bfd_mach_mcf_isa_aplus_mac 15 #define bfd_mach_mcf_isa_aplus_emac 16 #define bfd_mach_mcf_isa_b_nousp 17 #define bfd_mach_mcf_isa_b_nousp_mac 18 #define bfd_mach_mcf_isa_b_nousp_emac 19 #define bfd_mach_mcf_isa_b 20 #define bfd_mach_mcf_isa_b_mac 21 #define bfd_mach_mcf_isa_b_emac 22 #define bfd_mach_mcf_isa_b_float 23 #define bfd_mach_mcf_isa_b_float_mac 24 #define bfd_mach_mcf_isa_b_float_emac 25 #define bfd_mach_mcf_isa_c 26 #define bfd_mach_mcf_isa_c_mac 27 #define bfd_mach_mcf_isa_c_emac 28 bfd_arch_vax, /* DEC Vax */ bfd_arch_i960, /* Intel 960 */ /* The order of the following is important. lower number indicates a machine type that only accepts a subset of the instructions available to machines with higher numbers. The exception is the "ca", which is incompatible with all other machines except "core". */ #define bfd_mach_i960_core 1 #define bfd_mach_i960_ka_sa 2 #define bfd_mach_i960_kb_sb 3 #define bfd_mach_i960_mc 4 #define bfd_mach_i960_xa 5 #define bfd_mach_i960_ca 6 #define bfd_mach_i960_jx 7 #define bfd_mach_i960_hx 8 bfd_arch_or32, /* OpenRISC 32 */ bfd_arch_sparc, /* SPARC */ #define bfd_mach_sparc 1 /* The difference between v8plus and v9 is that v9 is a true 64 bit env. */ #define bfd_mach_sparc_sparclet 2 #define bfd_mach_sparc_sparclite 3 #define bfd_mach_sparc_v8plus 4 #define bfd_mach_sparc_v8plusa 5 /* with ultrasparc add'ns. */ #define bfd_mach_sparc_sparclite_le 6 #define bfd_mach_sparc_v9 7 #define bfd_mach_sparc_v9a 8 /* with ultrasparc add'ns. */ #define bfd_mach_sparc_v8plusb 9 /* with cheetah add'ns. */ #define bfd_mach_sparc_v9b 10 /* with cheetah add'ns. */ /* Nonzero if MACH has the v9 instruction set. */ #define bfd_mach_sparc_v9_p(mach) \ ((mach) >= bfd_mach_sparc_v8plus && (mach) <= bfd_mach_sparc_v9b \ && (mach) != bfd_mach_sparc_sparclite_le) /* Nonzero if MACH is a 64 bit sparc architecture. */ #define bfd_mach_sparc_64bit_p(mach) \ ((mach) >= bfd_mach_sparc_v9 && (mach) != bfd_mach_sparc_v8plusb) bfd_arch_spu, /* PowerPC SPU */ #define bfd_mach_spu 256 bfd_arch_mips, /* MIPS Rxxxx */ #define bfd_mach_mips3000 3000 #define bfd_mach_mips3900 3900 #define bfd_mach_mips4000 4000 #define bfd_mach_mips4010 4010 #define bfd_mach_mips4100 4100 #define bfd_mach_mips4111 4111 #define bfd_mach_mips4120 4120 #define bfd_mach_mips4300 4300 #define bfd_mach_mips4400 4400 #define bfd_mach_mips4600 4600 #define bfd_mach_mips4650 4650 #define bfd_mach_mips5000 5000 #define bfd_mach_mips5400 5400 #define bfd_mach_mips5500 5500 #define bfd_mach_mips6000 6000 #define bfd_mach_mips7000 7000 #define bfd_mach_mips8000 8000 #define bfd_mach_mips9000 9000 #define bfd_mach_mips10000 10000 #define bfd_mach_mips12000 12000 #define bfd_mach_mips16 16 #define bfd_mach_mips5 5 #define bfd_mach_mips_sb1 12310201 /* octal 'SB', 01 */ #define bfd_mach_mipsisa32 32 #define bfd_mach_mipsisa32r2 33 #define bfd_mach_mipsisa64 64 #define bfd_mach_mipsisa64r2 65 bfd_arch_i386, /* Intel 386 */ #define bfd_mach_i386_i386 1 #define bfd_mach_i386_i8086 2 #define bfd_mach_i386_i386_intel_syntax 3 #define bfd_mach_x86_64 64 #define bfd_mach_x86_64_intel_syntax 65 bfd_arch_we32k, /* AT&T WE32xxx */ bfd_arch_tahoe, /* CCI/Harris Tahoe */ bfd_arch_i860, /* Intel 860 */ bfd_arch_i370, /* IBM 360/370 Mainframes */ bfd_arch_romp, /* IBM ROMP PC/RT */ bfd_arch_convex, /* Convex */ bfd_arch_m88k, /* Motorola 88xxx */ bfd_arch_m98k, /* Motorola 98xxx */ bfd_arch_pyramid, /* Pyramid Technology */ bfd_arch_h8300, /* Renesas H8/300 (formerly Hitachi H8/300) */ #define bfd_mach_h8300 1 #define bfd_mach_h8300h 2 #define bfd_mach_h8300s 3 #define bfd_mach_h8300hn 4 #define bfd_mach_h8300sn 5 #define bfd_mach_h8300sx 6 #define bfd_mach_h8300sxn 7 bfd_arch_pdp11, /* DEC PDP-11 */ bfd_arch_powerpc, /* PowerPC */ #define bfd_mach_ppc 32 #define bfd_mach_ppc64 64 #define bfd_mach_ppc_403 403 #define bfd_mach_ppc_403gc 4030 #define bfd_mach_ppc_505 505 #define bfd_mach_ppc_601 601 #define bfd_mach_ppc_602 602 #define bfd_mach_ppc_603 603 #define bfd_mach_ppc_ec603e 6031 #define bfd_mach_ppc_604 604 #define bfd_mach_ppc_620 620 #define bfd_mach_ppc_630 630 #define bfd_mach_ppc_750 750 #define bfd_mach_ppc_860 860 #define bfd_mach_ppc_a35 35 #define bfd_mach_ppc_rs64ii 642 #define bfd_mach_ppc_rs64iii 643 #define bfd_mach_ppc_7400 7400 #define bfd_mach_ppc_e500 500 bfd_arch_rs6000, /* IBM RS/6000 */ #define bfd_mach_rs6k 6000 #define bfd_mach_rs6k_rs1 6001 #define bfd_mach_rs6k_rsc 6003 #define bfd_mach_rs6k_rs2 6002 bfd_arch_hppa, /* HP PA RISC */ #define bfd_mach_hppa10 10 #define bfd_mach_hppa11 11 #define bfd_mach_hppa20 20 #define bfd_mach_hppa20w 25 bfd_arch_d10v, /* Mitsubishi D10V */ #define bfd_mach_d10v 1 #define bfd_mach_d10v_ts2 2 #define bfd_mach_d10v_ts3 3 bfd_arch_d30v, /* Mitsubishi D30V */ bfd_arch_dlx, /* DLX */ bfd_arch_m68hc11, /* Motorola 68HC11 */ bfd_arch_m68hc12, /* Motorola 68HC12 */ #define bfd_mach_m6812_default 0 #define bfd_mach_m6812 1 #define bfd_mach_m6812s 2 bfd_arch_z8k, /* Zilog Z8000 */ #define bfd_mach_z8001 1 #define bfd_mach_z8002 2 bfd_arch_h8500, /* Renesas H8/500 (formerly Hitachi H8/500) */ bfd_arch_sh, /* Renesas / SuperH SH (formerly Hitachi SH) */ #define bfd_mach_sh 1 #define bfd_mach_sh2 0x20 #define bfd_mach_sh_dsp 0x2d #define bfd_mach_sh2a 0x2a #define bfd_mach_sh2a_nofpu 0x2b #define bfd_mach_sh2a_nofpu_or_sh4_nommu_nofpu 0x2a1 #define bfd_mach_sh2a_nofpu_or_sh3_nommu 0x2a2 #define bfd_mach_sh2a_or_sh4 0x2a3 #define bfd_mach_sh2a_or_sh3e 0x2a4 #define bfd_mach_sh2e 0x2e #define bfd_mach_sh3 0x30 #define bfd_mach_sh3_nommu 0x31 #define bfd_mach_sh3_dsp 0x3d #define bfd_mach_sh3e 0x3e #define bfd_mach_sh4 0x40 #define bfd_mach_sh4_nofpu 0x41 #define bfd_mach_sh4_nommu_nofpu 0x42 #define bfd_mach_sh4a 0x4a #define bfd_mach_sh4a_nofpu 0x4b #define bfd_mach_sh4al_dsp 0x4d #define bfd_mach_sh5 0x50 bfd_arch_alpha, /* Dec Alpha */ #define bfd_mach_alpha_ev4 0x10 #define bfd_mach_alpha_ev5 0x20 #define bfd_mach_alpha_ev6 0x30 bfd_arch_arm, /* Advanced Risc Machines ARM. */ #define bfd_mach_arm_unknown 0 #define bfd_mach_arm_2 1 #define bfd_mach_arm_2a 2 #define bfd_mach_arm_3 3 #define bfd_mach_arm_3M 4 #define bfd_mach_arm_4 5 #define bfd_mach_arm_4T 6 #define bfd_mach_arm_5 7 #define bfd_mach_arm_5T 8 #define bfd_mach_arm_5TE 9 #define bfd_mach_arm_XScale 10 #define bfd_mach_arm_ep9312 11 #define bfd_mach_arm_iWMMXt 12 #define bfd_mach_arm_iWMMXt2 13 bfd_arch_ns32k, /* National Semiconductors ns32000 */ bfd_arch_w65, /* WDC 65816 */ bfd_arch_tic30, /* Texas Instruments TMS320C30 */ bfd_arch_tic4x, /* Texas Instruments TMS320C3X/4X */ #define bfd_mach_tic3x 30 #define bfd_mach_tic4x 40 bfd_arch_tic54x, /* Texas Instruments TMS320C54X */ bfd_arch_tic80, /* TI TMS320c80 (MVP) */ bfd_arch_v850, /* NEC V850 */ #define bfd_mach_v850 1 #define bfd_mach_v850e 'E' #define bfd_mach_v850e1 '1' bfd_arch_arc, /* ARC Cores */ #define bfd_mach_arc_5 5 #define bfd_mach_arc_6 6 #define bfd_mach_arc_7 7 #define bfd_mach_arc_8 8 bfd_arch_m32c, /* Renesas M16C/M32C. */ #define bfd_mach_m16c 0x75 #define bfd_mach_m32c 0x78 bfd_arch_m32r, /* Renesas M32R (formerly Mitsubishi M32R/D) */ #define bfd_mach_m32r 1 /* For backwards compatibility. */ #define bfd_mach_m32rx 'x' #define bfd_mach_m32r2 '2' bfd_arch_mn10200, /* Matsushita MN10200 */ bfd_arch_mn10300, /* Matsushita MN10300 */ #define bfd_mach_mn10300 300 #define bfd_mach_am33 330 #define bfd_mach_am33_2 332 bfd_arch_fr30, #define bfd_mach_fr30 0x46523330 bfd_arch_frv, #define bfd_mach_frv 1 #define bfd_mach_frvsimple 2 #define bfd_mach_fr300 300 #define bfd_mach_fr400 400 #define bfd_mach_fr450 450 #define bfd_mach_frvtomcat 499 /* fr500 prototype */ #define bfd_mach_fr500 500 #define bfd_mach_fr550 550 bfd_arch_mcore, bfd_arch_mep, #define bfd_mach_mep 1 #define bfd_mach_mep_h1 0x6831 bfd_arch_ia64, /* HP/Intel ia64 */ #define bfd_mach_ia64_elf64 64 #define bfd_mach_ia64_elf32 32 bfd_arch_ip2k, /* Ubicom IP2K microcontrollers. */ #define bfd_mach_ip2022 1 #define bfd_mach_ip2022ext 2 bfd_arch_iq2000, /* Vitesse IQ2000. */ #define bfd_mach_iq2000 1 #define bfd_mach_iq10 2 bfd_arch_mt, #define bfd_mach_ms1 1 #define bfd_mach_mrisc2 2 #define bfd_mach_ms2 3 bfd_arch_pj, bfd_arch_avr, /* Atmel AVR microcontrollers. */ #define bfd_mach_avr1 1 #define bfd_mach_avr2 2 #define bfd_mach_avr3 3 #define bfd_mach_avr4 4 #define bfd_mach_avr5 5 #define bfd_mach_avr6 6 bfd_arch_bfin, /* ADI Blackfin */ #define bfd_mach_bfin 1 bfd_arch_cr16, /* National Semiconductor CompactRISC (ie CR16). */ #define bfd_mach_cr16 1 bfd_arch_cr16c, /* National Semiconductor CompactRISC. */ #define bfd_mach_cr16c 1 bfd_arch_crx, /* National Semiconductor CRX. */ #define bfd_mach_crx 1 bfd_arch_cris, /* Axis CRIS */ #define bfd_mach_cris_v0_v10 255 #define bfd_mach_cris_v32 32 #define bfd_mach_cris_v10_v32 1032 bfd_arch_s390, /* IBM s390 */ #define bfd_mach_s390_31 31 #define bfd_mach_s390_64 64 bfd_arch_score, /* Sunplus score */ bfd_arch_openrisc, /* OpenRISC */ bfd_arch_mmix, /* Donald Knuth's educational processor. */ bfd_arch_xstormy16, #define bfd_mach_xstormy16 1 bfd_arch_msp430, /* Texas Instruments MSP430 architecture. */ #define bfd_mach_msp11 11 #define bfd_mach_msp110 110 #define bfd_mach_msp12 12 #define bfd_mach_msp13 13 #define bfd_mach_msp14 14 #define bfd_mach_msp15 15 #define bfd_mach_msp16 16 #define bfd_mach_msp21 21 #define bfd_mach_msp31 31 #define bfd_mach_msp32 32 #define bfd_mach_msp33 33 #define bfd_mach_msp41 41 #define bfd_mach_msp42 42 #define bfd_mach_msp43 43 #define bfd_mach_msp44 44 bfd_arch_xc16x, /* Infineon's XC16X Series. */ #define bfd_mach_xc16x 1 #define bfd_mach_xc16xl 2 #define bfd_mach_xc16xs 3 bfd_arch_xtensa, /* Tensilica's Xtensa cores. */ #define bfd_mach_xtensa 1 bfd_arch_maxq, /* Dallas MAXQ 10/20 */ #define bfd_mach_maxq10 10 #define bfd_mach_maxq20 20 bfd_arch_z80, #define bfd_mach_z80strict 1 /* No undocumented opcodes. */ #define bfd_mach_z80 3 /* With ixl, ixh, iyl, and iyh. */ #define bfd_mach_z80full 7 /* All undocumented instructions. */ #define bfd_mach_r800 11 /* R800: successor with multiplication. */ bfd_arch_last }; typedef struct bfd_arch_info { int bits_per_word; int bits_per_address; int bits_per_byte; enum bfd_architecture arch; unsigned long mach; const char *arch_name; const char *printable_name; unsigned int section_align_power; /* TRUE if this is the default machine for the architecture. The default arch should be the first entry for an arch so that all the entries for that arch can be accessed via <>. */ bfd_boolean the_default; const struct bfd_arch_info * (*compatible) (const struct bfd_arch_info *a, const struct bfd_arch_info *b); bfd_boolean (*scan) (const struct bfd_arch_info *, const char *); const struct bfd_arch_info *next; } bfd_arch_info_type; const char *bfd_printable_name (bfd *abfd); const bfd_arch_info_type *bfd_scan_arch (const char *string); const char **bfd_arch_list (void); const bfd_arch_info_type *bfd_arch_get_compatible (const bfd *abfd, const bfd *bbfd, bfd_boolean accept_unknowns); void bfd_set_arch_info (bfd *abfd, const bfd_arch_info_type *arg); enum bfd_architecture bfd_get_arch (bfd *abfd); unsigned long bfd_get_mach (bfd *abfd); unsigned int bfd_arch_bits_per_byte (bfd *abfd); unsigned int bfd_arch_bits_per_address (bfd *abfd); const bfd_arch_info_type *bfd_get_arch_info (bfd *abfd); const bfd_arch_info_type *bfd_lookup_arch (enum bfd_architecture arch, unsigned long machine); const char *bfd_printable_arch_mach (enum bfd_architecture arch, unsigned long machine); unsigned int bfd_octets_per_byte (bfd *abfd); unsigned int bfd_arch_mach_octets_per_byte (enum bfd_architecture arch, unsigned long machine); /* Extracted from reloc.c. */ typedef enum bfd_reloc_status { /* No errors detected. */ bfd_reloc_ok, /* The relocation was performed, but there was an overflow. */ bfd_reloc_overflow, /* The address to relocate was not within the section supplied. */ bfd_reloc_outofrange, /* Used by special functions. */ bfd_reloc_continue, /* Unsupported relocation size requested. */ bfd_reloc_notsupported, /* Unused. */ bfd_reloc_other, /* The symbol to relocate against was undefined. */ bfd_reloc_undefined, /* The relocation was performed, but may not be ok - presently generated only when linking i960 coff files with i960 b.out symbols. If this type is returned, the error_message argument to bfd_perform_relocation will be set. */ bfd_reloc_dangerous } bfd_reloc_status_type; typedef struct reloc_cache_entry { /* A pointer into the canonical table of pointers. */ struct bfd_symbol **sym_ptr_ptr; /* offset in section. */ bfd_size_type address; /* addend for relocation value. */ bfd_vma addend; /* Pointer to how to perform the required relocation. */ reloc_howto_type *howto; } arelent; enum complain_overflow { /* Do not complain on overflow. */ complain_overflow_dont, /* Complain if the value overflows when considered as a signed number one bit larger than the field. ie. A bitfield of N bits is allowed to represent -2**n to 2**n-1. */ complain_overflow_bitfield, /* Complain if the value overflows when considered as a signed number. */ complain_overflow_signed, /* Complain if the value overflows when considered as an unsigned number. */ complain_overflow_unsigned }; struct reloc_howto_struct { /* The type field has mainly a documentary use - the back end can do what it wants with it, though normally the back end's external idea of what a reloc number is stored in this field. For example, a PC relative word relocation in a coff environment has the type 023 - because that's what the outside world calls a R_PCRWORD reloc. */ unsigned int type; /* The value the final relocation is shifted right by. This drops unwanted data from the relocation. */ unsigned int rightshift; /* The size of the item to be relocated. This is *not* a power-of-two measure. To get the number of bytes operated on by a type of relocation, use bfd_get_reloc_size. */ int size; /* The number of bits in the item to be relocated. This is used when doing overflow checking. */ unsigned int bitsize; /* Notes that the relocation is relative to the location in the data section of the addend. The relocation function will subtract from the relocation value the address of the location being relocated. */ bfd_boolean pc_relative; /* The bit position of the reloc value in the destination. The relocated value is left shifted by this amount. */ unsigned int bitpos; /* What type of overflow error should be checked for when relocating. */ enum complain_overflow complain_on_overflow; /* If this field is non null, then the supplied function is called rather than the normal function. This allows really strange relocation methods to be accommodated (e.g., i960 callj instructions). */ bfd_reloc_status_type (*special_function) (bfd *, arelent *, struct bfd_symbol *, void *, asection *, bfd *, char **); /* The textual name of the relocation type. */ char *name; /* Some formats record a relocation addend in the section contents rather than with the relocation. For ELF formats this is the distinction between USE_REL and USE_RELA (though the code checks for USE_REL == 1/0). The value of this field is TRUE if the addend is recorded with the section contents; when performing a partial link (ld -r) the section contents (the data) will be modified. The value of this field is FALSE if addends are recorded with the relocation (in arelent.addend); when performing a partial link the relocation will be modified. All relocations for all ELF USE_RELA targets should set this field to FALSE (values of TRUE should be looked on with suspicion). However, the converse is not true: not all relocations of all ELF USE_REL targets set this field to TRUE. Why this is so is peculiar to each particular target. For relocs that aren't used in partial links (e.g. GOT stuff) it doesn't matter what this is set to. */ bfd_boolean partial_inplace; /* src_mask selects the part of the instruction (or data) to be used in the relocation sum. If the target relocations don't have an addend in the reloc, eg. ELF USE_REL, src_mask will normally equal dst_mask to extract the addend from the section contents. If relocations do have an addend in the reloc, eg. ELF USE_RELA, this field should be zero. Non-zero values for ELF USE_RELA targets are bogus as in those cases the value in the dst_mask part of the section contents should be treated as garbage. */ bfd_vma src_mask; /* dst_mask selects which parts of the instruction (or data) are replaced with a relocated value. */ bfd_vma dst_mask; /* When some formats create PC relative instructions, they leave the value of the pc of the place being relocated in the offset slot of the instruction, so that a PC relative relocation can be made just by adding in an ordinary offset (e.g., sun3 a.out). Some formats leave the displacement part of an instruction empty (e.g., m88k bcs); this flag signals the fact. */ bfd_boolean pcrel_offset; }; #define HOWTO(C, R, S, B, P, BI, O, SF, NAME, INPLACE, MASKSRC, MASKDST, PC) \ { (unsigned) C, R, S, B, P, BI, O, SF, NAME, INPLACE, MASKSRC, MASKDST, PC } #define NEWHOWTO(FUNCTION, NAME, SIZE, REL, IN) \ HOWTO (0, 0, SIZE, 0, REL, 0, complain_overflow_dont, FUNCTION, \ NAME, FALSE, 0, 0, IN) #define EMPTY_HOWTO(C) \ HOWTO ((C), 0, 0, 0, FALSE, 0, complain_overflow_dont, NULL, \ NULL, FALSE, 0, 0, FALSE) #define HOWTO_PREPARE(relocation, symbol) \ { \ if (symbol != NULL) \ { \ if (bfd_is_com_section (symbol->section)) \ { \ relocation = 0; \ } \ else \ { \ relocation = symbol->value; \ } \ } \ } unsigned int bfd_get_reloc_size (reloc_howto_type *); typedef struct relent_chain { arelent relent; struct relent_chain *next; } arelent_chain; bfd_reloc_status_type bfd_check_overflow (enum complain_overflow how, unsigned int bitsize, unsigned int rightshift, unsigned int addrsize, bfd_vma relocation); bfd_reloc_status_type bfd_perform_relocation (bfd *abfd, arelent *reloc_entry, void *data, asection *input_section, bfd *output_bfd, char **error_message); bfd_reloc_status_type bfd_install_relocation (bfd *abfd, arelent *reloc_entry, void *data, bfd_vma data_start, asection *input_section, char **error_message); enum bfd_reloc_code_real { _dummy_first_bfd_reloc_code_real, /* Basic absolute relocations of N bits. */ BFD_RELOC_64, BFD_RELOC_32, BFD_RELOC_26, BFD_RELOC_24, BFD_RELOC_16, BFD_RELOC_14, BFD_RELOC_8, /* PC-relative relocations. Sometimes these are relative to the address of the relocation itself; sometimes they are relative to the start of the section containing the relocation. It depends on the specific target. The 24-bit relocation is used in some Intel 960 configurations. */ BFD_RELOC_64_PCREL, BFD_RELOC_32_PCREL, BFD_RELOC_24_PCREL, BFD_RELOC_16_PCREL, BFD_RELOC_12_PCREL, BFD_RELOC_8_PCREL, /* Section relative relocations. Some targets need this for DWARF2. */ BFD_RELOC_32_SECREL, /* For ELF. */ BFD_RELOC_32_GOT_PCREL, BFD_RELOC_16_GOT_PCREL, BFD_RELOC_8_GOT_PCREL, BFD_RELOC_32_GOTOFF, BFD_RELOC_16_GOTOFF, BFD_RELOC_LO16_GOTOFF, BFD_RELOC_HI16_GOTOFF, BFD_RELOC_HI16_S_GOTOFF, BFD_RELOC_8_GOTOFF, BFD_RELOC_64_PLT_PCREL, BFD_RELOC_32_PLT_PCREL, BFD_RELOC_24_PLT_PCREL, BFD_RELOC_16_PLT_PCREL, BFD_RELOC_8_PLT_PCREL, BFD_RELOC_64_PLTOFF, BFD_RELOC_32_PLTOFF, BFD_RELOC_16_PLTOFF, BFD_RELOC_LO16_PLTOFF, BFD_RELOC_HI16_PLTOFF, BFD_RELOC_HI16_S_PLTOFF, BFD_RELOC_8_PLTOFF, /* Relocations used by 68K ELF. */ BFD_RELOC_68K_GLOB_DAT, BFD_RELOC_68K_JMP_SLOT, BFD_RELOC_68K_RELATIVE, /* Linkage-table relative. */ BFD_RELOC_32_BASEREL, BFD_RELOC_16_BASEREL, BFD_RELOC_LO16_BASEREL, BFD_RELOC_HI16_BASEREL, BFD_RELOC_HI16_S_BASEREL, BFD_RELOC_8_BASEREL, BFD_RELOC_RVA, /* Absolute 8-bit relocation, but used to form an address like 0xFFnn. */ BFD_RELOC_8_FFnn, /* These PC-relative relocations are stored as word displacements -- i.e., byte displacements shifted right two bits. The 30-bit word displacement (<<32_PCREL_S2>> -- 32 bits, shifted 2) is used on the SPARC. (SPARC tools generally refer to this as <>.) The signed 16-bit displacement is used on the MIPS, and the 23-bit displacement is used on the Alpha. */ BFD_RELOC_32_PCREL_S2, BFD_RELOC_16_PCREL_S2, BFD_RELOC_23_PCREL_S2, /* High 22 bits and low 10 bits of 32-bit value, placed into lower bits of the target word. These are used on the SPARC. */ BFD_RELOC_HI22, BFD_RELOC_LO10, /* For systems that allocate a Global Pointer register, these are displacements off that register. These relocation types are handled specially, because the value the register will have is decided relatively late. */ BFD_RELOC_GPREL16, BFD_RELOC_GPREL32, /* Reloc types used for i960/b.out. */ BFD_RELOC_I960_CALLJ, /* SPARC ELF relocations. There is probably some overlap with other relocation types already defined. */ BFD_RELOC_NONE, BFD_RELOC_SPARC_WDISP22, BFD_RELOC_SPARC22, BFD_RELOC_SPARC13, BFD_RELOC_SPARC_GOT10, BFD_RELOC_SPARC_GOT13, BFD_RELOC_SPARC_GOT22, BFD_RELOC_SPARC_PC10, BFD_RELOC_SPARC_PC22, BFD_RELOC_SPARC_WPLT30, BFD_RELOC_SPARC_COPY, BFD_RELOC_SPARC_GLOB_DAT, BFD_RELOC_SPARC_JMP_SLOT, BFD_RELOC_SPARC_RELATIVE, BFD_RELOC_SPARC_UA16, BFD_RELOC_SPARC_UA32, BFD_RELOC_SPARC_UA64, /* I think these are specific to SPARC a.out (e.g., Sun 4). */ BFD_RELOC_SPARC_BASE13, BFD_RELOC_SPARC_BASE22, /* SPARC64 relocations */ #define BFD_RELOC_SPARC_64 BFD_RELOC_64 BFD_RELOC_SPARC_10, BFD_RELOC_SPARC_11, BFD_RELOC_SPARC_OLO10, BFD_RELOC_SPARC_HH22, BFD_RELOC_SPARC_HM10, BFD_RELOC_SPARC_LM22, BFD_RELOC_SPARC_PC_HH22, BFD_RELOC_SPARC_PC_HM10, BFD_RELOC_SPARC_PC_LM22, BFD_RELOC_SPARC_WDISP16, BFD_RELOC_SPARC_WDISP19, BFD_RELOC_SPARC_7, BFD_RELOC_SPARC_6, BFD_RELOC_SPARC_5, #define BFD_RELOC_SPARC_DISP64 BFD_RELOC_64_PCREL BFD_RELOC_SPARC_PLT32, BFD_RELOC_SPARC_PLT64, BFD_RELOC_SPARC_HIX22, BFD_RELOC_SPARC_LOX10, BFD_RELOC_SPARC_H44, BFD_RELOC_SPARC_M44, BFD_RELOC_SPARC_L44, BFD_RELOC_SPARC_REGISTER, /* SPARC little endian relocation */ BFD_RELOC_SPARC_REV32, /* SPARC TLS relocations */ BFD_RELOC_SPARC_TLS_GD_HI22, BFD_RELOC_SPARC_TLS_GD_LO10, BFD_RELOC_SPARC_TLS_GD_ADD, BFD_RELOC_SPARC_TLS_GD_CALL, BFD_RELOC_SPARC_TLS_LDM_HI22, BFD_RELOC_SPARC_TLS_LDM_LO10, BFD_RELOC_SPARC_TLS_LDM_ADD, BFD_RELOC_SPARC_TLS_LDM_CALL, BFD_RELOC_SPARC_TLS_LDO_HIX22, BFD_RELOC_SPARC_TLS_LDO_LOX10, BFD_RELOC_SPARC_TLS_LDO_ADD, BFD_RELOC_SPARC_TLS_IE_HI22, BFD_RELOC_SPARC_TLS_IE_LO10, BFD_RELOC_SPARC_TLS_IE_LD, BFD_RELOC_SPARC_TLS_IE_LDX, BFD_RELOC_SPARC_TLS_IE_ADD, BFD_RELOC_SPARC_TLS_LE_HIX22, BFD_RELOC_SPARC_TLS_LE_LOX10, BFD_RELOC_SPARC_TLS_DTPMOD32, BFD_RELOC_SPARC_TLS_DTPMOD64, BFD_RELOC_SPARC_TLS_DTPOFF32, BFD_RELOC_SPARC_TLS_DTPOFF64, BFD_RELOC_SPARC_TLS_TPOFF32, BFD_RELOC_SPARC_TLS_TPOFF64, /* SPU Relocations. */ BFD_RELOC_SPU_IMM7, BFD_RELOC_SPU_IMM8, BFD_RELOC_SPU_IMM10, BFD_RELOC_SPU_IMM10W, BFD_RELOC_SPU_IMM16, BFD_RELOC_SPU_IMM16W, BFD_RELOC_SPU_IMM18, BFD_RELOC_SPU_PCREL9a, BFD_RELOC_SPU_PCREL9b, BFD_RELOC_SPU_PCREL16, BFD_RELOC_SPU_LO16, BFD_RELOC_SPU_HI16, BFD_RELOC_SPU_PPU32, BFD_RELOC_SPU_PPU64, /* Alpha ECOFF and ELF relocations. Some of these treat the symbol or "addend" in some special way. For GPDISP_HI16 ("gpdisp") relocations, the symbol is ignored when writing; when reading, it will be the absolute section symbol. The addend is the displacement in bytes of the "lda" instruction from the "ldah" instruction (which is at the address of this reloc). */ BFD_RELOC_ALPHA_GPDISP_HI16, /* For GPDISP_LO16 ("ignore") relocations, the symbol is handled as with GPDISP_HI16 relocs. The addend is ignored when writing the relocations out, and is filled in with the file's GP value on reading, for convenience. */ BFD_RELOC_ALPHA_GPDISP_LO16, /* The ELF GPDISP relocation is exactly the same as the GPDISP_HI16 relocation except that there is no accompanying GPDISP_LO16 relocation. */ BFD_RELOC_ALPHA_GPDISP, /* The Alpha LITERAL/LITUSE relocs are produced by a symbol reference; the assembler turns it into a LDQ instruction to load the address of the symbol, and then fills in a register in the real instruction. The LITERAL reloc, at the LDQ instruction, refers to the .lita section symbol. The addend is ignored when writing, but is filled in with the file's GP value on reading, for convenience, as with the GPDISP_LO16 reloc. The ELF_LITERAL reloc is somewhere between 16_GOTOFF and GPDISP_LO16. It should refer to the symbol to be referenced, as with 16_GOTOFF, but it generates output not based on the position within the .got section, but relative to the GP value chosen for the file during the final link stage. The LITUSE reloc, on the instruction using the loaded address, gives information to the linker that it might be able to use to optimize away some literal section references. The symbol is ignored (read as the absolute section symbol), and the "addend" indicates the type of instruction using the register: 1 - "memory" fmt insn 2 - byte-manipulation (byte offset reg) 3 - jsr (target of branch) */ BFD_RELOC_ALPHA_LITERAL, BFD_RELOC_ALPHA_ELF_LITERAL, BFD_RELOC_ALPHA_LITUSE, /* The HINT relocation indicates a value that should be filled into the "hint" field of a jmp/jsr/ret instruction, for possible branch- prediction logic which may be provided on some processors. */ BFD_RELOC_ALPHA_HINT, /* The LINKAGE relocation outputs a linkage pair in the object file, which is filled by the linker. */ BFD_RELOC_ALPHA_LINKAGE, /* The CODEADDR relocation outputs a STO_CA in the object file, which is filled by the linker. */ BFD_RELOC_ALPHA_CODEADDR, /* The GPREL_HI/LO relocations together form a 32-bit offset from the GP register. */ BFD_RELOC_ALPHA_GPREL_HI16, BFD_RELOC_ALPHA_GPREL_LO16, /* Like BFD_RELOC_23_PCREL_S2, except that the source and target must share a common GP, and the target address is adjusted for STO_ALPHA_STD_GPLOAD. */ BFD_RELOC_ALPHA_BRSGP, /* Alpha thread-local storage relocations. */ BFD_RELOC_ALPHA_TLSGD, BFD_RELOC_ALPHA_TLSLDM, BFD_RELOC_ALPHA_DTPMOD64, BFD_RELOC_ALPHA_GOTDTPREL16, BFD_RELOC_ALPHA_DTPREL64, BFD_RELOC_ALPHA_DTPREL_HI16, BFD_RELOC_ALPHA_DTPREL_LO16, BFD_RELOC_ALPHA_DTPREL16, BFD_RELOC_ALPHA_GOTTPREL16, BFD_RELOC_ALPHA_TPREL64, BFD_RELOC_ALPHA_TPREL_HI16, BFD_RELOC_ALPHA_TPREL_LO16, BFD_RELOC_ALPHA_TPREL16, /* Bits 27..2 of the relocation address shifted right 2 bits; simple reloc otherwise. */ BFD_RELOC_MIPS_JMP, /* The MIPS16 jump instruction. */ BFD_RELOC_MIPS16_JMP, /* MIPS16 GP relative reloc. */ BFD_RELOC_MIPS16_GPREL, /* High 16 bits of 32-bit value; simple reloc. */ BFD_RELOC_HI16, /* High 16 bits of 32-bit value but the low 16 bits will be sign extended and added to form the final result. If the low 16 bits form a negative number, we need to add one to the high value to compensate for the borrow when the low bits are added. */ BFD_RELOC_HI16_S, /* Low 16 bits. */ BFD_RELOC_LO16, /* High 16 bits of 32-bit pc-relative value */ BFD_RELOC_HI16_PCREL, /* High 16 bits of 32-bit pc-relative value, adjusted */ BFD_RELOC_HI16_S_PCREL, /* Low 16 bits of pc-relative value */ BFD_RELOC_LO16_PCREL, /* MIPS16 high 16 bits of 32-bit value. */ BFD_RELOC_MIPS16_HI16, /* MIPS16 high 16 bits of 32-bit value but the low 16 bits will be sign extended and added to form the final result. If the low 16 bits form a negative number, we need to add one to the high value to compensate for the borrow when the low bits are added. */ BFD_RELOC_MIPS16_HI16_S, /* MIPS16 low 16 bits. */ BFD_RELOC_MIPS16_LO16, /* Relocation against a MIPS literal section. */ BFD_RELOC_MIPS_LITERAL, /* MIPS ELF relocations. */ BFD_RELOC_MIPS_GOT16, BFD_RELOC_MIPS_CALL16, BFD_RELOC_MIPS_GOT_HI16, BFD_RELOC_MIPS_GOT_LO16, BFD_RELOC_MIPS_CALL_HI16, BFD_RELOC_MIPS_CALL_LO16, BFD_RELOC_MIPS_SUB, BFD_RELOC_MIPS_GOT_PAGE, BFD_RELOC_MIPS_GOT_OFST, BFD_RELOC_MIPS_GOT_DISP, BFD_RELOC_MIPS_SHIFT5, BFD_RELOC_MIPS_SHIFT6, BFD_RELOC_MIPS_INSERT_A, BFD_RELOC_MIPS_INSERT_B, BFD_RELOC_MIPS_DELETE, BFD_RELOC_MIPS_HIGHEST, BFD_RELOC_MIPS_HIGHER, BFD_RELOC_MIPS_SCN_DISP, BFD_RELOC_MIPS_REL16, BFD_RELOC_MIPS_RELGOT, BFD_RELOC_MIPS_JALR, BFD_RELOC_MIPS_TLS_DTPMOD32, BFD_RELOC_MIPS_TLS_DTPREL32, BFD_RELOC_MIPS_TLS_DTPMOD64, BFD_RELOC_MIPS_TLS_DTPREL64, BFD_RELOC_MIPS_TLS_GD, BFD_RELOC_MIPS_TLS_LDM, BFD_RELOC_MIPS_TLS_DTPREL_HI16, BFD_RELOC_MIPS_TLS_DTPREL_LO16, BFD_RELOC_MIPS_TLS_GOTTPREL, BFD_RELOC_MIPS_TLS_TPREL32, BFD_RELOC_MIPS_TLS_TPREL64, BFD_RELOC_MIPS_TLS_TPREL_HI16, BFD_RELOC_MIPS_TLS_TPREL_LO16, /* MIPS ELF relocations (VxWorks extensions). */ BFD_RELOC_MIPS_COPY, BFD_RELOC_MIPS_JUMP_SLOT, /* Fujitsu Frv Relocations. */ BFD_RELOC_FRV_LABEL16, BFD_RELOC_FRV_LABEL24, BFD_RELOC_FRV_LO16, BFD_RELOC_FRV_HI16, BFD_RELOC_FRV_GPREL12, BFD_RELOC_FRV_GPRELU12, BFD_RELOC_FRV_GPREL32, BFD_RELOC_FRV_GPRELHI, BFD_RELOC_FRV_GPRELLO, BFD_RELOC_FRV_GOT12, BFD_RELOC_FRV_GOTHI, BFD_RELOC_FRV_GOTLO, BFD_RELOC_FRV_FUNCDESC, BFD_RELOC_FRV_FUNCDESC_GOT12, BFD_RELOC_FRV_FUNCDESC_GOTHI, BFD_RELOC_FRV_FUNCDESC_GOTLO, BFD_RELOC_FRV_FUNCDESC_VALUE, BFD_RELOC_FRV_FUNCDESC_GOTOFF12, BFD_RELOC_FRV_FUNCDESC_GOTOFFHI, BFD_RELOC_FRV_FUNCDESC_GOTOFFLO, BFD_RELOC_FRV_GOTOFF12, BFD_RELOC_FRV_GOTOFFHI, BFD_RELOC_FRV_GOTOFFLO, BFD_RELOC_FRV_GETTLSOFF, BFD_RELOC_FRV_TLSDESC_VALUE, BFD_RELOC_FRV_GOTTLSDESC12, BFD_RELOC_FRV_GOTTLSDESCHI, BFD_RELOC_FRV_GOTTLSDESCLO, BFD_RELOC_FRV_TLSMOFF12, BFD_RELOC_FRV_TLSMOFFHI, BFD_RELOC_FRV_TLSMOFFLO, BFD_RELOC_FRV_GOTTLSOFF12, BFD_RELOC_FRV_GOTTLSOFFHI, BFD_RELOC_FRV_GOTTLSOFFLO, BFD_RELOC_FRV_TLSOFF, BFD_RELOC_FRV_TLSDESC_RELAX, BFD_RELOC_FRV_GETTLSOFF_RELAX, BFD_RELOC_FRV_TLSOFF_RELAX, BFD_RELOC_FRV_TLSMOFF, /* This is a 24bit GOT-relative reloc for the mn10300. */ BFD_RELOC_MN10300_GOTOFF24, /* This is a 32bit GOT-relative reloc for the mn10300, offset by two bytes in the instruction. */ BFD_RELOC_MN10300_GOT32, /* This is a 24bit GOT-relative reloc for the mn10300, offset by two bytes in the instruction. */ BFD_RELOC_MN10300_GOT24, /* This is a 16bit GOT-relative reloc for the mn10300, offset by two bytes in the instruction. */ BFD_RELOC_MN10300_GOT16, /* Copy symbol at runtime. */ BFD_RELOC_MN10300_COPY, /* Create GOT entry. */ BFD_RELOC_MN10300_GLOB_DAT, /* Create PLT entry. */ BFD_RELOC_MN10300_JMP_SLOT, /* Adjust by program base. */ BFD_RELOC_MN10300_RELATIVE, /* i386/elf relocations */ BFD_RELOC_386_GOT32, BFD_RELOC_386_PLT32, BFD_RELOC_386_COPY, BFD_RELOC_386_GLOB_DAT, BFD_RELOC_386_JUMP_SLOT, BFD_RELOC_386_RELATIVE, BFD_RELOC_386_GOTOFF, BFD_RELOC_386_GOTPC, BFD_RELOC_386_TLS_TPOFF, BFD_RELOC_386_TLS_IE, BFD_RELOC_386_TLS_GOTIE, BFD_RELOC_386_TLS_LE, BFD_RELOC_386_TLS_GD, BFD_RELOC_386_TLS_LDM, BFD_RELOC_386_TLS_LDO_32, BFD_RELOC_386_TLS_IE_32, BFD_RELOC_386_TLS_LE_32, BFD_RELOC_386_TLS_DTPMOD32, BFD_RELOC_386_TLS_DTPOFF32, BFD_RELOC_386_TLS_TPOFF32, BFD_RELOC_386_TLS_GOTDESC, BFD_RELOC_386_TLS_DESC_CALL, BFD_RELOC_386_TLS_DESC, /* x86-64/elf relocations */ BFD_RELOC_X86_64_GOT32, BFD_RELOC_X86_64_PLT32, BFD_RELOC_X86_64_COPY, BFD_RELOC_X86_64_GLOB_DAT, BFD_RELOC_X86_64_JUMP_SLOT, BFD_RELOC_X86_64_RELATIVE, BFD_RELOC_X86_64_GOTPCREL, BFD_RELOC_X86_64_32S, BFD_RELOC_X86_64_DTPMOD64, BFD_RELOC_X86_64_DTPOFF64, BFD_RELOC_X86_64_TPOFF64, BFD_RELOC_X86_64_TLSGD, BFD_RELOC_X86_64_TLSLD, BFD_RELOC_X86_64_DTPOFF32, BFD_RELOC_X86_64_GOTTPOFF, BFD_RELOC_X86_64_TPOFF32, BFD_RELOC_X86_64_GOTOFF64, BFD_RELOC_X86_64_GOTPC32, BFD_RELOC_X86_64_GOT64, BFD_RELOC_X86_64_GOTPCREL64, BFD_RELOC_X86_64_GOTPC64, BFD_RELOC_X86_64_GOTPLT64, BFD_RELOC_X86_64_PLTOFF64, BFD_RELOC_X86_64_GOTPC32_TLSDESC, BFD_RELOC_X86_64_TLSDESC_CALL, BFD_RELOC_X86_64_TLSDESC, /* ns32k relocations */ BFD_RELOC_NS32K_IMM_8, BFD_RELOC_NS32K_IMM_16, BFD_RELOC_NS32K_IMM_32, BFD_RELOC_NS32K_IMM_8_PCREL, BFD_RELOC_NS32K_IMM_16_PCREL, BFD_RELOC_NS32K_IMM_32_PCREL, BFD_RELOC_NS32K_DISP_8, BFD_RELOC_NS32K_DISP_16, BFD_RELOC_NS32K_DISP_32, BFD_RELOC_NS32K_DISP_8_PCREL, BFD_RELOC_NS32K_DISP_16_PCREL, BFD_RELOC_NS32K_DISP_32_PCREL, /* PDP11 relocations */ BFD_RELOC_PDP11_DISP_8_PCREL, BFD_RELOC_PDP11_DISP_6_PCREL, /* Picojava relocs. Not all of these appear in object files. */ BFD_RELOC_PJ_CODE_HI16, BFD_RELOC_PJ_CODE_LO16, BFD_RELOC_PJ_CODE_DIR16, BFD_RELOC_PJ_CODE_DIR32, BFD_RELOC_PJ_CODE_REL16, BFD_RELOC_PJ_CODE_REL32, /* Power(rs6000) and PowerPC relocations. */ BFD_RELOC_PPC_B26, BFD_RELOC_PPC_BA26, BFD_RELOC_PPC_TOC16, BFD_RELOC_PPC_B16, BFD_RELOC_PPC_B16_BRTAKEN, BFD_RELOC_PPC_B16_BRNTAKEN, BFD_RELOC_PPC_BA16, BFD_RELOC_PPC_BA16_BRTAKEN, BFD_RELOC_PPC_BA16_BRNTAKEN, BFD_RELOC_PPC_COPY, BFD_RELOC_PPC_GLOB_DAT, BFD_RELOC_PPC_JMP_SLOT, BFD_RELOC_PPC_RELATIVE, BFD_RELOC_PPC_LOCAL24PC, BFD_RELOC_PPC_EMB_NADDR32, BFD_RELOC_PPC_EMB_NADDR16, BFD_RELOC_PPC_EMB_NADDR16_LO, BFD_RELOC_PPC_EMB_NADDR16_HI, BFD_RELOC_PPC_EMB_NADDR16_HA, BFD_RELOC_PPC_EMB_SDAI16, BFD_RELOC_PPC_EMB_SDA2I16, BFD_RELOC_PPC_EMB_SDA2REL, BFD_RELOC_PPC_EMB_SDA21, BFD_RELOC_PPC_EMB_MRKREF, BFD_RELOC_PPC_EMB_RELSEC16, BFD_RELOC_PPC_EMB_RELST_LO, BFD_RELOC_PPC_EMB_RELST_HI, BFD_RELOC_PPC_EMB_RELST_HA, BFD_RELOC_PPC_EMB_BIT_FLD, BFD_RELOC_PPC_EMB_RELSDA, BFD_RELOC_PPC64_HIGHER, BFD_RELOC_PPC64_HIGHER_S, BFD_RELOC_PPC64_HIGHEST, BFD_RELOC_PPC64_HIGHEST_S, BFD_RELOC_PPC64_TOC16_LO, BFD_RELOC_PPC64_TOC16_HI, BFD_RELOC_PPC64_TOC16_HA, BFD_RELOC_PPC64_TOC, BFD_RELOC_PPC64_PLTGOT16, BFD_RELOC_PPC64_PLTGOT16_LO, BFD_RELOC_PPC64_PLTGOT16_HI, BFD_RELOC_PPC64_PLTGOT16_HA, BFD_RELOC_PPC64_ADDR16_DS, BFD_RELOC_PPC64_ADDR16_LO_DS, BFD_RELOC_PPC64_GOT16_DS, BFD_RELOC_PPC64_GOT16_LO_DS, BFD_RELOC_PPC64_PLT16_LO_DS, BFD_RELOC_PPC64_SECTOFF_DS, BFD_RELOC_PPC64_SECTOFF_LO_DS, BFD_RELOC_PPC64_TOC16_DS, BFD_RELOC_PPC64_TOC16_LO_DS, BFD_RELOC_PPC64_PLTGOT16_DS, BFD_RELOC_PPC64_PLTGOT16_LO_DS, /* PowerPC and PowerPC64 thread-local storage relocations. */ BFD_RELOC_PPC_TLS, + BFD_RELOC_PPC_TLSGD, + BFD_RELOC_PPC_TLSLD, BFD_RELOC_PPC_DTPMOD, BFD_RELOC_PPC_TPREL16, BFD_RELOC_PPC_TPREL16_LO, BFD_RELOC_PPC_TPREL16_HI, BFD_RELOC_PPC_TPREL16_HA, BFD_RELOC_PPC_TPREL, BFD_RELOC_PPC_DTPREL16, BFD_RELOC_PPC_DTPREL16_LO, BFD_RELOC_PPC_DTPREL16_HI, BFD_RELOC_PPC_DTPREL16_HA, BFD_RELOC_PPC_DTPREL, BFD_RELOC_PPC_GOT_TLSGD16, BFD_RELOC_PPC_GOT_TLSGD16_LO, BFD_RELOC_PPC_GOT_TLSGD16_HI, BFD_RELOC_PPC_GOT_TLSGD16_HA, BFD_RELOC_PPC_GOT_TLSLD16, BFD_RELOC_PPC_GOT_TLSLD16_LO, BFD_RELOC_PPC_GOT_TLSLD16_HI, BFD_RELOC_PPC_GOT_TLSLD16_HA, BFD_RELOC_PPC_GOT_TPREL16, BFD_RELOC_PPC_GOT_TPREL16_LO, BFD_RELOC_PPC_GOT_TPREL16_HI, BFD_RELOC_PPC_GOT_TPREL16_HA, BFD_RELOC_PPC_GOT_DTPREL16, BFD_RELOC_PPC_GOT_DTPREL16_LO, BFD_RELOC_PPC_GOT_DTPREL16_HI, BFD_RELOC_PPC_GOT_DTPREL16_HA, BFD_RELOC_PPC64_TPREL16_DS, BFD_RELOC_PPC64_TPREL16_LO_DS, BFD_RELOC_PPC64_TPREL16_HIGHER, BFD_RELOC_PPC64_TPREL16_HIGHERA, BFD_RELOC_PPC64_TPREL16_HIGHEST, BFD_RELOC_PPC64_TPREL16_HIGHESTA, BFD_RELOC_PPC64_DTPREL16_DS, BFD_RELOC_PPC64_DTPREL16_LO_DS, BFD_RELOC_PPC64_DTPREL16_HIGHER, BFD_RELOC_PPC64_DTPREL16_HIGHERA, BFD_RELOC_PPC64_DTPREL16_HIGHEST, BFD_RELOC_PPC64_DTPREL16_HIGHESTA, /* IBM 370/390 relocations */ BFD_RELOC_I370_D12, /* The type of reloc used to build a constructor table - at the moment probably a 32 bit wide absolute relocation, but the target can choose. It generally does map to one of the other relocation types. */ BFD_RELOC_CTOR, /* ARM 26 bit pc-relative branch. The lowest two bits must be zero and are not stored in the instruction. */ BFD_RELOC_ARM_PCREL_BRANCH, /* ARM 26 bit pc-relative branch. The lowest bit must be zero and is not stored in the instruction. The 2nd lowest bit comes from a 1 bit field in the instruction. */ BFD_RELOC_ARM_PCREL_BLX, /* Thumb 22 bit pc-relative branch. The lowest bit must be zero and is not stored in the instruction. The 2nd lowest bit comes from a 1 bit field in the instruction. */ BFD_RELOC_THUMB_PCREL_BLX, /* ARM 26-bit pc-relative branch for an unconditional BL or BLX instruction. */ BFD_RELOC_ARM_PCREL_CALL, /* ARM 26-bit pc-relative branch for B or conditional BL instruction. */ BFD_RELOC_ARM_PCREL_JUMP, /* Thumb 7-, 9-, 12-, 20-, 23-, and 25-bit pc-relative branches. The lowest bit must be zero and is not stored in the instruction. Note that the corresponding ELF R_ARM_THM_JUMPnn constant has an "nn" one smaller in all cases. Note further that BRANCH23 corresponds to R_ARM_THM_CALL. */ BFD_RELOC_THUMB_PCREL_BRANCH7, BFD_RELOC_THUMB_PCREL_BRANCH9, BFD_RELOC_THUMB_PCREL_BRANCH12, BFD_RELOC_THUMB_PCREL_BRANCH20, BFD_RELOC_THUMB_PCREL_BRANCH23, BFD_RELOC_THUMB_PCREL_BRANCH25, /* 12-bit immediate offset, used in ARM-format ldr and str instructions. */ BFD_RELOC_ARM_OFFSET_IMM, /* 5-bit immediate offset, used in Thumb-format ldr and str instructions. */ BFD_RELOC_ARM_THUMB_OFFSET, /* Pc-relative or absolute relocation depending on target. Used for entries in .init_array sections. */ BFD_RELOC_ARM_TARGET1, /* Read-only segment base relative address. */ BFD_RELOC_ARM_ROSEGREL32, /* Data segment base relative address. */ BFD_RELOC_ARM_SBREL32, /* This reloc is used for references to RTTI data from exception handling tables. The actual definition depends on the target. It may be a pc-relative or some form of GOT-indirect relocation. */ BFD_RELOC_ARM_TARGET2, /* 31-bit PC relative address. */ BFD_RELOC_ARM_PREL31, /* Low and High halfword relocations for MOVW and MOVT instructions. */ BFD_RELOC_ARM_MOVW, BFD_RELOC_ARM_MOVT, BFD_RELOC_ARM_MOVW_PCREL, BFD_RELOC_ARM_MOVT_PCREL, BFD_RELOC_ARM_THUMB_MOVW, BFD_RELOC_ARM_THUMB_MOVT, BFD_RELOC_ARM_THUMB_MOVW_PCREL, BFD_RELOC_ARM_THUMB_MOVT_PCREL, /* Relocations for setting up GOTs and PLTs for shared libraries. */ BFD_RELOC_ARM_JUMP_SLOT, BFD_RELOC_ARM_GLOB_DAT, BFD_RELOC_ARM_GOT32, BFD_RELOC_ARM_PLT32, BFD_RELOC_ARM_RELATIVE, BFD_RELOC_ARM_GOTOFF, BFD_RELOC_ARM_GOTPC, /* ARM thread-local storage relocations. */ BFD_RELOC_ARM_TLS_GD32, BFD_RELOC_ARM_TLS_LDO32, BFD_RELOC_ARM_TLS_LDM32, BFD_RELOC_ARM_TLS_DTPOFF32, BFD_RELOC_ARM_TLS_DTPMOD32, BFD_RELOC_ARM_TLS_TPOFF32, BFD_RELOC_ARM_TLS_IE32, BFD_RELOC_ARM_TLS_LE32, /* ARM group relocations. */ BFD_RELOC_ARM_ALU_PC_G0_NC, BFD_RELOC_ARM_ALU_PC_G0, BFD_RELOC_ARM_ALU_PC_G1_NC, BFD_RELOC_ARM_ALU_PC_G1, BFD_RELOC_ARM_ALU_PC_G2, BFD_RELOC_ARM_LDR_PC_G0, BFD_RELOC_ARM_LDR_PC_G1, BFD_RELOC_ARM_LDR_PC_G2, BFD_RELOC_ARM_LDRS_PC_G0, BFD_RELOC_ARM_LDRS_PC_G1, BFD_RELOC_ARM_LDRS_PC_G2, BFD_RELOC_ARM_LDC_PC_G0, BFD_RELOC_ARM_LDC_PC_G1, BFD_RELOC_ARM_LDC_PC_G2, BFD_RELOC_ARM_ALU_SB_G0_NC, BFD_RELOC_ARM_ALU_SB_G0, BFD_RELOC_ARM_ALU_SB_G1_NC, BFD_RELOC_ARM_ALU_SB_G1, BFD_RELOC_ARM_ALU_SB_G2, BFD_RELOC_ARM_LDR_SB_G0, BFD_RELOC_ARM_LDR_SB_G1, BFD_RELOC_ARM_LDR_SB_G2, BFD_RELOC_ARM_LDRS_SB_G0, BFD_RELOC_ARM_LDRS_SB_G1, BFD_RELOC_ARM_LDRS_SB_G2, BFD_RELOC_ARM_LDC_SB_G0, BFD_RELOC_ARM_LDC_SB_G1, BFD_RELOC_ARM_LDC_SB_G2, /* These relocs are only used within the ARM assembler. They are not (at present) written to any object files. */ BFD_RELOC_ARM_IMMEDIATE, BFD_RELOC_ARM_ADRL_IMMEDIATE, BFD_RELOC_ARM_T32_IMMEDIATE, BFD_RELOC_ARM_T32_ADD_IMM, BFD_RELOC_ARM_T32_IMM12, BFD_RELOC_ARM_T32_ADD_PC12, BFD_RELOC_ARM_SHIFT_IMM, BFD_RELOC_ARM_SMC, BFD_RELOC_ARM_SWI, BFD_RELOC_ARM_MULTI, BFD_RELOC_ARM_CP_OFF_IMM, BFD_RELOC_ARM_CP_OFF_IMM_S2, BFD_RELOC_ARM_T32_CP_OFF_IMM, BFD_RELOC_ARM_T32_CP_OFF_IMM_S2, BFD_RELOC_ARM_ADR_IMM, BFD_RELOC_ARM_LDR_IMM, BFD_RELOC_ARM_LITERAL, BFD_RELOC_ARM_IN_POOL, BFD_RELOC_ARM_OFFSET_IMM8, BFD_RELOC_ARM_T32_OFFSET_U8, BFD_RELOC_ARM_T32_OFFSET_IMM, BFD_RELOC_ARM_HWLITERAL, BFD_RELOC_ARM_THUMB_ADD, BFD_RELOC_ARM_THUMB_IMM, BFD_RELOC_ARM_THUMB_SHIFT, /* Renesas / SuperH SH relocs. Not all of these appear in object files. */ BFD_RELOC_SH_PCDISP8BY2, BFD_RELOC_SH_PCDISP12BY2, BFD_RELOC_SH_IMM3, BFD_RELOC_SH_IMM3U, BFD_RELOC_SH_DISP12, BFD_RELOC_SH_DISP12BY2, BFD_RELOC_SH_DISP12BY4, BFD_RELOC_SH_DISP12BY8, BFD_RELOC_SH_DISP20, BFD_RELOC_SH_DISP20BY8, BFD_RELOC_SH_IMM4, BFD_RELOC_SH_IMM4BY2, BFD_RELOC_SH_IMM4BY4, BFD_RELOC_SH_IMM8, BFD_RELOC_SH_IMM8BY2, BFD_RELOC_SH_IMM8BY4, BFD_RELOC_SH_PCRELIMM8BY2, BFD_RELOC_SH_PCRELIMM8BY4, BFD_RELOC_SH_SWITCH16, BFD_RELOC_SH_SWITCH32, BFD_RELOC_SH_USES, BFD_RELOC_SH_COUNT, BFD_RELOC_SH_ALIGN, BFD_RELOC_SH_CODE, BFD_RELOC_SH_DATA, BFD_RELOC_SH_LABEL, BFD_RELOC_SH_LOOP_START, BFD_RELOC_SH_LOOP_END, BFD_RELOC_SH_COPY, BFD_RELOC_SH_GLOB_DAT, BFD_RELOC_SH_JMP_SLOT, BFD_RELOC_SH_RELATIVE, BFD_RELOC_SH_GOTPC, BFD_RELOC_SH_GOT_LOW16, BFD_RELOC_SH_GOT_MEDLOW16, BFD_RELOC_SH_GOT_MEDHI16, BFD_RELOC_SH_GOT_HI16, BFD_RELOC_SH_GOTPLT_LOW16, BFD_RELOC_SH_GOTPLT_MEDLOW16, BFD_RELOC_SH_GOTPLT_MEDHI16, BFD_RELOC_SH_GOTPLT_HI16, BFD_RELOC_SH_PLT_LOW16, BFD_RELOC_SH_PLT_MEDLOW16, BFD_RELOC_SH_PLT_MEDHI16, BFD_RELOC_SH_PLT_HI16, BFD_RELOC_SH_GOTOFF_LOW16, BFD_RELOC_SH_GOTOFF_MEDLOW16, BFD_RELOC_SH_GOTOFF_MEDHI16, BFD_RELOC_SH_GOTOFF_HI16, BFD_RELOC_SH_GOTPC_LOW16, BFD_RELOC_SH_GOTPC_MEDLOW16, BFD_RELOC_SH_GOTPC_MEDHI16, BFD_RELOC_SH_GOTPC_HI16, BFD_RELOC_SH_COPY64, BFD_RELOC_SH_GLOB_DAT64, BFD_RELOC_SH_JMP_SLOT64, BFD_RELOC_SH_RELATIVE64, BFD_RELOC_SH_GOT10BY4, BFD_RELOC_SH_GOT10BY8, BFD_RELOC_SH_GOTPLT10BY4, BFD_RELOC_SH_GOTPLT10BY8, BFD_RELOC_SH_GOTPLT32, BFD_RELOC_SH_SHMEDIA_CODE, BFD_RELOC_SH_IMMU5, BFD_RELOC_SH_IMMS6, BFD_RELOC_SH_IMMS6BY32, BFD_RELOC_SH_IMMU6, BFD_RELOC_SH_IMMS10, BFD_RELOC_SH_IMMS10BY2, BFD_RELOC_SH_IMMS10BY4, BFD_RELOC_SH_IMMS10BY8, BFD_RELOC_SH_IMMS16, BFD_RELOC_SH_IMMU16, BFD_RELOC_SH_IMM_LOW16, BFD_RELOC_SH_IMM_LOW16_PCREL, BFD_RELOC_SH_IMM_MEDLOW16, BFD_RELOC_SH_IMM_MEDLOW16_PCREL, BFD_RELOC_SH_IMM_MEDHI16, BFD_RELOC_SH_IMM_MEDHI16_PCREL, BFD_RELOC_SH_IMM_HI16, BFD_RELOC_SH_IMM_HI16_PCREL, BFD_RELOC_SH_PT_16, BFD_RELOC_SH_TLS_GD_32, BFD_RELOC_SH_TLS_LD_32, BFD_RELOC_SH_TLS_LDO_32, BFD_RELOC_SH_TLS_IE_32, BFD_RELOC_SH_TLS_LE_32, BFD_RELOC_SH_TLS_DTPMOD32, BFD_RELOC_SH_TLS_DTPOFF32, BFD_RELOC_SH_TLS_TPOFF32, /* ARC Cores relocs. ARC 22 bit pc-relative branch. The lowest two bits must be zero and are not stored in the instruction. The high 20 bits are installed in bits 26 through 7 of the instruction. */ BFD_RELOC_ARC_B22_PCREL, /* ARC 26 bit absolute branch. The lowest two bits must be zero and are not stored in the instruction. The high 24 bits are installed in bits 23 through 0. */ BFD_RELOC_ARC_B26, /* ADI Blackfin 16 bit immediate absolute reloc. */ BFD_RELOC_BFIN_16_IMM, /* ADI Blackfin 16 bit immediate absolute reloc higher 16 bits. */ BFD_RELOC_BFIN_16_HIGH, /* ADI Blackfin 'a' part of LSETUP. */ BFD_RELOC_BFIN_4_PCREL, /* ADI Blackfin. */ BFD_RELOC_BFIN_5_PCREL, /* ADI Blackfin 16 bit immediate absolute reloc lower 16 bits. */ BFD_RELOC_BFIN_16_LOW, /* ADI Blackfin. */ BFD_RELOC_BFIN_10_PCREL, /* ADI Blackfin 'b' part of LSETUP. */ BFD_RELOC_BFIN_11_PCREL, /* ADI Blackfin. */ BFD_RELOC_BFIN_12_PCREL_JUMP, /* ADI Blackfin Short jump, pcrel. */ BFD_RELOC_BFIN_12_PCREL_JUMP_S, /* ADI Blackfin Call.x not implemented. */ BFD_RELOC_BFIN_24_PCREL_CALL_X, /* ADI Blackfin Long Jump pcrel. */ BFD_RELOC_BFIN_24_PCREL_JUMP_L, /* ADI Blackfin FD-PIC relocations. */ BFD_RELOC_BFIN_GOT17M4, BFD_RELOC_BFIN_GOTHI, BFD_RELOC_BFIN_GOTLO, BFD_RELOC_BFIN_FUNCDESC, BFD_RELOC_BFIN_FUNCDESC_GOT17M4, BFD_RELOC_BFIN_FUNCDESC_GOTHI, BFD_RELOC_BFIN_FUNCDESC_GOTLO, BFD_RELOC_BFIN_FUNCDESC_VALUE, BFD_RELOC_BFIN_FUNCDESC_GOTOFF17M4, BFD_RELOC_BFIN_FUNCDESC_GOTOFFHI, BFD_RELOC_BFIN_FUNCDESC_GOTOFFLO, BFD_RELOC_BFIN_GOTOFF17M4, BFD_RELOC_BFIN_GOTOFFHI, BFD_RELOC_BFIN_GOTOFFLO, /* ADI Blackfin GOT relocation. */ BFD_RELOC_BFIN_GOT, /* ADI Blackfin PLTPC relocation. */ BFD_RELOC_BFIN_PLTPC, /* ADI Blackfin arithmetic relocation. */ BFD_ARELOC_BFIN_PUSH, /* ADI Blackfin arithmetic relocation. */ BFD_ARELOC_BFIN_CONST, /* ADI Blackfin arithmetic relocation. */ BFD_ARELOC_BFIN_ADD, /* ADI Blackfin arithmetic relocation. */ BFD_ARELOC_BFIN_SUB, /* ADI Blackfin arithmetic relocation. */ BFD_ARELOC_BFIN_MULT, /* ADI Blackfin arithmetic relocation. */ BFD_ARELOC_BFIN_DIV, /* ADI Blackfin arithmetic relocation. */ BFD_ARELOC_BFIN_MOD, /* ADI Blackfin arithmetic relocation. */ BFD_ARELOC_BFIN_LSHIFT, /* ADI Blackfin arithmetic relocation. */ BFD_ARELOC_BFIN_RSHIFT, /* ADI Blackfin arithmetic relocation. */ BFD_ARELOC_BFIN_AND, /* ADI Blackfin arithmetic relocation. */ BFD_ARELOC_BFIN_OR, /* ADI Blackfin arithmetic relocation. */ BFD_ARELOC_BFIN_XOR, /* ADI Blackfin arithmetic relocation. */ BFD_ARELOC_BFIN_LAND, /* ADI Blackfin arithmetic relocation. */ BFD_ARELOC_BFIN_LOR, /* ADI Blackfin arithmetic relocation. */ BFD_ARELOC_BFIN_LEN, /* ADI Blackfin arithmetic relocation. */ BFD_ARELOC_BFIN_NEG, /* ADI Blackfin arithmetic relocation. */ BFD_ARELOC_BFIN_COMP, /* ADI Blackfin arithmetic relocation. */ BFD_ARELOC_BFIN_PAGE, /* ADI Blackfin arithmetic relocation. */ BFD_ARELOC_BFIN_HWPAGE, /* ADI Blackfin arithmetic relocation. */ BFD_ARELOC_BFIN_ADDR, /* Mitsubishi D10V relocs. This is a 10-bit reloc with the right 2 bits assumed to be 0. */ BFD_RELOC_D10V_10_PCREL_R, /* Mitsubishi D10V relocs. This is a 10-bit reloc with the right 2 bits assumed to be 0. This is the same as the previous reloc except it is in the left container, i.e., shifted left 15 bits. */ BFD_RELOC_D10V_10_PCREL_L, /* This is an 18-bit reloc with the right 2 bits assumed to be 0. */ BFD_RELOC_D10V_18, /* This is an 18-bit reloc with the right 2 bits assumed to be 0. */ BFD_RELOC_D10V_18_PCREL, /* Mitsubishi D30V relocs. This is a 6-bit absolute reloc. */ BFD_RELOC_D30V_6, /* This is a 6-bit pc-relative reloc with the right 3 bits assumed to be 0. */ BFD_RELOC_D30V_9_PCREL, /* This is a 6-bit pc-relative reloc with the right 3 bits assumed to be 0. Same as the previous reloc but on the right side of the container. */ BFD_RELOC_D30V_9_PCREL_R, /* This is a 12-bit absolute reloc with the right 3 bitsassumed to be 0. */ BFD_RELOC_D30V_15, /* This is a 12-bit pc-relative reloc with the right 3 bits assumed to be 0. */ BFD_RELOC_D30V_15_PCREL, /* This is a 12-bit pc-relative reloc with the right 3 bits assumed to be 0. Same as the previous reloc but on the right side of the container. */ BFD_RELOC_D30V_15_PCREL_R, /* This is an 18-bit absolute reloc with the right 3 bits assumed to be 0. */ BFD_RELOC_D30V_21, /* This is an 18-bit pc-relative reloc with the right 3 bits assumed to be 0. */ BFD_RELOC_D30V_21_PCREL, /* This is an 18-bit pc-relative reloc with the right 3 bits assumed to be 0. Same as the previous reloc but on the right side of the container. */ BFD_RELOC_D30V_21_PCREL_R, /* This is a 32-bit absolute reloc. */ BFD_RELOC_D30V_32, /* This is a 32-bit pc-relative reloc. */ BFD_RELOC_D30V_32_PCREL, /* DLX relocs */ BFD_RELOC_DLX_HI16_S, /* DLX relocs */ BFD_RELOC_DLX_LO16, /* DLX relocs */ BFD_RELOC_DLX_JMP26, /* Renesas M16C/M32C Relocations. */ BFD_RELOC_M32C_HI8, BFD_RELOC_M32C_RL_JUMP, BFD_RELOC_M32C_RL_1ADDR, BFD_RELOC_M32C_RL_2ADDR, /* Renesas M32R (formerly Mitsubishi M32R) relocs. This is a 24 bit absolute address. */ BFD_RELOC_M32R_24, /* This is a 10-bit pc-relative reloc with the right 2 bits assumed to be 0. */ BFD_RELOC_M32R_10_PCREL, /* This is an 18-bit reloc with the right 2 bits assumed to be 0. */ BFD_RELOC_M32R_18_PCREL, /* This is a 26-bit reloc with the right 2 bits assumed to be 0. */ BFD_RELOC_M32R_26_PCREL, /* This is a 16-bit reloc containing the high 16 bits of an address used when the lower 16 bits are treated as unsigned. */ BFD_RELOC_M32R_HI16_ULO, /* This is a 16-bit reloc containing the high 16 bits of an address used when the lower 16 bits are treated as signed. */ BFD_RELOC_M32R_HI16_SLO, /* This is a 16-bit reloc containing the lower 16 bits of an address. */ BFD_RELOC_M32R_LO16, /* This is a 16-bit reloc containing the small data area offset for use in add3, load, and store instructions. */ BFD_RELOC_M32R_SDA16, /* For PIC. */ BFD_RELOC_M32R_GOT24, BFD_RELOC_M32R_26_PLTREL, BFD_RELOC_M32R_COPY, BFD_RELOC_M32R_GLOB_DAT, BFD_RELOC_M32R_JMP_SLOT, BFD_RELOC_M32R_RELATIVE, BFD_RELOC_M32R_GOTOFF, BFD_RELOC_M32R_GOTOFF_HI_ULO, BFD_RELOC_M32R_GOTOFF_HI_SLO, BFD_RELOC_M32R_GOTOFF_LO, BFD_RELOC_M32R_GOTPC24, BFD_RELOC_M32R_GOT16_HI_ULO, BFD_RELOC_M32R_GOT16_HI_SLO, BFD_RELOC_M32R_GOT16_LO, BFD_RELOC_M32R_GOTPC_HI_ULO, BFD_RELOC_M32R_GOTPC_HI_SLO, BFD_RELOC_M32R_GOTPC_LO, /* This is a 9-bit reloc */ BFD_RELOC_V850_9_PCREL, /* This is a 22-bit reloc */ BFD_RELOC_V850_22_PCREL, /* This is a 16 bit offset from the short data area pointer. */ BFD_RELOC_V850_SDA_16_16_OFFSET, /* This is a 16 bit offset (of which only 15 bits are used) from the short data area pointer. */ BFD_RELOC_V850_SDA_15_16_OFFSET, /* This is a 16 bit offset from the zero data area pointer. */ BFD_RELOC_V850_ZDA_16_16_OFFSET, /* This is a 16 bit offset (of which only 15 bits are used) from the zero data area pointer. */ BFD_RELOC_V850_ZDA_15_16_OFFSET, /* This is an 8 bit offset (of which only 6 bits are used) from the tiny data area pointer. */ BFD_RELOC_V850_TDA_6_8_OFFSET, /* This is an 8bit offset (of which only 7 bits are used) from the tiny data area pointer. */ BFD_RELOC_V850_TDA_7_8_OFFSET, /* This is a 7 bit offset from the tiny data area pointer. */ BFD_RELOC_V850_TDA_7_7_OFFSET, /* This is a 16 bit offset from the tiny data area pointer. */ BFD_RELOC_V850_TDA_16_16_OFFSET, /* This is a 5 bit offset (of which only 4 bits are used) from the tiny data area pointer. */ BFD_RELOC_V850_TDA_4_5_OFFSET, /* This is a 4 bit offset from the tiny data area pointer. */ BFD_RELOC_V850_TDA_4_4_OFFSET, /* This is a 16 bit offset from the short data area pointer, with the bits placed non-contiguously in the instruction. */ BFD_RELOC_V850_SDA_16_16_SPLIT_OFFSET, /* This is a 16 bit offset from the zero data area pointer, with the bits placed non-contiguously in the instruction. */ BFD_RELOC_V850_ZDA_16_16_SPLIT_OFFSET, /* This is a 6 bit offset from the call table base pointer. */ BFD_RELOC_V850_CALLT_6_7_OFFSET, /* This is a 16 bit offset from the call table base pointer. */ BFD_RELOC_V850_CALLT_16_16_OFFSET, /* Used for relaxing indirect function calls. */ BFD_RELOC_V850_LONGCALL, /* Used for relaxing indirect jumps. */ BFD_RELOC_V850_LONGJUMP, /* Used to maintain alignment whilst relaxing. */ BFD_RELOC_V850_ALIGN, /* This is a variation of BFD_RELOC_LO16 that can be used in v850e ld.bu instructions. */ BFD_RELOC_V850_LO16_SPLIT_OFFSET, /* This is a 32bit pcrel reloc for the mn10300, offset by two bytes in the instruction. */ BFD_RELOC_MN10300_32_PCREL, /* This is a 16bit pcrel reloc for the mn10300, offset by two bytes in the instruction. */ BFD_RELOC_MN10300_16_PCREL, /* This is a 8bit DP reloc for the tms320c30, where the most significant 8 bits of a 24 bit word are placed into the least significant 8 bits of the opcode. */ BFD_RELOC_TIC30_LDP, /* This is a 7bit reloc for the tms320c54x, where the least significant 7 bits of a 16 bit word are placed into the least significant 7 bits of the opcode. */ BFD_RELOC_TIC54X_PARTLS7, /* This is a 9bit DP reloc for the tms320c54x, where the most significant 9 bits of a 16 bit word are placed into the least significant 9 bits of the opcode. */ BFD_RELOC_TIC54X_PARTMS9, /* This is an extended address 23-bit reloc for the tms320c54x. */ BFD_RELOC_TIC54X_23, /* This is a 16-bit reloc for the tms320c54x, where the least significant 16 bits of a 23-bit extended address are placed into the opcode. */ BFD_RELOC_TIC54X_16_OF_23, /* This is a reloc for the tms320c54x, where the most significant 7 bits of a 23-bit extended address are placed into the opcode. */ BFD_RELOC_TIC54X_MS7_OF_23, /* This is a 48 bit reloc for the FR30 that stores 32 bits. */ BFD_RELOC_FR30_48, /* This is a 32 bit reloc for the FR30 that stores 20 bits split up into two sections. */ BFD_RELOC_FR30_20, /* This is a 16 bit reloc for the FR30 that stores a 6 bit word offset in 4 bits. */ BFD_RELOC_FR30_6_IN_4, /* This is a 16 bit reloc for the FR30 that stores an 8 bit byte offset into 8 bits. */ BFD_RELOC_FR30_8_IN_8, /* This is a 16 bit reloc for the FR30 that stores a 9 bit short offset into 8 bits. */ BFD_RELOC_FR30_9_IN_8, /* This is a 16 bit reloc for the FR30 that stores a 10 bit word offset into 8 bits. */ BFD_RELOC_FR30_10_IN_8, /* This is a 16 bit reloc for the FR30 that stores a 9 bit pc relative short offset into 8 bits. */ BFD_RELOC_FR30_9_PCREL, /* This is a 16 bit reloc for the FR30 that stores a 12 bit pc relative short offset into 11 bits. */ BFD_RELOC_FR30_12_PCREL, /* Motorola Mcore relocations. */ BFD_RELOC_MCORE_PCREL_IMM8BY4, BFD_RELOC_MCORE_PCREL_IMM11BY2, BFD_RELOC_MCORE_PCREL_IMM4BY2, BFD_RELOC_MCORE_PCREL_32, BFD_RELOC_MCORE_PCREL_JSR_IMM11BY2, BFD_RELOC_MCORE_RVA, /* Toshiba Media Processor Relocations. */ BFD_RELOC_MEP_8, BFD_RELOC_MEP_16, BFD_RELOC_MEP_32, BFD_RELOC_MEP_PCREL8A2, BFD_RELOC_MEP_PCREL12A2, BFD_RELOC_MEP_PCREL17A2, BFD_RELOC_MEP_PCREL24A2, BFD_RELOC_MEP_PCABS24A2, BFD_RELOC_MEP_LOW16, BFD_RELOC_MEP_HI16U, BFD_RELOC_MEP_HI16S, BFD_RELOC_MEP_GPREL, BFD_RELOC_MEP_TPREL, BFD_RELOC_MEP_TPREL7, BFD_RELOC_MEP_TPREL7A2, BFD_RELOC_MEP_TPREL7A4, BFD_RELOC_MEP_UIMM24, BFD_RELOC_MEP_ADDR24A4, BFD_RELOC_MEP_GNU_VTINHERIT, BFD_RELOC_MEP_GNU_VTENTRY, /* These are relocations for the GETA instruction. */ BFD_RELOC_MMIX_GETA, BFD_RELOC_MMIX_GETA_1, BFD_RELOC_MMIX_GETA_2, BFD_RELOC_MMIX_GETA_3, /* These are relocations for a conditional branch instruction. */ BFD_RELOC_MMIX_CBRANCH, BFD_RELOC_MMIX_CBRANCH_J, BFD_RELOC_MMIX_CBRANCH_1, BFD_RELOC_MMIX_CBRANCH_2, BFD_RELOC_MMIX_CBRANCH_3, /* These are relocations for the PUSHJ instruction. */ BFD_RELOC_MMIX_PUSHJ, BFD_RELOC_MMIX_PUSHJ_1, BFD_RELOC_MMIX_PUSHJ_2, BFD_RELOC_MMIX_PUSHJ_3, BFD_RELOC_MMIX_PUSHJ_STUBBABLE, /* These are relocations for the JMP instruction. */ BFD_RELOC_MMIX_JMP, BFD_RELOC_MMIX_JMP_1, BFD_RELOC_MMIX_JMP_2, BFD_RELOC_MMIX_JMP_3, /* This is a relocation for a relative address as in a GETA instruction or a branch. */ BFD_RELOC_MMIX_ADDR19, /* This is a relocation for a relative address as in a JMP instruction. */ BFD_RELOC_MMIX_ADDR27, /* This is a relocation for an instruction field that may be a general register or a value 0..255. */ BFD_RELOC_MMIX_REG_OR_BYTE, /* This is a relocation for an instruction field that may be a general register. */ BFD_RELOC_MMIX_REG, /* This is a relocation for two instruction fields holding a register and an offset, the equivalent of the relocation. */ BFD_RELOC_MMIX_BASE_PLUS_OFFSET, /* This relocation is an assertion that the expression is not allocated as a global register. It does not modify contents. */ BFD_RELOC_MMIX_LOCAL, /* This is a 16 bit reloc for the AVR that stores 8 bit pc relative short offset into 7 bits. */ BFD_RELOC_AVR_7_PCREL, /* This is a 16 bit reloc for the AVR that stores 13 bit pc relative short offset into 12 bits. */ BFD_RELOC_AVR_13_PCREL, /* This is a 16 bit reloc for the AVR that stores 17 bit value (usually program memory address) into 16 bits. */ BFD_RELOC_AVR_16_PM, /* This is a 16 bit reloc for the AVR that stores 8 bit value (usually data memory address) into 8 bit immediate value of LDI insn. */ BFD_RELOC_AVR_LO8_LDI, /* This is a 16 bit reloc for the AVR that stores 8 bit value (high 8 bit of data memory address) into 8 bit immediate value of LDI insn. */ BFD_RELOC_AVR_HI8_LDI, /* This is a 16 bit reloc for the AVR that stores 8 bit value (most high 8 bit of program memory address) into 8 bit immediate value of LDI insn. */ BFD_RELOC_AVR_HH8_LDI, /* This is a 16 bit reloc for the AVR that stores 8 bit value (most high 8 bit of 32 bit value) into 8 bit immediate value of LDI insn. */ BFD_RELOC_AVR_MS8_LDI, /* This is a 16 bit reloc for the AVR that stores negated 8 bit value (usually data memory address) into 8 bit immediate value of SUBI insn. */ BFD_RELOC_AVR_LO8_LDI_NEG, /* This is a 16 bit reloc for the AVR that stores negated 8 bit value (high 8 bit of data memory address) into 8 bit immediate value of SUBI insn. */ BFD_RELOC_AVR_HI8_LDI_NEG, /* This is a 16 bit reloc for the AVR that stores negated 8 bit value (most high 8 bit of program memory address) into 8 bit immediate value of LDI or SUBI insn. */ BFD_RELOC_AVR_HH8_LDI_NEG, /* This is a 16 bit reloc for the AVR that stores negated 8 bit value (msb of 32 bit value) into 8 bit immediate value of LDI insn. */ BFD_RELOC_AVR_MS8_LDI_NEG, /* This is a 16 bit reloc for the AVR that stores 8 bit value (usually command address) into 8 bit immediate value of LDI insn. */ BFD_RELOC_AVR_LO8_LDI_PM, /* This is a 16 bit reloc for the AVR that stores 8 bit value (command address) into 8 bit immediate value of LDI insn. If the address is beyond the 128k boundary, the linker inserts a jump stub for this reloc in the lower 128k. */ BFD_RELOC_AVR_LO8_LDI_GS, /* This is a 16 bit reloc for the AVR that stores 8 bit value (high 8 bit of command address) into 8 bit immediate value of LDI insn. */ BFD_RELOC_AVR_HI8_LDI_PM, /* This is a 16 bit reloc for the AVR that stores 8 bit value (high 8 bit of command address) into 8 bit immediate value of LDI insn. If the address is beyond the 128k boundary, the linker inserts a jump stub for this reloc below 128k. */ BFD_RELOC_AVR_HI8_LDI_GS, /* This is a 16 bit reloc for the AVR that stores 8 bit value (most high 8 bit of command address) into 8 bit immediate value of LDI insn. */ BFD_RELOC_AVR_HH8_LDI_PM, /* This is a 16 bit reloc for the AVR that stores negated 8 bit value (usually command address) into 8 bit immediate value of SUBI insn. */ BFD_RELOC_AVR_LO8_LDI_PM_NEG, /* This is a 16 bit reloc for the AVR that stores negated 8 bit value (high 8 bit of 16 bit command address) into 8 bit immediate value of SUBI insn. */ BFD_RELOC_AVR_HI8_LDI_PM_NEG, /* This is a 16 bit reloc for the AVR that stores negated 8 bit value (high 6 bit of 22 bit command address) into 8 bit immediate value of SUBI insn. */ BFD_RELOC_AVR_HH8_LDI_PM_NEG, /* This is a 32 bit reloc for the AVR that stores 23 bit value into 22 bits. */ BFD_RELOC_AVR_CALL, /* This is a 16 bit reloc for the AVR that stores all needed bits for absolute addressing with ldi with overflow check to linktime */ BFD_RELOC_AVR_LDI, /* This is a 6 bit reloc for the AVR that stores offset for ldd/std instructions */ BFD_RELOC_AVR_6, /* This is a 6 bit reloc for the AVR that stores offset for adiw/sbiw instructions */ BFD_RELOC_AVR_6_ADIW, /* Direct 12 bit. */ BFD_RELOC_390_12, /* 12 bit GOT offset. */ BFD_RELOC_390_GOT12, /* 32 bit PC relative PLT address. */ BFD_RELOC_390_PLT32, /* Copy symbol at runtime. */ BFD_RELOC_390_COPY, /* Create GOT entry. */ BFD_RELOC_390_GLOB_DAT, /* Create PLT entry. */ BFD_RELOC_390_JMP_SLOT, /* Adjust by program base. */ BFD_RELOC_390_RELATIVE, /* 32 bit PC relative offset to GOT. */ BFD_RELOC_390_GOTPC, /* 16 bit GOT offset. */ BFD_RELOC_390_GOT16, /* PC relative 16 bit shifted by 1. */ BFD_RELOC_390_PC16DBL, /* 16 bit PC rel. PLT shifted by 1. */ BFD_RELOC_390_PLT16DBL, /* PC relative 32 bit shifted by 1. */ BFD_RELOC_390_PC32DBL, /* 32 bit PC rel. PLT shifted by 1. */ BFD_RELOC_390_PLT32DBL, /* 32 bit PC rel. GOT shifted by 1. */ BFD_RELOC_390_GOTPCDBL, /* 64 bit GOT offset. */ BFD_RELOC_390_GOT64, /* 64 bit PC relative PLT address. */ BFD_RELOC_390_PLT64, /* 32 bit rel. offset to GOT entry. */ BFD_RELOC_390_GOTENT, /* 64 bit offset to GOT. */ BFD_RELOC_390_GOTOFF64, /* 12-bit offset to symbol-entry within GOT, with PLT handling. */ BFD_RELOC_390_GOTPLT12, /* 16-bit offset to symbol-entry within GOT, with PLT handling. */ BFD_RELOC_390_GOTPLT16, /* 32-bit offset to symbol-entry within GOT, with PLT handling. */ BFD_RELOC_390_GOTPLT32, /* 64-bit offset to symbol-entry within GOT, with PLT handling. */ BFD_RELOC_390_GOTPLT64, /* 32-bit rel. offset to symbol-entry within GOT, with PLT handling. */ BFD_RELOC_390_GOTPLTENT, /* 16-bit rel. offset from the GOT to a PLT entry. */ BFD_RELOC_390_PLTOFF16, /* 32-bit rel. offset from the GOT to a PLT entry. */ BFD_RELOC_390_PLTOFF32, /* 64-bit rel. offset from the GOT to a PLT entry. */ BFD_RELOC_390_PLTOFF64, /* s390 tls relocations. */ BFD_RELOC_390_TLS_LOAD, BFD_RELOC_390_TLS_GDCALL, BFD_RELOC_390_TLS_LDCALL, BFD_RELOC_390_TLS_GD32, BFD_RELOC_390_TLS_GD64, BFD_RELOC_390_TLS_GOTIE12, BFD_RELOC_390_TLS_GOTIE32, BFD_RELOC_390_TLS_GOTIE64, BFD_RELOC_390_TLS_LDM32, BFD_RELOC_390_TLS_LDM64, BFD_RELOC_390_TLS_IE32, BFD_RELOC_390_TLS_IE64, BFD_RELOC_390_TLS_IEENT, BFD_RELOC_390_TLS_LE32, BFD_RELOC_390_TLS_LE64, BFD_RELOC_390_TLS_LDO32, BFD_RELOC_390_TLS_LDO64, BFD_RELOC_390_TLS_DTPMOD, BFD_RELOC_390_TLS_DTPOFF, BFD_RELOC_390_TLS_TPOFF, /* Long displacement extension. */ BFD_RELOC_390_20, BFD_RELOC_390_GOT20, BFD_RELOC_390_GOTPLT20, BFD_RELOC_390_TLS_GOTIE20, /* Score relocations */ BFD_RELOC_SCORE_DUMMY1, /* Low 16 bit for load/store */ BFD_RELOC_SCORE_GPREL15, /* This is a 24-bit reloc with the right 1 bit assumed to be 0 */ BFD_RELOC_SCORE_DUMMY2, BFD_RELOC_SCORE_JMP, /* This is a 19-bit reloc with the right 1 bit assumed to be 0 */ BFD_RELOC_SCORE_BRANCH, /* This is a 11-bit reloc with the right 1 bit assumed to be 0 */ BFD_RELOC_SCORE16_JMP, /* This is a 8-bit reloc with the right 1 bit assumed to be 0 */ BFD_RELOC_SCORE16_BRANCH, /* Undocumented Score relocs */ BFD_RELOC_SCORE_GOT15, BFD_RELOC_SCORE_GOT_LO16, BFD_RELOC_SCORE_CALL15, BFD_RELOC_SCORE_DUMMY_HI16, /* Scenix IP2K - 9-bit register number / data address */ BFD_RELOC_IP2K_FR9, /* Scenix IP2K - 4-bit register/data bank number */ BFD_RELOC_IP2K_BANK, /* Scenix IP2K - low 13 bits of instruction word address */ BFD_RELOC_IP2K_ADDR16CJP, /* Scenix IP2K - high 3 bits of instruction word address */ BFD_RELOC_IP2K_PAGE3, /* Scenix IP2K - ext/low/high 8 bits of data address */ BFD_RELOC_IP2K_LO8DATA, BFD_RELOC_IP2K_HI8DATA, BFD_RELOC_IP2K_EX8DATA, /* Scenix IP2K - low/high 8 bits of instruction word address */ BFD_RELOC_IP2K_LO8INSN, BFD_RELOC_IP2K_HI8INSN, /* Scenix IP2K - even/odd PC modifier to modify snb pcl.0 */ BFD_RELOC_IP2K_PC_SKIP, /* Scenix IP2K - 16 bit word address in text section. */ BFD_RELOC_IP2K_TEXT, /* Scenix IP2K - 7-bit sp or dp offset */ BFD_RELOC_IP2K_FR_OFFSET, /* Scenix VPE4K coprocessor - data/insn-space addressing */ BFD_RELOC_VPE4KMATH_DATA, BFD_RELOC_VPE4KMATH_INSN, /* These two relocations are used by the linker to determine which of the entries in a C++ virtual function table are actually used. When the --gc-sections option is given, the linker will zero out the entries that are not used, so that the code for those functions need not be included in the output. VTABLE_INHERIT is a zero-space relocation used to describe to the linker the inheritance tree of a C++ virtual function table. The relocation's symbol should be the parent class' vtable, and the relocation should be located at the child vtable. VTABLE_ENTRY is a zero-space relocation that describes the use of a virtual function table entry. The reloc's symbol should refer to the table of the class mentioned in the code. Off of that base, an offset describes the entry that is being used. For Rela hosts, this offset is stored in the reloc's addend. For Rel hosts, we are forced to put this offset in the reloc's section offset. */ BFD_RELOC_VTABLE_INHERIT, BFD_RELOC_VTABLE_ENTRY, /* Intel IA64 Relocations. */ BFD_RELOC_IA64_IMM14, BFD_RELOC_IA64_IMM22, BFD_RELOC_IA64_IMM64, BFD_RELOC_IA64_DIR32MSB, BFD_RELOC_IA64_DIR32LSB, BFD_RELOC_IA64_DIR64MSB, BFD_RELOC_IA64_DIR64LSB, BFD_RELOC_IA64_GPREL22, BFD_RELOC_IA64_GPREL64I, BFD_RELOC_IA64_GPREL32MSB, BFD_RELOC_IA64_GPREL32LSB, BFD_RELOC_IA64_GPREL64MSB, BFD_RELOC_IA64_GPREL64LSB, BFD_RELOC_IA64_LTOFF22, BFD_RELOC_IA64_LTOFF64I, BFD_RELOC_IA64_PLTOFF22, BFD_RELOC_IA64_PLTOFF64I, BFD_RELOC_IA64_PLTOFF64MSB, BFD_RELOC_IA64_PLTOFF64LSB, BFD_RELOC_IA64_FPTR64I, BFD_RELOC_IA64_FPTR32MSB, BFD_RELOC_IA64_FPTR32LSB, BFD_RELOC_IA64_FPTR64MSB, BFD_RELOC_IA64_FPTR64LSB, BFD_RELOC_IA64_PCREL21B, BFD_RELOC_IA64_PCREL21BI, BFD_RELOC_IA64_PCREL21M, BFD_RELOC_IA64_PCREL21F, BFD_RELOC_IA64_PCREL22, BFD_RELOC_IA64_PCREL60B, BFD_RELOC_IA64_PCREL64I, BFD_RELOC_IA64_PCREL32MSB, BFD_RELOC_IA64_PCREL32LSB, BFD_RELOC_IA64_PCREL64MSB, BFD_RELOC_IA64_PCREL64LSB, BFD_RELOC_IA64_LTOFF_FPTR22, BFD_RELOC_IA64_LTOFF_FPTR64I, BFD_RELOC_IA64_LTOFF_FPTR32MSB, BFD_RELOC_IA64_LTOFF_FPTR32LSB, BFD_RELOC_IA64_LTOFF_FPTR64MSB, BFD_RELOC_IA64_LTOFF_FPTR64LSB, BFD_RELOC_IA64_SEGREL32MSB, BFD_RELOC_IA64_SEGREL32LSB, BFD_RELOC_IA64_SEGREL64MSB, BFD_RELOC_IA64_SEGREL64LSB, BFD_RELOC_IA64_SECREL32MSB, BFD_RELOC_IA64_SECREL32LSB, BFD_RELOC_IA64_SECREL64MSB, BFD_RELOC_IA64_SECREL64LSB, BFD_RELOC_IA64_REL32MSB, BFD_RELOC_IA64_REL32LSB, BFD_RELOC_IA64_REL64MSB, BFD_RELOC_IA64_REL64LSB, BFD_RELOC_IA64_LTV32MSB, BFD_RELOC_IA64_LTV32LSB, BFD_RELOC_IA64_LTV64MSB, BFD_RELOC_IA64_LTV64LSB, BFD_RELOC_IA64_IPLTMSB, BFD_RELOC_IA64_IPLTLSB, BFD_RELOC_IA64_COPY, BFD_RELOC_IA64_LTOFF22X, BFD_RELOC_IA64_LDXMOV, BFD_RELOC_IA64_TPREL14, BFD_RELOC_IA64_TPREL22, BFD_RELOC_IA64_TPREL64I, BFD_RELOC_IA64_TPREL64MSB, BFD_RELOC_IA64_TPREL64LSB, BFD_RELOC_IA64_LTOFF_TPREL22, BFD_RELOC_IA64_DTPMOD64MSB, BFD_RELOC_IA64_DTPMOD64LSB, BFD_RELOC_IA64_LTOFF_DTPMOD22, BFD_RELOC_IA64_DTPREL14, BFD_RELOC_IA64_DTPREL22, BFD_RELOC_IA64_DTPREL64I, BFD_RELOC_IA64_DTPREL32MSB, BFD_RELOC_IA64_DTPREL32LSB, BFD_RELOC_IA64_DTPREL64MSB, BFD_RELOC_IA64_DTPREL64LSB, BFD_RELOC_IA64_LTOFF_DTPREL22, /* Motorola 68HC11 reloc. This is the 8 bit high part of an absolute address. */ BFD_RELOC_M68HC11_HI8, /* Motorola 68HC11 reloc. This is the 8 bit low part of an absolute address. */ BFD_RELOC_M68HC11_LO8, /* Motorola 68HC11 reloc. This is the 3 bit of a value. */ BFD_RELOC_M68HC11_3B, /* Motorola 68HC11 reloc. This reloc marks the beginning of a jump/call instruction. It is used for linker relaxation to correctly identify beginning of instruction and change some branches to use PC-relative addressing mode. */ BFD_RELOC_M68HC11_RL_JUMP, /* Motorola 68HC11 reloc. This reloc marks a group of several instructions that gcc generates and for which the linker relaxation pass can modify and/or remove some of them. */ BFD_RELOC_M68HC11_RL_GROUP, /* Motorola 68HC11 reloc. This is the 16-bit lower part of an address. It is used for 'call' instruction to specify the symbol address without any special transformation (due to memory bank window). */ BFD_RELOC_M68HC11_LO16, /* Motorola 68HC11 reloc. This is a 8-bit reloc that specifies the page number of an address. It is used by 'call' instruction to specify the page number of the symbol. */ BFD_RELOC_M68HC11_PAGE, /* Motorola 68HC11 reloc. This is a 24-bit reloc that represents the address with a 16-bit value and a 8-bit page number. The symbol address is transformed to follow the 16K memory bank of 68HC12 (seen as mapped in the window). */ BFD_RELOC_M68HC11_24, /* Motorola 68HC12 reloc. This is the 5 bits of a value. */ BFD_RELOC_M68HC12_5B, /* NS CR16C Relocations. */ BFD_RELOC_16C_NUM08, BFD_RELOC_16C_NUM08_C, BFD_RELOC_16C_NUM16, BFD_RELOC_16C_NUM16_C, BFD_RELOC_16C_NUM32, BFD_RELOC_16C_NUM32_C, BFD_RELOC_16C_DISP04, BFD_RELOC_16C_DISP04_C, BFD_RELOC_16C_DISP08, BFD_RELOC_16C_DISP08_C, BFD_RELOC_16C_DISP16, BFD_RELOC_16C_DISP16_C, BFD_RELOC_16C_DISP24, BFD_RELOC_16C_DISP24_C, BFD_RELOC_16C_DISP24a, BFD_RELOC_16C_DISP24a_C, BFD_RELOC_16C_REG04, BFD_RELOC_16C_REG04_C, BFD_RELOC_16C_REG04a, BFD_RELOC_16C_REG04a_C, BFD_RELOC_16C_REG14, BFD_RELOC_16C_REG14_C, BFD_RELOC_16C_REG16, BFD_RELOC_16C_REG16_C, BFD_RELOC_16C_REG20, BFD_RELOC_16C_REG20_C, BFD_RELOC_16C_ABS20, BFD_RELOC_16C_ABS20_C, BFD_RELOC_16C_ABS24, BFD_RELOC_16C_ABS24_C, BFD_RELOC_16C_IMM04, BFD_RELOC_16C_IMM04_C, BFD_RELOC_16C_IMM16, BFD_RELOC_16C_IMM16_C, BFD_RELOC_16C_IMM20, BFD_RELOC_16C_IMM20_C, BFD_RELOC_16C_IMM24, BFD_RELOC_16C_IMM24_C, BFD_RELOC_16C_IMM32, BFD_RELOC_16C_IMM32_C, /* NS CR16 Relocations. */ BFD_RELOC_CR16_NUM8, BFD_RELOC_CR16_NUM16, BFD_RELOC_CR16_NUM32, BFD_RELOC_CR16_NUM32a, BFD_RELOC_CR16_REGREL0, BFD_RELOC_CR16_REGREL4, BFD_RELOC_CR16_REGREL4a, BFD_RELOC_CR16_REGREL14, BFD_RELOC_CR16_REGREL14a, BFD_RELOC_CR16_REGREL16, BFD_RELOC_CR16_REGREL20, BFD_RELOC_CR16_REGREL20a, BFD_RELOC_CR16_ABS20, BFD_RELOC_CR16_ABS24, BFD_RELOC_CR16_IMM4, BFD_RELOC_CR16_IMM8, BFD_RELOC_CR16_IMM16, BFD_RELOC_CR16_IMM20, BFD_RELOC_CR16_IMM24, BFD_RELOC_CR16_IMM32, BFD_RELOC_CR16_IMM32a, BFD_RELOC_CR16_DISP4, BFD_RELOC_CR16_DISP8, BFD_RELOC_CR16_DISP16, BFD_RELOC_CR16_DISP20, BFD_RELOC_CR16_DISP24, BFD_RELOC_CR16_DISP24a, /* NS CRX Relocations. */ BFD_RELOC_CRX_REL4, BFD_RELOC_CRX_REL8, BFD_RELOC_CRX_REL8_CMP, BFD_RELOC_CRX_REL16, BFD_RELOC_CRX_REL24, BFD_RELOC_CRX_REL32, BFD_RELOC_CRX_REGREL12, BFD_RELOC_CRX_REGREL22, BFD_RELOC_CRX_REGREL28, BFD_RELOC_CRX_REGREL32, BFD_RELOC_CRX_ABS16, BFD_RELOC_CRX_ABS32, BFD_RELOC_CRX_NUM8, BFD_RELOC_CRX_NUM16, BFD_RELOC_CRX_NUM32, BFD_RELOC_CRX_IMM16, BFD_RELOC_CRX_IMM32, BFD_RELOC_CRX_SWITCH8, BFD_RELOC_CRX_SWITCH16, BFD_RELOC_CRX_SWITCH32, /* These relocs are only used within the CRIS assembler. They are not (at present) written to any object files. */ BFD_RELOC_CRIS_BDISP8, BFD_RELOC_CRIS_UNSIGNED_5, BFD_RELOC_CRIS_SIGNED_6, BFD_RELOC_CRIS_UNSIGNED_6, BFD_RELOC_CRIS_SIGNED_8, BFD_RELOC_CRIS_UNSIGNED_8, BFD_RELOC_CRIS_SIGNED_16, BFD_RELOC_CRIS_UNSIGNED_16, BFD_RELOC_CRIS_LAPCQ_OFFSET, BFD_RELOC_CRIS_UNSIGNED_4, /* Relocs used in ELF shared libraries for CRIS. */ BFD_RELOC_CRIS_COPY, BFD_RELOC_CRIS_GLOB_DAT, BFD_RELOC_CRIS_JUMP_SLOT, BFD_RELOC_CRIS_RELATIVE, /* 32-bit offset to symbol-entry within GOT. */ BFD_RELOC_CRIS_32_GOT, /* 16-bit offset to symbol-entry within GOT. */ BFD_RELOC_CRIS_16_GOT, /* 32-bit offset to symbol-entry within GOT, with PLT handling. */ BFD_RELOC_CRIS_32_GOTPLT, /* 16-bit offset to symbol-entry within GOT, with PLT handling. */ BFD_RELOC_CRIS_16_GOTPLT, /* 32-bit offset to symbol, relative to GOT. */ BFD_RELOC_CRIS_32_GOTREL, /* 32-bit offset to symbol with PLT entry, relative to GOT. */ BFD_RELOC_CRIS_32_PLT_GOTREL, /* 32-bit offset to symbol with PLT entry, relative to this relocation. */ BFD_RELOC_CRIS_32_PLT_PCREL, /* Intel i860 Relocations. */ BFD_RELOC_860_COPY, BFD_RELOC_860_GLOB_DAT, BFD_RELOC_860_JUMP_SLOT, BFD_RELOC_860_RELATIVE, BFD_RELOC_860_PC26, BFD_RELOC_860_PLT26, BFD_RELOC_860_PC16, BFD_RELOC_860_LOW0, BFD_RELOC_860_SPLIT0, BFD_RELOC_860_LOW1, BFD_RELOC_860_SPLIT1, BFD_RELOC_860_LOW2, BFD_RELOC_860_SPLIT2, BFD_RELOC_860_LOW3, BFD_RELOC_860_LOGOT0, BFD_RELOC_860_SPGOT0, BFD_RELOC_860_LOGOT1, BFD_RELOC_860_SPGOT1, BFD_RELOC_860_LOGOTOFF0, BFD_RELOC_860_SPGOTOFF0, BFD_RELOC_860_LOGOTOFF1, BFD_RELOC_860_SPGOTOFF1, BFD_RELOC_860_LOGOTOFF2, BFD_RELOC_860_LOGOTOFF3, BFD_RELOC_860_LOPC, BFD_RELOC_860_HIGHADJ, BFD_RELOC_860_HAGOT, BFD_RELOC_860_HAGOTOFF, BFD_RELOC_860_HAPC, BFD_RELOC_860_HIGH, BFD_RELOC_860_HIGOT, BFD_RELOC_860_HIGOTOFF, /* OpenRISC Relocations. */ BFD_RELOC_OPENRISC_ABS_26, BFD_RELOC_OPENRISC_REL_26, /* H8 elf Relocations. */ BFD_RELOC_H8_DIR16A8, BFD_RELOC_H8_DIR16R8, BFD_RELOC_H8_DIR24A8, BFD_RELOC_H8_DIR24R8, BFD_RELOC_H8_DIR32A16, /* Sony Xstormy16 Relocations. */ BFD_RELOC_XSTORMY16_REL_12, BFD_RELOC_XSTORMY16_12, BFD_RELOC_XSTORMY16_24, BFD_RELOC_XSTORMY16_FPTR16, /* Self-describing complex relocations. */ BFD_RELOC_RELC, /* Infineon Relocations. */ BFD_RELOC_XC16X_PAG, BFD_RELOC_XC16X_POF, BFD_RELOC_XC16X_SEG, BFD_RELOC_XC16X_SOF, /* Relocations used by VAX ELF. */ BFD_RELOC_VAX_GLOB_DAT, BFD_RELOC_VAX_JMP_SLOT, BFD_RELOC_VAX_RELATIVE, /* Morpho MT - 16 bit immediate relocation. */ BFD_RELOC_MT_PC16, /* Morpho MT - Hi 16 bits of an address. */ BFD_RELOC_MT_HI16, /* Morpho MT - Low 16 bits of an address. */ BFD_RELOC_MT_LO16, /* Morpho MT - Used to tell the linker which vtable entries are used. */ BFD_RELOC_MT_GNU_VTINHERIT, /* Morpho MT - Used to tell the linker which vtable entries are used. */ BFD_RELOC_MT_GNU_VTENTRY, /* Morpho MT - 8 bit immediate relocation. */ BFD_RELOC_MT_PCINSN8, /* msp430 specific relocation codes */ BFD_RELOC_MSP430_10_PCREL, BFD_RELOC_MSP430_16_PCREL, BFD_RELOC_MSP430_16, BFD_RELOC_MSP430_16_PCREL_BYTE, BFD_RELOC_MSP430_16_BYTE, BFD_RELOC_MSP430_2X_PCREL, BFD_RELOC_MSP430_RL_PCREL, /* IQ2000 Relocations. */ BFD_RELOC_IQ2000_OFFSET_16, BFD_RELOC_IQ2000_OFFSET_21, BFD_RELOC_IQ2000_UHI16, /* Special Xtensa relocation used only by PLT entries in ELF shared objects to indicate that the runtime linker should set the value to one of its own internal functions or data structures. */ BFD_RELOC_XTENSA_RTLD, /* Xtensa relocations for ELF shared objects. */ BFD_RELOC_XTENSA_GLOB_DAT, BFD_RELOC_XTENSA_JMP_SLOT, BFD_RELOC_XTENSA_RELATIVE, /* Xtensa relocation used in ELF object files for symbols that may require PLT entries. Otherwise, this is just a generic 32-bit relocation. */ BFD_RELOC_XTENSA_PLT, /* Xtensa relocations to mark the difference of two local symbols. These are only needed to support linker relaxation and can be ignored when not relaxing. The field is set to the value of the difference assuming no relaxation. The relocation encodes the position of the first symbol so the linker can determine whether to adjust the field value. */ BFD_RELOC_XTENSA_DIFF8, BFD_RELOC_XTENSA_DIFF16, BFD_RELOC_XTENSA_DIFF32, /* Generic Xtensa relocations for instruction operands. Only the slot number is encoded in the relocation. The relocation applies to the last PC-relative immediate operand, or if there are no PC-relative immediates, to the last immediate operand. */ BFD_RELOC_XTENSA_SLOT0_OP, BFD_RELOC_XTENSA_SLOT1_OP, BFD_RELOC_XTENSA_SLOT2_OP, BFD_RELOC_XTENSA_SLOT3_OP, BFD_RELOC_XTENSA_SLOT4_OP, BFD_RELOC_XTENSA_SLOT5_OP, BFD_RELOC_XTENSA_SLOT6_OP, BFD_RELOC_XTENSA_SLOT7_OP, BFD_RELOC_XTENSA_SLOT8_OP, BFD_RELOC_XTENSA_SLOT9_OP, BFD_RELOC_XTENSA_SLOT10_OP, BFD_RELOC_XTENSA_SLOT11_OP, BFD_RELOC_XTENSA_SLOT12_OP, BFD_RELOC_XTENSA_SLOT13_OP, BFD_RELOC_XTENSA_SLOT14_OP, /* Alternate Xtensa relocations. Only the slot is encoded in the relocation. The meaning of these relocations is opcode-specific. */ BFD_RELOC_XTENSA_SLOT0_ALT, BFD_RELOC_XTENSA_SLOT1_ALT, BFD_RELOC_XTENSA_SLOT2_ALT, BFD_RELOC_XTENSA_SLOT3_ALT, BFD_RELOC_XTENSA_SLOT4_ALT, BFD_RELOC_XTENSA_SLOT5_ALT, BFD_RELOC_XTENSA_SLOT6_ALT, BFD_RELOC_XTENSA_SLOT7_ALT, BFD_RELOC_XTENSA_SLOT8_ALT, BFD_RELOC_XTENSA_SLOT9_ALT, BFD_RELOC_XTENSA_SLOT10_ALT, BFD_RELOC_XTENSA_SLOT11_ALT, BFD_RELOC_XTENSA_SLOT12_ALT, BFD_RELOC_XTENSA_SLOT13_ALT, BFD_RELOC_XTENSA_SLOT14_ALT, /* Xtensa relocations for backward compatibility. These have all been replaced by BFD_RELOC_XTENSA_SLOT0_OP. */ BFD_RELOC_XTENSA_OP0, BFD_RELOC_XTENSA_OP1, BFD_RELOC_XTENSA_OP2, /* Xtensa relocation to mark that the assembler expanded the instructions from an original target. The expansion size is encoded in the reloc size. */ BFD_RELOC_XTENSA_ASM_EXPAND, /* Xtensa relocation to mark that the linker should simplify assembler-expanded instructions. This is commonly used internally by the linker after analysis of a BFD_RELOC_XTENSA_ASM_EXPAND. */ BFD_RELOC_XTENSA_ASM_SIMPLIFY, /* 8 bit signed offset in (ix+d) or (iy+d). */ BFD_RELOC_Z80_DISP8, /* DJNZ offset. */ BFD_RELOC_Z8K_DISP7, /* CALR offset. */ BFD_RELOC_Z8K_CALLR, /* 4 bit value. */ BFD_RELOC_Z8K_IMM4L, BFD_RELOC_UNUSED }; typedef enum bfd_reloc_code_real bfd_reloc_code_real_type; reloc_howto_type *bfd_reloc_type_lookup (bfd *abfd, bfd_reloc_code_real_type code); reloc_howto_type *bfd_reloc_name_lookup (bfd *abfd, const char *reloc_name); const char *bfd_get_reloc_code_name (bfd_reloc_code_real_type code); /* Extracted from syms.c. */ typedef struct bfd_symbol { /* A pointer to the BFD which owns the symbol. This information is necessary so that a back end can work out what additional information (invisible to the application writer) is carried with the symbol. This field is *almost* redundant, since you can use section->owner instead, except that some symbols point to the global sections bfd_{abs,com,und}_section. This could be fixed by making these globals be per-bfd (or per-target-flavor). FIXME. */ struct bfd *the_bfd; /* Use bfd_asymbol_bfd(sym) to access this field. */ /* The text of the symbol. The name is left alone, and not copied; the application may not alter it. */ const char *name; /* The value of the symbol. This really should be a union of a numeric value with a pointer, since some flags indicate that a pointer to another symbol is stored here. */ symvalue value; /* Attributes of a symbol. */ #define BSF_NO_FLAGS 0x00 /* The symbol has local scope; <> in <>. The value is the offset into the section of the data. */ #define BSF_LOCAL 0x01 /* The symbol has global scope; initialized data in <>. The value is the offset into the section of the data. */ #define BSF_GLOBAL 0x02 /* The symbol has global scope and is exported. The value is the offset into the section of the data. */ #define BSF_EXPORT BSF_GLOBAL /* No real difference. */ /* A normal C symbol would be one of: <>, <>, <> or <>. */ /* The symbol is a debugging record. The value has an arbitrary meaning, unless BSF_DEBUGGING_RELOC is also set. */ #define BSF_DEBUGGING 0x08 /* The symbol denotes a function entry point. Used in ELF, perhaps others someday. */ #define BSF_FUNCTION 0x10 /* Used by the linker. */ #define BSF_KEEP 0x20 #define BSF_KEEP_G 0x40 /* A weak global symbol, overridable without warnings by a regular global symbol of the same name. */ #define BSF_WEAK 0x80 /* This symbol was created to point to a section, e.g. ELF's STT_SECTION symbols. */ #define BSF_SECTION_SYM 0x100 /* The symbol used to be a common symbol, but now it is allocated. */ #define BSF_OLD_COMMON 0x200 /* The default value for common data. */ #define BFD_FORT_COMM_DEFAULT_VALUE 0 /* In some files the type of a symbol sometimes alters its location in an output file - ie in coff a <> symbol which is also <> symbol appears where it was declared and not at the end of a section. This bit is set by the target BFD part to convey this information. */ #define BSF_NOT_AT_END 0x400 /* Signal that the symbol is the label of constructor section. */ #define BSF_CONSTRUCTOR 0x800 /* Signal that the symbol is a warning symbol. The name is a warning. The name of the next symbol is the one to warn about; if a reference is made to a symbol with the same name as the next symbol, a warning is issued by the linker. */ #define BSF_WARNING 0x1000 /* Signal that the symbol is indirect. This symbol is an indirect pointer to the symbol with the same name as the next symbol. */ #define BSF_INDIRECT 0x2000 /* BSF_FILE marks symbols that contain a file name. This is used for ELF STT_FILE symbols. */ #define BSF_FILE 0x4000 /* Symbol is from dynamic linking information. */ #define BSF_DYNAMIC 0x8000 /* The symbol denotes a data object. Used in ELF, and perhaps others someday. */ #define BSF_OBJECT 0x10000 /* This symbol is a debugging symbol. The value is the offset into the section of the data. BSF_DEBUGGING should be set as well. */ #define BSF_DEBUGGING_RELOC 0x20000 /* This symbol is thread local. Used in ELF. */ #define BSF_THREAD_LOCAL 0x40000 /* This symbol represents a complex relocation expression, with the expression tree serialized in the symbol name. */ #define BSF_RELC 0x80000 /* This symbol represents a signed complex relocation expression, with the expression tree serialized in the symbol name. */ #define BSF_SRELC 0x100000 flagword flags; /* A pointer to the section to which this symbol is relative. This will always be non NULL, there are special sections for undefined and absolute symbols. */ struct bfd_section *section; /* Back end special data. */ union { void *p; bfd_vma i; } udata; } asymbol; #define bfd_get_symtab_upper_bound(abfd) \ BFD_SEND (abfd, _bfd_get_symtab_upper_bound, (abfd)) bfd_boolean bfd_is_local_label (bfd *abfd, asymbol *sym); bfd_boolean bfd_is_local_label_name (bfd *abfd, const char *name); #define bfd_is_local_label_name(abfd, name) \ BFD_SEND (abfd, _bfd_is_local_label_name, (abfd, name)) bfd_boolean bfd_is_target_special_symbol (bfd *abfd, asymbol *sym); #define bfd_is_target_special_symbol(abfd, sym) \ BFD_SEND (abfd, _bfd_is_target_special_symbol, (abfd, sym)) #define bfd_canonicalize_symtab(abfd, location) \ BFD_SEND (abfd, _bfd_canonicalize_symtab, (abfd, location)) bfd_boolean bfd_set_symtab (bfd *abfd, asymbol **location, unsigned int count); void bfd_print_symbol_vandf (bfd *abfd, void *file, asymbol *symbol); #define bfd_make_empty_symbol(abfd) \ BFD_SEND (abfd, _bfd_make_empty_symbol, (abfd)) asymbol *_bfd_generic_make_empty_symbol (bfd *); #define bfd_make_debug_symbol(abfd,ptr,size) \ BFD_SEND (abfd, _bfd_make_debug_symbol, (abfd, ptr, size)) int bfd_decode_symclass (asymbol *symbol); bfd_boolean bfd_is_undefined_symclass (int symclass); void bfd_symbol_info (asymbol *symbol, symbol_info *ret); bfd_boolean bfd_copy_private_symbol_data (bfd *ibfd, asymbol *isym, bfd *obfd, asymbol *osym); #define bfd_copy_private_symbol_data(ibfd, isymbol, obfd, osymbol) \ BFD_SEND (obfd, _bfd_copy_private_symbol_data, \ (ibfd, isymbol, obfd, osymbol)) /* Extracted from bfd.c. */ struct bfd { /* A unique identifier of the BFD */ unsigned int id; /* The filename the application opened the BFD with. */ const char *filename; /* A pointer to the target jump table. */ const struct bfd_target *xvec; /* The IOSTREAM, and corresponding IO vector that provide access to the file backing the BFD. */ void *iostream; const struct bfd_iovec *iovec; /* Is the file descriptor being cached? That is, can it be closed as needed, and re-opened when accessed later? */ bfd_boolean cacheable; /* Marks whether there was a default target specified when the BFD was opened. This is used to select which matching algorithm to use to choose the back end. */ bfd_boolean target_defaulted; /* The caching routines use these to maintain a least-recently-used list of BFDs. */ struct bfd *lru_prev, *lru_next; /* When a file is closed by the caching routines, BFD retains state information on the file here... */ ufile_ptr where; /* ... and here: (``once'' means at least once). */ bfd_boolean opened_once; /* Set if we have a locally maintained mtime value, rather than getting it from the file each time. */ bfd_boolean mtime_set; /* File modified time, if mtime_set is TRUE. */ long mtime; /* Reserved for an unimplemented file locking extension. */ int ifd; /* The format which belongs to the BFD. (object, core, etc.) */ bfd_format format; /* The direction with which the BFD was opened. */ enum bfd_direction { no_direction = 0, read_direction = 1, write_direction = 2, both_direction = 3 } direction; /* Format_specific flags. */ flagword flags; /* Currently my_archive is tested before adding origin to anything. I believe that this can become always an add of origin, with origin set to 0 for non archive files. */ ufile_ptr origin; /* Remember when output has begun, to stop strange things from happening. */ bfd_boolean output_has_begun; /* A hash table for section names. */ struct bfd_hash_table section_htab; /* Pointer to linked list of sections. */ struct bfd_section *sections; /* The last section on the section list. */ struct bfd_section *section_last; /* The number of sections. */ unsigned int section_count; /* Stuff only useful for object files: The start address. */ bfd_vma start_address; /* Used for input and output. */ unsigned int symcount; /* Symbol table for output BFD (with symcount entries). */ struct bfd_symbol **outsymbols; /* Used for slurped dynamic symbol tables. */ unsigned int dynsymcount; /* Pointer to structure which contains architecture information. */ const struct bfd_arch_info *arch_info; /* Flag set if symbols from this BFD should not be exported. */ bfd_boolean no_export; /* Stuff only useful for archives. */ void *arelt_data; struct bfd *my_archive; /* The containing archive BFD. */ struct bfd *archive_next; /* The next BFD in the archive. */ struct bfd *archive_head; /* The first BFD in the archive. */ bfd_boolean has_armap; /* A chain of BFD structures involved in a link. */ struct bfd *link_next; /* A field used by _bfd_generic_link_add_archive_symbols. This will be used only for archive elements. */ int archive_pass; /* Used by the back end to hold private data. */ union { struct aout_data_struct *aout_data; struct artdata *aout_ar_data; struct _oasys_data *oasys_obj_data; struct _oasys_ar_data *oasys_ar_data; struct coff_tdata *coff_obj_data; struct pe_tdata *pe_obj_data; struct xcoff_tdata *xcoff_obj_data; struct ecoff_tdata *ecoff_obj_data; struct ieee_data_struct *ieee_data; struct ieee_ar_data_struct *ieee_ar_data; struct srec_data_struct *srec_data; struct ihex_data_struct *ihex_data; struct tekhex_data_struct *tekhex_data; struct elf_obj_tdata *elf_obj_data; struct nlm_obj_tdata *nlm_obj_data; struct bout_data_struct *bout_data; struct mmo_data_struct *mmo_data; struct sun_core_struct *sun_core_data; struct sco5_core_struct *sco5_core_data; struct trad_core_struct *trad_core_data; struct som_data_struct *som_data; struct hpux_core_struct *hpux_core_data; struct hppabsd_core_struct *hppabsd_core_data; struct sgi_core_struct *sgi_core_data; struct lynx_core_struct *lynx_core_data; struct osf_core_struct *osf_core_data; struct cisco_core_struct *cisco_core_data; struct versados_data_struct *versados_data; struct netbsd_core_struct *netbsd_core_data; struct mach_o_data_struct *mach_o_data; struct mach_o_fat_data_struct *mach_o_fat_data; struct bfd_pef_data_struct *pef_data; struct bfd_pef_xlib_data_struct *pef_xlib_data; struct bfd_sym_data_struct *sym_data; void *any; } tdata; /* Used by the application to hold private data. */ void *usrdata; /* Where all the allocated stuff under this BFD goes. This is a struct objalloc *, but we use void * to avoid requiring the inclusion of objalloc.h. */ void *memory; }; typedef enum bfd_error { bfd_error_no_error = 0, bfd_error_system_call, bfd_error_invalid_target, bfd_error_wrong_format, bfd_error_wrong_object_format, bfd_error_invalid_operation, bfd_error_no_memory, bfd_error_no_symbols, bfd_error_no_armap, bfd_error_no_more_archived_files, bfd_error_malformed_archive, bfd_error_file_not_recognized, bfd_error_file_ambiguously_recognized, bfd_error_no_contents, bfd_error_nonrepresentable_section, bfd_error_no_debug_section, bfd_error_bad_value, bfd_error_file_truncated, bfd_error_file_too_big, bfd_error_on_input, bfd_error_invalid_error_code } bfd_error_type; bfd_error_type bfd_get_error (void); void bfd_set_error (bfd_error_type error_tag, ...); const char *bfd_errmsg (bfd_error_type error_tag); void bfd_perror (const char *message); typedef void (*bfd_error_handler_type) (const char *, ...); bfd_error_handler_type bfd_set_error_handler (bfd_error_handler_type); void bfd_set_error_program_name (const char *); bfd_error_handler_type bfd_get_error_handler (void); long bfd_get_reloc_upper_bound (bfd *abfd, asection *sect); long bfd_canonicalize_reloc (bfd *abfd, asection *sec, arelent **loc, asymbol **syms); void bfd_set_reloc (bfd *abfd, asection *sec, arelent **rel, unsigned int count); bfd_boolean bfd_set_file_flags (bfd *abfd, flagword flags); int bfd_get_arch_size (bfd *abfd); int bfd_get_sign_extend_vma (bfd *abfd); bfd_boolean bfd_set_start_address (bfd *abfd, bfd_vma vma); unsigned int bfd_get_gp_size (bfd *abfd); void bfd_set_gp_size (bfd *abfd, unsigned int i); bfd_vma bfd_scan_vma (const char *string, const char **end, int base); bfd_boolean bfd_copy_private_header_data (bfd *ibfd, bfd *obfd); #define bfd_copy_private_header_data(ibfd, obfd) \ BFD_SEND (obfd, _bfd_copy_private_header_data, \ (ibfd, obfd)) bfd_boolean bfd_copy_private_bfd_data (bfd *ibfd, bfd *obfd); #define bfd_copy_private_bfd_data(ibfd, obfd) \ BFD_SEND (obfd, _bfd_copy_private_bfd_data, \ (ibfd, obfd)) bfd_boolean bfd_merge_private_bfd_data (bfd *ibfd, bfd *obfd); #define bfd_merge_private_bfd_data(ibfd, obfd) \ BFD_SEND (obfd, _bfd_merge_private_bfd_data, \ (ibfd, obfd)) bfd_boolean bfd_set_private_flags (bfd *abfd, flagword flags); #define bfd_set_private_flags(abfd, flags) \ BFD_SEND (abfd, _bfd_set_private_flags, (abfd, flags)) #define bfd_sizeof_headers(abfd, info) \ BFD_SEND (abfd, _bfd_sizeof_headers, (abfd, info)) #define bfd_find_nearest_line(abfd, sec, syms, off, file, func, line) \ BFD_SEND (abfd, _bfd_find_nearest_line, \ (abfd, sec, syms, off, file, func, line)) #define bfd_find_line(abfd, syms, sym, file, line) \ BFD_SEND (abfd, _bfd_find_line, \ (abfd, syms, sym, file, line)) #define bfd_find_inliner_info(abfd, file, func, line) \ BFD_SEND (abfd, _bfd_find_inliner_info, \ (abfd, file, func, line)) #define bfd_debug_info_start(abfd) \ BFD_SEND (abfd, _bfd_debug_info_start, (abfd)) #define bfd_debug_info_end(abfd) \ BFD_SEND (abfd, _bfd_debug_info_end, (abfd)) #define bfd_debug_info_accumulate(abfd, section) \ BFD_SEND (abfd, _bfd_debug_info_accumulate, (abfd, section)) #define bfd_stat_arch_elt(abfd, stat) \ BFD_SEND (abfd, _bfd_stat_arch_elt,(abfd, stat)) #define bfd_update_armap_timestamp(abfd) \ BFD_SEND (abfd, _bfd_update_armap_timestamp, (abfd)) #define bfd_set_arch_mach(abfd, arch, mach)\ BFD_SEND ( abfd, _bfd_set_arch_mach, (abfd, arch, mach)) #define bfd_relax_section(abfd, section, link_info, again) \ BFD_SEND (abfd, _bfd_relax_section, (abfd, section, link_info, again)) #define bfd_gc_sections(abfd, link_info) \ BFD_SEND (abfd, _bfd_gc_sections, (abfd, link_info)) #define bfd_merge_sections(abfd, link_info) \ BFD_SEND (abfd, _bfd_merge_sections, (abfd, link_info)) #define bfd_is_group_section(abfd, sec) \ BFD_SEND (abfd, _bfd_is_group_section, (abfd, sec)) #define bfd_discard_group(abfd, sec) \ BFD_SEND (abfd, _bfd_discard_group, (abfd, sec)) #define bfd_link_hash_table_create(abfd) \ BFD_SEND (abfd, _bfd_link_hash_table_create, (abfd)) #define bfd_link_hash_table_free(abfd, hash) \ BFD_SEND (abfd, _bfd_link_hash_table_free, (hash)) #define bfd_link_add_symbols(abfd, info) \ BFD_SEND (abfd, _bfd_link_add_symbols, (abfd, info)) #define bfd_link_just_syms(abfd, sec, info) \ BFD_SEND (abfd, _bfd_link_just_syms, (sec, info)) #define bfd_final_link(abfd, info) \ BFD_SEND (abfd, _bfd_final_link, (abfd, info)) #define bfd_free_cached_info(abfd) \ BFD_SEND (abfd, _bfd_free_cached_info, (abfd)) #define bfd_get_dynamic_symtab_upper_bound(abfd) \ BFD_SEND (abfd, _bfd_get_dynamic_symtab_upper_bound, (abfd)) #define bfd_print_private_bfd_data(abfd, file)\ BFD_SEND (abfd, _bfd_print_private_bfd_data, (abfd, file)) #define bfd_canonicalize_dynamic_symtab(abfd, asymbols) \ BFD_SEND (abfd, _bfd_canonicalize_dynamic_symtab, (abfd, asymbols)) #define bfd_get_synthetic_symtab(abfd, count, syms, dyncount, dynsyms, ret) \ BFD_SEND (abfd, _bfd_get_synthetic_symtab, (abfd, count, syms, \ dyncount, dynsyms, ret)) #define bfd_get_dynamic_reloc_upper_bound(abfd) \ BFD_SEND (abfd, _bfd_get_dynamic_reloc_upper_bound, (abfd)) #define bfd_canonicalize_dynamic_reloc(abfd, arels, asyms) \ BFD_SEND (abfd, _bfd_canonicalize_dynamic_reloc, (abfd, arels, asyms)) extern bfd_byte *bfd_get_relocated_section_contents (bfd *, struct bfd_link_info *, struct bfd_link_order *, bfd_byte *, bfd_boolean, asymbol **); bfd_boolean bfd_alt_mach_code (bfd *abfd, int alternative); struct bfd_preserve { void *marker; void *tdata; flagword flags; const struct bfd_arch_info *arch_info; struct bfd_section *sections; struct bfd_section *section_last; unsigned int section_count; struct bfd_hash_table section_htab; }; bfd_boolean bfd_preserve_save (bfd *, struct bfd_preserve *); void bfd_preserve_restore (bfd *, struct bfd_preserve *); void bfd_preserve_finish (bfd *, struct bfd_preserve *); bfd_vma bfd_emul_get_maxpagesize (const char *); void bfd_emul_set_maxpagesize (const char *, bfd_vma); bfd_vma bfd_emul_get_commonpagesize (const char *); void bfd_emul_set_commonpagesize (const char *, bfd_vma); char *bfd_demangle (bfd *, const char *, int); /* Extracted from archive.c. */ symindex bfd_get_next_mapent (bfd *abfd, symindex previous, carsym **sym); bfd_boolean bfd_set_archive_head (bfd *output, bfd *new_head); bfd *bfd_openr_next_archived_file (bfd *archive, bfd *previous); /* Extracted from corefile.c. */ const char *bfd_core_file_failing_command (bfd *abfd); int bfd_core_file_failing_signal (bfd *abfd); bfd_boolean core_file_matches_executable_p (bfd *core_bfd, bfd *exec_bfd); bfd_boolean generic_core_file_matches_executable_p (bfd *core_bfd, bfd *exec_bfd); /* Extracted from targets.c. */ #define BFD_SEND(bfd, message, arglist) \ ((*((bfd)->xvec->message)) arglist) #ifdef DEBUG_BFD_SEND #undef BFD_SEND #define BFD_SEND(bfd, message, arglist) \ (((bfd) && (bfd)->xvec && (bfd)->xvec->message) ? \ ((*((bfd)->xvec->message)) arglist) : \ (bfd_assert (__FILE__,__LINE__), NULL)) #endif #define BFD_SEND_FMT(bfd, message, arglist) \ (((bfd)->xvec->message[(int) ((bfd)->format)]) arglist) #ifdef DEBUG_BFD_SEND #undef BFD_SEND_FMT #define BFD_SEND_FMT(bfd, message, arglist) \ (((bfd) && (bfd)->xvec && (bfd)->xvec->message) ? \ (((bfd)->xvec->message[(int) ((bfd)->format)]) arglist) : \ (bfd_assert (__FILE__,__LINE__), NULL)) #endif enum bfd_flavour { bfd_target_unknown_flavour, bfd_target_aout_flavour, bfd_target_coff_flavour, bfd_target_ecoff_flavour, bfd_target_xcoff_flavour, bfd_target_elf_flavour, bfd_target_ieee_flavour, bfd_target_nlm_flavour, bfd_target_oasys_flavour, bfd_target_tekhex_flavour, bfd_target_srec_flavour, bfd_target_ihex_flavour, bfd_target_som_flavour, bfd_target_os9k_flavour, bfd_target_versados_flavour, bfd_target_msdos_flavour, bfd_target_ovax_flavour, bfd_target_evax_flavour, bfd_target_mmo_flavour, bfd_target_mach_o_flavour, bfd_target_pef_flavour, bfd_target_pef_xlib_flavour, bfd_target_sym_flavour }; enum bfd_endian { BFD_ENDIAN_BIG, BFD_ENDIAN_LITTLE, BFD_ENDIAN_UNKNOWN }; /* Forward declaration. */ typedef struct bfd_link_info _bfd_link_info; typedef struct bfd_target { /* Identifies the kind of target, e.g., SunOS4, Ultrix, etc. */ char *name; /* The "flavour" of a back end is a general indication about the contents of a file. */ enum bfd_flavour flavour; /* The order of bytes within the data area of a file. */ enum bfd_endian byteorder; /* The order of bytes within the header parts of a file. */ enum bfd_endian header_byteorder; /* A mask of all the flags which an executable may have set - from the set <>, <>, ...<>. */ flagword object_flags; /* A mask of all the flags which a section may have set - from the set <>, <>, ...<>. */ flagword section_flags; /* The character normally found at the front of a symbol. (if any), perhaps `_'. */ char symbol_leading_char; /* The pad character for file names within an archive header. */ char ar_pad_char; /* The maximum number of characters in an archive header. */ unsigned short ar_max_namelen; /* Entries for byte swapping for data. These are different from the other entry points, since they don't take a BFD as the first argument. Certain other handlers could do the same. */ bfd_uint64_t (*bfd_getx64) (const void *); bfd_int64_t (*bfd_getx_signed_64) (const void *); void (*bfd_putx64) (bfd_uint64_t, void *); bfd_vma (*bfd_getx32) (const void *); bfd_signed_vma (*bfd_getx_signed_32) (const void *); void (*bfd_putx32) (bfd_vma, void *); bfd_vma (*bfd_getx16) (const void *); bfd_signed_vma (*bfd_getx_signed_16) (const void *); void (*bfd_putx16) (bfd_vma, void *); /* Byte swapping for the headers. */ bfd_uint64_t (*bfd_h_getx64) (const void *); bfd_int64_t (*bfd_h_getx_signed_64) (const void *); void (*bfd_h_putx64) (bfd_uint64_t, void *); bfd_vma (*bfd_h_getx32) (const void *); bfd_signed_vma (*bfd_h_getx_signed_32) (const void *); void (*bfd_h_putx32) (bfd_vma, void *); bfd_vma (*bfd_h_getx16) (const void *); bfd_signed_vma (*bfd_h_getx_signed_16) (const void *); void (*bfd_h_putx16) (bfd_vma, void *); /* Format dependent routines: these are vectors of entry points within the target vector structure, one for each format to check. */ /* Check the format of a file being read. Return a <> or zero. */ const struct bfd_target *(*_bfd_check_format[bfd_type_end]) (bfd *); /* Set the format of a file being written. */ bfd_boolean (*_bfd_set_format[bfd_type_end]) (bfd *); /* Write cached information into a file being written, at <>. */ bfd_boolean (*_bfd_write_contents[bfd_type_end]) (bfd *); /* Generic entry points. */ #define BFD_JUMP_TABLE_GENERIC(NAME) \ NAME##_close_and_cleanup, \ NAME##_bfd_free_cached_info, \ NAME##_new_section_hook, \ NAME##_get_section_contents, \ NAME##_get_section_contents_in_window /* Called when the BFD is being closed to do any necessary cleanup. */ bfd_boolean (*_close_and_cleanup) (bfd *); /* Ask the BFD to free all cached information. */ bfd_boolean (*_bfd_free_cached_info) (bfd *); /* Called when a new section is created. */ bfd_boolean (*_new_section_hook) (bfd *, sec_ptr); /* Read the contents of a section. */ bfd_boolean (*_bfd_get_section_contents) (bfd *, sec_ptr, void *, file_ptr, bfd_size_type); bfd_boolean (*_bfd_get_section_contents_in_window) (bfd *, sec_ptr, bfd_window *, file_ptr, bfd_size_type); /* Entry points to copy private data. */ #define BFD_JUMP_TABLE_COPY(NAME) \ NAME##_bfd_copy_private_bfd_data, \ NAME##_bfd_merge_private_bfd_data, \ _bfd_generic_init_private_section_data, \ NAME##_bfd_copy_private_section_data, \ NAME##_bfd_copy_private_symbol_data, \ NAME##_bfd_copy_private_header_data, \ NAME##_bfd_set_private_flags, \ NAME##_bfd_print_private_bfd_data /* Called to copy BFD general private data from one object file to another. */ bfd_boolean (*_bfd_copy_private_bfd_data) (bfd *, bfd *); /* Called to merge BFD general private data from one object file to a common output file when linking. */ bfd_boolean (*_bfd_merge_private_bfd_data) (bfd *, bfd *); /* Called to initialize BFD private section data from one object file to another. */ #define bfd_init_private_section_data(ibfd, isec, obfd, osec, link_info) \ BFD_SEND (obfd, _bfd_init_private_section_data, (ibfd, isec, obfd, osec, link_info)) bfd_boolean (*_bfd_init_private_section_data) (bfd *, sec_ptr, bfd *, sec_ptr, struct bfd_link_info *); /* Called to copy BFD private section data from one object file to another. */ bfd_boolean (*_bfd_copy_private_section_data) (bfd *, sec_ptr, bfd *, sec_ptr); /* Called to copy BFD private symbol data from one symbol to another. */ bfd_boolean (*_bfd_copy_private_symbol_data) (bfd *, asymbol *, bfd *, asymbol *); /* Called to copy BFD private header data from one object file to another. */ bfd_boolean (*_bfd_copy_private_header_data) (bfd *, bfd *); /* Called to set private backend flags. */ bfd_boolean (*_bfd_set_private_flags) (bfd *, flagword); /* Called to print private BFD data. */ bfd_boolean (*_bfd_print_private_bfd_data) (bfd *, void *); /* Core file entry points. */ #define BFD_JUMP_TABLE_CORE(NAME) \ NAME##_core_file_failing_command, \ NAME##_core_file_failing_signal, \ NAME##_core_file_matches_executable_p char * (*_core_file_failing_command) (bfd *); int (*_core_file_failing_signal) (bfd *); bfd_boolean (*_core_file_matches_executable_p) (bfd *, bfd *); /* Archive entry points. */ #define BFD_JUMP_TABLE_ARCHIVE(NAME) \ NAME##_slurp_armap, \ NAME##_slurp_extended_name_table, \ NAME##_construct_extended_name_table, \ NAME##_truncate_arname, \ NAME##_write_armap, \ NAME##_read_ar_hdr, \ NAME##_openr_next_archived_file, \ NAME##_get_elt_at_index, \ NAME##_generic_stat_arch_elt, \ NAME##_update_armap_timestamp bfd_boolean (*_bfd_slurp_armap) (bfd *); bfd_boolean (*_bfd_slurp_extended_name_table) (bfd *); bfd_boolean (*_bfd_construct_extended_name_table) (bfd *, char **, bfd_size_type *, const char **); void (*_bfd_truncate_arname) (bfd *, const char *, char *); bfd_boolean (*write_armap) (bfd *, unsigned int, struct orl *, unsigned int, int); void * (*_bfd_read_ar_hdr_fn) (bfd *); bfd * (*openr_next_archived_file) (bfd *, bfd *); #define bfd_get_elt_at_index(b,i) BFD_SEND (b, _bfd_get_elt_at_index, (b,i)) bfd * (*_bfd_get_elt_at_index) (bfd *, symindex); int (*_bfd_stat_arch_elt) (bfd *, struct stat *); bfd_boolean (*_bfd_update_armap_timestamp) (bfd *); /* Entry points used for symbols. */ #define BFD_JUMP_TABLE_SYMBOLS(NAME) \ NAME##_get_symtab_upper_bound, \ NAME##_canonicalize_symtab, \ NAME##_make_empty_symbol, \ NAME##_print_symbol, \ NAME##_get_symbol_info, \ NAME##_bfd_is_local_label_name, \ NAME##_bfd_is_target_special_symbol, \ NAME##_get_lineno, \ NAME##_find_nearest_line, \ _bfd_generic_find_line, \ NAME##_find_inliner_info, \ NAME##_bfd_make_debug_symbol, \ NAME##_read_minisymbols, \ NAME##_minisymbol_to_symbol long (*_bfd_get_symtab_upper_bound) (bfd *); long (*_bfd_canonicalize_symtab) (bfd *, struct bfd_symbol **); struct bfd_symbol * (*_bfd_make_empty_symbol) (bfd *); void (*_bfd_print_symbol) (bfd *, void *, struct bfd_symbol *, bfd_print_symbol_type); #define bfd_print_symbol(b,p,s,e) BFD_SEND (b, _bfd_print_symbol, (b,p,s,e)) void (*_bfd_get_symbol_info) (bfd *, struct bfd_symbol *, symbol_info *); #define bfd_get_symbol_info(b,p,e) BFD_SEND (b, _bfd_get_symbol_info, (b,p,e)) bfd_boolean (*_bfd_is_local_label_name) (bfd *, const char *); bfd_boolean (*_bfd_is_target_special_symbol) (bfd *, asymbol *); alent * (*_get_lineno) (bfd *, struct bfd_symbol *); bfd_boolean (*_bfd_find_nearest_line) (bfd *, struct bfd_section *, struct bfd_symbol **, bfd_vma, const char **, const char **, unsigned int *); bfd_boolean (*_bfd_find_line) (bfd *, struct bfd_symbol **, struct bfd_symbol *, const char **, unsigned int *); bfd_boolean (*_bfd_find_inliner_info) (bfd *, const char **, const char **, unsigned int *); /* Back-door to allow format-aware applications to create debug symbols while using BFD for everything else. Currently used by the assembler when creating COFF files. */ asymbol * (*_bfd_make_debug_symbol) (bfd *, void *, unsigned long size); #define bfd_read_minisymbols(b, d, m, s) \ BFD_SEND (b, _read_minisymbols, (b, d, m, s)) long (*_read_minisymbols) (bfd *, bfd_boolean, void **, unsigned int *); #define bfd_minisymbol_to_symbol(b, d, m, f) \ BFD_SEND (b, _minisymbol_to_symbol, (b, d, m, f)) asymbol * (*_minisymbol_to_symbol) (bfd *, bfd_boolean, const void *, asymbol *); /* Routines for relocs. */ #define BFD_JUMP_TABLE_RELOCS(NAME) \ NAME##_get_reloc_upper_bound, \ NAME##_canonicalize_reloc, \ NAME##_bfd_reloc_type_lookup, \ NAME##_bfd_reloc_name_lookup long (*_get_reloc_upper_bound) (bfd *, sec_ptr); long (*_bfd_canonicalize_reloc) (bfd *, sec_ptr, arelent **, struct bfd_symbol **); /* See documentation on reloc types. */ reloc_howto_type * (*reloc_type_lookup) (bfd *, bfd_reloc_code_real_type); reloc_howto_type * (*reloc_name_lookup) (bfd *, const char *); /* Routines used when writing an object file. */ #define BFD_JUMP_TABLE_WRITE(NAME) \ NAME##_set_arch_mach, \ NAME##_set_section_contents bfd_boolean (*_bfd_set_arch_mach) (bfd *, enum bfd_architecture, unsigned long); bfd_boolean (*_bfd_set_section_contents) (bfd *, sec_ptr, const void *, file_ptr, bfd_size_type); /* Routines used by the linker. */ #define BFD_JUMP_TABLE_LINK(NAME) \ NAME##_sizeof_headers, \ NAME##_bfd_get_relocated_section_contents, \ NAME##_bfd_relax_section, \ NAME##_bfd_link_hash_table_create, \ NAME##_bfd_link_hash_table_free, \ NAME##_bfd_link_add_symbols, \ NAME##_bfd_link_just_syms, \ NAME##_bfd_final_link, \ NAME##_bfd_link_split_section, \ NAME##_bfd_gc_sections, \ NAME##_bfd_merge_sections, \ NAME##_bfd_is_group_section, \ NAME##_bfd_discard_group, \ NAME##_section_already_linked \ int (*_bfd_sizeof_headers) (bfd *, struct bfd_link_info *); bfd_byte * (*_bfd_get_relocated_section_contents) (bfd *, struct bfd_link_info *, struct bfd_link_order *, bfd_byte *, bfd_boolean, struct bfd_symbol **); bfd_boolean (*_bfd_relax_section) (bfd *, struct bfd_section *, struct bfd_link_info *, bfd_boolean *); /* Create a hash table for the linker. Different backends store different information in this table. */ struct bfd_link_hash_table * (*_bfd_link_hash_table_create) (bfd *); /* Release the memory associated with the linker hash table. */ void (*_bfd_link_hash_table_free) (struct bfd_link_hash_table *); /* Add symbols from this object file into the hash table. */ bfd_boolean (*_bfd_link_add_symbols) (bfd *, struct bfd_link_info *); /* Indicate that we are only retrieving symbol values from this section. */ void (*_bfd_link_just_syms) (asection *, struct bfd_link_info *); /* Do a link based on the link_order structures attached to each section of the BFD. */ bfd_boolean (*_bfd_final_link) (bfd *, struct bfd_link_info *); /* Should this section be split up into smaller pieces during linking. */ bfd_boolean (*_bfd_link_split_section) (bfd *, struct bfd_section *); /* Remove sections that are not referenced from the output. */ bfd_boolean (*_bfd_gc_sections) (bfd *, struct bfd_link_info *); /* Attempt to merge SEC_MERGE sections. */ bfd_boolean (*_bfd_merge_sections) (bfd *, struct bfd_link_info *); /* Is this section a member of a group? */ bfd_boolean (*_bfd_is_group_section) (bfd *, const struct bfd_section *); /* Discard members of a group. */ bfd_boolean (*_bfd_discard_group) (bfd *, struct bfd_section *); /* Check if SEC has been already linked during a reloceatable or final link. */ void (*_section_already_linked) (bfd *, struct bfd_section *, struct bfd_link_info *); /* Routines to handle dynamic symbols and relocs. */ #define BFD_JUMP_TABLE_DYNAMIC(NAME) \ NAME##_get_dynamic_symtab_upper_bound, \ NAME##_canonicalize_dynamic_symtab, \ NAME##_get_synthetic_symtab, \ NAME##_get_dynamic_reloc_upper_bound, \ NAME##_canonicalize_dynamic_reloc /* Get the amount of memory required to hold the dynamic symbols. */ long (*_bfd_get_dynamic_symtab_upper_bound) (bfd *); /* Read in the dynamic symbols. */ long (*_bfd_canonicalize_dynamic_symtab) (bfd *, struct bfd_symbol **); /* Create synthetized symbols. */ long (*_bfd_get_synthetic_symtab) (bfd *, long, struct bfd_symbol **, long, struct bfd_symbol **, struct bfd_symbol **); /* Get the amount of memory required to hold the dynamic relocs. */ long (*_bfd_get_dynamic_reloc_upper_bound) (bfd *); /* Read in the dynamic relocs. */ long (*_bfd_canonicalize_dynamic_reloc) (bfd *, arelent **, struct bfd_symbol **); /* Opposite endian version of this target. */ const struct bfd_target * alternative_target; /* Data for use by back-end routines, which isn't generic enough to belong in this structure. */ const void *backend_data; } bfd_target; bfd_boolean bfd_set_default_target (const char *name); const bfd_target *bfd_find_target (const char *target_name, bfd *abfd); const char ** bfd_target_list (void); const bfd_target *bfd_search_for_target (int (*search_func) (const bfd_target *, void *), void *); /* Extracted from format.c. */ bfd_boolean bfd_check_format (bfd *abfd, bfd_format format); bfd_boolean bfd_check_format_matches (bfd *abfd, bfd_format format, char ***matching); bfd_boolean bfd_set_format (bfd *abfd, bfd_format format); const char *bfd_format_string (bfd_format format); /* Extracted from linker.c. */ bfd_boolean bfd_link_split_section (bfd *abfd, asection *sec); #define bfd_link_split_section(abfd, sec) \ BFD_SEND (abfd, _bfd_link_split_section, (abfd, sec)) void bfd_section_already_linked (bfd *abfd, asection *sec, struct bfd_link_info *info); #define bfd_section_already_linked(abfd, sec, info) \ BFD_SEND (abfd, _section_already_linked, (abfd, sec, info)) /* Extracted from simple.c. */ bfd_byte *bfd_simple_get_relocated_section_contents (bfd *abfd, asection *sec, bfd_byte *outbuf, asymbol **symbol_table); #ifdef __cplusplus } #endif #endif Index: projects/clang350-import/contrib/binutils/bfd/ecoff.c =================================================================== --- projects/clang350-import/contrib/binutils/bfd/ecoff.c (revision 275748) +++ projects/clang350-import/contrib/binutils/bfd/ecoff.c (revision 275749) @@ -1,4634 +1,4634 @@ /* Generic ECOFF (Extended-COFF) routines. Copyright 1990, 1991, 1993, 1994, 1995, 1996, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007 Free Software Foundation, Inc. Original version by Per Bothner. Full support added by Ian Lance Taylor, ian@cygnus.com. This file is part of BFD, the Binary File Descriptor library. This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin Street - Fifth Floor, Boston, MA 02110-1301, USA. */ #include "sysdep.h" #include "bfd.h" #include "bfdlink.h" #include "libbfd.h" #include "aout/ar.h" #include "aout/ranlib.h" #include "aout/stab_gnu.h" /* FIXME: We need the definitions of N_SET[ADTB], but aout64.h defines some other stuff which we don't want and which conflicts with stuff we do want. */ #include "libaout.h" #include "aout/aout64.h" #undef N_ABS #undef exec_hdr #undef obj_sym_filepos #include "coff/internal.h" #include "coff/sym.h" #include "coff/symconst.h" #include "coff/ecoff.h" #include "libcoff.h" #include "libecoff.h" #include "libiberty.h" #define streq(a, b) (strcmp ((a), (b)) == 0) #define strneq(a, b, n) (strncmp ((a), (b), (n)) == 0) /* This stuff is somewhat copied from coffcode.h. */ static asection bfd_debug_section = { /* name, id, index, next, prev, flags, user_set_vma, */ "*DEBUG*", 0, 0, NULL, NULL, 0, 0, /* linker_mark, linker_has_input, gc_mark, gc_mark_from_eh, */ 0, 0, 1, 0, /* segment_mark, sec_info_type, use_rela_p, has_tls_reloc, */ 0, 0, 0, 0, - /* has_gp_reloc, need_finalize_relax, reloc_done, */ - 0, 0, 0, - /* vma, lma, size, rawsize, */ - 0, 0, 0, 0, + /* has_tls_get_addr_call, has_gp_reloc, need_finalize_relax, */ + 0, 0, 0, + /* reloc_done, vma, lma, size, rawsize, */ + 0, 0, 0, 0, 0, /* output_offset, output_section, alignment_power, */ 0, NULL, 0, /* relocation, orelocation, reloc_count, filepos, rel_filepos, */ NULL, NULL, 0, 0, 0, /* line_filepos, userdata, contents, lineno, lineno_count, */ 0, NULL, NULL, NULL, 0, /* entsize, kept_section, moving_line_filepos, */ 0, NULL, 0, /* target_index, used_by_bfd, constructor_chain, owner, */ 0, NULL, NULL, NULL, /* symbol, */ NULL, /* symbol_ptr_ptr, */ NULL, /* map_head, map_tail */ { NULL }, { NULL } }; /* Create an ECOFF object. */ bfd_boolean _bfd_ecoff_mkobject (bfd *abfd) { bfd_size_type amt = sizeof (ecoff_data_type); abfd->tdata.ecoff_obj_data = bfd_zalloc (abfd, amt); if (abfd->tdata.ecoff_obj_data == NULL) return FALSE; return TRUE; } /* This is a hook called by coff_real_object_p to create any backend specific information. */ void * _bfd_ecoff_mkobject_hook (bfd *abfd, void * filehdr, void * aouthdr) { struct internal_filehdr *internal_f = (struct internal_filehdr *) filehdr; struct internal_aouthdr *internal_a = (struct internal_aouthdr *) aouthdr; ecoff_data_type *ecoff; if (! _bfd_ecoff_mkobject (abfd)) return NULL; ecoff = ecoff_data (abfd); ecoff->gp_size = 8; ecoff->sym_filepos = internal_f->f_symptr; if (internal_a != NULL) { int i; ecoff->text_start = internal_a->text_start; ecoff->text_end = internal_a->text_start + internal_a->tsize; ecoff->gp = internal_a->gp_value; ecoff->gprmask = internal_a->gprmask; for (i = 0; i < 4; i++) ecoff->cprmask[i] = internal_a->cprmask[i]; ecoff->fprmask = internal_a->fprmask; if (internal_a->magic == ECOFF_AOUT_ZMAGIC) abfd->flags |= D_PAGED; else abfd->flags &=~ D_PAGED; } /* It turns out that no special action is required by the MIPS or Alpha ECOFF backends. They have different information in the a.out header, but we just copy it all (e.g., gprmask, cprmask and fprmask) and let the swapping routines ensure that only relevant information is written out. */ return (void *) ecoff; } /* Initialize a new section. */ bfd_boolean _bfd_ecoff_new_section_hook (bfd *abfd, asection *section) { unsigned int i; static struct { const char * name; flagword flags; } section_flags [] = { { _TEXT, SEC_ALLOC | SEC_CODE | SEC_LOAD }, { _INIT, SEC_ALLOC | SEC_CODE | SEC_LOAD }, { _FINI, SEC_ALLOC | SEC_CODE | SEC_LOAD }, { _DATA, SEC_ALLOC | SEC_DATA | SEC_LOAD }, { _SDATA, SEC_ALLOC | SEC_DATA | SEC_LOAD }, { _RDATA, SEC_ALLOC | SEC_DATA | SEC_LOAD | SEC_READONLY}, { _LIT8, SEC_ALLOC | SEC_DATA | SEC_LOAD | SEC_READONLY}, { _LIT4, SEC_ALLOC | SEC_DATA | SEC_LOAD | SEC_READONLY}, { _RCONST, SEC_ALLOC | SEC_DATA | SEC_LOAD | SEC_READONLY}, { _PDATA, SEC_ALLOC | SEC_DATA | SEC_LOAD | SEC_READONLY}, { _BSS, SEC_ALLOC}, { _SBSS, SEC_ALLOC}, /* An Irix 4 shared libary. */ { _LIB, SEC_COFF_SHARED_LIBRARY} }; section->alignment_power = 4; for (i = 0; i < ARRAY_SIZE (section_flags); i++) if (streq (section->name, section_flags[i].name)) { section->flags |= section_flags[i].flags; break; } /* Probably any other section name is SEC_NEVER_LOAD, but I'm uncertain about .init on some systems and I don't know how shared libraries work. */ return _bfd_generic_new_section_hook (abfd, section); } /* Determine the machine architecture and type. This is called from the generic COFF routines. It is the inverse of ecoff_get_magic, below. This could be an ECOFF backend routine, with one version for each target, but there aren't all that many ECOFF targets. */ bfd_boolean _bfd_ecoff_set_arch_mach_hook (bfd *abfd, void * filehdr) { struct internal_filehdr *internal_f = filehdr; enum bfd_architecture arch; unsigned long mach; switch (internal_f->f_magic) { case MIPS_MAGIC_1: case MIPS_MAGIC_LITTLE: case MIPS_MAGIC_BIG: arch = bfd_arch_mips; mach = bfd_mach_mips3000; break; case MIPS_MAGIC_LITTLE2: case MIPS_MAGIC_BIG2: /* MIPS ISA level 2: the r6000. */ arch = bfd_arch_mips; mach = bfd_mach_mips6000; break; case MIPS_MAGIC_LITTLE3: case MIPS_MAGIC_BIG3: /* MIPS ISA level 3: the r4000. */ arch = bfd_arch_mips; mach = bfd_mach_mips4000; break; case ALPHA_MAGIC: arch = bfd_arch_alpha; mach = 0; break; default: arch = bfd_arch_obscure; mach = 0; break; } return bfd_default_set_arch_mach (abfd, arch, mach); } /* Get the magic number to use based on the architecture and machine. This is the inverse of _bfd_ecoff_set_arch_mach_hook, above. */ static int ecoff_get_magic (bfd *abfd) { int big, little; switch (bfd_get_arch (abfd)) { case bfd_arch_mips: switch (bfd_get_mach (abfd)) { default: case 0: case bfd_mach_mips3000: big = MIPS_MAGIC_BIG; little = MIPS_MAGIC_LITTLE; break; case bfd_mach_mips6000: big = MIPS_MAGIC_BIG2; little = MIPS_MAGIC_LITTLE2; break; case bfd_mach_mips4000: big = MIPS_MAGIC_BIG3; little = MIPS_MAGIC_LITTLE3; break; } return bfd_big_endian (abfd) ? big : little; case bfd_arch_alpha: return ALPHA_MAGIC; default: abort (); return 0; } } /* Get the section s_flags to use for a section. */ static long ecoff_sec_to_styp_flags (const char *name, flagword flags) { unsigned int i; static struct { const char * name; long flags; } styp_flags [] = { { _TEXT, STYP_TEXT }, { _DATA, STYP_DATA }, { _SDATA, STYP_SDATA }, { _RDATA, STYP_RDATA }, { _LITA, STYP_LITA }, { _LIT8, STYP_LIT8 }, { _LIT4, STYP_LIT4 }, { _BSS, STYP_BSS }, { _SBSS, STYP_SBSS }, { _INIT, STYP_ECOFF_INIT }, { _FINI, STYP_ECOFF_FINI }, { _PDATA, STYP_PDATA }, { _XDATA, STYP_XDATA }, { _LIB, STYP_ECOFF_LIB }, { _GOT, STYP_GOT }, { _HASH, STYP_HASH }, { _DYNAMIC, STYP_DYNAMIC }, { _LIBLIST, STYP_LIBLIST }, { _RELDYN, STYP_RELDYN }, { _CONFLIC, STYP_CONFLIC }, { _DYNSTR, STYP_DYNSTR }, { _DYNSYM, STYP_DYNSYM }, { _RCONST, STYP_RCONST } }; long styp = 0; for (i = 0; i < ARRAY_SIZE (styp_flags); i++) if (streq (name, styp_flags[i].name)) { styp = styp_flags[i].flags; break; } if (styp == 0) { if (streq (name, _COMMENT)) { styp = STYP_COMMENT; flags &=~ SEC_NEVER_LOAD; } else if (flags & SEC_CODE) styp = STYP_TEXT; else if (flags & SEC_DATA) styp = STYP_DATA; else if (flags & SEC_READONLY) styp = STYP_RDATA; else if (flags & SEC_LOAD) styp = STYP_REG; else styp = STYP_BSS; } if (flags & SEC_NEVER_LOAD) styp |= STYP_NOLOAD; return styp; } /* Get the BFD flags to use for a section. */ bfd_boolean _bfd_ecoff_styp_to_sec_flags (bfd *abfd ATTRIBUTE_UNUSED, void * hdr, const char *name ATTRIBUTE_UNUSED, asection *section ATTRIBUTE_UNUSED, flagword * flags_ptr) { struct internal_scnhdr *internal_s = hdr; long styp_flags = internal_s->s_flags; flagword sec_flags = 0; if (styp_flags & STYP_NOLOAD) sec_flags |= SEC_NEVER_LOAD; /* For 386 COFF, at least, an unloadable text or data section is actually a shared library section. */ if ((styp_flags & STYP_TEXT) || (styp_flags & STYP_ECOFF_INIT) || (styp_flags & STYP_ECOFF_FINI) || (styp_flags & STYP_DYNAMIC) || (styp_flags & STYP_LIBLIST) || (styp_flags & STYP_RELDYN) || styp_flags == STYP_CONFLIC || (styp_flags & STYP_DYNSTR) || (styp_flags & STYP_DYNSYM) || (styp_flags & STYP_HASH)) { if (sec_flags & SEC_NEVER_LOAD) sec_flags |= SEC_CODE | SEC_COFF_SHARED_LIBRARY; else sec_flags |= SEC_CODE | SEC_LOAD | SEC_ALLOC; } else if ((styp_flags & STYP_DATA) || (styp_flags & STYP_RDATA) || (styp_flags & STYP_SDATA) || styp_flags == STYP_PDATA || styp_flags == STYP_XDATA || (styp_flags & STYP_GOT) || styp_flags == STYP_RCONST) { if (sec_flags & SEC_NEVER_LOAD) sec_flags |= SEC_DATA | SEC_COFF_SHARED_LIBRARY; else sec_flags |= SEC_DATA | SEC_LOAD | SEC_ALLOC; if ((styp_flags & STYP_RDATA) || styp_flags == STYP_PDATA || styp_flags == STYP_RCONST) sec_flags |= SEC_READONLY; } else if ((styp_flags & STYP_BSS) || (styp_flags & STYP_SBSS)) sec_flags |= SEC_ALLOC; else if ((styp_flags & STYP_INFO) || styp_flags == STYP_COMMENT) sec_flags |= SEC_NEVER_LOAD; else if ((styp_flags & STYP_LITA) || (styp_flags & STYP_LIT8) || (styp_flags & STYP_LIT4)) sec_flags |= SEC_DATA | SEC_LOAD | SEC_ALLOC | SEC_READONLY; else if (styp_flags & STYP_ECOFF_LIB) sec_flags |= SEC_COFF_SHARED_LIBRARY; else sec_flags |= SEC_ALLOC | SEC_LOAD; * flags_ptr = sec_flags; return TRUE; } /* Read in the symbolic header for an ECOFF object file. */ static bfd_boolean ecoff_slurp_symbolic_header (bfd *abfd) { const struct ecoff_backend_data * const backend = ecoff_backend (abfd); bfd_size_type external_hdr_size; void * raw = NULL; HDRR *internal_symhdr; /* See if we've already read it in. */ if (ecoff_data (abfd)->debug_info.symbolic_header.magic == backend->debug_swap.sym_magic) return TRUE; /* See whether there is a symbolic header. */ if (ecoff_data (abfd)->sym_filepos == 0) { bfd_get_symcount (abfd) = 0; return TRUE; } /* At this point bfd_get_symcount (abfd) holds the number of symbols as read from the file header, but on ECOFF this is always the size of the symbolic information header. It would be cleaner to handle this when we first read the file in coffgen.c. */ external_hdr_size = backend->debug_swap.external_hdr_size; if (bfd_get_symcount (abfd) != external_hdr_size) { bfd_set_error (bfd_error_bad_value); return FALSE; } /* Read the symbolic information header. */ raw = bfd_malloc (external_hdr_size); if (raw == NULL) goto error_return; if (bfd_seek (abfd, ecoff_data (abfd)->sym_filepos, SEEK_SET) != 0 || bfd_bread (raw, external_hdr_size, abfd) != external_hdr_size) goto error_return; internal_symhdr = &ecoff_data (abfd)->debug_info.symbolic_header; (*backend->debug_swap.swap_hdr_in) (abfd, raw, internal_symhdr); if (internal_symhdr->magic != backend->debug_swap.sym_magic) { bfd_set_error (bfd_error_bad_value); goto error_return; } /* Now we can get the correct number of symbols. */ bfd_get_symcount (abfd) = (internal_symhdr->isymMax + internal_symhdr->iextMax); if (raw != NULL) free (raw); return TRUE; error_return: if (raw != NULL) free (raw); return FALSE; } /* Read in and swap the important symbolic information for an ECOFF object file. This is called by gdb via the read_debug_info entry point in the backend structure. */ bfd_boolean _bfd_ecoff_slurp_symbolic_info (bfd *abfd, asection *ignore ATTRIBUTE_UNUSED, struct ecoff_debug_info *debug) { const struct ecoff_backend_data * const backend = ecoff_backend (abfd); HDRR *internal_symhdr; bfd_size_type raw_base; bfd_size_type raw_size; void * raw; bfd_size_type external_fdr_size; char *fraw_src; char *fraw_end; struct fdr *fdr_ptr; bfd_size_type raw_end; bfd_size_type cb_end; bfd_size_type amt; file_ptr pos; BFD_ASSERT (debug == &ecoff_data (abfd)->debug_info); /* Check whether we've already gotten it, and whether there's any to get. */ if (ecoff_data (abfd)->raw_syments != NULL) return TRUE; if (ecoff_data (abfd)->sym_filepos == 0) { bfd_get_symcount (abfd) = 0; return TRUE; } if (! ecoff_slurp_symbolic_header (abfd)) return FALSE; internal_symhdr = &debug->symbolic_header; /* Read all the symbolic information at once. */ raw_base = (ecoff_data (abfd)->sym_filepos + backend->debug_swap.external_hdr_size); /* Alpha ecoff makes the determination of raw_size difficult. It has an undocumented debug data section between the symhdr and the first documented section. And the ordering of the sections varies between statically and dynamically linked executables. If bfd supports SEEK_END someday, this code could be simplified. */ raw_end = 0; #define UPDATE_RAW_END(start, count, size) \ cb_end = internal_symhdr->start + internal_symhdr->count * (size); \ if (cb_end > raw_end) \ raw_end = cb_end UPDATE_RAW_END (cbLineOffset, cbLine, sizeof (unsigned char)); UPDATE_RAW_END (cbDnOffset, idnMax, backend->debug_swap.external_dnr_size); UPDATE_RAW_END (cbPdOffset, ipdMax, backend->debug_swap.external_pdr_size); UPDATE_RAW_END (cbSymOffset, isymMax, backend->debug_swap.external_sym_size); /* eraxxon@alumni.rice.edu: ioptMax refers to the size of the optimization symtab, not the number of entries. */ UPDATE_RAW_END (cbOptOffset, ioptMax, sizeof (char)); UPDATE_RAW_END (cbAuxOffset, iauxMax, sizeof (union aux_ext)); UPDATE_RAW_END (cbSsOffset, issMax, sizeof (char)); UPDATE_RAW_END (cbSsExtOffset, issExtMax, sizeof (char)); UPDATE_RAW_END (cbFdOffset, ifdMax, backend->debug_swap.external_fdr_size); UPDATE_RAW_END (cbRfdOffset, crfd, backend->debug_swap.external_rfd_size); UPDATE_RAW_END (cbExtOffset, iextMax, backend->debug_swap.external_ext_size); #undef UPDATE_RAW_END raw_size = raw_end - raw_base; if (raw_size == 0) { ecoff_data (abfd)->sym_filepos = 0; return TRUE; } raw = bfd_alloc (abfd, raw_size); if (raw == NULL) return FALSE; pos = ecoff_data (abfd)->sym_filepos; pos += backend->debug_swap.external_hdr_size; if (bfd_seek (abfd, pos, SEEK_SET) != 0 || bfd_bread (raw, raw_size, abfd) != raw_size) { bfd_release (abfd, raw); return FALSE; } ecoff_data (abfd)->raw_syments = raw; /* Get pointers for the numeric offsets in the HDRR structure. */ #define FIX(off1, off2, type) \ if (internal_symhdr->off1 == 0) \ debug->off2 = NULL; \ else \ debug->off2 = (type) ((char *) raw \ + (internal_symhdr->off1 \ - raw_base)) FIX (cbLineOffset, line, unsigned char *); FIX (cbDnOffset, external_dnr, void *); FIX (cbPdOffset, external_pdr, void *); FIX (cbSymOffset, external_sym, void *); FIX (cbOptOffset, external_opt, void *); FIX (cbAuxOffset, external_aux, union aux_ext *); FIX (cbSsOffset, ss, char *); FIX (cbSsExtOffset, ssext, char *); FIX (cbFdOffset, external_fdr, void *); FIX (cbRfdOffset, external_rfd, void *); FIX (cbExtOffset, external_ext, void *); #undef FIX /* I don't want to always swap all the data, because it will just waste time and most programs will never look at it. The only time the linker needs most of the debugging information swapped is when linking big-endian and little-endian MIPS object files together, which is not a common occurrence. We need to look at the fdr to deal with a lot of information in the symbols, so we swap them here. */ amt = internal_symhdr->ifdMax; amt *= sizeof (struct fdr); debug->fdr = bfd_alloc (abfd, amt); if (debug->fdr == NULL) return FALSE; external_fdr_size = backend->debug_swap.external_fdr_size; fdr_ptr = debug->fdr; fraw_src = (char *) debug->external_fdr; fraw_end = fraw_src + internal_symhdr->ifdMax * external_fdr_size; for (; fraw_src < fraw_end; fraw_src += external_fdr_size, fdr_ptr++) (*backend->debug_swap.swap_fdr_in) (abfd, (void *) fraw_src, fdr_ptr); return TRUE; } /* ECOFF symbol table routines. The ECOFF symbol table is described in gcc/mips-tfile.c. */ /* ECOFF uses two common sections. One is the usual one, and the other is for small objects. All the small objects are kept together, and then referenced via the gp pointer, which yields faster assembler code. This is what we use for the small common section. */ static asection ecoff_scom_section; static asymbol ecoff_scom_symbol; static asymbol *ecoff_scom_symbol_ptr; /* Create an empty symbol. */ asymbol * _bfd_ecoff_make_empty_symbol (bfd *abfd) { ecoff_symbol_type *new; bfd_size_type amt = sizeof (ecoff_symbol_type); new = bfd_zalloc (abfd, amt); if (new == NULL) return NULL; new->symbol.section = NULL; new->fdr = NULL; new->local = FALSE; new->native = NULL; new->symbol.the_bfd = abfd; return &new->symbol; } /* Set the BFD flags and section for an ECOFF symbol. */ static bfd_boolean ecoff_set_symbol_info (bfd *abfd, SYMR *ecoff_sym, asymbol *asym, int ext, int weak) { asym->the_bfd = abfd; asym->value = ecoff_sym->value; asym->section = &bfd_debug_section; asym->udata.i = 0; /* Most symbol types are just for debugging. */ switch (ecoff_sym->st) { case stGlobal: case stStatic: case stLabel: case stProc: case stStaticProc: break; case stNil: if (ECOFF_IS_STAB (ecoff_sym)) { asym->flags = BSF_DEBUGGING; return TRUE; } break; default: asym->flags = BSF_DEBUGGING; return TRUE; } if (weak) asym->flags = BSF_EXPORT | BSF_WEAK; else if (ext) asym->flags = BSF_EXPORT | BSF_GLOBAL; else { asym->flags = BSF_LOCAL; /* Normally, a local stProc symbol will have a corresponding external symbol. We mark the local symbol as a debugging symbol, in order to prevent nm from printing both out. Similarly, we mark stLabel and stabs symbols as debugging symbols. In both cases, we do want to set the value correctly based on the symbol class. */ if (ecoff_sym->st == stProc || ecoff_sym->st == stLabel || ECOFF_IS_STAB (ecoff_sym)) asym->flags |= BSF_DEBUGGING; } if (ecoff_sym->st == stProc || ecoff_sym->st == stStaticProc) asym->flags |= BSF_FUNCTION; switch (ecoff_sym->sc) { case scNil: /* Used for compiler generated labels. Leave them in the debugging section, and mark them as local. If BSF_DEBUGGING is set, then nm does not display them for some reason. If no flags are set then the linker whines about them. */ asym->flags = BSF_LOCAL; break; case scText: asym->section = bfd_make_section_old_way (abfd, _TEXT); asym->value -= asym->section->vma; break; case scData: asym->section = bfd_make_section_old_way (abfd, _DATA); asym->value -= asym->section->vma; break; case scBss: asym->section = bfd_make_section_old_way (abfd, _BSS); asym->value -= asym->section->vma; break; case scRegister: asym->flags = BSF_DEBUGGING; break; case scAbs: asym->section = bfd_abs_section_ptr; break; case scUndefined: asym->section = bfd_und_section_ptr; asym->flags = 0; asym->value = 0; break; case scCdbLocal: case scBits: case scCdbSystem: case scRegImage: case scInfo: case scUserStruct: asym->flags = BSF_DEBUGGING; break; case scSData: asym->section = bfd_make_section_old_way (abfd, ".sdata"); asym->value -= asym->section->vma; break; case scSBss: asym->section = bfd_make_section_old_way (abfd, ".sbss"); asym->value -= asym->section->vma; break; case scRData: asym->section = bfd_make_section_old_way (abfd, ".rdata"); asym->value -= asym->section->vma; break; case scVar: asym->flags = BSF_DEBUGGING; break; case scCommon: if (asym->value > ecoff_data (abfd)->gp_size) { asym->section = bfd_com_section_ptr; asym->flags = 0; break; } /* Fall through. */ case scSCommon: if (ecoff_scom_section.name == NULL) { /* Initialize the small common section. */ ecoff_scom_section.name = SCOMMON; ecoff_scom_section.flags = SEC_IS_COMMON; ecoff_scom_section.output_section = &ecoff_scom_section; ecoff_scom_section.symbol = &ecoff_scom_symbol; ecoff_scom_section.symbol_ptr_ptr = &ecoff_scom_symbol_ptr; ecoff_scom_symbol.name = SCOMMON; ecoff_scom_symbol.flags = BSF_SECTION_SYM; ecoff_scom_symbol.section = &ecoff_scom_section; ecoff_scom_symbol_ptr = &ecoff_scom_symbol; } asym->section = &ecoff_scom_section; asym->flags = 0; break; case scVarRegister: case scVariant: asym->flags = BSF_DEBUGGING; break; case scSUndefined: asym->section = bfd_und_section_ptr; asym->flags = 0; asym->value = 0; break; case scInit: asym->section = bfd_make_section_old_way (abfd, ".init"); asym->value -= asym->section->vma; break; case scBasedVar: case scXData: case scPData: asym->flags = BSF_DEBUGGING; break; case scFini: asym->section = bfd_make_section_old_way (abfd, ".fini"); asym->value -= asym->section->vma; break; case scRConst: asym->section = bfd_make_section_old_way (abfd, ".rconst"); asym->value -= asym->section->vma; break; default: break; } /* Look for special constructors symbols and make relocation entries in a special construction section. These are produced by the -fgnu-linker argument to g++. */ if (ECOFF_IS_STAB (ecoff_sym)) { switch (ECOFF_UNMARK_STAB (ecoff_sym->index)) { default: break; case N_SETA: case N_SETT: case N_SETD: case N_SETB: /* Mark the symbol as a constructor. */ asym->flags |= BSF_CONSTRUCTOR; break; } } return TRUE; } /* Read an ECOFF symbol table. */ bfd_boolean _bfd_ecoff_slurp_symbol_table (bfd *abfd) { const struct ecoff_backend_data * const backend = ecoff_backend (abfd); const bfd_size_type external_ext_size = backend->debug_swap.external_ext_size; const bfd_size_type external_sym_size = backend->debug_swap.external_sym_size; void (* const swap_ext_in) (bfd *, void *, EXTR *) = backend->debug_swap.swap_ext_in; void (* const swap_sym_in) (bfd *, void *, SYMR *) = backend->debug_swap.swap_sym_in; bfd_size_type internal_size; ecoff_symbol_type *internal; ecoff_symbol_type *internal_ptr; char *eraw_src; char *eraw_end; FDR *fdr_ptr; FDR *fdr_end; /* If we've already read in the symbol table, do nothing. */ if (ecoff_data (abfd)->canonical_symbols != NULL) return TRUE; /* Get the symbolic information. */ if (! _bfd_ecoff_slurp_symbolic_info (abfd, NULL, &ecoff_data (abfd)->debug_info)) return FALSE; if (bfd_get_symcount (abfd) == 0) return TRUE; internal_size = bfd_get_symcount (abfd); internal_size *= sizeof (ecoff_symbol_type); internal = bfd_alloc (abfd, internal_size); if (internal == NULL) return FALSE; internal_ptr = internal; eraw_src = (char *) ecoff_data (abfd)->debug_info.external_ext; eraw_end = (eraw_src + (ecoff_data (abfd)->debug_info.symbolic_header.iextMax * external_ext_size)); for (; eraw_src < eraw_end; eraw_src += external_ext_size, internal_ptr++) { EXTR internal_esym; (*swap_ext_in) (abfd, (void *) eraw_src, &internal_esym); internal_ptr->symbol.name = (ecoff_data (abfd)->debug_info.ssext + internal_esym.asym.iss); if (!ecoff_set_symbol_info (abfd, &internal_esym.asym, &internal_ptr->symbol, 1, internal_esym.weakext)) return FALSE; /* The alpha uses a negative ifd field for section symbols. */ if (internal_esym.ifd >= 0) internal_ptr->fdr = (ecoff_data (abfd)->debug_info.fdr + internal_esym.ifd); else internal_ptr->fdr = NULL; internal_ptr->local = FALSE; internal_ptr->native = (void *) eraw_src; } /* The local symbols must be accessed via the fdr's, because the string and aux indices are relative to the fdr information. */ fdr_ptr = ecoff_data (abfd)->debug_info.fdr; fdr_end = fdr_ptr + ecoff_data (abfd)->debug_info.symbolic_header.ifdMax; for (; fdr_ptr < fdr_end; fdr_ptr++) { char *lraw_src; char *lraw_end; lraw_src = ((char *) ecoff_data (abfd)->debug_info.external_sym + fdr_ptr->isymBase * external_sym_size); lraw_end = lraw_src + fdr_ptr->csym * external_sym_size; for (; lraw_src < lraw_end; lraw_src += external_sym_size, internal_ptr++) { SYMR internal_sym; (*swap_sym_in) (abfd, (void *) lraw_src, &internal_sym); internal_ptr->symbol.name = (ecoff_data (abfd)->debug_info.ss + fdr_ptr->issBase + internal_sym.iss); if (!ecoff_set_symbol_info (abfd, &internal_sym, &internal_ptr->symbol, 0, 0)) return FALSE; internal_ptr->fdr = fdr_ptr; internal_ptr->local = TRUE; internal_ptr->native = (void *) lraw_src; } } ecoff_data (abfd)->canonical_symbols = internal; return TRUE; } /* Return the amount of space needed for the canonical symbols. */ long _bfd_ecoff_get_symtab_upper_bound (bfd *abfd) { if (! _bfd_ecoff_slurp_symbolic_info (abfd, NULL, &ecoff_data (abfd)->debug_info)) return -1; if (bfd_get_symcount (abfd) == 0) return 0; return (bfd_get_symcount (abfd) + 1) * (sizeof (ecoff_symbol_type *)); } /* Get the canonical symbols. */ long _bfd_ecoff_canonicalize_symtab (bfd *abfd, asymbol **alocation) { unsigned int counter = 0; ecoff_symbol_type *symbase; ecoff_symbol_type **location = (ecoff_symbol_type **) alocation; if (! _bfd_ecoff_slurp_symbol_table (abfd)) return -1; if (bfd_get_symcount (abfd) == 0) return 0; symbase = ecoff_data (abfd)->canonical_symbols; while (counter < bfd_get_symcount (abfd)) { *(location++) = symbase++; counter++; } *location++ = NULL; return bfd_get_symcount (abfd); } /* Turn ECOFF type information into a printable string. ecoff_emit_aggregate and ecoff_type_to_string are from gcc/mips-tdump.c, with swapping added and used_ptr removed. */ /* Write aggregate information to a string. */ static void ecoff_emit_aggregate (bfd *abfd, FDR *fdr, char *string, RNDXR *rndx, long isym, const char *which) { const struct ecoff_debug_swap * const debug_swap = &ecoff_backend (abfd)->debug_swap; struct ecoff_debug_info * const debug_info = &ecoff_data (abfd)->debug_info; unsigned int ifd = rndx->rfd; unsigned int indx = rndx->index; const char *name; if (ifd == 0xfff) ifd = isym; /* An ifd of -1 is an opaque type. An escaped index of 0 is a struct return type of a procedure compiled without -g. */ if (ifd == 0xffffffff || (rndx->rfd == 0xfff && indx == 0)) name = ""; else if (indx == indexNil) name = ""; else { SYMR sym; if (debug_info->external_rfd == NULL) fdr = debug_info->fdr + ifd; else { RFDT rfd; (*debug_swap->swap_rfd_in) (abfd, ((char *) debug_info->external_rfd + ((fdr->rfdBase + ifd) * debug_swap->external_rfd_size)), &rfd); fdr = debug_info->fdr + rfd; } indx += fdr->isymBase; (*debug_swap->swap_sym_in) (abfd, ((char *) debug_info->external_sym + indx * debug_swap->external_sym_size), &sym); name = debug_info->ss + fdr->issBase + sym.iss; } sprintf (string, "%s %s { ifd = %u, index = %lu }", which, name, ifd, ((long) indx + debug_info->symbolic_header.iextMax)); } /* Convert the type information to string format. */ static char * ecoff_type_to_string (bfd *abfd, FDR *fdr, unsigned int indx) { union aux_ext *aux_ptr; int bigendian; AUXU u; struct qual { unsigned int type; int low_bound; int high_bound; int stride; } qualifiers[7]; unsigned int basic_type; int i; char buffer1[1024]; static char buffer2[1024]; char *p1 = buffer1; char *p2 = buffer2; RNDXR rndx; aux_ptr = ecoff_data (abfd)->debug_info.external_aux + fdr->iauxBase; bigendian = fdr->fBigendian; for (i = 0; i < 7; i++) { qualifiers[i].low_bound = 0; qualifiers[i].high_bound = 0; qualifiers[i].stride = 0; } if (AUX_GET_ISYM (bigendian, &aux_ptr[indx]) == (bfd_vma) -1) return "-1 (no type)"; _bfd_ecoff_swap_tir_in (bigendian, &aux_ptr[indx++].a_ti, &u.ti); basic_type = u.ti.bt; qualifiers[0].type = u.ti.tq0; qualifiers[1].type = u.ti.tq1; qualifiers[2].type = u.ti.tq2; qualifiers[3].type = u.ti.tq3; qualifiers[4].type = u.ti.tq4; qualifiers[5].type = u.ti.tq5; qualifiers[6].type = tqNil; /* Go get the basic type. */ switch (basic_type) { case btNil: /* Undefined. */ strcpy (p1, "nil"); break; case btAdr: /* Address - integer same size as pointer. */ strcpy (p1, "address"); break; case btChar: /* Character. */ strcpy (p1, "char"); break; case btUChar: /* Unsigned character. */ strcpy (p1, "unsigned char"); break; case btShort: /* Short. */ strcpy (p1, "short"); break; case btUShort: /* Unsigned short. */ strcpy (p1, "unsigned short"); break; case btInt: /* Int. */ strcpy (p1, "int"); break; case btUInt: /* Unsigned int. */ strcpy (p1, "unsigned int"); break; case btLong: /* Long. */ strcpy (p1, "long"); break; case btULong: /* Unsigned long. */ strcpy (p1, "unsigned long"); break; case btFloat: /* Float (real). */ strcpy (p1, "float"); break; case btDouble: /* Double (real). */ strcpy (p1, "double"); break; /* Structures add 1-2 aux words: 1st word is [ST_RFDESCAPE, offset] pointer to struct def; 2nd word is file index if 1st word rfd is ST_RFDESCAPE. */ case btStruct: /* Structure (Record). */ _bfd_ecoff_swap_rndx_in (bigendian, &aux_ptr[indx].a_rndx, &rndx); ecoff_emit_aggregate (abfd, fdr, p1, &rndx, (long) AUX_GET_ISYM (bigendian, &aux_ptr[indx+1]), "struct"); indx++; /* Skip aux words. */ break; /* Unions add 1-2 aux words: 1st word is [ST_RFDESCAPE, offset] pointer to union def; 2nd word is file index if 1st word rfd is ST_RFDESCAPE. */ case btUnion: /* Union. */ _bfd_ecoff_swap_rndx_in (bigendian, &aux_ptr[indx].a_rndx, &rndx); ecoff_emit_aggregate (abfd, fdr, p1, &rndx, (long) AUX_GET_ISYM (bigendian, &aux_ptr[indx+1]), "union"); indx++; /* Skip aux words. */ break; /* Enumerations add 1-2 aux words: 1st word is [ST_RFDESCAPE, offset] pointer to enum def; 2nd word is file index if 1st word rfd is ST_RFDESCAPE. */ case btEnum: /* Enumeration. */ _bfd_ecoff_swap_rndx_in (bigendian, &aux_ptr[indx].a_rndx, &rndx); ecoff_emit_aggregate (abfd, fdr, p1, &rndx, (long) AUX_GET_ISYM (bigendian, &aux_ptr[indx+1]), "enum"); indx++; /* Skip aux words. */ break; case btTypedef: /* Defined via a typedef, isymRef points. */ strcpy (p1, "typedef"); break; case btRange: /* Subrange of int. */ strcpy (p1, "subrange"); break; case btSet: /* Pascal sets. */ strcpy (p1, "set"); break; case btComplex: /* Fortran complex. */ strcpy (p1, "complex"); break; case btDComplex: /* Fortran double complex. */ strcpy (p1, "double complex"); break; case btIndirect: /* Forward or unnamed typedef. */ strcpy (p1, "forward/unamed typedef"); break; case btFixedDec: /* Fixed Decimal. */ strcpy (p1, "fixed decimal"); break; case btFloatDec: /* Float Decimal. */ strcpy (p1, "float decimal"); break; case btString: /* Varying Length Character String. */ strcpy (p1, "string"); break; case btBit: /* Aligned Bit String. */ strcpy (p1, "bit"); break; case btPicture: /* Picture. */ strcpy (p1, "picture"); break; case btVoid: /* Void. */ strcpy (p1, "void"); break; default: sprintf (p1, _("Unknown basic type %d"), (int) basic_type); break; } p1 += strlen (buffer1); /* If this is a bitfield, get the bitsize. */ if (u.ti.fBitfield) { int bitsize; bitsize = AUX_GET_WIDTH (bigendian, &aux_ptr[indx++]); sprintf (p1, " : %d", bitsize); p1 += strlen (buffer1); } /* Deal with any qualifiers. */ if (qualifiers[0].type != tqNil) { /* Snarf up any array bounds in the correct order. Arrays store 5 successive words in the aux. table: word 0 RNDXR to type of the bounds (ie, int) word 1 Current file descriptor index word 2 low bound word 3 high bound (or -1 if []) word 4 stride size in bits. */ for (i = 0; i < 7; i++) { if (qualifiers[i].type == tqArray) { qualifiers[i].low_bound = AUX_GET_DNLOW (bigendian, &aux_ptr[indx+2]); qualifiers[i].high_bound = AUX_GET_DNHIGH (bigendian, &aux_ptr[indx+3]); qualifiers[i].stride = AUX_GET_WIDTH (bigendian, &aux_ptr[indx+4]); indx += 5; } } /* Now print out the qualifiers. */ for (i = 0; i < 6; i++) { switch (qualifiers[i].type) { case tqNil: case tqMax: break; case tqPtr: strcpy (p2, "ptr to "); p2 += sizeof ("ptr to ")-1; break; case tqVol: strcpy (p2, "volatile "); p2 += sizeof ("volatile ")-1; break; case tqFar: strcpy (p2, "far "); p2 += sizeof ("far ")-1; break; case tqProc: strcpy (p2, "func. ret. "); p2 += sizeof ("func. ret. "); break; case tqArray: { int first_array = i; int j; /* Print array bounds reversed (ie, in the order the C programmer writes them). C is such a fun language.... */ while (i < 5 && qualifiers[i+1].type == tqArray) i++; for (j = i; j >= first_array; j--) { strcpy (p2, "array ["); p2 += sizeof ("array [")-1; if (qualifiers[j].low_bound != 0) sprintf (p2, "%ld:%ld {%ld bits}", (long) qualifiers[j].low_bound, (long) qualifiers[j].high_bound, (long) qualifiers[j].stride); else if (qualifiers[j].high_bound != -1) sprintf (p2, "%ld {%ld bits}", (long) (qualifiers[j].high_bound + 1), (long) (qualifiers[j].stride)); else sprintf (p2, " {%ld bits}", (long) (qualifiers[j].stride)); p2 += strlen (p2); strcpy (p2, "] of "); p2 += sizeof ("] of ")-1; } } break; } } } strcpy (p2, buffer1); return buffer2; } /* Return information about ECOFF symbol SYMBOL in RET. */ void _bfd_ecoff_get_symbol_info (bfd *abfd ATTRIBUTE_UNUSED, asymbol *symbol, symbol_info *ret) { bfd_symbol_info (symbol, ret); } /* Return whether this is a local label. */ bfd_boolean _bfd_ecoff_bfd_is_local_label_name (bfd *abfd ATTRIBUTE_UNUSED, const char *name) { return name[0] == '$'; } /* Print information about an ECOFF symbol. */ void _bfd_ecoff_print_symbol (bfd *abfd, void * filep, asymbol *symbol, bfd_print_symbol_type how) { const struct ecoff_debug_swap * const debug_swap = &ecoff_backend (abfd)->debug_swap; FILE *file = (FILE *)filep; switch (how) { case bfd_print_symbol_name: fprintf (file, "%s", symbol->name); break; case bfd_print_symbol_more: if (ecoffsymbol (symbol)->local) { SYMR ecoff_sym; (*debug_swap->swap_sym_in) (abfd, ecoffsymbol (symbol)->native, &ecoff_sym); fprintf (file, "ecoff local "); fprintf_vma (file, (bfd_vma) ecoff_sym.value); fprintf (file, " %x %x", (unsigned) ecoff_sym.st, (unsigned) ecoff_sym.sc); } else { EXTR ecoff_ext; (*debug_swap->swap_ext_in) (abfd, ecoffsymbol (symbol)->native, &ecoff_ext); fprintf (file, "ecoff extern "); fprintf_vma (file, (bfd_vma) ecoff_ext.asym.value); fprintf (file, " %x %x", (unsigned) ecoff_ext.asym.st, (unsigned) ecoff_ext.asym.sc); } break; case bfd_print_symbol_all: /* Print out the symbols in a reasonable way. */ { char type; int pos; EXTR ecoff_ext; char jmptbl; char cobol_main; char weakext; if (ecoffsymbol (symbol)->local) { (*debug_swap->swap_sym_in) (abfd, ecoffsymbol (symbol)->native, &ecoff_ext.asym); type = 'l'; pos = ((((char *) ecoffsymbol (symbol)->native - (char *) ecoff_data (abfd)->debug_info.external_sym) / debug_swap->external_sym_size) + ecoff_data (abfd)->debug_info.symbolic_header.iextMax); jmptbl = ' '; cobol_main = ' '; weakext = ' '; } else { (*debug_swap->swap_ext_in) (abfd, ecoffsymbol (symbol)->native, &ecoff_ext); type = 'e'; pos = (((char *) ecoffsymbol (symbol)->native - (char *) ecoff_data (abfd)->debug_info.external_ext) / debug_swap->external_ext_size); jmptbl = ecoff_ext.jmptbl ? 'j' : ' '; cobol_main = ecoff_ext.cobol_main ? 'c' : ' '; weakext = ecoff_ext.weakext ? 'w' : ' '; } fprintf (file, "[%3d] %c ", pos, type); fprintf_vma (file, (bfd_vma) ecoff_ext.asym.value); fprintf (file, " st %x sc %x indx %x %c%c%c %s", (unsigned) ecoff_ext.asym.st, (unsigned) ecoff_ext.asym.sc, (unsigned) ecoff_ext.asym.index, jmptbl, cobol_main, weakext, symbol->name); if (ecoffsymbol (symbol)->fdr != NULL && ecoff_ext.asym.index != indexNil) { FDR *fdr; unsigned int indx; int bigendian; bfd_size_type sym_base; union aux_ext *aux_base; fdr = ecoffsymbol (symbol)->fdr; indx = ecoff_ext.asym.index; /* sym_base is used to map the fdr relative indices which appear in the file to the position number which we are using. */ sym_base = fdr->isymBase; if (ecoffsymbol (symbol)->local) sym_base += ecoff_data (abfd)->debug_info.symbolic_header.iextMax; /* aux_base is the start of the aux entries for this file; asym.index is an offset from this. */ aux_base = (ecoff_data (abfd)->debug_info.external_aux + fdr->iauxBase); /* The aux entries are stored in host byte order; the order is indicated by a bit in the fdr. */ bigendian = fdr->fBigendian; /* This switch is basically from gcc/mips-tdump.c. */ switch (ecoff_ext.asym.st) { case stNil: case stLabel: break; case stFile: case stBlock: fprintf (file, _("\n End+1 symbol: %ld"), (long) (indx + sym_base)); break; case stEnd: if (ecoff_ext.asym.sc == scText || ecoff_ext.asym.sc == scInfo) fprintf (file, _("\n First symbol: %ld"), (long) (indx + sym_base)); else fprintf (file, _("\n First symbol: %ld"), ((long) (AUX_GET_ISYM (bigendian, &aux_base[ecoff_ext.asym.index]) + sym_base))); break; case stProc: case stStaticProc: if (ECOFF_IS_STAB (&ecoff_ext.asym)) ; else if (ecoffsymbol (symbol)->local) fprintf (file, _("\n End+1 symbol: %-7ld Type: %s"), ((long) (AUX_GET_ISYM (bigendian, &aux_base[ecoff_ext.asym.index]) + sym_base)), ecoff_type_to_string (abfd, fdr, indx + 1)); else fprintf (file, _("\n Local symbol: %ld"), ((long) indx + (long) sym_base + (ecoff_data (abfd) ->debug_info.symbolic_header.iextMax))); break; case stStruct: fprintf (file, _("\n struct; End+1 symbol: %ld"), (long) (indx + sym_base)); break; case stUnion: fprintf (file, _("\n union; End+1 symbol: %ld"), (long) (indx + sym_base)); break; case stEnum: fprintf (file, _("\n enum; End+1 symbol: %ld"), (long) (indx + sym_base)); break; default: if (! ECOFF_IS_STAB (&ecoff_ext.asym)) fprintf (file, _("\n Type: %s"), ecoff_type_to_string (abfd, fdr, indx)); break; } } } break; } } /* Read in the relocs for a section. */ static bfd_boolean ecoff_slurp_reloc_table (bfd *abfd, asection *section, asymbol **symbols) { const struct ecoff_backend_data * const backend = ecoff_backend (abfd); arelent *internal_relocs; bfd_size_type external_reloc_size; bfd_size_type amt; char *external_relocs; arelent *rptr; unsigned int i; if (section->relocation != NULL || section->reloc_count == 0 || (section->flags & SEC_CONSTRUCTOR) != 0) return TRUE; if (! _bfd_ecoff_slurp_symbol_table (abfd)) return FALSE; amt = section->reloc_count; amt *= sizeof (arelent); internal_relocs = bfd_alloc (abfd, amt); external_reloc_size = backend->external_reloc_size; amt = external_reloc_size * section->reloc_count; external_relocs = bfd_alloc (abfd, amt); if (internal_relocs == NULL || external_relocs == NULL) return FALSE; if (bfd_seek (abfd, section->rel_filepos, SEEK_SET) != 0) return FALSE; if (bfd_bread (external_relocs, amt, abfd) != amt) return FALSE; for (i = 0, rptr = internal_relocs; i < section->reloc_count; i++, rptr++) { struct internal_reloc intern; (*backend->swap_reloc_in) (abfd, external_relocs + i * external_reloc_size, &intern); if (intern.r_extern) { /* r_symndx is an index into the external symbols. */ BFD_ASSERT (intern.r_symndx >= 0 && (intern.r_symndx < (ecoff_data (abfd) ->debug_info.symbolic_header.iextMax))); rptr->sym_ptr_ptr = symbols + intern.r_symndx; rptr->addend = 0; } else if (intern.r_symndx == RELOC_SECTION_NONE || intern.r_symndx == RELOC_SECTION_ABS) { rptr->sym_ptr_ptr = bfd_abs_section_ptr->symbol_ptr_ptr; rptr->addend = 0; } else { const char *sec_name; asection *sec; /* r_symndx is a section key. */ switch (intern.r_symndx) { case RELOC_SECTION_TEXT: sec_name = _TEXT; break; case RELOC_SECTION_RDATA: sec_name = _RDATA; break; case RELOC_SECTION_DATA: sec_name = _DATA; break; case RELOC_SECTION_SDATA: sec_name = _SDATA; break; case RELOC_SECTION_SBSS: sec_name = _SBSS; break; case RELOC_SECTION_BSS: sec_name = _BSS; break; case RELOC_SECTION_INIT: sec_name = _INIT; break; case RELOC_SECTION_LIT8: sec_name = _LIT8; break; case RELOC_SECTION_LIT4: sec_name = _LIT4; break; case RELOC_SECTION_XDATA: sec_name = _XDATA; break; case RELOC_SECTION_PDATA: sec_name = _PDATA; break; case RELOC_SECTION_FINI: sec_name = _FINI; break; case RELOC_SECTION_LITA: sec_name = _LITA; break; case RELOC_SECTION_RCONST: sec_name = _RCONST; break; default: abort (); } sec = bfd_get_section_by_name (abfd, sec_name); if (sec == NULL) abort (); rptr->sym_ptr_ptr = sec->symbol_ptr_ptr; rptr->addend = - bfd_get_section_vma (abfd, sec); } rptr->address = intern.r_vaddr - bfd_get_section_vma (abfd, section); /* Let the backend select the howto field and do any other required processing. */ (*backend->adjust_reloc_in) (abfd, &intern, rptr); } bfd_release (abfd, external_relocs); section->relocation = internal_relocs; return TRUE; } /* Get a canonical list of relocs. */ long _bfd_ecoff_canonicalize_reloc (bfd *abfd, asection *section, arelent **relptr, asymbol **symbols) { unsigned int count; if (section->flags & SEC_CONSTRUCTOR) { arelent_chain *chain; /* This section has relocs made up by us, not the file, so take them out of their chain and place them into the data area provided. */ for (count = 0, chain = section->constructor_chain; count < section->reloc_count; count++, chain = chain->next) *relptr++ = &chain->relent; } else { arelent *tblptr; if (! ecoff_slurp_reloc_table (abfd, section, symbols)) return -1; tblptr = section->relocation; for (count = 0; count < section->reloc_count; count++) *relptr++ = tblptr++; } *relptr = NULL; return section->reloc_count; } /* Provided a BFD, a section and an offset into the section, calculate and return the name of the source file and the line nearest to the wanted location. */ bfd_boolean _bfd_ecoff_find_nearest_line (bfd *abfd, asection *section, asymbol **ignore_symbols ATTRIBUTE_UNUSED, bfd_vma offset, const char **filename_ptr, const char **functionname_ptr, unsigned int *retline_ptr) { const struct ecoff_debug_swap * const debug_swap = &ecoff_backend (abfd)->debug_swap; struct ecoff_debug_info * const debug_info = &ecoff_data (abfd)->debug_info; struct ecoff_find_line *line_info; /* Make sure we have the FDR's. */ if (! _bfd_ecoff_slurp_symbolic_info (abfd, NULL, debug_info) || bfd_get_symcount (abfd) == 0) return FALSE; if (ecoff_data (abfd)->find_line_info == NULL) { bfd_size_type amt = sizeof (struct ecoff_find_line); ecoff_data (abfd)->find_line_info = bfd_zalloc (abfd, amt); if (ecoff_data (abfd)->find_line_info == NULL) return FALSE; } line_info = ecoff_data (abfd)->find_line_info; return _bfd_ecoff_locate_line (abfd, section, offset, debug_info, debug_swap, line_info, filename_ptr, functionname_ptr, retline_ptr); } /* Copy private BFD data. This is called by objcopy and strip. We use it to copy the ECOFF debugging information from one BFD to the other. It would be theoretically possible to represent the ECOFF debugging information in the symbol table. However, it would be a lot of work, and there would be little gain (gas, gdb, and ld already access the ECOFF debugging information via the ecoff_debug_info structure, and that structure would have to be retained in order to support ECOFF debugging in MIPS ELF). The debugging information for the ECOFF external symbols comes from the symbol table, so this function only handles the other debugging information. */ bfd_boolean _bfd_ecoff_bfd_copy_private_bfd_data (bfd *ibfd, bfd *obfd) { struct ecoff_debug_info *iinfo = &ecoff_data (ibfd)->debug_info; struct ecoff_debug_info *oinfo = &ecoff_data (obfd)->debug_info; int i; asymbol **sym_ptr_ptr; size_t c; bfd_boolean local; /* We only want to copy information over if both BFD's use ECOFF format. */ if (bfd_get_flavour (ibfd) != bfd_target_ecoff_flavour || bfd_get_flavour (obfd) != bfd_target_ecoff_flavour) return TRUE; /* Copy the GP value and the register masks. */ ecoff_data (obfd)->gp = ecoff_data (ibfd)->gp; ecoff_data (obfd)->gprmask = ecoff_data (ibfd)->gprmask; ecoff_data (obfd)->fprmask = ecoff_data (ibfd)->fprmask; for (i = 0; i < 3; i++) ecoff_data (obfd)->cprmask[i] = ecoff_data (ibfd)->cprmask[i]; /* Copy the version stamp. */ oinfo->symbolic_header.vstamp = iinfo->symbolic_header.vstamp; /* If there are no symbols, don't copy any debugging information. */ c = bfd_get_symcount (obfd); sym_ptr_ptr = bfd_get_outsymbols (obfd); if (c == 0 || sym_ptr_ptr == NULL) return TRUE; /* See if there are any local symbols. */ local = FALSE; for (; c > 0; c--, sym_ptr_ptr++) { if (ecoffsymbol (*sym_ptr_ptr)->local) { local = TRUE; break; } } if (local) { /* There are some local symbols. We just bring over all the debugging information. FIXME: This is not quite the right thing to do. If the user has asked us to discard all debugging information, then we are probably going to wind up keeping it because there will probably be some local symbol which objcopy did not discard. We should actually break apart the debugging information and only keep that which applies to the symbols we want to keep. */ oinfo->symbolic_header.ilineMax = iinfo->symbolic_header.ilineMax; oinfo->symbolic_header.cbLine = iinfo->symbolic_header.cbLine; oinfo->line = iinfo->line; oinfo->symbolic_header.idnMax = iinfo->symbolic_header.idnMax; oinfo->external_dnr = iinfo->external_dnr; oinfo->symbolic_header.ipdMax = iinfo->symbolic_header.ipdMax; oinfo->external_pdr = iinfo->external_pdr; oinfo->symbolic_header.isymMax = iinfo->symbolic_header.isymMax; oinfo->external_sym = iinfo->external_sym; oinfo->symbolic_header.ioptMax = iinfo->symbolic_header.ioptMax; oinfo->external_opt = iinfo->external_opt; oinfo->symbolic_header.iauxMax = iinfo->symbolic_header.iauxMax; oinfo->external_aux = iinfo->external_aux; oinfo->symbolic_header.issMax = iinfo->symbolic_header.issMax; oinfo->ss = iinfo->ss; oinfo->symbolic_header.ifdMax = iinfo->symbolic_header.ifdMax; oinfo->external_fdr = iinfo->external_fdr; oinfo->symbolic_header.crfd = iinfo->symbolic_header.crfd; oinfo->external_rfd = iinfo->external_rfd; } else { /* We are discarding all the local symbol information. Look through the external symbols and remove all references to FDR or aux information. */ c = bfd_get_symcount (obfd); sym_ptr_ptr = bfd_get_outsymbols (obfd); for (; c > 0; c--, sym_ptr_ptr++) { EXTR esym; (*(ecoff_backend (obfd)->debug_swap.swap_ext_in)) (obfd, ecoffsymbol (*sym_ptr_ptr)->native, &esym); esym.ifd = ifdNil; esym.asym.index = indexNil; (*(ecoff_backend (obfd)->debug_swap.swap_ext_out)) (obfd, &esym, ecoffsymbol (*sym_ptr_ptr)->native); } } return TRUE; } /* Set the architecture. The supported architecture is stored in the backend pointer. We always set the architecture anyhow, since many callers ignore the return value. */ bfd_boolean _bfd_ecoff_set_arch_mach (bfd *abfd, enum bfd_architecture arch, unsigned long machine) { bfd_default_set_arch_mach (abfd, arch, machine); return arch == ecoff_backend (abfd)->arch; } /* Get the size of the section headers. */ int _bfd_ecoff_sizeof_headers (bfd *abfd, struct bfd_link_info *info ATTRIBUTE_UNUSED) { asection *current; int c; int ret; c = 0; for (current = abfd->sections; current != NULL; current = current->next) ++c; ret = (bfd_coff_filhsz (abfd) + bfd_coff_aoutsz (abfd) + c * bfd_coff_scnhsz (abfd)); return BFD_ALIGN (ret, 16); } /* Get the contents of a section. */ bfd_boolean _bfd_ecoff_get_section_contents (bfd *abfd, asection *section, void * location, file_ptr offset, bfd_size_type count) { return _bfd_generic_get_section_contents (abfd, section, location, offset, count); } /* Sort sections by VMA, but put SEC_ALLOC sections first. This is called via qsort. */ static int ecoff_sort_hdrs (const void * arg1, const void * arg2) { const asection *hdr1 = *(const asection **) arg1; const asection *hdr2 = *(const asection **) arg2; if ((hdr1->flags & SEC_ALLOC) != 0) { if ((hdr2->flags & SEC_ALLOC) == 0) return -1; } else { if ((hdr2->flags & SEC_ALLOC) != 0) return 1; } if (hdr1->vma < hdr2->vma) return -1; else if (hdr1->vma > hdr2->vma) return 1; else return 0; } /* Calculate the file position for each section, and set reloc_filepos. */ static bfd_boolean ecoff_compute_section_file_positions (bfd *abfd) { file_ptr sofar, file_sofar; asection **sorted_hdrs; asection *current; unsigned int i; file_ptr old_sofar; bfd_boolean rdata_in_text; bfd_boolean first_data, first_nonalloc; const bfd_vma round = ecoff_backend (abfd)->round; bfd_size_type amt; sofar = _bfd_ecoff_sizeof_headers (abfd, NULL); file_sofar = sofar; /* Sort the sections by VMA. */ amt = abfd->section_count; amt *= sizeof (asection *); sorted_hdrs = bfd_malloc (amt); if (sorted_hdrs == NULL) return FALSE; for (current = abfd->sections, i = 0; current != NULL; current = current->next, i++) sorted_hdrs[i] = current; BFD_ASSERT (i == abfd->section_count); qsort (sorted_hdrs, abfd->section_count, sizeof (asection *), ecoff_sort_hdrs); /* Some versions of the OSF linker put the .rdata section in the text segment, and some do not. */ rdata_in_text = ecoff_backend (abfd)->rdata_in_text; if (rdata_in_text) { for (i = 0; i < abfd->section_count; i++) { current = sorted_hdrs[i]; if (streq (current->name, _RDATA)) break; if ((current->flags & SEC_CODE) == 0 && ! streq (current->name, _PDATA) && ! streq (current->name, _RCONST)) { rdata_in_text = FALSE; break; } } } ecoff_data (abfd)->rdata_in_text = rdata_in_text; first_data = TRUE; first_nonalloc = TRUE; for (i = 0; i < abfd->section_count; i++) { unsigned int alignment_power; current = sorted_hdrs[i]; /* For the Alpha ECOFF .pdata section the lnnoptr field is supposed to indicate the number of .pdata entries that are really in the section. Each entry is 8 bytes. We store this away in line_filepos before increasing the section size. */ if (streq (current->name, _PDATA)) current->line_filepos = current->size / 8; alignment_power = current->alignment_power; /* On Ultrix, the data sections in an executable file must be aligned to a page boundary within the file. This does not affect the section size, though. FIXME: Does this work for other platforms? It requires some modification for the Alpha, because .rdata on the Alpha goes with the text, not the data. */ if ((abfd->flags & EXEC_P) != 0 && (abfd->flags & D_PAGED) != 0 && ! first_data && (current->flags & SEC_CODE) == 0 && (! rdata_in_text || ! streq (current->name, _RDATA)) && ! streq (current->name, _PDATA) && ! streq (current->name, _RCONST)) { sofar = (sofar + round - 1) &~ (round - 1); file_sofar = (file_sofar + round - 1) &~ (round - 1); first_data = FALSE; } else if (streq (current->name, _LIB)) { /* On Irix 4, the location of contents of the .lib section from a shared library section is also rounded up to a page boundary. */ sofar = (sofar + round - 1) &~ (round - 1); file_sofar = (file_sofar + round - 1) &~ (round - 1); } else if (first_nonalloc && (current->flags & SEC_ALLOC) == 0 && (abfd->flags & D_PAGED) != 0) { /* Skip up to the next page for an unallocated section, such as the .comment section on the Alpha. This leaves room for the .bss section. */ first_nonalloc = FALSE; sofar = (sofar + round - 1) &~ (round - 1); file_sofar = (file_sofar + round - 1) &~ (round - 1); } /* Align the sections in the file to the same boundary on which they are aligned in virtual memory. */ sofar = BFD_ALIGN (sofar, 1 << alignment_power); if ((current->flags & SEC_HAS_CONTENTS) != 0) file_sofar = BFD_ALIGN (file_sofar, 1 << alignment_power); if ((abfd->flags & D_PAGED) != 0 && (current->flags & SEC_ALLOC) != 0) { sofar += (current->vma - sofar) % round; if ((current->flags & SEC_HAS_CONTENTS) != 0) file_sofar += (current->vma - file_sofar) % round; } if ((current->flags & (SEC_HAS_CONTENTS | SEC_LOAD)) != 0) current->filepos = file_sofar; sofar += current->size; if ((current->flags & SEC_HAS_CONTENTS) != 0) file_sofar += current->size; /* Make sure that this section is of the right size too. */ old_sofar = sofar; sofar = BFD_ALIGN (sofar, 1 << alignment_power); if ((current->flags & SEC_HAS_CONTENTS) != 0) file_sofar = BFD_ALIGN (file_sofar, 1 << alignment_power); current->size += sofar - old_sofar; } free (sorted_hdrs); sorted_hdrs = NULL; ecoff_data (abfd)->reloc_filepos = file_sofar; return TRUE; } /* Determine the location of the relocs for all the sections in the output file, as well as the location of the symbolic debugging information. */ static bfd_size_type ecoff_compute_reloc_file_positions (bfd *abfd) { const bfd_size_type external_reloc_size = ecoff_backend (abfd)->external_reloc_size; file_ptr reloc_base; bfd_size_type reloc_size; asection *current; file_ptr sym_base; if (! abfd->output_has_begun) { if (! ecoff_compute_section_file_positions (abfd)) abort (); abfd->output_has_begun = TRUE; } reloc_base = ecoff_data (abfd)->reloc_filepos; reloc_size = 0; for (current = abfd->sections; current != NULL; current = current->next) { if (current->reloc_count == 0) current->rel_filepos = 0; else { bfd_size_type relsize; current->rel_filepos = reloc_base; relsize = current->reloc_count * external_reloc_size; reloc_size += relsize; reloc_base += relsize; } } sym_base = ecoff_data (abfd)->reloc_filepos + reloc_size; /* At least on Ultrix, the symbol table of an executable file must be aligned to a page boundary. FIXME: Is this true on other platforms? */ if ((abfd->flags & EXEC_P) != 0 && (abfd->flags & D_PAGED) != 0) sym_base = ((sym_base + ecoff_backend (abfd)->round - 1) &~ (ecoff_backend (abfd)->round - 1)); ecoff_data (abfd)->sym_filepos = sym_base; return reloc_size; } /* Set the contents of a section. */ bfd_boolean _bfd_ecoff_set_section_contents (bfd *abfd, asection *section, const void * location, file_ptr offset, bfd_size_type count) { file_ptr pos; /* This must be done first, because bfd_set_section_contents is going to set output_has_begun to TRUE. */ if (! abfd->output_has_begun && ! ecoff_compute_section_file_positions (abfd)) return FALSE; /* Handle the .lib section specially so that Irix 4 shared libraries work out. See coff_set_section_contents in coffcode.h. */ if (streq (section->name, _LIB)) { bfd_byte *rec, *recend; rec = (bfd_byte *) location; recend = rec + count; while (rec < recend) { ++section->lma; rec += bfd_get_32 (abfd, rec) * 4; } BFD_ASSERT (rec == recend); } if (count == 0) return TRUE; pos = section->filepos + offset; if (bfd_seek (abfd, pos, SEEK_SET) != 0 || bfd_bwrite (location, count, abfd) != count) return FALSE; return TRUE; } /* Get the GP value for an ECOFF file. This is a hook used by nlmconv. */ bfd_vma bfd_ecoff_get_gp_value (bfd *abfd) { if (bfd_get_flavour (abfd) != bfd_target_ecoff_flavour || bfd_get_format (abfd) != bfd_object) { bfd_set_error (bfd_error_invalid_operation); return 0; } return ecoff_data (abfd)->gp; } /* Set the GP value for an ECOFF file. This is a hook used by the assembler. */ bfd_boolean bfd_ecoff_set_gp_value (bfd *abfd, bfd_vma gp_value) { if (bfd_get_flavour (abfd) != bfd_target_ecoff_flavour || bfd_get_format (abfd) != bfd_object) { bfd_set_error (bfd_error_invalid_operation); return FALSE; } ecoff_data (abfd)->gp = gp_value; return TRUE; } /* Set the register masks for an ECOFF file. This is a hook used by the assembler. */ bfd_boolean bfd_ecoff_set_regmasks (bfd *abfd, unsigned long gprmask, unsigned long fprmask, unsigned long *cprmask) { ecoff_data_type *tdata; if (bfd_get_flavour (abfd) != bfd_target_ecoff_flavour || bfd_get_format (abfd) != bfd_object) { bfd_set_error (bfd_error_invalid_operation); return FALSE; } tdata = ecoff_data (abfd); tdata->gprmask = gprmask; tdata->fprmask = fprmask; if (cprmask != NULL) { int i; for (i = 0; i < 3; i++) tdata->cprmask[i] = cprmask[i]; } return TRUE; } /* Get ECOFF EXTR information for an external symbol. This function is passed to bfd_ecoff_debug_externals. */ static bfd_boolean ecoff_get_extr (asymbol *sym, EXTR *esym) { ecoff_symbol_type *ecoff_sym_ptr; bfd *input_bfd; if (bfd_asymbol_flavour (sym) != bfd_target_ecoff_flavour || ecoffsymbol (sym)->native == NULL) { /* Don't include debugging, local, or section symbols. */ if ((sym->flags & BSF_DEBUGGING) != 0 || (sym->flags & BSF_LOCAL) != 0 || (sym->flags & BSF_SECTION_SYM) != 0) return FALSE; esym->jmptbl = 0; esym->cobol_main = 0; esym->weakext = (sym->flags & BSF_WEAK) != 0; esym->reserved = 0; esym->ifd = ifdNil; /* FIXME: we can do better than this for st and sc. */ esym->asym.st = stGlobal; esym->asym.sc = scAbs; esym->asym.reserved = 0; esym->asym.index = indexNil; return TRUE; } ecoff_sym_ptr = ecoffsymbol (sym); if (ecoff_sym_ptr->local) return FALSE; input_bfd = bfd_asymbol_bfd (sym); (*(ecoff_backend (input_bfd)->debug_swap.swap_ext_in)) (input_bfd, ecoff_sym_ptr->native, esym); /* If the symbol was defined by the linker, then esym will be undefined but sym will not be. Get a better class for such a symbol. */ if ((esym->asym.sc == scUndefined || esym->asym.sc == scSUndefined) && ! bfd_is_und_section (bfd_get_section (sym))) esym->asym.sc = scAbs; /* Adjust the FDR index for the symbol by that used for the input BFD. */ if (esym->ifd != -1) { struct ecoff_debug_info *input_debug; input_debug = &ecoff_data (input_bfd)->debug_info; BFD_ASSERT (esym->ifd < input_debug->symbolic_header.ifdMax); if (input_debug->ifdmap != NULL) esym->ifd = input_debug->ifdmap[esym->ifd]; } return TRUE; } /* Set the external symbol index. This routine is passed to bfd_ecoff_debug_externals. */ static void ecoff_set_index (asymbol *sym, bfd_size_type indx) { ecoff_set_sym_index (sym, indx); } /* Write out an ECOFF file. */ bfd_boolean _bfd_ecoff_write_object_contents (bfd *abfd) { const struct ecoff_backend_data * const backend = ecoff_backend (abfd); const bfd_vma round = backend->round; const bfd_size_type filhsz = bfd_coff_filhsz (abfd); const bfd_size_type aoutsz = bfd_coff_aoutsz (abfd); const bfd_size_type scnhsz = bfd_coff_scnhsz (abfd); const bfd_size_type external_hdr_size = backend->debug_swap.external_hdr_size; const bfd_size_type external_reloc_size = backend->external_reloc_size; void (* const adjust_reloc_out) (bfd *, const arelent *, struct internal_reloc *) = backend->adjust_reloc_out; void (* const swap_reloc_out) (bfd *, const struct internal_reloc *, void *) = backend->swap_reloc_out; struct ecoff_debug_info * const debug = &ecoff_data (abfd)->debug_info; HDRR * const symhdr = &debug->symbolic_header; asection *current; unsigned int count; bfd_size_type reloc_size; bfd_size_type text_size; bfd_vma text_start; bfd_boolean set_text_start; bfd_size_type data_size; bfd_vma data_start; bfd_boolean set_data_start; bfd_size_type bss_size; void * buff = NULL; void * reloc_buff = NULL; struct internal_filehdr internal_f; struct internal_aouthdr internal_a; int i; /* Determine where the sections and relocs will go in the output file. */ reloc_size = ecoff_compute_reloc_file_positions (abfd); count = 1; for (current = abfd->sections; current != NULL; current = current->next) { current->target_index = count; ++count; } if ((abfd->flags & D_PAGED) != 0) text_size = _bfd_ecoff_sizeof_headers (abfd, NULL); else text_size = 0; text_start = 0; set_text_start = FALSE; data_size = 0; data_start = 0; set_data_start = FALSE; bss_size = 0; /* Write section headers to the file. */ /* Allocate buff big enough to hold a section header, file header, or a.out header. */ { bfd_size_type siz; siz = scnhsz; if (siz < filhsz) siz = filhsz; if (siz < aoutsz) siz = aoutsz; buff = bfd_malloc (siz); if (buff == NULL) goto error_return; } internal_f.f_nscns = 0; if (bfd_seek (abfd, (file_ptr) (filhsz + aoutsz), SEEK_SET) != 0) goto error_return; for (current = abfd->sections; current != NULL; current = current->next) { struct internal_scnhdr section; bfd_vma vma; ++internal_f.f_nscns; strncpy (section.s_name, current->name, sizeof section.s_name); /* This seems to be correct for Irix 4 shared libraries. */ vma = bfd_get_section_vma (abfd, current); if (streq (current->name, _LIB)) section.s_vaddr = 0; else section.s_vaddr = vma; section.s_paddr = current->lma; section.s_size = current->size; /* If this section is unloadable then the scnptr will be 0. */ if ((current->flags & (SEC_LOAD | SEC_HAS_CONTENTS)) == 0) section.s_scnptr = 0; else section.s_scnptr = current->filepos; section.s_relptr = current->rel_filepos; /* FIXME: the lnnoptr of the .sbss or .sdata section of an object file produced by the assembler is supposed to point to information about how much room is required by objects of various different sizes. I think this only matters if we want the linker to compute the best size to use, or something. I don't know what happens if the information is not present. */ if (! streq (current->name, _PDATA)) section.s_lnnoptr = 0; else { /* The Alpha ECOFF .pdata section uses the lnnoptr field to hold the number of entries in the section (each entry is 8 bytes). We stored this in the line_filepos field in ecoff_compute_section_file_positions. */ section.s_lnnoptr = current->line_filepos; } section.s_nreloc = current->reloc_count; section.s_nlnno = 0; section.s_flags = ecoff_sec_to_styp_flags (current->name, current->flags); if (bfd_coff_swap_scnhdr_out (abfd, (void *) §ion, buff) == 0 || bfd_bwrite (buff, scnhsz, abfd) != scnhsz) goto error_return; if ((section.s_flags & STYP_TEXT) != 0 || ((section.s_flags & STYP_RDATA) != 0 && ecoff_data (abfd)->rdata_in_text) || section.s_flags == STYP_PDATA || (section.s_flags & STYP_DYNAMIC) != 0 || (section.s_flags & STYP_LIBLIST) != 0 || (section.s_flags & STYP_RELDYN) != 0 || section.s_flags == STYP_CONFLIC || (section.s_flags & STYP_DYNSTR) != 0 || (section.s_flags & STYP_DYNSYM) != 0 || (section.s_flags & STYP_HASH) != 0 || (section.s_flags & STYP_ECOFF_INIT) != 0 || (section.s_flags & STYP_ECOFF_FINI) != 0 || section.s_flags == STYP_RCONST) { text_size += current->size; if (! set_text_start || text_start > vma) { text_start = vma; set_text_start = TRUE; } } else if ((section.s_flags & STYP_RDATA) != 0 || (section.s_flags & STYP_DATA) != 0 || (section.s_flags & STYP_LITA) != 0 || (section.s_flags & STYP_LIT8) != 0 || (section.s_flags & STYP_LIT4) != 0 || (section.s_flags & STYP_SDATA) != 0 || section.s_flags == STYP_XDATA || (section.s_flags & STYP_GOT) != 0) { data_size += current->size; if (! set_data_start || data_start > vma) { data_start = vma; set_data_start = TRUE; } } else if ((section.s_flags & STYP_BSS) != 0 || (section.s_flags & STYP_SBSS) != 0) bss_size += current->size; else if (section.s_flags == 0 || (section.s_flags & STYP_ECOFF_LIB) != 0 || section.s_flags == STYP_COMMENT) /* Do nothing. */ ; else abort (); } /* Set up the file header. */ internal_f.f_magic = ecoff_get_magic (abfd); /* We will NOT put a fucking timestamp in the header here. Every time you put it back, I will come in and take it out again. I'm sorry. This field does not belong here. We fill it with a 0 so it compares the same but is not a reasonable time. -- gnu@cygnus.com. */ internal_f.f_timdat = 0; if (bfd_get_symcount (abfd) != 0) { /* The ECOFF f_nsyms field is not actually the number of symbols, it's the size of symbolic information header. */ internal_f.f_nsyms = external_hdr_size; internal_f.f_symptr = ecoff_data (abfd)->sym_filepos; } else { internal_f.f_nsyms = 0; internal_f.f_symptr = 0; } internal_f.f_opthdr = aoutsz; internal_f.f_flags = F_LNNO; if (reloc_size == 0) internal_f.f_flags |= F_RELFLG; if (bfd_get_symcount (abfd) == 0) internal_f.f_flags |= F_LSYMS; if (abfd->flags & EXEC_P) internal_f.f_flags |= F_EXEC; if (bfd_little_endian (abfd)) internal_f.f_flags |= F_AR32WR; else internal_f.f_flags |= F_AR32W; /* Set up the ``optional'' header. */ if ((abfd->flags & D_PAGED) != 0) internal_a.magic = ECOFF_AOUT_ZMAGIC; else internal_a.magic = ECOFF_AOUT_OMAGIC; /* FIXME: Is this really correct? */ internal_a.vstamp = symhdr->vstamp; /* At least on Ultrix, these have to be rounded to page boundaries. FIXME: Is this true on other platforms? */ if ((abfd->flags & D_PAGED) != 0) { internal_a.tsize = (text_size + round - 1) &~ (round - 1); internal_a.text_start = text_start &~ (round - 1); internal_a.dsize = (data_size + round - 1) &~ (round - 1); internal_a.data_start = data_start &~ (round - 1); } else { internal_a.tsize = text_size; internal_a.text_start = text_start; internal_a.dsize = data_size; internal_a.data_start = data_start; } /* On Ultrix, the initial portions of the .sbss and .bss segments are at the end of the data section. The bsize field in the optional header records how many bss bytes are required beyond those in the data section. The value is not rounded to a page boundary. */ if (bss_size < internal_a.dsize - data_size) bss_size = 0; else bss_size -= internal_a.dsize - data_size; internal_a.bsize = bss_size; internal_a.bss_start = internal_a.data_start + internal_a.dsize; internal_a.entry = bfd_get_start_address (abfd); internal_a.gp_value = ecoff_data (abfd)->gp; internal_a.gprmask = ecoff_data (abfd)->gprmask; internal_a.fprmask = ecoff_data (abfd)->fprmask; for (i = 0; i < 4; i++) internal_a.cprmask[i] = ecoff_data (abfd)->cprmask[i]; /* Let the backend adjust the headers if necessary. */ if (backend->adjust_headers) { if (! (*backend->adjust_headers) (abfd, &internal_f, &internal_a)) goto error_return; } /* Write out the file header and the optional header. */ if (bfd_seek (abfd, (file_ptr) 0, SEEK_SET) != 0) goto error_return; bfd_coff_swap_filehdr_out (abfd, (void *) &internal_f, buff); if (bfd_bwrite (buff, filhsz, abfd) != filhsz) goto error_return; bfd_coff_swap_aouthdr_out (abfd, (void *) &internal_a, buff); if (bfd_bwrite (buff, aoutsz, abfd) != aoutsz) goto error_return; /* Build the external symbol information. This must be done before writing out the relocs so that we know the symbol indices. We don't do this if this BFD was created by the backend linker, since it will have already handled the symbols and relocs. */ if (! ecoff_data (abfd)->linker) { symhdr->iextMax = 0; symhdr->issExtMax = 0; debug->external_ext = debug->external_ext_end = NULL; debug->ssext = debug->ssext_end = NULL; if (! bfd_ecoff_debug_externals (abfd, debug, &backend->debug_swap, (abfd->flags & EXEC_P) == 0, ecoff_get_extr, ecoff_set_index)) goto error_return; /* Write out the relocs. */ for (current = abfd->sections; current != NULL; current = current->next) { arelent **reloc_ptr_ptr; arelent **reloc_end; char *out_ptr; bfd_size_type amt; if (current->reloc_count == 0) continue; amt = current->reloc_count * external_reloc_size; reloc_buff = bfd_alloc (abfd, amt); if (reloc_buff == NULL) goto error_return; reloc_ptr_ptr = current->orelocation; reloc_end = reloc_ptr_ptr + current->reloc_count; out_ptr = (char *) reloc_buff; for (; reloc_ptr_ptr < reloc_end; reloc_ptr_ptr++, out_ptr += external_reloc_size) { arelent *reloc; asymbol *sym; struct internal_reloc in; memset ((void *) &in, 0, sizeof in); reloc = *reloc_ptr_ptr; sym = *reloc->sym_ptr_ptr; /* If the howto field has not been initialised then skip this reloc. This assumes that an error message has been issued elsewhere. */ if (reloc->howto == NULL) continue; in.r_vaddr = (reloc->address + bfd_get_section_vma (abfd, current)); in.r_type = reloc->howto->type; if ((sym->flags & BSF_SECTION_SYM) == 0) { in.r_symndx = ecoff_get_sym_index (*reloc->sym_ptr_ptr); in.r_extern = 1; } else { const char *name; unsigned int i; static struct { const char * name; long r_symndx; } section_symndx [] = { { _TEXT, RELOC_SECTION_TEXT }, { _RDATA, RELOC_SECTION_RDATA }, { _DATA, RELOC_SECTION_DATA }, { _SDATA, RELOC_SECTION_SDATA }, { _SBSS, RELOC_SECTION_SBSS }, { _BSS, RELOC_SECTION_BSS }, { _INIT, RELOC_SECTION_INIT }, { _LIT8, RELOC_SECTION_LIT8 }, { _LIT4, RELOC_SECTION_LIT4 }, { _XDATA, RELOC_SECTION_XDATA }, { _PDATA, RELOC_SECTION_PDATA }, { _FINI, RELOC_SECTION_FINI }, { _LITA, RELOC_SECTION_LITA }, { "*ABS*", RELOC_SECTION_ABS }, { _RCONST, RELOC_SECTION_RCONST } }; name = bfd_get_section_name (abfd, bfd_get_section (sym)); for (i = 0; i < ARRAY_SIZE (section_symndx); i++) if (streq (name, section_symndx[i].name)) { in.r_symndx = section_symndx[i].r_symndx; break; } if (i == ARRAY_SIZE (section_symndx)) abort (); in.r_extern = 0; } (*adjust_reloc_out) (abfd, reloc, &in); (*swap_reloc_out) (abfd, &in, (void *) out_ptr); } if (bfd_seek (abfd, current->rel_filepos, SEEK_SET) != 0) goto error_return; amt = current->reloc_count * external_reloc_size; if (bfd_bwrite (reloc_buff, amt, abfd) != amt) goto error_return; bfd_release (abfd, reloc_buff); reloc_buff = NULL; } /* Write out the symbolic debugging information. */ if (bfd_get_symcount (abfd) > 0) { /* Write out the debugging information. */ if (! bfd_ecoff_write_debug (abfd, debug, &backend->debug_swap, ecoff_data (abfd)->sym_filepos)) goto error_return; } } /* The .bss section of a demand paged executable must receive an entire page. If there are symbols, the symbols will start on the next page. If there are no symbols, we must fill out the page by hand. */ if (bfd_get_symcount (abfd) == 0 && (abfd->flags & EXEC_P) != 0 && (abfd->flags & D_PAGED) != 0) { char c; if (bfd_seek (abfd, (file_ptr) ecoff_data (abfd)->sym_filepos - 1, SEEK_SET) != 0) goto error_return; if (bfd_bread (&c, (bfd_size_type) 1, abfd) == 0) c = 0; if (bfd_seek (abfd, (file_ptr) ecoff_data (abfd)->sym_filepos - 1, SEEK_SET) != 0) goto error_return; if (bfd_bwrite (&c, (bfd_size_type) 1, abfd) != 1) goto error_return; } if (reloc_buff != NULL) bfd_release (abfd, reloc_buff); if (buff != NULL) free (buff); return TRUE; error_return: if (reloc_buff != NULL) bfd_release (abfd, reloc_buff); if (buff != NULL) free (buff); return FALSE; } /* Archive handling. ECOFF uses what appears to be a unique type of archive header (armap). The byte ordering of the armap and the contents are encoded in the name of the armap itself. At least for now, we only support archives with the same byte ordering in the armap and the contents. The first four bytes in the armap are the number of symbol definitions. This is always a power of two. This is followed by the symbol definitions. Each symbol definition occupies 8 bytes. The first four bytes are the offset from the start of the armap strings to the null-terminated string naming this symbol. The second four bytes are the file offset to the archive member which defines this symbol. If the second four bytes are 0, then this is not actually a symbol definition, and it should be ignored. The symbols are hashed into the armap with a closed hashing scheme. See the functions below for the details of the algorithm. After the symbol definitions comes four bytes holding the size of the string table, followed by the string table itself. */ /* The name of an archive headers looks like this: __________E[BL]E[BL]_ (with a trailing space). The trailing space is changed to an X if the archive is changed to indicate that the armap is out of date. The Alpha seems to use ________64E[BL]E[BL]_. */ #define ARMAP_BIG_ENDIAN 'B' #define ARMAP_LITTLE_ENDIAN 'L' #define ARMAP_MARKER 'E' #define ARMAP_START_LENGTH 10 #define ARMAP_HEADER_MARKER_INDEX 10 #define ARMAP_HEADER_ENDIAN_INDEX 11 #define ARMAP_OBJECT_MARKER_INDEX 12 #define ARMAP_OBJECT_ENDIAN_INDEX 13 #define ARMAP_END_INDEX 14 #define ARMAP_END "_ " /* This is a magic number used in the hashing algorithm. */ #define ARMAP_HASH_MAGIC 0x9dd68ab5 /* This returns the hash value to use for a string. It also sets *REHASH to the rehash adjustment if the first slot is taken. SIZE is the number of entries in the hash table, and HLOG is the log base 2 of SIZE. */ static unsigned int ecoff_armap_hash (const char *s, unsigned int *rehash, unsigned int size, unsigned int hlog) { unsigned int hash; if (hlog == 0) return 0; hash = *s++; while (*s != '\0') hash = ((hash >> 27) | (hash << 5)) + *s++; hash *= ARMAP_HASH_MAGIC; *rehash = (hash & (size - 1)) | 1; return hash >> (32 - hlog); } /* Read in the armap. */ bfd_boolean _bfd_ecoff_slurp_armap (bfd *abfd) { char nextname[17]; unsigned int i; struct areltdata *mapdata; bfd_size_type parsed_size; char *raw_armap; struct artdata *ardata; unsigned int count; char *raw_ptr; struct symdef *symdef_ptr; char *stringbase; bfd_size_type amt; /* Get the name of the first element. */ i = bfd_bread ((void *) nextname, (bfd_size_type) 16, abfd); if (i == 0) return TRUE; if (i != 16) return FALSE; if (bfd_seek (abfd, (file_ptr) -16, SEEK_CUR) != 0) return FALSE; /* Irix 4.0.5F apparently can use either an ECOFF armap or a standard COFF armap. We could move the ECOFF armap stuff into bfd_slurp_armap, but that seems inappropriate since no other target uses this format. Instead, we check directly for a COFF armap. */ if (CONST_STRNEQ (nextname, "/ ")) return bfd_slurp_armap (abfd); /* See if the first element is an armap. */ if (! strneq (nextname, ecoff_backend (abfd)->armap_start, ARMAP_START_LENGTH) || nextname[ARMAP_HEADER_MARKER_INDEX] != ARMAP_MARKER || (nextname[ARMAP_HEADER_ENDIAN_INDEX] != ARMAP_BIG_ENDIAN && nextname[ARMAP_HEADER_ENDIAN_INDEX] != ARMAP_LITTLE_ENDIAN) || nextname[ARMAP_OBJECT_MARKER_INDEX] != ARMAP_MARKER || (nextname[ARMAP_OBJECT_ENDIAN_INDEX] != ARMAP_BIG_ENDIAN && nextname[ARMAP_OBJECT_ENDIAN_INDEX] != ARMAP_LITTLE_ENDIAN) || ! strneq (nextname + ARMAP_END_INDEX, ARMAP_END, sizeof ARMAP_END - 1)) { bfd_has_map (abfd) = FALSE; return TRUE; } /* Make sure we have the right byte ordering. */ if (((nextname[ARMAP_HEADER_ENDIAN_INDEX] == ARMAP_BIG_ENDIAN) ^ (bfd_header_big_endian (abfd))) || ((nextname[ARMAP_OBJECT_ENDIAN_INDEX] == ARMAP_BIG_ENDIAN) ^ (bfd_big_endian (abfd)))) { bfd_set_error (bfd_error_wrong_format); return FALSE; } /* Read in the armap. */ ardata = bfd_ardata (abfd); mapdata = (struct areltdata *) _bfd_read_ar_hdr (abfd); if (mapdata == NULL) return FALSE; parsed_size = mapdata->parsed_size; bfd_release (abfd, (void *) mapdata); raw_armap = bfd_alloc (abfd, parsed_size); if (raw_armap == NULL) return FALSE; if (bfd_bread ((void *) raw_armap, parsed_size, abfd) != parsed_size) { if (bfd_get_error () != bfd_error_system_call) bfd_set_error (bfd_error_malformed_archive); bfd_release (abfd, (void *) raw_armap); return FALSE; } ardata->tdata = (void *) raw_armap; count = H_GET_32 (abfd, raw_armap); ardata->symdef_count = 0; ardata->cache = NULL; /* This code used to overlay the symdefs over the raw archive data, but that doesn't work on a 64 bit host. */ stringbase = raw_armap + count * 8 + 8; #ifdef CHECK_ARMAP_HASH { unsigned int hlog; /* Double check that I have the hashing algorithm right by making sure that every symbol can be looked up successfully. */ hlog = 0; for (i = 1; i < count; i <<= 1) hlog++; BFD_ASSERT (i == count); raw_ptr = raw_armap + 4; for (i = 0; i < count; i++, raw_ptr += 8) { unsigned int name_offset, file_offset; unsigned int hash, rehash, srch; name_offset = H_GET_32 (abfd, raw_ptr); file_offset = H_GET_32 (abfd, (raw_ptr + 4)); if (file_offset == 0) continue; hash = ecoff_armap_hash (stringbase + name_offset, &rehash, count, hlog); if (hash == i) continue; /* See if we can rehash to this location. */ for (srch = (hash + rehash) & (count - 1); srch != hash && srch != i; srch = (srch + rehash) & (count - 1)) BFD_ASSERT (H_GET_32 (abfd, (raw_armap + 8 + srch * 8)) != 0); BFD_ASSERT (srch == i); } } #endif /* CHECK_ARMAP_HASH */ raw_ptr = raw_armap + 4; for (i = 0; i < count; i++, raw_ptr += 8) if (H_GET_32 (abfd, (raw_ptr + 4)) != 0) ++ardata->symdef_count; amt = ardata->symdef_count; amt *= sizeof (struct symdef); symdef_ptr = bfd_alloc (abfd, amt); if (!symdef_ptr) return FALSE; ardata->symdefs = (carsym *) symdef_ptr; raw_ptr = raw_armap + 4; for (i = 0; i < count; i++, raw_ptr += 8) { unsigned int name_offset, file_offset; file_offset = H_GET_32 (abfd, (raw_ptr + 4)); if (file_offset == 0) continue; name_offset = H_GET_32 (abfd, raw_ptr); symdef_ptr->s.name = stringbase + name_offset; symdef_ptr->file_offset = file_offset; ++symdef_ptr; } ardata->first_file_filepos = bfd_tell (abfd); /* Pad to an even boundary. */ ardata->first_file_filepos += ardata->first_file_filepos % 2; bfd_has_map (abfd) = TRUE; return TRUE; } /* Write out an armap. */ bfd_boolean _bfd_ecoff_write_armap (bfd *abfd, unsigned int elength, struct orl *map, unsigned int orl_count, int stridx) { unsigned int hashsize, hashlog; bfd_size_type symdefsize; int padit; unsigned int stringsize; unsigned int mapsize; file_ptr firstreal; struct ar_hdr hdr; struct stat statbuf; unsigned int i; bfd_byte temp[4]; bfd_byte *hashtable; bfd *current; bfd *last_elt; /* Ultrix appears to use as a hash table size the least power of two greater than twice the number of entries. */ for (hashlog = 0; ((unsigned int) 1 << hashlog) <= 2 * orl_count; hashlog++) ; hashsize = 1 << hashlog; symdefsize = hashsize * 8; padit = stridx % 2; stringsize = stridx + padit; /* Include 8 bytes to store symdefsize and stringsize in output. */ mapsize = symdefsize + stringsize + 8; firstreal = SARMAG + sizeof (struct ar_hdr) + mapsize + elength; memset ((void *) &hdr, 0, sizeof hdr); /* Work out the ECOFF armap name. */ strcpy (hdr.ar_name, ecoff_backend (abfd)->armap_start); hdr.ar_name[ARMAP_HEADER_MARKER_INDEX] = ARMAP_MARKER; hdr.ar_name[ARMAP_HEADER_ENDIAN_INDEX] = (bfd_header_big_endian (abfd) ? ARMAP_BIG_ENDIAN : ARMAP_LITTLE_ENDIAN); hdr.ar_name[ARMAP_OBJECT_MARKER_INDEX] = ARMAP_MARKER; hdr.ar_name[ARMAP_OBJECT_ENDIAN_INDEX] = bfd_big_endian (abfd) ? ARMAP_BIG_ENDIAN : ARMAP_LITTLE_ENDIAN; memcpy (hdr.ar_name + ARMAP_END_INDEX, ARMAP_END, sizeof ARMAP_END - 1); /* Write the timestamp of the archive header to be just a little bit later than the timestamp of the file, otherwise the linker will complain that the index is out of date. Actually, the Ultrix linker just checks the archive name; the GNU linker may check the date. */ stat (abfd->filename, &statbuf); sprintf (hdr.ar_date, "%ld", (long) (statbuf.st_mtime + 60)); /* The DECstation uses zeroes for the uid, gid and mode of the armap. */ hdr.ar_uid[0] = '0'; hdr.ar_gid[0] = '0'; /* Building gcc ends up extracting the armap as a file - twice. */ hdr.ar_mode[0] = '6'; hdr.ar_mode[1] = '4'; hdr.ar_mode[2] = '4'; sprintf (hdr.ar_size, "%-10d", (int) mapsize); hdr.ar_fmag[0] = '`'; hdr.ar_fmag[1] = '\012'; /* Turn all null bytes in the header into spaces. */ for (i = 0; i < sizeof (struct ar_hdr); i++) if (((char *) (&hdr))[i] == '\0') (((char *) (&hdr))[i]) = ' '; if (bfd_bwrite ((void *) &hdr, (bfd_size_type) sizeof (struct ar_hdr), abfd) != sizeof (struct ar_hdr)) return FALSE; H_PUT_32 (abfd, hashsize, temp); if (bfd_bwrite ((void *) temp, (bfd_size_type) 4, abfd) != 4) return FALSE; hashtable = bfd_zalloc (abfd, symdefsize); if (!hashtable) return FALSE; current = abfd->archive_head; last_elt = current; for (i = 0; i < orl_count; i++) { unsigned int hash, rehash = 0; /* Advance firstreal to the file position of this archive element. */ if (map[i].u.abfd != last_elt) { do { firstreal += arelt_size (current) + sizeof (struct ar_hdr); firstreal += firstreal % 2; current = current->archive_next; } while (current != map[i].u.abfd); } last_elt = current; hash = ecoff_armap_hash (*map[i].name, &rehash, hashsize, hashlog); if (H_GET_32 (abfd, (hashtable + (hash * 8) + 4)) != 0) { unsigned int srch; /* The desired slot is already taken. */ for (srch = (hash + rehash) & (hashsize - 1); srch != hash; srch = (srch + rehash) & (hashsize - 1)) if (H_GET_32 (abfd, (hashtable + (srch * 8) + 4)) == 0) break; BFD_ASSERT (srch != hash); hash = srch; } H_PUT_32 (abfd, map[i].namidx, (hashtable + hash * 8)); H_PUT_32 (abfd, firstreal, (hashtable + hash * 8 + 4)); } if (bfd_bwrite ((void *) hashtable, symdefsize, abfd) != symdefsize) return FALSE; bfd_release (abfd, hashtable); /* Now write the strings. */ H_PUT_32 (abfd, stringsize, temp); if (bfd_bwrite ((void *) temp, (bfd_size_type) 4, abfd) != 4) return FALSE; for (i = 0; i < orl_count; i++) { bfd_size_type len; len = strlen (*map[i].name) + 1; if (bfd_bwrite ((void *) (*map[i].name), len, abfd) != len) return FALSE; } /* The spec sez this should be a newline. But in order to be bug-compatible for DECstation ar we use a null. */ if (padit) { if (bfd_bwrite ("", (bfd_size_type) 1, abfd) != 1) return FALSE; } return TRUE; } /* See whether this BFD is an archive. If it is, read in the armap and the extended name table. */ const bfd_target * _bfd_ecoff_archive_p (bfd *abfd) { struct artdata *tdata_hold; char armag[SARMAG + 1]; bfd_size_type amt; if (bfd_bread ((void *) armag, (bfd_size_type) SARMAG, abfd) != SARMAG) { if (bfd_get_error () != bfd_error_system_call) bfd_set_error (bfd_error_wrong_format); return NULL; } if (! strneq (armag, ARMAG, SARMAG)) { bfd_set_error (bfd_error_wrong_format); return NULL; } tdata_hold = bfd_ardata (abfd); amt = sizeof (struct artdata); bfd_ardata (abfd) = bfd_zalloc (abfd, amt); if (bfd_ardata (abfd) == NULL) { bfd_ardata (abfd) = tdata_hold; return NULL; } bfd_ardata (abfd)->first_file_filepos = SARMAG; /* Already cleared by bfd_zalloc above. bfd_ardata (abfd)->cache = NULL; bfd_ardata (abfd)->archive_head = NULL; bfd_ardata (abfd)->symdefs = NULL; bfd_ardata (abfd)->extended_names = NULL; bfd_ardata (abfd)->extended_names_size = 0; bfd_ardata (abfd)->tdata = NULL; */ if (! _bfd_ecoff_slurp_armap (abfd) || ! _bfd_ecoff_slurp_extended_name_table (abfd)) { bfd_release (abfd, bfd_ardata (abfd)); bfd_ardata (abfd) = tdata_hold; return NULL; } if (bfd_has_map (abfd)) { bfd *first; /* This archive has a map, so we may presume that the contents are object files. Make sure that if the first file in the archive can be recognized as an object file, it is for this target. If not, assume that this is the wrong format. If the first file is not an object file, somebody is doing something weird, and we permit it so that ar -t will work. */ first = bfd_openr_next_archived_file (abfd, NULL); if (first != NULL) { first->target_defaulted = FALSE; if (bfd_check_format (first, bfd_object) && first->xvec != abfd->xvec) { /* We ought to close `first' here, but we can't, because we have no way to remove it from the archive cache. It's almost impossible to figure out when we can release bfd_ardata. FIXME. */ bfd_set_error (bfd_error_wrong_object_format); bfd_ardata (abfd) = tdata_hold; return NULL; } /* And we ought to close `first' here too. */ } } return abfd->xvec; } /* ECOFF linker code. */ /* Routine to create an entry in an ECOFF link hash table. */ static struct bfd_hash_entry * ecoff_link_hash_newfunc (struct bfd_hash_entry *entry, struct bfd_hash_table *table, const char *string) { struct ecoff_link_hash_entry *ret = (struct ecoff_link_hash_entry *) entry; /* Allocate the structure if it has not already been allocated by a subclass. */ if (ret == NULL) ret = ((struct ecoff_link_hash_entry *) bfd_hash_allocate (table, sizeof (struct ecoff_link_hash_entry))); if (ret == NULL) return NULL; /* Call the allocation method of the superclass. */ ret = ((struct ecoff_link_hash_entry *) _bfd_link_hash_newfunc ((struct bfd_hash_entry *) ret, table, string)); if (ret) { /* Set local fields. */ ret->indx = -1; ret->abfd = NULL; ret->written = 0; ret->small = 0; } memset ((void *) &ret->esym, 0, sizeof ret->esym); return (struct bfd_hash_entry *) ret; } /* Create an ECOFF link hash table. */ struct bfd_link_hash_table * _bfd_ecoff_bfd_link_hash_table_create (bfd *abfd) { struct ecoff_link_hash_table *ret; bfd_size_type amt = sizeof (struct ecoff_link_hash_table); ret = bfd_malloc (amt); if (ret == NULL) return NULL; if (!_bfd_link_hash_table_init (&ret->root, abfd, ecoff_link_hash_newfunc, sizeof (struct ecoff_link_hash_entry))) { free (ret); return NULL; } return &ret->root; } /* Look up an entry in an ECOFF link hash table. */ #define ecoff_link_hash_lookup(table, string, create, copy, follow) \ ((struct ecoff_link_hash_entry *) \ bfd_link_hash_lookup (&(table)->root, (string), (create), (copy), (follow))) /* Traverse an ECOFF link hash table. */ #define ecoff_link_hash_traverse(table, func, info) \ (bfd_link_hash_traverse \ (&(table)->root, \ (bfd_boolean (*) (struct bfd_link_hash_entry *, void *)) (func), \ (info))) /* Get the ECOFF link hash table from the info structure. This is just a cast. */ #define ecoff_hash_table(p) ((struct ecoff_link_hash_table *) ((p)->hash)) /* Add the external symbols of an object file to the global linker hash table. The external symbols and strings we are passed are just allocated on the stack, and will be discarded. We must explicitly save any information we may need later on in the link. We do not want to read the external symbol information again. */ static bfd_boolean ecoff_link_add_externals (bfd *abfd, struct bfd_link_info *info, void * external_ext, char *ssext) { const struct ecoff_backend_data * const backend = ecoff_backend (abfd); void (* const swap_ext_in) (bfd *, void *, EXTR *) = backend->debug_swap.swap_ext_in; bfd_size_type external_ext_size = backend->debug_swap.external_ext_size; unsigned long ext_count; struct bfd_link_hash_entry **sym_hash; char *ext_ptr; char *ext_end; bfd_size_type amt; ext_count = ecoff_data (abfd)->debug_info.symbolic_header.iextMax; amt = ext_count; amt *= sizeof (struct bfd_link_hash_entry *); sym_hash = bfd_alloc (abfd, amt); if (!sym_hash) return FALSE; ecoff_data (abfd)->sym_hashes = (struct ecoff_link_hash_entry **) sym_hash; ext_ptr = (char *) external_ext; ext_end = ext_ptr + ext_count * external_ext_size; for (; ext_ptr < ext_end; ext_ptr += external_ext_size, sym_hash++) { EXTR esym; bfd_boolean skip; bfd_vma value; asection *section; const char *name; struct ecoff_link_hash_entry *h; *sym_hash = NULL; (*swap_ext_in) (abfd, (void *) ext_ptr, &esym); /* Skip debugging symbols. */ skip = FALSE; switch (esym.asym.st) { case stGlobal: case stStatic: case stLabel: case stProc: case stStaticProc: break; default: skip = TRUE; break; } if (skip) continue; /* Get the information for this symbol. */ value = esym.asym.value; switch (esym.asym.sc) { default: case scNil: case scRegister: case scCdbLocal: case scBits: case scCdbSystem: case scRegImage: case scInfo: case scUserStruct: case scVar: case scVarRegister: case scVariant: case scBasedVar: case scXData: case scPData: section = NULL; break; case scText: section = bfd_make_section_old_way (abfd, _TEXT); value -= section->vma; break; case scData: section = bfd_make_section_old_way (abfd, _DATA); value -= section->vma; break; case scBss: section = bfd_make_section_old_way (abfd, _BSS); value -= section->vma; break; case scAbs: section = bfd_abs_section_ptr; break; case scUndefined: section = bfd_und_section_ptr; break; case scSData: section = bfd_make_section_old_way (abfd, _SDATA); value -= section->vma; break; case scSBss: section = bfd_make_section_old_way (abfd, _SBSS); value -= section->vma; break; case scRData: section = bfd_make_section_old_way (abfd, _RDATA); value -= section->vma; break; case scCommon: if (value > ecoff_data (abfd)->gp_size) { section = bfd_com_section_ptr; break; } /* Fall through. */ case scSCommon: if (ecoff_scom_section.name == NULL) { /* Initialize the small common section. */ ecoff_scom_section.name = SCOMMON; ecoff_scom_section.flags = SEC_IS_COMMON; ecoff_scom_section.output_section = &ecoff_scom_section; ecoff_scom_section.symbol = &ecoff_scom_symbol; ecoff_scom_section.symbol_ptr_ptr = &ecoff_scom_symbol_ptr; ecoff_scom_symbol.name = SCOMMON; ecoff_scom_symbol.flags = BSF_SECTION_SYM; ecoff_scom_symbol.section = &ecoff_scom_section; ecoff_scom_symbol_ptr = &ecoff_scom_symbol; } section = &ecoff_scom_section; break; case scSUndefined: section = bfd_und_section_ptr; break; case scInit: section = bfd_make_section_old_way (abfd, _INIT); value -= section->vma; break; case scFini: section = bfd_make_section_old_way (abfd, _FINI); value -= section->vma; break; case scRConst: section = bfd_make_section_old_way (abfd, _RCONST); value -= section->vma; break; } if (section == NULL) continue; name = ssext + esym.asym.iss; if (! (_bfd_generic_link_add_one_symbol (info, abfd, name, (flagword) (esym.weakext ? BSF_WEAK : BSF_GLOBAL), section, value, NULL, TRUE, TRUE, sym_hash))) return FALSE; h = (struct ecoff_link_hash_entry *) *sym_hash; /* If we are building an ECOFF hash table, save the external symbol information. */ if (info->hash->creator->flavour == bfd_get_flavour (abfd)) { if (h->abfd == NULL || (! bfd_is_und_section (section) && (! bfd_is_com_section (section) || (h->root.type != bfd_link_hash_defined && h->root.type != bfd_link_hash_defweak)))) { h->abfd = abfd; h->esym = esym; } /* Remember whether this symbol was small undefined. */ if (esym.asym.sc == scSUndefined) h->small = 1; /* If this symbol was ever small undefined, it needs to wind up in a GP relative section. We can't control the section of a defined symbol, but we can control the section of a common symbol. This case is actually needed on Ultrix 4.2 to handle the symbol cred in -lckrb. */ if (h->small && h->root.type == bfd_link_hash_common && streq (h->root.u.c.p->section->name, SCOMMON)) { h->root.u.c.p->section = bfd_make_section_old_way (abfd, SCOMMON); h->root.u.c.p->section->flags = SEC_ALLOC; if (h->esym.asym.sc == scCommon) h->esym.asym.sc = scSCommon; } } } return TRUE; } /* Add symbols from an ECOFF object file to the global linker hash table. */ static bfd_boolean ecoff_link_add_object_symbols (bfd *abfd, struct bfd_link_info *info) { HDRR *symhdr; bfd_size_type external_ext_size; void * external_ext = NULL; bfd_size_type esize; char *ssext = NULL; bfd_boolean result; if (! ecoff_slurp_symbolic_header (abfd)) return FALSE; /* If there are no symbols, we don't want it. */ if (bfd_get_symcount (abfd) == 0) return TRUE; symhdr = &ecoff_data (abfd)->debug_info.symbolic_header; /* Read in the external symbols and external strings. */ external_ext_size = ecoff_backend (abfd)->debug_swap.external_ext_size; esize = symhdr->iextMax * external_ext_size; external_ext = bfd_malloc (esize); if (external_ext == NULL && esize != 0) goto error_return; if (bfd_seek (abfd, (file_ptr) symhdr->cbExtOffset, SEEK_SET) != 0 || bfd_bread (external_ext, esize, abfd) != esize) goto error_return; ssext = bfd_malloc ((bfd_size_type) symhdr->issExtMax); if (ssext == NULL && symhdr->issExtMax != 0) goto error_return; if (bfd_seek (abfd, (file_ptr) symhdr->cbSsExtOffset, SEEK_SET) != 0 || (bfd_bread (ssext, (bfd_size_type) symhdr->issExtMax, abfd) != (bfd_size_type) symhdr->issExtMax)) goto error_return; result = ecoff_link_add_externals (abfd, info, external_ext, ssext); if (ssext != NULL) free (ssext); if (external_ext != NULL) free (external_ext); return result; error_return: if (ssext != NULL) free (ssext); if (external_ext != NULL) free (external_ext); return FALSE; } /* This is called if we used _bfd_generic_link_add_archive_symbols because we were not dealing with an ECOFF archive. */ static bfd_boolean ecoff_link_check_archive_element (bfd *abfd, struct bfd_link_info *info, bfd_boolean *pneeded) { const struct ecoff_backend_data * const backend = ecoff_backend (abfd); void (* const swap_ext_in) (bfd *, void *, EXTR *) = backend->debug_swap.swap_ext_in; HDRR *symhdr; bfd_size_type external_ext_size; void * external_ext = NULL; bfd_size_type esize; char *ssext = NULL; char *ext_ptr; char *ext_end; *pneeded = FALSE; if (! ecoff_slurp_symbolic_header (abfd)) goto error_return; /* If there are no symbols, we don't want it. */ if (bfd_get_symcount (abfd) == 0) goto successful_return; symhdr = &ecoff_data (abfd)->debug_info.symbolic_header; /* Read in the external symbols and external strings. */ external_ext_size = backend->debug_swap.external_ext_size; esize = symhdr->iextMax * external_ext_size; external_ext = bfd_malloc (esize); if (external_ext == NULL && esize != 0) goto error_return; if (bfd_seek (abfd, (file_ptr) symhdr->cbExtOffset, SEEK_SET) != 0 || bfd_bread (external_ext, esize, abfd) != esize) goto error_return; ssext = bfd_malloc ((bfd_size_type) symhdr->issExtMax); if (ssext == NULL && symhdr->issExtMax != 0) goto error_return; if (bfd_seek (abfd, (file_ptr) symhdr->cbSsExtOffset, SEEK_SET) != 0 || (bfd_bread (ssext, (bfd_size_type) symhdr->issExtMax, abfd) != (bfd_size_type) symhdr->issExtMax)) goto error_return; /* Look through the external symbols to see if they define some symbol that is currently undefined. */ ext_ptr = (char *) external_ext; ext_end = ext_ptr + esize; for (; ext_ptr < ext_end; ext_ptr += external_ext_size) { EXTR esym; bfd_boolean def; const char *name; struct bfd_link_hash_entry *h; (*swap_ext_in) (abfd, (void *) ext_ptr, &esym); /* See if this symbol defines something. */ if (esym.asym.st != stGlobal && esym.asym.st != stLabel && esym.asym.st != stProc) continue; switch (esym.asym.sc) { case scText: case scData: case scBss: case scAbs: case scSData: case scSBss: case scRData: case scCommon: case scSCommon: case scInit: case scFini: case scRConst: def = TRUE; break; default: def = FALSE; break; } if (! def) continue; name = ssext + esym.asym.iss; h = bfd_link_hash_lookup (info->hash, name, FALSE, FALSE, TRUE); /* Unlike the generic linker, we do not pull in elements because of common symbols. */ if (h == NULL || h->type != bfd_link_hash_undefined) continue; /* Include this element. */ if (! (*info->callbacks->add_archive_element) (info, abfd, name)) goto error_return; if (! ecoff_link_add_externals (abfd, info, external_ext, ssext)) goto error_return; *pneeded = TRUE; goto successful_return; } successful_return: if (external_ext != NULL) free (external_ext); if (ssext != NULL) free (ssext); return TRUE; error_return: if (external_ext != NULL) free (external_ext); if (ssext != NULL) free (ssext); return FALSE; } /* Add the symbols from an archive file to the global hash table. This looks through the undefined symbols, looks each one up in the archive hash table, and adds any associated object file. We do not use _bfd_generic_link_add_archive_symbols because ECOFF archives already have a hash table, so there is no reason to construct another one. */ static bfd_boolean ecoff_link_add_archive_symbols (bfd *abfd, struct bfd_link_info *info) { const struct ecoff_backend_data * const backend = ecoff_backend (abfd); const bfd_byte *raw_armap; struct bfd_link_hash_entry **pundef; unsigned int armap_count; unsigned int armap_log; unsigned int i; const bfd_byte *hashtable; const char *stringbase; if (! bfd_has_map (abfd)) { /* An empty archive is a special case. */ if (bfd_openr_next_archived_file (abfd, NULL) == NULL) return TRUE; bfd_set_error (bfd_error_no_armap); return FALSE; } /* If we don't have any raw data for this archive, as can happen on Irix 4.0.5F, we call the generic routine. FIXME: We should be more clever about this, since someday tdata may get to something for a generic archive. */ raw_armap = (const bfd_byte *) bfd_ardata (abfd)->tdata; if (raw_armap == NULL) return (_bfd_generic_link_add_archive_symbols (abfd, info, ecoff_link_check_archive_element)); armap_count = H_GET_32 (abfd, raw_armap); armap_log = 0; for (i = 1; i < armap_count; i <<= 1) armap_log++; BFD_ASSERT (i == armap_count); hashtable = raw_armap + 4; stringbase = (const char *) raw_armap + armap_count * 8 + 8; /* Look through the list of undefined symbols. */ pundef = &info->hash->undefs; while (*pundef != NULL) { struct bfd_link_hash_entry *h; unsigned int hash, rehash = 0; unsigned int file_offset; const char *name; bfd *element; h = *pundef; /* When a symbol is defined, it is not necessarily removed from the list. */ if (h->type != bfd_link_hash_undefined && h->type != bfd_link_hash_common) { /* Remove this entry from the list, for general cleanliness and because we are going to look through the list again if we search any more libraries. We can't remove the entry if it is the tail, because that would lose any entries we add to the list later on. */ if (*pundef != info->hash->undefs_tail) *pundef = (*pundef)->u.undef.next; else pundef = &(*pundef)->u.undef.next; continue; } /* Native ECOFF linkers do not pull in archive elements merely to satisfy common definitions, so neither do we. We leave them on the list, though, in case we are linking against some other object format. */ if (h->type != bfd_link_hash_undefined) { pundef = &(*pundef)->u.undef.next; continue; } /* Look for this symbol in the archive hash table. */ hash = ecoff_armap_hash (h->root.string, &rehash, armap_count, armap_log); file_offset = H_GET_32 (abfd, hashtable + (hash * 8) + 4); if (file_offset == 0) { /* Nothing in this slot. */ pundef = &(*pundef)->u.undef.next; continue; } name = stringbase + H_GET_32 (abfd, hashtable + (hash * 8)); if (name[0] != h->root.string[0] || ! streq (name, h->root.string)) { unsigned int srch; bfd_boolean found; /* That was the wrong symbol. Try rehashing. */ found = FALSE; for (srch = (hash + rehash) & (armap_count - 1); srch != hash; srch = (srch + rehash) & (armap_count - 1)) { file_offset = H_GET_32 (abfd, hashtable + (srch * 8) + 4); if (file_offset == 0) break; name = stringbase + H_GET_32 (abfd, hashtable + (srch * 8)); if (name[0] == h->root.string[0] && streq (name, h->root.string)) { found = TRUE; break; } } if (! found) { pundef = &(*pundef)->u.undef.next; continue; } hash = srch; } element = (*backend->get_elt_at_filepos) (abfd, (file_ptr) file_offset); if (element == NULL) return FALSE; if (! bfd_check_format (element, bfd_object)) return FALSE; /* Unlike the generic linker, we know that this element provides a definition for an undefined symbol and we know that we want to include it. We don't need to check anything. */ if (! (*info->callbacks->add_archive_element) (info, element, name)) return FALSE; if (! ecoff_link_add_object_symbols (element, info)) return FALSE; pundef = &(*pundef)->u.undef.next; } return TRUE; } /* Given an ECOFF BFD, add symbols to the global hash table as appropriate. */ bfd_boolean _bfd_ecoff_bfd_link_add_symbols (bfd *abfd, struct bfd_link_info *info) { switch (bfd_get_format (abfd)) { case bfd_object: return ecoff_link_add_object_symbols (abfd, info); case bfd_archive: return ecoff_link_add_archive_symbols (abfd, info); default: bfd_set_error (bfd_error_wrong_format); return FALSE; } } /* ECOFF final link routines. */ /* Structure used to pass information to ecoff_link_write_external. */ struct extsym_info { bfd *abfd; struct bfd_link_info *info; }; /* Accumulate the debugging information for an input BFD into the output BFD. This must read in the symbolic information of the input BFD. */ static bfd_boolean ecoff_final_link_debug_accumulate (bfd *output_bfd, bfd *input_bfd, struct bfd_link_info *info, void * handle) { struct ecoff_debug_info * const debug = &ecoff_data (input_bfd)->debug_info; const struct ecoff_debug_swap * const swap = &ecoff_backend (input_bfd)->debug_swap; HDRR *symhdr = &debug->symbolic_header; bfd_boolean ret; #define READ(ptr, offset, count, size, type) \ if (symhdr->count == 0) \ debug->ptr = NULL; \ else \ { \ bfd_size_type amt = (bfd_size_type) size * symhdr->count; \ debug->ptr = bfd_malloc (amt); \ if (debug->ptr == NULL) \ { \ ret = FALSE; \ goto return_something; \ } \ if (bfd_seek (input_bfd, (file_ptr) symhdr->offset, SEEK_SET) != 0 \ || bfd_bread (debug->ptr, amt, input_bfd) != amt) \ { \ ret = FALSE; \ goto return_something; \ } \ } /* If raw_syments is not NULL, then the data was already by read by _bfd_ecoff_slurp_symbolic_info. */ if (ecoff_data (input_bfd)->raw_syments == NULL) { READ (line, cbLineOffset, cbLine, sizeof (unsigned char), unsigned char *); READ (external_dnr, cbDnOffset, idnMax, swap->external_dnr_size, void *); READ (external_pdr, cbPdOffset, ipdMax, swap->external_pdr_size, void *); READ (external_sym, cbSymOffset, isymMax, swap->external_sym_size, void *); READ (external_opt, cbOptOffset, ioptMax, swap->external_opt_size, void *); READ (external_aux, cbAuxOffset, iauxMax, sizeof (union aux_ext), union aux_ext *); READ (ss, cbSsOffset, issMax, sizeof (char), char *); READ (external_fdr, cbFdOffset, ifdMax, swap->external_fdr_size, void *); READ (external_rfd, cbRfdOffset, crfd, swap->external_rfd_size, void *); } #undef READ /* We do not read the external strings or the external symbols. */ ret = (bfd_ecoff_debug_accumulate (handle, output_bfd, &ecoff_data (output_bfd)->debug_info, &ecoff_backend (output_bfd)->debug_swap, input_bfd, debug, swap, info)); return_something: if (ecoff_data (input_bfd)->raw_syments == NULL) { if (debug->line != NULL) free (debug->line); if (debug->external_dnr != NULL) free (debug->external_dnr); if (debug->external_pdr != NULL) free (debug->external_pdr); if (debug->external_sym != NULL) free (debug->external_sym); if (debug->external_opt != NULL) free (debug->external_opt); if (debug->external_aux != NULL) free (debug->external_aux); if (debug->ss != NULL) free (debug->ss); if (debug->external_fdr != NULL) free (debug->external_fdr); if (debug->external_rfd != NULL) free (debug->external_rfd); /* Make sure we don't accidentally follow one of these pointers into freed memory. */ debug->line = NULL; debug->external_dnr = NULL; debug->external_pdr = NULL; debug->external_sym = NULL; debug->external_opt = NULL; debug->external_aux = NULL; debug->ss = NULL; debug->external_fdr = NULL; debug->external_rfd = NULL; } return ret; } /* Relocate and write an ECOFF section into an ECOFF output file. */ static bfd_boolean ecoff_indirect_link_order (bfd *output_bfd, struct bfd_link_info *info, asection *output_section, struct bfd_link_order *link_order) { asection *input_section; bfd *input_bfd; bfd_byte *contents = NULL; bfd_size_type external_reloc_size; bfd_size_type external_relocs_size; void * external_relocs = NULL; BFD_ASSERT ((output_section->flags & SEC_HAS_CONTENTS) != 0); input_section = link_order->u.indirect.section; input_bfd = input_section->owner; if (input_section->size == 0) return TRUE; BFD_ASSERT (input_section->output_section == output_section); BFD_ASSERT (input_section->output_offset == link_order->offset); BFD_ASSERT (input_section->size == link_order->size); /* Get the section contents. */ if (!bfd_malloc_and_get_section (input_bfd, input_section, &contents)) goto error_return; /* Get the relocs. If we are relaxing MIPS code, they will already have been read in. Otherwise, we read them in now. */ external_reloc_size = ecoff_backend (input_bfd)->external_reloc_size; external_relocs_size = external_reloc_size * input_section->reloc_count; external_relocs = bfd_malloc (external_relocs_size); if (external_relocs == NULL && external_relocs_size != 0) goto error_return; if (bfd_seek (input_bfd, input_section->rel_filepos, SEEK_SET) != 0 || (bfd_bread (external_relocs, external_relocs_size, input_bfd) != external_relocs_size)) goto error_return; /* Relocate the section contents. */ if (! ((*ecoff_backend (input_bfd)->relocate_section) (output_bfd, info, input_bfd, input_section, contents, external_relocs))) goto error_return; /* Write out the relocated section. */ if (! bfd_set_section_contents (output_bfd, output_section, contents, input_section->output_offset, input_section->size)) goto error_return; /* If we are producing relocatable output, the relocs were modified, and we write them out now. We use the reloc_count field of output_section to keep track of the number of relocs we have output so far. */ if (info->relocatable) { file_ptr pos = (output_section->rel_filepos + output_section->reloc_count * external_reloc_size); if (bfd_seek (output_bfd, pos, SEEK_SET) != 0 || (bfd_bwrite (external_relocs, external_relocs_size, output_bfd) != external_relocs_size)) goto error_return; output_section->reloc_count += input_section->reloc_count; } if (contents != NULL) free (contents); if (external_relocs != NULL) free (external_relocs); return TRUE; error_return: if (contents != NULL) free (contents); if (external_relocs != NULL) free (external_relocs); return FALSE; } /* Generate a reloc when linking an ECOFF file. This is a reloc requested by the linker, and does come from any input file. This is used to build constructor and destructor tables when linking with -Ur. */ static bfd_boolean ecoff_reloc_link_order (bfd *output_bfd, struct bfd_link_info *info, asection *output_section, struct bfd_link_order *link_order) { enum bfd_link_order_type type; asection *section; bfd_vma addend; arelent rel; struct internal_reloc in; bfd_size_type external_reloc_size; bfd_byte *rbuf; bfd_boolean ok; file_ptr pos; type = link_order->type; section = NULL; addend = link_order->u.reloc.p->addend; /* We set up an arelent to pass to the backend adjust_reloc_out routine. */ rel.address = link_order->offset; rel.howto = bfd_reloc_type_lookup (output_bfd, link_order->u.reloc.p->reloc); if (rel.howto == 0) { bfd_set_error (bfd_error_bad_value); return FALSE; } if (type == bfd_section_reloc_link_order) { section = link_order->u.reloc.p->u.section; rel.sym_ptr_ptr = section->symbol_ptr_ptr; } else { struct bfd_link_hash_entry *h; /* Treat a reloc against a defined symbol as though it were actually against the section. */ h = bfd_wrapped_link_hash_lookup (output_bfd, info, link_order->u.reloc.p->u.name, FALSE, FALSE, FALSE); if (h != NULL && (h->type == bfd_link_hash_defined || h->type == bfd_link_hash_defweak)) { type = bfd_section_reloc_link_order; section = h->u.def.section->output_section; /* It seems that we ought to add the symbol value to the addend here, but in practice it has already been added because it was passed to constructor_callback. */ addend += section->vma + h->u.def.section->output_offset; } else { /* We can't set up a reloc against a symbol correctly, because we have no asymbol structure. Currently no adjust_reloc_out routine cares. */ rel.sym_ptr_ptr = NULL; } } /* All ECOFF relocs are in-place. Put the addend into the object file. */ BFD_ASSERT (rel.howto->partial_inplace); if (addend != 0) { bfd_size_type size; bfd_reloc_status_type rstat; bfd_byte *buf; size = bfd_get_reloc_size (rel.howto); buf = bfd_zmalloc (size); if (buf == NULL) return FALSE; rstat = _bfd_relocate_contents (rel.howto, output_bfd, (bfd_vma) addend, buf); switch (rstat) { case bfd_reloc_ok: break; default: case bfd_reloc_outofrange: abort (); case bfd_reloc_overflow: if (! ((*info->callbacks->reloc_overflow) (info, NULL, (link_order->type == bfd_section_reloc_link_order ? bfd_section_name (output_bfd, section) : link_order->u.reloc.p->u.name), rel.howto->name, addend, NULL, NULL, (bfd_vma) 0))) { free (buf); return FALSE; } break; } ok = bfd_set_section_contents (output_bfd, output_section, (void *) buf, (file_ptr) link_order->offset, size); free (buf); if (! ok) return FALSE; } rel.addend = 0; /* Move the information into an internal_reloc structure. */ in.r_vaddr = (rel.address + bfd_get_section_vma (output_bfd, output_section)); in.r_type = rel.howto->type; if (type == bfd_symbol_reloc_link_order) { struct ecoff_link_hash_entry *h; h = ((struct ecoff_link_hash_entry *) bfd_wrapped_link_hash_lookup (output_bfd, info, link_order->u.reloc.p->u.name, FALSE, FALSE, TRUE)); if (h != NULL && h->indx != -1) in.r_symndx = h->indx; else { if (! ((*info->callbacks->unattached_reloc) (info, link_order->u.reloc.p->u.name, NULL, NULL, (bfd_vma) 0))) return FALSE; in.r_symndx = 0; } in.r_extern = 1; } else { const char *name; unsigned int i; static struct { const char * name; long r_symndx; } section_symndx [] = { { _TEXT, RELOC_SECTION_TEXT }, { _RDATA, RELOC_SECTION_RDATA }, { _DATA, RELOC_SECTION_DATA }, { _SDATA, RELOC_SECTION_SDATA }, { _SBSS, RELOC_SECTION_SBSS }, { _BSS, RELOC_SECTION_BSS }, { _INIT, RELOC_SECTION_INIT }, { _LIT8, RELOC_SECTION_LIT8 }, { _LIT4, RELOC_SECTION_LIT4 }, { _XDATA, RELOC_SECTION_XDATA }, { _PDATA, RELOC_SECTION_PDATA }, { _FINI, RELOC_SECTION_FINI }, { _LITA, RELOC_SECTION_LITA }, { "*ABS*", RELOC_SECTION_ABS }, { _RCONST, RELOC_SECTION_RCONST } }; name = bfd_get_section_name (output_bfd, section); for (i = 0; i < ARRAY_SIZE (section_symndx); i++) if (streq (name, section_symndx[i].name)) { in.r_symndx = section_symndx[i].r_symndx; break; } if (i == ARRAY_SIZE (section_symndx)) abort (); in.r_extern = 0; } /* Let the BFD backend adjust the reloc. */ (*ecoff_backend (output_bfd)->adjust_reloc_out) (output_bfd, &rel, &in); /* Get some memory and swap out the reloc. */ external_reloc_size = ecoff_backend (output_bfd)->external_reloc_size; rbuf = bfd_malloc (external_reloc_size); if (rbuf == NULL) return FALSE; (*ecoff_backend (output_bfd)->swap_reloc_out) (output_bfd, &in, (void *) rbuf); pos = (output_section->rel_filepos + output_section->reloc_count * external_reloc_size); ok = (bfd_seek (output_bfd, pos, SEEK_SET) == 0 && (bfd_bwrite ((void *) rbuf, external_reloc_size, output_bfd) == external_reloc_size)); if (ok) ++output_section->reloc_count; free (rbuf); return ok; } /* Put out information for an external symbol. These come only from the hash table. */ static bfd_boolean ecoff_link_write_external (struct ecoff_link_hash_entry *h, void * data) { struct extsym_info *einfo = (struct extsym_info *) data; bfd *output_bfd = einfo->abfd; bfd_boolean strip; if (h->root.type == bfd_link_hash_warning) { h = (struct ecoff_link_hash_entry *) h->root.u.i.link; if (h->root.type == bfd_link_hash_new) return TRUE; } /* We need to check if this symbol is being stripped. */ if (h->root.type == bfd_link_hash_undefined || h->root.type == bfd_link_hash_undefweak) strip = FALSE; else if (einfo->info->strip == strip_all || (einfo->info->strip == strip_some && bfd_hash_lookup (einfo->info->keep_hash, h->root.root.string, FALSE, FALSE) == NULL)) strip = TRUE; else strip = FALSE; if (strip || h->written) return TRUE; if (h->abfd == NULL) { h->esym.jmptbl = 0; h->esym.cobol_main = 0; h->esym.weakext = 0; h->esym.reserved = 0; h->esym.ifd = ifdNil; h->esym.asym.value = 0; h->esym.asym.st = stGlobal; if (h->root.type != bfd_link_hash_defined && h->root.type != bfd_link_hash_defweak) h->esym.asym.sc = scAbs; else { asection *output_section; const char *name; unsigned int i; static struct { const char * name; int sc; } section_storage_classes [] = { { _TEXT, scText }, { _DATA, scData }, { _SDATA, scSData }, { _RDATA, scRData }, { _BSS, scBss }, { _SBSS, scSBss }, { _INIT, scInit }, { _FINI, scFini }, { _PDATA, scPData }, { _XDATA, scXData }, { _RCONST, scRConst } }; output_section = h->root.u.def.section->output_section; name = bfd_section_name (output_section->owner, output_section); for (i = 0; i < ARRAY_SIZE (section_storage_classes); i++) if (streq (name, section_storage_classes[i].name)) { h->esym.asym.sc = section_storage_classes[i].sc; break; } if (i == ARRAY_SIZE (section_storage_classes)) h->esym.asym.sc = scAbs; } h->esym.asym.reserved = 0; h->esym.asym.index = indexNil; } else if (h->esym.ifd != -1) { struct ecoff_debug_info *debug; /* Adjust the FDR index for the symbol by that used for the input BFD. */ debug = &ecoff_data (h->abfd)->debug_info; BFD_ASSERT (h->esym.ifd >= 0 && h->esym.ifd < debug->symbolic_header.ifdMax); h->esym.ifd = debug->ifdmap[h->esym.ifd]; } switch (h->root.type) { default: case bfd_link_hash_warning: case bfd_link_hash_new: abort (); case bfd_link_hash_undefined: case bfd_link_hash_undefweak: if (h->esym.asym.sc != scUndefined && h->esym.asym.sc != scSUndefined) h->esym.asym.sc = scUndefined; break; case bfd_link_hash_defined: case bfd_link_hash_defweak: if (h->esym.asym.sc == scUndefined || h->esym.asym.sc == scSUndefined) h->esym.asym.sc = scAbs; else if (h->esym.asym.sc == scCommon) h->esym.asym.sc = scBss; else if (h->esym.asym.sc == scSCommon) h->esym.asym.sc = scSBss; h->esym.asym.value = (h->root.u.def.value + h->root.u.def.section->output_section->vma + h->root.u.def.section->output_offset); break; case bfd_link_hash_common: if (h->esym.asym.sc != scCommon && h->esym.asym.sc != scSCommon) h->esym.asym.sc = scCommon; h->esym.asym.value = h->root.u.c.size; break; case bfd_link_hash_indirect: /* We ignore these symbols, since the indirected symbol is already in the hash table. */ return TRUE; } /* bfd_ecoff_debug_one_external uses iextMax to keep track of the symbol number. */ h->indx = ecoff_data (output_bfd)->debug_info.symbolic_header.iextMax; h->written = 1; return (bfd_ecoff_debug_one_external (output_bfd, &ecoff_data (output_bfd)->debug_info, &ecoff_backend (output_bfd)->debug_swap, h->root.root.string, &h->esym)); } /* ECOFF final link routine. This looks through all the input BFDs and gathers together all the debugging information, and then processes all the link order information. This may cause it to close and reopen some input BFDs; I'll see how bad this is. */ bfd_boolean _bfd_ecoff_bfd_final_link (bfd *abfd, struct bfd_link_info *info) { const struct ecoff_backend_data * const backend = ecoff_backend (abfd); struct ecoff_debug_info * const debug = &ecoff_data (abfd)->debug_info; HDRR *symhdr; void * handle; bfd *input_bfd; asection *o; struct bfd_link_order *p; struct extsym_info einfo; /* We accumulate the debugging information counts in the symbolic header. */ symhdr = &debug->symbolic_header; symhdr->vstamp = 0; symhdr->ilineMax = 0; symhdr->cbLine = 0; symhdr->idnMax = 0; symhdr->ipdMax = 0; symhdr->isymMax = 0; symhdr->ioptMax = 0; symhdr->iauxMax = 0; symhdr->issMax = 0; symhdr->issExtMax = 0; symhdr->ifdMax = 0; symhdr->crfd = 0; symhdr->iextMax = 0; /* We accumulate the debugging information itself in the debug_info structure. */ debug->line = NULL; debug->external_dnr = NULL; debug->external_pdr = NULL; debug->external_sym = NULL; debug->external_opt = NULL; debug->external_aux = NULL; debug->ss = NULL; debug->ssext = debug->ssext_end = NULL; debug->external_fdr = NULL; debug->external_rfd = NULL; debug->external_ext = debug->external_ext_end = NULL; handle = bfd_ecoff_debug_init (abfd, debug, &backend->debug_swap, info); if (handle == NULL) return FALSE; /* Accumulate the debugging symbols from each input BFD. */ for (input_bfd = info->input_bfds; input_bfd != NULL; input_bfd = input_bfd->link_next) { bfd_boolean ret; if (bfd_get_flavour (input_bfd) == bfd_target_ecoff_flavour) { /* Arbitrarily set the symbolic header vstamp to the vstamp of the first object file in the link. */ if (symhdr->vstamp == 0) symhdr->vstamp = ecoff_data (input_bfd)->debug_info.symbolic_header.vstamp; ret = ecoff_final_link_debug_accumulate (abfd, input_bfd, info, handle); } else ret = bfd_ecoff_debug_accumulate_other (handle, abfd, debug, &backend->debug_swap, input_bfd, info); if (! ret) return FALSE; /* Combine the register masks. */ ecoff_data (abfd)->gprmask |= ecoff_data (input_bfd)->gprmask; ecoff_data (abfd)->fprmask |= ecoff_data (input_bfd)->fprmask; ecoff_data (abfd)->cprmask[0] |= ecoff_data (input_bfd)->cprmask[0]; ecoff_data (abfd)->cprmask[1] |= ecoff_data (input_bfd)->cprmask[1]; ecoff_data (abfd)->cprmask[2] |= ecoff_data (input_bfd)->cprmask[2]; ecoff_data (abfd)->cprmask[3] |= ecoff_data (input_bfd)->cprmask[3]; } /* Write out the external symbols. */ einfo.abfd = abfd; einfo.info = info; ecoff_link_hash_traverse (ecoff_hash_table (info), ecoff_link_write_external, (void *) &einfo); if (info->relocatable) { /* We need to make a pass over the link_orders to count up the number of relocations we will need to output, so that we know how much space they will take up. */ for (o = abfd->sections; o != NULL; o = o->next) { o->reloc_count = 0; for (p = o->map_head.link_order; p != NULL; p = p->next) if (p->type == bfd_indirect_link_order) o->reloc_count += p->u.indirect.section->reloc_count; else if (p->type == bfd_section_reloc_link_order || p->type == bfd_symbol_reloc_link_order) ++o->reloc_count; } } /* Compute the reloc and symbol file positions. */ ecoff_compute_reloc_file_positions (abfd); /* Write out the debugging information. */ if (! bfd_ecoff_write_accumulated_debug (handle, abfd, debug, &backend->debug_swap, info, ecoff_data (abfd)->sym_filepos)) return FALSE; bfd_ecoff_debug_free (handle, abfd, debug, &backend->debug_swap, info); if (info->relocatable) { /* Now reset the reloc_count field of the sections in the output BFD to 0, so that we can use them to keep track of how many relocs we have output thus far. */ for (o = abfd->sections; o != NULL; o = o->next) o->reloc_count = 0; } /* Get a value for the GP register. */ if (ecoff_data (abfd)->gp == 0) { struct bfd_link_hash_entry *h; h = bfd_link_hash_lookup (info->hash, "_gp", FALSE, FALSE, TRUE); if (h != NULL && h->type == bfd_link_hash_defined) ecoff_data (abfd)->gp = (h->u.def.value + h->u.def.section->output_section->vma + h->u.def.section->output_offset); else if (info->relocatable) { bfd_vma lo; /* Make up a value. */ lo = (bfd_vma) -1; for (o = abfd->sections; o != NULL; o = o->next) { if (o->vma < lo && (streq (o->name, _SBSS) || streq (o->name, _SDATA) || streq (o->name, _LIT4) || streq (o->name, _LIT8) || streq (o->name, _LITA))) lo = o->vma; } ecoff_data (abfd)->gp = lo + 0x8000; } else { /* If the relocate_section function needs to do a reloc involving the GP value, it should make a reloc_dangerous callback to warn that GP is not defined. */ } } for (o = abfd->sections; o != NULL; o = o->next) { for (p = o->map_head.link_order; p != NULL; p = p->next) { if (p->type == bfd_indirect_link_order && (bfd_get_flavour (p->u.indirect.section->owner) == bfd_target_ecoff_flavour)) { if (! ecoff_indirect_link_order (abfd, info, o, p)) return FALSE; } else if (p->type == bfd_section_reloc_link_order || p->type == bfd_symbol_reloc_link_order) { if (! ecoff_reloc_link_order (abfd, info, o, p)) return FALSE; } else { if (! _bfd_default_link_order (abfd, info, o, p)) return FALSE; } } } bfd_get_symcount (abfd) = symhdr->iextMax + symhdr->isymMax; ecoff_data (abfd)->linker = TRUE; return TRUE; } Index: projects/clang350-import/contrib/binutils/bfd/elf32-ppc.c =================================================================== --- projects/clang350-import/contrib/binutils/bfd/elf32-ppc.c (revision 275748) +++ projects/clang350-import/contrib/binutils/bfd/elf32-ppc.c (revision 275749) @@ -1,7742 +1,7940 @@ /* PowerPC-specific support for 32-bit ELF Copyright 1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007 Free Software Foundation, Inc. Written by Ian Lance Taylor, Cygnus Support. This file is part of BFD, the Binary File Descriptor library. This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin Street - Fifth Floor, Boston, MA 02110-1301, USA. */ /* This file is based on a preliminary PowerPC ELF ABI. The information may not match the final PowerPC ELF ABI. It includes suggestions from the in-progress Embedded PowerPC ABI, and that information may also not match. */ #include "sysdep.h" #include #include "bfd.h" #include "bfdlink.h" #include "libbfd.h" #include "elf-bfd.h" #include "elf/ppc.h" #include "elf32-ppc.h" #include "elf-vxworks.h" /* RELA relocations are used here. */ static bfd_reloc_status_type ppc_elf_addr16_ha_reloc (bfd *, arelent *, asymbol *, void *, asection *, bfd *, char **); static bfd_reloc_status_type ppc_elf_unhandled_reloc (bfd *, arelent *, asymbol *, void *, asection *, bfd *, char **); /* Branch prediction bit for branch taken relocs. */ #define BRANCH_PREDICT_BIT 0x200000 /* Mask to set RA in memory instructions. */ #define RA_REGISTER_MASK 0x001f0000 /* Value to shift register by to insert RA. */ #define RA_REGISTER_SHIFT 16 /* The name of the dynamic interpreter. This is put in the .interp section. */ #define ELF_DYNAMIC_INTERPRETER "/usr/lib/ld.so.1" /* For old-style PLT. */ /* The number of single-slot PLT entries (the rest use two slots). */ #define PLT_NUM_SINGLE_ENTRIES 8192 /* For new-style .glink and .plt. */ #define GLINK_PLTRESOLVE 16*4 #define GLINK_ENTRY_SIZE 4*4 /* VxWorks uses its own plt layout, filled in by the static linker. */ /* The standard VxWorks PLT entry. */ #define VXWORKS_PLT_ENTRY_SIZE 32 static const bfd_vma ppc_elf_vxworks_plt_entry [VXWORKS_PLT_ENTRY_SIZE / 4] = { 0x3d800000, /* lis r12,0 */ 0x818c0000, /* lwz r12,0(r12) */ 0x7d8903a6, /* mtctr r12 */ 0x4e800420, /* bctr */ 0x39600000, /* li r11,0 */ 0x48000000, /* b 14 <.PLT0resolve+0x4> */ 0x60000000, /* nop */ 0x60000000, /* nop */ }; static const bfd_vma ppc_elf_vxworks_pic_plt_entry [VXWORKS_PLT_ENTRY_SIZE / 4] = { 0x3d9e0000, /* addis r12,r30,0 */ 0x818c0000, /* lwz r12,0(r12) */ 0x7d8903a6, /* mtctr r12 */ 0x4e800420, /* bctr */ 0x39600000, /* li r11,0 */ 0x48000000, /* b 14 <.PLT0resolve+0x4> 14: R_PPC_REL24 .PLTresolve */ 0x60000000, /* nop */ 0x60000000, /* nop */ }; /* The initial VxWorks PLT entry. */ #define VXWORKS_PLT_INITIAL_ENTRY_SIZE 32 static const bfd_vma ppc_elf_vxworks_plt0_entry [VXWORKS_PLT_INITIAL_ENTRY_SIZE / 4] = { 0x3d800000, /* lis r12,0 */ 0x398c0000, /* addi r12,r12,0 */ 0x800c0008, /* lwz r0,8(r12) */ 0x7c0903a6, /* mtctr r0 */ 0x818c0004, /* lwz r12,4(r12) */ 0x4e800420, /* bctr */ 0x60000000, /* nop */ 0x60000000, /* nop */ }; static const bfd_vma ppc_elf_vxworks_pic_plt0_entry [VXWORKS_PLT_INITIAL_ENTRY_SIZE / 4] = { 0x819e0008, /* lwz r12,8(r30) */ 0x7d8903a6, /* mtctr r12 */ 0x819e0004, /* lwz r12,4(r30) */ 0x4e800420, /* bctr */ 0x60000000, /* nop */ 0x60000000, /* nop */ 0x60000000, /* nop */ 0x60000000, /* nop */ }; /* For executables, we have some additional relocations in .rela.plt.unloaded, for the kernel loader. */ /* The number of non-JMP_SLOT relocations per PLT0 slot. */ #define VXWORKS_PLT_NON_JMP_SLOT_RELOCS 3 /* The number of relocations in the PLTResolve slot. */ #define VXWORKS_PLTRESOLVE_RELOCS 2 /* The number of relocations in the PLTResolve slot when when creating a shared library. */ #define VXWORKS_PLTRESOLVE_RELOCS_SHLIB 0 /* Some instructions. */ #define ADDIS_11_11 0x3d6b0000 #define ADDIS_11_30 0x3d7e0000 #define ADDIS_12_12 0x3d8c0000 #define ADDI_11_11 0x396b0000 #define ADD_0_11_11 0x7c0b5a14 #define ADD_11_0_11 0x7d605a14 #define B 0x48000000 #define BCL_20_31 0x429f0005 #define BCTR 0x4e800420 #define LIS_11 0x3d600000 #define LIS_12 0x3d800000 #define LWZU_0_12 0x840c0000 #define LWZ_0_12 0x800c0000 #define LWZ_11_11 0x816b0000 #define LWZ_11_30 0x817e0000 #define LWZ_12_12 0x818c0000 #define MFLR_0 0x7c0802a6 #define MFLR_12 0x7d8802a6 #define MTCTR_0 0x7c0903a6 #define MTCTR_11 0x7d6903a6 #define MTLR_0 0x7c0803a6 #define NOP 0x60000000 #define SUB_11_11_12 0x7d6c5850 /* Offset of tp and dtp pointers from start of TLS block. */ #define TP_OFFSET 0x7000 #define DTP_OFFSET 0x8000 static reloc_howto_type *ppc_elf_howto_table[R_PPC_max]; static reloc_howto_type ppc_elf_howto_raw[] = { /* This reloc does nothing. */ HOWTO (R_PPC_NONE, /* type */ 0, /* rightshift */ 2, /* size (0 = byte, 1 = short, 2 = long) */ 32, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_bitfield, /* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC_NONE", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0, /* dst_mask */ FALSE), /* pcrel_offset */ /* A standard 32 bit relocation. */ HOWTO (R_PPC_ADDR32, /* type */ 0, /* rightshift */ 2, /* size (0 = byte, 1 = short, 2 = long) */ 32, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_bitfield, /* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC_ADDR32", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffffffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* An absolute 26 bit branch; the lower two bits must be zero. FIXME: we don't check that, we just clear them. */ HOWTO (R_PPC_ADDR24, /* type */ 0, /* rightshift */ 2, /* size (0 = byte, 1 = short, 2 = long) */ 26, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_bitfield, /* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC_ADDR24", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0x3fffffc, /* dst_mask */ FALSE), /* pcrel_offset */ /* A standard 16 bit relocation. */ HOWTO (R_PPC_ADDR16, /* type */ 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_bitfield, /* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC_ADDR16", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* A 16 bit relocation without overflow. */ HOWTO (R_PPC_ADDR16_LO, /* type */ 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont,/* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC_ADDR16_LO", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* The high order 16 bits of an address. */ HOWTO (R_PPC_ADDR16_HI, /* type */ 16, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC_ADDR16_HI", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* The high order 16 bits of an address, plus 1 if the contents of the low 16 bits, treated as a signed number, is negative. */ HOWTO (R_PPC_ADDR16_HA, /* type */ 16, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc_elf_addr16_ha_reloc, /* special_function */ "R_PPC_ADDR16_HA", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* An absolute 16 bit branch; the lower two bits must be zero. FIXME: we don't check that, we just clear them. */ HOWTO (R_PPC_ADDR14, /* type */ 0, /* rightshift */ 2, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_bitfield, /* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC_ADDR14", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xfffc, /* dst_mask */ FALSE), /* pcrel_offset */ /* An absolute 16 bit branch, for which bit 10 should be set to indicate that the branch is expected to be taken. The lower two bits must be zero. */ HOWTO (R_PPC_ADDR14_BRTAKEN, /* type */ 0, /* rightshift */ 2, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_bitfield, /* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC_ADDR14_BRTAKEN",/* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xfffc, /* dst_mask */ FALSE), /* pcrel_offset */ /* An absolute 16 bit branch, for which bit 10 should be set to indicate that the branch is not expected to be taken. The lower two bits must be zero. */ HOWTO (R_PPC_ADDR14_BRNTAKEN, /* type */ 0, /* rightshift */ 2, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_bitfield, /* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC_ADDR14_BRNTAKEN",/* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xfffc, /* dst_mask */ FALSE), /* pcrel_offset */ /* A relative 26 bit branch; the lower two bits must be zero. */ HOWTO (R_PPC_REL24, /* type */ 0, /* rightshift */ 2, /* size (0 = byte, 1 = short, 2 = long) */ 26, /* bitsize */ TRUE, /* pc_relative */ 0, /* bitpos */ complain_overflow_signed, /* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC_REL24", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0x3fffffc, /* dst_mask */ TRUE), /* pcrel_offset */ /* A relative 16 bit branch; the lower two bits must be zero. */ HOWTO (R_PPC_REL14, /* type */ 0, /* rightshift */ 2, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ TRUE, /* pc_relative */ 0, /* bitpos */ complain_overflow_signed, /* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC_REL14", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xfffc, /* dst_mask */ TRUE), /* pcrel_offset */ /* A relative 16 bit branch. Bit 10 should be set to indicate that the branch is expected to be taken. The lower two bits must be zero. */ HOWTO (R_PPC_REL14_BRTAKEN, /* type */ 0, /* rightshift */ 2, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ TRUE, /* pc_relative */ 0, /* bitpos */ complain_overflow_signed, /* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC_REL14_BRTAKEN", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xfffc, /* dst_mask */ TRUE), /* pcrel_offset */ /* A relative 16 bit branch. Bit 10 should be set to indicate that the branch is not expected to be taken. The lower two bits must be zero. */ HOWTO (R_PPC_REL14_BRNTAKEN, /* type */ 0, /* rightshift */ 2, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ TRUE, /* pc_relative */ 0, /* bitpos */ complain_overflow_signed, /* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC_REL14_BRNTAKEN",/* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xfffc, /* dst_mask */ TRUE), /* pcrel_offset */ /* Like R_PPC_ADDR16, but referring to the GOT table entry for the symbol. */ HOWTO (R_PPC_GOT16, /* type */ 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_signed, /* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC_GOT16", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like R_PPC_ADDR16_LO, but referring to the GOT table entry for the symbol. */ HOWTO (R_PPC_GOT16_LO, /* type */ 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC_GOT16_LO", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like R_PPC_ADDR16_HI, but referring to the GOT table entry for the symbol. */ HOWTO (R_PPC_GOT16_HI, /* type */ 16, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_bitfield, /* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC_GOT16_HI", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like R_PPC_ADDR16_HA, but referring to the GOT table entry for the symbol. */ HOWTO (R_PPC_GOT16_HA, /* type */ 16, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_bitfield, /* complain_on_overflow */ ppc_elf_addr16_ha_reloc, /* special_function */ "R_PPC_GOT16_HA", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like R_PPC_REL24, but referring to the procedure linkage table entry for the symbol. */ HOWTO (R_PPC_PLTREL24, /* type */ 0, /* rightshift */ 2, /* size (0 = byte, 1 = short, 2 = long) */ 26, /* bitsize */ TRUE, /* pc_relative */ 0, /* bitpos */ complain_overflow_signed, /* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC_PLTREL24", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0x3fffffc, /* dst_mask */ TRUE), /* pcrel_offset */ /* This is used only by the dynamic linker. The symbol should exist both in the object being run and in some shared library. The dynamic linker copies the data addressed by the symbol from the shared library into the object, because the object being run has to have the data at some particular address. */ HOWTO (R_PPC_COPY, /* type */ 0, /* rightshift */ 2, /* size (0 = byte, 1 = short, 2 = long) */ 32, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_bitfield, /* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC_COPY", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like R_PPC_ADDR32, but used when setting global offset table entries. */ HOWTO (R_PPC_GLOB_DAT, /* type */ 0, /* rightshift */ 2, /* size (0 = byte, 1 = short, 2 = long) */ 32, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_bitfield, /* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC_GLOB_DAT", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffffffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Marks a procedure linkage table entry for a symbol. */ HOWTO (R_PPC_JMP_SLOT, /* type */ 0, /* rightshift */ 2, /* size (0 = byte, 1 = short, 2 = long) */ 32, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_bitfield, /* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC_JMP_SLOT", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0, /* dst_mask */ FALSE), /* pcrel_offset */ /* Used only by the dynamic linker. When the object is run, this longword is set to the load address of the object, plus the addend. */ HOWTO (R_PPC_RELATIVE, /* type */ 0, /* rightshift */ 2, /* size (0 = byte, 1 = short, 2 = long) */ 32, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_bitfield, /* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC_RELATIVE", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffffffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like R_PPC_REL24, but uses the value of the symbol within the object rather than the final value. Normally used for _GLOBAL_OFFSET_TABLE_. */ HOWTO (R_PPC_LOCAL24PC, /* type */ 0, /* rightshift */ 2, /* size (0 = byte, 1 = short, 2 = long) */ 26, /* bitsize */ TRUE, /* pc_relative */ 0, /* bitpos */ complain_overflow_signed, /* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC_LOCAL24PC", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0x3fffffc, /* dst_mask */ TRUE), /* pcrel_offset */ /* Like R_PPC_ADDR32, but may be unaligned. */ HOWTO (R_PPC_UADDR32, /* type */ 0, /* rightshift */ 2, /* size (0 = byte, 1 = short, 2 = long) */ 32, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_bitfield, /* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC_UADDR32", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffffffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like R_PPC_ADDR16, but may be unaligned. */ HOWTO (R_PPC_UADDR16, /* type */ 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_bitfield, /* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC_UADDR16", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* 32-bit PC relative */ HOWTO (R_PPC_REL32, /* type */ 0, /* rightshift */ 2, /* size (0 = byte, 1 = short, 2 = long) */ 32, /* bitsize */ TRUE, /* pc_relative */ 0, /* bitpos */ complain_overflow_bitfield, /* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC_REL32", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffffffff, /* dst_mask */ TRUE), /* pcrel_offset */ /* 32-bit relocation to the symbol's procedure linkage table. FIXME: not supported. */ HOWTO (R_PPC_PLT32, /* type */ 0, /* rightshift */ 2, /* size (0 = byte, 1 = short, 2 = long) */ 32, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_bitfield, /* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC_PLT32", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0, /* dst_mask */ FALSE), /* pcrel_offset */ /* 32-bit PC relative relocation to the symbol's procedure linkage table. FIXME: not supported. */ HOWTO (R_PPC_PLTREL32, /* type */ 0, /* rightshift */ 2, /* size (0 = byte, 1 = short, 2 = long) */ 32, /* bitsize */ TRUE, /* pc_relative */ 0, /* bitpos */ complain_overflow_bitfield, /* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC_PLTREL32", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0, /* dst_mask */ TRUE), /* pcrel_offset */ /* Like R_PPC_ADDR16_LO, but referring to the PLT table entry for the symbol. */ HOWTO (R_PPC_PLT16_LO, /* type */ 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC_PLT16_LO", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like R_PPC_ADDR16_HI, but referring to the PLT table entry for the symbol. */ HOWTO (R_PPC_PLT16_HI, /* type */ 16, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_bitfield, /* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC_PLT16_HI", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like R_PPC_ADDR16_HA, but referring to the PLT table entry for the symbol. */ HOWTO (R_PPC_PLT16_HA, /* type */ 16, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_bitfield, /* complain_on_overflow */ ppc_elf_addr16_ha_reloc, /* special_function */ "R_PPC_PLT16_HA", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* A sign-extended 16 bit value relative to _SDA_BASE_, for use with small data items. */ HOWTO (R_PPC_SDAREL16, /* type */ 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_signed, /* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC_SDAREL16", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* 16-bit section relative relocation. */ HOWTO (R_PPC_SECTOFF, /* type */ 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_bitfield, /* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC_SECTOFF", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* 16-bit lower half section relative relocation. */ HOWTO (R_PPC_SECTOFF_LO, /* type */ 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC_SECTOFF_LO", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* 16-bit upper half section relative relocation. */ HOWTO (R_PPC_SECTOFF_HI, /* type */ 16, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_bitfield, /* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC_SECTOFF_HI", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* 16-bit upper half adjusted section relative relocation. */ HOWTO (R_PPC_SECTOFF_HA, /* type */ 16, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_bitfield, /* complain_on_overflow */ ppc_elf_addr16_ha_reloc, /* special_function */ "R_PPC_SECTOFF_HA", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ - /* Marker reloc for TLS. */ + /* Marker relocs for TLS. */ HOWTO (R_PPC_TLS, 0, /* rightshift */ 2, /* size (0 = byte, 1 = short, 2 = long) */ 32, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC_TLS", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0, /* dst_mask */ FALSE), /* pcrel_offset */ + HOWTO (R_PPC_TLSGD, + 0, /* rightshift */ + 2, /* size (0 = byte, 1 = short, 2 = long) */ + 32, /* bitsize */ + FALSE, /* pc_relative */ + 0, /* bitpos */ + complain_overflow_dont, /* complain_on_overflow */ + bfd_elf_generic_reloc, /* special_function */ + "R_PPC_TLSGD", /* name */ + FALSE, /* partial_inplace */ + 0, /* src_mask */ + 0, /* dst_mask */ + FALSE), /* pcrel_offset */ + + HOWTO (R_PPC_TLSLD, + 0, /* rightshift */ + 2, /* size (0 = byte, 1 = short, 2 = long) */ + 32, /* bitsize */ + FALSE, /* pc_relative */ + 0, /* bitpos */ + complain_overflow_dont, /* complain_on_overflow */ + bfd_elf_generic_reloc, /* special_function */ + "R_PPC_TLSLD", /* name */ + FALSE, /* partial_inplace */ + 0, /* src_mask */ + 0, /* dst_mask */ + FALSE), /* pcrel_offset */ + /* Computes the load module index of the load module that contains the definition of its TLS sym. */ HOWTO (R_PPC_DTPMOD32, 0, /* rightshift */ 2, /* size (0 = byte, 1 = short, 2 = long) */ 32, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc_elf_unhandled_reloc, /* special_function */ "R_PPC_DTPMOD32", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffffffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Computes a dtv-relative displacement, the difference between the value of sym+add and the base address of the thread-local storage block that contains the definition of sym, minus 0x8000. */ HOWTO (R_PPC_DTPREL32, 0, /* rightshift */ 2, /* size (0 = byte, 1 = short, 2 = long) */ 32, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc_elf_unhandled_reloc, /* special_function */ "R_PPC_DTPREL32", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffffffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* A 16 bit dtprel reloc. */ HOWTO (R_PPC_DTPREL16, 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_signed, /* complain_on_overflow */ ppc_elf_unhandled_reloc, /* special_function */ "R_PPC_DTPREL16", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like DTPREL16, but no overflow. */ HOWTO (R_PPC_DTPREL16_LO, 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc_elf_unhandled_reloc, /* special_function */ "R_PPC_DTPREL16_LO", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like DTPREL16_LO, but next higher group of 16 bits. */ HOWTO (R_PPC_DTPREL16_HI, 16, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc_elf_unhandled_reloc, /* special_function */ "R_PPC_DTPREL16_HI", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like DTPREL16_HI, but adjust for low 16 bits. */ HOWTO (R_PPC_DTPREL16_HA, 16, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc_elf_unhandled_reloc, /* special_function */ "R_PPC_DTPREL16_HA", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Computes a tp-relative displacement, the difference between the value of sym+add and the value of the thread pointer (r13). */ HOWTO (R_PPC_TPREL32, 0, /* rightshift */ 2, /* size (0 = byte, 1 = short, 2 = long) */ 32, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc_elf_unhandled_reloc, /* special_function */ "R_PPC_TPREL32", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffffffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* A 16 bit tprel reloc. */ HOWTO (R_PPC_TPREL16, 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_signed, /* complain_on_overflow */ ppc_elf_unhandled_reloc, /* special_function */ "R_PPC_TPREL16", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like TPREL16, but no overflow. */ HOWTO (R_PPC_TPREL16_LO, 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc_elf_unhandled_reloc, /* special_function */ "R_PPC_TPREL16_LO", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like TPREL16_LO, but next higher group of 16 bits. */ HOWTO (R_PPC_TPREL16_HI, 16, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc_elf_unhandled_reloc, /* special_function */ "R_PPC_TPREL16_HI", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like TPREL16_HI, but adjust for low 16 bits. */ HOWTO (R_PPC_TPREL16_HA, 16, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc_elf_unhandled_reloc, /* special_function */ "R_PPC_TPREL16_HA", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Allocates two contiguous entries in the GOT to hold a tls_index structure, with values (sym+add)@dtpmod and (sym+add)@dtprel, and computes the offset to the first entry. */ HOWTO (R_PPC_GOT_TLSGD16, 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_signed, /* complain_on_overflow */ ppc_elf_unhandled_reloc, /* special_function */ "R_PPC_GOT_TLSGD16", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like GOT_TLSGD16, but no overflow. */ HOWTO (R_PPC_GOT_TLSGD16_LO, 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc_elf_unhandled_reloc, /* special_function */ "R_PPC_GOT_TLSGD16_LO", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like GOT_TLSGD16_LO, but next higher group of 16 bits. */ HOWTO (R_PPC_GOT_TLSGD16_HI, 16, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc_elf_unhandled_reloc, /* special_function */ "R_PPC_GOT_TLSGD16_HI", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like GOT_TLSGD16_HI, but adjust for low 16 bits. */ HOWTO (R_PPC_GOT_TLSGD16_HA, 16, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc_elf_unhandled_reloc, /* special_function */ "R_PPC_GOT_TLSGD16_HA", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Allocates two contiguous entries in the GOT to hold a tls_index structure, with values (sym+add)@dtpmod and zero, and computes the offset to the first entry. */ HOWTO (R_PPC_GOT_TLSLD16, 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_signed, /* complain_on_overflow */ ppc_elf_unhandled_reloc, /* special_function */ "R_PPC_GOT_TLSLD16", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like GOT_TLSLD16, but no overflow. */ HOWTO (R_PPC_GOT_TLSLD16_LO, 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc_elf_unhandled_reloc, /* special_function */ "R_PPC_GOT_TLSLD16_LO", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like GOT_TLSLD16_LO, but next higher group of 16 bits. */ HOWTO (R_PPC_GOT_TLSLD16_HI, 16, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc_elf_unhandled_reloc, /* special_function */ "R_PPC_GOT_TLSLD16_HI", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like GOT_TLSLD16_HI, but adjust for low 16 bits. */ HOWTO (R_PPC_GOT_TLSLD16_HA, 16, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc_elf_unhandled_reloc, /* special_function */ "R_PPC_GOT_TLSLD16_HA", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Allocates an entry in the GOT with value (sym+add)@dtprel, and computes the offset to the entry. */ HOWTO (R_PPC_GOT_DTPREL16, 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_signed, /* complain_on_overflow */ ppc_elf_unhandled_reloc, /* special_function */ "R_PPC_GOT_DTPREL16", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like GOT_DTPREL16, but no overflow. */ HOWTO (R_PPC_GOT_DTPREL16_LO, 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc_elf_unhandled_reloc, /* special_function */ "R_PPC_GOT_DTPREL16_LO", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like GOT_DTPREL16_LO, but next higher group of 16 bits. */ HOWTO (R_PPC_GOT_DTPREL16_HI, 16, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc_elf_unhandled_reloc, /* special_function */ "R_PPC_GOT_DTPREL16_HI", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like GOT_DTPREL16_HI, but adjust for low 16 bits. */ HOWTO (R_PPC_GOT_DTPREL16_HA, 16, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc_elf_unhandled_reloc, /* special_function */ "R_PPC_GOT_DTPREL16_HA", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Allocates an entry in the GOT with value (sym+add)@tprel, and computes the offset to the entry. */ HOWTO (R_PPC_GOT_TPREL16, 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_signed, /* complain_on_overflow */ ppc_elf_unhandled_reloc, /* special_function */ "R_PPC_GOT_TPREL16", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like GOT_TPREL16, but no overflow. */ HOWTO (R_PPC_GOT_TPREL16_LO, 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc_elf_unhandled_reloc, /* special_function */ "R_PPC_GOT_TPREL16_LO", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like GOT_TPREL16_LO, but next higher group of 16 bits. */ HOWTO (R_PPC_GOT_TPREL16_HI, 16, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc_elf_unhandled_reloc, /* special_function */ "R_PPC_GOT_TPREL16_HI", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like GOT_TPREL16_HI, but adjust for low 16 bits. */ HOWTO (R_PPC_GOT_TPREL16_HA, 16, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc_elf_unhandled_reloc, /* special_function */ "R_PPC_GOT_TPREL16_HA", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* The remaining relocs are from the Embedded ELF ABI, and are not in the SVR4 ELF ABI. */ /* 32 bit value resulting from the addend minus the symbol. */ HOWTO (R_PPC_EMB_NADDR32, /* type */ 0, /* rightshift */ 2, /* size (0 = byte, 1 = short, 2 = long) */ 32, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_bitfield, /* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC_EMB_NADDR32", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffffffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* 16 bit value resulting from the addend minus the symbol. */ HOWTO (R_PPC_EMB_NADDR16, /* type */ 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_bitfield, /* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC_EMB_NADDR16", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* 16 bit value resulting from the addend minus the symbol. */ HOWTO (R_PPC_EMB_NADDR16_LO, /* type */ 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont,/* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC_EMB_ADDR16_LO", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* The high order 16 bits of the addend minus the symbol. */ HOWTO (R_PPC_EMB_NADDR16_HI, /* type */ 16, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC_EMB_NADDR16_HI", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* The high order 16 bits of the result of the addend minus the address, plus 1 if the contents of the low 16 bits, treated as a signed number, is negative. */ HOWTO (R_PPC_EMB_NADDR16_HA, /* type */ 16, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc_elf_addr16_ha_reloc, /* special_function */ "R_PPC_EMB_NADDR16_HA", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* 16 bit value resulting from allocating a 4 byte word to hold an address in the .sdata section, and returning the offset from _SDA_BASE_ for that relocation. */ HOWTO (R_PPC_EMB_SDAI16, /* type */ 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_bitfield, /* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC_EMB_SDAI16", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* 16 bit value resulting from allocating a 4 byte word to hold an address in the .sdata2 section, and returning the offset from _SDA2_BASE_ for that relocation. */ HOWTO (R_PPC_EMB_SDA2I16, /* type */ 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_bitfield, /* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC_EMB_SDA2I16", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* A sign-extended 16 bit value relative to _SDA2_BASE_, for use with small data items. */ HOWTO (R_PPC_EMB_SDA2REL, /* type */ 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_signed, /* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC_EMB_SDA2REL", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Relocate against either _SDA_BASE_ or _SDA2_BASE_, filling in the 16 bit signed offset from the appropriate base, and filling in the register field with the appropriate register (0, 2, or 13). */ HOWTO (R_PPC_EMB_SDA21, /* type */ 0, /* rightshift */ 2, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_signed, /* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC_EMB_SDA21", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Relocation not handled: R_PPC_EMB_MRKREF */ /* Relocation not handled: R_PPC_EMB_RELSEC16 */ /* Relocation not handled: R_PPC_EMB_RELST_LO */ /* Relocation not handled: R_PPC_EMB_RELST_HI */ /* Relocation not handled: R_PPC_EMB_RELST_HA */ /* Relocation not handled: R_PPC_EMB_BIT_FLD */ /* PC relative relocation against either _SDA_BASE_ or _SDA2_BASE_, filling in the 16 bit signed offset from the appropriate base, and filling in the register field with the appropriate register (0, 2, or 13). */ HOWTO (R_PPC_EMB_RELSDA, /* type */ 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ TRUE, /* pc_relative */ 0, /* bitpos */ complain_overflow_signed, /* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC_EMB_RELSDA", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* A 16 bit relative relocation. */ HOWTO (R_PPC_REL16, /* type */ 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ TRUE, /* pc_relative */ 0, /* bitpos */ complain_overflow_bitfield, /* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC_REL16", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ TRUE), /* pcrel_offset */ /* A 16 bit relative relocation without overflow. */ HOWTO (R_PPC_REL16_LO, /* type */ 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ TRUE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont,/* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC_REL16_LO", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ TRUE), /* pcrel_offset */ /* The high order 16 bits of a relative address. */ HOWTO (R_PPC_REL16_HI, /* type */ 16, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ TRUE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC_REL16_HI", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ TRUE), /* pcrel_offset */ /* The high order 16 bits of a relative address, plus 1 if the contents of the low 16 bits, treated as a signed number, is negative. */ HOWTO (R_PPC_REL16_HA, /* type */ 16, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ TRUE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc_elf_addr16_ha_reloc, /* special_function */ "R_PPC_REL16_HA", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ TRUE), /* pcrel_offset */ /* GNU extension to record C++ vtable hierarchy. */ HOWTO (R_PPC_GNU_VTINHERIT, /* type */ 0, /* rightshift */ 0, /* size (0 = byte, 1 = short, 2 = long) */ 0, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ NULL, /* special_function */ "R_PPC_GNU_VTINHERIT", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0, /* dst_mask */ FALSE), /* pcrel_offset */ /* GNU extension to record C++ vtable member usage. */ HOWTO (R_PPC_GNU_VTENTRY, /* type */ 0, /* rightshift */ 0, /* size (0 = byte, 1 = short, 2 = long) */ 0, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ NULL, /* special_function */ "R_PPC_GNU_VTENTRY", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0, /* dst_mask */ FALSE), /* pcrel_offset */ /* Phony reloc to handle AIX style TOC entries. */ HOWTO (R_PPC_TOC16, /* type */ 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_signed, /* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC_TOC16", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ }; /* Initialize the ppc_elf_howto_table, so that linear accesses can be done. */ static void ppc_elf_howto_init (void) { unsigned int i, type; for (i = 0; i < sizeof (ppc_elf_howto_raw) / sizeof (ppc_elf_howto_raw[0]); i++) { type = ppc_elf_howto_raw[i].type; if (type >= (sizeof (ppc_elf_howto_table) / sizeof (ppc_elf_howto_table[0]))) abort (); ppc_elf_howto_table[type] = &ppc_elf_howto_raw[i]; } } static reloc_howto_type * ppc_elf_reloc_type_lookup (bfd *abfd ATTRIBUTE_UNUSED, bfd_reloc_code_real_type code) { enum elf_ppc_reloc_type r; /* Initialize howto table if not already done. */ if (!ppc_elf_howto_table[R_PPC_ADDR32]) ppc_elf_howto_init (); switch (code) { default: return NULL; case BFD_RELOC_NONE: r = R_PPC_NONE; break; case BFD_RELOC_32: r = R_PPC_ADDR32; break; case BFD_RELOC_PPC_BA26: r = R_PPC_ADDR24; break; case BFD_RELOC_16: r = R_PPC_ADDR16; break; case BFD_RELOC_LO16: r = R_PPC_ADDR16_LO; break; case BFD_RELOC_HI16: r = R_PPC_ADDR16_HI; break; case BFD_RELOC_HI16_S: r = R_PPC_ADDR16_HA; break; case BFD_RELOC_PPC_BA16: r = R_PPC_ADDR14; break; case BFD_RELOC_PPC_BA16_BRTAKEN: r = R_PPC_ADDR14_BRTAKEN; break; case BFD_RELOC_PPC_BA16_BRNTAKEN: r = R_PPC_ADDR14_BRNTAKEN; break; case BFD_RELOC_PPC_B26: r = R_PPC_REL24; break; case BFD_RELOC_PPC_B16: r = R_PPC_REL14; break; case BFD_RELOC_PPC_B16_BRTAKEN: r = R_PPC_REL14_BRTAKEN; break; case BFD_RELOC_PPC_B16_BRNTAKEN: r = R_PPC_REL14_BRNTAKEN; break; case BFD_RELOC_16_GOTOFF: r = R_PPC_GOT16; break; case BFD_RELOC_LO16_GOTOFF: r = R_PPC_GOT16_LO; break; case BFD_RELOC_HI16_GOTOFF: r = R_PPC_GOT16_HI; break; case BFD_RELOC_HI16_S_GOTOFF: r = R_PPC_GOT16_HA; break; case BFD_RELOC_24_PLT_PCREL: r = R_PPC_PLTREL24; break; case BFD_RELOC_PPC_COPY: r = R_PPC_COPY; break; case BFD_RELOC_PPC_GLOB_DAT: r = R_PPC_GLOB_DAT; break; case BFD_RELOC_PPC_LOCAL24PC: r = R_PPC_LOCAL24PC; break; case BFD_RELOC_32_PCREL: r = R_PPC_REL32; break; case BFD_RELOC_32_PLTOFF: r = R_PPC_PLT32; break; case BFD_RELOC_32_PLT_PCREL: r = R_PPC_PLTREL32; break; case BFD_RELOC_LO16_PLTOFF: r = R_PPC_PLT16_LO; break; case BFD_RELOC_HI16_PLTOFF: r = R_PPC_PLT16_HI; break; case BFD_RELOC_HI16_S_PLTOFF: r = R_PPC_PLT16_HA; break; case BFD_RELOC_GPREL16: r = R_PPC_SDAREL16; break; case BFD_RELOC_16_BASEREL: r = R_PPC_SECTOFF; break; case BFD_RELOC_LO16_BASEREL: r = R_PPC_SECTOFF_LO; break; case BFD_RELOC_HI16_BASEREL: r = R_PPC_SECTOFF_HI; break; case BFD_RELOC_HI16_S_BASEREL: r = R_PPC_SECTOFF_HA; break; case BFD_RELOC_CTOR: r = R_PPC_ADDR32; break; case BFD_RELOC_PPC_TOC16: r = R_PPC_TOC16; break; case BFD_RELOC_PPC_TLS: r = R_PPC_TLS; break; + case BFD_RELOC_PPC_TLSGD: r = R_PPC_TLSGD; break; + case BFD_RELOC_PPC_TLSLD: r = R_PPC_TLSLD; break; case BFD_RELOC_PPC_DTPMOD: r = R_PPC_DTPMOD32; break; case BFD_RELOC_PPC_TPREL16: r = R_PPC_TPREL16; break; case BFD_RELOC_PPC_TPREL16_LO: r = R_PPC_TPREL16_LO; break; case BFD_RELOC_PPC_TPREL16_HI: r = R_PPC_TPREL16_HI; break; case BFD_RELOC_PPC_TPREL16_HA: r = R_PPC_TPREL16_HA; break; case BFD_RELOC_PPC_TPREL: r = R_PPC_TPREL32; break; case BFD_RELOC_PPC_DTPREL16: r = R_PPC_DTPREL16; break; case BFD_RELOC_PPC_DTPREL16_LO: r = R_PPC_DTPREL16_LO; break; case BFD_RELOC_PPC_DTPREL16_HI: r = R_PPC_DTPREL16_HI; break; case BFD_RELOC_PPC_DTPREL16_HA: r = R_PPC_DTPREL16_HA; break; case BFD_RELOC_PPC_DTPREL: r = R_PPC_DTPREL32; break; case BFD_RELOC_PPC_GOT_TLSGD16: r = R_PPC_GOT_TLSGD16; break; case BFD_RELOC_PPC_GOT_TLSGD16_LO: r = R_PPC_GOT_TLSGD16_LO; break; case BFD_RELOC_PPC_GOT_TLSGD16_HI: r = R_PPC_GOT_TLSGD16_HI; break; case BFD_RELOC_PPC_GOT_TLSGD16_HA: r = R_PPC_GOT_TLSGD16_HA; break; case BFD_RELOC_PPC_GOT_TLSLD16: r = R_PPC_GOT_TLSLD16; break; case BFD_RELOC_PPC_GOT_TLSLD16_LO: r = R_PPC_GOT_TLSLD16_LO; break; case BFD_RELOC_PPC_GOT_TLSLD16_HI: r = R_PPC_GOT_TLSLD16_HI; break; case BFD_RELOC_PPC_GOT_TLSLD16_HA: r = R_PPC_GOT_TLSLD16_HA; break; case BFD_RELOC_PPC_GOT_TPREL16: r = R_PPC_GOT_TPREL16; break; case BFD_RELOC_PPC_GOT_TPREL16_LO: r = R_PPC_GOT_TPREL16_LO; break; case BFD_RELOC_PPC_GOT_TPREL16_HI: r = R_PPC_GOT_TPREL16_HI; break; case BFD_RELOC_PPC_GOT_TPREL16_HA: r = R_PPC_GOT_TPREL16_HA; break; case BFD_RELOC_PPC_GOT_DTPREL16: r = R_PPC_GOT_DTPREL16; break; case BFD_RELOC_PPC_GOT_DTPREL16_LO: r = R_PPC_GOT_DTPREL16_LO; break; case BFD_RELOC_PPC_GOT_DTPREL16_HI: r = R_PPC_GOT_DTPREL16_HI; break; case BFD_RELOC_PPC_GOT_DTPREL16_HA: r = R_PPC_GOT_DTPREL16_HA; break; case BFD_RELOC_PPC_EMB_NADDR32: r = R_PPC_EMB_NADDR32; break; case BFD_RELOC_PPC_EMB_NADDR16: r = R_PPC_EMB_NADDR16; break; case BFD_RELOC_PPC_EMB_NADDR16_LO: r = R_PPC_EMB_NADDR16_LO; break; case BFD_RELOC_PPC_EMB_NADDR16_HI: r = R_PPC_EMB_NADDR16_HI; break; case BFD_RELOC_PPC_EMB_NADDR16_HA: r = R_PPC_EMB_NADDR16_HA; break; case BFD_RELOC_PPC_EMB_SDAI16: r = R_PPC_EMB_SDAI16; break; case BFD_RELOC_PPC_EMB_SDA2I16: r = R_PPC_EMB_SDA2I16; break; case BFD_RELOC_PPC_EMB_SDA2REL: r = R_PPC_EMB_SDA2REL; break; case BFD_RELOC_PPC_EMB_SDA21: r = R_PPC_EMB_SDA21; break; case BFD_RELOC_PPC_EMB_MRKREF: r = R_PPC_EMB_MRKREF; break; case BFD_RELOC_PPC_EMB_RELSEC16: r = R_PPC_EMB_RELSEC16; break; case BFD_RELOC_PPC_EMB_RELST_LO: r = R_PPC_EMB_RELST_LO; break; case BFD_RELOC_PPC_EMB_RELST_HI: r = R_PPC_EMB_RELST_HI; break; case BFD_RELOC_PPC_EMB_RELST_HA: r = R_PPC_EMB_RELST_HA; break; case BFD_RELOC_PPC_EMB_BIT_FLD: r = R_PPC_EMB_BIT_FLD; break; case BFD_RELOC_PPC_EMB_RELSDA: r = R_PPC_EMB_RELSDA; break; case BFD_RELOC_16_PCREL: r = R_PPC_REL16; break; case BFD_RELOC_LO16_PCREL: r = R_PPC_REL16_LO; break; case BFD_RELOC_HI16_PCREL: r = R_PPC_REL16_HI; break; case BFD_RELOC_HI16_S_PCREL: r = R_PPC_REL16_HA; break; case BFD_RELOC_VTABLE_INHERIT: r = R_PPC_GNU_VTINHERIT; break; case BFD_RELOC_VTABLE_ENTRY: r = R_PPC_GNU_VTENTRY; break; } return ppc_elf_howto_table[r]; }; static reloc_howto_type * ppc_elf_reloc_name_lookup (bfd *abfd ATTRIBUTE_UNUSED, const char *r_name) { unsigned int i; for (i = 0; i < sizeof (ppc_elf_howto_raw) / sizeof (ppc_elf_howto_raw[0]); i++) if (ppc_elf_howto_raw[i].name != NULL && strcasecmp (ppc_elf_howto_raw[i].name, r_name) == 0) return &ppc_elf_howto_raw[i]; return NULL; } /* Set the howto pointer for a PowerPC ELF reloc. */ static void ppc_elf_info_to_howto (bfd *abfd ATTRIBUTE_UNUSED, arelent *cache_ptr, Elf_Internal_Rela *dst) { /* Initialize howto table if not already done. */ if (!ppc_elf_howto_table[R_PPC_ADDR32]) ppc_elf_howto_init (); BFD_ASSERT (ELF32_R_TYPE (dst->r_info) < (unsigned int) R_PPC_max); cache_ptr->howto = ppc_elf_howto_table[ELF32_R_TYPE (dst->r_info)]; /* Just because the above assert didn't trigger doesn't mean that ELF32_R_TYPE (dst->r_info) is necessarily a valid relocation. */ if (!cache_ptr->howto) { (*_bfd_error_handler) (_("%B: invalid relocation type %d"), abfd, ELF32_R_TYPE (dst->r_info)); bfd_set_error (bfd_error_bad_value); cache_ptr->howto = ppc_elf_howto_table[R_PPC_NONE]; } } /* Handle the R_PPC_ADDR16_HA and R_PPC_REL16_HA relocs. */ static bfd_reloc_status_type ppc_elf_addr16_ha_reloc (bfd *abfd ATTRIBUTE_UNUSED, arelent *reloc_entry, asymbol *symbol, void *data ATTRIBUTE_UNUSED, asection *input_section, bfd *output_bfd, char **error_message ATTRIBUTE_UNUSED) { bfd_vma relocation; if (output_bfd != NULL) { reloc_entry->address += input_section->output_offset; return bfd_reloc_ok; } if (reloc_entry->address > bfd_get_section_limit (abfd, input_section)) return bfd_reloc_outofrange; if (bfd_is_com_section (symbol->section)) relocation = 0; else relocation = symbol->value; relocation += symbol->section->output_section->vma; relocation += symbol->section->output_offset; relocation += reloc_entry->addend; if (reloc_entry->howto->pc_relative) relocation -= reloc_entry->address; reloc_entry->addend += (relocation & 0x8000) << 1; return bfd_reloc_continue; } static bfd_reloc_status_type ppc_elf_unhandled_reloc (bfd *abfd, arelent *reloc_entry, asymbol *symbol, void *data, asection *input_section, bfd *output_bfd, char **error_message) { /* If this is a relocatable link (output_bfd test tells us), just call the generic function. Any adjustment will be done at final link time. */ if (output_bfd != NULL) return bfd_elf_generic_reloc (abfd, reloc_entry, symbol, data, input_section, output_bfd, error_message); if (error_message != NULL) { static char buf[60]; sprintf (buf, _("generic linker can't handle %s"), reloc_entry->howto->name); *error_message = buf; } return bfd_reloc_dangerous; } /* Sections created by the linker. */ typedef struct elf_linker_section { /* Pointer to the bfd section. */ asection *section; /* Section name. */ const char *name; /* Associated bss section name. */ const char *bss_name; /* Associated symbol name. */ const char *sym_name; /* Associated symbol. */ struct elf_link_hash_entry *sym; } elf_linker_section_t; /* Linked list of allocated pointer entries. This hangs off of the symbol lists, and provides allows us to return different pointers, based on different addend's. */ typedef struct elf_linker_section_pointers { /* next allocated pointer for this symbol */ struct elf_linker_section_pointers *next; /* offset of pointer from beginning of section */ bfd_vma offset; /* addend used */ bfd_vma addend; /* which linker section this is */ elf_linker_section_t *lsect; } elf_linker_section_pointers_t; struct ppc_elf_obj_tdata { struct elf_obj_tdata elf; /* A mapping from local symbols to offsets into the various linker sections added. This is index by the symbol index. */ elf_linker_section_pointers_t **linker_section_pointers; /* Flags used to auto-detect plt type. */ unsigned int makes_plt_call : 1; unsigned int has_rel16 : 1; }; #define ppc_elf_tdata(bfd) \ ((struct ppc_elf_obj_tdata *) (bfd)->tdata.any) #define elf_local_ptr_offsets(bfd) \ (ppc_elf_tdata (bfd)->linker_section_pointers) /* Override the generic function because we store some extras. */ static bfd_boolean ppc_elf_mkobject (bfd *abfd) { if (abfd->tdata.any == NULL) { bfd_size_type amt = sizeof (struct ppc_elf_obj_tdata); abfd->tdata.any = bfd_zalloc (abfd, amt); if (abfd->tdata.any == NULL) return FALSE; } return bfd_elf_mkobject (abfd); } /* Fix bad default arch selected for a 32 bit input bfd when the default is 64 bit. */ static bfd_boolean ppc_elf_object_p (bfd *abfd) { if (abfd->arch_info->the_default && abfd->arch_info->bits_per_word == 64) { Elf_Internal_Ehdr *i_ehdr = elf_elfheader (abfd); if (i_ehdr->e_ident[EI_CLASS] == ELFCLASS32) { /* Relies on arch after 64 bit default being 32 bit default. */ abfd->arch_info = abfd->arch_info->next; BFD_ASSERT (abfd->arch_info->bits_per_word == 32); } } return TRUE; } /* Function to set whether a module needs the -mrelocatable bit set. */ static bfd_boolean ppc_elf_set_private_flags (bfd *abfd, flagword flags) { BFD_ASSERT (!elf_flags_init (abfd) || elf_elfheader (abfd)->e_flags == flags); elf_elfheader (abfd)->e_flags = flags; elf_flags_init (abfd) = TRUE; return TRUE; } /* Support for core dump NOTE sections. */ static bfd_boolean ppc_elf_grok_prstatus (bfd *abfd, Elf_Internal_Note *note) { int offset; unsigned int size; switch (note->descsz) { default: return FALSE; case 268: /* Linux/PPC. */ /* pr_cursig */ elf_tdata (abfd)->core_signal = bfd_get_16 (abfd, note->descdata + 12); /* pr_pid */ elf_tdata (abfd)->core_pid = bfd_get_32 (abfd, note->descdata + 24); /* pr_reg */ offset = 72; size = 192; break; } /* Make a ".reg/999" section. */ return _bfd_elfcore_make_pseudosection (abfd, ".reg", size, note->descpos + offset); } static bfd_boolean ppc_elf_grok_psinfo (bfd *abfd, Elf_Internal_Note *note) { switch (note->descsz) { default: return FALSE; case 128: /* Linux/PPC elf_prpsinfo. */ elf_tdata (abfd)->core_program = _bfd_elfcore_strndup (abfd, note->descdata + 32, 16); elf_tdata (abfd)->core_command = _bfd_elfcore_strndup (abfd, note->descdata + 48, 80); } /* Note that for some reason, a spurious space is tacked onto the end of the args in some (at least one anyway) implementations, so strip it off if it exists. */ { char *command = elf_tdata (abfd)->core_command; int n = strlen (command); if (0 < n && command[n - 1] == ' ') command[n - 1] = '\0'; } return TRUE; } static char * ppc_elf_write_core_note (bfd *abfd, char *buf, int *bufsiz, int note_type, ...) { switch (note_type) { default: return NULL; case NT_PRPSINFO: { char data[128]; va_list ap; va_start (ap, note_type); memset (data, 0, 32); strncpy (data + 32, va_arg (ap, const char *), 16); strncpy (data + 48, va_arg (ap, const char *), 80); va_end (ap); return elfcore_write_note (abfd, buf, bufsiz, "CORE", note_type, data, sizeof (data)); } case NT_PRSTATUS: { char data[268]; va_list ap; long pid; int cursig; const void *greg; va_start (ap, note_type); memset (data, 0, 72); pid = va_arg (ap, long); bfd_put_32 (abfd, pid, data + 24); cursig = va_arg (ap, int); bfd_put_16 (abfd, cursig, data + 12); greg = va_arg (ap, const void *); memcpy (data + 72, greg, 192); memset (data + 264, 0, 4); va_end (ap); return elfcore_write_note (abfd, buf, bufsiz, "CORE", note_type, data, sizeof (data)); } } } /* Return address for Ith PLT stub in section PLT, for relocation REL or (bfd_vma) -1 if it should not be included. */ static bfd_vma ppc_elf_plt_sym_val (bfd_vma i ATTRIBUTE_UNUSED, const asection *plt ATTRIBUTE_UNUSED, const arelent *rel) { return rel->address; } /* Handle a PowerPC specific section when reading an object file. This is called when bfd_section_from_shdr finds a section with an unknown type. */ static bfd_boolean ppc_elf_section_from_shdr (bfd *abfd, Elf_Internal_Shdr *hdr, const char *name, int shindex) { asection *newsect; flagword flags; if (! _bfd_elf_make_section_from_shdr (abfd, hdr, name, shindex)) return FALSE; newsect = hdr->bfd_section; flags = bfd_get_section_flags (abfd, newsect); if (hdr->sh_flags & SHF_EXCLUDE) flags |= SEC_EXCLUDE; if (hdr->sh_type == SHT_ORDERED) flags |= SEC_SORT_ENTRIES; bfd_set_section_flags (abfd, newsect, flags); return TRUE; } /* Set up any other section flags and such that may be necessary. */ static bfd_boolean ppc_elf_fake_sections (bfd *abfd ATTRIBUTE_UNUSED, Elf_Internal_Shdr *shdr, asection *asect) { if ((asect->flags & (SEC_GROUP | SEC_EXCLUDE)) == SEC_EXCLUDE) shdr->sh_flags |= SHF_EXCLUDE; if ((asect->flags & SEC_SORT_ENTRIES) != 0) shdr->sh_type = SHT_ORDERED; return TRUE; } /* If we have .sbss2 or .PPC.EMB.sbss0 output sections, we need to bump up the number of section headers. */ static int ppc_elf_additional_program_headers (bfd *abfd, struct bfd_link_info *info ATTRIBUTE_UNUSED) { asection *s; int ret = 0; s = bfd_get_section_by_name (abfd, ".sbss2"); if (s != NULL && (s->flags & SEC_ALLOC) != 0) ++ret; s = bfd_get_section_by_name (abfd, ".PPC.EMB.sbss0"); if (s != NULL && (s->flags & SEC_ALLOC) != 0) ++ret; return ret; } /* Add extra PPC sections -- Note, for now, make .sbss2 and .PPC.EMB.sbss0 a normal section, and not a bss section so that the linker doesn't crater when trying to make more than 2 sections. */ static const struct bfd_elf_special_section ppc_elf_special_sections[] = { { STRING_COMMA_LEN (".plt"), 0, SHT_NOBITS, SHF_ALLOC + SHF_EXECINSTR }, { STRING_COMMA_LEN (".sbss"), -2, SHT_NOBITS, SHF_ALLOC + SHF_WRITE }, { STRING_COMMA_LEN (".sbss2"), -2, SHT_PROGBITS, SHF_ALLOC }, { STRING_COMMA_LEN (".sdata"), -2, SHT_PROGBITS, SHF_ALLOC + SHF_WRITE }, { STRING_COMMA_LEN (".sdata2"), -2, SHT_PROGBITS, SHF_ALLOC }, { STRING_COMMA_LEN (".tags"), 0, SHT_ORDERED, SHF_ALLOC }, { STRING_COMMA_LEN (".PPC.EMB.apuinfo"), 0, SHT_NOTE, 0 }, { STRING_COMMA_LEN (".PPC.EMB.sbss0"), 0, SHT_PROGBITS, SHF_ALLOC }, { STRING_COMMA_LEN (".PPC.EMB.sdata0"), 0, SHT_PROGBITS, SHF_ALLOC }, { NULL, 0, 0, 0, 0 } }; /* This is what we want for new plt/got. */ static struct bfd_elf_special_section ppc_alt_plt = { STRING_COMMA_LEN (".plt"), 0, SHT_PROGBITS, SHF_ALLOC }; static const struct bfd_elf_special_section * ppc_elf_get_sec_type_attr (bfd *abfd ATTRIBUTE_UNUSED, asection *sec) { const struct bfd_elf_special_section *ssect; /* See if this is one of the special sections. */ if (sec->name == NULL) return NULL; ssect = _bfd_elf_get_special_section (sec->name, ppc_elf_special_sections, sec->use_rela_p); if (ssect != NULL) { if (ssect == ppc_elf_special_sections && (sec->flags & SEC_LOAD) != 0) ssect = &ppc_alt_plt; return ssect; } return _bfd_elf_get_sec_type_attr (abfd, sec); } /* Very simple linked list structure for recording apuinfo values. */ typedef struct apuinfo_list { struct apuinfo_list *next; unsigned long value; } apuinfo_list; static apuinfo_list *head; static void apuinfo_list_init (void) { head = NULL; } static void apuinfo_list_add (unsigned long value) { apuinfo_list *entry = head; while (entry != NULL) { if (entry->value == value) return; entry = entry->next; } entry = bfd_malloc (sizeof (* entry)); if (entry == NULL) return; entry->value = value; entry->next = head; head = entry; } static unsigned apuinfo_list_length (void) { apuinfo_list *entry; unsigned long count; for (entry = head, count = 0; entry; entry = entry->next) ++ count; return count; } static inline unsigned long apuinfo_list_element (unsigned long number) { apuinfo_list * entry; for (entry = head; entry && number --; entry = entry->next) ; return entry ? entry->value : 0; } static void apuinfo_list_finish (void) { apuinfo_list *entry; for (entry = head; entry;) { apuinfo_list *next = entry->next; free (entry); entry = next; } head = NULL; } #define APUINFO_SECTION_NAME ".PPC.EMB.apuinfo" #define APUINFO_LABEL "APUinfo" /* Scan the input BFDs and create a linked list of the APUinfo values that will need to be emitted. */ static void ppc_elf_begin_write_processing (bfd *abfd, struct bfd_link_info *link_info) { bfd *ibfd; asection *asec; char *buffer; unsigned num_input_sections; bfd_size_type output_section_size; unsigned i; unsigned num_entries; unsigned long offset; unsigned long length; const char *error_message = NULL; if (link_info == NULL) return; /* Scan the input bfds, looking for apuinfo sections. */ num_input_sections = 0; output_section_size = 0; for (ibfd = link_info->input_bfds; ibfd; ibfd = ibfd->link_next) { asec = bfd_get_section_by_name (ibfd, APUINFO_SECTION_NAME); if (asec) { ++ num_input_sections; output_section_size += asec->size; } } /* We need at least one input sections in order to make merging worthwhile. */ if (num_input_sections < 1) return; /* Just make sure that the output section exists as well. */ asec = bfd_get_section_by_name (abfd, APUINFO_SECTION_NAME); if (asec == NULL) return; /* Allocate a buffer for the contents of the input sections. */ buffer = bfd_malloc (output_section_size); if (buffer == NULL) return; offset = 0; apuinfo_list_init (); /* Read in the input sections contents. */ for (ibfd = link_info->input_bfds; ibfd; ibfd = ibfd->link_next) { unsigned long datum; char *ptr; asec = bfd_get_section_by_name (ibfd, APUINFO_SECTION_NAME); if (asec == NULL) continue; length = asec->size; if (length < 24) { error_message = _("corrupt or empty %s section in %B"); goto fail; } if (bfd_seek (ibfd, asec->filepos, SEEK_SET) != 0 || (bfd_bread (buffer + offset, length, ibfd) != length)) { error_message = _("unable to read in %s section from %B"); goto fail; } /* Process the contents of the section. */ ptr = buffer + offset; error_message = _("corrupt %s section in %B"); /* Verify the contents of the header. Note - we have to extract the values this way in order to allow for a host whose endian-ness is different from the target. */ datum = bfd_get_32 (ibfd, ptr); if (datum != sizeof APUINFO_LABEL) goto fail; datum = bfd_get_32 (ibfd, ptr + 8); if (datum != 0x2) goto fail; if (strcmp (ptr + 12, APUINFO_LABEL) != 0) goto fail; /* Get the number of bytes used for apuinfo entries. */ datum = bfd_get_32 (ibfd, ptr + 4); if (datum + 20 != length) goto fail; /* Make sure that we do not run off the end of the section. */ if (offset + length > output_section_size) goto fail; /* Scan the apuinfo section, building a list of apuinfo numbers. */ for (i = 0; i < datum; i += 4) apuinfo_list_add (bfd_get_32 (ibfd, ptr + 20 + i)); /* Update the offset. */ offset += length; } error_message = NULL; /* Compute the size of the output section. */ num_entries = apuinfo_list_length (); output_section_size = 20 + num_entries * 4; asec = bfd_get_section_by_name (abfd, APUINFO_SECTION_NAME); if (! bfd_set_section_size (abfd, asec, output_section_size)) ibfd = abfd, error_message = _("warning: unable to set size of %s section in %B"); fail: free (buffer); if (error_message) (*_bfd_error_handler) (error_message, ibfd, APUINFO_SECTION_NAME); } /* Prevent the output section from accumulating the input sections' contents. We have already stored this in our linked list structure. */ static bfd_boolean ppc_elf_write_section (bfd *abfd ATTRIBUTE_UNUSED, struct bfd_link_info *link_info ATTRIBUTE_UNUSED, asection *asec, bfd_byte *contents ATTRIBUTE_UNUSED) { return (apuinfo_list_length () && strcmp (asec->name, APUINFO_SECTION_NAME) == 0); } /* Finally we can generate the output section. */ static void ppc_elf_final_write_processing (bfd *abfd, bfd_boolean linker ATTRIBUTE_UNUSED) { bfd_byte *buffer; asection *asec; unsigned i; unsigned num_entries; bfd_size_type length; asec = bfd_get_section_by_name (abfd, APUINFO_SECTION_NAME); if (asec == NULL) return; if (apuinfo_list_length () == 0) return; length = asec->size; if (length < 20) return; buffer = bfd_malloc (length); if (buffer == NULL) { (*_bfd_error_handler) (_("failed to allocate space for new APUinfo section.")); return; } /* Create the apuinfo header. */ num_entries = apuinfo_list_length (); bfd_put_32 (abfd, sizeof APUINFO_LABEL, buffer); bfd_put_32 (abfd, num_entries * 4, buffer + 4); bfd_put_32 (abfd, 0x2, buffer + 8); strcpy ((char *) buffer + 12, APUINFO_LABEL); length = 20; for (i = 0; i < num_entries; i++) { bfd_put_32 (abfd, apuinfo_list_element (i), buffer + length); length += 4; } if (length != asec->size) (*_bfd_error_handler) (_("failed to compute new APUinfo section.")); if (! bfd_set_section_contents (abfd, asec, buffer, (file_ptr) 0, length)) (*_bfd_error_handler) (_("failed to install new APUinfo section.")); free (buffer); apuinfo_list_finish (); } /* The following functions are specific to the ELF linker, while functions above are used generally. They appear in this file more or less in the order in which they are called. eg. ppc_elf_check_relocs is called early in the link process, ppc_elf_finish_dynamic_sections is one of the last functions called. */ /* The PPC linker needs to keep track of the number of relocs that it decides to copy as dynamic relocs in check_relocs for each symbol. This is so that it can later discard them if they are found to be unnecessary. We store the information in a field extending the regular ELF linker hash table. */ struct ppc_elf_dyn_relocs { struct ppc_elf_dyn_relocs *next; /* The input section of the reloc. */ asection *sec; /* Total number of relocs copied for the input section. */ bfd_size_type count; /* Number of pc-relative relocs copied for the input section. */ bfd_size_type pc_count; }; /* Track PLT entries needed for a given symbol. We might need more than one glink entry per symbol. */ struct plt_entry { struct plt_entry *next; /* -fPIC uses multiple GOT sections, one per file, called ".got2". This field stores the offset into .got2 used to initialise the GOT pointer reg. It will always be at least 32768 (and for current gcc this is the only offset used). */ bfd_vma addend; /* The .got2 section. */ asection *sec; /* PLT refcount or offset. */ union { bfd_signed_vma refcount; bfd_vma offset; } plt; /* .glink stub offset. */ bfd_vma glink_offset; }; -/* Of those relocs that might be copied as dynamic relocs, this macro +/* Of those relocs that might be copied as dynamic relocs, this function selects those that must be copied when linking a shared library, even when the symbol is local. */ -#define MUST_BE_DYN_RELOC(RTYPE) \ - ((RTYPE) != R_PPC_REL24 \ - && (RTYPE) != R_PPC_REL14 \ - && (RTYPE) != R_PPC_REL14_BRTAKEN \ - && (RTYPE) != R_PPC_REL14_BRNTAKEN \ - && (RTYPE) != R_PPC_REL32) +static int +must_be_dyn_reloc (struct bfd_link_info *info, + enum elf_ppc_reloc_type r_type) +{ + switch (r_type) + { + default: + return 1; + case R_PPC_REL24: + case R_PPC_REL14: + case R_PPC_REL14_BRTAKEN: + case R_PPC_REL14_BRNTAKEN: + case R_PPC_REL32: + return 0; + + case R_PPC_TPREL32: + case R_PPC_TPREL16: + case R_PPC_TPREL16_LO: + case R_PPC_TPREL16_HI: + case R_PPC_TPREL16_HA: + return !info->executable; + } +} + /* If ELIMINATE_COPY_RELOCS is non-zero, the linker will try to avoid copying dynamic variables from a shared lib into an app's dynbss section, and instead use a dynamic relocation to point into the shared lib. */ #define ELIMINATE_COPY_RELOCS 1 /* PPC ELF linker hash entry. */ struct ppc_elf_link_hash_entry { struct elf_link_hash_entry elf; /* If this symbol is used in the linker created sections, the processor specific backend uses this field to map the field into the offset from the beginning of the section. */ elf_linker_section_pointers_t *linker_section_pointer; /* Track dynamic relocs copied for this symbol. */ struct ppc_elf_dyn_relocs *dyn_relocs; /* Contexts in which symbol is used in the GOT (or TOC). TLS_GD .. TLS_TLS bits are or'd into the mask as the corresponding relocs are encountered during check_relocs. tls_optimize clears TLS_GD .. TLS_TPREL when optimizing to indicate the corresponding GOT entry type is not needed. */ #define TLS_GD 1 /* GD reloc. */ #define TLS_LD 2 /* LD reloc. */ #define TLS_TPREL 4 /* TPREL reloc, => IE. */ #define TLS_DTPREL 8 /* DTPREL reloc, => LD. */ #define TLS_TLS 16 /* Any TLS reloc. */ #define TLS_TPRELGD 32 /* TPREL reloc resulting from GD->IE. */ char tls_mask; /* Nonzero if we have seen a small data relocation referring to this symbol. */ unsigned char has_sda_refs; }; #define ppc_elf_hash_entry(ent) ((struct ppc_elf_link_hash_entry *) (ent)) /* PPC ELF linker hash table. */ struct ppc_elf_link_hash_table { struct elf_link_hash_table elf; /* Short-cuts to get to dynamic linker sections. */ asection *got; asection *relgot; asection *glink; asection *plt; asection *relplt; asection *dynbss; asection *relbss; asection *dynsbss; asection *relsbss; elf_linker_section_t sdata[2]; asection *sbss; /* The (unloaded but important) .rela.plt.unloaded on VxWorks. */ asection *srelplt2; /* The .got.plt section (VxWorks only)*/ asection *sgotplt; - /* Shortcut to .__tls_get_addr. */ + /* Shortcut to __tls_get_addr. */ struct elf_link_hash_entry *tls_get_addr; /* The bfd that forced an old-style PLT. */ bfd *old_bfd; /* TLS local dynamic got entry handling. */ union { bfd_signed_vma refcount; bfd_vma offset; } tlsld_got; /* Offset of PltResolve function in glink. */ bfd_vma glink_pltresolve; /* Size of reserved GOT entries. */ unsigned int got_header_size; /* Non-zero if allocating the header left a gap. */ unsigned int got_gap; /* The type of PLT we have chosen to use. */ enum ppc_elf_plt_type plt_type; /* Set if we should emit symbols for stubs. */ unsigned int emit_stub_syms:1; /* True if the target system is VxWorks. */ unsigned int is_vxworks:1; /* The size of PLT entries. */ int plt_entry_size; /* The distance between adjacent PLT slots. */ int plt_slot_size; /* The size of the first PLT entry. */ int plt_initial_entry_size; /* Small local sym to section mapping cache. */ struct sym_sec_cache sym_sec; }; /* Get the PPC ELF linker hash table from a link_info structure. */ #define ppc_elf_hash_table(p) \ ((struct ppc_elf_link_hash_table *) (p)->hash) /* Create an entry in a PPC ELF linker hash table. */ static struct bfd_hash_entry * ppc_elf_link_hash_newfunc (struct bfd_hash_entry *entry, struct bfd_hash_table *table, const char *string) { /* Allocate the structure if it has not already been allocated by a subclass. */ if (entry == NULL) { entry = bfd_hash_allocate (table, sizeof (struct ppc_elf_link_hash_entry)); if (entry == NULL) return entry; } /* Call the allocation method of the superclass. */ entry = _bfd_elf_link_hash_newfunc (entry, table, string); if (entry != NULL) { ppc_elf_hash_entry (entry)->linker_section_pointer = NULL; ppc_elf_hash_entry (entry)->dyn_relocs = NULL; ppc_elf_hash_entry (entry)->tls_mask = 0; } return entry; } /* Create a PPC ELF linker hash table. */ static struct bfd_link_hash_table * ppc_elf_link_hash_table_create (bfd *abfd) { struct ppc_elf_link_hash_table *ret; ret = bfd_zmalloc (sizeof (struct ppc_elf_link_hash_table)); if (ret == NULL) return NULL; if (!_bfd_elf_link_hash_table_init (&ret->elf, abfd, ppc_elf_link_hash_newfunc, sizeof (struct ppc_elf_link_hash_entry))) { free (ret); return NULL; } ret->elf.init_plt_refcount.refcount = 0; ret->elf.init_plt_refcount.glist = NULL; ret->elf.init_plt_offset.offset = 0; ret->elf.init_plt_offset.glist = NULL; ret->sdata[0].name = ".sdata"; ret->sdata[0].sym_name = "_SDA_BASE_"; ret->sdata[0].bss_name = ".sbss"; ret->sdata[1].name = ".sdata2"; ret->sdata[1].sym_name = "_SDA2_BASE_"; ret->sdata[1].bss_name = ".sbss2"; ret->plt_entry_size = 12; ret->plt_slot_size = 8; ret->plt_initial_entry_size = 72; return &ret->elf.root; } /* Create .got and the related sections. */ static bfd_boolean ppc_elf_create_got (bfd *abfd, struct bfd_link_info *info) { struct ppc_elf_link_hash_table *htab; asection *s; flagword flags; if (!_bfd_elf_create_got_section (abfd, info)) return FALSE; htab = ppc_elf_hash_table (info); htab->got = s = bfd_get_section_by_name (abfd, ".got"); if (s == NULL) abort (); if (htab->is_vxworks) { htab->sgotplt = bfd_get_section_by_name (abfd, ".got.plt"); if (!htab->sgotplt) abort (); } else { /* The powerpc .got has a blrl instruction in it. Mark it executable. */ flags = (SEC_ALLOC | SEC_LOAD | SEC_CODE | SEC_HAS_CONTENTS | SEC_IN_MEMORY | SEC_LINKER_CREATED); if (!bfd_set_section_flags (abfd, s, flags)) return FALSE; } flags = (SEC_ALLOC | SEC_LOAD | SEC_HAS_CONTENTS | SEC_IN_MEMORY | SEC_LINKER_CREATED | SEC_READONLY); htab->relgot = bfd_make_section_with_flags (abfd, ".rela.got", flags); if (!htab->relgot || ! bfd_set_section_alignment (abfd, htab->relgot, 2)) return FALSE; return TRUE; } /* We have to create .dynsbss and .rela.sbss here so that they get mapped to output sections (just like _bfd_elf_create_dynamic_sections has to create .dynbss and .rela.bss). */ static bfd_boolean ppc_elf_create_dynamic_sections (bfd *abfd, struct bfd_link_info *info) { struct ppc_elf_link_hash_table *htab; asection *s; flagword flags; htab = ppc_elf_hash_table (info); if (htab->got == NULL && !ppc_elf_create_got (abfd, info)) return FALSE; if (!_bfd_elf_create_dynamic_sections (abfd, info)) return FALSE; flags = (SEC_ALLOC | SEC_LOAD | SEC_READONLY | SEC_HAS_CONTENTS | SEC_IN_MEMORY | SEC_LINKER_CREATED); s = bfd_make_section_anyway_with_flags (abfd, ".glink", flags | SEC_CODE); htab->glink = s; if (s == NULL || !bfd_set_section_alignment (abfd, s, 4)) return FALSE; htab->dynbss = bfd_get_section_by_name (abfd, ".dynbss"); s = bfd_make_section_with_flags (abfd, ".dynsbss", SEC_ALLOC | SEC_LINKER_CREATED); htab->dynsbss = s; if (s == NULL) return FALSE; if (! info->shared) { htab->relbss = bfd_get_section_by_name (abfd, ".rela.bss"); s = bfd_make_section_with_flags (abfd, ".rela.sbss", flags); htab->relsbss = s; if (s == NULL || ! bfd_set_section_alignment (abfd, s, 2)) return FALSE; } if (htab->is_vxworks && !elf_vxworks_create_dynamic_sections (abfd, info, &htab->srelplt2)) return FALSE; htab->relplt = bfd_get_section_by_name (abfd, ".rela.plt"); htab->plt = s = bfd_get_section_by_name (abfd, ".plt"); if (s == NULL) abort (); flags = SEC_ALLOC | SEC_CODE | SEC_LINKER_CREATED; if (htab->plt_type == PLT_VXWORKS) /* The VxWorks PLT is a loaded section with contents. */ flags |= SEC_HAS_CONTENTS | SEC_LOAD | SEC_READONLY; return bfd_set_section_flags (abfd, s, flags); } /* Copy the extra info we tack onto an elf_link_hash_entry. */ static void ppc_elf_copy_indirect_symbol (struct bfd_link_info *info, struct elf_link_hash_entry *dir, struct elf_link_hash_entry *ind) { struct ppc_elf_link_hash_entry *edir, *eind; edir = (struct ppc_elf_link_hash_entry *) dir; eind = (struct ppc_elf_link_hash_entry *) ind; if (eind->dyn_relocs != NULL) { if (edir->dyn_relocs != NULL) { struct ppc_elf_dyn_relocs **pp; struct ppc_elf_dyn_relocs *p; /* Add reloc counts against the indirect sym to the direct sym list. Merge any entries against the same section. */ for (pp = &eind->dyn_relocs; (p = *pp) != NULL; ) { struct ppc_elf_dyn_relocs *q; for (q = edir->dyn_relocs; q != NULL; q = q->next) if (q->sec == p->sec) { q->pc_count += p->pc_count; q->count += p->count; *pp = p->next; break; } if (q == NULL) pp = &p->next; } *pp = edir->dyn_relocs; } edir->dyn_relocs = eind->dyn_relocs; eind->dyn_relocs = NULL; } edir->tls_mask |= eind->tls_mask; edir->has_sda_refs |= eind->has_sda_refs; /* If called to transfer flags for a weakdef during processing of elf_adjust_dynamic_symbol, don't copy non_got_ref. We clear it ourselves for ELIMINATE_COPY_RELOCS. */ if (!(ELIMINATE_COPY_RELOCS && eind->elf.root.type != bfd_link_hash_indirect && edir->elf.dynamic_adjusted)) edir->elf.non_got_ref |= eind->elf.non_got_ref; edir->elf.ref_dynamic |= eind->elf.ref_dynamic; edir->elf.ref_regular |= eind->elf.ref_regular; edir->elf.ref_regular_nonweak |= eind->elf.ref_regular_nonweak; edir->elf.needs_plt |= eind->elf.needs_plt; /* If we were called to copy over info for a weak sym, that's all. */ if (eind->elf.root.type != bfd_link_hash_indirect) return; /* Copy over the GOT refcount entries that we may have already seen to the symbol which just became indirect. */ edir->elf.got.refcount += eind->elf.got.refcount; eind->elf.got.refcount = 0; /* And plt entries. */ if (eind->elf.plt.plist != NULL) { if (edir->elf.plt.plist != NULL) { struct plt_entry **entp; struct plt_entry *ent; for (entp = &eind->elf.plt.plist; (ent = *entp) != NULL; ) { struct plt_entry *dent; for (dent = edir->elf.plt.plist; dent != NULL; dent = dent->next) if (dent->sec == ent->sec && dent->addend == ent->addend) { dent->plt.refcount += ent->plt.refcount; *entp = ent->next; break; } if (dent == NULL) entp = &ent->next; } *entp = edir->elf.plt.plist; } edir->elf.plt.plist = eind->elf.plt.plist; eind->elf.plt.plist = NULL; } if (eind->elf.dynindx != -1) { if (edir->elf.dynindx != -1) _bfd_elf_strtab_delref (elf_hash_table (info)->dynstr, edir->elf.dynstr_index); edir->elf.dynindx = eind->elf.dynindx; edir->elf.dynstr_index = eind->elf.dynstr_index; eind->elf.dynindx = -1; eind->elf.dynstr_index = 0; } } /* Return 1 if target is one of ours. */ static bfd_boolean is_ppc_elf_target (const struct bfd_target *targ) { extern const bfd_target bfd_elf32_powerpc_vec; extern const bfd_target bfd_elf32_powerpc_vxworks_vec; extern const bfd_target bfd_elf32_powerpcle_vec; return (targ == &bfd_elf32_powerpc_vec || targ == &bfd_elf32_powerpc_vxworks_vec || targ == &bfd_elf32_powerpcle_vec); } /* Hook called by the linker routine which adds symbols from an object file. We use it to put .comm items in .sbss, and not .bss. */ static bfd_boolean ppc_elf_add_symbol_hook (bfd *abfd, struct bfd_link_info *info, Elf_Internal_Sym *sym, const char **namep ATTRIBUTE_UNUSED, flagword *flagsp ATTRIBUTE_UNUSED, asection **secp, bfd_vma *valp) { if (sym->st_shndx == SHN_COMMON && !info->relocatable && sym->st_size <= elf_gp_size (abfd) && is_ppc_elf_target (info->hash->creator)) { /* Common symbols less than or equal to -G nn bytes are automatically put into .sbss. */ struct ppc_elf_link_hash_table *htab; htab = ppc_elf_hash_table (info); if (htab->sbss == NULL) { flagword flags = SEC_IS_COMMON | SEC_LINKER_CREATED; if (!htab->elf.dynobj) htab->elf.dynobj = abfd; htab->sbss = bfd_make_section_anyway_with_flags (htab->elf.dynobj, ".sbss", flags); if (htab->sbss == NULL) return FALSE; } *secp = htab->sbss; *valp = sym->st_size; } return TRUE; } static bfd_boolean create_sdata_sym (struct ppc_elf_link_hash_table *htab, elf_linker_section_t *lsect) { lsect->sym = elf_link_hash_lookup (&htab->elf, lsect->sym_name, TRUE, FALSE, TRUE); if (lsect->sym == NULL) return FALSE; if (lsect->sym->root.type == bfd_link_hash_new) lsect->sym->non_elf = 0; lsect->sym->ref_regular = 1; return TRUE; } /* Create a special linker section. */ static bfd_boolean ppc_elf_create_linker_section (bfd *abfd, struct bfd_link_info *info, flagword flags, elf_linker_section_t *lsect) { struct ppc_elf_link_hash_table *htab = ppc_elf_hash_table (info); asection *s; flags |= (SEC_ALLOC | SEC_LOAD | SEC_HAS_CONTENTS | SEC_IN_MEMORY | SEC_LINKER_CREATED); /* Record the first bfd that needs the special sections. */ if (!htab->elf.dynobj) htab->elf.dynobj = abfd; s = bfd_make_section_anyway_with_flags (htab->elf.dynobj, lsect->name, flags); if (s == NULL || !bfd_set_section_alignment (htab->elf.dynobj, s, 2)) return FALSE; lsect->section = s; return create_sdata_sym (htab, lsect); } /* Find a linker generated pointer with a given addend and type. */ static elf_linker_section_pointers_t * elf_find_pointer_linker_section (elf_linker_section_pointers_t *linker_pointers, bfd_vma addend, elf_linker_section_t *lsect) { for ( ; linker_pointers != NULL; linker_pointers = linker_pointers->next) if (lsect == linker_pointers->lsect && addend == linker_pointers->addend) return linker_pointers; return NULL; } /* Allocate a pointer to live in a linker created section. */ static bfd_boolean elf_create_pointer_linker_section (bfd *abfd, elf_linker_section_t *lsect, struct elf_link_hash_entry *h, const Elf_Internal_Rela *rel) { elf_linker_section_pointers_t **ptr_linker_section_ptr = NULL; elf_linker_section_pointers_t *linker_section_ptr; unsigned long r_symndx = ELF32_R_SYM (rel->r_info); bfd_size_type amt; BFD_ASSERT (lsect != NULL); /* Is this a global symbol? */ if (h != NULL) { struct ppc_elf_link_hash_entry *eh; /* Has this symbol already been allocated? If so, our work is done. */ eh = (struct ppc_elf_link_hash_entry *) h; if (elf_find_pointer_linker_section (eh->linker_section_pointer, rel->r_addend, lsect)) return TRUE; ptr_linker_section_ptr = &eh->linker_section_pointer; } else { /* Allocation of a pointer to a local symbol. */ elf_linker_section_pointers_t **ptr = elf_local_ptr_offsets (abfd); /* Allocate a table to hold the local symbols if first time. */ if (!ptr) { unsigned int num_symbols = elf_tdata (abfd)->symtab_hdr.sh_info; amt = num_symbols; amt *= sizeof (elf_linker_section_pointers_t *); ptr = bfd_zalloc (abfd, amt); if (!ptr) return FALSE; elf_local_ptr_offsets (abfd) = ptr; } /* Has this symbol already been allocated? If so, our work is done. */ if (elf_find_pointer_linker_section (ptr[r_symndx], rel->r_addend, lsect)) return TRUE; ptr_linker_section_ptr = &ptr[r_symndx]; } /* Allocate space for a pointer in the linker section, and allocate a new pointer record from internal memory. */ BFD_ASSERT (ptr_linker_section_ptr != NULL); amt = sizeof (elf_linker_section_pointers_t); linker_section_ptr = bfd_alloc (abfd, amt); if (!linker_section_ptr) return FALSE; linker_section_ptr->next = *ptr_linker_section_ptr; linker_section_ptr->addend = rel->r_addend; linker_section_ptr->lsect = lsect; *ptr_linker_section_ptr = linker_section_ptr; linker_section_ptr->offset = lsect->section->size; lsect->section->size += 4; #ifdef DEBUG fprintf (stderr, "Create pointer in linker section %s, offset = %ld, section size = %ld\n", lsect->name, (long) linker_section_ptr->offset, (long) lsect->section->size); #endif return TRUE; } static bfd_boolean update_local_sym_info (bfd *abfd, Elf_Internal_Shdr *symtab_hdr, unsigned long r_symndx, int tls_type) { bfd_signed_vma *local_got_refcounts = elf_local_got_refcounts (abfd); char *local_got_tls_masks; if (local_got_refcounts == NULL) { bfd_size_type size = symtab_hdr->sh_info; size *= sizeof (*local_got_refcounts) + sizeof (*local_got_tls_masks); local_got_refcounts = bfd_zalloc (abfd, size); if (local_got_refcounts == NULL) return FALSE; elf_local_got_refcounts (abfd) = local_got_refcounts; } local_got_refcounts[r_symndx] += 1; local_got_tls_masks = (char *) (local_got_refcounts + symtab_hdr->sh_info); local_got_tls_masks[r_symndx] |= tls_type; return TRUE; } static bfd_boolean update_plt_info (bfd *abfd, struct elf_link_hash_entry *h, asection *sec, bfd_vma addend) { struct plt_entry *ent; if (addend < 32768) sec = NULL; for (ent = h->plt.plist; ent != NULL; ent = ent->next) if (ent->sec == sec && ent->addend == addend) break; if (ent == NULL) { bfd_size_type amt = sizeof (*ent); ent = bfd_alloc (abfd, amt); if (ent == NULL) return FALSE; ent->next = h->plt.plist; ent->sec = sec; ent->addend = addend; ent->plt.refcount = 0; h->plt.plist = ent; } ent->plt.refcount += 1; return TRUE; } static struct plt_entry * find_plt_ent (struct elf_link_hash_entry *h, asection *sec, bfd_vma addend) { struct plt_entry *ent; if (addend < 32768) sec = NULL; for (ent = h->plt.plist; ent != NULL; ent = ent->next) if (ent->sec == sec && ent->addend == addend) break; return ent; } static void bad_shared_reloc (bfd *abfd, enum elf_ppc_reloc_type r_type) { (*_bfd_error_handler) (_("%B: relocation %s cannot be used when making a shared object"), abfd, ppc_elf_howto_table[r_type]->name); bfd_set_error (bfd_error_bad_value); } /* Look through the relocs for a section during the first phase, and allocate space in the global offset table or procedure linkage table. */ static bfd_boolean ppc_elf_check_relocs (bfd *abfd, struct bfd_link_info *info, asection *sec, const Elf_Internal_Rela *relocs) { struct ppc_elf_link_hash_table *htab; Elf_Internal_Shdr *symtab_hdr; struct elf_link_hash_entry **sym_hashes; const Elf_Internal_Rela *rel; const Elf_Internal_Rela *rel_end; asection *got2, *sreloc; + struct elf_link_hash_entry *tga; if (info->relocatable) return TRUE; /* Don't do anything special with non-loaded, non-alloced sections. In particular, any relocs in such sections should not affect GOT and PLT reference counting (ie. we don't allow them to create GOT or PLT entries), there's no possibility or desire to optimize TLS relocs, and there's not much point in propagating relocs to shared libs that the dynamic linker won't relocate. */ if ((sec->flags & SEC_ALLOC) == 0) return TRUE; #ifdef DEBUG _bfd_error_handler ("ppc_elf_check_relocs called for section %A in %B", sec, abfd); #endif /* Initialize howto table if not already done. */ if (!ppc_elf_howto_table[R_PPC_ADDR32]) ppc_elf_howto_init (); htab = ppc_elf_hash_table (info); + tga = elf_link_hash_lookup (&htab->elf, "__tls_get_addr", + FALSE, FALSE, TRUE); symtab_hdr = &elf_tdata (abfd)->symtab_hdr; sym_hashes = elf_sym_hashes (abfd); got2 = bfd_get_section_by_name (abfd, ".got2"); sreloc = NULL; rel_end = relocs + sec->reloc_count; for (rel = relocs; rel < rel_end; rel++) { unsigned long r_symndx; enum elf_ppc_reloc_type r_type; struct elf_link_hash_entry *h; - int tls_type = 0; + int tls_type; r_symndx = ELF32_R_SYM (rel->r_info); if (r_symndx < symtab_hdr->sh_info) h = NULL; else { h = sym_hashes[r_symndx - symtab_hdr->sh_info]; while (h->root.type == bfd_link_hash_indirect || h->root.type == bfd_link_hash_warning) h = (struct elf_link_hash_entry *) h->root.u.i.link; } /* If a relocation refers to _GLOBAL_OFFSET_TABLE_, create the .got. This shows up in particular in an R_PPC_ADDR32 in the eabi startup code. */ if (h != NULL && htab->got == NULL && strcmp (h->root.root.string, "_GLOBAL_OFFSET_TABLE_") == 0) { if (htab->elf.dynobj == NULL) htab->elf.dynobj = abfd; if (!ppc_elf_create_got (htab->elf.dynobj, info)) return FALSE; BFD_ASSERT (h == htab->elf.hgot); } + tls_type = 0; r_type = ELF32_R_TYPE (rel->r_info); + if (h != NULL && h == tga) + switch (r_type) + { + default: + break; + + case R_PPC_PLTREL24: + case R_PPC_LOCAL24PC: + case R_PPC_REL24: + case R_PPC_REL14: + case R_PPC_REL14_BRTAKEN: + case R_PPC_REL14_BRNTAKEN: + case R_PPC_ADDR24: + case R_PPC_ADDR14: + case R_PPC_ADDR14_BRTAKEN: + case R_PPC_ADDR14_BRNTAKEN: + if (rel != relocs + && (ELF32_R_TYPE (rel[-1].r_info) == R_PPC_TLSGD + || ELF32_R_TYPE (rel[-1].r_info) == R_PPC_TLSLD)) + /* We have a new-style __tls_get_addr call with a marker + reloc. */ + ; + else + /* Mark this section as having an old-style call. */ + sec->has_tls_get_addr_call = 1; + break; + } + switch (r_type) { + case R_PPC_TLSGD: + case R_PPC_TLSLD: + /* These special tls relocs tie a call to __tls_get_addr with + its parameter symbol. */ + break; + case R_PPC_GOT_TLSLD16: case R_PPC_GOT_TLSLD16_LO: case R_PPC_GOT_TLSLD16_HI: case R_PPC_GOT_TLSLD16_HA: - htab->tlsld_got.refcount += 1; tls_type = TLS_TLS | TLS_LD; goto dogottls; case R_PPC_GOT_TLSGD16: case R_PPC_GOT_TLSGD16_LO: case R_PPC_GOT_TLSGD16_HI: case R_PPC_GOT_TLSGD16_HA: tls_type = TLS_TLS | TLS_GD; goto dogottls; case R_PPC_GOT_TPREL16: case R_PPC_GOT_TPREL16_LO: case R_PPC_GOT_TPREL16_HI: case R_PPC_GOT_TPREL16_HA: - if (info->shared) + if (!info->executable) info->flags |= DF_STATIC_TLS; tls_type = TLS_TLS | TLS_TPREL; goto dogottls; case R_PPC_GOT_DTPREL16: case R_PPC_GOT_DTPREL16_LO: case R_PPC_GOT_DTPREL16_HI: case R_PPC_GOT_DTPREL16_HA: tls_type = TLS_TLS | TLS_DTPREL; dogottls: sec->has_tls_reloc = 1; /* Fall thru */ /* GOT16 relocations */ case R_PPC_GOT16: case R_PPC_GOT16_LO: case R_PPC_GOT16_HI: case R_PPC_GOT16_HA: /* This symbol requires a global offset table entry. */ if (htab->got == NULL) { if (htab->elf.dynobj == NULL) htab->elf.dynobj = abfd; if (!ppc_elf_create_got (htab->elf.dynobj, info)) return FALSE; } if (h != NULL) { h->got.refcount += 1; ppc_elf_hash_entry (h)->tls_mask |= tls_type; } else /* This is a global offset table entry for a local symbol. */ if (!update_local_sym_info (abfd, symtab_hdr, r_symndx, tls_type)) return FALSE; break; /* Indirect .sdata relocation. */ case R_PPC_EMB_SDAI16: if (info->shared) { bad_shared_reloc (abfd, r_type); return FALSE; } if (htab->sdata[0].section == NULL && !ppc_elf_create_linker_section (abfd, info, 0, &htab->sdata[0])) return FALSE; if (!elf_create_pointer_linker_section (abfd, &htab->sdata[0], h, rel)) return FALSE; if (h != NULL) { ppc_elf_hash_entry (h)->has_sda_refs = TRUE; h->non_got_ref = TRUE; } break; /* Indirect .sdata2 relocation. */ case R_PPC_EMB_SDA2I16: if (info->shared) { bad_shared_reloc (abfd, r_type); return FALSE; } if (htab->sdata[1].section == NULL && !ppc_elf_create_linker_section (abfd, info, SEC_READONLY, &htab->sdata[1])) return FALSE; if (!elf_create_pointer_linker_section (abfd, &htab->sdata[1], h, rel)) return FALSE; if (h != NULL) { ppc_elf_hash_entry (h)->has_sda_refs = TRUE; h->non_got_ref = TRUE; } break; case R_PPC_SDAREL16: if (info->shared) { bad_shared_reloc (abfd, r_type); return FALSE; } if (htab->sdata[0].sym == NULL && !create_sdata_sym (htab, &htab->sdata[0])) return FALSE; if (h != NULL) { ppc_elf_hash_entry (h)->has_sda_refs = TRUE; h->non_got_ref = TRUE; } break; case R_PPC_EMB_SDA2REL: if (info->shared) { bad_shared_reloc (abfd, r_type); return FALSE; } if (htab->sdata[1].sym == NULL && !create_sdata_sym (htab, &htab->sdata[1])) return FALSE; if (h != NULL) { ppc_elf_hash_entry (h)->has_sda_refs = TRUE; h->non_got_ref = TRUE; } break; case R_PPC_EMB_SDA21: case R_PPC_EMB_RELSDA: if (info->shared) { bad_shared_reloc (abfd, r_type); return FALSE; } if (htab->sdata[0].sym == NULL && !create_sdata_sym (htab, &htab->sdata[0])) return FALSE; if (htab->sdata[1].sym == NULL && !create_sdata_sym (htab, &htab->sdata[1])) return FALSE; if (h != NULL) { ppc_elf_hash_entry (h)->has_sda_refs = TRUE; h->non_got_ref = TRUE; } break; case R_PPC_EMB_NADDR32: case R_PPC_EMB_NADDR16: case R_PPC_EMB_NADDR16_LO: case R_PPC_EMB_NADDR16_HI: case R_PPC_EMB_NADDR16_HA: if (info->shared) { bad_shared_reloc (abfd, r_type); return FALSE; } if (h != NULL) h->non_got_ref = TRUE; break; case R_PPC_PLT32: case R_PPC_PLTREL24: case R_PPC_PLTREL32: case R_PPC_PLT16_LO: case R_PPC_PLT16_HI: case R_PPC_PLT16_HA: #ifdef DEBUG fprintf (stderr, "Reloc requires a PLT entry\n"); #endif /* This symbol requires a procedure linkage table entry. We actually build the entry in finish_dynamic_symbol, because this might be a case of linking PIC code without linking in any dynamic objects, in which case we don't need to generate a procedure linkage table after all. */ if (h == NULL) { /* It does not make sense to have a procedure linkage table entry for a local symbol. */ (*_bfd_error_handler) (_("%B(%A+0x%lx): %s reloc against " "local symbol"), abfd, sec, (long) rel->r_offset, ppc_elf_howto_table[r_type]->name); bfd_set_error (bfd_error_bad_value); return FALSE; } else { bfd_vma addend = 0; if (r_type == R_PPC_PLTREL24) { ppc_elf_tdata (abfd)->makes_plt_call = 1; addend = rel->r_addend; } h->needs_plt = 1; if (!update_plt_info (abfd, h, got2, addend)) return FALSE; } break; /* The following relocations don't need to propagate the relocation if linking a shared object since they are section relative. */ case R_PPC_SECTOFF: case R_PPC_SECTOFF_LO: case R_PPC_SECTOFF_HI: case R_PPC_SECTOFF_HA: case R_PPC_DTPREL16: case R_PPC_DTPREL16_LO: case R_PPC_DTPREL16_HI: case R_PPC_DTPREL16_HA: case R_PPC_TOC16: break; case R_PPC_REL16: case R_PPC_REL16_LO: case R_PPC_REL16_HI: case R_PPC_REL16_HA: ppc_elf_tdata (abfd)->has_rel16 = 1; break; /* These are just markers. */ case R_PPC_TLS: case R_PPC_EMB_MRKREF: case R_PPC_NONE: case R_PPC_max: break; /* These should only appear in dynamic objects. */ case R_PPC_COPY: case R_PPC_GLOB_DAT: case R_PPC_JMP_SLOT: case R_PPC_RELATIVE: break; /* These aren't handled yet. We'll report an error later. */ case R_PPC_ADDR30: case R_PPC_EMB_RELSEC16: case R_PPC_EMB_RELST_LO: case R_PPC_EMB_RELST_HI: case R_PPC_EMB_RELST_HA: case R_PPC_EMB_BIT_FLD: break; /* This refers only to functions defined in the shared library. */ case R_PPC_LOCAL24PC: - if (h && h == htab->elf.hgot && htab->plt_type == PLT_UNSET) + if (h != NULL && h == htab->elf.hgot && htab->plt_type == PLT_UNSET) { htab->plt_type = PLT_OLD; htab->old_bfd = abfd; } break; /* This relocation describes the C++ object vtable hierarchy. Reconstruct it for later use during GC. */ case R_PPC_GNU_VTINHERIT: if (!bfd_elf_gc_record_vtinherit (abfd, sec, h, rel->r_offset)) return FALSE; break; /* This relocation describes which C++ vtable entries are actually used. Record for later use during GC. */ case R_PPC_GNU_VTENTRY: if (!bfd_elf_gc_record_vtentry (abfd, sec, h, rel->r_addend)) return FALSE; break; /* We shouldn't really be seeing these. */ case R_PPC_TPREL32: - if (info->shared) + case R_PPC_TPREL16: + case R_PPC_TPREL16_LO: + case R_PPC_TPREL16_HI: + case R_PPC_TPREL16_HA: + if (!info->executable) info->flags |= DF_STATIC_TLS; goto dodyn; /* Nor these. */ case R_PPC_DTPMOD32: case R_PPC_DTPREL32: goto dodyn; - case R_PPC_TPREL16: - case R_PPC_TPREL16_LO: - case R_PPC_TPREL16_HI: - case R_PPC_TPREL16_HA: - if (info->shared) - info->flags |= DF_STATIC_TLS; - goto dodyn; - case R_PPC_REL32: if (h == NULL && got2 != NULL && (sec->flags & SEC_CODE) != 0 && (info->shared || info->pie) && htab->plt_type == PLT_UNSET) { /* Old -fPIC gcc code has .long LCTOC1-LCFx just before the start of a function, which assembles to a REL32 reference to .got2. If we detect one of these, then force the old PLT layout because the linker cannot reliably deduce the GOT pointer value needed for PLT call stubs. */ asection *s; s = bfd_section_from_r_symndx (abfd, &htab->sym_sec, sec, r_symndx); if (s == got2) { htab->plt_type = PLT_OLD; htab->old_bfd = abfd; } } if (h == NULL || h == htab->elf.hgot) break; goto dodyn1; case R_PPC_REL24: case R_PPC_REL14: case R_PPC_REL14_BRTAKEN: case R_PPC_REL14_BRNTAKEN: if (h == NULL) break; if (h == htab->elf.hgot) { if (htab->plt_type == PLT_UNSET) { htab->plt_type = PLT_OLD; htab->old_bfd = abfd; } break; } /* fall through */ case R_PPC_ADDR32: case R_PPC_ADDR24: case R_PPC_ADDR16: case R_PPC_ADDR16_LO: case R_PPC_ADDR16_HI: case R_PPC_ADDR16_HA: case R_PPC_ADDR14: case R_PPC_ADDR14_BRTAKEN: case R_PPC_ADDR14_BRNTAKEN: case R_PPC_UADDR32: case R_PPC_UADDR16: dodyn1: if (h != NULL && !info->shared) { /* We may need a plt entry if the symbol turns out to be a function defined in a dynamic object. */ if (!update_plt_info (abfd, h, NULL, 0)) return FALSE; /* We may need a copy reloc too. */ h->non_got_ref = 1; } dodyn: /* If we are creating a shared library, and this is a reloc against a global symbol, or a non PC relative reloc against a local symbol, then we need to copy the reloc into the shared library. However, if we are linking with -Bsymbolic, we do not need to copy a reloc against a global symbol which is defined in an object we are including in the link (i.e., DEF_REGULAR is set). At this point we have not seen all the input files, so it is possible that DEF_REGULAR is not set now but will be set later (it is never cleared). In case of a weak definition, DEF_REGULAR may be cleared later by a strong definition in a shared library. We account for that possibility below by storing information in the dyn_relocs field of the hash table entry. A similar situation occurs when creating shared libraries and symbol visibility changes render the symbol local. If on the other hand, we are creating an executable, we may need to keep relocations for symbols satisfied by a dynamic library if we manage to avoid copy relocs for the symbol. */ if ((info->shared - && (MUST_BE_DYN_RELOC (r_type) + && (must_be_dyn_reloc (info, r_type) || (h != NULL && (! info->symbolic || h->root.type == bfd_link_hash_defweak || !h->def_regular)))) || (ELIMINATE_COPY_RELOCS && !info->shared && h != NULL && (h->root.type == bfd_link_hash_defweak || !h->def_regular))) { struct ppc_elf_dyn_relocs *p; struct ppc_elf_dyn_relocs **head; #ifdef DEBUG fprintf (stderr, "ppc_elf_check_relocs needs to " "create relocation for %s\n", (h && h->root.root.string ? h->root.root.string : "")); #endif if (sreloc == NULL) { const char *name; name = (bfd_elf_string_from_elf_section (abfd, elf_elfheader (abfd)->e_shstrndx, elf_section_data (sec)->rel_hdr.sh_name)); if (name == NULL) return FALSE; BFD_ASSERT (CONST_STRNEQ (name, ".rela") && strcmp (bfd_get_section_name (abfd, sec), name + 5) == 0); if (htab->elf.dynobj == NULL) htab->elf.dynobj = abfd; sreloc = bfd_get_section_by_name (htab->elf.dynobj, name); if (sreloc == NULL) { flagword flags; flags = (SEC_HAS_CONTENTS | SEC_READONLY | SEC_IN_MEMORY | SEC_LINKER_CREATED | SEC_ALLOC | SEC_LOAD); sreloc = bfd_make_section_with_flags (htab->elf.dynobj, name, flags); if (sreloc == NULL || ! bfd_set_section_alignment (htab->elf.dynobj, sreloc, 2)) return FALSE; } elf_section_data (sec)->sreloc = sreloc; } /* If this is a global symbol, we count the number of relocations we need for this symbol. */ if (h != NULL) { head = &ppc_elf_hash_entry (h)->dyn_relocs; } else { /* Track dynamic relocs needed for local syms too. We really need local syms available to do this easily. Oh well. */ asection *s; void *vpp; s = bfd_section_from_r_symndx (abfd, &htab->sym_sec, sec, r_symndx); if (s == NULL) return FALSE; vpp = &elf_section_data (s)->local_dynrel; head = (struct ppc_elf_dyn_relocs **) vpp; } p = *head; if (p == NULL || p->sec != sec) { p = bfd_alloc (htab->elf.dynobj, sizeof *p); if (p == NULL) return FALSE; p->next = *head; *head = p; p->sec = sec; p->count = 0; p->pc_count = 0; } p->count += 1; - if (!MUST_BE_DYN_RELOC (r_type)) + if (!must_be_dyn_reloc (info, r_type)) p->pc_count += 1; } break; } } return TRUE; } /* Merge object attributes from IBFD into OBFD. Raise an error if there are conflicting attributes. */ static bfd_boolean ppc_elf_merge_obj_attributes (bfd *ibfd, bfd *obfd) { obj_attribute *in_attr; obj_attribute *out_attr; if (!elf_known_obj_attributes_proc (obfd)[0].i) { /* This is the first object. Copy the attributes. */ _bfd_elf_copy_obj_attributes (ibfd, obfd); /* Use the Tag_null value to indicate the attributes have been initialized. */ elf_known_obj_attributes_proc (obfd)[0].i = 1; return TRUE; } /* Check for conflicting Tag_GNU_Power_ABI_FP attributes and merge non-conflicting ones. */ in_attr = elf_known_obj_attributes (ibfd)[OBJ_ATTR_GNU]; out_attr = elf_known_obj_attributes (obfd)[OBJ_ATTR_GNU]; if (in_attr[Tag_GNU_Power_ABI_FP].i != out_attr[Tag_GNU_Power_ABI_FP].i) { out_attr[Tag_GNU_Power_ABI_FP].type = 1; if (out_attr[Tag_GNU_Power_ABI_FP].i == 0) out_attr[Tag_GNU_Power_ABI_FP].i = in_attr[Tag_GNU_Power_ABI_FP].i; else if (in_attr[Tag_GNU_Power_ABI_FP].i == 0) ; else if (out_attr[Tag_GNU_Power_ABI_FP].i == 1 && in_attr[Tag_GNU_Power_ABI_FP].i == 2) _bfd_error_handler (_("Warning: %B uses hard float, %B uses soft float"), obfd, ibfd); else if (out_attr[Tag_GNU_Power_ABI_FP].i == 2 && in_attr[Tag_GNU_Power_ABI_FP].i == 1) _bfd_error_handler (_("Warning: %B uses hard float, %B uses soft float"), ibfd, obfd); else if (in_attr[Tag_GNU_Power_ABI_FP].i > 2) _bfd_error_handler (_("Warning: %B uses unknown floating point ABI %d"), ibfd, in_attr[Tag_GNU_Power_ABI_FP].i); else _bfd_error_handler (_("Warning: %B uses unknown floating point ABI %d"), obfd, out_attr[Tag_GNU_Power_ABI_FP].i); } /* Merge Tag_compatibility attributes and any common GNU ones. */ _bfd_elf_merge_object_attributes (ibfd, obfd); return TRUE; } /* Merge backend specific data from an object file to the output object file when linking. */ static bfd_boolean ppc_elf_merge_private_bfd_data (bfd *ibfd, bfd *obfd) { flagword old_flags; flagword new_flags; bfd_boolean error; if (!is_ppc_elf_target (ibfd->xvec) || !is_ppc_elf_target (obfd->xvec)) return TRUE; /* Check if we have the same endianess. */ if (! _bfd_generic_verify_endian_match (ibfd, obfd)) return FALSE; if (!ppc_elf_merge_obj_attributes (ibfd, obfd)) return FALSE; new_flags = elf_elfheader (ibfd)->e_flags; old_flags = elf_elfheader (obfd)->e_flags; if (!elf_flags_init (obfd)) { /* First call, no flags set. */ elf_flags_init (obfd) = TRUE; elf_elfheader (obfd)->e_flags = new_flags; } /* Compatible flags are ok. */ else if (new_flags == old_flags) ; /* Incompatible flags. */ else { /* Warn about -mrelocatable mismatch. Allow -mrelocatable-lib to be linked with either. */ error = FALSE; if ((new_flags & EF_PPC_RELOCATABLE) != 0 && (old_flags & (EF_PPC_RELOCATABLE | EF_PPC_RELOCATABLE_LIB)) == 0) { error = TRUE; (*_bfd_error_handler) (_("%B: compiled with -mrelocatable and linked with " "modules compiled normally"), ibfd); } else if ((new_flags & (EF_PPC_RELOCATABLE | EF_PPC_RELOCATABLE_LIB)) == 0 && (old_flags & EF_PPC_RELOCATABLE) != 0) { error = TRUE; (*_bfd_error_handler) (_("%B: compiled normally and linked with " "modules compiled with -mrelocatable"), ibfd); } /* The output is -mrelocatable-lib iff both the input files are. */ if (! (new_flags & EF_PPC_RELOCATABLE_LIB)) elf_elfheader (obfd)->e_flags &= ~EF_PPC_RELOCATABLE_LIB; /* The output is -mrelocatable iff it can't be -mrelocatable-lib, but each input file is either -mrelocatable or -mrelocatable-lib. */ if (! (elf_elfheader (obfd)->e_flags & EF_PPC_RELOCATABLE_LIB) && (new_flags & (EF_PPC_RELOCATABLE_LIB | EF_PPC_RELOCATABLE)) && (old_flags & (EF_PPC_RELOCATABLE_LIB | EF_PPC_RELOCATABLE))) elf_elfheader (obfd)->e_flags |= EF_PPC_RELOCATABLE; /* Do not warn about eabi vs. V.4 mismatch, just or in the bit if any module uses it. */ elf_elfheader (obfd)->e_flags |= (new_flags & EF_PPC_EMB); new_flags &= ~(EF_PPC_RELOCATABLE | EF_PPC_RELOCATABLE_LIB | EF_PPC_EMB); old_flags &= ~(EF_PPC_RELOCATABLE | EF_PPC_RELOCATABLE_LIB | EF_PPC_EMB); /* Warn about any other mismatches. */ if (new_flags != old_flags) { error = TRUE; (*_bfd_error_handler) (_("%B: uses different e_flags (0x%lx) fields " "than previous modules (0x%lx)"), ibfd, (long) new_flags, (long) old_flags); } if (error) { bfd_set_error (bfd_error_bad_value); return FALSE; } } return TRUE; } /* Choose which PLT scheme to use, and set .plt flags appropriately. Returns -1 on error, 0 for old PLT, 1 for new PLT. */ int ppc_elf_select_plt_layout (bfd *output_bfd ATTRIBUTE_UNUSED, struct bfd_link_info *info, enum ppc_elf_plt_type plt_style, int emit_stub_syms) { struct ppc_elf_link_hash_table *htab; flagword flags; htab = ppc_elf_hash_table (info); if (htab->plt_type == PLT_UNSET) { if (plt_style == PLT_OLD) htab->plt_type = PLT_OLD; else { bfd *ibfd; enum ppc_elf_plt_type plt_type = plt_style; /* Look through the reloc flags left by ppc_elf_check_relocs. Use the old style bss plt if a file makes plt calls without using the new relocs, and if ld isn't given --secure-plt and we never see REL16 relocs. */ if (plt_type == PLT_UNSET) plt_type = PLT_OLD; for (ibfd = info->input_bfds; ibfd; ibfd = ibfd->link_next) if (is_ppc_elf_target (ibfd->xvec)) { if (ppc_elf_tdata (ibfd)->has_rel16) plt_type = PLT_NEW; else if (ppc_elf_tdata (ibfd)->makes_plt_call) { plt_type = PLT_OLD; htab->old_bfd = ibfd; break; } } htab->plt_type = plt_type; } } if (htab->plt_type == PLT_OLD && plt_style == PLT_NEW) info->callbacks->info (_("Using bss-plt due to %B"), htab->old_bfd); htab->emit_stub_syms = emit_stub_syms; BFD_ASSERT (htab->plt_type != PLT_VXWORKS); if (htab->plt_type == PLT_NEW) { flags = (SEC_ALLOC | SEC_LOAD | SEC_HAS_CONTENTS | SEC_IN_MEMORY | SEC_LINKER_CREATED); /* The new PLT is a loaded section. */ if (htab->plt != NULL && !bfd_set_section_flags (htab->elf.dynobj, htab->plt, flags)) return -1; /* The new GOT is not executable. */ if (htab->got != NULL && !bfd_set_section_flags (htab->elf.dynobj, htab->got, flags)) return -1; } else { /* Stop an unused .glink section from affecting .text alignment. */ if (htab->glink != NULL && !bfd_set_section_alignment (htab->elf.dynobj, htab->glink, 0)) return -1; } return htab->plt_type == PLT_NEW; } /* Return the section that should be marked against GC for a given relocation. */ static asection * ppc_elf_gc_mark_hook (asection *sec, struct bfd_link_info *info, Elf_Internal_Rela *rel, struct elf_link_hash_entry *h, Elf_Internal_Sym *sym) { if (h != NULL) switch (ELF32_R_TYPE (rel->r_info)) { case R_PPC_GNU_VTINHERIT: case R_PPC_GNU_VTENTRY: return NULL; } return _bfd_elf_gc_mark_hook (sec, info, rel, h, sym); } /* Update the got, plt and dynamic reloc reference counts for the section being removed. */ static bfd_boolean ppc_elf_gc_sweep_hook (bfd *abfd, struct bfd_link_info *info, asection *sec, const Elf_Internal_Rela *relocs) { struct ppc_elf_link_hash_table *htab; Elf_Internal_Shdr *symtab_hdr; struct elf_link_hash_entry **sym_hashes; bfd_signed_vma *local_got_refcounts; const Elf_Internal_Rela *rel, *relend; asection *got2; if ((sec->flags & SEC_ALLOC) == 0) return TRUE; elf_section_data (sec)->local_dynrel = NULL; htab = ppc_elf_hash_table (info); symtab_hdr = &elf_tdata (abfd)->symtab_hdr; sym_hashes = elf_sym_hashes (abfd); local_got_refcounts = elf_local_got_refcounts (abfd); got2 = bfd_get_section_by_name (abfd, ".got2"); relend = relocs + sec->reloc_count; for (rel = relocs; rel < relend; rel++) { unsigned long r_symndx; enum elf_ppc_reloc_type r_type; struct elf_link_hash_entry *h = NULL; r_symndx = ELF32_R_SYM (rel->r_info); if (r_symndx >= symtab_hdr->sh_info) { struct ppc_elf_dyn_relocs **pp, *p; struct ppc_elf_link_hash_entry *eh; h = sym_hashes[r_symndx - symtab_hdr->sh_info]; while (h->root.type == bfd_link_hash_indirect || h->root.type == bfd_link_hash_warning) h = (struct elf_link_hash_entry *) h->root.u.i.link; eh = (struct ppc_elf_link_hash_entry *) h; for (pp = &eh->dyn_relocs; (p = *pp) != NULL; pp = &p->next) if (p->sec == sec) { /* Everything must go for SEC. */ *pp = p->next; break; } } r_type = ELF32_R_TYPE (rel->r_info); switch (r_type) { case R_PPC_GOT_TLSLD16: case R_PPC_GOT_TLSLD16_LO: case R_PPC_GOT_TLSLD16_HI: case R_PPC_GOT_TLSLD16_HA: - htab->tlsld_got.refcount -= 1; - /* Fall thru */ - case R_PPC_GOT_TLSGD16: case R_PPC_GOT_TLSGD16_LO: case R_PPC_GOT_TLSGD16_HI: case R_PPC_GOT_TLSGD16_HA: case R_PPC_GOT_TPREL16: case R_PPC_GOT_TPREL16_LO: case R_PPC_GOT_TPREL16_HI: case R_PPC_GOT_TPREL16_HA: case R_PPC_GOT_DTPREL16: case R_PPC_GOT_DTPREL16_LO: case R_PPC_GOT_DTPREL16_HI: case R_PPC_GOT_DTPREL16_HA: case R_PPC_GOT16: case R_PPC_GOT16_LO: case R_PPC_GOT16_HI: case R_PPC_GOT16_HA: if (h != NULL) { if (h->got.refcount > 0) h->got.refcount--; } else if (local_got_refcounts != NULL) { if (local_got_refcounts[r_symndx] > 0) local_got_refcounts[r_symndx]--; } break; case R_PPC_REL24: case R_PPC_REL14: case R_PPC_REL14_BRTAKEN: case R_PPC_REL14_BRNTAKEN: case R_PPC_REL32: if (h == NULL || h == htab->elf.hgot) break; /* Fall thru */ case R_PPC_ADDR32: case R_PPC_ADDR24: case R_PPC_ADDR16: case R_PPC_ADDR16_LO: case R_PPC_ADDR16_HI: case R_PPC_ADDR16_HA: case R_PPC_ADDR14: case R_PPC_ADDR14_BRTAKEN: case R_PPC_ADDR14_BRNTAKEN: case R_PPC_UADDR32: case R_PPC_UADDR16: if (info->shared) break; case R_PPC_PLT32: case R_PPC_PLTREL24: case R_PPC_PLTREL32: case R_PPC_PLT16_LO: case R_PPC_PLT16_HI: case R_PPC_PLT16_HA: if (h != NULL) { bfd_vma addend = r_type == R_PPC_PLTREL24 ? rel->r_addend : 0; struct plt_entry *ent = find_plt_ent (h, got2, addend); if (ent->plt.refcount > 0) ent->plt.refcount -= 1; } break; default: break; } } return TRUE; } -/* Set htab->tls_get_addr and call the generic ELF tls_setup function. */ +/* Set plt output section type, htab->tls_get_addr, and call the + generic ELF tls_setup function. */ asection * ppc_elf_tls_setup (bfd *obfd, struct bfd_link_info *info) { struct ppc_elf_link_hash_table *htab; htab = ppc_elf_hash_table (info); if (htab->plt_type == PLT_NEW && htab->plt != NULL && htab->plt->output_section != NULL) { elf_section_type (htab->plt->output_section) = SHT_PROGBITS; elf_section_flags (htab->plt->output_section) = SHF_ALLOC + SHF_WRITE; } htab->tls_get_addr = elf_link_hash_lookup (&htab->elf, "__tls_get_addr", FALSE, FALSE, TRUE); return _bfd_elf_tls_setup (obfd, info); } +/* Return TRUE iff REL is a branch reloc with a global symbol matching + HASH. */ + +static bfd_boolean +branch_reloc_hash_match (const bfd *ibfd, + const Elf_Internal_Rela *rel, + const struct elf_link_hash_entry *hash) +{ + Elf_Internal_Shdr *symtab_hdr = &elf_tdata (ibfd)->symtab_hdr; + enum elf_ppc_reloc_type r_type = ELF32_R_TYPE (rel->r_info); + unsigned int r_symndx = ELF32_R_SYM (rel->r_info); + + if (r_symndx >= symtab_hdr->sh_info + && (r_type == R_PPC_PLTREL24 + || r_type == R_PPC_LOCAL24PC + || r_type == R_PPC_REL14 + || r_type == R_PPC_REL14_BRTAKEN + || r_type == R_PPC_REL14_BRNTAKEN + || r_type == R_PPC_REL24 + || r_type == R_PPC_ADDR24 + || r_type == R_PPC_ADDR14 + || r_type == R_PPC_ADDR14_BRTAKEN + || r_type == R_PPC_ADDR14_BRNTAKEN)) + { + struct elf_link_hash_entry **sym_hashes = elf_sym_hashes (ibfd); + struct elf_link_hash_entry *h; + + h = sym_hashes[r_symndx - symtab_hdr->sh_info]; + while (h->root.type == bfd_link_hash_indirect + || h->root.type == bfd_link_hash_warning) + h = (struct elf_link_hash_entry *) h->root.u.i.link; + if (h == hash) + return TRUE; + } + return FALSE; +} + /* Run through all the TLS relocs looking for optimization opportunities. */ bfd_boolean ppc_elf_tls_optimize (bfd *obfd ATTRIBUTE_UNUSED, struct bfd_link_info *info) { bfd *ibfd; asection *sec; struct ppc_elf_link_hash_table *htab; + int pass; - if (info->relocatable || info->shared) + if (info->relocatable || !info->executable) return TRUE; htab = ppc_elf_hash_table (info); - for (ibfd = info->input_bfds; ibfd != NULL; ibfd = ibfd->link_next) - { - Elf_Internal_Sym *locsyms = NULL; - Elf_Internal_Shdr *symtab_hdr = &elf_tdata (ibfd)->symtab_hdr; + /* Make two passes through the relocs. First time check that tls + relocs involved in setting up a tls_get_addr call are indeed + followed by such a call. If they are not, exclude them from + the optimizations done on the second pass. */ + for (pass = 0; pass < 2; ++pass) + for (ibfd = info->input_bfds; ibfd != NULL; ibfd = ibfd->link_next) + { + Elf_Internal_Sym *locsyms = NULL; + Elf_Internal_Shdr *symtab_hdr = &elf_tdata (ibfd)->symtab_hdr; - for (sec = ibfd->sections; sec != NULL; sec = sec->next) - if (sec->has_tls_reloc && !bfd_is_abs_section (sec->output_section)) - { - Elf_Internal_Rela *relstart, *rel, *relend; - int expecting_tls_get_addr; + for (sec = ibfd->sections; sec != NULL; sec = sec->next) + if (sec->has_tls_reloc && !bfd_is_abs_section (sec->output_section)) + { + Elf_Internal_Rela *relstart, *rel, *relend; - /* Read the relocations. */ - relstart = _bfd_elf_link_read_relocs (ibfd, sec, NULL, NULL, - info->keep_memory); - if (relstart == NULL) - return FALSE; + /* Read the relocations. */ + relstart = _bfd_elf_link_read_relocs (ibfd, sec, NULL, NULL, + info->keep_memory); + if (relstart == NULL) + return FALSE; - expecting_tls_get_addr = 0; - relend = relstart + sec->reloc_count; - for (rel = relstart; rel < relend; rel++) - { - enum elf_ppc_reloc_type r_type; - unsigned long r_symndx; - struct elf_link_hash_entry *h = NULL; - char *tls_mask; - char tls_set, tls_clear; - bfd_boolean is_local; + relend = relstart + sec->reloc_count; + for (rel = relstart; rel < relend; rel++) + { + enum elf_ppc_reloc_type r_type; + unsigned long r_symndx; + struct elf_link_hash_entry *h = NULL; + char *tls_mask; + char tls_set, tls_clear; + bfd_boolean is_local; + int expecting_tls_get_addr; + bfd_signed_vma *got_count; - r_symndx = ELF32_R_SYM (rel->r_info); - if (r_symndx >= symtab_hdr->sh_info) - { - struct elf_link_hash_entry **sym_hashes; + r_symndx = ELF32_R_SYM (rel->r_info); + if (r_symndx >= symtab_hdr->sh_info) + { + struct elf_link_hash_entry **sym_hashes; - sym_hashes = elf_sym_hashes (ibfd); - h = sym_hashes[r_symndx - symtab_hdr->sh_info]; - while (h->root.type == bfd_link_hash_indirect - || h->root.type == bfd_link_hash_warning) - h = (struct elf_link_hash_entry *) h->root.u.i.link; - } + sym_hashes = elf_sym_hashes (ibfd); + h = sym_hashes[r_symndx - symtab_hdr->sh_info]; + while (h->root.type == bfd_link_hash_indirect + || h->root.type == bfd_link_hash_warning) + h = (struct elf_link_hash_entry *) h->root.u.i.link; + } - is_local = FALSE; - if (h == NULL - || !h->def_dynamic) - is_local = TRUE; + expecting_tls_get_addr = 0; + is_local = FALSE; + if (h == NULL + || !h->def_dynamic) + is_local = TRUE; - r_type = ELF32_R_TYPE (rel->r_info); - switch (r_type) - { - case R_PPC_GOT_TLSLD16: - case R_PPC_GOT_TLSLD16_LO: - case R_PPC_GOT_TLSLD16_HI: - case R_PPC_GOT_TLSLD16_HA: - /* These relocs should never be against a symbol - defined in a shared lib. Leave them alone if - that turns out to be the case. */ - expecting_tls_get_addr = 0; - htab->tlsld_got.refcount -= 1; - if (!is_local) - continue; + r_type = ELF32_R_TYPE (rel->r_info); + switch (r_type) + { + case R_PPC_GOT_TLSLD16: + case R_PPC_GOT_TLSLD16_LO: + expecting_tls_get_addr = 1; + /* Fall thru */ - /* LD -> LE */ - tls_set = 0; - tls_clear = TLS_LD; - expecting_tls_get_addr = 1; - break; + case R_PPC_GOT_TLSLD16_HI: + case R_PPC_GOT_TLSLD16_HA: + /* These relocs should never be against a symbol + defined in a shared lib. Leave them alone if + that turns out to be the case. */ + if (!is_local) + continue; - case R_PPC_GOT_TLSGD16: - case R_PPC_GOT_TLSGD16_LO: - case R_PPC_GOT_TLSGD16_HI: - case R_PPC_GOT_TLSGD16_HA: - if (is_local) - /* GD -> LE */ + /* LD -> LE */ tls_set = 0; - else - /* GD -> IE */ - tls_set = TLS_TLS | TLS_TPRELGD; - tls_clear = TLS_GD; - expecting_tls_get_addr = 1; - break; + tls_clear = TLS_LD; + break; - case R_PPC_GOT_TPREL16: - case R_PPC_GOT_TPREL16_LO: - case R_PPC_GOT_TPREL16_HI: - case R_PPC_GOT_TPREL16_HA: - expecting_tls_get_addr = 0; - if (is_local) - { - /* IE -> LE */ + case R_PPC_GOT_TLSGD16: + case R_PPC_GOT_TLSGD16_LO: + expecting_tls_get_addr = 1; + /* Fall thru */ + + case R_PPC_GOT_TLSGD16_HI: + case R_PPC_GOT_TLSGD16_HA: + if (is_local) + /* GD -> LE */ tls_set = 0; - tls_clear = TLS_TPREL; - break; - } - else + else + /* GD -> IE */ + tls_set = TLS_TLS | TLS_TPRELGD; + tls_clear = TLS_GD; + break; + + case R_PPC_GOT_TPREL16: + case R_PPC_GOT_TPREL16_LO: + case R_PPC_GOT_TPREL16_HI: + case R_PPC_GOT_TPREL16_HA: + if (is_local) + { + /* IE -> LE */ + tls_set = 0; + tls_clear = TLS_TPREL; + break; + } + else + continue; + + default: continue; + } - case R_PPC_REL14: - case R_PPC_REL14_BRTAKEN: - case R_PPC_REL14_BRNTAKEN: - case R_PPC_REL24: - if (expecting_tls_get_addr - && h != NULL - && h == htab->tls_get_addr) - { - struct plt_entry *ent = find_plt_ent (h, NULL, 0); - if (ent != NULL && ent->plt.refcount > 0) - ent->plt.refcount -= 1; - } - expecting_tls_get_addr = 0; - continue; + if (pass == 0) + { + if (!expecting_tls_get_addr + || !sec->has_tls_get_addr_call) + continue; - default: - expecting_tls_get_addr = 0; - continue; - } + if (rel + 1 < relend + && branch_reloc_hash_match (ibfd, rel + 1, + htab->tls_get_addr)) + continue; - if (h != NULL) - { - if (tls_set == 0) - { - /* We managed to get rid of a got entry. */ - if (h->got.refcount > 0) - h->got.refcount -= 1; - } - tls_mask = &ppc_elf_hash_entry (h)->tls_mask; - } - else - { - Elf_Internal_Sym *sym; - bfd_signed_vma *lgot_refs; - char *lgot_masks; + /* Uh oh, we didn't find the expected call. We + could just mark this symbol to exclude it + from tls optimization but it's safer to skip + the entire section. */ + sec->has_tls_reloc = 0; + break; + } - if (locsyms == NULL) - { - locsyms = (Elf_Internal_Sym *) symtab_hdr->contents; - if (locsyms == NULL) - locsyms = bfd_elf_get_elf_syms (ibfd, symtab_hdr, - symtab_hdr->sh_info, - 0, NULL, NULL, NULL); - if (locsyms == NULL) - { - if (elf_section_data (sec)->relocs != relstart) - free (relstart); - return FALSE; - } - } - sym = locsyms + r_symndx; - lgot_refs = elf_local_got_refcounts (ibfd); - if (lgot_refs == NULL) - abort (); - if (tls_set == 0) - { - /* We managed to get rid of a got entry. */ - if (lgot_refs[r_symndx] > 0) - lgot_refs[r_symndx] -= 1; - } - lgot_masks = (char *) (lgot_refs + symtab_hdr->sh_info); - tls_mask = &lgot_masks[r_symndx]; - } + if (h != NULL) + { + tls_mask = &ppc_elf_hash_entry (h)->tls_mask; + got_count = &h->got.refcount; + } + else + { + Elf_Internal_Sym *sym; + bfd_signed_vma *lgot_refs; + char *lgot_masks; - *tls_mask |= tls_set; - *tls_mask &= ~tls_clear; - } + if (locsyms == NULL) + { + locsyms = (Elf_Internal_Sym *) symtab_hdr->contents; + if (locsyms == NULL) + locsyms = bfd_elf_get_elf_syms (ibfd, symtab_hdr, + symtab_hdr->sh_info, + 0, NULL, NULL, NULL); + if (locsyms == NULL) + { + if (elf_section_data (sec)->relocs != relstart) + free (relstart); + return FALSE; + } + } + sym = locsyms + r_symndx; + lgot_refs = elf_local_got_refcounts (ibfd); + if (lgot_refs == NULL) + abort (); + lgot_masks = (char *) (lgot_refs + symtab_hdr->sh_info); + tls_mask = &lgot_masks[r_symndx]; + got_count = &lgot_refs[r_symndx]; + } - if (elf_section_data (sec)->relocs != relstart) - free (relstart); - } + if (tls_set == 0) + { + /* We managed to get rid of a got entry. */ + if (*got_count > 0) + *got_count -= 1; + } - if (locsyms != NULL - && (symtab_hdr->contents != (unsigned char *) locsyms)) - { - if (!info->keep_memory) - free (locsyms); - else - symtab_hdr->contents = (unsigned char *) locsyms; - } - } + if (expecting_tls_get_addr) + { + struct plt_entry *ent; + + ent = find_plt_ent (htab->tls_get_addr, NULL, 0); + if (ent != NULL && ent->plt.refcount > 0) + ent->plt.refcount -= 1; + } + + *tls_mask |= tls_set; + *tls_mask &= ~tls_clear; + } + + if (elf_section_data (sec)->relocs != relstart) + free (relstart); + } + + if (locsyms != NULL + && (symtab_hdr->contents != (unsigned char *) locsyms)) + { + if (!info->keep_memory) + free (locsyms); + else + symtab_hdr->contents = (unsigned char *) locsyms; + } + } return TRUE; } /* Adjust a symbol defined by a dynamic object and referenced by a regular object. The current definition is in some section of the dynamic object, but we're not including those sections. We have to change the definition to something the rest of the link can understand. */ static bfd_boolean ppc_elf_adjust_dynamic_symbol (struct bfd_link_info *info, struct elf_link_hash_entry *h) { struct ppc_elf_link_hash_table *htab; asection *s; #ifdef DEBUG fprintf (stderr, "ppc_elf_adjust_dynamic_symbol called for %s\n", h->root.root.string); #endif /* Make sure we know what is going on here. */ htab = ppc_elf_hash_table (info); BFD_ASSERT (htab->elf.dynobj != NULL && (h->needs_plt || h->u.weakdef != NULL || (h->def_dynamic && h->ref_regular && !h->def_regular))); /* Deal with function syms. */ if (h->type == STT_FUNC || h->needs_plt) { /* Clear procedure linkage table information for any symbol that won't need a .plt entry. */ struct plt_entry *ent; for (ent = h->plt.plist; ent != NULL; ent = ent->next) if (ent->plt.refcount > 0) break; if (ent == NULL || SYMBOL_CALLS_LOCAL (info, h) || (ELF_ST_VISIBILITY (h->other) != STV_DEFAULT && h->root.type == bfd_link_hash_undefweak)) { /* A PLT entry is not required/allowed when: 1. We are not using ld.so; because then the PLT entry can't be set up, so we can't use one. In this case, ppc_elf_adjust_dynamic_symbol won't even be called. 2. GC has rendered the entry unused. 3. We know for certain that a call to this symbol will go to this object, or will remain undefined. */ h->plt.plist = NULL; h->needs_plt = 0; } return TRUE; } else h->plt.plist = NULL; /* If this is a weak symbol, and there is a real definition, the processor independent code will have arranged for us to see the real definition first, and we can just use the same value. */ if (h->u.weakdef != NULL) { BFD_ASSERT (h->u.weakdef->root.type == bfd_link_hash_defined || h->u.weakdef->root.type == bfd_link_hash_defweak); h->root.u.def.section = h->u.weakdef->root.u.def.section; h->root.u.def.value = h->u.weakdef->root.u.def.value; if (ELIMINATE_COPY_RELOCS) h->non_got_ref = h->u.weakdef->non_got_ref; return TRUE; } /* This is a reference to a symbol defined by a dynamic object which is not a function. */ /* If we are creating a shared library, we must presume that the only references to the symbol are via the global offset table. For such cases we need not do anything here; the relocations will be handled correctly by relocate_section. */ if (info->shared) return TRUE; /* If there are no references to this symbol that do not use the GOT, we don't need to generate a copy reloc. */ if (!h->non_got_ref) return TRUE; /* If we didn't find any dynamic relocs in read-only sections, then we'll be keeping the dynamic relocs and avoiding the copy reloc. We can't do this if there are any small data relocations. */ if (ELIMINATE_COPY_RELOCS && !ppc_elf_hash_entry (h)->has_sda_refs) { struct ppc_elf_dyn_relocs *p; for (p = ppc_elf_hash_entry (h)->dyn_relocs; p != NULL; p = p->next) { s = p->sec->output_section; if (s != NULL && (s->flags & SEC_READONLY) != 0) break; } if (p == NULL) { h->non_got_ref = 0; return TRUE; } } if (h->size == 0) { (*_bfd_error_handler) (_("dynamic variable `%s' is zero size"), h->root.root.string); return TRUE; } /* We must allocate the symbol in our .dynbss section, which will become part of the .bss section of the executable. There will be an entry for this symbol in the .dynsym section. The dynamic object will contain position independent code, so all references from the dynamic object to this symbol will go through the global offset table. The dynamic linker will use the .dynsym entry to determine the address it must put in the global offset table, so both the dynamic object and the regular object will refer to the same memory location for the variable. Of course, if the symbol is referenced using SDAREL relocs, we must instead allocate it in .sbss. */ if (ppc_elf_hash_entry (h)->has_sda_refs) s = htab->dynsbss; else s = htab->dynbss; BFD_ASSERT (s != NULL); /* We must generate a R_PPC_COPY reloc to tell the dynamic linker to copy the initial value out of the dynamic object and into the runtime process image. We need to remember the offset into the .rela.bss section we are going to use. */ if ((h->root.u.def.section->flags & SEC_ALLOC) != 0) { asection *srel; if (ppc_elf_hash_entry (h)->has_sda_refs) srel = htab->relsbss; else srel = htab->relbss; BFD_ASSERT (srel != NULL); srel->size += sizeof (Elf32_External_Rela); h->needs_copy = 1; } return _bfd_elf_adjust_dynamic_copy (h, s); } /* Generate a symbol to mark plt call stubs. For non-PIC code the sym is xxxxxxxx.plt_call32. where xxxxxxxx is a hex number, usually 0, specifying the addend on the plt relocation. For -fpic code, the sym is xxxxxxxx.plt_pic32., and for -fPIC xxxxxxxx.got2.plt_pic32.. */ static bfd_boolean add_stub_sym (struct plt_entry *ent, struct elf_link_hash_entry *h, struct bfd_link_info *info) { struct elf_link_hash_entry *sh; size_t len1, len2, len3; char *name; const char *stub; struct ppc_elf_link_hash_table *htab = ppc_elf_hash_table (info); if (info->shared || info->pie) stub = ".plt_pic32."; else stub = ".plt_call32."; len1 = strlen (h->root.root.string); len2 = strlen (stub); len3 = 0; if (ent->sec) len3 = strlen (ent->sec->name); name = bfd_malloc (len1 + len2 + len3 + 9); if (name == NULL) return FALSE; sprintf (name, "%08x", (unsigned) ent->addend & 0xffffffff); if (ent->sec) memcpy (name + 8, ent->sec->name, len3); memcpy (name + 8 + len3, stub, len2); memcpy (name + 8 + len3 + len2, h->root.root.string, len1 + 1); sh = elf_link_hash_lookup (&htab->elf, name, TRUE, FALSE, FALSE); if (sh == NULL) return FALSE; if (sh->root.type == bfd_link_hash_new) { sh->root.type = bfd_link_hash_defined; sh->root.u.def.section = htab->glink; sh->root.u.def.value = ent->glink_offset; sh->ref_regular = 1; sh->def_regular = 1; sh->ref_regular_nonweak = 1; sh->forced_local = 1; sh->non_elf = 0; } return TRUE; } /* Allocate NEED contiguous space in .got, and return the offset. Handles allocation of the got header when crossing 32k. */ static bfd_vma allocate_got (struct ppc_elf_link_hash_table *htab, unsigned int need) { bfd_vma where; unsigned int max_before_header; if (htab->plt_type == PLT_VXWORKS) { where = htab->got->size; htab->got->size += need; } else { max_before_header = htab->plt_type == PLT_NEW ? 32768 : 32764; if (need <= htab->got_gap) { where = max_before_header - htab->got_gap; htab->got_gap -= need; } else { if (htab->got->size + need > max_before_header && htab->got->size <= max_before_header) { htab->got_gap = max_before_header - htab->got->size; htab->got->size = max_before_header + htab->got_header_size; } where = htab->got->size; htab->got->size += need; } } return where; } /* Allocate space in associated reloc sections for dynamic relocs. */ static bfd_boolean allocate_dynrelocs (struct elf_link_hash_entry *h, void *inf) { struct bfd_link_info *info = inf; struct ppc_elf_link_hash_entry *eh; struct ppc_elf_link_hash_table *htab; struct ppc_elf_dyn_relocs *p; if (h->root.type == bfd_link_hash_indirect) return TRUE; if (h->root.type == bfd_link_hash_warning) /* When warning symbols are created, they **replace** the "real" entry in the hash table, thus we never get to see the real symbol in a hash traversal. So look at it now. */ h = (struct elf_link_hash_entry *) h->root.u.i.link; htab = ppc_elf_hash_table (info); if (htab->elf.dynamic_sections_created) { struct plt_entry *ent; bfd_boolean doneone = FALSE; bfd_vma plt_offset = 0, glink_offset = 0; for (ent = h->plt.plist; ent != NULL; ent = ent->next) if (ent->plt.refcount > 0) { /* Make sure this symbol is output as a dynamic symbol. */ if (h->dynindx == -1 && !h->forced_local) { if (! bfd_elf_link_record_dynamic_symbol (info, h)) return FALSE; } if (info->shared || WILL_CALL_FINISH_DYNAMIC_SYMBOL (1, 0, h)) { asection *s = htab->plt; if (htab->plt_type == PLT_NEW) { if (!doneone) { plt_offset = s->size; s->size += 4; } ent->plt.offset = plt_offset; s = htab->glink; if (!doneone || info->shared || info->pie) { glink_offset = s->size; s->size += GLINK_ENTRY_SIZE; } if (!doneone && !info->shared && !h->def_regular) { h->root.u.def.section = s; h->root.u.def.value = glink_offset; } ent->glink_offset = glink_offset; if (htab->emit_stub_syms && !add_stub_sym (ent, h, info)) return FALSE; } else { if (!doneone) { /* If this is the first .plt entry, make room for the special first entry. */ if (s->size == 0) s->size += htab->plt_initial_entry_size; /* The PowerPC PLT is actually composed of two parts, the first part is 2 words (for a load and a jump), and then there is a remaining word available at the end. */ plt_offset = (htab->plt_initial_entry_size + (htab->plt_slot_size * ((s->size - htab->plt_initial_entry_size) / htab->plt_entry_size))); /* If this symbol is not defined in a regular file, and we are not generating a shared library, then set the symbol to this location in the .plt. This is required to make function pointers compare as equal between the normal executable and the shared library. */ if (! info->shared && !h->def_regular) { h->root.u.def.section = s; h->root.u.def.value = plt_offset; } /* Make room for this entry. */ s->size += htab->plt_entry_size; /* After the 8192nd entry, room for two entries is allocated. */ if (htab->plt_type == PLT_OLD && (s->size - htab->plt_initial_entry_size) / htab->plt_entry_size > PLT_NUM_SINGLE_ENTRIES) s->size += htab->plt_entry_size; } ent->plt.offset = plt_offset; } /* We also need to make an entry in the .rela.plt section. */ if (!doneone) { htab->relplt->size += sizeof (Elf32_External_Rela); if (htab->plt_type == PLT_VXWORKS) { /* Allocate space for the unloaded relocations. */ if (!info->shared) { if (ent->plt.offset == (bfd_vma) htab->plt_initial_entry_size) { htab->srelplt2->size += sizeof (Elf32_External_Rela) * VXWORKS_PLTRESOLVE_RELOCS; } htab->srelplt2->size += sizeof (Elf32_External_Rela) * VXWORKS_PLT_NON_JMP_SLOT_RELOCS; } /* Every PLT entry has an associated GOT entry in .got.plt. */ htab->sgotplt->size += 4; } doneone = TRUE; } } else ent->plt.offset = (bfd_vma) -1; } else ent->plt.offset = (bfd_vma) -1; if (!doneone) { h->plt.plist = NULL; h->needs_plt = 0; } } else { h->plt.plist = NULL; h->needs_plt = 0; } eh = (struct ppc_elf_link_hash_entry *) h; if (eh->elf.got.refcount > 0) { /* Make sure this symbol is output as a dynamic symbol. */ if (eh->elf.dynindx == -1 && !eh->elf.forced_local) { if (!bfd_elf_link_record_dynamic_symbol (info, &eh->elf)) return FALSE; } if (eh->tls_mask == (TLS_TLS | TLS_LD) && !eh->elf.def_dynamic) - /* If just an LD reloc, we'll just use htab->tlsld_got.offset. */ - eh->elf.got.offset = (bfd_vma) -1; + { + /* If just an LD reloc, we'll just use htab->tlsld_got.offset. */ + htab->tlsld_got.refcount += 1; + eh->elf.got.offset = (bfd_vma) -1; + } else { bfd_boolean dyn; unsigned int need = 0; if ((eh->tls_mask & TLS_TLS) != 0) { if ((eh->tls_mask & TLS_LD) != 0) need += 8; if ((eh->tls_mask & TLS_GD) != 0) need += 8; if ((eh->tls_mask & (TLS_TPREL | TLS_TPRELGD)) != 0) need += 4; if ((eh->tls_mask & TLS_DTPREL) != 0) need += 4; } else need += 4; eh->elf.got.offset = allocate_got (htab, need); dyn = htab->elf.dynamic_sections_created; if ((info->shared || WILL_CALL_FINISH_DYNAMIC_SYMBOL (dyn, 0, &eh->elf)) && (ELF_ST_VISIBILITY (eh->elf.other) == STV_DEFAULT || eh->elf.root.type != bfd_link_hash_undefweak)) { /* All the entries we allocated need relocs. Except LD only needs one. */ if ((eh->tls_mask & TLS_LD) != 0) need -= 4; htab->relgot->size += need * (sizeof (Elf32_External_Rela) / 4); } } } else eh->elf.got.offset = (bfd_vma) -1; if (eh->dyn_relocs == NULL) return TRUE; /* In the shared -Bsymbolic case, discard space allocated for dynamic pc-relative relocs against symbols which turn out to be defined in regular objects. For the normal shared case, discard space for relocs that have become local due to symbol visibility changes. */ if (info->shared) { /* Relocs that use pc_count are those that appear on a call insn, - or certain REL relocs (see MUST_BE_DYN_RELOC) that can be + or certain REL relocs (see must_be_dyn_reloc) that can be generated via assembly. We want calls to protected symbols to resolve directly to the function rather than going via the plt. If people want function pointer comparisons to work as expected then they should avoid writing weird assembly. */ if (SYMBOL_CALLS_LOCAL (info, h)) { struct ppc_elf_dyn_relocs **pp; for (pp = &eh->dyn_relocs; (p = *pp) != NULL; ) { p->count -= p->pc_count; p->pc_count = 0; if (p->count == 0) *pp = p->next; else pp = &p->next; } } /* Also discard relocs on undefined weak syms with non-default visibility. */ if (eh->dyn_relocs != NULL && h->root.type == bfd_link_hash_undefweak) { if (ELF_ST_VISIBILITY (h->other) != STV_DEFAULT) eh->dyn_relocs = NULL; /* Make sure undefined weak symbols are output as a dynamic symbol in PIEs. */ else if (h->dynindx == -1 && !h->forced_local) { if (! bfd_elf_link_record_dynamic_symbol (info, h)) return FALSE; } } } else if (ELIMINATE_COPY_RELOCS) { /* For the non-shared case, discard space for relocs against symbols which turn out to need copy relocs or are not dynamic. */ if (!h->non_got_ref && h->def_dynamic && !h->def_regular) { /* Make sure this symbol is output as a dynamic symbol. Undefined weak syms won't yet be marked as dynamic. */ if (h->dynindx == -1 && !h->forced_local) { if (! bfd_elf_link_record_dynamic_symbol (info, h)) return FALSE; } /* If that succeeded, we know we'll be keeping all the relocs. */ if (h->dynindx != -1) goto keep; } eh->dyn_relocs = NULL; keep: ; } /* Finally, allocate space. */ for (p = eh->dyn_relocs; p != NULL; p = p->next) { asection *sreloc = elf_section_data (p->sec)->sreloc; sreloc->size += p->count * sizeof (Elf32_External_Rela); } return TRUE; } /* Find any dynamic relocs that apply to read-only sections. */ static bfd_boolean readonly_dynrelocs (struct elf_link_hash_entry *h, void *info) { struct ppc_elf_dyn_relocs *p; if (h->root.type == bfd_link_hash_indirect) return TRUE; if (h->root.type == bfd_link_hash_warning) h = (struct elf_link_hash_entry *) h->root.u.i.link; for (p = ppc_elf_hash_entry (h)->dyn_relocs; p != NULL; p = p->next) { asection *s = p->sec->output_section; if (s != NULL && ((s->flags & (SEC_READONLY | SEC_ALLOC)) == (SEC_READONLY | SEC_ALLOC))) { ((struct bfd_link_info *) info)->flags |= DF_TEXTREL; /* Not an error, just cut short the traversal. */ return FALSE; } } return TRUE; } /* Set the sizes of the dynamic sections. */ static bfd_boolean ppc_elf_size_dynamic_sections (bfd *output_bfd ATTRIBUTE_UNUSED, struct bfd_link_info *info) { struct ppc_elf_link_hash_table *htab; asection *s; bfd_boolean relocs; bfd *ibfd; #ifdef DEBUG fprintf (stderr, "ppc_elf_size_dynamic_sections called\n"); #endif htab = ppc_elf_hash_table (info); BFD_ASSERT (htab->elf.dynobj != NULL); if (elf_hash_table (info)->dynamic_sections_created) { /* Set the contents of the .interp section to the interpreter. */ if (info->executable) { s = bfd_get_section_by_name (htab->elf.dynobj, ".interp"); BFD_ASSERT (s != NULL); s->size = sizeof ELF_DYNAMIC_INTERPRETER; s->contents = (unsigned char *) ELF_DYNAMIC_INTERPRETER; } } if (htab->plt_type == PLT_OLD) htab->got_header_size = 16; else if (htab->plt_type == PLT_NEW) htab->got_header_size = 12; /* Set up .got offsets for local syms, and space for local dynamic relocs. */ for (ibfd = info->input_bfds; ibfd != NULL; ibfd = ibfd->link_next) { bfd_signed_vma *local_got; bfd_signed_vma *end_local_got; char *lgot_masks; bfd_size_type locsymcount; Elf_Internal_Shdr *symtab_hdr; if (!is_ppc_elf_target (ibfd->xvec)) continue; for (s = ibfd->sections; s != NULL; s = s->next) { struct ppc_elf_dyn_relocs *p; for (p = ((struct ppc_elf_dyn_relocs *) elf_section_data (s)->local_dynrel); p != NULL; p = p->next) { if (!bfd_is_abs_section (p->sec) && bfd_is_abs_section (p->sec->output_section)) { /* Input section has been discarded, either because it is a copy of a linkonce section or due to linker script /DISCARD/, so we'll be discarding the relocs too. */ } else if (p->count != 0) { elf_section_data (p->sec)->sreloc->size += p->count * sizeof (Elf32_External_Rela); if ((p->sec->output_section->flags & (SEC_READONLY | SEC_ALLOC)) == (SEC_READONLY | SEC_ALLOC)) info->flags |= DF_TEXTREL; } } } local_got = elf_local_got_refcounts (ibfd); if (!local_got) continue; symtab_hdr = &elf_tdata (ibfd)->symtab_hdr; locsymcount = symtab_hdr->sh_info; end_local_got = local_got + locsymcount; lgot_masks = (char *) end_local_got; for (; local_got < end_local_got; ++local_got, ++lgot_masks) if (*local_got > 0) { if (*lgot_masks == (TLS_TLS | TLS_LD)) { /* If just an LD reloc, we'll just use htab->tlsld_got.offset. */ htab->tlsld_got.refcount += 1; *local_got = (bfd_vma) -1; } else { unsigned int need = 0; if ((*lgot_masks & TLS_TLS) != 0) { if ((*lgot_masks & TLS_GD) != 0) need += 8; if ((*lgot_masks & (TLS_TPREL | TLS_TPRELGD)) != 0) need += 4; if ((*lgot_masks & TLS_DTPREL) != 0) need += 4; } else need += 4; *local_got = allocate_got (htab, need); if (info->shared) htab->relgot->size += (need * (sizeof (Elf32_External_Rela) / 4)); } } else *local_got = (bfd_vma) -1; } + /* Allocate space for global sym dynamic relocs. */ + elf_link_hash_traverse (elf_hash_table (info), allocate_dynrelocs, info); + if (htab->tlsld_got.refcount > 0) { htab->tlsld_got.offset = allocate_got (htab, 8); if (info->shared) htab->relgot->size += sizeof (Elf32_External_Rela); } else htab->tlsld_got.offset = (bfd_vma) -1; - /* Allocate space for global sym dynamic relocs. */ - elf_link_hash_traverse (elf_hash_table (info), allocate_dynrelocs, info); - if (htab->got != NULL && htab->plt_type != PLT_VXWORKS) { unsigned int g_o_t = 32768; /* If we haven't allocated the header, do so now. When we get here, for old plt/got the got size will be 0 to 32764 (not allocated), or 32780 to 65536 (header allocated). For new plt/got, the corresponding ranges are 0 to 32768 and 32780 to 65536. */ if (htab->got->size <= 32768) { g_o_t = htab->got->size; if (htab->plt_type == PLT_OLD) g_o_t += 4; htab->got->size += htab->got_header_size; } htab->elf.hgot->root.u.def.value = g_o_t; } if (htab->glink != NULL && htab->glink->size != 0) { htab->glink_pltresolve = htab->glink->size; /* Space for the branch table. */ htab->glink->size += htab->glink->size / (GLINK_ENTRY_SIZE / 4) - 4; /* Pad out to align the start of PLTresolve. */ htab->glink->size += -htab->glink->size & 15; htab->glink->size += GLINK_PLTRESOLVE; if (htab->emit_stub_syms) { struct elf_link_hash_entry *sh; sh = elf_link_hash_lookup (&htab->elf, "__glink", TRUE, FALSE, FALSE); if (sh == NULL) return FALSE; if (sh->root.type == bfd_link_hash_new) { sh->root.type = bfd_link_hash_defined; sh->root.u.def.section = htab->glink; sh->root.u.def.value = htab->glink_pltresolve; sh->ref_regular = 1; sh->def_regular = 1; sh->ref_regular_nonweak = 1; sh->forced_local = 1; sh->non_elf = 0; } sh = elf_link_hash_lookup (&htab->elf, "__glink_PLTresolve", TRUE, FALSE, FALSE); if (sh == NULL) return FALSE; if (sh->root.type == bfd_link_hash_new) { sh->root.type = bfd_link_hash_defined; sh->root.u.def.section = htab->glink; sh->root.u.def.value = htab->glink->size - GLINK_PLTRESOLVE; sh->ref_regular = 1; sh->def_regular = 1; sh->ref_regular_nonweak = 1; sh->forced_local = 1; sh->non_elf = 0; } } } /* We've now determined the sizes of the various dynamic sections. Allocate memory for them. */ relocs = FALSE; for (s = htab->elf.dynobj->sections; s != NULL; s = s->next) { bfd_boolean strip_section = TRUE; if ((s->flags & SEC_LINKER_CREATED) == 0) continue; if (s == htab->plt || s == htab->glink || s == htab->got || s == htab->sgotplt || s == htab->sbss || s == htab->dynbss || s == htab->dynsbss) { /* We'd like to strip these sections if they aren't needed, but if we've exported dynamic symbols from them we must leave them. It's too late to tell BFD to get rid of the symbols. */ if ((s == htab->plt || s == htab->got) && htab->elf.hplt != NULL) strip_section = FALSE; /* Strip this section if we don't need it; see the comment below. */ } else if (s == htab->sdata[0].section || s == htab->sdata[1].section) { /* Strip these too. */ } else if (CONST_STRNEQ (bfd_get_section_name (dynobj, s), ".rela")) { if (s->size != 0) { /* Remember whether there are any relocation sections. */ relocs = TRUE; /* We use the reloc_count field as a counter if we need to copy relocs into the output file. */ s->reloc_count = 0; } } else { /* It's not one of our sections, so don't allocate space. */ continue; } if (s->size == 0 && strip_section) { /* If we don't need this section, strip it from the output file. This is mostly to handle .rela.bss and .rela.plt. We must create both sections in create_dynamic_sections, because they must be created before the linker maps input sections to output sections. The linker does that before adjust_dynamic_symbol is called, and it is that function which decides whether anything needs to go into these sections. */ s->flags |= SEC_EXCLUDE; continue; } if ((s->flags & SEC_HAS_CONTENTS) == 0) continue; /* Allocate memory for the section contents. */ s->contents = bfd_zalloc (htab->elf.dynobj, s->size); if (s->contents == NULL) return FALSE; } if (htab->elf.dynamic_sections_created) { /* Add some entries to the .dynamic section. We fill in the values later, in ppc_elf_finish_dynamic_sections, but we must add the entries now so that we get the correct size for the .dynamic section. The DT_DEBUG entry is filled in by the dynamic linker and used by the debugger. */ #define add_dynamic_entry(TAG, VAL) \ _bfd_elf_add_dynamic_entry (info, TAG, VAL) if (info->executable) { if (!add_dynamic_entry (DT_DEBUG, 0)) return FALSE; } if (htab->plt != NULL && htab->plt->size != 0) { if (!add_dynamic_entry (DT_PLTGOT, 0) || !add_dynamic_entry (DT_PLTRELSZ, 0) || !add_dynamic_entry (DT_PLTREL, DT_RELA) || !add_dynamic_entry (DT_JMPREL, 0)) return FALSE; } if (htab->glink != NULL && htab->glink->size != 0) { if (!add_dynamic_entry (DT_PPC_GOT, 0)) return FALSE; } if (relocs) { if (!add_dynamic_entry (DT_RELA, 0) || !add_dynamic_entry (DT_RELASZ, 0) || !add_dynamic_entry (DT_RELAENT, sizeof (Elf32_External_Rela))) return FALSE; } /* If any dynamic relocs apply to a read-only section, then we need a DT_TEXTREL entry. */ if ((info->flags & DF_TEXTREL) == 0) elf_link_hash_traverse (elf_hash_table (info), readonly_dynrelocs, info); if ((info->flags & DF_TEXTREL) != 0) { if (!add_dynamic_entry (DT_TEXTREL, 0)) return FALSE; } } #undef add_dynamic_entry return TRUE; } #define ARRAY_SIZE(a) (sizeof (a) / sizeof ((a)[0])) static const int shared_stub_entry[] = { 0x7c0802a6, /* mflr 0 */ 0x429f0005, /* bcl 20, 31, .Lxxx */ 0x7d6802a6, /* mflr 11 */ 0x3d6b0000, /* addis 11, 11, (xxx-.Lxxx)@ha */ 0x396b0018, /* addi 11, 11, (xxx-.Lxxx)@l */ 0x7c0803a6, /* mtlr 0 */ 0x7d6903a6, /* mtctr 11 */ 0x4e800420, /* bctr */ }; static const int stub_entry[] = { 0x3d600000, /* lis 11,xxx@ha */ 0x396b0000, /* addi 11,11,xxx@l */ 0x7d6903a6, /* mtctr 11 */ 0x4e800420, /* bctr */ }; static bfd_boolean ppc_elf_relax_section (bfd *abfd, asection *isec, struct bfd_link_info *link_info, bfd_boolean *again) { struct one_fixup { struct one_fixup *next; asection *tsec; bfd_vma toff; bfd_vma trampoff; }; Elf_Internal_Shdr *symtab_hdr; bfd_byte *contents = NULL; Elf_Internal_Sym *isymbuf = NULL; Elf_Internal_Rela *internal_relocs = NULL; Elf_Internal_Rela *irel, *irelend; struct one_fixup *fixups = NULL; bfd_boolean changed; struct ppc_elf_link_hash_table *htab; bfd_size_type trampoff; asection *got2; *again = FALSE; /* Nothing to do if there are no relocations, and no need to do anything with non-alloc sections. */ if ((isec->flags & SEC_ALLOC) == 0 || (isec->flags & SEC_RELOC) == 0 || isec->reloc_count == 0) return TRUE; trampoff = (isec->size + 3) & (bfd_vma) -4; /* Space for a branch around any trampolines. */ trampoff += 4; symtab_hdr = &elf_tdata (abfd)->symtab_hdr; /* Get a copy of the native relocations. */ internal_relocs = _bfd_elf_link_read_relocs (abfd, isec, NULL, NULL, link_info->keep_memory); if (internal_relocs == NULL) goto error_return; htab = ppc_elf_hash_table (link_info); got2 = bfd_get_section_by_name (abfd, ".got2"); irelend = internal_relocs + isec->reloc_count; for (irel = internal_relocs; irel < irelend; irel++) { unsigned long r_type = ELF32_R_TYPE (irel->r_info); bfd_vma symaddr, reladdr, toff, roff; asection *tsec; struct one_fixup *f; size_t insn_offset = 0; bfd_vma max_branch_offset, val; bfd_byte *hit_addr; unsigned long t0; unsigned char sym_type; switch (r_type) { case R_PPC_REL24: case R_PPC_LOCAL24PC: case R_PPC_PLTREL24: max_branch_offset = 1 << 25; break; case R_PPC_REL14: case R_PPC_REL14_BRTAKEN: case R_PPC_REL14_BRNTAKEN: max_branch_offset = 1 << 15; break; default: continue; } /* Get the value of the symbol referred to by the reloc. */ if (ELF32_R_SYM (irel->r_info) < symtab_hdr->sh_info) { /* A local symbol. */ Elf_Internal_Sym *isym; /* Read this BFD's local symbols. */ if (isymbuf == NULL) { isymbuf = (Elf_Internal_Sym *) symtab_hdr->contents; if (isymbuf == NULL) isymbuf = bfd_elf_get_elf_syms (abfd, symtab_hdr, symtab_hdr->sh_info, 0, NULL, NULL, NULL); if (isymbuf == 0) goto error_return; } isym = isymbuf + ELF32_R_SYM (irel->r_info); if (isym->st_shndx == SHN_UNDEF) continue; /* We can't do anything with undefined symbols. */ else if (isym->st_shndx == SHN_ABS) tsec = bfd_abs_section_ptr; else if (isym->st_shndx == SHN_COMMON) tsec = bfd_com_section_ptr; else tsec = bfd_section_from_elf_index (abfd, isym->st_shndx); toff = isym->st_value; sym_type = ELF_ST_TYPE (isym->st_info); } else { /* Global symbol handling. */ unsigned long indx; struct elf_link_hash_entry *h; indx = ELF32_R_SYM (irel->r_info) - symtab_hdr->sh_info; h = elf_sym_hashes (abfd)[indx]; while (h->root.type == bfd_link_hash_indirect || h->root.type == bfd_link_hash_warning) h = (struct elf_link_hash_entry *) h->root.u.i.link; tsec = NULL; toff = 0; if (r_type == R_PPC_PLTREL24 && htab->plt != NULL) { struct plt_entry *ent = find_plt_ent (h, got2, irel->r_addend); if (ent != NULL) { if (htab->plt_type == PLT_NEW) { tsec = htab->glink; toff = ent->glink_offset; } else { tsec = htab->plt; toff = ent->plt.offset; } } } if (tsec != NULL) ; else if (h->root.type == bfd_link_hash_defined || h->root.type == bfd_link_hash_defweak) { tsec = h->root.u.def.section; toff = h->root.u.def.value; } else continue; sym_type = h->type; } /* If the branch and target are in the same section, you have no hope of adding stubs. We'll error out later should the branch overflow. */ if (tsec == isec) continue; /* There probably isn't any reason to handle symbols in SEC_MERGE sections; SEC_MERGE doesn't seem a likely attribute for a code section, and we are only looking at branches. However, implement it correctly here as a reference for other target relax_section functions. */ if (0 && tsec->sec_info_type == ELF_INFO_TYPE_MERGE) { /* At this stage in linking, no SEC_MERGE symbol has been adjusted, so all references to such symbols need to be passed through _bfd_merged_section_offset. (Later, in relocate_section, all SEC_MERGE symbols *except* for section symbols have been adjusted.) gas may reduce relocations against symbols in SEC_MERGE sections to a relocation against the section symbol when the original addend was zero. When the reloc is against a section symbol we should include the addend in the offset passed to _bfd_merged_section_offset, since the location of interest is the original symbol. On the other hand, an access to "sym+addend" where "sym" is not a section symbol should not include the addend; Such an access is presumed to be an offset from "sym"; The location of interest is just "sym". */ if (sym_type == STT_SECTION) toff += irel->r_addend; toff = _bfd_merged_section_offset (abfd, &tsec, elf_section_data (tsec)->sec_info, toff); if (sym_type != STT_SECTION) toff += irel->r_addend; } /* PLTREL24 addends are special. */ else if (r_type != R_PPC_PLTREL24) toff += irel->r_addend; /* Attempted -shared link of non-pic code loses. */ if (tsec->output_section == NULL) continue; symaddr = tsec->output_section->vma + tsec->output_offset + toff; roff = irel->r_offset; reladdr = isec->output_section->vma + isec->output_offset + roff; /* If the branch is in range, no need to do anything. */ if (symaddr - reladdr + max_branch_offset < 2 * max_branch_offset) continue; /* Look for an existing fixup to this address. */ for (f = fixups; f ; f = f->next) if (f->tsec == tsec && f->toff == toff) break; if (f == NULL) { size_t size; unsigned long stub_rtype; val = trampoff - roff; if (val >= max_branch_offset) /* Oh dear, we can't reach a trampoline. Don't try to add one. We'll report an error later. */ continue; if (link_info->shared) { size = 4 * ARRAY_SIZE (shared_stub_entry); insn_offset = 12; stub_rtype = R_PPC_RELAX32PC; } else { size = 4 * ARRAY_SIZE (stub_entry); insn_offset = 0; stub_rtype = R_PPC_RELAX32; } if (R_PPC_RELAX32_PLT - R_PPC_RELAX32 != R_PPC_RELAX32PC_PLT - R_PPC_RELAX32PC) abort (); if (tsec == htab->plt || tsec == htab->glink) stub_rtype += R_PPC_RELAX32_PLT - R_PPC_RELAX32; /* Hijack the old relocation. Since we need two relocations for this use a "composite" reloc. */ irel->r_info = ELF32_R_INFO (ELF32_R_SYM (irel->r_info), stub_rtype); irel->r_offset = trampoff + insn_offset; /* Record the fixup so we don't do it again this section. */ f = bfd_malloc (sizeof (*f)); f->next = fixups; f->tsec = tsec; f->toff = toff; f->trampoff = trampoff; fixups = f; trampoff += size; } else { val = f->trampoff - roff; if (val >= max_branch_offset) continue; /* Nop out the reloc, since we're finalizing things here. */ irel->r_info = ELF32_R_INFO (0, R_PPC_NONE); } /* Get the section contents. */ if (contents == NULL) { /* Get cached copy if it exists. */ if (elf_section_data (isec)->this_hdr.contents != NULL) contents = elf_section_data (isec)->this_hdr.contents; else { /* Go get them off disk. */ if (!bfd_malloc_and_get_section (abfd, isec, &contents)) goto error_return; } } /* Fix up the existing branch to hit the trampoline. */ hit_addr = contents + roff; switch (r_type) { case R_PPC_REL24: case R_PPC_LOCAL24PC: case R_PPC_PLTREL24: t0 = bfd_get_32 (abfd, hit_addr); t0 &= ~0x3fffffc; t0 |= val & 0x3fffffc; bfd_put_32 (abfd, t0, hit_addr); break; case R_PPC_REL14: case R_PPC_REL14_BRTAKEN: case R_PPC_REL14_BRNTAKEN: t0 = bfd_get_32 (abfd, hit_addr); t0 &= ~0xfffc; t0 |= val & 0xfffc; bfd_put_32 (abfd, t0, hit_addr); break; } } /* Write out the trampolines. */ changed = fixups != NULL; if (fixups != NULL) { const int *stub; bfd_byte *dest; bfd_vma val; int i, size; do { struct one_fixup *f = fixups; fixups = fixups->next; free (f); } while (fixups); contents = bfd_realloc (contents, trampoff); if (contents == NULL) goto error_return; isec->size = (isec->size + 3) & (bfd_vma) -4; /* Branch around the trampolines. */ val = trampoff - isec->size + 0x48000000; dest = contents + isec->size; isec->size = trampoff; bfd_put_32 (abfd, val, dest); dest += 4; if (link_info->shared) { stub = shared_stub_entry; size = ARRAY_SIZE (shared_stub_entry); } else { stub = stub_entry; size = ARRAY_SIZE (stub_entry); } i = 0; while (dest < contents + trampoff) { bfd_put_32 (abfd, stub[i], dest); i++; if (i == size) i = 0; dest += 4; } BFD_ASSERT (i == 0); } if (isymbuf != NULL && symtab_hdr->contents != (unsigned char *) isymbuf) { if (! link_info->keep_memory) free (isymbuf); else { /* Cache the symbols for elf_link_input_bfd. */ symtab_hdr->contents = (unsigned char *) isymbuf; } } if (contents != NULL && elf_section_data (isec)->this_hdr.contents != contents) { if (!changed && !link_info->keep_memory) free (contents); else { /* Cache the section contents for elf_link_input_bfd. */ elf_section_data (isec)->this_hdr.contents = contents; } } if (elf_section_data (isec)->relocs != internal_relocs) { if (!changed) free (internal_relocs); else elf_section_data (isec)->relocs = internal_relocs; } *again = changed; return TRUE; error_return: if (isymbuf != NULL && (unsigned char *) isymbuf != symtab_hdr->contents) free (isymbuf); if (contents != NULL && elf_section_data (isec)->this_hdr.contents != contents) free (contents); if (internal_relocs != NULL && elf_section_data (isec)->relocs != internal_relocs) free (internal_relocs); return FALSE; } /* What to do when ld finds relocations against symbols defined in discarded sections. */ static unsigned int ppc_elf_action_discarded (asection *sec) { if (strcmp (".fixup", sec->name) == 0) return 0; if (strcmp (".got2", sec->name) == 0) return 0; return _bfd_elf_default_action_discarded (sec); } /* Fill in the address for a pointer generated in a linker section. */ static bfd_vma elf_finish_pointer_linker_section (bfd *input_bfd, elf_linker_section_t *lsect, struct elf_link_hash_entry *h, bfd_vma relocation, const Elf_Internal_Rela *rel) { elf_linker_section_pointers_t *linker_section_ptr; BFD_ASSERT (lsect != NULL); if (h != NULL) { /* Handle global symbol. */ struct ppc_elf_link_hash_entry *eh; eh = (struct ppc_elf_link_hash_entry *) h; BFD_ASSERT (eh->elf.def_regular); linker_section_ptr = eh->linker_section_pointer; } else { /* Handle local symbol. */ unsigned long r_symndx = ELF32_R_SYM (rel->r_info); BFD_ASSERT (elf_local_ptr_offsets (input_bfd) != NULL); linker_section_ptr = elf_local_ptr_offsets (input_bfd)[r_symndx]; } linker_section_ptr = elf_find_pointer_linker_section (linker_section_ptr, rel->r_addend, lsect); BFD_ASSERT (linker_section_ptr != NULL); /* Offset will always be a multiple of four, so use the bottom bit as a "written" flag. */ if ((linker_section_ptr->offset & 1) == 0) { bfd_put_32 (lsect->section->owner, relocation + linker_section_ptr->addend, lsect->section->contents + linker_section_ptr->offset); linker_section_ptr->offset += 1; } relocation = (lsect->section->output_offset + linker_section_ptr->offset - 1 - 0x8000); #ifdef DEBUG fprintf (stderr, "Finish pointer in linker section %s, offset = %ld (0x%lx)\n", lsect->name, (long) relocation, (long) relocation); #endif /* Subtract out the addend, because it will get added back in by the normal processing. */ return relocation - linker_section_ptr->addend; } /* The RELOCATE_SECTION function is called by the ELF backend linker to handle the relocations for a section. The relocs are always passed as Rela structures; if the section actually uses Rel structures, the r_addend field will always be zero. This function is responsible for adjust the section contents as necessary, and (if using Rela relocs and generating a relocatable output file) adjusting the reloc addend as necessary. This function does not have to worry about setting the reloc address or the reloc symbol index. LOCAL_SYMS is a pointer to the swapped in local symbols. LOCAL_SECTIONS is an array giving the section in the input file corresponding to the st_shndx field of each local symbol. The global hash table entry for the global symbols can be found via elf_sym_hashes (input_bfd). When generating relocatable output, this function must handle STB_LOCAL/STT_SECTION symbols specially. The output symbol is going to be the section symbol corresponding to the output section, which means that the addend must be adjusted accordingly. */ static bfd_boolean ppc_elf_relocate_section (bfd *output_bfd, struct bfd_link_info *info, bfd *input_bfd, asection *input_section, bfd_byte *contents, Elf_Internal_Rela *relocs, Elf_Internal_Sym *local_syms, asection **local_sections) { Elf_Internal_Shdr *symtab_hdr; struct elf_link_hash_entry **sym_hashes; struct ppc_elf_link_hash_table *htab; Elf_Internal_Rela *rel; Elf_Internal_Rela *relend; Elf_Internal_Rela outrel; bfd_byte *loc; asection *got2, *sreloc = NULL; bfd_vma *local_got_offsets; bfd_boolean ret = TRUE; bfd_vma d_offset = (bfd_big_endian (output_bfd) ? 2 : 0); #ifdef DEBUG _bfd_error_handler ("ppc_elf_relocate_section called for %B section %A, " "%ld relocations%s", input_bfd, input_section, (long) input_section->reloc_count, (info->relocatable) ? " (relocatable)" : ""); #endif got2 = bfd_get_section_by_name (input_bfd, ".got2"); /* Initialize howto table if not already done. */ if (!ppc_elf_howto_table[R_PPC_ADDR32]) ppc_elf_howto_init (); htab = ppc_elf_hash_table (info); local_got_offsets = elf_local_got_offsets (input_bfd); symtab_hdr = &elf_tdata (input_bfd)->symtab_hdr; sym_hashes = elf_sym_hashes (input_bfd); rel = relocs; relend = relocs + input_section->reloc_count; for (; rel < relend; rel++) { enum elf_ppc_reloc_type r_type; bfd_vma addend; bfd_reloc_status_type r; Elf_Internal_Sym *sym; asection *sec; struct elf_link_hash_entry *h; const char *sym_name; reloc_howto_type *howto; unsigned long r_symndx; bfd_vma relocation; bfd_vma branch_bit, insn, from; bfd_boolean unresolved_reloc; bfd_boolean warned; unsigned int tls_type, tls_mask, tls_gd; r_type = ELF32_R_TYPE (rel->r_info); sym = NULL; sec = NULL; h = NULL; unresolved_reloc = FALSE; warned = FALSE; r_symndx = ELF32_R_SYM (rel->r_info); if (r_symndx < symtab_hdr->sh_info) { sym = local_syms + r_symndx; sec = local_sections[r_symndx]; sym_name = bfd_elf_sym_name (input_bfd, symtab_hdr, sym, sec); relocation = _bfd_elf_rela_local_sym (output_bfd, sym, &sec, rel); } else { RELOC_FOR_GLOBAL_SYMBOL (info, input_bfd, input_section, rel, r_symndx, symtab_hdr, sym_hashes, h, sec, relocation, unresolved_reloc, warned); sym_name = h->root.root.string; } if (sec != NULL && elf_discarded_section (sec)) { /* For relocs against symbols from removed linkonce sections, or sections discarded by a linker script, we just want the section contents zeroed. Avoid any special processing. */ howto = NULL; if (r_type < R_PPC_max) howto = ppc_elf_howto_table[r_type]; _bfd_clear_contents (howto, input_bfd, contents + rel->r_offset); rel->r_info = 0; rel->r_addend = 0; continue; } if (info->relocatable) { if (got2 != NULL && r_type == R_PPC_PLTREL24 && rel->r_addend >= 32768) { /* R_PPC_PLTREL24 is rather special. If non-zero, the addend specifies the GOT pointer offset within .got2. */ rel->r_addend += got2->output_offset; } continue; } /* TLS optimizations. Replace instruction sequences and relocs based on information we collected in tls_optimize. We edit RELOCS so that --emit-relocs will output something sensible for the final instruction stream. */ tls_mask = 0; tls_gd = 0; - if (IS_PPC_TLS_RELOC (r_type)) + if (h != NULL) + tls_mask = ((struct ppc_elf_link_hash_entry *) h)->tls_mask; + else if (local_got_offsets != NULL) { - if (h != NULL) - tls_mask = ((struct ppc_elf_link_hash_entry *) h)->tls_mask; - else if (local_got_offsets != NULL) - { - char *lgot_masks; - lgot_masks = (char *) (local_got_offsets + symtab_hdr->sh_info); - tls_mask = lgot_masks[r_symndx]; - } + char *lgot_masks; + lgot_masks = (char *) (local_got_offsets + symtab_hdr->sh_info); + tls_mask = lgot_masks[r_symndx]; } /* Ensure reloc mapping code below stays sane. */ if ((R_PPC_GOT_TLSLD16 & 3) != (R_PPC_GOT_TLSGD16 & 3) || (R_PPC_GOT_TLSLD16_LO & 3) != (R_PPC_GOT_TLSGD16_LO & 3) || (R_PPC_GOT_TLSLD16_HI & 3) != (R_PPC_GOT_TLSGD16_HI & 3) || (R_PPC_GOT_TLSLD16_HA & 3) != (R_PPC_GOT_TLSGD16_HA & 3) || (R_PPC_GOT_TLSLD16 & 3) != (R_PPC_GOT_TPREL16 & 3) || (R_PPC_GOT_TLSLD16_LO & 3) != (R_PPC_GOT_TPREL16_LO & 3) || (R_PPC_GOT_TLSLD16_HI & 3) != (R_PPC_GOT_TPREL16_HI & 3) || (R_PPC_GOT_TLSLD16_HA & 3) != (R_PPC_GOT_TPREL16_HA & 3)) abort (); switch (r_type) { default: break; case R_PPC_GOT_TPREL16: case R_PPC_GOT_TPREL16_LO: if (tls_mask != 0 && (tls_mask & TLS_TPREL) == 0) { bfd_vma insn; insn = bfd_get_32 (output_bfd, contents + rel->r_offset - d_offset); insn &= 31 << 21; insn |= 0x3c020000; /* addis 0,2,0 */ bfd_put_32 (output_bfd, insn, contents + rel->r_offset - d_offset); r_type = R_PPC_TPREL16_HA; rel->r_info = ELF32_R_INFO (r_symndx, r_type); } break; case R_PPC_TLS: if (tls_mask != 0 && (tls_mask & TLS_TPREL) == 0) { bfd_vma insn, rtra; insn = bfd_get_32 (output_bfd, contents + rel->r_offset); if ((insn & ((31 << 26) | (31 << 11))) == ((31 << 26) | (2 << 11))) rtra = insn & ((1 << 26) - (1 << 16)); else if ((insn & ((31 << 26) | (31 << 16))) == ((31 << 26) | (2 << 16))) rtra = (insn & (31 << 21)) | ((insn & (31 << 11)) << 5); else abort (); if ((insn & ((1 << 11) - (1 << 1))) == 266 << 1) /* add -> addi. */ insn = 14 << 26; else if ((insn & (31 << 1)) == 23 << 1 && ((insn & (31 << 6)) < 14 << 6 || ((insn & (31 << 6)) >= 16 << 6 && (insn & (31 << 6)) < 24 << 6))) /* load and store indexed -> dform. */ insn = (32 | ((insn >> 6) & 31)) << 26; else if ((insn & (31 << 1)) == 21 << 1 && (insn & (0x1a << 6)) == 0) /* ldx, ldux, stdx, stdux -> ld, ldu, std, stdu. */ insn = (((58 | ((insn >> 6) & 4)) << 26) | ((insn >> 6) & 1)); else if ((insn & (31 << 1)) == 21 << 1 && (insn & ((1 << 11) - (1 << 1))) == 341 << 1) /* lwax -> lwa. */ insn = (58 << 26) | 2; else abort (); insn |= rtra; bfd_put_32 (output_bfd, insn, contents + rel->r_offset); r_type = R_PPC_TPREL16_LO; rel->r_info = ELF32_R_INFO (r_symndx, r_type); /* Was PPC_TLS which sits on insn boundary, now PPC_TPREL16_LO which is at low-order half-word. */ rel->r_offset += d_offset; } break; case R_PPC_GOT_TLSGD16_HI: case R_PPC_GOT_TLSGD16_HA: tls_gd = TLS_TPRELGD; if (tls_mask != 0 && (tls_mask & TLS_GD) == 0) goto tls_gdld_hi; break; case R_PPC_GOT_TLSLD16_HI: case R_PPC_GOT_TLSLD16_HA: if (tls_mask != 0 && (tls_mask & TLS_LD) == 0) { tls_gdld_hi: if ((tls_mask & tls_gd) != 0) r_type = (((r_type - (R_PPC_GOT_TLSGD16 & 3)) & 3) + R_PPC_GOT_TPREL16); else { bfd_put_32 (output_bfd, NOP, contents + rel->r_offset); rel->r_offset -= d_offset; r_type = R_PPC_NONE; } rel->r_info = ELF32_R_INFO (r_symndx, r_type); } break; case R_PPC_GOT_TLSGD16: case R_PPC_GOT_TLSGD16_LO: tls_gd = TLS_TPRELGD; if (tls_mask != 0 && (tls_mask & TLS_GD) == 0) - goto tls_get_addr_check; + goto tls_ldgd_opt; break; case R_PPC_GOT_TLSLD16: case R_PPC_GOT_TLSLD16_LO: if (tls_mask != 0 && (tls_mask & TLS_LD) == 0) { - tls_get_addr_check: - if (rel + 1 < relend) - { - enum elf_ppc_reloc_type r_type2; - unsigned long r_symndx2; - struct elf_link_hash_entry *h2; - bfd_vma insn1, insn2; - bfd_vma offset; + unsigned int insn1, insn2; + bfd_vma offset; - /* The next instruction should be a call to - __tls_get_addr. Peek at the reloc to be sure. */ - r_type2 = ELF32_R_TYPE (rel[1].r_info); - r_symndx2 = ELF32_R_SYM (rel[1].r_info); - if (r_symndx2 < symtab_hdr->sh_info - || (r_type2 != R_PPC_REL14 - && r_type2 != R_PPC_REL14_BRTAKEN - && r_type2 != R_PPC_REL14_BRNTAKEN - && r_type2 != R_PPC_REL24 - && r_type2 != R_PPC_PLTREL24)) - break; - - h2 = sym_hashes[r_symndx2 - symtab_hdr->sh_info]; - while (h2->root.type == bfd_link_hash_indirect - || h2->root.type == bfd_link_hash_warning) - h2 = (struct elf_link_hash_entry *) h2->root.u.i.link; - if (h2 == NULL || h2 != htab->tls_get_addr) - break; - - /* OK, it checks out. Replace the call. */ - offset = rel[1].r_offset; + tls_ldgd_opt: + offset = (bfd_vma) -1; + /* If not using the newer R_PPC_TLSGD/LD to mark + __tls_get_addr calls, we must trust that the call + stays with its arg setup insns, ie. that the next + reloc is the __tls_get_addr call associated with + the current reloc. Edit both insns. */ + if (input_section->has_tls_get_addr_call + && rel + 1 < relend + && branch_reloc_hash_match (input_bfd, rel + 1, + htab->tls_get_addr)) + offset = rel[1].r_offset; + if ((tls_mask & tls_gd) != 0) + { + /* IE */ insn1 = bfd_get_32 (output_bfd, contents + rel->r_offset - d_offset); - if ((tls_mask & tls_gd) != 0) + insn1 &= (1 << 26) - 1; + insn1 |= 32 << 26; /* lwz */ + if (offset != (bfd_vma) -1) { - /* IE */ - insn1 &= (1 << 26) - 1; - insn1 |= 32 << 26; /* lwz */ + rel[1].r_info + = ELF32_R_INFO (ELF32_R_SYM (rel[1].r_info), + R_PPC_NONE); insn2 = 0x7c631214; /* add 3,3,2 */ - rel[1].r_info = ELF32_R_INFO (r_symndx2, R_PPC_NONE); - rel[1].r_addend = 0; - r_type = (((r_type - (R_PPC_GOT_TLSGD16 & 3)) & 3) - + R_PPC_GOT_TPREL16); - rel->r_info = ELF32_R_INFO (r_symndx, r_type); + bfd_put_32 (output_bfd, insn2, contents + offset); } - else + r_type = (((r_type - (R_PPC_GOT_TLSGD16 & 3)) & 3) + + R_PPC_GOT_TPREL16); + rel->r_info = ELF32_R_INFO (r_symndx, r_type); + } + else + { + /* LE */ + insn1 = 0x3c620000; /* addis 3,2,0 */ + if (tls_gd == 0) { - /* LE */ - insn1 = 0x3c620000; /* addis 3,2,0 */ - insn2 = 0x38630000; /* addi 3,3,0 */ - if (tls_gd == 0) - { - /* Was an LD reloc. */ - r_symndx = 0; - rel->r_addend = htab->elf.tls_sec->vma + DTP_OFFSET; - } - r_type = R_PPC_TPREL16_HA; - rel->r_info = ELF32_R_INFO (r_symndx, r_type); - rel[1].r_info = ELF32_R_INFO (r_symndx, - R_PPC_TPREL16_LO); - rel[1].r_offset += d_offset; - rel[1].r_addend = rel->r_addend; + /* Was an LD reloc. */ + for (r_symndx = 0; + r_symndx < symtab_hdr->sh_info; + r_symndx++) + if (local_sections[r_symndx] == sec) + break; + if (r_symndx >= symtab_hdr->sh_info) + r_symndx = 0; + rel->r_addend = htab->elf.tls_sec->vma + DTP_OFFSET; + if (r_symndx != 0) + rel->r_addend -= (local_syms[r_symndx].st_value + + sec->output_offset + + sec->output_section->vma); } - bfd_put_32 (output_bfd, insn1, contents + rel->r_offset - d_offset); - bfd_put_32 (output_bfd, insn2, contents + offset); - if (tls_gd == 0) + r_type = R_PPC_TPREL16_HA; + rel->r_info = ELF32_R_INFO (r_symndx, r_type); + if (offset != (bfd_vma) -1) { - /* We changed the symbol on an LD reloc. Start over - in order to get h, sym, sec etc. right. */ - rel--; - continue; + rel[1].r_info = ELF32_R_INFO (r_symndx, R_PPC_TPREL16_LO); + rel[1].r_offset = offset + d_offset; + rel[1].r_addend = rel->r_addend; + insn2 = 0x38630000; /* addi 3,3,0 */ + bfd_put_32 (output_bfd, insn2, contents + offset); } } + bfd_put_32 (output_bfd, insn1, + contents + rel->r_offset - d_offset); + if (tls_gd == 0) + { + /* We changed the symbol on an LD reloc. Start over + in order to get h, sym, sec etc. right. */ + rel--; + continue; + } } break; + + case R_PPC_TLSGD: + if (tls_mask != 0 && (tls_mask & TLS_GD) == 0) + { + unsigned int insn2; + bfd_vma offset = rel->r_offset; + + if ((tls_mask & TLS_TPRELGD) != 0) + { + /* IE */ + r_type = R_PPC_NONE; + insn2 = 0x7c631214; /* add 3,3,2 */ + } + else + { + /* LE */ + r_type = R_PPC_TPREL16_LO; + rel->r_offset += d_offset; + insn2 = 0x38630000; /* addi 3,3,0 */ + } + rel->r_info = ELF32_R_INFO (r_symndx, r_type); + bfd_put_32 (output_bfd, insn2, contents + offset); + /* Zap the reloc on the _tls_get_addr call too. */ + BFD_ASSERT (offset == rel[1].r_offset); + rel[1].r_info = ELF32_R_INFO (ELF32_R_SYM (rel[1].r_info), + R_PPC_NONE); + } + break; + + case R_PPC_TLSLD: + if (tls_mask != 0 && (tls_mask & TLS_LD) == 0) + { + unsigned int insn2; + + for (r_symndx = 0; + r_symndx < symtab_hdr->sh_info; + r_symndx++) + if (local_sections[r_symndx] == sec) + break; + if (r_symndx >= symtab_hdr->sh_info) + r_symndx = 0; + rel->r_addend = htab->elf.tls_sec->vma + DTP_OFFSET; + if (r_symndx != 0) + rel->r_addend -= (local_syms[r_symndx].st_value + + sec->output_offset + + sec->output_section->vma); + + rel->r_info = ELF32_R_INFO (r_symndx, R_PPC_TPREL16_LO); + rel->r_offset += d_offset; + insn2 = 0x38630000; /* addi 3,3,0 */ + bfd_put_32 (output_bfd, insn2, + contents + rel->r_offset - d_offset); + /* Zap the reloc on the _tls_get_addr call too. */ + BFD_ASSERT (rel->r_offset - d_offset == rel[1].r_offset); + rel[1].r_info = ELF32_R_INFO (ELF32_R_SYM (rel[1].r_info), + R_PPC_NONE); + rel--; + continue; + } + break; } /* Handle other relocations that tweak non-addend part of insn. */ branch_bit = 0; switch (r_type) { default: break; /* Branch taken prediction relocations. */ case R_PPC_ADDR14_BRTAKEN: case R_PPC_REL14_BRTAKEN: branch_bit = BRANCH_PREDICT_BIT; /* Fall thru */ /* Branch not taken prediction relocations. */ case R_PPC_ADDR14_BRNTAKEN: case R_PPC_REL14_BRNTAKEN: insn = bfd_get_32 (output_bfd, contents + rel->r_offset); insn &= ~BRANCH_PREDICT_BIT; insn |= branch_bit; from = (rel->r_offset + input_section->output_offset + input_section->output_section->vma); /* Invert 'y' bit if not the default. */ if ((bfd_signed_vma) (relocation + rel->r_addend - from) < 0) insn ^= BRANCH_PREDICT_BIT; bfd_put_32 (output_bfd, insn, contents + rel->r_offset); break; } addend = rel->r_addend; tls_type = 0; howto = NULL; if (r_type < R_PPC_max) howto = ppc_elf_howto_table[r_type]; switch (r_type) { default: (*_bfd_error_handler) (_("%B: unknown relocation type %d for symbol %s"), input_bfd, (int) r_type, sym_name); bfd_set_error (bfd_error_bad_value); ret = FALSE; continue; case R_PPC_NONE: case R_PPC_TLS: + case R_PPC_TLSGD: + case R_PPC_TLSLD: case R_PPC_EMB_MRKREF: case R_PPC_GNU_VTINHERIT: case R_PPC_GNU_VTENTRY: continue; /* GOT16 relocations. Like an ADDR16 using the symbol's address in the GOT as relocation value instead of the symbol's value itself. Also, create a GOT entry for the symbol and put the symbol value there. */ case R_PPC_GOT_TLSGD16: case R_PPC_GOT_TLSGD16_LO: case R_PPC_GOT_TLSGD16_HI: case R_PPC_GOT_TLSGD16_HA: tls_type = TLS_TLS | TLS_GD; goto dogot; case R_PPC_GOT_TLSLD16: case R_PPC_GOT_TLSLD16_LO: case R_PPC_GOT_TLSLD16_HI: case R_PPC_GOT_TLSLD16_HA: tls_type = TLS_TLS | TLS_LD; goto dogot; case R_PPC_GOT_TPREL16: case R_PPC_GOT_TPREL16_LO: case R_PPC_GOT_TPREL16_HI: case R_PPC_GOT_TPREL16_HA: tls_type = TLS_TLS | TLS_TPREL; goto dogot; case R_PPC_GOT_DTPREL16: case R_PPC_GOT_DTPREL16_LO: case R_PPC_GOT_DTPREL16_HI: case R_PPC_GOT_DTPREL16_HA: tls_type = TLS_TLS | TLS_DTPREL; goto dogot; case R_PPC_GOT16: case R_PPC_GOT16_LO: case R_PPC_GOT16_HI: case R_PPC_GOT16_HA: + tls_mask = 0; dogot: { /* Relocation is to the entry for this symbol in the global offset table. */ bfd_vma off; bfd_vma *offp; unsigned long indx; if (htab->got == NULL) abort (); indx = 0; if (tls_type == (TLS_TLS | TLS_LD) && (h == NULL || !h->def_dynamic)) offp = &htab->tlsld_got.offset; else if (h != NULL) { bfd_boolean dyn; dyn = htab->elf.dynamic_sections_created; if (! WILL_CALL_FINISH_DYNAMIC_SYMBOL (dyn, info->shared, h) || (info->shared && SYMBOL_REFERENCES_LOCAL (info, h))) /* This is actually a static link, or it is a -Bsymbolic link and the symbol is defined locally, or the symbol was forced to be local because of a version file. */ ; else { indx = h->dynindx; unresolved_reloc = FALSE; } offp = &h->got.offset; } else { if (local_got_offsets == NULL) abort (); offp = &local_got_offsets[r_symndx]; } /* The offset must always be a multiple of 4. We use the least significant bit to record whether we have already processed this entry. */ off = *offp; if ((off & 1) != 0) off &= ~1; else { unsigned int tls_m = (tls_mask & (TLS_LD | TLS_GD | TLS_DTPREL | TLS_TPREL | TLS_TPRELGD)); if (offp == &htab->tlsld_got.offset) tls_m = TLS_LD; else if (h == NULL || !h->def_dynamic) tls_m &= ~TLS_LD; /* We might have multiple got entries for this sym. Initialize them all. */ do { int tls_ty = 0; if ((tls_m & TLS_LD) != 0) { tls_ty = TLS_TLS | TLS_LD; tls_m &= ~TLS_LD; } else if ((tls_m & TLS_GD) != 0) { tls_ty = TLS_TLS | TLS_GD; tls_m &= ~TLS_GD; } else if ((tls_m & TLS_DTPREL) != 0) { tls_ty = TLS_TLS | TLS_DTPREL; tls_m &= ~TLS_DTPREL; } else if ((tls_m & (TLS_TPREL | TLS_TPRELGD)) != 0) { tls_ty = TLS_TLS | TLS_TPREL; tls_m = 0; } /* Generate relocs for the dynamic linker. */ if ((info->shared || indx != 0) && (h == NULL || ELF_ST_VISIBILITY (h->other) == STV_DEFAULT || h->root.type != bfd_link_hash_undefweak)) { outrel.r_offset = (htab->got->output_section->vma + htab->got->output_offset + off); outrel.r_addend = 0; if (tls_ty & (TLS_LD | TLS_GD)) { outrel.r_info = ELF32_R_INFO (indx, R_PPC_DTPMOD32); if (tls_ty == (TLS_TLS | TLS_GD)) { loc = htab->relgot->contents; loc += (htab->relgot->reloc_count++ * sizeof (Elf32_External_Rela)); bfd_elf32_swap_reloca_out (output_bfd, &outrel, loc); outrel.r_offset += 4; outrel.r_info = ELF32_R_INFO (indx, R_PPC_DTPREL32); } } else if (tls_ty == (TLS_TLS | TLS_DTPREL)) outrel.r_info = ELF32_R_INFO (indx, R_PPC_DTPREL32); else if (tls_ty == (TLS_TLS | TLS_TPREL)) outrel.r_info = ELF32_R_INFO (indx, R_PPC_TPREL32); else if (indx == 0) outrel.r_info = ELF32_R_INFO (indx, R_PPC_RELATIVE); else outrel.r_info = ELF32_R_INFO (indx, R_PPC_GLOB_DAT); if (indx == 0) { outrel.r_addend += relocation; if (tls_ty & (TLS_GD | TLS_DTPREL | TLS_TPREL)) outrel.r_addend -= htab->elf.tls_sec->vma; } loc = htab->relgot->contents; loc += (htab->relgot->reloc_count++ * sizeof (Elf32_External_Rela)); bfd_elf32_swap_reloca_out (output_bfd, &outrel, loc); } /* Init the .got section contents if we're not emitting a reloc. */ else { bfd_vma value = relocation; if (tls_ty == (TLS_TLS | TLS_LD)) value = 1; else if (tls_ty != 0) { value -= htab->elf.tls_sec->vma + DTP_OFFSET; if (tls_ty == (TLS_TLS | TLS_TPREL)) value += DTP_OFFSET - TP_OFFSET; if (tls_ty == (TLS_TLS | TLS_GD)) { bfd_put_32 (output_bfd, value, htab->got->contents + off + 4); value = 1; } } bfd_put_32 (output_bfd, value, htab->got->contents + off); } off += 4; if (tls_ty & (TLS_LD | TLS_GD)) off += 4; } while (tls_m != 0); off = *offp; *offp = off | 1; } if (off >= (bfd_vma) -2) abort (); if ((tls_type & TLS_TLS) != 0) { if (tls_type != (TLS_TLS | TLS_LD)) { if ((tls_mask & TLS_LD) != 0 && !(h == NULL || !h->def_dynamic)) off += 8; if (tls_type != (TLS_TLS | TLS_GD)) { if ((tls_mask & TLS_GD) != 0) off += 8; if (tls_type != (TLS_TLS | TLS_DTPREL)) { if ((tls_mask & TLS_DTPREL) != 0) off += 4; } } } } relocation = htab->got->output_offset + off; relocation -= htab->elf.hgot->root.u.def.value; /* Addends on got relocations don't make much sense. x+off@got is actually x@got+off, and since the got is generated by a hash table traversal, the value in the got at entry m+n bears little relation to the entry m. */ if (addend != 0) (*_bfd_error_handler) (_("%B(%A+0x%lx): non-zero addend on %s reloc against `%s'"), input_bfd, input_section, (long) rel->r_offset, howto->name, sym_name); } break; /* Relocations that need no special processing. */ case R_PPC_LOCAL24PC: /* It makes no sense to point a local relocation at a symbol not in this object. */ if (unresolved_reloc) { if (! (*info->callbacks->undefined_symbol) (info, h->root.root.string, input_bfd, input_section, rel->r_offset, TRUE)) return FALSE; continue; } break; case R_PPC_DTPREL16: case R_PPC_DTPREL16_LO: case R_PPC_DTPREL16_HI: case R_PPC_DTPREL16_HA: addend -= htab->elf.tls_sec->vma + DTP_OFFSET; break; /* Relocations that may need to be propagated if this is a shared object. */ case R_PPC_TPREL16: case R_PPC_TPREL16_LO: case R_PPC_TPREL16_HI: case R_PPC_TPREL16_HA: addend -= htab->elf.tls_sec->vma + TP_OFFSET; /* The TPREL16 relocs shouldn't really be used in shared libs as they will result in DT_TEXTREL being set, but support them anyway. */ goto dodyn; case R_PPC_TPREL32: addend -= htab->elf.tls_sec->vma + TP_OFFSET; goto dodyn; case R_PPC_DTPREL32: addend -= htab->elf.tls_sec->vma + DTP_OFFSET; goto dodyn; case R_PPC_DTPMOD32: relocation = 1; addend = 0; goto dodyn; case R_PPC_REL16: case R_PPC_REL16_LO: case R_PPC_REL16_HI: case R_PPC_REL16_HA: break; case R_PPC_REL24: case R_PPC_REL32: case R_PPC_REL14: case R_PPC_REL14_BRTAKEN: case R_PPC_REL14_BRNTAKEN: /* If these relocations are not to a named symbol, they can be handled right here, no need to bother the dynamic linker. */ if (SYMBOL_REFERENCES_LOCAL (info, h) || h == htab->elf.hgot) break; /* fall through */ /* Relocations that always need to be propagated if this is a shared object. */ case R_PPC_ADDR32: case R_PPC_ADDR24: case R_PPC_ADDR16: case R_PPC_ADDR16_LO: case R_PPC_ADDR16_HI: case R_PPC_ADDR16_HA: case R_PPC_ADDR14: case R_PPC_ADDR14_BRTAKEN: case R_PPC_ADDR14_BRNTAKEN: case R_PPC_UADDR32: case R_PPC_UADDR16: dodyn: if ((input_section->flags & SEC_ALLOC) == 0) break; /* Fall thru. */ if ((info->shared && (h == NULL || ELF_ST_VISIBILITY (h->other) == STV_DEFAULT || h->root.type != bfd_link_hash_undefweak) - && (MUST_BE_DYN_RELOC (r_type) + && (must_be_dyn_reloc (info, r_type) || !SYMBOL_CALLS_LOCAL (info, h))) || (ELIMINATE_COPY_RELOCS && !info->shared && h != NULL && h->dynindx != -1 && !h->non_got_ref && h->def_dynamic && !h->def_regular)) { int skip; #ifdef DEBUG fprintf (stderr, "ppc_elf_relocate_section needs to " "create relocation for %s\n", (h && h->root.root.string ? h->root.root.string : "")); #endif /* When generating a shared object, these relocations are copied into the output file to be resolved at run time. */ if (sreloc == NULL) { const char *name; name = (bfd_elf_string_from_elf_section (input_bfd, elf_elfheader (input_bfd)->e_shstrndx, elf_section_data (input_section)->rel_hdr.sh_name)); if (name == NULL) return FALSE; BFD_ASSERT (CONST_STRNEQ (name, ".rela") && strcmp (bfd_get_section_name (input_bfd, input_section), name + 5) == 0); sreloc = bfd_get_section_by_name (htab->elf.dynobj, name); BFD_ASSERT (sreloc != NULL); } skip = 0; outrel.r_offset = _bfd_elf_section_offset (output_bfd, info, input_section, rel->r_offset); if (outrel.r_offset == (bfd_vma) -1 || outrel.r_offset == (bfd_vma) -2) skip = (int) outrel.r_offset; outrel.r_offset += (input_section->output_section->vma + input_section->output_offset); if (skip) memset (&outrel, 0, sizeof outrel); else if (!SYMBOL_REFERENCES_LOCAL (info, h)) { unresolved_reloc = FALSE; outrel.r_info = ELF32_R_INFO (h->dynindx, r_type); outrel.r_addend = rel->r_addend; } else { outrel.r_addend = relocation + rel->r_addend; if (r_type == R_PPC_ADDR32) outrel.r_info = ELF32_R_INFO (0, R_PPC_RELATIVE); else { - long indx; + long indx = 0; - if (bfd_is_abs_section (sec)) - indx = 0; + if (r_symndx == 0 || bfd_is_abs_section (sec)) + ; else if (sec == NULL || sec->owner == NULL) { bfd_set_error (bfd_error_bad_value); return FALSE; } else { asection *osec; /* We are turning this relocation into one against a section symbol. It would be proper to subtract the symbol's value, osec->vma, from the emitted reloc addend, but ld.so expects buggy relocs. */ osec = sec->output_section; indx = elf_section_data (osec)->dynindx; if (indx == 0) { osec = htab->elf.text_index_section; indx = elf_section_data (osec)->dynindx; } BFD_ASSERT (indx != 0); #ifdef DEBUG if (indx == 0) printf ("indx=%ld section=%s flags=%08x name=%s\n", indx, osec->name, osec->flags, h->root.root.string); #endif } outrel.r_info = ELF32_R_INFO (indx, r_type); } } loc = sreloc->contents; loc += sreloc->reloc_count++ * sizeof (Elf32_External_Rela); bfd_elf32_swap_reloca_out (output_bfd, &outrel, loc); if (skip == -1) continue; /* This reloc will be computed at runtime. We clear the memory so that it contains predictable value. */ if (! skip && ((input_section->flags & SEC_ALLOC) != 0 || ELF32_R_TYPE (outrel.r_info) != R_PPC_RELATIVE)) { relocation = howto->pc_relative ? outrel.r_offset : 0; addend = 0; break; } } break; case R_PPC_RELAX32PC_PLT: case R_PPC_RELAX32_PLT: { struct plt_entry *ent = find_plt_ent (h, got2, addend); if (htab->plt_type == PLT_NEW) relocation = (htab->glink->output_section->vma + htab->glink->output_offset + ent->glink_offset); else relocation = (htab->plt->output_section->vma + htab->plt->output_offset + ent->plt.offset); addend = 0; } if (r_type == R_PPC_RELAX32_PLT) goto relax32; /* Fall thru */ case R_PPC_RELAX32PC: relocation -= (input_section->output_section->vma + input_section->output_offset + rel->r_offset - 4); /* Fall thru */ case R_PPC_RELAX32: relax32: { unsigned long t0; unsigned long t1; t0 = bfd_get_32 (output_bfd, contents + rel->r_offset); t1 = bfd_get_32 (output_bfd, contents + rel->r_offset + 4); /* We're clearing the bits for R_PPC_ADDR16_HA and R_PPC_ADDR16_LO here. */ t0 &= ~0xffff; t1 &= ~0xffff; /* t0 is HA, t1 is LO */ relocation += addend; t0 |= ((relocation + 0x8000) >> 16) & 0xffff; t1 |= relocation & 0xffff; bfd_put_32 (output_bfd, t0, contents + rel->r_offset); bfd_put_32 (output_bfd, t1, contents + rel->r_offset + 4); } continue; /* Indirect .sdata relocation. */ case R_PPC_EMB_SDAI16: BFD_ASSERT (htab->sdata[0].section != NULL); relocation = elf_finish_pointer_linker_section (input_bfd, &htab->sdata[0], h, relocation, rel); break; /* Indirect .sdata2 relocation. */ case R_PPC_EMB_SDA2I16: BFD_ASSERT (htab->sdata[1].section != NULL); relocation = elf_finish_pointer_linker_section (input_bfd, &htab->sdata[1], h, relocation, rel); break; /* Handle the TOC16 reloc. We want to use the offset within the .got section, not the actual VMA. This is appropriate when generating an embedded ELF object, for which the .got section acts like the AIX .toc section. */ case R_PPC_TOC16: /* phony GOT16 relocations */ BFD_ASSERT (sec != NULL); BFD_ASSERT (bfd_is_und_section (sec) || strcmp (bfd_get_section_name (abfd, sec), ".got") == 0 || strcmp (bfd_get_section_name (abfd, sec), ".cgot") == 0); addend -= sec->output_section->vma + sec->output_offset + 0x8000; break; case R_PPC_PLTREL24: /* Relocation is to the entry for this symbol in the procedure linkage table. */ { struct plt_entry *ent = find_plt_ent (h, got2, addend); addend = 0; if (ent == NULL || htab->plt == NULL) { /* We didn't make a PLT entry for this symbol. This happens when statically linking PIC code, or when using -Bsymbolic. */ break; } unresolved_reloc = FALSE; if (htab->plt_type == PLT_NEW) relocation = (htab->glink->output_section->vma + htab->glink->output_offset + ent->glink_offset); else relocation = (htab->plt->output_section->vma + htab->plt->output_offset + ent->plt.offset); } break; /* Relocate against _SDA_BASE_. */ case R_PPC_SDAREL16: { const char *name; struct elf_link_hash_entry *sh; BFD_ASSERT (sec != NULL); name = bfd_get_section_name (abfd, sec->output_section); if (! ((CONST_STRNEQ (name, ".sdata") && (name[6] == 0 || name[6] == '.')) || (CONST_STRNEQ (name, ".sbss") && (name[5] == 0 || name[5] == '.')))) { (*_bfd_error_handler) (_("%B: the target (%s) of a %s relocation is " "in the wrong output section (%s)"), input_bfd, sym_name, howto->name, name); } sh = htab->sdata[0].sym; addend -= (sh->root.u.def.value + sh->root.u.def.section->output_offset + sh->root.u.def.section->output_section->vma); } break; /* Relocate against _SDA2_BASE_. */ case R_PPC_EMB_SDA2REL: { const char *name; struct elf_link_hash_entry *sh; BFD_ASSERT (sec != NULL); name = bfd_get_section_name (abfd, sec->output_section); if (! (CONST_STRNEQ (name, ".sdata2") || CONST_STRNEQ (name, ".sbss2"))) { (*_bfd_error_handler) (_("%B: the target (%s) of a %s relocation is " "in the wrong output section (%s)"), input_bfd, sym_name, howto->name, name); bfd_set_error (bfd_error_bad_value); ret = FALSE; continue; } sh = htab->sdata[1].sym; addend -= (sh->root.u.def.value + sh->root.u.def.section->output_offset + sh->root.u.def.section->output_section->vma); } break; /* Relocate against either _SDA_BASE_, _SDA2_BASE_, or 0. */ case R_PPC_EMB_SDA21: case R_PPC_EMB_RELSDA: { const char *name; int reg; struct elf_link_hash_entry *sh; BFD_ASSERT (sec != NULL); name = bfd_get_section_name (abfd, sec->output_section); if (((CONST_STRNEQ (name, ".sdata") && (name[6] == 0 || name[6] == '.')) || (CONST_STRNEQ (name, ".sbss") && (name[5] == 0 || name[5] == '.')))) { reg = 13; sh = htab->sdata[0].sym; addend -= (sh->root.u.def.value + sh->root.u.def.section->output_offset + sh->root.u.def.section->output_section->vma); } else if (CONST_STRNEQ (name, ".sdata2") || CONST_STRNEQ (name, ".sbss2")) { reg = 2; sh = htab->sdata[1].sym; addend -= (sh->root.u.def.value + sh->root.u.def.section->output_offset + sh->root.u.def.section->output_section->vma); } else if (strcmp (name, ".PPC.EMB.sdata0") == 0 || strcmp (name, ".PPC.EMB.sbss0") == 0) { reg = 0; } else { (*_bfd_error_handler) (_("%B: the target (%s) of a %s relocation is " "in the wrong output section (%s)"), input_bfd, sym_name, howto->name, name); bfd_set_error (bfd_error_bad_value); ret = FALSE; continue; } if (r_type == R_PPC_EMB_SDA21) { /* fill in register field */ insn = bfd_get_32 (output_bfd, contents + rel->r_offset); insn = (insn & ~RA_REGISTER_MASK) | (reg << RA_REGISTER_SHIFT); bfd_put_32 (output_bfd, insn, contents + rel->r_offset); } } break; /* Relocate against the beginning of the section. */ case R_PPC_SECTOFF: case R_PPC_SECTOFF_LO: case R_PPC_SECTOFF_HI: case R_PPC_SECTOFF_HA: BFD_ASSERT (sec != NULL); addend -= sec->output_section->vma; break; /* Negative relocations. */ case R_PPC_EMB_NADDR32: case R_PPC_EMB_NADDR16: case R_PPC_EMB_NADDR16_LO: case R_PPC_EMB_NADDR16_HI: case R_PPC_EMB_NADDR16_HA: addend -= 2 * relocation; break; case R_PPC_COPY: case R_PPC_GLOB_DAT: case R_PPC_JMP_SLOT: case R_PPC_RELATIVE: case R_PPC_PLT32: case R_PPC_PLTREL32: case R_PPC_PLT16_LO: case R_PPC_PLT16_HI: case R_PPC_PLT16_HA: case R_PPC_ADDR30: case R_PPC_EMB_RELSEC16: case R_PPC_EMB_RELST_LO: case R_PPC_EMB_RELST_HI: case R_PPC_EMB_RELST_HA: case R_PPC_EMB_BIT_FLD: (*_bfd_error_handler) (_("%B: relocation %s is not yet supported for symbol %s."), input_bfd, howto->name, sym_name); bfd_set_error (bfd_error_invalid_operation); ret = FALSE; continue; } /* Do any further special processing. */ switch (r_type) { default: break; case R_PPC_ADDR16_HA: case R_PPC_REL16_HA: case R_PPC_SECTOFF_HA: case R_PPC_TPREL16_HA: case R_PPC_DTPREL16_HA: case R_PPC_EMB_NADDR16_HA: case R_PPC_EMB_RELST_HA: /* It's just possible that this symbol is a weak symbol that's not actually defined anywhere. In that case, 'sec' would be NULL, and we should leave the symbol alone (it will be set to zero elsewhere in the link). */ if (sec == NULL) break; /* Fall thru */ case R_PPC_PLT16_HA: case R_PPC_GOT16_HA: case R_PPC_GOT_TLSGD16_HA: case R_PPC_GOT_TLSLD16_HA: case R_PPC_GOT_TPREL16_HA: case R_PPC_GOT_DTPREL16_HA: /* Add 0x10000 if sign bit in 0:15 is set. Bits 0:15 are not used. */ addend += 0x8000; break; } #ifdef DEBUG fprintf (stderr, "\ttype = %s (%d), name = %s, symbol index = %ld, " "offset = %ld, addend = %ld\n", howto->name, (int) r_type, sym_name, r_symndx, (long) rel->r_offset, (long) addend); #endif if (unresolved_reloc && !((input_section->flags & SEC_DEBUGGING) != 0 && h->def_dynamic)) { (*_bfd_error_handler) (_("%B(%A+0x%lx): unresolvable %s relocation against symbol `%s'"), input_bfd, input_section, (long) rel->r_offset, howto->name, sym_name); ret = FALSE; } r = _bfd_final_link_relocate (howto, input_bfd, input_section, contents, rel->r_offset, relocation, addend); if (r != bfd_reloc_ok) { if (r == bfd_reloc_overflow) { if (warned) continue; if (h != NULL && h->root.type == bfd_link_hash_undefweak && howto->pc_relative) { /* Assume this is a call protected by other code that detect the symbol is undefined. If this is the case, we can safely ignore the overflow. If not, the program is hosed anyway, and a little warning isn't going to help. */ continue; } if (! (*info->callbacks->reloc_overflow) (info, (h ? &h->root : NULL), sym_name, howto->name, rel->r_addend, input_bfd, input_section, rel->r_offset)) return FALSE; } else { (*_bfd_error_handler) (_("%B(%A+0x%lx): %s reloc against `%s': error %d"), input_bfd, input_section, (long) rel->r_offset, howto->name, sym_name, (int) r); ret = FALSE; } } } #ifdef DEBUG fprintf (stderr, "\n"); #endif return ret; } #define PPC_LO(v) ((v) & 0xffff) #define PPC_HI(v) (((v) >> 16) & 0xffff) #define PPC_HA(v) PPC_HI ((v) + 0x8000) /* Finish up dynamic symbol handling. We set the contents of various dynamic sections here. */ static bfd_boolean ppc_elf_finish_dynamic_symbol (bfd *output_bfd, struct bfd_link_info *info, struct elf_link_hash_entry *h, Elf_Internal_Sym *sym) { struct ppc_elf_link_hash_table *htab; struct plt_entry *ent; bfd_boolean doneone; #ifdef DEBUG fprintf (stderr, "ppc_elf_finish_dynamic_symbol called for %s", h->root.root.string); #endif htab = ppc_elf_hash_table (info); BFD_ASSERT (htab->elf.dynobj != NULL); doneone = FALSE; for (ent = h->plt.plist; ent != NULL; ent = ent->next) if (ent->plt.offset != (bfd_vma) -1) { if (!doneone) { Elf_Internal_Rela rela; bfd_byte *loc; bfd_vma reloc_index; if (htab->plt_type == PLT_NEW) reloc_index = ent->plt.offset / 4; else { reloc_index = ((ent->plt.offset - htab->plt_initial_entry_size) / htab->plt_slot_size); if (reloc_index > PLT_NUM_SINGLE_ENTRIES && htab->plt_type == PLT_OLD) reloc_index -= (reloc_index - PLT_NUM_SINGLE_ENTRIES) / 2; } /* This symbol has an entry in the procedure linkage table. Set it up. */ if (htab->plt_type == PLT_VXWORKS) { bfd_vma got_offset; const bfd_vma *plt_entry; /* The first three entries in .got.plt are reserved. */ got_offset = (reloc_index + 3) * 4; /* Use the right PLT. */ plt_entry = info->shared ? ppc_elf_vxworks_pic_plt_entry : ppc_elf_vxworks_plt_entry; /* Fill in the .plt on VxWorks. */ if (info->shared) { bfd_vma got_offset_hi = (got_offset >> 16) + ((got_offset & 0x8000) >> 15); bfd_put_32 (output_bfd, plt_entry[0] | (got_offset_hi & 0xffff), htab->plt->contents + ent->plt.offset + 0); bfd_put_32 (output_bfd, plt_entry[1] | (got_offset & 0xffff), htab->plt->contents + ent->plt.offset + 4); } else { bfd_vma got_loc = (got_offset + htab->elf.hgot->root.u.def.value + htab->elf.hgot->root.u.def.section->output_offset + htab->elf.hgot->root.u.def.section->output_section->vma); bfd_vma got_loc_hi = (got_loc >> 16) + ((got_loc & 0x8000) >> 15); bfd_put_32 (output_bfd, plt_entry[0] | (got_loc_hi & 0xffff), htab->plt->contents + ent->plt.offset + 0); bfd_put_32 (output_bfd, plt_entry[1] | (got_loc & 0xffff), htab->plt->contents + ent->plt.offset + 4); } bfd_put_32 (output_bfd, plt_entry[2], htab->plt->contents + ent->plt.offset + 8); bfd_put_32 (output_bfd, plt_entry[3], htab->plt->contents + ent->plt.offset + 12); /* This instruction is an immediate load. The value loaded is the byte offset of the R_PPC_JMP_SLOT relocation from the start of the .rela.plt section. The value is stored in the low-order 16 bits of the load instruction. */ /* NOTE: It appears that this is now an index rather than a prescaled offset. */ bfd_put_32 (output_bfd, plt_entry[4] | reloc_index, htab->plt->contents + ent->plt.offset + 16); /* This instruction is a PC-relative branch whose target is the start of the PLT section. The address of this branch instruction is 20 bytes beyond the start of this PLT entry. The address is encoded in bits 6-29, inclusive. The value stored is right-shifted by two bits, permitting a 26-bit offset. */ bfd_put_32 (output_bfd, (plt_entry[5] | (-(ent->plt.offset + 20) & 0x03fffffc)), htab->plt->contents + ent->plt.offset + 20); bfd_put_32 (output_bfd, plt_entry[6], htab->plt->contents + ent->plt.offset + 24); bfd_put_32 (output_bfd, plt_entry[7], htab->plt->contents + ent->plt.offset + 28); /* Fill in the GOT entry corresponding to this PLT slot with the address immediately after the the "bctr" instruction in this PLT entry. */ bfd_put_32 (output_bfd, (htab->plt->output_section->vma + htab->plt->output_offset + ent->plt.offset + 16), htab->sgotplt->contents + got_offset); if (!info->shared) { /* Fill in a couple of entries in .rela.plt.unloaded. */ loc = htab->srelplt2->contents + ((VXWORKS_PLTRESOLVE_RELOCS + reloc_index * VXWORKS_PLT_NON_JMP_SLOT_RELOCS) * sizeof (Elf32_External_Rela)); /* Provide the @ha relocation for the first instruction. */ rela.r_offset = (htab->plt->output_section->vma + htab->plt->output_offset + ent->plt.offset + 2); rela.r_info = ELF32_R_INFO (htab->elf.hgot->indx, R_PPC_ADDR16_HA); rela.r_addend = got_offset; bfd_elf32_swap_reloca_out (output_bfd, &rela, loc); loc += sizeof (Elf32_External_Rela); /* Provide the @l relocation for the second instruction. */ rela.r_offset = (htab->plt->output_section->vma + htab->plt->output_offset + ent->plt.offset + 6); rela.r_info = ELF32_R_INFO (htab->elf.hgot->indx, R_PPC_ADDR16_LO); rela.r_addend = got_offset; bfd_elf32_swap_reloca_out (output_bfd, &rela, loc); loc += sizeof (Elf32_External_Rela); /* Provide a relocation for the GOT entry corresponding to this PLT slot. Point it at the middle of the .plt entry. */ rela.r_offset = (htab->sgotplt->output_section->vma + htab->sgotplt->output_offset + got_offset); rela.r_info = ELF32_R_INFO (htab->elf.hplt->indx, R_PPC_ADDR32); rela.r_addend = ent->plt.offset + 16; bfd_elf32_swap_reloca_out (output_bfd, &rela, loc); } /* VxWorks uses non-standard semantics for R_PPC_JMP_SLOT. In particular, the offset for the relocation is not the address of the PLT entry for this function, as specified by the ABI. Instead, the offset is set to the address of the GOT slot for this function. See EABI 4.4.4.1. */ rela.r_offset = (htab->sgotplt->output_section->vma + htab->sgotplt->output_offset + got_offset); } else { rela.r_offset = (htab->plt->output_section->vma + htab->plt->output_offset + ent->plt.offset); if (htab->plt_type == PLT_OLD) { /* We don't need to fill in the .plt. The ppc dynamic linker will fill it in. */ } else { bfd_vma val = (htab->glink_pltresolve + ent->plt.offset + htab->glink->output_section->vma + htab->glink->output_offset); bfd_put_32 (output_bfd, val, htab->plt->contents + ent->plt.offset); } } /* Fill in the entry in the .rela.plt section. */ rela.r_info = ELF32_R_INFO (h->dynindx, R_PPC_JMP_SLOT); rela.r_addend = 0; loc = (htab->relplt->contents + reloc_index * sizeof (Elf32_External_Rela)); bfd_elf32_swap_reloca_out (output_bfd, &rela, loc); if (!h->def_regular) { /* Mark the symbol as undefined, rather than as defined in the .plt section. Leave the value alone. */ sym->st_shndx = SHN_UNDEF; /* If the symbol is weak, we do need to clear the value. Otherwise, the PLT entry would provide a definition for the symbol even if the symbol wasn't defined anywhere, and so the symbol would never be NULL. */ if (!h->ref_regular_nonweak) sym->st_value = 0; } doneone = TRUE; } if (htab->plt_type == PLT_NEW) { bfd_vma plt; unsigned char *p; plt = (ent->plt.offset + htab->plt->output_section->vma + htab->plt->output_offset); p = (unsigned char *) htab->glink->contents + ent->glink_offset; if (info->shared || info->pie) { bfd_vma got = 0; if (ent->addend >= 32768) got = (ent->addend + ent->sec->output_section->vma + ent->sec->output_offset); else if (htab->elf.hgot != NULL) got = (htab->elf.hgot->root.u.def.value + htab->elf.hgot->root.u.def.section->output_section->vma + htab->elf.hgot->root.u.def.section->output_offset); plt -= got; if (plt + 0x8000 < 0x10000) { bfd_put_32 (output_bfd, LWZ_11_30 + PPC_LO (plt), p); p += 4; bfd_put_32 (output_bfd, MTCTR_11, p); p += 4; bfd_put_32 (output_bfd, BCTR, p); p += 4; bfd_put_32 (output_bfd, NOP, p); p += 4; } else { bfd_put_32 (output_bfd, ADDIS_11_30 + PPC_HA (plt), p); p += 4; bfd_put_32 (output_bfd, LWZ_11_11 + PPC_LO (plt), p); p += 4; bfd_put_32 (output_bfd, MTCTR_11, p); p += 4; bfd_put_32 (output_bfd, BCTR, p); p += 4; } } else { bfd_put_32 (output_bfd, LIS_11 + PPC_HA (plt), p); p += 4; bfd_put_32 (output_bfd, LWZ_11_11 + PPC_LO (plt), p); p += 4; bfd_put_32 (output_bfd, MTCTR_11, p); p += 4; bfd_put_32 (output_bfd, BCTR, p); p += 4; /* We only need one non-PIC glink stub. */ break; } } else break; } if (h->needs_copy) { asection *s; Elf_Internal_Rela rela; bfd_byte *loc; /* This symbols needs a copy reloc. Set it up. */ #ifdef DEBUG fprintf (stderr, ", copy"); #endif BFD_ASSERT (h->dynindx != -1); if (ppc_elf_hash_entry (h)->has_sda_refs) s = htab->relsbss; else s = htab->relbss; BFD_ASSERT (s != NULL); rela.r_offset = (h->root.u.def.value + h->root.u.def.section->output_section->vma + h->root.u.def.section->output_offset); rela.r_info = ELF32_R_INFO (h->dynindx, R_PPC_COPY); rela.r_addend = 0; loc = s->contents + s->reloc_count++ * sizeof (Elf32_External_Rela); bfd_elf32_swap_reloca_out (output_bfd, &rela, loc); } #ifdef DEBUG fprintf (stderr, "\n"); #endif /* Mark some specially defined symbols as absolute. */ if (strcmp (h->root.root.string, "_DYNAMIC") == 0 || (!htab->is_vxworks && (h == htab->elf.hgot || strcmp (h->root.root.string, "_PROCEDURE_LINKAGE_TABLE_") == 0))) sym->st_shndx = SHN_ABS; return TRUE; } static enum elf_reloc_type_class ppc_elf_reloc_type_class (const Elf_Internal_Rela *rela) { switch (ELF32_R_TYPE (rela->r_info)) { case R_PPC_RELATIVE: return reloc_class_relative; case R_PPC_REL24: case R_PPC_ADDR24: case R_PPC_JMP_SLOT: return reloc_class_plt; case R_PPC_COPY: return reloc_class_copy; default: return reloc_class_normal; } } /* Finish up the dynamic sections. */ static bfd_boolean ppc_elf_finish_dynamic_sections (bfd *output_bfd, struct bfd_link_info *info) { asection *sdyn; asection *splt; struct ppc_elf_link_hash_table *htab; bfd_vma got; bfd * dynobj; #ifdef DEBUG fprintf (stderr, "ppc_elf_finish_dynamic_sections called\n"); #endif htab = ppc_elf_hash_table (info); dynobj = elf_hash_table (info)->dynobj; sdyn = bfd_get_section_by_name (dynobj, ".dynamic"); if (htab->is_vxworks) splt = bfd_get_section_by_name (dynobj, ".plt"); else splt = NULL; got = 0; if (htab->elf.hgot != NULL) got = (htab->elf.hgot->root.u.def.value + htab->elf.hgot->root.u.def.section->output_section->vma + htab->elf.hgot->root.u.def.section->output_offset); if (htab->elf.dynamic_sections_created) { Elf32_External_Dyn *dyncon, *dynconend; BFD_ASSERT (htab->plt != NULL && sdyn != NULL); dyncon = (Elf32_External_Dyn *) sdyn->contents; dynconend = (Elf32_External_Dyn *) (sdyn->contents + sdyn->size); for (; dyncon < dynconend; dyncon++) { Elf_Internal_Dyn dyn; asection *s; bfd_elf32_swap_dyn_in (dynobj, dyncon, &dyn); switch (dyn.d_tag) { case DT_PLTGOT: if (htab->is_vxworks) s = htab->sgotplt; else s = htab->plt; dyn.d_un.d_ptr = s->output_section->vma + s->output_offset; break; case DT_PLTRELSZ: dyn.d_un.d_val = htab->relplt->size; break; case DT_JMPREL: s = htab->relplt; dyn.d_un.d_ptr = s->output_section->vma + s->output_offset; break; case DT_PPC_GOT: dyn.d_un.d_ptr = got; break; case DT_RELASZ: if (htab->is_vxworks) { if (htab->relplt) dyn.d_un.d_ptr -= htab->relplt->size; break; } continue; default: continue; } bfd_elf32_swap_dyn_out (output_bfd, &dyn, dyncon); } } /* Add a blrl instruction at _GLOBAL_OFFSET_TABLE_-4 so that a function can easily find the address of the _GLOBAL_OFFSET_TABLE_. */ if (htab->got != NULL) { unsigned char *p = htab->got->contents; bfd_vma val; p += htab->elf.hgot->root.u.def.value; if (htab->plt_type == PLT_OLD) bfd_put_32 (output_bfd, 0x4e800021 /* blrl */, p - 4); val = 0; if (sdyn != NULL) val = sdyn->output_section->vma + sdyn->output_offset; bfd_put_32 (output_bfd, val, p); elf_section_data (htab->got->output_section)->this_hdr.sh_entsize = 4; } /* Fill in the first entry in the VxWorks procedure linkage table. */ if (splt && splt->size > 0) { /* Use the right PLT. */ static const bfd_vma *plt_entry = NULL; plt_entry = info->shared ? ppc_elf_vxworks_pic_plt0_entry : ppc_elf_vxworks_plt0_entry; if (!info->shared) { bfd_vma got_value = (htab->elf.hgot->root.u.def.section->output_section->vma + htab->elf.hgot->root.u.def.section->output_offset + htab->elf.hgot->root.u.def.value); bfd_vma got_hi = (got_value >> 16) + ((got_value & 0x8000) >> 15); bfd_put_32 (output_bfd, plt_entry[0] | (got_hi & 0xffff), splt->contents + 0); bfd_put_32 (output_bfd, plt_entry[1] | (got_value & 0xffff), splt->contents + 4); } else { bfd_put_32 (output_bfd, plt_entry[0], splt->contents + 0); bfd_put_32 (output_bfd, plt_entry[1], splt->contents + 4); } bfd_put_32 (output_bfd, plt_entry[2], splt->contents + 8); bfd_put_32 (output_bfd, plt_entry[3], splt->contents + 12); bfd_put_32 (output_bfd, plt_entry[4], splt->contents + 16); bfd_put_32 (output_bfd, plt_entry[5], splt->contents + 20); bfd_put_32 (output_bfd, plt_entry[6], splt->contents + 24); bfd_put_32 (output_bfd, plt_entry[7], splt->contents + 28); if (! info->shared) { Elf_Internal_Rela rela; bfd_byte *loc; loc = htab->srelplt2->contents; /* Output the @ha relocation for the first instruction. */ rela.r_offset = (htab->plt->output_section->vma + htab->plt->output_offset + 2); rela.r_info = ELF32_R_INFO (htab->elf.hgot->indx, R_PPC_ADDR16_HA); rela.r_addend = 0; bfd_elf32_swap_reloca_out (output_bfd, &rela, loc); loc += sizeof (Elf32_External_Rela); /* Output the @l relocation for the second instruction. */ rela.r_offset = (htab->plt->output_section->vma + htab->plt->output_offset + 6); rela.r_info = ELF32_R_INFO (htab->elf.hgot->indx, R_PPC_ADDR16_LO); rela.r_addend = 0; bfd_elf32_swap_reloca_out (output_bfd, &rela, loc); loc += sizeof (Elf32_External_Rela); /* Fix up the remaining relocations. They may have the wrong symbol index for _G_O_T_ or _P_L_T_ depending on the order in which symbols were output. */ while (loc < htab->srelplt2->contents + htab->srelplt2->size) { Elf_Internal_Rela rel; bfd_elf32_swap_reloc_in (output_bfd, loc, &rel); rel.r_info = ELF32_R_INFO (htab->elf.hgot->indx, R_PPC_ADDR16_HA); bfd_elf32_swap_reloc_out (output_bfd, &rel, loc); loc += sizeof (Elf32_External_Rela); bfd_elf32_swap_reloc_in (output_bfd, loc, &rel); rel.r_info = ELF32_R_INFO (htab->elf.hgot->indx, R_PPC_ADDR16_LO); bfd_elf32_swap_reloc_out (output_bfd, &rel, loc); loc += sizeof (Elf32_External_Rela); bfd_elf32_swap_reloc_in (output_bfd, loc, &rel); rel.r_info = ELF32_R_INFO (htab->elf.hplt->indx, R_PPC_ADDR32); bfd_elf32_swap_reloc_out (output_bfd, &rel, loc); loc += sizeof (Elf32_External_Rela); } } } if (htab->glink != NULL && htab->glink->contents != NULL) { unsigned char *p; unsigned char *endp; bfd_vma res0; unsigned int i; /* * PIC glink code is the following: * * # ith PLT code stub. * addis 11,30,(plt+(i-1)*4-got)@ha * lwz 11,(plt+(i-1)*4-got)@l(11) * mtctr 11 * bctr * * # A table of branches, one for each plt entry. * # The idea is that the plt call stub loads ctr (and r11) with these * # addresses, so (r11 - res_0) gives the plt index * 4. * res_0: b PLTresolve * res_1: b PLTresolve * . * # Some number of entries towards the end can be nops * res_n_m3: nop * res_n_m2: nop * res_n_m1: * * PLTresolve: * addis 11,11,(1f-res_0)@ha * mflr 0 * bcl 20,31,1f * 1: addi 11,11,(1b-res_0)@l * mflr 12 * mtlr 0 * sub 11,11,12 # r11 = index * 4 * addis 12,12,(got+4-1b)@ha * lwz 0,(got+4-1b)@l(12) # got[1] address of dl_runtime_resolve * lwz 12,(got+8-1b)@l(12) # got[2] contains the map address * mtctr 0 * add 0,11,11 * add 11,0,11 # r11 = index * 12 = reloc offset. * bctr */ static const unsigned int pic_plt_resolve[] = { ADDIS_11_11, MFLR_0, BCL_20_31, ADDI_11_11, MFLR_12, MTLR_0, SUB_11_11_12, ADDIS_12_12, LWZ_0_12, LWZ_12_12, MTCTR_0, ADD_0_11_11, ADD_11_0_11, BCTR, NOP, NOP }; static const unsigned int plt_resolve[] = { LIS_12, ADDIS_11_11, LWZ_0_12, ADDI_11_11, MTCTR_0, ADD_0_11_11, LWZ_12_12, ADD_11_0_11, BCTR, NOP, NOP, NOP, NOP, NOP, NOP, NOP }; if (ARRAY_SIZE (pic_plt_resolve) != GLINK_PLTRESOLVE / 4) abort (); if (ARRAY_SIZE (plt_resolve) != GLINK_PLTRESOLVE / 4) abort (); /* Build the branch table, one for each plt entry (less one), and perhaps some padding. */ p = htab->glink->contents; p += htab->glink_pltresolve; endp = htab->glink->contents; endp += htab->glink->size - GLINK_PLTRESOLVE; while (p < endp - 8 * 4) { bfd_put_32 (output_bfd, B + endp - p, p); p += 4; } while (p < endp) { bfd_put_32 (output_bfd, NOP, p); p += 4; } res0 = (htab->glink_pltresolve + htab->glink->output_section->vma + htab->glink->output_offset); /* Last comes the PLTresolve stub. */ if (info->shared || info->pie) { bfd_vma bcl; for (i = 0; i < ARRAY_SIZE (pic_plt_resolve); i++) { bfd_put_32 (output_bfd, pic_plt_resolve[i], p); p += 4; } p -= 4 * ARRAY_SIZE (pic_plt_resolve); bcl = (htab->glink->size - GLINK_PLTRESOLVE + 3*4 + htab->glink->output_section->vma + htab->glink->output_offset); bfd_put_32 (output_bfd, ADDIS_11_11 + PPC_HA (bcl - res0), p + 0*4); bfd_put_32 (output_bfd, ADDI_11_11 + PPC_LO (bcl - res0), p + 3*4); bfd_put_32 (output_bfd, ADDIS_12_12 + PPC_HA (got + 4 - bcl), p + 7*4); if (PPC_HA (got + 4 - bcl) == PPC_HA (got + 8 - bcl)) { bfd_put_32 (output_bfd, LWZ_0_12 + PPC_LO (got + 4 - bcl), p + 8*4); bfd_put_32 (output_bfd, LWZ_12_12 + PPC_LO (got + 8 - bcl), p + 9*4); } else { bfd_put_32 (output_bfd, LWZU_0_12 + PPC_LO (got + 4 - bcl), p + 8*4); bfd_put_32 (output_bfd, LWZ_12_12 + 4, p + 9*4); } } else { for (i = 0; i < ARRAY_SIZE (plt_resolve); i++) { bfd_put_32 (output_bfd, plt_resolve[i], p); p += 4; } p -= 4 * ARRAY_SIZE (plt_resolve); bfd_put_32 (output_bfd, LIS_12 + PPC_HA (got + 4), p + 0*4); bfd_put_32 (output_bfd, ADDIS_11_11 + PPC_HA (-res0), p + 1*4); bfd_put_32 (output_bfd, ADDI_11_11 + PPC_LO (-res0), p + 3*4); if (PPC_HA (got + 4) == PPC_HA (got + 8)) { bfd_put_32 (output_bfd, LWZ_0_12 + PPC_LO (got + 4), p + 2*4); bfd_put_32 (output_bfd, LWZ_12_12 + PPC_LO (got + 8), p + 6*4); } else { bfd_put_32 (output_bfd, LWZU_0_12 + PPC_LO (got + 4), p + 2*4); bfd_put_32 (output_bfd, LWZ_12_12 + 4, p + 6*4); } } } return TRUE; } #define TARGET_LITTLE_SYM bfd_elf32_powerpcle_vec #define TARGET_LITTLE_NAME "elf32-powerpcle" #define TARGET_BIG_SYM bfd_elf32_powerpc_vec #define TARGET_BIG_NAME "elf32-powerpc-freebsd" #define ELF_ARCH bfd_arch_powerpc #define ELF_MACHINE_CODE EM_PPC #ifdef __QNXTARGET__ #define ELF_MAXPAGESIZE 0x1000 #else #define ELF_MAXPAGESIZE 0x10000 #endif #define ELF_MINPAGESIZE 0x1000 #define ELF_COMMONPAGESIZE 0x1000 #define elf_info_to_howto ppc_elf_info_to_howto #ifdef EM_CYGNUS_POWERPC #define ELF_MACHINE_ALT1 EM_CYGNUS_POWERPC #endif #ifdef EM_PPC_OLD #define ELF_MACHINE_ALT2 EM_PPC_OLD #endif #define elf_backend_plt_not_loaded 1 #define elf_backend_can_gc_sections 1 #define elf_backend_can_refcount 1 #define elf_backend_rela_normal 1 #define bfd_elf32_mkobject ppc_elf_mkobject #define bfd_elf32_bfd_merge_private_bfd_data ppc_elf_merge_private_bfd_data #define bfd_elf32_bfd_relax_section ppc_elf_relax_section #define bfd_elf32_bfd_reloc_type_lookup ppc_elf_reloc_type_lookup #define bfd_elf32_bfd_reloc_name_lookup ppc_elf_reloc_name_lookup #define bfd_elf32_bfd_set_private_flags ppc_elf_set_private_flags #define bfd_elf32_bfd_link_hash_table_create ppc_elf_link_hash_table_create #define elf_backend_object_p ppc_elf_object_p #define elf_backend_gc_mark_hook ppc_elf_gc_mark_hook #define elf_backend_gc_sweep_hook ppc_elf_gc_sweep_hook #define elf_backend_section_from_shdr ppc_elf_section_from_shdr #define elf_backend_relocate_section ppc_elf_relocate_section #define elf_backend_create_dynamic_sections ppc_elf_create_dynamic_sections #define elf_backend_check_relocs ppc_elf_check_relocs #define elf_backend_copy_indirect_symbol ppc_elf_copy_indirect_symbol #define elf_backend_adjust_dynamic_symbol ppc_elf_adjust_dynamic_symbol #define elf_backend_add_symbol_hook ppc_elf_add_symbol_hook #define elf_backend_size_dynamic_sections ppc_elf_size_dynamic_sections #define elf_backend_finish_dynamic_symbol ppc_elf_finish_dynamic_symbol #define elf_backend_finish_dynamic_sections ppc_elf_finish_dynamic_sections #define elf_backend_fake_sections ppc_elf_fake_sections #define elf_backend_additional_program_headers ppc_elf_additional_program_headers #define elf_backend_grok_prstatus ppc_elf_grok_prstatus #define elf_backend_grok_psinfo ppc_elf_grok_psinfo #define elf_backend_write_core_note ppc_elf_write_core_note #define elf_backend_reloc_type_class ppc_elf_reloc_type_class #define elf_backend_begin_write_processing ppc_elf_begin_write_processing #define elf_backend_final_write_processing ppc_elf_final_write_processing #define elf_backend_write_section ppc_elf_write_section #define elf_backend_get_sec_type_attr ppc_elf_get_sec_type_attr #define elf_backend_plt_sym_val ppc_elf_plt_sym_val #define elf_backend_action_discarded ppc_elf_action_discarded #define elf_backend_init_index_section _bfd_elf_init_1_index_section #include "elf32-target.h" /* VxWorks Target */ #undef TARGET_LITTLE_SYM #undef TARGET_LITTLE_NAME #undef TARGET_BIG_SYM #define TARGET_BIG_SYM bfd_elf32_powerpc_vxworks_vec #undef TARGET_BIG_NAME #define TARGET_BIG_NAME "elf32-powerpc-vxworks" /* VxWorks uses the elf default section flags for .plt. */ static const struct bfd_elf_special_section * ppc_elf_vxworks_get_sec_type_attr (bfd *abfd ATTRIBUTE_UNUSED, asection *sec) { if (sec->name == NULL) return NULL; if (strcmp (sec->name, ".plt") == 0) return _bfd_elf_get_sec_type_attr (abfd, sec); return ppc_elf_get_sec_type_attr (abfd, sec); } /* Like ppc_elf_link_hash_table_create, but overrides appropriately for VxWorks. */ static struct bfd_link_hash_table * ppc_elf_vxworks_link_hash_table_create (bfd *abfd) { struct bfd_link_hash_table *ret; ret = ppc_elf_link_hash_table_create (abfd); if (ret) { struct ppc_elf_link_hash_table *htab = (struct ppc_elf_link_hash_table *)ret; htab->is_vxworks = 1; htab->plt_type = PLT_VXWORKS; htab->plt_entry_size = VXWORKS_PLT_ENTRY_SIZE; htab->plt_slot_size = VXWORKS_PLT_ENTRY_SIZE; htab->plt_initial_entry_size = VXWORKS_PLT_INITIAL_ENTRY_SIZE; } return ret; } /* Tweak magic VxWorks symbols as they are loaded. */ static bfd_boolean ppc_elf_vxworks_add_symbol_hook (bfd *abfd, struct bfd_link_info *info, Elf_Internal_Sym *sym, const char **namep ATTRIBUTE_UNUSED, flagword *flagsp ATTRIBUTE_UNUSED, asection **secp, bfd_vma *valp) { if (!elf_vxworks_add_symbol_hook(abfd, info, sym,namep, flagsp, secp, valp)) return FALSE; return ppc_elf_add_symbol_hook(abfd, info, sym,namep, flagsp, secp, valp); } static void ppc_elf_vxworks_final_write_processing (bfd *abfd, bfd_boolean linker) { ppc_elf_final_write_processing(abfd, linker); elf_vxworks_final_write_processing(abfd, linker); } /* On VxWorks, we emit relocations against _PROCEDURE_LINKAGE_TABLE_, so define it. */ #undef elf_backend_want_plt_sym #define elf_backend_want_plt_sym 1 #undef elf_backend_want_got_plt #define elf_backend_want_got_plt 1 #undef elf_backend_got_symbol_offset #define elf_backend_got_symbol_offset 0 #undef elf_backend_plt_not_loaded #define elf_backend_plt_not_loaded 0 #undef elf_backend_plt_readonly #define elf_backend_plt_readonly 1 #undef elf_backend_got_header_size #define elf_backend_got_header_size 12 #undef bfd_elf32_bfd_link_hash_table_create #define bfd_elf32_bfd_link_hash_table_create \ ppc_elf_vxworks_link_hash_table_create #undef elf_backend_add_symbol_hook #define elf_backend_add_symbol_hook \ ppc_elf_vxworks_add_symbol_hook #undef elf_backend_link_output_symbol_hook #define elf_backend_link_output_symbol_hook \ elf_vxworks_link_output_symbol_hook #undef elf_backend_final_write_processing #define elf_backend_final_write_processing \ ppc_elf_vxworks_final_write_processing #undef elf_backend_get_sec_type_attr #define elf_backend_get_sec_type_attr \ ppc_elf_vxworks_get_sec_type_attr #undef elf_backend_emit_relocs #define elf_backend_emit_relocs \ elf_vxworks_emit_relocs #undef elf32_bed #define elf32_bed ppc_elf_vxworks_bed #include "elf32-target.h" Index: projects/clang350-import/contrib/binutils/bfd/elf64-ppc.c =================================================================== --- projects/clang350-import/contrib/binutils/bfd/elf64-ppc.c (revision 275748) +++ projects/clang350-import/contrib/binutils/bfd/elf64-ppc.c (revision 275749) @@ -1,11500 +1,11835 @@ /* PowerPC64-specific support for 64-bit ELF. Copyright 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007 Free Software Foundation, Inc. Written by Linus Nordberg, Swox AB , based on elf32-ppc.c by Ian Lance Taylor. Largely rewritten by Alan Modra This file is part of BFD, the Binary File Descriptor library. This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin Street - Fifth Floor, Boston, MA 02110-1301, USA. */ /* The 64-bit PowerPC ELF ABI may be found at http://www.linuxbase.org/spec/ELF/ppc64/PPC-elf64abi.txt, and http://www.linuxbase.org/spec/ELF/ppc64/spec/book1.html */ #include "sysdep.h" #include #include "bfd.h" #include "bfdlink.h" #include "libbfd.h" #include "elf-bfd.h" #include "elf/ppc64.h" #include "elf64-ppc.h" static bfd_reloc_status_type ppc64_elf_ha_reloc (bfd *, arelent *, asymbol *, void *, asection *, bfd *, char **); static bfd_reloc_status_type ppc64_elf_branch_reloc (bfd *, arelent *, asymbol *, void *, asection *, bfd *, char **); static bfd_reloc_status_type ppc64_elf_brtaken_reloc (bfd *, arelent *, asymbol *, void *, asection *, bfd *, char **); static bfd_reloc_status_type ppc64_elf_sectoff_reloc (bfd *, arelent *, asymbol *, void *, asection *, bfd *, char **); static bfd_reloc_status_type ppc64_elf_sectoff_ha_reloc (bfd *, arelent *, asymbol *, void *, asection *, bfd *, char **); static bfd_reloc_status_type ppc64_elf_toc_reloc (bfd *, arelent *, asymbol *, void *, asection *, bfd *, char **); static bfd_reloc_status_type ppc64_elf_toc_ha_reloc (bfd *, arelent *, asymbol *, void *, asection *, bfd *, char **); static bfd_reloc_status_type ppc64_elf_toc64_reloc (bfd *, arelent *, asymbol *, void *, asection *, bfd *, char **); static bfd_reloc_status_type ppc64_elf_unhandled_reloc (bfd *, arelent *, asymbol *, void *, asection *, bfd *, char **); static bfd_vma opd_entry_value (asection *, bfd_vma, asection **, bfd_vma *); #define TARGET_LITTLE_SYM bfd_elf64_powerpcle_vec #define TARGET_LITTLE_NAME "elf64-powerpcle" #define TARGET_BIG_SYM bfd_elf64_powerpc_vec #define TARGET_BIG_NAME "elf64-powerpc-freebsd" #define ELF_ARCH bfd_arch_powerpc #define ELF_MACHINE_CODE EM_PPC64 #define ELF_MAXPAGESIZE 0x10000 #define ELF_COMMONPAGESIZE 0x1000 #define elf_info_to_howto ppc64_elf_info_to_howto #define elf_backend_want_got_sym 0 #define elf_backend_want_plt_sym 0 #define elf_backend_plt_alignment 3 #define elf_backend_plt_not_loaded 1 #define elf_backend_got_header_size 8 #define elf_backend_can_gc_sections 1 #define elf_backend_can_refcount 1 #define elf_backend_rela_normal 1 #define elf_backend_default_execstack 0 #define bfd_elf64_mkobject ppc64_elf_mkobject #define bfd_elf64_bfd_reloc_type_lookup ppc64_elf_reloc_type_lookup #define bfd_elf64_bfd_reloc_name_lookup ppc64_elf_reloc_name_lookup #define bfd_elf64_bfd_merge_private_bfd_data ppc64_elf_merge_private_bfd_data #define bfd_elf64_new_section_hook ppc64_elf_new_section_hook #define bfd_elf64_bfd_link_hash_table_create ppc64_elf_link_hash_table_create #define bfd_elf64_bfd_link_hash_table_free ppc64_elf_link_hash_table_free #define bfd_elf64_get_synthetic_symtab ppc64_elf_get_synthetic_symtab #define elf_backend_object_p ppc64_elf_object_p #define elf_backend_grok_prstatus ppc64_elf_grok_prstatus #define elf_backend_grok_psinfo ppc64_elf_grok_psinfo #define elf_backend_write_core_note ppc64_elf_write_core_note #define elf_backend_create_dynamic_sections ppc64_elf_create_dynamic_sections #define elf_backend_copy_indirect_symbol ppc64_elf_copy_indirect_symbol #define elf_backend_add_symbol_hook ppc64_elf_add_symbol_hook #define elf_backend_check_directives ppc64_elf_check_directives #define elf_backend_as_needed_cleanup ppc64_elf_as_needed_cleanup #define elf_backend_archive_symbol_lookup ppc64_elf_archive_symbol_lookup #define elf_backend_check_relocs ppc64_elf_check_relocs #define elf_backend_gc_mark_dynamic_ref ppc64_elf_gc_mark_dynamic_ref #define elf_backend_gc_mark_hook ppc64_elf_gc_mark_hook #define elf_backend_gc_sweep_hook ppc64_elf_gc_sweep_hook #define elf_backend_adjust_dynamic_symbol ppc64_elf_adjust_dynamic_symbol #define elf_backend_hide_symbol ppc64_elf_hide_symbol #define elf_backend_always_size_sections ppc64_elf_func_desc_adjust #define elf_backend_size_dynamic_sections ppc64_elf_size_dynamic_sections #define elf_backend_init_index_section _bfd_elf_init_2_index_sections #define elf_backend_action_discarded ppc64_elf_action_discarded #define elf_backend_relocate_section ppc64_elf_relocate_section #define elf_backend_finish_dynamic_symbol ppc64_elf_finish_dynamic_symbol #define elf_backend_reloc_type_class ppc64_elf_reloc_type_class #define elf_backend_finish_dynamic_sections ppc64_elf_finish_dynamic_sections #define elf_backend_link_output_symbol_hook ppc64_elf_output_symbol_hook #define elf_backend_special_sections ppc64_elf_special_sections /* The name of the dynamic interpreter. This is put in the .interp section. */ #define ELF_DYNAMIC_INTERPRETER "/usr/lib/ld.so.1" /* The size in bytes of an entry in the procedure linkage table. */ #define PLT_ENTRY_SIZE 24 /* The initial size of the plt reserved for the dynamic linker. */ #define PLT_INITIAL_ENTRY_SIZE PLT_ENTRY_SIZE /* TOC base pointers offset from start of TOC. */ #define TOC_BASE_OFF 0x8000 /* Offset of tp and dtp pointers from start of TLS block. */ #define TP_OFFSET 0x7000 #define DTP_OFFSET 0x8000 /* .plt call stub instructions. The normal stub is like this, but sometimes the .plt entry crosses a 64k boundary and we need to - insert an addis to adjust r12. */ + insert an addi to adjust r12. */ #define PLT_CALL_STUB_SIZE (7*4) #define ADDIS_R12_R2 0x3d820000 /* addis %r12,%r2,xxx@ha */ #define STD_R2_40R1 0xf8410028 /* std %r2,40(%r1) */ #define LD_R11_0R12 0xe96c0000 /* ld %r11,xxx+0@l(%r12) */ -#define LD_R2_0R12 0xe84c0000 /* ld %r2,xxx+8@l(%r12) */ #define MTCTR_R11 0x7d6903a6 /* mtctr %r11 */ +#define LD_R2_0R12 0xe84c0000 /* ld %r2,xxx+8@l(%r12) */ /* ld %r11,xxx+16@l(%r12) */ #define BCTR 0x4e800420 /* bctr */ #define ADDIS_R12_R12 0x3d8c0000 /* addis %r12,%r12,off@ha */ +#define ADDI_R12_R12 0x398c0000 /* addi %r12,%r12,off@l */ #define ADDIS_R2_R2 0x3c420000 /* addis %r2,%r2,off@ha */ #define ADDI_R2_R2 0x38420000 /* addi %r2,%r2,off@l */ +#define LD_R11_0R2 0xe9620000 /* ld %r11,xxx+0(%r2) */ +#define LD_R2_0R2 0xe8420000 /* ld %r2,xxx+0(%r2) */ + #define LD_R2_40R1 0xe8410028 /* ld %r2,40(%r1) */ /* glink call stub instructions. We enter with the index in R0. */ #define GLINK_CALL_STUB_SIZE (16*4) /* 0: */ /* .quad plt0-1f */ /* __glink: */ #define MFLR_R12 0x7d8802a6 /* mflr %12 */ #define BCL_20_31 0x429f0005 /* bcl 20,31,1f */ /* 1: */ #define MFLR_R11 0x7d6802a6 /* mflr %11 */ #define LD_R2_M16R11 0xe84bfff0 /* ld %2,(0b-1b)(%11) */ #define MTLR_R12 0x7d8803a6 /* mtlr %12 */ #define ADD_R12_R2_R11 0x7d825a14 /* add %12,%2,%11 */ /* ld %11,0(%12) */ /* ld %2,8(%12) */ /* mtctr %11 */ /* ld %11,16(%12) */ /* bctr */ /* Pad with this. */ #define NOP 0x60000000 /* Some other nops. */ #define CROR_151515 0x4def7b82 #define CROR_313131 0x4ffffb82 /* .glink entries for the first 32k functions are two instructions. */ #define LI_R0_0 0x38000000 /* li %r0,0 */ #define B_DOT 0x48000000 /* b . */ /* After that, we need two instructions to load the index, followed by a branch. */ #define LIS_R0_0 0x3c000000 /* lis %r0,0 */ #define ORI_R0_R0_0 0x60000000 /* ori %r0,%r0,0 */ /* Instructions used by the save and restore reg functions. */ #define STD_R0_0R1 0xf8010000 /* std %r0,0(%r1) */ #define STD_R0_0R12 0xf80c0000 /* std %r0,0(%r12) */ #define LD_R0_0R1 0xe8010000 /* ld %r0,0(%r1) */ #define LD_R0_0R12 0xe80c0000 /* ld %r0,0(%r12) */ #define STFD_FR0_0R1 0xd8010000 /* stfd %fr0,0(%r1) */ #define LFD_FR0_0R1 0xc8010000 /* lfd %fr0,0(%r1) */ #define LI_R12_0 0x39800000 /* li %r12,0 */ #define STVX_VR0_R12_R0 0x7c0c01ce /* stvx %v0,%r12,%r0 */ #define LVX_VR0_R12_R0 0x7c0c00ce /* lvx %v0,%r12,%r0 */ #define MTLR_R0 0x7c0803a6 /* mtlr %r0 */ #define BLR 0x4e800020 /* blr */ /* Since .opd is an array of descriptors and each entry will end up with identical R_PPC64_RELATIVE relocs, there is really no need to propagate .opd relocs; The dynamic linker should be taught to relocate .opd without reloc entries. */ #ifndef NO_OPD_RELOCS #define NO_OPD_RELOCS 0 #endif #define ONES(n) (((bfd_vma) 1 << ((n) - 1) << 1) - 1) /* Relocation HOWTO's. */ static reloc_howto_type *ppc64_elf_howto_table[(int) R_PPC64_max]; static reloc_howto_type ppc64_elf_howto_raw[] = { /* This reloc does nothing. */ HOWTO (R_PPC64_NONE, /* type */ 0, /* rightshift */ 2, /* size (0 = byte, 1 = short, 2 = long) */ 32, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC64_NONE", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0, /* dst_mask */ FALSE), /* pcrel_offset */ /* A standard 32 bit relocation. */ HOWTO (R_PPC64_ADDR32, /* type */ 0, /* rightshift */ 2, /* size (0 = byte, 1 = short, 2 = long) */ 32, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_bitfield, /* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC64_ADDR32", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffffffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* An absolute 26 bit branch; the lower two bits must be zero. FIXME: we don't check that, we just clear them. */ HOWTO (R_PPC64_ADDR24, /* type */ 0, /* rightshift */ 2, /* size (0 = byte, 1 = short, 2 = long) */ 26, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_bitfield, /* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC64_ADDR24", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0x03fffffc, /* dst_mask */ FALSE), /* pcrel_offset */ /* A standard 16 bit relocation. */ HOWTO (R_PPC64_ADDR16, /* type */ 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_bitfield, /* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC64_ADDR16", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* A 16 bit relocation without overflow. */ HOWTO (R_PPC64_ADDR16_LO, /* type */ 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont,/* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC64_ADDR16_LO", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Bits 16-31 of an address. */ HOWTO (R_PPC64_ADDR16_HI, /* type */ 16, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC64_ADDR16_HI", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Bits 16-31 of an address, plus 1 if the contents of the low 16 bits, treated as a signed number, is negative. */ HOWTO (R_PPC64_ADDR16_HA, /* type */ 16, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc64_elf_ha_reloc, /* special_function */ "R_PPC64_ADDR16_HA", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* An absolute 16 bit branch; the lower two bits must be zero. FIXME: we don't check that, we just clear them. */ HOWTO (R_PPC64_ADDR14, /* type */ 0, /* rightshift */ 2, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_bitfield, /* complain_on_overflow */ ppc64_elf_branch_reloc, /* special_function */ "R_PPC64_ADDR14", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0x0000fffc, /* dst_mask */ FALSE), /* pcrel_offset */ /* An absolute 16 bit branch, for which bit 10 should be set to indicate that the branch is expected to be taken. The lower two bits must be zero. */ HOWTO (R_PPC64_ADDR14_BRTAKEN, /* type */ 0, /* rightshift */ 2, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_bitfield, /* complain_on_overflow */ ppc64_elf_brtaken_reloc, /* special_function */ "R_PPC64_ADDR14_BRTAKEN",/* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0x0000fffc, /* dst_mask */ FALSE), /* pcrel_offset */ /* An absolute 16 bit branch, for which bit 10 should be set to indicate that the branch is not expected to be taken. The lower two bits must be zero. */ HOWTO (R_PPC64_ADDR14_BRNTAKEN, /* type */ 0, /* rightshift */ 2, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_bitfield, /* complain_on_overflow */ ppc64_elf_brtaken_reloc, /* special_function */ "R_PPC64_ADDR14_BRNTAKEN",/* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0x0000fffc, /* dst_mask */ FALSE), /* pcrel_offset */ /* A relative 26 bit branch; the lower two bits must be zero. */ HOWTO (R_PPC64_REL24, /* type */ 0, /* rightshift */ 2, /* size (0 = byte, 1 = short, 2 = long) */ 26, /* bitsize */ TRUE, /* pc_relative */ 0, /* bitpos */ complain_overflow_signed, /* complain_on_overflow */ ppc64_elf_branch_reloc, /* special_function */ "R_PPC64_REL24", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0x03fffffc, /* dst_mask */ TRUE), /* pcrel_offset */ /* A relative 16 bit branch; the lower two bits must be zero. */ HOWTO (R_PPC64_REL14, /* type */ 0, /* rightshift */ 2, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ TRUE, /* pc_relative */ 0, /* bitpos */ complain_overflow_signed, /* complain_on_overflow */ ppc64_elf_branch_reloc, /* special_function */ "R_PPC64_REL14", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0x0000fffc, /* dst_mask */ TRUE), /* pcrel_offset */ /* A relative 16 bit branch. Bit 10 should be set to indicate that the branch is expected to be taken. The lower two bits must be zero. */ HOWTO (R_PPC64_REL14_BRTAKEN, /* type */ 0, /* rightshift */ 2, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ TRUE, /* pc_relative */ 0, /* bitpos */ complain_overflow_signed, /* complain_on_overflow */ ppc64_elf_brtaken_reloc, /* special_function */ "R_PPC64_REL14_BRTAKEN", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0x0000fffc, /* dst_mask */ TRUE), /* pcrel_offset */ /* A relative 16 bit branch. Bit 10 should be set to indicate that the branch is not expected to be taken. The lower two bits must be zero. */ HOWTO (R_PPC64_REL14_BRNTAKEN, /* type */ 0, /* rightshift */ 2, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ TRUE, /* pc_relative */ 0, /* bitpos */ complain_overflow_signed, /* complain_on_overflow */ ppc64_elf_brtaken_reloc, /* special_function */ "R_PPC64_REL14_BRNTAKEN",/* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0x0000fffc, /* dst_mask */ TRUE), /* pcrel_offset */ /* Like R_PPC64_ADDR16, but referring to the GOT table entry for the symbol. */ HOWTO (R_PPC64_GOT16, /* type */ 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_signed, /* complain_on_overflow */ ppc64_elf_unhandled_reloc, /* special_function */ "R_PPC64_GOT16", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like R_PPC64_ADDR16_LO, but referring to the GOT table entry for the symbol. */ HOWTO (R_PPC64_GOT16_LO, /* type */ 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc64_elf_unhandled_reloc, /* special_function */ "R_PPC64_GOT16_LO", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like R_PPC64_ADDR16_HI, but referring to the GOT table entry for the symbol. */ HOWTO (R_PPC64_GOT16_HI, /* type */ 16, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont,/* complain_on_overflow */ ppc64_elf_unhandled_reloc, /* special_function */ "R_PPC64_GOT16_HI", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like R_PPC64_ADDR16_HA, but referring to the GOT table entry for the symbol. */ HOWTO (R_PPC64_GOT16_HA, /* type */ 16, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont,/* complain_on_overflow */ ppc64_elf_unhandled_reloc, /* special_function */ "R_PPC64_GOT16_HA", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* This is used only by the dynamic linker. The symbol should exist both in the object being run and in some shared library. The dynamic linker copies the data addressed by the symbol from the shared library into the object, because the object being run has to have the data at some particular address. */ HOWTO (R_PPC64_COPY, /* type */ 0, /* rightshift */ 0, /* this one is variable size */ 0, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc64_elf_unhandled_reloc, /* special_function */ "R_PPC64_COPY", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like R_PPC64_ADDR64, but used when setting global offset table entries. */ HOWTO (R_PPC64_GLOB_DAT, /* type */ 0, /* rightshift */ 4, /* size (0=byte, 1=short, 2=long, 4=64 bits) */ 64, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc64_elf_unhandled_reloc, /* special_function */ "R_PPC64_GLOB_DAT", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ ONES (64), /* dst_mask */ FALSE), /* pcrel_offset */ /* Created by the link editor. Marks a procedure linkage table entry for a symbol. */ HOWTO (R_PPC64_JMP_SLOT, /* type */ 0, /* rightshift */ 0, /* size (0 = byte, 1 = short, 2 = long) */ 0, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc64_elf_unhandled_reloc, /* special_function */ "R_PPC64_JMP_SLOT", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0, /* dst_mask */ FALSE), /* pcrel_offset */ /* Used only by the dynamic linker. When the object is run, this doubleword64 is set to the load address of the object, plus the addend. */ HOWTO (R_PPC64_RELATIVE, /* type */ 0, /* rightshift */ 4, /* size (0=byte, 1=short, 2=long, 4=64 bits) */ 64, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC64_RELATIVE", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ ONES (64), /* dst_mask */ FALSE), /* pcrel_offset */ /* Like R_PPC64_ADDR32, but may be unaligned. */ HOWTO (R_PPC64_UADDR32, /* type */ 0, /* rightshift */ 2, /* size (0 = byte, 1 = short, 2 = long) */ 32, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_bitfield, /* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC64_UADDR32", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffffffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like R_PPC64_ADDR16, but may be unaligned. */ HOWTO (R_PPC64_UADDR16, /* type */ 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_bitfield, /* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC64_UADDR16", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* 32-bit PC relative. */ HOWTO (R_PPC64_REL32, /* type */ 0, /* rightshift */ 2, /* size (0 = byte, 1 = short, 2 = long) */ 32, /* bitsize */ TRUE, /* pc_relative */ 0, /* bitpos */ /* FIXME: Verify. Was complain_overflow_bitfield. */ complain_overflow_signed, /* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC64_REL32", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffffffff, /* dst_mask */ TRUE), /* pcrel_offset */ /* 32-bit relocation to the symbol's procedure linkage table. */ HOWTO (R_PPC64_PLT32, /* type */ 0, /* rightshift */ 2, /* size (0 = byte, 1 = short, 2 = long) */ 32, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_bitfield, /* complain_on_overflow */ ppc64_elf_unhandled_reloc, /* special_function */ "R_PPC64_PLT32", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffffffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* 32-bit PC relative relocation to the symbol's procedure linkage table. FIXME: R_PPC64_PLTREL32 not supported. */ HOWTO (R_PPC64_PLTREL32, /* type */ 0, /* rightshift */ 2, /* size (0 = byte, 1 = short, 2 = long) */ 32, /* bitsize */ TRUE, /* pc_relative */ 0, /* bitpos */ complain_overflow_signed, /* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC64_PLTREL32", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffffffff, /* dst_mask */ TRUE), /* pcrel_offset */ /* Like R_PPC64_ADDR16_LO, but referring to the PLT table entry for the symbol. */ HOWTO (R_PPC64_PLT16_LO, /* type */ 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc64_elf_unhandled_reloc, /* special_function */ "R_PPC64_PLT16_LO", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like R_PPC64_ADDR16_HI, but referring to the PLT table entry for the symbol. */ HOWTO (R_PPC64_PLT16_HI, /* type */ 16, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc64_elf_unhandled_reloc, /* special_function */ "R_PPC64_PLT16_HI", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like R_PPC64_ADDR16_HA, but referring to the PLT table entry for the symbol. */ HOWTO (R_PPC64_PLT16_HA, /* type */ 16, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc64_elf_unhandled_reloc, /* special_function */ "R_PPC64_PLT16_HA", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* 16-bit section relative relocation. */ HOWTO (R_PPC64_SECTOFF, /* type */ 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_bitfield, /* complain_on_overflow */ ppc64_elf_sectoff_reloc, /* special_function */ "R_PPC64_SECTOFF", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like R_PPC64_SECTOFF, but no overflow warning. */ HOWTO (R_PPC64_SECTOFF_LO, /* type */ 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc64_elf_sectoff_reloc, /* special_function */ "R_PPC64_SECTOFF_LO", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* 16-bit upper half section relative relocation. */ HOWTO (R_PPC64_SECTOFF_HI, /* type */ 16, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc64_elf_sectoff_reloc, /* special_function */ "R_PPC64_SECTOFF_HI", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* 16-bit upper half adjusted section relative relocation. */ HOWTO (R_PPC64_SECTOFF_HA, /* type */ 16, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc64_elf_sectoff_ha_reloc, /* special_function */ "R_PPC64_SECTOFF_HA", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like R_PPC64_REL24 without touching the two least significant bits. */ HOWTO (R_PPC64_REL30, /* type */ 2, /* rightshift */ 2, /* size (0 = byte, 1 = short, 2 = long) */ 30, /* bitsize */ TRUE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC64_REL30", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xfffffffc, /* dst_mask */ TRUE), /* pcrel_offset */ /* Relocs in the 64-bit PowerPC ELF ABI, not in the 32-bit ABI. */ /* A standard 64-bit relocation. */ HOWTO (R_PPC64_ADDR64, /* type */ 0, /* rightshift */ 4, /* size (0=byte, 1=short, 2=long, 4=64 bits) */ 64, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC64_ADDR64", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ ONES (64), /* dst_mask */ FALSE), /* pcrel_offset */ /* The bits 32-47 of an address. */ HOWTO (R_PPC64_ADDR16_HIGHER, /* type */ 32, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC64_ADDR16_HIGHER", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* The bits 32-47 of an address, plus 1 if the contents of the low 16 bits, treated as a signed number, is negative. */ HOWTO (R_PPC64_ADDR16_HIGHERA, /* type */ 32, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc64_elf_ha_reloc, /* special_function */ "R_PPC64_ADDR16_HIGHERA", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* The bits 48-63 of an address. */ HOWTO (R_PPC64_ADDR16_HIGHEST,/* type */ 48, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC64_ADDR16_HIGHEST", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* The bits 48-63 of an address, plus 1 if the contents of the low 16 bits, treated as a signed number, is negative. */ HOWTO (R_PPC64_ADDR16_HIGHESTA,/* type */ 48, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc64_elf_ha_reloc, /* special_function */ "R_PPC64_ADDR16_HIGHESTA", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like ADDR64, but may be unaligned. */ HOWTO (R_PPC64_UADDR64, /* type */ 0, /* rightshift */ 4, /* size (0=byte, 1=short, 2=long, 4=64 bits) */ 64, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC64_UADDR64", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ ONES (64), /* dst_mask */ FALSE), /* pcrel_offset */ /* 64-bit relative relocation. */ HOWTO (R_PPC64_REL64, /* type */ 0, /* rightshift */ 4, /* size (0=byte, 1=short, 2=long, 4=64 bits) */ 64, /* bitsize */ TRUE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC64_REL64", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ ONES (64), /* dst_mask */ TRUE), /* pcrel_offset */ /* 64-bit relocation to the symbol's procedure linkage table. */ HOWTO (R_PPC64_PLT64, /* type */ 0, /* rightshift */ 4, /* size (0=byte, 1=short, 2=long, 4=64 bits) */ 64, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc64_elf_unhandled_reloc, /* special_function */ "R_PPC64_PLT64", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ ONES (64), /* dst_mask */ FALSE), /* pcrel_offset */ /* 64-bit PC relative relocation to the symbol's procedure linkage table. */ /* FIXME: R_PPC64_PLTREL64 not supported. */ HOWTO (R_PPC64_PLTREL64, /* type */ 0, /* rightshift */ 4, /* size (0=byte, 1=short, 2=long, 4=64 bits) */ 64, /* bitsize */ TRUE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc64_elf_unhandled_reloc, /* special_function */ "R_PPC64_PLTREL64", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ ONES (64), /* dst_mask */ TRUE), /* pcrel_offset */ /* 16 bit TOC-relative relocation. */ /* R_PPC64_TOC16 47 half16* S + A - .TOC. */ HOWTO (R_PPC64_TOC16, /* type */ 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_signed, /* complain_on_overflow */ ppc64_elf_toc_reloc, /* special_function */ "R_PPC64_TOC16", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* 16 bit TOC-relative relocation without overflow. */ /* R_PPC64_TOC16_LO 48 half16 #lo (S + A - .TOC.) */ HOWTO (R_PPC64_TOC16_LO, /* type */ 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc64_elf_toc_reloc, /* special_function */ "R_PPC64_TOC16_LO", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* 16 bit TOC-relative relocation, high 16 bits. */ /* R_PPC64_TOC16_HI 49 half16 #hi (S + A - .TOC.) */ HOWTO (R_PPC64_TOC16_HI, /* type */ 16, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc64_elf_toc_reloc, /* special_function */ "R_PPC64_TOC16_HI", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* 16 bit TOC-relative relocation, high 16 bits, plus 1 if the contents of the low 16 bits, treated as a signed number, is negative. */ /* R_PPC64_TOC16_HA 50 half16 #ha (S + A - .TOC.) */ HOWTO (R_PPC64_TOC16_HA, /* type */ 16, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc64_elf_toc_ha_reloc, /* special_function */ "R_PPC64_TOC16_HA", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* 64-bit relocation; insert value of TOC base (.TOC.). */ /* R_PPC64_TOC 51 doubleword64 .TOC. */ HOWTO (R_PPC64_TOC, /* type */ 0, /* rightshift */ 4, /* size (0=byte, 1=short, 2=long, 4=64 bits) */ 64, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_bitfield, /* complain_on_overflow */ ppc64_elf_toc64_reloc, /* special_function */ "R_PPC64_TOC", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ ONES (64), /* dst_mask */ FALSE), /* pcrel_offset */ /* Like R_PPC64_GOT16, but also informs the link editor that the value to relocate may (!) refer to a PLT entry which the link editor (a) may replace with the symbol value. If the link editor is unable to fully resolve the symbol, it may (b) create a PLT entry and store the address to the new PLT entry in the GOT. This permits lazy resolution of function symbols at run time. The link editor may also skip all of this and just (c) emit a R_PPC64_GLOB_DAT to tie the symbol to the GOT entry. */ /* FIXME: R_PPC64_PLTGOT16 not implemented. */ HOWTO (R_PPC64_PLTGOT16, /* type */ 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_signed, /* complain_on_overflow */ ppc64_elf_unhandled_reloc, /* special_function */ "R_PPC64_PLTGOT16", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like R_PPC64_PLTGOT16, but without overflow. */ /* FIXME: R_PPC64_PLTGOT16_LO not implemented. */ HOWTO (R_PPC64_PLTGOT16_LO, /* type */ 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc64_elf_unhandled_reloc, /* special_function */ "R_PPC64_PLTGOT16_LO", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like R_PPC64_PLT_GOT16, but using bits 16-31 of the address. */ /* FIXME: R_PPC64_PLTGOT16_HI not implemented. */ HOWTO (R_PPC64_PLTGOT16_HI, /* type */ 16, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc64_elf_unhandled_reloc, /* special_function */ "R_PPC64_PLTGOT16_HI", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like R_PPC64_PLT_GOT16, but using bits 16-31 of the address, plus 1 if the contents of the low 16 bits, treated as a signed number, is negative. */ /* FIXME: R_PPC64_PLTGOT16_HA not implemented. */ HOWTO (R_PPC64_PLTGOT16_HA, /* type */ 16, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont,/* complain_on_overflow */ ppc64_elf_unhandled_reloc, /* special_function */ "R_PPC64_PLTGOT16_HA", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like R_PPC64_ADDR16, but for instructions with a DS field. */ HOWTO (R_PPC64_ADDR16_DS, /* type */ 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_bitfield, /* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC64_ADDR16_DS", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xfffc, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like R_PPC64_ADDR16_LO, but for instructions with a DS field. */ HOWTO (R_PPC64_ADDR16_LO_DS, /* type */ 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont,/* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC64_ADDR16_LO_DS",/* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xfffc, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like R_PPC64_GOT16, but for instructions with a DS field. */ HOWTO (R_PPC64_GOT16_DS, /* type */ 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_signed, /* complain_on_overflow */ ppc64_elf_unhandled_reloc, /* special_function */ "R_PPC64_GOT16_DS", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xfffc, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like R_PPC64_GOT16_LO, but for instructions with a DS field. */ HOWTO (R_PPC64_GOT16_LO_DS, /* type */ 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc64_elf_unhandled_reloc, /* special_function */ "R_PPC64_GOT16_LO_DS", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xfffc, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like R_PPC64_PLT16_LO, but for instructions with a DS field. */ HOWTO (R_PPC64_PLT16_LO_DS, /* type */ 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc64_elf_unhandled_reloc, /* special_function */ "R_PPC64_PLT16_LO_DS", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xfffc, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like R_PPC64_SECTOFF, but for instructions with a DS field. */ HOWTO (R_PPC64_SECTOFF_DS, /* type */ 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_bitfield, /* complain_on_overflow */ ppc64_elf_sectoff_reloc, /* special_function */ "R_PPC64_SECTOFF_DS", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xfffc, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like R_PPC64_SECTOFF_LO, but for instructions with a DS field. */ HOWTO (R_PPC64_SECTOFF_LO_DS, /* type */ 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc64_elf_sectoff_reloc, /* special_function */ "R_PPC64_SECTOFF_LO_DS",/* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xfffc, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like R_PPC64_TOC16, but for instructions with a DS field. */ HOWTO (R_PPC64_TOC16_DS, /* type */ 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_signed, /* complain_on_overflow */ ppc64_elf_toc_reloc, /* special_function */ "R_PPC64_TOC16_DS", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xfffc, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like R_PPC64_TOC16_LO, but for instructions with a DS field. */ HOWTO (R_PPC64_TOC16_LO_DS, /* type */ 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc64_elf_toc_reloc, /* special_function */ "R_PPC64_TOC16_LO_DS", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xfffc, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like R_PPC64_PLTGOT16, but for instructions with a DS field. */ /* FIXME: R_PPC64_PLTGOT16_DS not implemented. */ HOWTO (R_PPC64_PLTGOT16_DS, /* type */ 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_signed, /* complain_on_overflow */ ppc64_elf_unhandled_reloc, /* special_function */ "R_PPC64_PLTGOT16_DS", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xfffc, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like R_PPC64_PLTGOT16_LO, but for instructions with a DS field. */ /* FIXME: R_PPC64_PLTGOT16_LO not implemented. */ HOWTO (R_PPC64_PLTGOT16_LO_DS,/* type */ 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc64_elf_unhandled_reloc, /* special_function */ "R_PPC64_PLTGOT16_LO_DS",/* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xfffc, /* dst_mask */ FALSE), /* pcrel_offset */ - /* Marker reloc for TLS. */ + /* Marker relocs for TLS. */ HOWTO (R_PPC64_TLS, 0, /* rightshift */ 2, /* size (0 = byte, 1 = short, 2 = long) */ 32, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ bfd_elf_generic_reloc, /* special_function */ "R_PPC64_TLS", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0, /* dst_mask */ FALSE), /* pcrel_offset */ + HOWTO (R_PPC64_TLSGD, + 0, /* rightshift */ + 2, /* size (0 = byte, 1 = short, 2 = long) */ + 32, /* bitsize */ + FALSE, /* pc_relative */ + 0, /* bitpos */ + complain_overflow_dont, /* complain_on_overflow */ + bfd_elf_generic_reloc, /* special_function */ + "R_PPC64_TLSGD", /* name */ + FALSE, /* partial_inplace */ + 0, /* src_mask */ + 0, /* dst_mask */ + FALSE), /* pcrel_offset */ + + HOWTO (R_PPC64_TLSLD, + 0, /* rightshift */ + 2, /* size (0 = byte, 1 = short, 2 = long) */ + 32, /* bitsize */ + FALSE, /* pc_relative */ + 0, /* bitpos */ + complain_overflow_dont, /* complain_on_overflow */ + bfd_elf_generic_reloc, /* special_function */ + "R_PPC64_TLSLD", /* name */ + FALSE, /* partial_inplace */ + 0, /* src_mask */ + 0, /* dst_mask */ + FALSE), /* pcrel_offset */ + /* Computes the load module index of the load module that contains the definition of its TLS sym. */ HOWTO (R_PPC64_DTPMOD64, 0, /* rightshift */ 4, /* size (0 = byte, 1 = short, 2 = long) */ 64, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc64_elf_unhandled_reloc, /* special_function */ "R_PPC64_DTPMOD64", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ ONES (64), /* dst_mask */ FALSE), /* pcrel_offset */ /* Computes a dtv-relative displacement, the difference between the value of sym+add and the base address of the thread-local storage block that contains the definition of sym, minus 0x8000. */ HOWTO (R_PPC64_DTPREL64, 0, /* rightshift */ 4, /* size (0 = byte, 1 = short, 2 = long) */ 64, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc64_elf_unhandled_reloc, /* special_function */ "R_PPC64_DTPREL64", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ ONES (64), /* dst_mask */ FALSE), /* pcrel_offset */ /* A 16 bit dtprel reloc. */ HOWTO (R_PPC64_DTPREL16, 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_signed, /* complain_on_overflow */ ppc64_elf_unhandled_reloc, /* special_function */ "R_PPC64_DTPREL16", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like DTPREL16, but no overflow. */ HOWTO (R_PPC64_DTPREL16_LO, 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc64_elf_unhandled_reloc, /* special_function */ "R_PPC64_DTPREL16_LO", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like DTPREL16_LO, but next higher group of 16 bits. */ HOWTO (R_PPC64_DTPREL16_HI, 16, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc64_elf_unhandled_reloc, /* special_function */ "R_PPC64_DTPREL16_HI", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like DTPREL16_HI, but adjust for low 16 bits. */ HOWTO (R_PPC64_DTPREL16_HA, 16, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc64_elf_unhandled_reloc, /* special_function */ "R_PPC64_DTPREL16_HA", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like DTPREL16_HI, but next higher group of 16 bits. */ HOWTO (R_PPC64_DTPREL16_HIGHER, 32, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc64_elf_unhandled_reloc, /* special_function */ "R_PPC64_DTPREL16_HIGHER", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like DTPREL16_HIGHER, but adjust for low 16 bits. */ HOWTO (R_PPC64_DTPREL16_HIGHERA, 32, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc64_elf_unhandled_reloc, /* special_function */ "R_PPC64_DTPREL16_HIGHERA", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like DTPREL16_HIGHER, but next higher group of 16 bits. */ HOWTO (R_PPC64_DTPREL16_HIGHEST, 48, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc64_elf_unhandled_reloc, /* special_function */ "R_PPC64_DTPREL16_HIGHEST", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like DTPREL16_HIGHEST, but adjust for low 16 bits. */ HOWTO (R_PPC64_DTPREL16_HIGHESTA, 48, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc64_elf_unhandled_reloc, /* special_function */ "R_PPC64_DTPREL16_HIGHESTA", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like DTPREL16, but for insns with a DS field. */ HOWTO (R_PPC64_DTPREL16_DS, 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_signed, /* complain_on_overflow */ ppc64_elf_unhandled_reloc, /* special_function */ "R_PPC64_DTPREL16_DS", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xfffc, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like DTPREL16_DS, but no overflow. */ HOWTO (R_PPC64_DTPREL16_LO_DS, 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc64_elf_unhandled_reloc, /* special_function */ "R_PPC64_DTPREL16_LO_DS", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xfffc, /* dst_mask */ FALSE), /* pcrel_offset */ /* Computes a tp-relative displacement, the difference between the value of sym+add and the value of the thread pointer (r13). */ HOWTO (R_PPC64_TPREL64, 0, /* rightshift */ 4, /* size (0 = byte, 1 = short, 2 = long) */ 64, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc64_elf_unhandled_reloc, /* special_function */ "R_PPC64_TPREL64", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ ONES (64), /* dst_mask */ FALSE), /* pcrel_offset */ /* A 16 bit tprel reloc. */ HOWTO (R_PPC64_TPREL16, 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_signed, /* complain_on_overflow */ ppc64_elf_unhandled_reloc, /* special_function */ "R_PPC64_TPREL16", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like TPREL16, but no overflow. */ HOWTO (R_PPC64_TPREL16_LO, 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc64_elf_unhandled_reloc, /* special_function */ "R_PPC64_TPREL16_LO", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like TPREL16_LO, but next higher group of 16 bits. */ HOWTO (R_PPC64_TPREL16_HI, 16, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc64_elf_unhandled_reloc, /* special_function */ "R_PPC64_TPREL16_HI", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like TPREL16_HI, but adjust for low 16 bits. */ HOWTO (R_PPC64_TPREL16_HA, 16, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc64_elf_unhandled_reloc, /* special_function */ "R_PPC64_TPREL16_HA", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like TPREL16_HI, but next higher group of 16 bits. */ HOWTO (R_PPC64_TPREL16_HIGHER, 32, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc64_elf_unhandled_reloc, /* special_function */ "R_PPC64_TPREL16_HIGHER", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like TPREL16_HIGHER, but adjust for low 16 bits. */ HOWTO (R_PPC64_TPREL16_HIGHERA, 32, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc64_elf_unhandled_reloc, /* special_function */ "R_PPC64_TPREL16_HIGHERA", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like TPREL16_HIGHER, but next higher group of 16 bits. */ HOWTO (R_PPC64_TPREL16_HIGHEST, 48, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc64_elf_unhandled_reloc, /* special_function */ "R_PPC64_TPREL16_HIGHEST", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like TPREL16_HIGHEST, but adjust for low 16 bits. */ HOWTO (R_PPC64_TPREL16_HIGHESTA, 48, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc64_elf_unhandled_reloc, /* special_function */ "R_PPC64_TPREL16_HIGHESTA", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like TPREL16, but for insns with a DS field. */ HOWTO (R_PPC64_TPREL16_DS, 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_signed, /* complain_on_overflow */ ppc64_elf_unhandled_reloc, /* special_function */ "R_PPC64_TPREL16_DS", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xfffc, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like TPREL16_DS, but no overflow. */ HOWTO (R_PPC64_TPREL16_LO_DS, 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc64_elf_unhandled_reloc, /* special_function */ "R_PPC64_TPREL16_LO_DS", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xfffc, /* dst_mask */ FALSE), /* pcrel_offset */ /* Allocates two contiguous entries in the GOT to hold a tls_index structure, with values (sym+add)@dtpmod and (sym+add)@dtprel, and computes the offset to the first entry relative to the TOC base (r2). */ HOWTO (R_PPC64_GOT_TLSGD16, 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_signed, /* complain_on_overflow */ ppc64_elf_unhandled_reloc, /* special_function */ "R_PPC64_GOT_TLSGD16", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like GOT_TLSGD16, but no overflow. */ HOWTO (R_PPC64_GOT_TLSGD16_LO, 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc64_elf_unhandled_reloc, /* special_function */ "R_PPC64_GOT_TLSGD16_LO", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like GOT_TLSGD16_LO, but next higher group of 16 bits. */ HOWTO (R_PPC64_GOT_TLSGD16_HI, 16, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc64_elf_unhandled_reloc, /* special_function */ "R_PPC64_GOT_TLSGD16_HI", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like GOT_TLSGD16_HI, but adjust for low 16 bits. */ HOWTO (R_PPC64_GOT_TLSGD16_HA, 16, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc64_elf_unhandled_reloc, /* special_function */ "R_PPC64_GOT_TLSGD16_HA", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Allocates two contiguous entries in the GOT to hold a tls_index structure, with values (sym+add)@dtpmod and zero, and computes the offset to the first entry relative to the TOC base (r2). */ HOWTO (R_PPC64_GOT_TLSLD16, 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_signed, /* complain_on_overflow */ ppc64_elf_unhandled_reloc, /* special_function */ "R_PPC64_GOT_TLSLD16", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like GOT_TLSLD16, but no overflow. */ HOWTO (R_PPC64_GOT_TLSLD16_LO, 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc64_elf_unhandled_reloc, /* special_function */ "R_PPC64_GOT_TLSLD16_LO", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like GOT_TLSLD16_LO, but next higher group of 16 bits. */ HOWTO (R_PPC64_GOT_TLSLD16_HI, 16, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc64_elf_unhandled_reloc, /* special_function */ "R_PPC64_GOT_TLSLD16_HI", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like GOT_TLSLD16_HI, but adjust for low 16 bits. */ HOWTO (R_PPC64_GOT_TLSLD16_HA, 16, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc64_elf_unhandled_reloc, /* special_function */ "R_PPC64_GOT_TLSLD16_HA", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Allocates an entry in the GOT with value (sym+add)@dtprel, and computes the offset to the entry relative to the TOC base (r2). */ HOWTO (R_PPC64_GOT_DTPREL16_DS, 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_signed, /* complain_on_overflow */ ppc64_elf_unhandled_reloc, /* special_function */ "R_PPC64_GOT_DTPREL16_DS", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xfffc, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like GOT_DTPREL16_DS, but no overflow. */ HOWTO (R_PPC64_GOT_DTPREL16_LO_DS, 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc64_elf_unhandled_reloc, /* special_function */ "R_PPC64_GOT_DTPREL16_LO_DS", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xfffc, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like GOT_DTPREL16_LO_DS, but next higher group of 16 bits. */ HOWTO (R_PPC64_GOT_DTPREL16_HI, 16, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc64_elf_unhandled_reloc, /* special_function */ "R_PPC64_GOT_DTPREL16_HI", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like GOT_DTPREL16_HI, but adjust for low 16 bits. */ HOWTO (R_PPC64_GOT_DTPREL16_HA, 16, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc64_elf_unhandled_reloc, /* special_function */ "R_PPC64_GOT_DTPREL16_HA", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Allocates an entry in the GOT with value (sym+add)@tprel, and computes the offset to the entry relative to the TOC base (r2). */ HOWTO (R_PPC64_GOT_TPREL16_DS, 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_signed, /* complain_on_overflow */ ppc64_elf_unhandled_reloc, /* special_function */ "R_PPC64_GOT_TPREL16_DS", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xfffc, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like GOT_TPREL16_DS, but no overflow. */ HOWTO (R_PPC64_GOT_TPREL16_LO_DS, 0, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc64_elf_unhandled_reloc, /* special_function */ "R_PPC64_GOT_TPREL16_LO_DS", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xfffc, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like GOT_TPREL16_LO_DS, but next higher group of 16 bits. */ HOWTO (R_PPC64_GOT_TPREL16_HI, 16, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc64_elf_unhandled_reloc, /* special_function */ "R_PPC64_GOT_TPREL16_HI", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* Like GOT_TPREL16_HI, but adjust for low 16 bits. */ HOWTO (R_PPC64_GOT_TPREL16_HA, 16, /* rightshift */ 1, /* size (0 = byte, 1 = short, 2 = long) */ 16, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ ppc64_elf_unhandled_reloc, /* special_function */ "R_PPC64_GOT_TPREL16_HA", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0xffff, /* dst_mask */ FALSE), /* pcrel_offset */ /* GNU extension to record C++ vtable hierarchy. */ HOWTO (R_PPC64_GNU_VTINHERIT, /* type */ 0, /* rightshift */ 0, /* size (0 = byte, 1 = short, 2 = long) */ 0, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ NULL, /* special_function */ "R_PPC64_GNU_VTINHERIT", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0, /* dst_mask */ FALSE), /* pcrel_offset */ /* GNU extension to record C++ vtable member usage. */ HOWTO (R_PPC64_GNU_VTENTRY, /* type */ 0, /* rightshift */ 0, /* size (0 = byte, 1 = short, 2 = long) */ 0, /* bitsize */ FALSE, /* pc_relative */ 0, /* bitpos */ complain_overflow_dont, /* complain_on_overflow */ NULL, /* special_function */ "R_PPC64_GNU_VTENTRY", /* name */ FALSE, /* partial_inplace */ 0, /* src_mask */ 0, /* dst_mask */ FALSE), /* pcrel_offset */ }; /* Initialize the ppc64_elf_howto_table, so that linear accesses can be done. */ static void ppc_howto_init (void) { unsigned int i, type; for (i = 0; i < sizeof (ppc64_elf_howto_raw) / sizeof (ppc64_elf_howto_raw[0]); i++) { type = ppc64_elf_howto_raw[i].type; BFD_ASSERT (type < (sizeof (ppc64_elf_howto_table) / sizeof (ppc64_elf_howto_table[0]))); ppc64_elf_howto_table[type] = &ppc64_elf_howto_raw[i]; } } static reloc_howto_type * ppc64_elf_reloc_type_lookup (bfd *abfd ATTRIBUTE_UNUSED, bfd_reloc_code_real_type code) { enum elf_ppc64_reloc_type r = R_PPC64_NONE; if (!ppc64_elf_howto_table[R_PPC64_ADDR32]) /* Initialize howto table if needed. */ ppc_howto_init (); switch (code) { default: return NULL; case BFD_RELOC_NONE: r = R_PPC64_NONE; break; case BFD_RELOC_32: r = R_PPC64_ADDR32; break; case BFD_RELOC_PPC_BA26: r = R_PPC64_ADDR24; break; case BFD_RELOC_16: r = R_PPC64_ADDR16; break; case BFD_RELOC_LO16: r = R_PPC64_ADDR16_LO; break; case BFD_RELOC_HI16: r = R_PPC64_ADDR16_HI; break; case BFD_RELOC_HI16_S: r = R_PPC64_ADDR16_HA; break; case BFD_RELOC_PPC_BA16: r = R_PPC64_ADDR14; break; case BFD_RELOC_PPC_BA16_BRTAKEN: r = R_PPC64_ADDR14_BRTAKEN; break; case BFD_RELOC_PPC_BA16_BRNTAKEN: r = R_PPC64_ADDR14_BRNTAKEN; break; case BFD_RELOC_PPC_B26: r = R_PPC64_REL24; break; case BFD_RELOC_PPC_B16: r = R_PPC64_REL14; break; case BFD_RELOC_PPC_B16_BRTAKEN: r = R_PPC64_REL14_BRTAKEN; break; case BFD_RELOC_PPC_B16_BRNTAKEN: r = R_PPC64_REL14_BRNTAKEN; break; case BFD_RELOC_16_GOTOFF: r = R_PPC64_GOT16; break; case BFD_RELOC_LO16_GOTOFF: r = R_PPC64_GOT16_LO; break; case BFD_RELOC_HI16_GOTOFF: r = R_PPC64_GOT16_HI; break; case BFD_RELOC_HI16_S_GOTOFF: r = R_PPC64_GOT16_HA; break; case BFD_RELOC_PPC_COPY: r = R_PPC64_COPY; break; case BFD_RELOC_PPC_GLOB_DAT: r = R_PPC64_GLOB_DAT; break; case BFD_RELOC_32_PCREL: r = R_PPC64_REL32; break; case BFD_RELOC_32_PLTOFF: r = R_PPC64_PLT32; break; case BFD_RELOC_32_PLT_PCREL: r = R_PPC64_PLTREL32; break; case BFD_RELOC_LO16_PLTOFF: r = R_PPC64_PLT16_LO; break; case BFD_RELOC_HI16_PLTOFF: r = R_PPC64_PLT16_HI; break; case BFD_RELOC_HI16_S_PLTOFF: r = R_PPC64_PLT16_HA; break; case BFD_RELOC_16_BASEREL: r = R_PPC64_SECTOFF; break; case BFD_RELOC_LO16_BASEREL: r = R_PPC64_SECTOFF_LO; break; case BFD_RELOC_HI16_BASEREL: r = R_PPC64_SECTOFF_HI; break; case BFD_RELOC_HI16_S_BASEREL: r = R_PPC64_SECTOFF_HA; break; case BFD_RELOC_CTOR: r = R_PPC64_ADDR64; break; case BFD_RELOC_64: r = R_PPC64_ADDR64; break; case BFD_RELOC_PPC64_HIGHER: r = R_PPC64_ADDR16_HIGHER; break; case BFD_RELOC_PPC64_HIGHER_S: r = R_PPC64_ADDR16_HIGHERA; break; case BFD_RELOC_PPC64_HIGHEST: r = R_PPC64_ADDR16_HIGHEST; break; case BFD_RELOC_PPC64_HIGHEST_S: r = R_PPC64_ADDR16_HIGHESTA; break; case BFD_RELOC_64_PCREL: r = R_PPC64_REL64; break; case BFD_RELOC_64_PLTOFF: r = R_PPC64_PLT64; break; case BFD_RELOC_64_PLT_PCREL: r = R_PPC64_PLTREL64; break; case BFD_RELOC_PPC_TOC16: r = R_PPC64_TOC16; break; case BFD_RELOC_PPC64_TOC16_LO: r = R_PPC64_TOC16_LO; break; case BFD_RELOC_PPC64_TOC16_HI: r = R_PPC64_TOC16_HI; break; case BFD_RELOC_PPC64_TOC16_HA: r = R_PPC64_TOC16_HA; break; case BFD_RELOC_PPC64_TOC: r = R_PPC64_TOC; break; case BFD_RELOC_PPC64_PLTGOT16: r = R_PPC64_PLTGOT16; break; case BFD_RELOC_PPC64_PLTGOT16_LO: r = R_PPC64_PLTGOT16_LO; break; case BFD_RELOC_PPC64_PLTGOT16_HI: r = R_PPC64_PLTGOT16_HI; break; case BFD_RELOC_PPC64_PLTGOT16_HA: r = R_PPC64_PLTGOT16_HA; break; case BFD_RELOC_PPC64_ADDR16_DS: r = R_PPC64_ADDR16_DS; break; case BFD_RELOC_PPC64_ADDR16_LO_DS: r = R_PPC64_ADDR16_LO_DS; break; case BFD_RELOC_PPC64_GOT16_DS: r = R_PPC64_GOT16_DS; break; case BFD_RELOC_PPC64_GOT16_LO_DS: r = R_PPC64_GOT16_LO_DS; break; case BFD_RELOC_PPC64_PLT16_LO_DS: r = R_PPC64_PLT16_LO_DS; break; case BFD_RELOC_PPC64_SECTOFF_DS: r = R_PPC64_SECTOFF_DS; break; case BFD_RELOC_PPC64_SECTOFF_LO_DS: r = R_PPC64_SECTOFF_LO_DS; break; case BFD_RELOC_PPC64_TOC16_DS: r = R_PPC64_TOC16_DS; break; case BFD_RELOC_PPC64_TOC16_LO_DS: r = R_PPC64_TOC16_LO_DS; break; case BFD_RELOC_PPC64_PLTGOT16_DS: r = R_PPC64_PLTGOT16_DS; break; case BFD_RELOC_PPC64_PLTGOT16_LO_DS: r = R_PPC64_PLTGOT16_LO_DS; break; case BFD_RELOC_PPC_TLS: r = R_PPC64_TLS; break; + case BFD_RELOC_PPC_TLSGD: r = R_PPC64_TLSGD; + break; + case BFD_RELOC_PPC_TLSLD: r = R_PPC64_TLSLD; + break; case BFD_RELOC_PPC_DTPMOD: r = R_PPC64_DTPMOD64; break; case BFD_RELOC_PPC_TPREL16: r = R_PPC64_TPREL16; break; case BFD_RELOC_PPC_TPREL16_LO: r = R_PPC64_TPREL16_LO; break; case BFD_RELOC_PPC_TPREL16_HI: r = R_PPC64_TPREL16_HI; break; case BFD_RELOC_PPC_TPREL16_HA: r = R_PPC64_TPREL16_HA; break; case BFD_RELOC_PPC_TPREL: r = R_PPC64_TPREL64; break; case BFD_RELOC_PPC_DTPREL16: r = R_PPC64_DTPREL16; break; case BFD_RELOC_PPC_DTPREL16_LO: r = R_PPC64_DTPREL16_LO; break; case BFD_RELOC_PPC_DTPREL16_HI: r = R_PPC64_DTPREL16_HI; break; case BFD_RELOC_PPC_DTPREL16_HA: r = R_PPC64_DTPREL16_HA; break; case BFD_RELOC_PPC_DTPREL: r = R_PPC64_DTPREL64; break; case BFD_RELOC_PPC_GOT_TLSGD16: r = R_PPC64_GOT_TLSGD16; break; case BFD_RELOC_PPC_GOT_TLSGD16_LO: r = R_PPC64_GOT_TLSGD16_LO; break; case BFD_RELOC_PPC_GOT_TLSGD16_HI: r = R_PPC64_GOT_TLSGD16_HI; break; case BFD_RELOC_PPC_GOT_TLSGD16_HA: r = R_PPC64_GOT_TLSGD16_HA; break; case BFD_RELOC_PPC_GOT_TLSLD16: r = R_PPC64_GOT_TLSLD16; break; case BFD_RELOC_PPC_GOT_TLSLD16_LO: r = R_PPC64_GOT_TLSLD16_LO; break; case BFD_RELOC_PPC_GOT_TLSLD16_HI: r = R_PPC64_GOT_TLSLD16_HI; break; case BFD_RELOC_PPC_GOT_TLSLD16_HA: r = R_PPC64_GOT_TLSLD16_HA; break; case BFD_RELOC_PPC_GOT_TPREL16: r = R_PPC64_GOT_TPREL16_DS; break; case BFD_RELOC_PPC_GOT_TPREL16_LO: r = R_PPC64_GOT_TPREL16_LO_DS; break; case BFD_RELOC_PPC_GOT_TPREL16_HI: r = R_PPC64_GOT_TPREL16_HI; break; case BFD_RELOC_PPC_GOT_TPREL16_HA: r = R_PPC64_GOT_TPREL16_HA; break; case BFD_RELOC_PPC_GOT_DTPREL16: r = R_PPC64_GOT_DTPREL16_DS; break; case BFD_RELOC_PPC_GOT_DTPREL16_LO: r = R_PPC64_GOT_DTPREL16_LO_DS; break; case BFD_RELOC_PPC_GOT_DTPREL16_HI: r = R_PPC64_GOT_DTPREL16_HI; break; case BFD_RELOC_PPC_GOT_DTPREL16_HA: r = R_PPC64_GOT_DTPREL16_HA; break; case BFD_RELOC_PPC64_TPREL16_DS: r = R_PPC64_TPREL16_DS; break; case BFD_RELOC_PPC64_TPREL16_LO_DS: r = R_PPC64_TPREL16_LO_DS; break; case BFD_RELOC_PPC64_TPREL16_HIGHER: r = R_PPC64_TPREL16_HIGHER; break; case BFD_RELOC_PPC64_TPREL16_HIGHERA: r = R_PPC64_TPREL16_HIGHERA; break; case BFD_RELOC_PPC64_TPREL16_HIGHEST: r = R_PPC64_TPREL16_HIGHEST; break; case BFD_RELOC_PPC64_TPREL16_HIGHESTA: r = R_PPC64_TPREL16_HIGHESTA; break; case BFD_RELOC_PPC64_DTPREL16_DS: r = R_PPC64_DTPREL16_DS; break; case BFD_RELOC_PPC64_DTPREL16_LO_DS: r = R_PPC64_DTPREL16_LO_DS; break; case BFD_RELOC_PPC64_DTPREL16_HIGHER: r = R_PPC64_DTPREL16_HIGHER; break; case BFD_RELOC_PPC64_DTPREL16_HIGHERA: r = R_PPC64_DTPREL16_HIGHERA; break; case BFD_RELOC_PPC64_DTPREL16_HIGHEST: r = R_PPC64_DTPREL16_HIGHEST; break; case BFD_RELOC_PPC64_DTPREL16_HIGHESTA: r = R_PPC64_DTPREL16_HIGHESTA; break; case BFD_RELOC_VTABLE_INHERIT: r = R_PPC64_GNU_VTINHERIT; break; case BFD_RELOC_VTABLE_ENTRY: r = R_PPC64_GNU_VTENTRY; break; } return ppc64_elf_howto_table[r]; }; static reloc_howto_type * ppc64_elf_reloc_name_lookup (bfd *abfd ATTRIBUTE_UNUSED, const char *r_name) { unsigned int i; for (i = 0; i < sizeof (ppc64_elf_howto_raw) / sizeof (ppc64_elf_howto_raw[0]); i++) if (ppc64_elf_howto_raw[i].name != NULL && strcasecmp (ppc64_elf_howto_raw[i].name, r_name) == 0) return &ppc64_elf_howto_raw[i]; return NULL; } /* Set the howto pointer for a PowerPC ELF reloc. */ static void ppc64_elf_info_to_howto (bfd *abfd ATTRIBUTE_UNUSED, arelent *cache_ptr, Elf_Internal_Rela *dst) { unsigned int type; /* Initialize howto table if needed. */ if (!ppc64_elf_howto_table[R_PPC64_ADDR32]) ppc_howto_init (); type = ELF64_R_TYPE (dst->r_info); if (type >= (sizeof (ppc64_elf_howto_table) / sizeof (ppc64_elf_howto_table[0]))) { (*_bfd_error_handler) (_("%B: invalid relocation type %d"), abfd, (int) type); type = R_PPC64_NONE; } cache_ptr->howto = ppc64_elf_howto_table[type]; } /* Handle the R_PPC64_ADDR16_HA and similar relocs. */ static bfd_reloc_status_type ppc64_elf_ha_reloc (bfd *abfd, arelent *reloc_entry, asymbol *symbol, void *data, asection *input_section, bfd *output_bfd, char **error_message) { /* If this is a relocatable link (output_bfd test tells us), just call the generic function. Any adjustment will be done at final link time. */ if (output_bfd != NULL) return bfd_elf_generic_reloc (abfd, reloc_entry, symbol, data, input_section, output_bfd, error_message); /* Adjust the addend for sign extension of the low 16 bits. We won't actually be using the low 16 bits, so trashing them doesn't matter. */ reloc_entry->addend += 0x8000; return bfd_reloc_continue; } static bfd_reloc_status_type ppc64_elf_branch_reloc (bfd *abfd, arelent *reloc_entry, asymbol *symbol, void *data, asection *input_section, bfd *output_bfd, char **error_message) { if (output_bfd != NULL) return bfd_elf_generic_reloc (abfd, reloc_entry, symbol, data, input_section, output_bfd, error_message); if (strcmp (symbol->section->name, ".opd") == 0 && (symbol->section->owner->flags & DYNAMIC) == 0) { bfd_vma dest = opd_entry_value (symbol->section, symbol->value + reloc_entry->addend, NULL, NULL); if (dest != (bfd_vma) -1) reloc_entry->addend = dest - (symbol->value + symbol->section->output_section->vma + symbol->section->output_offset); } return bfd_reloc_continue; } static bfd_reloc_status_type ppc64_elf_brtaken_reloc (bfd *abfd, arelent *reloc_entry, asymbol *symbol, void *data, asection *input_section, bfd *output_bfd, char **error_message) { long insn; enum elf_ppc64_reloc_type r_type; bfd_size_type octets; /* Disabled until we sort out how ld should choose 'y' vs 'at'. */ bfd_boolean is_power4 = FALSE; /* If this is a relocatable link (output_bfd test tells us), just call the generic function. Any adjustment will be done at final link time. */ if (output_bfd != NULL) return bfd_elf_generic_reloc (abfd, reloc_entry, symbol, data, input_section, output_bfd, error_message); octets = reloc_entry->address * bfd_octets_per_byte (abfd); insn = bfd_get_32 (abfd, (bfd_byte *) data + octets); insn &= ~(0x01 << 21); r_type = reloc_entry->howto->type; if (r_type == R_PPC64_ADDR14_BRTAKEN || r_type == R_PPC64_REL14_BRTAKEN) insn |= 0x01 << 21; /* 'y' or 't' bit, lowest bit of BO field. */ if (is_power4) { /* Set 'a' bit. This is 0b00010 in BO field for branch on CR(BI) insns (BO == 001at or 011at), and 0b01000 for branch on CTR insns (BO == 1a00t or 1a01t). */ if ((insn & (0x14 << 21)) == (0x04 << 21)) insn |= 0x02 << 21; else if ((insn & (0x14 << 21)) == (0x10 << 21)) insn |= 0x08 << 21; else goto out; } else { bfd_vma target = 0; bfd_vma from; if (!bfd_is_com_section (symbol->section)) target = symbol->value; target += symbol->section->output_section->vma; target += symbol->section->output_offset; target += reloc_entry->addend; from = (reloc_entry->address + input_section->output_offset + input_section->output_section->vma); /* Invert 'y' bit if not the default. */ if ((bfd_signed_vma) (target - from) < 0) insn ^= 0x01 << 21; } bfd_put_32 (abfd, insn, (bfd_byte *) data + octets); out: return ppc64_elf_branch_reloc (abfd, reloc_entry, symbol, data, input_section, output_bfd, error_message); } static bfd_reloc_status_type ppc64_elf_sectoff_reloc (bfd *abfd, arelent *reloc_entry, asymbol *symbol, void *data, asection *input_section, bfd *output_bfd, char **error_message) { /* If this is a relocatable link (output_bfd test tells us), just call the generic function. Any adjustment will be done at final link time. */ if (output_bfd != NULL) return bfd_elf_generic_reloc (abfd, reloc_entry, symbol, data, input_section, output_bfd, error_message); /* Subtract the symbol section base address. */ reloc_entry->addend -= symbol->section->output_section->vma; return bfd_reloc_continue; } static bfd_reloc_status_type ppc64_elf_sectoff_ha_reloc (bfd *abfd, arelent *reloc_entry, asymbol *symbol, void *data, asection *input_section, bfd *output_bfd, char **error_message) { /* If this is a relocatable link (output_bfd test tells us), just call the generic function. Any adjustment will be done at final link time. */ if (output_bfd != NULL) return bfd_elf_generic_reloc (abfd, reloc_entry, symbol, data, input_section, output_bfd, error_message); /* Subtract the symbol section base address. */ reloc_entry->addend -= symbol->section->output_section->vma; /* Adjust the addend for sign extension of the low 16 bits. */ reloc_entry->addend += 0x8000; return bfd_reloc_continue; } static bfd_reloc_status_type ppc64_elf_toc_reloc (bfd *abfd, arelent *reloc_entry, asymbol *symbol, void *data, asection *input_section, bfd *output_bfd, char **error_message) { bfd_vma TOCstart; /* If this is a relocatable link (output_bfd test tells us), just call the generic function. Any adjustment will be done at final link time. */ if (output_bfd != NULL) return bfd_elf_generic_reloc (abfd, reloc_entry, symbol, data, input_section, output_bfd, error_message); TOCstart = _bfd_get_gp_value (input_section->output_section->owner); if (TOCstart == 0) TOCstart = ppc64_elf_toc (input_section->output_section->owner); /* Subtract the TOC base address. */ reloc_entry->addend -= TOCstart + TOC_BASE_OFF; return bfd_reloc_continue; } static bfd_reloc_status_type ppc64_elf_toc_ha_reloc (bfd *abfd, arelent *reloc_entry, asymbol *symbol, void *data, asection *input_section, bfd *output_bfd, char **error_message) { bfd_vma TOCstart; /* If this is a relocatable link (output_bfd test tells us), just call the generic function. Any adjustment will be done at final link time. */ if (output_bfd != NULL) return bfd_elf_generic_reloc (abfd, reloc_entry, symbol, data, input_section, output_bfd, error_message); TOCstart = _bfd_get_gp_value (input_section->output_section->owner); if (TOCstart == 0) TOCstart = ppc64_elf_toc (input_section->output_section->owner); /* Subtract the TOC base address. */ reloc_entry->addend -= TOCstart + TOC_BASE_OFF; /* Adjust the addend for sign extension of the low 16 bits. */ reloc_entry->addend += 0x8000; return bfd_reloc_continue; } static bfd_reloc_status_type ppc64_elf_toc64_reloc (bfd *abfd, arelent *reloc_entry, asymbol *symbol, void *data, asection *input_section, bfd *output_bfd, char **error_message) { bfd_vma TOCstart; bfd_size_type octets; /* If this is a relocatable link (output_bfd test tells us), just call the generic function. Any adjustment will be done at final link time. */ if (output_bfd != NULL) return bfd_elf_generic_reloc (abfd, reloc_entry, symbol, data, input_section, output_bfd, error_message); TOCstart = _bfd_get_gp_value (input_section->output_section->owner); if (TOCstart == 0) TOCstart = ppc64_elf_toc (input_section->output_section->owner); octets = reloc_entry->address * bfd_octets_per_byte (abfd); bfd_put_64 (abfd, TOCstart + TOC_BASE_OFF, (bfd_byte *) data + octets); return bfd_reloc_ok; } static bfd_reloc_status_type ppc64_elf_unhandled_reloc (bfd *abfd, arelent *reloc_entry, asymbol *symbol, void *data, asection *input_section, bfd *output_bfd, char **error_message) { /* If this is a relocatable link (output_bfd test tells us), just call the generic function. Any adjustment will be done at final link time. */ if (output_bfd != NULL) return bfd_elf_generic_reloc (abfd, reloc_entry, symbol, data, input_section, output_bfd, error_message); if (error_message != NULL) { static char buf[60]; sprintf (buf, "generic linker can't handle %s", reloc_entry->howto->name); *error_message = buf; } return bfd_reloc_dangerous; } struct ppc64_elf_obj_tdata { struct elf_obj_tdata elf; /* Shortcuts to dynamic linker sections. */ asection *got; asection *relgot; /* Used during garbage collection. We attach global symbols defined on removed .opd entries to this section so that the sym is removed. */ asection *deleted_section; /* TLS local dynamic got entry handling. Suppose for multiple GOT sections means we potentially need one of these for each input bfd. */ union { bfd_signed_vma refcount; bfd_vma offset; } tlsld_got; /* A copy of relocs before they are modified for --emit-relocs. */ Elf_Internal_Rela *opd_relocs; }; #define ppc64_elf_tdata(bfd) \ ((struct ppc64_elf_obj_tdata *) (bfd)->tdata.any) #define ppc64_tlsld_got(bfd) \ (&ppc64_elf_tdata (bfd)->tlsld_got) /* Override the generic function because we store some extras. */ static bfd_boolean ppc64_elf_mkobject (bfd *abfd) { if (abfd->tdata.any == NULL) { bfd_size_type amt = sizeof (struct ppc64_elf_obj_tdata); abfd->tdata.any = bfd_zalloc (abfd, amt); if (abfd->tdata.any == NULL) return FALSE; } return bfd_elf_mkobject (abfd); } /* Return 1 if target is one of ours. */ static bfd_boolean is_ppc64_elf_target (const struct bfd_target *targ) { extern const bfd_target bfd_elf64_powerpc_vec; extern const bfd_target bfd_elf64_powerpcle_vec; return targ == &bfd_elf64_powerpc_vec || targ == &bfd_elf64_powerpcle_vec; } /* Fix bad default arch selected for a 64 bit input bfd when the default is 32 bit. */ static bfd_boolean ppc64_elf_object_p (bfd *abfd) { if (abfd->arch_info->the_default && abfd->arch_info->bits_per_word == 32) { Elf_Internal_Ehdr *i_ehdr = elf_elfheader (abfd); if (i_ehdr->e_ident[EI_CLASS] == ELFCLASS64) { /* Relies on arch after 32 bit default being 64 bit default. */ abfd->arch_info = abfd->arch_info->next; BFD_ASSERT (abfd->arch_info->bits_per_word == 64); } } return TRUE; } /* Support for core dump NOTE sections. */ static bfd_boolean ppc64_elf_grok_prstatus (bfd *abfd, Elf_Internal_Note *note) { size_t offset, size; if (note->descsz != 504) return FALSE; /* pr_cursig */ elf_tdata (abfd)->core_signal = bfd_get_16 (abfd, note->descdata + 12); /* pr_pid */ elf_tdata (abfd)->core_pid = bfd_get_32 (abfd, note->descdata + 32); /* pr_reg */ offset = 112; size = 384; /* Make a ".reg/999" section. */ return _bfd_elfcore_make_pseudosection (abfd, ".reg", size, note->descpos + offset); } static bfd_boolean ppc64_elf_grok_psinfo (bfd *abfd, Elf_Internal_Note *note) { if (note->descsz != 136) return FALSE; elf_tdata (abfd)->core_program = _bfd_elfcore_strndup (abfd, note->descdata + 40, 16); elf_tdata (abfd)->core_command = _bfd_elfcore_strndup (abfd, note->descdata + 56, 80); return TRUE; } static char * ppc64_elf_write_core_note (bfd *abfd, char *buf, int *bufsiz, int note_type, ...) { switch (note_type) { default: return NULL; case NT_PRPSINFO: { char data[136]; va_list ap; va_start (ap, note_type); memset (data, 0, 40); strncpy (data + 40, va_arg (ap, const char *), 16); strncpy (data + 56, va_arg (ap, const char *), 80); va_end (ap); return elfcore_write_note (abfd, buf, bufsiz, "CORE", note_type, data, sizeof (data)); } case NT_PRSTATUS: { char data[504]; va_list ap; long pid; int cursig; const void *greg; va_start (ap, note_type); memset (data, 0, 112); pid = va_arg (ap, long); bfd_put_32 (abfd, pid, data + 32); cursig = va_arg (ap, int); bfd_put_16 (abfd, cursig, data + 12); greg = va_arg (ap, const void *); memcpy (data + 112, greg, 384); memset (data + 496, 0, 8); va_end (ap); return elfcore_write_note (abfd, buf, bufsiz, "CORE", note_type, data, sizeof (data)); } } } /* Merge backend specific data from an object file to the output object file when linking. */ static bfd_boolean ppc64_elf_merge_private_bfd_data (bfd *ibfd, bfd *obfd) { /* Check if we have the same endianess. */ if (ibfd->xvec->byteorder != obfd->xvec->byteorder && ibfd->xvec->byteorder != BFD_ENDIAN_UNKNOWN && obfd->xvec->byteorder != BFD_ENDIAN_UNKNOWN) { const char *msg; if (bfd_big_endian (ibfd)) msg = _("%B: compiled for a big endian system " "and target is little endian"); else msg = _("%B: compiled for a little endian system " "and target is big endian"); (*_bfd_error_handler) (msg, ibfd); bfd_set_error (bfd_error_wrong_format); return FALSE; } return TRUE; } /* Add extra PPC sections. */ static const struct bfd_elf_special_section ppc64_elf_special_sections[]= { { STRING_COMMA_LEN (".plt"), 0, SHT_NOBITS, 0 }, { STRING_COMMA_LEN (".sbss"), -2, SHT_NOBITS, SHF_ALLOC + SHF_WRITE }, { STRING_COMMA_LEN (".sdata"), -2, SHT_PROGBITS, SHF_ALLOC + SHF_WRITE }, { STRING_COMMA_LEN (".toc"), 0, SHT_PROGBITS, SHF_ALLOC + SHF_WRITE }, { STRING_COMMA_LEN (".toc1"), 0, SHT_PROGBITS, SHF_ALLOC + SHF_WRITE }, { STRING_COMMA_LEN (".tocbss"), 0, SHT_NOBITS, SHF_ALLOC + SHF_WRITE }, { NULL, 0, 0, 0, 0 } }; enum _ppc64_sec_type { sec_normal = 0, sec_opd = 1, sec_toc = 2 }; struct _ppc64_elf_section_data { struct bfd_elf_section_data elf; /* An array with one entry for each opd function descriptor. */ union { /* Points to the function code section for local opd entries. */ asection **opd_func_sec; /* After editing .opd, adjust references to opd local syms. */ long *opd_adjust; - /* An array for toc sections, indexed by offset/8. - Specifies the relocation symbol index used at a given toc offset. */ - unsigned *t_symndx; + /* An array for toc sections, indexed by offset/8. */ + struct _toc_sec_data + { + /* Specifies the relocation symbol index used at a given toc offset. */ + unsigned *symndx; + + /* And the relocation addend. */ + bfd_vma *add; + } toc; } u; enum _ppc64_sec_type sec_type:2; /* Flag set when small branches are detected. Used to select suitable defaults for the stub group size. */ unsigned int has_14bit_branch:1; }; #define ppc64_elf_section_data(sec) \ ((struct _ppc64_elf_section_data *) elf_section_data (sec)) static bfd_boolean ppc64_elf_new_section_hook (bfd *abfd, asection *sec) { if (!sec->used_by_bfd) { struct _ppc64_elf_section_data *sdata; bfd_size_type amt = sizeof (*sdata); sdata = bfd_zalloc (abfd, amt); if (sdata == NULL) return FALSE; sec->used_by_bfd = sdata; } return _bfd_elf_new_section_hook (abfd, sec); } static void * get_opd_info (asection * sec) { if (sec != NULL && ppc64_elf_section_data (sec) != NULL && ppc64_elf_section_data (sec)->sec_type == sec_opd) return ppc64_elf_section_data (sec)->u.opd_adjust; return NULL; } /* Parameters for the qsort hook. */ static asection *synthetic_opd; static bfd_boolean synthetic_relocatable; /* qsort comparison function for ppc64_elf_get_synthetic_symtab. */ static int compare_symbols (const void *ap, const void *bp) { const asymbol *a = * (const asymbol **) ap; const asymbol *b = * (const asymbol **) bp; /* Section symbols first. */ if ((a->flags & BSF_SECTION_SYM) && !(b->flags & BSF_SECTION_SYM)) return -1; if (!(a->flags & BSF_SECTION_SYM) && (b->flags & BSF_SECTION_SYM)) return 1; /* then .opd symbols. */ if (a->section == synthetic_opd && b->section != synthetic_opd) return -1; if (a->section != synthetic_opd && b->section == synthetic_opd) return 1; /* then other code symbols. */ if ((a->section->flags & (SEC_CODE | SEC_ALLOC | SEC_THREAD_LOCAL)) == (SEC_CODE | SEC_ALLOC) && (b->section->flags & (SEC_CODE | SEC_ALLOC | SEC_THREAD_LOCAL)) != (SEC_CODE | SEC_ALLOC)) return -1; if ((a->section->flags & (SEC_CODE | SEC_ALLOC | SEC_THREAD_LOCAL)) != (SEC_CODE | SEC_ALLOC) && (b->section->flags & (SEC_CODE | SEC_ALLOC | SEC_THREAD_LOCAL)) == (SEC_CODE | SEC_ALLOC)) return 1; if (synthetic_relocatable) { if (a->section->id < b->section->id) return -1; if (a->section->id > b->section->id) return 1; } if (a->value + a->section->vma < b->value + b->section->vma) return -1; if (a->value + a->section->vma > b->value + b->section->vma) return 1; /* For syms with the same value, prefer strong dynamic global function syms over other syms. */ if ((a->flags & BSF_GLOBAL) != 0 && (b->flags & BSF_GLOBAL) == 0) return -1; if ((a->flags & BSF_GLOBAL) == 0 && (b->flags & BSF_GLOBAL) != 0) return 1; if ((a->flags & BSF_FUNCTION) != 0 && (b->flags & BSF_FUNCTION) == 0) return -1; if ((a->flags & BSF_FUNCTION) == 0 && (b->flags & BSF_FUNCTION) != 0) return 1; if ((a->flags & BSF_WEAK) == 0 && (b->flags & BSF_WEAK) != 0) return -1; if ((a->flags & BSF_WEAK) != 0 && (b->flags & BSF_WEAK) == 0) return 1; if ((a->flags & BSF_DYNAMIC) != 0 && (b->flags & BSF_DYNAMIC) == 0) return -1; if ((a->flags & BSF_DYNAMIC) == 0 && (b->flags & BSF_DYNAMIC) != 0) return 1; return 0; } /* Search SYMS for a symbol of the given VALUE. */ static asymbol * sym_exists_at (asymbol **syms, long lo, long hi, int id, bfd_vma value) { long mid; if (id == -1) { while (lo < hi) { mid = (lo + hi) >> 1; if (syms[mid]->value + syms[mid]->section->vma < value) lo = mid + 1; else if (syms[mid]->value + syms[mid]->section->vma > value) hi = mid; else return syms[mid]; } } else { while (lo < hi) { mid = (lo + hi) >> 1; if (syms[mid]->section->id < id) lo = mid + 1; else if (syms[mid]->section->id > id) hi = mid; else if (syms[mid]->value < value) lo = mid + 1; else if (syms[mid]->value > value) hi = mid; else return syms[mid]; } } return NULL; } /* Create synthetic symbols, effectively restoring "dot-symbol" function entry syms. */ static long ppc64_elf_get_synthetic_symtab (bfd *abfd, long static_count, asymbol **static_syms, long dyn_count, asymbol **dyn_syms, asymbol **ret) { asymbol *s; long i; long count; char *names; long symcount, codesecsym, codesecsymend, secsymend, opdsymend; asection *opd; bfd_boolean relocatable = (abfd->flags & (EXEC_P | DYNAMIC)) == 0; asymbol **syms; *ret = NULL; opd = bfd_get_section_by_name (abfd, ".opd"); if (opd == NULL) return 0; symcount = static_count; if (!relocatable) symcount += dyn_count; if (symcount == 0) return 0; syms = bfd_malloc ((symcount + 1) * sizeof (*syms)); if (syms == NULL) return -1; if (!relocatable && static_count != 0 && dyn_count != 0) { /* Use both symbol tables. */ memcpy (syms, static_syms, static_count * sizeof (*syms)); memcpy (syms + static_count, dyn_syms, (dyn_count + 1) * sizeof (*syms)); } else if (!relocatable && static_count == 0) memcpy (syms, dyn_syms, (symcount + 1) * sizeof (*syms)); else memcpy (syms, static_syms, (symcount + 1) * sizeof (*syms)); synthetic_opd = opd; synthetic_relocatable = relocatable; qsort (syms, symcount, sizeof (*syms), compare_symbols); if (!relocatable && symcount > 1) { long j; /* Trim duplicate syms, since we may have merged the normal and dynamic symbols. Actually, we only care about syms that have different values, so trim any with the same value. */ for (i = 1, j = 1; i < symcount; ++i) if (syms[i - 1]->value + syms[i - 1]->section->vma != syms[i]->value + syms[i]->section->vma) syms[j++] = syms[i]; symcount = j; } i = 0; if (syms[i]->section == opd) ++i; codesecsym = i; for (; i < symcount; ++i) if (((syms[i]->section->flags & (SEC_CODE | SEC_ALLOC | SEC_THREAD_LOCAL)) != (SEC_CODE | SEC_ALLOC)) || (syms[i]->flags & BSF_SECTION_SYM) == 0) break; codesecsymend = i; for (; i < symcount; ++i) if ((syms[i]->flags & BSF_SECTION_SYM) == 0) break; secsymend = i; for (; i < symcount; ++i) if (syms[i]->section != opd) break; opdsymend = i; for (; i < symcount; ++i) if ((syms[i]->section->flags & (SEC_CODE | SEC_ALLOC | SEC_THREAD_LOCAL)) != (SEC_CODE | SEC_ALLOC)) break; symcount = i; count = 0; if (opdsymend == secsymend) goto done; if (relocatable) { bfd_boolean (*slurp_relocs) (bfd *, asection *, asymbol **, bfd_boolean); arelent *r; size_t size; long relcount; slurp_relocs = get_elf_backend_data (abfd)->s->slurp_reloc_table; relcount = (opd->flags & SEC_RELOC) ? opd->reloc_count : 0; if (relcount == 0) goto done; if (!(*slurp_relocs) (abfd, opd, static_syms, FALSE)) { count = -1; goto done; } size = 0; for (i = secsymend, r = opd->relocation; i < opdsymend; ++i) { asymbol *sym; while (r < opd->relocation + relcount && r->address < syms[i]->value + opd->vma) ++r; if (r == opd->relocation + relcount) break; if (r->address != syms[i]->value + opd->vma) continue; if (r->howto->type != R_PPC64_ADDR64) continue; sym = *r->sym_ptr_ptr; if (!sym_exists_at (syms, opdsymend, symcount, sym->section->id, sym->value + r->addend)) { ++count; size += sizeof (asymbol); size += strlen (syms[i]->name) + 2; } } s = *ret = bfd_malloc (size); if (s == NULL) { count = -1; goto done; } names = (char *) (s + count); for (i = secsymend, r = opd->relocation; i < opdsymend; ++i) { asymbol *sym; while (r < opd->relocation + relcount && r->address < syms[i]->value + opd->vma) ++r; if (r == opd->relocation + relcount) break; if (r->address != syms[i]->value + opd->vma) continue; if (r->howto->type != R_PPC64_ADDR64) continue; sym = *r->sym_ptr_ptr; if (!sym_exists_at (syms, opdsymend, symcount, sym->section->id, sym->value + r->addend)) { size_t len; *s = *syms[i]; s->section = sym->section; s->value = sym->value + r->addend; s->name = names; *names++ = '.'; len = strlen (syms[i]->name); memcpy (names, syms[i]->name, len + 1); names += len + 1; s++; } } } else { bfd_byte *contents; size_t size; if (!bfd_malloc_and_get_section (abfd, opd, &contents)) { if (contents) { free_contents_and_exit: free (contents); } count = -1; goto done; } size = 0; for (i = secsymend; i < opdsymend; ++i) { bfd_vma ent; ent = bfd_get_64 (abfd, contents + syms[i]->value); if (!sym_exists_at (syms, opdsymend, symcount, -1, ent)) { ++count; size += sizeof (asymbol); size += strlen (syms[i]->name) + 2; } } s = *ret = bfd_malloc (size); if (s == NULL) goto free_contents_and_exit; names = (char *) (s + count); for (i = secsymend; i < opdsymend; ++i) { bfd_vma ent; ent = bfd_get_64 (abfd, contents + syms[i]->value); if (!sym_exists_at (syms, opdsymend, symcount, -1, ent)) { long lo, hi; size_t len; asection *sec = abfd->sections; *s = *syms[i]; lo = codesecsym; hi = codesecsymend; while (lo < hi) { long mid = (lo + hi) >> 1; if (syms[mid]->section->vma < ent) lo = mid + 1; else if (syms[mid]->section->vma > ent) hi = mid; else { sec = syms[mid]->section; break; } } if (lo >= hi && lo > codesecsym) sec = syms[lo - 1]->section; for (; sec != NULL; sec = sec->next) { if (sec->vma > ent) break; if ((sec->flags & SEC_ALLOC) == 0 || (sec->flags & SEC_LOAD) == 0) break; if ((sec->flags & SEC_CODE) != 0) s->section = sec; } s->value = ent - s->section->vma; s->name = names; *names++ = '.'; len = strlen (syms[i]->name); memcpy (names, syms[i]->name, len + 1); names += len + 1; s++; } } free (contents); } done: free (syms); return count; } /* The following functions are specific to the ELF linker, while functions above are used generally. Those named ppc64_elf_* are called by the main ELF linker code. They appear in this file more or less in the order in which they are called. eg. ppc64_elf_check_relocs is called early in the link process, ppc64_elf_finish_dynamic_sections is one of the last functions called. PowerPC64-ELF uses a similar scheme to PowerPC64-XCOFF in that functions have both a function code symbol and a function descriptor symbol. A call to foo in a relocatable object file looks like: . .text . x: . bl .foo . nop The function definition in another object file might be: . .section .opd . foo: .quad .foo . .quad .TOC.@tocbase . .quad 0 . . .text . .foo: blr When the linker resolves the call during a static link, the branch unsurprisingly just goes to .foo and the .opd information is unused. If the function definition is in a shared library, things are a little different: The call goes via a plt call stub, the opd information gets copied to the plt, and the linker patches the nop. . x: . bl .foo_stub . ld 2,40(1) . . . .foo_stub: . addis 12,2,Lfoo@toc@ha # in practice, the call stub . addi 12,12,Lfoo@toc@l # is slightly optimized, but . std 2,40(1) # this is the general idea . ld 11,0(12) . ld 2,8(12) . mtctr 11 . ld 11,16(12) . bctr . . .section .plt . Lfoo: reloc (R_PPC64_JMP_SLOT, foo) The "reloc ()" notation is supposed to indicate that the linker emits an R_PPC64_JMP_SLOT reloc against foo. The dynamic linker does the opd copying. What are the difficulties here? Well, firstly, the relocations examined by the linker in check_relocs are against the function code sym .foo, while the dynamic relocation in the plt is emitted against the function descriptor symbol, foo. Somewhere along the line, we need to carefully copy dynamic link information from one symbol to the other. Secondly, the generic part of the elf linker will make .foo a dynamic symbol as is normal for most other backends. We need foo dynamic instead, at least for an application final link. However, when creating a shared library containing foo, we need to have both symbols dynamic so that references to .foo are satisfied during the early stages of linking. Otherwise the linker might decide to pull in a definition from some other object, eg. a static library. Update: As of August 2004, we support a new convention. Function calls may use the function descriptor symbol, ie. "bl foo". This behaves exactly as "bl .foo". */ /* The linker needs to keep track of the number of relocs that it decides to copy as dynamic relocs in check_relocs for each symbol. This is so that it can later discard them if they are found to be unnecessary. We store the information in a field extending the regular ELF linker hash table. */ struct ppc_dyn_relocs { struct ppc_dyn_relocs *next; /* The input section of the reloc. */ asection *sec; /* Total number of relocs copied for the input section. */ bfd_size_type count; /* Number of pc-relative relocs copied for the input section. */ bfd_size_type pc_count; }; /* Track GOT entries needed for a given symbol. We might need more than one got entry per symbol. */ struct got_entry { struct got_entry *next; /* The symbol addend that we'll be placing in the GOT. */ bfd_vma addend; /* Unlike other ELF targets, we use separate GOT entries for the same symbol referenced from different input files. This is to support automatic multiple TOC/GOT sections, where the TOC base can vary - from one input file to another. + from one input file to another. FIXME: After group_sections we + ought to merge entries within the group. Point to the BFD owning this GOT entry. */ bfd *owner; /* Zero for non-tls entries, or TLS_TLS and one of TLS_GD, TLS_LD, TLS_TPREL or TLS_DTPREL for tls entries. */ char tls_type; /* Reference count until size_dynamic_sections, GOT offset thereafter. */ union { bfd_signed_vma refcount; bfd_vma offset; } got; }; /* The same for PLT. */ struct plt_entry { struct plt_entry *next; bfd_vma addend; union { bfd_signed_vma refcount; bfd_vma offset; } plt; }; -/* Of those relocs that might be copied as dynamic relocs, this macro +/* Of those relocs that might be copied as dynamic relocs, this function selects those that must be copied when linking a shared library, even when the symbol is local. */ -#define MUST_BE_DYN_RELOC(RTYPE) \ - ((RTYPE) != R_PPC64_REL32 \ - && (RTYPE) != R_PPC64_REL64 \ - && (RTYPE) != R_PPC64_REL30) +static int +must_be_dyn_reloc (struct bfd_link_info *info, + enum elf_ppc64_reloc_type r_type) +{ + switch (r_type) + { + default: + return 1; + case R_PPC64_REL32: + case R_PPC64_REL64: + case R_PPC64_REL30: + return 0; + + case R_PPC64_TPREL16: + case R_PPC64_TPREL16_LO: + case R_PPC64_TPREL16_HI: + case R_PPC64_TPREL16_HA: + case R_PPC64_TPREL16_DS: + case R_PPC64_TPREL16_LO_DS: + case R_PPC64_TPREL16_HIGHER: + case R_PPC64_TPREL16_HIGHERA: + case R_PPC64_TPREL16_HIGHEST: + case R_PPC64_TPREL16_HIGHESTA: + case R_PPC64_TPREL64: + return !info->executable; + } +} + /* If ELIMINATE_COPY_RELOCS is non-zero, the linker will try to avoid copying dynamic variables from a shared lib into an app's dynbss section, and instead use a dynamic relocation to point into the shared lib. With code that gcc generates, it's vital that this be enabled; In the PowerPC64 ABI, the address of a function is actually the address of a function descriptor, which resides in the .opd section. gcc uses the descriptor directly rather than going via the GOT as some other ABI's do, which means that initialized function pointers must reference the descriptor. Thus, a function pointer initialized to the address of a function in a shared library will either require a copy reloc, or a dynamic reloc. Using a copy reloc redefines the function descriptor symbol to point to the copy. This presents a problem as a plt entry for that function is also initialized from the function descriptor symbol and the copy reloc may not be initialized first. */ #define ELIMINATE_COPY_RELOCS 1 /* Section name for stubs is the associated section name plus this string. */ #define STUB_SUFFIX ".stub" /* Linker stubs. ppc_stub_long_branch: Used when a 14 bit branch (or even a 24 bit branch) can't reach its destination, but a 24 bit branch in a stub section will reach. . b dest ppc_stub_plt_branch: Similar to the above, but a 24 bit branch in the stub section won't reach its destination. . addis %r12,%r2,xxx@toc@ha . ld %r11,xxx@toc@l(%r12) . mtctr %r11 . bctr ppc_stub_plt_call: Used to call a function in a shared library. If it so happens that the plt entry referenced crosses a 64k boundary, then an extra - "addis %r12,%r12,1" will be inserted before the load at xxx+8 or - xxx+16 as appropriate. + "addi %r12,%r12,xxx@toc@l" will be inserted before the "mtctr". . addis %r12,%r2,xxx@toc@ha . std %r2,40(%r1) . ld %r11,xxx+0@toc@l(%r12) - . ld %r2,xxx+8@toc@l(%r12) . mtctr %r11 + . ld %r2,xxx+8@toc@l(%r12) . ld %r11,xxx+16@toc@l(%r12) . bctr ppc_stub_long_branch and ppc_stub_plt_branch may also have additional code to adjust the value and save r2 to support multiple toc sections. A ppc_stub_long_branch with an r2 offset looks like: . std %r2,40(%r1) . addis %r2,%r2,off@ha . addi %r2,%r2,off@l . b dest A ppc_stub_plt_branch with an r2 offset looks like: . std %r2,40(%r1) . addis %r12,%r2,xxx@toc@ha . ld %r11,xxx@toc@l(%r12) . addis %r2,%r2,off@ha . addi %r2,%r2,off@l . mtctr %r11 . bctr + + In cases where the "addis" instruction would add zero, the "addis" is + omitted and following instructions modified slightly in some cases. */ enum ppc_stub_type { ppc_stub_none, ppc_stub_long_branch, ppc_stub_long_branch_r2off, ppc_stub_plt_branch, ppc_stub_plt_branch_r2off, ppc_stub_plt_call }; struct ppc_stub_hash_entry { /* Base hash table entry structure. */ struct bfd_hash_entry root; enum ppc_stub_type stub_type; /* The stub section. */ asection *stub_sec; /* Offset within stub_sec of the beginning of this stub. */ bfd_vma stub_offset; /* Given the symbol's value and its section we can determine its final value when building the stubs (so the stub knows where to jump. */ bfd_vma target_value; asection *target_section; /* The symbol table entry, if any, that this was derived from. */ struct ppc_link_hash_entry *h; /* And the reloc addend that this was derived from. */ bfd_vma addend; /* Where this stub is being called from, or, in the case of combined stub sections, the first input section in the group. */ asection *id_sec; }; struct ppc_branch_hash_entry { /* Base hash table entry structure. */ struct bfd_hash_entry root; /* Offset within branch lookup table. */ unsigned int offset; /* Generation marker. */ unsigned int iter; }; struct ppc_link_hash_entry { struct elf_link_hash_entry elf; union { /* A pointer to the most recently used stub hash entry against this symbol. */ struct ppc_stub_hash_entry *stub_cache; /* A pointer to the next symbol starting with a '.' */ struct ppc_link_hash_entry *next_dot_sym; } u; /* Track dynamic relocs copied for this symbol. */ struct ppc_dyn_relocs *dyn_relocs; /* Link between function code and descriptor symbols. */ struct ppc_link_hash_entry *oh; /* Flag function code and descriptor symbols. */ unsigned int is_func:1; unsigned int is_func_descriptor:1; unsigned int fake:1; /* Whether global opd/toc sym has been adjusted or not. After ppc64_elf_edit_opd/ppc64_elf_edit_toc has run, this flag should be set for all globals defined in any opd/toc section. */ unsigned int adjust_done:1; /* Set if we twiddled this symbol to weak at some stage. */ unsigned int was_undefined:1; /* Contexts in which symbol is used in the GOT (or TOC). TLS_GD .. TLS_EXPLICIT bits are or'd into the mask as the corresponding relocs are encountered during check_relocs. tls_optimize clears TLS_GD .. TLS_TPREL when optimizing to indicate the corresponding GOT entry type is not needed. tls_optimize may also set TLS_TPRELGD when a GD reloc turns into a TPREL one. We use a separate flag rather than setting TPREL just for convenience in distinguishing the two cases. */ #define TLS_GD 1 /* GD reloc. */ #define TLS_LD 2 /* LD reloc. */ #define TLS_TPREL 4 /* TPREL reloc, => IE. */ #define TLS_DTPREL 8 /* DTPREL reloc, => LD. */ #define TLS_TLS 16 /* Any TLS reloc. */ #define TLS_EXPLICIT 32 /* Marks TOC section TLS relocs. */ #define TLS_TPRELGD 64 /* TPREL reloc resulting from GD->IE. */ char tls_mask; }; /* ppc64 ELF linker hash table. */ struct ppc_link_hash_table { struct elf_link_hash_table elf; /* The stub hash table. */ struct bfd_hash_table stub_hash_table; /* Another hash table for plt_branch stubs. */ struct bfd_hash_table branch_hash_table; /* Linker stub bfd. */ bfd *stub_bfd; /* Linker call-backs. */ asection * (*add_stub_section) (const char *, asection *); void (*layout_sections_again) (void); /* Array to keep track of which stub sections have been created, and information on stub grouping. */ struct map_stub { /* This is the section to which stubs in the group will be attached. */ asection *link_sec; /* The stub section. */ asection *stub_sec; /* Along with elf_gp, specifies the TOC pointer used in this group. */ bfd_vma toc_off; } *stub_group; /* Temp used when calculating TOC pointers. */ bfd_vma toc_curr; /* Highest input section id. */ int top_id; /* Highest output section index. */ int top_index; /* Used when adding symbols. */ struct ppc_link_hash_entry *dot_syms; /* List of input sections for each output section. */ asection **input_list; /* Short-cuts to get to dynamic linker sections. */ asection *got; asection *plt; asection *relplt; asection *dynbss; asection *relbss; asection *glink; asection *sfpr; asection *brlt; asection *relbrlt; /* Shortcut to .__tls_get_addr and __tls_get_addr. */ struct ppc_link_hash_entry *tls_get_addr; struct ppc_link_hash_entry *tls_get_addr_fd; /* Statistics. */ unsigned long stub_count[ppc_stub_plt_call]; /* Number of stubs against global syms. */ unsigned long stub_globals; /* Set if we should emit symbols for stubs. */ unsigned int emit_stub_syms:1; /* Support for multiple toc sections. */ unsigned int no_multi_toc:1; unsigned int multi_toc_needed:1; /* Set on error. */ unsigned int stub_error:1; /* Temp used by ppc64_elf_check_directives. */ unsigned int twiddled_syms:1; /* Incremented every time we size stubs. */ unsigned int stub_iteration; /* Small local sym to section mapping cache. */ struct sym_sec_cache sym_sec; }; /* Rename some of the generic section flags to better document how they are used here. */ #define has_toc_reloc has_gp_reloc #define makes_toc_func_call need_finalize_relax #define call_check_in_progress reloc_done /* Get the ppc64 ELF linker hash table from a link_info structure. */ #define ppc_hash_table(p) \ ((struct ppc_link_hash_table *) ((p)->hash)) #define ppc_stub_hash_lookup(table, string, create, copy) \ ((struct ppc_stub_hash_entry *) \ bfd_hash_lookup ((table), (string), (create), (copy))) #define ppc_branch_hash_lookup(table, string, create, copy) \ ((struct ppc_branch_hash_entry *) \ bfd_hash_lookup ((table), (string), (create), (copy))) /* Create an entry in the stub hash table. */ static struct bfd_hash_entry * stub_hash_newfunc (struct bfd_hash_entry *entry, struct bfd_hash_table *table, const char *string) { /* Allocate the structure if it has not already been allocated by a subclass. */ if (entry == NULL) { entry = bfd_hash_allocate (table, sizeof (struct ppc_stub_hash_entry)); if (entry == NULL) return entry; } /* Call the allocation method of the superclass. */ entry = bfd_hash_newfunc (entry, table, string); if (entry != NULL) { struct ppc_stub_hash_entry *eh; /* Initialize the local fields. */ eh = (struct ppc_stub_hash_entry *) entry; eh->stub_type = ppc_stub_none; eh->stub_sec = NULL; eh->stub_offset = 0; eh->target_value = 0; eh->target_section = NULL; eh->h = NULL; eh->id_sec = NULL; } return entry; } /* Create an entry in the branch hash table. */ static struct bfd_hash_entry * branch_hash_newfunc (struct bfd_hash_entry *entry, struct bfd_hash_table *table, const char *string) { /* Allocate the structure if it has not already been allocated by a subclass. */ if (entry == NULL) { entry = bfd_hash_allocate (table, sizeof (struct ppc_branch_hash_entry)); if (entry == NULL) return entry; } /* Call the allocation method of the superclass. */ entry = bfd_hash_newfunc (entry, table, string); if (entry != NULL) { struct ppc_branch_hash_entry *eh; /* Initialize the local fields. */ eh = (struct ppc_branch_hash_entry *) entry; eh->offset = 0; eh->iter = 0; } return entry; } /* Create an entry in a ppc64 ELF linker hash table. */ static struct bfd_hash_entry * link_hash_newfunc (struct bfd_hash_entry *entry, struct bfd_hash_table *table, const char *string) { /* Allocate the structure if it has not already been allocated by a subclass. */ if (entry == NULL) { entry = bfd_hash_allocate (table, sizeof (struct ppc_link_hash_entry)); if (entry == NULL) return entry; } /* Call the allocation method of the superclass. */ entry = _bfd_elf_link_hash_newfunc (entry, table, string); if (entry != NULL) { struct ppc_link_hash_entry *eh = (struct ppc_link_hash_entry *) entry; memset (&eh->u.stub_cache, 0, (sizeof (struct ppc_link_hash_entry) - offsetof (struct ppc_link_hash_entry, u.stub_cache))); /* When making function calls, old ABI code references function entry points (dot symbols), while new ABI code references the function descriptor symbol. We need to make any combination of reference and definition work together, without breaking archive linking. For a defined function "foo" and an undefined call to "bar": An old object defines "foo" and ".foo", references ".bar" (possibly "bar" too). A new object defines "foo" and references "bar". A new object thus has no problem with its undefined symbols being satisfied by definitions in an old object. On the other hand, the old object won't have ".bar" satisfied by a new object. Keep a list of newly added dot-symbols. */ if (string[0] == '.') { struct ppc_link_hash_table *htab; htab = (struct ppc_link_hash_table *) table; eh->u.next_dot_sym = htab->dot_syms; htab->dot_syms = eh; } } return entry; } /* Create a ppc64 ELF linker hash table. */ static struct bfd_link_hash_table * ppc64_elf_link_hash_table_create (bfd *abfd) { struct ppc_link_hash_table *htab; bfd_size_type amt = sizeof (struct ppc_link_hash_table); htab = bfd_zmalloc (amt); if (htab == NULL) return NULL; if (!_bfd_elf_link_hash_table_init (&htab->elf, abfd, link_hash_newfunc, sizeof (struct ppc_link_hash_entry))) { free (htab); return NULL; } /* Init the stub hash table too. */ if (!bfd_hash_table_init (&htab->stub_hash_table, stub_hash_newfunc, sizeof (struct ppc_stub_hash_entry))) return NULL; /* And the branch hash table. */ if (!bfd_hash_table_init (&htab->branch_hash_table, branch_hash_newfunc, sizeof (struct ppc_branch_hash_entry))) return NULL; /* Initializing two fields of the union is just cosmetic. We really only care about glist, but when compiled on a 32-bit host the bfd_vma fields are larger. Setting the bfd_vma to zero makes debugger inspection of these fields look nicer. */ htab->elf.init_got_refcount.refcount = 0; htab->elf.init_got_refcount.glist = NULL; htab->elf.init_plt_refcount.refcount = 0; htab->elf.init_plt_refcount.glist = NULL; htab->elf.init_got_offset.offset = 0; htab->elf.init_got_offset.glist = NULL; htab->elf.init_plt_offset.offset = 0; htab->elf.init_plt_offset.glist = NULL; return &htab->elf.root; } /* Free the derived linker hash table. */ static void ppc64_elf_link_hash_table_free (struct bfd_link_hash_table *hash) { struct ppc_link_hash_table *ret = (struct ppc_link_hash_table *) hash; bfd_hash_table_free (&ret->stub_hash_table); bfd_hash_table_free (&ret->branch_hash_table); _bfd_generic_link_hash_table_free (hash); } /* Satisfy the ELF linker by filling in some fields in our fake bfd. */ void ppc64_elf_init_stub_bfd (bfd *abfd, struct bfd_link_info *info) { struct ppc_link_hash_table *htab; elf_elfheader (abfd)->e_ident[EI_CLASS] = ELFCLASS64; /* Always hook our dynamic sections into the first bfd, which is the linker created stub bfd. This ensures that the GOT header is at the start of the output TOC section. */ htab = ppc_hash_table (info); htab->stub_bfd = abfd; htab->elf.dynobj = abfd; } /* Build a name for an entry in the stub hash table. */ static char * ppc_stub_name (const asection *input_section, const asection *sym_sec, const struct ppc_link_hash_entry *h, const Elf_Internal_Rela *rel) { char *stub_name; bfd_size_type len; /* rel->r_addend is actually 64 bit, but who uses more than +/- 2^31 offsets from a sym as a branch target? In fact, we could probably assume the addend is always zero. */ BFD_ASSERT (((int) rel->r_addend & 0xffffffff) == rel->r_addend); if (h) { len = 8 + 1 + strlen (h->elf.root.root.string) + 1 + 8 + 1; stub_name = bfd_malloc (len); if (stub_name == NULL) return stub_name; sprintf (stub_name, "%08x.%s+%x", input_section->id & 0xffffffff, h->elf.root.root.string, (int) rel->r_addend & 0xffffffff); } else { len = 8 + 1 + 8 + 1 + 8 + 1 + 8 + 1; stub_name = bfd_malloc (len); if (stub_name == NULL) return stub_name; sprintf (stub_name, "%08x.%x:%x+%x", input_section->id & 0xffffffff, sym_sec->id & 0xffffffff, (int) ELF64_R_SYM (rel->r_info) & 0xffffffff, (int) rel->r_addend & 0xffffffff); } if (stub_name[len - 2] == '+' && stub_name[len - 1] == '0') stub_name[len - 2] = 0; return stub_name; } /* Look up an entry in the stub hash. Stub entries are cached because creating the stub name takes a bit of time. */ static struct ppc_stub_hash_entry * ppc_get_stub_entry (const asection *input_section, const asection *sym_sec, struct ppc_link_hash_entry *h, const Elf_Internal_Rela *rel, struct ppc_link_hash_table *htab) { struct ppc_stub_hash_entry *stub_entry; const asection *id_sec; /* If this input section is part of a group of sections sharing one stub section, then use the id of the first section in the group. Stub names need to include a section id, as there may well be more than one stub used to reach say, printf, and we need to distinguish between them. */ id_sec = htab->stub_group[input_section->id].link_sec; if (h != NULL && h->u.stub_cache != NULL && h->u.stub_cache->h == h && h->u.stub_cache->id_sec == id_sec) { stub_entry = h->u.stub_cache; } else { char *stub_name; stub_name = ppc_stub_name (id_sec, sym_sec, h, rel); if (stub_name == NULL) return NULL; stub_entry = ppc_stub_hash_lookup (&htab->stub_hash_table, stub_name, FALSE, FALSE); if (h != NULL) h->u.stub_cache = stub_entry; free (stub_name); } return stub_entry; } /* Add a new stub entry to the stub hash. Not all fields of the new stub entry are initialised. */ static struct ppc_stub_hash_entry * ppc_add_stub (const char *stub_name, asection *section, struct ppc_link_hash_table *htab) { asection *link_sec; asection *stub_sec; struct ppc_stub_hash_entry *stub_entry; link_sec = htab->stub_group[section->id].link_sec; stub_sec = htab->stub_group[section->id].stub_sec; if (stub_sec == NULL) { stub_sec = htab->stub_group[link_sec->id].stub_sec; if (stub_sec == NULL) { size_t namelen; bfd_size_type len; char *s_name; namelen = strlen (link_sec->name); len = namelen + sizeof (STUB_SUFFIX); s_name = bfd_alloc (htab->stub_bfd, len); if (s_name == NULL) return NULL; memcpy (s_name, link_sec->name, namelen); memcpy (s_name + namelen, STUB_SUFFIX, sizeof (STUB_SUFFIX)); stub_sec = (*htab->add_stub_section) (s_name, link_sec); if (stub_sec == NULL) return NULL; htab->stub_group[link_sec->id].stub_sec = stub_sec; } htab->stub_group[section->id].stub_sec = stub_sec; } /* Enter this entry into the linker stub hash table. */ stub_entry = ppc_stub_hash_lookup (&htab->stub_hash_table, stub_name, TRUE, FALSE); if (stub_entry == NULL) { (*_bfd_error_handler) (_("%B: cannot create stub entry %s"), section->owner, stub_name); return NULL; } stub_entry->stub_sec = stub_sec; stub_entry->stub_offset = 0; stub_entry->id_sec = link_sec; return stub_entry; } /* Create sections for linker generated code. */ static bfd_boolean create_linkage_sections (bfd *dynobj, struct bfd_link_info *info) { struct ppc_link_hash_table *htab; flagword flags; htab = ppc_hash_table (info); /* Create .sfpr for code to save and restore fp regs. */ flags = (SEC_ALLOC | SEC_LOAD | SEC_CODE | SEC_READONLY | SEC_HAS_CONTENTS | SEC_IN_MEMORY | SEC_LINKER_CREATED); htab->sfpr = bfd_make_section_anyway_with_flags (dynobj, ".sfpr", flags); if (htab->sfpr == NULL || ! bfd_set_section_alignment (dynobj, htab->sfpr, 2)) return FALSE; /* Create .glink for lazy dynamic linking support. */ htab->glink = bfd_make_section_anyway_with_flags (dynobj, ".glink", flags); if (htab->glink == NULL || ! bfd_set_section_alignment (dynobj, htab->glink, 3)) return FALSE; /* Create branch lookup table for plt_branch stubs. */ flags = (SEC_ALLOC | SEC_LOAD | SEC_HAS_CONTENTS | SEC_IN_MEMORY | SEC_LINKER_CREATED); htab->brlt = bfd_make_section_anyway_with_flags (dynobj, ".branch_lt", flags); if (htab->brlt == NULL || ! bfd_set_section_alignment (dynobj, htab->brlt, 3)) return FALSE; if (!info->shared) return TRUE; flags = (SEC_ALLOC | SEC_LOAD | SEC_READONLY | SEC_HAS_CONTENTS | SEC_IN_MEMORY | SEC_LINKER_CREATED); htab->relbrlt = bfd_make_section_anyway_with_flags (dynobj, ".rela.branch_lt", flags); if (!htab->relbrlt || ! bfd_set_section_alignment (dynobj, htab->relbrlt, 3)) return FALSE; return TRUE; } /* Create .got and .rela.got sections in ABFD, and .got in dynobj if not already done. */ static bfd_boolean create_got_section (bfd *abfd, struct bfd_link_info *info) { asection *got, *relgot; flagword flags; struct ppc_link_hash_table *htab = ppc_hash_table (info); if (!htab->got) { if (! _bfd_elf_create_got_section (htab->elf.dynobj, info)) return FALSE; htab->got = bfd_get_section_by_name (htab->elf.dynobj, ".got"); if (!htab->got) abort (); } flags = (SEC_ALLOC | SEC_LOAD | SEC_HAS_CONTENTS | SEC_IN_MEMORY | SEC_LINKER_CREATED); got = bfd_make_section_anyway_with_flags (abfd, ".got", flags); if (!got || !bfd_set_section_alignment (abfd, got, 3)) return FALSE; relgot = bfd_make_section_anyway_with_flags (abfd, ".rela.got", flags | SEC_READONLY); if (!relgot || ! bfd_set_section_alignment (abfd, relgot, 3)) return FALSE; ppc64_elf_tdata (abfd)->got = got; ppc64_elf_tdata (abfd)->relgot = relgot; return TRUE; } /* Create the dynamic sections, and set up shortcuts. */ static bfd_boolean ppc64_elf_create_dynamic_sections (bfd *dynobj, struct bfd_link_info *info) { struct ppc_link_hash_table *htab; if (!_bfd_elf_create_dynamic_sections (dynobj, info)) return FALSE; htab = ppc_hash_table (info); if (!htab->got) htab->got = bfd_get_section_by_name (dynobj, ".got"); htab->plt = bfd_get_section_by_name (dynobj, ".plt"); htab->relplt = bfd_get_section_by_name (dynobj, ".rela.plt"); htab->dynbss = bfd_get_section_by_name (dynobj, ".dynbss"); if (!info->shared) htab->relbss = bfd_get_section_by_name (dynobj, ".rela.bss"); if (!htab->got || !htab->plt || !htab->relplt || !htab->dynbss || (!info->shared && !htab->relbss)) abort (); return TRUE; } /* Merge PLT info on FROM with that on TO. */ static void move_plt_plist (struct ppc_link_hash_entry *from, struct ppc_link_hash_entry *to) { if (from->elf.plt.plist != NULL) { if (to->elf.plt.plist != NULL) { struct plt_entry **entp; struct plt_entry *ent; for (entp = &from->elf.plt.plist; (ent = *entp) != NULL; ) { struct plt_entry *dent; for (dent = to->elf.plt.plist; dent != NULL; dent = dent->next) if (dent->addend == ent->addend) { dent->plt.refcount += ent->plt.refcount; *entp = ent->next; break; } if (dent == NULL) entp = &ent->next; } *entp = to->elf.plt.plist; } to->elf.plt.plist = from->elf.plt.plist; from->elf.plt.plist = NULL; } } /* Copy the extra info we tack onto an elf_link_hash_entry. */ static void ppc64_elf_copy_indirect_symbol (struct bfd_link_info *info, struct elf_link_hash_entry *dir, struct elf_link_hash_entry *ind) { struct ppc_link_hash_entry *edir, *eind; edir = (struct ppc_link_hash_entry *) dir; eind = (struct ppc_link_hash_entry *) ind; /* Copy over any dynamic relocs we may have on the indirect sym. */ if (eind->dyn_relocs != NULL) { if (edir->dyn_relocs != NULL) { struct ppc_dyn_relocs **pp; struct ppc_dyn_relocs *p; /* Add reloc counts against the indirect sym to the direct sym list. Merge any entries against the same section. */ for (pp = &eind->dyn_relocs; (p = *pp) != NULL; ) { struct ppc_dyn_relocs *q; for (q = edir->dyn_relocs; q != NULL; q = q->next) if (q->sec == p->sec) { q->pc_count += p->pc_count; q->count += p->count; *pp = p->next; break; } if (q == NULL) pp = &p->next; } *pp = edir->dyn_relocs; } edir->dyn_relocs = eind->dyn_relocs; eind->dyn_relocs = NULL; } edir->is_func |= eind->is_func; edir->is_func_descriptor |= eind->is_func_descriptor; edir->tls_mask |= eind->tls_mask; /* If called to transfer flags for a weakdef during processing of elf_adjust_dynamic_symbol, don't copy NON_GOT_REF. We clear it ourselves for ELIMINATE_COPY_RELOCS. */ if (!(ELIMINATE_COPY_RELOCS && eind->elf.root.type != bfd_link_hash_indirect && edir->elf.dynamic_adjusted)) edir->elf.non_got_ref |= eind->elf.non_got_ref; edir->elf.ref_dynamic |= eind->elf.ref_dynamic; edir->elf.ref_regular |= eind->elf.ref_regular; edir->elf.ref_regular_nonweak |= eind->elf.ref_regular_nonweak; edir->elf.needs_plt |= eind->elf.needs_plt; /* If we were called to copy over info for a weak sym, that's all. */ if (eind->elf.root.type != bfd_link_hash_indirect) return; /* Copy over got entries that we may have already seen to the symbol which just became indirect. */ if (eind->elf.got.glist != NULL) { if (edir->elf.got.glist != NULL) { struct got_entry **entp; struct got_entry *ent; for (entp = &eind->elf.got.glist; (ent = *entp) != NULL; ) { struct got_entry *dent; for (dent = edir->elf.got.glist; dent != NULL; dent = dent->next) if (dent->addend == ent->addend && dent->owner == ent->owner && dent->tls_type == ent->tls_type) { dent->got.refcount += ent->got.refcount; *entp = ent->next; break; } if (dent == NULL) entp = &ent->next; } *entp = edir->elf.got.glist; } edir->elf.got.glist = eind->elf.got.glist; eind->elf.got.glist = NULL; } /* And plt entries. */ move_plt_plist (eind, edir); if (eind->elf.dynindx != -1) { if (edir->elf.dynindx != -1) _bfd_elf_strtab_delref (elf_hash_table (info)->dynstr, edir->elf.dynstr_index); edir->elf.dynindx = eind->elf.dynindx; edir->elf.dynstr_index = eind->elf.dynstr_index; eind->elf.dynindx = -1; eind->elf.dynstr_index = 0; } } /* Find the function descriptor hash entry from the given function code hash entry FH. Link the entries via their OH fields. */ static struct ppc_link_hash_entry * get_fdh (struct ppc_link_hash_entry *fh, struct ppc_link_hash_table *htab) { struct ppc_link_hash_entry *fdh = fh->oh; if (fdh == NULL) { const char *fd_name = fh->elf.root.root.string + 1; fdh = (struct ppc_link_hash_entry *) elf_link_hash_lookup (&htab->elf, fd_name, FALSE, FALSE, FALSE); if (fdh != NULL) { fdh->is_func_descriptor = 1; fdh->oh = fh; fh->is_func = 1; fh->oh = fdh; } } return fdh; } /* Make a fake function descriptor sym for the code sym FH. */ static struct ppc_link_hash_entry * make_fdh (struct bfd_link_info *info, struct ppc_link_hash_entry *fh) { bfd *abfd; asymbol *newsym; struct bfd_link_hash_entry *bh; struct ppc_link_hash_entry *fdh; abfd = fh->elf.root.u.undef.abfd; newsym = bfd_make_empty_symbol (abfd); newsym->name = fh->elf.root.root.string + 1; newsym->section = bfd_und_section_ptr; newsym->value = 0; newsym->flags = BSF_WEAK; bh = NULL; if (!_bfd_generic_link_add_one_symbol (info, abfd, newsym->name, newsym->flags, newsym->section, newsym->value, NULL, FALSE, FALSE, &bh)) return NULL; fdh = (struct ppc_link_hash_entry *) bh; fdh->elf.non_elf = 0; fdh->fake = 1; fdh->is_func_descriptor = 1; fdh->oh = fh; fh->is_func = 1; fh->oh = fdh; return fdh; } /* Fix function descriptor symbols defined in .opd sections to be function type. */ static bfd_boolean ppc64_elf_add_symbol_hook (bfd *ibfd ATTRIBUTE_UNUSED, struct bfd_link_info *info ATTRIBUTE_UNUSED, Elf_Internal_Sym *isym, const char **name ATTRIBUTE_UNUSED, flagword *flags ATTRIBUTE_UNUSED, asection **sec, bfd_vma *value ATTRIBUTE_UNUSED) { if (*sec != NULL && strcmp (bfd_get_section_name (ibfd, *sec), ".opd") == 0) isym->st_info = ELF_ST_INFO (ELF_ST_BIND (isym->st_info), STT_FUNC); return TRUE; } /* This function makes an old ABI object reference to ".bar" cause the inclusion of a new ABI object archive that defines "bar". NAME is a symbol defined in an archive. Return a symbol in the hash table that might be satisfied by the archive symbols. */ static struct elf_link_hash_entry * ppc64_elf_archive_symbol_lookup (bfd *abfd, struct bfd_link_info *info, const char *name) { struct elf_link_hash_entry *h; char *dot_name; size_t len; h = _bfd_elf_archive_symbol_lookup (abfd, info, name); if (h != NULL /* Don't return this sym if it is a fake function descriptor created by add_symbol_adjust. */ && !(h->root.type == bfd_link_hash_undefweak && ((struct ppc_link_hash_entry *) h)->fake)) return h; if (name[0] == '.') return h; len = strlen (name); dot_name = bfd_alloc (abfd, len + 2); if (dot_name == NULL) return (struct elf_link_hash_entry *) 0 - 1; dot_name[0] = '.'; memcpy (dot_name + 1, name, len + 1); h = _bfd_elf_archive_symbol_lookup (abfd, info, dot_name); bfd_release (abfd, dot_name); return h; } /* This function satisfies all old ABI object references to ".bar" if a new ABI object defines "bar". Well, at least, undefined dot symbols are made weak. This stops later archive searches from including an object if we already have a function descriptor definition. It also prevents the linker complaining about undefined symbols. We also check and correct mismatched symbol visibility here. The most restrictive visibility of the function descriptor and the function entry symbol is used. */ static bfd_boolean add_symbol_adjust (struct ppc_link_hash_entry *eh, struct bfd_link_info *info) { struct ppc_link_hash_table *htab; struct ppc_link_hash_entry *fdh; if (eh->elf.root.type == bfd_link_hash_indirect) return TRUE; if (eh->elf.root.type == bfd_link_hash_warning) eh = (struct ppc_link_hash_entry *) eh->elf.root.u.i.link; if (eh->elf.root.root.string[0] != '.') abort (); htab = ppc_hash_table (info); fdh = get_fdh (eh, htab); if (fdh == NULL && !info->relocatable && (eh->elf.root.type == bfd_link_hash_undefined || eh->elf.root.type == bfd_link_hash_undefweak) && eh->elf.ref_regular) { /* Make an undefweak function descriptor sym, which is enough to pull in an --as-needed shared lib, but won't cause link errors. Archives are handled elsewhere. */ fdh = make_fdh (info, eh); if (fdh == NULL) return FALSE; else fdh->elf.ref_regular = 1; } else if (fdh != NULL) { unsigned entry_vis = ELF_ST_VISIBILITY (eh->elf.other) - 1; unsigned descr_vis = ELF_ST_VISIBILITY (fdh->elf.other) - 1; if (entry_vis < descr_vis) fdh->elf.other += entry_vis - descr_vis; else if (entry_vis > descr_vis) eh->elf.other += descr_vis - entry_vis; if ((fdh->elf.root.type == bfd_link_hash_defined || fdh->elf.root.type == bfd_link_hash_defweak) && eh->elf.root.type == bfd_link_hash_undefined) { eh->elf.root.type = bfd_link_hash_undefweak; eh->was_undefined = 1; htab->twiddled_syms = 1; } } return TRUE; } /* Process list of dot-symbols we made in link_hash_newfunc. */ static bfd_boolean ppc64_elf_check_directives (bfd *ibfd, struct bfd_link_info *info) { struct ppc_link_hash_table *htab; struct ppc_link_hash_entry **p, *eh; htab = ppc_hash_table (info); if (!is_ppc64_elf_target (htab->elf.root.creator)) return TRUE; if (is_ppc64_elf_target (ibfd->xvec)) { p = &htab->dot_syms; while ((eh = *p) != NULL) { *p = NULL; if (!add_symbol_adjust (eh, info)) return FALSE; p = &eh->u.next_dot_sym; } } /* Clear the list for non-ppc64 input files. */ p = &htab->dot_syms; while ((eh = *p) != NULL) { *p = NULL; p = &eh->u.next_dot_sym; } /* We need to fix the undefs list for any syms we have twiddled to undef_weak. */ if (htab->twiddled_syms) { bfd_link_repair_undef_list (&htab->elf.root); htab->twiddled_syms = 0; } return TRUE; } /* Undo hash table changes when an --as-needed input file is determined not to be needed. */ static bfd_boolean ppc64_elf_as_needed_cleanup (bfd *ibfd ATTRIBUTE_UNUSED, struct bfd_link_info *info) { ppc_hash_table (info)->dot_syms = NULL; return TRUE; } static bfd_boolean update_local_sym_info (bfd *abfd, Elf_Internal_Shdr *symtab_hdr, unsigned long r_symndx, bfd_vma r_addend, int tls_type) { struct got_entry **local_got_ents = elf_local_got_ents (abfd); char *local_got_tls_masks; if (local_got_ents == NULL) { bfd_size_type size = symtab_hdr->sh_info; size *= sizeof (*local_got_ents) + sizeof (*local_got_tls_masks); local_got_ents = bfd_zalloc (abfd, size); if (local_got_ents == NULL) return FALSE; elf_local_got_ents (abfd) = local_got_ents; } if ((tls_type & TLS_EXPLICIT) == 0) { struct got_entry *ent; for (ent = local_got_ents[r_symndx]; ent != NULL; ent = ent->next) if (ent->addend == r_addend && ent->owner == abfd && ent->tls_type == tls_type) break; if (ent == NULL) { bfd_size_type amt = sizeof (*ent); ent = bfd_alloc (abfd, amt); if (ent == NULL) return FALSE; ent->next = local_got_ents[r_symndx]; ent->addend = r_addend; ent->owner = abfd; ent->tls_type = tls_type; ent->got.refcount = 0; local_got_ents[r_symndx] = ent; } ent->got.refcount += 1; } local_got_tls_masks = (char *) (local_got_ents + symtab_hdr->sh_info); local_got_tls_masks[r_symndx] |= tls_type; return TRUE; } static bfd_boolean update_plt_info (bfd *abfd, struct ppc_link_hash_entry *eh, bfd_vma addend) { struct plt_entry *ent; for (ent = eh->elf.plt.plist; ent != NULL; ent = ent->next) if (ent->addend == addend) break; if (ent == NULL) { bfd_size_type amt = sizeof (*ent); ent = bfd_alloc (abfd, amt); if (ent == NULL) return FALSE; ent->next = eh->elf.plt.plist; ent->addend = addend; ent->plt.refcount = 0; eh->elf.plt.plist = ent; } ent->plt.refcount += 1; eh->elf.needs_plt = 1; if (eh->elf.root.root.string[0] == '.' && eh->elf.root.root.string[1] != '\0') eh->is_func = 1; return TRUE; } /* Look through the relocs for a section during the first phase, and calculate needed space in the global offset table, procedure linkage table, and dynamic reloc sections. */ static bfd_boolean ppc64_elf_check_relocs (bfd *abfd, struct bfd_link_info *info, asection *sec, const Elf_Internal_Rela *relocs) { struct ppc_link_hash_table *htab; Elf_Internal_Shdr *symtab_hdr; struct elf_link_hash_entry **sym_hashes, **sym_hashes_end; const Elf_Internal_Rela *rel; const Elf_Internal_Rela *rel_end; asection *sreloc; asection **opd_sym_map; + struct elf_link_hash_entry *tga, *dottga; if (info->relocatable) return TRUE; /* Don't do anything special with non-loaded, non-alloced sections. In particular, any relocs in such sections should not affect GOT and PLT reference counting (ie. we don't allow them to create GOT or PLT entries), there's no possibility or desire to optimize TLS relocs, and there's not much point in propagating relocs to shared libs that the dynamic linker won't relocate. */ if ((sec->flags & SEC_ALLOC) == 0) return TRUE; htab = ppc_hash_table (info); + tga = elf_link_hash_lookup (&htab->elf, "__tls_get_addr", + FALSE, FALSE, TRUE); + dottga = elf_link_hash_lookup (&htab->elf, ".__tls_get_addr", + FALSE, FALSE, TRUE); symtab_hdr = &elf_tdata (abfd)->symtab_hdr; sym_hashes = elf_sym_hashes (abfd); sym_hashes_end = (sym_hashes + symtab_hdr->sh_size / sizeof (Elf64_External_Sym) - symtab_hdr->sh_info); sreloc = NULL; opd_sym_map = NULL; if (strcmp (bfd_get_section_name (abfd, sec), ".opd") == 0) { /* Garbage collection needs some extra help with .opd sections. We don't want to necessarily keep everything referenced by relocs in .opd, as that would keep all functions. Instead, if we reference an .opd symbol (a function descriptor), we want to keep the function code symbol's section. This is easy for global symbols, but for local syms we need to keep information about the associated function section. Later, if edit_opd deletes entries, we'll use this array to adjust local syms in .opd. */ union opd_info { asection *func_section; long entry_adjust; }; bfd_size_type amt; amt = sec->size * sizeof (union opd_info) / 8; opd_sym_map = bfd_zalloc (abfd, amt); if (opd_sym_map == NULL) return FALSE; ppc64_elf_section_data (sec)->u.opd_func_sec = opd_sym_map; BFD_ASSERT (ppc64_elf_section_data (sec)->sec_type == sec_normal); ppc64_elf_section_data (sec)->sec_type = sec_opd; } if (htab->sfpr == NULL && !create_linkage_sections (htab->elf.dynobj, info)) return FALSE; rel_end = relocs + sec->reloc_count; for (rel = relocs; rel < rel_end; rel++) { unsigned long r_symndx; struct elf_link_hash_entry *h; enum elf_ppc64_reloc_type r_type; - int tls_type = 0; + int tls_type; struct _ppc64_elf_section_data *ppc64_sec; r_symndx = ELF64_R_SYM (rel->r_info); if (r_symndx < symtab_hdr->sh_info) h = NULL; else { h = sym_hashes[r_symndx - symtab_hdr->sh_info]; while (h->root.type == bfd_link_hash_indirect || h->root.type == bfd_link_hash_warning) h = (struct elf_link_hash_entry *) h->root.u.i.link; } + tls_type = 0; r_type = ELF64_R_TYPE (rel->r_info); + if (h != NULL && (h == tga || h == dottga)) + switch (r_type) + { + default: + break; + + case R_PPC64_REL24: + case R_PPC64_REL14: + case R_PPC64_REL14_BRTAKEN: + case R_PPC64_REL14_BRNTAKEN: + case R_PPC64_ADDR24: + case R_PPC64_ADDR14: + case R_PPC64_ADDR14_BRTAKEN: + case R_PPC64_ADDR14_BRNTAKEN: + if (rel != relocs + && (ELF64_R_TYPE (rel[-1].r_info) == R_PPC64_TLSGD + || ELF64_R_TYPE (rel[-1].r_info) == R_PPC64_TLSLD)) + /* We have a new-style __tls_get_addr call with a marker + reloc. */ + ; + else + /* Mark this section as having an old-style call. */ + sec->has_tls_get_addr_call = 1; + break; + } + switch (r_type) { + case R_PPC64_TLSGD: + case R_PPC64_TLSLD: + /* These special tls relocs tie a call to __tls_get_addr with + its parameter symbol. */ + break; + case R_PPC64_GOT_TLSLD16: case R_PPC64_GOT_TLSLD16_LO: case R_PPC64_GOT_TLSLD16_HI: case R_PPC64_GOT_TLSLD16_HA: - ppc64_tlsld_got (abfd)->refcount += 1; tls_type = TLS_TLS | TLS_LD; goto dogottls; case R_PPC64_GOT_TLSGD16: case R_PPC64_GOT_TLSGD16_LO: case R_PPC64_GOT_TLSGD16_HI: case R_PPC64_GOT_TLSGD16_HA: tls_type = TLS_TLS | TLS_GD; goto dogottls; case R_PPC64_GOT_TPREL16_DS: case R_PPC64_GOT_TPREL16_LO_DS: case R_PPC64_GOT_TPREL16_HI: case R_PPC64_GOT_TPREL16_HA: - if (info->shared) + if (!info->executable) info->flags |= DF_STATIC_TLS; tls_type = TLS_TLS | TLS_TPREL; goto dogottls; case R_PPC64_GOT_DTPREL16_DS: case R_PPC64_GOT_DTPREL16_LO_DS: case R_PPC64_GOT_DTPREL16_HI: case R_PPC64_GOT_DTPREL16_HA: tls_type = TLS_TLS | TLS_DTPREL; dogottls: sec->has_tls_reloc = 1; /* Fall thru */ case R_PPC64_GOT16: case R_PPC64_GOT16_DS: case R_PPC64_GOT16_HA: case R_PPC64_GOT16_HI: case R_PPC64_GOT16_LO: case R_PPC64_GOT16_LO_DS: /* This symbol requires a global offset table entry. */ sec->has_toc_reloc = 1; if (ppc64_elf_tdata (abfd)->got == NULL && !create_got_section (abfd, info)) return FALSE; if (h != NULL) { struct ppc_link_hash_entry *eh; struct got_entry *ent; eh = (struct ppc_link_hash_entry *) h; for (ent = eh->elf.got.glist; ent != NULL; ent = ent->next) if (ent->addend == rel->r_addend && ent->owner == abfd && ent->tls_type == tls_type) break; if (ent == NULL) { bfd_size_type amt = sizeof (*ent); ent = bfd_alloc (abfd, amt); if (ent == NULL) return FALSE; ent->next = eh->elf.got.glist; ent->addend = rel->r_addend; ent->owner = abfd; ent->tls_type = tls_type; ent->got.refcount = 0; eh->elf.got.glist = ent; } ent->got.refcount += 1; eh->tls_mask |= tls_type; } else /* This is a global offset table entry for a local symbol. */ if (!update_local_sym_info (abfd, symtab_hdr, r_symndx, rel->r_addend, tls_type)) return FALSE; break; case R_PPC64_PLT16_HA: case R_PPC64_PLT16_HI: case R_PPC64_PLT16_LO: case R_PPC64_PLT32: case R_PPC64_PLT64: /* This symbol requires a procedure linkage table entry. We actually build the entry in adjust_dynamic_symbol, because this might be a case of linking PIC code without linking in any dynamic objects, in which case we don't need to generate a procedure linkage table after all. */ if (h == NULL) { /* It does not make sense to have a procedure linkage table entry for a local symbol. */ bfd_set_error (bfd_error_bad_value); return FALSE; } else if (!update_plt_info (abfd, (struct ppc_link_hash_entry *) h, rel->r_addend)) return FALSE; break; /* The following relocations don't need to propagate the relocation if linking a shared object since they are section relative. */ case R_PPC64_SECTOFF: case R_PPC64_SECTOFF_LO: case R_PPC64_SECTOFF_HI: case R_PPC64_SECTOFF_HA: case R_PPC64_SECTOFF_DS: case R_PPC64_SECTOFF_LO_DS: case R_PPC64_DTPREL16: case R_PPC64_DTPREL16_LO: case R_PPC64_DTPREL16_HI: case R_PPC64_DTPREL16_HA: case R_PPC64_DTPREL16_DS: case R_PPC64_DTPREL16_LO_DS: case R_PPC64_DTPREL16_HIGHER: case R_PPC64_DTPREL16_HIGHERA: case R_PPC64_DTPREL16_HIGHEST: case R_PPC64_DTPREL16_HIGHESTA: break; /* Nor do these. */ case R_PPC64_TOC16: case R_PPC64_TOC16_LO: case R_PPC64_TOC16_HI: case R_PPC64_TOC16_HA: case R_PPC64_TOC16_DS: case R_PPC64_TOC16_LO_DS: sec->has_toc_reloc = 1; break; /* This relocation describes the C++ object vtable hierarchy. Reconstruct it for later use during GC. */ case R_PPC64_GNU_VTINHERIT: if (!bfd_elf_gc_record_vtinherit (abfd, sec, h, rel->r_offset)) return FALSE; break; /* This relocation describes which C++ vtable entries are actually used. Record for later use during GC. */ case R_PPC64_GNU_VTENTRY: if (!bfd_elf_gc_record_vtentry (abfd, sec, h, rel->r_addend)) return FALSE; break; case R_PPC64_REL14: case R_PPC64_REL14_BRTAKEN: case R_PPC64_REL14_BRNTAKEN: { asection *dest = NULL; /* Heuristic: If jumping outside our section, chances are we are going to need a stub. */ if (h != NULL) { /* If the sym is weak it may be overridden later, so don't assume we know where a weak sym lives. */ if (h->root.type == bfd_link_hash_defined) dest = h->root.u.def.section; } else dest = bfd_section_from_r_symndx (abfd, &htab->sym_sec, sec, r_symndx); if (dest != sec) ppc64_elf_section_data (sec)->has_14bit_branch = 1; } /* Fall through. */ case R_PPC64_REL24: if (h != NULL) { /* We may need a .plt entry if the function this reloc refers to is in a shared lib. */ if (!update_plt_info (abfd, (struct ppc_link_hash_entry *) h, rel->r_addend)) return FALSE; - if (h == &htab->tls_get_addr->elf - || h == &htab->tls_get_addr_fd->elf) + if (h == tga || h == dottga) sec->has_tls_reloc = 1; - else if (htab->tls_get_addr == NULL - && CONST_STRNEQ (h->root.root.string, ".__tls_get_addr") - && (h->root.root.string[15] == 0 - || h->root.root.string[15] == '@')) - { - htab->tls_get_addr = (struct ppc_link_hash_entry *) h; - sec->has_tls_reloc = 1; - } - else if (htab->tls_get_addr_fd == NULL - && CONST_STRNEQ (h->root.root.string, "__tls_get_addr") - && (h->root.root.string[14] == 0 - || h->root.root.string[14] == '@')) - { - htab->tls_get_addr_fd = (struct ppc_link_hash_entry *) h; - sec->has_tls_reloc = 1; - } } break; case R_PPC64_TPREL64: tls_type = TLS_EXPLICIT | TLS_TLS | TLS_TPREL; - if (info->shared) + if (!info->executable) info->flags |= DF_STATIC_TLS; goto dotlstoc; case R_PPC64_DTPMOD64: if (rel + 1 < rel_end && rel[1].r_info == ELF64_R_INFO (r_symndx, R_PPC64_DTPREL64) && rel[1].r_offset == rel->r_offset + 8) tls_type = TLS_EXPLICIT | TLS_TLS | TLS_GD; else tls_type = TLS_EXPLICIT | TLS_TLS | TLS_LD; goto dotlstoc; case R_PPC64_DTPREL64: tls_type = TLS_EXPLICIT | TLS_TLS | TLS_DTPREL; if (rel != relocs && rel[-1].r_info == ELF64_R_INFO (r_symndx, R_PPC64_DTPMOD64) && rel[-1].r_offset == rel->r_offset - 8) /* This is the second reloc of a dtpmod, dtprel pair. Don't mark with TLS_DTPREL. */ goto dodyn; dotlstoc: sec->has_tls_reloc = 1; if (h != NULL) { struct ppc_link_hash_entry *eh; eh = (struct ppc_link_hash_entry *) h; eh->tls_mask |= tls_type; } else if (!update_local_sym_info (abfd, symtab_hdr, r_symndx, rel->r_addend, tls_type)) return FALSE; ppc64_sec = ppc64_elf_section_data (sec); if (ppc64_sec->sec_type != sec_toc) { + bfd_size_type amt; + /* One extra to simplify get_tls_mask. */ - bfd_size_type amt = sec->size * sizeof (unsigned) / 8 + 1; - ppc64_sec->u.t_symndx = bfd_zalloc (abfd, amt); - if (ppc64_sec->u.t_symndx == NULL) + amt = sec->size * sizeof (unsigned) / 8 + sizeof (unsigned); + ppc64_sec->u.toc.symndx = bfd_zalloc (abfd, amt); + if (ppc64_sec->u.toc.symndx == NULL) return FALSE; + amt = sec->size * sizeof (bfd_vma) / 8; + ppc64_sec->u.toc.add = bfd_zalloc (abfd, amt); + if (ppc64_sec->u.toc.add == NULL) + return FALSE; BFD_ASSERT (ppc64_sec->sec_type == sec_normal); ppc64_sec->sec_type = sec_toc; } BFD_ASSERT (rel->r_offset % 8 == 0); - ppc64_sec->u.t_symndx[rel->r_offset / 8] = r_symndx; + ppc64_sec->u.toc.symndx[rel->r_offset / 8] = r_symndx; + ppc64_sec->u.toc.add[rel->r_offset / 8] = rel->r_addend; /* Mark the second slot of a GD or LD entry. -1 to indicate GD and -2 to indicate LD. */ if (tls_type == (TLS_EXPLICIT | TLS_TLS | TLS_GD)) - ppc64_sec->u.t_symndx[rel->r_offset / 8 + 1] = -1; + ppc64_sec->u.toc.symndx[rel->r_offset / 8 + 1] = -1; else if (tls_type == (TLS_EXPLICIT | TLS_TLS | TLS_LD)) - ppc64_sec->u.t_symndx[rel->r_offset / 8 + 1] = -2; + ppc64_sec->u.toc.symndx[rel->r_offset / 8 + 1] = -2; goto dodyn; case R_PPC64_TPREL16: case R_PPC64_TPREL16_LO: case R_PPC64_TPREL16_HI: case R_PPC64_TPREL16_HA: case R_PPC64_TPREL16_DS: case R_PPC64_TPREL16_LO_DS: case R_PPC64_TPREL16_HIGHER: case R_PPC64_TPREL16_HIGHERA: case R_PPC64_TPREL16_HIGHEST: case R_PPC64_TPREL16_HIGHESTA: if (info->shared) { - info->flags |= DF_STATIC_TLS; + if (!info->executable) + info->flags |= DF_STATIC_TLS; goto dodyn; } break; case R_PPC64_ADDR64: if (opd_sym_map != NULL && rel + 1 < rel_end && ELF64_R_TYPE ((rel + 1)->r_info) == R_PPC64_TOC) { if (h != NULL) { if (h->root.root.string[0] == '.' && h->root.root.string[1] != 0 && get_fdh ((struct ppc_link_hash_entry *) h, htab)) ; else ((struct ppc_link_hash_entry *) h)->is_func = 1; } else { asection *s; s = bfd_section_from_r_symndx (abfd, &htab->sym_sec, sec, r_symndx); if (s == NULL) return FALSE; else if (s != sec) opd_sym_map[rel->r_offset / 8] = s; } } /* Fall through. */ case R_PPC64_REL30: case R_PPC64_REL32: case R_PPC64_REL64: case R_PPC64_ADDR14: case R_PPC64_ADDR14_BRNTAKEN: case R_PPC64_ADDR14_BRTAKEN: case R_PPC64_ADDR16: case R_PPC64_ADDR16_DS: case R_PPC64_ADDR16_HA: case R_PPC64_ADDR16_HI: case R_PPC64_ADDR16_HIGHER: case R_PPC64_ADDR16_HIGHERA: case R_PPC64_ADDR16_HIGHEST: case R_PPC64_ADDR16_HIGHESTA: case R_PPC64_ADDR16_LO: case R_PPC64_ADDR16_LO_DS: case R_PPC64_ADDR24: case R_PPC64_ADDR32: case R_PPC64_UADDR16: case R_PPC64_UADDR32: case R_PPC64_UADDR64: case R_PPC64_TOC: if (h != NULL && !info->shared) /* We may need a copy reloc. */ h->non_got_ref = 1; /* Don't propagate .opd relocs. */ if (NO_OPD_RELOCS && opd_sym_map != NULL) break; /* If we are creating a shared library, and this is a reloc against a global symbol, or a non PC relative reloc against a local symbol, then we need to copy the reloc into the shared library. However, if we are linking with -Bsymbolic, we do not need to copy a reloc against a global symbol which is defined in an object we are including in the link (i.e., DEF_REGULAR is set). At this point we have not seen all the input files, so it is possible that DEF_REGULAR is not set now but will be set later (it is never cleared). In case of a weak definition, DEF_REGULAR may be cleared later by a strong definition in a shared library. We account for that possibility below by storing information in the dyn_relocs field of the hash table entry. A similar situation occurs when creating shared libraries and symbol visibility changes render the symbol local. If on the other hand, we are creating an executable, we may need to keep relocations for symbols satisfied by a dynamic library if we manage to avoid copy relocs for the symbol. */ dodyn: if ((info->shared - && (MUST_BE_DYN_RELOC (r_type) + && (must_be_dyn_reloc (info, r_type) || (h != NULL && (! info->symbolic || h->root.type == bfd_link_hash_defweak || !h->def_regular)))) || (ELIMINATE_COPY_RELOCS && !info->shared && h != NULL && (h->root.type == bfd_link_hash_defweak || !h->def_regular))) { struct ppc_dyn_relocs *p; struct ppc_dyn_relocs **head; /* We must copy these reloc types into the output file. Create a reloc section in dynobj and make room for this reloc. */ if (sreloc == NULL) { const char *name; bfd *dynobj; name = (bfd_elf_string_from_elf_section (abfd, elf_elfheader (abfd)->e_shstrndx, elf_section_data (sec)->rel_hdr.sh_name)); if (name == NULL) return FALSE; if (! CONST_STRNEQ (name, ".rela") || strcmp (bfd_get_section_name (abfd, sec), name + 5) != 0) { (*_bfd_error_handler) (_("%B: bad relocation section name `%s\'"), abfd, name); bfd_set_error (bfd_error_bad_value); } dynobj = htab->elf.dynobj; sreloc = bfd_get_section_by_name (dynobj, name); if (sreloc == NULL) { flagword flags; flags = (SEC_HAS_CONTENTS | SEC_READONLY | SEC_IN_MEMORY | SEC_LINKER_CREATED | SEC_ALLOC | SEC_LOAD); sreloc = bfd_make_section_with_flags (dynobj, name, flags); if (sreloc == NULL || ! bfd_set_section_alignment (dynobj, sreloc, 3)) return FALSE; } elf_section_data (sec)->sreloc = sreloc; } /* If this is a global symbol, we count the number of relocations we need for this symbol. */ if (h != NULL) { head = &((struct ppc_link_hash_entry *) h)->dyn_relocs; } else { /* Track dynamic relocs needed for local syms too. We really need local syms available to do this easily. Oh well. */ asection *s; void *vpp; s = bfd_section_from_r_symndx (abfd, &htab->sym_sec, sec, r_symndx); if (s == NULL) return FALSE; vpp = &elf_section_data (s)->local_dynrel; head = (struct ppc_dyn_relocs **) vpp; } p = *head; if (p == NULL || p->sec != sec) { p = bfd_alloc (htab->elf.dynobj, sizeof *p); if (p == NULL) return FALSE; p->next = *head; *head = p; p->sec = sec; p->count = 0; p->pc_count = 0; } p->count += 1; - if (!MUST_BE_DYN_RELOC (r_type)) + if (!must_be_dyn_reloc (info, r_type)) p->pc_count += 1; } break; default: break; } } return TRUE; } /* OFFSET in OPD_SEC specifies a function descriptor. Return the address of the code entry point, and its section. */ static bfd_vma opd_entry_value (asection *opd_sec, bfd_vma offset, asection **code_sec, bfd_vma *code_off) { bfd *opd_bfd = opd_sec->owner; Elf_Internal_Rela *relocs; Elf_Internal_Rela *lo, *hi, *look; bfd_vma val; /* No relocs implies we are linking a --just-symbols object. */ if (opd_sec->reloc_count == 0) { bfd_vma val; if (!bfd_get_section_contents (opd_bfd, opd_sec, &val, offset, 8)) return (bfd_vma) -1; if (code_sec != NULL) { asection *sec, *likely = NULL; for (sec = opd_bfd->sections; sec != NULL; sec = sec->next) if (sec->vma <= val && (sec->flags & SEC_LOAD) != 0 && (sec->flags & SEC_ALLOC) != 0) likely = sec; if (likely != NULL) { *code_sec = likely; if (code_off != NULL) *code_off = val - likely->vma; } } return val; } relocs = ppc64_elf_tdata (opd_bfd)->opd_relocs; if (relocs == NULL) relocs = _bfd_elf_link_read_relocs (opd_bfd, opd_sec, NULL, NULL, TRUE); /* Go find the opd reloc at the sym address. */ lo = relocs; BFD_ASSERT (lo != NULL); hi = lo + opd_sec->reloc_count - 1; /* ignore last reloc */ val = (bfd_vma) -1; while (lo < hi) { look = lo + (hi - lo) / 2; if (look->r_offset < offset) lo = look + 1; else if (look->r_offset > offset) hi = look; else { Elf_Internal_Shdr *symtab_hdr = &elf_tdata (opd_bfd)->symtab_hdr; if (ELF64_R_TYPE (look->r_info) == R_PPC64_ADDR64 && ELF64_R_TYPE ((look + 1)->r_info) == R_PPC64_TOC) { unsigned long symndx = ELF64_R_SYM (look->r_info); asection *sec; if (symndx < symtab_hdr->sh_info) { Elf_Internal_Sym *sym; sym = (Elf_Internal_Sym *) symtab_hdr->contents; if (sym == NULL) { sym = bfd_elf_get_elf_syms (opd_bfd, symtab_hdr, symtab_hdr->sh_info, 0, NULL, NULL, NULL); if (sym == NULL) break; symtab_hdr->contents = (bfd_byte *) sym; } sym += symndx; val = sym->st_value; sec = NULL; if ((sym->st_shndx != SHN_UNDEF && sym->st_shndx < SHN_LORESERVE) || sym->st_shndx > SHN_HIRESERVE) sec = bfd_section_from_elf_index (opd_bfd, sym->st_shndx); BFD_ASSERT ((sec->flags & SEC_MERGE) == 0); } else { struct elf_link_hash_entry **sym_hashes; struct elf_link_hash_entry *rh; sym_hashes = elf_sym_hashes (opd_bfd); rh = sym_hashes[symndx - symtab_hdr->sh_info]; while (rh->root.type == bfd_link_hash_indirect || rh->root.type == bfd_link_hash_warning) rh = ((struct elf_link_hash_entry *) rh->root.u.i.link); BFD_ASSERT (rh->root.type == bfd_link_hash_defined || rh->root.type == bfd_link_hash_defweak); val = rh->root.u.def.value; sec = rh->root.u.def.section; } val += look->r_addend; if (code_off != NULL) *code_off = val; if (code_sec != NULL) *code_sec = sec; if (sec != NULL && sec->output_section != NULL) val += sec->output_section->vma + sec->output_offset; } break; } } return val; } /* Mark sections containing dynamically referenced symbols. When building shared libraries, we must assume that any visible symbol is referenced. */ static bfd_boolean ppc64_elf_gc_mark_dynamic_ref (struct elf_link_hash_entry *h, void *inf) { struct bfd_link_info *info = (struct bfd_link_info *) inf; struct ppc_link_hash_entry *eh = (struct ppc_link_hash_entry *) h; if (eh->elf.root.type == bfd_link_hash_warning) eh = (struct ppc_link_hash_entry *) eh->elf.root.u.i.link; /* Dynamic linking info is on the func descriptor sym. */ if (eh->oh != NULL && eh->oh->is_func_descriptor && (eh->oh->elf.root.type == bfd_link_hash_defined || eh->oh->elf.root.type == bfd_link_hash_defweak)) eh = eh->oh; if ((eh->elf.root.type == bfd_link_hash_defined || eh->elf.root.type == bfd_link_hash_defweak) && (eh->elf.ref_dynamic || (!info->executable && eh->elf.def_regular && ELF_ST_VISIBILITY (eh->elf.other) != STV_INTERNAL && ELF_ST_VISIBILITY (eh->elf.other) != STV_HIDDEN))) { asection *code_sec; eh->elf.root.u.def.section->flags |= SEC_KEEP; /* Function descriptor syms cause the associated function code sym section to be marked. */ if (eh->is_func_descriptor && (eh->oh->elf.root.type == bfd_link_hash_defined || eh->oh->elf.root.type == bfd_link_hash_defweak)) eh->oh->elf.root.u.def.section->flags |= SEC_KEEP; else if (get_opd_info (eh->elf.root.u.def.section) != NULL && opd_entry_value (eh->elf.root.u.def.section, eh->elf.root.u.def.value, &code_sec, NULL) != (bfd_vma) -1) code_sec->flags |= SEC_KEEP; } return TRUE; } /* Return the section that should be marked against GC for a given relocation. */ static asection * ppc64_elf_gc_mark_hook (asection *sec, struct bfd_link_info *info, Elf_Internal_Rela *rel, struct elf_link_hash_entry *h, Elf_Internal_Sym *sym) { asection *rsec; /* First mark all our entry sym sections. */ if (info->gc_sym_list != NULL) { struct ppc_link_hash_table *htab = ppc_hash_table (info); struct bfd_sym_chain *sym = info->gc_sym_list; info->gc_sym_list = NULL; for (; sym != NULL; sym = sym->next) { struct ppc_link_hash_entry *eh; eh = (struct ppc_link_hash_entry *) elf_link_hash_lookup (&htab->elf, sym->name, FALSE, FALSE, FALSE); if (eh == NULL) continue; if (eh->elf.root.type != bfd_link_hash_defined && eh->elf.root.type != bfd_link_hash_defweak) continue; if (eh->is_func_descriptor && (eh->oh->elf.root.type == bfd_link_hash_defined || eh->oh->elf.root.type == bfd_link_hash_defweak)) rsec = eh->oh->elf.root.u.def.section; else if (get_opd_info (eh->elf.root.u.def.section) != NULL && opd_entry_value (eh->elf.root.u.def.section, eh->elf.root.u.def.value, &rsec, NULL) != (bfd_vma) -1) ; else continue; if (!rsec->gc_mark) _bfd_elf_gc_mark (info, rsec, ppc64_elf_gc_mark_hook); rsec = eh->elf.root.u.def.section; if (!rsec->gc_mark) _bfd_elf_gc_mark (info, rsec, ppc64_elf_gc_mark_hook); } } /* Syms return NULL if we're marking .opd, so we avoid marking all function sections, as all functions are referenced in .opd. */ rsec = NULL; if (get_opd_info (sec) != NULL) return rsec; if (h != NULL) { enum elf_ppc64_reloc_type r_type; struct ppc_link_hash_entry *eh; r_type = ELF64_R_TYPE (rel->r_info); switch (r_type) { case R_PPC64_GNU_VTINHERIT: case R_PPC64_GNU_VTENTRY: break; default: switch (h->root.type) { case bfd_link_hash_defined: case bfd_link_hash_defweak: eh = (struct ppc_link_hash_entry *) h; if (eh->oh != NULL && eh->oh->is_func_descriptor && (eh->oh->elf.root.type == bfd_link_hash_defined || eh->oh->elf.root.type == bfd_link_hash_defweak)) eh = eh->oh; /* Function descriptor syms cause the associated function code sym section to be marked. */ if (eh->is_func_descriptor && (eh->oh->elf.root.type == bfd_link_hash_defined || eh->oh->elf.root.type == bfd_link_hash_defweak)) { /* They also mark their opd section. */ if (!eh->elf.root.u.def.section->gc_mark) _bfd_elf_gc_mark (info, eh->elf.root.u.def.section, ppc64_elf_gc_mark_hook); rsec = eh->oh->elf.root.u.def.section; } else if (get_opd_info (eh->elf.root.u.def.section) != NULL && opd_entry_value (eh->elf.root.u.def.section, eh->elf.root.u.def.value, &rsec, NULL) != (bfd_vma) -1) { if (!eh->elf.root.u.def.section->gc_mark) _bfd_elf_gc_mark (info, eh->elf.root.u.def.section, ppc64_elf_gc_mark_hook); } else rsec = h->root.u.def.section; break; case bfd_link_hash_common: rsec = h->root.u.c.p->section; break; default: break; } } } else { asection **opd_sym_section; rsec = bfd_section_from_elf_index (sec->owner, sym->st_shndx); opd_sym_section = get_opd_info (rsec); if (opd_sym_section != NULL) { if (!rsec->gc_mark) _bfd_elf_gc_mark (info, rsec, ppc64_elf_gc_mark_hook); rsec = opd_sym_section[(sym->st_value + rel->r_addend) / 8]; } } return rsec; } /* Update the .got, .plt. and dynamic reloc reference counts for the section being removed. */ static bfd_boolean ppc64_elf_gc_sweep_hook (bfd *abfd, struct bfd_link_info *info, asection *sec, const Elf_Internal_Rela *relocs) { struct ppc_link_hash_table *htab; Elf_Internal_Shdr *symtab_hdr; struct elf_link_hash_entry **sym_hashes; struct got_entry **local_got_ents; const Elf_Internal_Rela *rel, *relend; if ((sec->flags & SEC_ALLOC) == 0) return TRUE; elf_section_data (sec)->local_dynrel = NULL; htab = ppc_hash_table (info); symtab_hdr = &elf_tdata (abfd)->symtab_hdr; sym_hashes = elf_sym_hashes (abfd); local_got_ents = elf_local_got_ents (abfd); relend = relocs + sec->reloc_count; for (rel = relocs; rel < relend; rel++) { unsigned long r_symndx; enum elf_ppc64_reloc_type r_type; struct elf_link_hash_entry *h = NULL; char tls_type = 0; r_symndx = ELF64_R_SYM (rel->r_info); r_type = ELF64_R_TYPE (rel->r_info); if (r_symndx >= symtab_hdr->sh_info) { struct ppc_link_hash_entry *eh; struct ppc_dyn_relocs **pp; struct ppc_dyn_relocs *p; h = sym_hashes[r_symndx - symtab_hdr->sh_info]; while (h->root.type == bfd_link_hash_indirect || h->root.type == bfd_link_hash_warning) h = (struct elf_link_hash_entry *) h->root.u.i.link; eh = (struct ppc_link_hash_entry *) h; for (pp = &eh->dyn_relocs; (p = *pp) != NULL; pp = &p->next) if (p->sec == sec) { /* Everything must go for SEC. */ *pp = p->next; break; } } switch (r_type) { case R_PPC64_GOT_TLSLD16: case R_PPC64_GOT_TLSLD16_LO: case R_PPC64_GOT_TLSLD16_HI: case R_PPC64_GOT_TLSLD16_HA: - ppc64_tlsld_got (abfd)->refcount -= 1; tls_type = TLS_TLS | TLS_LD; goto dogot; case R_PPC64_GOT_TLSGD16: case R_PPC64_GOT_TLSGD16_LO: case R_PPC64_GOT_TLSGD16_HI: case R_PPC64_GOT_TLSGD16_HA: tls_type = TLS_TLS | TLS_GD; goto dogot; case R_PPC64_GOT_TPREL16_DS: case R_PPC64_GOT_TPREL16_LO_DS: case R_PPC64_GOT_TPREL16_HI: case R_PPC64_GOT_TPREL16_HA: tls_type = TLS_TLS | TLS_TPREL; goto dogot; case R_PPC64_GOT_DTPREL16_DS: case R_PPC64_GOT_DTPREL16_LO_DS: case R_PPC64_GOT_DTPREL16_HI: case R_PPC64_GOT_DTPREL16_HA: tls_type = TLS_TLS | TLS_DTPREL; goto dogot; case R_PPC64_GOT16: case R_PPC64_GOT16_DS: case R_PPC64_GOT16_HA: case R_PPC64_GOT16_HI: case R_PPC64_GOT16_LO: case R_PPC64_GOT16_LO_DS: dogot: { struct got_entry *ent; if (h != NULL) ent = h->got.glist; else ent = local_got_ents[r_symndx]; for (; ent != NULL; ent = ent->next) if (ent->addend == rel->r_addend && ent->owner == abfd && ent->tls_type == tls_type) break; if (ent == NULL) abort (); if (ent->got.refcount > 0) ent->got.refcount -= 1; } break; case R_PPC64_PLT16_HA: case R_PPC64_PLT16_HI: case R_PPC64_PLT16_LO: case R_PPC64_PLT32: case R_PPC64_PLT64: case R_PPC64_REL14: case R_PPC64_REL14_BRNTAKEN: case R_PPC64_REL14_BRTAKEN: case R_PPC64_REL24: if (h != NULL) { struct plt_entry *ent; for (ent = h->plt.plist; ent != NULL; ent = ent->next) if (ent->addend == rel->r_addend) break; if (ent != NULL && ent->plt.refcount > 0) ent->plt.refcount -= 1; } break; default: break; } } return TRUE; } /* The maximum size of .sfpr. */ #define SFPR_MAX (218*4) struct sfpr_def_parms { const char name[12]; unsigned char lo, hi; bfd_byte * (*write_ent) (bfd *, bfd_byte *, int); bfd_byte * (*write_tail) (bfd *, bfd_byte *, int); }; /* Auto-generate _save*, _rest* functions in .sfpr. */ static unsigned int sfpr_define (struct bfd_link_info *info, const struct sfpr_def_parms *parm) { struct ppc_link_hash_table *htab = ppc_hash_table (info); unsigned int i; size_t len = strlen (parm->name); bfd_boolean writing = FALSE; char sym[16]; memcpy (sym, parm->name, len); sym[len + 2] = 0; for (i = parm->lo; i <= parm->hi; i++) { struct elf_link_hash_entry *h; sym[len + 0] = i / 10 + '0'; sym[len + 1] = i % 10 + '0'; h = elf_link_hash_lookup (&htab->elf, sym, FALSE, FALSE, TRUE); if (h != NULL && !h->def_regular) { h->root.type = bfd_link_hash_defined; h->root.u.def.section = htab->sfpr; h->root.u.def.value = htab->sfpr->size; h->type = STT_FUNC; h->def_regular = 1; _bfd_elf_link_hash_hide_symbol (info, h, TRUE); writing = TRUE; if (htab->sfpr->contents == NULL) { htab->sfpr->contents = bfd_alloc (htab->elf.dynobj, SFPR_MAX); if (htab->sfpr->contents == NULL) return FALSE; } } if (writing) { bfd_byte *p = htab->sfpr->contents + htab->sfpr->size; if (i != parm->hi) p = (*parm->write_ent) (htab->elf.dynobj, p, i); else p = (*parm->write_tail) (htab->elf.dynobj, p, i); htab->sfpr->size = p - htab->sfpr->contents; } } return TRUE; } static bfd_byte * savegpr0 (bfd *abfd, bfd_byte *p, int r) { bfd_put_32 (abfd, STD_R0_0R1 + (r << 21) + (1 << 16) - (32 - r) * 8, p); return p + 4; } static bfd_byte * savegpr0_tail (bfd *abfd, bfd_byte *p, int r) { p = savegpr0 (abfd, p, r); bfd_put_32 (abfd, STD_R0_0R1 + 16, p); p = p + 4; bfd_put_32 (abfd, BLR, p); return p + 4; } static bfd_byte * restgpr0 (bfd *abfd, bfd_byte *p, int r) { bfd_put_32 (abfd, LD_R0_0R1 + (r << 21) + (1 << 16) - (32 - r) * 8, p); return p + 4; } static bfd_byte * restgpr0_tail (bfd *abfd, bfd_byte *p, int r) { bfd_put_32 (abfd, LD_R0_0R1 + 16, p); p = p + 4; p = restgpr0 (abfd, p, r); bfd_put_32 (abfd, MTLR_R0, p); p = p + 4; if (r == 29) { p = restgpr0 (abfd, p, 30); p = restgpr0 (abfd, p, 31); } bfd_put_32 (abfd, BLR, p); return p + 4; } static bfd_byte * savegpr1 (bfd *abfd, bfd_byte *p, int r) { bfd_put_32 (abfd, STD_R0_0R12 + (r << 21) + (1 << 16) - (32 - r) * 8, p); return p + 4; } static bfd_byte * savegpr1_tail (bfd *abfd, bfd_byte *p, int r) { p = savegpr1 (abfd, p, r); bfd_put_32 (abfd, BLR, p); return p + 4; } static bfd_byte * restgpr1 (bfd *abfd, bfd_byte *p, int r) { bfd_put_32 (abfd, LD_R0_0R12 + (r << 21) + (1 << 16) - (32 - r) * 8, p); return p + 4; } static bfd_byte * restgpr1_tail (bfd *abfd, bfd_byte *p, int r) { p = restgpr1 (abfd, p, r); bfd_put_32 (abfd, BLR, p); return p + 4; } static bfd_byte * savefpr (bfd *abfd, bfd_byte *p, int r) { bfd_put_32 (abfd, STFD_FR0_0R1 + (r << 21) + (1 << 16) - (32 - r) * 8, p); return p + 4; } static bfd_byte * savefpr0_tail (bfd *abfd, bfd_byte *p, int r) { p = savefpr (abfd, p, r); bfd_put_32 (abfd, STD_R0_0R1 + 16, p); p = p + 4; bfd_put_32 (abfd, BLR, p); return p + 4; } static bfd_byte * restfpr (bfd *abfd, bfd_byte *p, int r) { bfd_put_32 (abfd, LFD_FR0_0R1 + (r << 21) + (1 << 16) - (32 - r) * 8, p); return p + 4; } static bfd_byte * restfpr0_tail (bfd *abfd, bfd_byte *p, int r) { bfd_put_32 (abfd, LD_R0_0R1 + 16, p); p = p + 4; p = restfpr (abfd, p, r); bfd_put_32 (abfd, MTLR_R0, p); p = p + 4; if (r == 29) { p = restfpr (abfd, p, 30); p = restfpr (abfd, p, 31); } bfd_put_32 (abfd, BLR, p); return p + 4; } static bfd_byte * savefpr1_tail (bfd *abfd, bfd_byte *p, int r) { p = savefpr (abfd, p, r); bfd_put_32 (abfd, BLR, p); return p + 4; } static bfd_byte * restfpr1_tail (bfd *abfd, bfd_byte *p, int r) { p = restfpr (abfd, p, r); bfd_put_32 (abfd, BLR, p); return p + 4; } static bfd_byte * savevr (bfd *abfd, bfd_byte *p, int r) { bfd_put_32 (abfd, LI_R12_0 + (1 << 16) - (32 - r) * 16, p); p = p + 4; bfd_put_32 (abfd, STVX_VR0_R12_R0 + (r << 21), p); return p + 4; } static bfd_byte * savevr_tail (bfd *abfd, bfd_byte *p, int r) { p = savevr (abfd, p, r); bfd_put_32 (abfd, BLR, p); return p + 4; } static bfd_byte * restvr (bfd *abfd, bfd_byte *p, int r) { bfd_put_32 (abfd, LI_R12_0 + (1 << 16) - (32 - r) * 16, p); p = p + 4; bfd_put_32 (abfd, LVX_VR0_R12_R0 + (r << 21), p); return p + 4; } static bfd_byte * restvr_tail (bfd *abfd, bfd_byte *p, int r) { p = restvr (abfd, p, r); bfd_put_32 (abfd, BLR, p); return p + 4; } /* Called via elf_link_hash_traverse to transfer dynamic linking information on function code symbol entries to their corresponding function descriptor symbol entries. */ static bfd_boolean func_desc_adjust (struct elf_link_hash_entry *h, void *inf) { struct bfd_link_info *info; struct ppc_link_hash_table *htab; struct plt_entry *ent; struct ppc_link_hash_entry *fh; struct ppc_link_hash_entry *fdh; bfd_boolean force_local; fh = (struct ppc_link_hash_entry *) h; if (fh->elf.root.type == bfd_link_hash_indirect) return TRUE; if (fh->elf.root.type == bfd_link_hash_warning) fh = (struct ppc_link_hash_entry *) fh->elf.root.u.i.link; info = inf; htab = ppc_hash_table (info); /* Resolve undefined references to dot-symbols as the value in the function descriptor, if we have one in a regular object. This is to satisfy cases like ".quad .foo". Calls to functions in dynamic objects are handled elsewhere. */ if (fh->elf.root.type == bfd_link_hash_undefweak && fh->was_undefined && (fh->oh->elf.root.type == bfd_link_hash_defined || fh->oh->elf.root.type == bfd_link_hash_defweak) && get_opd_info (fh->oh->elf.root.u.def.section) != NULL && opd_entry_value (fh->oh->elf.root.u.def.section, fh->oh->elf.root.u.def.value, &fh->elf.root.u.def.section, &fh->elf.root.u.def.value) != (bfd_vma) -1) { fh->elf.root.type = fh->oh->elf.root.type; fh->elf.forced_local = 1; + fh->elf.def_regular = fh->oh->elf.def_regular; + fh->elf.def_dynamic = fh->oh->elf.def_dynamic; } /* If this is a function code symbol, transfer dynamic linking information to the function descriptor symbol. */ if (!fh->is_func) return TRUE; for (ent = fh->elf.plt.plist; ent != NULL; ent = ent->next) if (ent->plt.refcount > 0) break; if (ent == NULL || fh->elf.root.root.string[0] != '.' || fh->elf.root.root.string[1] == '\0') return TRUE; /* Find the corresponding function descriptor symbol. Create it as undefined if necessary. */ fdh = get_fdh (fh, htab); if (fdh != NULL) while (fdh->elf.root.type == bfd_link_hash_indirect || fdh->elf.root.type == bfd_link_hash_warning) fdh = (struct ppc_link_hash_entry *) fdh->elf.root.u.i.link; if (fdh == NULL && info->shared && (fh->elf.root.type == bfd_link_hash_undefined || fh->elf.root.type == bfd_link_hash_undefweak)) { fdh = make_fdh (info, fh); if (fdh == NULL) return FALSE; } /* Fake function descriptors are made undefweak. If the function code symbol is strong undefined, make the fake sym the same. If the function code symbol is defined, then force the fake descriptor local; We can't support overriding of symbols in a shared library on a fake descriptor. */ if (fdh != NULL && fdh->fake && fdh->elf.root.type == bfd_link_hash_undefweak) { if (fh->elf.root.type == bfd_link_hash_undefined) { fdh->elf.root.type = bfd_link_hash_undefined; bfd_link_add_undef (&htab->elf.root, &fdh->elf.root); } else if (fh->elf.root.type == bfd_link_hash_defined || fh->elf.root.type == bfd_link_hash_defweak) { _bfd_elf_link_hash_hide_symbol (info, &fdh->elf, TRUE); } } if (fdh != NULL && !fdh->elf.forced_local && (info->shared || fdh->elf.def_dynamic || fdh->elf.ref_dynamic || (fdh->elf.root.type == bfd_link_hash_undefweak && ELF_ST_VISIBILITY (fdh->elf.other) == STV_DEFAULT))) { if (fdh->elf.dynindx == -1) if (! bfd_elf_link_record_dynamic_symbol (info, &fdh->elf)) return FALSE; fdh->elf.ref_regular |= fh->elf.ref_regular; fdh->elf.ref_dynamic |= fh->elf.ref_dynamic; fdh->elf.ref_regular_nonweak |= fh->elf.ref_regular_nonweak; fdh->elf.non_got_ref |= fh->elf.non_got_ref; if (ELF_ST_VISIBILITY (fh->elf.other) == STV_DEFAULT) { move_plt_plist (fh, fdh); fdh->elf.needs_plt = 1; } fdh->is_func_descriptor = 1; fdh->oh = fh; fh->oh = fdh; } /* Now that the info is on the function descriptor, clear the function code sym info. Any function code syms for which we don't have a definition in a regular file, we force local. This prevents a shared library from exporting syms that have been imported from another library. Function code syms that are really in the library we must leave global to prevent the linker dragging in a definition from a static library. */ force_local = (!fh->elf.def_regular || fdh == NULL || !fdh->elf.def_regular || fdh->elf.forced_local); _bfd_elf_link_hash_hide_symbol (info, &fh->elf, force_local); return TRUE; } /* Called near the start of bfd_elf_size_dynamic_sections. We use this hook to a) provide some gcc support functions, and b) transfer dynamic linking information gathered so far on function code symbol entries, to their corresponding function descriptor symbol entries. */ static bfd_boolean ppc64_elf_func_desc_adjust (bfd *obfd ATTRIBUTE_UNUSED, struct bfd_link_info *info) { struct ppc_link_hash_table *htab; unsigned int i; const struct sfpr_def_parms funcs[] = { { "_savegpr0_", 14, 31, savegpr0, savegpr0_tail }, { "_restgpr0_", 14, 29, restgpr0, restgpr0_tail }, { "_restgpr0_", 30, 31, restgpr0, restgpr0_tail }, { "_savegpr1_", 14, 31, savegpr1, savegpr1_tail }, { "_restgpr1_", 14, 31, restgpr1, restgpr1_tail }, { "_savefpr_", 14, 31, savefpr, savefpr0_tail }, { "_restfpr_", 14, 29, restfpr, restfpr0_tail }, { "_restfpr_", 30, 31, restfpr, restfpr0_tail }, { "._savef", 14, 31, savefpr, savefpr1_tail }, { "._restf", 14, 31, restfpr, restfpr1_tail }, { "_savevr_", 20, 31, savevr, savevr_tail }, { "_restvr_", 20, 31, restvr, restvr_tail } }; htab = ppc_hash_table (info); if (htab->sfpr == NULL) /* We don't have any relocs. */ return TRUE; /* Provide any missing _save* and _rest* functions. */ htab->sfpr->size = 0; for (i = 0; i < sizeof (funcs) / sizeof (funcs[0]); i++) if (!sfpr_define (info, &funcs[i])) return FALSE; elf_link_hash_traverse (&htab->elf, func_desc_adjust, info); if (htab->sfpr->size == 0) htab->sfpr->flags |= SEC_EXCLUDE; return TRUE; } /* Adjust a symbol defined by a dynamic object and referenced by a regular object. The current definition is in some section of the dynamic object, but we're not including those sections. We have to change the definition to something the rest of the link can understand. */ static bfd_boolean ppc64_elf_adjust_dynamic_symbol (struct bfd_link_info *info, struct elf_link_hash_entry *h) { struct ppc_link_hash_table *htab; asection *s; htab = ppc_hash_table (info); /* Deal with function syms. */ if (h->type == STT_FUNC || h->needs_plt) { /* Clear procedure linkage table information for any symbol that won't need a .plt entry. */ struct plt_entry *ent; for (ent = h->plt.plist; ent != NULL; ent = ent->next) if (ent->plt.refcount > 0) break; if (ent == NULL || SYMBOL_CALLS_LOCAL (info, h) || (ELF_ST_VISIBILITY (h->other) != STV_DEFAULT && h->root.type == bfd_link_hash_undefweak)) { h->plt.plist = NULL; h->needs_plt = 0; } } else h->plt.plist = NULL; /* If this is a weak symbol, and there is a real definition, the processor independent code will have arranged for us to see the real definition first, and we can just use the same value. */ if (h->u.weakdef != NULL) { BFD_ASSERT (h->u.weakdef->root.type == bfd_link_hash_defined || h->u.weakdef->root.type == bfd_link_hash_defweak); h->root.u.def.section = h->u.weakdef->root.u.def.section; h->root.u.def.value = h->u.weakdef->root.u.def.value; if (ELIMINATE_COPY_RELOCS) h->non_got_ref = h->u.weakdef->non_got_ref; return TRUE; } /* If we are creating a shared library, we must presume that the only references to the symbol are via the global offset table. For such cases we need not do anything here; the relocations will be handled correctly by relocate_section. */ if (info->shared) return TRUE; /* If there are no references to this symbol that do not use the GOT, we don't need to generate a copy reloc. */ if (!h->non_got_ref) return TRUE; /* Don't generate a copy reloc for symbols defined in the executable. */ if (!h->def_dynamic || !h->ref_regular || h->def_regular) return TRUE; if (ELIMINATE_COPY_RELOCS) { struct ppc_link_hash_entry * eh; struct ppc_dyn_relocs *p; eh = (struct ppc_link_hash_entry *) h; for (p = eh->dyn_relocs; p != NULL; p = p->next) { s = p->sec->output_section; if (s != NULL && (s->flags & SEC_READONLY) != 0) break; } /* If we didn't find any dynamic relocs in read-only sections, then we'll be keeping the dynamic relocs and avoiding the copy reloc. */ if (p == NULL) { h->non_got_ref = 0; return TRUE; } } if (h->plt.plist != NULL) { /* We should never get here, but unfortunately there are versions of gcc out there that improperly (for this ABI) put initialized function pointers, vtable refs and suchlike in read-only sections. Allow them to proceed, but warn that this might break at runtime. */ (*_bfd_error_handler) (_("copy reloc against `%s' requires lazy plt linking; " "avoid setting LD_BIND_NOW=1 or upgrade gcc"), h->root.root.string); } /* This is a reference to a symbol defined by a dynamic object which is not a function. */ if (h->size == 0) { (*_bfd_error_handler) (_("dynamic variable `%s' is zero size"), h->root.root.string); return TRUE; } /* We must allocate the symbol in our .dynbss section, which will become part of the .bss section of the executable. There will be an entry for this symbol in the .dynsym section. The dynamic object will contain position independent code, so all references from the dynamic object to this symbol will go through the global offset table. The dynamic linker will use the .dynsym entry to determine the address it must put in the global offset table, so both the dynamic object and the regular object will refer to the same memory location for the variable. */ /* We must generate a R_PPC64_COPY reloc to tell the dynamic linker to copy the initial value out of the dynamic object and into the runtime process image. We need to remember the offset into the .rela.bss section we are going to use. */ if ((h->root.u.def.section->flags & SEC_ALLOC) != 0) { htab->relbss->size += sizeof (Elf64_External_Rela); h->needs_copy = 1; } s = htab->dynbss; return _bfd_elf_adjust_dynamic_copy (h, s); } /* If given a function descriptor symbol, hide both the function code sym and the descriptor. */ static void ppc64_elf_hide_symbol (struct bfd_link_info *info, struct elf_link_hash_entry *h, bfd_boolean force_local) { struct ppc_link_hash_entry *eh; _bfd_elf_link_hash_hide_symbol (info, h, force_local); eh = (struct ppc_link_hash_entry *) h; if (eh->is_func_descriptor) { struct ppc_link_hash_entry *fh = eh->oh; if (fh == NULL) { const char *p, *q; struct ppc_link_hash_table *htab; char save; /* We aren't supposed to use alloca in BFD because on systems which do not have alloca the version in libiberty calls xmalloc, which might cause the program to crash when it runs out of memory. This function doesn't have a return status, so there's no way to gracefully return an error. So cheat. We know that string[-1] can be safely accessed; It's either a string in an ELF string table, or allocated in an objalloc structure. */ p = eh->elf.root.root.string - 1; save = *p; *(char *) p = '.'; htab = ppc_hash_table (info); fh = (struct ppc_link_hash_entry *) elf_link_hash_lookup (&htab->elf, p, FALSE, FALSE, FALSE); *(char *) p = save; /* Unfortunately, if it so happens that the string we were looking for was allocated immediately before this string, then we overwrote the string terminator. That's the only reason the lookup should fail. */ if (fh == NULL) { q = eh->elf.root.root.string + strlen (eh->elf.root.root.string); while (q >= eh->elf.root.root.string && *q == *p) --q, --p; if (q < eh->elf.root.root.string && *p == '.') fh = (struct ppc_link_hash_entry *) elf_link_hash_lookup (&htab->elf, p, FALSE, FALSE, FALSE); } if (fh != NULL) { eh->oh = fh; fh->oh = eh; } } if (fh != NULL) _bfd_elf_link_hash_hide_symbol (info, &fh->elf, force_local); } } static bfd_boolean get_sym_h (struct elf_link_hash_entry **hp, Elf_Internal_Sym **symp, asection **symsecp, char **tls_maskp, Elf_Internal_Sym **locsymsp, unsigned long r_symndx, bfd *ibfd) { Elf_Internal_Shdr *symtab_hdr = &elf_tdata (ibfd)->symtab_hdr; if (r_symndx >= symtab_hdr->sh_info) { struct elf_link_hash_entry **sym_hashes = elf_sym_hashes (ibfd); struct elf_link_hash_entry *h; h = sym_hashes[r_symndx - symtab_hdr->sh_info]; while (h->root.type == bfd_link_hash_indirect || h->root.type == bfd_link_hash_warning) h = (struct elf_link_hash_entry *) h->root.u.i.link; if (hp != NULL) *hp = h; if (symp != NULL) *symp = NULL; if (symsecp != NULL) { asection *symsec = NULL; if (h->root.type == bfd_link_hash_defined || h->root.type == bfd_link_hash_defweak) symsec = h->root.u.def.section; *symsecp = symsec; } if (tls_maskp != NULL) { struct ppc_link_hash_entry *eh; eh = (struct ppc_link_hash_entry *) h; *tls_maskp = &eh->tls_mask; } } else { Elf_Internal_Sym *sym; Elf_Internal_Sym *locsyms = *locsymsp; if (locsyms == NULL) { locsyms = (Elf_Internal_Sym *) symtab_hdr->contents; if (locsyms == NULL) locsyms = bfd_elf_get_elf_syms (ibfd, symtab_hdr, symtab_hdr->sh_info, 0, NULL, NULL, NULL); if (locsyms == NULL) return FALSE; *locsymsp = locsyms; } sym = locsyms + r_symndx; if (hp != NULL) *hp = NULL; if (symp != NULL) *symp = sym; if (symsecp != NULL) { asection *symsec = NULL; if ((sym->st_shndx != SHN_UNDEF && sym->st_shndx < SHN_LORESERVE) || sym->st_shndx > SHN_HIRESERVE) symsec = bfd_section_from_elf_index (ibfd, sym->st_shndx); *symsecp = symsec; } if (tls_maskp != NULL) { struct got_entry **lgot_ents; char *tls_mask; tls_mask = NULL; lgot_ents = elf_local_got_ents (ibfd); if (lgot_ents != NULL) { char *lgot_masks = (char *) (lgot_ents + symtab_hdr->sh_info); tls_mask = &lgot_masks[r_symndx]; } *tls_maskp = tls_mask; } } return TRUE; } /* Returns TLS_MASKP for the given REL symbol. Function return is 0 on error, 2 on a toc GD type suitable for optimization, 3 on a toc LD type suitable for optimization, and 1 otherwise. */ static int -get_tls_mask (char **tls_maskp, unsigned long *toc_symndx, +get_tls_mask (char **tls_maskp, + unsigned long *toc_symndx, + bfd_vma *toc_addend, Elf_Internal_Sym **locsymsp, - const Elf_Internal_Rela *rel, bfd *ibfd) + const Elf_Internal_Rela *rel, + bfd *ibfd) { unsigned long r_symndx; int next_r; struct elf_link_hash_entry *h; Elf_Internal_Sym *sym; asection *sec; bfd_vma off; r_symndx = ELF64_R_SYM (rel->r_info); if (!get_sym_h (&h, &sym, &sec, tls_maskp, locsymsp, r_symndx, ibfd)) return 0; if ((*tls_maskp != NULL && **tls_maskp != 0) || sec == NULL || ppc64_elf_section_data (sec)->sec_type != sec_toc) return 1; /* Look inside a TOC section too. */ if (h != NULL) { BFD_ASSERT (h->root.type == bfd_link_hash_defined); off = h->root.u.def.value; } else off = sym->st_value; off += rel->r_addend; BFD_ASSERT (off % 8 == 0); - r_symndx = ppc64_elf_section_data (sec)->u.t_symndx[off / 8]; - next_r = ppc64_elf_section_data (sec)->u.t_symndx[off / 8 + 1]; - if (!get_sym_h (&h, &sym, &sec, tls_maskp, locsymsp, r_symndx, ibfd)) - return 0; + r_symndx = ppc64_elf_section_data (sec)->u.toc.symndx[off / 8]; + next_r = ppc64_elf_section_data (sec)->u.toc.symndx[off / 8 + 1]; if (toc_symndx != NULL) *toc_symndx = r_symndx; + if (toc_addend != NULL) + *toc_addend = ppc64_elf_section_data (sec)->u.toc.add[off / 8]; + if (!get_sym_h (&h, &sym, &sec, tls_maskp, locsymsp, r_symndx, ibfd)) + return 0; if ((h == NULL || ((h->root.type == bfd_link_hash_defined || h->root.type == bfd_link_hash_defweak) && !h->def_dynamic)) && (next_r == -1 || next_r == -2)) return 1 - next_r; return 1; } /* Adjust all global syms defined in opd sections. In gcc generated code for the old ABI, these will already have been done. */ static bfd_boolean adjust_opd_syms (struct elf_link_hash_entry *h, void *inf ATTRIBUTE_UNUSED) { struct ppc_link_hash_entry *eh; asection *sym_sec; long *opd_adjust; if (h->root.type == bfd_link_hash_indirect) return TRUE; if (h->root.type == bfd_link_hash_warning) h = (struct elf_link_hash_entry *) h->root.u.i.link; if (h->root.type != bfd_link_hash_defined && h->root.type != bfd_link_hash_defweak) return TRUE; eh = (struct ppc_link_hash_entry *) h; if (eh->adjust_done) return TRUE; sym_sec = eh->elf.root.u.def.section; opd_adjust = get_opd_info (sym_sec); if (opd_adjust != NULL) { long adjust = opd_adjust[eh->elf.root.u.def.value / 8]; if (adjust == -1) { /* This entry has been deleted. */ asection *dsec = ppc64_elf_tdata (sym_sec->owner)->deleted_section; if (dsec == NULL) { for (dsec = sym_sec->owner->sections; dsec; dsec = dsec->next) if (elf_discarded_section (dsec)) { ppc64_elf_tdata (sym_sec->owner)->deleted_section = dsec; break; } } eh->elf.root.u.def.value = 0; eh->elf.root.u.def.section = dsec; } else eh->elf.root.u.def.value += adjust; eh->adjust_done = 1; } return TRUE; } /* Handles decrementing dynamic reloc counts for the reloc specified by R_INFO in section SEC. If LOCAL_SYMS is NULL, then H and SYM_SEC have already been determined. */ static bfd_boolean dec_dynrel_count (bfd_vma r_info, asection *sec, struct bfd_link_info *info, Elf_Internal_Sym **local_syms, struct elf_link_hash_entry *h, asection *sym_sec) { enum elf_ppc64_reloc_type r_type; struct ppc_dyn_relocs *p; struct ppc_dyn_relocs **pp; /* Can this reloc be dynamic? This switch, and later tests here should be kept in sync with the code in check_relocs. */ r_type = ELF64_R_TYPE (r_info); switch (r_type) { default: return TRUE; case R_PPC64_TPREL16: case R_PPC64_TPREL16_LO: case R_PPC64_TPREL16_HI: case R_PPC64_TPREL16_HA: case R_PPC64_TPREL16_DS: case R_PPC64_TPREL16_LO_DS: case R_PPC64_TPREL16_HIGHER: case R_PPC64_TPREL16_HIGHERA: case R_PPC64_TPREL16_HIGHEST: case R_PPC64_TPREL16_HIGHESTA: if (!info->shared) return TRUE; case R_PPC64_TPREL64: case R_PPC64_DTPMOD64: case R_PPC64_DTPREL64: case R_PPC64_ADDR64: case R_PPC64_REL30: case R_PPC64_REL32: case R_PPC64_REL64: case R_PPC64_ADDR14: case R_PPC64_ADDR14_BRNTAKEN: case R_PPC64_ADDR14_BRTAKEN: case R_PPC64_ADDR16: case R_PPC64_ADDR16_DS: case R_PPC64_ADDR16_HA: case R_PPC64_ADDR16_HI: case R_PPC64_ADDR16_HIGHER: case R_PPC64_ADDR16_HIGHERA: case R_PPC64_ADDR16_HIGHEST: case R_PPC64_ADDR16_HIGHESTA: case R_PPC64_ADDR16_LO: case R_PPC64_ADDR16_LO_DS: case R_PPC64_ADDR24: case R_PPC64_ADDR32: case R_PPC64_UADDR16: case R_PPC64_UADDR32: case R_PPC64_UADDR64: case R_PPC64_TOC: break; } if (local_syms != NULL) { unsigned long r_symndx; Elf_Internal_Sym *sym; bfd *ibfd = sec->owner; r_symndx = ELF64_R_SYM (r_info); if (!get_sym_h (&h, &sym, &sym_sec, NULL, local_syms, r_symndx, ibfd)) return FALSE; } if ((info->shared - && (MUST_BE_DYN_RELOC (r_type) + && (must_be_dyn_reloc (info, r_type) || (h != NULL && (!info->symbolic || h->root.type == bfd_link_hash_defweak || !h->def_regular)))) || (ELIMINATE_COPY_RELOCS && !info->shared && h != NULL && (h->root.type == bfd_link_hash_defweak || !h->def_regular))) ; else return TRUE; if (h != NULL) pp = &((struct ppc_link_hash_entry *) h)->dyn_relocs; else { if (sym_sec != NULL) { void *vpp = &elf_section_data (sym_sec)->local_dynrel; pp = (struct ppc_dyn_relocs **) vpp; } else { void *vpp = &elf_section_data (sec)->local_dynrel; pp = (struct ppc_dyn_relocs **) vpp; } /* elf_gc_sweep may have already removed all dyn relocs associated with local syms for a given section. Don't report a dynreloc miscount. */ if (*pp == NULL) return TRUE; } while ((p = *pp) != NULL) { if (p->sec == sec) { - if (!MUST_BE_DYN_RELOC (r_type)) + if (!must_be_dyn_reloc (info, r_type)) p->pc_count -= 1; p->count -= 1; if (p->count == 0) *pp = p->next; return TRUE; } pp = &p->next; } (*_bfd_error_handler) (_("dynreloc miscount for %B, section %A"), sec->owner, sec); bfd_set_error (bfd_error_bad_value); return FALSE; } /* Remove unused Official Procedure Descriptor entries. Currently we only remove those associated with functions in discarded link-once sections, or weakly defined functions that have been overridden. It would be possible to remove many more entries for statically linked applications. */ bfd_boolean ppc64_elf_edit_opd (bfd *obfd, struct bfd_link_info *info, bfd_boolean no_opd_opt, bfd_boolean non_overlapping) { bfd *ibfd; bfd_boolean some_edited = FALSE; asection *need_pad = NULL; for (ibfd = info->input_bfds; ibfd != NULL; ibfd = ibfd->link_next) { asection *sec; Elf_Internal_Rela *relstart, *rel, *relend; Elf_Internal_Shdr *symtab_hdr; Elf_Internal_Sym *local_syms; struct elf_link_hash_entry **sym_hashes; bfd_vma offset; bfd_size_type amt; long *opd_adjust; bfd_boolean need_edit, add_aux_fields; bfd_size_type cnt_16b = 0; sec = bfd_get_section_by_name (ibfd, ".opd"); if (sec == NULL || sec->size == 0) continue; amt = sec->size * sizeof (long) / 8; opd_adjust = get_opd_info (sec); if (opd_adjust == NULL) { /* check_relocs hasn't been called. Must be a ld -r link or --just-symbols object. */ opd_adjust = bfd_alloc (obfd, amt); if (opd_adjust == NULL) return FALSE; ppc64_elf_section_data (sec)->u.opd_adjust = opd_adjust; BFD_ASSERT (ppc64_elf_section_data (sec)->sec_type == sec_normal); ppc64_elf_section_data (sec)->sec_type = sec_opd; } memset (opd_adjust, 0, amt); if (no_opd_opt) continue; if (sec->sec_info_type == ELF_INFO_TYPE_JUST_SYMS) continue; if (sec->output_section == bfd_abs_section_ptr) continue; /* Look through the section relocs. */ if ((sec->flags & SEC_RELOC) == 0 || sec->reloc_count == 0) continue; local_syms = NULL; symtab_hdr = &elf_tdata (ibfd)->symtab_hdr; sym_hashes = elf_sym_hashes (ibfd); /* Read the relocations. */ relstart = _bfd_elf_link_read_relocs (ibfd, sec, NULL, NULL, info->keep_memory); if (relstart == NULL) return FALSE; /* First run through the relocs to check they are sane, and to determine whether we need to edit this opd section. */ need_edit = FALSE; need_pad = sec; offset = 0; relend = relstart + sec->reloc_count; for (rel = relstart; rel < relend; ) { enum elf_ppc64_reloc_type r_type; unsigned long r_symndx; asection *sym_sec; struct elf_link_hash_entry *h; Elf_Internal_Sym *sym; /* .opd contains a regular array of 16 or 24 byte entries. We're only interested in the reloc pointing to a function entry point. */ if (rel->r_offset != offset || rel + 1 >= relend || (rel + 1)->r_offset != offset + 8) { /* If someone messes with .opd alignment then after a "ld -r" we might have padding in the middle of .opd. Also, there's nothing to prevent someone putting something silly in .opd with the assembler. No .opd optimization for them! */ broken_opd: (*_bfd_error_handler) (_("%B: .opd is not a regular array of opd entries"), ibfd); need_edit = FALSE; break; } if ((r_type = ELF64_R_TYPE (rel->r_info)) != R_PPC64_ADDR64 || (r_type = ELF64_R_TYPE ((rel + 1)->r_info)) != R_PPC64_TOC) { (*_bfd_error_handler) (_("%B: unexpected reloc type %u in .opd section"), ibfd, r_type); need_edit = FALSE; break; } r_symndx = ELF64_R_SYM (rel->r_info); if (!get_sym_h (&h, &sym, &sym_sec, NULL, &local_syms, r_symndx, ibfd)) goto error_ret; if (sym_sec == NULL || sym_sec->owner == NULL) { const char *sym_name; if (h != NULL) sym_name = h->root.root.string; else sym_name = bfd_elf_sym_name (ibfd, symtab_hdr, sym, sym_sec); (*_bfd_error_handler) (_("%B: undefined sym `%s' in .opd section"), ibfd, sym_name); need_edit = FALSE; break; } /* opd entries are always for functions defined in the current input bfd. If the symbol isn't defined in the input bfd, then we won't be using the function in this bfd; It must be defined in a linkonce section in another bfd, or is weak. It's also possible that we are discarding the function due to a linker script /DISCARD/, which we test for via the output_section. */ if (sym_sec->owner != ibfd || sym_sec->output_section == bfd_abs_section_ptr) need_edit = TRUE; rel += 2; if (rel == relend || (rel + 1 == relend && rel->r_offset == offset + 16)) { if (sec->size == offset + 24) { need_pad = NULL; break; } if (rel == relend && sec->size == offset + 16) { cnt_16b++; break; } goto broken_opd; } if (rel->r_offset == offset + 24) offset += 24; else if (rel->r_offset != offset + 16) goto broken_opd; else if (rel + 1 < relend && ELF64_R_TYPE (rel[0].r_info) == R_PPC64_ADDR64 && ELF64_R_TYPE (rel[1].r_info) == R_PPC64_TOC) { offset += 16; cnt_16b++; } else if (rel + 2 < relend && ELF64_R_TYPE (rel[1].r_info) == R_PPC64_ADDR64 && ELF64_R_TYPE (rel[2].r_info) == R_PPC64_TOC) { offset += 24; rel += 1; } else goto broken_opd; } add_aux_fields = non_overlapping && cnt_16b > 0; if (need_edit || add_aux_fields) { Elf_Internal_Rela *write_rel; bfd_byte *rptr, *wptr; bfd_byte *new_contents = NULL; bfd_boolean skip; long opd_ent_size; /* This seems a waste of time as input .opd sections are all zeros as generated by gcc, but I suppose there's no reason this will always be so. We might start putting something in the third word of .opd entries. */ if ((sec->flags & SEC_IN_MEMORY) == 0) { bfd_byte *loc; if (!bfd_malloc_and_get_section (ibfd, sec, &loc)) { if (loc != NULL) free (loc); error_ret: if (local_syms != NULL && symtab_hdr->contents != (unsigned char *) local_syms) free (local_syms); if (elf_section_data (sec)->relocs != relstart) free (relstart); return FALSE; } sec->contents = loc; sec->flags |= (SEC_IN_MEMORY | SEC_HAS_CONTENTS); } elf_section_data (sec)->relocs = relstart; new_contents = sec->contents; if (add_aux_fields) { new_contents = bfd_malloc (sec->size + cnt_16b * 8); if (new_contents == NULL) return FALSE; need_pad = FALSE; } wptr = new_contents; rptr = sec->contents; write_rel = relstart; skip = FALSE; offset = 0; opd_ent_size = 0; for (rel = relstart; rel < relend; rel++) { unsigned long r_symndx; asection *sym_sec; struct elf_link_hash_entry *h; Elf_Internal_Sym *sym; r_symndx = ELF64_R_SYM (rel->r_info); if (!get_sym_h (&h, &sym, &sym_sec, NULL, &local_syms, r_symndx, ibfd)) goto error_ret; if (rel->r_offset == offset) { struct ppc_link_hash_entry *fdh = NULL; /* See if the .opd entry is full 24 byte or 16 byte (with fd_aux entry overlapped with next fd_func). */ opd_ent_size = 24; if ((rel + 2 == relend && sec->size == offset + 16) || (rel + 3 < relend && rel[2].r_offset == offset + 16 && rel[3].r_offset == offset + 24 && ELF64_R_TYPE (rel[2].r_info) == R_PPC64_ADDR64 && ELF64_R_TYPE (rel[3].r_info) == R_PPC64_TOC)) opd_ent_size = 16; if (h != NULL && h->root.root.string[0] == '.') { fdh = get_fdh ((struct ppc_link_hash_entry *) h, ppc_hash_table (info)); if (fdh != NULL && fdh->elf.root.type != bfd_link_hash_defined && fdh->elf.root.type != bfd_link_hash_defweak) fdh = NULL; } skip = (sym_sec->owner != ibfd || sym_sec->output_section == bfd_abs_section_ptr); if (skip) { if (fdh != NULL && sym_sec->owner == ibfd) { /* Arrange for the function descriptor sym to be dropped. */ fdh->elf.root.u.def.value = 0; fdh->elf.root.u.def.section = sym_sec; } opd_adjust[rel->r_offset / 8] = -1; } else { /* We'll be keeping this opd entry. */ if (fdh != NULL) { /* Redefine the function descriptor symbol to this location in the opd section. It is necessary to update the value here rather than using an array of adjustments as we do for local symbols, because various places in the generic ELF code use the value stored in u.def.value. */ fdh->elf.root.u.def.value = wptr - new_contents; fdh->adjust_done = 1; } /* Local syms are a bit tricky. We could tweak them as they can be cached, but we'd need to look through the local syms for the function descriptor sym which we don't have at the moment. So keep an array of adjustments. */ opd_adjust[rel->r_offset / 8] = (wptr - new_contents) - (rptr - sec->contents); if (wptr != rptr) memcpy (wptr, rptr, opd_ent_size); wptr += opd_ent_size; if (add_aux_fields && opd_ent_size == 16) { memset (wptr, '\0', 8); wptr += 8; } } rptr += opd_ent_size; offset += opd_ent_size; } if (skip) { if (!NO_OPD_RELOCS && !info->relocatable && !dec_dynrel_count (rel->r_info, sec, info, NULL, h, sym_sec)) goto error_ret; } else { /* We need to adjust any reloc offsets to point to the new opd entries. While we're at it, we may as well remove redundant relocs. */ rel->r_offset += opd_adjust[(offset - opd_ent_size) / 8]; if (write_rel != rel) memcpy (write_rel, rel, sizeof (*rel)); ++write_rel; } } sec->size = wptr - new_contents; sec->reloc_count = write_rel - relstart; if (add_aux_fields) { free (sec->contents); sec->contents = new_contents; } /* Fudge the header size too, as this is used later in elf_bfd_final_link if we are emitting relocs. */ elf_section_data (sec)->rel_hdr.sh_size = sec->reloc_count * elf_section_data (sec)->rel_hdr.sh_entsize; BFD_ASSERT (elf_section_data (sec)->rel_hdr2 == NULL); some_edited = TRUE; } else if (elf_section_data (sec)->relocs != relstart) free (relstart); if (local_syms != NULL && symtab_hdr->contents != (unsigned char *) local_syms) { if (!info->keep_memory) free (local_syms); else symtab_hdr->contents = (unsigned char *) local_syms; } } if (some_edited) elf_link_hash_traverse (elf_hash_table (info), adjust_opd_syms, NULL); /* If we are doing a final link and the last .opd entry is just 16 byte long, add a 8 byte padding after it. */ if (need_pad != NULL && !info->relocatable) { bfd_byte *p; if ((need_pad->flags & SEC_IN_MEMORY) == 0) { BFD_ASSERT (need_pad->size > 0); p = bfd_malloc (need_pad->size + 8); if (p == NULL) return FALSE; if (! bfd_get_section_contents (need_pad->owner, need_pad, p, 0, need_pad->size)) return FALSE; need_pad->contents = p; need_pad->flags |= (SEC_IN_MEMORY | SEC_HAS_CONTENTS); } else { p = bfd_realloc (need_pad->contents, need_pad->size + 8); if (p == NULL) return FALSE; need_pad->contents = p; } memset (need_pad->contents + need_pad->size, 0, 8); need_pad->size += 8; } return TRUE; } /* Set htab->tls_get_addr and call the generic ELF tls_setup function. */ asection * ppc64_elf_tls_setup (bfd *obfd, struct bfd_link_info *info) { struct ppc_link_hash_table *htab; htab = ppc_hash_table (info); - if (htab->tls_get_addr != NULL) - { - struct ppc_link_hash_entry *h = htab->tls_get_addr; + htab->tls_get_addr = ((struct ppc_link_hash_entry *) + elf_link_hash_lookup (&htab->elf, ".__tls_get_addr", + FALSE, FALSE, TRUE)); + htab->tls_get_addr_fd = ((struct ppc_link_hash_entry *) + elf_link_hash_lookup (&htab->elf, "__tls_get_addr", + FALSE, FALSE, TRUE)); + return _bfd_elf_tls_setup (obfd, info); +} - while (h->elf.root.type == bfd_link_hash_indirect - || h->elf.root.type == bfd_link_hash_warning) - h = (struct ppc_link_hash_entry *) h->elf.root.u.i.link; +/* Return TRUE iff REL is a branch reloc with a global symbol matching + HASH1 or HASH2. */ - htab->tls_get_addr = h; +static bfd_boolean +branch_reloc_hash_match (const bfd *ibfd, + const Elf_Internal_Rela *rel, + const struct ppc_link_hash_entry *hash1, + const struct ppc_link_hash_entry *hash2) +{ + Elf_Internal_Shdr *symtab_hdr = &elf_tdata (ibfd)->symtab_hdr; + enum elf_ppc64_reloc_type r_type = ELF64_R_TYPE (rel->r_info); + unsigned int r_symndx = ELF64_R_SYM (rel->r_info); - if (htab->tls_get_addr_fd == NULL - && h->oh != NULL - && h->oh->is_func_descriptor - && (h->oh->elf.root.type == bfd_link_hash_defined - || h->oh->elf.root.type == bfd_link_hash_defweak)) - htab->tls_get_addr_fd = h->oh; - } - - if (htab->tls_get_addr_fd != NULL) + if (r_symndx >= symtab_hdr->sh_info + && (r_type == R_PPC64_REL24 + || r_type == R_PPC64_REL14 + || r_type == R_PPC64_REL14_BRTAKEN + || r_type == R_PPC64_REL14_BRNTAKEN + || r_type == R_PPC64_ADDR24 + || r_type == R_PPC64_ADDR14 + || r_type == R_PPC64_ADDR14_BRTAKEN + || r_type == R_PPC64_ADDR14_BRNTAKEN)) { - struct ppc_link_hash_entry *h = htab->tls_get_addr_fd; + struct elf_link_hash_entry **sym_hashes = elf_sym_hashes (ibfd); + struct elf_link_hash_entry *h; - while (h->elf.root.type == bfd_link_hash_indirect - || h->elf.root.type == bfd_link_hash_warning) - h = (struct ppc_link_hash_entry *) h->elf.root.u.i.link; - - htab->tls_get_addr_fd = h; + h = sym_hashes[r_symndx - symtab_hdr->sh_info]; + while (h->root.type == bfd_link_hash_indirect + || h->root.type == bfd_link_hash_warning) + h = (struct elf_link_hash_entry *) h->root.u.i.link; + if (h == &hash1->elf || h == &hash2->elf) + return TRUE; } - - return _bfd_elf_tls_setup (obfd, info); + return FALSE; } /* Run through all the TLS relocs looking for optimization opportunities. The linker has been hacked (see ppc64elf.em) to do a preliminary section layout so that we know the TLS segment offsets. We can't optimize earlier because some optimizations need to know the tp offset, and we need to optimize before allocating dynamic relocations. */ bfd_boolean ppc64_elf_tls_optimize (bfd *obfd ATTRIBUTE_UNUSED, struct bfd_link_info *info) { bfd *ibfd; asection *sec; struct ppc_link_hash_table *htab; + int pass; - if (info->relocatable || info->shared) + if (info->relocatable || !info->executable) return TRUE; htab = ppc_hash_table (info); for (ibfd = info->input_bfds; ibfd != NULL; ibfd = ibfd->link_next) { Elf_Internal_Sym *locsyms = NULL; asection *toc = bfd_get_section_by_name (ibfd, ".toc"); unsigned char *toc_ref = NULL; - /* Look at all the sections for this file, with TOC last. */ - for (sec = (ibfd->sections == toc && toc && toc->next ? toc->next - : ibfd->sections); - sec != NULL; - sec = (sec == toc ? NULL - : sec->next == NULL ? toc - : sec->next == toc && toc->next ? toc->next - : sec->next)) - if (sec->has_tls_reloc && !bfd_is_abs_section (sec->output_section)) - { - Elf_Internal_Rela *relstart, *rel, *relend; - int expecting_tls_get_addr; - long toc_ref_index = 0; + /* Look at all the sections for this file. Make two passes over + the relocs. On the first pass, mark toc entries involved + with tls relocs, and check that tls relocs involved in + setting up a tls_get_addr call are indeed followed by such a + call. If they are not, exclude them from the optimizations + done on the second pass. */ + for (pass = 0; pass < 2; ++pass) + for (sec = ibfd->sections; sec != NULL; sec = sec->next) + if (sec->has_tls_reloc && !bfd_is_abs_section (sec->output_section)) + { + Elf_Internal_Rela *relstart, *rel, *relend; - /* Read the relocations. */ - relstart = _bfd_elf_link_read_relocs (ibfd, sec, NULL, NULL, - info->keep_memory); - if (relstart == NULL) - return FALSE; + /* Read the relocations. */ + relstart = _bfd_elf_link_read_relocs (ibfd, sec, NULL, NULL, + info->keep_memory); + if (relstart == NULL) + return FALSE; - expecting_tls_get_addr = 0; - relend = relstart + sec->reloc_count; - for (rel = relstart; rel < relend; rel++) - { - enum elf_ppc64_reloc_type r_type; - unsigned long r_symndx; - struct elf_link_hash_entry *h; - Elf_Internal_Sym *sym; - asection *sym_sec; - char *tls_mask; - char tls_set, tls_clear, tls_type = 0; - bfd_vma value; - bfd_boolean ok_tprel, is_local; + relend = relstart + sec->reloc_count; + for (rel = relstart; rel < relend; rel++) + { + enum elf_ppc64_reloc_type r_type; + unsigned long r_symndx; + struct elf_link_hash_entry *h; + Elf_Internal_Sym *sym; + asection *sym_sec; + char *tls_mask; + char tls_set, tls_clear, tls_type = 0; + bfd_vma value; + bfd_boolean ok_tprel, is_local; + long toc_ref_index = 0; + int expecting_tls_get_addr = 0; - r_symndx = ELF64_R_SYM (rel->r_info); - if (!get_sym_h (&h, &sym, &sym_sec, &tls_mask, &locsyms, - r_symndx, ibfd)) - { - err_free_rel: - if (elf_section_data (sec)->relocs != relstart) - free (relstart); - if (toc_ref != NULL) - free (toc_ref); - if (locsyms != NULL - && (elf_tdata (ibfd)->symtab_hdr.contents - != (unsigned char *) locsyms)) - free (locsyms); - return FALSE; - } + r_symndx = ELF64_R_SYM (rel->r_info); + if (!get_sym_h (&h, &sym, &sym_sec, &tls_mask, &locsyms, + r_symndx, ibfd)) + { + err_free_rel: + if (elf_section_data (sec)->relocs != relstart) + free (relstart); + if (toc_ref != NULL) + free (toc_ref); + if (locsyms != NULL + && (elf_tdata (ibfd)->symtab_hdr.contents + != (unsigned char *) locsyms)) + free (locsyms); + return FALSE; + } - if (h != NULL) - { - if (h->root.type != bfd_link_hash_defined - && h->root.type != bfd_link_hash_defweak) - continue; - value = h->root.u.def.value; - } - else - /* Symbols referenced by TLS relocs must be of type - STT_TLS. So no need for .opd local sym adjust. */ - value = sym->st_value; + if (h != NULL) + { + if (h->root.type != bfd_link_hash_defined + && h->root.type != bfd_link_hash_defweak) + continue; + value = h->root.u.def.value; + } + else + /* Symbols referenced by TLS relocs must be of type + STT_TLS. So no need for .opd local sym adjust. */ + value = sym->st_value; - ok_tprel = FALSE; - is_local = FALSE; - if (h == NULL - || !h->def_dynamic) - { - is_local = TRUE; - value += sym_sec->output_offset; - value += sym_sec->output_section->vma; - value -= htab->elf.tls_sec->vma; - ok_tprel = (value + TP_OFFSET + ((bfd_vma) 1 << 31) - < (bfd_vma) 1 << 32); - } + ok_tprel = FALSE; + is_local = FALSE; + if (h == NULL + || !h->def_dynamic) + { + is_local = TRUE; + value += sym_sec->output_offset; + value += sym_sec->output_section->vma; + value -= htab->elf.tls_sec->vma; + ok_tprel = (value + TP_OFFSET + ((bfd_vma) 1 << 31) + < (bfd_vma) 1 << 32); + } - r_type = ELF64_R_TYPE (rel->r_info); - switch (r_type) - { - case R_PPC64_GOT_TLSLD16: - case R_PPC64_GOT_TLSLD16_LO: - case R_PPC64_GOT_TLSLD16_HI: - case R_PPC64_GOT_TLSLD16_HA: - /* These relocs should never be against a symbol - defined in a shared lib. Leave them alone if - that turns out to be the case. */ - ppc64_tlsld_got (ibfd)->refcount -= 1; - if (!is_local) - continue; + r_type = ELF64_R_TYPE (rel->r_info); + switch (r_type) + { + case R_PPC64_GOT_TLSLD16: + case R_PPC64_GOT_TLSLD16_LO: + expecting_tls_get_addr = 1; + /* Fall thru */ - /* LD -> LE */ - tls_set = 0; - tls_clear = TLS_LD; - tls_type = TLS_TLS | TLS_LD; - expecting_tls_get_addr = 1; - break; + case R_PPC64_GOT_TLSLD16_HI: + case R_PPC64_GOT_TLSLD16_HA: + /* These relocs should never be against a symbol + defined in a shared lib. Leave them alone if + that turns out to be the case. */ + if (!is_local) + continue; - case R_PPC64_GOT_TLSGD16: - case R_PPC64_GOT_TLSGD16_LO: - case R_PPC64_GOT_TLSGD16_HI: - case R_PPC64_GOT_TLSGD16_HA: - if (ok_tprel) - /* GD -> LE */ + /* LD -> LE */ tls_set = 0; - else - /* GD -> IE */ - tls_set = TLS_TLS | TLS_TPRELGD; - tls_clear = TLS_GD; - tls_type = TLS_TLS | TLS_GD; - expecting_tls_get_addr = 1; - break; + tls_clear = TLS_LD; + tls_type = TLS_TLS | TLS_LD; + break; - case R_PPC64_GOT_TPREL16_DS: - case R_PPC64_GOT_TPREL16_LO_DS: - case R_PPC64_GOT_TPREL16_HI: - case R_PPC64_GOT_TPREL16_HA: - expecting_tls_get_addr = 0; - if (ok_tprel) - { - /* IE -> LE */ + case R_PPC64_GOT_TLSGD16: + case R_PPC64_GOT_TLSGD16_LO: + expecting_tls_get_addr = 1; + /* Fall thru */ + + case R_PPC64_GOT_TLSGD16_HI: + case R_PPC64_GOT_TLSGD16_HA: + if (ok_tprel) + /* GD -> LE */ tls_set = 0; - tls_clear = TLS_TPREL; - tls_type = TLS_TLS | TLS_TPREL; - break; - } - else + else + /* GD -> IE */ + tls_set = TLS_TLS | TLS_TPRELGD; + tls_clear = TLS_GD; + tls_type = TLS_TLS | TLS_GD; + break; + + case R_PPC64_GOT_TPREL16_DS: + case R_PPC64_GOT_TPREL16_LO_DS: + case R_PPC64_GOT_TPREL16_HI: + case R_PPC64_GOT_TPREL16_HA: + if (ok_tprel) + { + /* IE -> LE */ + tls_set = 0; + tls_clear = TLS_TPREL; + tls_type = TLS_TLS | TLS_TPREL; + break; + } continue; - case R_PPC64_REL14: - case R_PPC64_REL14_BRTAKEN: - case R_PPC64_REL14_BRNTAKEN: - case R_PPC64_REL24: - if (h != NULL - && (h == &htab->tls_get_addr->elf - || h == &htab->tls_get_addr_fd->elf)) - { - if (!expecting_tls_get_addr - && rel != relstart - && ((ELF64_R_TYPE (rel[-1].r_info) - == R_PPC64_TOC16) - || (ELF64_R_TYPE (rel[-1].r_info) - == R_PPC64_TOC16_LO))) - { - /* Check for toc tls entries. */ - char *toc_tls; - int retval; + case R_PPC64_TOC16: + case R_PPC64_TOC16_LO: + case R_PPC64_TLS: + case R_PPC64_TLSGD: + case R_PPC64_TLSLD: + if (sym_sec == NULL || sym_sec != toc) + continue; - retval = get_tls_mask (&toc_tls, NULL, &locsyms, - rel - 1, ibfd); - if (retval == 0) - goto err_free_rel; - if (retval > 1 && toc_tls != NULL) - { - expecting_tls_get_addr = 1; - if (toc_ref != NULL) - toc_ref[toc_ref_index] = 1; - } - } + /* Mark this toc entry as referenced by a TLS + code sequence. We can do that now in the + case of R_PPC64_TLS, and after checking for + tls_get_addr for the TOC16 relocs. */ + if (toc_ref == NULL) + { + toc_ref = bfd_zmalloc (toc->size / 8); + if (toc_ref == NULL) + goto err_free_rel; + } + if (h != NULL) + value = h->root.u.def.value; + else + value = sym->st_value; + value += rel->r_addend; + BFD_ASSERT (value < toc->size && value % 8 == 0); + toc_ref_index = value / 8; + if (r_type == R_PPC64_TLS + || r_type == R_PPC64_TLSGD + || r_type == R_PPC64_TLSLD) + { + toc_ref[toc_ref_index] = 1; + continue; + } - if (expecting_tls_get_addr) - { - struct plt_entry *ent; - for (ent = h->plt.plist; ent; ent = ent->next) - if (ent->addend == 0) - { - if (ent->plt.refcount > 0) - ent->plt.refcount -= 1; - break; - } - } - } - expecting_tls_get_addr = 0; - continue; + if (pass != 0 && toc_ref[toc_ref_index] == 0) + continue; - case R_PPC64_TOC16: - case R_PPC64_TOC16_LO: - case R_PPC64_TLS: - expecting_tls_get_addr = 0; - if (sym_sec == toc && toc != NULL) - { - /* Mark this toc entry as referenced by a TLS - code sequence. We can do that now in the - case of R_PPC64_TLS, and after checking for - tls_get_addr for the TOC16 relocs. */ - if (toc_ref == NULL) - { - toc_ref = bfd_zmalloc (toc->size / 8); - if (toc_ref == NULL) - goto err_free_rel; - } - if (h != NULL) - value = h->root.u.def.value; - else - value = sym->st_value; - value += rel->r_addend; - BFD_ASSERT (value < toc->size && value % 8 == 0); - toc_ref_index = value / 8; - if (r_type == R_PPC64_TLS) - toc_ref[toc_ref_index] = 1; - } - continue; + tls_set = 0; + tls_clear = 0; + expecting_tls_get_addr = 2; + break; - case R_PPC64_TPREL64: - expecting_tls_get_addr = 0; - if (sec != toc - || toc_ref == NULL - || !toc_ref[rel->r_offset / 8]) + case R_PPC64_TPREL64: + if (pass == 0 + || sec != toc + || toc_ref == NULL + || !toc_ref[rel->r_offset / 8]) + continue; + if (ok_tprel) + { + /* IE -> LE */ + tls_set = TLS_EXPLICIT; + tls_clear = TLS_TPREL; + break; + } continue; - if (ok_tprel) - { - /* IE -> LE */ - tls_set = TLS_EXPLICIT; - tls_clear = TLS_TPREL; - break; - } - else - continue; - case R_PPC64_DTPMOD64: - expecting_tls_get_addr = 0; - if (sec != toc - || toc_ref == NULL - || !toc_ref[rel->r_offset / 8]) + case R_PPC64_DTPMOD64: + if (pass == 0 + || sec != toc + || toc_ref == NULL + || !toc_ref[rel->r_offset / 8]) + continue; + if (rel + 1 < relend + && (rel[1].r_info + == ELF64_R_INFO (r_symndx, R_PPC64_DTPREL64)) + && rel[1].r_offset == rel->r_offset + 8) + { + if (ok_tprel) + /* GD -> LE */ + tls_set = TLS_EXPLICIT | TLS_GD; + else + /* GD -> IE */ + tls_set = TLS_EXPLICIT | TLS_GD | TLS_TPRELGD; + tls_clear = TLS_GD; + } + else + { + if (!is_local) + continue; + + /* LD -> LE */ + tls_set = TLS_EXPLICIT; + tls_clear = TLS_LD; + } + break; + + default: continue; - if (rel + 1 < relend - && (rel[1].r_info - == ELF64_R_INFO (r_symndx, R_PPC64_DTPREL64)) - && rel[1].r_offset == rel->r_offset + 8) - { - if (ok_tprel) - /* GD -> LE */ - tls_set = TLS_EXPLICIT | TLS_GD; - else - /* GD -> IE */ - tls_set = TLS_EXPLICIT | TLS_GD | TLS_TPRELGD; - tls_clear = TLS_GD; - } - else - { - if (!is_local) + } + + if (pass == 0) + { + if (!expecting_tls_get_addr + || !sec->has_tls_get_addr_call) + continue; + + if (rel + 1 < relend + && branch_reloc_hash_match (ibfd, rel + 1, + htab->tls_get_addr, + htab->tls_get_addr_fd)) + { + if (expecting_tls_get_addr == 2) + { + /* Check for toc tls entries. */ + char *toc_tls; + int retval; + + retval = get_tls_mask (&toc_tls, NULL, NULL, + &locsyms, + rel, ibfd); + if (retval == 0) + goto err_free_rel; + if (retval > 1 && toc_tls != NULL) + toc_ref[toc_ref_index] = 1; + } continue; + } - /* LD -> LE */ - tls_set = TLS_EXPLICIT; - tls_clear = TLS_LD; - } - break; + if (expecting_tls_get_addr != 1) + continue; - default: - expecting_tls_get_addr = 0; + /* Uh oh, we didn't find the expected call. We + could just mark this symbol to exclude it + from tls optimization but it's safer to skip + the entire section. */ + sec->has_tls_reloc = 0; + break; + } + + if (expecting_tls_get_addr) + { + struct plt_entry *ent; + for (ent = htab->tls_get_addr->elf.plt.plist; + ent != NULL; + ent = ent->next) + if (ent->addend == 0) + { + if (ent->plt.refcount > 0) + { + ent->plt.refcount -= 1; + expecting_tls_get_addr = 0; + } + break; + } + } + + if (expecting_tls_get_addr) + { + struct plt_entry *ent; + for (ent = htab->tls_get_addr_fd->elf.plt.plist; + ent != NULL; + ent = ent->next) + if (ent->addend == 0) + { + if (ent->plt.refcount > 0) + ent->plt.refcount -= 1; + break; + } + } + + if (tls_clear == 0) continue; - } - if ((tls_set & TLS_EXPLICIT) == 0) - { - struct got_entry *ent; + if ((tls_set & TLS_EXPLICIT) == 0) + { + struct got_entry *ent; - /* Adjust got entry for this reloc. */ - if (h != NULL) - ent = h->got.glist; - else - ent = elf_local_got_ents (ibfd)[r_symndx]; + /* Adjust got entry for this reloc. */ + if (h != NULL) + ent = h->got.glist; + else + ent = elf_local_got_ents (ibfd)[r_symndx]; - for (; ent != NULL; ent = ent->next) - if (ent->addend == rel->r_addend - && ent->owner == ibfd - && ent->tls_type == tls_type) - break; - if (ent == NULL) - abort (); + for (; ent != NULL; ent = ent->next) + if (ent->addend == rel->r_addend + && ent->owner == ibfd + && ent->tls_type == tls_type) + break; + if (ent == NULL) + abort (); - if (tls_set == 0) - { - /* We managed to get rid of a got entry. */ - if (ent->got.refcount > 0) - ent->got.refcount -= 1; - } - } - else - { - /* If we got rid of a DTPMOD/DTPREL reloc pair then - we'll lose one or two dyn relocs. */ - if (!dec_dynrel_count (rel->r_info, sec, info, - NULL, h, sym_sec)) - return FALSE; + if (tls_set == 0) + { + /* We managed to get rid of a got entry. */ + if (ent->got.refcount > 0) + ent->got.refcount -= 1; + } + } + else + { + /* If we got rid of a DTPMOD/DTPREL reloc pair then + we'll lose one or two dyn relocs. */ + if (!dec_dynrel_count (rel->r_info, sec, info, + NULL, h, sym_sec)) + return FALSE; - if (tls_set == (TLS_EXPLICIT | TLS_GD)) - { - if (!dec_dynrel_count ((rel + 1)->r_info, sec, info, - NULL, h, sym_sec)) - return FALSE; - } - } + if (tls_set == (TLS_EXPLICIT | TLS_GD)) + { + if (!dec_dynrel_count ((rel + 1)->r_info, sec, info, + NULL, h, sym_sec)) + return FALSE; + } + } - *tls_mask |= tls_set; - *tls_mask &= ~tls_clear; - } + *tls_mask |= tls_set; + *tls_mask &= ~tls_clear; + } - if (elf_section_data (sec)->relocs != relstart) - free (relstart); - } + if (elf_section_data (sec)->relocs != relstart) + free (relstart); + } - if (toc_ref != NULL) - free (toc_ref); + if (toc_ref != NULL) + free (toc_ref); - if (locsyms != NULL - && (elf_tdata (ibfd)->symtab_hdr.contents - != (unsigned char *) locsyms)) - { - if (!info->keep_memory) - free (locsyms); - else - elf_tdata (ibfd)->symtab_hdr.contents = (unsigned char *) locsyms; - } - } + if (locsyms != NULL + && (elf_tdata (ibfd)->symtab_hdr.contents + != (unsigned char *) locsyms)) + { + if (!info->keep_memory) + free (locsyms); + else + elf_tdata (ibfd)->symtab_hdr.contents = (unsigned char *) locsyms; + } + } return TRUE; } /* Called via elf_link_hash_traverse from ppc64_elf_edit_toc to adjust the values of any global symbols in a toc section that has been edited. Globals in toc sections should be a rarity, so this function sets a flag if any are found in toc sections other than the one just edited, so that futher hash table traversals can be avoided. */ struct adjust_toc_info { asection *toc; unsigned long *skip; bfd_boolean global_toc_syms; }; static bfd_boolean adjust_toc_syms (struct elf_link_hash_entry *h, void *inf) { struct ppc_link_hash_entry *eh; struct adjust_toc_info *toc_inf = (struct adjust_toc_info *) inf; if (h->root.type == bfd_link_hash_indirect) return TRUE; if (h->root.type == bfd_link_hash_warning) h = (struct elf_link_hash_entry *) h->root.u.i.link; if (h->root.type != bfd_link_hash_defined && h->root.type != bfd_link_hash_defweak) return TRUE; eh = (struct ppc_link_hash_entry *) h; if (eh->adjust_done) return TRUE; if (eh->elf.root.u.def.section == toc_inf->toc) { unsigned long skip = toc_inf->skip[eh->elf.root.u.def.value >> 3]; if (skip != (unsigned long) -1) eh->elf.root.u.def.value -= skip; else { (*_bfd_error_handler) (_("%s defined in removed toc entry"), eh->elf.root.root.string); eh->elf.root.u.def.section = &bfd_abs_section; eh->elf.root.u.def.value = 0; } eh->adjust_done = 1; } else if (strcmp (eh->elf.root.u.def.section->name, ".toc") == 0) toc_inf->global_toc_syms = TRUE; return TRUE; } /* Examine all relocs referencing .toc sections in order to remove unused .toc entries. */ bfd_boolean ppc64_elf_edit_toc (bfd *obfd ATTRIBUTE_UNUSED, struct bfd_link_info *info) { bfd *ibfd; struct adjust_toc_info toc_inf; toc_inf.global_toc_syms = TRUE; for (ibfd = info->input_bfds; ibfd != NULL; ibfd = ibfd->link_next) { asection *toc, *sec; Elf_Internal_Shdr *symtab_hdr; Elf_Internal_Sym *local_syms; struct elf_link_hash_entry **sym_hashes; Elf_Internal_Rela *relstart, *rel; unsigned long *skip, *drop; unsigned char *used; unsigned char *keep, last, some_unused; toc = bfd_get_section_by_name (ibfd, ".toc"); if (toc == NULL || toc->size == 0 || toc->sec_info_type == ELF_INFO_TYPE_JUST_SYMS || elf_discarded_section (toc)) continue; local_syms = NULL; symtab_hdr = &elf_tdata (ibfd)->symtab_hdr; sym_hashes = elf_sym_hashes (ibfd); /* Look at sections dropped from the final link. */ skip = NULL; relstart = NULL; for (sec = ibfd->sections; sec != NULL; sec = sec->next) { if (sec->reloc_count == 0 || !elf_discarded_section (sec) || get_opd_info (sec) || (sec->flags & SEC_ALLOC) == 0 || (sec->flags & SEC_DEBUGGING) != 0) continue; relstart = _bfd_elf_link_read_relocs (ibfd, sec, NULL, NULL, FALSE); if (relstart == NULL) goto error_ret; /* Run through the relocs to see which toc entries might be unused. */ for (rel = relstart; rel < relstart + sec->reloc_count; ++rel) { enum elf_ppc64_reloc_type r_type; unsigned long r_symndx; asection *sym_sec; struct elf_link_hash_entry *h; Elf_Internal_Sym *sym; bfd_vma val; r_type = ELF64_R_TYPE (rel->r_info); switch (r_type) { default: continue; case R_PPC64_TOC16: case R_PPC64_TOC16_LO: case R_PPC64_TOC16_HI: case R_PPC64_TOC16_HA: case R_PPC64_TOC16_DS: case R_PPC64_TOC16_LO_DS: break; } r_symndx = ELF64_R_SYM (rel->r_info); if (!get_sym_h (&h, &sym, &sym_sec, NULL, &local_syms, r_symndx, ibfd)) goto error_ret; if (sym_sec != toc) continue; if (h != NULL) val = h->root.u.def.value; else val = sym->st_value; val += rel->r_addend; if (val >= toc->size) continue; /* Anything in the toc ought to be aligned to 8 bytes. If not, don't mark as unused. */ if (val & 7) continue; if (skip == NULL) { skip = bfd_zmalloc (sizeof (*skip) * (toc->size + 7) / 8); if (skip == NULL) goto error_ret; } skip[val >> 3] = 1; } if (elf_section_data (sec)->relocs != relstart) free (relstart); } if (skip == NULL) continue; used = bfd_zmalloc (sizeof (*used) * (toc->size + 7) / 8); if (used == NULL) { error_ret: if (local_syms != NULL && symtab_hdr->contents != (unsigned char *) local_syms) free (local_syms); if (sec != NULL && relstart != NULL && elf_section_data (sec)->relocs != relstart) free (relstart); if (skip != NULL) free (skip); return FALSE; } /* Now check all kept sections that might reference the toc. Check the toc itself last. */ for (sec = (ibfd->sections == toc && toc->next ? toc->next : ibfd->sections); sec != NULL; sec = (sec == toc ? NULL : sec->next == NULL ? toc : sec->next == toc && toc->next ? toc->next : sec->next)) { int repeat; if (sec->reloc_count == 0 || elf_discarded_section (sec) || get_opd_info (sec) || (sec->flags & SEC_ALLOC) == 0 || (sec->flags & SEC_DEBUGGING) != 0) continue; relstart = _bfd_elf_link_read_relocs (ibfd, sec, NULL, NULL, TRUE); if (relstart == NULL) goto error_ret; /* Mark toc entries referenced as used. */ repeat = 0; do for (rel = relstart; rel < relstart + sec->reloc_count; ++rel) { enum elf_ppc64_reloc_type r_type; unsigned long r_symndx; asection *sym_sec; struct elf_link_hash_entry *h; Elf_Internal_Sym *sym; bfd_vma val; r_type = ELF64_R_TYPE (rel->r_info); switch (r_type) { case R_PPC64_TOC16: case R_PPC64_TOC16_LO: case R_PPC64_TOC16_HI: case R_PPC64_TOC16_HA: case R_PPC64_TOC16_DS: case R_PPC64_TOC16_LO_DS: /* In case we're taking addresses of toc entries. */ case R_PPC64_ADDR64: break; default: continue; } r_symndx = ELF64_R_SYM (rel->r_info); if (!get_sym_h (&h, &sym, &sym_sec, NULL, &local_syms, r_symndx, ibfd)) { free (used); goto error_ret; } if (sym_sec != toc) continue; if (h != NULL) val = h->root.u.def.value; else val = sym->st_value; val += rel->r_addend; if (val >= toc->size) continue; /* For the toc section, we only mark as used if this entry itself isn't unused. */ if (sec == toc && !used[val >> 3] && (used[rel->r_offset >> 3] || !skip[rel->r_offset >> 3])) /* Do all the relocs again, to catch reference chains. */ repeat = 1; used[val >> 3] = 1; } while (repeat); } /* Merge the used and skip arrays. Assume that TOC doublewords not appearing as either used or unused belong to to an entry more than one doubleword in size. */ for (drop = skip, keep = used, last = 0, some_unused = 0; drop < skip + (toc->size + 7) / 8; ++drop, ++keep) { if (*keep) { *drop = 0; last = 0; } else if (*drop) { some_unused = 1; last = 1; } else *drop = last; } free (used); if (some_unused) { bfd_byte *contents, *src; unsigned long off; /* Shuffle the toc contents, and at the same time convert the skip array from booleans into offsets. */ if (!bfd_malloc_and_get_section (ibfd, toc, &contents)) goto error_ret; elf_section_data (toc)->this_hdr.contents = contents; for (src = contents, off = 0, drop = skip; src < contents + toc->size; src += 8, ++drop) { if (*drop) { *drop = (unsigned long) -1; off += 8; } else if (off != 0) { *drop = off; memcpy (src - off, src, 8); } } toc->rawsize = toc->size; toc->size = src - contents - off; if (toc->reloc_count != 0) { Elf_Internal_Rela *wrel; bfd_size_type sz; /* Read toc relocs. */ relstart = _bfd_elf_link_read_relocs (ibfd, toc, NULL, NULL, TRUE); if (relstart == NULL) goto error_ret; /* Remove unused toc relocs, and adjust those we keep. */ wrel = relstart; for (rel = relstart; rel < relstart + toc->reloc_count; ++rel) if (skip[rel->r_offset >> 3] != (unsigned long) -1) { wrel->r_offset = rel->r_offset - skip[rel->r_offset >> 3]; wrel->r_info = rel->r_info; wrel->r_addend = rel->r_addend; ++wrel; } else if (!dec_dynrel_count (rel->r_info, toc, info, &local_syms, NULL, NULL)) goto error_ret; toc->reloc_count = wrel - relstart; sz = elf_section_data (toc)->rel_hdr.sh_entsize; elf_section_data (toc)->rel_hdr.sh_size = toc->reloc_count * sz; BFD_ASSERT (elf_section_data (toc)->rel_hdr2 == NULL); } /* Adjust addends for relocs against the toc section sym. */ for (sec = ibfd->sections; sec != NULL; sec = sec->next) { if (sec->reloc_count == 0 || elf_discarded_section (sec)) continue; relstart = _bfd_elf_link_read_relocs (ibfd, sec, NULL, NULL, TRUE); if (relstart == NULL) goto error_ret; for (rel = relstart; rel < relstart + sec->reloc_count; ++rel) { enum elf_ppc64_reloc_type r_type; unsigned long r_symndx; asection *sym_sec; struct elf_link_hash_entry *h; Elf_Internal_Sym *sym; r_type = ELF64_R_TYPE (rel->r_info); switch (r_type) { default: continue; case R_PPC64_TOC16: case R_PPC64_TOC16_LO: case R_PPC64_TOC16_HI: case R_PPC64_TOC16_HA: case R_PPC64_TOC16_DS: case R_PPC64_TOC16_LO_DS: case R_PPC64_ADDR64: break; } r_symndx = ELF64_R_SYM (rel->r_info); if (!get_sym_h (&h, &sym, &sym_sec, NULL, &local_syms, r_symndx, ibfd)) goto error_ret; if (sym_sec != toc || h != NULL || sym->st_value != 0) continue; rel->r_addend -= skip[rel->r_addend >> 3]; } } /* We shouldn't have local or global symbols defined in the TOC, but handle them anyway. */ if (local_syms != NULL) { Elf_Internal_Sym *sym; for (sym = local_syms; sym < local_syms + symtab_hdr->sh_info; ++sym) if (sym->st_shndx != SHN_UNDEF && (sym->st_shndx < SHN_LORESERVE || sym->st_shndx > SHN_HIRESERVE) && sym->st_value != 0 && bfd_section_from_elf_index (ibfd, sym->st_shndx) == toc) { if (skip[sym->st_value >> 3] != (unsigned long) -1) sym->st_value -= skip[sym->st_value >> 3]; else { (*_bfd_error_handler) (_("%s defined in removed toc entry"), bfd_elf_sym_name (ibfd, symtab_hdr, sym, NULL)); sym->st_value = 0; sym->st_shndx = SHN_ABS; } symtab_hdr->contents = (unsigned char *) local_syms; } } /* Finally, adjust any global syms defined in the toc. */ if (toc_inf.global_toc_syms) { toc_inf.toc = toc; toc_inf.skip = skip; toc_inf.global_toc_syms = FALSE; elf_link_hash_traverse (elf_hash_table (info), adjust_toc_syms, &toc_inf); } } if (local_syms != NULL && symtab_hdr->contents != (unsigned char *) local_syms) { if (!info->keep_memory) free (local_syms); else symtab_hdr->contents = (unsigned char *) local_syms; } free (skip); } return TRUE; } /* Allocate space in .plt, .got and associated reloc sections for dynamic relocs. */ static bfd_boolean allocate_dynrelocs (struct elf_link_hash_entry *h, void *inf) { struct bfd_link_info *info; struct ppc_link_hash_table *htab; asection *s; struct ppc_link_hash_entry *eh; struct ppc_dyn_relocs *p; struct got_entry *gent; if (h->root.type == bfd_link_hash_indirect) return TRUE; if (h->root.type == bfd_link_hash_warning) h = (struct elf_link_hash_entry *) h->root.u.i.link; info = (struct bfd_link_info *) inf; htab = ppc_hash_table (info); if (htab->elf.dynamic_sections_created && h->dynindx != -1 && WILL_CALL_FINISH_DYNAMIC_SYMBOL (1, info->shared, h)) { struct plt_entry *pent; bfd_boolean doneone = FALSE; for (pent = h->plt.plist; pent != NULL; pent = pent->next) if (pent->plt.refcount > 0) { /* If this is the first .plt entry, make room for the special first entry. */ s = htab->plt; if (s->size == 0) s->size += PLT_INITIAL_ENTRY_SIZE; pent->plt.offset = s->size; /* Make room for this entry. */ s->size += PLT_ENTRY_SIZE; /* Make room for the .glink code. */ s = htab->glink; if (s->size == 0) s->size += GLINK_CALL_STUB_SIZE; /* We need bigger stubs past index 32767. */ if (s->size >= GLINK_CALL_STUB_SIZE + 32768*2*4) s->size += 4; s->size += 2*4; /* We also need to make an entry in the .rela.plt section. */ s = htab->relplt; s->size += sizeof (Elf64_External_Rela); doneone = TRUE; } else pent->plt.offset = (bfd_vma) -1; if (!doneone) { h->plt.plist = NULL; h->needs_plt = 0; } } else { h->plt.plist = NULL; h->needs_plt = 0; } eh = (struct ppc_link_hash_entry *) h; /* Run through the TLS GD got entries first if we're changing them to TPREL. */ if ((eh->tls_mask & TLS_TPRELGD) != 0) for (gent = h->got.glist; gent != NULL; gent = gent->next) if (gent->got.refcount > 0 && (gent->tls_type & TLS_GD) != 0) { /* This was a GD entry that has been converted to TPREL. If there happens to be a TPREL entry we can use that one. */ struct got_entry *ent; for (ent = h->got.glist; ent != NULL; ent = ent->next) if (ent->got.refcount > 0 && (ent->tls_type & TLS_TPREL) != 0 && ent->addend == gent->addend && ent->owner == gent->owner) { gent->got.refcount = 0; break; } /* If not, then we'll be using our own TPREL entry. */ if (gent->got.refcount != 0) gent->tls_type = TLS_TLS | TLS_TPREL; } for (gent = h->got.glist; gent != NULL; gent = gent->next) if (gent->got.refcount > 0) { bfd_boolean dyn; /* Make sure this symbol is output as a dynamic symbol. Undefined weak syms won't yet be marked as dynamic, nor will all TLS symbols. */ if (h->dynindx == -1 && !h->forced_local) { if (! bfd_elf_link_record_dynamic_symbol (info, h)) return FALSE; } if ((gent->tls_type & TLS_LD) != 0 && !h->def_dynamic) { - gent->got.offset = ppc64_tlsld_got (gent->owner)->offset; + ppc64_tlsld_got (gent->owner)->refcount += 1; + gent->got.offset = (bfd_vma) -1; continue; } s = ppc64_elf_tdata (gent->owner)->got; gent->got.offset = s->size; s->size += (gent->tls_type & eh->tls_mask & (TLS_GD | TLS_LD)) ? 16 : 8; dyn = htab->elf.dynamic_sections_created; if ((info->shared || WILL_CALL_FINISH_DYNAMIC_SYMBOL (dyn, 0, h)) && (ELF_ST_VISIBILITY (h->other) == STV_DEFAULT || h->root.type != bfd_link_hash_undefweak)) ppc64_elf_tdata (gent->owner)->relgot->size += (gent->tls_type & eh->tls_mask & TLS_GD ? 2 * sizeof (Elf64_External_Rela) : sizeof (Elf64_External_Rela)); } else gent->got.offset = (bfd_vma) -1; if (eh->dyn_relocs == NULL) return TRUE; /* In the shared -Bsymbolic case, discard space allocated for dynamic pc-relative relocs against symbols which turn out to be defined in regular objects. For the normal shared case, discard space for relocs that have become local due to symbol visibility changes. */ if (info->shared) { /* Relocs that use pc_count are those that appear on a call insn, - or certain REL relocs (see MUST_BE_DYN_RELOC) that can be + or certain REL relocs (see must_be_dyn_reloc) that can be generated via assembly. We want calls to protected symbols to resolve directly to the function rather than going via the plt. If people want function pointer comparisons to work as expected then they should avoid writing weird assembly. */ if (SYMBOL_CALLS_LOCAL (info, h)) { struct ppc_dyn_relocs **pp; for (pp = &eh->dyn_relocs; (p = *pp) != NULL; ) { p->count -= p->pc_count; p->pc_count = 0; if (p->count == 0) *pp = p->next; else pp = &p->next; } } /* Also discard relocs on undefined weak syms with non-default visibility. */ if (eh->dyn_relocs != NULL && h->root.type == bfd_link_hash_undefweak) { if (ELF_ST_VISIBILITY (h->other) != STV_DEFAULT) eh->dyn_relocs = NULL; /* Make sure this symbol is output as a dynamic symbol. Undefined weak syms won't yet be marked as dynamic. */ else if (h->dynindx == -1 && !h->forced_local) { if (! bfd_elf_link_record_dynamic_symbol (info, h)) return FALSE; } } } else if (ELIMINATE_COPY_RELOCS) { /* For the non-shared case, discard space for relocs against symbols which turn out to need copy relocs or are not dynamic. */ if (!h->non_got_ref && h->def_dynamic && !h->def_regular) { /* Make sure this symbol is output as a dynamic symbol. Undefined weak syms won't yet be marked as dynamic. */ if (h->dynindx == -1 && !h->forced_local) { if (! bfd_elf_link_record_dynamic_symbol (info, h)) return FALSE; } /* If that succeeded, we know we'll be keeping all the relocs. */ if (h->dynindx != -1) goto keep; } eh->dyn_relocs = NULL; keep: ; } /* Finally, allocate space. */ for (p = eh->dyn_relocs; p != NULL; p = p->next) { asection *sreloc = elf_section_data (p->sec)->sreloc; sreloc->size += p->count * sizeof (Elf64_External_Rela); } return TRUE; } /* Find any dynamic relocs that apply to read-only sections. */ static bfd_boolean readonly_dynrelocs (struct elf_link_hash_entry *h, void *inf) { struct ppc_link_hash_entry *eh; struct ppc_dyn_relocs *p; if (h->root.type == bfd_link_hash_warning) h = (struct elf_link_hash_entry *) h->root.u.i.link; eh = (struct ppc_link_hash_entry *) h; for (p = eh->dyn_relocs; p != NULL; p = p->next) { asection *s = p->sec->output_section; if (s != NULL && (s->flags & SEC_READONLY) != 0) { struct bfd_link_info *info = inf; info->flags |= DF_TEXTREL; /* Not an error, just cut short the traversal. */ return FALSE; } } return TRUE; } /* Set the sizes of the dynamic sections. */ static bfd_boolean ppc64_elf_size_dynamic_sections (bfd *output_bfd ATTRIBUTE_UNUSED, struct bfd_link_info *info) { struct ppc_link_hash_table *htab; bfd *dynobj; asection *s; bfd_boolean relocs; bfd *ibfd; htab = ppc_hash_table (info); dynobj = htab->elf.dynobj; if (dynobj == NULL) abort (); if (htab->elf.dynamic_sections_created) { /* Set the contents of the .interp section to the interpreter. */ if (info->executable) { s = bfd_get_section_by_name (dynobj, ".interp"); if (s == NULL) abort (); s->size = sizeof ELF_DYNAMIC_INTERPRETER; s->contents = (unsigned char *) ELF_DYNAMIC_INTERPRETER; } } /* Set up .got offsets for local syms, and space for local dynamic relocs. */ for (ibfd = info->input_bfds; ibfd != NULL; ibfd = ibfd->link_next) { struct got_entry **lgot_ents; struct got_entry **end_lgot_ents; char *lgot_masks; bfd_size_type locsymcount; Elf_Internal_Shdr *symtab_hdr; asection *srel; if (!is_ppc64_elf_target (ibfd->xvec)) continue; - if (ppc64_tlsld_got (ibfd)->refcount > 0) - { - s = ppc64_elf_tdata (ibfd)->got; - ppc64_tlsld_got (ibfd)->offset = s->size; - s->size += 16; - if (info->shared) - { - srel = ppc64_elf_tdata (ibfd)->relgot; - srel->size += sizeof (Elf64_External_Rela); - } - } - else - ppc64_tlsld_got (ibfd)->offset = (bfd_vma) -1; - for (s = ibfd->sections; s != NULL; s = s->next) { struct ppc_dyn_relocs *p; for (p = elf_section_data (s)->local_dynrel; p != NULL; p = p->next) { if (!bfd_is_abs_section (p->sec) && bfd_is_abs_section (p->sec->output_section)) { /* Input section has been discarded, either because it is a copy of a linkonce section or due to linker script /DISCARD/, so we'll be discarding the relocs too. */ } else if (p->count != 0) { srel = elf_section_data (p->sec)->sreloc; srel->size += p->count * sizeof (Elf64_External_Rela); if ((p->sec->output_section->flags & SEC_READONLY) != 0) info->flags |= DF_TEXTREL; } } } lgot_ents = elf_local_got_ents (ibfd); if (!lgot_ents) continue; symtab_hdr = &elf_tdata (ibfd)->symtab_hdr; locsymcount = symtab_hdr->sh_info; end_lgot_ents = lgot_ents + locsymcount; lgot_masks = (char *) end_lgot_ents; s = ppc64_elf_tdata (ibfd)->got; srel = ppc64_elf_tdata (ibfd)->relgot; for (; lgot_ents < end_lgot_ents; ++lgot_ents, ++lgot_masks) { struct got_entry *ent; for (ent = *lgot_ents; ent != NULL; ent = ent->next) if (ent->got.refcount > 0) { if ((ent->tls_type & *lgot_masks & TLS_LD) != 0) { - if (ppc64_tlsld_got (ibfd)->offset == (bfd_vma) -1) - { - ppc64_tlsld_got (ibfd)->offset = s->size; - s->size += 16; - if (info->shared) - srel->size += sizeof (Elf64_External_Rela); - } - ent->got.offset = ppc64_tlsld_got (ibfd)->offset; + ppc64_tlsld_got (ibfd)->refcount += 1; + ent->got.offset = (bfd_vma) -1; } else { ent->got.offset = s->size; if ((ent->tls_type & *lgot_masks & TLS_GD) != 0) { s->size += 16; if (info->shared) srel->size += 2 * sizeof (Elf64_External_Rela); } else { s->size += 8; if (info->shared) srel->size += sizeof (Elf64_External_Rela); } } } else ent->got.offset = (bfd_vma) -1; } } /* Allocate global sym .plt and .got entries, and space for global sym dynamic relocs. */ elf_link_hash_traverse (&htab->elf, allocate_dynrelocs, info); + for (ibfd = info->input_bfds; ibfd != NULL; ibfd = ibfd->link_next) + { + if (!is_ppc64_elf_target (ibfd->xvec)) + continue; + + if (ppc64_tlsld_got (ibfd)->refcount > 0) + { + s = ppc64_elf_tdata (ibfd)->got; + ppc64_tlsld_got (ibfd)->offset = s->size; + s->size += 16; + if (info->shared) + { + asection *srel = ppc64_elf_tdata (ibfd)->relgot; + srel->size += sizeof (Elf64_External_Rela); + } + } + else + ppc64_tlsld_got (ibfd)->offset = (bfd_vma) -1; + } + /* We now have determined the sizes of the various dynamic sections. Allocate memory for them. */ relocs = FALSE; for (s = dynobj->sections; s != NULL; s = s->next) { if ((s->flags & SEC_LINKER_CREATED) == 0) continue; if (s == htab->brlt || s == htab->relbrlt) /* These haven't been allocated yet; don't strip. */ continue; else if (s == htab->got || s == htab->plt || s == htab->glink || s == htab->dynbss) { /* Strip this section if we don't need it; see the comment below. */ } else if (CONST_STRNEQ (bfd_get_section_name (dynobj, s), ".rela")) { if (s->size != 0) { if (s != htab->relplt) relocs = TRUE; /* We use the reloc_count field as a counter if we need to copy relocs into the output file. */ s->reloc_count = 0; } } else { /* It's not one of our sections, so don't allocate space. */ continue; } if (s->size == 0) { /* If we don't need this section, strip it from the output file. This is mostly to handle .rela.bss and .rela.plt. We must create both sections in create_dynamic_sections, because they must be created before the linker maps input sections to output sections. The linker does that before adjust_dynamic_symbol is called, and it is that function which decides whether anything needs to go into these sections. */ s->flags |= SEC_EXCLUDE; continue; } if ((s->flags & SEC_HAS_CONTENTS) == 0) continue; /* Allocate memory for the section contents. We use bfd_zalloc here in case unused entries are not reclaimed before the section's contents are written out. This should not happen, but this way if it does we get a R_PPC64_NONE reloc in .rela sections instead of garbage. We also rely on the section contents being zero when writing the GOT. */ s->contents = bfd_zalloc (dynobj, s->size); if (s->contents == NULL) return FALSE; } for (ibfd = info->input_bfds; ibfd != NULL; ibfd = ibfd->link_next) { if (!is_ppc64_elf_target (ibfd->xvec)) continue; s = ppc64_elf_tdata (ibfd)->got; if (s != NULL && s != htab->got) { if (s->size == 0) s->flags |= SEC_EXCLUDE; else { s->contents = bfd_zalloc (ibfd, s->size); if (s->contents == NULL) return FALSE; } } s = ppc64_elf_tdata (ibfd)->relgot; if (s != NULL) { if (s->size == 0) s->flags |= SEC_EXCLUDE; else { s->contents = bfd_zalloc (ibfd, s->size); if (s->contents == NULL) return FALSE; relocs = TRUE; s->reloc_count = 0; } } } if (htab->elf.dynamic_sections_created) { /* Add some entries to the .dynamic section. We fill in the values later, in ppc64_elf_finish_dynamic_sections, but we must add the entries now so that we get the correct size for the .dynamic section. The DT_DEBUG entry is filled in by the dynamic linker and used by the debugger. */ #define add_dynamic_entry(TAG, VAL) \ _bfd_elf_add_dynamic_entry (info, TAG, VAL) if (info->executable) { if (!add_dynamic_entry (DT_DEBUG, 0)) return FALSE; } if (htab->plt != NULL && htab->plt->size != 0) { if (!add_dynamic_entry (DT_PLTGOT, 0) || !add_dynamic_entry (DT_PLTRELSZ, 0) || !add_dynamic_entry (DT_PLTREL, DT_RELA) || !add_dynamic_entry (DT_JMPREL, 0) || !add_dynamic_entry (DT_PPC64_GLINK, 0)) return FALSE; } if (NO_OPD_RELOCS) { if (!add_dynamic_entry (DT_PPC64_OPD, 0) || !add_dynamic_entry (DT_PPC64_OPDSZ, 0)) return FALSE; } if (relocs) { if (!add_dynamic_entry (DT_RELA, 0) || !add_dynamic_entry (DT_RELASZ, 0) || !add_dynamic_entry (DT_RELAENT, sizeof (Elf64_External_Rela))) return FALSE; /* If any dynamic relocs apply to a read-only section, then we need a DT_TEXTREL entry. */ if ((info->flags & DF_TEXTREL) == 0) elf_link_hash_traverse (&htab->elf, readonly_dynrelocs, info); if ((info->flags & DF_TEXTREL) != 0) { if (!add_dynamic_entry (DT_TEXTREL, 0)) return FALSE; } } } #undef add_dynamic_entry return TRUE; } /* Determine the type of stub needed, if any, for a call. */ static inline enum ppc_stub_type ppc_type_of_stub (asection *input_sec, const Elf_Internal_Rela *rel, struct ppc_link_hash_entry **hash, bfd_vma destination) { struct ppc_link_hash_entry *h = *hash; bfd_vma location; bfd_vma branch_offset; bfd_vma max_branch_offset; enum elf_ppc64_reloc_type r_type; if (h != NULL) { struct ppc_link_hash_entry *fdh = h; if (fdh->oh != NULL && fdh->oh->is_func_descriptor) fdh = fdh->oh; if (fdh->elf.dynindx != -1) { struct plt_entry *ent; for (ent = fdh->elf.plt.plist; ent != NULL; ent = ent->next) if (ent->addend == rel->r_addend && ent->plt.offset != (bfd_vma) -1) { *hash = fdh; return ppc_stub_plt_call; } } /* Here, we know we don't have a plt entry. If we don't have a either a defined function descriptor or a defined entry symbol in a regular object file, then it is pointless trying to make any other type of stub. */ if (!((fdh->elf.root.type == bfd_link_hash_defined || fdh->elf.root.type == bfd_link_hash_defweak) && fdh->elf.root.u.def.section->output_section != NULL) && !((h->elf.root.type == bfd_link_hash_defined || h->elf.root.type == bfd_link_hash_defweak) && h->elf.root.u.def.section->output_section != NULL)) return ppc_stub_none; } /* Determine where the call point is. */ location = (input_sec->output_offset + input_sec->output_section->vma + rel->r_offset); branch_offset = destination - location; r_type = ELF64_R_TYPE (rel->r_info); /* Determine if a long branch stub is needed. */ max_branch_offset = 1 << 25; if (r_type != R_PPC64_REL24) max_branch_offset = 1 << 15; if (branch_offset + max_branch_offset >= 2 * max_branch_offset) /* We need a stub. Figure out whether a long_branch or plt_branch is needed later. */ return ppc_stub_long_branch; return ppc_stub_none; } /* Build a .plt call stub. */ static inline bfd_byte * build_plt_stub (bfd *obfd, bfd_byte *p, int offset) { #define PPC_LO(v) ((v) & 0xffff) #define PPC_HI(v) (((v) >> 16) & 0xffff) #define PPC_HA(v) PPC_HI ((v) + 0x8000) - bfd_put_32 (obfd, ADDIS_R12_R2 | PPC_HA (offset), p), p += 4; - bfd_put_32 (obfd, STD_R2_40R1, p), p += 4; - bfd_put_32 (obfd, LD_R11_0R12 | PPC_LO (offset), p), p += 4; - if (PPC_HA (offset + 8) != PPC_HA (offset)) - bfd_put_32 (obfd, ADDIS_R12_R12 | 1, p), p += 4; - offset += 8; - bfd_put_32 (obfd, LD_R2_0R12 | PPC_LO (offset), p), p += 4; - if (PPC_HA (offset + 8) != PPC_HA (offset)) - bfd_put_32 (obfd, ADDIS_R12_R12 | 1, p), p += 4; - offset += 8; - bfd_put_32 (obfd, MTCTR_R11, p), p += 4; - bfd_put_32 (obfd, LD_R11_0R12 | PPC_LO (offset), p), p += 4; - bfd_put_32 (obfd, BCTR, p), p += 4; + if (PPC_HA (offset) != 0) + { + bfd_put_32 (obfd, ADDIS_R12_R2 | PPC_HA (offset), p), p += 4; + bfd_put_32 (obfd, STD_R2_40R1, p), p += 4; + bfd_put_32 (obfd, LD_R11_0R12 | PPC_LO (offset), p), p += 4; + if (PPC_HA (offset + 16) != PPC_HA (offset)) + { + bfd_put_32 (obfd, ADDI_R12_R12 | PPC_LO (offset), p), p += 4; + offset = 0; + } + bfd_put_32 (obfd, MTCTR_R11, p), p += 4; + bfd_put_32 (obfd, LD_R2_0R12 | PPC_LO (offset + 8), p), p += 4; + bfd_put_32 (obfd, LD_R11_0R12 | PPC_LO (offset + 16), p), p += 4; + bfd_put_32 (obfd, BCTR, p), p += 4; + } + else + { + bfd_put_32 (obfd, STD_R2_40R1, p), p += 4; + bfd_put_32 (obfd, LD_R11_0R2 | PPC_LO (offset), p), p += 4; + if (PPC_HA (offset + 16) != PPC_HA (offset)) + { + bfd_put_32 (obfd, ADDI_R2_R2 | PPC_LO (offset), p), p += 4; + offset = 0; + } + bfd_put_32 (obfd, MTCTR_R11, p), p += 4; + bfd_put_32 (obfd, LD_R11_0R2 | PPC_LO (offset + 16), p), p += 4; + bfd_put_32 (obfd, LD_R2_0R2 | PPC_LO (offset + 8), p), p += 4; + bfd_put_32 (obfd, BCTR, p), p += 4; + } return p; } static bfd_boolean ppc_build_one_stub (struct bfd_hash_entry *gen_entry, void *in_arg) { struct ppc_stub_hash_entry *stub_entry; struct ppc_branch_hash_entry *br_entry; struct bfd_link_info *info; struct ppc_link_hash_table *htab; bfd_byte *loc; bfd_byte *p; unsigned int indx; struct plt_entry *ent; bfd_vma dest, off; int size; /* Massage our args to the form they really have. */ stub_entry = (struct ppc_stub_hash_entry *) gen_entry; info = in_arg; htab = ppc_hash_table (info); /* Make a note of the offset within the stubs for this entry. */ stub_entry->stub_offset = stub_entry->stub_sec->size; loc = stub_entry->stub_sec->contents + stub_entry->stub_offset; htab->stub_count[stub_entry->stub_type - 1] += 1; switch (stub_entry->stub_type) { case ppc_stub_long_branch: case ppc_stub_long_branch_r2off: /* Branches are relative. This is where we are going to. */ off = dest = (stub_entry->target_value + stub_entry->target_section->output_offset + stub_entry->target_section->output_section->vma); /* And this is where we are coming from. */ off -= (stub_entry->stub_offset + stub_entry->stub_sec->output_offset + stub_entry->stub_sec->output_section->vma); - if (stub_entry->stub_type != ppc_stub_long_branch_r2off) - size = 4; - else + size = 4; + if (stub_entry->stub_type == ppc_stub_long_branch_r2off) { bfd_vma r2off; r2off = (htab->stub_group[stub_entry->target_section->id].toc_off - htab->stub_group[stub_entry->id_sec->id].toc_off); bfd_put_32 (htab->stub_bfd, STD_R2_40R1, loc); loc += 4; - bfd_put_32 (htab->stub_bfd, ADDIS_R2_R2 | PPC_HA (r2off), loc); - loc += 4; + size = 12; + if (PPC_HA (r2off) != 0) + { + size = 16; + bfd_put_32 (htab->stub_bfd, ADDIS_R2_R2 | PPC_HA (r2off), loc); + loc += 4; + } bfd_put_32 (htab->stub_bfd, ADDI_R2_R2 | PPC_LO (r2off), loc); loc += 4; - off -= 12; - size = 16; + off -= size - 4; } bfd_put_32 (htab->stub_bfd, B_DOT | (off & 0x3fffffc), loc); if (off + (1 << 25) >= (bfd_vma) (1 << 26)) { (*_bfd_error_handler) (_("long branch stub `%s' offset overflow"), stub_entry->root.string); htab->stub_error = TRUE; return FALSE; } if (info->emitrelocations) { Elf_Internal_Rela *relocs, *r; struct bfd_elf_section_data *elfsec_data; elfsec_data = elf_section_data (stub_entry->stub_sec); relocs = elfsec_data->relocs; if (relocs == NULL) { bfd_size_type relsize; relsize = stub_entry->stub_sec->reloc_count * sizeof (*relocs); relocs = bfd_alloc (htab->stub_bfd, relsize); if (relocs == NULL) return FALSE; elfsec_data->relocs = relocs; elfsec_data->rel_hdr.sh_size = relsize; elfsec_data->rel_hdr.sh_entsize = 24; stub_entry->stub_sec->reloc_count = 0; } r = relocs + stub_entry->stub_sec->reloc_count; stub_entry->stub_sec->reloc_count += 1; r->r_offset = loc - stub_entry->stub_sec->contents; r->r_info = ELF64_R_INFO (0, R_PPC64_REL24); r->r_addend = dest; if (stub_entry->h != NULL) { struct elf_link_hash_entry **hashes; unsigned long symndx; struct ppc_link_hash_entry *h; hashes = elf_sym_hashes (htab->stub_bfd); if (hashes == NULL) { bfd_size_type hsize; hsize = (htab->stub_globals + 1) * sizeof (*hashes); hashes = bfd_zalloc (htab->stub_bfd, hsize); if (hashes == NULL) return FALSE; elf_sym_hashes (htab->stub_bfd) = hashes; htab->stub_globals = 1; } symndx = htab->stub_globals++; h = stub_entry->h; hashes[symndx] = &h->elf; r->r_info = ELF64_R_INFO (symndx, R_PPC64_REL24); if (h->oh != NULL && h->oh->is_func) h = h->oh; if (h->elf.root.u.def.section != stub_entry->target_section) /* H is an opd symbol. The addend must be zero. */ r->r_addend = 0; else { off = (h->elf.root.u.def.value + h->elf.root.u.def.section->output_offset + h->elf.root.u.def.section->output_section->vma); r->r_addend -= off; } } } break; case ppc_stub_plt_branch: case ppc_stub_plt_branch_r2off: br_entry = ppc_branch_hash_lookup (&htab->branch_hash_table, stub_entry->root.string + 9, FALSE, FALSE); if (br_entry == NULL) { (*_bfd_error_handler) (_("can't find branch stub `%s'"), stub_entry->root.string); htab->stub_error = TRUE; return FALSE; } off = (stub_entry->target_value + stub_entry->target_section->output_offset + stub_entry->target_section->output_section->vma); bfd_put_64 (htab->brlt->owner, off, htab->brlt->contents + br_entry->offset); if (htab->relbrlt != NULL) { /* Create a reloc for the branch lookup table entry. */ Elf_Internal_Rela rela; bfd_byte *rl; rela.r_offset = (br_entry->offset + htab->brlt->output_offset + htab->brlt->output_section->vma); rela.r_info = ELF64_R_INFO (0, R_PPC64_RELATIVE); rela.r_addend = off; rl = htab->relbrlt->contents; rl += htab->relbrlt->reloc_count++ * sizeof (Elf64_External_Rela); bfd_elf64_swap_reloca_out (htab->relbrlt->owner, &rela, rl); } else if (info->emitrelocations) { Elf_Internal_Rela *relocs, *r; struct bfd_elf_section_data *elfsec_data; elfsec_data = elf_section_data (htab->brlt); relocs = elfsec_data->relocs; if (relocs == NULL) { bfd_size_type relsize; relsize = htab->brlt->reloc_count * sizeof (*relocs); relocs = bfd_alloc (htab->brlt->owner, relsize); if (relocs == NULL) return FALSE; elfsec_data->relocs = relocs; elfsec_data->rel_hdr.sh_size = relsize; elfsec_data->rel_hdr.sh_entsize = 24; htab->brlt->reloc_count = 0; } r = relocs + htab->brlt->reloc_count; htab->brlt->reloc_count += 1; r->r_offset = (br_entry->offset + htab->brlt->output_offset + htab->brlt->output_section->vma); r->r_info = ELF64_R_INFO (0, R_PPC64_RELATIVE); r->r_addend = off; } off = (br_entry->offset + htab->brlt->output_offset + htab->brlt->output_section->vma - elf_gp (htab->brlt->output_section->owner) - htab->stub_group[stub_entry->id_sec->id].toc_off); if (off + 0x80008000 > 0xffffffff || (off & 7) != 0) { (*_bfd_error_handler) (_("linkage table error against `%s'"), stub_entry->root.string); bfd_set_error (bfd_error_bad_value); htab->stub_error = TRUE; return FALSE; } indx = off; if (stub_entry->stub_type != ppc_stub_plt_branch_r2off) { - bfd_put_32 (htab->stub_bfd, ADDIS_R12_R2 | PPC_HA (indx), loc); - loc += 4; - bfd_put_32 (htab->stub_bfd, LD_R11_0R12 | PPC_LO (indx), loc); - size = 16; + if (PPC_HA (indx) != 0) + { + size = 16; + bfd_put_32 (htab->stub_bfd, ADDIS_R12_R2 | PPC_HA (indx), loc); + loc += 4; + bfd_put_32 (htab->stub_bfd, LD_R11_0R12 | PPC_LO (indx), loc); + } + else + { + size = 12; + bfd_put_32 (htab->stub_bfd, LD_R11_0R2 | PPC_LO (indx), loc); + } } else { bfd_vma r2off; r2off = (htab->stub_group[stub_entry->target_section->id].toc_off - htab->stub_group[stub_entry->id_sec->id].toc_off); bfd_put_32 (htab->stub_bfd, STD_R2_40R1, loc); loc += 4; - bfd_put_32 (htab->stub_bfd, ADDIS_R12_R2 | PPC_HA (indx), loc); - loc += 4; - bfd_put_32 (htab->stub_bfd, LD_R11_0R12 | PPC_LO (indx), loc); - loc += 4; - bfd_put_32 (htab->stub_bfd, ADDIS_R2_R2 | PPC_HA (r2off), loc); - loc += 4; + size = 20; + if (PPC_HA (indx) != 0) + { + size += 4; + bfd_put_32 (htab->stub_bfd, ADDIS_R12_R2 | PPC_HA (indx), loc); + loc += 4; + bfd_put_32 (htab->stub_bfd, LD_R11_0R12 | PPC_LO (indx), loc); + loc += 4; + } + else + { + bfd_put_32 (htab->stub_bfd, LD_R11_0R2 | PPC_LO (indx), loc); + loc += 4; + } + + if (PPC_HA (r2off) != 0) + { + size += 4; + bfd_put_32 (htab->stub_bfd, ADDIS_R2_R2 | PPC_HA (r2off), loc); + loc += 4; + } bfd_put_32 (htab->stub_bfd, ADDI_R2_R2 | PPC_LO (r2off), loc); - size = 28; } loc += 4; bfd_put_32 (htab->stub_bfd, MTCTR_R11, loc); loc += 4; bfd_put_32 (htab->stub_bfd, BCTR, loc); break; case ppc_stub_plt_call: /* Do the best we can for shared libraries built without exporting ".foo" for each "foo". This can happen when symbol versioning scripts strip all bar a subset of symbols. */ if (stub_entry->h->oh != NULL && stub_entry->h->oh->elf.root.type != bfd_link_hash_defined && stub_entry->h->oh->elf.root.type != bfd_link_hash_defweak) { /* Point the symbol at the stub. There may be multiple stubs, we don't really care; The main thing is to make this sym defined somewhere. Maybe defining the symbol in the stub section is a silly idea. If we didn't do this, htab->top_id could disappear. */ stub_entry->h->oh->elf.root.type = bfd_link_hash_defined; stub_entry->h->oh->elf.root.u.def.section = stub_entry->stub_sec; stub_entry->h->oh->elf.root.u.def.value = stub_entry->stub_offset; } /* Now build the stub. */ off = (bfd_vma) -1; for (ent = stub_entry->h->elf.plt.plist; ent != NULL; ent = ent->next) if (ent->addend == stub_entry->addend) { off = ent->plt.offset; break; } if (off >= (bfd_vma) -2) abort (); off &= ~ (bfd_vma) 1; off += (htab->plt->output_offset + htab->plt->output_section->vma - elf_gp (htab->plt->output_section->owner) - htab->stub_group[stub_entry->id_sec->id].toc_off); if (off + 0x80008000 > 0xffffffff || (off & 7) != 0) { (*_bfd_error_handler) (_("linkage table error against `%s'"), stub_entry->h->elf.root.root.string); bfd_set_error (bfd_error_bad_value); htab->stub_error = TRUE; return FALSE; } p = build_plt_stub (htab->stub_bfd, loc, off); size = p - loc; break; default: BFD_FAIL (); return FALSE; } stub_entry->stub_sec->size += size; if (htab->emit_stub_syms) { struct elf_link_hash_entry *h; size_t len1, len2; char *name; const char *const stub_str[] = { "long_branch", "long_branch_r2off", "plt_branch", "plt_branch_r2off", "plt_call" }; len1 = strlen (stub_str[stub_entry->stub_type - 1]); len2 = strlen (stub_entry->root.string); name = bfd_malloc (len1 + len2 + 2); if (name == NULL) return FALSE; memcpy (name, stub_entry->root.string, 9); memcpy (name + 9, stub_str[stub_entry->stub_type - 1], len1); memcpy (name + len1 + 9, stub_entry->root.string + 8, len2 - 8 + 1); h = elf_link_hash_lookup (&htab->elf, name, TRUE, FALSE, FALSE); if (h == NULL) return FALSE; if (h->root.type == bfd_link_hash_new) { h->root.type = bfd_link_hash_defined; h->root.u.def.section = stub_entry->stub_sec; h->root.u.def.value = stub_entry->stub_offset; h->ref_regular = 1; h->def_regular = 1; h->ref_regular_nonweak = 1; h->forced_local = 1; h->non_elf = 0; } } return TRUE; } /* As above, but don't actually build the stub. Just bump offset so we know stub section sizes, and select plt_branch stubs where long_branch stubs won't do. */ static bfd_boolean ppc_size_one_stub (struct bfd_hash_entry *gen_entry, void *in_arg) { struct ppc_stub_hash_entry *stub_entry; struct bfd_link_info *info; struct ppc_link_hash_table *htab; bfd_vma off; int size; /* Massage our args to the form they really have. */ stub_entry = (struct ppc_stub_hash_entry *) gen_entry; info = in_arg; htab = ppc_hash_table (info); if (stub_entry->stub_type == ppc_stub_plt_call) { struct plt_entry *ent; off = (bfd_vma) -1; for (ent = stub_entry->h->elf.plt.plist; ent != NULL; ent = ent->next) if (ent->addend == stub_entry->addend) { off = ent->plt.offset & ~(bfd_vma) 1; break; } if (off >= (bfd_vma) -2) abort (); off += (htab->plt->output_offset + htab->plt->output_section->vma - elf_gp (htab->plt->output_section->owner) - htab->stub_group[stub_entry->id_sec->id].toc_off); size = PLT_CALL_STUB_SIZE; + if (PPC_HA (off) == 0) + size -= 4; if (PPC_HA (off + 16) != PPC_HA (off)) size += 4; } else { /* ppc_stub_long_branch or ppc_stub_plt_branch, or their r2off variants. */ + bfd_vma r2off = 0; + off = (stub_entry->target_value + stub_entry->target_section->output_offset + stub_entry->target_section->output_section->vma); off -= (stub_entry->stub_sec->size + stub_entry->stub_sec->output_offset + stub_entry->stub_sec->output_section->vma); /* Reset the stub type from the plt variant in case we now can reach with a shorter stub. */ if (stub_entry->stub_type >= ppc_stub_plt_branch) stub_entry->stub_type += ppc_stub_long_branch - ppc_stub_plt_branch; size = 4; if (stub_entry->stub_type == ppc_stub_long_branch_r2off) { - off -= 12; - size = 16; + r2off = (htab->stub_group[stub_entry->target_section->id].toc_off + - htab->stub_group[stub_entry->id_sec->id].toc_off); + size = 12; + if (PPC_HA (r2off) != 0) + size = 16; + off -= size - 4; } /* If the branch offset if too big, use a ppc_stub_plt_branch. */ if (off + (1 << 25) >= (bfd_vma) (1 << 26)) { struct ppc_branch_hash_entry *br_entry; + unsigned int indx; br_entry = ppc_branch_hash_lookup (&htab->branch_hash_table, stub_entry->root.string + 9, TRUE, FALSE); if (br_entry == NULL) { (*_bfd_error_handler) (_("can't build branch stub `%s'"), stub_entry->root.string); htab->stub_error = TRUE; return FALSE; } if (br_entry->iter != htab->stub_iteration) { br_entry->iter = htab->stub_iteration; br_entry->offset = htab->brlt->size; htab->brlt->size += 8; if (htab->relbrlt != NULL) htab->relbrlt->size += sizeof (Elf64_External_Rela); else if (info->emitrelocations) { htab->brlt->reloc_count += 1; htab->brlt->flags |= SEC_RELOC; } } stub_entry->stub_type += ppc_stub_plt_branch - ppc_stub_long_branch; - size = 16; - if (stub_entry->stub_type != ppc_stub_plt_branch) - size = 28; + off = (br_entry->offset + + htab->brlt->output_offset + + htab->brlt->output_section->vma + - elf_gp (htab->brlt->output_section->owner) + - htab->stub_group[stub_entry->id_sec->id].toc_off); + + indx = off; + if (stub_entry->stub_type != ppc_stub_plt_branch_r2off) + { + size = 12; + if (PPC_HA (indx) != 0) + size = 16; + } + else + { + size = 20; + if (PPC_HA (indx) != 0) + size += 4; + + if (PPC_HA (r2off) != 0) + size += 4; + } } else if (info->emitrelocations) { stub_entry->stub_sec->reloc_count += 1; stub_entry->stub_sec->flags |= SEC_RELOC; } } stub_entry->stub_sec->size += size; return TRUE; } /* Set up various things so that we can make a list of input sections for each output section included in the link. Returns -1 on error, 0 when no stubs will be needed, and 1 on success. */ int ppc64_elf_setup_section_lists (bfd *output_bfd, struct bfd_link_info *info, int no_multi_toc) { bfd *input_bfd; int top_id, top_index, id; asection *section; asection **input_list; bfd_size_type amt; struct ppc_link_hash_table *htab = ppc_hash_table (info); htab->no_multi_toc = no_multi_toc; if (htab->brlt == NULL) return 0; /* Find the top input section id. */ for (input_bfd = info->input_bfds, top_id = 3; input_bfd != NULL; input_bfd = input_bfd->link_next) { for (section = input_bfd->sections; section != NULL; section = section->next) { if (top_id < section->id) top_id = section->id; } } htab->top_id = top_id; amt = sizeof (struct map_stub) * (top_id + 1); htab->stub_group = bfd_zmalloc (amt); if (htab->stub_group == NULL) return -1; /* Set toc_off for com, und, abs and ind sections. */ for (id = 0; id < 3; id++) htab->stub_group[id].toc_off = TOC_BASE_OFF; elf_gp (output_bfd) = htab->toc_curr = ppc64_elf_toc (output_bfd); /* We can't use output_bfd->section_count here to find the top output section index as some sections may have been removed, and strip_excluded_output_sections doesn't renumber the indices. */ for (section = output_bfd->sections, top_index = 0; section != NULL; section = section->next) { if (top_index < section->index) top_index = section->index; } htab->top_index = top_index; amt = sizeof (asection *) * (top_index + 1); input_list = bfd_zmalloc (amt); htab->input_list = input_list; if (input_list == NULL) return -1; return 1; } /* The linker repeatedly calls this function for each TOC input section and linker generated GOT section. Group input bfds such that the toc within a group is less than 64k in size. Will break with cute linker scripts that play games with dot in the output toc section. */ void ppc64_elf_next_toc_section (struct bfd_link_info *info, asection *isec) { struct ppc_link_hash_table *htab = ppc_hash_table (info); if (!htab->no_multi_toc) { bfd_vma addr = isec->output_offset + isec->output_section->vma; bfd_vma off = addr - htab->toc_curr; if (off + isec->size > 0x10000) htab->toc_curr = addr; elf_gp (isec->owner) = (htab->toc_curr - elf_gp (isec->output_section->owner) + TOC_BASE_OFF); } } /* Called after the last call to the above function. */ void ppc64_elf_reinit_toc (bfd *output_bfd, struct bfd_link_info *info) { struct ppc_link_hash_table *htab = ppc_hash_table (info); htab->multi_toc_needed = htab->toc_curr != elf_gp (output_bfd); /* toc_curr tracks the TOC offset used for code sections below in ppc64_elf_next_input_section. Start off at 0x8000. */ htab->toc_curr = TOC_BASE_OFF; } /* No toc references were found in ISEC. If the code in ISEC makes no calls, then there's no need to use toc adjusting stubs when branching into ISEC. Actually, indirect calls from ISEC are OK as they will load r2. Returns -1 on error, 0 for no stub needed, 1 for stub needed, and 2 if a cyclical call-graph was found but no other reason for a stub was detected. If called from the top level, a return of 2 means the same as a return of 0. */ static int toc_adjusting_stub_needed (struct bfd_link_info *info, asection *isec) { Elf_Internal_Rela *relstart, *rel; Elf_Internal_Sym *local_syms; int ret; struct ppc_link_hash_table *htab; /* We know none of our code bearing sections will need toc stubs. */ if ((isec->flags & SEC_LINKER_CREATED) != 0) return 0; if (isec->size == 0) return 0; if (isec->output_section == NULL) return 0; /* Hack for linux kernel. .fixup contains branches, but only back to the function that hit an exception. */ if (strcmp (isec->name, ".fixup") == 0) return 0; if (isec->reloc_count == 0) return 0; relstart = _bfd_elf_link_read_relocs (isec->owner, isec, NULL, NULL, info->keep_memory); if (relstart == NULL) return -1; /* Look for branches to outside of this section. */ local_syms = NULL; ret = 0; htab = ppc_hash_table (info); for (rel = relstart; rel < relstart + isec->reloc_count; ++rel) { enum elf_ppc64_reloc_type r_type; unsigned long r_symndx; struct elf_link_hash_entry *h; + struct ppc_link_hash_entry *eh; Elf_Internal_Sym *sym; asection *sym_sec; long *opd_adjust; bfd_vma sym_value; bfd_vma dest; r_type = ELF64_R_TYPE (rel->r_info); if (r_type != R_PPC64_REL24 && r_type != R_PPC64_REL14 && r_type != R_PPC64_REL14_BRTAKEN && r_type != R_PPC64_REL14_BRNTAKEN) continue; r_symndx = ELF64_R_SYM (rel->r_info); if (!get_sym_h (&h, &sym, &sym_sec, NULL, &local_syms, r_symndx, isec->owner)) { ret = -1; break; } /* Calls to dynamic lib functions go through a plt call stub - that uses r2. Branches to undefined symbols might be a call - using old-style dot symbols that can be satisfied by a plt - call into a new-style dynamic library. */ - if (sym_sec == NULL) + that uses r2. */ + eh = (struct ppc_link_hash_entry *) h; + if (eh != NULL + && (eh->elf.plt.plist != NULL + || (eh->oh != NULL + && eh->oh->elf.plt.plist != NULL))) { - struct ppc_link_hash_entry *eh = (struct ppc_link_hash_entry *) h; - if (eh != NULL - && eh->oh != NULL - && eh->oh->elf.plt.plist != NULL) - { - ret = 1; - break; - } - - /* Ignore other undefined symbols. */ - continue; + ret = 1; + break; } + if (sym_sec == NULL) + /* Ignore other undefined symbols. */ + continue; + /* Assume branches to other sections not included in the link need stubs too, to cover -R and absolute syms. */ if (sym_sec->output_section == NULL) { ret = 1; break; } if (h == NULL) sym_value = sym->st_value; else { if (h->root.type != bfd_link_hash_defined && h->root.type != bfd_link_hash_defweak) abort (); sym_value = h->root.u.def.value; } sym_value += rel->r_addend; /* If this branch reloc uses an opd sym, find the code section. */ opd_adjust = get_opd_info (sym_sec); if (opd_adjust != NULL) { if (h == NULL) { long adjust; adjust = opd_adjust[sym->st_value / 8]; if (adjust == -1) /* Assume deleted functions won't ever be called. */ continue; sym_value += adjust; } dest = opd_entry_value (sym_sec, sym_value, &sym_sec, NULL); if (dest == (bfd_vma) -1) continue; } else dest = (sym_value + sym_sec->output_offset + sym_sec->output_section->vma); /* Ignore branch to self. */ if (sym_sec == isec) continue; /* If the called function uses the toc, we need a stub. */ if (sym_sec->has_toc_reloc || sym_sec->makes_toc_func_call) { ret = 1; break; } /* Assume any branch that needs a long branch stub might in fact need a plt_branch stub. A plt_branch stub uses r2. */ else if (dest - (isec->output_offset + isec->output_section->vma + rel->r_offset) + (1 << 25) >= (2 << 25)) { ret = 1; break; } /* If calling back to a section in the process of being tested, we can't say for sure that no toc adjusting stubs are needed, so don't return zero. */ else if (sym_sec->call_check_in_progress) ret = 2; /* Branches to another section that itself doesn't have any TOC references are OK. Recursively call ourselves to check. */ else if (sym_sec->id <= htab->top_id && htab->stub_group[sym_sec->id].toc_off == 0) { int recur; /* Mark current section as indeterminate, so that other sections that call back to current won't be marked as known. */ isec->call_check_in_progress = 1; recur = toc_adjusting_stub_needed (info, sym_sec); isec->call_check_in_progress = 0; if (recur < 0) { /* An error. Exit. */ ret = -1; break; } else if (recur <= 1) { /* Known result. Mark as checked and set section flag. */ htab->stub_group[sym_sec->id].toc_off = 1; if (recur != 0) { sym_sec->makes_toc_func_call = 1; ret = 1; break; } } else { /* Unknown result. Continue checking. */ ret = 2; } } } if (local_syms != NULL && (elf_tdata (isec->owner)->symtab_hdr.contents != (unsigned char *) local_syms)) free (local_syms); if (elf_section_data (isec)->relocs != relstart) free (relstart); return ret; } /* The linker repeatedly calls this function for each input section, in the order that input sections are linked into output sections. Build lists of input sections to determine groupings between which we may insert linker stubs. */ bfd_boolean ppc64_elf_next_input_section (struct bfd_link_info *info, asection *isec) { struct ppc_link_hash_table *htab = ppc_hash_table (info); if ((isec->output_section->flags & SEC_CODE) != 0 && isec->output_section->index <= htab->top_index) { asection **list = htab->input_list + isec->output_section->index; /* Steal the link_sec pointer for our list. */ #define PREV_SEC(sec) (htab->stub_group[(sec)->id].link_sec) /* This happens to make the list in reverse order, which is what we want. */ PREV_SEC (isec) = *list; *list = isec; } if (htab->multi_toc_needed) { /* If a code section has a function that uses the TOC then we need to use the right TOC (obviously). Also, make sure that .opd gets the correct TOC value for R_PPC64_TOC relocs that don't have or can't find their function symbol (shouldn't ever happen now). */ if (isec->has_toc_reloc || (isec->flags & SEC_CODE) == 0) { if (elf_gp (isec->owner) != 0) htab->toc_curr = elf_gp (isec->owner); } else if (htab->stub_group[isec->id].toc_off == 0) { int ret = toc_adjusting_stub_needed (info, isec); if (ret < 0) return FALSE; else isec->makes_toc_func_call = ret & 1; } } /* Functions that don't use the TOC can belong in any TOC group. Use the last TOC base. This happens to make _init and _fini pasting work. */ htab->stub_group[isec->id].toc_off = htab->toc_curr; return TRUE; } /* See whether we can group stub sections together. Grouping stub sections may result in fewer stubs. More importantly, we need to put all .init* and .fini* stubs at the beginning of the .init or .fini output sections respectively, because glibc splits the _init and _fini functions into multiple parts. Putting a stub in the middle of a function is not a good idea. */ static void group_sections (struct ppc_link_hash_table *htab, bfd_size_type stub_group_size, bfd_boolean stubs_always_before_branch) { asection **list; bfd_size_type stub14_group_size; bfd_boolean suppress_size_errors; suppress_size_errors = FALSE; stub14_group_size = stub_group_size; if (stub_group_size == 1) { /* Default values. */ if (stubs_always_before_branch) { stub_group_size = 0x1e00000; stub14_group_size = 0x7800; } else { stub_group_size = 0x1c00000; stub14_group_size = 0x7000; } suppress_size_errors = TRUE; } list = htab->input_list + htab->top_index; do { asection *tail = *list; while (tail != NULL) { asection *curr; asection *prev; bfd_size_type total; bfd_boolean big_sec; bfd_vma curr_toc; curr = tail; total = tail->size; big_sec = total > (ppc64_elf_section_data (tail)->has_14bit_branch ? stub14_group_size : stub_group_size); if (big_sec && !suppress_size_errors) (*_bfd_error_handler) (_("%B section %A exceeds stub group size"), tail->owner, tail); curr_toc = htab->stub_group[tail->id].toc_off; while ((prev = PREV_SEC (curr)) != NULL && ((total += curr->output_offset - prev->output_offset) < (ppc64_elf_section_data (prev)->has_14bit_branch ? stub14_group_size : stub_group_size)) && htab->stub_group[prev->id].toc_off == curr_toc) curr = prev; /* OK, the size from the start of CURR to the end is less than stub_group_size and thus can be handled by one stub section. (or the tail section is itself larger than stub_group_size, in which case we may be toast.) We should really be keeping track of the total size of stubs added here, as stubs contribute to the final output section size. That's a little tricky, and this way will only break if stubs added make the total size more than 2^25, ie. for the default stub_group_size, if stubs total more than 2097152 bytes, or nearly 75000 plt call stubs. */ do { prev = PREV_SEC (tail); /* Set up this stub group. */ htab->stub_group[tail->id].link_sec = curr; } while (tail != curr && (tail = prev) != NULL); /* But wait, there's more! Input sections up to stub_group_size bytes before the stub section can be handled by it too. Don't do this if we have a really large section after the stubs, as adding more stubs increases the chance that branches may not reach into the stub section. */ if (!stubs_always_before_branch && !big_sec) { total = 0; while (prev != NULL && ((total += tail->output_offset - prev->output_offset) < (ppc64_elf_section_data (prev)->has_14bit_branch ? stub14_group_size : stub_group_size)) && htab->stub_group[prev->id].toc_off == curr_toc) { tail = prev; prev = PREV_SEC (tail); htab->stub_group[tail->id].link_sec = curr; } } tail = prev; } } while (list-- != htab->input_list); free (htab->input_list); #undef PREV_SEC } /* Determine and set the size of the stub section for a final link. The basic idea here is to examine all the relocations looking for PC-relative calls to a target that is unreachable with a "bl" instruction. */ bfd_boolean ppc64_elf_size_stubs (bfd *output_bfd, struct bfd_link_info *info, bfd_signed_vma group_size, asection *(*add_stub_section) (const char *, asection *), void (*layout_sections_again) (void)) { bfd_size_type stub_group_size; bfd_boolean stubs_always_before_branch; struct ppc_link_hash_table *htab = ppc_hash_table (info); /* Stash our params away. */ htab->add_stub_section = add_stub_section; htab->layout_sections_again = layout_sections_again; stubs_always_before_branch = group_size < 0; if (group_size < 0) stub_group_size = -group_size; else stub_group_size = group_size; group_sections (htab, stub_group_size, stubs_always_before_branch); while (1) { bfd *input_bfd; unsigned int bfd_indx; asection *stub_sec; htab->stub_iteration += 1; for (input_bfd = info->input_bfds, bfd_indx = 0; input_bfd != NULL; input_bfd = input_bfd->link_next, bfd_indx++) { Elf_Internal_Shdr *symtab_hdr; asection *section; Elf_Internal_Sym *local_syms = NULL; if (!is_ppc64_elf_target (input_bfd->xvec)) continue; /* We'll need the symbol table in a second. */ symtab_hdr = &elf_tdata (input_bfd)->symtab_hdr; if (symtab_hdr->sh_info == 0) continue; /* Walk over each section attached to the input bfd. */ for (section = input_bfd->sections; section != NULL; section = section->next) { Elf_Internal_Rela *internal_relocs, *irelaend, *irela; /* If there aren't any relocs, then there's nothing more to do. */ if ((section->flags & SEC_RELOC) == 0 || (section->flags & SEC_ALLOC) == 0 || (section->flags & SEC_LOAD) == 0 || (section->flags & SEC_CODE) == 0 || section->reloc_count == 0) continue; /* If this section is a link-once section that will be discarded, then don't create any stubs. */ if (section->output_section == NULL || section->output_section->owner != output_bfd) continue; /* Get the relocs. */ internal_relocs = _bfd_elf_link_read_relocs (input_bfd, section, NULL, NULL, info->keep_memory); if (internal_relocs == NULL) goto error_ret_free_local; /* Now examine each relocation. */ irela = internal_relocs; irelaend = irela + section->reloc_count; for (; irela < irelaend; irela++) { enum elf_ppc64_reloc_type r_type; unsigned int r_indx; enum ppc_stub_type stub_type; struct ppc_stub_hash_entry *stub_entry; asection *sym_sec, *code_sec; bfd_vma sym_value; bfd_vma destination; bfd_boolean ok_dest; struct ppc_link_hash_entry *hash; struct ppc_link_hash_entry *fdh; struct elf_link_hash_entry *h; Elf_Internal_Sym *sym; char *stub_name; const asection *id_sec; long *opd_adjust; r_type = ELF64_R_TYPE (irela->r_info); r_indx = ELF64_R_SYM (irela->r_info); if (r_type >= R_PPC64_max) { bfd_set_error (bfd_error_bad_value); goto error_ret_free_internal; } /* Only look for stubs on branch instructions. */ if (r_type != R_PPC64_REL24 && r_type != R_PPC64_REL14 && r_type != R_PPC64_REL14_BRTAKEN && r_type != R_PPC64_REL14_BRNTAKEN) continue; /* Now determine the call target, its name, value, section. */ if (!get_sym_h (&h, &sym, &sym_sec, NULL, &local_syms, r_indx, input_bfd)) goto error_ret_free_internal; hash = (struct ppc_link_hash_entry *) h; ok_dest = FALSE; fdh = NULL; sym_value = 0; if (hash == NULL) { sym_value = sym->st_value; ok_dest = TRUE; } else if (hash->elf.root.type == bfd_link_hash_defined || hash->elf.root.type == bfd_link_hash_defweak) { sym_value = hash->elf.root.u.def.value; if (sym_sec->output_section != NULL) ok_dest = TRUE; } else if (hash->elf.root.type == bfd_link_hash_undefweak || hash->elf.root.type == bfd_link_hash_undefined) { /* Recognise an old ABI func code entry sym, and use the func descriptor sym instead if it is defined. */ if (hash->elf.root.root.string[0] == '.' && (fdh = get_fdh (hash, htab)) != NULL) { if (fdh->elf.root.type == bfd_link_hash_defined || fdh->elf.root.type == bfd_link_hash_defweak) { sym_sec = fdh->elf.root.u.def.section; sym_value = fdh->elf.root.u.def.value; if (sym_sec->output_section != NULL) ok_dest = TRUE; } else fdh = NULL; } } else { bfd_set_error (bfd_error_bad_value); goto error_ret_free_internal; } destination = 0; if (ok_dest) { sym_value += irela->r_addend; destination = (sym_value + sym_sec->output_offset + sym_sec->output_section->vma); } code_sec = sym_sec; opd_adjust = get_opd_info (sym_sec); if (opd_adjust != NULL) { bfd_vma dest; if (hash == NULL) { long adjust = opd_adjust[sym_value / 8]; if (adjust == -1) continue; sym_value += adjust; } dest = opd_entry_value (sym_sec, sym_value, &code_sec, &sym_value); if (dest != (bfd_vma) -1) { destination = dest; if (fdh != NULL) { /* Fixup old ABI sym to point at code entry. */ hash->elf.root.type = bfd_link_hash_defweak; hash->elf.root.u.def.section = code_sec; hash->elf.root.u.def.value = sym_value; } } } /* Determine what (if any) linker stub is needed. */ stub_type = ppc_type_of_stub (section, irela, &hash, destination); if (stub_type != ppc_stub_plt_call) { /* Check whether we need a TOC adjusting stub. Since the linker pastes together pieces from different object files when creating the _init and _fini functions, it may be that a call to what looks like a local sym is in fact a call needing a TOC adjustment. */ if (code_sec != NULL && code_sec->output_section != NULL && (htab->stub_group[code_sec->id].toc_off != htab->stub_group[section->id].toc_off) && (code_sec->has_toc_reloc || code_sec->makes_toc_func_call)) stub_type = ppc_stub_long_branch_r2off; } if (stub_type == ppc_stub_none) continue; /* __tls_get_addr calls might be eliminated. */ if (stub_type != ppc_stub_plt_call && hash != NULL && (hash == htab->tls_get_addr || hash == htab->tls_get_addr_fd) && section->has_tls_reloc && irela != internal_relocs) { /* Get tls info. */ char *tls_mask; - if (!get_tls_mask (&tls_mask, NULL, &local_syms, + if (!get_tls_mask (&tls_mask, NULL, NULL, &local_syms, irela - 1, input_bfd)) goto error_ret_free_internal; if (*tls_mask != 0) continue; } /* Support for grouping stub sections. */ id_sec = htab->stub_group[section->id].link_sec; /* Get the name of this stub. */ stub_name = ppc_stub_name (id_sec, sym_sec, hash, irela); if (!stub_name) goto error_ret_free_internal; stub_entry = ppc_stub_hash_lookup (&htab->stub_hash_table, stub_name, FALSE, FALSE); if (stub_entry != NULL) { /* The proper stub has already been created. */ free (stub_name); continue; } stub_entry = ppc_add_stub (stub_name, section, htab); if (stub_entry == NULL) { free (stub_name); error_ret_free_internal: if (elf_section_data (section)->relocs == NULL) free (internal_relocs); error_ret_free_local: if (local_syms != NULL && (symtab_hdr->contents != (unsigned char *) local_syms)) free (local_syms); return FALSE; } stub_entry->stub_type = stub_type; stub_entry->target_value = sym_value; stub_entry->target_section = code_sec; stub_entry->h = hash; stub_entry->addend = irela->r_addend; if (stub_entry->h != NULL) htab->stub_globals += 1; } /* We're done with the internal relocs, free them. */ if (elf_section_data (section)->relocs != internal_relocs) free (internal_relocs); } if (local_syms != NULL && symtab_hdr->contents != (unsigned char *) local_syms) { if (!info->keep_memory) free (local_syms); else symtab_hdr->contents = (unsigned char *) local_syms; } } /* We may have added some stubs. Find out the new size of the stub sections. */ for (stub_sec = htab->stub_bfd->sections; stub_sec != NULL; stub_sec = stub_sec->next) if ((stub_sec->flags & SEC_LINKER_CREATED) == 0) { stub_sec->rawsize = stub_sec->size; stub_sec->size = 0; stub_sec->reloc_count = 0; stub_sec->flags &= ~SEC_RELOC; } htab->brlt->size = 0; htab->brlt->reloc_count = 0; htab->brlt->flags &= ~SEC_RELOC; if (htab->relbrlt != NULL) htab->relbrlt->size = 0; bfd_hash_traverse (&htab->stub_hash_table, ppc_size_one_stub, info); for (stub_sec = htab->stub_bfd->sections; stub_sec != NULL; stub_sec = stub_sec->next) if ((stub_sec->flags & SEC_LINKER_CREATED) == 0 && stub_sec->rawsize != stub_sec->size) break; /* Exit from this loop when no stubs have been added, and no stubs have changed size. */ if (stub_sec == NULL) break; /* Ask the linker to do its stuff. */ (*htab->layout_sections_again) (); } /* It would be nice to strip htab->brlt from the output if the section is empty, but it's too late. If we strip sections here, the dynamic symbol table is corrupted since the section symbol for the stripped section isn't written. */ return TRUE; } /* Called after we have determined section placement. If sections move, we'll be called again. Provide a value for TOCstart. */ bfd_vma ppc64_elf_toc (bfd *obfd) { asection *s; bfd_vma TOCstart; /* The TOC consists of sections .got, .toc, .tocbss, .plt in that order. The TOC starts where the first of these sections starts. */ s = bfd_get_section_by_name (obfd, ".got"); if (s == NULL) s = bfd_get_section_by_name (obfd, ".toc"); if (s == NULL) s = bfd_get_section_by_name (obfd, ".tocbss"); if (s == NULL) s = bfd_get_section_by_name (obfd, ".plt"); if (s == NULL) { /* This may happen for o references to TOC base (SYM@toc / TOC[tc0]) without a .toc directive o bad linker script o --gc-sections and empty TOC sections FIXME: Warn user? */ /* Look for a likely section. We probably won't even be using TOCstart. */ for (s = obfd->sections; s != NULL; s = s->next) if ((s->flags & (SEC_ALLOC | SEC_SMALL_DATA | SEC_READONLY)) == (SEC_ALLOC | SEC_SMALL_DATA)) break; if (s == NULL) for (s = obfd->sections; s != NULL; s = s->next) if ((s->flags & (SEC_ALLOC | SEC_SMALL_DATA)) == (SEC_ALLOC | SEC_SMALL_DATA)) break; if (s == NULL) for (s = obfd->sections; s != NULL; s = s->next) if ((s->flags & (SEC_ALLOC | SEC_READONLY)) == SEC_ALLOC) break; if (s == NULL) for (s = obfd->sections; s != NULL; s = s->next) if ((s->flags & SEC_ALLOC) == SEC_ALLOC) break; } TOCstart = 0; if (s != NULL) TOCstart = s->output_section->vma + s->output_offset; return TOCstart; } /* Build all the stubs associated with the current output file. The stubs are kept in a hash table attached to the main linker hash table. This function is called via gldelf64ppc_finish. */ bfd_boolean ppc64_elf_build_stubs (bfd_boolean emit_stub_syms, struct bfd_link_info *info, char **stats) { struct ppc_link_hash_table *htab = ppc_hash_table (info); asection *stub_sec; bfd_byte *p; int stub_sec_count = 0; htab->emit_stub_syms = emit_stub_syms; /* Allocate memory to hold the linker stubs. */ for (stub_sec = htab->stub_bfd->sections; stub_sec != NULL; stub_sec = stub_sec->next) if ((stub_sec->flags & SEC_LINKER_CREATED) == 0 && stub_sec->size != 0) { stub_sec->contents = bfd_zalloc (htab->stub_bfd, stub_sec->size); if (stub_sec->contents == NULL) return FALSE; /* We want to check that built size is the same as calculated size. rawsize is a convenient location to use. */ stub_sec->rawsize = stub_sec->size; stub_sec->size = 0; } if (htab->glink != NULL && htab->glink->size != 0) { unsigned int indx; bfd_vma plt0; /* Build the .glink plt call stub. */ if (htab->emit_stub_syms) { struct elf_link_hash_entry *h; h = elf_link_hash_lookup (&htab->elf, "__glink", TRUE, FALSE, FALSE); if (h == NULL) return FALSE; if (h->root.type == bfd_link_hash_new) { h->root.type = bfd_link_hash_defined; h->root.u.def.section = htab->glink; h->root.u.def.value = 8; h->ref_regular = 1; h->def_regular = 1; h->ref_regular_nonweak = 1; h->forced_local = 1; h->non_elf = 0; } } p = htab->glink->contents; plt0 = (htab->plt->output_section->vma + htab->plt->output_offset - (htab->glink->output_section->vma + htab->glink->output_offset + 16)); bfd_put_64 (htab->glink->owner, plt0, p); p += 8; bfd_put_32 (htab->glink->owner, MFLR_R12, p); p += 4; bfd_put_32 (htab->glink->owner, BCL_20_31, p); p += 4; bfd_put_32 (htab->glink->owner, MFLR_R11, p); p += 4; bfd_put_32 (htab->glink->owner, LD_R2_M16R11, p); p += 4; bfd_put_32 (htab->glink->owner, MTLR_R12, p); p += 4; bfd_put_32 (htab->glink->owner, ADD_R12_R2_R11, p); p += 4; bfd_put_32 (htab->glink->owner, LD_R11_0R12, p); p += 4; bfd_put_32 (htab->glink->owner, LD_R2_0R12 | 8, p); p += 4; bfd_put_32 (htab->glink->owner, MTCTR_R11, p); p += 4; bfd_put_32 (htab->glink->owner, LD_R11_0R12 | 16, p); p += 4; bfd_put_32 (htab->glink->owner, BCTR, p); p += 4; while (p - htab->glink->contents < GLINK_CALL_STUB_SIZE) { bfd_put_32 (htab->glink->owner, NOP, p); p += 4; } /* Build the .glink lazy link call stubs. */ indx = 0; while (p < htab->glink->contents + htab->glink->size) { if (indx < 0x8000) { bfd_put_32 (htab->glink->owner, LI_R0_0 | indx, p); p += 4; } else { bfd_put_32 (htab->glink->owner, LIS_R0_0 | PPC_HI (indx), p); p += 4; bfd_put_32 (htab->glink->owner, ORI_R0_R0_0 | PPC_LO (indx), p); p += 4; } bfd_put_32 (htab->glink->owner, B_DOT | ((htab->glink->contents - p + 8) & 0x3fffffc), p); indx++; p += 4; } htab->glink->rawsize = p - htab->glink->contents; } if (htab->brlt->size != 0) { htab->brlt->contents = bfd_zalloc (htab->brlt->owner, htab->brlt->size); if (htab->brlt->contents == NULL) return FALSE; } if (htab->relbrlt != NULL && htab->relbrlt->size != 0) { htab->relbrlt->contents = bfd_zalloc (htab->relbrlt->owner, htab->relbrlt->size); if (htab->relbrlt->contents == NULL) return FALSE; } /* Build the stubs as directed by the stub hash table. */ bfd_hash_traverse (&htab->stub_hash_table, ppc_build_one_stub, info); if (htab->relbrlt != NULL) htab->relbrlt->reloc_count = 0; for (stub_sec = htab->stub_bfd->sections; stub_sec != NULL; stub_sec = stub_sec->next) if ((stub_sec->flags & SEC_LINKER_CREATED) == 0) { stub_sec_count += 1; if (stub_sec->rawsize != stub_sec->size) break; } if (stub_sec != NULL || htab->glink->rawsize != htab->glink->size) { htab->stub_error = TRUE; (*_bfd_error_handler) (_("stubs don't match calculated size")); } if (htab->stub_error) return FALSE; if (stats != NULL) { *stats = bfd_malloc (500); if (*stats == NULL) return FALSE; sprintf (*stats, _("linker stubs in %u group%s\n" " branch %lu\n" " toc adjust %lu\n" " long branch %lu\n" " long toc adj %lu\n" " plt call %lu"), stub_sec_count, stub_sec_count == 1 ? "" : "s", htab->stub_count[ppc_stub_long_branch - 1], htab->stub_count[ppc_stub_long_branch_r2off - 1], htab->stub_count[ppc_stub_plt_branch - 1], htab->stub_count[ppc_stub_plt_branch_r2off - 1], htab->stub_count[ppc_stub_plt_call - 1]); } return TRUE; } /* This function undoes the changes made by add_symbol_adjust. */ static bfd_boolean undo_symbol_twiddle (struct elf_link_hash_entry *h, void *inf ATTRIBUTE_UNUSED) { struct ppc_link_hash_entry *eh; if (h->root.type == bfd_link_hash_indirect) return TRUE; if (h->root.type == bfd_link_hash_warning) h = (struct elf_link_hash_entry *) h->root.u.i.link; eh = (struct ppc_link_hash_entry *) h; if (eh->elf.root.type != bfd_link_hash_undefweak || !eh->was_undefined) return TRUE; eh->elf.root.type = bfd_link_hash_undefined; return TRUE; } void ppc64_elf_restore_symbols (struct bfd_link_info *info) { struct ppc_link_hash_table *htab = ppc_hash_table (info); elf_link_hash_traverse (&htab->elf, undo_symbol_twiddle, info); } /* What to do when ld finds relocations against symbols defined in discarded sections. */ static unsigned int ppc64_elf_action_discarded (asection *sec) { if (strcmp (".opd", sec->name) == 0) return 0; if (strcmp (".toc", sec->name) == 0) return 0; if (strcmp (".toc1", sec->name) == 0) return 0; if (sec->flags & SEC_RELOC) return 0; return _bfd_elf_default_action_discarded (sec); } /* The RELOCATE_SECTION function is called by the ELF backend linker to handle the relocations for a section. The relocs are always passed as Rela structures; if the section actually uses Rel structures, the r_addend field will always be zero. This function is responsible for adjust the section contents as necessary, and (if using Rela relocs and generating a relocatable output file) adjusting the reloc addend as necessary. This function does not have to worry about setting the reloc address or the reloc symbol index. LOCAL_SYMS is a pointer to the swapped in local symbols. LOCAL_SECTIONS is an array giving the section in the input file corresponding to the st_shndx field of each local symbol. The global hash table entry for the global symbols can be found via elf_sym_hashes (input_bfd). When generating relocatable output, this function must handle STB_LOCAL/STT_SECTION symbols specially. The output symbol is going to be the section symbol corresponding to the output section, which means that the addend must be adjusted accordingly. */ static bfd_boolean ppc64_elf_relocate_section (bfd *output_bfd, struct bfd_link_info *info, bfd *input_bfd, asection *input_section, bfd_byte *contents, Elf_Internal_Rela *relocs, Elf_Internal_Sym *local_syms, asection **local_sections) { struct ppc_link_hash_table *htab; Elf_Internal_Shdr *symtab_hdr; struct elf_link_hash_entry **sym_hashes; Elf_Internal_Rela *rel; Elf_Internal_Rela *relend; Elf_Internal_Rela outrel; bfd_byte *loc; struct got_entry **local_got_ents; bfd_vma TOCstart; bfd_boolean ret = TRUE; bfd_boolean is_opd; /* Disabled until we sort out how ld should choose 'y' vs 'at'. */ bfd_boolean is_power4 = FALSE; bfd_vma d_offset = (bfd_big_endian (output_bfd) ? 2 : 0); /* Initialize howto table if needed. */ if (!ppc64_elf_howto_table[R_PPC64_ADDR32]) ppc_howto_init (); htab = ppc_hash_table (info); /* Don't relocate stub sections. */ if (input_section->owner == htab->stub_bfd) return TRUE; local_got_ents = elf_local_got_ents (input_bfd); TOCstart = elf_gp (output_bfd); symtab_hdr = &elf_tdata (input_bfd)->symtab_hdr; sym_hashes = elf_sym_hashes (input_bfd); is_opd = ppc64_elf_section_data (input_section)->sec_type == sec_opd; rel = relocs; relend = relocs + input_section->reloc_count; for (; rel < relend; rel++) { enum elf_ppc64_reloc_type r_type; bfd_vma addend, orig_addend; bfd_reloc_status_type r; Elf_Internal_Sym *sym; asection *sec; struct elf_link_hash_entry *h_elf; struct ppc_link_hash_entry *h; struct ppc_link_hash_entry *fdh; const char *sym_name; unsigned long r_symndx, toc_symndx; + bfd_vma toc_addend; char tls_mask, tls_gd, tls_type; char sym_type; bfd_vma relocation; bfd_boolean unresolved_reloc; bfd_boolean warned; unsigned long insn, mask; struct ppc_stub_hash_entry *stub_entry; bfd_vma max_br_offset; bfd_vma from; r_type = ELF64_R_TYPE (rel->r_info); r_symndx = ELF64_R_SYM (rel->r_info); /* For old style R_PPC64_TOC relocs with a zero symbol, use the symbol of the previous ADDR64 reloc. The symbol gives us the proper TOC base to use. */ if (rel->r_info == ELF64_R_INFO (0, R_PPC64_TOC) && rel != relocs && ELF64_R_TYPE (rel[-1].r_info) == R_PPC64_ADDR64 && is_opd) r_symndx = ELF64_R_SYM (rel[-1].r_info); sym = NULL; sec = NULL; h_elf = NULL; sym_name = NULL; unresolved_reloc = FALSE; warned = FALSE; orig_addend = rel->r_addend; if (r_symndx < symtab_hdr->sh_info) { /* It's a local symbol. */ long *opd_adjust; sym = local_syms + r_symndx; sec = local_sections[r_symndx]; sym_name = bfd_elf_sym_name (input_bfd, symtab_hdr, sym, sec); sym_type = ELF64_ST_TYPE (sym->st_info); relocation = _bfd_elf_rela_local_sym (output_bfd, sym, &sec, rel); opd_adjust = get_opd_info (sec); if (opd_adjust != NULL) { long adjust = opd_adjust[(sym->st_value + rel->r_addend) / 8]; if (adjust == -1) relocation = 0; else { /* If this is a relocation against the opd section sym and we have edited .opd, adjust the reloc addend so that ld -r and ld --emit-relocs output is correct. If it is a reloc against some other .opd symbol, then the symbol value will be adjusted later. */ if (ELF_ST_TYPE (sym->st_info) == STT_SECTION) rel->r_addend += adjust; else relocation += adjust; } } } else { RELOC_FOR_GLOBAL_SYMBOL (info, input_bfd, input_section, rel, r_symndx, symtab_hdr, sym_hashes, h_elf, sec, relocation, unresolved_reloc, warned); sym_name = h_elf->root.root.string; sym_type = h_elf->type; } h = (struct ppc_link_hash_entry *) h_elf; if (sec != NULL && elf_discarded_section (sec)) { /* For relocs against symbols from removed linkonce sections, or sections discarded by a linker script, we just want the section contents zeroed. Avoid any special processing. */ _bfd_clear_contents (ppc64_elf_howto_table[r_type], input_bfd, contents + rel->r_offset); rel->r_info = 0; rel->r_addend = 0; continue; } if (info->relocatable) continue; /* TLS optimizations. Replace instruction sequences and relocs based on information we collected in tls_optimize. We edit RELOCS so that --emit-relocs will output something sensible for the final instruction stream. */ tls_mask = 0; tls_gd = 0; toc_symndx = 0; - if (IS_PPC64_TLS_RELOC (r_type)) + if (h != NULL) + tls_mask = h->tls_mask; + else if (local_got_ents != NULL) { - if (h != NULL) - tls_mask = h->tls_mask; - else if (local_got_ents != NULL) - { - char *lgot_masks; - lgot_masks = (char *) (local_got_ents + symtab_hdr->sh_info); - tls_mask = lgot_masks[r_symndx]; - } - if (tls_mask == 0 && r_type == R_PPC64_TLS) - { - /* Check for toc tls entries. */ - char *toc_tls; + char *lgot_masks; + lgot_masks = (char *) (local_got_ents + symtab_hdr->sh_info); + tls_mask = lgot_masks[r_symndx]; + } + if (tls_mask == 0 + && (r_type == R_PPC64_TLS + || r_type == R_PPC64_TLSGD + || r_type == R_PPC64_TLSLD)) + { + /* Check for toc tls entries. */ + char *toc_tls; - if (!get_tls_mask (&toc_tls, &toc_symndx, &local_syms, - rel, input_bfd)) - return FALSE; + if (!get_tls_mask (&toc_tls, &toc_symndx, &toc_addend, + &local_syms, rel, input_bfd)) + return FALSE; - if (toc_tls) - tls_mask = *toc_tls; - } + if (toc_tls) + tls_mask = *toc_tls; } /* Check that tls relocs are used with tls syms, and non-tls relocs are used with non-tls syms. */ if (r_symndx != 0 && r_type != R_PPC64_NONE && (h == NULL || h->elf.root.type == bfd_link_hash_defined || h->elf.root.type == bfd_link_hash_defweak) - && IS_PPC64_TLS_RELOC (r_type) != (sym_type == STT_TLS)) + && (IS_PPC64_TLS_RELOC (r_type) + != (sym_type == STT_TLS + || (sym_type == STT_SECTION + && (sec->flags & SEC_THREAD_LOCAL) != 0)))) { - if (r_type == R_PPC64_TLS && tls_mask != 0) + if (tls_mask != 0 + && (r_type == R_PPC64_TLS + || r_type == R_PPC64_TLSGD + || r_type == R_PPC64_TLSLD)) /* R_PPC64_TLS is OK against a symbol in the TOC. */ ; else (*_bfd_error_handler) - (sym_type == STT_TLS + (!IS_PPC64_TLS_RELOC (r_type) ? _("%B(%A+0x%lx): %s used with TLS symbol %s") : _("%B(%A+0x%lx): %s used with non-TLS symbol %s"), input_bfd, input_section, (long) rel->r_offset, ppc64_elf_howto_table[r_type]->name, sym_name); } /* Ensure reloc mapping code below stays sane. */ if (R_PPC64_TOC16_LO_DS != R_PPC64_TOC16_DS + 1 || R_PPC64_TOC16_LO != R_PPC64_TOC16 + 1 || (R_PPC64_GOT_TLSLD16 & 3) != (R_PPC64_GOT_TLSGD16 & 3) || (R_PPC64_GOT_TLSLD16_LO & 3) != (R_PPC64_GOT_TLSGD16_LO & 3) || (R_PPC64_GOT_TLSLD16_HI & 3) != (R_PPC64_GOT_TLSGD16_HI & 3) || (R_PPC64_GOT_TLSLD16_HA & 3) != (R_PPC64_GOT_TLSGD16_HA & 3) || (R_PPC64_GOT_TLSLD16 & 3) != (R_PPC64_GOT_TPREL16_DS & 3) || (R_PPC64_GOT_TLSLD16_LO & 3) != (R_PPC64_GOT_TPREL16_LO_DS & 3) || (R_PPC64_GOT_TLSLD16_HI & 3) != (R_PPC64_GOT_TPREL16_HI & 3) || (R_PPC64_GOT_TLSLD16_HA & 3) != (R_PPC64_GOT_TPREL16_HA & 3)) abort (); switch (r_type) { default: break; case R_PPC64_TOC16: case R_PPC64_TOC16_LO: case R_PPC64_TOC16_DS: case R_PPC64_TOC16_LO_DS: { /* Check for toc tls entries. */ char *toc_tls; int retval; - retval = get_tls_mask (&toc_tls, &toc_symndx, &local_syms, - rel, input_bfd); + retval = get_tls_mask (&toc_tls, &toc_symndx, &toc_addend, + &local_syms, rel, input_bfd); if (retval == 0) return FALSE; if (toc_tls) { tls_mask = *toc_tls; if (r_type == R_PPC64_TOC16_DS || r_type == R_PPC64_TOC16_LO_DS) { if (tls_mask != 0 && (tls_mask & (TLS_DTPREL | TLS_TPREL)) == 0) goto toctprel; } else { /* If we found a GD reloc pair, then we might be doing a GD->IE transition. */ if (retval == 2) { tls_gd = TLS_TPRELGD; if (tls_mask != 0 && (tls_mask & TLS_GD) == 0) - goto tls_get_addr_check; + goto tls_ldgd_opt; } else if (retval == 3) { if (tls_mask != 0 && (tls_mask & TLS_LD) == 0) - goto tls_get_addr_check; + goto tls_ldgd_opt; } } } } break; case R_PPC64_GOT_TPREL16_DS: case R_PPC64_GOT_TPREL16_LO_DS: if (tls_mask != 0 && (tls_mask & TLS_TPREL) == 0) { toctprel: insn = bfd_get_32 (output_bfd, contents + rel->r_offset - d_offset); insn &= 31 << 21; insn |= 0x3c0d0000; /* addis 0,13,0 */ bfd_put_32 (output_bfd, insn, contents + rel->r_offset - d_offset); r_type = R_PPC64_TPREL16_HA; if (toc_symndx != 0) { rel->r_info = ELF64_R_INFO (toc_symndx, r_type); /* We changed the symbol. Start over in order to get h, sym, sec etc. right. */ rel--; continue; } else rel->r_info = ELF64_R_INFO (r_symndx, r_type); } break; case R_PPC64_TLS: if (tls_mask != 0 && (tls_mask & TLS_TPREL) == 0) { bfd_vma rtra; insn = bfd_get_32 (output_bfd, contents + rel->r_offset); if ((insn & ((0x3f << 26) | (31 << 11))) == ((31 << 26) | (13 << 11))) rtra = insn & ((1 << 26) - (1 << 16)); else if ((insn & ((0x3f << 26) | (31 << 16))) == ((31 << 26) | (13 << 16))) rtra = (insn & (31 << 21)) | ((insn & (31 << 11)) << 5); else abort (); if ((insn & ((1 << 11) - (1 << 1))) == 266 << 1) /* add -> addi. */ insn = 14 << 26; else if ((insn & (31 << 1)) == 23 << 1 && ((insn & (31 << 6)) < 14 << 6 || ((insn & (31 << 6)) >= 16 << 6 && (insn & (31 << 6)) < 24 << 6))) /* load and store indexed -> dform. */ insn = (32 | ((insn >> 6) & 31)) << 26; else if ((insn & (31 << 1)) == 21 << 1 && (insn & (0x1a << 6)) == 0) /* ldx, ldux, stdx, stdux -> ld, ldu, std, stdu. */ insn = (((58 | ((insn >> 6) & 4)) << 26) | ((insn >> 6) & 1)); else if ((insn & (31 << 1)) == 21 << 1 && (insn & ((1 << 11) - (1 << 1))) == 341 << 1) /* lwax -> lwa. */ insn = (58 << 26) | 2; else abort (); insn |= rtra; bfd_put_32 (output_bfd, insn, contents + rel->r_offset); /* Was PPC64_TLS which sits on insn boundary, now PPC64_TPREL16_LO which is at low-order half-word. */ rel->r_offset += d_offset; r_type = R_PPC64_TPREL16_LO; if (toc_symndx != 0) { rel->r_info = ELF64_R_INFO (toc_symndx, r_type); + rel->r_addend = toc_addend; + rel->r_addend = toc_addend; /* We changed the symbol. Start over in order to get h, sym, sec etc. right. */ rel--; continue; } else rel->r_info = ELF64_R_INFO (r_symndx, r_type); } break; case R_PPC64_GOT_TLSGD16_HI: case R_PPC64_GOT_TLSGD16_HA: tls_gd = TLS_TPRELGD; if (tls_mask != 0 && (tls_mask & TLS_GD) == 0) goto tls_gdld_hi; break; case R_PPC64_GOT_TLSLD16_HI: case R_PPC64_GOT_TLSLD16_HA: if (tls_mask != 0 && (tls_mask & TLS_LD) == 0) { tls_gdld_hi: if ((tls_mask & tls_gd) != 0) r_type = (((r_type - (R_PPC64_GOT_TLSGD16 & 3)) & 3) + R_PPC64_GOT_TPREL16_DS); else { bfd_put_32 (output_bfd, NOP, contents + rel->r_offset); rel->r_offset -= d_offset; r_type = R_PPC64_NONE; } rel->r_info = ELF64_R_INFO (r_symndx, r_type); } break; case R_PPC64_GOT_TLSGD16: case R_PPC64_GOT_TLSGD16_LO: tls_gd = TLS_TPRELGD; if (tls_mask != 0 && (tls_mask & TLS_GD) == 0) - goto tls_get_addr_check; + goto tls_ldgd_opt; break; case R_PPC64_GOT_TLSLD16: case R_PPC64_GOT_TLSLD16_LO: if (tls_mask != 0 && (tls_mask & TLS_LD) == 0) { - tls_get_addr_check: - if (rel + 1 < relend) - { - enum elf_ppc64_reloc_type r_type2; - unsigned long r_symndx2; - struct elf_link_hash_entry *h2; - bfd_vma insn1, insn2, insn3; - bfd_vma offset; + unsigned int insn1, insn2, insn3; + bfd_vma offset; - /* The next instruction should be a call to - __tls_get_addr. Peek at the reloc to be sure. */ - r_type2 = ELF64_R_TYPE (rel[1].r_info); - r_symndx2 = ELF64_R_SYM (rel[1].r_info); - if (r_symndx2 < symtab_hdr->sh_info - || (r_type2 != R_PPC64_REL14 - && r_type2 != R_PPC64_REL14_BRTAKEN - && r_type2 != R_PPC64_REL14_BRNTAKEN - && r_type2 != R_PPC64_REL24)) - break; - - h2 = sym_hashes[r_symndx2 - symtab_hdr->sh_info]; - while (h2->root.type == bfd_link_hash_indirect - || h2->root.type == bfd_link_hash_warning) - h2 = (struct elf_link_hash_entry *) h2->root.u.i.link; - if (h2 == NULL || (h2 != &htab->tls_get_addr->elf - && h2 != &htab->tls_get_addr_fd->elf)) - break; - - /* OK, it checks out. Replace the call. */ - offset = rel[1].r_offset; + tls_ldgd_opt: + offset = (bfd_vma) -1; + /* If not using the newer R_PPC64_TLSGD/LD to mark + __tls_get_addr calls, we must trust that the call + stays with its arg setup insns, ie. that the next + reloc is the __tls_get_addr call associated with + the current reloc. Edit both insns. */ + if (input_section->has_tls_get_addr_call + && rel + 1 < relend + && branch_reloc_hash_match (input_bfd, rel + 1, + htab->tls_get_addr, + htab->tls_get_addr_fd)) + offset = rel[1].r_offset; + if ((tls_mask & tls_gd) != 0) + { + /* IE */ insn1 = bfd_get_32 (output_bfd, contents + rel->r_offset - d_offset); - insn3 = bfd_get_32 (output_bfd, - contents + offset + 4); - if ((tls_mask & tls_gd) != 0) + insn1 &= (1 << 26) - (1 << 2); + insn1 |= 58 << 26; /* ld */ + insn2 = 0x7c636a14; /* add 3,3,13 */ + if (offset != (bfd_vma) -1) + rel[1].r_info = ELF64_R_INFO (ELF64_R_SYM (rel[1].r_info), + R_PPC64_NONE); + if ((tls_mask & TLS_EXPLICIT) == 0) + r_type = (((r_type - (R_PPC64_GOT_TLSGD16 & 3)) & 3) + + R_PPC64_GOT_TPREL16_DS); + else + r_type += R_PPC64_TOC16_DS - R_PPC64_TOC16; + rel->r_info = ELF64_R_INFO (r_symndx, r_type); + } + else + { + /* LE */ + insn1 = 0x3c6d0000; /* addis 3,13,0 */ + insn2 = 0x38630000; /* addi 3,3,0 */ + if (tls_gd == 0) { - /* IE */ - insn1 &= (1 << 26) - (1 << 2); - insn1 |= 58 << 26; /* ld */ - insn2 = 0x7c636a14; /* add 3,3,13 */ - rel[1].r_info = ELF64_R_INFO (r_symndx2, R_PPC64_NONE); - if ((tls_mask & TLS_EXPLICIT) == 0) - r_type = (((r_type - (R_PPC64_GOT_TLSGD16 & 3)) & 3) - + R_PPC64_GOT_TPREL16_DS); - else - r_type += R_PPC64_TOC16_DS - R_PPC64_TOC16; - rel->r_info = ELF64_R_INFO (r_symndx, r_type); + /* Was an LD reloc. */ + if (toc_symndx) + sec = local_sections[toc_symndx]; + for (r_symndx = 0; + r_symndx < symtab_hdr->sh_info; + r_symndx++) + if (local_sections[r_symndx] == sec) + break; + if (r_symndx >= symtab_hdr->sh_info) + r_symndx = 0; + rel->r_addend = htab->elf.tls_sec->vma + DTP_OFFSET; + if (r_symndx != 0) + rel->r_addend -= (local_syms[r_symndx].st_value + + sec->output_offset + + sec->output_section->vma); } - else + else if (toc_symndx != 0) { - /* LE */ - insn1 = 0x3c6d0000; /* addis 3,13,0 */ - insn2 = 0x38630000; /* addi 3,3,0 */ - if (tls_gd == 0) - { - /* Was an LD reloc. */ - r_symndx = 0; - rel->r_addend = htab->elf.tls_sec->vma + DTP_OFFSET; - rel[1].r_addend = htab->elf.tls_sec->vma + DTP_OFFSET; - } - else if (toc_symndx != 0) - r_symndx = toc_symndx; - r_type = R_PPC64_TPREL16_HA; - rel->r_info = ELF64_R_INFO (r_symndx, r_type); + r_symndx = toc_symndx; + rel->r_addend = toc_addend; + } + r_type = R_PPC64_TPREL16_HA; + rel->r_info = ELF64_R_INFO (r_symndx, r_type); + if (offset != (bfd_vma) -1) + { rel[1].r_info = ELF64_R_INFO (r_symndx, R_PPC64_TPREL16_LO); - rel[1].r_offset += d_offset; + rel[1].r_offset = offset + d_offset; + rel[1].r_addend = rel->r_addend; } + } + bfd_put_32 (output_bfd, insn1, + contents + rel->r_offset - d_offset); + if (offset != (bfd_vma) -1) + { + insn3 = bfd_get_32 (output_bfd, + contents + offset + 4); if (insn3 == NOP || insn3 == CROR_151515 || insn3 == CROR_313131) { - insn3 = insn2; - insn2 = NOP; rel[1].r_offset += 4; + bfd_put_32 (output_bfd, insn2, contents + offset + 4); + insn2 = NOP; } - bfd_put_32 (output_bfd, insn1, contents + rel->r_offset - d_offset); bfd_put_32 (output_bfd, insn2, contents + offset); - bfd_put_32 (output_bfd, insn3, contents + offset + 4); - if (tls_gd == 0 || toc_symndx != 0) + } + if ((tls_mask & tls_gd) == 0 + && (tls_gd == 0 || toc_symndx != 0)) + { + /* We changed the symbol. Start over in order + to get h, sym, sec etc. right. */ + rel--; + continue; + } + } + break; + + case R_PPC64_TLSGD: + if (tls_mask != 0 && (tls_mask & TLS_GD) == 0) + { + unsigned int insn2, insn3; + bfd_vma offset = rel->r_offset; + + if ((tls_mask & TLS_TPRELGD) != 0) + { + /* IE */ + r_type = R_PPC64_NONE; + insn2 = 0x7c636a14; /* add 3,3,13 */ + } + else + { + /* LE */ + if (toc_symndx != 0) { - /* We changed the symbol. Start over in order - to get h, sym, sec etc. right. */ - rel--; - continue; + r_symndx = toc_symndx; + rel->r_addend = toc_addend; } + r_type = R_PPC64_TPREL16_LO; + rel->r_offset = offset + d_offset; + insn2 = 0x38630000; /* addi 3,3,0 */ } + rel->r_info = ELF64_R_INFO (r_symndx, r_type); + /* Zap the reloc on the _tls_get_addr call too. */ + BFD_ASSERT (offset == rel[1].r_offset); + rel[1].r_info = ELF64_R_INFO (ELF64_R_SYM (rel[1].r_info), + R_PPC64_NONE); + insn3 = bfd_get_32 (output_bfd, + contents + offset + 4); + if (insn3 == NOP + || insn3 == CROR_151515 || insn3 == CROR_313131) + { + rel->r_offset += 4; + bfd_put_32 (output_bfd, insn2, contents + offset + 4); + insn2 = NOP; + } + bfd_put_32 (output_bfd, insn2, contents + offset); + if ((tls_mask & TLS_TPRELGD) == 0 && toc_symndx != 0) + { + rel--; + continue; + } } break; + case R_PPC64_TLSLD: + if (tls_mask != 0 && (tls_mask & TLS_LD) == 0) + { + unsigned int insn2, insn3; + bfd_vma offset = rel->r_offset; + + if (toc_symndx) + sec = local_sections[toc_symndx]; + for (r_symndx = 0; + r_symndx < symtab_hdr->sh_info; + r_symndx++) + if (local_sections[r_symndx] == sec) + break; + if (r_symndx >= symtab_hdr->sh_info) + r_symndx = 0; + rel->r_addend = htab->elf.tls_sec->vma + DTP_OFFSET; + if (r_symndx != 0) + rel->r_addend -= (local_syms[r_symndx].st_value + + sec->output_offset + + sec->output_section->vma); + + r_type = R_PPC64_TPREL16_LO; + rel->r_info = ELF64_R_INFO (r_symndx, r_type); + rel->r_offset = offset + d_offset; + /* Zap the reloc on the _tls_get_addr call too. */ + BFD_ASSERT (offset == rel[1].r_offset); + rel[1].r_info = ELF64_R_INFO (ELF64_R_SYM (rel[1].r_info), + R_PPC64_NONE); + insn2 = 0x38630000; /* addi 3,3,0 */ + insn3 = bfd_get_32 (output_bfd, + contents + offset + 4); + if (insn3 == NOP + || insn3 == CROR_151515 || insn3 == CROR_313131) + { + rel->r_offset += 4; + bfd_put_32 (output_bfd, insn2, contents + offset + 4); + insn2 = NOP; + } + bfd_put_32 (output_bfd, insn2, contents + offset); + rel--; + continue; + } + break; + case R_PPC64_DTPMOD64: if (rel + 1 < relend && rel[1].r_info == ELF64_R_INFO (r_symndx, R_PPC64_DTPREL64) && rel[1].r_offset == rel->r_offset + 8) { if ((tls_mask & TLS_GD) == 0) { rel[1].r_info = ELF64_R_INFO (r_symndx, R_PPC64_NONE); if ((tls_mask & TLS_TPRELGD) != 0) r_type = R_PPC64_TPREL64; else { bfd_put_64 (output_bfd, 1, contents + rel->r_offset); r_type = R_PPC64_NONE; } rel->r_info = ELF64_R_INFO (r_symndx, r_type); } } else { if ((tls_mask & TLS_LD) == 0) { bfd_put_64 (output_bfd, 1, contents + rel->r_offset); r_type = R_PPC64_NONE; rel->r_info = ELF64_R_INFO (r_symndx, r_type); } } break; case R_PPC64_TPREL64: if ((tls_mask & TLS_TPREL) == 0) { r_type = R_PPC64_NONE; rel->r_info = ELF64_R_INFO (r_symndx, r_type); } break; } /* Handle other relocations that tweak non-addend part of insn. */ insn = 0; max_br_offset = 1 << 25; addend = rel->r_addend; switch (r_type) { default: break; /* Branch taken prediction relocations. */ case R_PPC64_ADDR14_BRTAKEN: case R_PPC64_REL14_BRTAKEN: insn = 0x01 << 21; /* 'y' or 't' bit, lowest bit of BO field. */ /* Fall thru. */ /* Branch not taken prediction relocations. */ case R_PPC64_ADDR14_BRNTAKEN: case R_PPC64_REL14_BRNTAKEN: insn |= bfd_get_32 (output_bfd, contents + rel->r_offset) & ~(0x01 << 21); /* Fall thru. */ case R_PPC64_REL14: max_br_offset = 1 << 15; /* Fall thru. */ case R_PPC64_REL24: /* Calls to functions with a different TOC, such as calls to shared objects, need to alter the TOC pointer. This is done using a linkage stub. A REL24 branching to these linkage stubs needs to be followed by a nop, as the nop will be replaced with an instruction to restore the TOC base pointer. */ stub_entry = NULL; fdh = h; if (((h != NULL && (((fdh = h->oh) != NULL && fdh->elf.plt.plist != NULL) || (fdh = h)->elf.plt.plist != NULL)) || (sec != NULL && sec->output_section != NULL && sec->id <= htab->top_id && (htab->stub_group[sec->id].toc_off != htab->stub_group[input_section->id].toc_off))) && (stub_entry = ppc_get_stub_entry (input_section, sec, fdh, rel, htab)) != NULL && (stub_entry->stub_type == ppc_stub_plt_call || stub_entry->stub_type == ppc_stub_plt_branch_r2off || stub_entry->stub_type == ppc_stub_long_branch_r2off)) { bfd_boolean can_plt_call = FALSE; if (rel->r_offset + 8 <= input_section->size) { unsigned long nop; nop = bfd_get_32 (input_bfd, contents + rel->r_offset + 4); if (nop == NOP || nop == CROR_151515 || nop == CROR_313131) { bfd_put_32 (input_bfd, LD_R2_40R1, contents + rel->r_offset + 4); can_plt_call = TRUE; } } if (!can_plt_call) { if (stub_entry->stub_type == ppc_stub_plt_call) { /* If this is a plain branch rather than a branch and link, don't require a nop. However, don't allow tail calls in a shared library as they will result in r2 being corrupted. */ unsigned long br; br = bfd_get_32 (input_bfd, contents + rel->r_offset); if (info->executable && (br & 1) == 0) can_plt_call = TRUE; else stub_entry = NULL; } else if (h != NULL && strcmp (h->elf.root.root.string, ".__libc_start_main") == 0) { /* Allow crt1 branch to go via a toc adjusting stub. */ can_plt_call = TRUE; } else { if (strcmp (input_section->output_section->name, ".init") == 0 || strcmp (input_section->output_section->name, ".fini") == 0) (*_bfd_error_handler) (_("%B(%A+0x%lx): automatic multiple TOCs " "not supported using your crt files; " "recompile with -mminimal-toc or upgrade gcc"), input_bfd, input_section, (long) rel->r_offset); else (*_bfd_error_handler) (_("%B(%A+0x%lx): sibling call optimization to `%s' " "does not allow automatic multiple TOCs; " "recompile with -mminimal-toc or " "-fno-optimize-sibling-calls, " "or make `%s' extern"), input_bfd, input_section, (long) rel->r_offset, sym_name, sym_name); bfd_set_error (bfd_error_bad_value); ret = FALSE; } } if (can_plt_call && stub_entry->stub_type == ppc_stub_plt_call) unresolved_reloc = FALSE; } if (stub_entry == NULL && get_opd_info (sec) != NULL) { /* The branch destination is the value of the opd entry. */ bfd_vma off = (relocation + addend - sec->output_section->vma - sec->output_offset); bfd_vma dest = opd_entry_value (sec, off, NULL, NULL); if (dest != (bfd_vma) -1) { relocation = dest; addend = 0; } } /* If the branch is out of reach we ought to have a long branch stub. */ from = (rel->r_offset + input_section->output_offset + input_section->output_section->vma); if (stub_entry == NULL && (relocation + addend - from + max_br_offset >= 2 * max_br_offset) && r_type != R_PPC64_ADDR14_BRTAKEN && r_type != R_PPC64_ADDR14_BRNTAKEN) stub_entry = ppc_get_stub_entry (input_section, sec, h, rel, htab); if (stub_entry != NULL) { /* Munge up the value and addend so that we call the stub rather than the procedure directly. */ relocation = (stub_entry->stub_offset + stub_entry->stub_sec->output_offset + stub_entry->stub_sec->output_section->vma); addend = 0; } if (insn != 0) { if (is_power4) { /* Set 'a' bit. This is 0b00010 in BO field for branch on CR(BI) insns (BO == 001at or 011at), and 0b01000 for branch on CTR insns (BO == 1a00t or 1a01t). */ if ((insn & (0x14 << 21)) == (0x04 << 21)) insn |= 0x02 << 21; else if ((insn & (0x14 << 21)) == (0x10 << 21)) insn |= 0x08 << 21; else break; } else { /* Invert 'y' bit if not the default. */ if ((bfd_signed_vma) (relocation + addend - from) < 0) insn ^= 0x01 << 21; } bfd_put_32 (output_bfd, insn, contents + rel->r_offset); } /* NOP out calls to undefined weak functions. We can thus call a weak function without first checking whether the function is defined. */ else if (h != NULL && h->elf.root.type == bfd_link_hash_undefweak && r_type == R_PPC64_REL24 && relocation == 0 && addend == 0) { bfd_put_32 (output_bfd, NOP, contents + rel->r_offset); continue; } break; } /* Set `addend'. */ tls_type = 0; switch (r_type) { default: (*_bfd_error_handler) (_("%B: unknown relocation type %d for symbol %s"), input_bfd, (int) r_type, sym_name); bfd_set_error (bfd_error_bad_value); ret = FALSE; continue; case R_PPC64_NONE: case R_PPC64_TLS: + case R_PPC64_TLSGD: + case R_PPC64_TLSLD: case R_PPC64_GNU_VTINHERIT: case R_PPC64_GNU_VTENTRY: continue; /* GOT16 relocations. Like an ADDR16 using the symbol's address in the GOT as relocation value instead of the symbol's value itself. Also, create a GOT entry for the symbol and put the symbol value there. */ case R_PPC64_GOT_TLSGD16: case R_PPC64_GOT_TLSGD16_LO: case R_PPC64_GOT_TLSGD16_HI: case R_PPC64_GOT_TLSGD16_HA: tls_type = TLS_TLS | TLS_GD; goto dogot; case R_PPC64_GOT_TLSLD16: case R_PPC64_GOT_TLSLD16_LO: case R_PPC64_GOT_TLSLD16_HI: case R_PPC64_GOT_TLSLD16_HA: tls_type = TLS_TLS | TLS_LD; goto dogot; case R_PPC64_GOT_TPREL16_DS: case R_PPC64_GOT_TPREL16_LO_DS: case R_PPC64_GOT_TPREL16_HI: case R_PPC64_GOT_TPREL16_HA: tls_type = TLS_TLS | TLS_TPREL; goto dogot; case R_PPC64_GOT_DTPREL16_DS: case R_PPC64_GOT_DTPREL16_LO_DS: case R_PPC64_GOT_DTPREL16_HI: case R_PPC64_GOT_DTPREL16_HA: tls_type = TLS_TLS | TLS_DTPREL; goto dogot; case R_PPC64_GOT16: case R_PPC64_GOT16_LO: case R_PPC64_GOT16_HI: case R_PPC64_GOT16_HA: case R_PPC64_GOT16_DS: case R_PPC64_GOT16_LO_DS: dogot: { /* Relocation is to the entry for this symbol in the global offset table. */ asection *got; bfd_vma *offp; bfd_vma off; unsigned long indx = 0; if (tls_type == (TLS_TLS | TLS_LD) && (h == NULL || !h->elf.def_dynamic)) offp = &ppc64_tlsld_got (input_bfd)->offset; else { struct got_entry *ent; if (h != NULL) { bfd_boolean dyn = htab->elf.dynamic_sections_created; if (!WILL_CALL_FINISH_DYNAMIC_SYMBOL (dyn, info->shared, &h->elf) || (info->shared && SYMBOL_REFERENCES_LOCAL (info, &h->elf))) /* This is actually a static link, or it is a -Bsymbolic link and the symbol is defined locally, or the symbol was forced to be local because of a version file. */ ; else { indx = h->elf.dynindx; unresolved_reloc = FALSE; } ent = h->elf.got.glist; } else { if (local_got_ents == NULL) abort (); ent = local_got_ents[r_symndx]; } for (; ent != NULL; ent = ent->next) if (ent->addend == orig_addend && ent->owner == input_bfd && ent->tls_type == tls_type) break; if (ent == NULL) abort (); offp = &ent->got.offset; } got = ppc64_elf_tdata (input_bfd)->got; if (got == NULL) abort (); /* The offset must always be a multiple of 8. We use the least significant bit to record whether we have already processed this entry. */ off = *offp; if ((off & 1) != 0) off &= ~1; else { /* Generate relocs for the dynamic linker, except in the case of TLSLD where we'll use one entry per module. */ asection *relgot = ppc64_elf_tdata (input_bfd)->relgot; *offp = off | 1; if ((info->shared || indx != 0) - && (h == NULL + && (offp == &ppc64_tlsld_got (input_bfd)->offset + || h == NULL || ELF_ST_VISIBILITY (h->elf.other) == STV_DEFAULT || h->elf.root.type != bfd_link_hash_undefweak)) { outrel.r_offset = (got->output_section->vma + got->output_offset + off); outrel.r_addend = addend; if (tls_type & (TLS_LD | TLS_GD)) { outrel.r_addend = 0; outrel.r_info = ELF64_R_INFO (indx, R_PPC64_DTPMOD64); if (tls_type == (TLS_TLS | TLS_GD)) { loc = relgot->contents; loc += (relgot->reloc_count++ * sizeof (Elf64_External_Rela)); bfd_elf64_swap_reloca_out (output_bfd, &outrel, loc); outrel.r_offset += 8; outrel.r_addend = addend; outrel.r_info = ELF64_R_INFO (indx, R_PPC64_DTPREL64); } } else if (tls_type == (TLS_TLS | TLS_DTPREL)) outrel.r_info = ELF64_R_INFO (indx, R_PPC64_DTPREL64); else if (tls_type == (TLS_TLS | TLS_TPREL)) outrel.r_info = ELF64_R_INFO (indx, R_PPC64_TPREL64); else if (indx == 0) { outrel.r_info = ELF64_R_INFO (indx, R_PPC64_RELATIVE); /* Write the .got section contents for the sake of prelink. */ loc = got->contents + off; bfd_put_64 (output_bfd, outrel.r_addend + relocation, loc); } else outrel.r_info = ELF64_R_INFO (indx, R_PPC64_GLOB_DAT); if (indx == 0 && tls_type != (TLS_TLS | TLS_LD)) { outrel.r_addend += relocation; if (tls_type & (TLS_GD | TLS_DTPREL | TLS_TPREL)) outrel.r_addend -= htab->elf.tls_sec->vma; } loc = relgot->contents; loc += (relgot->reloc_count++ * sizeof (Elf64_External_Rela)); bfd_elf64_swap_reloca_out (output_bfd, &outrel, loc); } /* Init the .got section contents here if we're not emitting a reloc. */ else { relocation += addend; if (tls_type == (TLS_TLS | TLS_LD)) relocation = 1; else if (tls_type != 0) { relocation -= htab->elf.tls_sec->vma + DTP_OFFSET; if (tls_type == (TLS_TLS | TLS_TPREL)) relocation += DTP_OFFSET - TP_OFFSET; if (tls_type == (TLS_TLS | TLS_GD)) { bfd_put_64 (output_bfd, relocation, got->contents + off + 8); relocation = 1; } } bfd_put_64 (output_bfd, relocation, got->contents + off); } } if (off >= (bfd_vma) -2) abort (); relocation = got->output_offset + off; /* TOC base (r2) is TOC start plus 0x8000. */ addend = -TOC_BASE_OFF; } break; case R_PPC64_PLT16_HA: case R_PPC64_PLT16_HI: case R_PPC64_PLT16_LO: case R_PPC64_PLT32: case R_PPC64_PLT64: /* Relocation is to the entry for this symbol in the procedure linkage table. */ /* Resolve a PLT reloc against a local symbol directly, without using the procedure linkage table. */ if (h == NULL) break; /* It's possible that we didn't make a PLT entry for this symbol. This happens when statically linking PIC code, or when using -Bsymbolic. Go find a match if there is a PLT entry. */ if (htab->plt != NULL) { struct plt_entry *ent; for (ent = h->elf.plt.plist; ent != NULL; ent = ent->next) if (ent->addend == orig_addend && ent->plt.offset != (bfd_vma) -1) { relocation = (htab->plt->output_section->vma + htab->plt->output_offset + ent->plt.offset); unresolved_reloc = FALSE; } } break; case R_PPC64_TOC: /* Relocation value is TOC base. */ relocation = TOCstart; if (r_symndx == 0) relocation += htab->stub_group[input_section->id].toc_off; else if (unresolved_reloc) ; else if (sec != NULL && sec->id <= htab->top_id) relocation += htab->stub_group[sec->id].toc_off; else unresolved_reloc = TRUE; goto dodyn; /* TOC16 relocs. We want the offset relative to the TOC base, which is the address of the start of the TOC plus 0x8000. The TOC consists of sections .got, .toc, .tocbss, and .plt, in this order. */ case R_PPC64_TOC16: case R_PPC64_TOC16_LO: case R_PPC64_TOC16_HI: case R_PPC64_TOC16_DS: case R_PPC64_TOC16_LO_DS: case R_PPC64_TOC16_HA: addend -= TOCstart + htab->stub_group[input_section->id].toc_off; break; /* Relocate against the beginning of the section. */ case R_PPC64_SECTOFF: case R_PPC64_SECTOFF_LO: case R_PPC64_SECTOFF_HI: case R_PPC64_SECTOFF_DS: case R_PPC64_SECTOFF_LO_DS: case R_PPC64_SECTOFF_HA: if (sec != NULL) addend -= sec->output_section->vma; break; case R_PPC64_REL14: case R_PPC64_REL14_BRNTAKEN: case R_PPC64_REL14_BRTAKEN: case R_PPC64_REL24: break; case R_PPC64_TPREL16: case R_PPC64_TPREL16_LO: case R_PPC64_TPREL16_HI: case R_PPC64_TPREL16_HA: case R_PPC64_TPREL16_DS: case R_PPC64_TPREL16_LO_DS: case R_PPC64_TPREL16_HIGHER: case R_PPC64_TPREL16_HIGHERA: case R_PPC64_TPREL16_HIGHEST: case R_PPC64_TPREL16_HIGHESTA: addend -= htab->elf.tls_sec->vma + TP_OFFSET; if (info->shared) /* The TPREL16 relocs shouldn't really be used in shared libs as they will result in DT_TEXTREL being set, but support them anyway. */ goto dodyn; break; case R_PPC64_DTPREL16: case R_PPC64_DTPREL16_LO: case R_PPC64_DTPREL16_HI: case R_PPC64_DTPREL16_HA: case R_PPC64_DTPREL16_DS: case R_PPC64_DTPREL16_LO_DS: case R_PPC64_DTPREL16_HIGHER: case R_PPC64_DTPREL16_HIGHERA: case R_PPC64_DTPREL16_HIGHEST: case R_PPC64_DTPREL16_HIGHESTA: addend -= htab->elf.tls_sec->vma + DTP_OFFSET; break; case R_PPC64_DTPMOD64: relocation = 1; addend = 0; goto dodyn; case R_PPC64_TPREL64: addend -= htab->elf.tls_sec->vma + TP_OFFSET; goto dodyn; case R_PPC64_DTPREL64: addend -= htab->elf.tls_sec->vma + DTP_OFFSET; /* Fall thru */ /* Relocations that may need to be propagated if this is a dynamic object. */ case R_PPC64_REL30: case R_PPC64_REL32: case R_PPC64_REL64: case R_PPC64_ADDR14: case R_PPC64_ADDR14_BRNTAKEN: case R_PPC64_ADDR14_BRTAKEN: case R_PPC64_ADDR16: case R_PPC64_ADDR16_DS: case R_PPC64_ADDR16_HA: case R_PPC64_ADDR16_HI: case R_PPC64_ADDR16_HIGHER: case R_PPC64_ADDR16_HIGHERA: case R_PPC64_ADDR16_HIGHEST: case R_PPC64_ADDR16_HIGHESTA: case R_PPC64_ADDR16_LO: case R_PPC64_ADDR16_LO_DS: case R_PPC64_ADDR24: case R_PPC64_ADDR32: case R_PPC64_ADDR64: case R_PPC64_UADDR16: case R_PPC64_UADDR32: case R_PPC64_UADDR64: dodyn: if ((input_section->flags & SEC_ALLOC) == 0) break; if (NO_OPD_RELOCS && is_opd) break; if ((info->shared && (h == NULL || ELF_ST_VISIBILITY (h->elf.other) == STV_DEFAULT || h->elf.root.type != bfd_link_hash_undefweak) - && (MUST_BE_DYN_RELOC (r_type) + && (must_be_dyn_reloc (info, r_type) || !SYMBOL_CALLS_LOCAL (info, &h->elf))) || (ELIMINATE_COPY_RELOCS && !info->shared && h != NULL && h->elf.dynindx != -1 && !h->elf.non_got_ref && h->elf.def_dynamic && !h->elf.def_regular)) { Elf_Internal_Rela outrel; bfd_boolean skip, relocate; asection *sreloc; bfd_byte *loc; bfd_vma out_off; /* When generating a dynamic object, these relocations are copied into the output file to be resolved at run time. */ skip = FALSE; relocate = FALSE; out_off = _bfd_elf_section_offset (output_bfd, info, input_section, rel->r_offset); if (out_off == (bfd_vma) -1) skip = TRUE; else if (out_off == (bfd_vma) -2) skip = TRUE, relocate = TRUE; out_off += (input_section->output_section->vma + input_section->output_offset); outrel.r_offset = out_off; outrel.r_addend = rel->r_addend; /* Optimize unaligned reloc use. */ if ((r_type == R_PPC64_ADDR64 && (out_off & 7) != 0) || (r_type == R_PPC64_UADDR64 && (out_off & 7) == 0)) r_type ^= R_PPC64_ADDR64 ^ R_PPC64_UADDR64; else if ((r_type == R_PPC64_ADDR32 && (out_off & 3) != 0) || (r_type == R_PPC64_UADDR32 && (out_off & 3) == 0)) r_type ^= R_PPC64_ADDR32 ^ R_PPC64_UADDR32; else if ((r_type == R_PPC64_ADDR16 && (out_off & 1) != 0) || (r_type == R_PPC64_UADDR16 && (out_off & 1) == 0)) r_type ^= R_PPC64_ADDR16 ^ R_PPC64_UADDR16; if (skip) memset (&outrel, 0, sizeof outrel); else if (!SYMBOL_REFERENCES_LOCAL (info, &h->elf) && !is_opd && r_type != R_PPC64_TOC) outrel.r_info = ELF64_R_INFO (h->elf.dynindx, r_type); else { /* This symbol is local, or marked to become local, or this is an opd section reloc which must point at a local function. */ outrel.r_addend += relocation; if (r_type == R_PPC64_ADDR64 || r_type == R_PPC64_TOC) { if (is_opd && h != NULL) { /* Lie about opd entries. This case occurs when building shared libraries and we reference a function in another shared lib. The same thing happens for a weak definition in an application that's overridden by a strong definition in a shared lib. (I believe this is a generic bug in binutils handling of weak syms.) In these cases we won't use the opd entry in this lib. */ unresolved_reloc = FALSE; } outrel.r_info = ELF64_R_INFO (0, R_PPC64_RELATIVE); /* We need to relocate .opd contents for ld.so. Prelink also wants simple and consistent rules for relocs. This make all RELATIVE relocs have *r_offset equal to r_addend. */ relocate = TRUE; } else { long indx = 0; - if (bfd_is_abs_section (sec)) + if (r_symndx == 0 || bfd_is_abs_section (sec)) ; else if (sec == NULL || sec->owner == NULL) { bfd_set_error (bfd_error_bad_value); return FALSE; } else { asection *osec; osec = sec->output_section; indx = elf_section_data (osec)->dynindx; if (indx == 0) { if ((osec->flags & SEC_READONLY) == 0 && htab->elf.data_index_section != NULL) osec = htab->elf.data_index_section; else osec = htab->elf.text_index_section; indx = elf_section_data (osec)->dynindx; } BFD_ASSERT (indx != 0); /* We are turning this relocation into one against a section symbol, so subtract out the output section's address but not the offset of the input section in the output section. */ outrel.r_addend -= osec->vma; } outrel.r_info = ELF64_R_INFO (indx, r_type); } } sreloc = elf_section_data (input_section)->sreloc; if (sreloc == NULL) abort (); if (sreloc->reloc_count * sizeof (Elf64_External_Rela) >= sreloc->size) abort (); loc = sreloc->contents; loc += sreloc->reloc_count++ * sizeof (Elf64_External_Rela); bfd_elf64_swap_reloca_out (output_bfd, &outrel, loc); /* If this reloc is against an external symbol, it will be computed at runtime, so there's no need to do anything now. However, for the sake of prelink ensure that the section contents are a known value. */ if (! relocate) { unresolved_reloc = FALSE; /* The value chosen here is quite arbitrary as ld.so ignores section contents except for the special case of .opd where the contents might be accessed before relocation. Choose zero, as that won't cause reloc overflow. */ relocation = 0; addend = 0; /* Use *r_offset == r_addend for R_PPC64_ADDR64 relocs to improve backward compatibility with older versions of ld. */ if (r_type == R_PPC64_ADDR64) addend = outrel.r_addend; /* Adjust pc_relative relocs to have zero in *r_offset. */ else if (ppc64_elf_howto_table[r_type]->pc_relative) addend = (input_section->output_section->vma + input_section->output_offset + rel->r_offset); } } break; case R_PPC64_COPY: case R_PPC64_GLOB_DAT: case R_PPC64_JMP_SLOT: case R_PPC64_RELATIVE: /* We shouldn't ever see these dynamic relocs in relocatable files. */ /* Fall through. */ case R_PPC64_PLTGOT16: case R_PPC64_PLTGOT16_DS: case R_PPC64_PLTGOT16_HA: case R_PPC64_PLTGOT16_HI: case R_PPC64_PLTGOT16_LO: case R_PPC64_PLTGOT16_LO_DS: case R_PPC64_PLTREL32: case R_PPC64_PLTREL64: /* These ones haven't been implemented yet. */ (*_bfd_error_handler) (_("%B: relocation %s is not supported for symbol %s."), input_bfd, ppc64_elf_howto_table[r_type]->name, sym_name); bfd_set_error (bfd_error_invalid_operation); ret = FALSE; continue; } /* Do any further special processing. */ switch (r_type) { default: break; case R_PPC64_ADDR16_HA: case R_PPC64_ADDR16_HIGHERA: case R_PPC64_ADDR16_HIGHESTA: case R_PPC64_TOC16_HA: case R_PPC64_SECTOFF_HA: case R_PPC64_TPREL16_HA: case R_PPC64_DTPREL16_HA: case R_PPC64_TPREL16_HIGHER: case R_PPC64_TPREL16_HIGHERA: case R_PPC64_TPREL16_HIGHEST: case R_PPC64_TPREL16_HIGHESTA: case R_PPC64_DTPREL16_HIGHER: case R_PPC64_DTPREL16_HIGHERA: case R_PPC64_DTPREL16_HIGHEST: case R_PPC64_DTPREL16_HIGHESTA: /* It's just possible that this symbol is a weak symbol that's not actually defined anywhere. In that case, 'sec' would be NULL, and we should leave the symbol alone (it will be set to zero elsewhere in the link). */ if (sec == NULL) break; /* Fall thru */ case R_PPC64_GOT16_HA: case R_PPC64_PLTGOT16_HA: case R_PPC64_PLT16_HA: case R_PPC64_GOT_TLSGD16_HA: case R_PPC64_GOT_TLSLD16_HA: case R_PPC64_GOT_TPREL16_HA: case R_PPC64_GOT_DTPREL16_HA: /* Add 0x10000 if sign bit in 0:15 is set. Bits 0:15 are not used. */ addend += 0x8000; break; case R_PPC64_ADDR16_DS: case R_PPC64_ADDR16_LO_DS: case R_PPC64_GOT16_DS: case R_PPC64_GOT16_LO_DS: case R_PPC64_PLT16_LO_DS: case R_PPC64_SECTOFF_DS: case R_PPC64_SECTOFF_LO_DS: case R_PPC64_TOC16_DS: case R_PPC64_TOC16_LO_DS: case R_PPC64_PLTGOT16_DS: case R_PPC64_PLTGOT16_LO_DS: case R_PPC64_GOT_TPREL16_DS: case R_PPC64_GOT_TPREL16_LO_DS: case R_PPC64_GOT_DTPREL16_DS: case R_PPC64_GOT_DTPREL16_LO_DS: case R_PPC64_TPREL16_DS: case R_PPC64_TPREL16_LO_DS: case R_PPC64_DTPREL16_DS: case R_PPC64_DTPREL16_LO_DS: insn = bfd_get_32 (input_bfd, contents + (rel->r_offset & ~3)); mask = 3; /* If this reloc is against an lq insn, then the value must be a multiple of 16. This is somewhat of a hack, but the "correct" way to do this by defining _DQ forms of all the _DS relocs bloats all reloc switches in this file. It doesn't seem to make much sense to use any of these relocs in data, so testing the insn should be safe. */ if ((insn & (0x3f << 26)) == (56u << 26)) mask = 15; if (((relocation + addend) & mask) != 0) { (*_bfd_error_handler) (_("%B: error: relocation %s not a multiple of %d"), input_bfd, ppc64_elf_howto_table[r_type]->name, mask + 1); bfd_set_error (bfd_error_bad_value); ret = FALSE; continue; } break; } /* Dynamic relocs are not propagated for SEC_DEBUGGING sections because such sections are not SEC_ALLOC and thus ld.so will not process them. */ if (unresolved_reloc && !((input_section->flags & SEC_DEBUGGING) != 0 && h->elf.def_dynamic)) { (*_bfd_error_handler) (_("%B(%A+0x%lx): unresolvable %s relocation against symbol `%s'"), input_bfd, input_section, (long) rel->r_offset, ppc64_elf_howto_table[(int) r_type]->name, h->elf.root.root.string); ret = FALSE; } r = _bfd_final_link_relocate (ppc64_elf_howto_table[(int) r_type], input_bfd, input_section, contents, rel->r_offset, relocation, addend); if (r != bfd_reloc_ok) { if (sym_name == NULL) sym_name = "(null)"; if (r == bfd_reloc_overflow) { if (warned) continue; if (h != NULL && h->elf.root.type == bfd_link_hash_undefweak && ppc64_elf_howto_table[r_type]->pc_relative) { /* Assume this is a call protected by other code that detects the symbol is undefined. If this is the case, we can safely ignore the overflow. If not, the program is hosed anyway, and a little warning isn't going to help. */ continue; } if (!((*info->callbacks->reloc_overflow) (info, (h ? &h->elf.root : NULL), sym_name, ppc64_elf_howto_table[r_type]->name, orig_addend, input_bfd, input_section, rel->r_offset))) return FALSE; } else { (*_bfd_error_handler) (_("%B(%A+0x%lx): %s reloc against `%s': error %d"), input_bfd, input_section, (long) rel->r_offset, ppc64_elf_howto_table[r_type]->name, sym_name, (int) r); ret = FALSE; } } } /* If we're emitting relocations, then shortly after this function returns, reloc offsets and addends for this section will be adjusted. Worse, reloc symbol indices will be for the output file rather than the input. Save a copy of the relocs for opd_entry_value. */ if (is_opd && (info->emitrelocations || info->relocatable)) { bfd_size_type amt; amt = input_section->reloc_count * sizeof (Elf_Internal_Rela); rel = bfd_alloc (input_bfd, amt); BFD_ASSERT (ppc64_elf_tdata (input_bfd)->opd_relocs == NULL); ppc64_elf_tdata (input_bfd)->opd_relocs = rel; if (rel == NULL) return FALSE; memcpy (rel, relocs, amt); } return ret; } /* Adjust the value of any local symbols in opd sections. */ static bfd_boolean ppc64_elf_output_symbol_hook (struct bfd_link_info *info, const char *name ATTRIBUTE_UNUSED, Elf_Internal_Sym *elfsym, asection *input_sec, struct elf_link_hash_entry *h) { long *opd_adjust, adjust; bfd_vma value; if (h != NULL) return TRUE; opd_adjust = get_opd_info (input_sec); if (opd_adjust == NULL) return TRUE; value = elfsym->st_value - input_sec->output_offset; if (!info->relocatable) value -= input_sec->output_section->vma; adjust = opd_adjust[value / 8]; if (adjust == -1) elfsym->st_value = 0; else elfsym->st_value += adjust; return TRUE; } /* Finish up dynamic symbol handling. We set the contents of various dynamic sections here. */ static bfd_boolean ppc64_elf_finish_dynamic_symbol (bfd *output_bfd, struct bfd_link_info *info, struct elf_link_hash_entry *h, Elf_Internal_Sym *sym) { struct ppc_link_hash_table *htab; struct plt_entry *ent; Elf_Internal_Rela rela; bfd_byte *loc; htab = ppc_hash_table (info); for (ent = h->plt.plist; ent != NULL; ent = ent->next) if (ent->plt.offset != (bfd_vma) -1) { /* This symbol has an entry in the procedure linkage table. Set it up. */ if (htab->plt == NULL || htab->relplt == NULL || htab->glink == NULL) abort (); /* Create a JMP_SLOT reloc to inform the dynamic linker to fill in the PLT entry. */ rela.r_offset = (htab->plt->output_section->vma + htab->plt->output_offset + ent->plt.offset); rela.r_info = ELF64_R_INFO (h->dynindx, R_PPC64_JMP_SLOT); rela.r_addend = ent->addend; loc = htab->relplt->contents; loc += ((ent->plt.offset - PLT_INITIAL_ENTRY_SIZE) / PLT_ENTRY_SIZE * sizeof (Elf64_External_Rela)); bfd_elf64_swap_reloca_out (output_bfd, &rela, loc); } if (h->needs_copy) { Elf_Internal_Rela rela; bfd_byte *loc; /* This symbol needs a copy reloc. Set it up. */ if (h->dynindx == -1 || (h->root.type != bfd_link_hash_defined && h->root.type != bfd_link_hash_defweak) || htab->relbss == NULL) abort (); rela.r_offset = (h->root.u.def.value + h->root.u.def.section->output_section->vma + h->root.u.def.section->output_offset); rela.r_info = ELF64_R_INFO (h->dynindx, R_PPC64_COPY); rela.r_addend = 0; loc = htab->relbss->contents; loc += htab->relbss->reloc_count++ * sizeof (Elf64_External_Rela); bfd_elf64_swap_reloca_out (output_bfd, &rela, loc); } /* Mark some specially defined symbols as absolute. */ if (strcmp (h->root.root.string, "_DYNAMIC") == 0) sym->st_shndx = SHN_ABS; return TRUE; } /* Used to decide how to sort relocs in an optimal manner for the dynamic linker, before writing them out. */ static enum elf_reloc_type_class ppc64_elf_reloc_type_class (const Elf_Internal_Rela *rela) { enum elf_ppc64_reloc_type r_type; r_type = ELF64_R_TYPE (rela->r_info); switch (r_type) { case R_PPC64_RELATIVE: return reloc_class_relative; case R_PPC64_JMP_SLOT: return reloc_class_plt; case R_PPC64_COPY: return reloc_class_copy; default: return reloc_class_normal; } } /* Finish up the dynamic sections. */ static bfd_boolean ppc64_elf_finish_dynamic_sections (bfd *output_bfd, struct bfd_link_info *info) { struct ppc_link_hash_table *htab; bfd *dynobj; asection *sdyn; htab = ppc_hash_table (info); dynobj = htab->elf.dynobj; sdyn = bfd_get_section_by_name (dynobj, ".dynamic"); if (htab->elf.dynamic_sections_created) { Elf64_External_Dyn *dyncon, *dynconend; if (sdyn == NULL || htab->got == NULL) abort (); dyncon = (Elf64_External_Dyn *) sdyn->contents; dynconend = (Elf64_External_Dyn *) (sdyn->contents + sdyn->size); for (; dyncon < dynconend; dyncon++) { Elf_Internal_Dyn dyn; asection *s; bfd_elf64_swap_dyn_in (dynobj, dyncon, &dyn); switch (dyn.d_tag) { default: continue; case DT_PPC64_GLINK: s = htab->glink; dyn.d_un.d_ptr = s->output_section->vma + s->output_offset; /* We stupidly defined DT_PPC64_GLINK to be the start of glink rather than the first entry point, which is what ld.so needs, and now have a bigger stub to support automatic multiple TOCs. */ dyn.d_un.d_ptr += GLINK_CALL_STUB_SIZE - 32; break; case DT_PPC64_OPD: s = bfd_get_section_by_name (output_bfd, ".opd"); if (s == NULL) continue; dyn.d_un.d_ptr = s->vma; break; case DT_PPC64_OPDSZ: s = bfd_get_section_by_name (output_bfd, ".opd"); if (s == NULL) continue; dyn.d_un.d_val = s->size; break; case DT_PLTGOT: s = htab->plt; dyn.d_un.d_ptr = s->output_section->vma + s->output_offset; break; case DT_JMPREL: s = htab->relplt; dyn.d_un.d_ptr = s->output_section->vma + s->output_offset; break; case DT_PLTRELSZ: dyn.d_un.d_val = htab->relplt->size; break; case DT_RELASZ: /* Don't count procedure linkage table relocs in the overall reloc count. */ s = htab->relplt; if (s == NULL) continue; dyn.d_un.d_val -= s->size; break; case DT_RELA: /* We may not be using the standard ELF linker script. If .rela.plt is the first .rela section, we adjust DT_RELA to not include it. */ s = htab->relplt; if (s == NULL) continue; if (dyn.d_un.d_ptr != s->output_section->vma + s->output_offset) continue; dyn.d_un.d_ptr += s->size; break; } bfd_elf64_swap_dyn_out (output_bfd, &dyn, dyncon); } } if (htab->got != NULL && htab->got->size != 0) { /* Fill in the first entry in the global offset table. We use it to hold the link-time TOCbase. */ bfd_put_64 (output_bfd, elf_gp (output_bfd) + TOC_BASE_OFF, htab->got->contents); /* Set .got entry size. */ elf_section_data (htab->got->output_section)->this_hdr.sh_entsize = 8; } if (htab->plt != NULL && htab->plt->size != 0) { /* Set .plt entry size. */ elf_section_data (htab->plt->output_section)->this_hdr.sh_entsize = PLT_ENTRY_SIZE; } /* brlt is SEC_LINKER_CREATED, so we need to write out relocs for brlt ourselves if emitrelocations. */ if (htab->brlt != NULL && htab->brlt->reloc_count != 0 && !_bfd_elf_link_output_relocs (output_bfd, htab->brlt, &elf_section_data (htab->brlt)->rel_hdr, elf_section_data (htab->brlt)->relocs, NULL)) return FALSE; /* We need to handle writing out multiple GOT sections ourselves, since we didn't add them to DYNOBJ. We know dynobj is the first bfd. */ while ((dynobj = dynobj->link_next) != NULL) { asection *s; if (!is_ppc64_elf_target (dynobj->xvec)) continue; s = ppc64_elf_tdata (dynobj)->got; if (s != NULL && s->size != 0 && s->output_section != bfd_abs_section_ptr && !bfd_set_section_contents (output_bfd, s->output_section, s->contents, s->output_offset, s->size)) return FALSE; s = ppc64_elf_tdata (dynobj)->relgot; if (s != NULL && s->size != 0 && s->output_section != bfd_abs_section_ptr && !bfd_set_section_contents (output_bfd, s->output_section, s->contents, s->output_offset, s->size)) return FALSE; } return TRUE; } #include "elf64-target.h" Index: projects/clang350-import/contrib/binutils/bfd/libbfd.h =================================================================== --- projects/clang350-import/contrib/binutils/bfd/libbfd.h (revision 275748) +++ projects/clang350-import/contrib/binutils/bfd/libbfd.h (revision 275749) @@ -1,2031 +1,2033 @@ /* DO NOT EDIT! -*- buffer-read-only: t -*- This file is automatically generated from "libbfd-in.h", "init.c", "libbfd.c", "bfdio.c", "bfdwin.c", "cache.c", "reloc.c", "archures.c" and "elf.c". Run "make headers" in your build bfd/ to regenerate. */ /* libbfd.h -- Declarations used by bfd library *implementation*. (This include file is not for users of the library.) Copyright 1990, 1991, 1992, 1993, 1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007 Free Software Foundation, Inc. Written by Cygnus Support. This file is part of BFD, the Binary File Descriptor library. This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin Street - Fifth Floor, Boston, MA 02110-1301, USA. */ #include "hashtab.h" /* Align an address upward to a boundary, expressed as a number of bytes. E.g. align to an 8-byte boundary with argument of 8. Take care never to wrap around if the address is within boundary-1 of the end of the address space. */ #define BFD_ALIGN(this, boundary) \ ((((bfd_vma) (this) + (boundary) - 1) >= (bfd_vma) (this)) \ ? (((bfd_vma) (this) + ((boundary) - 1)) & ~ (bfd_vma) ((boundary)-1)) \ : ~ (bfd_vma) 0) /* If you want to read and write large blocks, you might want to do it in quanta of this amount */ #define DEFAULT_BUFFERSIZE 8192 /* Set a tdata field. Can't use the other macros for this, since they do casts, and casting to the left of assignment isn't portable. */ #define set_tdata(bfd, v) ((bfd)->tdata.any = (v)) /* If BFD_IN_MEMORY is set for a BFD, then the iostream fields points to an instance of this structure. */ struct bfd_in_memory { /* Size of buffer. */ bfd_size_type size; /* Buffer holding contents of BFD. */ bfd_byte *buffer; }; struct section_hash_entry { struct bfd_hash_entry root; asection section; }; /* tdata for an archive. For an input archive, cache needs to be free()'d. For an output archive, symdefs do. */ struct artdata { file_ptr first_file_filepos; /* Speed up searching the armap */ htab_t cache; bfd *archive_head; /* Only interesting in output routines */ carsym *symdefs; /* the symdef entries */ symindex symdef_count; /* how many there are */ char *extended_names; /* clever intel extension */ bfd_size_type extended_names_size; /* Size of extended names */ /* when more compilers are standard C, this can be a time_t */ long armap_timestamp; /* Timestamp value written into armap. This is used for BSD archives to check that the timestamp is recent enough for the BSD linker to not complain, just before we finish writing an archive. */ file_ptr armap_datepos; /* Position within archive to seek to rewrite the date field. */ void *tdata; /* Backend specific information. */ }; #define bfd_ardata(bfd) ((bfd)->tdata.aout_ar_data) /* Goes in bfd's arelt_data slot */ struct areltdata { char * arch_header; /* it's actually a string */ unsigned int parsed_size; /* octets of filesize not including ar_hdr */ char *filename; /* null-terminated */ }; #define arelt_size(bfd) (((struct areltdata *)((bfd)->arelt_data))->parsed_size) extern void *bfd_malloc (bfd_size_type); extern void *bfd_realloc (void *, bfd_size_type); extern void *bfd_zmalloc (bfd_size_type); extern void *bfd_malloc2 (bfd_size_type, bfd_size_type); extern void *bfd_realloc2 (void *, bfd_size_type, bfd_size_type); extern void *bfd_zmalloc2 (bfd_size_type, bfd_size_type); extern void _bfd_default_error_handler (const char *s, ...); extern bfd_error_handler_type _bfd_error_handler; /* These routines allocate and free things on the BFD's objalloc. */ extern void *bfd_alloc (bfd *, bfd_size_type); extern void *bfd_zalloc (bfd *, bfd_size_type); extern void *bfd_alloc2 (bfd *, bfd_size_type, bfd_size_type); extern void *bfd_zalloc2 (bfd *, bfd_size_type, bfd_size_type); extern void bfd_release (bfd *, void *); bfd * _bfd_create_empty_archive_element_shell (bfd *obfd); bfd * _bfd_look_for_bfd_in_cache (bfd *, file_ptr); bfd_boolean _bfd_add_bfd_to_archive_cache (bfd *, file_ptr, bfd *); bfd_boolean _bfd_generic_mkarchive (bfd *abfd); const bfd_target *bfd_generic_archive_p (bfd *abfd); bfd_boolean bfd_slurp_armap (bfd *abfd); bfd_boolean bfd_slurp_bsd_armap_f2 (bfd *abfd); #define bfd_slurp_bsd_armap bfd_slurp_armap #define bfd_slurp_coff_armap bfd_slurp_armap bfd_boolean _bfd_slurp_extended_name_table (bfd *abfd); extern bfd_boolean _bfd_construct_extended_name_table (bfd *, bfd_boolean, char **, bfd_size_type *); bfd_boolean _bfd_write_archive_contents (bfd *abfd); bfd_boolean _bfd_compute_and_write_armap (bfd *, unsigned int elength); bfd *_bfd_get_elt_at_filepos (bfd *archive, file_ptr filepos); extern bfd *_bfd_generic_get_elt_at_index (bfd *, symindex); bfd * _bfd_new_bfd (void); void _bfd_delete_bfd (bfd *); bfd_boolean _bfd_free_cached_info (bfd *); bfd_boolean bfd_false (bfd *ignore); bfd_boolean bfd_true (bfd *ignore); void *bfd_nullvoidptr (bfd *ignore); int bfd_0 (bfd *ignore); unsigned int bfd_0u (bfd *ignore); long bfd_0l (bfd *ignore); long _bfd_n1 (bfd *ignore); void bfd_void (bfd *ignore); bfd *_bfd_new_bfd_contained_in (bfd *); const bfd_target *_bfd_dummy_target (bfd *abfd); void bfd_dont_truncate_arname (bfd *abfd, const char *filename, char *hdr); void bfd_bsd_truncate_arname (bfd *abfd, const char *filename, char *hdr); void bfd_gnu_truncate_arname (bfd *abfd, const char *filename, char *hdr); bfd_boolean bsd_write_armap (bfd *arch, unsigned int elength, struct orl *map, unsigned int orl_count, int stridx); bfd_boolean coff_write_armap (bfd *arch, unsigned int elength, struct orl *map, unsigned int orl_count, int stridx); extern void *_bfd_generic_read_ar_hdr (bfd *); extern void _bfd_ar_spacepad (char *, size_t, const char *, long); extern void *_bfd_generic_read_ar_hdr_mag (bfd *, const char *); bfd * bfd_generic_openr_next_archived_file (bfd *archive, bfd *last_file); int bfd_generic_stat_arch_elt (bfd *, struct stat *); #define _bfd_read_ar_hdr(abfd) \ BFD_SEND (abfd, _bfd_read_ar_hdr_fn, (abfd)) /* Generic routines to use for BFD_JUMP_TABLE_GENERIC. Use BFD_JUMP_TABLE_GENERIC (_bfd_generic). */ #define _bfd_generic_close_and_cleanup bfd_true #define _bfd_generic_bfd_free_cached_info bfd_true extern bfd_boolean _bfd_generic_new_section_hook (bfd *, asection *); extern bfd_boolean _bfd_generic_get_section_contents (bfd *, asection *, void *, file_ptr, bfd_size_type); extern bfd_boolean _bfd_generic_get_section_contents_in_window (bfd *, asection *, bfd_window *, file_ptr, bfd_size_type); /* Generic routines to use for BFD_JUMP_TABLE_COPY. Use BFD_JUMP_TABLE_COPY (_bfd_generic). */ #define _bfd_generic_bfd_copy_private_bfd_data \ ((bfd_boolean (*) (bfd *, bfd *)) bfd_true) #define _bfd_generic_bfd_merge_private_bfd_data \ ((bfd_boolean (*) (bfd *, bfd *)) bfd_true) #define _bfd_generic_bfd_set_private_flags \ ((bfd_boolean (*) (bfd *, flagword)) bfd_true) #define _bfd_generic_bfd_copy_private_section_data \ ((bfd_boolean (*) (bfd *, asection *, bfd *, asection *)) bfd_true) #define _bfd_generic_bfd_copy_private_symbol_data \ ((bfd_boolean (*) (bfd *, asymbol *, bfd *, asymbol *)) bfd_true) #define _bfd_generic_bfd_copy_private_header_data \ ((bfd_boolean (*) (bfd *, bfd *)) bfd_true) #define _bfd_generic_bfd_print_private_bfd_data \ ((bfd_boolean (*) (bfd *, void *)) bfd_true) extern bfd_boolean _bfd_generic_init_private_section_data (bfd *, asection *, bfd *, asection *, struct bfd_link_info *); /* Routines to use for BFD_JUMP_TABLE_CORE when there is no core file support. Use BFD_JUMP_TABLE_CORE (_bfd_nocore). */ extern char *_bfd_nocore_core_file_failing_command (bfd *); extern int _bfd_nocore_core_file_failing_signal (bfd *); extern bfd_boolean _bfd_nocore_core_file_matches_executable_p (bfd *, bfd *); /* Routines to use for BFD_JUMP_TABLE_ARCHIVE when there is no archive file support. Use BFD_JUMP_TABLE_ARCHIVE (_bfd_noarchive). */ #define _bfd_noarchive_slurp_armap bfd_false #define _bfd_noarchive_slurp_extended_name_table bfd_false #define _bfd_noarchive_construct_extended_name_table \ ((bfd_boolean (*) (bfd *, char **, bfd_size_type *, const char **)) \ bfd_false) #define _bfd_noarchive_truncate_arname \ ((void (*) (bfd *, const char *, char *)) bfd_void) #define _bfd_noarchive_write_armap \ ((bfd_boolean (*) (bfd *, unsigned int, struct orl *, unsigned int, int)) \ bfd_false) #define _bfd_noarchive_read_ar_hdr bfd_nullvoidptr #define _bfd_noarchive_openr_next_archived_file \ ((bfd *(*) (bfd *, bfd *)) bfd_nullvoidptr) #define _bfd_noarchive_get_elt_at_index \ ((bfd *(*) (bfd *, symindex)) bfd_nullvoidptr) #define _bfd_noarchive_generic_stat_arch_elt bfd_generic_stat_arch_elt #define _bfd_noarchive_update_armap_timestamp bfd_false /* Routines to use for BFD_JUMP_TABLE_ARCHIVE to get BSD style archives. Use BFD_JUMP_TABLE_ARCHIVE (_bfd_archive_bsd). */ #define _bfd_archive_bsd_slurp_armap bfd_slurp_bsd_armap #define _bfd_archive_bsd_slurp_extended_name_table \ _bfd_slurp_extended_name_table extern bfd_boolean _bfd_archive_bsd_construct_extended_name_table (bfd *, char **, bfd_size_type *, const char **); #define _bfd_archive_bsd_truncate_arname bfd_bsd_truncate_arname #define _bfd_archive_bsd_write_armap bsd_write_armap #define _bfd_archive_bsd_read_ar_hdr _bfd_generic_read_ar_hdr #define _bfd_archive_bsd_openr_next_archived_file \ bfd_generic_openr_next_archived_file #define _bfd_archive_bsd_get_elt_at_index _bfd_generic_get_elt_at_index #define _bfd_archive_bsd_generic_stat_arch_elt \ bfd_generic_stat_arch_elt extern bfd_boolean _bfd_archive_bsd_update_armap_timestamp (bfd *); /* Routines to use for BFD_JUMP_TABLE_ARCHIVE to get COFF style archives. Use BFD_JUMP_TABLE_ARCHIVE (_bfd_archive_coff). */ #define _bfd_archive_coff_slurp_armap bfd_slurp_coff_armap #define _bfd_archive_coff_slurp_extended_name_table \ _bfd_slurp_extended_name_table extern bfd_boolean _bfd_archive_coff_construct_extended_name_table (bfd *, char **, bfd_size_type *, const char **); #define _bfd_archive_coff_truncate_arname bfd_dont_truncate_arname #define _bfd_archive_coff_write_armap coff_write_armap #define _bfd_archive_coff_read_ar_hdr _bfd_generic_read_ar_hdr #define _bfd_archive_coff_openr_next_archived_file \ bfd_generic_openr_next_archived_file #define _bfd_archive_coff_get_elt_at_index _bfd_generic_get_elt_at_index #define _bfd_archive_coff_generic_stat_arch_elt \ bfd_generic_stat_arch_elt #define _bfd_archive_coff_update_armap_timestamp bfd_true /* Routines to use for BFD_JUMP_TABLE_SYMBOLS where there is no symbol support. Use BFD_JUMP_TABLE_SYMBOLS (_bfd_nosymbols). */ #define _bfd_nosymbols_get_symtab_upper_bound _bfd_n1 #define _bfd_nosymbols_canonicalize_symtab \ ((long (*) (bfd *, asymbol **)) _bfd_n1) #define _bfd_nosymbols_make_empty_symbol _bfd_generic_make_empty_symbol #define _bfd_nosymbols_print_symbol \ ((void (*) (bfd *, void *, asymbol *, bfd_print_symbol_type)) bfd_void) #define _bfd_nosymbols_get_symbol_info \ ((void (*) (bfd *, asymbol *, symbol_info *)) bfd_void) #define _bfd_nosymbols_bfd_is_local_label_name \ ((bfd_boolean (*) (bfd *, const char *)) bfd_false) #define _bfd_nosymbols_bfd_is_target_special_symbol \ ((bfd_boolean (*) (bfd *, asymbol *)) bfd_false) #define _bfd_nosymbols_get_lineno \ ((alent *(*) (bfd *, asymbol *)) bfd_nullvoidptr) #define _bfd_nosymbols_find_nearest_line \ ((bfd_boolean (*) (bfd *, asection *, asymbol **, bfd_vma, const char **, \ const char **, unsigned int *)) \ bfd_false) #define _bfd_nosymbols_find_inliner_info \ ((bfd_boolean (*) (bfd *, const char **, const char **, unsigned int *)) \ bfd_false) #define _bfd_nosymbols_bfd_make_debug_symbol \ ((asymbol *(*) (bfd *, void *, unsigned long)) bfd_nullvoidptr) #define _bfd_nosymbols_read_minisymbols \ ((long (*) (bfd *, bfd_boolean, void **, unsigned int *)) _bfd_n1) #define _bfd_nosymbols_minisymbol_to_symbol \ ((asymbol *(*) (bfd *, bfd_boolean, const void *, asymbol *)) \ bfd_nullvoidptr) /* Routines to use for BFD_JUMP_TABLE_RELOCS when there is no reloc support. Use BFD_JUMP_TABLE_RELOCS (_bfd_norelocs). */ extern long _bfd_norelocs_get_reloc_upper_bound (bfd *, asection *); extern long _bfd_norelocs_canonicalize_reloc (bfd *, asection *, arelent **, asymbol **); #define _bfd_norelocs_bfd_reloc_type_lookup \ ((reloc_howto_type *(*) (bfd *, bfd_reloc_code_real_type)) bfd_nullvoidptr) #define _bfd_norelocs_bfd_reloc_name_lookup \ ((reloc_howto_type *(*) (bfd *, const char *)) bfd_nullvoidptr) /* Routines to use for BFD_JUMP_TABLE_WRITE for targets which may not be written. Use BFD_JUMP_TABLE_WRITE (_bfd_nowrite). */ #define _bfd_nowrite_set_arch_mach \ ((bfd_boolean (*) (bfd *, enum bfd_architecture, unsigned long)) \ bfd_false) #define _bfd_nowrite_set_section_contents \ ((bfd_boolean (*) (bfd *, asection *, const void *, file_ptr, bfd_size_type)) \ bfd_false) /* Generic routines to use for BFD_JUMP_TABLE_WRITE. Use BFD_JUMP_TABLE_WRITE (_bfd_generic). */ #define _bfd_generic_set_arch_mach bfd_default_set_arch_mach extern bfd_boolean _bfd_generic_set_section_contents (bfd *, asection *, const void *, file_ptr, bfd_size_type); /* Routines to use for BFD_JUMP_TABLE_LINK for targets which do not support linking. Use BFD_JUMP_TABLE_LINK (_bfd_nolink). */ #define _bfd_nolink_sizeof_headers \ ((int (*) (bfd *, struct bfd_link_info *)) bfd_0) #define _bfd_nolink_bfd_get_relocated_section_contents \ ((bfd_byte *(*) (bfd *, struct bfd_link_info *, struct bfd_link_order *, \ bfd_byte *, bfd_boolean, asymbol **)) \ bfd_nullvoidptr) #define _bfd_nolink_bfd_relax_section \ ((bfd_boolean (*) \ (bfd *, asection *, struct bfd_link_info *, bfd_boolean *)) \ bfd_false) #define _bfd_nolink_bfd_gc_sections \ ((bfd_boolean (*) (bfd *, struct bfd_link_info *)) \ bfd_false) #define _bfd_nolink_bfd_merge_sections \ ((bfd_boolean (*) (bfd *, struct bfd_link_info *)) \ bfd_false) #define _bfd_nolink_bfd_is_group_section \ ((bfd_boolean (*) (bfd *, const struct bfd_section *)) \ bfd_false) #define _bfd_nolink_bfd_discard_group \ ((bfd_boolean (*) (bfd *, struct bfd_section *)) \ bfd_false) #define _bfd_nolink_bfd_link_hash_table_create \ ((struct bfd_link_hash_table *(*) (bfd *)) bfd_nullvoidptr) #define _bfd_nolink_bfd_link_hash_table_free \ ((void (*) (struct bfd_link_hash_table *)) bfd_void) #define _bfd_nolink_bfd_link_add_symbols \ ((bfd_boolean (*) (bfd *, struct bfd_link_info *)) bfd_false) #define _bfd_nolink_bfd_link_just_syms \ ((void (*) (asection *, struct bfd_link_info *)) bfd_void) #define _bfd_nolink_bfd_final_link \ ((bfd_boolean (*) (bfd *, struct bfd_link_info *)) bfd_false) #define _bfd_nolink_bfd_link_split_section \ ((bfd_boolean (*) (bfd *, struct bfd_section *)) bfd_false) #define _bfd_nolink_section_already_linked \ ((void (*) (bfd *, struct bfd_section *, struct bfd_link_info *)) bfd_void) /* Routines to use for BFD_JUMP_TABLE_DYNAMIC for targets which do not have dynamic symbols or relocs. Use BFD_JUMP_TABLE_DYNAMIC (_bfd_nodynamic). */ #define _bfd_nodynamic_get_dynamic_symtab_upper_bound _bfd_n1 #define _bfd_nodynamic_canonicalize_dynamic_symtab \ ((long (*) (bfd *, asymbol **)) _bfd_n1) #define _bfd_nodynamic_get_synthetic_symtab \ ((long (*) (bfd *, long, asymbol **, long, asymbol **, asymbol **)) _bfd_n1) #define _bfd_nodynamic_get_dynamic_reloc_upper_bound _bfd_n1 #define _bfd_nodynamic_canonicalize_dynamic_reloc \ ((long (*) (bfd *, arelent **, asymbol **)) _bfd_n1) /* Generic routine to determine of the given symbol is a local label. */ extern bfd_boolean bfd_generic_is_local_label_name (bfd *, const char *); /* Generic minisymbol routines. */ extern long _bfd_generic_read_minisymbols (bfd *, bfd_boolean, void **, unsigned int *); extern asymbol *_bfd_generic_minisymbol_to_symbol (bfd *, bfd_boolean, const void *, asymbol *); /* Find the nearest line using .stab/.stabstr sections. */ extern bfd_boolean _bfd_stab_section_find_nearest_line (bfd *, asymbol **, asection *, bfd_vma, bfd_boolean *, const char **, const char **, unsigned int *, void **); /* Find the nearest line using DWARF 1 debugging information. */ extern bfd_boolean _bfd_dwarf1_find_nearest_line (bfd *, asection *, asymbol **, bfd_vma, const char **, const char **, unsigned int *); /* Find the nearest line using DWARF 2 debugging information. */ extern bfd_boolean _bfd_dwarf2_find_nearest_line (bfd *, asection *, asymbol **, bfd_vma, const char **, const char **, unsigned int *, unsigned int, void **); /* Find the line using DWARF 2 debugging information. */ extern bfd_boolean _bfd_dwarf2_find_line (bfd *, asymbol **, asymbol *, const char **, unsigned int *, unsigned int, void **); bfd_boolean _bfd_generic_find_line (bfd *, asymbol **, asymbol *, const char **, unsigned int *); /* Find inliner info after calling bfd_find_nearest_line. */ extern bfd_boolean _bfd_dwarf2_find_inliner_info (bfd *, const char **, const char **, unsigned int *, void **); /* Create a new section entry. */ extern struct bfd_hash_entry *bfd_section_hash_newfunc (struct bfd_hash_entry *, struct bfd_hash_table *, const char *); /* A routine to create entries for a bfd_link_hash_table. */ extern struct bfd_hash_entry *_bfd_link_hash_newfunc (struct bfd_hash_entry *entry, struct bfd_hash_table *table, const char *string); /* Initialize a bfd_link_hash_table. */ extern bfd_boolean _bfd_link_hash_table_init (struct bfd_link_hash_table *, bfd *, struct bfd_hash_entry *(*) (struct bfd_hash_entry *, struct bfd_hash_table *, const char *), unsigned int); /* Generic link hash table creation routine. */ extern struct bfd_link_hash_table *_bfd_generic_link_hash_table_create (bfd *); /* Generic link hash table destruction routine. */ extern void _bfd_generic_link_hash_table_free (struct bfd_link_hash_table *); /* Generic add symbol routine. */ extern bfd_boolean _bfd_generic_link_add_symbols (bfd *, struct bfd_link_info *); /* Generic add symbol routine. This version is used by targets for which the linker must collect constructors and destructors by name, as the collect2 program does. */ extern bfd_boolean _bfd_generic_link_add_symbols_collect (bfd *, struct bfd_link_info *); /* Generic archive add symbol routine. */ extern bfd_boolean _bfd_generic_link_add_archive_symbols (bfd *, struct bfd_link_info *, bfd_boolean (*) (bfd *, struct bfd_link_info *, bfd_boolean *)); /* Forward declaration to avoid prototype errors. */ typedef struct bfd_link_hash_entry _bfd_link_hash_entry; /* Generic routine to add a single symbol. */ extern bfd_boolean _bfd_generic_link_add_one_symbol (struct bfd_link_info *, bfd *, const char *name, flagword, asection *, bfd_vma, const char *, bfd_boolean copy, bfd_boolean constructor, struct bfd_link_hash_entry **); /* Generic routine to mark section as supplying symbols only. */ extern void _bfd_generic_link_just_syms (asection *, struct bfd_link_info *); /* Generic link routine. */ extern bfd_boolean _bfd_generic_final_link (bfd *, struct bfd_link_info *); extern bfd_boolean _bfd_generic_link_split_section (bfd *, struct bfd_section *); extern void _bfd_generic_section_already_linked (bfd *, struct bfd_section *, struct bfd_link_info *); /* Generic reloc_link_order processing routine. */ extern bfd_boolean _bfd_generic_reloc_link_order (bfd *, struct bfd_link_info *, asection *, struct bfd_link_order *); /* Default link order processing routine. */ extern bfd_boolean _bfd_default_link_order (bfd *, struct bfd_link_info *, asection *, struct bfd_link_order *); /* Count the number of reloc entries in a link order list. */ extern unsigned int _bfd_count_link_order_relocs (struct bfd_link_order *); /* Final link relocation routine. */ extern bfd_reloc_status_type _bfd_final_link_relocate (reloc_howto_type *, bfd *, asection *, bfd_byte *, bfd_vma, bfd_vma, bfd_vma); /* Relocate a particular location by a howto and a value. */ extern bfd_reloc_status_type _bfd_relocate_contents (reloc_howto_type *, bfd *, bfd_vma, bfd_byte *); /* Clear a given location using a given howto. */ extern void _bfd_clear_contents (reloc_howto_type *howto, bfd *input_bfd, bfd_byte *location); /* Link stabs in sections in the first pass. */ extern bfd_boolean _bfd_link_section_stabs (bfd *, struct stab_info *, asection *, asection *, void **, bfd_size_type *); /* Eliminate stabs for discarded functions and symbols. */ extern bfd_boolean _bfd_discard_section_stabs (bfd *, asection *, void *, bfd_boolean (*) (bfd_vma, void *), void *); /* Write out the .stab section when linking stabs in sections. */ extern bfd_boolean _bfd_write_section_stabs (bfd *, struct stab_info *, asection *, void **, bfd_byte *); /* Write out the .stabstr string table when linking stabs in sections. */ extern bfd_boolean _bfd_write_stab_strings (bfd *, struct stab_info *); /* Find an offset within a .stab section when linking stabs in sections. */ extern bfd_vma _bfd_stab_section_offset (asection *, void *, bfd_vma); /* Register a SEC_MERGE section as a candidate for merging. */ extern bfd_boolean _bfd_add_merge_section (bfd *, void **, asection *, void **); /* Attempt to merge SEC_MERGE sections. */ extern bfd_boolean _bfd_merge_sections (bfd *, struct bfd_link_info *, void *, void (*) (bfd *, asection *)); /* Write out a merged section. */ extern bfd_boolean _bfd_write_merged_section (bfd *, asection *, void *); /* Find an offset within a modified SEC_MERGE section. */ extern bfd_vma _bfd_merged_section_offset (bfd *, asection **, void *, bfd_vma); /* Create a string table. */ extern struct bfd_strtab_hash *_bfd_stringtab_init (void); /* Create an XCOFF .debug section style string table. */ extern struct bfd_strtab_hash *_bfd_xcoff_stringtab_init (void); /* Free a string table. */ extern void _bfd_stringtab_free (struct bfd_strtab_hash *); /* Get the size of a string table. */ extern bfd_size_type _bfd_stringtab_size (struct bfd_strtab_hash *); /* Add a string to a string table. */ extern bfd_size_type _bfd_stringtab_add (struct bfd_strtab_hash *, const char *, bfd_boolean hash, bfd_boolean copy); /* Write out a string table. */ extern bfd_boolean _bfd_stringtab_emit (bfd *, struct bfd_strtab_hash *); /* Check that endianness of input and output file match. */ extern bfd_boolean _bfd_generic_verify_endian_match (bfd *, bfd *); /* Macros to tell if bfds are read or write enabled. Note that bfds open for read may be scribbled into if the fd passed to bfd_fdopenr is actually open both for read and write simultaneously. However an output bfd will never be open for read. Therefore sometimes you want to check bfd_read_p or !bfd_read_p, and only sometimes bfd_write_p. */ #define bfd_read_p(abfd) \ ((abfd)->direction == read_direction || (abfd)->direction == both_direction) #define bfd_write_p(abfd) \ ((abfd)->direction == write_direction || (abfd)->direction == both_direction) void bfd_assert (const char*,int); #define BFD_ASSERT(x) \ do { if (!(x)) bfd_assert(__FILE__,__LINE__); } while (0) #define BFD_FAIL() \ do { bfd_assert(__FILE__,__LINE__); } while (0) extern void _bfd_abort (const char *, int, const char *) ATTRIBUTE_NORETURN; /* if gcc >= 2.6, we can give a function name, too */ #if __GNUC__ < 2 || (__GNUC__ == 2 && __GNUC_MINOR__ < 6) #define __PRETTY_FUNCTION__ ((char *) NULL) #endif #undef abort #define abort() _bfd_abort (__FILE__, __LINE__, __PRETTY_FUNCTION__) /* Manipulate a system FILE but using BFD's "file_ptr", rather than the system "off_t" or "off64_t", as the offset. */ extern file_ptr real_ftell (FILE *file); extern int real_fseek (FILE *file, file_ptr offset, int whence); extern FILE *real_fopen (const char *filename, const char *modes); /* List of supported target vectors, and the default vector (if bfd_default_vector[0] is NULL, there is no default). */ extern const bfd_target * const *bfd_target_vector; extern const bfd_target *bfd_default_vector[]; /* List of associated target vectors. */ extern const bfd_target * const *bfd_associated_vector; /* Functions shared by the ECOFF and MIPS ELF backends, which have no other common header files. */ #if defined(__STDC__) || defined(ALMOST_STDC) struct ecoff_find_line; #endif extern bfd_boolean _bfd_ecoff_locate_line (bfd *, asection *, bfd_vma, struct ecoff_debug_info * const, const struct ecoff_debug_swap * const, struct ecoff_find_line *, const char **, const char **, unsigned int *); extern bfd_boolean _bfd_ecoff_get_accumulated_pdr (void *, bfd_byte *); extern bfd_boolean _bfd_ecoff_get_accumulated_sym (void *, bfd_byte *); extern bfd_boolean _bfd_ecoff_get_accumulated_ss (void *, bfd_byte *); extern bfd_vma _bfd_get_gp_value (bfd *); extern void _bfd_set_gp_value (bfd *, bfd_vma); /* Function shared by the COFF and ELF SH backends, which have no other common header files. */ #ifndef _bfd_sh_align_load_span extern bfd_boolean _bfd_sh_align_load_span (bfd *, asection *, bfd_byte *, bfd_boolean (*) (bfd *, asection *, void *, bfd_byte *, bfd_vma), void *, bfd_vma **, bfd_vma *, bfd_vma, bfd_vma, bfd_boolean *); #endif /* This is the shape of the elements inside the already_linked hash table. It maps a name onto a list of already_linked elements with the same name. */ struct bfd_section_already_linked_hash_entry { struct bfd_hash_entry root; struct bfd_section_already_linked *entry; }; struct bfd_section_already_linked { struct bfd_section_already_linked *next; asection *sec; }; extern struct bfd_section_already_linked_hash_entry * bfd_section_already_linked_table_lookup (const char *); extern void bfd_section_already_linked_table_insert (struct bfd_section_already_linked_hash_entry *, asection *); extern void bfd_section_already_linked_table_traverse (bfd_boolean (*) (struct bfd_section_already_linked_hash_entry *, void *), void *); extern bfd_vma read_unsigned_leb128 (bfd *, bfd_byte *, unsigned int *); extern bfd_signed_vma read_signed_leb128 (bfd *, bfd_byte *, unsigned int *); /* Extracted from init.c. */ /* Extracted from libbfd.c. */ bfd_boolean bfd_write_bigendian_4byte_int (bfd *, unsigned int); unsigned int bfd_log2 (bfd_vma x); /* Extracted from bfdio.c. */ struct bfd_iovec { /* To avoid problems with macros, a "b" rather than "f" prefix is prepended to each method name. */ /* Attempt to read/write NBYTES on ABFD's IOSTREAM storing/fetching bytes starting at PTR. Return the number of bytes actually transfered (a read past end-of-file returns less than NBYTES), or -1 (setting <>) if an error occurs. */ file_ptr (*bread) (struct bfd *abfd, void *ptr, file_ptr nbytes); file_ptr (*bwrite) (struct bfd *abfd, const void *ptr, file_ptr nbytes); /* Return the current IOSTREAM file offset, or -1 (setting <> if an error occurs. */ file_ptr (*btell) (struct bfd *abfd); /* For the following, on successful completion a value of 0 is returned. Otherwise, a value of -1 is returned (and <> is set). */ int (*bseek) (struct bfd *abfd, file_ptr offset, int whence); int (*bclose) (struct bfd *abfd); int (*bflush) (struct bfd *abfd); int (*bstat) (struct bfd *abfd, struct stat *sb); }; /* Extracted from bfdwin.c. */ struct _bfd_window_internal { struct _bfd_window_internal *next; void *data; bfd_size_type size; int refcount : 31; /* should be enough... */ unsigned mapped : 1; /* 1 = mmap, 0 = malloc */ }; /* Extracted from cache.c. */ bfd_boolean bfd_cache_init (bfd *abfd); bfd_boolean bfd_cache_close (bfd *abfd); FILE* bfd_open_file (bfd *abfd); /* Extracted from reloc.c. */ #ifdef _BFD_MAKE_TABLE_bfd_reloc_code_real static const char *const bfd_reloc_code_real_names[] = { "@@uninitialized@@", "BFD_RELOC_64", "BFD_RELOC_32", "BFD_RELOC_26", "BFD_RELOC_24", "BFD_RELOC_16", "BFD_RELOC_14", "BFD_RELOC_8", "BFD_RELOC_64_PCREL", "BFD_RELOC_32_PCREL", "BFD_RELOC_24_PCREL", "BFD_RELOC_16_PCREL", "BFD_RELOC_12_PCREL", "BFD_RELOC_8_PCREL", "BFD_RELOC_32_SECREL", "BFD_RELOC_32_GOT_PCREL", "BFD_RELOC_16_GOT_PCREL", "BFD_RELOC_8_GOT_PCREL", "BFD_RELOC_32_GOTOFF", "BFD_RELOC_16_GOTOFF", "BFD_RELOC_LO16_GOTOFF", "BFD_RELOC_HI16_GOTOFF", "BFD_RELOC_HI16_S_GOTOFF", "BFD_RELOC_8_GOTOFF", "BFD_RELOC_64_PLT_PCREL", "BFD_RELOC_32_PLT_PCREL", "BFD_RELOC_24_PLT_PCREL", "BFD_RELOC_16_PLT_PCREL", "BFD_RELOC_8_PLT_PCREL", "BFD_RELOC_64_PLTOFF", "BFD_RELOC_32_PLTOFF", "BFD_RELOC_16_PLTOFF", "BFD_RELOC_LO16_PLTOFF", "BFD_RELOC_HI16_PLTOFF", "BFD_RELOC_HI16_S_PLTOFF", "BFD_RELOC_8_PLTOFF", "BFD_RELOC_68K_GLOB_DAT", "BFD_RELOC_68K_JMP_SLOT", "BFD_RELOC_68K_RELATIVE", "BFD_RELOC_32_BASEREL", "BFD_RELOC_16_BASEREL", "BFD_RELOC_LO16_BASEREL", "BFD_RELOC_HI16_BASEREL", "BFD_RELOC_HI16_S_BASEREL", "BFD_RELOC_8_BASEREL", "BFD_RELOC_RVA", "BFD_RELOC_8_FFnn", "BFD_RELOC_32_PCREL_S2", "BFD_RELOC_16_PCREL_S2", "BFD_RELOC_23_PCREL_S2", "BFD_RELOC_HI22", "BFD_RELOC_LO10", "BFD_RELOC_GPREL16", "BFD_RELOC_GPREL32", "BFD_RELOC_I960_CALLJ", "BFD_RELOC_NONE", "BFD_RELOC_SPARC_WDISP22", "BFD_RELOC_SPARC22", "BFD_RELOC_SPARC13", "BFD_RELOC_SPARC_GOT10", "BFD_RELOC_SPARC_GOT13", "BFD_RELOC_SPARC_GOT22", "BFD_RELOC_SPARC_PC10", "BFD_RELOC_SPARC_PC22", "BFD_RELOC_SPARC_WPLT30", "BFD_RELOC_SPARC_COPY", "BFD_RELOC_SPARC_GLOB_DAT", "BFD_RELOC_SPARC_JMP_SLOT", "BFD_RELOC_SPARC_RELATIVE", "BFD_RELOC_SPARC_UA16", "BFD_RELOC_SPARC_UA32", "BFD_RELOC_SPARC_UA64", "BFD_RELOC_SPARC_BASE13", "BFD_RELOC_SPARC_BASE22", "BFD_RELOC_SPARC_10", "BFD_RELOC_SPARC_11", "BFD_RELOC_SPARC_OLO10", "BFD_RELOC_SPARC_HH22", "BFD_RELOC_SPARC_HM10", "BFD_RELOC_SPARC_LM22", "BFD_RELOC_SPARC_PC_HH22", "BFD_RELOC_SPARC_PC_HM10", "BFD_RELOC_SPARC_PC_LM22", "BFD_RELOC_SPARC_WDISP16", "BFD_RELOC_SPARC_WDISP19", "BFD_RELOC_SPARC_7", "BFD_RELOC_SPARC_6", "BFD_RELOC_SPARC_5", "BFD_RELOC_SPARC_PLT32", "BFD_RELOC_SPARC_PLT64", "BFD_RELOC_SPARC_HIX22", "BFD_RELOC_SPARC_LOX10", "BFD_RELOC_SPARC_H44", "BFD_RELOC_SPARC_M44", "BFD_RELOC_SPARC_L44", "BFD_RELOC_SPARC_REGISTER", "BFD_RELOC_SPARC_REV32", "BFD_RELOC_SPARC_TLS_GD_HI22", "BFD_RELOC_SPARC_TLS_GD_LO10", "BFD_RELOC_SPARC_TLS_GD_ADD", "BFD_RELOC_SPARC_TLS_GD_CALL", "BFD_RELOC_SPARC_TLS_LDM_HI22", "BFD_RELOC_SPARC_TLS_LDM_LO10", "BFD_RELOC_SPARC_TLS_LDM_ADD", "BFD_RELOC_SPARC_TLS_LDM_CALL", "BFD_RELOC_SPARC_TLS_LDO_HIX22", "BFD_RELOC_SPARC_TLS_LDO_LOX10", "BFD_RELOC_SPARC_TLS_LDO_ADD", "BFD_RELOC_SPARC_TLS_IE_HI22", "BFD_RELOC_SPARC_TLS_IE_LO10", "BFD_RELOC_SPARC_TLS_IE_LD", "BFD_RELOC_SPARC_TLS_IE_LDX", "BFD_RELOC_SPARC_TLS_IE_ADD", "BFD_RELOC_SPARC_TLS_LE_HIX22", "BFD_RELOC_SPARC_TLS_LE_LOX10", "BFD_RELOC_SPARC_TLS_DTPMOD32", "BFD_RELOC_SPARC_TLS_DTPMOD64", "BFD_RELOC_SPARC_TLS_DTPOFF32", "BFD_RELOC_SPARC_TLS_DTPOFF64", "BFD_RELOC_SPARC_TLS_TPOFF32", "BFD_RELOC_SPARC_TLS_TPOFF64", "BFD_RELOC_SPU_IMM7", "BFD_RELOC_SPU_IMM8", "BFD_RELOC_SPU_IMM10", "BFD_RELOC_SPU_IMM10W", "BFD_RELOC_SPU_IMM16", "BFD_RELOC_SPU_IMM16W", "BFD_RELOC_SPU_IMM18", "BFD_RELOC_SPU_PCREL9a", "BFD_RELOC_SPU_PCREL9b", "BFD_RELOC_SPU_PCREL16", "BFD_RELOC_SPU_LO16", "BFD_RELOC_SPU_HI16", "BFD_RELOC_SPU_PPU32", "BFD_RELOC_SPU_PPU64", "BFD_RELOC_ALPHA_GPDISP_HI16", "BFD_RELOC_ALPHA_GPDISP_LO16", "BFD_RELOC_ALPHA_GPDISP", "BFD_RELOC_ALPHA_LITERAL", "BFD_RELOC_ALPHA_ELF_LITERAL", "BFD_RELOC_ALPHA_LITUSE", "BFD_RELOC_ALPHA_HINT", "BFD_RELOC_ALPHA_LINKAGE", "BFD_RELOC_ALPHA_CODEADDR", "BFD_RELOC_ALPHA_GPREL_HI16", "BFD_RELOC_ALPHA_GPREL_LO16", "BFD_RELOC_ALPHA_BRSGP", "BFD_RELOC_ALPHA_TLSGD", "BFD_RELOC_ALPHA_TLSLDM", "BFD_RELOC_ALPHA_DTPMOD64", "BFD_RELOC_ALPHA_GOTDTPREL16", "BFD_RELOC_ALPHA_DTPREL64", "BFD_RELOC_ALPHA_DTPREL_HI16", "BFD_RELOC_ALPHA_DTPREL_LO16", "BFD_RELOC_ALPHA_DTPREL16", "BFD_RELOC_ALPHA_GOTTPREL16", "BFD_RELOC_ALPHA_TPREL64", "BFD_RELOC_ALPHA_TPREL_HI16", "BFD_RELOC_ALPHA_TPREL_LO16", "BFD_RELOC_ALPHA_TPREL16", "BFD_RELOC_MIPS_JMP", "BFD_RELOC_MIPS16_JMP", "BFD_RELOC_MIPS16_GPREL", "BFD_RELOC_HI16", "BFD_RELOC_HI16_S", "BFD_RELOC_LO16", "BFD_RELOC_HI16_PCREL", "BFD_RELOC_HI16_S_PCREL", "BFD_RELOC_LO16_PCREL", "BFD_RELOC_MIPS16_HI16", "BFD_RELOC_MIPS16_HI16_S", "BFD_RELOC_MIPS16_LO16", "BFD_RELOC_MIPS_LITERAL", "BFD_RELOC_MIPS_GOT16", "BFD_RELOC_MIPS_CALL16", "BFD_RELOC_MIPS_GOT_HI16", "BFD_RELOC_MIPS_GOT_LO16", "BFD_RELOC_MIPS_CALL_HI16", "BFD_RELOC_MIPS_CALL_LO16", "BFD_RELOC_MIPS_SUB", "BFD_RELOC_MIPS_GOT_PAGE", "BFD_RELOC_MIPS_GOT_OFST", "BFD_RELOC_MIPS_GOT_DISP", "BFD_RELOC_MIPS_SHIFT5", "BFD_RELOC_MIPS_SHIFT6", "BFD_RELOC_MIPS_INSERT_A", "BFD_RELOC_MIPS_INSERT_B", "BFD_RELOC_MIPS_DELETE", "BFD_RELOC_MIPS_HIGHEST", "BFD_RELOC_MIPS_HIGHER", "BFD_RELOC_MIPS_SCN_DISP", "BFD_RELOC_MIPS_REL16", "BFD_RELOC_MIPS_RELGOT", "BFD_RELOC_MIPS_JALR", "BFD_RELOC_MIPS_TLS_DTPMOD32", "BFD_RELOC_MIPS_TLS_DTPREL32", "BFD_RELOC_MIPS_TLS_DTPMOD64", "BFD_RELOC_MIPS_TLS_DTPREL64", "BFD_RELOC_MIPS_TLS_GD", "BFD_RELOC_MIPS_TLS_LDM", "BFD_RELOC_MIPS_TLS_DTPREL_HI16", "BFD_RELOC_MIPS_TLS_DTPREL_LO16", "BFD_RELOC_MIPS_TLS_GOTTPREL", "BFD_RELOC_MIPS_TLS_TPREL32", "BFD_RELOC_MIPS_TLS_TPREL64", "BFD_RELOC_MIPS_TLS_TPREL_HI16", "BFD_RELOC_MIPS_TLS_TPREL_LO16", "BFD_RELOC_MIPS_COPY", "BFD_RELOC_MIPS_JUMP_SLOT", "BFD_RELOC_FRV_LABEL16", "BFD_RELOC_FRV_LABEL24", "BFD_RELOC_FRV_LO16", "BFD_RELOC_FRV_HI16", "BFD_RELOC_FRV_GPREL12", "BFD_RELOC_FRV_GPRELU12", "BFD_RELOC_FRV_GPREL32", "BFD_RELOC_FRV_GPRELHI", "BFD_RELOC_FRV_GPRELLO", "BFD_RELOC_FRV_GOT12", "BFD_RELOC_FRV_GOTHI", "BFD_RELOC_FRV_GOTLO", "BFD_RELOC_FRV_FUNCDESC", "BFD_RELOC_FRV_FUNCDESC_GOT12", "BFD_RELOC_FRV_FUNCDESC_GOTHI", "BFD_RELOC_FRV_FUNCDESC_GOTLO", "BFD_RELOC_FRV_FUNCDESC_VALUE", "BFD_RELOC_FRV_FUNCDESC_GOTOFF12", "BFD_RELOC_FRV_FUNCDESC_GOTOFFHI", "BFD_RELOC_FRV_FUNCDESC_GOTOFFLO", "BFD_RELOC_FRV_GOTOFF12", "BFD_RELOC_FRV_GOTOFFHI", "BFD_RELOC_FRV_GOTOFFLO", "BFD_RELOC_FRV_GETTLSOFF", "BFD_RELOC_FRV_TLSDESC_VALUE", "BFD_RELOC_FRV_GOTTLSDESC12", "BFD_RELOC_FRV_GOTTLSDESCHI", "BFD_RELOC_FRV_GOTTLSDESCLO", "BFD_RELOC_FRV_TLSMOFF12", "BFD_RELOC_FRV_TLSMOFFHI", "BFD_RELOC_FRV_TLSMOFFLO", "BFD_RELOC_FRV_GOTTLSOFF12", "BFD_RELOC_FRV_GOTTLSOFFHI", "BFD_RELOC_FRV_GOTTLSOFFLO", "BFD_RELOC_FRV_TLSOFF", "BFD_RELOC_FRV_TLSDESC_RELAX", "BFD_RELOC_FRV_GETTLSOFF_RELAX", "BFD_RELOC_FRV_TLSOFF_RELAX", "BFD_RELOC_FRV_TLSMOFF", "BFD_RELOC_MN10300_GOTOFF24", "BFD_RELOC_MN10300_GOT32", "BFD_RELOC_MN10300_GOT24", "BFD_RELOC_MN10300_GOT16", "BFD_RELOC_MN10300_COPY", "BFD_RELOC_MN10300_GLOB_DAT", "BFD_RELOC_MN10300_JMP_SLOT", "BFD_RELOC_MN10300_RELATIVE", "BFD_RELOC_386_GOT32", "BFD_RELOC_386_PLT32", "BFD_RELOC_386_COPY", "BFD_RELOC_386_GLOB_DAT", "BFD_RELOC_386_JUMP_SLOT", "BFD_RELOC_386_RELATIVE", "BFD_RELOC_386_GOTOFF", "BFD_RELOC_386_GOTPC", "BFD_RELOC_386_TLS_TPOFF", "BFD_RELOC_386_TLS_IE", "BFD_RELOC_386_TLS_GOTIE", "BFD_RELOC_386_TLS_LE", "BFD_RELOC_386_TLS_GD", "BFD_RELOC_386_TLS_LDM", "BFD_RELOC_386_TLS_LDO_32", "BFD_RELOC_386_TLS_IE_32", "BFD_RELOC_386_TLS_LE_32", "BFD_RELOC_386_TLS_DTPMOD32", "BFD_RELOC_386_TLS_DTPOFF32", "BFD_RELOC_386_TLS_TPOFF32", "BFD_RELOC_386_TLS_GOTDESC", "BFD_RELOC_386_TLS_DESC_CALL", "BFD_RELOC_386_TLS_DESC", "BFD_RELOC_X86_64_GOT32", "BFD_RELOC_X86_64_PLT32", "BFD_RELOC_X86_64_COPY", "BFD_RELOC_X86_64_GLOB_DAT", "BFD_RELOC_X86_64_JUMP_SLOT", "BFD_RELOC_X86_64_RELATIVE", "BFD_RELOC_X86_64_GOTPCREL", "BFD_RELOC_X86_64_32S", "BFD_RELOC_X86_64_DTPMOD64", "BFD_RELOC_X86_64_DTPOFF64", "BFD_RELOC_X86_64_TPOFF64", "BFD_RELOC_X86_64_TLSGD", "BFD_RELOC_X86_64_TLSLD", "BFD_RELOC_X86_64_DTPOFF32", "BFD_RELOC_X86_64_GOTTPOFF", "BFD_RELOC_X86_64_TPOFF32", "BFD_RELOC_X86_64_GOTOFF64", "BFD_RELOC_X86_64_GOTPC32", "BFD_RELOC_X86_64_GOT64", "BFD_RELOC_X86_64_GOTPCREL64", "BFD_RELOC_X86_64_GOTPC64", "BFD_RELOC_X86_64_GOTPLT64", "BFD_RELOC_X86_64_PLTOFF64", "BFD_RELOC_X86_64_GOTPC32_TLSDESC", "BFD_RELOC_X86_64_TLSDESC_CALL", "BFD_RELOC_X86_64_TLSDESC", "BFD_RELOC_NS32K_IMM_8", "BFD_RELOC_NS32K_IMM_16", "BFD_RELOC_NS32K_IMM_32", "BFD_RELOC_NS32K_IMM_8_PCREL", "BFD_RELOC_NS32K_IMM_16_PCREL", "BFD_RELOC_NS32K_IMM_32_PCREL", "BFD_RELOC_NS32K_DISP_8", "BFD_RELOC_NS32K_DISP_16", "BFD_RELOC_NS32K_DISP_32", "BFD_RELOC_NS32K_DISP_8_PCREL", "BFD_RELOC_NS32K_DISP_16_PCREL", "BFD_RELOC_NS32K_DISP_32_PCREL", "BFD_RELOC_PDP11_DISP_8_PCREL", "BFD_RELOC_PDP11_DISP_6_PCREL", "BFD_RELOC_PJ_CODE_HI16", "BFD_RELOC_PJ_CODE_LO16", "BFD_RELOC_PJ_CODE_DIR16", "BFD_RELOC_PJ_CODE_DIR32", "BFD_RELOC_PJ_CODE_REL16", "BFD_RELOC_PJ_CODE_REL32", "BFD_RELOC_PPC_B26", "BFD_RELOC_PPC_BA26", "BFD_RELOC_PPC_TOC16", "BFD_RELOC_PPC_B16", "BFD_RELOC_PPC_B16_BRTAKEN", "BFD_RELOC_PPC_B16_BRNTAKEN", "BFD_RELOC_PPC_BA16", "BFD_RELOC_PPC_BA16_BRTAKEN", "BFD_RELOC_PPC_BA16_BRNTAKEN", "BFD_RELOC_PPC_COPY", "BFD_RELOC_PPC_GLOB_DAT", "BFD_RELOC_PPC_JMP_SLOT", "BFD_RELOC_PPC_RELATIVE", "BFD_RELOC_PPC_LOCAL24PC", "BFD_RELOC_PPC_EMB_NADDR32", "BFD_RELOC_PPC_EMB_NADDR16", "BFD_RELOC_PPC_EMB_NADDR16_LO", "BFD_RELOC_PPC_EMB_NADDR16_HI", "BFD_RELOC_PPC_EMB_NADDR16_HA", "BFD_RELOC_PPC_EMB_SDAI16", "BFD_RELOC_PPC_EMB_SDA2I16", "BFD_RELOC_PPC_EMB_SDA2REL", "BFD_RELOC_PPC_EMB_SDA21", "BFD_RELOC_PPC_EMB_MRKREF", "BFD_RELOC_PPC_EMB_RELSEC16", "BFD_RELOC_PPC_EMB_RELST_LO", "BFD_RELOC_PPC_EMB_RELST_HI", "BFD_RELOC_PPC_EMB_RELST_HA", "BFD_RELOC_PPC_EMB_BIT_FLD", "BFD_RELOC_PPC_EMB_RELSDA", "BFD_RELOC_PPC64_HIGHER", "BFD_RELOC_PPC64_HIGHER_S", "BFD_RELOC_PPC64_HIGHEST", "BFD_RELOC_PPC64_HIGHEST_S", "BFD_RELOC_PPC64_TOC16_LO", "BFD_RELOC_PPC64_TOC16_HI", "BFD_RELOC_PPC64_TOC16_HA", "BFD_RELOC_PPC64_TOC", "BFD_RELOC_PPC64_PLTGOT16", "BFD_RELOC_PPC64_PLTGOT16_LO", "BFD_RELOC_PPC64_PLTGOT16_HI", "BFD_RELOC_PPC64_PLTGOT16_HA", "BFD_RELOC_PPC64_ADDR16_DS", "BFD_RELOC_PPC64_ADDR16_LO_DS", "BFD_RELOC_PPC64_GOT16_DS", "BFD_RELOC_PPC64_GOT16_LO_DS", "BFD_RELOC_PPC64_PLT16_LO_DS", "BFD_RELOC_PPC64_SECTOFF_DS", "BFD_RELOC_PPC64_SECTOFF_LO_DS", "BFD_RELOC_PPC64_TOC16_DS", "BFD_RELOC_PPC64_TOC16_LO_DS", "BFD_RELOC_PPC64_PLTGOT16_DS", "BFD_RELOC_PPC64_PLTGOT16_LO_DS", "BFD_RELOC_PPC_TLS", + "BFD_RELOC_PPC_TLSGD", + "BFD_RELOC_PPC_TLSLD", "BFD_RELOC_PPC_DTPMOD", "BFD_RELOC_PPC_TPREL16", "BFD_RELOC_PPC_TPREL16_LO", "BFD_RELOC_PPC_TPREL16_HI", "BFD_RELOC_PPC_TPREL16_HA", "BFD_RELOC_PPC_TPREL", "BFD_RELOC_PPC_DTPREL16", "BFD_RELOC_PPC_DTPREL16_LO", "BFD_RELOC_PPC_DTPREL16_HI", "BFD_RELOC_PPC_DTPREL16_HA", "BFD_RELOC_PPC_DTPREL", "BFD_RELOC_PPC_GOT_TLSGD16", "BFD_RELOC_PPC_GOT_TLSGD16_LO", "BFD_RELOC_PPC_GOT_TLSGD16_HI", "BFD_RELOC_PPC_GOT_TLSGD16_HA", "BFD_RELOC_PPC_GOT_TLSLD16", "BFD_RELOC_PPC_GOT_TLSLD16_LO", "BFD_RELOC_PPC_GOT_TLSLD16_HI", "BFD_RELOC_PPC_GOT_TLSLD16_HA", "BFD_RELOC_PPC_GOT_TPREL16", "BFD_RELOC_PPC_GOT_TPREL16_LO", "BFD_RELOC_PPC_GOT_TPREL16_HI", "BFD_RELOC_PPC_GOT_TPREL16_HA", "BFD_RELOC_PPC_GOT_DTPREL16", "BFD_RELOC_PPC_GOT_DTPREL16_LO", "BFD_RELOC_PPC_GOT_DTPREL16_HI", "BFD_RELOC_PPC_GOT_DTPREL16_HA", "BFD_RELOC_PPC64_TPREL16_DS", "BFD_RELOC_PPC64_TPREL16_LO_DS", "BFD_RELOC_PPC64_TPREL16_HIGHER", "BFD_RELOC_PPC64_TPREL16_HIGHERA", "BFD_RELOC_PPC64_TPREL16_HIGHEST", "BFD_RELOC_PPC64_TPREL16_HIGHESTA", "BFD_RELOC_PPC64_DTPREL16_DS", "BFD_RELOC_PPC64_DTPREL16_LO_DS", "BFD_RELOC_PPC64_DTPREL16_HIGHER", "BFD_RELOC_PPC64_DTPREL16_HIGHERA", "BFD_RELOC_PPC64_DTPREL16_HIGHEST", "BFD_RELOC_PPC64_DTPREL16_HIGHESTA", "BFD_RELOC_I370_D12", "BFD_RELOC_CTOR", "BFD_RELOC_ARM_PCREL_BRANCH", "BFD_RELOC_ARM_PCREL_BLX", "BFD_RELOC_THUMB_PCREL_BLX", "BFD_RELOC_ARM_PCREL_CALL", "BFD_RELOC_ARM_PCREL_JUMP", "BFD_RELOC_THUMB_PCREL_BRANCH7", "BFD_RELOC_THUMB_PCREL_BRANCH9", "BFD_RELOC_THUMB_PCREL_BRANCH12", "BFD_RELOC_THUMB_PCREL_BRANCH20", "BFD_RELOC_THUMB_PCREL_BRANCH23", "BFD_RELOC_THUMB_PCREL_BRANCH25", "BFD_RELOC_ARM_OFFSET_IMM", "BFD_RELOC_ARM_THUMB_OFFSET", "BFD_RELOC_ARM_TARGET1", "BFD_RELOC_ARM_ROSEGREL32", "BFD_RELOC_ARM_SBREL32", "BFD_RELOC_ARM_TARGET2", "BFD_RELOC_ARM_PREL31", "BFD_RELOC_ARM_MOVW", "BFD_RELOC_ARM_MOVT", "BFD_RELOC_ARM_MOVW_PCREL", "BFD_RELOC_ARM_MOVT_PCREL", "BFD_RELOC_ARM_THUMB_MOVW", "BFD_RELOC_ARM_THUMB_MOVT", "BFD_RELOC_ARM_THUMB_MOVW_PCREL", "BFD_RELOC_ARM_THUMB_MOVT_PCREL", "BFD_RELOC_ARM_JUMP_SLOT", "BFD_RELOC_ARM_GLOB_DAT", "BFD_RELOC_ARM_GOT32", "BFD_RELOC_ARM_PLT32", "BFD_RELOC_ARM_RELATIVE", "BFD_RELOC_ARM_GOTOFF", "BFD_RELOC_ARM_GOTPC", "BFD_RELOC_ARM_TLS_GD32", "BFD_RELOC_ARM_TLS_LDO32", "BFD_RELOC_ARM_TLS_LDM32", "BFD_RELOC_ARM_TLS_DTPOFF32", "BFD_RELOC_ARM_TLS_DTPMOD32", "BFD_RELOC_ARM_TLS_TPOFF32", "BFD_RELOC_ARM_TLS_IE32", "BFD_RELOC_ARM_TLS_LE32", "BFD_RELOC_ARM_ALU_PC_G0_NC", "BFD_RELOC_ARM_ALU_PC_G0", "BFD_RELOC_ARM_ALU_PC_G1_NC", "BFD_RELOC_ARM_ALU_PC_G1", "BFD_RELOC_ARM_ALU_PC_G2", "BFD_RELOC_ARM_LDR_PC_G0", "BFD_RELOC_ARM_LDR_PC_G1", "BFD_RELOC_ARM_LDR_PC_G2", "BFD_RELOC_ARM_LDRS_PC_G0", "BFD_RELOC_ARM_LDRS_PC_G1", "BFD_RELOC_ARM_LDRS_PC_G2", "BFD_RELOC_ARM_LDC_PC_G0", "BFD_RELOC_ARM_LDC_PC_G1", "BFD_RELOC_ARM_LDC_PC_G2", "BFD_RELOC_ARM_ALU_SB_G0_NC", "BFD_RELOC_ARM_ALU_SB_G0", "BFD_RELOC_ARM_ALU_SB_G1_NC", "BFD_RELOC_ARM_ALU_SB_G1", "BFD_RELOC_ARM_ALU_SB_G2", "BFD_RELOC_ARM_LDR_SB_G0", "BFD_RELOC_ARM_LDR_SB_G1", "BFD_RELOC_ARM_LDR_SB_G2", "BFD_RELOC_ARM_LDRS_SB_G0", "BFD_RELOC_ARM_LDRS_SB_G1", "BFD_RELOC_ARM_LDRS_SB_G2", "BFD_RELOC_ARM_LDC_SB_G0", "BFD_RELOC_ARM_LDC_SB_G1", "BFD_RELOC_ARM_LDC_SB_G2", "BFD_RELOC_ARM_IMMEDIATE", "BFD_RELOC_ARM_ADRL_IMMEDIATE", "BFD_RELOC_ARM_T32_IMMEDIATE", "BFD_RELOC_ARM_T32_ADD_IMM", "BFD_RELOC_ARM_T32_IMM12", "BFD_RELOC_ARM_T32_ADD_PC12", "BFD_RELOC_ARM_SHIFT_IMM", "BFD_RELOC_ARM_SMC", "BFD_RELOC_ARM_SWI", "BFD_RELOC_ARM_MULTI", "BFD_RELOC_ARM_CP_OFF_IMM", "BFD_RELOC_ARM_CP_OFF_IMM_S2", "BFD_RELOC_ARM_T32_CP_OFF_IMM", "BFD_RELOC_ARM_T32_CP_OFF_IMM_S2", "BFD_RELOC_ARM_ADR_IMM", "BFD_RELOC_ARM_LDR_IMM", "BFD_RELOC_ARM_LITERAL", "BFD_RELOC_ARM_IN_POOL", "BFD_RELOC_ARM_OFFSET_IMM8", "BFD_RELOC_ARM_T32_OFFSET_U8", "BFD_RELOC_ARM_T32_OFFSET_IMM", "BFD_RELOC_ARM_HWLITERAL", "BFD_RELOC_ARM_THUMB_ADD", "BFD_RELOC_ARM_THUMB_IMM", "BFD_RELOC_ARM_THUMB_SHIFT", "BFD_RELOC_SH_PCDISP8BY2", "BFD_RELOC_SH_PCDISP12BY2", "BFD_RELOC_SH_IMM3", "BFD_RELOC_SH_IMM3U", "BFD_RELOC_SH_DISP12", "BFD_RELOC_SH_DISP12BY2", "BFD_RELOC_SH_DISP12BY4", "BFD_RELOC_SH_DISP12BY8", "BFD_RELOC_SH_DISP20", "BFD_RELOC_SH_DISP20BY8", "BFD_RELOC_SH_IMM4", "BFD_RELOC_SH_IMM4BY2", "BFD_RELOC_SH_IMM4BY4", "BFD_RELOC_SH_IMM8", "BFD_RELOC_SH_IMM8BY2", "BFD_RELOC_SH_IMM8BY4", "BFD_RELOC_SH_PCRELIMM8BY2", "BFD_RELOC_SH_PCRELIMM8BY4", "BFD_RELOC_SH_SWITCH16", "BFD_RELOC_SH_SWITCH32", "BFD_RELOC_SH_USES", "BFD_RELOC_SH_COUNT", "BFD_RELOC_SH_ALIGN", "BFD_RELOC_SH_CODE", "BFD_RELOC_SH_DATA", "BFD_RELOC_SH_LABEL", "BFD_RELOC_SH_LOOP_START", "BFD_RELOC_SH_LOOP_END", "BFD_RELOC_SH_COPY", "BFD_RELOC_SH_GLOB_DAT", "BFD_RELOC_SH_JMP_SLOT", "BFD_RELOC_SH_RELATIVE", "BFD_RELOC_SH_GOTPC", "BFD_RELOC_SH_GOT_LOW16", "BFD_RELOC_SH_GOT_MEDLOW16", "BFD_RELOC_SH_GOT_MEDHI16", "BFD_RELOC_SH_GOT_HI16", "BFD_RELOC_SH_GOTPLT_LOW16", "BFD_RELOC_SH_GOTPLT_MEDLOW16", "BFD_RELOC_SH_GOTPLT_MEDHI16", "BFD_RELOC_SH_GOTPLT_HI16", "BFD_RELOC_SH_PLT_LOW16", "BFD_RELOC_SH_PLT_MEDLOW16", "BFD_RELOC_SH_PLT_MEDHI16", "BFD_RELOC_SH_PLT_HI16", "BFD_RELOC_SH_GOTOFF_LOW16", "BFD_RELOC_SH_GOTOFF_MEDLOW16", "BFD_RELOC_SH_GOTOFF_MEDHI16", "BFD_RELOC_SH_GOTOFF_HI16", "BFD_RELOC_SH_GOTPC_LOW16", "BFD_RELOC_SH_GOTPC_MEDLOW16", "BFD_RELOC_SH_GOTPC_MEDHI16", "BFD_RELOC_SH_GOTPC_HI16", "BFD_RELOC_SH_COPY64", "BFD_RELOC_SH_GLOB_DAT64", "BFD_RELOC_SH_JMP_SLOT64", "BFD_RELOC_SH_RELATIVE64", "BFD_RELOC_SH_GOT10BY4", "BFD_RELOC_SH_GOT10BY8", "BFD_RELOC_SH_GOTPLT10BY4", "BFD_RELOC_SH_GOTPLT10BY8", "BFD_RELOC_SH_GOTPLT32", "BFD_RELOC_SH_SHMEDIA_CODE", "BFD_RELOC_SH_IMMU5", "BFD_RELOC_SH_IMMS6", "BFD_RELOC_SH_IMMS6BY32", "BFD_RELOC_SH_IMMU6", "BFD_RELOC_SH_IMMS10", "BFD_RELOC_SH_IMMS10BY2", "BFD_RELOC_SH_IMMS10BY4", "BFD_RELOC_SH_IMMS10BY8", "BFD_RELOC_SH_IMMS16", "BFD_RELOC_SH_IMMU16", "BFD_RELOC_SH_IMM_LOW16", "BFD_RELOC_SH_IMM_LOW16_PCREL", "BFD_RELOC_SH_IMM_MEDLOW16", "BFD_RELOC_SH_IMM_MEDLOW16_PCREL", "BFD_RELOC_SH_IMM_MEDHI16", "BFD_RELOC_SH_IMM_MEDHI16_PCREL", "BFD_RELOC_SH_IMM_HI16", "BFD_RELOC_SH_IMM_HI16_PCREL", "BFD_RELOC_SH_PT_16", "BFD_RELOC_SH_TLS_GD_32", "BFD_RELOC_SH_TLS_LD_32", "BFD_RELOC_SH_TLS_LDO_32", "BFD_RELOC_SH_TLS_IE_32", "BFD_RELOC_SH_TLS_LE_32", "BFD_RELOC_SH_TLS_DTPMOD32", "BFD_RELOC_SH_TLS_DTPOFF32", "BFD_RELOC_SH_TLS_TPOFF32", "BFD_RELOC_ARC_B22_PCREL", "BFD_RELOC_ARC_B26", "BFD_RELOC_BFIN_16_IMM", "BFD_RELOC_BFIN_16_HIGH", "BFD_RELOC_BFIN_4_PCREL", "BFD_RELOC_BFIN_5_PCREL", "BFD_RELOC_BFIN_16_LOW", "BFD_RELOC_BFIN_10_PCREL", "BFD_RELOC_BFIN_11_PCREL", "BFD_RELOC_BFIN_12_PCREL_JUMP", "BFD_RELOC_BFIN_12_PCREL_JUMP_S", "BFD_RELOC_BFIN_24_PCREL_CALL_X", "BFD_RELOC_BFIN_24_PCREL_JUMP_L", "BFD_RELOC_BFIN_GOT17M4", "BFD_RELOC_BFIN_GOTHI", "BFD_RELOC_BFIN_GOTLO", "BFD_RELOC_BFIN_FUNCDESC", "BFD_RELOC_BFIN_FUNCDESC_GOT17M4", "BFD_RELOC_BFIN_FUNCDESC_GOTHI", "BFD_RELOC_BFIN_FUNCDESC_GOTLO", "BFD_RELOC_BFIN_FUNCDESC_VALUE", "BFD_RELOC_BFIN_FUNCDESC_GOTOFF17M4", "BFD_RELOC_BFIN_FUNCDESC_GOTOFFHI", "BFD_RELOC_BFIN_FUNCDESC_GOTOFFLO", "BFD_RELOC_BFIN_GOTOFF17M4", "BFD_RELOC_BFIN_GOTOFFHI", "BFD_RELOC_BFIN_GOTOFFLO", "BFD_RELOC_BFIN_GOT", "BFD_RELOC_BFIN_PLTPC", "BFD_ARELOC_BFIN_PUSH", "BFD_ARELOC_BFIN_CONST", "BFD_ARELOC_BFIN_ADD", "BFD_ARELOC_BFIN_SUB", "BFD_ARELOC_BFIN_MULT", "BFD_ARELOC_BFIN_DIV", "BFD_ARELOC_BFIN_MOD", "BFD_ARELOC_BFIN_LSHIFT", "BFD_ARELOC_BFIN_RSHIFT", "BFD_ARELOC_BFIN_AND", "BFD_ARELOC_BFIN_OR", "BFD_ARELOC_BFIN_XOR", "BFD_ARELOC_BFIN_LAND", "BFD_ARELOC_BFIN_LOR", "BFD_ARELOC_BFIN_LEN", "BFD_ARELOC_BFIN_NEG", "BFD_ARELOC_BFIN_COMP", "BFD_ARELOC_BFIN_PAGE", "BFD_ARELOC_BFIN_HWPAGE", "BFD_ARELOC_BFIN_ADDR", "BFD_RELOC_D10V_10_PCREL_R", "BFD_RELOC_D10V_10_PCREL_L", "BFD_RELOC_D10V_18", "BFD_RELOC_D10V_18_PCREL", "BFD_RELOC_D30V_6", "BFD_RELOC_D30V_9_PCREL", "BFD_RELOC_D30V_9_PCREL_R", "BFD_RELOC_D30V_15", "BFD_RELOC_D30V_15_PCREL", "BFD_RELOC_D30V_15_PCREL_R", "BFD_RELOC_D30V_21", "BFD_RELOC_D30V_21_PCREL", "BFD_RELOC_D30V_21_PCREL_R", "BFD_RELOC_D30V_32", "BFD_RELOC_D30V_32_PCREL", "BFD_RELOC_DLX_HI16_S", "BFD_RELOC_DLX_LO16", "BFD_RELOC_DLX_JMP26", "BFD_RELOC_M32C_HI8", "BFD_RELOC_M32C_RL_JUMP", "BFD_RELOC_M32C_RL_1ADDR", "BFD_RELOC_M32C_RL_2ADDR", "BFD_RELOC_M32R_24", "BFD_RELOC_M32R_10_PCREL", "BFD_RELOC_M32R_18_PCREL", "BFD_RELOC_M32R_26_PCREL", "BFD_RELOC_M32R_HI16_ULO", "BFD_RELOC_M32R_HI16_SLO", "BFD_RELOC_M32R_LO16", "BFD_RELOC_M32R_SDA16", "BFD_RELOC_M32R_GOT24", "BFD_RELOC_M32R_26_PLTREL", "BFD_RELOC_M32R_COPY", "BFD_RELOC_M32R_GLOB_DAT", "BFD_RELOC_M32R_JMP_SLOT", "BFD_RELOC_M32R_RELATIVE", "BFD_RELOC_M32R_GOTOFF", "BFD_RELOC_M32R_GOTOFF_HI_ULO", "BFD_RELOC_M32R_GOTOFF_HI_SLO", "BFD_RELOC_M32R_GOTOFF_LO", "BFD_RELOC_M32R_GOTPC24", "BFD_RELOC_M32R_GOT16_HI_ULO", "BFD_RELOC_M32R_GOT16_HI_SLO", "BFD_RELOC_M32R_GOT16_LO", "BFD_RELOC_M32R_GOTPC_HI_ULO", "BFD_RELOC_M32R_GOTPC_HI_SLO", "BFD_RELOC_M32R_GOTPC_LO", "BFD_RELOC_V850_9_PCREL", "BFD_RELOC_V850_22_PCREL", "BFD_RELOC_V850_SDA_16_16_OFFSET", "BFD_RELOC_V850_SDA_15_16_OFFSET", "BFD_RELOC_V850_ZDA_16_16_OFFSET", "BFD_RELOC_V850_ZDA_15_16_OFFSET", "BFD_RELOC_V850_TDA_6_8_OFFSET", "BFD_RELOC_V850_TDA_7_8_OFFSET", "BFD_RELOC_V850_TDA_7_7_OFFSET", "BFD_RELOC_V850_TDA_16_16_OFFSET", "BFD_RELOC_V850_TDA_4_5_OFFSET", "BFD_RELOC_V850_TDA_4_4_OFFSET", "BFD_RELOC_V850_SDA_16_16_SPLIT_OFFSET", "BFD_RELOC_V850_ZDA_16_16_SPLIT_OFFSET", "BFD_RELOC_V850_CALLT_6_7_OFFSET", "BFD_RELOC_V850_CALLT_16_16_OFFSET", "BFD_RELOC_V850_LONGCALL", "BFD_RELOC_V850_LONGJUMP", "BFD_RELOC_V850_ALIGN", "BFD_RELOC_V850_LO16_SPLIT_OFFSET", "BFD_RELOC_MN10300_32_PCREL", "BFD_RELOC_MN10300_16_PCREL", "BFD_RELOC_TIC30_LDP", "BFD_RELOC_TIC54X_PARTLS7", "BFD_RELOC_TIC54X_PARTMS9", "BFD_RELOC_TIC54X_23", "BFD_RELOC_TIC54X_16_OF_23", "BFD_RELOC_TIC54X_MS7_OF_23", "BFD_RELOC_FR30_48", "BFD_RELOC_FR30_20", "BFD_RELOC_FR30_6_IN_4", "BFD_RELOC_FR30_8_IN_8", "BFD_RELOC_FR30_9_IN_8", "BFD_RELOC_FR30_10_IN_8", "BFD_RELOC_FR30_9_PCREL", "BFD_RELOC_FR30_12_PCREL", "BFD_RELOC_MCORE_PCREL_IMM8BY4", "BFD_RELOC_MCORE_PCREL_IMM11BY2", "BFD_RELOC_MCORE_PCREL_IMM4BY2", "BFD_RELOC_MCORE_PCREL_32", "BFD_RELOC_MCORE_PCREL_JSR_IMM11BY2", "BFD_RELOC_MCORE_RVA", "BFD_RELOC_MEP_8", "BFD_RELOC_MEP_16", "BFD_RELOC_MEP_32", "BFD_RELOC_MEP_PCREL8A2", "BFD_RELOC_MEP_PCREL12A2", "BFD_RELOC_MEP_PCREL17A2", "BFD_RELOC_MEP_PCREL24A2", "BFD_RELOC_MEP_PCABS24A2", "BFD_RELOC_MEP_LOW16", "BFD_RELOC_MEP_HI16U", "BFD_RELOC_MEP_HI16S", "BFD_RELOC_MEP_GPREL", "BFD_RELOC_MEP_TPREL", "BFD_RELOC_MEP_TPREL7", "BFD_RELOC_MEP_TPREL7A2", "BFD_RELOC_MEP_TPREL7A4", "BFD_RELOC_MEP_UIMM24", "BFD_RELOC_MEP_ADDR24A4", "BFD_RELOC_MEP_GNU_VTINHERIT", "BFD_RELOC_MEP_GNU_VTENTRY", "BFD_RELOC_MMIX_GETA", "BFD_RELOC_MMIX_GETA_1", "BFD_RELOC_MMIX_GETA_2", "BFD_RELOC_MMIX_GETA_3", "BFD_RELOC_MMIX_CBRANCH", "BFD_RELOC_MMIX_CBRANCH_J", "BFD_RELOC_MMIX_CBRANCH_1", "BFD_RELOC_MMIX_CBRANCH_2", "BFD_RELOC_MMIX_CBRANCH_3", "BFD_RELOC_MMIX_PUSHJ", "BFD_RELOC_MMIX_PUSHJ_1", "BFD_RELOC_MMIX_PUSHJ_2", "BFD_RELOC_MMIX_PUSHJ_3", "BFD_RELOC_MMIX_PUSHJ_STUBBABLE", "BFD_RELOC_MMIX_JMP", "BFD_RELOC_MMIX_JMP_1", "BFD_RELOC_MMIX_JMP_2", "BFD_RELOC_MMIX_JMP_3", "BFD_RELOC_MMIX_ADDR19", "BFD_RELOC_MMIX_ADDR27", "BFD_RELOC_MMIX_REG_OR_BYTE", "BFD_RELOC_MMIX_REG", "BFD_RELOC_MMIX_BASE_PLUS_OFFSET", "BFD_RELOC_MMIX_LOCAL", "BFD_RELOC_AVR_7_PCREL", "BFD_RELOC_AVR_13_PCREL", "BFD_RELOC_AVR_16_PM", "BFD_RELOC_AVR_LO8_LDI", "BFD_RELOC_AVR_HI8_LDI", "BFD_RELOC_AVR_HH8_LDI", "BFD_RELOC_AVR_MS8_LDI", "BFD_RELOC_AVR_LO8_LDI_NEG", "BFD_RELOC_AVR_HI8_LDI_NEG", "BFD_RELOC_AVR_HH8_LDI_NEG", "BFD_RELOC_AVR_MS8_LDI_NEG", "BFD_RELOC_AVR_LO8_LDI_PM", "BFD_RELOC_AVR_LO8_LDI_GS", "BFD_RELOC_AVR_HI8_LDI_PM", "BFD_RELOC_AVR_HI8_LDI_GS", "BFD_RELOC_AVR_HH8_LDI_PM", "BFD_RELOC_AVR_LO8_LDI_PM_NEG", "BFD_RELOC_AVR_HI8_LDI_PM_NEG", "BFD_RELOC_AVR_HH8_LDI_PM_NEG", "BFD_RELOC_AVR_CALL", "BFD_RELOC_AVR_LDI", "BFD_RELOC_AVR_6", "BFD_RELOC_AVR_6_ADIW", "BFD_RELOC_390_12", "BFD_RELOC_390_GOT12", "BFD_RELOC_390_PLT32", "BFD_RELOC_390_COPY", "BFD_RELOC_390_GLOB_DAT", "BFD_RELOC_390_JMP_SLOT", "BFD_RELOC_390_RELATIVE", "BFD_RELOC_390_GOTPC", "BFD_RELOC_390_GOT16", "BFD_RELOC_390_PC16DBL", "BFD_RELOC_390_PLT16DBL", "BFD_RELOC_390_PC32DBL", "BFD_RELOC_390_PLT32DBL", "BFD_RELOC_390_GOTPCDBL", "BFD_RELOC_390_GOT64", "BFD_RELOC_390_PLT64", "BFD_RELOC_390_GOTENT", "BFD_RELOC_390_GOTOFF64", "BFD_RELOC_390_GOTPLT12", "BFD_RELOC_390_GOTPLT16", "BFD_RELOC_390_GOTPLT32", "BFD_RELOC_390_GOTPLT64", "BFD_RELOC_390_GOTPLTENT", "BFD_RELOC_390_PLTOFF16", "BFD_RELOC_390_PLTOFF32", "BFD_RELOC_390_PLTOFF64", "BFD_RELOC_390_TLS_LOAD", "BFD_RELOC_390_TLS_GDCALL", "BFD_RELOC_390_TLS_LDCALL", "BFD_RELOC_390_TLS_GD32", "BFD_RELOC_390_TLS_GD64", "BFD_RELOC_390_TLS_GOTIE12", "BFD_RELOC_390_TLS_GOTIE32", "BFD_RELOC_390_TLS_GOTIE64", "BFD_RELOC_390_TLS_LDM32", "BFD_RELOC_390_TLS_LDM64", "BFD_RELOC_390_TLS_IE32", "BFD_RELOC_390_TLS_IE64", "BFD_RELOC_390_TLS_IEENT", "BFD_RELOC_390_TLS_LE32", "BFD_RELOC_390_TLS_LE64", "BFD_RELOC_390_TLS_LDO32", "BFD_RELOC_390_TLS_LDO64", "BFD_RELOC_390_TLS_DTPMOD", "BFD_RELOC_390_TLS_DTPOFF", "BFD_RELOC_390_TLS_TPOFF", "BFD_RELOC_390_20", "BFD_RELOC_390_GOT20", "BFD_RELOC_390_GOTPLT20", "BFD_RELOC_390_TLS_GOTIE20", "BFD_RELOC_SCORE_DUMMY1", "BFD_RELOC_SCORE_GPREL15", "BFD_RELOC_SCORE_DUMMY2", "BFD_RELOC_SCORE_JMP", "BFD_RELOC_SCORE_BRANCH", "BFD_RELOC_SCORE16_JMP", "BFD_RELOC_SCORE16_BRANCH", "BFD_RELOC_SCORE_GOT15", "BFD_RELOC_SCORE_GOT_LO16", "BFD_RELOC_SCORE_CALL15", "BFD_RELOC_SCORE_DUMMY_HI16", "BFD_RELOC_IP2K_FR9", "BFD_RELOC_IP2K_BANK", "BFD_RELOC_IP2K_ADDR16CJP", "BFD_RELOC_IP2K_PAGE3", "BFD_RELOC_IP2K_LO8DATA", "BFD_RELOC_IP2K_HI8DATA", "BFD_RELOC_IP2K_EX8DATA", "BFD_RELOC_IP2K_LO8INSN", "BFD_RELOC_IP2K_HI8INSN", "BFD_RELOC_IP2K_PC_SKIP", "BFD_RELOC_IP2K_TEXT", "BFD_RELOC_IP2K_FR_OFFSET", "BFD_RELOC_VPE4KMATH_DATA", "BFD_RELOC_VPE4KMATH_INSN", "BFD_RELOC_VTABLE_INHERIT", "BFD_RELOC_VTABLE_ENTRY", "BFD_RELOC_IA64_IMM14", "BFD_RELOC_IA64_IMM22", "BFD_RELOC_IA64_IMM64", "BFD_RELOC_IA64_DIR32MSB", "BFD_RELOC_IA64_DIR32LSB", "BFD_RELOC_IA64_DIR64MSB", "BFD_RELOC_IA64_DIR64LSB", "BFD_RELOC_IA64_GPREL22", "BFD_RELOC_IA64_GPREL64I", "BFD_RELOC_IA64_GPREL32MSB", "BFD_RELOC_IA64_GPREL32LSB", "BFD_RELOC_IA64_GPREL64MSB", "BFD_RELOC_IA64_GPREL64LSB", "BFD_RELOC_IA64_LTOFF22", "BFD_RELOC_IA64_LTOFF64I", "BFD_RELOC_IA64_PLTOFF22", "BFD_RELOC_IA64_PLTOFF64I", "BFD_RELOC_IA64_PLTOFF64MSB", "BFD_RELOC_IA64_PLTOFF64LSB", "BFD_RELOC_IA64_FPTR64I", "BFD_RELOC_IA64_FPTR32MSB", "BFD_RELOC_IA64_FPTR32LSB", "BFD_RELOC_IA64_FPTR64MSB", "BFD_RELOC_IA64_FPTR64LSB", "BFD_RELOC_IA64_PCREL21B", "BFD_RELOC_IA64_PCREL21BI", "BFD_RELOC_IA64_PCREL21M", "BFD_RELOC_IA64_PCREL21F", "BFD_RELOC_IA64_PCREL22", "BFD_RELOC_IA64_PCREL60B", "BFD_RELOC_IA64_PCREL64I", "BFD_RELOC_IA64_PCREL32MSB", "BFD_RELOC_IA64_PCREL32LSB", "BFD_RELOC_IA64_PCREL64MSB", "BFD_RELOC_IA64_PCREL64LSB", "BFD_RELOC_IA64_LTOFF_FPTR22", "BFD_RELOC_IA64_LTOFF_FPTR64I", "BFD_RELOC_IA64_LTOFF_FPTR32MSB", "BFD_RELOC_IA64_LTOFF_FPTR32LSB", "BFD_RELOC_IA64_LTOFF_FPTR64MSB", "BFD_RELOC_IA64_LTOFF_FPTR64LSB", "BFD_RELOC_IA64_SEGREL32MSB", "BFD_RELOC_IA64_SEGREL32LSB", "BFD_RELOC_IA64_SEGREL64MSB", "BFD_RELOC_IA64_SEGREL64LSB", "BFD_RELOC_IA64_SECREL32MSB", "BFD_RELOC_IA64_SECREL32LSB", "BFD_RELOC_IA64_SECREL64MSB", "BFD_RELOC_IA64_SECREL64LSB", "BFD_RELOC_IA64_REL32MSB", "BFD_RELOC_IA64_REL32LSB", "BFD_RELOC_IA64_REL64MSB", "BFD_RELOC_IA64_REL64LSB", "BFD_RELOC_IA64_LTV32MSB", "BFD_RELOC_IA64_LTV32LSB", "BFD_RELOC_IA64_LTV64MSB", "BFD_RELOC_IA64_LTV64LSB", "BFD_RELOC_IA64_IPLTMSB", "BFD_RELOC_IA64_IPLTLSB", "BFD_RELOC_IA64_COPY", "BFD_RELOC_IA64_LTOFF22X", "BFD_RELOC_IA64_LDXMOV", "BFD_RELOC_IA64_TPREL14", "BFD_RELOC_IA64_TPREL22", "BFD_RELOC_IA64_TPREL64I", "BFD_RELOC_IA64_TPREL64MSB", "BFD_RELOC_IA64_TPREL64LSB", "BFD_RELOC_IA64_LTOFF_TPREL22", "BFD_RELOC_IA64_DTPMOD64MSB", "BFD_RELOC_IA64_DTPMOD64LSB", "BFD_RELOC_IA64_LTOFF_DTPMOD22", "BFD_RELOC_IA64_DTPREL14", "BFD_RELOC_IA64_DTPREL22", "BFD_RELOC_IA64_DTPREL64I", "BFD_RELOC_IA64_DTPREL32MSB", "BFD_RELOC_IA64_DTPREL32LSB", "BFD_RELOC_IA64_DTPREL64MSB", "BFD_RELOC_IA64_DTPREL64LSB", "BFD_RELOC_IA64_LTOFF_DTPREL22", "BFD_RELOC_M68HC11_HI8", "BFD_RELOC_M68HC11_LO8", "BFD_RELOC_M68HC11_3B", "BFD_RELOC_M68HC11_RL_JUMP", "BFD_RELOC_M68HC11_RL_GROUP", "BFD_RELOC_M68HC11_LO16", "BFD_RELOC_M68HC11_PAGE", "BFD_RELOC_M68HC11_24", "BFD_RELOC_M68HC12_5B", "BFD_RELOC_16C_NUM08", "BFD_RELOC_16C_NUM08_C", "BFD_RELOC_16C_NUM16", "BFD_RELOC_16C_NUM16_C", "BFD_RELOC_16C_NUM32", "BFD_RELOC_16C_NUM32_C", "BFD_RELOC_16C_DISP04", "BFD_RELOC_16C_DISP04_C", "BFD_RELOC_16C_DISP08", "BFD_RELOC_16C_DISP08_C", "BFD_RELOC_16C_DISP16", "BFD_RELOC_16C_DISP16_C", "BFD_RELOC_16C_DISP24", "BFD_RELOC_16C_DISP24_C", "BFD_RELOC_16C_DISP24a", "BFD_RELOC_16C_DISP24a_C", "BFD_RELOC_16C_REG04", "BFD_RELOC_16C_REG04_C", "BFD_RELOC_16C_REG04a", "BFD_RELOC_16C_REG04a_C", "BFD_RELOC_16C_REG14", "BFD_RELOC_16C_REG14_C", "BFD_RELOC_16C_REG16", "BFD_RELOC_16C_REG16_C", "BFD_RELOC_16C_REG20", "BFD_RELOC_16C_REG20_C", "BFD_RELOC_16C_ABS20", "BFD_RELOC_16C_ABS20_C", "BFD_RELOC_16C_ABS24", "BFD_RELOC_16C_ABS24_C", "BFD_RELOC_16C_IMM04", "BFD_RELOC_16C_IMM04_C", "BFD_RELOC_16C_IMM16", "BFD_RELOC_16C_IMM16_C", "BFD_RELOC_16C_IMM20", "BFD_RELOC_16C_IMM20_C", "BFD_RELOC_16C_IMM24", "BFD_RELOC_16C_IMM24_C", "BFD_RELOC_16C_IMM32", "BFD_RELOC_16C_IMM32_C", "BFD_RELOC_CR16_NUM8", "BFD_RELOC_CR16_NUM16", "BFD_RELOC_CR16_NUM32", "BFD_RELOC_CR16_NUM32a", "BFD_RELOC_CR16_REGREL0", "BFD_RELOC_CR16_REGREL4", "BFD_RELOC_CR16_REGREL4a", "BFD_RELOC_CR16_REGREL14", "BFD_RELOC_CR16_REGREL14a", "BFD_RELOC_CR16_REGREL16", "BFD_RELOC_CR16_REGREL20", "BFD_RELOC_CR16_REGREL20a", "BFD_RELOC_CR16_ABS20", "BFD_RELOC_CR16_ABS24", "BFD_RELOC_CR16_IMM4", "BFD_RELOC_CR16_IMM8", "BFD_RELOC_CR16_IMM16", "BFD_RELOC_CR16_IMM20", "BFD_RELOC_CR16_IMM24", "BFD_RELOC_CR16_IMM32", "BFD_RELOC_CR16_IMM32a", "BFD_RELOC_CR16_DISP4", "BFD_RELOC_CR16_DISP8", "BFD_RELOC_CR16_DISP16", "BFD_RELOC_CR16_DISP20", "BFD_RELOC_CR16_DISP24", "BFD_RELOC_CR16_DISP24a", "BFD_RELOC_CRX_REL4", "BFD_RELOC_CRX_REL8", "BFD_RELOC_CRX_REL8_CMP", "BFD_RELOC_CRX_REL16", "BFD_RELOC_CRX_REL24", "BFD_RELOC_CRX_REL32", "BFD_RELOC_CRX_REGREL12", "BFD_RELOC_CRX_REGREL22", "BFD_RELOC_CRX_REGREL28", "BFD_RELOC_CRX_REGREL32", "BFD_RELOC_CRX_ABS16", "BFD_RELOC_CRX_ABS32", "BFD_RELOC_CRX_NUM8", "BFD_RELOC_CRX_NUM16", "BFD_RELOC_CRX_NUM32", "BFD_RELOC_CRX_IMM16", "BFD_RELOC_CRX_IMM32", "BFD_RELOC_CRX_SWITCH8", "BFD_RELOC_CRX_SWITCH16", "BFD_RELOC_CRX_SWITCH32", "BFD_RELOC_CRIS_BDISP8", "BFD_RELOC_CRIS_UNSIGNED_5", "BFD_RELOC_CRIS_SIGNED_6", "BFD_RELOC_CRIS_UNSIGNED_6", "BFD_RELOC_CRIS_SIGNED_8", "BFD_RELOC_CRIS_UNSIGNED_8", "BFD_RELOC_CRIS_SIGNED_16", "BFD_RELOC_CRIS_UNSIGNED_16", "BFD_RELOC_CRIS_LAPCQ_OFFSET", "BFD_RELOC_CRIS_UNSIGNED_4", "BFD_RELOC_CRIS_COPY", "BFD_RELOC_CRIS_GLOB_DAT", "BFD_RELOC_CRIS_JUMP_SLOT", "BFD_RELOC_CRIS_RELATIVE", "BFD_RELOC_CRIS_32_GOT", "BFD_RELOC_CRIS_16_GOT", "BFD_RELOC_CRIS_32_GOTPLT", "BFD_RELOC_CRIS_16_GOTPLT", "BFD_RELOC_CRIS_32_GOTREL", "BFD_RELOC_CRIS_32_PLT_GOTREL", "BFD_RELOC_CRIS_32_PLT_PCREL", "BFD_RELOC_860_COPY", "BFD_RELOC_860_GLOB_DAT", "BFD_RELOC_860_JUMP_SLOT", "BFD_RELOC_860_RELATIVE", "BFD_RELOC_860_PC26", "BFD_RELOC_860_PLT26", "BFD_RELOC_860_PC16", "BFD_RELOC_860_LOW0", "BFD_RELOC_860_SPLIT0", "BFD_RELOC_860_LOW1", "BFD_RELOC_860_SPLIT1", "BFD_RELOC_860_LOW2", "BFD_RELOC_860_SPLIT2", "BFD_RELOC_860_LOW3", "BFD_RELOC_860_LOGOT0", "BFD_RELOC_860_SPGOT0", "BFD_RELOC_860_LOGOT1", "BFD_RELOC_860_SPGOT1", "BFD_RELOC_860_LOGOTOFF0", "BFD_RELOC_860_SPGOTOFF0", "BFD_RELOC_860_LOGOTOFF1", "BFD_RELOC_860_SPGOTOFF1", "BFD_RELOC_860_LOGOTOFF2", "BFD_RELOC_860_LOGOTOFF3", "BFD_RELOC_860_LOPC", "BFD_RELOC_860_HIGHADJ", "BFD_RELOC_860_HAGOT", "BFD_RELOC_860_HAGOTOFF", "BFD_RELOC_860_HAPC", "BFD_RELOC_860_HIGH", "BFD_RELOC_860_HIGOT", "BFD_RELOC_860_HIGOTOFF", "BFD_RELOC_OPENRISC_ABS_26", "BFD_RELOC_OPENRISC_REL_26", "BFD_RELOC_H8_DIR16A8", "BFD_RELOC_H8_DIR16R8", "BFD_RELOC_H8_DIR24A8", "BFD_RELOC_H8_DIR24R8", "BFD_RELOC_H8_DIR32A16", "BFD_RELOC_XSTORMY16_REL_12", "BFD_RELOC_XSTORMY16_12", "BFD_RELOC_XSTORMY16_24", "BFD_RELOC_XSTORMY16_FPTR16", "BFD_RELOC_RELC", "BFD_RELOC_XC16X_PAG", "BFD_RELOC_XC16X_POF", "BFD_RELOC_XC16X_SEG", "BFD_RELOC_XC16X_SOF", "BFD_RELOC_VAX_GLOB_DAT", "BFD_RELOC_VAX_JMP_SLOT", "BFD_RELOC_VAX_RELATIVE", "BFD_RELOC_MT_PC16", "BFD_RELOC_MT_HI16", "BFD_RELOC_MT_LO16", "BFD_RELOC_MT_GNU_VTINHERIT", "BFD_RELOC_MT_GNU_VTENTRY", "BFD_RELOC_MT_PCINSN8", "BFD_RELOC_MSP430_10_PCREL", "BFD_RELOC_MSP430_16_PCREL", "BFD_RELOC_MSP430_16", "BFD_RELOC_MSP430_16_PCREL_BYTE", "BFD_RELOC_MSP430_16_BYTE", "BFD_RELOC_MSP430_2X_PCREL", "BFD_RELOC_MSP430_RL_PCREL", "BFD_RELOC_IQ2000_OFFSET_16", "BFD_RELOC_IQ2000_OFFSET_21", "BFD_RELOC_IQ2000_UHI16", "BFD_RELOC_XTENSA_RTLD", "BFD_RELOC_XTENSA_GLOB_DAT", "BFD_RELOC_XTENSA_JMP_SLOT", "BFD_RELOC_XTENSA_RELATIVE", "BFD_RELOC_XTENSA_PLT", "BFD_RELOC_XTENSA_DIFF8", "BFD_RELOC_XTENSA_DIFF16", "BFD_RELOC_XTENSA_DIFF32", "BFD_RELOC_XTENSA_SLOT0_OP", "BFD_RELOC_XTENSA_SLOT1_OP", "BFD_RELOC_XTENSA_SLOT2_OP", "BFD_RELOC_XTENSA_SLOT3_OP", "BFD_RELOC_XTENSA_SLOT4_OP", "BFD_RELOC_XTENSA_SLOT5_OP", "BFD_RELOC_XTENSA_SLOT6_OP", "BFD_RELOC_XTENSA_SLOT7_OP", "BFD_RELOC_XTENSA_SLOT8_OP", "BFD_RELOC_XTENSA_SLOT9_OP", "BFD_RELOC_XTENSA_SLOT10_OP", "BFD_RELOC_XTENSA_SLOT11_OP", "BFD_RELOC_XTENSA_SLOT12_OP", "BFD_RELOC_XTENSA_SLOT13_OP", "BFD_RELOC_XTENSA_SLOT14_OP", "BFD_RELOC_XTENSA_SLOT0_ALT", "BFD_RELOC_XTENSA_SLOT1_ALT", "BFD_RELOC_XTENSA_SLOT2_ALT", "BFD_RELOC_XTENSA_SLOT3_ALT", "BFD_RELOC_XTENSA_SLOT4_ALT", "BFD_RELOC_XTENSA_SLOT5_ALT", "BFD_RELOC_XTENSA_SLOT6_ALT", "BFD_RELOC_XTENSA_SLOT7_ALT", "BFD_RELOC_XTENSA_SLOT8_ALT", "BFD_RELOC_XTENSA_SLOT9_ALT", "BFD_RELOC_XTENSA_SLOT10_ALT", "BFD_RELOC_XTENSA_SLOT11_ALT", "BFD_RELOC_XTENSA_SLOT12_ALT", "BFD_RELOC_XTENSA_SLOT13_ALT", "BFD_RELOC_XTENSA_SLOT14_ALT", "BFD_RELOC_XTENSA_OP0", "BFD_RELOC_XTENSA_OP1", "BFD_RELOC_XTENSA_OP2", "BFD_RELOC_XTENSA_ASM_EXPAND", "BFD_RELOC_XTENSA_ASM_SIMPLIFY", "BFD_RELOC_Z80_DISP8", "BFD_RELOC_Z8K_DISP7", "BFD_RELOC_Z8K_CALLR", "BFD_RELOC_Z8K_IMM4L", "@@overflow: BFD_RELOC_UNUSED@@", }; #endif reloc_howto_type *bfd_default_reloc_type_lookup (bfd *abfd, bfd_reloc_code_real_type code); bfd_boolean bfd_generic_relax_section (bfd *abfd, asection *section, struct bfd_link_info *, bfd_boolean *); bfd_boolean bfd_generic_gc_sections (bfd *, struct bfd_link_info *); bfd_boolean bfd_generic_merge_sections (bfd *, struct bfd_link_info *); bfd_byte *bfd_generic_get_relocated_section_contents (bfd *abfd, struct bfd_link_info *link_info, struct bfd_link_order *link_order, bfd_byte *data, bfd_boolean relocatable, asymbol **symbols); /* Extracted from archures.c. */ extern const bfd_arch_info_type bfd_default_arch_struct; bfd_boolean bfd_default_set_arch_mach (bfd *abfd, enum bfd_architecture arch, unsigned long mach); const bfd_arch_info_type *bfd_default_compatible (const bfd_arch_info_type *a, const bfd_arch_info_type *b); bfd_boolean bfd_default_scan (const struct bfd_arch_info *info, const char *string); /* Extracted from elf.c. */ struct elf_internal_shdr *bfd_elf_find_section (bfd *abfd, char *name); Index: projects/clang350-import/contrib/binutils/bfd/reloc.c =================================================================== --- projects/clang350-import/contrib/binutils/bfd/reloc.c (revision 275748) +++ projects/clang350-import/contrib/binutils/bfd/reloc.c (revision 275749) @@ -1,5299 +1,5303 @@ /* BFD support for handling relocation entries. Copyright 1990, 1991, 1992, 1993, 1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007 Free Software Foundation, Inc. Written by Cygnus Support. This file is part of BFD, the Binary File Descriptor library. This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin Street - Fifth Floor, Boston, MA 02110-1301, USA. */ /* SECTION Relocations BFD maintains relocations in much the same way it maintains symbols: they are left alone until required, then read in en-masse and translated into an internal form. A common routine <> acts upon the canonical form to do the fixup. Relocations are maintained on a per section basis, while symbols are maintained on a per BFD basis. All that a back end has to do to fit the BFD interface is to create a <> for each relocation in a particular section, and fill in the right bits of the structures. @menu @* typedef arelent:: @* howto manager:: @end menu */ /* DO compile in the reloc_code name table from libbfd.h. */ #define _BFD_MAKE_TABLE_bfd_reloc_code_real #include "sysdep.h" #include "bfd.h" #include "bfdlink.h" #include "libbfd.h" /* DOCDD INODE typedef arelent, howto manager, Relocations, Relocations SUBSECTION typedef arelent This is the structure of a relocation entry: CODE_FRAGMENT . .typedef enum bfd_reloc_status .{ . {* No errors detected. *} . bfd_reloc_ok, . . {* The relocation was performed, but there was an overflow. *} . bfd_reloc_overflow, . . {* The address to relocate was not within the section supplied. *} . bfd_reloc_outofrange, . . {* Used by special functions. *} . bfd_reloc_continue, . . {* Unsupported relocation size requested. *} . bfd_reloc_notsupported, . . {* Unused. *} . bfd_reloc_other, . . {* The symbol to relocate against was undefined. *} . bfd_reloc_undefined, . . {* The relocation was performed, but may not be ok - presently . generated only when linking i960 coff files with i960 b.out . symbols. If this type is returned, the error_message argument . to bfd_perform_relocation will be set. *} . bfd_reloc_dangerous . } . bfd_reloc_status_type; . . .typedef struct reloc_cache_entry .{ . {* A pointer into the canonical table of pointers. *} . struct bfd_symbol **sym_ptr_ptr; . . {* offset in section. *} . bfd_size_type address; . . {* addend for relocation value. *} . bfd_vma addend; . . {* Pointer to how to perform the required relocation. *} . reloc_howto_type *howto; . .} .arelent; . */ /* DESCRIPTION Here is a description of each of the fields within an <>: o <> The symbol table pointer points to a pointer to the symbol associated with the relocation request. It is the pointer into the table returned by the back end's <> action. @xref{Symbols}. The symbol is referenced through a pointer to a pointer so that tools like the linker can fix up all the symbols of the same name by modifying only one pointer. The relocation routine looks in the symbol and uses the base of the section the symbol is attached to and the value of the symbol as the initial relocation offset. If the symbol pointer is zero, then the section provided is looked up. o <
> The <
> field gives the offset in bytes from the base of the section data which owns the relocation record to the first byte of relocatable information. The actual data relocated will be relative to this point; for example, a relocation type which modifies the bottom two bytes of a four byte word would not touch the first byte pointed to in a big endian world. o <> The <> is a value provided by the back end to be added (!) to the relocation offset. Its interpretation is dependent upon the howto. For example, on the 68k the code: | char foo[]; | main() | { | return foo[0x12345678]; | } Could be compiled into: | linkw fp,#-4 | moveb @@#12345678,d0 | extbl d0 | unlk fp | rts This could create a reloc pointing to <>, but leave the offset in the data, something like: |RELOCATION RECORDS FOR [.text]: |offset type value |00000006 32 _foo | |00000000 4e56 fffc ; linkw fp,#-4 |00000004 1039 1234 5678 ; moveb @@#12345678,d0 |0000000a 49c0 ; extbl d0 |0000000c 4e5e ; unlk fp |0000000e 4e75 ; rts Using coff and an 88k, some instructions don't have enough space in them to represent the full address range, and pointers have to be loaded in two parts. So you'd get something like: | or.u r13,r0,hi16(_foo+0x12345678) | ld.b r2,r13,lo16(_foo+0x12345678) | jmp r1 This should create two relocs, both pointing to <<_foo>>, and with 0x12340000 in their addend field. The data would consist of: |RELOCATION RECORDS FOR [.text]: |offset type value |00000002 HVRT16 _foo+0x12340000 |00000006 LVRT16 _foo+0x12340000 | |00000000 5da05678 ; or.u r13,r0,0x5678 |00000004 1c4d5678 ; ld.b r2,r13,0x5678 |00000008 f400c001 ; jmp r1 The relocation routine digs out the value from the data, adds it to the addend to get the original offset, and then adds the value of <<_foo>>. Note that all 32 bits have to be kept around somewhere, to cope with carry from bit 15 to bit 16. One further example is the sparc and the a.out format. The sparc has a similar problem to the 88k, in that some instructions don't have room for an entire offset, but on the sparc the parts are created in odd sized lumps. The designers of the a.out format chose to not use the data within the section for storing part of the offset; all the offset is kept within the reloc. Anything in the data should be ignored. | save %sp,-112,%sp | sethi %hi(_foo+0x12345678),%g2 | ldsb [%g2+%lo(_foo+0x12345678)],%i0 | ret | restore Both relocs contain a pointer to <>, and the offsets contain junk. |RELOCATION RECORDS FOR [.text]: |offset type value |00000004 HI22 _foo+0x12345678 |00000008 LO10 _foo+0x12345678 | |00000000 9de3bf90 ; save %sp,-112,%sp |00000004 05000000 ; sethi %hi(_foo+0),%g2 |00000008 f048a000 ; ldsb [%g2+%lo(_foo+0)],%i0 |0000000c 81c7e008 ; ret |00000010 81e80000 ; restore o <> The <> field can be imagined as a relocation instruction. It is a pointer to a structure which contains information on what to do with all of the other information in the reloc record and data section. A back end would normally have a relocation instruction set and turn relocations into pointers to the correct structure on input - but it would be possible to create each howto field on demand. */ /* SUBSUBSECTION <> Indicates what sort of overflow checking should be done when performing a relocation. CODE_FRAGMENT . .enum complain_overflow .{ . {* Do not complain on overflow. *} . complain_overflow_dont, . . {* Complain if the value overflows when considered as a signed . number one bit larger than the field. ie. A bitfield of N bits . is allowed to represent -2**n to 2**n-1. *} . complain_overflow_bitfield, . . {* Complain if the value overflows when considered as a signed . number. *} . complain_overflow_signed, . . {* Complain if the value overflows when considered as an . unsigned number. *} . complain_overflow_unsigned .}; */ /* SUBSUBSECTION <> The <> is a structure which contains all the information that libbfd needs to know to tie up a back end's data. CODE_FRAGMENT .struct bfd_symbol; {* Forward declaration. *} . .struct reloc_howto_struct .{ . {* The type field has mainly a documentary use - the back end can . do what it wants with it, though normally the back end's . external idea of what a reloc number is stored . in this field. For example, a PC relative word relocation . in a coff environment has the type 023 - because that's . what the outside world calls a R_PCRWORD reloc. *} . unsigned int type; . . {* The value the final relocation is shifted right by. This drops . unwanted data from the relocation. *} . unsigned int rightshift; . . {* The size of the item to be relocated. This is *not* a . power-of-two measure. To get the number of bytes operated . on by a type of relocation, use bfd_get_reloc_size. *} . int size; . . {* The number of bits in the item to be relocated. This is used . when doing overflow checking. *} . unsigned int bitsize; . . {* Notes that the relocation is relative to the location in the . data section of the addend. The relocation function will . subtract from the relocation value the address of the location . being relocated. *} . bfd_boolean pc_relative; . . {* The bit position of the reloc value in the destination. . The relocated value is left shifted by this amount. *} . unsigned int bitpos; . . {* What type of overflow error should be checked for when . relocating. *} . enum complain_overflow complain_on_overflow; . . {* If this field is non null, then the supplied function is . called rather than the normal function. This allows really . strange relocation methods to be accommodated (e.g., i960 callj . instructions). *} . bfd_reloc_status_type (*special_function) . (bfd *, arelent *, struct bfd_symbol *, void *, asection *, . bfd *, char **); . . {* The textual name of the relocation type. *} . char *name; . . {* Some formats record a relocation addend in the section contents . rather than with the relocation. For ELF formats this is the . distinction between USE_REL and USE_RELA (though the code checks . for USE_REL == 1/0). The value of this field is TRUE if the . addend is recorded with the section contents; when performing a . partial link (ld -r) the section contents (the data) will be . modified. The value of this field is FALSE if addends are . recorded with the relocation (in arelent.addend); when performing . a partial link the relocation will be modified. . All relocations for all ELF USE_RELA targets should set this field . to FALSE (values of TRUE should be looked on with suspicion). . However, the converse is not true: not all relocations of all ELF . USE_REL targets set this field to TRUE. Why this is so is peculiar . to each particular target. For relocs that aren't used in partial . links (e.g. GOT stuff) it doesn't matter what this is set to. *} . bfd_boolean partial_inplace; . . {* src_mask selects the part of the instruction (or data) to be used . in the relocation sum. If the target relocations don't have an . addend in the reloc, eg. ELF USE_REL, src_mask will normally equal . dst_mask to extract the addend from the section contents. If . relocations do have an addend in the reloc, eg. ELF USE_RELA, this . field should be zero. Non-zero values for ELF USE_RELA targets are . bogus as in those cases the value in the dst_mask part of the . section contents should be treated as garbage. *} . bfd_vma src_mask; . . {* dst_mask selects which parts of the instruction (or data) are . replaced with a relocated value. *} . bfd_vma dst_mask; . . {* When some formats create PC relative instructions, they leave . the value of the pc of the place being relocated in the offset . slot of the instruction, so that a PC relative relocation can . be made just by adding in an ordinary offset (e.g., sun3 a.out). . Some formats leave the displacement part of an instruction . empty (e.g., m88k bcs); this flag signals the fact. *} . bfd_boolean pcrel_offset; .}; . */ /* FUNCTION The HOWTO Macro DESCRIPTION The HOWTO define is horrible and will go away. .#define HOWTO(C, R, S, B, P, BI, O, SF, NAME, INPLACE, MASKSRC, MASKDST, PC) \ . { (unsigned) C, R, S, B, P, BI, O, SF, NAME, INPLACE, MASKSRC, MASKDST, PC } DESCRIPTION And will be replaced with the totally magic way. But for the moment, we are compatible, so do it this way. .#define NEWHOWTO(FUNCTION, NAME, SIZE, REL, IN) \ . HOWTO (0, 0, SIZE, 0, REL, 0, complain_overflow_dont, FUNCTION, \ . NAME, FALSE, 0, 0, IN) . DESCRIPTION This is used to fill in an empty howto entry in an array. .#define EMPTY_HOWTO(C) \ . HOWTO ((C), 0, 0, 0, FALSE, 0, complain_overflow_dont, NULL, \ . NULL, FALSE, 0, 0, FALSE) . DESCRIPTION Helper routine to turn a symbol into a relocation value. .#define HOWTO_PREPARE(relocation, symbol) \ . { \ . if (symbol != NULL) \ . { \ . if (bfd_is_com_section (symbol->section)) \ . { \ . relocation = 0; \ . } \ . else \ . { \ . relocation = symbol->value; \ . } \ . } \ . } . */ /* FUNCTION bfd_get_reloc_size SYNOPSIS unsigned int bfd_get_reloc_size (reloc_howto_type *); DESCRIPTION For a reloc_howto_type that operates on a fixed number of bytes, this returns the number of bytes operated on. */ unsigned int bfd_get_reloc_size (reloc_howto_type *howto) { switch (howto->size) { case 0: return 1; case 1: return 2; case 2: return 4; case 3: return 0; case 4: return 8; case 8: return 16; case -2: return 4; default: abort (); } } /* TYPEDEF arelent_chain DESCRIPTION How relocs are tied together in an <>: .typedef struct relent_chain .{ . arelent relent; . struct relent_chain *next; .} .arelent_chain; . */ /* N_ONES produces N one bits, without overflowing machine arithmetic. */ #define N_ONES(n) (((((bfd_vma) 1 << ((n) - 1)) - 1) << 1) | 1) /* FUNCTION bfd_check_overflow SYNOPSIS bfd_reloc_status_type bfd_check_overflow (enum complain_overflow how, unsigned int bitsize, unsigned int rightshift, unsigned int addrsize, bfd_vma relocation); DESCRIPTION Perform overflow checking on @var{relocation} which has @var{bitsize} significant bits and will be shifted right by @var{rightshift} bits, on a machine with addresses containing @var{addrsize} significant bits. The result is either of @code{bfd_reloc_ok} or @code{bfd_reloc_overflow}. */ bfd_reloc_status_type bfd_check_overflow (enum complain_overflow how, unsigned int bitsize, unsigned int rightshift, unsigned int addrsize, bfd_vma relocation) { bfd_vma fieldmask, addrmask, signmask, ss, a; bfd_reloc_status_type flag = bfd_reloc_ok; /* Note: BITSIZE should always be <= ADDRSIZE, but in case it's not, we'll be permissive: extra bits in the field mask will automatically extend the address mask for purposes of the overflow check. */ fieldmask = N_ONES (bitsize); signmask = ~fieldmask; addrmask = N_ONES (addrsize) | fieldmask; a = (relocation & addrmask) >> rightshift;; switch (how) { case complain_overflow_dont: break; case complain_overflow_signed: /* If any sign bits are set, all sign bits must be set. That is, A must be a valid negative address after shifting. */ signmask = ~ (fieldmask >> 1); /* Fall thru */ case complain_overflow_bitfield: /* Bitfields are sometimes signed, sometimes unsigned. We explicitly allow an address wrap too, which means a bitfield of n bits is allowed to store -2**n to 2**n-1. Thus overflow if the value has some, but not all, bits set outside the field. */ ss = a & signmask; if (ss != 0 && ss != ((addrmask >> rightshift) & signmask)) flag = bfd_reloc_overflow; break; case complain_overflow_unsigned: /* We have an overflow if the address does not fit in the field. */ if ((a & signmask) != 0) flag = bfd_reloc_overflow; break; default: abort (); } return flag; } /* FUNCTION bfd_perform_relocation SYNOPSIS bfd_reloc_status_type bfd_perform_relocation (bfd *abfd, arelent *reloc_entry, void *data, asection *input_section, bfd *output_bfd, char **error_message); DESCRIPTION If @var{output_bfd} is supplied to this function, the generated image will be relocatable; the relocations are copied to the output file after they have been changed to reflect the new state of the world. There are two ways of reflecting the results of partial linkage in an output file: by modifying the output data in place, and by modifying the relocation record. Some native formats (e.g., basic a.out and basic coff) have no way of specifying an addend in the relocation type, so the addend has to go in the output data. This is no big deal since in these formats the output data slot will always be big enough for the addend. Complex reloc types with addends were invented to solve just this problem. The @var{error_message} argument is set to an error message if this return @code{bfd_reloc_dangerous}. */ bfd_reloc_status_type bfd_perform_relocation (bfd *abfd, arelent *reloc_entry, void *data, asection *input_section, bfd *output_bfd, char **error_message) { bfd_vma relocation; bfd_reloc_status_type flag = bfd_reloc_ok; bfd_size_type octets = reloc_entry->address * bfd_octets_per_byte (abfd); bfd_vma output_base = 0; reloc_howto_type *howto = reloc_entry->howto; asection *reloc_target_output_section; asymbol *symbol; symbol = *(reloc_entry->sym_ptr_ptr); if (bfd_is_abs_section (symbol->section) && output_bfd != NULL) { reloc_entry->address += input_section->output_offset; return bfd_reloc_ok; } /* If we are not producing relocatable output, return an error if the symbol is not defined. An undefined weak symbol is considered to have a value of zero (SVR4 ABI, p. 4-27). */ if (bfd_is_und_section (symbol->section) && (symbol->flags & BSF_WEAK) == 0 && output_bfd == NULL) flag = bfd_reloc_undefined; /* If there is a function supplied to handle this relocation type, call it. It'll return `bfd_reloc_continue' if further processing can be done. */ if (howto->special_function) { bfd_reloc_status_type cont; cont = howto->special_function (abfd, reloc_entry, symbol, data, input_section, output_bfd, error_message); if (cont != bfd_reloc_continue) return cont; } /* Is the address of the relocation really within the section? */ if (reloc_entry->address > bfd_get_section_limit (abfd, input_section)) return bfd_reloc_outofrange; /* Work out which section the relocation is targeted at and the initial relocation command value. */ /* Get symbol value. (Common symbols are special.) */ if (bfd_is_com_section (symbol->section)) relocation = 0; else relocation = symbol->value; reloc_target_output_section = symbol->section->output_section; /* Convert input-section-relative symbol value to absolute. */ if ((output_bfd && ! howto->partial_inplace) || reloc_target_output_section == NULL) output_base = 0; else output_base = reloc_target_output_section->vma; relocation += output_base + symbol->section->output_offset; /* Add in supplied addend. */ relocation += reloc_entry->addend; /* Here the variable relocation holds the final address of the symbol we are relocating against, plus any addend. */ if (howto->pc_relative) { /* This is a PC relative relocation. We want to set RELOCATION to the distance between the address of the symbol and the location. RELOCATION is already the address of the symbol. We start by subtracting the address of the section containing the location. If pcrel_offset is set, we must further subtract the position of the location within the section. Some targets arrange for the addend to be the negative of the position of the location within the section; for example, i386-aout does this. For i386-aout, pcrel_offset is FALSE. Some other targets do not include the position of the location; for example, m88kbcs, or ELF. For those targets, pcrel_offset is TRUE. If we are producing relocatable output, then we must ensure that this reloc will be correctly computed when the final relocation is done. If pcrel_offset is FALSE we want to wind up with the negative of the location within the section, which means we must adjust the existing addend by the change in the location within the section. If pcrel_offset is TRUE we do not want to adjust the existing addend at all. FIXME: This seems logical to me, but for the case of producing relocatable output it is not what the code actually does. I don't want to change it, because it seems far too likely that something will break. */ relocation -= input_section->output_section->vma + input_section->output_offset; if (howto->pcrel_offset) relocation -= reloc_entry->address; } if (output_bfd != NULL) { if (! howto->partial_inplace) { /* This is a partial relocation, and we want to apply the relocation to the reloc entry rather than the raw data. Modify the reloc inplace to reflect what we now know. */ reloc_entry->addend = relocation; reloc_entry->address += input_section->output_offset; return flag; } else { /* This is a partial relocation, but inplace, so modify the reloc record a bit. If we've relocated with a symbol with a section, change into a ref to the section belonging to the symbol. */ reloc_entry->address += input_section->output_offset; /* WTF?? */ if (abfd->xvec->flavour == bfd_target_coff_flavour && strcmp (abfd->xvec->name, "coff-Intel-little") != 0 && strcmp (abfd->xvec->name, "coff-Intel-big") != 0) { /* For m68k-coff, the addend was being subtracted twice during relocation with -r. Removing the line below this comment fixes that problem; see PR 2953. However, Ian wrote the following, regarding removing the line below, which explains why it is still enabled: --djm If you put a patch like that into BFD you need to check all the COFF linkers. I am fairly certain that patch will break coff-i386 (e.g., SCO); see coff_i386_reloc in coff-i386.c where I worked around the problem in a different way. There may very well be a reason that the code works as it does. Hmmm. The first obvious point is that bfd_perform_relocation should not have any tests that depend upon the flavour. It's seem like entirely the wrong place for such a thing. The second obvious point is that the current code ignores the reloc addend when producing relocatable output for COFF. That's peculiar. In fact, I really have no idea what the point of the line you want to remove is. A typical COFF reloc subtracts the old value of the symbol and adds in the new value to the location in the object file (if it's a pc relative reloc it adds the difference between the symbol value and the location). When relocating we need to preserve that property. BFD handles this by setting the addend to the negative of the old value of the symbol. Unfortunately it handles common symbols in a non-standard way (it doesn't subtract the old value) but that's a different story (we can't change it without losing backward compatibility with old object files) (coff-i386 does subtract the old value, to be compatible with existing coff-i386 targets, like SCO). So everything works fine when not producing relocatable output. When we are producing relocatable output, logically we should do exactly what we do when not producing relocatable output. Therefore, your patch is correct. In fact, it should probably always just set reloc_entry->addend to 0 for all cases, since it is, in fact, going to add the value into the object file. This won't hurt the COFF code, which doesn't use the addend; I'm not sure what it will do to other formats (the thing to check for would be whether any formats both use the addend and set partial_inplace). When I wanted to make coff-i386 produce relocatable output, I ran into the problem that you are running into: I wanted to remove that line. Rather than risk it, I made the coff-i386 relocs use a special function; it's coff_i386_reloc in coff-i386.c. The function specifically adds the addend field into the object file, knowing that bfd_perform_relocation is not going to. If you remove that line, then coff-i386.c will wind up adding the addend field in twice. It's trivial to fix; it just needs to be done. The problem with removing the line is just that it may break some working code. With BFD it's hard to be sure of anything. The right way to deal with this is simply to build and test at least all the supported COFF targets. It should be straightforward if time and disk space consuming. For each target: 1) build the linker 2) generate some executable, and link it using -r (I would probably use paranoia.o and link against newlib/libc.a, which for all the supported targets would be available in /usr/cygnus/progressive/H-host/target/lib/libc.a). 3) make the change to reloc.c 4) rebuild the linker 5) repeat step 2 6) if the resulting object files are the same, you have at least made it no worse 7) if they are different you have to figure out which version is right */ relocation -= reloc_entry->addend; reloc_entry->addend = 0; } else { reloc_entry->addend = relocation; } } } else { reloc_entry->addend = 0; } /* FIXME: This overflow checking is incomplete, because the value might have overflowed before we get here. For a correct check we need to compute the value in a size larger than bitsize, but we can't reasonably do that for a reloc the same size as a host machine word. FIXME: We should also do overflow checking on the result after adding in the value contained in the object file. */ if (howto->complain_on_overflow != complain_overflow_dont && flag == bfd_reloc_ok) flag = bfd_check_overflow (howto->complain_on_overflow, howto->bitsize, howto->rightshift, bfd_arch_bits_per_address (abfd), relocation); /* Either we are relocating all the way, or we don't want to apply the relocation to the reloc entry (probably because there isn't any room in the output format to describe addends to relocs). */ /* The cast to bfd_vma avoids a bug in the Alpha OSF/1 C compiler (OSF version 1.3, compiler version 3.11). It miscompiles the following program: struct str { unsigned int i0; } s = { 0 }; int main () { unsigned long x; x = 0x100000000; x <<= (unsigned long) s.i0; if (x == 0) printf ("failed\n"); else printf ("succeeded (%lx)\n", x); } */ relocation >>= (bfd_vma) howto->rightshift; /* Shift everything up to where it's going to be used. */ relocation <<= (bfd_vma) howto->bitpos; /* Wait for the day when all have the mask in them. */ /* What we do: i instruction to be left alone o offset within instruction r relocation offset to apply S src mask D dst mask N ~dst mask A part 1 B part 2 R result Do this: (( i i i i i o o o o o from bfd_get and S S S S S) to get the size offset we want + r r r r r r r r r r) to get the final value to place and D D D D D to chop to right size ----------------------- = A A A A A And this: ( i i i i i o o o o o from bfd_get and N N N N N ) get instruction ----------------------- = B B B B B And then: ( B B B B B or A A A A A) ----------------------- = R R R R R R R R R R put into bfd_put */ #define DOIT(x) \ x = ( (x & ~howto->dst_mask) | (((x & howto->src_mask) + relocation) & howto->dst_mask)) switch (howto->size) { case 0: { char x = bfd_get_8 (abfd, (char *) data + octets); DOIT (x); bfd_put_8 (abfd, x, (unsigned char *) data + octets); } break; case 1: { short x = bfd_get_16 (abfd, (bfd_byte *) data + octets); DOIT (x); bfd_put_16 (abfd, (bfd_vma) x, (unsigned char *) data + octets); } break; case 2: { long x = bfd_get_32 (abfd, (bfd_byte *) data + octets); DOIT (x); bfd_put_32 (abfd, (bfd_vma) x, (bfd_byte *) data + octets); } break; case -2: { long x = bfd_get_32 (abfd, (bfd_byte *) data + octets); relocation = -relocation; DOIT (x); bfd_put_32 (abfd, (bfd_vma) x, (bfd_byte *) data + octets); } break; case -1: { long x = bfd_get_16 (abfd, (bfd_byte *) data + octets); relocation = -relocation; DOIT (x); bfd_put_16 (abfd, (bfd_vma) x, (bfd_byte *) data + octets); } break; case 3: /* Do nothing */ break; case 4: #ifdef BFD64 { bfd_vma x = bfd_get_64 (abfd, (bfd_byte *) data + octets); DOIT (x); bfd_put_64 (abfd, x, (bfd_byte *) data + octets); } #else abort (); #endif break; default: return bfd_reloc_other; } return flag; } /* FUNCTION bfd_install_relocation SYNOPSIS bfd_reloc_status_type bfd_install_relocation (bfd *abfd, arelent *reloc_entry, void *data, bfd_vma data_start, asection *input_section, char **error_message); DESCRIPTION This looks remarkably like <>, except it does not expect that the section contents have been filled in. I.e., it's suitable for use when creating, rather than applying a relocation. For now, this function should be considered reserved for the assembler. */ bfd_reloc_status_type bfd_install_relocation (bfd *abfd, arelent *reloc_entry, void *data_start, bfd_vma data_start_offset, asection *input_section, char **error_message) { bfd_vma relocation; bfd_reloc_status_type flag = bfd_reloc_ok; bfd_size_type octets = reloc_entry->address * bfd_octets_per_byte (abfd); bfd_vma output_base = 0; reloc_howto_type *howto = reloc_entry->howto; asection *reloc_target_output_section; asymbol *symbol; bfd_byte *data; symbol = *(reloc_entry->sym_ptr_ptr); if (bfd_is_abs_section (symbol->section)) { reloc_entry->address += input_section->output_offset; return bfd_reloc_ok; } /* If there is a function supplied to handle this relocation type, call it. It'll return `bfd_reloc_continue' if further processing can be done. */ if (howto->special_function) { bfd_reloc_status_type cont; /* XXX - The special_function calls haven't been fixed up to deal with creating new relocations and section contents. */ cont = howto->special_function (abfd, reloc_entry, symbol, /* XXX - Non-portable! */ ((bfd_byte *) data_start - data_start_offset), input_section, abfd, error_message); if (cont != bfd_reloc_continue) return cont; } /* Is the address of the relocation really within the section? */ if (reloc_entry->address > bfd_get_section_limit (abfd, input_section)) return bfd_reloc_outofrange; /* Work out which section the relocation is targeted at and the initial relocation command value. */ /* Get symbol value. (Common symbols are special.) */ if (bfd_is_com_section (symbol->section)) relocation = 0; else relocation = symbol->value; reloc_target_output_section = symbol->section->output_section; /* Convert input-section-relative symbol value to absolute. */ if (! howto->partial_inplace) output_base = 0; else output_base = reloc_target_output_section->vma; relocation += output_base + symbol->section->output_offset; /* Add in supplied addend. */ relocation += reloc_entry->addend; /* Here the variable relocation holds the final address of the symbol we are relocating against, plus any addend. */ if (howto->pc_relative) { /* This is a PC relative relocation. We want to set RELOCATION to the distance between the address of the symbol and the location. RELOCATION is already the address of the symbol. We start by subtracting the address of the section containing the location. If pcrel_offset is set, we must further subtract the position of the location within the section. Some targets arrange for the addend to be the negative of the position of the location within the section; for example, i386-aout does this. For i386-aout, pcrel_offset is FALSE. Some other targets do not include the position of the location; for example, m88kbcs, or ELF. For those targets, pcrel_offset is TRUE. If we are producing relocatable output, then we must ensure that this reloc will be correctly computed when the final relocation is done. If pcrel_offset is FALSE we want to wind up with the negative of the location within the section, which means we must adjust the existing addend by the change in the location within the section. If pcrel_offset is TRUE we do not want to adjust the existing addend at all. FIXME: This seems logical to me, but for the case of producing relocatable output it is not what the code actually does. I don't want to change it, because it seems far too likely that something will break. */ relocation -= input_section->output_section->vma + input_section->output_offset; if (howto->pcrel_offset && howto->partial_inplace) relocation -= reloc_entry->address; } if (! howto->partial_inplace) { /* This is a partial relocation, and we want to apply the relocation to the reloc entry rather than the raw data. Modify the reloc inplace to reflect what we now know. */ reloc_entry->addend = relocation; reloc_entry->address += input_section->output_offset; return flag; } else { /* This is a partial relocation, but inplace, so modify the reloc record a bit. If we've relocated with a symbol with a section, change into a ref to the section belonging to the symbol. */ reloc_entry->address += input_section->output_offset; /* WTF?? */ if (abfd->xvec->flavour == bfd_target_coff_flavour && strcmp (abfd->xvec->name, "coff-Intel-little") != 0 && strcmp (abfd->xvec->name, "coff-Intel-big") != 0) { /* For m68k-coff, the addend was being subtracted twice during relocation with -r. Removing the line below this comment fixes that problem; see PR 2953. However, Ian wrote the following, regarding removing the line below, which explains why it is still enabled: --djm If you put a patch like that into BFD you need to check all the COFF linkers. I am fairly certain that patch will break coff-i386 (e.g., SCO); see coff_i386_reloc in coff-i386.c where I worked around the problem in a different way. There may very well be a reason that the code works as it does. Hmmm. The first obvious point is that bfd_install_relocation should not have any tests that depend upon the flavour. It's seem like entirely the wrong place for such a thing. The second obvious point is that the current code ignores the reloc addend when producing relocatable output for COFF. That's peculiar. In fact, I really have no idea what the point of the line you want to remove is. A typical COFF reloc subtracts the old value of the symbol and adds in the new value to the location in the object file (if it's a pc relative reloc it adds the difference between the symbol value and the location). When relocating we need to preserve that property. BFD handles this by setting the addend to the negative of the old value of the symbol. Unfortunately it handles common symbols in a non-standard way (it doesn't subtract the old value) but that's a different story (we can't change it without losing backward compatibility with old object files) (coff-i386 does subtract the old value, to be compatible with existing coff-i386 targets, like SCO). So everything works fine when not producing relocatable output. When we are producing relocatable output, logically we should do exactly what we do when not producing relocatable output. Therefore, your patch is correct. In fact, it should probably always just set reloc_entry->addend to 0 for all cases, since it is, in fact, going to add the value into the object file. This won't hurt the COFF code, which doesn't use the addend; I'm not sure what it will do to other formats (the thing to check for would be whether any formats both use the addend and set partial_inplace). When I wanted to make coff-i386 produce relocatable output, I ran into the problem that you are running into: I wanted to remove that line. Rather than risk it, I made the coff-i386 relocs use a special function; it's coff_i386_reloc in coff-i386.c. The function specifically adds the addend field into the object file, knowing that bfd_install_relocation is not going to. If you remove that line, then coff-i386.c will wind up adding the addend field in twice. It's trivial to fix; it just needs to be done. The problem with removing the line is just that it may break some working code. With BFD it's hard to be sure of anything. The right way to deal with this is simply to build and test at least all the supported COFF targets. It should be straightforward if time and disk space consuming. For each target: 1) build the linker 2) generate some executable, and link it using -r (I would probably use paranoia.o and link against newlib/libc.a, which for all the supported targets would be available in /usr/cygnus/progressive/H-host/target/lib/libc.a). 3) make the change to reloc.c 4) rebuild the linker 5) repeat step 2 6) if the resulting object files are the same, you have at least made it no worse 7) if they are different you have to figure out which version is right. */ relocation -= reloc_entry->addend; /* FIXME: There should be no target specific code here... */ if (strcmp (abfd->xvec->name, "coff-z8k") != 0) reloc_entry->addend = 0; } else { reloc_entry->addend = relocation; } } /* FIXME: This overflow checking is incomplete, because the value might have overflowed before we get here. For a correct check we need to compute the value in a size larger than bitsize, but we can't reasonably do that for a reloc the same size as a host machine word. FIXME: We should also do overflow checking on the result after adding in the value contained in the object file. */ if (howto->complain_on_overflow != complain_overflow_dont) flag = bfd_check_overflow (howto->complain_on_overflow, howto->bitsize, howto->rightshift, bfd_arch_bits_per_address (abfd), relocation); /* Either we are relocating all the way, or we don't want to apply the relocation to the reloc entry (probably because there isn't any room in the output format to describe addends to relocs). */ /* The cast to bfd_vma avoids a bug in the Alpha OSF/1 C compiler (OSF version 1.3, compiler version 3.11). It miscompiles the following program: struct str { unsigned int i0; } s = { 0 }; int main () { unsigned long x; x = 0x100000000; x <<= (unsigned long) s.i0; if (x == 0) printf ("failed\n"); else printf ("succeeded (%lx)\n", x); } */ relocation >>= (bfd_vma) howto->rightshift; /* Shift everything up to where it's going to be used. */ relocation <<= (bfd_vma) howto->bitpos; /* Wait for the day when all have the mask in them. */ /* What we do: i instruction to be left alone o offset within instruction r relocation offset to apply S src mask D dst mask N ~dst mask A part 1 B part 2 R result Do this: (( i i i i i o o o o o from bfd_get and S S S S S) to get the size offset we want + r r r r r r r r r r) to get the final value to place and D D D D D to chop to right size ----------------------- = A A A A A And this: ( i i i i i o o o o o from bfd_get and N N N N N ) get instruction ----------------------- = B B B B B And then: ( B B B B B or A A A A A) ----------------------- = R R R R R R R R R R put into bfd_put */ #define DOIT(x) \ x = ( (x & ~howto->dst_mask) | (((x & howto->src_mask) + relocation) & howto->dst_mask)) data = (bfd_byte *) data_start + (octets - data_start_offset); switch (howto->size) { case 0: { char x = bfd_get_8 (abfd, data); DOIT (x); bfd_put_8 (abfd, x, data); } break; case 1: { short x = bfd_get_16 (abfd, data); DOIT (x); bfd_put_16 (abfd, (bfd_vma) x, data); } break; case 2: { long x = bfd_get_32 (abfd, data); DOIT (x); bfd_put_32 (abfd, (bfd_vma) x, data); } break; case -2: { long x = bfd_get_32 (abfd, data); relocation = -relocation; DOIT (x); bfd_put_32 (abfd, (bfd_vma) x, data); } break; case 3: /* Do nothing */ break; case 4: { bfd_vma x = bfd_get_64 (abfd, data); DOIT (x); bfd_put_64 (abfd, x, data); } break; default: return bfd_reloc_other; } return flag; } /* This relocation routine is used by some of the backend linkers. They do not construct asymbol or arelent structures, so there is no reason for them to use bfd_perform_relocation. Also, bfd_perform_relocation is so hacked up it is easier to write a new function than to try to deal with it. This routine does a final relocation. Whether it is useful for a relocatable link depends upon how the object format defines relocations. FIXME: This routine ignores any special_function in the HOWTO, since the existing special_function values have been written for bfd_perform_relocation. HOWTO is the reloc howto information. INPUT_BFD is the BFD which the reloc applies to. INPUT_SECTION is the section which the reloc applies to. CONTENTS is the contents of the section. ADDRESS is the address of the reloc within INPUT_SECTION. VALUE is the value of the symbol the reloc refers to. ADDEND is the addend of the reloc. */ bfd_reloc_status_type _bfd_final_link_relocate (reloc_howto_type *howto, bfd *input_bfd, asection *input_section, bfd_byte *contents, bfd_vma address, bfd_vma value, bfd_vma addend) { bfd_vma relocation; /* Sanity check the address. */ if (address > bfd_get_section_limit (input_bfd, input_section)) return bfd_reloc_outofrange; /* This function assumes that we are dealing with a basic relocation against a symbol. We want to compute the value of the symbol to relocate to. This is just VALUE, the value of the symbol, plus ADDEND, any addend associated with the reloc. */ relocation = value + addend; /* If the relocation is PC relative, we want to set RELOCATION to the distance between the symbol (currently in RELOCATION) and the location we are relocating. Some targets (e.g., i386-aout) arrange for the contents of the section to be the negative of the offset of the location within the section; for such targets pcrel_offset is FALSE. Other targets (e.g., m88kbcs or ELF) simply leave the contents of the section as zero; for such targets pcrel_offset is TRUE. If pcrel_offset is FALSE we do not need to subtract out the offset of the location within the section (which is just ADDRESS). */ if (howto->pc_relative) { relocation -= (input_section->output_section->vma + input_section->output_offset); if (howto->pcrel_offset) relocation -= address; } return _bfd_relocate_contents (howto, input_bfd, relocation, contents + address); } /* Relocate a given location using a given value and howto. */ bfd_reloc_status_type _bfd_relocate_contents (reloc_howto_type *howto, bfd *input_bfd, bfd_vma relocation, bfd_byte *location) { int size; bfd_vma x = 0; bfd_reloc_status_type flag; unsigned int rightshift = howto->rightshift; unsigned int bitpos = howto->bitpos; /* If the size is negative, negate RELOCATION. This isn't very general. */ if (howto->size < 0) relocation = -relocation; /* Get the value we are going to relocate. */ size = bfd_get_reloc_size (howto); switch (size) { default: case 0: abort (); case 1: x = bfd_get_8 (input_bfd, location); break; case 2: x = bfd_get_16 (input_bfd, location); break; case 4: x = bfd_get_32 (input_bfd, location); break; case 8: #ifdef BFD64 x = bfd_get_64 (input_bfd, location); #else abort (); #endif break; } /* Check for overflow. FIXME: We may drop bits during the addition which we don't check for. We must either check at every single operation, which would be tedious, or we must do the computations in a type larger than bfd_vma, which would be inefficient. */ flag = bfd_reloc_ok; if (howto->complain_on_overflow != complain_overflow_dont) { bfd_vma addrmask, fieldmask, signmask, ss; bfd_vma a, b, sum; /* Get the values to be added together. For signed and unsigned relocations, we assume that all values should be truncated to the size of an address. For bitfields, all the bits matter. See also bfd_check_overflow. */ fieldmask = N_ONES (howto->bitsize); signmask = ~fieldmask; addrmask = N_ONES (bfd_arch_bits_per_address (input_bfd)) | fieldmask; a = (relocation & addrmask) >> rightshift; b = (x & howto->src_mask & addrmask) >> bitpos; switch (howto->complain_on_overflow) { case complain_overflow_signed: /* If any sign bits are set, all sign bits must be set. That is, A must be a valid negative address after shifting. */ signmask = ~(fieldmask >> 1); /* Fall thru */ case complain_overflow_bitfield: /* Much like the signed check, but for a field one bit wider. We allow a bitfield to represent numbers in the range -2**n to 2**n-1, where n is the number of bits in the field. Note that when bfd_vma is 32 bits, a 32-bit reloc can't overflow, which is exactly what we want. */ ss = a & signmask; if (ss != 0 && ss != ((addrmask >> rightshift) & signmask)) flag = bfd_reloc_overflow; /* We only need this next bit of code if the sign bit of B is below the sign bit of A. This would only happen if SRC_MASK had fewer bits than BITSIZE. Note that if SRC_MASK has more bits than BITSIZE, we can get into trouble; we would need to verify that B is in range, as we do for A above. */ ss = ((~howto->src_mask) >> 1) & howto->src_mask; ss >>= bitpos; /* Set all the bits above the sign bit. */ b = (b ^ ss) - ss; /* Now we can do the addition. */ sum = a + b; /* See if the result has the correct sign. Bits above the sign bit are junk now; ignore them. If the sum is positive, make sure we did not have all negative inputs; if the sum is negative, make sure we did not have all positive inputs. The test below looks only at the sign bits, and it really just SIGN (A) == SIGN (B) && SIGN (A) != SIGN (SUM) We mask with addrmask here to explicitly allow an address wrap-around. The Linux kernel relies on it, and it is the only way to write assembler code which can run when loaded at a location 0x80000000 away from the location at which it is linked. */ if (((~(a ^ b)) & (a ^ sum)) & signmask & addrmask) flag = bfd_reloc_overflow; break; case complain_overflow_unsigned: /* Checking for an unsigned overflow is relatively easy: trim the addresses and add, and trim the result as well. Overflow is normally indicated when the result does not fit in the field. However, we also need to consider the case when, e.g., fieldmask is 0x7fffffff or smaller, an input is 0x80000000, and bfd_vma is only 32 bits; then we will get sum == 0, but there is an overflow, since the inputs did not fit in the field. Instead of doing a separate test, we can check for this by or-ing in the operands when testing for the sum overflowing its final field. */ sum = (a + b) & addrmask; if ((a | b | sum) & signmask) flag = bfd_reloc_overflow; break; default: abort (); } } /* Put RELOCATION in the right bits. */ relocation >>= (bfd_vma) rightshift; relocation <<= (bfd_vma) bitpos; /* Add RELOCATION to the right bits of X. */ x = ((x & ~howto->dst_mask) | (((x & howto->src_mask) + relocation) & howto->dst_mask)); /* Put the relocated value back in the object file. */ switch (size) { default: abort (); case 1: bfd_put_8 (input_bfd, x, location); break; case 2: bfd_put_16 (input_bfd, x, location); break; case 4: bfd_put_32 (input_bfd, x, location); break; case 8: #ifdef BFD64 bfd_put_64 (input_bfd, x, location); #else abort (); #endif break; } return flag; } /* Clear a given location using a given howto, by applying a relocation value of zero and discarding any in-place addend. This is used for fixed-up relocations against discarded symbols, to make ignorable debug or unwind information more obvious. */ void _bfd_clear_contents (reloc_howto_type *howto, bfd *input_bfd, bfd_byte *location) { int size; bfd_vma x = 0; /* Get the value we are going to relocate. */ size = bfd_get_reloc_size (howto); switch (size) { default: case 0: abort (); case 1: x = bfd_get_8 (input_bfd, location); break; case 2: x = bfd_get_16 (input_bfd, location); break; case 4: x = bfd_get_32 (input_bfd, location); break; case 8: #ifdef BFD64 x = bfd_get_64 (input_bfd, location); #else abort (); #endif break; } /* Zero out the unwanted bits of X. */ x &= ~howto->dst_mask; /* Put the relocated value back in the object file. */ switch (size) { default: case 0: abort (); case 1: bfd_put_8 (input_bfd, x, location); break; case 2: bfd_put_16 (input_bfd, x, location); break; case 4: bfd_put_32 (input_bfd, x, location); break; case 8: #ifdef BFD64 bfd_put_64 (input_bfd, x, location); #else abort (); #endif break; } } /* DOCDD INODE howto manager, , typedef arelent, Relocations SUBSECTION The howto manager When an application wants to create a relocation, but doesn't know what the target machine might call it, it can find out by using this bit of code. */ /* TYPEDEF bfd_reloc_code_type DESCRIPTION The insides of a reloc code. The idea is that, eventually, there will be one enumerator for every type of relocation we ever do. Pass one of these values to <>, and it'll return a howto pointer. This does mean that the application must determine the correct enumerator value; you can't get a howto pointer from a random set of attributes. SENUM bfd_reloc_code_real ENUM BFD_RELOC_64 ENUMX BFD_RELOC_32 ENUMX BFD_RELOC_26 ENUMX BFD_RELOC_24 ENUMX BFD_RELOC_16 ENUMX BFD_RELOC_14 ENUMX BFD_RELOC_8 ENUMDOC Basic absolute relocations of N bits. ENUM BFD_RELOC_64_PCREL ENUMX BFD_RELOC_32_PCREL ENUMX BFD_RELOC_24_PCREL ENUMX BFD_RELOC_16_PCREL ENUMX BFD_RELOC_12_PCREL ENUMX BFD_RELOC_8_PCREL ENUMDOC PC-relative relocations. Sometimes these are relative to the address of the relocation itself; sometimes they are relative to the start of the section containing the relocation. It depends on the specific target. The 24-bit relocation is used in some Intel 960 configurations. ENUM BFD_RELOC_32_SECREL ENUMDOC Section relative relocations. Some targets need this for DWARF2. ENUM BFD_RELOC_32_GOT_PCREL ENUMX BFD_RELOC_16_GOT_PCREL ENUMX BFD_RELOC_8_GOT_PCREL ENUMX BFD_RELOC_32_GOTOFF ENUMX BFD_RELOC_16_GOTOFF ENUMX BFD_RELOC_LO16_GOTOFF ENUMX BFD_RELOC_HI16_GOTOFF ENUMX BFD_RELOC_HI16_S_GOTOFF ENUMX BFD_RELOC_8_GOTOFF ENUMX BFD_RELOC_64_PLT_PCREL ENUMX BFD_RELOC_32_PLT_PCREL ENUMX BFD_RELOC_24_PLT_PCREL ENUMX BFD_RELOC_16_PLT_PCREL ENUMX BFD_RELOC_8_PLT_PCREL ENUMX BFD_RELOC_64_PLTOFF ENUMX BFD_RELOC_32_PLTOFF ENUMX BFD_RELOC_16_PLTOFF ENUMX BFD_RELOC_LO16_PLTOFF ENUMX BFD_RELOC_HI16_PLTOFF ENUMX BFD_RELOC_HI16_S_PLTOFF ENUMX BFD_RELOC_8_PLTOFF ENUMDOC For ELF. ENUM BFD_RELOC_68K_GLOB_DAT ENUMX BFD_RELOC_68K_JMP_SLOT ENUMX BFD_RELOC_68K_RELATIVE ENUMDOC Relocations used by 68K ELF. ENUM BFD_RELOC_32_BASEREL ENUMX BFD_RELOC_16_BASEREL ENUMX BFD_RELOC_LO16_BASEREL ENUMX BFD_RELOC_HI16_BASEREL ENUMX BFD_RELOC_HI16_S_BASEREL ENUMX BFD_RELOC_8_BASEREL ENUMX BFD_RELOC_RVA ENUMDOC Linkage-table relative. ENUM BFD_RELOC_8_FFnn ENUMDOC Absolute 8-bit relocation, but used to form an address like 0xFFnn. ENUM BFD_RELOC_32_PCREL_S2 ENUMX BFD_RELOC_16_PCREL_S2 ENUMX BFD_RELOC_23_PCREL_S2 ENUMDOC These PC-relative relocations are stored as word displacements -- i.e., byte displacements shifted right two bits. The 30-bit word displacement (<<32_PCREL_S2>> -- 32 bits, shifted 2) is used on the SPARC. (SPARC tools generally refer to this as <>.) The signed 16-bit displacement is used on the MIPS, and the 23-bit displacement is used on the Alpha. ENUM BFD_RELOC_HI22 ENUMX BFD_RELOC_LO10 ENUMDOC High 22 bits and low 10 bits of 32-bit value, placed into lower bits of the target word. These are used on the SPARC. ENUM BFD_RELOC_GPREL16 ENUMX BFD_RELOC_GPREL32 ENUMDOC For systems that allocate a Global Pointer register, these are displacements off that register. These relocation types are handled specially, because the value the register will have is decided relatively late. ENUM BFD_RELOC_I960_CALLJ ENUMDOC Reloc types used for i960/b.out. ENUM BFD_RELOC_NONE ENUMX BFD_RELOC_SPARC_WDISP22 ENUMX BFD_RELOC_SPARC22 ENUMX BFD_RELOC_SPARC13 ENUMX BFD_RELOC_SPARC_GOT10 ENUMX BFD_RELOC_SPARC_GOT13 ENUMX BFD_RELOC_SPARC_GOT22 ENUMX BFD_RELOC_SPARC_PC10 ENUMX BFD_RELOC_SPARC_PC22 ENUMX BFD_RELOC_SPARC_WPLT30 ENUMX BFD_RELOC_SPARC_COPY ENUMX BFD_RELOC_SPARC_GLOB_DAT ENUMX BFD_RELOC_SPARC_JMP_SLOT ENUMX BFD_RELOC_SPARC_RELATIVE ENUMX BFD_RELOC_SPARC_UA16 ENUMX BFD_RELOC_SPARC_UA32 ENUMX BFD_RELOC_SPARC_UA64 ENUMDOC SPARC ELF relocations. There is probably some overlap with other relocation types already defined. ENUM BFD_RELOC_SPARC_BASE13 ENUMX BFD_RELOC_SPARC_BASE22 ENUMDOC I think these are specific to SPARC a.out (e.g., Sun 4). ENUMEQ BFD_RELOC_SPARC_64 BFD_RELOC_64 ENUMX BFD_RELOC_SPARC_10 ENUMX BFD_RELOC_SPARC_11 ENUMX BFD_RELOC_SPARC_OLO10 ENUMX BFD_RELOC_SPARC_HH22 ENUMX BFD_RELOC_SPARC_HM10 ENUMX BFD_RELOC_SPARC_LM22 ENUMX BFD_RELOC_SPARC_PC_HH22 ENUMX BFD_RELOC_SPARC_PC_HM10 ENUMX BFD_RELOC_SPARC_PC_LM22 ENUMX BFD_RELOC_SPARC_WDISP16 ENUMX BFD_RELOC_SPARC_WDISP19 ENUMX BFD_RELOC_SPARC_7 ENUMX BFD_RELOC_SPARC_6 ENUMX BFD_RELOC_SPARC_5 ENUMEQX BFD_RELOC_SPARC_DISP64 BFD_RELOC_64_PCREL ENUMX BFD_RELOC_SPARC_PLT32 ENUMX BFD_RELOC_SPARC_PLT64 ENUMX BFD_RELOC_SPARC_HIX22 ENUMX BFD_RELOC_SPARC_LOX10 ENUMX BFD_RELOC_SPARC_H44 ENUMX BFD_RELOC_SPARC_M44 ENUMX BFD_RELOC_SPARC_L44 ENUMX BFD_RELOC_SPARC_REGISTER ENUMDOC SPARC64 relocations ENUM BFD_RELOC_SPARC_REV32 ENUMDOC SPARC little endian relocation ENUM BFD_RELOC_SPARC_TLS_GD_HI22 ENUMX BFD_RELOC_SPARC_TLS_GD_LO10 ENUMX BFD_RELOC_SPARC_TLS_GD_ADD ENUMX BFD_RELOC_SPARC_TLS_GD_CALL ENUMX BFD_RELOC_SPARC_TLS_LDM_HI22 ENUMX BFD_RELOC_SPARC_TLS_LDM_LO10 ENUMX BFD_RELOC_SPARC_TLS_LDM_ADD ENUMX BFD_RELOC_SPARC_TLS_LDM_CALL ENUMX BFD_RELOC_SPARC_TLS_LDO_HIX22 ENUMX BFD_RELOC_SPARC_TLS_LDO_LOX10 ENUMX BFD_RELOC_SPARC_TLS_LDO_ADD ENUMX BFD_RELOC_SPARC_TLS_IE_HI22 ENUMX BFD_RELOC_SPARC_TLS_IE_LO10 ENUMX BFD_RELOC_SPARC_TLS_IE_LD ENUMX BFD_RELOC_SPARC_TLS_IE_LDX ENUMX BFD_RELOC_SPARC_TLS_IE_ADD ENUMX BFD_RELOC_SPARC_TLS_LE_HIX22 ENUMX BFD_RELOC_SPARC_TLS_LE_LOX10 ENUMX BFD_RELOC_SPARC_TLS_DTPMOD32 ENUMX BFD_RELOC_SPARC_TLS_DTPMOD64 ENUMX BFD_RELOC_SPARC_TLS_DTPOFF32 ENUMX BFD_RELOC_SPARC_TLS_DTPOFF64 ENUMX BFD_RELOC_SPARC_TLS_TPOFF32 ENUMX BFD_RELOC_SPARC_TLS_TPOFF64 ENUMDOC SPARC TLS relocations ENUM BFD_RELOC_SPU_IMM7 ENUMX BFD_RELOC_SPU_IMM8 ENUMX BFD_RELOC_SPU_IMM10 ENUMX BFD_RELOC_SPU_IMM10W ENUMX BFD_RELOC_SPU_IMM16 ENUMX BFD_RELOC_SPU_IMM16W ENUMX BFD_RELOC_SPU_IMM18 ENUMX BFD_RELOC_SPU_PCREL9a ENUMX BFD_RELOC_SPU_PCREL9b ENUMX BFD_RELOC_SPU_PCREL16 ENUMX BFD_RELOC_SPU_LO16 ENUMX BFD_RELOC_SPU_HI16 ENUMX BFD_RELOC_SPU_PPU32 ENUMX BFD_RELOC_SPU_PPU64 ENUMDOC SPU Relocations. ENUM BFD_RELOC_ALPHA_GPDISP_HI16 ENUMDOC Alpha ECOFF and ELF relocations. Some of these treat the symbol or "addend" in some special way. For GPDISP_HI16 ("gpdisp") relocations, the symbol is ignored when writing; when reading, it will be the absolute section symbol. The addend is the displacement in bytes of the "lda" instruction from the "ldah" instruction (which is at the address of this reloc). ENUM BFD_RELOC_ALPHA_GPDISP_LO16 ENUMDOC For GPDISP_LO16 ("ignore") relocations, the symbol is handled as with GPDISP_HI16 relocs. The addend is ignored when writing the relocations out, and is filled in with the file's GP value on reading, for convenience. ENUM BFD_RELOC_ALPHA_GPDISP ENUMDOC The ELF GPDISP relocation is exactly the same as the GPDISP_HI16 relocation except that there is no accompanying GPDISP_LO16 relocation. ENUM BFD_RELOC_ALPHA_LITERAL ENUMX BFD_RELOC_ALPHA_ELF_LITERAL ENUMX BFD_RELOC_ALPHA_LITUSE ENUMDOC The Alpha LITERAL/LITUSE relocs are produced by a symbol reference; the assembler turns it into a LDQ instruction to load the address of the symbol, and then fills in a register in the real instruction. The LITERAL reloc, at the LDQ instruction, refers to the .lita section symbol. The addend is ignored when writing, but is filled in with the file's GP value on reading, for convenience, as with the GPDISP_LO16 reloc. The ELF_LITERAL reloc is somewhere between 16_GOTOFF and GPDISP_LO16. It should refer to the symbol to be referenced, as with 16_GOTOFF, but it generates output not based on the position within the .got section, but relative to the GP value chosen for the file during the final link stage. The LITUSE reloc, on the instruction using the loaded address, gives information to the linker that it might be able to use to optimize away some literal section references. The symbol is ignored (read as the absolute section symbol), and the "addend" indicates the type of instruction using the register: 1 - "memory" fmt insn 2 - byte-manipulation (byte offset reg) 3 - jsr (target of branch) ENUM BFD_RELOC_ALPHA_HINT ENUMDOC The HINT relocation indicates a value that should be filled into the "hint" field of a jmp/jsr/ret instruction, for possible branch- prediction logic which may be provided on some processors. ENUM BFD_RELOC_ALPHA_LINKAGE ENUMDOC The LINKAGE relocation outputs a linkage pair in the object file, which is filled by the linker. ENUM BFD_RELOC_ALPHA_CODEADDR ENUMDOC The CODEADDR relocation outputs a STO_CA in the object file, which is filled by the linker. ENUM BFD_RELOC_ALPHA_GPREL_HI16 ENUMX BFD_RELOC_ALPHA_GPREL_LO16 ENUMDOC The GPREL_HI/LO relocations together form a 32-bit offset from the GP register. ENUM BFD_RELOC_ALPHA_BRSGP ENUMDOC Like BFD_RELOC_23_PCREL_S2, except that the source and target must share a common GP, and the target address is adjusted for STO_ALPHA_STD_GPLOAD. ENUM BFD_RELOC_ALPHA_TLSGD ENUMX BFD_RELOC_ALPHA_TLSLDM ENUMX BFD_RELOC_ALPHA_DTPMOD64 ENUMX BFD_RELOC_ALPHA_GOTDTPREL16 ENUMX BFD_RELOC_ALPHA_DTPREL64 ENUMX BFD_RELOC_ALPHA_DTPREL_HI16 ENUMX BFD_RELOC_ALPHA_DTPREL_LO16 ENUMX BFD_RELOC_ALPHA_DTPREL16 ENUMX BFD_RELOC_ALPHA_GOTTPREL16 ENUMX BFD_RELOC_ALPHA_TPREL64 ENUMX BFD_RELOC_ALPHA_TPREL_HI16 ENUMX BFD_RELOC_ALPHA_TPREL_LO16 ENUMX BFD_RELOC_ALPHA_TPREL16 ENUMDOC Alpha thread-local storage relocations. ENUM BFD_RELOC_MIPS_JMP ENUMDOC Bits 27..2 of the relocation address shifted right 2 bits; simple reloc otherwise. ENUM BFD_RELOC_MIPS16_JMP ENUMDOC The MIPS16 jump instruction. ENUM BFD_RELOC_MIPS16_GPREL ENUMDOC MIPS16 GP relative reloc. ENUM BFD_RELOC_HI16 ENUMDOC High 16 bits of 32-bit value; simple reloc. ENUM BFD_RELOC_HI16_S ENUMDOC High 16 bits of 32-bit value but the low 16 bits will be sign extended and added to form the final result. If the low 16 bits form a negative number, we need to add one to the high value to compensate for the borrow when the low bits are added. ENUM BFD_RELOC_LO16 ENUMDOC Low 16 bits. ENUM BFD_RELOC_HI16_PCREL ENUMDOC High 16 bits of 32-bit pc-relative value ENUM BFD_RELOC_HI16_S_PCREL ENUMDOC High 16 bits of 32-bit pc-relative value, adjusted ENUM BFD_RELOC_LO16_PCREL ENUMDOC Low 16 bits of pc-relative value ENUM BFD_RELOC_MIPS16_HI16 ENUMDOC MIPS16 high 16 bits of 32-bit value. ENUM BFD_RELOC_MIPS16_HI16_S ENUMDOC MIPS16 high 16 bits of 32-bit value but the low 16 bits will be sign extended and added to form the final result. If the low 16 bits form a negative number, we need to add one to the high value to compensate for the borrow when the low bits are added. ENUM BFD_RELOC_MIPS16_LO16 ENUMDOC MIPS16 low 16 bits. ENUM BFD_RELOC_MIPS_LITERAL ENUMDOC Relocation against a MIPS literal section. ENUM BFD_RELOC_MIPS_GOT16 ENUMX BFD_RELOC_MIPS_CALL16 ENUMX BFD_RELOC_MIPS_GOT_HI16 ENUMX BFD_RELOC_MIPS_GOT_LO16 ENUMX BFD_RELOC_MIPS_CALL_HI16 ENUMX BFD_RELOC_MIPS_CALL_LO16 ENUMX BFD_RELOC_MIPS_SUB ENUMX BFD_RELOC_MIPS_GOT_PAGE ENUMX BFD_RELOC_MIPS_GOT_OFST ENUMX BFD_RELOC_MIPS_GOT_DISP ENUMX BFD_RELOC_MIPS_SHIFT5 ENUMX BFD_RELOC_MIPS_SHIFT6 ENUMX BFD_RELOC_MIPS_INSERT_A ENUMX BFD_RELOC_MIPS_INSERT_B ENUMX BFD_RELOC_MIPS_DELETE ENUMX BFD_RELOC_MIPS_HIGHEST ENUMX BFD_RELOC_MIPS_HIGHER ENUMX BFD_RELOC_MIPS_SCN_DISP ENUMX BFD_RELOC_MIPS_REL16 ENUMX BFD_RELOC_MIPS_RELGOT ENUMX BFD_RELOC_MIPS_JALR ENUMX BFD_RELOC_MIPS_TLS_DTPMOD32 ENUMX BFD_RELOC_MIPS_TLS_DTPREL32 ENUMX BFD_RELOC_MIPS_TLS_DTPMOD64 ENUMX BFD_RELOC_MIPS_TLS_DTPREL64 ENUMX BFD_RELOC_MIPS_TLS_GD ENUMX BFD_RELOC_MIPS_TLS_LDM ENUMX BFD_RELOC_MIPS_TLS_DTPREL_HI16 ENUMX BFD_RELOC_MIPS_TLS_DTPREL_LO16 ENUMX BFD_RELOC_MIPS_TLS_GOTTPREL ENUMX BFD_RELOC_MIPS_TLS_TPREL32 ENUMX BFD_RELOC_MIPS_TLS_TPREL64 ENUMX BFD_RELOC_MIPS_TLS_TPREL_HI16 ENUMX BFD_RELOC_MIPS_TLS_TPREL_LO16 ENUMDOC MIPS ELF relocations. COMMENT ENUM BFD_RELOC_MIPS_COPY ENUMX BFD_RELOC_MIPS_JUMP_SLOT ENUMDOC MIPS ELF relocations (VxWorks extensions). COMMENT ENUM BFD_RELOC_FRV_LABEL16 ENUMX BFD_RELOC_FRV_LABEL24 ENUMX BFD_RELOC_FRV_LO16 ENUMX BFD_RELOC_FRV_HI16 ENUMX BFD_RELOC_FRV_GPREL12 ENUMX BFD_RELOC_FRV_GPRELU12 ENUMX BFD_RELOC_FRV_GPREL32 ENUMX BFD_RELOC_FRV_GPRELHI ENUMX BFD_RELOC_FRV_GPRELLO ENUMX BFD_RELOC_FRV_GOT12 ENUMX BFD_RELOC_FRV_GOTHI ENUMX BFD_RELOC_FRV_GOTLO ENUMX BFD_RELOC_FRV_FUNCDESC ENUMX BFD_RELOC_FRV_FUNCDESC_GOT12 ENUMX BFD_RELOC_FRV_FUNCDESC_GOTHI ENUMX BFD_RELOC_FRV_FUNCDESC_GOTLO ENUMX BFD_RELOC_FRV_FUNCDESC_VALUE ENUMX BFD_RELOC_FRV_FUNCDESC_GOTOFF12 ENUMX BFD_RELOC_FRV_FUNCDESC_GOTOFFHI ENUMX BFD_RELOC_FRV_FUNCDESC_GOTOFFLO ENUMX BFD_RELOC_FRV_GOTOFF12 ENUMX BFD_RELOC_FRV_GOTOFFHI ENUMX BFD_RELOC_FRV_GOTOFFLO ENUMX BFD_RELOC_FRV_GETTLSOFF ENUMX BFD_RELOC_FRV_TLSDESC_VALUE ENUMX BFD_RELOC_FRV_GOTTLSDESC12 ENUMX BFD_RELOC_FRV_GOTTLSDESCHI ENUMX BFD_RELOC_FRV_GOTTLSDESCLO ENUMX BFD_RELOC_FRV_TLSMOFF12 ENUMX BFD_RELOC_FRV_TLSMOFFHI ENUMX BFD_RELOC_FRV_TLSMOFFLO ENUMX BFD_RELOC_FRV_GOTTLSOFF12 ENUMX BFD_RELOC_FRV_GOTTLSOFFHI ENUMX BFD_RELOC_FRV_GOTTLSOFFLO ENUMX BFD_RELOC_FRV_TLSOFF ENUMX BFD_RELOC_FRV_TLSDESC_RELAX ENUMX BFD_RELOC_FRV_GETTLSOFF_RELAX ENUMX BFD_RELOC_FRV_TLSOFF_RELAX ENUMX BFD_RELOC_FRV_TLSMOFF ENUMDOC Fujitsu Frv Relocations. COMMENT ENUM BFD_RELOC_MN10300_GOTOFF24 ENUMDOC This is a 24bit GOT-relative reloc for the mn10300. ENUM BFD_RELOC_MN10300_GOT32 ENUMDOC This is a 32bit GOT-relative reloc for the mn10300, offset by two bytes in the instruction. ENUM BFD_RELOC_MN10300_GOT24 ENUMDOC This is a 24bit GOT-relative reloc for the mn10300, offset by two bytes in the instruction. ENUM BFD_RELOC_MN10300_GOT16 ENUMDOC This is a 16bit GOT-relative reloc for the mn10300, offset by two bytes in the instruction. ENUM BFD_RELOC_MN10300_COPY ENUMDOC Copy symbol at runtime. ENUM BFD_RELOC_MN10300_GLOB_DAT ENUMDOC Create GOT entry. ENUM BFD_RELOC_MN10300_JMP_SLOT ENUMDOC Create PLT entry. ENUM BFD_RELOC_MN10300_RELATIVE ENUMDOC Adjust by program base. COMMENT ENUM BFD_RELOC_386_GOT32 ENUMX BFD_RELOC_386_PLT32 ENUMX BFD_RELOC_386_COPY ENUMX BFD_RELOC_386_GLOB_DAT ENUMX BFD_RELOC_386_JUMP_SLOT ENUMX BFD_RELOC_386_RELATIVE ENUMX BFD_RELOC_386_GOTOFF ENUMX BFD_RELOC_386_GOTPC ENUMX BFD_RELOC_386_TLS_TPOFF ENUMX BFD_RELOC_386_TLS_IE ENUMX BFD_RELOC_386_TLS_GOTIE ENUMX BFD_RELOC_386_TLS_LE ENUMX BFD_RELOC_386_TLS_GD ENUMX BFD_RELOC_386_TLS_LDM ENUMX BFD_RELOC_386_TLS_LDO_32 ENUMX BFD_RELOC_386_TLS_IE_32 ENUMX BFD_RELOC_386_TLS_LE_32 ENUMX BFD_RELOC_386_TLS_DTPMOD32 ENUMX BFD_RELOC_386_TLS_DTPOFF32 ENUMX BFD_RELOC_386_TLS_TPOFF32 ENUMX BFD_RELOC_386_TLS_GOTDESC ENUMX BFD_RELOC_386_TLS_DESC_CALL ENUMX BFD_RELOC_386_TLS_DESC ENUMDOC i386/elf relocations ENUM BFD_RELOC_X86_64_GOT32 ENUMX BFD_RELOC_X86_64_PLT32 ENUMX BFD_RELOC_X86_64_COPY ENUMX BFD_RELOC_X86_64_GLOB_DAT ENUMX BFD_RELOC_X86_64_JUMP_SLOT ENUMX BFD_RELOC_X86_64_RELATIVE ENUMX BFD_RELOC_X86_64_GOTPCREL ENUMX BFD_RELOC_X86_64_32S ENUMX BFD_RELOC_X86_64_DTPMOD64 ENUMX BFD_RELOC_X86_64_DTPOFF64 ENUMX BFD_RELOC_X86_64_TPOFF64 ENUMX BFD_RELOC_X86_64_TLSGD ENUMX BFD_RELOC_X86_64_TLSLD ENUMX BFD_RELOC_X86_64_DTPOFF32 ENUMX BFD_RELOC_X86_64_GOTTPOFF ENUMX BFD_RELOC_X86_64_TPOFF32 ENUMX BFD_RELOC_X86_64_GOTOFF64 ENUMX BFD_RELOC_X86_64_GOTPC32 ENUMX BFD_RELOC_X86_64_GOT64 ENUMX BFD_RELOC_X86_64_GOTPCREL64 ENUMX BFD_RELOC_X86_64_GOTPC64 ENUMX BFD_RELOC_X86_64_GOTPLT64 ENUMX BFD_RELOC_X86_64_PLTOFF64 ENUMX BFD_RELOC_X86_64_GOTPC32_TLSDESC ENUMX BFD_RELOC_X86_64_TLSDESC_CALL ENUMX BFD_RELOC_X86_64_TLSDESC ENUMDOC x86-64/elf relocations ENUM BFD_RELOC_NS32K_IMM_8 ENUMX BFD_RELOC_NS32K_IMM_16 ENUMX BFD_RELOC_NS32K_IMM_32 ENUMX BFD_RELOC_NS32K_IMM_8_PCREL ENUMX BFD_RELOC_NS32K_IMM_16_PCREL ENUMX BFD_RELOC_NS32K_IMM_32_PCREL ENUMX BFD_RELOC_NS32K_DISP_8 ENUMX BFD_RELOC_NS32K_DISP_16 ENUMX BFD_RELOC_NS32K_DISP_32 ENUMX BFD_RELOC_NS32K_DISP_8_PCREL ENUMX BFD_RELOC_NS32K_DISP_16_PCREL ENUMX BFD_RELOC_NS32K_DISP_32_PCREL ENUMDOC ns32k relocations ENUM BFD_RELOC_PDP11_DISP_8_PCREL ENUMX BFD_RELOC_PDP11_DISP_6_PCREL ENUMDOC PDP11 relocations ENUM BFD_RELOC_PJ_CODE_HI16 ENUMX BFD_RELOC_PJ_CODE_LO16 ENUMX BFD_RELOC_PJ_CODE_DIR16 ENUMX BFD_RELOC_PJ_CODE_DIR32 ENUMX BFD_RELOC_PJ_CODE_REL16 ENUMX BFD_RELOC_PJ_CODE_REL32 ENUMDOC Picojava relocs. Not all of these appear in object files. ENUM BFD_RELOC_PPC_B26 ENUMX BFD_RELOC_PPC_BA26 ENUMX BFD_RELOC_PPC_TOC16 ENUMX BFD_RELOC_PPC_B16 ENUMX BFD_RELOC_PPC_B16_BRTAKEN ENUMX BFD_RELOC_PPC_B16_BRNTAKEN ENUMX BFD_RELOC_PPC_BA16 ENUMX BFD_RELOC_PPC_BA16_BRTAKEN ENUMX BFD_RELOC_PPC_BA16_BRNTAKEN ENUMX BFD_RELOC_PPC_COPY ENUMX BFD_RELOC_PPC_GLOB_DAT ENUMX BFD_RELOC_PPC_JMP_SLOT ENUMX BFD_RELOC_PPC_RELATIVE ENUMX BFD_RELOC_PPC_LOCAL24PC ENUMX BFD_RELOC_PPC_EMB_NADDR32 ENUMX BFD_RELOC_PPC_EMB_NADDR16 ENUMX BFD_RELOC_PPC_EMB_NADDR16_LO ENUMX BFD_RELOC_PPC_EMB_NADDR16_HI ENUMX BFD_RELOC_PPC_EMB_NADDR16_HA ENUMX BFD_RELOC_PPC_EMB_SDAI16 ENUMX BFD_RELOC_PPC_EMB_SDA2I16 ENUMX BFD_RELOC_PPC_EMB_SDA2REL ENUMX BFD_RELOC_PPC_EMB_SDA21 ENUMX BFD_RELOC_PPC_EMB_MRKREF ENUMX BFD_RELOC_PPC_EMB_RELSEC16 ENUMX BFD_RELOC_PPC_EMB_RELST_LO ENUMX BFD_RELOC_PPC_EMB_RELST_HI ENUMX BFD_RELOC_PPC_EMB_RELST_HA ENUMX BFD_RELOC_PPC_EMB_BIT_FLD ENUMX BFD_RELOC_PPC_EMB_RELSDA ENUMX BFD_RELOC_PPC64_HIGHER ENUMX BFD_RELOC_PPC64_HIGHER_S ENUMX BFD_RELOC_PPC64_HIGHEST ENUMX BFD_RELOC_PPC64_HIGHEST_S ENUMX BFD_RELOC_PPC64_TOC16_LO ENUMX BFD_RELOC_PPC64_TOC16_HI ENUMX BFD_RELOC_PPC64_TOC16_HA ENUMX BFD_RELOC_PPC64_TOC ENUMX BFD_RELOC_PPC64_PLTGOT16 ENUMX BFD_RELOC_PPC64_PLTGOT16_LO ENUMX BFD_RELOC_PPC64_PLTGOT16_HI ENUMX BFD_RELOC_PPC64_PLTGOT16_HA ENUMX BFD_RELOC_PPC64_ADDR16_DS ENUMX BFD_RELOC_PPC64_ADDR16_LO_DS ENUMX BFD_RELOC_PPC64_GOT16_DS ENUMX BFD_RELOC_PPC64_GOT16_LO_DS ENUMX BFD_RELOC_PPC64_PLT16_LO_DS ENUMX BFD_RELOC_PPC64_SECTOFF_DS ENUMX BFD_RELOC_PPC64_SECTOFF_LO_DS ENUMX BFD_RELOC_PPC64_TOC16_DS ENUMX BFD_RELOC_PPC64_TOC16_LO_DS ENUMX BFD_RELOC_PPC64_PLTGOT16_DS ENUMX BFD_RELOC_PPC64_PLTGOT16_LO_DS ENUMDOC Power(rs6000) and PowerPC relocations. ENUM BFD_RELOC_PPC_TLS ENUMX + BFD_RELOC_PPC_TLSGD +ENUMX + BFD_RELOC_PPC_TLSLD +ENUMX BFD_RELOC_PPC_DTPMOD ENUMX BFD_RELOC_PPC_TPREL16 ENUMX BFD_RELOC_PPC_TPREL16_LO ENUMX BFD_RELOC_PPC_TPREL16_HI ENUMX BFD_RELOC_PPC_TPREL16_HA ENUMX BFD_RELOC_PPC_TPREL ENUMX BFD_RELOC_PPC_DTPREL16 ENUMX BFD_RELOC_PPC_DTPREL16_LO ENUMX BFD_RELOC_PPC_DTPREL16_HI ENUMX BFD_RELOC_PPC_DTPREL16_HA ENUMX BFD_RELOC_PPC_DTPREL ENUMX BFD_RELOC_PPC_GOT_TLSGD16 ENUMX BFD_RELOC_PPC_GOT_TLSGD16_LO ENUMX BFD_RELOC_PPC_GOT_TLSGD16_HI ENUMX BFD_RELOC_PPC_GOT_TLSGD16_HA ENUMX BFD_RELOC_PPC_GOT_TLSLD16 ENUMX BFD_RELOC_PPC_GOT_TLSLD16_LO ENUMX BFD_RELOC_PPC_GOT_TLSLD16_HI ENUMX BFD_RELOC_PPC_GOT_TLSLD16_HA ENUMX BFD_RELOC_PPC_GOT_TPREL16 ENUMX BFD_RELOC_PPC_GOT_TPREL16_LO ENUMX BFD_RELOC_PPC_GOT_TPREL16_HI ENUMX BFD_RELOC_PPC_GOT_TPREL16_HA ENUMX BFD_RELOC_PPC_GOT_DTPREL16 ENUMX BFD_RELOC_PPC_GOT_DTPREL16_LO ENUMX BFD_RELOC_PPC_GOT_DTPREL16_HI ENUMX BFD_RELOC_PPC_GOT_DTPREL16_HA ENUMX BFD_RELOC_PPC64_TPREL16_DS ENUMX BFD_RELOC_PPC64_TPREL16_LO_DS ENUMX BFD_RELOC_PPC64_TPREL16_HIGHER ENUMX BFD_RELOC_PPC64_TPREL16_HIGHERA ENUMX BFD_RELOC_PPC64_TPREL16_HIGHEST ENUMX BFD_RELOC_PPC64_TPREL16_HIGHESTA ENUMX BFD_RELOC_PPC64_DTPREL16_DS ENUMX BFD_RELOC_PPC64_DTPREL16_LO_DS ENUMX BFD_RELOC_PPC64_DTPREL16_HIGHER ENUMX BFD_RELOC_PPC64_DTPREL16_HIGHERA ENUMX BFD_RELOC_PPC64_DTPREL16_HIGHEST ENUMX BFD_RELOC_PPC64_DTPREL16_HIGHESTA ENUMDOC PowerPC and PowerPC64 thread-local storage relocations. ENUM BFD_RELOC_I370_D12 ENUMDOC IBM 370/390 relocations ENUM BFD_RELOC_CTOR ENUMDOC The type of reloc used to build a constructor table - at the moment probably a 32 bit wide absolute relocation, but the target can choose. It generally does map to one of the other relocation types. ENUM BFD_RELOC_ARM_PCREL_BRANCH ENUMDOC ARM 26 bit pc-relative branch. The lowest two bits must be zero and are not stored in the instruction. ENUM BFD_RELOC_ARM_PCREL_BLX ENUMDOC ARM 26 bit pc-relative branch. The lowest bit must be zero and is not stored in the instruction. The 2nd lowest bit comes from a 1 bit field in the instruction. ENUM BFD_RELOC_THUMB_PCREL_BLX ENUMDOC Thumb 22 bit pc-relative branch. The lowest bit must be zero and is not stored in the instruction. The 2nd lowest bit comes from a 1 bit field in the instruction. ENUM BFD_RELOC_ARM_PCREL_CALL ENUMDOC ARM 26-bit pc-relative branch for an unconditional BL or BLX instruction. ENUM BFD_RELOC_ARM_PCREL_JUMP ENUMDOC ARM 26-bit pc-relative branch for B or conditional BL instruction. ENUM BFD_RELOC_THUMB_PCREL_BRANCH7 ENUMX BFD_RELOC_THUMB_PCREL_BRANCH9 ENUMX BFD_RELOC_THUMB_PCREL_BRANCH12 ENUMX BFD_RELOC_THUMB_PCREL_BRANCH20 ENUMX BFD_RELOC_THUMB_PCREL_BRANCH23 ENUMX BFD_RELOC_THUMB_PCREL_BRANCH25 ENUMDOC Thumb 7-, 9-, 12-, 20-, 23-, and 25-bit pc-relative branches. The lowest bit must be zero and is not stored in the instruction. Note that the corresponding ELF R_ARM_THM_JUMPnn constant has an "nn" one smaller in all cases. Note further that BRANCH23 corresponds to R_ARM_THM_CALL. ENUM BFD_RELOC_ARM_OFFSET_IMM ENUMDOC 12-bit immediate offset, used in ARM-format ldr and str instructions. ENUM BFD_RELOC_ARM_THUMB_OFFSET ENUMDOC 5-bit immediate offset, used in Thumb-format ldr and str instructions. ENUM BFD_RELOC_ARM_TARGET1 ENUMDOC Pc-relative or absolute relocation depending on target. Used for entries in .init_array sections. ENUM BFD_RELOC_ARM_ROSEGREL32 ENUMDOC Read-only segment base relative address. ENUM BFD_RELOC_ARM_SBREL32 ENUMDOC Data segment base relative address. ENUM BFD_RELOC_ARM_TARGET2 ENUMDOC This reloc is used for references to RTTI data from exception handling tables. The actual definition depends on the target. It may be a pc-relative or some form of GOT-indirect relocation. ENUM BFD_RELOC_ARM_PREL31 ENUMDOC 31-bit PC relative address. ENUM BFD_RELOC_ARM_MOVW ENUMX BFD_RELOC_ARM_MOVT ENUMX BFD_RELOC_ARM_MOVW_PCREL ENUMX BFD_RELOC_ARM_MOVT_PCREL ENUMX BFD_RELOC_ARM_THUMB_MOVW ENUMX BFD_RELOC_ARM_THUMB_MOVT ENUMX BFD_RELOC_ARM_THUMB_MOVW_PCREL ENUMX BFD_RELOC_ARM_THUMB_MOVT_PCREL ENUMDOC Low and High halfword relocations for MOVW and MOVT instructions. ENUM BFD_RELOC_ARM_JUMP_SLOT ENUMX BFD_RELOC_ARM_GLOB_DAT ENUMX BFD_RELOC_ARM_GOT32 ENUMX BFD_RELOC_ARM_PLT32 ENUMX BFD_RELOC_ARM_RELATIVE ENUMX BFD_RELOC_ARM_GOTOFF ENUMX BFD_RELOC_ARM_GOTPC ENUMDOC Relocations for setting up GOTs and PLTs for shared libraries. ENUM BFD_RELOC_ARM_TLS_GD32 ENUMX BFD_RELOC_ARM_TLS_LDO32 ENUMX BFD_RELOC_ARM_TLS_LDM32 ENUMX BFD_RELOC_ARM_TLS_DTPOFF32 ENUMX BFD_RELOC_ARM_TLS_DTPMOD32 ENUMX BFD_RELOC_ARM_TLS_TPOFF32 ENUMX BFD_RELOC_ARM_TLS_IE32 ENUMX BFD_RELOC_ARM_TLS_LE32 ENUMDOC ARM thread-local storage relocations. ENUM BFD_RELOC_ARM_ALU_PC_G0_NC ENUMX BFD_RELOC_ARM_ALU_PC_G0 ENUMX BFD_RELOC_ARM_ALU_PC_G1_NC ENUMX BFD_RELOC_ARM_ALU_PC_G1 ENUMX BFD_RELOC_ARM_ALU_PC_G2 ENUMX BFD_RELOC_ARM_LDR_PC_G0 ENUMX BFD_RELOC_ARM_LDR_PC_G1 ENUMX BFD_RELOC_ARM_LDR_PC_G2 ENUMX BFD_RELOC_ARM_LDRS_PC_G0 ENUMX BFD_RELOC_ARM_LDRS_PC_G1 ENUMX BFD_RELOC_ARM_LDRS_PC_G2 ENUMX BFD_RELOC_ARM_LDC_PC_G0 ENUMX BFD_RELOC_ARM_LDC_PC_G1 ENUMX BFD_RELOC_ARM_LDC_PC_G2 ENUMX BFD_RELOC_ARM_ALU_SB_G0_NC ENUMX BFD_RELOC_ARM_ALU_SB_G0 ENUMX BFD_RELOC_ARM_ALU_SB_G1_NC ENUMX BFD_RELOC_ARM_ALU_SB_G1 ENUMX BFD_RELOC_ARM_ALU_SB_G2 ENUMX BFD_RELOC_ARM_LDR_SB_G0 ENUMX BFD_RELOC_ARM_LDR_SB_G1 ENUMX BFD_RELOC_ARM_LDR_SB_G2 ENUMX BFD_RELOC_ARM_LDRS_SB_G0 ENUMX BFD_RELOC_ARM_LDRS_SB_G1 ENUMX BFD_RELOC_ARM_LDRS_SB_G2 ENUMX BFD_RELOC_ARM_LDC_SB_G0 ENUMX BFD_RELOC_ARM_LDC_SB_G1 ENUMX BFD_RELOC_ARM_LDC_SB_G2 ENUMDOC ARM group relocations. ENUM BFD_RELOC_ARM_IMMEDIATE ENUMX BFD_RELOC_ARM_ADRL_IMMEDIATE ENUMX BFD_RELOC_ARM_T32_IMMEDIATE ENUMX BFD_RELOC_ARM_T32_ADD_IMM ENUMX BFD_RELOC_ARM_T32_IMM12 ENUMX BFD_RELOC_ARM_T32_ADD_PC12 ENUMX BFD_RELOC_ARM_SHIFT_IMM ENUMX BFD_RELOC_ARM_SMC ENUMX BFD_RELOC_ARM_SWI ENUMX BFD_RELOC_ARM_MULTI ENUMX BFD_RELOC_ARM_CP_OFF_IMM ENUMX BFD_RELOC_ARM_CP_OFF_IMM_S2 ENUMX BFD_RELOC_ARM_T32_CP_OFF_IMM ENUMX BFD_RELOC_ARM_T32_CP_OFF_IMM_S2 ENUMX BFD_RELOC_ARM_ADR_IMM ENUMX BFD_RELOC_ARM_LDR_IMM ENUMX BFD_RELOC_ARM_LITERAL ENUMX BFD_RELOC_ARM_IN_POOL ENUMX BFD_RELOC_ARM_OFFSET_IMM8 ENUMX BFD_RELOC_ARM_T32_OFFSET_U8 ENUMX BFD_RELOC_ARM_T32_OFFSET_IMM ENUMX BFD_RELOC_ARM_HWLITERAL ENUMX BFD_RELOC_ARM_THUMB_ADD ENUMX BFD_RELOC_ARM_THUMB_IMM ENUMX BFD_RELOC_ARM_THUMB_SHIFT ENUMDOC These relocs are only used within the ARM assembler. They are not (at present) written to any object files. ENUM BFD_RELOC_SH_PCDISP8BY2 ENUMX BFD_RELOC_SH_PCDISP12BY2 ENUMX BFD_RELOC_SH_IMM3 ENUMX BFD_RELOC_SH_IMM3U ENUMX BFD_RELOC_SH_DISP12 ENUMX BFD_RELOC_SH_DISP12BY2 ENUMX BFD_RELOC_SH_DISP12BY4 ENUMX BFD_RELOC_SH_DISP12BY8 ENUMX BFD_RELOC_SH_DISP20 ENUMX BFD_RELOC_SH_DISP20BY8 ENUMX BFD_RELOC_SH_IMM4 ENUMX BFD_RELOC_SH_IMM4BY2 ENUMX BFD_RELOC_SH_IMM4BY4 ENUMX BFD_RELOC_SH_IMM8 ENUMX BFD_RELOC_SH_IMM8BY2 ENUMX BFD_RELOC_SH_IMM8BY4 ENUMX BFD_RELOC_SH_PCRELIMM8BY2 ENUMX BFD_RELOC_SH_PCRELIMM8BY4 ENUMX BFD_RELOC_SH_SWITCH16 ENUMX BFD_RELOC_SH_SWITCH32 ENUMX BFD_RELOC_SH_USES ENUMX BFD_RELOC_SH_COUNT ENUMX BFD_RELOC_SH_ALIGN ENUMX BFD_RELOC_SH_CODE ENUMX BFD_RELOC_SH_DATA ENUMX BFD_RELOC_SH_LABEL ENUMX BFD_RELOC_SH_LOOP_START ENUMX BFD_RELOC_SH_LOOP_END ENUMX BFD_RELOC_SH_COPY ENUMX BFD_RELOC_SH_GLOB_DAT ENUMX BFD_RELOC_SH_JMP_SLOT ENUMX BFD_RELOC_SH_RELATIVE ENUMX BFD_RELOC_SH_GOTPC ENUMX BFD_RELOC_SH_GOT_LOW16 ENUMX BFD_RELOC_SH_GOT_MEDLOW16 ENUMX BFD_RELOC_SH_GOT_MEDHI16 ENUMX BFD_RELOC_SH_GOT_HI16 ENUMX BFD_RELOC_SH_GOTPLT_LOW16 ENUMX BFD_RELOC_SH_GOTPLT_MEDLOW16 ENUMX BFD_RELOC_SH_GOTPLT_MEDHI16 ENUMX BFD_RELOC_SH_GOTPLT_HI16 ENUMX BFD_RELOC_SH_PLT_LOW16 ENUMX BFD_RELOC_SH_PLT_MEDLOW16 ENUMX BFD_RELOC_SH_PLT_MEDHI16 ENUMX BFD_RELOC_SH_PLT_HI16 ENUMX BFD_RELOC_SH_GOTOFF_LOW16 ENUMX BFD_RELOC_SH_GOTOFF_MEDLOW16 ENUMX BFD_RELOC_SH_GOTOFF_MEDHI16 ENUMX BFD_RELOC_SH_GOTOFF_HI16 ENUMX BFD_RELOC_SH_GOTPC_LOW16 ENUMX BFD_RELOC_SH_GOTPC_MEDLOW16 ENUMX BFD_RELOC_SH_GOTPC_MEDHI16 ENUMX BFD_RELOC_SH_GOTPC_HI16 ENUMX BFD_RELOC_SH_COPY64 ENUMX BFD_RELOC_SH_GLOB_DAT64 ENUMX BFD_RELOC_SH_JMP_SLOT64 ENUMX BFD_RELOC_SH_RELATIVE64 ENUMX BFD_RELOC_SH_GOT10BY4 ENUMX BFD_RELOC_SH_GOT10BY8 ENUMX BFD_RELOC_SH_GOTPLT10BY4 ENUMX BFD_RELOC_SH_GOTPLT10BY8 ENUMX BFD_RELOC_SH_GOTPLT32 ENUMX BFD_RELOC_SH_SHMEDIA_CODE ENUMX BFD_RELOC_SH_IMMU5 ENUMX BFD_RELOC_SH_IMMS6 ENUMX BFD_RELOC_SH_IMMS6BY32 ENUMX BFD_RELOC_SH_IMMU6 ENUMX BFD_RELOC_SH_IMMS10 ENUMX BFD_RELOC_SH_IMMS10BY2 ENUMX BFD_RELOC_SH_IMMS10BY4 ENUMX BFD_RELOC_SH_IMMS10BY8 ENUMX BFD_RELOC_SH_IMMS16 ENUMX BFD_RELOC_SH_IMMU16 ENUMX BFD_RELOC_SH_IMM_LOW16 ENUMX BFD_RELOC_SH_IMM_LOW16_PCREL ENUMX BFD_RELOC_SH_IMM_MEDLOW16 ENUMX BFD_RELOC_SH_IMM_MEDLOW16_PCREL ENUMX BFD_RELOC_SH_IMM_MEDHI16 ENUMX BFD_RELOC_SH_IMM_MEDHI16_PCREL ENUMX BFD_RELOC_SH_IMM_HI16 ENUMX BFD_RELOC_SH_IMM_HI16_PCREL ENUMX BFD_RELOC_SH_PT_16 ENUMX BFD_RELOC_SH_TLS_GD_32 ENUMX BFD_RELOC_SH_TLS_LD_32 ENUMX BFD_RELOC_SH_TLS_LDO_32 ENUMX BFD_RELOC_SH_TLS_IE_32 ENUMX BFD_RELOC_SH_TLS_LE_32 ENUMX BFD_RELOC_SH_TLS_DTPMOD32 ENUMX BFD_RELOC_SH_TLS_DTPOFF32 ENUMX BFD_RELOC_SH_TLS_TPOFF32 ENUMDOC Renesas / SuperH SH relocs. Not all of these appear in object files. ENUM BFD_RELOC_ARC_B22_PCREL ENUMDOC ARC Cores relocs. ARC 22 bit pc-relative branch. The lowest two bits must be zero and are not stored in the instruction. The high 20 bits are installed in bits 26 through 7 of the instruction. ENUM BFD_RELOC_ARC_B26 ENUMDOC ARC 26 bit absolute branch. The lowest two bits must be zero and are not stored in the instruction. The high 24 bits are installed in bits 23 through 0. ENUM BFD_RELOC_BFIN_16_IMM ENUMDOC ADI Blackfin 16 bit immediate absolute reloc. ENUM BFD_RELOC_BFIN_16_HIGH ENUMDOC ADI Blackfin 16 bit immediate absolute reloc higher 16 bits. ENUM BFD_RELOC_BFIN_4_PCREL ENUMDOC ADI Blackfin 'a' part of LSETUP. ENUM BFD_RELOC_BFIN_5_PCREL ENUMDOC ADI Blackfin. ENUM BFD_RELOC_BFIN_16_LOW ENUMDOC ADI Blackfin 16 bit immediate absolute reloc lower 16 bits. ENUM BFD_RELOC_BFIN_10_PCREL ENUMDOC ADI Blackfin. ENUM BFD_RELOC_BFIN_11_PCREL ENUMDOC ADI Blackfin 'b' part of LSETUP. ENUM BFD_RELOC_BFIN_12_PCREL_JUMP ENUMDOC ADI Blackfin. ENUM BFD_RELOC_BFIN_12_PCREL_JUMP_S ENUMDOC ADI Blackfin Short jump, pcrel. ENUM BFD_RELOC_BFIN_24_PCREL_CALL_X ENUMDOC ADI Blackfin Call.x not implemented. ENUM BFD_RELOC_BFIN_24_PCREL_JUMP_L ENUMDOC ADI Blackfin Long Jump pcrel. ENUM BFD_RELOC_BFIN_GOT17M4 ENUMX BFD_RELOC_BFIN_GOTHI ENUMX BFD_RELOC_BFIN_GOTLO ENUMX BFD_RELOC_BFIN_FUNCDESC ENUMX BFD_RELOC_BFIN_FUNCDESC_GOT17M4 ENUMX BFD_RELOC_BFIN_FUNCDESC_GOTHI ENUMX BFD_RELOC_BFIN_FUNCDESC_GOTLO ENUMX BFD_RELOC_BFIN_FUNCDESC_VALUE ENUMX BFD_RELOC_BFIN_FUNCDESC_GOTOFF17M4 ENUMX BFD_RELOC_BFIN_FUNCDESC_GOTOFFHI ENUMX BFD_RELOC_BFIN_FUNCDESC_GOTOFFLO ENUMX BFD_RELOC_BFIN_GOTOFF17M4 ENUMX BFD_RELOC_BFIN_GOTOFFHI ENUMX BFD_RELOC_BFIN_GOTOFFLO ENUMDOC ADI Blackfin FD-PIC relocations. ENUM BFD_RELOC_BFIN_GOT ENUMDOC ADI Blackfin GOT relocation. ENUM BFD_RELOC_BFIN_PLTPC ENUMDOC ADI Blackfin PLTPC relocation. ENUM BFD_ARELOC_BFIN_PUSH ENUMDOC ADI Blackfin arithmetic relocation. ENUM BFD_ARELOC_BFIN_CONST ENUMDOC ADI Blackfin arithmetic relocation. ENUM BFD_ARELOC_BFIN_ADD ENUMDOC ADI Blackfin arithmetic relocation. ENUM BFD_ARELOC_BFIN_SUB ENUMDOC ADI Blackfin arithmetic relocation. ENUM BFD_ARELOC_BFIN_MULT ENUMDOC ADI Blackfin arithmetic relocation. ENUM BFD_ARELOC_BFIN_DIV ENUMDOC ADI Blackfin arithmetic relocation. ENUM BFD_ARELOC_BFIN_MOD ENUMDOC ADI Blackfin arithmetic relocation. ENUM BFD_ARELOC_BFIN_LSHIFT ENUMDOC ADI Blackfin arithmetic relocation. ENUM BFD_ARELOC_BFIN_RSHIFT ENUMDOC ADI Blackfin arithmetic relocation. ENUM BFD_ARELOC_BFIN_AND ENUMDOC ADI Blackfin arithmetic relocation. ENUM BFD_ARELOC_BFIN_OR ENUMDOC ADI Blackfin arithmetic relocation. ENUM BFD_ARELOC_BFIN_XOR ENUMDOC ADI Blackfin arithmetic relocation. ENUM BFD_ARELOC_BFIN_LAND ENUMDOC ADI Blackfin arithmetic relocation. ENUM BFD_ARELOC_BFIN_LOR ENUMDOC ADI Blackfin arithmetic relocation. ENUM BFD_ARELOC_BFIN_LEN ENUMDOC ADI Blackfin arithmetic relocation. ENUM BFD_ARELOC_BFIN_NEG ENUMDOC ADI Blackfin arithmetic relocation. ENUM BFD_ARELOC_BFIN_COMP ENUMDOC ADI Blackfin arithmetic relocation. ENUM BFD_ARELOC_BFIN_PAGE ENUMDOC ADI Blackfin arithmetic relocation. ENUM BFD_ARELOC_BFIN_HWPAGE ENUMDOC ADI Blackfin arithmetic relocation. ENUM BFD_ARELOC_BFIN_ADDR ENUMDOC ADI Blackfin arithmetic relocation. ENUM BFD_RELOC_D10V_10_PCREL_R ENUMDOC Mitsubishi D10V relocs. This is a 10-bit reloc with the right 2 bits assumed to be 0. ENUM BFD_RELOC_D10V_10_PCREL_L ENUMDOC Mitsubishi D10V relocs. This is a 10-bit reloc with the right 2 bits assumed to be 0. This is the same as the previous reloc except it is in the left container, i.e., shifted left 15 bits. ENUM BFD_RELOC_D10V_18 ENUMDOC This is an 18-bit reloc with the right 2 bits assumed to be 0. ENUM BFD_RELOC_D10V_18_PCREL ENUMDOC This is an 18-bit reloc with the right 2 bits assumed to be 0. ENUM BFD_RELOC_D30V_6 ENUMDOC Mitsubishi D30V relocs. This is a 6-bit absolute reloc. ENUM BFD_RELOC_D30V_9_PCREL ENUMDOC This is a 6-bit pc-relative reloc with the right 3 bits assumed to be 0. ENUM BFD_RELOC_D30V_9_PCREL_R ENUMDOC This is a 6-bit pc-relative reloc with the right 3 bits assumed to be 0. Same as the previous reloc but on the right side of the container. ENUM BFD_RELOC_D30V_15 ENUMDOC This is a 12-bit absolute reloc with the right 3 bitsassumed to be 0. ENUM BFD_RELOC_D30V_15_PCREL ENUMDOC This is a 12-bit pc-relative reloc with the right 3 bits assumed to be 0. ENUM BFD_RELOC_D30V_15_PCREL_R ENUMDOC This is a 12-bit pc-relative reloc with the right 3 bits assumed to be 0. Same as the previous reloc but on the right side of the container. ENUM BFD_RELOC_D30V_21 ENUMDOC This is an 18-bit absolute reloc with the right 3 bits assumed to be 0. ENUM BFD_RELOC_D30V_21_PCREL ENUMDOC This is an 18-bit pc-relative reloc with the right 3 bits assumed to be 0. ENUM BFD_RELOC_D30V_21_PCREL_R ENUMDOC This is an 18-bit pc-relative reloc with the right 3 bits assumed to be 0. Same as the previous reloc but on the right side of the container. ENUM BFD_RELOC_D30V_32 ENUMDOC This is a 32-bit absolute reloc. ENUM BFD_RELOC_D30V_32_PCREL ENUMDOC This is a 32-bit pc-relative reloc. ENUM BFD_RELOC_DLX_HI16_S ENUMDOC DLX relocs ENUM BFD_RELOC_DLX_LO16 ENUMDOC DLX relocs ENUM BFD_RELOC_DLX_JMP26 ENUMDOC DLX relocs ENUM BFD_RELOC_M32C_HI8 ENUMX BFD_RELOC_M32C_RL_JUMP ENUMX BFD_RELOC_M32C_RL_1ADDR ENUMX BFD_RELOC_M32C_RL_2ADDR ENUMDOC Renesas M16C/M32C Relocations. ENUM BFD_RELOC_M32R_24 ENUMDOC Renesas M32R (formerly Mitsubishi M32R) relocs. This is a 24 bit absolute address. ENUM BFD_RELOC_M32R_10_PCREL ENUMDOC This is a 10-bit pc-relative reloc with the right 2 bits assumed to be 0. ENUM BFD_RELOC_M32R_18_PCREL ENUMDOC This is an 18-bit reloc with the right 2 bits assumed to be 0. ENUM BFD_RELOC_M32R_26_PCREL ENUMDOC This is a 26-bit reloc with the right 2 bits assumed to be 0. ENUM BFD_RELOC_M32R_HI16_ULO ENUMDOC This is a 16-bit reloc containing the high 16 bits of an address used when the lower 16 bits are treated as unsigned. ENUM BFD_RELOC_M32R_HI16_SLO ENUMDOC This is a 16-bit reloc containing the high 16 bits of an address used when the lower 16 bits are treated as signed. ENUM BFD_RELOC_M32R_LO16 ENUMDOC This is a 16-bit reloc containing the lower 16 bits of an address. ENUM BFD_RELOC_M32R_SDA16 ENUMDOC This is a 16-bit reloc containing the small data area offset for use in add3, load, and store instructions. ENUM BFD_RELOC_M32R_GOT24 ENUMX BFD_RELOC_M32R_26_PLTREL ENUMX BFD_RELOC_M32R_COPY ENUMX BFD_RELOC_M32R_GLOB_DAT ENUMX BFD_RELOC_M32R_JMP_SLOT ENUMX BFD_RELOC_M32R_RELATIVE ENUMX BFD_RELOC_M32R_GOTOFF ENUMX BFD_RELOC_M32R_GOTOFF_HI_ULO ENUMX BFD_RELOC_M32R_GOTOFF_HI_SLO ENUMX BFD_RELOC_M32R_GOTOFF_LO ENUMX BFD_RELOC_M32R_GOTPC24 ENUMX BFD_RELOC_M32R_GOT16_HI_ULO ENUMX BFD_RELOC_M32R_GOT16_HI_SLO ENUMX BFD_RELOC_M32R_GOT16_LO ENUMX BFD_RELOC_M32R_GOTPC_HI_ULO ENUMX BFD_RELOC_M32R_GOTPC_HI_SLO ENUMX BFD_RELOC_M32R_GOTPC_LO ENUMDOC For PIC. ENUM BFD_RELOC_V850_9_PCREL ENUMDOC This is a 9-bit reloc ENUM BFD_RELOC_V850_22_PCREL ENUMDOC This is a 22-bit reloc ENUM BFD_RELOC_V850_SDA_16_16_OFFSET ENUMDOC This is a 16 bit offset from the short data area pointer. ENUM BFD_RELOC_V850_SDA_15_16_OFFSET ENUMDOC This is a 16 bit offset (of which only 15 bits are used) from the short data area pointer. ENUM BFD_RELOC_V850_ZDA_16_16_OFFSET ENUMDOC This is a 16 bit offset from the zero data area pointer. ENUM BFD_RELOC_V850_ZDA_15_16_OFFSET ENUMDOC This is a 16 bit offset (of which only 15 bits are used) from the zero data area pointer. ENUM BFD_RELOC_V850_TDA_6_8_OFFSET ENUMDOC This is an 8 bit offset (of which only 6 bits are used) from the tiny data area pointer. ENUM BFD_RELOC_V850_TDA_7_8_OFFSET ENUMDOC This is an 8bit offset (of which only 7 bits are used) from the tiny data area pointer. ENUM BFD_RELOC_V850_TDA_7_7_OFFSET ENUMDOC This is a 7 bit offset from the tiny data area pointer. ENUM BFD_RELOC_V850_TDA_16_16_OFFSET ENUMDOC This is a 16 bit offset from the tiny data area pointer. COMMENT ENUM BFD_RELOC_V850_TDA_4_5_OFFSET ENUMDOC This is a 5 bit offset (of which only 4 bits are used) from the tiny data area pointer. ENUM BFD_RELOC_V850_TDA_4_4_OFFSET ENUMDOC This is a 4 bit offset from the tiny data area pointer. ENUM BFD_RELOC_V850_SDA_16_16_SPLIT_OFFSET ENUMDOC This is a 16 bit offset from the short data area pointer, with the bits placed non-contiguously in the instruction. ENUM BFD_RELOC_V850_ZDA_16_16_SPLIT_OFFSET ENUMDOC This is a 16 bit offset from the zero data area pointer, with the bits placed non-contiguously in the instruction. ENUM BFD_RELOC_V850_CALLT_6_7_OFFSET ENUMDOC This is a 6 bit offset from the call table base pointer. ENUM BFD_RELOC_V850_CALLT_16_16_OFFSET ENUMDOC This is a 16 bit offset from the call table base pointer. ENUM BFD_RELOC_V850_LONGCALL ENUMDOC Used for relaxing indirect function calls. ENUM BFD_RELOC_V850_LONGJUMP ENUMDOC Used for relaxing indirect jumps. ENUM BFD_RELOC_V850_ALIGN ENUMDOC Used to maintain alignment whilst relaxing. ENUM BFD_RELOC_V850_LO16_SPLIT_OFFSET ENUMDOC This is a variation of BFD_RELOC_LO16 that can be used in v850e ld.bu instructions. ENUM BFD_RELOC_MN10300_32_PCREL ENUMDOC This is a 32bit pcrel reloc for the mn10300, offset by two bytes in the instruction. ENUM BFD_RELOC_MN10300_16_PCREL ENUMDOC This is a 16bit pcrel reloc for the mn10300, offset by two bytes in the instruction. ENUM BFD_RELOC_TIC30_LDP ENUMDOC This is a 8bit DP reloc for the tms320c30, where the most significant 8 bits of a 24 bit word are placed into the least significant 8 bits of the opcode. ENUM BFD_RELOC_TIC54X_PARTLS7 ENUMDOC This is a 7bit reloc for the tms320c54x, where the least significant 7 bits of a 16 bit word are placed into the least significant 7 bits of the opcode. ENUM BFD_RELOC_TIC54X_PARTMS9 ENUMDOC This is a 9bit DP reloc for the tms320c54x, where the most significant 9 bits of a 16 bit word are placed into the least significant 9 bits of the opcode. ENUM BFD_RELOC_TIC54X_23 ENUMDOC This is an extended address 23-bit reloc for the tms320c54x. ENUM BFD_RELOC_TIC54X_16_OF_23 ENUMDOC This is a 16-bit reloc for the tms320c54x, where the least significant 16 bits of a 23-bit extended address are placed into the opcode. ENUM BFD_RELOC_TIC54X_MS7_OF_23 ENUMDOC This is a reloc for the tms320c54x, where the most significant 7 bits of a 23-bit extended address are placed into the opcode. ENUM BFD_RELOC_FR30_48 ENUMDOC This is a 48 bit reloc for the FR30 that stores 32 bits. ENUM BFD_RELOC_FR30_20 ENUMDOC This is a 32 bit reloc for the FR30 that stores 20 bits split up into two sections. ENUM BFD_RELOC_FR30_6_IN_4 ENUMDOC This is a 16 bit reloc for the FR30 that stores a 6 bit word offset in 4 bits. ENUM BFD_RELOC_FR30_8_IN_8 ENUMDOC This is a 16 bit reloc for the FR30 that stores an 8 bit byte offset into 8 bits. ENUM BFD_RELOC_FR30_9_IN_8 ENUMDOC This is a 16 bit reloc for the FR30 that stores a 9 bit short offset into 8 bits. ENUM BFD_RELOC_FR30_10_IN_8 ENUMDOC This is a 16 bit reloc for the FR30 that stores a 10 bit word offset into 8 bits. ENUM BFD_RELOC_FR30_9_PCREL ENUMDOC This is a 16 bit reloc for the FR30 that stores a 9 bit pc relative short offset into 8 bits. ENUM BFD_RELOC_FR30_12_PCREL ENUMDOC This is a 16 bit reloc for the FR30 that stores a 12 bit pc relative short offset into 11 bits. ENUM BFD_RELOC_MCORE_PCREL_IMM8BY4 ENUMX BFD_RELOC_MCORE_PCREL_IMM11BY2 ENUMX BFD_RELOC_MCORE_PCREL_IMM4BY2 ENUMX BFD_RELOC_MCORE_PCREL_32 ENUMX BFD_RELOC_MCORE_PCREL_JSR_IMM11BY2 ENUMX BFD_RELOC_MCORE_RVA ENUMDOC Motorola Mcore relocations. ENUM BFD_RELOC_MEP_8 ENUMX BFD_RELOC_MEP_16 ENUMX BFD_RELOC_MEP_32 ENUMX BFD_RELOC_MEP_PCREL8A2 ENUMX BFD_RELOC_MEP_PCREL12A2 ENUMX BFD_RELOC_MEP_PCREL17A2 ENUMX BFD_RELOC_MEP_PCREL24A2 ENUMX BFD_RELOC_MEP_PCABS24A2 ENUMX BFD_RELOC_MEP_LOW16 ENUMX BFD_RELOC_MEP_HI16U ENUMX BFD_RELOC_MEP_HI16S ENUMX BFD_RELOC_MEP_GPREL ENUMX BFD_RELOC_MEP_TPREL ENUMX BFD_RELOC_MEP_TPREL7 ENUMX BFD_RELOC_MEP_TPREL7A2 ENUMX BFD_RELOC_MEP_TPREL7A4 ENUMX BFD_RELOC_MEP_UIMM24 ENUMX BFD_RELOC_MEP_ADDR24A4 ENUMX BFD_RELOC_MEP_GNU_VTINHERIT ENUMX BFD_RELOC_MEP_GNU_VTENTRY ENUMDOC Toshiba Media Processor Relocations. COMMENT ENUM BFD_RELOC_MMIX_GETA ENUMX BFD_RELOC_MMIX_GETA_1 ENUMX BFD_RELOC_MMIX_GETA_2 ENUMX BFD_RELOC_MMIX_GETA_3 ENUMDOC These are relocations for the GETA instruction. ENUM BFD_RELOC_MMIX_CBRANCH ENUMX BFD_RELOC_MMIX_CBRANCH_J ENUMX BFD_RELOC_MMIX_CBRANCH_1 ENUMX BFD_RELOC_MMIX_CBRANCH_2 ENUMX BFD_RELOC_MMIX_CBRANCH_3 ENUMDOC These are relocations for a conditional branch instruction. ENUM BFD_RELOC_MMIX_PUSHJ ENUMX BFD_RELOC_MMIX_PUSHJ_1 ENUMX BFD_RELOC_MMIX_PUSHJ_2 ENUMX BFD_RELOC_MMIX_PUSHJ_3 ENUMX BFD_RELOC_MMIX_PUSHJ_STUBBABLE ENUMDOC These are relocations for the PUSHJ instruction. ENUM BFD_RELOC_MMIX_JMP ENUMX BFD_RELOC_MMIX_JMP_1 ENUMX BFD_RELOC_MMIX_JMP_2 ENUMX BFD_RELOC_MMIX_JMP_3 ENUMDOC These are relocations for the JMP instruction. ENUM BFD_RELOC_MMIX_ADDR19 ENUMDOC This is a relocation for a relative address as in a GETA instruction or a branch. ENUM BFD_RELOC_MMIX_ADDR27 ENUMDOC This is a relocation for a relative address as in a JMP instruction. ENUM BFD_RELOC_MMIX_REG_OR_BYTE ENUMDOC This is a relocation for an instruction field that may be a general register or a value 0..255. ENUM BFD_RELOC_MMIX_REG ENUMDOC This is a relocation for an instruction field that may be a general register. ENUM BFD_RELOC_MMIX_BASE_PLUS_OFFSET ENUMDOC This is a relocation for two instruction fields holding a register and an offset, the equivalent of the relocation. ENUM BFD_RELOC_MMIX_LOCAL ENUMDOC This relocation is an assertion that the expression is not allocated as a global register. It does not modify contents. ENUM BFD_RELOC_AVR_7_PCREL ENUMDOC This is a 16 bit reloc for the AVR that stores 8 bit pc relative short offset into 7 bits. ENUM BFD_RELOC_AVR_13_PCREL ENUMDOC This is a 16 bit reloc for the AVR that stores 13 bit pc relative short offset into 12 bits. ENUM BFD_RELOC_AVR_16_PM ENUMDOC This is a 16 bit reloc for the AVR that stores 17 bit value (usually program memory address) into 16 bits. ENUM BFD_RELOC_AVR_LO8_LDI ENUMDOC This is a 16 bit reloc for the AVR that stores 8 bit value (usually data memory address) into 8 bit immediate value of LDI insn. ENUM BFD_RELOC_AVR_HI8_LDI ENUMDOC This is a 16 bit reloc for the AVR that stores 8 bit value (high 8 bit of data memory address) into 8 bit immediate value of LDI insn. ENUM BFD_RELOC_AVR_HH8_LDI ENUMDOC This is a 16 bit reloc for the AVR that stores 8 bit value (most high 8 bit of program memory address) into 8 bit immediate value of LDI insn. ENUM BFD_RELOC_AVR_MS8_LDI ENUMDOC This is a 16 bit reloc for the AVR that stores 8 bit value (most high 8 bit of 32 bit value) into 8 bit immediate value of LDI insn. ENUM BFD_RELOC_AVR_LO8_LDI_NEG ENUMDOC This is a 16 bit reloc for the AVR that stores negated 8 bit value (usually data memory address) into 8 bit immediate value of SUBI insn. ENUM BFD_RELOC_AVR_HI8_LDI_NEG ENUMDOC This is a 16 bit reloc for the AVR that stores negated 8 bit value (high 8 bit of data memory address) into 8 bit immediate value of SUBI insn. ENUM BFD_RELOC_AVR_HH8_LDI_NEG ENUMDOC This is a 16 bit reloc for the AVR that stores negated 8 bit value (most high 8 bit of program memory address) into 8 bit immediate value of LDI or SUBI insn. ENUM BFD_RELOC_AVR_MS8_LDI_NEG ENUMDOC This is a 16 bit reloc for the AVR that stores negated 8 bit value (msb of 32 bit value) into 8 bit immediate value of LDI insn. ENUM BFD_RELOC_AVR_LO8_LDI_PM ENUMDOC This is a 16 bit reloc for the AVR that stores 8 bit value (usually command address) into 8 bit immediate value of LDI insn. ENUM BFD_RELOC_AVR_LO8_LDI_GS ENUMDOC This is a 16 bit reloc for the AVR that stores 8 bit value (command address) into 8 bit immediate value of LDI insn. If the address is beyond the 128k boundary, the linker inserts a jump stub for this reloc in the lower 128k. ENUM BFD_RELOC_AVR_HI8_LDI_PM ENUMDOC This is a 16 bit reloc for the AVR that stores 8 bit value (high 8 bit of command address) into 8 bit immediate value of LDI insn. ENUM BFD_RELOC_AVR_HI8_LDI_GS ENUMDOC This is a 16 bit reloc for the AVR that stores 8 bit value (high 8 bit of command address) into 8 bit immediate value of LDI insn. If the address is beyond the 128k boundary, the linker inserts a jump stub for this reloc below 128k. ENUM BFD_RELOC_AVR_HH8_LDI_PM ENUMDOC This is a 16 bit reloc for the AVR that stores 8 bit value (most high 8 bit of command address) into 8 bit immediate value of LDI insn. ENUM BFD_RELOC_AVR_LO8_LDI_PM_NEG ENUMDOC This is a 16 bit reloc for the AVR that stores negated 8 bit value (usually command address) into 8 bit immediate value of SUBI insn. ENUM BFD_RELOC_AVR_HI8_LDI_PM_NEG ENUMDOC This is a 16 bit reloc for the AVR that stores negated 8 bit value (high 8 bit of 16 bit command address) into 8 bit immediate value of SUBI insn. ENUM BFD_RELOC_AVR_HH8_LDI_PM_NEG ENUMDOC This is a 16 bit reloc for the AVR that stores negated 8 bit value (high 6 bit of 22 bit command address) into 8 bit immediate value of SUBI insn. ENUM BFD_RELOC_AVR_CALL ENUMDOC This is a 32 bit reloc for the AVR that stores 23 bit value into 22 bits. ENUM BFD_RELOC_AVR_LDI ENUMDOC This is a 16 bit reloc for the AVR that stores all needed bits for absolute addressing with ldi with overflow check to linktime ENUM BFD_RELOC_AVR_6 ENUMDOC This is a 6 bit reloc for the AVR that stores offset for ldd/std instructions ENUM BFD_RELOC_AVR_6_ADIW ENUMDOC This is a 6 bit reloc for the AVR that stores offset for adiw/sbiw instructions ENUM BFD_RELOC_390_12 ENUMDOC Direct 12 bit. ENUM BFD_RELOC_390_GOT12 ENUMDOC 12 bit GOT offset. ENUM BFD_RELOC_390_PLT32 ENUMDOC 32 bit PC relative PLT address. ENUM BFD_RELOC_390_COPY ENUMDOC Copy symbol at runtime. ENUM BFD_RELOC_390_GLOB_DAT ENUMDOC Create GOT entry. ENUM BFD_RELOC_390_JMP_SLOT ENUMDOC Create PLT entry. ENUM BFD_RELOC_390_RELATIVE ENUMDOC Adjust by program base. ENUM BFD_RELOC_390_GOTPC ENUMDOC 32 bit PC relative offset to GOT. ENUM BFD_RELOC_390_GOT16 ENUMDOC 16 bit GOT offset. ENUM BFD_RELOC_390_PC16DBL ENUMDOC PC relative 16 bit shifted by 1. ENUM BFD_RELOC_390_PLT16DBL ENUMDOC 16 bit PC rel. PLT shifted by 1. ENUM BFD_RELOC_390_PC32DBL ENUMDOC PC relative 32 bit shifted by 1. ENUM BFD_RELOC_390_PLT32DBL ENUMDOC 32 bit PC rel. PLT shifted by 1. ENUM BFD_RELOC_390_GOTPCDBL ENUMDOC 32 bit PC rel. GOT shifted by 1. ENUM BFD_RELOC_390_GOT64 ENUMDOC 64 bit GOT offset. ENUM BFD_RELOC_390_PLT64 ENUMDOC 64 bit PC relative PLT address. ENUM BFD_RELOC_390_GOTENT ENUMDOC 32 bit rel. offset to GOT entry. ENUM BFD_RELOC_390_GOTOFF64 ENUMDOC 64 bit offset to GOT. ENUM BFD_RELOC_390_GOTPLT12 ENUMDOC 12-bit offset to symbol-entry within GOT, with PLT handling. ENUM BFD_RELOC_390_GOTPLT16 ENUMDOC 16-bit offset to symbol-entry within GOT, with PLT handling. ENUM BFD_RELOC_390_GOTPLT32 ENUMDOC 32-bit offset to symbol-entry within GOT, with PLT handling. ENUM BFD_RELOC_390_GOTPLT64 ENUMDOC 64-bit offset to symbol-entry within GOT, with PLT handling. ENUM BFD_RELOC_390_GOTPLTENT ENUMDOC 32-bit rel. offset to symbol-entry within GOT, with PLT handling. ENUM BFD_RELOC_390_PLTOFF16 ENUMDOC 16-bit rel. offset from the GOT to a PLT entry. ENUM BFD_RELOC_390_PLTOFF32 ENUMDOC 32-bit rel. offset from the GOT to a PLT entry. ENUM BFD_RELOC_390_PLTOFF64 ENUMDOC 64-bit rel. offset from the GOT to a PLT entry. ENUM BFD_RELOC_390_TLS_LOAD ENUMX BFD_RELOC_390_TLS_GDCALL ENUMX BFD_RELOC_390_TLS_LDCALL ENUMX BFD_RELOC_390_TLS_GD32 ENUMX BFD_RELOC_390_TLS_GD64 ENUMX BFD_RELOC_390_TLS_GOTIE12 ENUMX BFD_RELOC_390_TLS_GOTIE32 ENUMX BFD_RELOC_390_TLS_GOTIE64 ENUMX BFD_RELOC_390_TLS_LDM32 ENUMX BFD_RELOC_390_TLS_LDM64 ENUMX BFD_RELOC_390_TLS_IE32 ENUMX BFD_RELOC_390_TLS_IE64 ENUMX BFD_RELOC_390_TLS_IEENT ENUMX BFD_RELOC_390_TLS_LE32 ENUMX BFD_RELOC_390_TLS_LE64 ENUMX BFD_RELOC_390_TLS_LDO32 ENUMX BFD_RELOC_390_TLS_LDO64 ENUMX BFD_RELOC_390_TLS_DTPMOD ENUMX BFD_RELOC_390_TLS_DTPOFF ENUMX BFD_RELOC_390_TLS_TPOFF ENUMDOC s390 tls relocations. ENUM BFD_RELOC_390_20 ENUMX BFD_RELOC_390_GOT20 ENUMX BFD_RELOC_390_GOTPLT20 ENUMX BFD_RELOC_390_TLS_GOTIE20 ENUMDOC Long displacement extension. ENUM BFD_RELOC_SCORE_DUMMY1 ENUMDOC Score relocations ENUM BFD_RELOC_SCORE_GPREL15 ENUMDOC Low 16 bit for load/store ENUM BFD_RELOC_SCORE_DUMMY2 ENUMX BFD_RELOC_SCORE_JMP ENUMDOC This is a 24-bit reloc with the right 1 bit assumed to be 0 ENUM BFD_RELOC_SCORE_BRANCH ENUMDOC This is a 19-bit reloc with the right 1 bit assumed to be 0 ENUM BFD_RELOC_SCORE16_JMP ENUMDOC This is a 11-bit reloc with the right 1 bit assumed to be 0 ENUM BFD_RELOC_SCORE16_BRANCH ENUMDOC This is a 8-bit reloc with the right 1 bit assumed to be 0 ENUM BFD_RELOC_SCORE_GOT15 ENUMX BFD_RELOC_SCORE_GOT_LO16 ENUMX BFD_RELOC_SCORE_CALL15 ENUMX BFD_RELOC_SCORE_DUMMY_HI16 ENUMDOC Undocumented Score relocs ENUM BFD_RELOC_IP2K_FR9 ENUMDOC Scenix IP2K - 9-bit register number / data address ENUM BFD_RELOC_IP2K_BANK ENUMDOC Scenix IP2K - 4-bit register/data bank number ENUM BFD_RELOC_IP2K_ADDR16CJP ENUMDOC Scenix IP2K - low 13 bits of instruction word address ENUM BFD_RELOC_IP2K_PAGE3 ENUMDOC Scenix IP2K - high 3 bits of instruction word address ENUM BFD_RELOC_IP2K_LO8DATA ENUMX BFD_RELOC_IP2K_HI8DATA ENUMX BFD_RELOC_IP2K_EX8DATA ENUMDOC Scenix IP2K - ext/low/high 8 bits of data address ENUM BFD_RELOC_IP2K_LO8INSN ENUMX BFD_RELOC_IP2K_HI8INSN ENUMDOC Scenix IP2K - low/high 8 bits of instruction word address ENUM BFD_RELOC_IP2K_PC_SKIP ENUMDOC Scenix IP2K - even/odd PC modifier to modify snb pcl.0 ENUM BFD_RELOC_IP2K_TEXT ENUMDOC Scenix IP2K - 16 bit word address in text section. ENUM BFD_RELOC_IP2K_FR_OFFSET ENUMDOC Scenix IP2K - 7-bit sp or dp offset ENUM BFD_RELOC_VPE4KMATH_DATA ENUMX BFD_RELOC_VPE4KMATH_INSN ENUMDOC Scenix VPE4K coprocessor - data/insn-space addressing ENUM BFD_RELOC_VTABLE_INHERIT ENUMX BFD_RELOC_VTABLE_ENTRY ENUMDOC These two relocations are used by the linker to determine which of the entries in a C++ virtual function table are actually used. When the --gc-sections option is given, the linker will zero out the entries that are not used, so that the code for those functions need not be included in the output. VTABLE_INHERIT is a zero-space relocation used to describe to the linker the inheritance tree of a C++ virtual function table. The relocation's symbol should be the parent class' vtable, and the relocation should be located at the child vtable. VTABLE_ENTRY is a zero-space relocation that describes the use of a virtual function table entry. The reloc's symbol should refer to the table of the class mentioned in the code. Off of that base, an offset describes the entry that is being used. For Rela hosts, this offset is stored in the reloc's addend. For Rel hosts, we are forced to put this offset in the reloc's section offset. ENUM BFD_RELOC_IA64_IMM14 ENUMX BFD_RELOC_IA64_IMM22 ENUMX BFD_RELOC_IA64_IMM64 ENUMX BFD_RELOC_IA64_DIR32MSB ENUMX BFD_RELOC_IA64_DIR32LSB ENUMX BFD_RELOC_IA64_DIR64MSB ENUMX BFD_RELOC_IA64_DIR64LSB ENUMX BFD_RELOC_IA64_GPREL22 ENUMX BFD_RELOC_IA64_GPREL64I ENUMX BFD_RELOC_IA64_GPREL32MSB ENUMX BFD_RELOC_IA64_GPREL32LSB ENUMX BFD_RELOC_IA64_GPREL64MSB ENUMX BFD_RELOC_IA64_GPREL64LSB ENUMX BFD_RELOC_IA64_LTOFF22 ENUMX BFD_RELOC_IA64_LTOFF64I ENUMX BFD_RELOC_IA64_PLTOFF22 ENUMX BFD_RELOC_IA64_PLTOFF64I ENUMX BFD_RELOC_IA64_PLTOFF64MSB ENUMX BFD_RELOC_IA64_PLTOFF64LSB ENUMX BFD_RELOC_IA64_FPTR64I ENUMX BFD_RELOC_IA64_FPTR32MSB ENUMX BFD_RELOC_IA64_FPTR32LSB ENUMX BFD_RELOC_IA64_FPTR64MSB ENUMX BFD_RELOC_IA64_FPTR64LSB ENUMX BFD_RELOC_IA64_PCREL21B ENUMX BFD_RELOC_IA64_PCREL21BI ENUMX BFD_RELOC_IA64_PCREL21M ENUMX BFD_RELOC_IA64_PCREL21F ENUMX BFD_RELOC_IA64_PCREL22 ENUMX BFD_RELOC_IA64_PCREL60B ENUMX BFD_RELOC_IA64_PCREL64I ENUMX BFD_RELOC_IA64_PCREL32MSB ENUMX BFD_RELOC_IA64_PCREL32LSB ENUMX BFD_RELOC_IA64_PCREL64MSB ENUMX BFD_RELOC_IA64_PCREL64LSB ENUMX BFD_RELOC_IA64_LTOFF_FPTR22 ENUMX BFD_RELOC_IA64_LTOFF_FPTR64I ENUMX BFD_RELOC_IA64_LTOFF_FPTR32MSB ENUMX BFD_RELOC_IA64_LTOFF_FPTR32LSB ENUMX BFD_RELOC_IA64_LTOFF_FPTR64MSB ENUMX BFD_RELOC_IA64_LTOFF_FPTR64LSB ENUMX BFD_RELOC_IA64_SEGREL32MSB ENUMX BFD_RELOC_IA64_SEGREL32LSB ENUMX BFD_RELOC_IA64_SEGREL64MSB ENUMX BFD_RELOC_IA64_SEGREL64LSB ENUMX BFD_RELOC_IA64_SECREL32MSB ENUMX BFD_RELOC_IA64_SECREL32LSB ENUMX BFD_RELOC_IA64_SECREL64MSB ENUMX BFD_RELOC_IA64_SECREL64LSB ENUMX BFD_RELOC_IA64_REL32MSB ENUMX BFD_RELOC_IA64_REL32LSB ENUMX BFD_RELOC_IA64_REL64MSB ENUMX BFD_RELOC_IA64_REL64LSB ENUMX BFD_RELOC_IA64_LTV32MSB ENUMX BFD_RELOC_IA64_LTV32LSB ENUMX BFD_RELOC_IA64_LTV64MSB ENUMX BFD_RELOC_IA64_LTV64LSB ENUMX BFD_RELOC_IA64_IPLTMSB ENUMX BFD_RELOC_IA64_IPLTLSB ENUMX BFD_RELOC_IA64_COPY ENUMX BFD_RELOC_IA64_LTOFF22X ENUMX BFD_RELOC_IA64_LDXMOV ENUMX BFD_RELOC_IA64_TPREL14 ENUMX BFD_RELOC_IA64_TPREL22 ENUMX BFD_RELOC_IA64_TPREL64I ENUMX BFD_RELOC_IA64_TPREL64MSB ENUMX BFD_RELOC_IA64_TPREL64LSB ENUMX BFD_RELOC_IA64_LTOFF_TPREL22 ENUMX BFD_RELOC_IA64_DTPMOD64MSB ENUMX BFD_RELOC_IA64_DTPMOD64LSB ENUMX BFD_RELOC_IA64_LTOFF_DTPMOD22 ENUMX BFD_RELOC_IA64_DTPREL14 ENUMX BFD_RELOC_IA64_DTPREL22 ENUMX BFD_RELOC_IA64_DTPREL64I ENUMX BFD_RELOC_IA64_DTPREL32MSB ENUMX BFD_RELOC_IA64_DTPREL32LSB ENUMX BFD_RELOC_IA64_DTPREL64MSB ENUMX BFD_RELOC_IA64_DTPREL64LSB ENUMX BFD_RELOC_IA64_LTOFF_DTPREL22 ENUMDOC Intel IA64 Relocations. ENUM BFD_RELOC_M68HC11_HI8 ENUMDOC Motorola 68HC11 reloc. This is the 8 bit high part of an absolute address. ENUM BFD_RELOC_M68HC11_LO8 ENUMDOC Motorola 68HC11 reloc. This is the 8 bit low part of an absolute address. ENUM BFD_RELOC_M68HC11_3B ENUMDOC Motorola 68HC11 reloc. This is the 3 bit of a value. ENUM BFD_RELOC_M68HC11_RL_JUMP ENUMDOC Motorola 68HC11 reloc. This reloc marks the beginning of a jump/call instruction. It is used for linker relaxation to correctly identify beginning of instruction and change some branches to use PC-relative addressing mode. ENUM BFD_RELOC_M68HC11_RL_GROUP ENUMDOC Motorola 68HC11 reloc. This reloc marks a group of several instructions that gcc generates and for which the linker relaxation pass can modify and/or remove some of them. ENUM BFD_RELOC_M68HC11_LO16 ENUMDOC Motorola 68HC11 reloc. This is the 16-bit lower part of an address. It is used for 'call' instruction to specify the symbol address without any special transformation (due to memory bank window). ENUM BFD_RELOC_M68HC11_PAGE ENUMDOC Motorola 68HC11 reloc. This is a 8-bit reloc that specifies the page number of an address. It is used by 'call' instruction to specify the page number of the symbol. ENUM BFD_RELOC_M68HC11_24 ENUMDOC Motorola 68HC11 reloc. This is a 24-bit reloc that represents the address with a 16-bit value and a 8-bit page number. The symbol address is transformed to follow the 16K memory bank of 68HC12 (seen as mapped in the window). ENUM BFD_RELOC_M68HC12_5B ENUMDOC Motorola 68HC12 reloc. This is the 5 bits of a value. ENUM BFD_RELOC_16C_NUM08 ENUMX BFD_RELOC_16C_NUM08_C ENUMX BFD_RELOC_16C_NUM16 ENUMX BFD_RELOC_16C_NUM16_C ENUMX BFD_RELOC_16C_NUM32 ENUMX BFD_RELOC_16C_NUM32_C ENUMX BFD_RELOC_16C_DISP04 ENUMX BFD_RELOC_16C_DISP04_C ENUMX BFD_RELOC_16C_DISP08 ENUMX BFD_RELOC_16C_DISP08_C ENUMX BFD_RELOC_16C_DISP16 ENUMX BFD_RELOC_16C_DISP16_C ENUMX BFD_RELOC_16C_DISP24 ENUMX BFD_RELOC_16C_DISP24_C ENUMX BFD_RELOC_16C_DISP24a ENUMX BFD_RELOC_16C_DISP24a_C ENUMX BFD_RELOC_16C_REG04 ENUMX BFD_RELOC_16C_REG04_C ENUMX BFD_RELOC_16C_REG04a ENUMX BFD_RELOC_16C_REG04a_C ENUMX BFD_RELOC_16C_REG14 ENUMX BFD_RELOC_16C_REG14_C ENUMX BFD_RELOC_16C_REG16 ENUMX BFD_RELOC_16C_REG16_C ENUMX BFD_RELOC_16C_REG20 ENUMX BFD_RELOC_16C_REG20_C ENUMX BFD_RELOC_16C_ABS20 ENUMX BFD_RELOC_16C_ABS20_C ENUMX BFD_RELOC_16C_ABS24 ENUMX BFD_RELOC_16C_ABS24_C ENUMX BFD_RELOC_16C_IMM04 ENUMX BFD_RELOC_16C_IMM04_C ENUMX BFD_RELOC_16C_IMM16 ENUMX BFD_RELOC_16C_IMM16_C ENUMX BFD_RELOC_16C_IMM20 ENUMX BFD_RELOC_16C_IMM20_C ENUMX BFD_RELOC_16C_IMM24 ENUMX BFD_RELOC_16C_IMM24_C ENUMX BFD_RELOC_16C_IMM32 ENUMX BFD_RELOC_16C_IMM32_C ENUMDOC NS CR16C Relocations. ENUM BFD_RELOC_CR16_NUM8 ENUMX BFD_RELOC_CR16_NUM16 ENUMX BFD_RELOC_CR16_NUM32 ENUMX BFD_RELOC_CR16_NUM32a ENUMX BFD_RELOC_CR16_REGREL0 ENUMX BFD_RELOC_CR16_REGREL4 ENUMX BFD_RELOC_CR16_REGREL4a ENUMX BFD_RELOC_CR16_REGREL14 ENUMX BFD_RELOC_CR16_REGREL14a ENUMX BFD_RELOC_CR16_REGREL16 ENUMX BFD_RELOC_CR16_REGREL20 ENUMX BFD_RELOC_CR16_REGREL20a ENUMX BFD_RELOC_CR16_ABS20 ENUMX BFD_RELOC_CR16_ABS24 ENUMX BFD_RELOC_CR16_IMM4 ENUMX BFD_RELOC_CR16_IMM8 ENUMX BFD_RELOC_CR16_IMM16 ENUMX BFD_RELOC_CR16_IMM20 ENUMX BFD_RELOC_CR16_IMM24 ENUMX BFD_RELOC_CR16_IMM32 ENUMX BFD_RELOC_CR16_IMM32a ENUMX BFD_RELOC_CR16_DISP4 ENUMX BFD_RELOC_CR16_DISP8 ENUMX BFD_RELOC_CR16_DISP16 ENUMX BFD_RELOC_CR16_DISP20 ENUMX BFD_RELOC_CR16_DISP24 ENUMX BFD_RELOC_CR16_DISP24a ENUMDOC NS CR16 Relocations. ENUM BFD_RELOC_CRX_REL4 ENUMX BFD_RELOC_CRX_REL8 ENUMX BFD_RELOC_CRX_REL8_CMP ENUMX BFD_RELOC_CRX_REL16 ENUMX BFD_RELOC_CRX_REL24 ENUMX BFD_RELOC_CRX_REL32 ENUMX BFD_RELOC_CRX_REGREL12 ENUMX BFD_RELOC_CRX_REGREL22 ENUMX BFD_RELOC_CRX_REGREL28 ENUMX BFD_RELOC_CRX_REGREL32 ENUMX BFD_RELOC_CRX_ABS16 ENUMX BFD_RELOC_CRX_ABS32 ENUMX BFD_RELOC_CRX_NUM8 ENUMX BFD_RELOC_CRX_NUM16 ENUMX BFD_RELOC_CRX_NUM32 ENUMX BFD_RELOC_CRX_IMM16 ENUMX BFD_RELOC_CRX_IMM32 ENUMX BFD_RELOC_CRX_SWITCH8 ENUMX BFD_RELOC_CRX_SWITCH16 ENUMX BFD_RELOC_CRX_SWITCH32 ENUMDOC NS CRX Relocations. ENUM BFD_RELOC_CRIS_BDISP8 ENUMX BFD_RELOC_CRIS_UNSIGNED_5 ENUMX BFD_RELOC_CRIS_SIGNED_6 ENUMX BFD_RELOC_CRIS_UNSIGNED_6 ENUMX BFD_RELOC_CRIS_SIGNED_8 ENUMX BFD_RELOC_CRIS_UNSIGNED_8 ENUMX BFD_RELOC_CRIS_SIGNED_16 ENUMX BFD_RELOC_CRIS_UNSIGNED_16 ENUMX BFD_RELOC_CRIS_LAPCQ_OFFSET ENUMX BFD_RELOC_CRIS_UNSIGNED_4 ENUMDOC These relocs are only used within the CRIS assembler. They are not (at present) written to any object files. ENUM BFD_RELOC_CRIS_COPY ENUMX BFD_RELOC_CRIS_GLOB_DAT ENUMX BFD_RELOC_CRIS_JUMP_SLOT ENUMX BFD_RELOC_CRIS_RELATIVE ENUMDOC Relocs used in ELF shared libraries for CRIS. ENUM BFD_RELOC_CRIS_32_GOT ENUMDOC 32-bit offset to symbol-entry within GOT. ENUM BFD_RELOC_CRIS_16_GOT ENUMDOC 16-bit offset to symbol-entry within GOT. ENUM BFD_RELOC_CRIS_32_GOTPLT ENUMDOC 32-bit offset to symbol-entry within GOT, with PLT handling. ENUM BFD_RELOC_CRIS_16_GOTPLT ENUMDOC 16-bit offset to symbol-entry within GOT, with PLT handling. ENUM BFD_RELOC_CRIS_32_GOTREL ENUMDOC 32-bit offset to symbol, relative to GOT. ENUM BFD_RELOC_CRIS_32_PLT_GOTREL ENUMDOC 32-bit offset to symbol with PLT entry, relative to GOT. ENUM BFD_RELOC_CRIS_32_PLT_PCREL ENUMDOC 32-bit offset to symbol with PLT entry, relative to this relocation. ENUM BFD_RELOC_860_COPY ENUMX BFD_RELOC_860_GLOB_DAT ENUMX BFD_RELOC_860_JUMP_SLOT ENUMX BFD_RELOC_860_RELATIVE ENUMX BFD_RELOC_860_PC26 ENUMX BFD_RELOC_860_PLT26 ENUMX BFD_RELOC_860_PC16 ENUMX BFD_RELOC_860_LOW0 ENUMX BFD_RELOC_860_SPLIT0 ENUMX BFD_RELOC_860_LOW1 ENUMX BFD_RELOC_860_SPLIT1 ENUMX BFD_RELOC_860_LOW2 ENUMX BFD_RELOC_860_SPLIT2 ENUMX BFD_RELOC_860_LOW3 ENUMX BFD_RELOC_860_LOGOT0 ENUMX BFD_RELOC_860_SPGOT0 ENUMX BFD_RELOC_860_LOGOT1 ENUMX BFD_RELOC_860_SPGOT1 ENUMX BFD_RELOC_860_LOGOTOFF0 ENUMX BFD_RELOC_860_SPGOTOFF0 ENUMX BFD_RELOC_860_LOGOTOFF1 ENUMX BFD_RELOC_860_SPGOTOFF1 ENUMX BFD_RELOC_860_LOGOTOFF2 ENUMX BFD_RELOC_860_LOGOTOFF3 ENUMX BFD_RELOC_860_LOPC ENUMX BFD_RELOC_860_HIGHADJ ENUMX BFD_RELOC_860_HAGOT ENUMX BFD_RELOC_860_HAGOTOFF ENUMX BFD_RELOC_860_HAPC ENUMX BFD_RELOC_860_HIGH ENUMX BFD_RELOC_860_HIGOT ENUMX BFD_RELOC_860_HIGOTOFF ENUMDOC Intel i860 Relocations. ENUM BFD_RELOC_OPENRISC_ABS_26 ENUMX BFD_RELOC_OPENRISC_REL_26 ENUMDOC OpenRISC Relocations. ENUM BFD_RELOC_H8_DIR16A8 ENUMX BFD_RELOC_H8_DIR16R8 ENUMX BFD_RELOC_H8_DIR24A8 ENUMX BFD_RELOC_H8_DIR24R8 ENUMX BFD_RELOC_H8_DIR32A16 ENUMDOC H8 elf Relocations. ENUM BFD_RELOC_XSTORMY16_REL_12 ENUMX BFD_RELOC_XSTORMY16_12 ENUMX BFD_RELOC_XSTORMY16_24 ENUMX BFD_RELOC_XSTORMY16_FPTR16 ENUMDOC Sony Xstormy16 Relocations. ENUM BFD_RELOC_RELC ENUMDOC Self-describing complex relocations. COMMENT ENUM BFD_RELOC_XC16X_PAG ENUMX BFD_RELOC_XC16X_POF ENUMX BFD_RELOC_XC16X_SEG ENUMX BFD_RELOC_XC16X_SOF ENUMDOC Infineon Relocations. ENUM BFD_RELOC_VAX_GLOB_DAT ENUMX BFD_RELOC_VAX_JMP_SLOT ENUMX BFD_RELOC_VAX_RELATIVE ENUMDOC Relocations used by VAX ELF. ENUM BFD_RELOC_MT_PC16 ENUMDOC Morpho MT - 16 bit immediate relocation. ENUM BFD_RELOC_MT_HI16 ENUMDOC Morpho MT - Hi 16 bits of an address. ENUM BFD_RELOC_MT_LO16 ENUMDOC Morpho MT - Low 16 bits of an address. ENUM BFD_RELOC_MT_GNU_VTINHERIT ENUMDOC Morpho MT - Used to tell the linker which vtable entries are used. ENUM BFD_RELOC_MT_GNU_VTENTRY ENUMDOC Morpho MT - Used to tell the linker which vtable entries are used. ENUM BFD_RELOC_MT_PCINSN8 ENUMDOC Morpho MT - 8 bit immediate relocation. ENUM BFD_RELOC_MSP430_10_PCREL ENUMX BFD_RELOC_MSP430_16_PCREL ENUMX BFD_RELOC_MSP430_16 ENUMX BFD_RELOC_MSP430_16_PCREL_BYTE ENUMX BFD_RELOC_MSP430_16_BYTE ENUMX BFD_RELOC_MSP430_2X_PCREL ENUMX BFD_RELOC_MSP430_RL_PCREL ENUMDOC msp430 specific relocation codes ENUM BFD_RELOC_IQ2000_OFFSET_16 ENUMX BFD_RELOC_IQ2000_OFFSET_21 ENUMX BFD_RELOC_IQ2000_UHI16 ENUMDOC IQ2000 Relocations. ENUM BFD_RELOC_XTENSA_RTLD ENUMDOC Special Xtensa relocation used only by PLT entries in ELF shared objects to indicate that the runtime linker should set the value to one of its own internal functions or data structures. ENUM BFD_RELOC_XTENSA_GLOB_DAT ENUMX BFD_RELOC_XTENSA_JMP_SLOT ENUMX BFD_RELOC_XTENSA_RELATIVE ENUMDOC Xtensa relocations for ELF shared objects. ENUM BFD_RELOC_XTENSA_PLT ENUMDOC Xtensa relocation used in ELF object files for symbols that may require PLT entries. Otherwise, this is just a generic 32-bit relocation. ENUM BFD_RELOC_XTENSA_DIFF8 ENUMX BFD_RELOC_XTENSA_DIFF16 ENUMX BFD_RELOC_XTENSA_DIFF32 ENUMDOC Xtensa relocations to mark the difference of two local symbols. These are only needed to support linker relaxation and can be ignored when not relaxing. The field is set to the value of the difference assuming no relaxation. The relocation encodes the position of the first symbol so the linker can determine whether to adjust the field value. ENUM BFD_RELOC_XTENSA_SLOT0_OP ENUMX BFD_RELOC_XTENSA_SLOT1_OP ENUMX BFD_RELOC_XTENSA_SLOT2_OP ENUMX BFD_RELOC_XTENSA_SLOT3_OP ENUMX BFD_RELOC_XTENSA_SLOT4_OP ENUMX BFD_RELOC_XTENSA_SLOT5_OP ENUMX BFD_RELOC_XTENSA_SLOT6_OP ENUMX BFD_RELOC_XTENSA_SLOT7_OP ENUMX BFD_RELOC_XTENSA_SLOT8_OP ENUMX BFD_RELOC_XTENSA_SLOT9_OP ENUMX BFD_RELOC_XTENSA_SLOT10_OP ENUMX BFD_RELOC_XTENSA_SLOT11_OP ENUMX BFD_RELOC_XTENSA_SLOT12_OP ENUMX BFD_RELOC_XTENSA_SLOT13_OP ENUMX BFD_RELOC_XTENSA_SLOT14_OP ENUMDOC Generic Xtensa relocations for instruction operands. Only the slot number is encoded in the relocation. The relocation applies to the last PC-relative immediate operand, or if there are no PC-relative immediates, to the last immediate operand. ENUM BFD_RELOC_XTENSA_SLOT0_ALT ENUMX BFD_RELOC_XTENSA_SLOT1_ALT ENUMX BFD_RELOC_XTENSA_SLOT2_ALT ENUMX BFD_RELOC_XTENSA_SLOT3_ALT ENUMX BFD_RELOC_XTENSA_SLOT4_ALT ENUMX BFD_RELOC_XTENSA_SLOT5_ALT ENUMX BFD_RELOC_XTENSA_SLOT6_ALT ENUMX BFD_RELOC_XTENSA_SLOT7_ALT ENUMX BFD_RELOC_XTENSA_SLOT8_ALT ENUMX BFD_RELOC_XTENSA_SLOT9_ALT ENUMX BFD_RELOC_XTENSA_SLOT10_ALT ENUMX BFD_RELOC_XTENSA_SLOT11_ALT ENUMX BFD_RELOC_XTENSA_SLOT12_ALT ENUMX BFD_RELOC_XTENSA_SLOT13_ALT ENUMX BFD_RELOC_XTENSA_SLOT14_ALT ENUMDOC Alternate Xtensa relocations. Only the slot is encoded in the relocation. The meaning of these relocations is opcode-specific. ENUM BFD_RELOC_XTENSA_OP0 ENUMX BFD_RELOC_XTENSA_OP1 ENUMX BFD_RELOC_XTENSA_OP2 ENUMDOC Xtensa relocations for backward compatibility. These have all been replaced by BFD_RELOC_XTENSA_SLOT0_OP. ENUM BFD_RELOC_XTENSA_ASM_EXPAND ENUMDOC Xtensa relocation to mark that the assembler expanded the instructions from an original target. The expansion size is encoded in the reloc size. ENUM BFD_RELOC_XTENSA_ASM_SIMPLIFY ENUMDOC Xtensa relocation to mark that the linker should simplify assembler-expanded instructions. This is commonly used internally by the linker after analysis of a BFD_RELOC_XTENSA_ASM_EXPAND. ENUM BFD_RELOC_Z80_DISP8 ENUMDOC 8 bit signed offset in (ix+d) or (iy+d). ENUM BFD_RELOC_Z8K_DISP7 ENUMDOC DJNZ offset. ENUM BFD_RELOC_Z8K_CALLR ENUMDOC CALR offset. ENUM BFD_RELOC_Z8K_IMM4L ENUMDOC 4 bit value. ENDSENUM BFD_RELOC_UNUSED CODE_FRAGMENT . .typedef enum bfd_reloc_code_real bfd_reloc_code_real_type; */ /* FUNCTION bfd_reloc_type_lookup bfd_reloc_name_lookup SYNOPSIS reloc_howto_type *bfd_reloc_type_lookup (bfd *abfd, bfd_reloc_code_real_type code); reloc_howto_type *bfd_reloc_name_lookup (bfd *abfd, const char *reloc_name); DESCRIPTION Return a pointer to a howto structure which, when invoked, will perform the relocation @var{code} on data from the architecture noted. */ reloc_howto_type * bfd_reloc_type_lookup (bfd *abfd, bfd_reloc_code_real_type code) { return BFD_SEND (abfd, reloc_type_lookup, (abfd, code)); } reloc_howto_type * bfd_reloc_name_lookup (bfd *abfd, const char *reloc_name) { return BFD_SEND (abfd, reloc_name_lookup, (abfd, reloc_name)); } static reloc_howto_type bfd_howto_32 = HOWTO (0, 00, 2, 32, FALSE, 0, complain_overflow_dont, 0, "VRT32", FALSE, 0xffffffff, 0xffffffff, TRUE); /* INTERNAL_FUNCTION bfd_default_reloc_type_lookup SYNOPSIS reloc_howto_type *bfd_default_reloc_type_lookup (bfd *abfd, bfd_reloc_code_real_type code); DESCRIPTION Provides a default relocation lookup routine for any architecture. */ reloc_howto_type * bfd_default_reloc_type_lookup (bfd *abfd, bfd_reloc_code_real_type code) { switch (code) { case BFD_RELOC_CTOR: /* The type of reloc used in a ctor, which will be as wide as the address - so either a 64, 32, or 16 bitter. */ switch (bfd_get_arch_info (abfd)->bits_per_address) { case 64: BFD_FAIL (); case 32: return &bfd_howto_32; case 16: BFD_FAIL (); default: BFD_FAIL (); } default: BFD_FAIL (); } return NULL; } /* FUNCTION bfd_get_reloc_code_name SYNOPSIS const char *bfd_get_reloc_code_name (bfd_reloc_code_real_type code); DESCRIPTION Provides a printable name for the supplied relocation code. Useful mainly for printing error messages. */ const char * bfd_get_reloc_code_name (bfd_reloc_code_real_type code) { if (code > BFD_RELOC_UNUSED) return 0; return bfd_reloc_code_real_names[code]; } /* INTERNAL_FUNCTION bfd_generic_relax_section SYNOPSIS bfd_boolean bfd_generic_relax_section (bfd *abfd, asection *section, struct bfd_link_info *, bfd_boolean *); DESCRIPTION Provides default handling for relaxing for back ends which don't do relaxing. */ bfd_boolean bfd_generic_relax_section (bfd *abfd ATTRIBUTE_UNUSED, asection *section ATTRIBUTE_UNUSED, struct bfd_link_info *link_info ATTRIBUTE_UNUSED, bfd_boolean *again) { *again = FALSE; return TRUE; } /* INTERNAL_FUNCTION bfd_generic_gc_sections SYNOPSIS bfd_boolean bfd_generic_gc_sections (bfd *, struct bfd_link_info *); DESCRIPTION Provides default handling for relaxing for back ends which don't do section gc -- i.e., does nothing. */ bfd_boolean bfd_generic_gc_sections (bfd *abfd ATTRIBUTE_UNUSED, struct bfd_link_info *info ATTRIBUTE_UNUSED) { return TRUE; } /* INTERNAL_FUNCTION bfd_generic_merge_sections SYNOPSIS bfd_boolean bfd_generic_merge_sections (bfd *, struct bfd_link_info *); DESCRIPTION Provides default handling for SEC_MERGE section merging for back ends which don't have SEC_MERGE support -- i.e., does nothing. */ bfd_boolean bfd_generic_merge_sections (bfd *abfd ATTRIBUTE_UNUSED, struct bfd_link_info *link_info ATTRIBUTE_UNUSED) { return TRUE; } /* INTERNAL_FUNCTION bfd_generic_get_relocated_section_contents SYNOPSIS bfd_byte *bfd_generic_get_relocated_section_contents (bfd *abfd, struct bfd_link_info *link_info, struct bfd_link_order *link_order, bfd_byte *data, bfd_boolean relocatable, asymbol **symbols); DESCRIPTION Provides default handling of relocation effort for back ends which can't be bothered to do it efficiently. */ bfd_byte * bfd_generic_get_relocated_section_contents (bfd *abfd, struct bfd_link_info *link_info, struct bfd_link_order *link_order, bfd_byte *data, bfd_boolean relocatable, asymbol **symbols) { /* Get enough memory to hold the stuff. */ bfd *input_bfd = link_order->u.indirect.section->owner; asection *input_section = link_order->u.indirect.section; long reloc_size = bfd_get_reloc_upper_bound (input_bfd, input_section); arelent **reloc_vector = NULL; long reloc_count; bfd_size_type sz; if (reloc_size < 0) goto error_return; reloc_vector = bfd_malloc (reloc_size); if (reloc_vector == NULL && reloc_size != 0) goto error_return; /* Read in the section. */ sz = input_section->rawsize ? input_section->rawsize : input_section->size; if (!bfd_get_section_contents (input_bfd, input_section, data, 0, sz)) goto error_return; reloc_count = bfd_canonicalize_reloc (input_bfd, input_section, reloc_vector, symbols); if (reloc_count < 0) goto error_return; if (reloc_count > 0) { arelent **parent; for (parent = reloc_vector; *parent != NULL; parent++) { char *error_message = NULL; asymbol *symbol; bfd_reloc_status_type r; symbol = *(*parent)->sym_ptr_ptr; if (symbol->section && elf_discarded_section (symbol->section)) { bfd_byte *p; static reloc_howto_type none_howto = HOWTO (0, 0, 0, 0, FALSE, 0, complain_overflow_dont, NULL, "unused", FALSE, 0, 0, FALSE); p = data + (*parent)->address * bfd_octets_per_byte (input_bfd); _bfd_clear_contents ((*parent)->howto, input_bfd, p); (*parent)->sym_ptr_ptr = bfd_abs_section.symbol_ptr_ptr; (*parent)->addend = 0; (*parent)->howto = &none_howto; r = bfd_reloc_ok; } else r = bfd_perform_relocation (input_bfd, *parent, data, input_section, relocatable ? abfd : NULL, &error_message); if (relocatable) { asection *os = input_section->output_section; /* A partial link, so keep the relocs. */ os->orelocation[os->reloc_count] = *parent; os->reloc_count++; } if (r != bfd_reloc_ok) { switch (r) { case bfd_reloc_undefined: if (!((*link_info->callbacks->undefined_symbol) (link_info, bfd_asymbol_name (*(*parent)->sym_ptr_ptr), input_bfd, input_section, (*parent)->address, TRUE))) goto error_return; break; case bfd_reloc_dangerous: BFD_ASSERT (error_message != NULL); if (!((*link_info->callbacks->reloc_dangerous) (link_info, error_message, input_bfd, input_section, (*parent)->address))) goto error_return; break; case bfd_reloc_overflow: if (!((*link_info->callbacks->reloc_overflow) (link_info, NULL, bfd_asymbol_name (*(*parent)->sym_ptr_ptr), (*parent)->howto->name, (*parent)->addend, input_bfd, input_section, (*parent)->address))) goto error_return; break; case bfd_reloc_outofrange: default: abort (); break; } } } } if (reloc_vector != NULL) free (reloc_vector); return data; error_return: if (reloc_vector != NULL) free (reloc_vector); return NULL; } Index: projects/clang350-import/contrib/binutils/bfd/section.c =================================================================== --- projects/clang350-import/contrib/binutils/bfd/section.c (revision 275748) +++ projects/clang350-import/contrib/binutils/bfd/section.c (revision 275749) @@ -1,1526 +1,1529 @@ /* Object file "section" support for the BFD library. Copyright 1990, 1991, 1992, 1993, 1994, 1995, 1996, 1997, 1998, 1999, - 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007 + 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009 Free Software Foundation, Inc. Written by Cygnus Support. This file is part of BFD, the Binary File Descriptor library. This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin Street - Fifth Floor, Boston, MA 02110-1301, USA. */ /* SECTION Sections The raw data contained within a BFD is maintained through the section abstraction. A single BFD may have any number of sections. It keeps hold of them by pointing to the first; each one points to the next in the list. Sections are supported in BFD in <>. @menu @* Section Input:: @* Section Output:: @* typedef asection:: @* section prototypes:: @end menu INODE Section Input, Section Output, Sections, Sections SUBSECTION Section input When a BFD is opened for reading, the section structures are created and attached to the BFD. Each section has a name which describes the section in the outside world---for example, <> would contain at least three sections, called <<.text>>, <<.data>> and <<.bss>>. Names need not be unique; for example a COFF file may have several sections named <<.data>>. Sometimes a BFD will contain more than the ``natural'' number of sections. A back end may attach other sections containing constructor data, or an application may add a section (using <>) to the sections attached to an already open BFD. For example, the linker creates an extra section <> for each input file's BFD to hold information about common storage. The raw data is not necessarily read in when the section descriptor is created. Some targets may leave the data in place until a <> call is made. Other back ends may read in all the data at once. For example, an S-record file has to be read once to determine the size of the data. An IEEE-695 file doesn't contain raw data in sections, but data and relocation expressions intermixed, so the data area has to be parsed to get out the data and relocations. INODE Section Output, typedef asection, Section Input, Sections SUBSECTION Section output To write a new object style BFD, the various sections to be written have to be created. They are attached to the BFD in the same way as input sections; data is written to the sections using <>. Any program that creates or combines sections (e.g., the assembler and linker) must use the <> fields <> and <> to indicate the file sections to which each section must be written. (If the section is being created from scratch, <> should probably point to the section itself and <> should probably be zero.) The data to be written comes from input sections attached (via <> pointers) to the output sections. The output section structure can be considered a filter for the input section: the output section determines the vma of the output data and the name, but the input section determines the offset into the output section of the data to be written. E.g., to create a section "O", starting at 0x100, 0x123 long, containing two subsections, "A" at offset 0x0 (i.e., at vma 0x100) and "B" at offset 0x20 (i.e., at vma 0x120) the <> structures would look like: | section name "A" | output_offset 0x00 | size 0x20 | output_section -----------> section name "O" | | vma 0x100 | section name "B" | size 0x123 | output_offset 0x20 | | size 0x103 | | output_section --------| SUBSECTION Link orders The data within a section is stored in a @dfn{link_order}. These are much like the fixups in <>. The link_order abstraction allows a section to grow and shrink within itself. A link_order knows how big it is, and which is the next link_order and where the raw data for it is; it also points to a list of relocations which apply to it. The link_order is used by the linker to perform relaxing on final code. The compiler creates code which is as big as necessary to make it work without relaxing, and the user can select whether to relax. Sometimes relaxing takes a lot of time. The linker runs around the relocations to see if any are attached to data which can be shrunk, if so it does it on a link_order by link_order basis. */ #include "sysdep.h" #include "bfd.h" #include "libbfd.h" #include "bfdlink.h" /* DOCDD INODE typedef asection, section prototypes, Section Output, Sections SUBSECTION typedef asection Here is the section structure: CODE_FRAGMENT . .typedef struct bfd_section .{ . {* The name of the section; the name isn't a copy, the pointer is . the same as that passed to bfd_make_section. *} . const char *name; . . {* A unique sequence number. *} . int id; . . {* Which section in the bfd; 0..n-1 as sections are created in a bfd. *} . int index; . . {* The next section in the list belonging to the BFD, or NULL. *} . struct bfd_section *next; . . {* The previous section in the list belonging to the BFD, or NULL. *} . struct bfd_section *prev; . . {* The field flags contains attributes of the section. Some . flags are read in from the object file, and some are . synthesized from other information. *} . flagword flags; . .#define SEC_NO_FLAGS 0x000 . . {* Tells the OS to allocate space for this section when loading. . This is clear for a section containing debug information only. *} .#define SEC_ALLOC 0x001 . . {* Tells the OS to load the section from the file when loading. . This is clear for a .bss section. *} .#define SEC_LOAD 0x002 . . {* The section contains data still to be relocated, so there is . some relocation information too. *} .#define SEC_RELOC 0x004 . . {* A signal to the OS that the section contains read only data. *} .#define SEC_READONLY 0x008 . . {* The section contains code only. *} .#define SEC_CODE 0x010 . . {* The section contains data only. *} .#define SEC_DATA 0x020 . . {* The section will reside in ROM. *} .#define SEC_ROM 0x040 . . {* The section contains constructor information. This section . type is used by the linker to create lists of constructors and . destructors used by <>. When a back end sees a symbol . which should be used in a constructor list, it creates a new . section for the type of name (e.g., <<__CTOR_LIST__>>), attaches . the symbol to it, and builds a relocation. To build the lists . of constructors, all the linker has to do is catenate all the . sections called <<__CTOR_LIST__>> and relocate the data . contained within - exactly the operations it would peform on . standard data. *} .#define SEC_CONSTRUCTOR 0x080 . . {* The section has contents - a data section could be . <> | <>; a debug section could be . <> *} .#define SEC_HAS_CONTENTS 0x100 . . {* An instruction to the linker to not output the section . even if it has information which would normally be written. *} .#define SEC_NEVER_LOAD 0x200 . . {* The section contains thread local data. *} .#define SEC_THREAD_LOCAL 0x400 . . {* The section has GOT references. This flag is only for the . linker, and is currently only used by the elf32-hppa back end. . It will be set if global offset table references were detected . in this section, which indicate to the linker that the section . contains PIC code, and must be handled specially when doing a . static link. *} .#define SEC_HAS_GOT_REF 0x800 . . {* The section contains common symbols (symbols may be defined . multiple times, the value of a symbol is the amount of . space it requires, and the largest symbol value is the one . used). Most targets have exactly one of these (which we . translate to bfd_com_section_ptr), but ECOFF has two. *} .#define SEC_IS_COMMON 0x1000 . . {* The section contains only debugging information. For . example, this is set for ELF .debug and .stab sections. . strip tests this flag to see if a section can be . discarded. *} .#define SEC_DEBUGGING 0x2000 . . {* The contents of this section are held in memory pointed to . by the contents field. This is checked by bfd_get_section_contents, . and the data is retrieved from memory if appropriate. *} .#define SEC_IN_MEMORY 0x4000 . . {* The contents of this section are to be excluded by the . linker for executable and shared objects unless those . objects are to be further relocated. *} .#define SEC_EXCLUDE 0x8000 . . {* The contents of this section are to be sorted based on the sum of . the symbol and addend values specified by the associated relocation . entries. Entries without associated relocation entries will be . appended to the end of the section in an unspecified order. *} .#define SEC_SORT_ENTRIES 0x10000 . . {* When linking, duplicate sections of the same name should be . discarded, rather than being combined into a single section as . is usually done. This is similar to how common symbols are . handled. See SEC_LINK_DUPLICATES below. *} .#define SEC_LINK_ONCE 0x20000 . . {* If SEC_LINK_ONCE is set, this bitfield describes how the linker . should handle duplicate sections. *} .#define SEC_LINK_DUPLICATES 0x40000 . . {* This value for SEC_LINK_DUPLICATES means that duplicate . sections with the same name should simply be discarded. *} .#define SEC_LINK_DUPLICATES_DISCARD 0x0 . . {* This value for SEC_LINK_DUPLICATES means that the linker . should warn if there are any duplicate sections, although . it should still only link one copy. *} .#define SEC_LINK_DUPLICATES_ONE_ONLY 0x80000 . . {* This value for SEC_LINK_DUPLICATES means that the linker . should warn if any duplicate sections are a different size. *} .#define SEC_LINK_DUPLICATES_SAME_SIZE 0x100000 . . {* This value for SEC_LINK_DUPLICATES means that the linker . should warn if any duplicate sections contain different . contents. *} .#define SEC_LINK_DUPLICATES_SAME_CONTENTS \ . (SEC_LINK_DUPLICATES_ONE_ONLY | SEC_LINK_DUPLICATES_SAME_SIZE) . . {* This section was created by the linker as part of dynamic . relocation or other arcane processing. It is skipped when . going through the first-pass output, trusting that someone . else up the line will take care of it later. *} .#define SEC_LINKER_CREATED 0x200000 . . {* This section should not be subject to garbage collection. . Also set to inform the linker that this section should not be . listed in the link map as discarded. *} .#define SEC_KEEP 0x400000 . . {* This section contains "short" data, and should be placed . "near" the GP. *} .#define SEC_SMALL_DATA 0x800000 . . {* Attempt to merge identical entities in the section. . Entity size is given in the entsize field. *} .#define SEC_MERGE 0x1000000 . . {* If given with SEC_MERGE, entities to merge are zero terminated . strings where entsize specifies character size instead of fixed . size entries. *} .#define SEC_STRINGS 0x2000000 . . {* This section contains data about section groups. *} .#define SEC_GROUP 0x4000000 . . {* The section is a COFF shared library section. This flag is . only for the linker. If this type of section appears in . the input file, the linker must copy it to the output file . without changing the vma or size. FIXME: Although this . was originally intended to be general, it really is COFF . specific (and the flag was renamed to indicate this). It . might be cleaner to have some more general mechanism to . allow the back end to control what the linker does with . sections. *} .#define SEC_COFF_SHARED_LIBRARY 0x10000000 . . {* This section contains data which may be shared with other . executables or shared objects. This is for COFF only. *} .#define SEC_COFF_SHARED 0x20000000 . . {* When a section with this flag is being linked, then if the size of . the input section is less than a page, it should not cross a page . boundary. If the size of the input section is one page or more, . it should be aligned on a page boundary. This is for TI . TMS320C54X only. *} .#define SEC_TIC54X_BLOCK 0x40000000 . . {* Conditionally link this section; do not link if there are no . references found to any symbol in the section. This is for TI . TMS320C54X only. *} .#define SEC_TIC54X_CLINK 0x80000000 . . {* End of section flags. *} . . {* Some internal packed boolean fields. *} . . {* See the vma field. *} . unsigned int user_set_vma : 1; . . {* A mark flag used by some of the linker backends. *} . unsigned int linker_mark : 1; . . {* Another mark flag used by some of the linker backends. Set for . output sections that have an input section. *} . unsigned int linker_has_input : 1; . . {* Mark flags used by some linker backends for garbage collection. *} . unsigned int gc_mark : 1; . unsigned int gc_mark_from_eh : 1; . . {* The following flags are used by the ELF linker. *} . . {* Mark sections which have been allocated to segments. *} . unsigned int segment_mark : 1; . . {* Type of sec_info information. *} . unsigned int sec_info_type:3; .#define ELF_INFO_TYPE_NONE 0 .#define ELF_INFO_TYPE_STABS 1 .#define ELF_INFO_TYPE_MERGE 2 .#define ELF_INFO_TYPE_EH_FRAME 3 .#define ELF_INFO_TYPE_JUST_SYMS 4 . . {* Nonzero if this section uses RELA relocations, rather than REL. *} . unsigned int use_rela_p:1; . . {* Bits used by various backends. The generic code doesn't touch . these fields. *} . . {* Nonzero if this section has TLS related relocations. *} . unsigned int has_tls_reloc:1; . +. {* Nonzero if this section has a call to __tls_get_addr. *} +. unsigned int has_tls_get_addr_call:1; +. . {* Nonzero if this section has a gp reloc. *} . unsigned int has_gp_reloc:1; . . {* Nonzero if this section needs the relax finalize pass. *} . unsigned int need_finalize_relax:1; . . {* Whether relocations have been processed. *} . unsigned int reloc_done : 1; . . {* End of internal packed boolean fields. *} . . {* The virtual memory address of the section - where it will be . at run time. The symbols are relocated against this. The . user_set_vma flag is maintained by bfd; if it's not set, the . backend can assign addresses (for example, in <>, where . the default address for <<.data>> is dependent on the specific . target and various flags). *} . bfd_vma vma; . . {* The load address of the section - where it would be in a . rom image; really only used for writing section header . information. *} . bfd_vma lma; . . {* The size of the section in octets, as it will be output. . Contains a value even if the section has no contents (e.g., the . size of <<.bss>>). *} . bfd_size_type size; . . {* For input sections, the original size on disk of the section, in . octets. This field is used by the linker relaxation code. It is . currently only set for sections where the linker relaxation scheme . doesn't cache altered section and reloc contents (stabs, eh_frame, . SEC_MERGE, some coff relaxing targets), and thus the original size . needs to be kept to read the section multiple times. . For output sections, rawsize holds the section size calculated on . a previous linker relaxation pass. *} . bfd_size_type rawsize; . . {* If this section is going to be output, then this value is the . offset in *bytes* into the output section of the first byte in the . input section (byte ==> smallest addressable unit on the . target). In most cases, if this was going to start at the . 100th octet (8-bit quantity) in the output section, this value . would be 100. However, if the target byte size is 16 bits . (bfd_octets_per_byte is "2"), this value would be 50. *} . bfd_vma output_offset; . . {* The output section through which to map on output. *} . struct bfd_section *output_section; . . {* The alignment requirement of the section, as an exponent of 2 - . e.g., 3 aligns to 2^3 (or 8). *} . unsigned int alignment_power; . . {* If an input section, a pointer to a vector of relocation . records for the data in this section. *} . struct reloc_cache_entry *relocation; . . {* If an output section, a pointer to a vector of pointers to . relocation records for the data in this section. *} . struct reloc_cache_entry **orelocation; . . {* The number of relocation records in one of the above. *} . unsigned reloc_count; . . {* Information below is back end specific - and not always used . or updated. *} . . {* File position of section data. *} . file_ptr filepos; . . {* File position of relocation info. *} . file_ptr rel_filepos; . . {* File position of line data. *} . file_ptr line_filepos; . . {* Pointer to data for applications. *} . void *userdata; . . {* If the SEC_IN_MEMORY flag is set, this points to the actual . contents. *} . unsigned char *contents; . . {* Attached line number information. *} . alent *lineno; . . {* Number of line number records. *} . unsigned int lineno_count; . . {* Entity size for merging purposes. *} . unsigned int entsize; . . {* Points to the kept section if this section is a link-once section, . and is discarded. *} . struct bfd_section *kept_section; . . {* When a section is being output, this value changes as more . linenumbers are written out. *} . file_ptr moving_line_filepos; . . {* What the section number is in the target world. *} . int target_index; . . void *used_by_bfd; . . {* If this is a constructor section then here is a list of the . relocations created to relocate items within it. *} . struct relent_chain *constructor_chain; . . {* The BFD which owns the section. *} . bfd *owner; . . {* A symbol which points at this section only. *} . struct bfd_symbol *symbol; . struct bfd_symbol **symbol_ptr_ptr; . . {* Early in the link process, map_head and map_tail are used to build . a list of input sections attached to an output section. Later, . output sections use these fields for a list of bfd_link_order . structs. *} . union { . struct bfd_link_order *link_order; . struct bfd_section *s; . } map_head, map_tail; .} asection; . .{* These sections are global, and are managed by BFD. The application . and target back end are not permitted to change the values in . these sections. New code should use the section_ptr macros rather . than referring directly to the const sections. The const sections . may eventually vanish. *} .#define BFD_ABS_SECTION_NAME "*ABS*" .#define BFD_UND_SECTION_NAME "*UND*" .#define BFD_COM_SECTION_NAME "*COM*" .#define BFD_IND_SECTION_NAME "*IND*" . .{* The absolute section. *} .extern asection bfd_abs_section; .#define bfd_abs_section_ptr ((asection *) &bfd_abs_section) .#define bfd_is_abs_section(sec) ((sec) == bfd_abs_section_ptr) .{* Pointer to the undefined section. *} .extern asection bfd_und_section; .#define bfd_und_section_ptr ((asection *) &bfd_und_section) .#define bfd_is_und_section(sec) ((sec) == bfd_und_section_ptr) .{* Pointer to the common section. *} .extern asection bfd_com_section; .#define bfd_com_section_ptr ((asection *) &bfd_com_section) .{* Pointer to the indirect section. *} .extern asection bfd_ind_section; .#define bfd_ind_section_ptr ((asection *) &bfd_ind_section) .#define bfd_is_ind_section(sec) ((sec) == bfd_ind_section_ptr) . .#define bfd_is_const_section(SEC) \ . ( ((SEC) == bfd_abs_section_ptr) \ . || ((SEC) == bfd_und_section_ptr) \ . || ((SEC) == bfd_com_section_ptr) \ . || ((SEC) == bfd_ind_section_ptr)) . .{* Macros to handle insertion and deletion of a bfd's sections. These . only handle the list pointers, ie. do not adjust section_count, . target_index etc. *} .#define bfd_section_list_remove(ABFD, S) \ . do \ . { \ . asection *_s = S; \ . asection *_next = _s->next; \ . asection *_prev = _s->prev; \ . if (_prev) \ . _prev->next = _next; \ . else \ . (ABFD)->sections = _next; \ . if (_next) \ . _next->prev = _prev; \ . else \ . (ABFD)->section_last = _prev; \ . } \ . while (0) .#define bfd_section_list_append(ABFD, S) \ . do \ . { \ . asection *_s = S; \ . bfd *_abfd = ABFD; \ . _s->next = NULL; \ . if (_abfd->section_last) \ . { \ . _s->prev = _abfd->section_last; \ . _abfd->section_last->next = _s; \ . } \ . else \ . { \ . _s->prev = NULL; \ . _abfd->sections = _s; \ . } \ . _abfd->section_last = _s; \ . } \ . while (0) .#define bfd_section_list_prepend(ABFD, S) \ . do \ . { \ . asection *_s = S; \ . bfd *_abfd = ABFD; \ . _s->prev = NULL; \ . if (_abfd->sections) \ . { \ . _s->next = _abfd->sections; \ . _abfd->sections->prev = _s; \ . } \ . else \ . { \ . _s->next = NULL; \ . _abfd->section_last = _s; \ . } \ . _abfd->sections = _s; \ . } \ . while (0) .#define bfd_section_list_insert_after(ABFD, A, S) \ . do \ . { \ . asection *_a = A; \ . asection *_s = S; \ . asection *_next = _a->next; \ . _s->next = _next; \ . _s->prev = _a; \ . _a->next = _s; \ . if (_next) \ . _next->prev = _s; \ . else \ . (ABFD)->section_last = _s; \ . } \ . while (0) .#define bfd_section_list_insert_before(ABFD, B, S) \ . do \ . { \ . asection *_b = B; \ . asection *_s = S; \ . asection *_prev = _b->prev; \ . _s->prev = _prev; \ . _s->next = _b; \ . _b->prev = _s; \ . if (_prev) \ . _prev->next = _s; \ . else \ . (ABFD)->sections = _s; \ . } \ . while (0) .#define bfd_section_removed_from_list(ABFD, S) \ . ((S)->next == NULL ? (ABFD)->section_last != (S) : (S)->next->prev != (S)) . .#define BFD_FAKE_SECTION(SEC, FLAGS, SYM, NAME, IDX) \ . {* name, id, index, next, prev, flags, user_set_vma, *} \ . { NAME, IDX, 0, NULL, NULL, FLAGS, 0, \ . \ . {* linker_mark, linker_has_input, gc_mark, gc_mark_from_eh, *} \ . 0, 0, 1, 0, \ . \ . {* segment_mark, sec_info_type, use_rela_p, has_tls_reloc, *} \ . 0, 0, 0, 0, \ . \ -. {* has_gp_reloc, need_finalize_relax, reloc_done, *} \ -. 0, 0, 0, \ +. {* has_tls_get_addr_call, has_gp_reloc, need_finalize_relax, *} \ +. 0, 0, 0, \ . \ -. {* vma, lma, size, rawsize *} \ -. 0, 0, 0, 0, \ +. {* reloc_done, vma, lma, size, rawsize *} \ +. 0, 0, 0, 0, 0, \ . \ . {* output_offset, output_section, alignment_power, *} \ . 0, (struct bfd_section *) &SEC, 0, \ . \ . {* relocation, orelocation, reloc_count, filepos, rel_filepos, *} \ . NULL, NULL, 0, 0, 0, \ . \ . {* line_filepos, userdata, contents, lineno, lineno_count, *} \ . 0, NULL, NULL, NULL, 0, \ . \ . {* entsize, kept_section, moving_line_filepos, *} \ . 0, NULL, 0, \ . \ . {* target_index, used_by_bfd, constructor_chain, owner, *} \ . 0, NULL, NULL, NULL, \ . \ . {* symbol, symbol_ptr_ptr, *} \ . (struct bfd_symbol *) SYM, &SEC.symbol, \ . \ . {* map_head, map_tail *} \ . { NULL }, { NULL } \ . } . */ /* We use a macro to initialize the static asymbol structures because traditional C does not permit us to initialize a union member while gcc warns if we don't initialize it. */ /* the_bfd, name, value, attr, section [, udata] */ #ifdef __STDC__ #define GLOBAL_SYM_INIT(NAME, SECTION) \ { 0, NAME, 0, BSF_SECTION_SYM, (asection *) SECTION, { 0 }} #else #define GLOBAL_SYM_INIT(NAME, SECTION) \ { 0, NAME, 0, BSF_SECTION_SYM, (asection *) SECTION } #endif /* These symbols are global, not specific to any BFD. Therefore, anything that tries to change them is broken, and should be repaired. */ static const asymbol global_syms[] = { GLOBAL_SYM_INIT (BFD_COM_SECTION_NAME, &bfd_com_section), GLOBAL_SYM_INIT (BFD_UND_SECTION_NAME, &bfd_und_section), GLOBAL_SYM_INIT (BFD_ABS_SECTION_NAME, &bfd_abs_section), GLOBAL_SYM_INIT (BFD_IND_SECTION_NAME, &bfd_ind_section) }; #define STD_SECTION(SEC, FLAGS, NAME, IDX) \ asection SEC = BFD_FAKE_SECTION(SEC, FLAGS, &global_syms[IDX], \ NAME, IDX) STD_SECTION (bfd_com_section, SEC_IS_COMMON, BFD_COM_SECTION_NAME, 0); STD_SECTION (bfd_und_section, 0, BFD_UND_SECTION_NAME, 1); STD_SECTION (bfd_abs_section, 0, BFD_ABS_SECTION_NAME, 2); STD_SECTION (bfd_ind_section, 0, BFD_IND_SECTION_NAME, 3); #undef STD_SECTION /* Initialize an entry in the section hash table. */ struct bfd_hash_entry * bfd_section_hash_newfunc (struct bfd_hash_entry *entry, struct bfd_hash_table *table, const char *string) { /* Allocate the structure if it has not already been allocated by a subclass. */ if (entry == NULL) { entry = (struct bfd_hash_entry *) bfd_hash_allocate (table, sizeof (struct section_hash_entry)); if (entry == NULL) return entry; } /* Call the allocation method of the superclass. */ entry = bfd_hash_newfunc (entry, table, string); if (entry != NULL) memset (&((struct section_hash_entry *) entry)->section, 0, sizeof (asection)); return entry; } #define section_hash_lookup(table, string, create, copy) \ ((struct section_hash_entry *) \ bfd_hash_lookup ((table), (string), (create), (copy))) /* Create a symbol whose only job is to point to this section. This is useful for things like relocs which are relative to the base of a section. */ bfd_boolean _bfd_generic_new_section_hook (bfd *abfd, asection *newsect) { newsect->symbol = bfd_make_empty_symbol (abfd); if (newsect->symbol == NULL) return FALSE; newsect->symbol->name = newsect->name; newsect->symbol->value = 0; newsect->symbol->section = newsect; newsect->symbol->flags = BSF_SECTION_SYM; newsect->symbol_ptr_ptr = &newsect->symbol; return TRUE; } /* Initializes a new section. NEWSECT->NAME is already set. */ static asection * bfd_section_init (bfd *abfd, asection *newsect) { static int section_id = 0x10; /* id 0 to 3 used by STD_SECTION. */ newsect->id = section_id; newsect->index = abfd->section_count; newsect->owner = abfd; if (! BFD_SEND (abfd, _new_section_hook, (abfd, newsect))) return NULL; section_id++; abfd->section_count++; bfd_section_list_append (abfd, newsect); return newsect; } /* DOCDD INODE section prototypes, , typedef asection, Sections SUBSECTION Section prototypes These are the functions exported by the section handling part of BFD. */ /* FUNCTION bfd_section_list_clear SYNOPSIS void bfd_section_list_clear (bfd *); DESCRIPTION Clears the section list, and also resets the section count and hash table entries. */ void bfd_section_list_clear (bfd *abfd) { abfd->sections = NULL; abfd->section_last = NULL; abfd->section_count = 0; memset (abfd->section_htab.table, 0, abfd->section_htab.size * sizeof (struct bfd_hash_entry *)); } /* FUNCTION bfd_get_section_by_name SYNOPSIS asection *bfd_get_section_by_name (bfd *abfd, const char *name); DESCRIPTION Run through @var{abfd} and return the one of the <>s whose name matches @var{name}, otherwise <>. @xref{Sections}, for more information. This should only be used in special cases; the normal way to process all sections of a given name is to use <> and <> on the name (or better yet, base it on the section flags or something else) for each section. */ asection * bfd_get_section_by_name (bfd *abfd, const char *name) { struct section_hash_entry *sh; sh = section_hash_lookup (&abfd->section_htab, name, FALSE, FALSE); if (sh != NULL) return &sh->section; return NULL; } /* FUNCTION bfd_get_section_by_name_if SYNOPSIS asection *bfd_get_section_by_name_if (bfd *abfd, const char *name, bfd_boolean (*func) (bfd *abfd, asection *sect, void *obj), void *obj); DESCRIPTION Call the provided function @var{func} for each section attached to the BFD @var{abfd} whose name matches @var{name}, passing @var{obj} as an argument. The function will be called as if by | func (abfd, the_section, obj); It returns the first section for which @var{func} returns true, otherwise <>. */ asection * bfd_get_section_by_name_if (bfd *abfd, const char *name, bfd_boolean (*operation) (bfd *, asection *, void *), void *user_storage) { struct section_hash_entry *sh; unsigned long hash; sh = section_hash_lookup (&abfd->section_htab, name, FALSE, FALSE); if (sh == NULL) return NULL; hash = sh->root.hash; do { if ((*operation) (abfd, &sh->section, user_storage)) return &sh->section; sh = (struct section_hash_entry *) sh->root.next; } while (sh != NULL && sh->root.hash == hash && strcmp (sh->root.string, name) == 0); return NULL; } /* FUNCTION bfd_get_unique_section_name SYNOPSIS char *bfd_get_unique_section_name (bfd *abfd, const char *templat, int *count); DESCRIPTION Invent a section name that is unique in @var{abfd} by tacking a dot and a digit suffix onto the original @var{templat}. If @var{count} is non-NULL, then it specifies the first number tried as a suffix to generate a unique name. The value pointed to by @var{count} will be incremented in this case. */ char * bfd_get_unique_section_name (bfd *abfd, const char *templat, int *count) { int num; unsigned int len; char *sname; len = strlen (templat); sname = bfd_malloc (len + 8); if (sname == NULL) return NULL; memcpy (sname, templat, len); num = 1; if (count != NULL) num = *count; do { /* If we have a million sections, something is badly wrong. */ if (num > 999999) abort (); sprintf (sname + len, ".%d", num++); } while (section_hash_lookup (&abfd->section_htab, sname, FALSE, FALSE)); if (count != NULL) *count = num; return sname; } /* FUNCTION bfd_make_section_old_way SYNOPSIS asection *bfd_make_section_old_way (bfd *abfd, const char *name); DESCRIPTION Create a new empty section called @var{name} and attach it to the end of the chain of sections for the BFD @var{abfd}. An attempt to create a section with a name which is already in use returns its pointer without changing the section chain. It has the funny name since this is the way it used to be before it was rewritten.... Possible errors are: o <> - If output has already started for this BFD. o <> - If memory allocation fails. */ asection * bfd_make_section_old_way (bfd *abfd, const char *name) { asection *newsect; if (abfd->output_has_begun) { bfd_set_error (bfd_error_invalid_operation); return NULL; } if (strcmp (name, BFD_ABS_SECTION_NAME) == 0) newsect = bfd_abs_section_ptr; else if (strcmp (name, BFD_COM_SECTION_NAME) == 0) newsect = bfd_com_section_ptr; else if (strcmp (name, BFD_UND_SECTION_NAME) == 0) newsect = bfd_und_section_ptr; else if (strcmp (name, BFD_IND_SECTION_NAME) == 0) newsect = bfd_ind_section_ptr; else { struct section_hash_entry *sh; sh = section_hash_lookup (&abfd->section_htab, name, TRUE, FALSE); if (sh == NULL) return NULL; newsect = &sh->section; if (newsect->name != NULL) { /* Section already exists. */ return newsect; } newsect->name = name; return bfd_section_init (abfd, newsect); } /* Call new_section_hook when "creating" the standard abs, com, und and ind sections to tack on format specific section data. Also, create a proper section symbol. */ if (! BFD_SEND (abfd, _new_section_hook, (abfd, newsect))) return NULL; return newsect; } /* FUNCTION bfd_make_section_anyway_with_flags SYNOPSIS asection *bfd_make_section_anyway_with_flags (bfd *abfd, const char *name, flagword flags); DESCRIPTION Create a new empty section called @var{name} and attach it to the end of the chain of sections for @var{abfd}. Create a new section even if there is already a section with that name. Also set the attributes of the new section to the value @var{flags}. Return <> and set <> on error; possible errors are: o <> - If output has already started for @var{abfd}. o <> - If memory allocation fails. */ sec_ptr bfd_make_section_anyway_with_flags (bfd *abfd, const char *name, flagword flags) { struct section_hash_entry *sh; asection *newsect; if (abfd->output_has_begun) { bfd_set_error (bfd_error_invalid_operation); return NULL; } sh = section_hash_lookup (&abfd->section_htab, name, TRUE, FALSE); if (sh == NULL) return NULL; newsect = &sh->section; if (newsect->name != NULL) { /* We are making a section of the same name. Put it in the section hash table. Even though we can't find it directly by a hash lookup, we'll be able to find the section by traversing sh->root.next quicker than looking at all the bfd sections. */ struct section_hash_entry *new_sh; new_sh = (struct section_hash_entry *) bfd_section_hash_newfunc (NULL, &abfd->section_htab, name); if (new_sh == NULL) return NULL; new_sh->root = sh->root; sh->root.next = &new_sh->root; newsect = &new_sh->section; } newsect->flags = flags; newsect->name = name; return bfd_section_init (abfd, newsect); } /* FUNCTION bfd_make_section_anyway SYNOPSIS asection *bfd_make_section_anyway (bfd *abfd, const char *name); DESCRIPTION Create a new empty section called @var{name} and attach it to the end of the chain of sections for @var{abfd}. Create a new section even if there is already a section with that name. Return <> and set <> on error; possible errors are: o <> - If output has already started for @var{abfd}. o <> - If memory allocation fails. */ sec_ptr bfd_make_section_anyway (bfd *abfd, const char *name) { return bfd_make_section_anyway_with_flags (abfd, name, 0); } /* FUNCTION bfd_make_section_with_flags SYNOPSIS asection *bfd_make_section_with_flags (bfd *, const char *name, flagword flags); DESCRIPTION Like <>, but return <> (without calling bfd_set_error ()) without changing the section chain if there is already a section named @var{name}. Also set the attributes of the new section to the value @var{flags}. If there is an error, return <> and set <>. */ asection * bfd_make_section_with_flags (bfd *abfd, const char *name, flagword flags) { struct section_hash_entry *sh; asection *newsect; if (abfd->output_has_begun) { bfd_set_error (bfd_error_invalid_operation); return NULL; } if (strcmp (name, BFD_ABS_SECTION_NAME) == 0 || strcmp (name, BFD_COM_SECTION_NAME) == 0 || strcmp (name, BFD_UND_SECTION_NAME) == 0 || strcmp (name, BFD_IND_SECTION_NAME) == 0) return NULL; sh = section_hash_lookup (&abfd->section_htab, name, TRUE, FALSE); if (sh == NULL) return NULL; newsect = &sh->section; if (newsect->name != NULL) { /* Section already exists. */ return NULL; } newsect->name = name; newsect->flags = flags; return bfd_section_init (abfd, newsect); } /* FUNCTION bfd_make_section SYNOPSIS asection *bfd_make_section (bfd *, const char *name); DESCRIPTION Like <>, but return <> (without calling bfd_set_error ()) without changing the section chain if there is already a section named @var{name}. If there is an error, return <> and set <>. */ asection * bfd_make_section (bfd *abfd, const char *name) { return bfd_make_section_with_flags (abfd, name, 0); } /* FUNCTION bfd_set_section_flags SYNOPSIS bfd_boolean bfd_set_section_flags (bfd *abfd, asection *sec, flagword flags); DESCRIPTION Set the attributes of the section @var{sec} in the BFD @var{abfd} to the value @var{flags}. Return <> on success, <> on error. Possible error returns are: o <> - The section cannot have one or more of the attributes requested. For example, a .bss section in <> may not have the <> field set. */ bfd_boolean bfd_set_section_flags (bfd *abfd ATTRIBUTE_UNUSED, sec_ptr section, flagword flags) { section->flags = flags; return TRUE; } /* FUNCTION bfd_map_over_sections SYNOPSIS void bfd_map_over_sections (bfd *abfd, void (*func) (bfd *abfd, asection *sect, void *obj), void *obj); DESCRIPTION Call the provided function @var{func} for each section attached to the BFD @var{abfd}, passing @var{obj} as an argument. The function will be called as if by | func (abfd, the_section, obj); This is the preferred method for iterating over sections; an alternative would be to use a loop: | section *p; | for (p = abfd->sections; p != NULL; p = p->next) | func (abfd, p, ...) */ void bfd_map_over_sections (bfd *abfd, void (*operation) (bfd *, asection *, void *), void *user_storage) { asection *sect; unsigned int i = 0; for (sect = abfd->sections; sect != NULL; i++, sect = sect->next) (*operation) (abfd, sect, user_storage); if (i != abfd->section_count) /* Debugging */ abort (); } /* FUNCTION bfd_sections_find_if SYNOPSIS asection *bfd_sections_find_if (bfd *abfd, bfd_boolean (*operation) (bfd *abfd, asection *sect, void *obj), void *obj); DESCRIPTION Call the provided function @var{operation} for each section attached to the BFD @var{abfd}, passing @var{obj} as an argument. The function will be called as if by | operation (abfd, the_section, obj); It returns the first section for which @var{operation} returns true. */ asection * bfd_sections_find_if (bfd *abfd, bfd_boolean (*operation) (bfd *, asection *, void *), void *user_storage) { asection *sect; for (sect = abfd->sections; sect != NULL; sect = sect->next) if ((*operation) (abfd, sect, user_storage)) break; return sect; } /* FUNCTION bfd_set_section_size SYNOPSIS bfd_boolean bfd_set_section_size (bfd *abfd, asection *sec, bfd_size_type val); DESCRIPTION Set @var{sec} to the size @var{val}. If the operation is ok, then <> is returned, else <>. Possible error returns: o <> - Writing has started to the BFD, so setting the size is invalid. */ bfd_boolean bfd_set_section_size (bfd *abfd, sec_ptr ptr, bfd_size_type val) { /* Once you've started writing to any section you cannot create or change the size of any others. */ if (abfd->output_has_begun) { bfd_set_error (bfd_error_invalid_operation); return FALSE; } ptr->size = val; return TRUE; } /* FUNCTION bfd_set_section_contents SYNOPSIS bfd_boolean bfd_set_section_contents (bfd *abfd, asection *section, const void *data, file_ptr offset, bfd_size_type count); DESCRIPTION Sets the contents of the section @var{section} in BFD @var{abfd} to the data starting in memory at @var{data}. The data is written to the output section starting at offset @var{offset} for @var{count} octets. Normally <> is returned, else <>. Possible error returns are: o <> - The output section does not have the <> attribute, so nothing can be written to it. o and some more too This routine is front end to the back end function <<_bfd_set_section_contents>>. */ bfd_boolean bfd_set_section_contents (bfd *abfd, sec_ptr section, const void *location, file_ptr offset, bfd_size_type count) { bfd_size_type sz; if (!(bfd_get_section_flags (abfd, section) & SEC_HAS_CONTENTS)) { bfd_set_error (bfd_error_no_contents); return FALSE; } sz = section->size; if ((bfd_size_type) offset > sz || count > sz || offset + count > sz || count != (size_t) count) { bfd_set_error (bfd_error_bad_value); return FALSE; } if (!bfd_write_p (abfd)) { bfd_set_error (bfd_error_invalid_operation); return FALSE; } /* Record a copy of the data in memory if desired. */ if (section->contents && location != section->contents + offset) memcpy (section->contents + offset, location, (size_t) count); if (BFD_SEND (abfd, _bfd_set_section_contents, (abfd, section, location, offset, count))) { abfd->output_has_begun = TRUE; return TRUE; } return FALSE; } /* FUNCTION bfd_get_section_contents SYNOPSIS bfd_boolean bfd_get_section_contents (bfd *abfd, asection *section, void *location, file_ptr offset, bfd_size_type count); DESCRIPTION Read data from @var{section} in BFD @var{abfd} into memory starting at @var{location}. The data is read at an offset of @var{offset} from the start of the input section, and is read for @var{count} bytes. If the contents of a constructor with the <> flag set are requested or if the section does not have the <> flag set, then the @var{location} is filled with zeroes. If no errors occur, <> is returned, else <>. */ bfd_boolean bfd_get_section_contents (bfd *abfd, sec_ptr section, void *location, file_ptr offset, bfd_size_type count) { bfd_size_type sz; if (section->flags & SEC_CONSTRUCTOR) { memset (location, 0, (size_t) count); return TRUE; } sz = section->rawsize ? section->rawsize : section->size; if ((bfd_size_type) offset > sz || count > sz || offset + count > sz || count != (size_t) count) { bfd_set_error (bfd_error_bad_value); return FALSE; } if (count == 0) /* Don't bother. */ return TRUE; if ((section->flags & SEC_HAS_CONTENTS) == 0) { memset (location, 0, (size_t) count); return TRUE; } if ((section->flags & SEC_IN_MEMORY) != 0) { memcpy (location, section->contents + offset, (size_t) count); return TRUE; } return BFD_SEND (abfd, _bfd_get_section_contents, (abfd, section, location, offset, count)); } /* FUNCTION bfd_malloc_and_get_section SYNOPSIS bfd_boolean bfd_malloc_and_get_section (bfd *abfd, asection *section, bfd_byte **buf); DESCRIPTION Read all data from @var{section} in BFD @var{abfd} into a buffer, *@var{buf}, malloc'd by this function. */ bfd_boolean bfd_malloc_and_get_section (bfd *abfd, sec_ptr sec, bfd_byte **buf) { bfd_size_type sz = sec->rawsize ? sec->rawsize : sec->size; bfd_byte *p = NULL; *buf = p; if (sz == 0) return TRUE; p = bfd_malloc (sec->rawsize > sec->size ? sec->rawsize : sec->size); if (p == NULL) return FALSE; *buf = p; return bfd_get_section_contents (abfd, sec, p, 0, sz); } /* FUNCTION bfd_copy_private_section_data SYNOPSIS bfd_boolean bfd_copy_private_section_data (bfd *ibfd, asection *isec, bfd *obfd, asection *osec); DESCRIPTION Copy private section information from @var{isec} in the BFD @var{ibfd} to the section @var{osec} in the BFD @var{obfd}. Return <> on success, <> on error. Possible error returns are: o <> - Not enough memory exists to create private data for @var{osec}. .#define bfd_copy_private_section_data(ibfd, isection, obfd, osection) \ . BFD_SEND (obfd, _bfd_copy_private_section_data, \ . (ibfd, isection, obfd, osection)) */ /* FUNCTION bfd_generic_is_group_section SYNOPSIS bfd_boolean bfd_generic_is_group_section (bfd *, const asection *sec); DESCRIPTION Returns TRUE if @var{sec} is a member of a group. */ bfd_boolean bfd_generic_is_group_section (bfd *abfd ATTRIBUTE_UNUSED, const asection *sec ATTRIBUTE_UNUSED) { return FALSE; } /* FUNCTION bfd_generic_discard_group SYNOPSIS bfd_boolean bfd_generic_discard_group (bfd *abfd, asection *group); DESCRIPTION Remove all members of @var{group} from the output. */ bfd_boolean bfd_generic_discard_group (bfd *abfd ATTRIBUTE_UNUSED, asection *group ATTRIBUTE_UNUSED) { return TRUE; } Index: projects/clang350-import/contrib/binutils/include/elf/ChangeLog =================================================================== --- projects/clang350-import/contrib/binutils/include/elf/ChangeLog (revision 275748) +++ projects/clang350-import/contrib/binutils/include/elf/ChangeLog (revision 275749) @@ -1,636 +1,641 @@ +2009-03-04 Alan Modra + + * ppc.h (R_PPC_TLSGD, R_PPC_TLSLD): Add new relocs. + * ppc64.h (R_PPC64_TLSGD, R_PPC64_TLSLD): Add new relocs. + 2007-06-29 Joseph Myers * ppc.h (Tag_GNU_Power_ABI_FP): Define. 2007-06-29 Joseph Myers * mips.h (Tag_GNU_MIPS_ABI_FP): Define. 2007-06-29 Joseph Myers * arm.h (elf32_arm_add_eabi_attr_int, elf32_arm_add_eabi_attr_string, elf32_arm_add_eabi_attr_compat, elf32_arm_get_eabi_attr_int, elf32_arm_set_eabi_attr_contents, elf32_arm_eabi_attr_size, Tag_NULL, Tag_File, Tag_Section, Tag_Symbol, Tag_compatibility): Remove. * common.h (SHT_GNU_ATTRIBUTES): Define. 2007-06-29 M R Swami Reddy * common.h (EM_CR16): New entry for CR16 cpu. * cr16.h: New file. 2007-06-11 Sterling Augustine Bob Wilson * xtensa.h (XTENSA_PROP_INSN_NO_TRANSFORM): Renamed to... (XTENSA_PROP_NO_TRANSFORM): ...this. 2007-05-18 Caroline Tice * dwarf2.h: (enum dwarf_location_atom): Add new DW_OP, DW_OP_GNU_uninit. 2007-05-12 Alan Modra * spu.h (R_SPU_ADDR16X): Define. (R_SPU_PPU32, R_SPU_PPU64): Renumber. 2007-05-11 Alan Modra * spu.h (R_SPU_PPU32, R_SPU_PPU64): Define. 2007-05-02 Alan Modra * internal.h (ELF_IS_SECTION_IN_SEGMENT): Check both file offset and vma for appropriate sections. 2007-04-26 Jan Beulich * common.h (DT_ENCODING): Correct value (back to spec mandated value). 2007-03-08 Alan Modra * v850.h (V850_OTHER_TDA_BYTE): Delete. (V850_OTHER_SDA, V850_OTHER_ZDA, V850_OTHER_TDA): Assign bits that don't clash with visibility bits. 2007-03-07 Alan Modra * common.h (ELF_ST_VISIBILITY): Comment typo fix. 2007-02-05 Dave Brolley Richard Sandiford Richard Henderson DJ Delorie Ben Elliston * mep.h: New file. * common.h (EM_CYGNUS_MEP): Define. 2007-02-15 Dave Brolley From Graydon Hoare : * common.h (STT_RELC, STT_SRELC, R_RELC): New macros. 2007-01-08 Kazu Hirata * m68k.h (EF_M68K_FIDO): New. (EF_M68K_ARCH_MASK): OR EF_M68K_FIDO. (EF_M68K_CPU32_FIDO_A, EF_M68K_CPU32_MASK): Remove. 2006-12-25 Kazu Hirata * m68k.h (EF_M68K_CPU32_FIDO_A, EF_M68K_CPU32_MASK): New. 2006-12-19 Kazu Hirata * m68k.h (EF_M68K_ARCH_MASK): New. 2006-12-19 Nathan Sidwell * internal.h (struct elf_segment_map): Add p_vaddr_offset field. 2006-12-07 Kazu Hirata * m68k.h (EF_M68K_ISA_MASK, EF_M68K_ISA_A_NODIV, EF_M68K_ISA_A, EF_M68K_ISA_A_PLUS, EF_M68K_ISA_B_NOUSP, EF_M68K_ISA_B, EF_M68K_ISA_C, EF_M68K_MAC_MASK, EF_M68K_MAC, EF_M68K_EMAC, EF_M68K_EMAC_B, EF_M68K_FLOAT): Rename to EF_M68K_CF_ISA_MASK, EF_M68K_CF_ISA_A_NODIV, EF_M68K_CF_ISA_A, EF_M68K_CF_ISA_A_PLUS, EF_M68K_CF_ISA_B_NOUSP, EF_M68K_CF_ISA_B, EF_M68K_CF_ISA_C, EF_M68K_CF_MAC_MASK, EF_M68K_CF_MAC, EF_M68K_CF_EMAC, EF_M68K_CF_EMAC_B, EF_M68K_CF_FLOAT, respectively. 2006-12-05 Michael Tautschnig Nick Clifton * external.h (struct Elf_External_Versym): Use ATTRIBUTE_PACKED. 2006-10-28 Richard Sandiford * mips.h (R_MIPS_GLOB_DAT): Define (R_MIPS_max): Bump by 1. 2006-10-25 Trevor Smigiel Yukishige Shibata Nobuhisa Fujinami Takeaki Fukuoka Alan Modra * common.h (EM_SPU): Define. * spu.h: New file. 2006-10-19 Mei Ligang * score.h (EF_SCORE_PIC): Redefine EF_SCORE_PIC as 0x80000000. (EF_SCORE_FIXDEP): Redefine EF_SCORE_FIXDEP as 0x40000000. (EF_SCORE_HASENTRY): Delete. 2006-09-17 Mei Ligang * score.h: New file. * common.h: Add Score machine number. 2006-07-10 Jakub Jelinek * common.h (SHT_GNU_HASH, DT_GNU_HASH): Define. 2006-05-31 H.J. Lu * internal.h (ELF_SECTION_SIZE): New. (ELF_IS_SECTION_IN_SEGMENT): Likewise. (ELF_IS_SECTION_IN_SEGMENT_FILE): Updated. (ELF_IS_SECTION_IN_SEGMENT_MEMORY): Likewise. 2006-05-27 H.J. Lu * internal.h (struct elf_segment_map): Add p_align and p_align_valid. 2006-05-24 Carlos O'Donell Randolph Chung * hppa.h (R_PARISC_TLS_GD21L, R_PARISC_TLS_GD14R, R_PARISC_TLS_GDCALL, R_PARISC_TLS_LDM21L, R_PARISC_TLS_LDM14R, R_PARISC_TLS_LDMCALL, R_PARISC_TLS_LDO21L, R_PARISC_TLS_LDO14R, R_PARISC_TLS_DTPMOD32, R_PARISC_TLS_DTPMOD64, R_PARISC_TLS_DTPOFF32, R_PARISC_TLS_DTPOFF64): New TLS relocs. (R_PARISC_TLS_LE21L, R_PARISC_TLS_LE14R, R_PARISC_TLS_IE21L, R_PARISC_TLS_IE14R, R_PARISC_TLS_TPREL32, R_PARISC_TLS_TPREL64): Define TLS relocs using existing equivalents. 2006-05-24 Bjoern Haase * avr.h: Add E_AVR_MACH_AVR6, R_AVR_LO8_LDI_GS and R_AVR_HI8_LDI_GS. 2006-03-22 Richard Sandiford Daniel Jacobowitz Phil Edwards Zack Weinberg Mark Mitchell Nathan Sidwell * mips.h (R_MIPS_COPY, R_MIPS_JUMP_SLOT): New relocs. 2006-03-19 John David Anglin * hppa.h (SHF_HP_TLS, SHF_HP_NEAR_SHARED, SHF_HP_FAR_SHARED, SHF_HP_COMDAT, SHF_HP_CONST, SHN_TLS_COMMON, SHN_NS_COMMON, SHN_NS_UNDEF, SHN_FS_UNDEF, SHN_HP_EXTERN, SHN_HP_EXTHINT, SHN_HP_UNDEF_BIND_IMM, SHT_HP_OVLBITS, SHT_HP_DLKM, SHT_HP_COMDAT, SHT_HP_OBJDICT, SHT_HP_ANNOT, STB_HP_ALIAS): Define. 2006-03-10 Paul Brook * arm.h (EF_ARM_EABI_VER5): Define. 2006-03-06 Nathan Sidwell * m68k.h (EF_M68K_ISA_MASK, EF_M68K_ISA_A, EF_M68K_ISA_A_PLUS, EF_M68K_ISA_B, EF_M68K_ISA_C): Adjust. (EF_M68K_ISA_A_NODIV, EF_M68K_ISA_B_NOUSP): New. (EF_M68K_HW_DIV, EF_M68K_USP): Remove. (EF_M68K_MAC, EF_M68K_EMAC, EF_M68K_FLOAT): Adjust. (EF_M68K_EMAC_B): New. 2006-03-03 Bjoern Haase * avr.h (R_AVR_MS8_LDI,R_AVR_MS8_LDI_NEG): Add. (EF_AVR_LINKRELAX_PREPARED): Add. 2006-03-02 Ben Elliston Import from the GCC tree: 2006-03-01 Jakub Jelinek * dwarf2.h (DW_TAG_condition, DW_TAG_shared_type): New constants from DWARF 3. (DW_AT_description, DW_AT_binary_scale, DW_AT_decimal_scale, DW_AT_small, DW_AT_decimal_sign, DW_AT_digit_count, DW_AT_picture_string, DW_AT_mutable, DW_AT_threads_scaled, DW_AT_explicit, DW_AT_object_pointer, DW_AT_endianity, DW_AT_elemental, DW_AT_pure, DW_AT_recursive): New. (DW_OP_form_tls_address, DW_OP_call_frame_cfa, DW_OP_bit_piece): New. (DW_ATE_packed_decimal, DW_ATE_numeric_string, DW_ATE_edited, DW_ATE_signed_fixed, DW_ATE_unsigned_fixed): New. (DW_DS_unsigned, DW_DS_leading_overpunch, DW_DS_trailing_overpunch, DW_DS_leading_separate, DW_DS_trailing_separate): New. (DW_END_default, DW_END_big, DW_END_little): New. (DW_END_lo_user, DW_END_hi_user): Define. (DW_LNE_lo_user, DW_LNE_hi_user): Define. (DW_CFA_val_offset, DW_CFA_val_offset_sf, DW_CFA_val_expression): New. (DW_LANG_PLI, DW_LANG_ObjC, DW_LANG_ObjC_plus_plus, DW_LANG_UPC, DW_LANG_D): New. 2006-02-24 DJ Delorie * m32c.h: Add relax relocs. 2006-02-17 Shrirang Khisti Anil Paranjape Shilin Shakti * common.h (EM_XC16X): New entry for xc16x cpu. Sort other EM_* numbers into numerical order. * xc16x.h: New file. 2006-02-10 H.J. Lu PR binutils/2258 * internal.h (ELF_IS_SECTION_IN_SEGMENT_FILE): New. (ELF_IS_SECTION_IN_SEGMENT_MEMORY): Likewise. 2006-02-07 Nathan Sidwell * m68k.h (EF_CPU32, EF_M68000, EF_CFV4E): Rename to ... (EF_M68K_CPU32, EF_M68K_M68000, EF_M68K_CFV4E): ... here. (EF_M68K_ISA_MASK, EF_M68K_ISA_A, EF_M68K_M68K_ISA_A_PLUS, EF_M68K_ISA_B, EF_M68K_HW_DIV, EF_M68K_MAC_MASK, EF_M68K_MAC, EF_M68K_EMAC, EF_M68K_USP, EF_M68K_FLOAT): New. 2006-02-06 Steve Ellcey * ia64.h (SHF_IA_64_HP_TLS): New. 2006-01-18 Alexandre Oliva Introduce TLS descriptors for i386 and x86_64. * common.h (DT_TLSDESC_GOT, DT_TLSDESC_PLT): New. * i386.h (R_386_TLS_GOTDESC, R_386_TLS_DESC_CALL, R_386_TLS_DESC): New. * x86-64.h (R_X86_64_GOTPC32_TLSDESC, R_X86_64_TLSDESC_CALL, R_X86_64_TLSDESC): New. 2006-01-09 Mike Frysinger : * common.h (EM_ALTERA_NIOS2, EM_NIOS32) Define. 2005-12-16 Nathan Sidwell Second part of ms1 to mt renaming. * common.h (EM_MT): Renamed. * mt.h: Rename relocs, cpu & other defines. 2005-12-12 Paul Brook * arm.h (elf32_arm_get_eabi_attr_int): Add prototype. 2005-11-11 Nick Clifton PR 1150 * mips.h (STO_OPTIONAL): Define. (ELF_MIPS_IS_OPTIONAL): Define. 2005-09-30 Catherine Moore * bfin.h: New file. * common.h (EM_BLACKFIN): Define. 2005-10-08 Paul Brook * arm.h: Add prototypes for BFD object attribute routines. 2005-09-09 Richard Earnshaw * arm.h (SHT_ARM_PREEMPTMAP, SHT_ARM_ATTRIBUTES): New defines. 2005-08-09 John David Anglin * hppa.h (SHT_PARISC_DLKM, SHF_PARISC_WEAKORDER, PT_PARISC_WEAKORDER): New defines. 2005-08-04 John David Anglin * hppa.h (PF_HP_CODE, PF_HP_MODIFY, PF_HP_LAZYSWAP): Revise defines. (PF_HP_CODE_DEPR, PF_HP_MODIFY_DEPR, PF_HP_LAZYSWAP_DEPR): New deprecated defines. (DT_HP_EPLTREL, DT_HP_EPLTRELSZ, DT_HP_FILTERED, DT_HP_FILTER_TLS, DT_HP_COMPAT_FILTERED, DT_HP_LAZYLOAD, DT_HP_BIND_NOW_COUNT, DT_PLT, DT_PLT_SIZE, DT_DLT, DT_DLT_SIZE, DT_HP_BIND_DEPTH_FIRST, DT_HP_GST, DT_HP_SHLIB_FIXED, DT_HP_MERGE_SHLIB_SEG, DT_HP_NODELETE, DT_HP_GROUP, DT_HP_PROTECT_LINKAGE_TABLE, PT_HP_OPT_ANNOT, PT_HP_HSL_ANNOT, PT_HP_STACK, PT_HP_CORE_UTSNAME, NT_HP_COMPILER, NT_HP_COPYRIGHT, NT_HP_VERSION, NT_HP_SRCFILE_INFO, NT_HP_LINKER, NT_HP_INSTRUMENTED, NT_HP_UX_OPTIONS): Define. 2005-07-25 DJ Delorie * m32c.h: Add R_M32C_8, R_M32C_LO16, R_M32C_HI8, and R_M32C_HI16. 2005-07-25 Jan Hubicka * x86-64.h (SHN_X86_64_LCOMMON): New. (SHF_X86_64_LARGE): New. 2005-07-20 Kazuhiro Inaoka * m32r.h (R_M32R_REL32): Added. 2005-07-18 Ben Elliston * dwarf2.h (enum dwarf_type): Remove DW_AT_GNU_decimal_float. Replace with DW_ATE_decimal_float (now in DWARF 3). 2005-07-14 Jim Blandy Add support for Renesas M32C and M16C. * common.h (EM_M32C): New machine number. * m32c.h: New file. 2005-06-17 Jan Beulich * x86-64.h (elf_x86_64_reloc_type): Adjust comment for R_X86_64_GOTPCREL. Add R_X86_64_PC64, R_X86_64_GOTOFF64, and R_X86_64_GOTPC32. 2005-06-07 Aldy Hernandez Michael Snyder Stan Cox * common.h (EM_MS1): Define. * ms1.h: New file. 2005-05-31 Richard Henderson * alpha.h (LITUSE_ALPHA_JSRDIRECT): New. 2005-05-29 Richard Henderson * alpha.h (DT_ALPHA_PLTRO): New. 2005-05-19 Ben Elliston * dwarf2.h (enum dwarf_type): Assign DW_ATE_GNU_decimal_float from the user-defined encoding space pending inclusion in the standard. 2005-05-18 Zack Weinberg * arm.h: Make all #ifndef OLD_ARM_ABI blocks unconditional, delete all #ifdef OLD_ARM_ABI blocks. 2005-05-17 Zack Weinberg * arm.h: Import complete list of official relocation names and numbers from AAELF. Define FAKE_RELOCs for old names. Remove a few old names no longer used anywhere. 2005-05-14 Alan Modra * ppc.h (DT_PPC_GOT): Rename from DT_PPC_GLINK. 2005-05-11 Alan Modra * ppc.h (R_PPC_RELAX32, R_PPC_RELAX32PC, R_PPC_RELAX32_PLT, R_PPC_RELAX32PC_PLT) Adjust. (R_PPC_REL16, R_PPC_REL16_LO, R_PPC_REL16_HI, R_PPC_REL16_HA): Define. (DT_PPC_GLINK): Define. 2005-05-10 Nick Clifton * Update the address and phone number of the FSF organization in the GPL notices in the following files: alpha.h, arc.h, arm.h, avr.h, common.h, cr16c.h, cris.h, crx.h, d10v.h, d30v.h, dlx.h, dwarf.h, dwarf2.h, external.h, fr30.h, frv.h, h8.h, hppa.h, i370.h, i386.h, i860.h, i960.h, ia64.h, internal.h, ip2k.h, iq2000.h, m32r.h, m68hc11.h, m68k.h, mcore.h, mips.h, mmix.h, mn10200.h, mn10300.h, msp430.h, openrisc.h, or32.h, pj.h, ppc.h, ppc64.h, reloc-macros.h, s390.h, sh.h, sparc.h, v850.h, vax.h, x86-64.h, xstormy16.h, xtensa.h 2005-04-13 H.J. Lu Moved from ../ChangeLog 2004-10-27 Richard Earnshaw * arm.h: Add R_ARM_CALL and R_ARM_JUMP32. 2004-10-12 Paul Brook * arm.h (EF_ARM_EABI_VER4): Define. 2004-10-08 Daniel Jacobowitz * common.h (PT_SUNW_EH_FRAME): Define. * x86-64.h (SHT_X86_64_UNWIND): Define. 2004-10-07 Bob Wilson * xtensa.h (R_XTENSA_DIFF8, R_XTENSA_DIFF16, R_XTENSA_DIFF32, R_XTENSA_SLOT*_OP, R_XTENSA_SLOT*_ALT): New relocations. (XTENSA_PROP_SEC_NAME): Define. (property_table_entry): Add flags field. (XTENSA_PROP_*, GET_XTENSA_PROP_*, SET_XTENSA_PROP_*): Define. 2004-09-17 Paul Brook * arm.h: Remove R_ARM_STKCHK and R_ARM_THM_STKCHK. Add R_ARM_TARGET2, R_ARM_PREL31, R_ARM_GOT_ABS, R_ARM_GOT_PREL, R_ARM_GOT_BREL12, R_ARM_GOTOFF12 and R_ARM_GOTRELAX. 2004-09-13 Paul Brook * arm.h: Rename RELABS to TARGET1. 2004-05-11 Jakub Jelinek * common.h (PT_GNU_EH_FRAME, PT_GNU_STACK): Add comments. (PT_GNU_RELRO): Define. 2005-03-29 Daniel Jacobowitz Phil Blundell * arm.h: Add TLS relocations. 2005-03-23 Ben Elliston * dwarf.h: Merge with GCC's dwarf.h. 2005-03-18 C Jaipraash * m68k.h (EF_CFV4E): Define. 2005-03-17 Paul Brook Dan Jacobowitz Mark Mitchell * arm.h (PT_ARM_EXIDX): Define. 2005-03-02 Daniel Jacobowitz Joseph Myers * mips.h: Define MIPS TLS relocations. 2005-02-15 Nigel Stephens Maciej W. Rozycki * mips.h (R_MIPS16_GOT16): New reloc code. (R_MIPS16_CALL16): Likewise. (R_MIPS16_HI16): Likewise. (R_MIPS16_LO16): Likewise. (R_MIPS16_min): New fake reloc code. (R_MIPS16_max): Likewise. 2005-02-11 Maciej W. Rozycki * mips.h (R_MIPS_max): Use FAKE_RELOC to define. 2005-01-25 Alexandre Oliva 2004-12-10 Alexandre Oliva * frv.h: Add R_FRV_TLSMOFF. 2004-11-10 Alexandre Oliva * frv.h: Add TLS relocations. 2005-01-17 Nick Clifton * sh.h (EF_SH2A_SH4_NOFPU, EF_SH2A_SH3_NOFPU, EF_SH2A_SH4, EF_SH2A_SH3E): New flags. (EF_SH_BFD_TABLE): Add these new flags to the table. 2005-01-12 Alan Modra * ppc.h (R_PPC_RELAX32_PLT, R_PPC_RELAX32PC_PLT): Define. (R_PPC_RELAX32, R_PPC_RELAX32PC): Adjust value. 2004-12-22 Klaus Rudolph * avr.h (R_AVR_LDI, R_AVR_6, R_AVR_6_ADIW): New relocs. 2004-12-16 Richard Sandiford * v850.h (R_V850_LO16_SPLIT_OFFSET): New reloc. 2004-12-09 Ian Lance Taylor * mips.h (E_MIPS_MACH_9000): Define. 2004-11-04 Hans-Peter Nilsson * cris.h (EF_CRIS_VARIANT_MASK, EF_CRIS_VARIANT_ANY_V0_V10) (EF_CRIS_VARIANT_V32, EF_CRIS_VARIANT_COMMON_V10_V32): New macros. 2004-10-06 Eric Christopher * dwarf2.h: Sync with gcc dwarf2.h. Fix typo. 2004-10-01 Paul Brook * arm.h (SHT_ARM_EXIDX): Define. (ELF_STRING_ARM_unwind, ELF_STRING_ARM_unwind, ELF_STRING_ARM_unwind_once, ELF_STRING_ARM_unwind_info_once): Define. 2004-08-25 Dmitry Diky * msp430.h: Add new relocs. 2004-08-12 H.J. Lu * i386.h (R_386_USED_BY_INTEL_200): New. 2004-07-29 Alexandre Oliva Introduce SH2a support. 2004-02-18 Corinna Vinschen * sh.h (EF_SH2A_NOFPU): New. 2003-12-01 Michael Snyder * sh.h (EF_SH2A): New. 2004-07-27 Tomer Levi * crx.h: Add BFD_RELOC_CRX_SWITCH8, BFD_RELOC_CRX_SWITCH16, BFD_RELOC_CRX_SWITCH32. 2004-07-06 Tomer Levi * common.h (EM_CRX): Define. * crx.h: New file. 2004-06-25 Kazuhiro Inaoka * m32r.h: Add defintions of R_M32R_GOTOFF_HI_ULO, R_M32R_GOTOFF_HI_SLO and R_M32R_GOTOFF_LO. 2004-06-19 Alan Modra * common.h (ELF64_R_INFO): Warning fix. 2004-06-14 Chris Demetriou * mips.h (R_MIPS_PC32): Add back (undoing removal on 2004-04-24), with an updated comment. 2004-05-28 Andrew Stubbs * sh.h (EF_SH_HAS_DSP): Remove. (EF_SH_HAS_FP): Remove. (EF_SH_MERGE_MACH): Remove. (EF_SH4_NOFPU): Convert to decimal. (EF_SH4A_NOFPU): Likewise. (EF_SH4_NOMMU_NOFPU): Likewise. (EF_SH3_NOMMU): Add new macro. (EF_SH_BFD_TABLE): Likewise. (sh_find_elf_flags): Add prototype. (sh_elf_get_flags_from_mach): Likewise. 2004-04-24 Chris Demetriou * mips.h (R_MIPS_PC32, R_MIPS_PC64, R_MIPS_GNU_REL_LO16) (R_MIPS_GNU_REL_HI16): Remove. (R_MIPS_GNU_REL16_S2): Update comment. 2004-30-30 Galit Heller Tomer Levi * common.h (EM_CR): Define. * cr16c.h: New file. 2004-03-23 Paul Brook * arm.h (EF_ERM_BE8, EF_ARM_LE8, EF_ARM_EABI_VER3): Add. 2003-03-03 Andrew Stubbs * sh.h: Add EF_SH4_NOMMU_NOFPU. 2004-03-01 Richard Sandiford * frv.h (EF_FRV_CPU_FR405, EF_FRV_CPU_FR450): Define. 2004-01-28 Roland McGrath * common.h (AT_SECURE): New macro. 2004-01-21 Roland McGrath * common.h (AT_SUN_UID, AT_SUN_RUID, AT_SUN_GID): New macros. (AT_SUN_RGID, AT_SUN_LDELF, AT_SUN_LDSHDR, AT_SUN_LDNAME, AT_SUN_LPAGESZ, AT_SUN_PLATFORM, AT_SUN_HWCAP, AT_SUN_IFLUSH, AT_SUN_CPU, AT_SUN_EMUL_ENTRY, AT_SUN_EMUL_EXECFD, AT_SUN_EXECNAME) AT_SUN_MMU, AT_SUN_LDDATA): Likewise. 2004-01-17 Mark Kettenis * common.h (NT_OPENBSD_IDENT): Define. 2004-01-06 Alexandre Oliva 2003-09-18 Alexandre Oliva * frv.h (EF_FRV_FDPIC): New macro. (EF_FRV_PIC_FLAGS): Adjust. 2003-08-08 Alexandre Oliva * frv.h (R_FRV_FUNCDESC_VALUE, R_FRV_FUNCDESC_GOTOFF12, R_FRV_FUNCDESC_GOTOFFLO, R_FRV_FUNCDESC_GOTOFFHI, R_FRV_GOTOFF12, R_FRV_GOTOFFLO, R_FRV_GOTOFFHI): New. 2003-08-04 Alexandre Oliva * frv.h (R_FRV_GOT12, R_FRV_GOTHI, R_FRV_GOTLO, R_FRV_FUNCDESC, R_FRV_FUNCDESC_GOT12, R_FRV_FUNCDESC_GOTHI, R_FRV_FUNCDESC_GOTLO): New. For older changes see ChangeLog-9103 Local Variables: mode: change-log left-margin: 8 fill-column: 74 version-control: never End: Index: projects/clang350-import/contrib/binutils/include/elf/ppc.h =================================================================== --- projects/clang350-import/contrib/binutils/include/elf/ppc.h (revision 275748) +++ projects/clang350-import/contrib/binutils/include/elf/ppc.h (revision 275749) @@ -1,185 +1,187 @@ /* PPC ELF support for BFD. Copyright 1995, 1996, 1998, 2000, 2001, 2002, 2003, 2005 Free Software Foundation, Inc. By Michael Meissner, Cygnus Support, , from information in the System V Application Binary Interface, PowerPC Processor Supplement and the PowerPC Embedded Application Binary Interface (eabi). This file is part of BFD, the Binary File Descriptor library. This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin Street - Fifth Floor, Boston, MA 02110-1301, USA. */ /* This file holds definitions specific to the PPC ELF ABI. Note that most of this is not actually implemented by BFD. */ #ifndef _ELF_PPC_H #define _ELF_PPC_H #include "elf/reloc-macros.h" /* Relocations. */ START_RELOC_NUMBERS (elf_ppc_reloc_type) RELOC_NUMBER (R_PPC_NONE, 0) RELOC_NUMBER (R_PPC_ADDR32, 1) RELOC_NUMBER (R_PPC_ADDR24, 2) RELOC_NUMBER (R_PPC_ADDR16, 3) RELOC_NUMBER (R_PPC_ADDR16_LO, 4) RELOC_NUMBER (R_PPC_ADDR16_HI, 5) RELOC_NUMBER (R_PPC_ADDR16_HA, 6) RELOC_NUMBER (R_PPC_ADDR14, 7) RELOC_NUMBER (R_PPC_ADDR14_BRTAKEN, 8) RELOC_NUMBER (R_PPC_ADDR14_BRNTAKEN, 9) RELOC_NUMBER (R_PPC_REL24, 10) RELOC_NUMBER (R_PPC_REL14, 11) RELOC_NUMBER (R_PPC_REL14_BRTAKEN, 12) RELOC_NUMBER (R_PPC_REL14_BRNTAKEN, 13) RELOC_NUMBER (R_PPC_GOT16, 14) RELOC_NUMBER (R_PPC_GOT16_LO, 15) RELOC_NUMBER (R_PPC_GOT16_HI, 16) RELOC_NUMBER (R_PPC_GOT16_HA, 17) RELOC_NUMBER (R_PPC_PLTREL24, 18) RELOC_NUMBER (R_PPC_COPY, 19) RELOC_NUMBER (R_PPC_GLOB_DAT, 20) RELOC_NUMBER (R_PPC_JMP_SLOT, 21) RELOC_NUMBER (R_PPC_RELATIVE, 22) RELOC_NUMBER (R_PPC_LOCAL24PC, 23) RELOC_NUMBER (R_PPC_UADDR32, 24) RELOC_NUMBER (R_PPC_UADDR16, 25) RELOC_NUMBER (R_PPC_REL32, 26) RELOC_NUMBER (R_PPC_PLT32, 27) RELOC_NUMBER (R_PPC_PLTREL32, 28) RELOC_NUMBER (R_PPC_PLT16_LO, 29) RELOC_NUMBER (R_PPC_PLT16_HI, 30) RELOC_NUMBER (R_PPC_PLT16_HA, 31) RELOC_NUMBER (R_PPC_SDAREL16, 32) RELOC_NUMBER (R_PPC_SECTOFF, 33) RELOC_NUMBER (R_PPC_SECTOFF_LO, 34) RELOC_NUMBER (R_PPC_SECTOFF_HI, 35) RELOC_NUMBER (R_PPC_SECTOFF_HA, 36) RELOC_NUMBER (R_PPC_ADDR30, 37) /* Relocs added to support TLS. */ RELOC_NUMBER (R_PPC_TLS, 67) RELOC_NUMBER (R_PPC_DTPMOD32, 68) RELOC_NUMBER (R_PPC_TPREL16, 69) RELOC_NUMBER (R_PPC_TPREL16_LO, 70) RELOC_NUMBER (R_PPC_TPREL16_HI, 71) RELOC_NUMBER (R_PPC_TPREL16_HA, 72) RELOC_NUMBER (R_PPC_TPREL32, 73) RELOC_NUMBER (R_PPC_DTPREL16, 74) RELOC_NUMBER (R_PPC_DTPREL16_LO, 75) RELOC_NUMBER (R_PPC_DTPREL16_HI, 76) RELOC_NUMBER (R_PPC_DTPREL16_HA, 77) RELOC_NUMBER (R_PPC_DTPREL32, 78) RELOC_NUMBER (R_PPC_GOT_TLSGD16, 79) RELOC_NUMBER (R_PPC_GOT_TLSGD16_LO, 80) RELOC_NUMBER (R_PPC_GOT_TLSGD16_HI, 81) RELOC_NUMBER (R_PPC_GOT_TLSGD16_HA, 82) RELOC_NUMBER (R_PPC_GOT_TLSLD16, 83) RELOC_NUMBER (R_PPC_GOT_TLSLD16_LO, 84) RELOC_NUMBER (R_PPC_GOT_TLSLD16_HI, 85) RELOC_NUMBER (R_PPC_GOT_TLSLD16_HA, 86) RELOC_NUMBER (R_PPC_GOT_TPREL16, 87) RELOC_NUMBER (R_PPC_GOT_TPREL16_LO, 88) RELOC_NUMBER (R_PPC_GOT_TPREL16_HI, 89) RELOC_NUMBER (R_PPC_GOT_TPREL16_HA, 90) RELOC_NUMBER (R_PPC_GOT_DTPREL16, 91) RELOC_NUMBER (R_PPC_GOT_DTPREL16_LO, 92) RELOC_NUMBER (R_PPC_GOT_DTPREL16_HI, 93) RELOC_NUMBER (R_PPC_GOT_DTPREL16_HA, 94) + RELOC_NUMBER (R_PPC_TLSGD, 95) + RELOC_NUMBER (R_PPC_TLSLD, 96) /* The remaining relocs are from the Embedded ELF ABI, and are not in the SVR4 ELF ABI. */ RELOC_NUMBER (R_PPC_EMB_NADDR32, 101) RELOC_NUMBER (R_PPC_EMB_NADDR16, 102) RELOC_NUMBER (R_PPC_EMB_NADDR16_LO, 103) RELOC_NUMBER (R_PPC_EMB_NADDR16_HI, 104) RELOC_NUMBER (R_PPC_EMB_NADDR16_HA, 105) RELOC_NUMBER (R_PPC_EMB_SDAI16, 106) RELOC_NUMBER (R_PPC_EMB_SDA2I16, 107) RELOC_NUMBER (R_PPC_EMB_SDA2REL, 108) RELOC_NUMBER (R_PPC_EMB_SDA21, 109) RELOC_NUMBER (R_PPC_EMB_MRKREF, 110) RELOC_NUMBER (R_PPC_EMB_RELSEC16, 111) RELOC_NUMBER (R_PPC_EMB_RELST_LO, 112) RELOC_NUMBER (R_PPC_EMB_RELST_HI, 113) RELOC_NUMBER (R_PPC_EMB_RELST_HA, 114) RELOC_NUMBER (R_PPC_EMB_BIT_FLD, 115) RELOC_NUMBER (R_PPC_EMB_RELSDA, 116) /* Fake relocations for branch stubs, only used internally by ld. */ #define R_PPC_RELAX32 245 #define R_PPC_RELAX32PC 246 #define R_PPC_RELAX32_PLT 247 #define R_PPC_RELAX32PC_PLT 248 /* These are GNU extensions used in PIC code sequences. */ RELOC_NUMBER (R_PPC_REL16, 249) RELOC_NUMBER (R_PPC_REL16_LO, 250) RELOC_NUMBER (R_PPC_REL16_HI, 251) RELOC_NUMBER (R_PPC_REL16_HA, 252) /* These are GNU extensions to enable C++ vtable garbage collection. */ RELOC_NUMBER (R_PPC_GNU_VTINHERIT, 253) RELOC_NUMBER (R_PPC_GNU_VTENTRY, 254) /* This is a phony reloc to handle any old fashioned TOC16 references that may still be in object files. */ RELOC_NUMBER (R_PPC_TOC16, 255) END_RELOC_NUMBERS (R_PPC_max) #define IS_PPC_TLS_RELOC(R) \ ((R) >= R_PPC_TLS && (R) <= R_PPC_GOT_DTPREL16_HA) /* Specify the value of _GLOBAL_OFFSET_TABLE_. */ #define DT_PPC_GOT DT_LOPROC /* Processor specific flags for the ELF header e_flags field. */ #define EF_PPC_EMB 0x80000000 /* PowerPC embedded flag. */ #define EF_PPC_RELOCATABLE 0x00010000 /* PowerPC -mrelocatable flag. */ #define EF_PPC_RELOCATABLE_LIB 0x00008000 /* PowerPC -mrelocatable-lib flag. */ /* Processor specific section headers, sh_type field. */ #define SHT_ORDERED SHT_HIPROC /* Link editor is to sort the \ entries in this section \ based on the address \ specified in the associated \ symbol table entry. */ /* Processor specific section flags, sh_flags field. */ #define SHF_EXCLUDE 0x80000000 /* Link editor is to exclude \ this section from executable \ and shared objects that it \ builds when those objects \ are not to be furhter \ relocated. */ /* Object attribute tags. */ enum { /* 0-3 are generic. */ Tag_GNU_Power_ABI_FP = 4, /* Value 1 for hard-float, 2 for soft-float; 0 for not tagged or not using any ABIs affected by the differences. */ }; #endif /* _ELF_PPC_H */ Index: projects/clang350-import/contrib/binutils/include/elf/ppc64.h =================================================================== --- projects/clang350-import/contrib/binutils/include/elf/ppc64.h (revision 275748) +++ projects/clang350-import/contrib/binutils/include/elf/ppc64.h (revision 275749) @@ -1,156 +1,158 @@ /* PPC64 ELF support for BFD. - Copyright 2003 Free Software Foundation, Inc. + Copyright 2003, 2005, 2009 Free Software Foundation, Inc. This file is part of BFD, the Binary File Descriptor library. This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin Street - Fifth Floor, Boston, MA 02110-1301, USA. */ #ifndef _ELF_PPC64_H #define _ELF_PPC64_H #include "elf/reloc-macros.h" /* Relocations. */ START_RELOC_NUMBERS (elf_ppc64_reloc_type) RELOC_NUMBER (R_PPC64_NONE, 0) RELOC_NUMBER (R_PPC64_ADDR32, 1) RELOC_NUMBER (R_PPC64_ADDR24, 2) RELOC_NUMBER (R_PPC64_ADDR16, 3) RELOC_NUMBER (R_PPC64_ADDR16_LO, 4) RELOC_NUMBER (R_PPC64_ADDR16_HI, 5) RELOC_NUMBER (R_PPC64_ADDR16_HA, 6) RELOC_NUMBER (R_PPC64_ADDR14, 7) RELOC_NUMBER (R_PPC64_ADDR14_BRTAKEN, 8) RELOC_NUMBER (R_PPC64_ADDR14_BRNTAKEN, 9) RELOC_NUMBER (R_PPC64_REL24, 10) RELOC_NUMBER (R_PPC64_REL14, 11) RELOC_NUMBER (R_PPC64_REL14_BRTAKEN, 12) RELOC_NUMBER (R_PPC64_REL14_BRNTAKEN, 13) RELOC_NUMBER (R_PPC64_GOT16, 14) RELOC_NUMBER (R_PPC64_GOT16_LO, 15) RELOC_NUMBER (R_PPC64_GOT16_HI, 16) RELOC_NUMBER (R_PPC64_GOT16_HA, 17) /* 18 unused. 32-bit reloc is R_PPC_PLTREL24. */ RELOC_NUMBER (R_PPC64_COPY, 19) RELOC_NUMBER (R_PPC64_GLOB_DAT, 20) RELOC_NUMBER (R_PPC64_JMP_SLOT, 21) RELOC_NUMBER (R_PPC64_RELATIVE, 22) /* 23 unused. 32-bit reloc is R_PPC_LOCAL24PC. */ RELOC_NUMBER (R_PPC64_UADDR32, 24) RELOC_NUMBER (R_PPC64_UADDR16, 25) RELOC_NUMBER (R_PPC64_REL32, 26) RELOC_NUMBER (R_PPC64_PLT32, 27) RELOC_NUMBER (R_PPC64_PLTREL32, 28) RELOC_NUMBER (R_PPC64_PLT16_LO, 29) RELOC_NUMBER (R_PPC64_PLT16_HI, 30) RELOC_NUMBER (R_PPC64_PLT16_HA, 31) /* 32 unused. 32-bit reloc is R_PPC_SDAREL16. */ RELOC_NUMBER (R_PPC64_SECTOFF, 33) RELOC_NUMBER (R_PPC64_SECTOFF_LO, 34) RELOC_NUMBER (R_PPC64_SECTOFF_HI, 35) RELOC_NUMBER (R_PPC64_SECTOFF_HA, 36) RELOC_NUMBER (R_PPC64_REL30, 37) RELOC_NUMBER (R_PPC64_ADDR64, 38) RELOC_NUMBER (R_PPC64_ADDR16_HIGHER, 39) RELOC_NUMBER (R_PPC64_ADDR16_HIGHERA, 40) RELOC_NUMBER (R_PPC64_ADDR16_HIGHEST, 41) RELOC_NUMBER (R_PPC64_ADDR16_HIGHESTA, 42) RELOC_NUMBER (R_PPC64_UADDR64, 43) RELOC_NUMBER (R_PPC64_REL64, 44) RELOC_NUMBER (R_PPC64_PLT64, 45) RELOC_NUMBER (R_PPC64_PLTREL64, 46) RELOC_NUMBER (R_PPC64_TOC16, 47) RELOC_NUMBER (R_PPC64_TOC16_LO, 48) RELOC_NUMBER (R_PPC64_TOC16_HI, 49) RELOC_NUMBER (R_PPC64_TOC16_HA, 50) RELOC_NUMBER (R_PPC64_TOC, 51) RELOC_NUMBER (R_PPC64_PLTGOT16, 52) RELOC_NUMBER (R_PPC64_PLTGOT16_LO, 53) RELOC_NUMBER (R_PPC64_PLTGOT16_HI, 54) RELOC_NUMBER (R_PPC64_PLTGOT16_HA, 55) /* The following relocs were added in the 64-bit PowerPC ELF ABI revision 1.2. */ RELOC_NUMBER (R_PPC64_ADDR16_DS, 56) RELOC_NUMBER (R_PPC64_ADDR16_LO_DS, 57) RELOC_NUMBER (R_PPC64_GOT16_DS, 58) RELOC_NUMBER (R_PPC64_GOT16_LO_DS, 59) RELOC_NUMBER (R_PPC64_PLT16_LO_DS, 60) RELOC_NUMBER (R_PPC64_SECTOFF_DS, 61) RELOC_NUMBER (R_PPC64_SECTOFF_LO_DS, 62) RELOC_NUMBER (R_PPC64_TOC16_DS, 63) RELOC_NUMBER (R_PPC64_TOC16_LO_DS, 64) RELOC_NUMBER (R_PPC64_PLTGOT16_DS, 65) RELOC_NUMBER (R_PPC64_PLTGOT16_LO_DS, 66) /* Relocs added to support TLS. PowerPC64 ELF ABI revision 1.5. */ RELOC_NUMBER (R_PPC64_TLS, 67) RELOC_NUMBER (R_PPC64_DTPMOD64, 68) RELOC_NUMBER (R_PPC64_TPREL16, 69) RELOC_NUMBER (R_PPC64_TPREL16_LO, 70) RELOC_NUMBER (R_PPC64_TPREL16_HI, 71) RELOC_NUMBER (R_PPC64_TPREL16_HA, 72) RELOC_NUMBER (R_PPC64_TPREL64, 73) RELOC_NUMBER (R_PPC64_DTPREL16, 74) RELOC_NUMBER (R_PPC64_DTPREL16_LO, 75) RELOC_NUMBER (R_PPC64_DTPREL16_HI, 76) RELOC_NUMBER (R_PPC64_DTPREL16_HA, 77) RELOC_NUMBER (R_PPC64_DTPREL64, 78) RELOC_NUMBER (R_PPC64_GOT_TLSGD16, 79) RELOC_NUMBER (R_PPC64_GOT_TLSGD16_LO, 80) RELOC_NUMBER (R_PPC64_GOT_TLSGD16_HI, 81) RELOC_NUMBER (R_PPC64_GOT_TLSGD16_HA, 82) RELOC_NUMBER (R_PPC64_GOT_TLSLD16, 83) RELOC_NUMBER (R_PPC64_GOT_TLSLD16_LO, 84) RELOC_NUMBER (R_PPC64_GOT_TLSLD16_HI, 85) RELOC_NUMBER (R_PPC64_GOT_TLSLD16_HA, 86) RELOC_NUMBER (R_PPC64_GOT_TPREL16_DS, 87) RELOC_NUMBER (R_PPC64_GOT_TPREL16_LO_DS, 88) RELOC_NUMBER (R_PPC64_GOT_TPREL16_HI, 89) RELOC_NUMBER (R_PPC64_GOT_TPREL16_HA, 90) RELOC_NUMBER (R_PPC64_GOT_DTPREL16_DS, 91) RELOC_NUMBER (R_PPC64_GOT_DTPREL16_LO_DS, 92) RELOC_NUMBER (R_PPC64_GOT_DTPREL16_HI, 93) RELOC_NUMBER (R_PPC64_GOT_DTPREL16_HA, 94) RELOC_NUMBER (R_PPC64_TPREL16_DS, 95) RELOC_NUMBER (R_PPC64_TPREL16_LO_DS, 96) RELOC_NUMBER (R_PPC64_TPREL16_HIGHER, 97) RELOC_NUMBER (R_PPC64_TPREL16_HIGHERA, 98) RELOC_NUMBER (R_PPC64_TPREL16_HIGHEST, 99) RELOC_NUMBER (R_PPC64_TPREL16_HIGHESTA, 100) RELOC_NUMBER (R_PPC64_DTPREL16_DS, 101) RELOC_NUMBER (R_PPC64_DTPREL16_LO_DS, 102) RELOC_NUMBER (R_PPC64_DTPREL16_HIGHER, 103) RELOC_NUMBER (R_PPC64_DTPREL16_HIGHERA, 104) RELOC_NUMBER (R_PPC64_DTPREL16_HIGHEST, 105) RELOC_NUMBER (R_PPC64_DTPREL16_HIGHESTA, 106) + RELOC_NUMBER (R_PPC64_TLSGD, 107) + RELOC_NUMBER (R_PPC64_TLSLD, 108) /* These are GNU extensions to enable C++ vtable garbage collection. */ RELOC_NUMBER (R_PPC64_GNU_VTINHERIT, 253) RELOC_NUMBER (R_PPC64_GNU_VTENTRY, 254) END_RELOC_NUMBERS (R_PPC64_max) #define IS_PPC64_TLS_RELOC(R) \ ((R) >= R_PPC64_TLS && (R) <= R_PPC64_DTPREL16_HIGHESTA) /* Specify the start of the .glink section. */ #define DT_PPC64_GLINK DT_LOPROC /* Specify the start and size of the .opd section. */ #define DT_PPC64_OPD (DT_LOPROC + 1) #define DT_PPC64_OPDSZ (DT_LOPROC + 2) #endif /* _ELF_PPC64_H */ Index: projects/clang350-import/contrib/binutils =================================================================== --- projects/clang350-import/contrib/binutils (revision 275748) +++ projects/clang350-import/contrib/binutils (revision 275749) Property changes on: projects/clang350-import/contrib/binutils ___________________________________________________________________ Modified: svn:mergeinfo ## -0,0 +0,1 ## Merged /head/contrib/binutils:r275623-275748 Index: projects/clang350-import/contrib/llvm =================================================================== --- projects/clang350-import/contrib/llvm (revision 275748) +++ projects/clang350-import/contrib/llvm (revision 275749) Property changes on: projects/clang350-import/contrib/llvm ___________________________________________________________________ Modified: svn:mergeinfo ## -0,0 +0,1 ## Merged /head/contrib/llvm:r275685-275748 Index: projects/clang350-import/etc/mtree/BSD.tests.dist =================================================================== --- projects/clang350-import/etc/mtree/BSD.tests.dist (revision 275748) +++ projects/clang350-import/etc/mtree/BSD.tests.dist (revision 275749) @@ -1,412 +1,414 @@ # $FreeBSD$ # # Please see the file src/etc/mtree/README before making changes to this file. # /set type=dir uname=root gname=wheel mode=0755 . include atf-c .. atf-c++ .. .. share atf .. doc atf .. pjdfstest .. .. .. tests bin chown .. date .. mv .. pax .. pkill .. sh builtins .. errors .. execution .. expansion .. parameters .. parser .. set-e .. .. sleep .. test .. .. cddl lib .. sbin .. usr.bin .. usr.sbin .. .. etc .. games .. gnu lib .. usr.bin diff .. .. .. lib atf libatf-c detail .. .. libatf-c++ detail .. .. test-programs .. .. libc c063 .. db .. gen execve .. posix_spawn .. .. hash data .. .. inet .. locale .. net getaddrinfo data .. .. .. regex data .. .. ssp .. stdio .. stdlib .. string .. sys .. time .. tls dso .. .. termios .. ttyio .. .. libcrypt .. libmp .. libnv .. libpam .. libproc .. librt .. libthr dlopen .. .. libutil .. msun .. .. libexec atf atf-check .. atf-sh .. .. rtld-elf .. .. sbin dhclient .. devd .. growfs .. mdconfig .. .. secure lib .. libexec .. usr.bin .. usr.sbin .. .. share examples tests atf .. plain .. .. .. .. sys kern .. netinet .. + opencrypto + .. pjdfstest chflags .. chmod .. chown .. ftruncate .. granular .. link .. mkdir .. mkfifo .. mknod .. open .. rename .. rmdir .. symlink .. truncate .. unlink .. .. .. usr.bin apply .. basename .. bmake archives fmt_44bsd .. fmt_44bsd_mod .. fmt_oldbsd .. .. basic t0 .. t1 .. t2 .. t3 .. .. execution ellipsis .. empty .. joberr .. plus .. .. shell builtin .. meta .. path .. path_select .. replace .. select .. .. suffixes basic .. src_wild1 .. src_wild2 .. .. syntax directive-t0 .. enl .. funny-targets .. semi .. .. sysmk t0 2 1 .. .. mk .. .. t1 2 1 .. .. mk .. .. t2 2 1 .. .. mk .. .. .. variables modifier_M .. modifier_t .. opt_V .. t0 .. .. .. calendar .. cmp .. comm .. cut .. dirname .. file2c .. grep .. gzip .. join .. jot .. lastcomm .. m4 .. mkimg .. ncal .. printf .. sed regress.multitest.out .. .. timeout .. tr .. truncate .. units .. uudecode .. uuencode .. xargs .. yacc yacc .. .. .. usr.sbin etcupdate .. newsyslog .. nmtree .. pw .. sa .. .. .. .. # vim: set expandtab ts=4 sw=4: Index: projects/clang350-import/etc =================================================================== --- projects/clang350-import/etc (revision 275748) +++ projects/clang350-import/etc (revision 275749) Property changes on: projects/clang350-import/etc ___________________________________________________________________ Modified: svn:mergeinfo ## -0,0 +0,1 ## Merged /head/etc:r275685-275748 Index: projects/clang350-import/gnu/usr.bin/binutils/libbfd/bfd.h =================================================================== --- projects/clang350-import/gnu/usr.bin/binutils/libbfd/bfd.h (revision 275748) +++ projects/clang350-import/gnu/usr.bin/binutils/libbfd/bfd.h (revision 275749) @@ -1,5405 +1,5410 @@ /* $FreeBSD$ */ /* DO NOT EDIT! -*- buffer-read-only: t -*- This file is automatically generated from "bfd-in.h", "init.c", "opncls.c", "libbfd.c", "bfdio.c", "bfdwin.c", "section.c", "archures.c", "reloc.c", "syms.c", "bfd.c", "archive.c", "corefile.c", "targets.c", "format.c", "linker.c" and "simple.c". Run "make headers" in your build bfd/ to regenerate. */ /* Main header file for the bfd library -- portable access to object files. Copyright 1990, 1991, 1992, 1993, 1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007 Free Software Foundation, Inc. Contributed by Cygnus Support. This file is part of BFD, the Binary File Descriptor library. This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin Street - Fifth Floor, Boston, MA 02110-1301, USA. */ #ifndef __BFD_H_SEEN__ #define __BFD_H_SEEN__ #ifdef __cplusplus extern "C" { #endif #include "ansidecl.h" #include "symcat.h" #if defined (__STDC__) || defined (ALMOST_STDC) || defined (HAVE_STRINGIZE) #ifndef SABER /* This hack is to avoid a problem with some strict ANSI C preprocessors. The problem is, "32_" is not a valid preprocessing token, and we don't want extra underscores (e.g., "nlm_32_"). The XCONCAT2 macro will cause the inner CONCAT2 macros to be evaluated first, producing still-valid pp-tokens. Then the final concatenation can be done. */ #undef CONCAT4 #define CONCAT4(a,b,c,d) XCONCAT2(CONCAT2(a,b),CONCAT2(c,d)) #endif #endif /* This is a utility macro to handle the situation where the code wants to place a constant string into the code, followed by a comma and then the length of the string. Doing this by hand is error prone, so using this macro is safer. The macro will also safely handle the case where a NULL is passed as the arg. */ #define STRING_COMMA_LEN(STR) (STR), ((STR) ? sizeof (STR) - 1 : 0) /* Unfortunately it is not possible to use the STRING_COMMA_LEN macro to create the arguments to another macro, since the preprocessor will mis-count the number of arguments to the outer macro (by not evaluating STRING_COMMA_LEN and so missing the comma). This is a problem for example when trying to use STRING_COMMA_LEN to build the arguments to the strncmp() macro. Hence this alternative definition of strncmp is provided here. Note - these macros do NOT work if STR2 is not a constant string. */ #define CONST_STRNEQ(STR1,STR2) (strncmp ((STR1), (STR2), sizeof (STR2) - 1) == 0) /* strcpy() can have a similar problem, but since we know we are copying a constant string, we can use memcpy which will be faster since there is no need to check for a NUL byte inside STR. We can also save time if we do not need to copy the terminating NUL. */ #define LITMEMCPY(DEST,STR2) memcpy ((DEST), (STR2), sizeof (STR2) - 1) #define LITSTRCPY(DEST,STR2) memcpy ((DEST), (STR2), sizeof (STR2)) /* The word size used by BFD on the host. This may be 64 with a 32 bit target if the host is 64 bit, or if other 64 bit targets have been selected with --enable-targets, or if --enable-64-bit-bfd. */ #define BFD_ARCH_SIZE 64 #if defined(__i386__) || defined(__powerpc__) || defined(__arm__) || defined(__mips__) #define BFD_HOST_64BIT_LONG 0 #define BFD_HOST_64_BIT long long #define BFD_HOST_U_64_BIT unsigned long long #elif defined(__alpha__) || defined(__sparc64__) || defined(__amd64__) #define BFD_HOST_64BIT_LONG 1 #define BFD_HOST_64_BIT long #define BFD_HOST_U_64_BIT unsigned long #else #error Unsupported architecture/platform. #endif /* 64-bit host */ #define BFD_HOST_LONG_LONG 1 typedef BFD_HOST_64_BIT bfd_int64_t; typedef BFD_HOST_U_64_BIT bfd_uint64_t; #if BFD_ARCH_SIZE >= 64 #define BFD64 #endif #ifndef INLINE #if __GNUC__ >= 2 #define INLINE __inline__ #else #define INLINE #endif #endif /* Forward declaration. */ typedef struct bfd bfd; /* Boolean type used in bfd. Too many systems define their own versions of "boolean" for us to safely typedef a "boolean" of our own. Using an enum for "bfd_boolean" has its own set of problems, with strange looking casts required to avoid warnings on some older compilers. Thus we just use an int. General rule: Functions which are bfd_boolean return TRUE on success and FALSE on failure (unless they're a predicate). */ typedef int bfd_boolean; #undef FALSE #undef TRUE #define FALSE 0 #define TRUE 1 #ifdef BFD64 #ifndef BFD_HOST_64_BIT #error No 64 bit integer type available #endif /* ! defined (BFD_HOST_64_BIT) */ typedef BFD_HOST_U_64_BIT bfd_vma; typedef BFD_HOST_64_BIT bfd_signed_vma; typedef BFD_HOST_U_64_BIT bfd_size_type; typedef BFD_HOST_U_64_BIT symvalue; #ifndef fprintf_vma #if BFD_HOST_64BIT_LONG #define sprintf_vma(s,x) sprintf (s, "%016lx", x) #define fprintf_vma(f,x) fprintf (f, "%016lx", x) #else #define _bfd_int64_low(x) ((unsigned long) (((x) & 0xffffffff))) #define _bfd_int64_high(x) ((unsigned long) (((x) >> 32) & 0xffffffff)) #define fprintf_vma(s,x) \ fprintf ((s), "%08lx%08lx", _bfd_int64_high (x), _bfd_int64_low (x)) #define sprintf_vma(s,x) \ sprintf ((s), "%08lx%08lx", _bfd_int64_high (x), _bfd_int64_low (x)) #endif #endif #else /* not BFD64 */ /* Represent a target address. Also used as a generic unsigned type which is guaranteed to be big enough to hold any arithmetic types we need to deal with. */ typedef unsigned long bfd_vma; /* A generic signed type which is guaranteed to be big enough to hold any arithmetic types we need to deal with. Can be assumed to be compatible with bfd_vma in the same way that signed and unsigned ints are compatible (as parameters, in assignment, etc). */ typedef long bfd_signed_vma; typedef unsigned long symvalue; typedef unsigned long bfd_size_type; /* Print a bfd_vma x on stream s. */ #define fprintf_vma(s,x) fprintf (s, "%08lx", x) #define sprintf_vma(s,x) sprintf (s, "%08lx", x) #endif /* not BFD64 */ #define HALF_BFD_SIZE_TYPE \ (((bfd_size_type) 1) << (8 * sizeof (bfd_size_type) / 2)) #ifndef BFD_HOST_64_BIT /* Fall back on a 32 bit type. The idea is to make these types always available for function return types, but in the case that BFD_HOST_64_BIT is undefined such a function should abort or otherwise signal an error. */ typedef bfd_signed_vma bfd_int64_t; typedef bfd_vma bfd_uint64_t; #endif /* An offset into a file. BFD always uses the largest possible offset based on the build time availability of fseek, fseeko, or fseeko64. */ typedef BFD_HOST_64_BIT file_ptr; typedef unsigned BFD_HOST_64_BIT ufile_ptr; extern void bfd_sprintf_vma (bfd *, char *, bfd_vma); extern void bfd_fprintf_vma (bfd *, void *, bfd_vma); #define printf_vma(x) fprintf_vma(stdout,x) #define bfd_printf_vma(abfd,x) bfd_fprintf_vma (abfd,stdout,x) typedef unsigned int flagword; /* 32 bits of flags */ typedef unsigned char bfd_byte; /* File formats. */ typedef enum bfd_format { bfd_unknown = 0, /* File format is unknown. */ bfd_object, /* Linker/assembler/compiler output. */ bfd_archive, /* Object archive file. */ bfd_core, /* Core dump. */ bfd_type_end /* Marks the end; don't use it! */ } bfd_format; /* Values that may appear in the flags field of a BFD. These also appear in the object_flags field of the bfd_target structure, where they indicate the set of flags used by that backend (not all flags are meaningful for all object file formats) (FIXME: at the moment, the object_flags values have mostly just been copied from backend to another, and are not necessarily correct). */ /* No flags. */ #define BFD_NO_FLAGS 0x00 /* BFD contains relocation entries. */ #define HAS_RELOC 0x01 /* BFD is directly executable. */ #define EXEC_P 0x02 /* BFD has line number information (basically used for F_LNNO in a COFF header). */ #define HAS_LINENO 0x04 /* BFD has debugging information. */ #define HAS_DEBUG 0x08 /* BFD has symbols. */ #define HAS_SYMS 0x10 /* BFD has local symbols (basically used for F_LSYMS in a COFF header). */ #define HAS_LOCALS 0x20 /* BFD is a dynamic object. */ #define DYNAMIC 0x40 /* Text section is write protected (if D_PAGED is not set, this is like an a.out NMAGIC file) (the linker sets this by default, but clears it for -r or -N). */ #define WP_TEXT 0x80 /* BFD is dynamically paged (this is like an a.out ZMAGIC file) (the linker sets this by default, but clears it for -r or -n or -N). */ #define D_PAGED 0x100 /* BFD is relaxable (this means that bfd_relax_section may be able to do something) (sometimes bfd_relax_section can do something even if this is not set). */ #define BFD_IS_RELAXABLE 0x200 /* This may be set before writing out a BFD to request using a traditional format. For example, this is used to request that when writing out an a.out object the symbols not be hashed to eliminate duplicates. */ #define BFD_TRADITIONAL_FORMAT 0x400 /* This flag indicates that the BFD contents are actually cached in memory. If this is set, iostream points to a bfd_in_memory struct. */ #define BFD_IN_MEMORY 0x800 /* The sections in this BFD specify a memory page. */ #define HAS_LOAD_PAGE 0x1000 /* This BFD has been created by the linker and doesn't correspond to any input file. */ #define BFD_LINKER_CREATED 0x2000 /* Symbols and relocation. */ /* A count of carsyms (canonical archive symbols). */ typedef unsigned long symindex; /* How to perform a relocation. */ typedef const struct reloc_howto_struct reloc_howto_type; #define BFD_NO_MORE_SYMBOLS ((symindex) ~0) /* General purpose part of a symbol X; target specific parts are in libcoff.h, libaout.h, etc. */ #define bfd_get_section(x) ((x)->section) #define bfd_get_output_section(x) ((x)->section->output_section) #define bfd_set_section(x,y) ((x)->section) = (y) #define bfd_asymbol_base(x) ((x)->section->vma) #define bfd_asymbol_value(x) (bfd_asymbol_base(x) + (x)->value) #define bfd_asymbol_name(x) ((x)->name) /*Perhaps future: #define bfd_asymbol_bfd(x) ((x)->section->owner)*/ #define bfd_asymbol_bfd(x) ((x)->the_bfd) #define bfd_asymbol_flavour(x) (bfd_asymbol_bfd(x)->xvec->flavour) /* A canonical archive symbol. */ /* This is a type pun with struct ranlib on purpose! */ typedef struct carsym { char *name; file_ptr file_offset; /* Look here to find the file. */ } carsym; /* To make these you call a carsymogen. */ /* Used in generating armaps (archive tables of contents). Perhaps just a forward definition would do? */ struct orl /* Output ranlib. */ { char **name; /* Symbol name. */ union { file_ptr pos; bfd *abfd; } u; /* bfd* or file position. */ int namidx; /* Index into string table. */ }; /* Linenumber stuff. */ typedef struct lineno_cache_entry { unsigned int line_number; /* Linenumber from start of function. */ union { struct bfd_symbol *sym; /* Function name. */ bfd_vma offset; /* Offset into section. */ } u; } alent; /* Object and core file sections. */ #define align_power(addr, align) \ (((addr) + ((bfd_vma) 1 << (align)) - 1) & ((bfd_vma) -1 << (align))) typedef struct bfd_section *sec_ptr; #define bfd_get_section_name(bfd, ptr) ((ptr)->name + 0) #define bfd_get_section_vma(bfd, ptr) ((ptr)->vma + 0) #define bfd_get_section_lma(bfd, ptr) ((ptr)->lma + 0) #define bfd_get_section_alignment(bfd, ptr) ((ptr)->alignment_power + 0) #define bfd_section_name(bfd, ptr) ((ptr)->name) #define bfd_section_size(bfd, ptr) ((ptr)->size) #define bfd_get_section_size(ptr) ((ptr)->size) #define bfd_section_vma(bfd, ptr) ((ptr)->vma) #define bfd_section_lma(bfd, ptr) ((ptr)->lma) #define bfd_section_alignment(bfd, ptr) ((ptr)->alignment_power) #define bfd_get_section_flags(bfd, ptr) ((ptr)->flags + 0) #define bfd_get_section_userdata(bfd, ptr) ((ptr)->userdata) #define bfd_is_com_section(ptr) (((ptr)->flags & SEC_IS_COMMON) != 0) #define bfd_set_section_vma(bfd, ptr, val) (((ptr)->vma = (ptr)->lma = (val)), ((ptr)->user_set_vma = TRUE), TRUE) #define bfd_set_section_alignment(bfd, ptr, val) (((ptr)->alignment_power = (val)),TRUE) #define bfd_set_section_userdata(bfd, ptr, val) (((ptr)->userdata = (val)),TRUE) /* Find the address one past the end of SEC. */ #define bfd_get_section_limit(bfd, sec) \ (((sec)->rawsize ? (sec)->rawsize : (sec)->size) \ / bfd_octets_per_byte (bfd)) /* Return TRUE if section has been discarded. */ #define elf_discarded_section(sec) \ (!bfd_is_abs_section (sec) \ && bfd_is_abs_section ((sec)->output_section) \ && (sec)->sec_info_type != ELF_INFO_TYPE_MERGE \ && (sec)->sec_info_type != ELF_INFO_TYPE_JUST_SYMS) /* Forward define. */ struct stat; typedef enum bfd_print_symbol { bfd_print_symbol_name, bfd_print_symbol_more, bfd_print_symbol_all } bfd_print_symbol_type; /* Information about a symbol that nm needs. */ typedef struct _symbol_info { symvalue value; char type; const char *name; /* Symbol name. */ unsigned char stab_type; /* Stab type. */ char stab_other; /* Stab other. */ short stab_desc; /* Stab desc. */ const char *stab_name; /* String for stab type. */ } symbol_info; /* Get the name of a stabs type code. */ extern const char *bfd_get_stab_name (int); /* Hash table routines. There is no way to free up a hash table. */ /* An element in the hash table. Most uses will actually use a larger structure, and an instance of this will be the first field. */ struct bfd_hash_entry { /* Next entry for this hash code. */ struct bfd_hash_entry *next; /* String being hashed. */ const char *string; /* Hash code. This is the full hash code, not the index into the table. */ unsigned long hash; }; /* A hash table. */ struct bfd_hash_table { /* The hash array. */ struct bfd_hash_entry **table; /* A function used to create new elements in the hash table. The first entry is itself a pointer to an element. When this function is first invoked, this pointer will be NULL. However, having the pointer permits a hierarchy of method functions to be built each of which calls the function in the superclass. Thus each function should be written to allocate a new block of memory only if the argument is NULL. */ struct bfd_hash_entry *(*newfunc) (struct bfd_hash_entry *, struct bfd_hash_table *, const char *); /* An objalloc for this hash table. This is a struct objalloc *, but we use void * to avoid requiring the inclusion of objalloc.h. */ void *memory; /* The number of slots in the hash table. */ unsigned int size; /* The number of entries in the hash table. */ unsigned int count; /* The size of elements. */ unsigned int entsize; /* If non-zero, don't grow the hash table. */ unsigned int frozen:1; }; /* Initialize a hash table. */ extern bfd_boolean bfd_hash_table_init (struct bfd_hash_table *, struct bfd_hash_entry *(*) (struct bfd_hash_entry *, struct bfd_hash_table *, const char *), unsigned int); /* Initialize a hash table specifying a size. */ extern bfd_boolean bfd_hash_table_init_n (struct bfd_hash_table *, struct bfd_hash_entry *(*) (struct bfd_hash_entry *, struct bfd_hash_table *, const char *), unsigned int, unsigned int); /* Free up a hash table. */ extern void bfd_hash_table_free (struct bfd_hash_table *); /* Look up a string in a hash table. If CREATE is TRUE, a new entry will be created for this string if one does not already exist. The COPY argument must be TRUE if this routine should copy the string into newly allocated memory when adding an entry. */ extern struct bfd_hash_entry *bfd_hash_lookup (struct bfd_hash_table *, const char *, bfd_boolean create, bfd_boolean copy); /* Replace an entry in a hash table. */ extern void bfd_hash_replace (struct bfd_hash_table *, struct bfd_hash_entry *old, struct bfd_hash_entry *nw); /* Base method for creating a hash table entry. */ extern struct bfd_hash_entry *bfd_hash_newfunc (struct bfd_hash_entry *, struct bfd_hash_table *, const char *); /* Grab some space for a hash table entry. */ extern void *bfd_hash_allocate (struct bfd_hash_table *, unsigned int); /* Traverse a hash table in a random order, calling a function on each element. If the function returns FALSE, the traversal stops. The INFO argument is passed to the function. */ extern void bfd_hash_traverse (struct bfd_hash_table *, bfd_boolean (*) (struct bfd_hash_entry *, void *), void *info); /* Allows the default size of a hash table to be configured. New hash tables allocated using bfd_hash_table_init will be created with this size. */ extern void bfd_hash_set_default_size (bfd_size_type); /* This structure is used to keep track of stabs in sections information while linking. */ struct stab_info { /* A hash table used to hold stabs strings. */ struct bfd_strtab_hash *strings; /* The header file hash table. */ struct bfd_hash_table includes; /* The first .stabstr section. */ struct bfd_section *stabstr; }; #define COFF_SWAP_TABLE (void *) &bfd_coff_std_swap_table /* User program access to BFD facilities. */ /* Direct I/O routines, for programs which know more about the object file than BFD does. Use higher level routines if possible. */ extern bfd_size_type bfd_bread (void *, bfd_size_type, bfd *); extern bfd_size_type bfd_bwrite (const void *, bfd_size_type, bfd *); extern int bfd_seek (bfd *, file_ptr, int); extern file_ptr bfd_tell (bfd *); extern int bfd_flush (bfd *); extern int bfd_stat (bfd *, struct stat *); /* Deprecated old routines. */ #if __GNUC__ #define bfd_read(BUF, ELTSIZE, NITEMS, ABFD) \ (warn_deprecated ("bfd_read", __FILE__, __LINE__, __FUNCTION__), \ bfd_bread ((BUF), (ELTSIZE) * (NITEMS), (ABFD))) #define bfd_write(BUF, ELTSIZE, NITEMS, ABFD) \ (warn_deprecated ("bfd_write", __FILE__, __LINE__, __FUNCTION__), \ bfd_bwrite ((BUF), (ELTSIZE) * (NITEMS), (ABFD))) #else #define bfd_read(BUF, ELTSIZE, NITEMS, ABFD) \ (warn_deprecated ("bfd_read", (const char *) 0, 0, (const char *) 0), \ bfd_bread ((BUF), (ELTSIZE) * (NITEMS), (ABFD))) #define bfd_write(BUF, ELTSIZE, NITEMS, ABFD) \ (warn_deprecated ("bfd_write", (const char *) 0, 0, (const char *) 0),\ bfd_bwrite ((BUF), (ELTSIZE) * (NITEMS), (ABFD))) #endif extern void warn_deprecated (const char *, const char *, int, const char *); /* Cast from const char * to char * so that caller can assign to a char * without a warning. */ #define bfd_get_filename(abfd) ((char *) (abfd)->filename) #define bfd_get_cacheable(abfd) ((abfd)->cacheable) #define bfd_get_format(abfd) ((abfd)->format) #define bfd_get_target(abfd) ((abfd)->xvec->name) #define bfd_get_flavour(abfd) ((abfd)->xvec->flavour) #define bfd_family_coff(abfd) \ (bfd_get_flavour (abfd) == bfd_target_coff_flavour || \ bfd_get_flavour (abfd) == bfd_target_xcoff_flavour) #define bfd_big_endian(abfd) ((abfd)->xvec->byteorder == BFD_ENDIAN_BIG) #define bfd_little_endian(abfd) ((abfd)->xvec->byteorder == BFD_ENDIAN_LITTLE) #define bfd_header_big_endian(abfd) \ ((abfd)->xvec->header_byteorder == BFD_ENDIAN_BIG) #define bfd_header_little_endian(abfd) \ ((abfd)->xvec->header_byteorder == BFD_ENDIAN_LITTLE) #define bfd_get_file_flags(abfd) ((abfd)->flags) #define bfd_applicable_file_flags(abfd) ((abfd)->xvec->object_flags) #define bfd_applicable_section_flags(abfd) ((abfd)->xvec->section_flags) #define bfd_my_archive(abfd) ((abfd)->my_archive) #define bfd_has_map(abfd) ((abfd)->has_armap) #define bfd_valid_reloc_types(abfd) ((abfd)->xvec->valid_reloc_types) #define bfd_usrdata(abfd) ((abfd)->usrdata) #define bfd_get_start_address(abfd) ((abfd)->start_address) #define bfd_get_symcount(abfd) ((abfd)->symcount) #define bfd_get_outsymbols(abfd) ((abfd)->outsymbols) #define bfd_count_sections(abfd) ((abfd)->section_count) #define bfd_get_dynamic_symcount(abfd) ((abfd)->dynsymcount) #define bfd_get_symbol_leading_char(abfd) ((abfd)->xvec->symbol_leading_char) #define bfd_set_cacheable(abfd,bool) (((abfd)->cacheable = bool), TRUE) extern bfd_boolean bfd_cache_close (bfd *abfd); /* NB: This declaration should match the autogenerated one in libbfd.h. */ extern bfd_boolean bfd_cache_close_all (void); extern bfd_boolean bfd_record_phdr (bfd *, unsigned long, bfd_boolean, flagword, bfd_boolean, bfd_vma, bfd_boolean, bfd_boolean, unsigned int, struct bfd_section **); /* Byte swapping routines. */ bfd_uint64_t bfd_getb64 (const void *); bfd_uint64_t bfd_getl64 (const void *); bfd_int64_t bfd_getb_signed_64 (const void *); bfd_int64_t bfd_getl_signed_64 (const void *); bfd_vma bfd_getb32 (const void *); bfd_vma bfd_getl32 (const void *); bfd_signed_vma bfd_getb_signed_32 (const void *); bfd_signed_vma bfd_getl_signed_32 (const void *); bfd_vma bfd_getb16 (const void *); bfd_vma bfd_getl16 (const void *); bfd_signed_vma bfd_getb_signed_16 (const void *); bfd_signed_vma bfd_getl_signed_16 (const void *); void bfd_putb64 (bfd_uint64_t, void *); void bfd_putl64 (bfd_uint64_t, void *); void bfd_putb32 (bfd_vma, void *); void bfd_putl32 (bfd_vma, void *); void bfd_putb16 (bfd_vma, void *); void bfd_putl16 (bfd_vma, void *); /* Byte swapping routines which take size and endiannes as arguments. */ bfd_uint64_t bfd_get_bits (const void *, int, bfd_boolean); void bfd_put_bits (bfd_uint64_t, void *, int, bfd_boolean); extern bfd_boolean bfd_section_already_linked_table_init (void); extern void bfd_section_already_linked_table_free (void); /* Externally visible ECOFF routines. */ #if defined(__STDC__) || defined(ALMOST_STDC) struct ecoff_debug_info; struct ecoff_debug_swap; struct ecoff_extr; struct bfd_symbol; struct bfd_link_info; struct bfd_link_hash_entry; struct bfd_elf_version_tree; #endif extern bfd_vma bfd_ecoff_get_gp_value (bfd * abfd); extern bfd_boolean bfd_ecoff_set_gp_value (bfd *abfd, bfd_vma gp_value); extern bfd_boolean bfd_ecoff_set_regmasks (bfd *abfd, unsigned long gprmask, unsigned long fprmask, unsigned long *cprmask); extern void *bfd_ecoff_debug_init (bfd *output_bfd, struct ecoff_debug_info *output_debug, const struct ecoff_debug_swap *output_swap, struct bfd_link_info *); extern void bfd_ecoff_debug_free (void *handle, bfd *output_bfd, struct ecoff_debug_info *output_debug, const struct ecoff_debug_swap *output_swap, struct bfd_link_info *); extern bfd_boolean bfd_ecoff_debug_accumulate (void *handle, bfd *output_bfd, struct ecoff_debug_info *output_debug, const struct ecoff_debug_swap *output_swap, bfd *input_bfd, struct ecoff_debug_info *input_debug, const struct ecoff_debug_swap *input_swap, struct bfd_link_info *); extern bfd_boolean bfd_ecoff_debug_accumulate_other (void *handle, bfd *output_bfd, struct ecoff_debug_info *output_debug, const struct ecoff_debug_swap *output_swap, bfd *input_bfd, struct bfd_link_info *); extern bfd_boolean bfd_ecoff_debug_externals (bfd *abfd, struct ecoff_debug_info *debug, const struct ecoff_debug_swap *swap, bfd_boolean relocatable, bfd_boolean (*get_extr) (struct bfd_symbol *, struct ecoff_extr *), void (*set_index) (struct bfd_symbol *, bfd_size_type)); extern bfd_boolean bfd_ecoff_debug_one_external (bfd *abfd, struct ecoff_debug_info *debug, const struct ecoff_debug_swap *swap, const char *name, struct ecoff_extr *esym); extern bfd_size_type bfd_ecoff_debug_size (bfd *abfd, struct ecoff_debug_info *debug, const struct ecoff_debug_swap *swap); extern bfd_boolean bfd_ecoff_write_debug (bfd *abfd, struct ecoff_debug_info *debug, const struct ecoff_debug_swap *swap, file_ptr where); extern bfd_boolean bfd_ecoff_write_accumulated_debug (void *handle, bfd *abfd, struct ecoff_debug_info *debug, const struct ecoff_debug_swap *swap, struct bfd_link_info *info, file_ptr where); /* Externally visible ELF routines. */ struct bfd_link_needed_list { struct bfd_link_needed_list *next; bfd *by; const char *name; }; enum dynamic_lib_link_class { DYN_NORMAL = 0, DYN_AS_NEEDED = 1, DYN_DT_NEEDED = 2, DYN_NO_ADD_NEEDED = 4, DYN_NO_NEEDED = 8 }; enum notice_asneeded_action { notice_as_needed, notice_not_needed, notice_needed }; extern bfd_boolean bfd_elf_record_link_assignment (bfd *, struct bfd_link_info *, const char *, bfd_boolean, bfd_boolean); extern struct bfd_link_needed_list *bfd_elf_get_needed_list (bfd *, struct bfd_link_info *); extern bfd_boolean bfd_elf_get_bfd_needed_list (bfd *, struct bfd_link_needed_list **); extern bfd_boolean bfd_elf_size_dynamic_sections (bfd *, const char *, const char *, const char *, const char * const *, struct bfd_link_info *, struct bfd_section **, struct bfd_elf_version_tree *); extern bfd_boolean bfd_elf_size_dynsym_hash_dynstr (bfd *, struct bfd_link_info *); extern void bfd_elf_set_dt_needed_name (bfd *, const char *); extern const char *bfd_elf_get_dt_soname (bfd *); extern void bfd_elf_set_dyn_lib_class (bfd *, enum dynamic_lib_link_class); extern int bfd_elf_get_dyn_lib_class (bfd *); extern struct bfd_link_needed_list *bfd_elf_get_runpath_list (bfd *, struct bfd_link_info *); extern bfd_boolean bfd_elf_discard_info (bfd *, struct bfd_link_info *); extern unsigned int _bfd_elf_default_action_discarded (struct bfd_section *); /* Return an upper bound on the number of bytes required to store a copy of ABFD's program header table entries. Return -1 if an error occurs; bfd_get_error will return an appropriate code. */ extern long bfd_get_elf_phdr_upper_bound (bfd *abfd); /* Copy ABFD's program header table entries to *PHDRS. The entries will be stored as an array of Elf_Internal_Phdr structures, as defined in include/elf/internal.h. To find out how large the buffer needs to be, call bfd_get_elf_phdr_upper_bound. Return the number of program header table entries read, or -1 if an error occurs; bfd_get_error will return an appropriate code. */ extern int bfd_get_elf_phdrs (bfd *abfd, void *phdrs); /* Create a new BFD as if by bfd_openr. Rather than opening a file, reconstruct an ELF file by reading the segments out of remote memory based on the ELF file header at EHDR_VMA and the ELF program headers it points to. If not null, *LOADBASEP is filled in with the difference between the VMAs from which the segments were read, and the VMAs the file headers (and hence BFD's idea of each section's VMA) put them at. The function TARGET_READ_MEMORY is called to copy LEN bytes from the remote memory at target address VMA into the local buffer at MYADDR; it should return zero on success or an `errno' code on failure. TEMPL must be a BFD for an ELF target with the word size and byte order found in the remote memory. */ extern bfd *bfd_elf_bfd_from_remote_memory (bfd *templ, bfd_vma ehdr_vma, bfd_vma *loadbasep, int (*target_read_memory) (bfd_vma vma, bfd_byte *myaddr, int len)); /* Return the arch_size field of an elf bfd, or -1 if not elf. */ extern int bfd_get_arch_size (bfd *); /* Return TRUE if address "naturally" sign extends, or -1 if not elf. */ extern int bfd_get_sign_extend_vma (bfd *); extern struct bfd_section *_bfd_elf_tls_setup (bfd *, struct bfd_link_info *); extern void _bfd_fix_excluded_sec_syms (bfd *, struct bfd_link_info *); extern unsigned bfd_m68k_mach_to_features (int); extern int bfd_m68k_features_to_mach (unsigned); extern bfd_boolean bfd_m68k_elf32_create_embedded_relocs (bfd *, struct bfd_link_info *, struct bfd_section *, struct bfd_section *, char **); extern bfd_boolean bfd_bfin_elf32_create_embedded_relocs (bfd *, struct bfd_link_info *, struct bfd_section *, struct bfd_section *, char **); /* SunOS shared library support routines for the linker. */ extern struct bfd_link_needed_list *bfd_sunos_get_needed_list (bfd *, struct bfd_link_info *); extern bfd_boolean bfd_sunos_record_link_assignment (bfd *, struct bfd_link_info *, const char *); extern bfd_boolean bfd_sunos_size_dynamic_sections (bfd *, struct bfd_link_info *, struct bfd_section **, struct bfd_section **, struct bfd_section **); /* Linux shared library support routines for the linker. */ extern bfd_boolean bfd_i386linux_size_dynamic_sections (bfd *, struct bfd_link_info *); extern bfd_boolean bfd_m68klinux_size_dynamic_sections (bfd *, struct bfd_link_info *); extern bfd_boolean bfd_sparclinux_size_dynamic_sections (bfd *, struct bfd_link_info *); /* mmap hacks */ struct _bfd_window_internal; typedef struct _bfd_window_internal bfd_window_internal; typedef struct _bfd_window { /* What the user asked for. */ void *data; bfd_size_type size; /* The actual window used by BFD. Small user-requested read-only regions sharing a page may share a single window into the object file. Read-write versions shouldn't until I've fixed things to keep track of which portions have been claimed by the application; don't want to give the same region back when the application wants two writable copies! */ struct _bfd_window_internal *i; } bfd_window; extern void bfd_init_window (bfd_window *); extern void bfd_free_window (bfd_window *); extern bfd_boolean bfd_get_file_window (bfd *, file_ptr, bfd_size_type, bfd_window *, bfd_boolean); /* XCOFF support routines for the linker. */ extern bfd_boolean bfd_xcoff_link_record_set (bfd *, struct bfd_link_info *, struct bfd_link_hash_entry *, bfd_size_type); extern bfd_boolean bfd_xcoff_import_symbol (bfd *, struct bfd_link_info *, struct bfd_link_hash_entry *, bfd_vma, const char *, const char *, const char *, unsigned int); extern bfd_boolean bfd_xcoff_export_symbol (bfd *, struct bfd_link_info *, struct bfd_link_hash_entry *); extern bfd_boolean bfd_xcoff_link_count_reloc (bfd *, struct bfd_link_info *, const char *); extern bfd_boolean bfd_xcoff_record_link_assignment (bfd *, struct bfd_link_info *, const char *); extern bfd_boolean bfd_xcoff_size_dynamic_sections (bfd *, struct bfd_link_info *, const char *, const char *, unsigned long, unsigned long, unsigned long, bfd_boolean, int, bfd_boolean, bfd_boolean, struct bfd_section **, bfd_boolean); extern bfd_boolean bfd_xcoff_link_generate_rtinit (bfd *, const char *, const char *, bfd_boolean); /* XCOFF support routines for ar. */ extern bfd_boolean bfd_xcoff_ar_archive_set_magic (bfd *, char *); /* Externally visible COFF routines. */ #if defined(__STDC__) || defined(ALMOST_STDC) struct internal_syment; union internal_auxent; #endif extern bfd_boolean bfd_coff_get_syment (bfd *, struct bfd_symbol *, struct internal_syment *); extern bfd_boolean bfd_coff_get_auxent (bfd *, struct bfd_symbol *, int, union internal_auxent *); extern bfd_boolean bfd_coff_set_symbol_class (bfd *, struct bfd_symbol *, unsigned int); extern bfd_boolean bfd_m68k_coff_create_embedded_relocs (bfd *, struct bfd_link_info *, struct bfd_section *, struct bfd_section *, char **); /* ARM VFP11 erratum workaround support. */ typedef enum { BFD_ARM_VFP11_FIX_DEFAULT, BFD_ARM_VFP11_FIX_NONE, BFD_ARM_VFP11_FIX_SCALAR, BFD_ARM_VFP11_FIX_VECTOR } bfd_arm_vfp11_fix; extern void bfd_elf32_arm_init_maps (bfd *); extern void bfd_elf32_arm_set_vfp11_fix (bfd *, struct bfd_link_info *); extern bfd_boolean bfd_elf32_arm_vfp11_erratum_scan (bfd *, struct bfd_link_info *); extern void bfd_elf32_arm_vfp11_fix_veneer_locations (bfd *, struct bfd_link_info *); /* ARM Interworking support. Called from linker. */ extern bfd_boolean bfd_arm_allocate_interworking_sections (struct bfd_link_info *); extern bfd_boolean bfd_arm_process_before_allocation (bfd *, struct bfd_link_info *, int); extern bfd_boolean bfd_arm_get_bfd_for_interworking (bfd *, struct bfd_link_info *); /* PE ARM Interworking support. Called from linker. */ extern bfd_boolean bfd_arm_pe_allocate_interworking_sections (struct bfd_link_info *); extern bfd_boolean bfd_arm_pe_process_before_allocation (bfd *, struct bfd_link_info *, int); extern bfd_boolean bfd_arm_pe_get_bfd_for_interworking (bfd *, struct bfd_link_info *); /* ELF ARM Interworking support. Called from linker. */ extern bfd_boolean bfd_elf32_arm_allocate_interworking_sections (struct bfd_link_info *); extern bfd_boolean bfd_elf32_arm_process_before_allocation (bfd *, struct bfd_link_info *); void bfd_elf32_arm_set_target_relocs (bfd *, struct bfd_link_info *, int, char *, int, int, bfd_arm_vfp11_fix, int, int); extern bfd_boolean bfd_elf32_arm_get_bfd_for_interworking (bfd *, struct bfd_link_info *); extern bfd_boolean bfd_elf32_arm_add_glue_sections_to_bfd (bfd *, struct bfd_link_info *); /* ELF ARM mapping symbol support */ #define BFD_ARM_SPECIAL_SYM_TYPE_MAP (1 << 0) #define BFD_ARM_SPECIAL_SYM_TYPE_TAG (1 << 1) #define BFD_ARM_SPECIAL_SYM_TYPE_OTHER (1 << 2) #define BFD_ARM_SPECIAL_SYM_TYPE_ANY (~0) extern bfd_boolean bfd_is_arm_special_symbol_name (const char * name, int type); extern void bfd_elf32_arm_set_byteswap_code (struct bfd_link_info *, int); /* ARM Note section processing. */ extern bfd_boolean bfd_arm_merge_machines (bfd *, bfd *); extern bfd_boolean bfd_arm_update_notes (bfd *, const char *); extern unsigned int bfd_arm_get_mach_from_notes (bfd *, const char *); /* TI COFF load page support. */ extern void bfd_ticoff_set_section_load_page (struct bfd_section *, int); extern int bfd_ticoff_get_section_load_page (struct bfd_section *); /* H8/300 functions. */ extern bfd_vma bfd_h8300_pad_address (bfd *, bfd_vma); /* IA64 Itanium code generation. Called from linker. */ extern void bfd_elf32_ia64_after_parse (int); extern void bfd_elf64_ia64_after_parse (int); /* This structure is used for a comdat section, as in PE. A comdat section is associated with a particular symbol. When the linker sees a comdat section, it keeps only one of the sections with a given name and associated with a given symbol. */ struct coff_comdat_info { /* The name of the symbol associated with a comdat section. */ const char *name; /* The local symbol table index of the symbol associated with a comdat section. This is only meaningful to the object file format specific code; it is not an index into the list returned by bfd_canonicalize_symtab. */ long symbol; }; extern struct coff_comdat_info *bfd_coff_get_comdat_section (bfd *, struct bfd_section *); /* Extracted from init.c. */ void bfd_init (void); /* Extracted from opncls.c. */ bfd *bfd_fopen (const char *filename, const char *target, const char *mode, int fd); bfd *bfd_openr (const char *filename, const char *target); bfd *bfd_fdopenr (const char *filename, const char *target, int fd); bfd *bfd_openstreamr (const char *, const char *, void *); bfd *bfd_openr_iovec (const char *filename, const char *target, void *(*open) (struct bfd *nbfd, void *open_closure), void *open_closure, file_ptr (*pread) (struct bfd *nbfd, void *stream, void *buf, file_ptr nbytes, file_ptr offset), int (*close) (struct bfd *nbfd, void *stream), int (*stat) (struct bfd *abfd, void *stream, struct stat *sb)); bfd *bfd_openw (const char *filename, const char *target); bfd_boolean bfd_close (bfd *abfd); bfd_boolean bfd_close_all_done (bfd *); bfd *bfd_create (const char *filename, bfd *templ); bfd_boolean bfd_make_writable (bfd *abfd); bfd_boolean bfd_make_readable (bfd *abfd); unsigned long bfd_calc_gnu_debuglink_crc32 (unsigned long crc, const unsigned char *buf, bfd_size_type len); char *bfd_follow_gnu_debuglink (bfd *abfd, const char *dir); struct bfd_section *bfd_create_gnu_debuglink_section (bfd *abfd, const char *filename); bfd_boolean bfd_fill_in_gnu_debuglink_section (bfd *abfd, struct bfd_section *sect, const char *filename); /* Extracted from libbfd.c. */ /* Byte swapping macros for user section data. */ #define bfd_put_8(abfd, val, ptr) \ ((void) (*((unsigned char *) (ptr)) = (val) & 0xff)) #define bfd_put_signed_8 \ bfd_put_8 #define bfd_get_8(abfd, ptr) \ (*(unsigned char *) (ptr) & 0xff) #define bfd_get_signed_8(abfd, ptr) \ (((*(unsigned char *) (ptr) & 0xff) ^ 0x80) - 0x80) #define bfd_put_16(abfd, val, ptr) \ BFD_SEND (abfd, bfd_putx16, ((val),(ptr))) #define bfd_put_signed_16 \ bfd_put_16 #define bfd_get_16(abfd, ptr) \ BFD_SEND (abfd, bfd_getx16, (ptr)) #define bfd_get_signed_16(abfd, ptr) \ BFD_SEND (abfd, bfd_getx_signed_16, (ptr)) #define bfd_put_32(abfd, val, ptr) \ BFD_SEND (abfd, bfd_putx32, ((val),(ptr))) #define bfd_put_signed_32 \ bfd_put_32 #define bfd_get_32(abfd, ptr) \ BFD_SEND (abfd, bfd_getx32, (ptr)) #define bfd_get_signed_32(abfd, ptr) \ BFD_SEND (abfd, bfd_getx_signed_32, (ptr)) #define bfd_put_64(abfd, val, ptr) \ BFD_SEND (abfd, bfd_putx64, ((val), (ptr))) #define bfd_put_signed_64 \ bfd_put_64 #define bfd_get_64(abfd, ptr) \ BFD_SEND (abfd, bfd_getx64, (ptr)) #define bfd_get_signed_64(abfd, ptr) \ BFD_SEND (abfd, bfd_getx_signed_64, (ptr)) #define bfd_get(bits, abfd, ptr) \ ((bits) == 8 ? (bfd_vma) bfd_get_8 (abfd, ptr) \ : (bits) == 16 ? bfd_get_16 (abfd, ptr) \ : (bits) == 32 ? bfd_get_32 (abfd, ptr) \ : (bits) == 64 ? bfd_get_64 (abfd, ptr) \ : (abort (), (bfd_vma) - 1)) #define bfd_put(bits, abfd, val, ptr) \ ((bits) == 8 ? bfd_put_8 (abfd, val, ptr) \ : (bits) == 16 ? bfd_put_16 (abfd, val, ptr) \ : (bits) == 32 ? bfd_put_32 (abfd, val, ptr) \ : (bits) == 64 ? bfd_put_64 (abfd, val, ptr) \ : (abort (), (void) 0)) /* Byte swapping macros for file header data. */ #define bfd_h_put_8(abfd, val, ptr) \ bfd_put_8 (abfd, val, ptr) #define bfd_h_put_signed_8(abfd, val, ptr) \ bfd_put_8 (abfd, val, ptr) #define bfd_h_get_8(abfd, ptr) \ bfd_get_8 (abfd, ptr) #define bfd_h_get_signed_8(abfd, ptr) \ bfd_get_signed_8 (abfd, ptr) #define bfd_h_put_16(abfd, val, ptr) \ BFD_SEND (abfd, bfd_h_putx16, (val, ptr)) #define bfd_h_put_signed_16 \ bfd_h_put_16 #define bfd_h_get_16(abfd, ptr) \ BFD_SEND (abfd, bfd_h_getx16, (ptr)) #define bfd_h_get_signed_16(abfd, ptr) \ BFD_SEND (abfd, bfd_h_getx_signed_16, (ptr)) #define bfd_h_put_32(abfd, val, ptr) \ BFD_SEND (abfd, bfd_h_putx32, (val, ptr)) #define bfd_h_put_signed_32 \ bfd_h_put_32 #define bfd_h_get_32(abfd, ptr) \ BFD_SEND (abfd, bfd_h_getx32, (ptr)) #define bfd_h_get_signed_32(abfd, ptr) \ BFD_SEND (abfd, bfd_h_getx_signed_32, (ptr)) #define bfd_h_put_64(abfd, val, ptr) \ BFD_SEND (abfd, bfd_h_putx64, (val, ptr)) #define bfd_h_put_signed_64 \ bfd_h_put_64 #define bfd_h_get_64(abfd, ptr) \ BFD_SEND (abfd, bfd_h_getx64, (ptr)) #define bfd_h_get_signed_64(abfd, ptr) \ BFD_SEND (abfd, bfd_h_getx_signed_64, (ptr)) /* Aliases for the above, which should eventually go away. */ #define H_PUT_64 bfd_h_put_64 #define H_PUT_32 bfd_h_put_32 #define H_PUT_16 bfd_h_put_16 #define H_PUT_8 bfd_h_put_8 #define H_PUT_S64 bfd_h_put_signed_64 #define H_PUT_S32 bfd_h_put_signed_32 #define H_PUT_S16 bfd_h_put_signed_16 #define H_PUT_S8 bfd_h_put_signed_8 #define H_GET_64 bfd_h_get_64 #define H_GET_32 bfd_h_get_32 #define H_GET_16 bfd_h_get_16 #define H_GET_8 bfd_h_get_8 #define H_GET_S64 bfd_h_get_signed_64 #define H_GET_S32 bfd_h_get_signed_32 #define H_GET_S16 bfd_h_get_signed_16 #define H_GET_S8 bfd_h_get_signed_8 /* Extracted from bfdio.c. */ long bfd_get_mtime (bfd *abfd); file_ptr bfd_get_size (bfd *abfd); /* Extracted from bfdwin.c. */ /* Extracted from section.c. */ typedef struct bfd_section { /* The name of the section; the name isn't a copy, the pointer is the same as that passed to bfd_make_section. */ const char *name; /* A unique sequence number. */ int id; /* Which section in the bfd; 0..n-1 as sections are created in a bfd. */ int index; /* The next section in the list belonging to the BFD, or NULL. */ struct bfd_section *next; /* The previous section in the list belonging to the BFD, or NULL. */ struct bfd_section *prev; /* The field flags contains attributes of the section. Some flags are read in from the object file, and some are synthesized from other information. */ flagword flags; #define SEC_NO_FLAGS 0x000 /* Tells the OS to allocate space for this section when loading. This is clear for a section containing debug information only. */ #define SEC_ALLOC 0x001 /* Tells the OS to load the section from the file when loading. This is clear for a .bss section. */ #define SEC_LOAD 0x002 /* The section contains data still to be relocated, so there is some relocation information too. */ #define SEC_RELOC 0x004 /* A signal to the OS that the section contains read only data. */ #define SEC_READONLY 0x008 /* The section contains code only. */ #define SEC_CODE 0x010 /* The section contains data only. */ #define SEC_DATA 0x020 /* The section will reside in ROM. */ #define SEC_ROM 0x040 /* The section contains constructor information. This section type is used by the linker to create lists of constructors and destructors used by <>. When a back end sees a symbol which should be used in a constructor list, it creates a new section for the type of name (e.g., <<__CTOR_LIST__>>), attaches the symbol to it, and builds a relocation. To build the lists of constructors, all the linker has to do is catenate all the sections called <<__CTOR_LIST__>> and relocate the data contained within - exactly the operations it would peform on standard data. */ #define SEC_CONSTRUCTOR 0x080 /* The section has contents - a data section could be <> | <>; a debug section could be <> */ #define SEC_HAS_CONTENTS 0x100 /* An instruction to the linker to not output the section even if it has information which would normally be written. */ #define SEC_NEVER_LOAD 0x200 /* The section contains thread local data. */ #define SEC_THREAD_LOCAL 0x400 /* The section has GOT references. This flag is only for the linker, and is currently only used by the elf32-hppa back end. It will be set if global offset table references were detected in this section, which indicate to the linker that the section contains PIC code, and must be handled specially when doing a static link. */ #define SEC_HAS_GOT_REF 0x800 /* The section contains common symbols (symbols may be defined multiple times, the value of a symbol is the amount of space it requires, and the largest symbol value is the one used). Most targets have exactly one of these (which we translate to bfd_com_section_ptr), but ECOFF has two. */ #define SEC_IS_COMMON 0x1000 /* The section contains only debugging information. For example, this is set for ELF .debug and .stab sections. strip tests this flag to see if a section can be discarded. */ #define SEC_DEBUGGING 0x2000 /* The contents of this section are held in memory pointed to by the contents field. This is checked by bfd_get_section_contents, and the data is retrieved from memory if appropriate. */ #define SEC_IN_MEMORY 0x4000 /* The contents of this section are to be excluded by the linker for executable and shared objects unless those objects are to be further relocated. */ #define SEC_EXCLUDE 0x8000 /* The contents of this section are to be sorted based on the sum of the symbol and addend values specified by the associated relocation entries. Entries without associated relocation entries will be appended to the end of the section in an unspecified order. */ #define SEC_SORT_ENTRIES 0x10000 /* When linking, duplicate sections of the same name should be discarded, rather than being combined into a single section as is usually done. This is similar to how common symbols are handled. See SEC_LINK_DUPLICATES below. */ #define SEC_LINK_ONCE 0x20000 /* If SEC_LINK_ONCE is set, this bitfield describes how the linker should handle duplicate sections. */ #define SEC_LINK_DUPLICATES 0x40000 /* This value for SEC_LINK_DUPLICATES means that duplicate sections with the same name should simply be discarded. */ #define SEC_LINK_DUPLICATES_DISCARD 0x0 /* This value for SEC_LINK_DUPLICATES means that the linker should warn if there are any duplicate sections, although it should still only link one copy. */ #define SEC_LINK_DUPLICATES_ONE_ONLY 0x80000 /* This value for SEC_LINK_DUPLICATES means that the linker should warn if any duplicate sections are a different size. */ #define SEC_LINK_DUPLICATES_SAME_SIZE 0x100000 /* This value for SEC_LINK_DUPLICATES means that the linker should warn if any duplicate sections contain different contents. */ #define SEC_LINK_DUPLICATES_SAME_CONTENTS \ (SEC_LINK_DUPLICATES_ONE_ONLY | SEC_LINK_DUPLICATES_SAME_SIZE) /* This section was created by the linker as part of dynamic relocation or other arcane processing. It is skipped when going through the first-pass output, trusting that someone else up the line will take care of it later. */ #define SEC_LINKER_CREATED 0x200000 /* This section should not be subject to garbage collection. Also set to inform the linker that this section should not be listed in the link map as discarded. */ #define SEC_KEEP 0x400000 /* This section contains "short" data, and should be placed "near" the GP. */ #define SEC_SMALL_DATA 0x800000 /* Attempt to merge identical entities in the section. Entity size is given in the entsize field. */ #define SEC_MERGE 0x1000000 /* If given with SEC_MERGE, entities to merge are zero terminated strings where entsize specifies character size instead of fixed size entries. */ #define SEC_STRINGS 0x2000000 /* This section contains data about section groups. */ #define SEC_GROUP 0x4000000 /* The section is a COFF shared library section. This flag is only for the linker. If this type of section appears in the input file, the linker must copy it to the output file without changing the vma or size. FIXME: Although this was originally intended to be general, it really is COFF specific (and the flag was renamed to indicate this). It might be cleaner to have some more general mechanism to allow the back end to control what the linker does with sections. */ #define SEC_COFF_SHARED_LIBRARY 0x10000000 /* This section contains data which may be shared with other executables or shared objects. This is for COFF only. */ #define SEC_COFF_SHARED 0x20000000 /* When a section with this flag is being linked, then if the size of the input section is less than a page, it should not cross a page boundary. If the size of the input section is one page or more, it should be aligned on a page boundary. This is for TI TMS320C54X only. */ #define SEC_TIC54X_BLOCK 0x40000000 /* Conditionally link this section; do not link if there are no references found to any symbol in the section. This is for TI TMS320C54X only. */ #define SEC_TIC54X_CLINK 0x80000000 /* End of section flags. */ /* Some internal packed boolean fields. */ /* See the vma field. */ unsigned int user_set_vma : 1; /* A mark flag used by some of the linker backends. */ unsigned int linker_mark : 1; /* Another mark flag used by some of the linker backends. Set for output sections that have an input section. */ unsigned int linker_has_input : 1; /* Mark flags used by some linker backends for garbage collection. */ unsigned int gc_mark : 1; unsigned int gc_mark_from_eh : 1; /* The following flags are used by the ELF linker. */ /* Mark sections which have been allocated to segments. */ unsigned int segment_mark : 1; /* Type of sec_info information. */ unsigned int sec_info_type:3; #define ELF_INFO_TYPE_NONE 0 #define ELF_INFO_TYPE_STABS 1 #define ELF_INFO_TYPE_MERGE 2 #define ELF_INFO_TYPE_EH_FRAME 3 #define ELF_INFO_TYPE_JUST_SYMS 4 /* Nonzero if this section uses RELA relocations, rather than REL. */ unsigned int use_rela_p:1; /* Bits used by various backends. The generic code doesn't touch these fields. */ /* Nonzero if this section has TLS related relocations. */ unsigned int has_tls_reloc:1; + /* Nonzero if this section has a call to __tls_get_addr. */ + unsigned int has_tls_get_addr_call:1; + /* Nonzero if this section has a gp reloc. */ unsigned int has_gp_reloc:1; /* Nonzero if this section needs the relax finalize pass. */ unsigned int need_finalize_relax:1; /* Whether relocations have been processed. */ unsigned int reloc_done : 1; /* End of internal packed boolean fields. */ /* The virtual memory address of the section - where it will be at run time. The symbols are relocated against this. The user_set_vma flag is maintained by bfd; if it's not set, the backend can assign addresses (for example, in <>, where the default address for <<.data>> is dependent on the specific target and various flags). */ bfd_vma vma; /* The load address of the section - where it would be in a rom image; really only used for writing section header information. */ bfd_vma lma; /* The size of the section in octets, as it will be output. Contains a value even if the section has no contents (e.g., the size of <<.bss>>). */ bfd_size_type size; /* For input sections, the original size on disk of the section, in octets. This field is used by the linker relaxation code. It is currently only set for sections where the linker relaxation scheme doesn't cache altered section and reloc contents (stabs, eh_frame, SEC_MERGE, some coff relaxing targets), and thus the original size needs to be kept to read the section multiple times. For output sections, rawsize holds the section size calculated on a previous linker relaxation pass. */ bfd_size_type rawsize; /* If this section is going to be output, then this value is the offset in *bytes* into the output section of the first byte in the input section (byte ==> smallest addressable unit on the target). In most cases, if this was going to start at the 100th octet (8-bit quantity) in the output section, this value would be 100. However, if the target byte size is 16 bits (bfd_octets_per_byte is "2"), this value would be 50. */ bfd_vma output_offset; /* The output section through which to map on output. */ struct bfd_section *output_section; /* The alignment requirement of the section, as an exponent of 2 - e.g., 3 aligns to 2^3 (or 8). */ unsigned int alignment_power; /* If an input section, a pointer to a vector of relocation records for the data in this section. */ struct reloc_cache_entry *relocation; /* If an output section, a pointer to a vector of pointers to relocation records for the data in this section. */ struct reloc_cache_entry **orelocation; /* The number of relocation records in one of the above. */ unsigned reloc_count; /* Information below is back end specific - and not always used or updated. */ /* File position of section data. */ file_ptr filepos; /* File position of relocation info. */ file_ptr rel_filepos; /* File position of line data. */ file_ptr line_filepos; /* Pointer to data for applications. */ void *userdata; /* If the SEC_IN_MEMORY flag is set, this points to the actual contents. */ unsigned char *contents; /* Attached line number information. */ alent *lineno; /* Number of line number records. */ unsigned int lineno_count; /* Entity size for merging purposes. */ unsigned int entsize; /* Points to the kept section if this section is a link-once section, and is discarded. */ struct bfd_section *kept_section; /* When a section is being output, this value changes as more linenumbers are written out. */ file_ptr moving_line_filepos; /* What the section number is in the target world. */ int target_index; void *used_by_bfd; /* If this is a constructor section then here is a list of the relocations created to relocate items within it. */ struct relent_chain *constructor_chain; /* The BFD which owns the section. */ bfd *owner; /* A symbol which points at this section only. */ struct bfd_symbol *symbol; struct bfd_symbol **symbol_ptr_ptr; /* Early in the link process, map_head and map_tail are used to build a list of input sections attached to an output section. Later, output sections use these fields for a list of bfd_link_order structs. */ union { struct bfd_link_order *link_order; struct bfd_section *s; } map_head, map_tail; } asection; /* These sections are global, and are managed by BFD. The application and target back end are not permitted to change the values in these sections. New code should use the section_ptr macros rather than referring directly to the const sections. The const sections may eventually vanish. */ #define BFD_ABS_SECTION_NAME "*ABS*" #define BFD_UND_SECTION_NAME "*UND*" #define BFD_COM_SECTION_NAME "*COM*" #define BFD_IND_SECTION_NAME "*IND*" /* The absolute section. */ extern asection bfd_abs_section; #define bfd_abs_section_ptr ((asection *) &bfd_abs_section) #define bfd_is_abs_section(sec) ((sec) == bfd_abs_section_ptr) /* Pointer to the undefined section. */ extern asection bfd_und_section; #define bfd_und_section_ptr ((asection *) &bfd_und_section) #define bfd_is_und_section(sec) ((sec) == bfd_und_section_ptr) /* Pointer to the common section. */ extern asection bfd_com_section; #define bfd_com_section_ptr ((asection *) &bfd_com_section) /* Pointer to the indirect section. */ extern asection bfd_ind_section; #define bfd_ind_section_ptr ((asection *) &bfd_ind_section) #define bfd_is_ind_section(sec) ((sec) == bfd_ind_section_ptr) #define bfd_is_const_section(SEC) \ ( ((SEC) == bfd_abs_section_ptr) \ || ((SEC) == bfd_und_section_ptr) \ || ((SEC) == bfd_com_section_ptr) \ || ((SEC) == bfd_ind_section_ptr)) /* Macros to handle insertion and deletion of a bfd's sections. These only handle the list pointers, ie. do not adjust section_count, target_index etc. */ #define bfd_section_list_remove(ABFD, S) \ do \ { \ asection *_s = S; \ asection *_next = _s->next; \ asection *_prev = _s->prev; \ if (_prev) \ _prev->next = _next; \ else \ (ABFD)->sections = _next; \ if (_next) \ _next->prev = _prev; \ else \ (ABFD)->section_last = _prev; \ } \ while (0) #define bfd_section_list_append(ABFD, S) \ do \ { \ asection *_s = S; \ bfd *_abfd = ABFD; \ _s->next = NULL; \ if (_abfd->section_last) \ { \ _s->prev = _abfd->section_last; \ _abfd->section_last->next = _s; \ } \ else \ { \ _s->prev = NULL; \ _abfd->sections = _s; \ } \ _abfd->section_last = _s; \ } \ while (0) #define bfd_section_list_prepend(ABFD, S) \ do \ { \ asection *_s = S; \ bfd *_abfd = ABFD; \ _s->prev = NULL; \ if (_abfd->sections) \ { \ _s->next = _abfd->sections; \ _abfd->sections->prev = _s; \ } \ else \ { \ _s->next = NULL; \ _abfd->section_last = _s; \ } \ _abfd->sections = _s; \ } \ while (0) #define bfd_section_list_insert_after(ABFD, A, S) \ do \ { \ asection *_a = A; \ asection *_s = S; \ asection *_next = _a->next; \ _s->next = _next; \ _s->prev = _a; \ _a->next = _s; \ if (_next) \ _next->prev = _s; \ else \ (ABFD)->section_last = _s; \ } \ while (0) #define bfd_section_list_insert_before(ABFD, B, S) \ do \ { \ asection *_b = B; \ asection *_s = S; \ asection *_prev = _b->prev; \ _s->prev = _prev; \ _s->next = _b; \ _b->prev = _s; \ if (_prev) \ _prev->next = _s; \ else \ (ABFD)->sections = _s; \ } \ while (0) #define bfd_section_removed_from_list(ABFD, S) \ ((S)->next == NULL ? (ABFD)->section_last != (S) : (S)->next->prev != (S)) #define BFD_FAKE_SECTION(SEC, FLAGS, SYM, NAME, IDX) \ /* name, id, index, next, prev, flags, user_set_vma, */ \ { NAME, IDX, 0, NULL, NULL, FLAGS, 0, \ \ /* linker_mark, linker_has_input, gc_mark, gc_mark_from_eh, */ \ 0, 0, 1, 0, \ \ /* segment_mark, sec_info_type, use_rela_p, has_tls_reloc, */ \ 0, 0, 0, 0, \ \ - /* has_gp_reloc, need_finalize_relax, reloc_done, */ \ - 0, 0, 0, \ + /* has_tls_get_addr_call, has_gp_reloc, need_finalize_relax, */ \ + 0, 0, 0, \ \ - /* vma, lma, size, rawsize */ \ - 0, 0, 0, 0, \ + /* reloc_done, vma, lma, size, rawsize */ \ + 0, 0, 0, 0, 0, \ \ /* output_offset, output_section, alignment_power, */ \ 0, (struct bfd_section *) &SEC, 0, \ \ /* relocation, orelocation, reloc_count, filepos, rel_filepos, */ \ NULL, NULL, 0, 0, 0, \ \ /* line_filepos, userdata, contents, lineno, lineno_count, */ \ 0, NULL, NULL, NULL, 0, \ \ /* entsize, kept_section, moving_line_filepos, */ \ 0, NULL, 0, \ \ /* target_index, used_by_bfd, constructor_chain, owner, */ \ 0, NULL, NULL, NULL, \ \ /* symbol, symbol_ptr_ptr, */ \ (struct bfd_symbol *) SYM, &SEC.symbol, \ \ /* map_head, map_tail */ \ { NULL }, { NULL } \ } void bfd_section_list_clear (bfd *); asection *bfd_get_section_by_name (bfd *abfd, const char *name); asection *bfd_get_section_by_name_if (bfd *abfd, const char *name, bfd_boolean (*func) (bfd *abfd, asection *sect, void *obj), void *obj); char *bfd_get_unique_section_name (bfd *abfd, const char *templat, int *count); asection *bfd_make_section_old_way (bfd *abfd, const char *name); asection *bfd_make_section_anyway_with_flags (bfd *abfd, const char *name, flagword flags); asection *bfd_make_section_anyway (bfd *abfd, const char *name); asection *bfd_make_section_with_flags (bfd *, const char *name, flagword flags); asection *bfd_make_section (bfd *, const char *name); bfd_boolean bfd_set_section_flags (bfd *abfd, asection *sec, flagword flags); void bfd_map_over_sections (bfd *abfd, void (*func) (bfd *abfd, asection *sect, void *obj), void *obj); asection *bfd_sections_find_if (bfd *abfd, bfd_boolean (*operation) (bfd *abfd, asection *sect, void *obj), void *obj); bfd_boolean bfd_set_section_size (bfd *abfd, asection *sec, bfd_size_type val); bfd_boolean bfd_set_section_contents (bfd *abfd, asection *section, const void *data, file_ptr offset, bfd_size_type count); bfd_boolean bfd_get_section_contents (bfd *abfd, asection *section, void *location, file_ptr offset, bfd_size_type count); bfd_boolean bfd_malloc_and_get_section (bfd *abfd, asection *section, bfd_byte **buf); bfd_boolean bfd_copy_private_section_data (bfd *ibfd, asection *isec, bfd *obfd, asection *osec); #define bfd_copy_private_section_data(ibfd, isection, obfd, osection) \ BFD_SEND (obfd, _bfd_copy_private_section_data, \ (ibfd, isection, obfd, osection)) bfd_boolean bfd_generic_is_group_section (bfd *, const asection *sec); bfd_boolean bfd_generic_discard_group (bfd *abfd, asection *group); /* Extracted from archures.c. */ enum bfd_architecture { bfd_arch_unknown, /* File arch not known. */ bfd_arch_obscure, /* Arch known, not one of these. */ bfd_arch_m68k, /* Motorola 68xxx */ #define bfd_mach_m68000 1 #define bfd_mach_m68008 2 #define bfd_mach_m68010 3 #define bfd_mach_m68020 4 #define bfd_mach_m68030 5 #define bfd_mach_m68040 6 #define bfd_mach_m68060 7 #define bfd_mach_cpu32 8 #define bfd_mach_fido 9 #define bfd_mach_mcf_isa_a_nodiv 10 #define bfd_mach_mcf_isa_a 11 #define bfd_mach_mcf_isa_a_mac 12 #define bfd_mach_mcf_isa_a_emac 13 #define bfd_mach_mcf_isa_aplus 14 #define bfd_mach_mcf_isa_aplus_mac 15 #define bfd_mach_mcf_isa_aplus_emac 16 #define bfd_mach_mcf_isa_b_nousp 17 #define bfd_mach_mcf_isa_b_nousp_mac 18 #define bfd_mach_mcf_isa_b_nousp_emac 19 #define bfd_mach_mcf_isa_b 20 #define bfd_mach_mcf_isa_b_mac 21 #define bfd_mach_mcf_isa_b_emac 22 #define bfd_mach_mcf_isa_b_float 23 #define bfd_mach_mcf_isa_b_float_mac 24 #define bfd_mach_mcf_isa_b_float_emac 25 #define bfd_mach_mcf_isa_c 26 #define bfd_mach_mcf_isa_c_mac 27 #define bfd_mach_mcf_isa_c_emac 28 bfd_arch_vax, /* DEC Vax */ bfd_arch_i960, /* Intel 960 */ /* The order of the following is important. lower number indicates a machine type that only accepts a subset of the instructions available to machines with higher numbers. The exception is the "ca", which is incompatible with all other machines except "core". */ #define bfd_mach_i960_core 1 #define bfd_mach_i960_ka_sa 2 #define bfd_mach_i960_kb_sb 3 #define bfd_mach_i960_mc 4 #define bfd_mach_i960_xa 5 #define bfd_mach_i960_ca 6 #define bfd_mach_i960_jx 7 #define bfd_mach_i960_hx 8 bfd_arch_or32, /* OpenRISC 32 */ bfd_arch_sparc, /* SPARC */ #define bfd_mach_sparc 1 /* The difference between v8plus and v9 is that v9 is a true 64 bit env. */ #define bfd_mach_sparc_sparclet 2 #define bfd_mach_sparc_sparclite 3 #define bfd_mach_sparc_v8plus 4 #define bfd_mach_sparc_v8plusa 5 /* with ultrasparc add'ns. */ #define bfd_mach_sparc_sparclite_le 6 #define bfd_mach_sparc_v9 7 #define bfd_mach_sparc_v9a 8 /* with ultrasparc add'ns. */ #define bfd_mach_sparc_v8plusb 9 /* with cheetah add'ns. */ #define bfd_mach_sparc_v9b 10 /* with cheetah add'ns. */ /* Nonzero if MACH has the v9 instruction set. */ #define bfd_mach_sparc_v9_p(mach) \ ((mach) >= bfd_mach_sparc_v8plus && (mach) <= bfd_mach_sparc_v9b \ && (mach) != bfd_mach_sparc_sparclite_le) /* Nonzero if MACH is a 64 bit sparc architecture. */ #define bfd_mach_sparc_64bit_p(mach) \ ((mach) >= bfd_mach_sparc_v9 && (mach) != bfd_mach_sparc_v8plusb) bfd_arch_spu, /* PowerPC SPU */ #define bfd_mach_spu 256 bfd_arch_mips, /* MIPS Rxxxx */ #define bfd_mach_mips3000 3000 #define bfd_mach_mips3900 3900 #define bfd_mach_mips4000 4000 #define bfd_mach_mips4010 4010 #define bfd_mach_mips4100 4100 #define bfd_mach_mips4111 4111 #define bfd_mach_mips4120 4120 #define bfd_mach_mips4300 4300 #define bfd_mach_mips4400 4400 #define bfd_mach_mips4600 4600 #define bfd_mach_mips4650 4650 #define bfd_mach_mips5000 5000 #define bfd_mach_mips5400 5400 #define bfd_mach_mips5500 5500 #define bfd_mach_mips6000 6000 #define bfd_mach_mips7000 7000 #define bfd_mach_mips8000 8000 #define bfd_mach_mips9000 9000 #define bfd_mach_mips10000 10000 #define bfd_mach_mips12000 12000 #define bfd_mach_mips16 16 #define bfd_mach_mips5 5 #define bfd_mach_mips_octeon 6502 #define bfd_mach_mips_sb1 12310201 /* octal 'SB', 01 */ #define bfd_mach_mipsisa32 32 #define bfd_mach_mipsisa32r2 33 #define bfd_mach_mipsisa64 64 #define bfd_mach_mipsisa64r2 65 bfd_arch_i386, /* Intel 386 */ #define bfd_mach_i386_i386 1 #define bfd_mach_i386_i8086 2 #define bfd_mach_i386_i386_intel_syntax 3 #define bfd_mach_x86_64 64 #define bfd_mach_x86_64_intel_syntax 65 bfd_arch_we32k, /* AT&T WE32xxx */ bfd_arch_tahoe, /* CCI/Harris Tahoe */ bfd_arch_i860, /* Intel 860 */ bfd_arch_i370, /* IBM 360/370 Mainframes */ bfd_arch_romp, /* IBM ROMP PC/RT */ bfd_arch_convex, /* Convex */ bfd_arch_m88k, /* Motorola 88xxx */ bfd_arch_m98k, /* Motorola 98xxx */ bfd_arch_pyramid, /* Pyramid Technology */ bfd_arch_h8300, /* Renesas H8/300 (formerly Hitachi H8/300) */ #define bfd_mach_h8300 1 #define bfd_mach_h8300h 2 #define bfd_mach_h8300s 3 #define bfd_mach_h8300hn 4 #define bfd_mach_h8300sn 5 #define bfd_mach_h8300sx 6 #define bfd_mach_h8300sxn 7 bfd_arch_pdp11, /* DEC PDP-11 */ bfd_arch_powerpc, /* PowerPC */ #define bfd_mach_ppc 32 #define bfd_mach_ppc64 64 #define bfd_mach_ppc_403 403 #define bfd_mach_ppc_403gc 4030 #define bfd_mach_ppc_505 505 #define bfd_mach_ppc_601 601 #define bfd_mach_ppc_602 602 #define bfd_mach_ppc_603 603 #define bfd_mach_ppc_ec603e 6031 #define bfd_mach_ppc_604 604 #define bfd_mach_ppc_620 620 #define bfd_mach_ppc_630 630 #define bfd_mach_ppc_750 750 #define bfd_mach_ppc_860 860 #define bfd_mach_ppc_a35 35 #define bfd_mach_ppc_rs64ii 642 #define bfd_mach_ppc_rs64iii 643 #define bfd_mach_ppc_7400 7400 #define bfd_mach_ppc_e500 500 bfd_arch_rs6000, /* IBM RS/6000 */ #define bfd_mach_rs6k 6000 #define bfd_mach_rs6k_rs1 6001 #define bfd_mach_rs6k_rsc 6003 #define bfd_mach_rs6k_rs2 6002 bfd_arch_hppa, /* HP PA RISC */ #define bfd_mach_hppa10 10 #define bfd_mach_hppa11 11 #define bfd_mach_hppa20 20 #define bfd_mach_hppa20w 25 bfd_arch_d10v, /* Mitsubishi D10V */ #define bfd_mach_d10v 1 #define bfd_mach_d10v_ts2 2 #define bfd_mach_d10v_ts3 3 bfd_arch_d30v, /* Mitsubishi D30V */ bfd_arch_dlx, /* DLX */ bfd_arch_m68hc11, /* Motorola 68HC11 */ bfd_arch_m68hc12, /* Motorola 68HC12 */ #define bfd_mach_m6812_default 0 #define bfd_mach_m6812 1 #define bfd_mach_m6812s 2 bfd_arch_z8k, /* Zilog Z8000 */ #define bfd_mach_z8001 1 #define bfd_mach_z8002 2 bfd_arch_h8500, /* Renesas H8/500 (formerly Hitachi H8/500) */ bfd_arch_sh, /* Renesas / SuperH SH (formerly Hitachi SH) */ #define bfd_mach_sh 1 #define bfd_mach_sh2 0x20 #define bfd_mach_sh_dsp 0x2d #define bfd_mach_sh2a 0x2a #define bfd_mach_sh2a_nofpu 0x2b #define bfd_mach_sh2a_nofpu_or_sh4_nommu_nofpu 0x2a1 #define bfd_mach_sh2a_nofpu_or_sh3_nommu 0x2a2 #define bfd_mach_sh2a_or_sh4 0x2a3 #define bfd_mach_sh2a_or_sh3e 0x2a4 #define bfd_mach_sh2e 0x2e #define bfd_mach_sh3 0x30 #define bfd_mach_sh3_nommu 0x31 #define bfd_mach_sh3_dsp 0x3d #define bfd_mach_sh3e 0x3e #define bfd_mach_sh4 0x40 #define bfd_mach_sh4_nofpu 0x41 #define bfd_mach_sh4_nommu_nofpu 0x42 #define bfd_mach_sh4a 0x4a #define bfd_mach_sh4a_nofpu 0x4b #define bfd_mach_sh4al_dsp 0x4d #define bfd_mach_sh5 0x50 bfd_arch_alpha, /* Dec Alpha */ #define bfd_mach_alpha_ev4 0x10 #define bfd_mach_alpha_ev5 0x20 #define bfd_mach_alpha_ev6 0x30 bfd_arch_arm, /* Advanced Risc Machines ARM. */ #define bfd_mach_arm_unknown 0 #define bfd_mach_arm_2 1 #define bfd_mach_arm_2a 2 #define bfd_mach_arm_3 3 #define bfd_mach_arm_3M 4 #define bfd_mach_arm_4 5 #define bfd_mach_arm_4T 6 #define bfd_mach_arm_5 7 #define bfd_mach_arm_5T 8 #define bfd_mach_arm_5TE 9 #define bfd_mach_arm_XScale 10 #define bfd_mach_arm_ep9312 11 #define bfd_mach_arm_iWMMXt 12 #define bfd_mach_arm_iWMMXt2 13 bfd_arch_ns32k, /* National Semiconductors ns32000 */ bfd_arch_w65, /* WDC 65816 */ bfd_arch_tic30, /* Texas Instruments TMS320C30 */ bfd_arch_tic4x, /* Texas Instruments TMS320C3X/4X */ #define bfd_mach_tic3x 30 #define bfd_mach_tic4x 40 bfd_arch_tic54x, /* Texas Instruments TMS320C54X */ bfd_arch_tic80, /* TI TMS320c80 (MVP) */ bfd_arch_v850, /* NEC V850 */ #define bfd_mach_v850 1 #define bfd_mach_v850e 'E' #define bfd_mach_v850e1 '1' bfd_arch_arc, /* ARC Cores */ #define bfd_mach_arc_5 5 #define bfd_mach_arc_6 6 #define bfd_mach_arc_7 7 #define bfd_mach_arc_8 8 bfd_arch_m32c, /* Renesas M16C/M32C. */ #define bfd_mach_m16c 0x75 #define bfd_mach_m32c 0x78 bfd_arch_m32r, /* Renesas M32R (formerly Mitsubishi M32R/D) */ #define bfd_mach_m32r 1 /* For backwards compatibility. */ #define bfd_mach_m32rx 'x' #define bfd_mach_m32r2 '2' bfd_arch_mn10200, /* Matsushita MN10200 */ bfd_arch_mn10300, /* Matsushita MN10300 */ #define bfd_mach_mn10300 300 #define bfd_mach_am33 330 #define bfd_mach_am33_2 332 bfd_arch_fr30, #define bfd_mach_fr30 0x46523330 bfd_arch_frv, #define bfd_mach_frv 1 #define bfd_mach_frvsimple 2 #define bfd_mach_fr300 300 #define bfd_mach_fr400 400 #define bfd_mach_fr450 450 #define bfd_mach_frvtomcat 499 /* fr500 prototype */ #define bfd_mach_fr500 500 #define bfd_mach_fr550 550 bfd_arch_mcore, bfd_arch_mep, #define bfd_mach_mep 1 #define bfd_mach_mep_h1 0x6831 bfd_arch_ia64, /* HP/Intel ia64 */ #define bfd_mach_ia64_elf64 64 #define bfd_mach_ia64_elf32 32 bfd_arch_ip2k, /* Ubicom IP2K microcontrollers. */ #define bfd_mach_ip2022 1 #define bfd_mach_ip2022ext 2 bfd_arch_iq2000, /* Vitesse IQ2000. */ #define bfd_mach_iq2000 1 #define bfd_mach_iq10 2 bfd_arch_mt, #define bfd_mach_ms1 1 #define bfd_mach_mrisc2 2 #define bfd_mach_ms2 3 bfd_arch_pj, bfd_arch_avr, /* Atmel AVR microcontrollers. */ #define bfd_mach_avr1 1 #define bfd_mach_avr2 2 #define bfd_mach_avr3 3 #define bfd_mach_avr4 4 #define bfd_mach_avr5 5 #define bfd_mach_avr6 6 bfd_arch_bfin, /* ADI Blackfin */ #define bfd_mach_bfin 1 bfd_arch_cr16, /* National Semiconductor CompactRISC (ie CR16). */ #define bfd_mach_cr16 1 bfd_arch_cr16c, /* National Semiconductor CompactRISC. */ #define bfd_mach_cr16c 1 bfd_arch_crx, /* National Semiconductor CRX. */ #define bfd_mach_crx 1 bfd_arch_cris, /* Axis CRIS */ #define bfd_mach_cris_v0_v10 255 #define bfd_mach_cris_v32 32 #define bfd_mach_cris_v10_v32 1032 bfd_arch_s390, /* IBM s390 */ #define bfd_mach_s390_31 31 #define bfd_mach_s390_64 64 bfd_arch_score, /* Sunplus score */ bfd_arch_openrisc, /* OpenRISC */ bfd_arch_mmix, /* Donald Knuth's educational processor. */ bfd_arch_xstormy16, #define bfd_mach_xstormy16 1 bfd_arch_msp430, /* Texas Instruments MSP430 architecture. */ #define bfd_mach_msp11 11 #define bfd_mach_msp110 110 #define bfd_mach_msp12 12 #define bfd_mach_msp13 13 #define bfd_mach_msp14 14 #define bfd_mach_msp15 15 #define bfd_mach_msp16 16 #define bfd_mach_msp21 21 #define bfd_mach_msp31 31 #define bfd_mach_msp32 32 #define bfd_mach_msp33 33 #define bfd_mach_msp41 41 #define bfd_mach_msp42 42 #define bfd_mach_msp43 43 #define bfd_mach_msp44 44 bfd_arch_xc16x, /* Infineon's XC16X Series. */ #define bfd_mach_xc16x 1 #define bfd_mach_xc16xl 2 #define bfd_mach_xc16xs 3 bfd_arch_xtensa, /* Tensilica's Xtensa cores. */ #define bfd_mach_xtensa 1 bfd_arch_maxq, /* Dallas MAXQ 10/20 */ #define bfd_mach_maxq10 10 #define bfd_mach_maxq20 20 bfd_arch_z80, #define bfd_mach_z80strict 1 /* No undocumented opcodes. */ #define bfd_mach_z80 3 /* With ixl, ixh, iyl, and iyh. */ #define bfd_mach_z80full 7 /* All undocumented instructions. */ #define bfd_mach_r800 11 /* R800: successor with multiplication. */ bfd_arch_last }; typedef struct bfd_arch_info { int bits_per_word; int bits_per_address; int bits_per_byte; enum bfd_architecture arch; unsigned long mach; const char *arch_name; const char *printable_name; unsigned int section_align_power; /* TRUE if this is the default machine for the architecture. The default arch should be the first entry for an arch so that all the entries for that arch can be accessed via <>. */ bfd_boolean the_default; const struct bfd_arch_info * (*compatible) (const struct bfd_arch_info *a, const struct bfd_arch_info *b); bfd_boolean (*scan) (const struct bfd_arch_info *, const char *); const struct bfd_arch_info *next; } bfd_arch_info_type; const char *bfd_printable_name (bfd *abfd); const bfd_arch_info_type *bfd_scan_arch (const char *string); const char **bfd_arch_list (void); const bfd_arch_info_type *bfd_arch_get_compatible (const bfd *abfd, const bfd *bbfd, bfd_boolean accept_unknowns); void bfd_set_arch_info (bfd *abfd, const bfd_arch_info_type *arg); enum bfd_architecture bfd_get_arch (bfd *abfd); unsigned long bfd_get_mach (bfd *abfd); unsigned int bfd_arch_bits_per_byte (bfd *abfd); unsigned int bfd_arch_bits_per_address (bfd *abfd); const bfd_arch_info_type *bfd_get_arch_info (bfd *abfd); const bfd_arch_info_type *bfd_lookup_arch (enum bfd_architecture arch, unsigned long machine); const char *bfd_printable_arch_mach (enum bfd_architecture arch, unsigned long machine); unsigned int bfd_octets_per_byte (bfd *abfd); unsigned int bfd_arch_mach_octets_per_byte (enum bfd_architecture arch, unsigned long machine); /* Extracted from reloc.c. */ typedef enum bfd_reloc_status { /* No errors detected. */ bfd_reloc_ok, /* The relocation was performed, but there was an overflow. */ bfd_reloc_overflow, /* The address to relocate was not within the section supplied. */ bfd_reloc_outofrange, /* Used by special functions. */ bfd_reloc_continue, /* Unsupported relocation size requested. */ bfd_reloc_notsupported, /* Unused. */ bfd_reloc_other, /* The symbol to relocate against was undefined. */ bfd_reloc_undefined, /* The relocation was performed, but may not be ok - presently generated only when linking i960 coff files with i960 b.out symbols. If this type is returned, the error_message argument to bfd_perform_relocation will be set. */ bfd_reloc_dangerous } bfd_reloc_status_type; typedef struct reloc_cache_entry { /* A pointer into the canonical table of pointers. */ struct bfd_symbol **sym_ptr_ptr; /* offset in section. */ bfd_size_type address; /* addend for relocation value. */ bfd_vma addend; /* Pointer to how to perform the required relocation. */ reloc_howto_type *howto; } arelent; enum complain_overflow { /* Do not complain on overflow. */ complain_overflow_dont, /* Complain if the value overflows when considered as a signed number one bit larger than the field. ie. A bitfield of N bits is allowed to represent -2**n to 2**n-1. */ complain_overflow_bitfield, /* Complain if the value overflows when considered as a signed number. */ complain_overflow_signed, /* Complain if the value overflows when considered as an unsigned number. */ complain_overflow_unsigned }; struct reloc_howto_struct { /* The type field has mainly a documentary use - the back end can do what it wants with it, though normally the back end's external idea of what a reloc number is stored in this field. For example, a PC relative word relocation in a coff environment has the type 023 - because that's what the outside world calls a R_PCRWORD reloc. */ unsigned int type; /* The value the final relocation is shifted right by. This drops unwanted data from the relocation. */ unsigned int rightshift; /* The size of the item to be relocated. This is *not* a power-of-two measure. To get the number of bytes operated on by a type of relocation, use bfd_get_reloc_size. */ int size; /* The number of bits in the item to be relocated. This is used when doing overflow checking. */ unsigned int bitsize; /* Notes that the relocation is relative to the location in the data section of the addend. The relocation function will subtract from the relocation value the address of the location being relocated. */ bfd_boolean pc_relative; /* The bit position of the reloc value in the destination. The relocated value is left shifted by this amount. */ unsigned int bitpos; /* What type of overflow error should be checked for when relocating. */ enum complain_overflow complain_on_overflow; /* If this field is non null, then the supplied function is called rather than the normal function. This allows really strange relocation methods to be accommodated (e.g., i960 callj instructions). */ bfd_reloc_status_type (*special_function) (bfd *, arelent *, struct bfd_symbol *, void *, asection *, bfd *, char **); /* The textual name of the relocation type. */ char *name; /* Some formats record a relocation addend in the section contents rather than with the relocation. For ELF formats this is the distinction between USE_REL and USE_RELA (though the code checks for USE_REL == 1/0). The value of this field is TRUE if the addend is recorded with the section contents; when performing a partial link (ld -r) the section contents (the data) will be modified. The value of this field is FALSE if addends are recorded with the relocation (in arelent.addend); when performing a partial link the relocation will be modified. All relocations for all ELF USE_RELA targets should set this field to FALSE (values of TRUE should be looked on with suspicion). However, the converse is not true: not all relocations of all ELF USE_REL targets set this field to TRUE. Why this is so is peculiar to each particular target. For relocs that aren't used in partial links (e.g. GOT stuff) it doesn't matter what this is set to. */ bfd_boolean partial_inplace; /* src_mask selects the part of the instruction (or data) to be used in the relocation sum. If the target relocations don't have an addend in the reloc, eg. ELF USE_REL, src_mask will normally equal dst_mask to extract the addend from the section contents. If relocations do have an addend in the reloc, eg. ELF USE_RELA, this field should be zero. Non-zero values for ELF USE_RELA targets are bogus as in those cases the value in the dst_mask part of the section contents should be treated as garbage. */ bfd_vma src_mask; /* dst_mask selects which parts of the instruction (or data) are replaced with a relocated value. */ bfd_vma dst_mask; /* When some formats create PC relative instructions, they leave the value of the pc of the place being relocated in the offset slot of the instruction, so that a PC relative relocation can be made just by adding in an ordinary offset (e.g., sun3 a.out). Some formats leave the displacement part of an instruction empty (e.g., m88k bcs); this flag signals the fact. */ bfd_boolean pcrel_offset; }; #define HOWTO(C, R, S, B, P, BI, O, SF, NAME, INPLACE, MASKSRC, MASKDST, PC) \ { (unsigned) C, R, S, B, P, BI, O, SF, NAME, INPLACE, MASKSRC, MASKDST, PC } #define NEWHOWTO(FUNCTION, NAME, SIZE, REL, IN) \ HOWTO (0, 0, SIZE, 0, REL, 0, complain_overflow_dont, FUNCTION, \ NAME, FALSE, 0, 0, IN) #define EMPTY_HOWTO(C) \ HOWTO ((C), 0, 0, 0, FALSE, 0, complain_overflow_dont, NULL, \ NULL, FALSE, 0, 0, FALSE) #define HOWTO_PREPARE(relocation, symbol) \ { \ if (symbol != NULL) \ { \ if (bfd_is_com_section (symbol->section)) \ { \ relocation = 0; \ } \ else \ { \ relocation = symbol->value; \ } \ } \ } unsigned int bfd_get_reloc_size (reloc_howto_type *); typedef struct relent_chain { arelent relent; struct relent_chain *next; } arelent_chain; bfd_reloc_status_type bfd_check_overflow (enum complain_overflow how, unsigned int bitsize, unsigned int rightshift, unsigned int addrsize, bfd_vma relocation); bfd_reloc_status_type bfd_perform_relocation (bfd *abfd, arelent *reloc_entry, void *data, asection *input_section, bfd *output_bfd, char **error_message); bfd_reloc_status_type bfd_install_relocation (bfd *abfd, arelent *reloc_entry, void *data, bfd_vma data_start, asection *input_section, char **error_message); enum bfd_reloc_code_real { _dummy_first_bfd_reloc_code_real, /* Basic absolute relocations of N bits. */ BFD_RELOC_64, BFD_RELOC_32, BFD_RELOC_26, BFD_RELOC_24, BFD_RELOC_16, BFD_RELOC_14, BFD_RELOC_8, /* PC-relative relocations. Sometimes these are relative to the address of the relocation itself; sometimes they are relative to the start of the section containing the relocation. It depends on the specific target. The 24-bit relocation is used in some Intel 960 configurations. */ BFD_RELOC_64_PCREL, BFD_RELOC_32_PCREL, BFD_RELOC_24_PCREL, BFD_RELOC_16_PCREL, BFD_RELOC_12_PCREL, BFD_RELOC_8_PCREL, /* Section relative relocations. Some targets need this for DWARF2. */ BFD_RELOC_32_SECREL, /* For ELF. */ BFD_RELOC_32_GOT_PCREL, BFD_RELOC_16_GOT_PCREL, BFD_RELOC_8_GOT_PCREL, BFD_RELOC_32_GOTOFF, BFD_RELOC_16_GOTOFF, BFD_RELOC_LO16_GOTOFF, BFD_RELOC_HI16_GOTOFF, BFD_RELOC_HI16_S_GOTOFF, BFD_RELOC_8_GOTOFF, BFD_RELOC_64_PLT_PCREL, BFD_RELOC_32_PLT_PCREL, BFD_RELOC_24_PLT_PCREL, BFD_RELOC_16_PLT_PCREL, BFD_RELOC_8_PLT_PCREL, BFD_RELOC_64_PLTOFF, BFD_RELOC_32_PLTOFF, BFD_RELOC_16_PLTOFF, BFD_RELOC_LO16_PLTOFF, BFD_RELOC_HI16_PLTOFF, BFD_RELOC_HI16_S_PLTOFF, BFD_RELOC_8_PLTOFF, /* Relocations used by 68K ELF. */ BFD_RELOC_68K_GLOB_DAT, BFD_RELOC_68K_JMP_SLOT, BFD_RELOC_68K_RELATIVE, /* Linkage-table relative. */ BFD_RELOC_32_BASEREL, BFD_RELOC_16_BASEREL, BFD_RELOC_LO16_BASEREL, BFD_RELOC_HI16_BASEREL, BFD_RELOC_HI16_S_BASEREL, BFD_RELOC_8_BASEREL, BFD_RELOC_RVA, /* Absolute 8-bit relocation, but used to form an address like 0xFFnn. */ BFD_RELOC_8_FFnn, /* These PC-relative relocations are stored as word displacements -- i.e., byte displacements shifted right two bits. The 30-bit word displacement (<<32_PCREL_S2>> -- 32 bits, shifted 2) is used on the SPARC. (SPARC tools generally refer to this as <>.) The signed 16-bit displacement is used on the MIPS, and the 23-bit displacement is used on the Alpha. */ BFD_RELOC_32_PCREL_S2, BFD_RELOC_16_PCREL_S2, BFD_RELOC_23_PCREL_S2, /* High 22 bits and low 10 bits of 32-bit value, placed into lower bits of the target word. These are used on the SPARC. */ BFD_RELOC_HI22, BFD_RELOC_LO10, /* For systems that allocate a Global Pointer register, these are displacements off that register. These relocation types are handled specially, because the value the register will have is decided relatively late. */ BFD_RELOC_GPREL16, BFD_RELOC_GPREL32, /* Reloc types used for i960/b.out. */ BFD_RELOC_I960_CALLJ, /* SPARC ELF relocations. There is probably some overlap with other relocation types already defined. */ BFD_RELOC_NONE, BFD_RELOC_SPARC_WDISP22, BFD_RELOC_SPARC22, BFD_RELOC_SPARC13, BFD_RELOC_SPARC_GOT10, BFD_RELOC_SPARC_GOT13, BFD_RELOC_SPARC_GOT22, BFD_RELOC_SPARC_PC10, BFD_RELOC_SPARC_PC22, BFD_RELOC_SPARC_WPLT30, BFD_RELOC_SPARC_COPY, BFD_RELOC_SPARC_GLOB_DAT, BFD_RELOC_SPARC_JMP_SLOT, BFD_RELOC_SPARC_RELATIVE, BFD_RELOC_SPARC_UA16, BFD_RELOC_SPARC_UA32, BFD_RELOC_SPARC_UA64, /* I think these are specific to SPARC a.out (e.g., Sun 4). */ BFD_RELOC_SPARC_BASE13, BFD_RELOC_SPARC_BASE22, /* SPARC64 relocations */ #define BFD_RELOC_SPARC_64 BFD_RELOC_64 BFD_RELOC_SPARC_10, BFD_RELOC_SPARC_11, BFD_RELOC_SPARC_OLO10, BFD_RELOC_SPARC_HH22, BFD_RELOC_SPARC_HM10, BFD_RELOC_SPARC_LM22, BFD_RELOC_SPARC_PC_HH22, BFD_RELOC_SPARC_PC_HM10, BFD_RELOC_SPARC_PC_LM22, BFD_RELOC_SPARC_WDISP16, BFD_RELOC_SPARC_WDISP19, BFD_RELOC_SPARC_7, BFD_RELOC_SPARC_6, BFD_RELOC_SPARC_5, #define BFD_RELOC_SPARC_DISP64 BFD_RELOC_64_PCREL BFD_RELOC_SPARC_PLT32, BFD_RELOC_SPARC_PLT64, BFD_RELOC_SPARC_HIX22, BFD_RELOC_SPARC_LOX10, BFD_RELOC_SPARC_H44, BFD_RELOC_SPARC_M44, BFD_RELOC_SPARC_L44, BFD_RELOC_SPARC_REGISTER, /* SPARC little endian relocation */ BFD_RELOC_SPARC_REV32, /* SPARC TLS relocations */ BFD_RELOC_SPARC_TLS_GD_HI22, BFD_RELOC_SPARC_TLS_GD_LO10, BFD_RELOC_SPARC_TLS_GD_ADD, BFD_RELOC_SPARC_TLS_GD_CALL, BFD_RELOC_SPARC_TLS_LDM_HI22, BFD_RELOC_SPARC_TLS_LDM_LO10, BFD_RELOC_SPARC_TLS_LDM_ADD, BFD_RELOC_SPARC_TLS_LDM_CALL, BFD_RELOC_SPARC_TLS_LDO_HIX22, BFD_RELOC_SPARC_TLS_LDO_LOX10, BFD_RELOC_SPARC_TLS_LDO_ADD, BFD_RELOC_SPARC_TLS_IE_HI22, BFD_RELOC_SPARC_TLS_IE_LO10, BFD_RELOC_SPARC_TLS_IE_LD, BFD_RELOC_SPARC_TLS_IE_LDX, BFD_RELOC_SPARC_TLS_IE_ADD, BFD_RELOC_SPARC_TLS_LE_HIX22, BFD_RELOC_SPARC_TLS_LE_LOX10, BFD_RELOC_SPARC_TLS_DTPMOD32, BFD_RELOC_SPARC_TLS_DTPMOD64, BFD_RELOC_SPARC_TLS_DTPOFF32, BFD_RELOC_SPARC_TLS_DTPOFF64, BFD_RELOC_SPARC_TLS_TPOFF32, BFD_RELOC_SPARC_TLS_TPOFF64, /* SPU Relocations. */ BFD_RELOC_SPU_IMM7, BFD_RELOC_SPU_IMM8, BFD_RELOC_SPU_IMM10, BFD_RELOC_SPU_IMM10W, BFD_RELOC_SPU_IMM16, BFD_RELOC_SPU_IMM16W, BFD_RELOC_SPU_IMM18, BFD_RELOC_SPU_PCREL9a, BFD_RELOC_SPU_PCREL9b, BFD_RELOC_SPU_PCREL16, BFD_RELOC_SPU_LO16, BFD_RELOC_SPU_HI16, BFD_RELOC_SPU_PPU32, BFD_RELOC_SPU_PPU64, /* Alpha ECOFF and ELF relocations. Some of these treat the symbol or "addend" in some special way. For GPDISP_HI16 ("gpdisp") relocations, the symbol is ignored when writing; when reading, it will be the absolute section symbol. The addend is the displacement in bytes of the "lda" instruction from the "ldah" instruction (which is at the address of this reloc). */ BFD_RELOC_ALPHA_GPDISP_HI16, /* For GPDISP_LO16 ("ignore") relocations, the symbol is handled as with GPDISP_HI16 relocs. The addend is ignored when writing the relocations out, and is filled in with the file's GP value on reading, for convenience. */ BFD_RELOC_ALPHA_GPDISP_LO16, /* The ELF GPDISP relocation is exactly the same as the GPDISP_HI16 relocation except that there is no accompanying GPDISP_LO16 relocation. */ BFD_RELOC_ALPHA_GPDISP, /* The Alpha LITERAL/LITUSE relocs are produced by a symbol reference; the assembler turns it into a LDQ instruction to load the address of the symbol, and then fills in a register in the real instruction. The LITERAL reloc, at the LDQ instruction, refers to the .lita section symbol. The addend is ignored when writing, but is filled in with the file's GP value on reading, for convenience, as with the GPDISP_LO16 reloc. The ELF_LITERAL reloc is somewhere between 16_GOTOFF and GPDISP_LO16. It should refer to the symbol to be referenced, as with 16_GOTOFF, but it generates output not based on the position within the .got section, but relative to the GP value chosen for the file during the final link stage. The LITUSE reloc, on the instruction using the loaded address, gives information to the linker that it might be able to use to optimize away some literal section references. The symbol is ignored (read as the absolute section symbol), and the "addend" indicates the type of instruction using the register: 1 - "memory" fmt insn 2 - byte-manipulation (byte offset reg) 3 - jsr (target of branch) */ BFD_RELOC_ALPHA_LITERAL, BFD_RELOC_ALPHA_ELF_LITERAL, BFD_RELOC_ALPHA_LITUSE, /* The HINT relocation indicates a value that should be filled into the "hint" field of a jmp/jsr/ret instruction, for possible branch- prediction logic which may be provided on some processors. */ BFD_RELOC_ALPHA_HINT, /* The LINKAGE relocation outputs a linkage pair in the object file, which is filled by the linker. */ BFD_RELOC_ALPHA_LINKAGE, /* The CODEADDR relocation outputs a STO_CA in the object file, which is filled by the linker. */ BFD_RELOC_ALPHA_CODEADDR, /* The GPREL_HI/LO relocations together form a 32-bit offset from the GP register. */ BFD_RELOC_ALPHA_GPREL_HI16, BFD_RELOC_ALPHA_GPREL_LO16, /* Like BFD_RELOC_23_PCREL_S2, except that the source and target must share a common GP, and the target address is adjusted for STO_ALPHA_STD_GPLOAD. */ BFD_RELOC_ALPHA_BRSGP, /* Alpha thread-local storage relocations. */ BFD_RELOC_ALPHA_TLSGD, BFD_RELOC_ALPHA_TLSLDM, BFD_RELOC_ALPHA_DTPMOD64, BFD_RELOC_ALPHA_GOTDTPREL16, BFD_RELOC_ALPHA_DTPREL64, BFD_RELOC_ALPHA_DTPREL_HI16, BFD_RELOC_ALPHA_DTPREL_LO16, BFD_RELOC_ALPHA_DTPREL16, BFD_RELOC_ALPHA_GOTTPREL16, BFD_RELOC_ALPHA_TPREL64, BFD_RELOC_ALPHA_TPREL_HI16, BFD_RELOC_ALPHA_TPREL_LO16, BFD_RELOC_ALPHA_TPREL16, /* Bits 27..2 of the relocation address shifted right 2 bits; simple reloc otherwise. */ BFD_RELOC_MIPS_JMP, /* The MIPS16 jump instruction. */ BFD_RELOC_MIPS16_JMP, /* MIPS16 GP relative reloc. */ BFD_RELOC_MIPS16_GPREL, /* High 16 bits of 32-bit value; simple reloc. */ BFD_RELOC_HI16, /* High 16 bits of 32-bit value but the low 16 bits will be sign extended and added to form the final result. If the low 16 bits form a negative number, we need to add one to the high value to compensate for the borrow when the low bits are added. */ BFD_RELOC_HI16_S, /* Low 16 bits. */ BFD_RELOC_LO16, /* High 16 bits of 32-bit pc-relative value */ BFD_RELOC_HI16_PCREL, /* High 16 bits of 32-bit pc-relative value, adjusted */ BFD_RELOC_HI16_S_PCREL, /* Low 16 bits of pc-relative value */ BFD_RELOC_LO16_PCREL, /* MIPS16 high 16 bits of 32-bit value. */ BFD_RELOC_MIPS16_HI16, /* MIPS16 high 16 bits of 32-bit value but the low 16 bits will be sign extended and added to form the final result. If the low 16 bits form a negative number, we need to add one to the high value to compensate for the borrow when the low bits are added. */ BFD_RELOC_MIPS16_HI16_S, /* MIPS16 low 16 bits. */ BFD_RELOC_MIPS16_LO16, /* Relocation against a MIPS literal section. */ BFD_RELOC_MIPS_LITERAL, /* MIPS ELF relocations. */ BFD_RELOC_MIPS_GOT16, BFD_RELOC_MIPS_CALL16, BFD_RELOC_MIPS_GOT_HI16, BFD_RELOC_MIPS_GOT_LO16, BFD_RELOC_MIPS_CALL_HI16, BFD_RELOC_MIPS_CALL_LO16, BFD_RELOC_MIPS_SUB, BFD_RELOC_MIPS_GOT_PAGE, BFD_RELOC_MIPS_GOT_OFST, BFD_RELOC_MIPS_GOT_DISP, BFD_RELOC_MIPS_SHIFT5, BFD_RELOC_MIPS_SHIFT6, BFD_RELOC_MIPS_INSERT_A, BFD_RELOC_MIPS_INSERT_B, BFD_RELOC_MIPS_DELETE, BFD_RELOC_MIPS_HIGHEST, BFD_RELOC_MIPS_HIGHER, BFD_RELOC_MIPS_SCN_DISP, BFD_RELOC_MIPS_REL16, BFD_RELOC_MIPS_RELGOT, BFD_RELOC_MIPS_JALR, BFD_RELOC_MIPS_TLS_DTPMOD32, BFD_RELOC_MIPS_TLS_DTPREL32, BFD_RELOC_MIPS_TLS_DTPMOD64, BFD_RELOC_MIPS_TLS_DTPREL64, BFD_RELOC_MIPS_TLS_GD, BFD_RELOC_MIPS_TLS_LDM, BFD_RELOC_MIPS_TLS_DTPREL_HI16, BFD_RELOC_MIPS_TLS_DTPREL_LO16, BFD_RELOC_MIPS_TLS_GOTTPREL, BFD_RELOC_MIPS_TLS_TPREL32, BFD_RELOC_MIPS_TLS_TPREL64, BFD_RELOC_MIPS_TLS_TPREL_HI16, BFD_RELOC_MIPS_TLS_TPREL_LO16, /* MIPS ELF relocations (VxWorks extensions). */ BFD_RELOC_MIPS_COPY, BFD_RELOC_MIPS_JUMP_SLOT, /* Fujitsu Frv Relocations. */ BFD_RELOC_FRV_LABEL16, BFD_RELOC_FRV_LABEL24, BFD_RELOC_FRV_LO16, BFD_RELOC_FRV_HI16, BFD_RELOC_FRV_GPREL12, BFD_RELOC_FRV_GPRELU12, BFD_RELOC_FRV_GPREL32, BFD_RELOC_FRV_GPRELHI, BFD_RELOC_FRV_GPRELLO, BFD_RELOC_FRV_GOT12, BFD_RELOC_FRV_GOTHI, BFD_RELOC_FRV_GOTLO, BFD_RELOC_FRV_FUNCDESC, BFD_RELOC_FRV_FUNCDESC_GOT12, BFD_RELOC_FRV_FUNCDESC_GOTHI, BFD_RELOC_FRV_FUNCDESC_GOTLO, BFD_RELOC_FRV_FUNCDESC_VALUE, BFD_RELOC_FRV_FUNCDESC_GOTOFF12, BFD_RELOC_FRV_FUNCDESC_GOTOFFHI, BFD_RELOC_FRV_FUNCDESC_GOTOFFLO, BFD_RELOC_FRV_GOTOFF12, BFD_RELOC_FRV_GOTOFFHI, BFD_RELOC_FRV_GOTOFFLO, BFD_RELOC_FRV_GETTLSOFF, BFD_RELOC_FRV_TLSDESC_VALUE, BFD_RELOC_FRV_GOTTLSDESC12, BFD_RELOC_FRV_GOTTLSDESCHI, BFD_RELOC_FRV_GOTTLSDESCLO, BFD_RELOC_FRV_TLSMOFF12, BFD_RELOC_FRV_TLSMOFFHI, BFD_RELOC_FRV_TLSMOFFLO, BFD_RELOC_FRV_GOTTLSOFF12, BFD_RELOC_FRV_GOTTLSOFFHI, BFD_RELOC_FRV_GOTTLSOFFLO, BFD_RELOC_FRV_TLSOFF, BFD_RELOC_FRV_TLSDESC_RELAX, BFD_RELOC_FRV_GETTLSOFF_RELAX, BFD_RELOC_FRV_TLSOFF_RELAX, BFD_RELOC_FRV_TLSMOFF, /* This is a 24bit GOT-relative reloc for the mn10300. */ BFD_RELOC_MN10300_GOTOFF24, /* This is a 32bit GOT-relative reloc for the mn10300, offset by two bytes in the instruction. */ BFD_RELOC_MN10300_GOT32, /* This is a 24bit GOT-relative reloc for the mn10300, offset by two bytes in the instruction. */ BFD_RELOC_MN10300_GOT24, /* This is a 16bit GOT-relative reloc for the mn10300, offset by two bytes in the instruction. */ BFD_RELOC_MN10300_GOT16, /* Copy symbol at runtime. */ BFD_RELOC_MN10300_COPY, /* Create GOT entry. */ BFD_RELOC_MN10300_GLOB_DAT, /* Create PLT entry. */ BFD_RELOC_MN10300_JMP_SLOT, /* Adjust by program base. */ BFD_RELOC_MN10300_RELATIVE, /* i386/elf relocations */ BFD_RELOC_386_GOT32, BFD_RELOC_386_PLT32, BFD_RELOC_386_COPY, BFD_RELOC_386_GLOB_DAT, BFD_RELOC_386_JUMP_SLOT, BFD_RELOC_386_RELATIVE, BFD_RELOC_386_GOTOFF, BFD_RELOC_386_GOTPC, BFD_RELOC_386_TLS_TPOFF, BFD_RELOC_386_TLS_IE, BFD_RELOC_386_TLS_GOTIE, BFD_RELOC_386_TLS_LE, BFD_RELOC_386_TLS_GD, BFD_RELOC_386_TLS_LDM, BFD_RELOC_386_TLS_LDO_32, BFD_RELOC_386_TLS_IE_32, BFD_RELOC_386_TLS_LE_32, BFD_RELOC_386_TLS_DTPMOD32, BFD_RELOC_386_TLS_DTPOFF32, BFD_RELOC_386_TLS_TPOFF32, BFD_RELOC_386_TLS_GOTDESC, BFD_RELOC_386_TLS_DESC_CALL, BFD_RELOC_386_TLS_DESC, /* x86-64/elf relocations */ BFD_RELOC_X86_64_GOT32, BFD_RELOC_X86_64_PLT32, BFD_RELOC_X86_64_COPY, BFD_RELOC_X86_64_GLOB_DAT, BFD_RELOC_X86_64_JUMP_SLOT, BFD_RELOC_X86_64_RELATIVE, BFD_RELOC_X86_64_GOTPCREL, BFD_RELOC_X86_64_32S, BFD_RELOC_X86_64_DTPMOD64, BFD_RELOC_X86_64_DTPOFF64, BFD_RELOC_X86_64_TPOFF64, BFD_RELOC_X86_64_TLSGD, BFD_RELOC_X86_64_TLSLD, BFD_RELOC_X86_64_DTPOFF32, BFD_RELOC_X86_64_GOTTPOFF, BFD_RELOC_X86_64_TPOFF32, BFD_RELOC_X86_64_GOTOFF64, BFD_RELOC_X86_64_GOTPC32, BFD_RELOC_X86_64_GOT64, BFD_RELOC_X86_64_GOTPCREL64, BFD_RELOC_X86_64_GOTPC64, BFD_RELOC_X86_64_GOTPLT64, BFD_RELOC_X86_64_PLTOFF64, BFD_RELOC_X86_64_GOTPC32_TLSDESC, BFD_RELOC_X86_64_TLSDESC_CALL, BFD_RELOC_X86_64_TLSDESC, /* ns32k relocations */ BFD_RELOC_NS32K_IMM_8, BFD_RELOC_NS32K_IMM_16, BFD_RELOC_NS32K_IMM_32, BFD_RELOC_NS32K_IMM_8_PCREL, BFD_RELOC_NS32K_IMM_16_PCREL, BFD_RELOC_NS32K_IMM_32_PCREL, BFD_RELOC_NS32K_DISP_8, BFD_RELOC_NS32K_DISP_16, BFD_RELOC_NS32K_DISP_32, BFD_RELOC_NS32K_DISP_8_PCREL, BFD_RELOC_NS32K_DISP_16_PCREL, BFD_RELOC_NS32K_DISP_32_PCREL, /* PDP11 relocations */ BFD_RELOC_PDP11_DISP_8_PCREL, BFD_RELOC_PDP11_DISP_6_PCREL, /* Picojava relocs. Not all of these appear in object files. */ BFD_RELOC_PJ_CODE_HI16, BFD_RELOC_PJ_CODE_LO16, BFD_RELOC_PJ_CODE_DIR16, BFD_RELOC_PJ_CODE_DIR32, BFD_RELOC_PJ_CODE_REL16, BFD_RELOC_PJ_CODE_REL32, /* Power(rs6000) and PowerPC relocations. */ BFD_RELOC_PPC_B26, BFD_RELOC_PPC_BA26, BFD_RELOC_PPC_TOC16, BFD_RELOC_PPC_B16, BFD_RELOC_PPC_B16_BRTAKEN, BFD_RELOC_PPC_B16_BRNTAKEN, BFD_RELOC_PPC_BA16, BFD_RELOC_PPC_BA16_BRTAKEN, BFD_RELOC_PPC_BA16_BRNTAKEN, BFD_RELOC_PPC_COPY, BFD_RELOC_PPC_GLOB_DAT, BFD_RELOC_PPC_JMP_SLOT, BFD_RELOC_PPC_RELATIVE, BFD_RELOC_PPC_LOCAL24PC, BFD_RELOC_PPC_EMB_NADDR32, BFD_RELOC_PPC_EMB_NADDR16, BFD_RELOC_PPC_EMB_NADDR16_LO, BFD_RELOC_PPC_EMB_NADDR16_HI, BFD_RELOC_PPC_EMB_NADDR16_HA, BFD_RELOC_PPC_EMB_SDAI16, BFD_RELOC_PPC_EMB_SDA2I16, BFD_RELOC_PPC_EMB_SDA2REL, BFD_RELOC_PPC_EMB_SDA21, BFD_RELOC_PPC_EMB_MRKREF, BFD_RELOC_PPC_EMB_RELSEC16, BFD_RELOC_PPC_EMB_RELST_LO, BFD_RELOC_PPC_EMB_RELST_HI, BFD_RELOC_PPC_EMB_RELST_HA, BFD_RELOC_PPC_EMB_BIT_FLD, BFD_RELOC_PPC_EMB_RELSDA, BFD_RELOC_PPC64_HIGHER, BFD_RELOC_PPC64_HIGHER_S, BFD_RELOC_PPC64_HIGHEST, BFD_RELOC_PPC64_HIGHEST_S, BFD_RELOC_PPC64_TOC16_LO, BFD_RELOC_PPC64_TOC16_HI, BFD_RELOC_PPC64_TOC16_HA, BFD_RELOC_PPC64_TOC, BFD_RELOC_PPC64_PLTGOT16, BFD_RELOC_PPC64_PLTGOT16_LO, BFD_RELOC_PPC64_PLTGOT16_HI, BFD_RELOC_PPC64_PLTGOT16_HA, BFD_RELOC_PPC64_ADDR16_DS, BFD_RELOC_PPC64_ADDR16_LO_DS, BFD_RELOC_PPC64_GOT16_DS, BFD_RELOC_PPC64_GOT16_LO_DS, BFD_RELOC_PPC64_PLT16_LO_DS, BFD_RELOC_PPC64_SECTOFF_DS, BFD_RELOC_PPC64_SECTOFF_LO_DS, BFD_RELOC_PPC64_TOC16_DS, BFD_RELOC_PPC64_TOC16_LO_DS, BFD_RELOC_PPC64_PLTGOT16_DS, BFD_RELOC_PPC64_PLTGOT16_LO_DS, /* PowerPC and PowerPC64 thread-local storage relocations. */ BFD_RELOC_PPC_TLS, + BFD_RELOC_PPC_TLSGD, + BFD_RELOC_PPC_TLSLD, BFD_RELOC_PPC_DTPMOD, BFD_RELOC_PPC_TPREL16, BFD_RELOC_PPC_TPREL16_LO, BFD_RELOC_PPC_TPREL16_HI, BFD_RELOC_PPC_TPREL16_HA, BFD_RELOC_PPC_TPREL, BFD_RELOC_PPC_DTPREL16, BFD_RELOC_PPC_DTPREL16_LO, BFD_RELOC_PPC_DTPREL16_HI, BFD_RELOC_PPC_DTPREL16_HA, BFD_RELOC_PPC_DTPREL, BFD_RELOC_PPC_GOT_TLSGD16, BFD_RELOC_PPC_GOT_TLSGD16_LO, BFD_RELOC_PPC_GOT_TLSGD16_HI, BFD_RELOC_PPC_GOT_TLSGD16_HA, BFD_RELOC_PPC_GOT_TLSLD16, BFD_RELOC_PPC_GOT_TLSLD16_LO, BFD_RELOC_PPC_GOT_TLSLD16_HI, BFD_RELOC_PPC_GOT_TLSLD16_HA, BFD_RELOC_PPC_GOT_TPREL16, BFD_RELOC_PPC_GOT_TPREL16_LO, BFD_RELOC_PPC_GOT_TPREL16_HI, BFD_RELOC_PPC_GOT_TPREL16_HA, BFD_RELOC_PPC_GOT_DTPREL16, BFD_RELOC_PPC_GOT_DTPREL16_LO, BFD_RELOC_PPC_GOT_DTPREL16_HI, BFD_RELOC_PPC_GOT_DTPREL16_HA, BFD_RELOC_PPC64_TPREL16_DS, BFD_RELOC_PPC64_TPREL16_LO_DS, BFD_RELOC_PPC64_TPREL16_HIGHER, BFD_RELOC_PPC64_TPREL16_HIGHERA, BFD_RELOC_PPC64_TPREL16_HIGHEST, BFD_RELOC_PPC64_TPREL16_HIGHESTA, BFD_RELOC_PPC64_DTPREL16_DS, BFD_RELOC_PPC64_DTPREL16_LO_DS, BFD_RELOC_PPC64_DTPREL16_HIGHER, BFD_RELOC_PPC64_DTPREL16_HIGHERA, BFD_RELOC_PPC64_DTPREL16_HIGHEST, BFD_RELOC_PPC64_DTPREL16_HIGHESTA, /* IBM 370/390 relocations */ BFD_RELOC_I370_D12, /* The type of reloc used to build a constructor table - at the moment probably a 32 bit wide absolute relocation, but the target can choose. It generally does map to one of the other relocation types. */ BFD_RELOC_CTOR, /* ARM 26 bit pc-relative branch. The lowest two bits must be zero and are not stored in the instruction. */ BFD_RELOC_ARM_PCREL_BRANCH, /* ARM 26 bit pc-relative branch. The lowest bit must be zero and is not stored in the instruction. The 2nd lowest bit comes from a 1 bit field in the instruction. */ BFD_RELOC_ARM_PCREL_BLX, /* Thumb 22 bit pc-relative branch. The lowest bit must be zero and is not stored in the instruction. The 2nd lowest bit comes from a 1 bit field in the instruction. */ BFD_RELOC_THUMB_PCREL_BLX, /* ARM 26-bit pc-relative branch for an unconditional BL or BLX instruction. */ BFD_RELOC_ARM_PCREL_CALL, /* ARM 26-bit pc-relative branch for B or conditional BL instruction. */ BFD_RELOC_ARM_PCREL_JUMP, /* Thumb 7-, 9-, 12-, 20-, 23-, and 25-bit pc-relative branches. The lowest bit must be zero and is not stored in the instruction. Note that the corresponding ELF R_ARM_THM_JUMPnn constant has an "nn" one smaller in all cases. Note further that BRANCH23 corresponds to R_ARM_THM_CALL. */ BFD_RELOC_THUMB_PCREL_BRANCH7, BFD_RELOC_THUMB_PCREL_BRANCH9, BFD_RELOC_THUMB_PCREL_BRANCH12, BFD_RELOC_THUMB_PCREL_BRANCH20, BFD_RELOC_THUMB_PCREL_BRANCH23, BFD_RELOC_THUMB_PCREL_BRANCH25, /* 12-bit immediate offset, used in ARM-format ldr and str instructions. */ BFD_RELOC_ARM_OFFSET_IMM, /* 5-bit immediate offset, used in Thumb-format ldr and str instructions. */ BFD_RELOC_ARM_THUMB_OFFSET, /* Pc-relative or absolute relocation depending on target. Used for entries in .init_array sections. */ BFD_RELOC_ARM_TARGET1, /* Read-only segment base relative address. */ BFD_RELOC_ARM_ROSEGREL32, /* Data segment base relative address. */ BFD_RELOC_ARM_SBREL32, /* This reloc is used for references to RTTI data from exception handling tables. The actual definition depends on the target. It may be a pc-relative or some form of GOT-indirect relocation. */ BFD_RELOC_ARM_TARGET2, /* 31-bit PC relative address. */ BFD_RELOC_ARM_PREL31, /* Low and High halfword relocations for MOVW and MOVT instructions. */ BFD_RELOC_ARM_MOVW, BFD_RELOC_ARM_MOVT, BFD_RELOC_ARM_MOVW_PCREL, BFD_RELOC_ARM_MOVT_PCREL, BFD_RELOC_ARM_THUMB_MOVW, BFD_RELOC_ARM_THUMB_MOVT, BFD_RELOC_ARM_THUMB_MOVW_PCREL, BFD_RELOC_ARM_THUMB_MOVT_PCREL, /* Relocations for setting up GOTs and PLTs for shared libraries. */ BFD_RELOC_ARM_JUMP_SLOT, BFD_RELOC_ARM_GLOB_DAT, BFD_RELOC_ARM_GOT32, BFD_RELOC_ARM_PLT32, BFD_RELOC_ARM_RELATIVE, BFD_RELOC_ARM_GOTOFF, BFD_RELOC_ARM_GOTPC, /* ARM thread-local storage relocations. */ BFD_RELOC_ARM_TLS_GD32, BFD_RELOC_ARM_TLS_LDO32, BFD_RELOC_ARM_TLS_LDM32, BFD_RELOC_ARM_TLS_DTPOFF32, BFD_RELOC_ARM_TLS_DTPMOD32, BFD_RELOC_ARM_TLS_TPOFF32, BFD_RELOC_ARM_TLS_IE32, BFD_RELOC_ARM_TLS_LE32, /* ARM group relocations. */ BFD_RELOC_ARM_ALU_PC_G0_NC, BFD_RELOC_ARM_ALU_PC_G0, BFD_RELOC_ARM_ALU_PC_G1_NC, BFD_RELOC_ARM_ALU_PC_G1, BFD_RELOC_ARM_ALU_PC_G2, BFD_RELOC_ARM_LDR_PC_G0, BFD_RELOC_ARM_LDR_PC_G1, BFD_RELOC_ARM_LDR_PC_G2, BFD_RELOC_ARM_LDRS_PC_G0, BFD_RELOC_ARM_LDRS_PC_G1, BFD_RELOC_ARM_LDRS_PC_G2, BFD_RELOC_ARM_LDC_PC_G0, BFD_RELOC_ARM_LDC_PC_G1, BFD_RELOC_ARM_LDC_PC_G2, BFD_RELOC_ARM_ALU_SB_G0_NC, BFD_RELOC_ARM_ALU_SB_G0, BFD_RELOC_ARM_ALU_SB_G1_NC, BFD_RELOC_ARM_ALU_SB_G1, BFD_RELOC_ARM_ALU_SB_G2, BFD_RELOC_ARM_LDR_SB_G0, BFD_RELOC_ARM_LDR_SB_G1, BFD_RELOC_ARM_LDR_SB_G2, BFD_RELOC_ARM_LDRS_SB_G0, BFD_RELOC_ARM_LDRS_SB_G1, BFD_RELOC_ARM_LDRS_SB_G2, BFD_RELOC_ARM_LDC_SB_G0, BFD_RELOC_ARM_LDC_SB_G1, BFD_RELOC_ARM_LDC_SB_G2, /* These relocs are only used within the ARM assembler. They are not (at present) written to any object files. */ BFD_RELOC_ARM_IMMEDIATE, BFD_RELOC_ARM_ADRL_IMMEDIATE, BFD_RELOC_ARM_T32_IMMEDIATE, BFD_RELOC_ARM_T32_ADD_IMM, BFD_RELOC_ARM_T32_IMM12, BFD_RELOC_ARM_T32_ADD_PC12, BFD_RELOC_ARM_SHIFT_IMM, BFD_RELOC_ARM_SMC, BFD_RELOC_ARM_SWI, BFD_RELOC_ARM_MULTI, BFD_RELOC_ARM_CP_OFF_IMM, BFD_RELOC_ARM_CP_OFF_IMM_S2, BFD_RELOC_ARM_T32_CP_OFF_IMM, BFD_RELOC_ARM_T32_CP_OFF_IMM_S2, BFD_RELOC_ARM_ADR_IMM, BFD_RELOC_ARM_LDR_IMM, BFD_RELOC_ARM_LITERAL, BFD_RELOC_ARM_IN_POOL, BFD_RELOC_ARM_OFFSET_IMM8, BFD_RELOC_ARM_T32_OFFSET_U8, BFD_RELOC_ARM_T32_OFFSET_IMM, BFD_RELOC_ARM_HWLITERAL, BFD_RELOC_ARM_THUMB_ADD, BFD_RELOC_ARM_THUMB_IMM, BFD_RELOC_ARM_THUMB_SHIFT, /* Renesas / SuperH SH relocs. Not all of these appear in object files. */ BFD_RELOC_SH_PCDISP8BY2, BFD_RELOC_SH_PCDISP12BY2, BFD_RELOC_SH_IMM3, BFD_RELOC_SH_IMM3U, BFD_RELOC_SH_DISP12, BFD_RELOC_SH_DISP12BY2, BFD_RELOC_SH_DISP12BY4, BFD_RELOC_SH_DISP12BY8, BFD_RELOC_SH_DISP20, BFD_RELOC_SH_DISP20BY8, BFD_RELOC_SH_IMM4, BFD_RELOC_SH_IMM4BY2, BFD_RELOC_SH_IMM4BY4, BFD_RELOC_SH_IMM8, BFD_RELOC_SH_IMM8BY2, BFD_RELOC_SH_IMM8BY4, BFD_RELOC_SH_PCRELIMM8BY2, BFD_RELOC_SH_PCRELIMM8BY4, BFD_RELOC_SH_SWITCH16, BFD_RELOC_SH_SWITCH32, BFD_RELOC_SH_USES, BFD_RELOC_SH_COUNT, BFD_RELOC_SH_ALIGN, BFD_RELOC_SH_CODE, BFD_RELOC_SH_DATA, BFD_RELOC_SH_LABEL, BFD_RELOC_SH_LOOP_START, BFD_RELOC_SH_LOOP_END, BFD_RELOC_SH_COPY, BFD_RELOC_SH_GLOB_DAT, BFD_RELOC_SH_JMP_SLOT, BFD_RELOC_SH_RELATIVE, BFD_RELOC_SH_GOTPC, BFD_RELOC_SH_GOT_LOW16, BFD_RELOC_SH_GOT_MEDLOW16, BFD_RELOC_SH_GOT_MEDHI16, BFD_RELOC_SH_GOT_HI16, BFD_RELOC_SH_GOTPLT_LOW16, BFD_RELOC_SH_GOTPLT_MEDLOW16, BFD_RELOC_SH_GOTPLT_MEDHI16, BFD_RELOC_SH_GOTPLT_HI16, BFD_RELOC_SH_PLT_LOW16, BFD_RELOC_SH_PLT_MEDLOW16, BFD_RELOC_SH_PLT_MEDHI16, BFD_RELOC_SH_PLT_HI16, BFD_RELOC_SH_GOTOFF_LOW16, BFD_RELOC_SH_GOTOFF_MEDLOW16, BFD_RELOC_SH_GOTOFF_MEDHI16, BFD_RELOC_SH_GOTOFF_HI16, BFD_RELOC_SH_GOTPC_LOW16, BFD_RELOC_SH_GOTPC_MEDLOW16, BFD_RELOC_SH_GOTPC_MEDHI16, BFD_RELOC_SH_GOTPC_HI16, BFD_RELOC_SH_COPY64, BFD_RELOC_SH_GLOB_DAT64, BFD_RELOC_SH_JMP_SLOT64, BFD_RELOC_SH_RELATIVE64, BFD_RELOC_SH_GOT10BY4, BFD_RELOC_SH_GOT10BY8, BFD_RELOC_SH_GOTPLT10BY4, BFD_RELOC_SH_GOTPLT10BY8, BFD_RELOC_SH_GOTPLT32, BFD_RELOC_SH_SHMEDIA_CODE, BFD_RELOC_SH_IMMU5, BFD_RELOC_SH_IMMS6, BFD_RELOC_SH_IMMS6BY32, BFD_RELOC_SH_IMMU6, BFD_RELOC_SH_IMMS10, BFD_RELOC_SH_IMMS10BY2, BFD_RELOC_SH_IMMS10BY4, BFD_RELOC_SH_IMMS10BY8, BFD_RELOC_SH_IMMS16, BFD_RELOC_SH_IMMU16, BFD_RELOC_SH_IMM_LOW16, BFD_RELOC_SH_IMM_LOW16_PCREL, BFD_RELOC_SH_IMM_MEDLOW16, BFD_RELOC_SH_IMM_MEDLOW16_PCREL, BFD_RELOC_SH_IMM_MEDHI16, BFD_RELOC_SH_IMM_MEDHI16_PCREL, BFD_RELOC_SH_IMM_HI16, BFD_RELOC_SH_IMM_HI16_PCREL, BFD_RELOC_SH_PT_16, BFD_RELOC_SH_TLS_GD_32, BFD_RELOC_SH_TLS_LD_32, BFD_RELOC_SH_TLS_LDO_32, BFD_RELOC_SH_TLS_IE_32, BFD_RELOC_SH_TLS_LE_32, BFD_RELOC_SH_TLS_DTPMOD32, BFD_RELOC_SH_TLS_DTPOFF32, BFD_RELOC_SH_TLS_TPOFF32, /* ARC Cores relocs. ARC 22 bit pc-relative branch. The lowest two bits must be zero and are not stored in the instruction. The high 20 bits are installed in bits 26 through 7 of the instruction. */ BFD_RELOC_ARC_B22_PCREL, /* ARC 26 bit absolute branch. The lowest two bits must be zero and are not stored in the instruction. The high 24 bits are installed in bits 23 through 0. */ BFD_RELOC_ARC_B26, /* ADI Blackfin 16 bit immediate absolute reloc. */ BFD_RELOC_BFIN_16_IMM, /* ADI Blackfin 16 bit immediate absolute reloc higher 16 bits. */ BFD_RELOC_BFIN_16_HIGH, /* ADI Blackfin 'a' part of LSETUP. */ BFD_RELOC_BFIN_4_PCREL, /* ADI Blackfin. */ BFD_RELOC_BFIN_5_PCREL, /* ADI Blackfin 16 bit immediate absolute reloc lower 16 bits. */ BFD_RELOC_BFIN_16_LOW, /* ADI Blackfin. */ BFD_RELOC_BFIN_10_PCREL, /* ADI Blackfin 'b' part of LSETUP. */ BFD_RELOC_BFIN_11_PCREL, /* ADI Blackfin. */ BFD_RELOC_BFIN_12_PCREL_JUMP, /* ADI Blackfin Short jump, pcrel. */ BFD_RELOC_BFIN_12_PCREL_JUMP_S, /* ADI Blackfin Call.x not implemented. */ BFD_RELOC_BFIN_24_PCREL_CALL_X, /* ADI Blackfin Long Jump pcrel. */ BFD_RELOC_BFIN_24_PCREL_JUMP_L, /* ADI Blackfin FD-PIC relocations. */ BFD_RELOC_BFIN_GOT17M4, BFD_RELOC_BFIN_GOTHI, BFD_RELOC_BFIN_GOTLO, BFD_RELOC_BFIN_FUNCDESC, BFD_RELOC_BFIN_FUNCDESC_GOT17M4, BFD_RELOC_BFIN_FUNCDESC_GOTHI, BFD_RELOC_BFIN_FUNCDESC_GOTLO, BFD_RELOC_BFIN_FUNCDESC_VALUE, BFD_RELOC_BFIN_FUNCDESC_GOTOFF17M4, BFD_RELOC_BFIN_FUNCDESC_GOTOFFHI, BFD_RELOC_BFIN_FUNCDESC_GOTOFFLO, BFD_RELOC_BFIN_GOTOFF17M4, BFD_RELOC_BFIN_GOTOFFHI, BFD_RELOC_BFIN_GOTOFFLO, /* ADI Blackfin GOT relocation. */ BFD_RELOC_BFIN_GOT, /* ADI Blackfin PLTPC relocation. */ BFD_RELOC_BFIN_PLTPC, /* ADI Blackfin arithmetic relocation. */ BFD_ARELOC_BFIN_PUSH, /* ADI Blackfin arithmetic relocation. */ BFD_ARELOC_BFIN_CONST, /* ADI Blackfin arithmetic relocation. */ BFD_ARELOC_BFIN_ADD, /* ADI Blackfin arithmetic relocation. */ BFD_ARELOC_BFIN_SUB, /* ADI Blackfin arithmetic relocation. */ BFD_ARELOC_BFIN_MULT, /* ADI Blackfin arithmetic relocation. */ BFD_ARELOC_BFIN_DIV, /* ADI Blackfin arithmetic relocation. */ BFD_ARELOC_BFIN_MOD, /* ADI Blackfin arithmetic relocation. */ BFD_ARELOC_BFIN_LSHIFT, /* ADI Blackfin arithmetic relocation. */ BFD_ARELOC_BFIN_RSHIFT, /* ADI Blackfin arithmetic relocation. */ BFD_ARELOC_BFIN_AND, /* ADI Blackfin arithmetic relocation. */ BFD_ARELOC_BFIN_OR, /* ADI Blackfin arithmetic relocation. */ BFD_ARELOC_BFIN_XOR, /* ADI Blackfin arithmetic relocation. */ BFD_ARELOC_BFIN_LAND, /* ADI Blackfin arithmetic relocation. */ BFD_ARELOC_BFIN_LOR, /* ADI Blackfin arithmetic relocation. */ BFD_ARELOC_BFIN_LEN, /* ADI Blackfin arithmetic relocation. */ BFD_ARELOC_BFIN_NEG, /* ADI Blackfin arithmetic relocation. */ BFD_ARELOC_BFIN_COMP, /* ADI Blackfin arithmetic relocation. */ BFD_ARELOC_BFIN_PAGE, /* ADI Blackfin arithmetic relocation. */ BFD_ARELOC_BFIN_HWPAGE, /* ADI Blackfin arithmetic relocation. */ BFD_ARELOC_BFIN_ADDR, /* Mitsubishi D10V relocs. This is a 10-bit reloc with the right 2 bits assumed to be 0. */ BFD_RELOC_D10V_10_PCREL_R, /* Mitsubishi D10V relocs. This is a 10-bit reloc with the right 2 bits assumed to be 0. This is the same as the previous reloc except it is in the left container, i.e., shifted left 15 bits. */ BFD_RELOC_D10V_10_PCREL_L, /* This is an 18-bit reloc with the right 2 bits assumed to be 0. */ BFD_RELOC_D10V_18, /* This is an 18-bit reloc with the right 2 bits assumed to be 0. */ BFD_RELOC_D10V_18_PCREL, /* Mitsubishi D30V relocs. This is a 6-bit absolute reloc. */ BFD_RELOC_D30V_6, /* This is a 6-bit pc-relative reloc with the right 3 bits assumed to be 0. */ BFD_RELOC_D30V_9_PCREL, /* This is a 6-bit pc-relative reloc with the right 3 bits assumed to be 0. Same as the previous reloc but on the right side of the container. */ BFD_RELOC_D30V_9_PCREL_R, /* This is a 12-bit absolute reloc with the right 3 bitsassumed to be 0. */ BFD_RELOC_D30V_15, /* This is a 12-bit pc-relative reloc with the right 3 bits assumed to be 0. */ BFD_RELOC_D30V_15_PCREL, /* This is a 12-bit pc-relative reloc with the right 3 bits assumed to be 0. Same as the previous reloc but on the right side of the container. */ BFD_RELOC_D30V_15_PCREL_R, /* This is an 18-bit absolute reloc with the right 3 bits assumed to be 0. */ BFD_RELOC_D30V_21, /* This is an 18-bit pc-relative reloc with the right 3 bits assumed to be 0. */ BFD_RELOC_D30V_21_PCREL, /* This is an 18-bit pc-relative reloc with the right 3 bits assumed to be 0. Same as the previous reloc but on the right side of the container. */ BFD_RELOC_D30V_21_PCREL_R, /* This is a 32-bit absolute reloc. */ BFD_RELOC_D30V_32, /* This is a 32-bit pc-relative reloc. */ BFD_RELOC_D30V_32_PCREL, /* DLX relocs */ BFD_RELOC_DLX_HI16_S, /* DLX relocs */ BFD_RELOC_DLX_LO16, /* DLX relocs */ BFD_RELOC_DLX_JMP26, /* Renesas M16C/M32C Relocations. */ BFD_RELOC_M32C_HI8, BFD_RELOC_M32C_RL_JUMP, BFD_RELOC_M32C_RL_1ADDR, BFD_RELOC_M32C_RL_2ADDR, /* Renesas M32R (formerly Mitsubishi M32R) relocs. This is a 24 bit absolute address. */ BFD_RELOC_M32R_24, /* This is a 10-bit pc-relative reloc with the right 2 bits assumed to be 0. */ BFD_RELOC_M32R_10_PCREL, /* This is an 18-bit reloc with the right 2 bits assumed to be 0. */ BFD_RELOC_M32R_18_PCREL, /* This is a 26-bit reloc with the right 2 bits assumed to be 0. */ BFD_RELOC_M32R_26_PCREL, /* This is a 16-bit reloc containing the high 16 bits of an address used when the lower 16 bits are treated as unsigned. */ BFD_RELOC_M32R_HI16_ULO, /* This is a 16-bit reloc containing the high 16 bits of an address used when the lower 16 bits are treated as signed. */ BFD_RELOC_M32R_HI16_SLO, /* This is a 16-bit reloc containing the lower 16 bits of an address. */ BFD_RELOC_M32R_LO16, /* This is a 16-bit reloc containing the small data area offset for use in add3, load, and store instructions. */ BFD_RELOC_M32R_SDA16, /* For PIC. */ BFD_RELOC_M32R_GOT24, BFD_RELOC_M32R_26_PLTREL, BFD_RELOC_M32R_COPY, BFD_RELOC_M32R_GLOB_DAT, BFD_RELOC_M32R_JMP_SLOT, BFD_RELOC_M32R_RELATIVE, BFD_RELOC_M32R_GOTOFF, BFD_RELOC_M32R_GOTOFF_HI_ULO, BFD_RELOC_M32R_GOTOFF_HI_SLO, BFD_RELOC_M32R_GOTOFF_LO, BFD_RELOC_M32R_GOTPC24, BFD_RELOC_M32R_GOT16_HI_ULO, BFD_RELOC_M32R_GOT16_HI_SLO, BFD_RELOC_M32R_GOT16_LO, BFD_RELOC_M32R_GOTPC_HI_ULO, BFD_RELOC_M32R_GOTPC_HI_SLO, BFD_RELOC_M32R_GOTPC_LO, /* This is a 9-bit reloc */ BFD_RELOC_V850_9_PCREL, /* This is a 22-bit reloc */ BFD_RELOC_V850_22_PCREL, /* This is a 16 bit offset from the short data area pointer. */ BFD_RELOC_V850_SDA_16_16_OFFSET, /* This is a 16 bit offset (of which only 15 bits are used) from the short data area pointer. */ BFD_RELOC_V850_SDA_15_16_OFFSET, /* This is a 16 bit offset from the zero data area pointer. */ BFD_RELOC_V850_ZDA_16_16_OFFSET, /* This is a 16 bit offset (of which only 15 bits are used) from the zero data area pointer. */ BFD_RELOC_V850_ZDA_15_16_OFFSET, /* This is an 8 bit offset (of which only 6 bits are used) from the tiny data area pointer. */ BFD_RELOC_V850_TDA_6_8_OFFSET, /* This is an 8bit offset (of which only 7 bits are used) from the tiny data area pointer. */ BFD_RELOC_V850_TDA_7_8_OFFSET, /* This is a 7 bit offset from the tiny data area pointer. */ BFD_RELOC_V850_TDA_7_7_OFFSET, /* This is a 16 bit offset from the tiny data area pointer. */ BFD_RELOC_V850_TDA_16_16_OFFSET, /* This is a 5 bit offset (of which only 4 bits are used) from the tiny data area pointer. */ BFD_RELOC_V850_TDA_4_5_OFFSET, /* This is a 4 bit offset from the tiny data area pointer. */ BFD_RELOC_V850_TDA_4_4_OFFSET, /* This is a 16 bit offset from the short data area pointer, with the bits placed non-contiguously in the instruction. */ BFD_RELOC_V850_SDA_16_16_SPLIT_OFFSET, /* This is a 16 bit offset from the zero data area pointer, with the bits placed non-contiguously in the instruction. */ BFD_RELOC_V850_ZDA_16_16_SPLIT_OFFSET, /* This is a 6 bit offset from the call table base pointer. */ BFD_RELOC_V850_CALLT_6_7_OFFSET, /* This is a 16 bit offset from the call table base pointer. */ BFD_RELOC_V850_CALLT_16_16_OFFSET, /* Used for relaxing indirect function calls. */ BFD_RELOC_V850_LONGCALL, /* Used for relaxing indirect jumps. */ BFD_RELOC_V850_LONGJUMP, /* Used to maintain alignment whilst relaxing. */ BFD_RELOC_V850_ALIGN, /* This is a variation of BFD_RELOC_LO16 that can be used in v850e ld.bu instructions. */ BFD_RELOC_V850_LO16_SPLIT_OFFSET, /* This is a 32bit pcrel reloc for the mn10300, offset by two bytes in the instruction. */ BFD_RELOC_MN10300_32_PCREL, /* This is a 16bit pcrel reloc for the mn10300, offset by two bytes in the instruction. */ BFD_RELOC_MN10300_16_PCREL, /* This is a 8bit DP reloc for the tms320c30, where the most significant 8 bits of a 24 bit word are placed into the least significant 8 bits of the opcode. */ BFD_RELOC_TIC30_LDP, /* This is a 7bit reloc for the tms320c54x, where the least significant 7 bits of a 16 bit word are placed into the least significant 7 bits of the opcode. */ BFD_RELOC_TIC54X_PARTLS7, /* This is a 9bit DP reloc for the tms320c54x, where the most significant 9 bits of a 16 bit word are placed into the least significant 9 bits of the opcode. */ BFD_RELOC_TIC54X_PARTMS9, /* This is an extended address 23-bit reloc for the tms320c54x. */ BFD_RELOC_TIC54X_23, /* This is a 16-bit reloc for the tms320c54x, where the least significant 16 bits of a 23-bit extended address are placed into the opcode. */ BFD_RELOC_TIC54X_16_OF_23, /* This is a reloc for the tms320c54x, where the most significant 7 bits of a 23-bit extended address are placed into the opcode. */ BFD_RELOC_TIC54X_MS7_OF_23, /* This is a 48 bit reloc for the FR30 that stores 32 bits. */ BFD_RELOC_FR30_48, /* This is a 32 bit reloc for the FR30 that stores 20 bits split up into two sections. */ BFD_RELOC_FR30_20, /* This is a 16 bit reloc for the FR30 that stores a 6 bit word offset in 4 bits. */ BFD_RELOC_FR30_6_IN_4, /* This is a 16 bit reloc for the FR30 that stores an 8 bit byte offset into 8 bits. */ BFD_RELOC_FR30_8_IN_8, /* This is a 16 bit reloc for the FR30 that stores a 9 bit short offset into 8 bits. */ BFD_RELOC_FR30_9_IN_8, /* This is a 16 bit reloc for the FR30 that stores a 10 bit word offset into 8 bits. */ BFD_RELOC_FR30_10_IN_8, /* This is a 16 bit reloc for the FR30 that stores a 9 bit pc relative short offset into 8 bits. */ BFD_RELOC_FR30_9_PCREL, /* This is a 16 bit reloc for the FR30 that stores a 12 bit pc relative short offset into 11 bits. */ BFD_RELOC_FR30_12_PCREL, /* Motorola Mcore relocations. */ BFD_RELOC_MCORE_PCREL_IMM8BY4, BFD_RELOC_MCORE_PCREL_IMM11BY2, BFD_RELOC_MCORE_PCREL_IMM4BY2, BFD_RELOC_MCORE_PCREL_32, BFD_RELOC_MCORE_PCREL_JSR_IMM11BY2, BFD_RELOC_MCORE_RVA, /* Toshiba Media Processor Relocations. */ BFD_RELOC_MEP_8, BFD_RELOC_MEP_16, BFD_RELOC_MEP_32, BFD_RELOC_MEP_PCREL8A2, BFD_RELOC_MEP_PCREL12A2, BFD_RELOC_MEP_PCREL17A2, BFD_RELOC_MEP_PCREL24A2, BFD_RELOC_MEP_PCABS24A2, BFD_RELOC_MEP_LOW16, BFD_RELOC_MEP_HI16U, BFD_RELOC_MEP_HI16S, BFD_RELOC_MEP_GPREL, BFD_RELOC_MEP_TPREL, BFD_RELOC_MEP_TPREL7, BFD_RELOC_MEP_TPREL7A2, BFD_RELOC_MEP_TPREL7A4, BFD_RELOC_MEP_UIMM24, BFD_RELOC_MEP_ADDR24A4, BFD_RELOC_MEP_GNU_VTINHERIT, BFD_RELOC_MEP_GNU_VTENTRY, /* These are relocations for the GETA instruction. */ BFD_RELOC_MMIX_GETA, BFD_RELOC_MMIX_GETA_1, BFD_RELOC_MMIX_GETA_2, BFD_RELOC_MMIX_GETA_3, /* These are relocations for a conditional branch instruction. */ BFD_RELOC_MMIX_CBRANCH, BFD_RELOC_MMIX_CBRANCH_J, BFD_RELOC_MMIX_CBRANCH_1, BFD_RELOC_MMIX_CBRANCH_2, BFD_RELOC_MMIX_CBRANCH_3, /* These are relocations for the PUSHJ instruction. */ BFD_RELOC_MMIX_PUSHJ, BFD_RELOC_MMIX_PUSHJ_1, BFD_RELOC_MMIX_PUSHJ_2, BFD_RELOC_MMIX_PUSHJ_3, BFD_RELOC_MMIX_PUSHJ_STUBBABLE, /* These are relocations for the JMP instruction. */ BFD_RELOC_MMIX_JMP, BFD_RELOC_MMIX_JMP_1, BFD_RELOC_MMIX_JMP_2, BFD_RELOC_MMIX_JMP_3, /* This is a relocation for a relative address as in a GETA instruction or a branch. */ BFD_RELOC_MMIX_ADDR19, /* This is a relocation for a relative address as in a JMP instruction. */ BFD_RELOC_MMIX_ADDR27, /* This is a relocation for an instruction field that may be a general register or a value 0..255. */ BFD_RELOC_MMIX_REG_OR_BYTE, /* This is a relocation for an instruction field that may be a general register. */ BFD_RELOC_MMIX_REG, /* This is a relocation for two instruction fields holding a register and an offset, the equivalent of the relocation. */ BFD_RELOC_MMIX_BASE_PLUS_OFFSET, /* This relocation is an assertion that the expression is not allocated as a global register. It does not modify contents. */ BFD_RELOC_MMIX_LOCAL, /* This is a 16 bit reloc for the AVR that stores 8 bit pc relative short offset into 7 bits. */ BFD_RELOC_AVR_7_PCREL, /* This is a 16 bit reloc for the AVR that stores 13 bit pc relative short offset into 12 bits. */ BFD_RELOC_AVR_13_PCREL, /* This is a 16 bit reloc for the AVR that stores 17 bit value (usually program memory address) into 16 bits. */ BFD_RELOC_AVR_16_PM, /* This is a 16 bit reloc for the AVR that stores 8 bit value (usually data memory address) into 8 bit immediate value of LDI insn. */ BFD_RELOC_AVR_LO8_LDI, /* This is a 16 bit reloc for the AVR that stores 8 bit value (high 8 bit of data memory address) into 8 bit immediate value of LDI insn. */ BFD_RELOC_AVR_HI8_LDI, /* This is a 16 bit reloc for the AVR that stores 8 bit value (most high 8 bit of program memory address) into 8 bit immediate value of LDI insn. */ BFD_RELOC_AVR_HH8_LDI, /* This is a 16 bit reloc for the AVR that stores 8 bit value (most high 8 bit of 32 bit value) into 8 bit immediate value of LDI insn. */ BFD_RELOC_AVR_MS8_LDI, /* This is a 16 bit reloc for the AVR that stores negated 8 bit value (usually data memory address) into 8 bit immediate value of SUBI insn. */ BFD_RELOC_AVR_LO8_LDI_NEG, /* This is a 16 bit reloc for the AVR that stores negated 8 bit value (high 8 bit of data memory address) into 8 bit immediate value of SUBI insn. */ BFD_RELOC_AVR_HI8_LDI_NEG, /* This is a 16 bit reloc for the AVR that stores negated 8 bit value (most high 8 bit of program memory address) into 8 bit immediate value of LDI or SUBI insn. */ BFD_RELOC_AVR_HH8_LDI_NEG, /* This is a 16 bit reloc for the AVR that stores negated 8 bit value (msb of 32 bit value) into 8 bit immediate value of LDI insn. */ BFD_RELOC_AVR_MS8_LDI_NEG, /* This is a 16 bit reloc for the AVR that stores 8 bit value (usually command address) into 8 bit immediate value of LDI insn. */ BFD_RELOC_AVR_LO8_LDI_PM, /* This is a 16 bit reloc for the AVR that stores 8 bit value (command address) into 8 bit immediate value of LDI insn. If the address is beyond the 128k boundary, the linker inserts a jump stub for this reloc in the lower 128k. */ BFD_RELOC_AVR_LO8_LDI_GS, /* This is a 16 bit reloc for the AVR that stores 8 bit value (high 8 bit of command address) into 8 bit immediate value of LDI insn. */ BFD_RELOC_AVR_HI8_LDI_PM, /* This is a 16 bit reloc for the AVR that stores 8 bit value (high 8 bit of command address) into 8 bit immediate value of LDI insn. If the address is beyond the 128k boundary, the linker inserts a jump stub for this reloc below 128k. */ BFD_RELOC_AVR_HI8_LDI_GS, /* This is a 16 bit reloc for the AVR that stores 8 bit value (most high 8 bit of command address) into 8 bit immediate value of LDI insn. */ BFD_RELOC_AVR_HH8_LDI_PM, /* This is a 16 bit reloc for the AVR that stores negated 8 bit value (usually command address) into 8 bit immediate value of SUBI insn. */ BFD_RELOC_AVR_LO8_LDI_PM_NEG, /* This is a 16 bit reloc for the AVR that stores negated 8 bit value (high 8 bit of 16 bit command address) into 8 bit immediate value of SUBI insn. */ BFD_RELOC_AVR_HI8_LDI_PM_NEG, /* This is a 16 bit reloc for the AVR that stores negated 8 bit value (high 6 bit of 22 bit command address) into 8 bit immediate value of SUBI insn. */ BFD_RELOC_AVR_HH8_LDI_PM_NEG, /* This is a 32 bit reloc for the AVR that stores 23 bit value into 22 bits. */ BFD_RELOC_AVR_CALL, /* This is a 16 bit reloc for the AVR that stores all needed bits for absolute addressing with ldi with overflow check to linktime */ BFD_RELOC_AVR_LDI, /* This is a 6 bit reloc for the AVR that stores offset for ldd/std instructions */ BFD_RELOC_AVR_6, /* This is a 6 bit reloc for the AVR that stores offset for adiw/sbiw instructions */ BFD_RELOC_AVR_6_ADIW, /* Direct 12 bit. */ BFD_RELOC_390_12, /* 12 bit GOT offset. */ BFD_RELOC_390_GOT12, /* 32 bit PC relative PLT address. */ BFD_RELOC_390_PLT32, /* Copy symbol at runtime. */ BFD_RELOC_390_COPY, /* Create GOT entry. */ BFD_RELOC_390_GLOB_DAT, /* Create PLT entry. */ BFD_RELOC_390_JMP_SLOT, /* Adjust by program base. */ BFD_RELOC_390_RELATIVE, /* 32 bit PC relative offset to GOT. */ BFD_RELOC_390_GOTPC, /* 16 bit GOT offset. */ BFD_RELOC_390_GOT16, /* PC relative 16 bit shifted by 1. */ BFD_RELOC_390_PC16DBL, /* 16 bit PC rel. PLT shifted by 1. */ BFD_RELOC_390_PLT16DBL, /* PC relative 32 bit shifted by 1. */ BFD_RELOC_390_PC32DBL, /* 32 bit PC rel. PLT shifted by 1. */ BFD_RELOC_390_PLT32DBL, /* 32 bit PC rel. GOT shifted by 1. */ BFD_RELOC_390_GOTPCDBL, /* 64 bit GOT offset. */ BFD_RELOC_390_GOT64, /* 64 bit PC relative PLT address. */ BFD_RELOC_390_PLT64, /* 32 bit rel. offset to GOT entry. */ BFD_RELOC_390_GOTENT, /* 64 bit offset to GOT. */ BFD_RELOC_390_GOTOFF64, /* 12-bit offset to symbol-entry within GOT, with PLT handling. */ BFD_RELOC_390_GOTPLT12, /* 16-bit offset to symbol-entry within GOT, with PLT handling. */ BFD_RELOC_390_GOTPLT16, /* 32-bit offset to symbol-entry within GOT, with PLT handling. */ BFD_RELOC_390_GOTPLT32, /* 64-bit offset to symbol-entry within GOT, with PLT handling. */ BFD_RELOC_390_GOTPLT64, /* 32-bit rel. offset to symbol-entry within GOT, with PLT handling. */ BFD_RELOC_390_GOTPLTENT, /* 16-bit rel. offset from the GOT to a PLT entry. */ BFD_RELOC_390_PLTOFF16, /* 32-bit rel. offset from the GOT to a PLT entry. */ BFD_RELOC_390_PLTOFF32, /* 64-bit rel. offset from the GOT to a PLT entry. */ BFD_RELOC_390_PLTOFF64, /* s390 tls relocations. */ BFD_RELOC_390_TLS_LOAD, BFD_RELOC_390_TLS_GDCALL, BFD_RELOC_390_TLS_LDCALL, BFD_RELOC_390_TLS_GD32, BFD_RELOC_390_TLS_GD64, BFD_RELOC_390_TLS_GOTIE12, BFD_RELOC_390_TLS_GOTIE32, BFD_RELOC_390_TLS_GOTIE64, BFD_RELOC_390_TLS_LDM32, BFD_RELOC_390_TLS_LDM64, BFD_RELOC_390_TLS_IE32, BFD_RELOC_390_TLS_IE64, BFD_RELOC_390_TLS_IEENT, BFD_RELOC_390_TLS_LE32, BFD_RELOC_390_TLS_LE64, BFD_RELOC_390_TLS_LDO32, BFD_RELOC_390_TLS_LDO64, BFD_RELOC_390_TLS_DTPMOD, BFD_RELOC_390_TLS_DTPOFF, BFD_RELOC_390_TLS_TPOFF, /* Long displacement extension. */ BFD_RELOC_390_20, BFD_RELOC_390_GOT20, BFD_RELOC_390_GOTPLT20, BFD_RELOC_390_TLS_GOTIE20, /* Score relocations */ BFD_RELOC_SCORE_DUMMY1, /* Low 16 bit for load/store */ BFD_RELOC_SCORE_GPREL15, /* This is a 24-bit reloc with the right 1 bit assumed to be 0 */ BFD_RELOC_SCORE_DUMMY2, BFD_RELOC_SCORE_JMP, /* This is a 19-bit reloc with the right 1 bit assumed to be 0 */ BFD_RELOC_SCORE_BRANCH, /* This is a 11-bit reloc with the right 1 bit assumed to be 0 */ BFD_RELOC_SCORE16_JMP, /* This is a 8-bit reloc with the right 1 bit assumed to be 0 */ BFD_RELOC_SCORE16_BRANCH, /* Undocumented Score relocs */ BFD_RELOC_SCORE_GOT15, BFD_RELOC_SCORE_GOT_LO16, BFD_RELOC_SCORE_CALL15, BFD_RELOC_SCORE_DUMMY_HI16, /* Scenix IP2K - 9-bit register number / data address */ BFD_RELOC_IP2K_FR9, /* Scenix IP2K - 4-bit register/data bank number */ BFD_RELOC_IP2K_BANK, /* Scenix IP2K - low 13 bits of instruction word address */ BFD_RELOC_IP2K_ADDR16CJP, /* Scenix IP2K - high 3 bits of instruction word address */ BFD_RELOC_IP2K_PAGE3, /* Scenix IP2K - ext/low/high 8 bits of data address */ BFD_RELOC_IP2K_LO8DATA, BFD_RELOC_IP2K_HI8DATA, BFD_RELOC_IP2K_EX8DATA, /* Scenix IP2K - low/high 8 bits of instruction word address */ BFD_RELOC_IP2K_LO8INSN, BFD_RELOC_IP2K_HI8INSN, /* Scenix IP2K - even/odd PC modifier to modify snb pcl.0 */ BFD_RELOC_IP2K_PC_SKIP, /* Scenix IP2K - 16 bit word address in text section. */ BFD_RELOC_IP2K_TEXT, /* Scenix IP2K - 7-bit sp or dp offset */ BFD_RELOC_IP2K_FR_OFFSET, /* Scenix VPE4K coprocessor - data/insn-space addressing */ BFD_RELOC_VPE4KMATH_DATA, BFD_RELOC_VPE4KMATH_INSN, /* These two relocations are used by the linker to determine which of the entries in a C++ virtual function table are actually used. When the --gc-sections option is given, the linker will zero out the entries that are not used, so that the code for those functions need not be included in the output. VTABLE_INHERIT is a zero-space relocation used to describe to the linker the inheritance tree of a C++ virtual function table. The relocation's symbol should be the parent class' vtable, and the relocation should be located at the child vtable. VTABLE_ENTRY is a zero-space relocation that describes the use of a virtual function table entry. The reloc's symbol should refer to the table of the class mentioned in the code. Off of that base, an offset describes the entry that is being used. For Rela hosts, this offset is stored in the reloc's addend. For Rel hosts, we are forced to put this offset in the reloc's section offset. */ BFD_RELOC_VTABLE_INHERIT, BFD_RELOC_VTABLE_ENTRY, /* Intel IA64 Relocations. */ BFD_RELOC_IA64_IMM14, BFD_RELOC_IA64_IMM22, BFD_RELOC_IA64_IMM64, BFD_RELOC_IA64_DIR32MSB, BFD_RELOC_IA64_DIR32LSB, BFD_RELOC_IA64_DIR64MSB, BFD_RELOC_IA64_DIR64LSB, BFD_RELOC_IA64_GPREL22, BFD_RELOC_IA64_GPREL64I, BFD_RELOC_IA64_GPREL32MSB, BFD_RELOC_IA64_GPREL32LSB, BFD_RELOC_IA64_GPREL64MSB, BFD_RELOC_IA64_GPREL64LSB, BFD_RELOC_IA64_LTOFF22, BFD_RELOC_IA64_LTOFF64I, BFD_RELOC_IA64_PLTOFF22, BFD_RELOC_IA64_PLTOFF64I, BFD_RELOC_IA64_PLTOFF64MSB, BFD_RELOC_IA64_PLTOFF64LSB, BFD_RELOC_IA64_FPTR64I, BFD_RELOC_IA64_FPTR32MSB, BFD_RELOC_IA64_FPTR32LSB, BFD_RELOC_IA64_FPTR64MSB, BFD_RELOC_IA64_FPTR64LSB, BFD_RELOC_IA64_PCREL21B, BFD_RELOC_IA64_PCREL21BI, BFD_RELOC_IA64_PCREL21M, BFD_RELOC_IA64_PCREL21F, BFD_RELOC_IA64_PCREL22, BFD_RELOC_IA64_PCREL60B, BFD_RELOC_IA64_PCREL64I, BFD_RELOC_IA64_PCREL32MSB, BFD_RELOC_IA64_PCREL32LSB, BFD_RELOC_IA64_PCREL64MSB, BFD_RELOC_IA64_PCREL64LSB, BFD_RELOC_IA64_LTOFF_FPTR22, BFD_RELOC_IA64_LTOFF_FPTR64I, BFD_RELOC_IA64_LTOFF_FPTR32MSB, BFD_RELOC_IA64_LTOFF_FPTR32LSB, BFD_RELOC_IA64_LTOFF_FPTR64MSB, BFD_RELOC_IA64_LTOFF_FPTR64LSB, BFD_RELOC_IA64_SEGREL32MSB, BFD_RELOC_IA64_SEGREL32LSB, BFD_RELOC_IA64_SEGREL64MSB, BFD_RELOC_IA64_SEGREL64LSB, BFD_RELOC_IA64_SECREL32MSB, BFD_RELOC_IA64_SECREL32LSB, BFD_RELOC_IA64_SECREL64MSB, BFD_RELOC_IA64_SECREL64LSB, BFD_RELOC_IA64_REL32MSB, BFD_RELOC_IA64_REL32LSB, BFD_RELOC_IA64_REL64MSB, BFD_RELOC_IA64_REL64LSB, BFD_RELOC_IA64_LTV32MSB, BFD_RELOC_IA64_LTV32LSB, BFD_RELOC_IA64_LTV64MSB, BFD_RELOC_IA64_LTV64LSB, BFD_RELOC_IA64_IPLTMSB, BFD_RELOC_IA64_IPLTLSB, BFD_RELOC_IA64_COPY, BFD_RELOC_IA64_LTOFF22X, BFD_RELOC_IA64_LDXMOV, BFD_RELOC_IA64_TPREL14, BFD_RELOC_IA64_TPREL22, BFD_RELOC_IA64_TPREL64I, BFD_RELOC_IA64_TPREL64MSB, BFD_RELOC_IA64_TPREL64LSB, BFD_RELOC_IA64_LTOFF_TPREL22, BFD_RELOC_IA64_DTPMOD64MSB, BFD_RELOC_IA64_DTPMOD64LSB, BFD_RELOC_IA64_LTOFF_DTPMOD22, BFD_RELOC_IA64_DTPREL14, BFD_RELOC_IA64_DTPREL22, BFD_RELOC_IA64_DTPREL64I, BFD_RELOC_IA64_DTPREL32MSB, BFD_RELOC_IA64_DTPREL32LSB, BFD_RELOC_IA64_DTPREL64MSB, BFD_RELOC_IA64_DTPREL64LSB, BFD_RELOC_IA64_LTOFF_DTPREL22, /* Motorola 68HC11 reloc. This is the 8 bit high part of an absolute address. */ BFD_RELOC_M68HC11_HI8, /* Motorola 68HC11 reloc. This is the 8 bit low part of an absolute address. */ BFD_RELOC_M68HC11_LO8, /* Motorola 68HC11 reloc. This is the 3 bit of a value. */ BFD_RELOC_M68HC11_3B, /* Motorola 68HC11 reloc. This reloc marks the beginning of a jump/call instruction. It is used for linker relaxation to correctly identify beginning of instruction and change some branches to use PC-relative addressing mode. */ BFD_RELOC_M68HC11_RL_JUMP, /* Motorola 68HC11 reloc. This reloc marks a group of several instructions that gcc generates and for which the linker relaxation pass can modify and/or remove some of them. */ BFD_RELOC_M68HC11_RL_GROUP, /* Motorola 68HC11 reloc. This is the 16-bit lower part of an address. It is used for 'call' instruction to specify the symbol address without any special transformation (due to memory bank window). */ BFD_RELOC_M68HC11_LO16, /* Motorola 68HC11 reloc. This is a 8-bit reloc that specifies the page number of an address. It is used by 'call' instruction to specify the page number of the symbol. */ BFD_RELOC_M68HC11_PAGE, /* Motorola 68HC11 reloc. This is a 24-bit reloc that represents the address with a 16-bit value and a 8-bit page number. The symbol address is transformed to follow the 16K memory bank of 68HC12 (seen as mapped in the window). */ BFD_RELOC_M68HC11_24, /* Motorola 68HC12 reloc. This is the 5 bits of a value. */ BFD_RELOC_M68HC12_5B, /* NS CR16C Relocations. */ BFD_RELOC_16C_NUM08, BFD_RELOC_16C_NUM08_C, BFD_RELOC_16C_NUM16, BFD_RELOC_16C_NUM16_C, BFD_RELOC_16C_NUM32, BFD_RELOC_16C_NUM32_C, BFD_RELOC_16C_DISP04, BFD_RELOC_16C_DISP04_C, BFD_RELOC_16C_DISP08, BFD_RELOC_16C_DISP08_C, BFD_RELOC_16C_DISP16, BFD_RELOC_16C_DISP16_C, BFD_RELOC_16C_DISP24, BFD_RELOC_16C_DISP24_C, BFD_RELOC_16C_DISP24a, BFD_RELOC_16C_DISP24a_C, BFD_RELOC_16C_REG04, BFD_RELOC_16C_REG04_C, BFD_RELOC_16C_REG04a, BFD_RELOC_16C_REG04a_C, BFD_RELOC_16C_REG14, BFD_RELOC_16C_REG14_C, BFD_RELOC_16C_REG16, BFD_RELOC_16C_REG16_C, BFD_RELOC_16C_REG20, BFD_RELOC_16C_REG20_C, BFD_RELOC_16C_ABS20, BFD_RELOC_16C_ABS20_C, BFD_RELOC_16C_ABS24, BFD_RELOC_16C_ABS24_C, BFD_RELOC_16C_IMM04, BFD_RELOC_16C_IMM04_C, BFD_RELOC_16C_IMM16, BFD_RELOC_16C_IMM16_C, BFD_RELOC_16C_IMM20, BFD_RELOC_16C_IMM20_C, BFD_RELOC_16C_IMM24, BFD_RELOC_16C_IMM24_C, BFD_RELOC_16C_IMM32, BFD_RELOC_16C_IMM32_C, /* NS CR16 Relocations. */ BFD_RELOC_CR16_NUM8, BFD_RELOC_CR16_NUM16, BFD_RELOC_CR16_NUM32, BFD_RELOC_CR16_NUM32a, BFD_RELOC_CR16_REGREL0, BFD_RELOC_CR16_REGREL4, BFD_RELOC_CR16_REGREL4a, BFD_RELOC_CR16_REGREL14, BFD_RELOC_CR16_REGREL14a, BFD_RELOC_CR16_REGREL16, BFD_RELOC_CR16_REGREL20, BFD_RELOC_CR16_REGREL20a, BFD_RELOC_CR16_ABS20, BFD_RELOC_CR16_ABS24, BFD_RELOC_CR16_IMM4, BFD_RELOC_CR16_IMM8, BFD_RELOC_CR16_IMM16, BFD_RELOC_CR16_IMM20, BFD_RELOC_CR16_IMM24, BFD_RELOC_CR16_IMM32, BFD_RELOC_CR16_IMM32a, BFD_RELOC_CR16_DISP4, BFD_RELOC_CR16_DISP8, BFD_RELOC_CR16_DISP16, BFD_RELOC_CR16_DISP20, BFD_RELOC_CR16_DISP24, BFD_RELOC_CR16_DISP24a, /* NS CRX Relocations. */ BFD_RELOC_CRX_REL4, BFD_RELOC_CRX_REL8, BFD_RELOC_CRX_REL8_CMP, BFD_RELOC_CRX_REL16, BFD_RELOC_CRX_REL24, BFD_RELOC_CRX_REL32, BFD_RELOC_CRX_REGREL12, BFD_RELOC_CRX_REGREL22, BFD_RELOC_CRX_REGREL28, BFD_RELOC_CRX_REGREL32, BFD_RELOC_CRX_ABS16, BFD_RELOC_CRX_ABS32, BFD_RELOC_CRX_NUM8, BFD_RELOC_CRX_NUM16, BFD_RELOC_CRX_NUM32, BFD_RELOC_CRX_IMM16, BFD_RELOC_CRX_IMM32, BFD_RELOC_CRX_SWITCH8, BFD_RELOC_CRX_SWITCH16, BFD_RELOC_CRX_SWITCH32, /* These relocs are only used within the CRIS assembler. They are not (at present) written to any object files. */ BFD_RELOC_CRIS_BDISP8, BFD_RELOC_CRIS_UNSIGNED_5, BFD_RELOC_CRIS_SIGNED_6, BFD_RELOC_CRIS_UNSIGNED_6, BFD_RELOC_CRIS_SIGNED_8, BFD_RELOC_CRIS_UNSIGNED_8, BFD_RELOC_CRIS_SIGNED_16, BFD_RELOC_CRIS_UNSIGNED_16, BFD_RELOC_CRIS_LAPCQ_OFFSET, BFD_RELOC_CRIS_UNSIGNED_4, /* Relocs used in ELF shared libraries for CRIS. */ BFD_RELOC_CRIS_COPY, BFD_RELOC_CRIS_GLOB_DAT, BFD_RELOC_CRIS_JUMP_SLOT, BFD_RELOC_CRIS_RELATIVE, /* 32-bit offset to symbol-entry within GOT. */ BFD_RELOC_CRIS_32_GOT, /* 16-bit offset to symbol-entry within GOT. */ BFD_RELOC_CRIS_16_GOT, /* 32-bit offset to symbol-entry within GOT, with PLT handling. */ BFD_RELOC_CRIS_32_GOTPLT, /* 16-bit offset to symbol-entry within GOT, with PLT handling. */ BFD_RELOC_CRIS_16_GOTPLT, /* 32-bit offset to symbol, relative to GOT. */ BFD_RELOC_CRIS_32_GOTREL, /* 32-bit offset to symbol with PLT entry, relative to GOT. */ BFD_RELOC_CRIS_32_PLT_GOTREL, /* 32-bit offset to symbol with PLT entry, relative to this relocation. */ BFD_RELOC_CRIS_32_PLT_PCREL, /* Intel i860 Relocations. */ BFD_RELOC_860_COPY, BFD_RELOC_860_GLOB_DAT, BFD_RELOC_860_JUMP_SLOT, BFD_RELOC_860_RELATIVE, BFD_RELOC_860_PC26, BFD_RELOC_860_PLT26, BFD_RELOC_860_PC16, BFD_RELOC_860_LOW0, BFD_RELOC_860_SPLIT0, BFD_RELOC_860_LOW1, BFD_RELOC_860_SPLIT1, BFD_RELOC_860_LOW2, BFD_RELOC_860_SPLIT2, BFD_RELOC_860_LOW3, BFD_RELOC_860_LOGOT0, BFD_RELOC_860_SPGOT0, BFD_RELOC_860_LOGOT1, BFD_RELOC_860_SPGOT1, BFD_RELOC_860_LOGOTOFF0, BFD_RELOC_860_SPGOTOFF0, BFD_RELOC_860_LOGOTOFF1, BFD_RELOC_860_SPGOTOFF1, BFD_RELOC_860_LOGOTOFF2, BFD_RELOC_860_LOGOTOFF3, BFD_RELOC_860_LOPC, BFD_RELOC_860_HIGHADJ, BFD_RELOC_860_HAGOT, BFD_RELOC_860_HAGOTOFF, BFD_RELOC_860_HAPC, BFD_RELOC_860_HIGH, BFD_RELOC_860_HIGOT, BFD_RELOC_860_HIGOTOFF, /* OpenRISC Relocations. */ BFD_RELOC_OPENRISC_ABS_26, BFD_RELOC_OPENRISC_REL_26, /* H8 elf Relocations. */ BFD_RELOC_H8_DIR16A8, BFD_RELOC_H8_DIR16R8, BFD_RELOC_H8_DIR24A8, BFD_RELOC_H8_DIR24R8, BFD_RELOC_H8_DIR32A16, /* Sony Xstormy16 Relocations. */ BFD_RELOC_XSTORMY16_REL_12, BFD_RELOC_XSTORMY16_12, BFD_RELOC_XSTORMY16_24, BFD_RELOC_XSTORMY16_FPTR16, /* Self-describing complex relocations. */ BFD_RELOC_RELC, /* Infineon Relocations. */ BFD_RELOC_XC16X_PAG, BFD_RELOC_XC16X_POF, BFD_RELOC_XC16X_SEG, BFD_RELOC_XC16X_SOF, /* Relocations used by VAX ELF. */ BFD_RELOC_VAX_GLOB_DAT, BFD_RELOC_VAX_JMP_SLOT, BFD_RELOC_VAX_RELATIVE, /* Morpho MT - 16 bit immediate relocation. */ BFD_RELOC_MT_PC16, /* Morpho MT - Hi 16 bits of an address. */ BFD_RELOC_MT_HI16, /* Morpho MT - Low 16 bits of an address. */ BFD_RELOC_MT_LO16, /* Morpho MT - Used to tell the linker which vtable entries are used. */ BFD_RELOC_MT_GNU_VTINHERIT, /* Morpho MT - Used to tell the linker which vtable entries are used. */ BFD_RELOC_MT_GNU_VTENTRY, /* Morpho MT - 8 bit immediate relocation. */ BFD_RELOC_MT_PCINSN8, /* msp430 specific relocation codes */ BFD_RELOC_MSP430_10_PCREL, BFD_RELOC_MSP430_16_PCREL, BFD_RELOC_MSP430_16, BFD_RELOC_MSP430_16_PCREL_BYTE, BFD_RELOC_MSP430_16_BYTE, BFD_RELOC_MSP430_2X_PCREL, BFD_RELOC_MSP430_RL_PCREL, /* IQ2000 Relocations. */ BFD_RELOC_IQ2000_OFFSET_16, BFD_RELOC_IQ2000_OFFSET_21, BFD_RELOC_IQ2000_UHI16, /* Special Xtensa relocation used only by PLT entries in ELF shared objects to indicate that the runtime linker should set the value to one of its own internal functions or data structures. */ BFD_RELOC_XTENSA_RTLD, /* Xtensa relocations for ELF shared objects. */ BFD_RELOC_XTENSA_GLOB_DAT, BFD_RELOC_XTENSA_JMP_SLOT, BFD_RELOC_XTENSA_RELATIVE, /* Xtensa relocation used in ELF object files for symbols that may require PLT entries. Otherwise, this is just a generic 32-bit relocation. */ BFD_RELOC_XTENSA_PLT, /* Xtensa relocations to mark the difference of two local symbols. These are only needed to support linker relaxation and can be ignored when not relaxing. The field is set to the value of the difference assuming no relaxation. The relocation encodes the position of the first symbol so the linker can determine whether to adjust the field value. */ BFD_RELOC_XTENSA_DIFF8, BFD_RELOC_XTENSA_DIFF16, BFD_RELOC_XTENSA_DIFF32, /* Generic Xtensa relocations for instruction operands. Only the slot number is encoded in the relocation. The relocation applies to the last PC-relative immediate operand, or if there are no PC-relative immediates, to the last immediate operand. */ BFD_RELOC_XTENSA_SLOT0_OP, BFD_RELOC_XTENSA_SLOT1_OP, BFD_RELOC_XTENSA_SLOT2_OP, BFD_RELOC_XTENSA_SLOT3_OP, BFD_RELOC_XTENSA_SLOT4_OP, BFD_RELOC_XTENSA_SLOT5_OP, BFD_RELOC_XTENSA_SLOT6_OP, BFD_RELOC_XTENSA_SLOT7_OP, BFD_RELOC_XTENSA_SLOT8_OP, BFD_RELOC_XTENSA_SLOT9_OP, BFD_RELOC_XTENSA_SLOT10_OP, BFD_RELOC_XTENSA_SLOT11_OP, BFD_RELOC_XTENSA_SLOT12_OP, BFD_RELOC_XTENSA_SLOT13_OP, BFD_RELOC_XTENSA_SLOT14_OP, /* Alternate Xtensa relocations. Only the slot is encoded in the relocation. The meaning of these relocations is opcode-specific. */ BFD_RELOC_XTENSA_SLOT0_ALT, BFD_RELOC_XTENSA_SLOT1_ALT, BFD_RELOC_XTENSA_SLOT2_ALT, BFD_RELOC_XTENSA_SLOT3_ALT, BFD_RELOC_XTENSA_SLOT4_ALT, BFD_RELOC_XTENSA_SLOT5_ALT, BFD_RELOC_XTENSA_SLOT6_ALT, BFD_RELOC_XTENSA_SLOT7_ALT, BFD_RELOC_XTENSA_SLOT8_ALT, BFD_RELOC_XTENSA_SLOT9_ALT, BFD_RELOC_XTENSA_SLOT10_ALT, BFD_RELOC_XTENSA_SLOT11_ALT, BFD_RELOC_XTENSA_SLOT12_ALT, BFD_RELOC_XTENSA_SLOT13_ALT, BFD_RELOC_XTENSA_SLOT14_ALT, /* Xtensa relocations for backward compatibility. These have all been replaced by BFD_RELOC_XTENSA_SLOT0_OP. */ BFD_RELOC_XTENSA_OP0, BFD_RELOC_XTENSA_OP1, BFD_RELOC_XTENSA_OP2, /* Xtensa relocation to mark that the assembler expanded the instructions from an original target. The expansion size is encoded in the reloc size. */ BFD_RELOC_XTENSA_ASM_EXPAND, /* Xtensa relocation to mark that the linker should simplify assembler-expanded instructions. This is commonly used internally by the linker after analysis of a BFD_RELOC_XTENSA_ASM_EXPAND. */ BFD_RELOC_XTENSA_ASM_SIMPLIFY, /* 8 bit signed offset in (ix+d) or (iy+d). */ BFD_RELOC_Z80_DISP8, /* DJNZ offset. */ BFD_RELOC_Z8K_DISP7, /* CALR offset. */ BFD_RELOC_Z8K_CALLR, /* 4 bit value. */ BFD_RELOC_Z8K_IMM4L, BFD_RELOC_UNUSED }; typedef enum bfd_reloc_code_real bfd_reloc_code_real_type; reloc_howto_type *bfd_reloc_type_lookup (bfd *abfd, bfd_reloc_code_real_type code); reloc_howto_type *bfd_reloc_name_lookup (bfd *abfd, const char *reloc_name); const char *bfd_get_reloc_code_name (bfd_reloc_code_real_type code); /* Extracted from syms.c. */ typedef struct bfd_symbol { /* A pointer to the BFD which owns the symbol. This information is necessary so that a back end can work out what additional information (invisible to the application writer) is carried with the symbol. This field is *almost* redundant, since you can use section->owner instead, except that some symbols point to the global sections bfd_{abs,com,und}_section. This could be fixed by making these globals be per-bfd (or per-target-flavor). FIXME. */ struct bfd *the_bfd; /* Use bfd_asymbol_bfd(sym) to access this field. */ /* The text of the symbol. The name is left alone, and not copied; the application may not alter it. */ const char *name; /* The value of the symbol. This really should be a union of a numeric value with a pointer, since some flags indicate that a pointer to another symbol is stored here. */ symvalue value; /* Attributes of a symbol. */ #define BSF_NO_FLAGS 0x00 /* The symbol has local scope; <> in <>. The value is the offset into the section of the data. */ #define BSF_LOCAL 0x01 /* The symbol has global scope; initialized data in <>. The value is the offset into the section of the data. */ #define BSF_GLOBAL 0x02 /* The symbol has global scope and is exported. The value is the offset into the section of the data. */ #define BSF_EXPORT BSF_GLOBAL /* No real difference. */ /* A normal C symbol would be one of: <>, <>, <> or <>. */ /* The symbol is a debugging record. The value has an arbitrary meaning, unless BSF_DEBUGGING_RELOC is also set. */ #define BSF_DEBUGGING 0x08 /* The symbol denotes a function entry point. Used in ELF, perhaps others someday. */ #define BSF_FUNCTION 0x10 /* Used by the linker. */ #define BSF_KEEP 0x20 #define BSF_KEEP_G 0x40 /* A weak global symbol, overridable without warnings by a regular global symbol of the same name. */ #define BSF_WEAK 0x80 /* This symbol was created to point to a section, e.g. ELF's STT_SECTION symbols. */ #define BSF_SECTION_SYM 0x100 /* The symbol used to be a common symbol, but now it is allocated. */ #define BSF_OLD_COMMON 0x200 /* The default value for common data. */ #define BFD_FORT_COMM_DEFAULT_VALUE 0 /* In some files the type of a symbol sometimes alters its location in an output file - ie in coff a <> symbol which is also <> symbol appears where it was declared and not at the end of a section. This bit is set by the target BFD part to convey this information. */ #define BSF_NOT_AT_END 0x400 /* Signal that the symbol is the label of constructor section. */ #define BSF_CONSTRUCTOR 0x800 /* Signal that the symbol is a warning symbol. The name is a warning. The name of the next symbol is the one to warn about; if a reference is made to a symbol with the same name as the next symbol, a warning is issued by the linker. */ #define BSF_WARNING 0x1000 /* Signal that the symbol is indirect. This symbol is an indirect pointer to the symbol with the same name as the next symbol. */ #define BSF_INDIRECT 0x2000 /* BSF_FILE marks symbols that contain a file name. This is used for ELF STT_FILE symbols. */ #define BSF_FILE 0x4000 /* Symbol is from dynamic linking information. */ #define BSF_DYNAMIC 0x8000 /* The symbol denotes a data object. Used in ELF, and perhaps others someday. */ #define BSF_OBJECT 0x10000 /* This symbol is a debugging symbol. The value is the offset into the section of the data. BSF_DEBUGGING should be set as well. */ #define BSF_DEBUGGING_RELOC 0x20000 /* This symbol is thread local. Used in ELF. */ #define BSF_THREAD_LOCAL 0x40000 /* This symbol represents a complex relocation expression, with the expression tree serialized in the symbol name. */ #define BSF_RELC 0x80000 /* This symbol represents a signed complex relocation expression, with the expression tree serialized in the symbol name. */ #define BSF_SRELC 0x100000 flagword flags; /* A pointer to the section to which this symbol is relative. This will always be non NULL, there are special sections for undefined and absolute symbols. */ struct bfd_section *section; /* Back end special data. */ union { void *p; bfd_vma i; } udata; } asymbol; #define bfd_get_symtab_upper_bound(abfd) \ BFD_SEND (abfd, _bfd_get_symtab_upper_bound, (abfd)) bfd_boolean bfd_is_local_label (bfd *abfd, asymbol *sym); bfd_boolean bfd_is_local_label_name (bfd *abfd, const char *name); #define bfd_is_local_label_name(abfd, name) \ BFD_SEND (abfd, _bfd_is_local_label_name, (abfd, name)) bfd_boolean bfd_is_target_special_symbol (bfd *abfd, asymbol *sym); #define bfd_is_target_special_symbol(abfd, sym) \ BFD_SEND (abfd, _bfd_is_target_special_symbol, (abfd, sym)) #define bfd_canonicalize_symtab(abfd, location) \ BFD_SEND (abfd, _bfd_canonicalize_symtab, (abfd, location)) bfd_boolean bfd_set_symtab (bfd *abfd, asymbol **location, unsigned int count); void bfd_print_symbol_vandf (bfd *abfd, void *file, asymbol *symbol); #define bfd_make_empty_symbol(abfd) \ BFD_SEND (abfd, _bfd_make_empty_symbol, (abfd)) asymbol *_bfd_generic_make_empty_symbol (bfd *); #define bfd_make_debug_symbol(abfd,ptr,size) \ BFD_SEND (abfd, _bfd_make_debug_symbol, (abfd, ptr, size)) int bfd_decode_symclass (asymbol *symbol); bfd_boolean bfd_is_undefined_symclass (int symclass); void bfd_symbol_info (asymbol *symbol, symbol_info *ret); bfd_boolean bfd_copy_private_symbol_data (bfd *ibfd, asymbol *isym, bfd *obfd, asymbol *osym); #define bfd_copy_private_symbol_data(ibfd, isymbol, obfd, osymbol) \ BFD_SEND (obfd, _bfd_copy_private_symbol_data, \ (ibfd, isymbol, obfd, osymbol)) /* Extracted from bfd.c. */ struct bfd { /* A unique identifier of the BFD */ unsigned int id; /* The filename the application opened the BFD with. */ const char *filename; /* A pointer to the target jump table. */ const struct bfd_target *xvec; /* The IOSTREAM, and corresponding IO vector that provide access to the file backing the BFD. */ void *iostream; const struct bfd_iovec *iovec; /* Is the file descriptor being cached? That is, can it be closed as needed, and re-opened when accessed later? */ bfd_boolean cacheable; /* Marks whether there was a default target specified when the BFD was opened. This is used to select which matching algorithm to use to choose the back end. */ bfd_boolean target_defaulted; /* The caching routines use these to maintain a least-recently-used list of BFDs. */ struct bfd *lru_prev, *lru_next; /* When a file is closed by the caching routines, BFD retains state information on the file here... */ ufile_ptr where; /* ... and here: (``once'' means at least once). */ bfd_boolean opened_once; /* Set if we have a locally maintained mtime value, rather than getting it from the file each time. */ bfd_boolean mtime_set; /* File modified time, if mtime_set is TRUE. */ long mtime; /* Reserved for an unimplemented file locking extension. */ int ifd; /* The format which belongs to the BFD. (object, core, etc.) */ bfd_format format; /* The direction with which the BFD was opened. */ enum bfd_direction { no_direction = 0, read_direction = 1, write_direction = 2, both_direction = 3 } direction; /* Format_specific flags. */ flagword flags; /* Currently my_archive is tested before adding origin to anything. I believe that this can become always an add of origin, with origin set to 0 for non archive files. */ ufile_ptr origin; /* Remember when output has begun, to stop strange things from happening. */ bfd_boolean output_has_begun; /* A hash table for section names. */ struct bfd_hash_table section_htab; /* Pointer to linked list of sections. */ struct bfd_section *sections; /* The last section on the section list. */ struct bfd_section *section_last; /* The number of sections. */ unsigned int section_count; /* Stuff only useful for object files: The start address. */ bfd_vma start_address; /* Used for input and output. */ unsigned int symcount; /* Symbol table for output BFD (with symcount entries). */ struct bfd_symbol **outsymbols; /* Used for slurped dynamic symbol tables. */ unsigned int dynsymcount; /* Pointer to structure which contains architecture information. */ const struct bfd_arch_info *arch_info; /* Flag set if symbols from this BFD should not be exported. */ bfd_boolean no_export; /* Stuff only useful for archives. */ void *arelt_data; struct bfd *my_archive; /* The containing archive BFD. */ struct bfd *archive_next; /* The next BFD in the archive. */ struct bfd *archive_head; /* The first BFD in the archive. */ bfd_boolean has_armap; /* A chain of BFD structures involved in a link. */ struct bfd *link_next; /* A field used by _bfd_generic_link_add_archive_symbols. This will be used only for archive elements. */ int archive_pass; /* Used by the back end to hold private data. */ union { struct aout_data_struct *aout_data; struct artdata *aout_ar_data; struct _oasys_data *oasys_obj_data; struct _oasys_ar_data *oasys_ar_data; struct coff_tdata *coff_obj_data; struct pe_tdata *pe_obj_data; struct xcoff_tdata *xcoff_obj_data; struct ecoff_tdata *ecoff_obj_data; struct ieee_data_struct *ieee_data; struct ieee_ar_data_struct *ieee_ar_data; struct srec_data_struct *srec_data; struct ihex_data_struct *ihex_data; struct tekhex_data_struct *tekhex_data; struct elf_obj_tdata *elf_obj_data; struct nlm_obj_tdata *nlm_obj_data; struct bout_data_struct *bout_data; struct mmo_data_struct *mmo_data; struct sun_core_struct *sun_core_data; struct sco5_core_struct *sco5_core_data; struct trad_core_struct *trad_core_data; struct som_data_struct *som_data; struct hpux_core_struct *hpux_core_data; struct hppabsd_core_struct *hppabsd_core_data; struct sgi_core_struct *sgi_core_data; struct lynx_core_struct *lynx_core_data; struct osf_core_struct *osf_core_data; struct cisco_core_struct *cisco_core_data; struct versados_data_struct *versados_data; struct netbsd_core_struct *netbsd_core_data; struct mach_o_data_struct *mach_o_data; struct mach_o_fat_data_struct *mach_o_fat_data; struct bfd_pef_data_struct *pef_data; struct bfd_pef_xlib_data_struct *pef_xlib_data; struct bfd_sym_data_struct *sym_data; void *any; } tdata; /* Used by the application to hold private data. */ void *usrdata; /* Where all the allocated stuff under this BFD goes. This is a struct objalloc *, but we use void * to avoid requiring the inclusion of objalloc.h. */ void *memory; }; typedef enum bfd_error { bfd_error_no_error = 0, bfd_error_system_call, bfd_error_invalid_target, bfd_error_wrong_format, bfd_error_wrong_object_format, bfd_error_invalid_operation, bfd_error_no_memory, bfd_error_no_symbols, bfd_error_no_armap, bfd_error_no_more_archived_files, bfd_error_malformed_archive, bfd_error_file_not_recognized, bfd_error_file_ambiguously_recognized, bfd_error_no_contents, bfd_error_nonrepresentable_section, bfd_error_no_debug_section, bfd_error_bad_value, bfd_error_file_truncated, bfd_error_file_too_big, bfd_error_on_input, bfd_error_invalid_error_code } bfd_error_type; bfd_error_type bfd_get_error (void); void bfd_set_error (bfd_error_type error_tag, ...); const char *bfd_errmsg (bfd_error_type error_tag); void bfd_perror (const char *message); typedef void (*bfd_error_handler_type) (const char *, ...); bfd_error_handler_type bfd_set_error_handler (bfd_error_handler_type); void bfd_set_error_program_name (const char *); bfd_error_handler_type bfd_get_error_handler (void); long bfd_get_reloc_upper_bound (bfd *abfd, asection *sect); long bfd_canonicalize_reloc (bfd *abfd, asection *sec, arelent **loc, asymbol **syms); void bfd_set_reloc (bfd *abfd, asection *sec, arelent **rel, unsigned int count); bfd_boolean bfd_set_file_flags (bfd *abfd, flagword flags); int bfd_get_arch_size (bfd *abfd); int bfd_get_sign_extend_vma (bfd *abfd); bfd_boolean bfd_set_start_address (bfd *abfd, bfd_vma vma); unsigned int bfd_get_gp_size (bfd *abfd); void bfd_set_gp_size (bfd *abfd, unsigned int i); bfd_vma bfd_scan_vma (const char *string, const char **end, int base); bfd_boolean bfd_copy_private_header_data (bfd *ibfd, bfd *obfd); #define bfd_copy_private_header_data(ibfd, obfd) \ BFD_SEND (obfd, _bfd_copy_private_header_data, \ (ibfd, obfd)) bfd_boolean bfd_copy_private_bfd_data (bfd *ibfd, bfd *obfd); #define bfd_copy_private_bfd_data(ibfd, obfd) \ BFD_SEND (obfd, _bfd_copy_private_bfd_data, \ (ibfd, obfd)) bfd_boolean bfd_merge_private_bfd_data (bfd *ibfd, bfd *obfd); #define bfd_merge_private_bfd_data(ibfd, obfd) \ BFD_SEND (obfd, _bfd_merge_private_bfd_data, \ (ibfd, obfd)) bfd_boolean bfd_set_private_flags (bfd *abfd, flagword flags); #define bfd_set_private_flags(abfd, flags) \ BFD_SEND (abfd, _bfd_set_private_flags, (abfd, flags)) #define bfd_sizeof_headers(abfd, info) \ BFD_SEND (abfd, _bfd_sizeof_headers, (abfd, info)) #define bfd_find_nearest_line(abfd, sec, syms, off, file, func, line) \ BFD_SEND (abfd, _bfd_find_nearest_line, \ (abfd, sec, syms, off, file, func, line)) #define bfd_find_line(abfd, syms, sym, file, line) \ BFD_SEND (abfd, _bfd_find_line, \ (abfd, syms, sym, file, line)) #define bfd_find_inliner_info(abfd, file, func, line) \ BFD_SEND (abfd, _bfd_find_inliner_info, \ (abfd, file, func, line)) #define bfd_debug_info_start(abfd) \ BFD_SEND (abfd, _bfd_debug_info_start, (abfd)) #define bfd_debug_info_end(abfd) \ BFD_SEND (abfd, _bfd_debug_info_end, (abfd)) #define bfd_debug_info_accumulate(abfd, section) \ BFD_SEND (abfd, _bfd_debug_info_accumulate, (abfd, section)) #define bfd_stat_arch_elt(abfd, stat) \ BFD_SEND (abfd, _bfd_stat_arch_elt,(abfd, stat)) #define bfd_update_armap_timestamp(abfd) \ BFD_SEND (abfd, _bfd_update_armap_timestamp, (abfd)) #define bfd_set_arch_mach(abfd, arch, mach)\ BFD_SEND ( abfd, _bfd_set_arch_mach, (abfd, arch, mach)) #define bfd_relax_section(abfd, section, link_info, again) \ BFD_SEND (abfd, _bfd_relax_section, (abfd, section, link_info, again)) #define bfd_gc_sections(abfd, link_info) \ BFD_SEND (abfd, _bfd_gc_sections, (abfd, link_info)) #define bfd_merge_sections(abfd, link_info) \ BFD_SEND (abfd, _bfd_merge_sections, (abfd, link_info)) #define bfd_is_group_section(abfd, sec) \ BFD_SEND (abfd, _bfd_is_group_section, (abfd, sec)) #define bfd_discard_group(abfd, sec) \ BFD_SEND (abfd, _bfd_discard_group, (abfd, sec)) #define bfd_link_hash_table_create(abfd) \ BFD_SEND (abfd, _bfd_link_hash_table_create, (abfd)) #define bfd_link_hash_table_free(abfd, hash) \ BFD_SEND (abfd, _bfd_link_hash_table_free, (hash)) #define bfd_link_add_symbols(abfd, info) \ BFD_SEND (abfd, _bfd_link_add_symbols, (abfd, info)) #define bfd_link_just_syms(abfd, sec, info) \ BFD_SEND (abfd, _bfd_link_just_syms, (sec, info)) #define bfd_final_link(abfd, info) \ BFD_SEND (abfd, _bfd_final_link, (abfd, info)) #define bfd_free_cached_info(abfd) \ BFD_SEND (abfd, _bfd_free_cached_info, (abfd)) #define bfd_get_dynamic_symtab_upper_bound(abfd) \ BFD_SEND (abfd, _bfd_get_dynamic_symtab_upper_bound, (abfd)) #define bfd_print_private_bfd_data(abfd, file)\ BFD_SEND (abfd, _bfd_print_private_bfd_data, (abfd, file)) #define bfd_canonicalize_dynamic_symtab(abfd, asymbols) \ BFD_SEND (abfd, _bfd_canonicalize_dynamic_symtab, (abfd, asymbols)) #define bfd_get_synthetic_symtab(abfd, count, syms, dyncount, dynsyms, ret) \ BFD_SEND (abfd, _bfd_get_synthetic_symtab, (abfd, count, syms, \ dyncount, dynsyms, ret)) #define bfd_get_dynamic_reloc_upper_bound(abfd) \ BFD_SEND (abfd, _bfd_get_dynamic_reloc_upper_bound, (abfd)) #define bfd_canonicalize_dynamic_reloc(abfd, arels, asyms) \ BFD_SEND (abfd, _bfd_canonicalize_dynamic_reloc, (abfd, arels, asyms)) extern bfd_byte *bfd_get_relocated_section_contents (bfd *, struct bfd_link_info *, struct bfd_link_order *, bfd_byte *, bfd_boolean, asymbol **); bfd_boolean bfd_alt_mach_code (bfd *abfd, int alternative); struct bfd_preserve { void *marker; void *tdata; flagword flags; const struct bfd_arch_info *arch_info; struct bfd_section *sections; struct bfd_section *section_last; unsigned int section_count; struct bfd_hash_table section_htab; }; bfd_boolean bfd_preserve_save (bfd *, struct bfd_preserve *); void bfd_preserve_restore (bfd *, struct bfd_preserve *); void bfd_preserve_finish (bfd *, struct bfd_preserve *); bfd_vma bfd_emul_get_maxpagesize (const char *); void bfd_emul_set_maxpagesize (const char *, bfd_vma); bfd_vma bfd_emul_get_commonpagesize (const char *); void bfd_emul_set_commonpagesize (const char *, bfd_vma); char *bfd_demangle (bfd *, const char *, int); /* Extracted from archive.c. */ symindex bfd_get_next_mapent (bfd *abfd, symindex previous, carsym **sym); bfd_boolean bfd_set_archive_head (bfd *output, bfd *new_head); bfd *bfd_openr_next_archived_file (bfd *archive, bfd *previous); /* Extracted from corefile.c. */ const char *bfd_core_file_failing_command (bfd *abfd); int bfd_core_file_failing_signal (bfd *abfd); bfd_boolean core_file_matches_executable_p (bfd *core_bfd, bfd *exec_bfd); bfd_boolean generic_core_file_matches_executable_p (bfd *core_bfd, bfd *exec_bfd); /* Extracted from targets.c. */ #define BFD_SEND(bfd, message, arglist) \ ((*((bfd)->xvec->message)) arglist) #ifdef DEBUG_BFD_SEND #undef BFD_SEND #define BFD_SEND(bfd, message, arglist) \ (((bfd) && (bfd)->xvec && (bfd)->xvec->message) ? \ ((*((bfd)->xvec->message)) arglist) : \ (bfd_assert (__FILE__,__LINE__), NULL)) #endif #define BFD_SEND_FMT(bfd, message, arglist) \ (((bfd)->xvec->message[(int) ((bfd)->format)]) arglist) #ifdef DEBUG_BFD_SEND #undef BFD_SEND_FMT #define BFD_SEND_FMT(bfd, message, arglist) \ (((bfd) && (bfd)->xvec && (bfd)->xvec->message) ? \ (((bfd)->xvec->message[(int) ((bfd)->format)]) arglist) : \ (bfd_assert (__FILE__,__LINE__), NULL)) #endif enum bfd_flavour { bfd_target_unknown_flavour, bfd_target_aout_flavour, bfd_target_coff_flavour, bfd_target_ecoff_flavour, bfd_target_xcoff_flavour, bfd_target_elf_flavour, bfd_target_ieee_flavour, bfd_target_nlm_flavour, bfd_target_oasys_flavour, bfd_target_tekhex_flavour, bfd_target_srec_flavour, bfd_target_ihex_flavour, bfd_target_som_flavour, bfd_target_os9k_flavour, bfd_target_versados_flavour, bfd_target_msdos_flavour, bfd_target_ovax_flavour, bfd_target_evax_flavour, bfd_target_mmo_flavour, bfd_target_mach_o_flavour, bfd_target_pef_flavour, bfd_target_pef_xlib_flavour, bfd_target_sym_flavour }; enum bfd_endian { BFD_ENDIAN_BIG, BFD_ENDIAN_LITTLE, BFD_ENDIAN_UNKNOWN }; /* Forward declaration. */ typedef struct bfd_link_info _bfd_link_info; typedef struct bfd_target { /* Identifies the kind of target, e.g., SunOS4, Ultrix, etc. */ char *name; /* The "flavour" of a back end is a general indication about the contents of a file. */ enum bfd_flavour flavour; /* The order of bytes within the data area of a file. */ enum bfd_endian byteorder; /* The order of bytes within the header parts of a file. */ enum bfd_endian header_byteorder; /* A mask of all the flags which an executable may have set - from the set <>, <>, ...<>. */ flagword object_flags; /* A mask of all the flags which a section may have set - from the set <>, <>, ...<>. */ flagword section_flags; /* The character normally found at the front of a symbol. (if any), perhaps `_'. */ char symbol_leading_char; /* The pad character for file names within an archive header. */ char ar_pad_char; /* The maximum number of characters in an archive header. */ unsigned short ar_max_namelen; /* Entries for byte swapping for data. These are different from the other entry points, since they don't take a BFD as the first argument. Certain other handlers could do the same. */ bfd_uint64_t (*bfd_getx64) (const void *); bfd_int64_t (*bfd_getx_signed_64) (const void *); void (*bfd_putx64) (bfd_uint64_t, void *); bfd_vma (*bfd_getx32) (const void *); bfd_signed_vma (*bfd_getx_signed_32) (const void *); void (*bfd_putx32) (bfd_vma, void *); bfd_vma (*bfd_getx16) (const void *); bfd_signed_vma (*bfd_getx_signed_16) (const void *); void (*bfd_putx16) (bfd_vma, void *); /* Byte swapping for the headers. */ bfd_uint64_t (*bfd_h_getx64) (const void *); bfd_int64_t (*bfd_h_getx_signed_64) (const void *); void (*bfd_h_putx64) (bfd_uint64_t, void *); bfd_vma (*bfd_h_getx32) (const void *); bfd_signed_vma (*bfd_h_getx_signed_32) (const void *); void (*bfd_h_putx32) (bfd_vma, void *); bfd_vma (*bfd_h_getx16) (const void *); bfd_signed_vma (*bfd_h_getx_signed_16) (const void *); void (*bfd_h_putx16) (bfd_vma, void *); /* Format dependent routines: these are vectors of entry points within the target vector structure, one for each format to check. */ /* Check the format of a file being read. Return a <> or zero. */ const struct bfd_target *(*_bfd_check_format[bfd_type_end]) (bfd *); /* Set the format of a file being written. */ bfd_boolean (*_bfd_set_format[bfd_type_end]) (bfd *); /* Write cached information into a file being written, at <>. */ bfd_boolean (*_bfd_write_contents[bfd_type_end]) (bfd *); /* Generic entry points. */ #define BFD_JUMP_TABLE_GENERIC(NAME) \ NAME##_close_and_cleanup, \ NAME##_bfd_free_cached_info, \ NAME##_new_section_hook, \ NAME##_get_section_contents, \ NAME##_get_section_contents_in_window /* Called when the BFD is being closed to do any necessary cleanup. */ bfd_boolean (*_close_and_cleanup) (bfd *); /* Ask the BFD to free all cached information. */ bfd_boolean (*_bfd_free_cached_info) (bfd *); /* Called when a new section is created. */ bfd_boolean (*_new_section_hook) (bfd *, sec_ptr); /* Read the contents of a section. */ bfd_boolean (*_bfd_get_section_contents) (bfd *, sec_ptr, void *, file_ptr, bfd_size_type); bfd_boolean (*_bfd_get_section_contents_in_window) (bfd *, sec_ptr, bfd_window *, file_ptr, bfd_size_type); /* Entry points to copy private data. */ #define BFD_JUMP_TABLE_COPY(NAME) \ NAME##_bfd_copy_private_bfd_data, \ NAME##_bfd_merge_private_bfd_data, \ _bfd_generic_init_private_section_data, \ NAME##_bfd_copy_private_section_data, \ NAME##_bfd_copy_private_symbol_data, \ NAME##_bfd_copy_private_header_data, \ NAME##_bfd_set_private_flags, \ NAME##_bfd_print_private_bfd_data /* Called to copy BFD general private data from one object file to another. */ bfd_boolean (*_bfd_copy_private_bfd_data) (bfd *, bfd *); /* Called to merge BFD general private data from one object file to a common output file when linking. */ bfd_boolean (*_bfd_merge_private_bfd_data) (bfd *, bfd *); /* Called to initialize BFD private section data from one object file to another. */ #define bfd_init_private_section_data(ibfd, isec, obfd, osec, link_info) \ BFD_SEND (obfd, _bfd_init_private_section_data, (ibfd, isec, obfd, osec, link_info)) bfd_boolean (*_bfd_init_private_section_data) (bfd *, sec_ptr, bfd *, sec_ptr, struct bfd_link_info *); /* Called to copy BFD private section data from one object file to another. */ bfd_boolean (*_bfd_copy_private_section_data) (bfd *, sec_ptr, bfd *, sec_ptr); /* Called to copy BFD private symbol data from one symbol to another. */ bfd_boolean (*_bfd_copy_private_symbol_data) (bfd *, asymbol *, bfd *, asymbol *); /* Called to copy BFD private header data from one object file to another. */ bfd_boolean (*_bfd_copy_private_header_data) (bfd *, bfd *); /* Called to set private backend flags. */ bfd_boolean (*_bfd_set_private_flags) (bfd *, flagword); /* Called to print private BFD data. */ bfd_boolean (*_bfd_print_private_bfd_data) (bfd *, void *); /* Core file entry points. */ #define BFD_JUMP_TABLE_CORE(NAME) \ NAME##_core_file_failing_command, \ NAME##_core_file_failing_signal, \ NAME##_core_file_matches_executable_p char * (*_core_file_failing_command) (bfd *); int (*_core_file_failing_signal) (bfd *); bfd_boolean (*_core_file_matches_executable_p) (bfd *, bfd *); /* Archive entry points. */ #define BFD_JUMP_TABLE_ARCHIVE(NAME) \ NAME##_slurp_armap, \ NAME##_slurp_extended_name_table, \ NAME##_construct_extended_name_table, \ NAME##_truncate_arname, \ NAME##_write_armap, \ NAME##_read_ar_hdr, \ NAME##_openr_next_archived_file, \ NAME##_get_elt_at_index, \ NAME##_generic_stat_arch_elt, \ NAME##_update_armap_timestamp bfd_boolean (*_bfd_slurp_armap) (bfd *); bfd_boolean (*_bfd_slurp_extended_name_table) (bfd *); bfd_boolean (*_bfd_construct_extended_name_table) (bfd *, char **, bfd_size_type *, const char **); void (*_bfd_truncate_arname) (bfd *, const char *, char *); bfd_boolean (*write_armap) (bfd *, unsigned int, struct orl *, unsigned int, int); void * (*_bfd_read_ar_hdr_fn) (bfd *); bfd * (*openr_next_archived_file) (bfd *, bfd *); #define bfd_get_elt_at_index(b,i) BFD_SEND (b, _bfd_get_elt_at_index, (b,i)) bfd * (*_bfd_get_elt_at_index) (bfd *, symindex); int (*_bfd_stat_arch_elt) (bfd *, struct stat *); bfd_boolean (*_bfd_update_armap_timestamp) (bfd *); /* Entry points used for symbols. */ #define BFD_JUMP_TABLE_SYMBOLS(NAME) \ NAME##_get_symtab_upper_bound, \ NAME##_canonicalize_symtab, \ NAME##_make_empty_symbol, \ NAME##_print_symbol, \ NAME##_get_symbol_info, \ NAME##_bfd_is_local_label_name, \ NAME##_bfd_is_target_special_symbol, \ NAME##_get_lineno, \ NAME##_find_nearest_line, \ _bfd_generic_find_line, \ NAME##_find_inliner_info, \ NAME##_bfd_make_debug_symbol, \ NAME##_read_minisymbols, \ NAME##_minisymbol_to_symbol long (*_bfd_get_symtab_upper_bound) (bfd *); long (*_bfd_canonicalize_symtab) (bfd *, struct bfd_symbol **); struct bfd_symbol * (*_bfd_make_empty_symbol) (bfd *); void (*_bfd_print_symbol) (bfd *, void *, struct bfd_symbol *, bfd_print_symbol_type); #define bfd_print_symbol(b,p,s,e) BFD_SEND (b, _bfd_print_symbol, (b,p,s,e)) void (*_bfd_get_symbol_info) (bfd *, struct bfd_symbol *, symbol_info *); #define bfd_get_symbol_info(b,p,e) BFD_SEND (b, _bfd_get_symbol_info, (b,p,e)) bfd_boolean (*_bfd_is_local_label_name) (bfd *, const char *); bfd_boolean (*_bfd_is_target_special_symbol) (bfd *, asymbol *); alent * (*_get_lineno) (bfd *, struct bfd_symbol *); bfd_boolean (*_bfd_find_nearest_line) (bfd *, struct bfd_section *, struct bfd_symbol **, bfd_vma, const char **, const char **, unsigned int *); bfd_boolean (*_bfd_find_line) (bfd *, struct bfd_symbol **, struct bfd_symbol *, const char **, unsigned int *); bfd_boolean (*_bfd_find_inliner_info) (bfd *, const char **, const char **, unsigned int *); /* Back-door to allow format-aware applications to create debug symbols while using BFD for everything else. Currently used by the assembler when creating COFF files. */ asymbol * (*_bfd_make_debug_symbol) (bfd *, void *, unsigned long size); #define bfd_read_minisymbols(b, d, m, s) \ BFD_SEND (b, _read_minisymbols, (b, d, m, s)) long (*_read_minisymbols) (bfd *, bfd_boolean, void **, unsigned int *); #define bfd_minisymbol_to_symbol(b, d, m, f) \ BFD_SEND (b, _minisymbol_to_symbol, (b, d, m, f)) asymbol * (*_minisymbol_to_symbol) (bfd *, bfd_boolean, const void *, asymbol *); /* Routines for relocs. */ #define BFD_JUMP_TABLE_RELOCS(NAME) \ NAME##_get_reloc_upper_bound, \ NAME##_canonicalize_reloc, \ NAME##_bfd_reloc_type_lookup, \ NAME##_bfd_reloc_name_lookup long (*_get_reloc_upper_bound) (bfd *, sec_ptr); long (*_bfd_canonicalize_reloc) (bfd *, sec_ptr, arelent **, struct bfd_symbol **); /* See documentation on reloc types. */ reloc_howto_type * (*reloc_type_lookup) (bfd *, bfd_reloc_code_real_type); reloc_howto_type * (*reloc_name_lookup) (bfd *, const char *); /* Routines used when writing an object file. */ #define BFD_JUMP_TABLE_WRITE(NAME) \ NAME##_set_arch_mach, \ NAME##_set_section_contents bfd_boolean (*_bfd_set_arch_mach) (bfd *, enum bfd_architecture, unsigned long); bfd_boolean (*_bfd_set_section_contents) (bfd *, sec_ptr, const void *, file_ptr, bfd_size_type); /* Routines used by the linker. */ #define BFD_JUMP_TABLE_LINK(NAME) \ NAME##_sizeof_headers, \ NAME##_bfd_get_relocated_section_contents, \ NAME##_bfd_relax_section, \ NAME##_bfd_link_hash_table_create, \ NAME##_bfd_link_hash_table_free, \ NAME##_bfd_link_add_symbols, \ NAME##_bfd_link_just_syms, \ NAME##_bfd_final_link, \ NAME##_bfd_link_split_section, \ NAME##_bfd_gc_sections, \ NAME##_bfd_merge_sections, \ NAME##_bfd_is_group_section, \ NAME##_bfd_discard_group, \ NAME##_section_already_linked \ int (*_bfd_sizeof_headers) (bfd *, struct bfd_link_info *); bfd_byte * (*_bfd_get_relocated_section_contents) (bfd *, struct bfd_link_info *, struct bfd_link_order *, bfd_byte *, bfd_boolean, struct bfd_symbol **); bfd_boolean (*_bfd_relax_section) (bfd *, struct bfd_section *, struct bfd_link_info *, bfd_boolean *); /* Create a hash table for the linker. Different backends store different information in this table. */ struct bfd_link_hash_table * (*_bfd_link_hash_table_create) (bfd *); /* Release the memory associated with the linker hash table. */ void (*_bfd_link_hash_table_free) (struct bfd_link_hash_table *); /* Add symbols from this object file into the hash table. */ bfd_boolean (*_bfd_link_add_symbols) (bfd *, struct bfd_link_info *); /* Indicate that we are only retrieving symbol values from this section. */ void (*_bfd_link_just_syms) (asection *, struct bfd_link_info *); /* Do a link based on the link_order structures attached to each section of the BFD. */ bfd_boolean (*_bfd_final_link) (bfd *, struct bfd_link_info *); /* Should this section be split up into smaller pieces during linking. */ bfd_boolean (*_bfd_link_split_section) (bfd *, struct bfd_section *); /* Remove sections that are not referenced from the output. */ bfd_boolean (*_bfd_gc_sections) (bfd *, struct bfd_link_info *); /* Attempt to merge SEC_MERGE sections. */ bfd_boolean (*_bfd_merge_sections) (bfd *, struct bfd_link_info *); /* Is this section a member of a group? */ bfd_boolean (*_bfd_is_group_section) (bfd *, const struct bfd_section *); /* Discard members of a group. */ bfd_boolean (*_bfd_discard_group) (bfd *, struct bfd_section *); /* Check if SEC has been already linked during a reloceatable or final link. */ void (*_section_already_linked) (bfd *, struct bfd_section *, struct bfd_link_info *); /* Routines to handle dynamic symbols and relocs. */ #define BFD_JUMP_TABLE_DYNAMIC(NAME) \ NAME##_get_dynamic_symtab_upper_bound, \ NAME##_canonicalize_dynamic_symtab, \ NAME##_get_synthetic_symtab, \ NAME##_get_dynamic_reloc_upper_bound, \ NAME##_canonicalize_dynamic_reloc /* Get the amount of memory required to hold the dynamic symbols. */ long (*_bfd_get_dynamic_symtab_upper_bound) (bfd *); /* Read in the dynamic symbols. */ long (*_bfd_canonicalize_dynamic_symtab) (bfd *, struct bfd_symbol **); /* Create synthetized symbols. */ long (*_bfd_get_synthetic_symtab) (bfd *, long, struct bfd_symbol **, long, struct bfd_symbol **, struct bfd_symbol **); /* Get the amount of memory required to hold the dynamic relocs. */ long (*_bfd_get_dynamic_reloc_upper_bound) (bfd *); /* Read in the dynamic relocs. */ long (*_bfd_canonicalize_dynamic_reloc) (bfd *, arelent **, struct bfd_symbol **); /* Opposite endian version of this target. */ const struct bfd_target * alternative_target; /* Data for use by back-end routines, which isn't generic enough to belong in this structure. */ const void *backend_data; } bfd_target; bfd_boolean bfd_set_default_target (const char *name); const bfd_target *bfd_find_target (const char *target_name, bfd *abfd); const char ** bfd_target_list (void); const bfd_target *bfd_search_for_target (int (*search_func) (const bfd_target *, void *), void *); /* Extracted from format.c. */ bfd_boolean bfd_check_format (bfd *abfd, bfd_format format); bfd_boolean bfd_check_format_matches (bfd *abfd, bfd_format format, char ***matching); bfd_boolean bfd_set_format (bfd *abfd, bfd_format format); const char *bfd_format_string (bfd_format format); /* Extracted from linker.c. */ bfd_boolean bfd_link_split_section (bfd *abfd, asection *sec); #define bfd_link_split_section(abfd, sec) \ BFD_SEND (abfd, _bfd_link_split_section, (abfd, sec)) void bfd_section_already_linked (bfd *abfd, asection *sec, struct bfd_link_info *info); #define bfd_section_already_linked(abfd, sec, info) \ BFD_SEND (abfd, _section_already_linked, (abfd, sec, info)) /* Extracted from simple.c. */ bfd_byte *bfd_simple_get_relocated_section_contents (bfd *abfd, asection *sec, bfd_byte *outbuf, asymbol **symbol_table); #ifdef __cplusplus } #endif #endif Index: projects/clang350-import/gnu/usr.bin/binutils =================================================================== --- projects/clang350-import/gnu/usr.bin/binutils (revision 275748) +++ projects/clang350-import/gnu/usr.bin/binutils (revision 275749) Property changes on: projects/clang350-import/gnu/usr.bin/binutils ___________________________________________________________________ Modified: svn:mergeinfo ## -0,0 +0,1 ## Merged /head/gnu/usr.bin/binutils:r275478-275748 Index: projects/clang350-import/share/man/man4/crypto.4 =================================================================== --- projects/clang350-import/share/man/man4/crypto.4 (revision 275748) +++ projects/clang350-import/share/man/man4/crypto.4 (revision 275749) @@ -1,125 +1,434 @@ -.\" $OpenBSD: crypto.4,v 1.4 2002/09/12 07:15:03 deraadt Exp $ +.\" $NetBSD: crypto.4,v 1.24 2014/01/27 21:23:59 pgoyette Exp $ .\" -.\" Copyright (c) 2001 Theo de Raadt +.\" Copyright (c) 2008 The NetBSD Foundation, Inc. +.\" Copyright (c) 2014 The FreeBSD Foundation .\" All rights reserved. .\" +.\" Portions of this documentation were written by John-Mark Gurney +.\" under sponsorship of the FreeBSD Foundation and +.\" Rubicon Communications, LLC (Netgate). +.\" +.\" This code is derived from software contributed to The NetBSD Foundation +.\" by Coyote Point Systems, Inc. +.\" .\" Redistribution and use in source and binary forms, with or without .\" modification, are permitted provided that the following conditions .\" are met: .\" 1. Redistributions of source code must retain the above copyright .\" notice, this list of conditions and the following disclaimer. .\" 2. Redistributions in binary form must reproduce the above copyright .\" notice, this list of conditions and the following disclaimer in the .\" documentation and/or other materials provided with the distribution. -.\" 3. The name of the author may not be used to endorse or promote products -.\" derived from this software without specific prior written permission. .\" -.\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR -.\" IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED -.\" WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE -.\" DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, -.\" INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES -.\" (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR -.\" SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) -.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, -.\" STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN -.\" ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE +.\" THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS +.\" ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED +.\" TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +.\" PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS +.\" BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR +.\" CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF +.\" SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS +.\" INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN +.\" CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) +.\" ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE .\" POSSIBILITY OF SUCH DAMAGE. .\" +.\" +.\" +.\" Copyright (c) 2004 +.\" Jonathan Stone . All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions +.\" are met: +.\" 1. Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" 2. Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" +.\" THIS SOFTWARE IS PROVIDED BY Jonathan Stone AND CONTRIBUTORS ``AS IS'' AND +.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +.\" ARE DISCLAIMED. IN NO EVENT SHALL Jonathan Stone OR THE VOICES IN HIS HEAD +.\" BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR +.\" CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF +.\" SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS +.\" INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN +.\" CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) +.\" ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF +.\" THE POSSIBILITY OF SUCH DAMAGE. +.\" .\" $FreeBSD$ .\" -.Dd September 7, 2010 +.Dd December 12, 2014 .Dt CRYPTO 4 .Os .Sh NAME .Nm crypto , .Nm cryptodev -.Nd hardware crypto access driver +.Nd user-mode access to hardware-accelerated cryptography .Sh SYNOPSIS .Cd device crypto .Cd device cryptodev +.Pp +.In sys/ioctl.h +.In sys/time.h +.In crypto/cryptodev.h .Sh DESCRIPTION The .Nm -driver provides a device-independent framework to support -cryptographic operations in the kernel. +driver gives user-mode applications access to hardware-accelerated +cryptographic transforms, as implemented by the +.Xr opencrypto 9 +in-kernel interface. +.Pp The -.Nm cryptodev -driver provides userland applications access to this support -through the .Pa /dev/crypto -device. -This node primarily operates in an +special device provides an .Xr ioctl 2 -based model, permitting a variety of applications to query device capabilities, -submit transactions, and get results. +based interface. +User-mode applications should open the special device, +then issue +.Xr ioctl 2 +calls on the descriptor. +User-mode access to +.Pa /dev/crypto +is controlled by three +.Xr sysctl 8 +variables, +.Ic kern.userasymcrypto +and +.Ic kern.cryptodevallowsoft . +See +.Xr sysctl 7 +for additional details. .Pp -If -.Ar count -given in the specification, and is greater than 0, a maximum of one +The .Nm -device is created. +device provides two distinct modes of operation: one mode for +symmetric-keyed cryptographic requests, and a second mode for +both asymmetric-key (public-key/private-key) requests, and for +modular arithmetic (for Diffie-Hellman key exchange and other +cryptographic protocols). +The two modes are described separately below. +.Sh THEORY OF OPERATION +Regardless of whether symmetric-key or asymmetric-key operations are +to be performed, use of the device requires a basic series of steps: .Pp -The following +.Bl -enum +.It +Open a file descriptor for the device. +See +.Xr open 2 . +.It +If any symmetric operation will be performed, +create one session, with +.Dv CIOCGSESSION . +Most applications will require at least one symmetric session. +Since cipher and MAC keys are tied to sessions, many +applications will require more. +Asymmetric operations do not use sessions. +.It +Submit requests, synchronously with +.Dv CIOCCRYPT +(symmetric) +or +.Dv CIOCKEY +(asymmetric). +.It +Destroy one session with +.Dv CIOCFSESSION . +.It +Close the device with +.Xr close 2 . +.El +.Sh SYMMETRIC-KEY OPERATION +The symmetric-key operation mode provides a context-based API +to traditional symmetric-key encryption (or privacy) algorithms, +or to keyed and unkeyed one-way hash (HMAC and MAC) algorithms. +The symmetric-key mode also permits fused operation, +where the hardware performs both a privacy algorithm and an integrity-check +algorithm in a single pass over the data: either a fused +encrypt/HMAC-generate operation, or a fused HMAC-verify/decrypt operation. +.Pp +To use symmetric mode, you must first create a session specifying +the algorithm(s) and key(s) to use; then issue encrypt or decrypt +requests against the session. +.Ss Algorithms +For a list of supported algorithms, see +.Xr crypto 7 +and +.Xr crypto 9 . +.Ss IOCTL Request Descriptions +.\" +.Bl -tag -width CIOCGSESSION +.\" +.It Dv CRIOGET Fa int *fd +Clone the fd argument to +.Xr ioctl 2 , +yielding a new file descriptor for the creation of sessions. +.\" +.It Dv CIOCFINDDEV Fa struct crypt_find_op *fop +.Bd -literal +struct crypt_find_op { + int crid; /* driver id + flags */ + char name[32]; /* device/driver name */ +}; + +.Ed +If +.Fa crid +is -1, then find the driver named +.Fa name +and return the id in +.Fa crid . +If +.Fa crid +is not -1, return the name of the driver with +.Fa crid +in +.Fa name . +In either case, if the driver is not found, +.Dv ENOENT +is returned. +.It Dv CIOCGSESSION Fa struct session_op *sessp +.Bd -literal +struct session_op { + u_int32_t cipher; /* e.g. CRYPTO_DES_CBC */ + u_int32_t mac; /* e.g. CRYPTO_MD5_HMAC */ + + u_int32_t keylen; /* cipher key */ + void * key; + int mackeylen; /* mac key */ + void * mackey; + + u_int32_t ses; /* returns: ses # */ +}; + +.Ed +Create a new cryptographic session on a file descriptor for the device; +that is, a persistent object specific to the chosen +privacy algorithm, integrity algorithm, and keys specified in +.Fa sessp . +The special value 0 for either privacy or integrity +is reserved to indicate that the indicated operation (privacy or integrity) +is not desired for this session. +.Pp +Multiple sessions may be bound to a single file descriptor. +The session ID returned in +.Fa sessp-\*[Gt]ses +is supplied as a required field in the symmetric-operation structure +.Fa crypt_op +for future encryption or hashing requests. +.\" .Pp +.\" This implementation will never return a session ID of 0 for a successful +.\" creation of a session, which is a +.\" .Nx +.\" extension. +.Pp +For non-zero symmetric-key privacy algorithms, the privacy algorithm +must be specified in +.Fa sessp-\*[Gt]cipher , +the key length in +.Fa sessp-\*[Gt]keylen , +and the key value in the octets addressed by +.Fa sessp-\*[Gt]key . +.Pp +For keyed one-way hash algorithms, the one-way hash must be specified +in +.Fa sessp-\*[Gt]mac , +the key length in +.Fa sessp-\*[Gt]mackey , +and the key value in the octets addressed by +.Fa sessp-\*[Gt]mackeylen . +.\" +.Pp +Support for a specific combination of fused privacy and +integrity-check algorithms depends on whether the underlying +hardware supports that combination. +Not all combinations are supported +by all hardware, even if the hardware supports each operation as a +stand-alone non-fused operation. +.It Dv CIOCCRYPT Fa struct crypt_op *cr_op +.Bd -literal +struct crypt_op { + u_int32_t ses; + u_int16_t op; /* e.g. COP_ENCRYPT */ + u_int16_t flags; + u_int len; + caddr_t src, dst; + caddr_t mac; /* must be large enough for result */ + caddr_t iv; +}; + +.Ed +Request a symmetric-key (or hash) operation. +The file descriptor argument to .Xr ioctl 2 -calls apply only to the -.Nm -devices: -.Bl -tag -width ".Dv CIOCGSESSION" -.It Dv CIOCGSESSION -Setup a new crypto session for a new type of operation. -.It Dv CIOCFSESSION -Free a previously established session. -.It Dv CIOCCRYPT -Perform a crypto operation against a previously setup session. +must have been bound to a valid session. +To encrypt, set +.Fa cr_op-\*[Gt]op +to +.Dv COP_ENCRYPT . +To decrypt, set +.Fa cr_op-\*[Gt]op +to +.Dv COP_DECRYPT . +The field +.Fa cr_op-\*[Gt]len +supplies the length of the input buffer; the fields +.Fa cr_op-\*[Gt]src , +.Fa cr_op-\*[Gt]dst , +.Fa cr_op-\*[Gt]mac , +.Fa cr_op-\*[Gt]iv +supply the addresses of the input buffer, output buffer, +one-way hash, and initialization vector, respectively. +.It Dv CIOCCRYPTAEAD Fa struct crypt_aead *cr_aead +.Bd -literal +struct crypt_aead { + u_int32_t ses; + u_int16_t op; /* e.g. COP_ENCRYPT */ + u_int16_t flags; + u_int len; + u_int aadlen; + u_int ivlen; + caddr_t src, dst; + caddr_t aad; + caddr_t tag; /* must be large enough for result */ + caddr_t iv; +}; + +.Ed +The +.Dv CIOCCRYPTAEAD +is similar to the +.Dv CIOCCRYPT +but provides additional data in +.Fa cr_aead-\*[Gt]aad +to include in the authentication mode. +.It Dv CIOCFSESSION Fa u_int32_t ses_id +Destroys the /dev/crypto session associated with the file-descriptor +argument. +.It Dv CIOCNFSESSION Fa struct crypt_sfop *sfop ; +.Bd -literal +struct crypt_sfop { + size_t count; + u_int32_t *sesid; +}; + +.Ed +Destroys the +.Fa sfop-\*[Gt]count +sessions specified by the +.Fa sfop +array of session identifiers. .El -.Sh FEATURES -Depending on hardware being present, the following symmetric and -asymmetric cryptographic features are potentially available from -.Pa /dev/crypto : +.\" +.Sh ASYMMETRIC-KEY OPERATION +.Ss Asymmetric-key algorithms +Contingent upon hardware support, the following asymmetric +(public-key/private-key; or key-exchange subroutine) operations may +also be available: .Pp -.Bl -tag -width ".Dv CRYPTO_RIPEMD160_HMAC" -offset indent -compact -.It Dv CRYPTO_DES_CBC -.It Dv CRYPTO_3DES_CBC -.It Dv CRYPTO_BLF_CBC -.It Dv CRYPTO_CAMELLIA_CBC -.It Dv CRYPTO_CAST_CBC -.It Dv CRYPTO_SKIPJACK_CBC -.It Dv CRYPTO_MD5_HMAC -.It Dv CRYPTO_SHA1_HMAC -.It Dv CRYPTO_RIPEMD160_HMAC -.It Dv CRYPTO_MD5_KPDK -.It Dv CRYPTO_SHA1_KPDK -.It Dv CRYPTO_AES_CBC -.It Dv CRYPTO_ARC4 -.It Dv CRYPTO_MD5 -.It Dv CRYPTO_SHA1 -.It Dv CRK_MOD_EXP -.It Dv CRK_MOD_EXP_CRT -.It Dv CRK_DSA_SIGN -.It Dv CRK_DSA_VERIFY -.It Dv CRK_DH_COMPUTE_KEY +.Bl -column "CRK_DH_COMPUTE_KEY" "Input parameter" "Output parameter" -offset indent -compact +.It Em "Algorithm" Ta "Input parameter" Ta "Output parameter" +.It Em " " Ta "Count" Ta "Count" +.It Dv CRK_MOD_EXP Ta 3 Ta 1 +.It Dv CRK_MOD_EXP_CRT Ta 6 Ta 1 +.It Dv CRK_DSA_SIGN Ta 5 Ta 2 +.It Dv CRK_DSA_VERIFY Ta 7 Ta 0 +.It Dv CRK_DH_COMPUTE_KEY Ta 3 Ta 1 .El -.Sh FILES -.Bl -tag -width ".Pa /dev/crypto" -compact -.It Pa /dev/crypto -crypto access device +.Pp +See below for discussion of the input and output parameter counts. +.Ss Asymmetric-key commands +.Bl -tag -width CIOCKEY +.It Dv CIOCASYMFEAT Fa int *feature_mask +Returns a bitmask of supported asymmetric-key operations. +Each of the above-listed asymmetric operations is present +if and only if the bit position numbered by the code for that operation +is set. +For example, +.Dv CRK_MOD_EXP +is available if and only if the bit +.Pq 1 \*[Lt]\*[Lt] Dv CRK_MOD_EXP +is set. +.It Dv CIOCKEY Fa struct crypt_kop *kop +.Bd -literal +struct crypt_kop { + u_int crk_op; /* e.g. CRK_MOD_EXP */ + u_int crk_status; /* return status */ + u_short crk_iparams; /* # of input params */ + u_short crk_oparams; /* # of output params */ + u_int crk_pad1; + struct crparam crk_param[CRK_MAXPARAM]; +}; + +/* Bignum parameter, in packed bytes. */ +struct crparam { + void * crp_p; + u_int crp_nbits; +}; + +.Ed +Performs an asymmetric-key operation from the list above. +The specific operation is supplied in +.Fa kop-\*[Gt]crk_op ; +final status for the operation is returned in +.Fa kop-\*[Gt]crk_status . +The number of input arguments and the number of output arguments +is specified in +.Fa kop-\*[Gt]crk_iparams +and +.Fa kop-\*[Gt]crk_iparams , +respectively. +The field +.Fa crk_param[] +must be filled in with exactly +.Fa kop-\*[Gt]crk_iparams + kop-\*[Gt]crk_oparams +arguments, each encoded as a +.Fa struct crparam +(address, bitlength) pair. +.Pp +The semantics of these arguments are currently undocumented. .El .Sh SEE ALSO .Xr aesni 4 , .Xr hifn 4 , .Xr ipsec 4 , .Xr padlock 4 , .Xr safe 4 , .Xr ubsec 4 , +.Xr crypto 7 , .Xr geli 8 , .Xr crypto 9 .Sh HISTORY The .Nm driver first appeared in .Ox 3.0 . The .Nm driver was imported to .Fx 5.0 . +.Sh BUGS +Error checking and reporting is weak. +.Pp +The values specified for symmetric-key key sizes to +.Dv CIOCGSESSION +must exactly match the values expected by +.Xr opencrypto 9 . +The output buffer and MAC buffers supplied to +.Dv CIOCCRYPT +must follow whether privacy or integrity algorithms were specified for +session: if you request a +.No non- Ns Dv NULL +algorithm, you must supply a suitably-sized buffer. +.Pp +The scheme for passing arguments for asymmetric requests is baroque. +.Pp +The naming inconsistency between +.Dv CRIOGET +and the various +.Dv CIOC Ns \&* +names is an unfortunate historical artifact. Index: projects/clang350-import/share/man/man4 =================================================================== --- projects/clang350-import/share/man/man4 (revision 275748) +++ projects/clang350-import/share/man/man4 (revision 275749) Property changes on: projects/clang350-import/share/man/man4 ___________________________________________________________________ Modified: svn:mergeinfo ## -0,0 +0,1 ## Merged /head/share/man/man4:r275623-275748 Index: projects/clang350-import/share/man/man7/Makefile =================================================================== --- projects/clang350-import/share/man/man7/Makefile (revision 275748) +++ projects/clang350-import/share/man/man7/Makefile (revision 275749) @@ -1,52 +1,53 @@ # @(#)Makefile 8.1 (Berkeley) 6/5/93 # $FreeBSD$ .include #MISSING: eqnchar.7 ms.7 term.7 MAN= adding_user.7 \ ascii.7 \ bsd.snmpmod.mk.7 \ build.7 \ clocks.7 \ + crypto.7 \ c99.7 \ development.7 \ environ.7 \ ffs.7 \ firewall.7 \ growfs.7 \ hier.7 \ hostname.7 \ intro.7 \ maclabel.7 \ mailaddr.7 \ operator.7 \ ports.7 \ release.7 \ sdoc.7 \ security.7 \ sprog.7 \ stdint.7 \ sticky.7 \ tests.7 \ tuning.7 MLINKS= intro.7 miscellaneous.7 MLINKS+= security.7 securelevel.7 MLINKS+= c99.7 c.7 MLINKS+= c99.7 c78.7 MLINKS+= c99.7 c89.7 MLINKS+= c99.7 c90.7 .if ${MK_TESTS} != "no" ATF= ${.CURDIR}/../../../contrib/atf .PATH: ${ATF}/doc MAN+= atf.7 CLEANFILES+= atf.7 atf.7: atf.7.in sed -e 's,__DOCDIR__,/usr/share/doc/atf,g' \ <"${ATF}/doc/atf.7.in" >atf.7 .endif .include Index: projects/clang350-import/share/man/man7/crypto.7 =================================================================== --- projects/clang350-import/share/man/man7/crypto.7 (nonexistent) +++ projects/clang350-import/share/man/man7/crypto.7 (revision 275749) @@ -0,0 +1,141 @@ +.\" Copyright (c) 2014 The FreeBSD Foundation +.\" All rights reserved. +.\" +.\" This documentation was written by John-Mark Gurney under +.\" the sponsorship of the FreeBSD Foundation and +.\" Rubicon Communications, LLC (Netgate). +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions +.\" are met: +.\" 1. Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" 2. Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND +.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +.\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE +.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS +.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) +.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT +.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY +.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF +.\" SUCH DAMAGE. +.\" +.\" $FreeBSD$ +.\" +.Dd December 12, 2014 +.Dt CRYPTO 7 +.Os +.Sh NAME +.Nm crypto +.Nd OpenCrypto algorithms +.Sh SYNOPSIS +In the kernel configuration file: +.Cd "device crypto" +.Pp +Or load the crypto.ko module. +.Sh DESCRIPTION +The following cryptographic algorithms that are part of the OpenCrypto +framework have the following requirements. +.Pp +Cipher algorithms: +.Bl -tag -width ".Dv CRYPTO_AES_CBC" +.It Dv CRYPTO_AES_CBC +.Bl -tag -width "Block size :" -compact -offset indent +.It IV size : +16 +.It Block size : +16 +.It Key size : +16, 24 or 32 +.El +.Pp +This algorithm implements Cipher-block chaining. +.It Dv CRYPTO_AES_NIST_GCM_16 +.Bl -tag -width "Block size :" -compact -offset indent +.It IV size : +12 +.It Block size : +1 +.It Key size : +16, 24 or 32 +.It Digest size : +16 +.El +.Pp +This algorithm implements Galois/Counter Mode. +This is the cipher part of an AEAD +.Pq Authenticated Encryption with Associated Data +mode. +This requires use of the use of a proper authentication mode, one of +.Dv CRYPTO_AES_128_NIST_GMAC , +.Dv CRYPTO_AES_192_NIST_GMAC +or +.Dv CRYPTO_AES_256_NIST_GMAC , +that corresponds with the number of bits in the key that you are using. +.Pp +The associated data (if any) must be provided by the authentication mode op. +The authentication tag will be read/written from/to the offset crd_inject +specified in the descriptor for the authentication mode. +.Pp +Note: You must provide an IV on every call. +.It Dv CRYPTO_AES_ICM +.Bl -tag -width "Block size :" -compact -offset indent +.It IV size : +16 +.It Block size : +1 (aesni), 16 (software) +.It Key size : +16, 24 or 32 +.El +.Pp +This algorithm implements Integer Counter Mode. +This is similar to what most people call counter mode, but instead of the +counter being split into a nonce and a counter part, then entire nonce is +used as the initial counter. +This does mean that if a counter is required that rolls over at 32 bits, +the transaction need to be split into two parts where the counter rolls over. +The counter incremented as a 128-bit big endian number. +.Pp +Note: You must provide an IV on every call. +.It Dv CRYPTO_AES_XTS +.Bl -tag -width "Block size :" -compact -offset indent +.It IV size : +16 +.It Block size : +16 +.It Key size : +32 or 64 +.El +.Pp +This algorithm implements XEX Tweakable Block Cipher with Ciphertext Stealing +as defined in NIST SP 800-38E. +.Pp +NOTE: The ciphertext stealing part is not implemented which is why this cipher +is listed as having a block size of 16 instead of 1. +.El +.Pp +Authentication algorithms: +.Bl -tag -width ".Dv CRYPTO_AES_256_NIST_GMAC" +.It CRYPTO_AES_128_NIST_GMAC +See +.Dv CRYPTO_AES_NIST_GCM_16 +in the cipher mode section. +.It CRYPTO_AES_192_NIST_GMAC +See +.Dv CRYPTO_AES_NIST_GCM_16 +in the cipher mode section. +.It CRYPTO_AES_256_NIST_GMAC +See +.Dv CRYPTO_AES_NIST_GCM_16 +in the cipher mode section. +.El +.Sh SEE ALSO +.Xr crypto 4 , +.Xr crypto 9 +.Sh BUGS +Not all the implemented algorithms are listed. Property changes on: projects/clang350-import/share/man/man7/crypto.7 ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: projects/clang350-import/share/man/man9/crypto.9 =================================================================== --- projects/clang350-import/share/man/man9/crypto.9 (revision 275748) +++ projects/clang350-import/share/man/man9/crypto.9 (revision 275749) @@ -1,666 +1,676 @@ .\" $OpenBSD: crypto.9,v 1.19 2002/07/16 06:31:57 angelos Exp $ .\" .\" The author of this manual page is Angelos D. Keromytis (angelos@cis.upenn.edu) .\" .\" Copyright (c) 2000, 2001 Angelos D. Keromytis .\" .\" Permission to use, copy, and modify this software with or without fee .\" is hereby granted, provided that this entire notice is included in .\" all source code copies of any software which is or includes a copy or .\" modification of this software. .\" .\" THIS SOFTWARE IS BEING PROVIDED "AS IS", WITHOUT ANY EXPRESS OR .\" IMPLIED WARRANTY. IN PARTICULAR, NONE OF THE AUTHORS MAKES ANY .\" REPRESENTATION OR WARRANTY OF ANY KIND CONCERNING THE .\" MERCHANTABILITY OF THIS SOFTWARE OR ITS FITNESS FOR ANY PARTICULAR .\" PURPOSE. .\" .\" $FreeBSD$ .\" -.Dd April 18, 2014 +.Dd December 12, 2014 .Dt CRYPTO 9 .Os .Sh NAME .Nm crypto .Nd API for cryptographic services in the kernel .Sh SYNOPSIS .In opencrypto/cryptodev.h .Ft int32_t .Fn crypto_get_driverid uint8_t .Ft int .Fn crypto_register uint32_t int uint16_t uint32_t "int \*[lp]*\*[rp]\*[lp]void *, uint32_t *, struct cryptoini *\*[rp]" "int \*[lp]*\*[rp]\*[lp]void *, uint64_t\*[rp]" "int \*[lp]*\*[rp]\*[lp]void *, struct cryptop *\*[rp]" "void *" .Ft int .Fn crypto_kregister uint32_t int uint32_t "int \*[lp]*\*[rp]\*[lp]void *, struct cryptkop *\*[rp]" "void *" .Ft int .Fn crypto_unregister uint32_t int .Ft int .Fn crypto_unregister_all uint32_t .Ft void .Fn crypto_done "struct cryptop *" .Ft void .Fn crypto_kdone "struct cryptkop *" .Ft int .Fn crypto_newsession "uint64_t *" "struct cryptoini *" int .Ft int .Fn crypto_freesession uint64_t .Ft int .Fn crypto_dispatch "struct cryptop *" .Ft int .Fn crypto_kdispatch "struct cryptkop *" .Ft int .Fn crypto_unblock uint32_t int .Ft "struct cryptop *" .Fn crypto_getreq int .Ft void .Fn crypto_freereq void .Bd -literal #define CRYPTO_SYMQ 0x1 #define CRYPTO_ASYMQ 0x2 #define EALG_MAX_BLOCK_LEN 16 struct cryptoini { int cri_alg; int cri_klen; int cri_mlen; caddr_t cri_key; uint8_t cri_iv[EALG_MAX_BLOCK_LEN]; struct cryptoini *cri_next; }; struct cryptodesc { int crd_skip; int crd_len; int crd_inject; int crd_flags; struct cryptoini CRD_INI; #define crd_iv CRD_INI.cri_iv #define crd_key CRD_INI.cri_key #define crd_alg CRD_INI.cri_alg #define crd_klen CRD_INI.cri_klen struct cryptodesc *crd_next; }; struct cryptop { TAILQ_ENTRY(cryptop) crp_next; uint64_t crp_sid; int crp_ilen; int crp_olen; int crp_etype; int crp_flags; caddr_t crp_buf; caddr_t crp_opaque; struct cryptodesc *crp_desc; int (*crp_callback) (struct cryptop *); caddr_t crp_mac; }; struct crparam { caddr_t crp_p; u_int crp_nbits; }; #define CRK_MAXPARAM 8 struct cryptkop { TAILQ_ENTRY(cryptkop) krp_next; u_int krp_op; /* ie. CRK_MOD_EXP or other */ u_int krp_status; /* return status */ u_short krp_iparams; /* # of input parameters */ u_short krp_oparams; /* # of output parameters */ uint32_t krp_hid; struct crparam krp_param[CRK_MAXPARAM]; int (*krp_callback)(struct cryptkop *); }; .Ed .Sh DESCRIPTION .Nm is a framework for drivers of cryptographic hardware to register with the kernel so .Dq consumers (other kernel subsystems, and users through the .Pa /dev/crypto device) are able to make use of it. Drivers register with the framework the algorithms they support, and provide entry points (functions) the framework may call to establish, use, and tear down sessions. Sessions are used to cache cryptographic information in a particular driver (or associated hardware), so initialization is not needed with every request. Consumers of cryptographic services pass a set of descriptors that instruct the framework (and the drivers registered with it) of the operations that should be applied on the data (more than one cryptographic operation can be requested). .Pp Keying operations are supported as well. Unlike the symmetric operators described above, these sessionless commands perform mathematical operations using input and output parameters. .Pp Since the consumers may not be associated with a process, drivers may not .Xr sleep 9 . The same holds for the framework. Thus, a callback mechanism is used to notify a consumer that a request has been completed (the callback is specified by the consumer on a per-request basis). The callback is invoked by the framework whether the request was successfully completed or not. An error indication is provided in the latter case. A specific error code, .Er EAGAIN , is used to indicate that a session number has changed and that the request may be re-submitted immediately with the new session number. Errors are only returned to the invoking function if not enough information to call the callback is available (meaning, there was a fatal error in verifying the arguments). For session initialization and teardown there is no callback mechanism used. .Pp The .Fn crypto_newsession routine is called by consumers of cryptographic services (such as the .Xr ipsec 4 stack) that wish to establish a new session with the framework. On success, the first argument will contain the Session Identifier (SID). The second argument contains all the necessary information for the driver to establish the session. The third argument indicates whether a hardware driver (1) should be used or not (0). The various fields in the .Vt cryptoini structure are: .Bl -tag -width ".Va cri_next" .It Va cri_alg Contains an algorithm identifier. Currently supported algorithms are: .Pp .Bl -tag -width ".Dv CRYPTO_RIPEMD160_HMAC" -compact +.It Dv CRYPTO_AES_128_NIST_GMAC +.It Dv CRYPTO_AES_192_NIST_GMAC +.It Dv CRYPTO_AES_256_NIST_GMAC .It Dv CRYPTO_AES_CBC +.It Dv CRYPTO_AES_ICM +.It Dv CRYPTO_AES_NIST_GCM_16 +.It Dv CRYPTO_AES_NIST_GMAC +.It Dv CRYPTO_AES_XTS .It Dv CRYPTO_ARC4 .It Dv CRYPTO_BLF_CBC .It Dv CRYPTO_CAMELLIA_CBC .It Dv CRYPTO_CAST_CBC +.It Dv CRYPTO_DEFLATE_COMP .It Dv CRYPTO_DES_CBC .It Dv CRYPTO_3DES_CBC -.It Dv CRYPTO_SKIPJACK_CBC .It Dv CRYPTO_MD5 .It Dv CRYPTO_MD5_HMAC .It Dv CRYPTO_MD5_KPDK +.It Dv CRYPTO_NULL_HMAC +.It Dv CRYPTO_NULL_CBC .It Dv CRYPTO_RIPEMD160_HMAC .It Dv CRYPTO_SHA1 .It Dv CRYPTO_SHA1_HMAC .It Dv CRYPTO_SHA1_KPDK .It Dv CRYPTO_SHA2_256_HMAC .It Dv CRYPTO_SHA2_384_HMAC .It Dv CRYPTO_SHA2_512_HMAC -.It Dv CRYPTO_NULL_HMAC -.It Dv CRYPTO_NULL_CBC +.It Dv CRYPTO_SKIPJACK_CBC .El .It Va cri_klen Specifies the length of the key in bits, for variable-size key algorithms. .It Va cri_mlen Specifies how many bytes from the calculated hash should be copied back. 0 means entire hash. .It Va cri_key Contains the key to be used with the algorithm. .It Va cri_iv Contains an explicit initialization vector (IV), if it does not prefix the data. -This field is ignored during initialization. +This field is ignored during initialization +.Pq Nm crypto_newsession . If no IV is explicitly passed (see below on details), a random IV is used by the device driver processing the request. .It Va cri_next Contains a pointer to another .Vt cryptoini structure. Multiple such structures may be linked to establish multi-algorithm sessions .Xr ( ipsec 4 is an example consumer of such a feature). .El .Pp The .Vt cryptoini structure and its contents will not be modified by the framework (or the drivers used). Subsequent requests for processing that use the SID returned will avoid the cost of re-initializing the hardware (in essence, SID acts as an index in the session cache of the driver). .Pp .Fn crypto_freesession is called with the SID returned by .Fn crypto_newsession to disestablish the session. .Pp .Fn crypto_dispatch is called to process a request. The various fields in the .Vt cryptop structure are: .Bl -tag -width ".Va crp_callback" .It Va crp_sid Contains the SID. .It Va crp_ilen Indicates the total length in bytes of the buffer to be processed. .It Va crp_olen On return, contains the total length of the result. For symmetric crypto operations, this will be the same as the input length. This will be used if the framework needs to allocate a new buffer for the result (or for re-formatting the input). .It Va crp_callback This routine is invoked upon completion of the request, whether successful or not. It is invoked through the .Fn crypto_done routine. If the request was not successful, an error code is set in the .Va crp_etype field. It is the responsibility of the callback routine to set the appropriate .Xr spl 9 level. .It Va crp_etype Contains the error type, if any errors were encountered, or zero if the request was successfully processed. If the .Er EAGAIN error code is returned, the SID has changed (and has been recorded in the .Va crp_sid field). The consumer should record the new SID and use it in all subsequent requests. In this case, the request may be re-submitted immediately. This mechanism is used by the framework to perform session migration (move a session from one driver to another, because of availability, performance, or other considerations). .Pp Note that this field only makes sense when examined by the callback routine specified in .Va crp_callback . Errors are returned to the invoker of .Fn crypto_process only when enough information is not present to call the callback routine (i.e., if the pointer passed is .Dv NULL or if no callback routine was specified). .It Va crp_flags Is a bitmask of flags associated with this request. Currently defined flags are: .Bl -tag -width ".Dv CRYPTO_F_CBIFSYNC" .It Dv CRYPTO_F_IMBUF The buffer pointed to by .Va crp_buf is an mbuf chain. .It Dv CRYPTO_F_IOV The buffer pointed to by .Va crp_buf is an .Vt uio structure. -.It Dv CRYPTO_F_REL -Must return data in the same place. .It Dv CRYPTO_F_BATCH Batch operation if possible. .It Dv CRYPTO_F_CBIMM Do callback immediately instead of doing it from a dedicated kernel thread. .It Dv CRYPTO_F_DONE Operation completed. .It Dv CRYPTO_F_CBIFSYNC Do callback immediately if operation is synchronous. .El .It Va crp_buf Points to the input buffer. On return (when the callback is invoked), it contains the result of the request. The input buffer may be an mbuf chain or a contiguous buffer, depending on .Va crp_flags . .It Va crp_opaque This is passed through the crypto framework untouched and is intended for the invoking application's use. .It Va crp_desc This is a linked list of descriptors. Each descriptor provides information about what type of cryptographic operation should be done on the input buffer. The various fields are: .Bl -tag -width ".Va crd_inject" .It Va crd_iv The field where IV should be provided when the .Dv CRD_F_IV_EXPLICIT flag is given. .It Va crd_key When the .Dv CRD_F_KEY_EXPLICIT flag is given, the .Va crd_key points to a buffer with encryption or authentication key. .It Va crd_alg An algorithm to use. Must be the same as the one given at newsession time. .It Va crd_klen The .Va crd_key key length. .It Va crd_skip The offset in the input buffer where processing should start. .It Va crd_len How many bytes, after .Va crd_skip , should be processed. .It Va crd_inject Offset from the beginning of the buffer to insert any results. For encryption algorithms, this is where the initialization vector (IV) will be inserted when encrypting or where it can be found when decrypting (subject to .Va crd_flags ) . For MAC algorithms, this is where the result of the keyed hash will be inserted. .It Va crd_flags The following flags are defined: .Bl -tag -width 3n .It Dv CRD_F_ENCRYPT For encryption algorithms, this bit is set when encryption is required (when not set, decryption is performed). .It Dv CRD_F_IV_PRESENT -For encryption algorithms, this bit is set when the IV already +For encryption, this bit is set when the IV already precedes the data, so the .Va crd_inject value will be ignored and no IV will be written in the buffer. Otherwise, the IV used to encrypt the packet will be written at the location pointed to by .Va crd_inject . The IV length is assumed to be equal to the blocksize of the encryption algorithm. -Some applications that do special +Applications that do special .Dq "IV cooking" , such as the half-IV mode in .Xr ipsec 4 , can use this flag to indicate that the IV should not be written on the packet. This flag is typically used in conjunction with the .Dv CRD_F_IV_EXPLICIT flag. .It Dv CRD_F_IV_EXPLICIT For encryption algorithms, this bit is set when the IV is explicitly provided by the consumer in the .Va crd_iv field. Otherwise, for encryption operations the IV is provided for by the driver used to perform the operation, whereas for decryption operations it is pointed to by the .Va crd_inject field. This flag is typically used when the IV is calculated .Dq "on the fly" by the consumer, and does not precede the data (some .Xr ipsec 4 configurations, and the encrypted swap are two such examples). .It Dv CRD_F_KEY_EXPLICIT For encryption and authentication (MAC) algorithms, this bit is set when the key is explicitly provided by the consumer in the .Va crd_key field for the given operation. Otherwise, the key is taken at newsession time from the .Va cri_key field. +As calculating the key schedule may take a while, it is recommended that often +used keys are given their own session. .It Dv CRD_F_COMP For compression algorithms, this bit is set when compression is required (when not set, decompression is performed). .El .It Va CRD_INI This .Vt cryptoini structure will not be modified by the framework or the device drivers. Since this information accompanies every cryptographic operation request, drivers may re-initialize state on-demand (typically an expensive operation). Furthermore, the cryptographic framework may re-route requests as a result of full queues or hardware failure, as described above. .It Va crd_next Point to the next descriptor. Linked operations are useful in protocols such as .Xr ipsec 4 , where multiple cryptographic transforms may be applied on the same block of data. .El .El .Pp .Fn crypto_getreq allocates a .Vt cryptop structure with a linked list of as many .Vt cryptodesc structures as were specified in the argument passed to it. .Pp .Fn crypto_freereq deallocates a structure .Vt cryptop and any .Vt cryptodesc structures linked to it. Note that it is the responsibility of the callback routine to do the necessary cleanups associated with the opaque field in the .Vt cryptop structure. .Pp .Fn crypto_kdispatch is called to perform a keying operation. The various fields in the .Vt cryptkop structure are: .Bl -tag -width ".Va krp_callback" .It Va krp_op Operation code, such as .Dv CRK_MOD_EXP . .It Va krp_status Return code. This .Va errno Ns -style variable indicates whether lower level reasons for operation failure. .It Va krp_iparams Number if input parameters to the specified operation. Note that each operation has a (typically hardwired) number of such parameters. .It Va krp_oparams Number if output parameters from the specified operation. Note that each operation has a (typically hardwired) number of such parameters. .It Va krp_kvp An array of kernel memory blocks containing the parameters. .It Va krp_hid Identifier specifying which low-level driver is being used. .It Va krp_callback Callback called on completion of a keying operation. .El .Sh DRIVER-SIDE API The .Fn crypto_get_driverid , .Fn crypto_register , .Fn crypto_kregister , .Fn crypto_unregister , .Fn crypto_unblock , and .Fn crypto_done routines are used by drivers that provide support for cryptographic primitives to register and unregister with the kernel crypto services framework. Drivers must first use the .Fn crypto_get_driverid function to acquire a driver identifier, specifying the .Fa cc_flags as an argument (normally 0, but software-only drivers should specify .Dv CRYPTOCAP_F_SOFTWARE ) . For each algorithm the driver supports, it must then call .Fn crypto_register . The first two arguments are the driver and algorithm identifiers. The next two arguments specify the largest possible operator length (in bits, important for public key operations) and flags for this algorithm. The last four arguments must be provided in the first call to .Fn crypto_register and are ignored in all subsequent calls. They are pointers to three driver-provided functions that the framework may call to establish new cryptographic context with the driver, free already established context, and ask for a request to be processed (encrypt, decrypt, etc.); and an opaque parameter to pass when calling each of these routines. .Fn crypto_unregister is called by drivers that wish to withdraw support for an algorithm. The two arguments are the driver and algorithm identifiers, respectively. Typically, drivers for PCMCIA crypto cards that are being ejected will invoke this routine for all algorithms supported by the card. .Fn crypto_unregister_all will unregister all algorithms registered by a driver and the driver will be disabled (no new sessions will be allocated on that driver, and any existing sessions will be migrated to other drivers). The same will be done if all algorithms associated with a driver are unregistered one by one. .Pp The calling convention for the three driver-supplied routines is: .Pp .Bl -item -compact .It .Ft int .Fn \*[lp]*newsession\*[rp] "void *" "uint32_t *" "struct cryptoini *" ; .It .Ft int .Fn \*[lp]*freesession\*[rp] "void *" "uint64_t" ; .It .Ft int .Fn \*[lp]*process\*[rp] "void *" "struct cryptop *" ; .It .Ft int .Fn \*[lp]*kprocess\*[rp] "void *" "struct cryptkop *" ; .El .Pp On invocation, the first argument to all routines is an opaque data value supplied when the algorithm is registered with .Fn crypto_register . The second argument to .Fn newsession contains the driver identifier obtained via .Fn crypto_get_driverid . On successful return, it should contain a driver-specific session identifier. The third argument is identical to that of .Fn crypto_newsession . .Pp The .Fn freesession routine takes as arguments the opaque data value and the SID (which is the concatenation of the driver identifier and the driver-specific session identifier). It should clear any context associated with the session (clear hardware registers, memory, etc.). .Pp The .Fn process routine is invoked with a request to perform crypto processing. This routine must not block, but should queue the request and return immediately. Upon processing the request, the callback routine should be invoked. In case of an unrecoverable error, the error indication must be placed in the .Va crp_etype field of the .Vt cryptop structure. When the request is completed, or an error is detected, the .Fn process routine should invoke .Fn crypto_done . Session migration may be performed, as mentioned previously. .Pp In case of a temporary resource exhaustion, the .Fn process routine may return .Er ERESTART in which case the crypto services will requeue the request, mark the driver as .Dq blocked , and stop submitting requests for processing. The driver is then responsible for notifying the crypto services when it is again able to process requests through the .Fn crypto_unblock routine. This simple flow control mechanism should only be used for short-lived resource exhaustion as it causes operations to be queued in the crypto layer. Doing so is preferable to returning an error in such cases as it can cause network protocols to degrade performance by treating the failure much like a lost packet. .Pp The .Fn kprocess routine is invoked with a request to perform crypto key processing. This routine must not block, but should queue the request and return immediately. Upon processing the request, the callback routine should be invoked. In case of an unrecoverable error, the error indication must be placed in the .Va krp_status field of the .Vt cryptkop structure. When the request is completed, or an error is detected, the .Fn kprocess routine should invoked .Fn crypto_kdone . .Sh RETURN VALUES .Fn crypto_register , .Fn crypto_kregister , .Fn crypto_unregister , .Fn crypto_newsession , .Fn crypto_freesession , and .Fn crypto_unblock return 0 on success, or an error code on failure. .Fn crypto_get_driverid returns a non-negative value on error, and \-1 on failure. .Fn crypto_getreq returns a pointer to a .Vt cryptop structure and .Dv NULL on failure. .Fn crypto_dispatch returns .Er EINVAL if its argument or the callback function was .Dv NULL , and 0 otherwise. The callback is provided with an error code in case of failure, in the .Va crp_etype field. .Sh FILES .Bl -tag -width ".Pa sys/opencrypto/crypto.c" .It Pa sys/opencrypto/crypto.c most of the framework code .El .Sh SEE ALSO .Xr crypto 4 , .Xr ipsec 4 , +.Xr crypto 7 , .Xr malloc 9 , .Xr sleep 9 .Sh HISTORY The cryptographic framework first appeared in .Ox 2.7 and was written by .An Angelos D. Keromytis Aq Mt angelos@openbsd.org . .Sh BUGS The framework currently assumes that all the algorithms in a .Fn crypto_newsession operation must be available by the same driver. If that is not the case, session initialization will fail. .Pp The framework also needs a mechanism for determining which driver is best for a specific set of algorithms associated with a session. Some type of benchmarking is in order here. .Pp Multiple instances of the same algorithm in the same session are not supported. Note that 3DES is considered one algorithm (and not three instances of DES). Thus, 3DES and DES could be mixed in the same request. Index: projects/clang350-import/share/mk/bsd.opts.mk =================================================================== --- projects/clang350-import/share/mk/bsd.opts.mk (revision 275748) +++ projects/clang350-import/share/mk/bsd.opts.mk (revision 275749) @@ -1,98 +1,98 @@ # $FreeBSD$ # # Option file for src builds. # # Users define WITH_FOO and WITHOUT_FOO on the command line or in /etc/src.conf # and /etc/make.conf files. These translate in the build system to MK_FOO={yes,no} # with sensible (usually) defaults. # # Makefiles must include bsd.opts.mk after defining specific MK_FOO options that # are applicable for that Makefile (typically there are none, but sometimes there # are exceptions). Recursive makes usually add MK_FOO=no for options that they wish # to omit from that make. # -# Makefiles must include bsd.srcpot.mk before they test the value of any MK_FOO +# Makefiles must include bsd.mkopt.mk before they test the value of any MK_FOO # variable. # # Makefiles may also assume that this file is included by bsd.own.mk should it # need variables defined there prior to the end of the Makefile where # bsd.{subdir,lib.bin}.mk is traditionally included. # # The old-style YES_FOO and NO_FOO are being phased out. No new instances of them # should be added. Old instances should be removed since they were just to # bridge the gap between FreeBSD 4 and FreeBSD 5. # # Makefiles should never test WITH_FOO or WITHOUT_FOO directly (although an # exception is made for _WITHOUT_SRCONF which turns off this mechanism # completely). # .if !target(____) ____: .if !defined(_WITHOUT_SRCCONF) # # Define MK_* variables (which are either "yes" or "no") for users # to set via WITH_*/WITHOUT_* in /etc/src.conf and override in the # make(1) environment. # These should be tested with `== "no"' or `!= "no"' in makefiles. # The NO_* variables should only be set by makefiles for variables # that haven't been converted over. # # Only these options are used by bsd.*.mk. KERBEROS and OPENSSH are # unforutnately needed to support statically linking the entire # tree. su(1) wouldn't link since it depends on PAM which depends on # ssh libraries when building with OPENSSH, and likewise for KERBEROS. # All other variables used to build /usr/src live in src.opts.mk # and variables from both files are documented in src.conf(5). __DEFAULT_YES_OPTIONS = \ ASSERT_DEBUG \ DOCCOMPRESS \ INCLUDES \ INSTALLLIB \ KERBEROS \ MAN \ MANCOMPRESS \ NIS \ NLS \ OPENSSH \ PROFILE \ SSP \ SYMVER \ TOOLCHAIN \ WARNS __DEFAULT_NO_OPTIONS = \ CTF \ DEBUG_FILES \ INSTALL_AS_USER \ INFO .include # # Supported NO_* options (if defined, MK_* will be forced to "no", # regardless of user's setting). # # These are transitional and will disappaer in the FreeBSD 12. # .for var in \ CTF \ DEBUG_FILES \ INSTALLLIB \ MAN \ PROFILE \ WARNS .if defined(NO_${var}) # This warning may be premature... #.warning "NO_${var} is defined, but deprecated. Please use MK_${var}=no instead." MK_${var}:=no .endif .endfor .endif # !_WITHOUT_SRCCONF .endif Index: projects/clang350-import/share/mk/src.opts.mk =================================================================== --- projects/clang350-import/share/mk/src.opts.mk (revision 275748) +++ projects/clang350-import/share/mk/src.opts.mk (revision 275749) @@ -1,374 +1,374 @@ # $FreeBSD$ # # Option file for FreeBSD /usr/src builds. # # Users define WITH_FOO and WITHOUT_FOO on the command line or in /etc/src.conf # and /etc/make.conf files. These translate in the build system to MK_FOO={yes,no} # with sensible (usually) defaults. # # Makefiles must include bsd.opts.mk after defining specific MK_FOO options that # are applicable for that Makefile (typically there are none, but sometimes there # are exceptions). Recursive makes usually add MK_FOO=no for options that they wish # to omit from that make. # -# Makefiles must include bsd.srcpot.mk before they test the value of any MK_FOO +# Makefiles must include bsd.mkopt.mk before they test the value of any MK_FOO # variable. # # Makefiles may also assume that this file is included by src.opts.mk should it # need variables defined there prior to the end of the Makefile where # bsd.{subdir,lib.bin}.mk is traditionally included. # # The old-style YES_FOO and NO_FOO are being phased out. No new instances of them # should be added. Old instances should be removed since they were just to # bridge the gap between FreeBSD 4 and FreeBSD 5. # # Makefiles should never test WITH_FOO or WITHOUT_FOO directly (although an # exception is made for _WITHOUT_SRCONF which turns off this mechanism # completely inside bsd.*.mk files). # .if !target(____) ____: .include # # Define MK_* variables (which are either "yes" or "no") for users # to set via WITH_*/WITHOUT_* in /etc/src.conf and override in the # make(1) environment. # These should be tested with `== "no"' or `!= "no"' in makefiles. # The NO_* variables should only be set by makefiles for variables # that haven't been converted over. # # These options are used by src the builds __DEFAULT_YES_OPTIONS = \ ACCT \ ACPI \ AMD \ APM \ AT \ ATM \ AUDIT \ AUTHPF \ BINUTILS \ BINUTILS_BOOTSTRAP \ BLUETOOTH \ BOOT \ BSD_CPIO \ BSNMP \ BZIP2 \ CALENDAR \ CAPSICUM \ CASPER \ CDDL \ CPP \ CROSS_COMPILER \ CRYPT \ CTM \ CUSE \ CXX \ DICT \ DMAGENT \ DYNAMICROOT \ ED_CRYPTO \ EXAMPLES \ FDT \ FLOPPY \ FMTREE \ FORTH \ FP_LIBC \ FREEBSD_UPDATE \ GAMES \ GCOV \ GDB \ GNU \ GNU_GREP_COMPAT \ GPIB \ GPIO \ GPL_DTC \ GROFF \ HTML \ HYPERV \ ICONV \ INET \ INET6 \ IPFILTER \ IPFW \ JAIL \ KDUMP \ KVM \ LDNS \ LDNS_UTILS \ LEGACY_CONSOLE \ LIB32 \ LIBPTHREAD \ LIBTHR \ LOCALES \ LOCATE \ LPR \ LS_COLORS \ LZMA_SUPPORT \ MAIL \ MAILWRAPPER \ MAKE \ NDIS \ NETCAT \ NETGRAPH \ NLS_CATALOGS \ NS_CACHING \ NTP \ OPENSSL \ PAM \ PC_SYSINSTALL \ PF \ PKGBOOTSTRAP \ PMC \ PORTSNAP \ PPP \ QUOTAS \ RCMDS \ RCS \ RESCUE \ ROUTED \ SENDMAIL \ SETUID_LOGIN \ SHAREDOCS \ SOURCELESS \ SOURCELESS_HOST \ SOURCELESS_UCODE \ SVNLITE \ SYSCALL_COMPAT \ SYSCONS \ SYSINSTALL \ TCSH \ TELNET \ TESTS \ TEXTPROC \ UNBOUND \ USB \ UTMPX \ VI \ VT \ WIRELESS \ WPA_SUPPLICANT_EAPOL \ ZFS \ ZONEINFO __DEFAULT_NO_OPTIONS = \ BSD_GREP \ CLANG_EXTRAS \ EISA \ ELFTOOLCHAIN_TOOLS \ FMAKE \ HESIOD \ LLDB \ NAND \ OFED \ OPENLDAP \ OPENSSH_NONE_CIPHER \ SHARED_TOOLCHAIN \ SORT_THREADS \ SVN # # Default behaviour of some options depends on the architecture. Unfortunately # this means that we have to test TARGET_ARCH (the buildworld case) as well # as MACHINE_ARCH (the non-buildworld case). Normally TARGET_ARCH is not # used at all in bsd.*.mk, but we have to make an exception here if we want # to allow defaults for some things like clang to vary by target architecture. # Additional, per-target behavior should be rarely added only after much # gnashing of teeth and grinding of gears. # .if defined(TARGET_ARCH) __T=${TARGET_ARCH} .else __T=${MACHINE_ARCH} .endif .if defined(TARGET) __TT=${TARGET} .else __TT=${MACHINE} .endif .include .if !${COMPILER_FEATURES:Mc++11} # If the compiler is not C++11 capable, disable clang and use gcc instead. __DEFAULT_YES_OPTIONS+=GCC GCC_BOOTSTRAP GNUCXX __DEFAULT_NO_OPTIONS+=CLANG CLANG_BOOTSTRAP CLANG_FULL CLANG_IS_CC .elif ${__T} == "amd64" || ${__T} == "i386" # On x86, clang is enabled, and will be installed as the default cc. __DEFAULT_YES_OPTIONS+=CLANG CLANG_BOOTSTRAP CLANG_FULL CLANG_IS_CC __DEFAULT_NO_OPTIONS+=GCC GCC_BOOTSTRAP GNUCXX .elif ${__TT} == "arm" && ${__T:Marm*eb*} == "" # On little-endian arm, clang is enabled, and it is installed as the default # cc, but since gcc is unable to build the full clang, disable it by default. __DEFAULT_YES_OPTIONS+=CLANG CLANG_BOOTSTRAP CLANG_IS_CC __DEFAULT_NO_OPTIONS+=CLANG_FULL GCC GCC_BOOTSTRAP GNUCXX .elif ${__T:Mpowerpc*} # On powerpc, clang is enabled, but gcc is installed as the default cc. __DEFAULT_YES_OPTIONS+=CLANG CLANG_FULL GCC GCC_BOOTSTRAP GNUCXX __DEFAULT_NO_OPTIONS+=CLANG_BOOTSTRAP CLANG_IS_CC .else # Everything else disables clang, and uses gcc instead. __DEFAULT_YES_OPTIONS+=GCC GCC_BOOTSTRAP GNUCXX __DEFAULT_NO_OPTIONS+=CLANG CLANG_BOOTSTRAP CLANG_FULL CLANG_IS_CC .endif .include # # MK_* options that default to "yes" if the compiler is a C++11 compiler. # .for var in \ LIBCPLUSPLUS .if !defined(MK_${var}) .if ${COMPILER_FEATURES:Mc++11} .if defined(WITHOUT_${var}) MK_${var}:= no .else MK_${var}:= yes .endif .else .if defined(WITH_${var}) MK_${var}:= yes .else MK_${var}:= no .endif .endif .endif .endfor # # Force some options off if their dependencies are off. # Order is somewhat important. # .if ${MK_LIBPTHREAD} == "no" MK_LIBTHR:= no .endif .if ${MK_LDNS} == "no" MK_LDNS_UTILS:= no MK_UNBOUND:= no .endif .if ${MK_SOURCELESS} == "no" MK_SOURCELESS_HOST:= no MK_SOURCELESS_UCODE:= no .endif .if ${MK_CDDL} == "no" MK_ZFS:= no MK_CTF:= no .endif .if ${MK_CRYPT} == "no" MK_OPENSSL:= no MK_OPENSSH:= no MK_KERBEROS:= no .endif .if ${MK_CXX} == "no" MK_CLANG:= no MK_GROFF:= no MK_GNUCXX:= no .endif .if ${MK_MAIL} == "no" MK_MAILWRAPPER:= no MK_SENDMAIL:= no MK_DMAGENT:= no .endif .if ${MK_NETGRAPH} == "no" MK_ATM:= no MK_BLUETOOTH:= no .endif .if ${MK_OPENSSL} == "no" MK_OPENSSH:= no MK_KERBEROS:= no .endif .if ${MK_PF} == "no" MK_AUTHPF:= no .endif .if ${MK_TEXTPROC} == "no" MK_GROFF:= no .endif .if ${MK_CROSS_COMPILER} == "no" MK_BINUTILS_BOOTSTRAP:= no MK_CLANG_BOOTSTRAP:= no MK_GCC_BOOTSTRAP:= no .endif .if ${MK_TOOLCHAIN} == "no" MK_BINUTILS:= no MK_CLANG:= no MK_GCC:= no MK_GDB:= no MK_INCLUDES:= no .endif .if ${MK_CLANG} == "no" MK_CLANG_EXTRAS:= no MK_CLANG_FULL:= no .endif # # Set defaults for the MK_*_SUPPORT variables. # # # MK_*_SUPPORT options which default to "yes" unless their corresponding # MK_* variable is set to "no". # .for var in \ BZIP2 \ GNU \ INET \ INET6 \ KERBEROS \ KVM \ NETGRAPH \ PAM \ TESTS \ WIRELESS .if defined(WITHOUT_${var}_SUPPORT) || ${MK_${var}} == "no" MK_${var}_SUPPORT:= no .else MK_${var}_SUPPORT:= yes .endif .endfor # # MK_* options whose default value depends on another option. # .for vv in \ GSSAPI/KERBEROS \ MAN_UTILS/MAN .if defined(WITH_${vv:H}) MK_${vv:H}:= yes .elif defined(WITHOUT_${vv:H}) MK_${vv:H}:= no .else MK_${vv:H}:= ${MK_${vv:T}} .endif .endfor .if !${COMPILER_FEATURES:Mc++11} MK_LLDB:= no .endif # gcc 4.8 and newer supports libc++, so suppress gnuc++ in that case. # while in theory we could build it with that, we don't want to do # that since it creates too much confusion for too little gain. .if ${COMPILER_TYPE} == "gcc" && ${COMPILER_VERSION} >= 40800 MK_GNUCXX:=no MK_GCC:=no .endif .endif # !target(____) Index: projects/clang350-import/share =================================================================== --- projects/clang350-import/share (revision 275748) +++ projects/clang350-import/share (revision 275749) Property changes on: projects/clang350-import/share ___________________________________________________________________ Modified: svn:mergeinfo ## -0,0 +0,1 ## Merged /head/share:r275715-275748 Index: projects/clang350-import/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c =================================================================== --- projects/clang350-import/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c (revision 275748) +++ projects/clang350-import/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c (revision 275749) @@ -1,5715 +1,5747 @@ /* * CDDL HEADER START * * The contents of this file are subject to the terms of the * Common Development and Distribution License (the "License"). * You may not use this file except in compliance with the License. * * You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE * or http://www.opensolaris.org/os/licensing. * See the License for the specific language governing permissions * and limitations under the License. * * When distributing Covered Code, include this CDDL HEADER in each * file and include the License file at usr/src/OPENSOLARIS.LICENSE. * If applicable, add the following below this CDDL HEADER, with the * fields enclosed by brackets "[]" replaced with your own identifying * information: Portions Copyright [yyyy] [name of copyright owner] * * CDDL HEADER END */ /* * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. * Copyright (c) 2011, 2014 by Delphix. All rights reserved. * Copyright (c) 2014 by Saso Kiselkov. All rights reserved. * Copyright 2014 Nexenta Systems, Inc. All rights reserved. */ /* * DVA-based Adjustable Replacement Cache * * While much of the theory of operation used here is * based on the self-tuning, low overhead replacement cache * presented by Megiddo and Modha at FAST 2003, there are some * significant differences: * * 1. The Megiddo and Modha model assumes any page is evictable. * Pages in its cache cannot be "locked" into memory. This makes * the eviction algorithm simple: evict the last page in the list. * This also make the performance characteristics easy to reason * about. Our cache is not so simple. At any given moment, some * subset of the blocks in the cache are un-evictable because we * have handed out a reference to them. Blocks are only evictable * when there are no external references active. This makes * eviction far more problematic: we choose to evict the evictable * blocks that are the "lowest" in the list. * * There are times when it is not possible to evict the requested * space. In these circumstances we are unable to adjust the cache * size. To prevent the cache growing unbounded at these times we * implement a "cache throttle" that slows the flow of new data * into the cache until we can make space available. * * 2. The Megiddo and Modha model assumes a fixed cache size. * Pages are evicted when the cache is full and there is a cache * miss. Our model has a variable sized cache. It grows with * high use, but also tries to react to memory pressure from the * operating system: decreasing its size when system memory is * tight. * * 3. The Megiddo and Modha model assumes a fixed page size. All * elements of the cache are therefore exactly the same size. So * when adjusting the cache size following a cache miss, its simply * a matter of choosing a single page to evict. In our model, we * have variable sized cache blocks (rangeing from 512 bytes to * 128K bytes). We therefore choose a set of blocks to evict to make * space for a cache miss that approximates as closely as possible * the space used by the new block. * * See also: "ARC: A Self-Tuning, Low Overhead Replacement Cache" * by N. Megiddo & D. Modha, FAST 2003 */ /* * The locking model: * * A new reference to a cache buffer can be obtained in two * ways: 1) via a hash table lookup using the DVA as a key, * or 2) via one of the ARC lists. The arc_read() interface * uses method 1, while the internal arc algorithms for * adjusting the cache use method 2. We therefore provide two * types of locks: 1) the hash table lock array, and 2) the * arc list locks. * * Buffers do not have their own mutexs, rather they rely on the * hash table mutexs for the bulk of their protection (i.e. most * fields in the arc_buf_hdr_t are protected by these mutexs). * * buf_hash_find() returns the appropriate mutex (held) when it * locates the requested buffer in the hash table. It returns * NULL for the mutex if the buffer was not in the table. * * buf_hash_remove() expects the appropriate hash mutex to be * already held before it is invoked. * * Each arc state also has a mutex which is used to protect the * buffer list associated with the state. When attempting to * obtain a hash table lock while holding an arc list lock you * must use: mutex_tryenter() to avoid deadlock. Also note that * the active state mutex must be held before the ghost state mutex. * * Arc buffers may have an associated eviction callback function. * This function will be invoked prior to removing the buffer (e.g. * in arc_do_user_evicts()). Note however that the data associated * with the buffer may be evicted prior to the callback. The callback * must be made with *no locks held* (to prevent deadlock). Additionally, * the users of callbacks must ensure that their private data is * protected from simultaneous callbacks from arc_clear_callback() * and arc_do_user_evicts(). * * Note that the majority of the performance stats are manipulated * with atomic operations. * * The L2ARC uses the l2arc_buflist_mtx global mutex for the following: * * - L2ARC buflist creation * - L2ARC buflist eviction * - L2ARC write completion, which walks L2ARC buflists * - ARC header destruction, as it removes from L2ARC buflists * - ARC header release, as it removes from L2ARC buflists */ #include #include #include #include #include #include #include #include #include #ifdef _KERNEL #include #endif #include #include #include #include #include #include #include #ifdef illumos #ifndef _KERNEL /* set with ZFS_DEBUG=watch, to enable watchpoints on frozen buffers */ boolean_t arc_watch = B_FALSE; int arc_procfd; #endif #endif /* illumos */ static kmutex_t arc_reclaim_thr_lock; static kcondvar_t arc_reclaim_thr_cv; /* used to signal reclaim thr */ static uint8_t arc_thread_exit; #define ARC_REDUCE_DNLC_PERCENT 3 uint_t arc_reduce_dnlc_percent = ARC_REDUCE_DNLC_PERCENT; typedef enum arc_reclaim_strategy { ARC_RECLAIM_AGGR, /* Aggressive reclaim strategy */ ARC_RECLAIM_CONS /* Conservative reclaim strategy */ } arc_reclaim_strategy_t; /* * The number of iterations through arc_evict_*() before we * drop & reacquire the lock. */ int arc_evict_iterations = 100; /* number of seconds before growing cache again */ static int arc_grow_retry = 60; /* shift of arc_c for calculating both min and max arc_p */ static int arc_p_min_shift = 4; /* log2(fraction of arc to reclaim) */ static int arc_shrink_shift = 5; /* * minimum lifespan of a prefetch block in clock ticks * (initialized in arc_init()) */ static int arc_min_prefetch_lifespan; /* * If this percent of memory is free, don't throttle. */ int arc_lotsfree_percent = 10; static int arc_dead; extern int zfs_prefetch_disable; /* * The arc has filled available memory and has now warmed up. */ static boolean_t arc_warm; uint64_t zfs_arc_max; uint64_t zfs_arc_min; uint64_t zfs_arc_meta_limit = 0; int zfs_arc_grow_retry = 0; int zfs_arc_shrink_shift = 0; int zfs_arc_p_min_shift = 0; int zfs_disable_dup_eviction = 0; uint64_t zfs_arc_average_blocksize = 8 * 1024; /* 8KB */ u_int zfs_arc_free_target = 0; static int sysctl_vfs_zfs_arc_free_target(SYSCTL_HANDLER_ARGS); +static int sysctl_vfs_zfs_arc_meta_limit(SYSCTL_HANDLER_ARGS); #ifdef _KERNEL static void arc_free_target_init(void *unused __unused) { zfs_arc_free_target = vm_pageout_wakeup_thresh; } SYSINIT(arc_free_target_init, SI_SUB_KTHREAD_PAGE, SI_ORDER_ANY, arc_free_target_init, NULL); TUNABLE_QUAD("vfs.zfs.arc_meta_limit", &zfs_arc_meta_limit); TUNABLE_INT("vfs.zfs.arc_shrink_shift", &zfs_arc_shrink_shift); SYSCTL_DECL(_vfs_zfs); SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, arc_max, CTLFLAG_RDTUN, &zfs_arc_max, 0, "Maximum ARC size"); SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, arc_min, CTLFLAG_RDTUN, &zfs_arc_min, 0, "Minimum ARC size"); SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, arc_average_blocksize, CTLFLAG_RDTUN, &zfs_arc_average_blocksize, 0, "ARC average blocksize"); SYSCTL_INT(_vfs_zfs, OID_AUTO, arc_shrink_shift, CTLFLAG_RW, &arc_shrink_shift, 0, "log2(fraction of arc to reclaim)"); /* * We don't have a tunable for arc_free_target due to the dependency on * pagedaemon initialisation. */ SYSCTL_PROC(_vfs_zfs, OID_AUTO, arc_free_target, CTLTYPE_UINT | CTLFLAG_MPSAFE | CTLFLAG_RW, 0, sizeof(u_int), sysctl_vfs_zfs_arc_free_target, "IU", "Desired number of free pages below which ARC triggers reclaim"); static int sysctl_vfs_zfs_arc_free_target(SYSCTL_HANDLER_ARGS) { u_int val; int err; val = zfs_arc_free_target; err = sysctl_handle_int(oidp, &val, 0, req); if (err != 0 || req->newptr == NULL) return (err); if (val < minfree) return (EINVAL); if (val > vm_cnt.v_page_count) return (EINVAL); zfs_arc_free_target = val; return (0); } + +/* + * Must be declared here, before the definition of corresponding kstat + * macro which uses the same names will confuse the compiler. + */ +SYSCTL_PROC(_vfs_zfs, OID_AUTO, arc_meta_limit, + CTLTYPE_U64 | CTLFLAG_MPSAFE | CTLFLAG_RW, 0, sizeof(uint64_t), + sysctl_vfs_zfs_arc_meta_limit, "QU", + "ARC metadata limit"); #endif /* * Note that buffers can be in one of 6 states: * ARC_anon - anonymous (discussed below) * ARC_mru - recently used, currently cached * ARC_mru_ghost - recentely used, no longer in cache * ARC_mfu - frequently used, currently cached * ARC_mfu_ghost - frequently used, no longer in cache * ARC_l2c_only - exists in L2ARC but not other states * When there are no active references to the buffer, they are * are linked onto a list in one of these arc states. These are * the only buffers that can be evicted or deleted. Within each * state there are multiple lists, one for meta-data and one for * non-meta-data. Meta-data (indirect blocks, blocks of dnodes, * etc.) is tracked separately so that it can be managed more * explicitly: favored over data, limited explicitly. * * Anonymous buffers are buffers that are not associated with * a DVA. These are buffers that hold dirty block copies * before they are written to stable storage. By definition, * they are "ref'd" and are considered part of arc_mru * that cannot be freed. Generally, they will aquire a DVA * as they are written and migrate onto the arc_mru list. * * The ARC_l2c_only state is for buffers that are in the second * level ARC but no longer in any of the ARC_m* lists. The second * level ARC itself may also contain buffers that are in any of * the ARC_m* states - meaning that a buffer can exist in two * places. The reason for the ARC_l2c_only state is to keep the * buffer header in the hash table, so that reads that hit the * second level ARC benefit from these fast lookups. */ #define ARCS_LOCK_PAD CACHE_LINE_SIZE struct arcs_lock { kmutex_t arcs_lock; #ifdef _KERNEL unsigned char pad[(ARCS_LOCK_PAD - sizeof (kmutex_t))]; #endif }; /* * must be power of two for mask use to work * */ #define ARC_BUFC_NUMDATALISTS 16 #define ARC_BUFC_NUMMETADATALISTS 16 #define ARC_BUFC_NUMLISTS (ARC_BUFC_NUMMETADATALISTS + ARC_BUFC_NUMDATALISTS) typedef struct arc_state { uint64_t arcs_lsize[ARC_BUFC_NUMTYPES]; /* amount of evictable data */ uint64_t arcs_size; /* total amount of data in this state */ list_t arcs_lists[ARC_BUFC_NUMLISTS]; /* list of evictable buffers */ struct arcs_lock arcs_locks[ARC_BUFC_NUMLISTS] __aligned(CACHE_LINE_SIZE); } arc_state_t; #define ARCS_LOCK(s, i) (&((s)->arcs_locks[(i)].arcs_lock)) /* The 6 states: */ static arc_state_t ARC_anon; static arc_state_t ARC_mru; static arc_state_t ARC_mru_ghost; static arc_state_t ARC_mfu; static arc_state_t ARC_mfu_ghost; static arc_state_t ARC_l2c_only; typedef struct arc_stats { kstat_named_t arcstat_hits; kstat_named_t arcstat_misses; kstat_named_t arcstat_demand_data_hits; kstat_named_t arcstat_demand_data_misses; kstat_named_t arcstat_demand_metadata_hits; kstat_named_t arcstat_demand_metadata_misses; kstat_named_t arcstat_prefetch_data_hits; kstat_named_t arcstat_prefetch_data_misses; kstat_named_t arcstat_prefetch_metadata_hits; kstat_named_t arcstat_prefetch_metadata_misses; kstat_named_t arcstat_mru_hits; kstat_named_t arcstat_mru_ghost_hits; kstat_named_t arcstat_mfu_hits; kstat_named_t arcstat_mfu_ghost_hits; kstat_named_t arcstat_allocated; kstat_named_t arcstat_deleted; kstat_named_t arcstat_stolen; kstat_named_t arcstat_recycle_miss; /* * Number of buffers that could not be evicted because the hash lock * was held by another thread. The lock may not necessarily be held * by something using the same buffer, since hash locks are shared * by multiple buffers. */ kstat_named_t arcstat_mutex_miss; /* * Number of buffers skipped because they have I/O in progress, are * indrect prefetch buffers that have not lived long enough, or are * not from the spa we're trying to evict from. */ kstat_named_t arcstat_evict_skip; kstat_named_t arcstat_evict_l2_cached; kstat_named_t arcstat_evict_l2_eligible; kstat_named_t arcstat_evict_l2_ineligible; kstat_named_t arcstat_hash_elements; kstat_named_t arcstat_hash_elements_max; kstat_named_t arcstat_hash_collisions; kstat_named_t arcstat_hash_chains; kstat_named_t arcstat_hash_chain_max; kstat_named_t arcstat_p; kstat_named_t arcstat_c; kstat_named_t arcstat_c_min; kstat_named_t arcstat_c_max; kstat_named_t arcstat_size; kstat_named_t arcstat_hdr_size; kstat_named_t arcstat_data_size; kstat_named_t arcstat_other_size; kstat_named_t arcstat_l2_hits; kstat_named_t arcstat_l2_misses; kstat_named_t arcstat_l2_feeds; kstat_named_t arcstat_l2_rw_clash; kstat_named_t arcstat_l2_read_bytes; kstat_named_t arcstat_l2_write_bytes; kstat_named_t arcstat_l2_writes_sent; kstat_named_t arcstat_l2_writes_done; kstat_named_t arcstat_l2_writes_error; kstat_named_t arcstat_l2_writes_hdr_miss; kstat_named_t arcstat_l2_evict_lock_retry; kstat_named_t arcstat_l2_evict_reading; kstat_named_t arcstat_l2_free_on_write; kstat_named_t arcstat_l2_cdata_free_on_write; kstat_named_t arcstat_l2_abort_lowmem; kstat_named_t arcstat_l2_cksum_bad; kstat_named_t arcstat_l2_io_error; kstat_named_t arcstat_l2_size; kstat_named_t arcstat_l2_asize; kstat_named_t arcstat_l2_hdr_size; kstat_named_t arcstat_l2_compress_successes; kstat_named_t arcstat_l2_compress_zeros; kstat_named_t arcstat_l2_compress_failures; kstat_named_t arcstat_l2_write_trylock_fail; kstat_named_t arcstat_l2_write_passed_headroom; kstat_named_t arcstat_l2_write_spa_mismatch; kstat_named_t arcstat_l2_write_in_l2; kstat_named_t arcstat_l2_write_hdr_io_in_progress; kstat_named_t arcstat_l2_write_not_cacheable; kstat_named_t arcstat_l2_write_full; kstat_named_t arcstat_l2_write_buffer_iter; kstat_named_t arcstat_l2_write_pios; kstat_named_t arcstat_l2_write_buffer_bytes_scanned; kstat_named_t arcstat_l2_write_buffer_list_iter; kstat_named_t arcstat_l2_write_buffer_list_null_iter; kstat_named_t arcstat_memory_throttle_count; kstat_named_t arcstat_duplicate_buffers; kstat_named_t arcstat_duplicate_buffers_size; kstat_named_t arcstat_duplicate_reads; + kstat_named_t arcstat_meta_used; + kstat_named_t arcstat_meta_limit; + kstat_named_t arcstat_meta_max; } arc_stats_t; static arc_stats_t arc_stats = { { "hits", KSTAT_DATA_UINT64 }, { "misses", KSTAT_DATA_UINT64 }, { "demand_data_hits", KSTAT_DATA_UINT64 }, { "demand_data_misses", KSTAT_DATA_UINT64 }, { "demand_metadata_hits", KSTAT_DATA_UINT64 }, { "demand_metadata_misses", KSTAT_DATA_UINT64 }, { "prefetch_data_hits", KSTAT_DATA_UINT64 }, { "prefetch_data_misses", KSTAT_DATA_UINT64 }, { "prefetch_metadata_hits", KSTAT_DATA_UINT64 }, { "prefetch_metadata_misses", KSTAT_DATA_UINT64 }, { "mru_hits", KSTAT_DATA_UINT64 }, { "mru_ghost_hits", KSTAT_DATA_UINT64 }, { "mfu_hits", KSTAT_DATA_UINT64 }, { "mfu_ghost_hits", KSTAT_DATA_UINT64 }, { "allocated", KSTAT_DATA_UINT64 }, { "deleted", KSTAT_DATA_UINT64 }, { "stolen", KSTAT_DATA_UINT64 }, { "recycle_miss", KSTAT_DATA_UINT64 }, { "mutex_miss", KSTAT_DATA_UINT64 }, { "evict_skip", KSTAT_DATA_UINT64 }, { "evict_l2_cached", KSTAT_DATA_UINT64 }, { "evict_l2_eligible", KSTAT_DATA_UINT64 }, { "evict_l2_ineligible", KSTAT_DATA_UINT64 }, { "hash_elements", KSTAT_DATA_UINT64 }, { "hash_elements_max", KSTAT_DATA_UINT64 }, { "hash_collisions", KSTAT_DATA_UINT64 }, { "hash_chains", KSTAT_DATA_UINT64 }, { "hash_chain_max", KSTAT_DATA_UINT64 }, { "p", KSTAT_DATA_UINT64 }, { "c", KSTAT_DATA_UINT64 }, { "c_min", KSTAT_DATA_UINT64 }, { "c_max", KSTAT_DATA_UINT64 }, { "size", KSTAT_DATA_UINT64 }, { "hdr_size", KSTAT_DATA_UINT64 }, { "data_size", KSTAT_DATA_UINT64 }, { "other_size", KSTAT_DATA_UINT64 }, { "l2_hits", KSTAT_DATA_UINT64 }, { "l2_misses", KSTAT_DATA_UINT64 }, { "l2_feeds", KSTAT_DATA_UINT64 }, { "l2_rw_clash", KSTAT_DATA_UINT64 }, { "l2_read_bytes", KSTAT_DATA_UINT64 }, { "l2_write_bytes", KSTAT_DATA_UINT64 }, { "l2_writes_sent", KSTAT_DATA_UINT64 }, { "l2_writes_done", KSTAT_DATA_UINT64 }, { "l2_writes_error", KSTAT_DATA_UINT64 }, { "l2_writes_hdr_miss", KSTAT_DATA_UINT64 }, { "l2_evict_lock_retry", KSTAT_DATA_UINT64 }, { "l2_evict_reading", KSTAT_DATA_UINT64 }, { "l2_free_on_write", KSTAT_DATA_UINT64 }, { "l2_cdata_free_on_write", KSTAT_DATA_UINT64 }, { "l2_abort_lowmem", KSTAT_DATA_UINT64 }, { "l2_cksum_bad", KSTAT_DATA_UINT64 }, { "l2_io_error", KSTAT_DATA_UINT64 }, { "l2_size", KSTAT_DATA_UINT64 }, { "l2_asize", KSTAT_DATA_UINT64 }, { "l2_hdr_size", KSTAT_DATA_UINT64 }, { "l2_compress_successes", KSTAT_DATA_UINT64 }, { "l2_compress_zeros", KSTAT_DATA_UINT64 }, { "l2_compress_failures", KSTAT_DATA_UINT64 }, { "l2_write_trylock_fail", KSTAT_DATA_UINT64 }, { "l2_write_passed_headroom", KSTAT_DATA_UINT64 }, { "l2_write_spa_mismatch", KSTAT_DATA_UINT64 }, { "l2_write_in_l2", KSTAT_DATA_UINT64 }, { "l2_write_io_in_progress", KSTAT_DATA_UINT64 }, { "l2_write_not_cacheable", KSTAT_DATA_UINT64 }, { "l2_write_full", KSTAT_DATA_UINT64 }, { "l2_write_buffer_iter", KSTAT_DATA_UINT64 }, { "l2_write_pios", KSTAT_DATA_UINT64 }, { "l2_write_buffer_bytes_scanned", KSTAT_DATA_UINT64 }, { "l2_write_buffer_list_iter", KSTAT_DATA_UINT64 }, { "l2_write_buffer_list_null_iter", KSTAT_DATA_UINT64 }, { "memory_throttle_count", KSTAT_DATA_UINT64 }, { "duplicate_buffers", KSTAT_DATA_UINT64 }, { "duplicate_buffers_size", KSTAT_DATA_UINT64 }, - { "duplicate_reads", KSTAT_DATA_UINT64 } + { "duplicate_reads", KSTAT_DATA_UINT64 }, + { "arc_meta_used", KSTAT_DATA_UINT64 }, + { "arc_meta_limit", KSTAT_DATA_UINT64 }, + { "arc_meta_max", KSTAT_DATA_UINT64 } }; #define ARCSTAT(stat) (arc_stats.stat.value.ui64) #define ARCSTAT_INCR(stat, val) \ atomic_add_64(&arc_stats.stat.value.ui64, (val)) #define ARCSTAT_BUMP(stat) ARCSTAT_INCR(stat, 1) #define ARCSTAT_BUMPDOWN(stat) ARCSTAT_INCR(stat, -1) #define ARCSTAT_MAX(stat, val) { \ uint64_t m; \ while ((val) > (m = arc_stats.stat.value.ui64) && \ (m != atomic_cas_64(&arc_stats.stat.value.ui64, m, (val)))) \ continue; \ } #define ARCSTAT_MAXSTAT(stat) \ ARCSTAT_MAX(stat##_max, arc_stats.stat.value.ui64) /* * We define a macro to allow ARC hits/misses to be easily broken down by * two separate conditions, giving a total of four different subtypes for * each of hits and misses (so eight statistics total). */ #define ARCSTAT_CONDSTAT(cond1, stat1, notstat1, cond2, stat2, notstat2, stat) \ if (cond1) { \ if (cond2) { \ ARCSTAT_BUMP(arcstat_##stat1##_##stat2##_##stat); \ } else { \ ARCSTAT_BUMP(arcstat_##stat1##_##notstat2##_##stat); \ } \ } else { \ if (cond2) { \ ARCSTAT_BUMP(arcstat_##notstat1##_##stat2##_##stat); \ } else { \ ARCSTAT_BUMP(arcstat_##notstat1##_##notstat2##_##stat);\ } \ } kstat_t *arc_ksp; static arc_state_t *arc_anon; static arc_state_t *arc_mru; static arc_state_t *arc_mru_ghost; static arc_state_t *arc_mfu; static arc_state_t *arc_mfu_ghost; static arc_state_t *arc_l2c_only; /* * There are several ARC variables that are critical to export as kstats -- * but we don't want to have to grovel around in the kstat whenever we wish to * manipulate them. For these variables, we therefore define them to be in * terms of the statistic variable. This assures that we are not introducing * the possibility of inconsistency by having shadow copies of the variables, * while still allowing the code to be readable. */ #define arc_size ARCSTAT(arcstat_size) /* actual total arc size */ #define arc_p ARCSTAT(arcstat_p) /* target size of MRU */ #define arc_c ARCSTAT(arcstat_c) /* target size of cache */ #define arc_c_min ARCSTAT(arcstat_c_min) /* min target cache size */ #define arc_c_max ARCSTAT(arcstat_c_max) /* max target cache size */ +#define arc_meta_limit ARCSTAT(arcstat_meta_limit) /* max size for metadata */ +#define arc_meta_used ARCSTAT(arcstat_meta_used) /* size of metadata */ +#define arc_meta_max ARCSTAT(arcstat_meta_max) /* max size of metadata */ #define L2ARC_IS_VALID_COMPRESS(_c_) \ ((_c_) == ZIO_COMPRESS_LZ4 || (_c_) == ZIO_COMPRESS_EMPTY) static int arc_no_grow; /* Don't try to grow cache size */ static uint64_t arc_tempreserve; static uint64_t arc_loaned_bytes; -static uint64_t arc_meta_used; -static uint64_t arc_meta_limit; -static uint64_t arc_meta_max = 0; -SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, arc_meta_used, CTLFLAG_RD, &arc_meta_used, 0, - "ARC metadata used"); -SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, arc_meta_limit, CTLFLAG_RW, &arc_meta_limit, 0, - "ARC metadata limit"); typedef struct l2arc_buf_hdr l2arc_buf_hdr_t; typedef struct arc_callback arc_callback_t; struct arc_callback { void *acb_private; arc_done_func_t *acb_done; arc_buf_t *acb_buf; zio_t *acb_zio_dummy; arc_callback_t *acb_next; }; typedef struct arc_write_callback arc_write_callback_t; struct arc_write_callback { void *awcb_private; arc_done_func_t *awcb_ready; arc_done_func_t *awcb_physdone; arc_done_func_t *awcb_done; arc_buf_t *awcb_buf; }; struct arc_buf_hdr { /* protected by hash lock */ dva_t b_dva; uint64_t b_birth; uint64_t b_cksum0; kmutex_t b_freeze_lock; zio_cksum_t *b_freeze_cksum; void *b_thawed; arc_buf_hdr_t *b_hash_next; arc_buf_t *b_buf; uint32_t b_flags; uint32_t b_datacnt; arc_callback_t *b_acb; kcondvar_t b_cv; /* immutable */ arc_buf_contents_t b_type; uint64_t b_size; uint64_t b_spa; /* protected by arc state mutex */ arc_state_t *b_state; list_node_t b_arc_node; /* updated atomically */ clock_t b_arc_access; /* self protecting */ refcount_t b_refcnt; l2arc_buf_hdr_t *b_l2hdr; list_node_t b_l2node; }; +#ifdef _KERNEL +static int +sysctl_vfs_zfs_arc_meta_limit(SYSCTL_HANDLER_ARGS) +{ + uint64_t val; + int err; + + val = arc_meta_limit; + err = sysctl_handle_64(oidp, &val, 0, req); + if (err != 0 || req->newptr == NULL) + return (err); + + if (val <= 0 || val > arc_c_max) + return (EINVAL); + + arc_meta_limit = val; + return (0); +} +#endif + static arc_buf_t *arc_eviction_list; static kmutex_t arc_eviction_mtx; static arc_buf_hdr_t arc_eviction_hdr; static void arc_get_data_buf(arc_buf_t *buf); static void arc_access(arc_buf_hdr_t *buf, kmutex_t *hash_lock); static int arc_evict_needed(arc_buf_contents_t type); static void arc_evict_ghost(arc_state_t *state, uint64_t spa, int64_t bytes); #ifdef illumos static void arc_buf_watch(arc_buf_t *buf); #endif /* illumos */ static boolean_t l2arc_write_eligible(uint64_t spa_guid, arc_buf_hdr_t *ab); #define GHOST_STATE(state) \ ((state) == arc_mru_ghost || (state) == arc_mfu_ghost || \ (state) == arc_l2c_only) /* * Private ARC flags. These flags are private ARC only flags that will show up * in b_flags in the arc_hdr_buf_t. Some flags are publicly declared, and can * be passed in as arc_flags in things like arc_read. However, these flags * should never be passed and should only be set by ARC code. When adding new * public flags, make sure not to smash the private ones. */ #define ARC_IN_HASH_TABLE (1 << 9) /* this buffer is hashed */ #define ARC_IO_IN_PROGRESS (1 << 10) /* I/O in progress for buf */ #define ARC_IO_ERROR (1 << 11) /* I/O failed for buf */ #define ARC_FREED_IN_READ (1 << 12) /* buf freed while in read */ #define ARC_BUF_AVAILABLE (1 << 13) /* block not in active use */ #define ARC_INDIRECT (1 << 14) /* this is an indirect block */ #define ARC_FREE_IN_PROGRESS (1 << 15) /* hdr about to be freed */ #define ARC_L2_WRITING (1 << 16) /* L2ARC write in progress */ #define ARC_L2_EVICTED (1 << 17) /* evicted during I/O */ #define ARC_L2_WRITE_HEAD (1 << 18) /* head of write list */ #define HDR_IN_HASH_TABLE(hdr) ((hdr)->b_flags & ARC_IN_HASH_TABLE) #define HDR_IO_IN_PROGRESS(hdr) ((hdr)->b_flags & ARC_IO_IN_PROGRESS) #define HDR_IO_ERROR(hdr) ((hdr)->b_flags & ARC_IO_ERROR) #define HDR_PREFETCH(hdr) ((hdr)->b_flags & ARC_PREFETCH) #define HDR_FREED_IN_READ(hdr) ((hdr)->b_flags & ARC_FREED_IN_READ) #define HDR_BUF_AVAILABLE(hdr) ((hdr)->b_flags & ARC_BUF_AVAILABLE) #define HDR_FREE_IN_PROGRESS(hdr) ((hdr)->b_flags & ARC_FREE_IN_PROGRESS) #define HDR_L2CACHE(hdr) ((hdr)->b_flags & ARC_L2CACHE) #define HDR_L2_READING(hdr) ((hdr)->b_flags & ARC_IO_IN_PROGRESS && \ (hdr)->b_l2hdr != NULL) #define HDR_L2_WRITING(hdr) ((hdr)->b_flags & ARC_L2_WRITING) #define HDR_L2_EVICTED(hdr) ((hdr)->b_flags & ARC_L2_EVICTED) #define HDR_L2_WRITE_HEAD(hdr) ((hdr)->b_flags & ARC_L2_WRITE_HEAD) /* * Other sizes */ #define HDR_SIZE ((int64_t)sizeof (arc_buf_hdr_t)) #define L2HDR_SIZE ((int64_t)sizeof (l2arc_buf_hdr_t)) /* * Hash table routines */ #define HT_LOCK_PAD CACHE_LINE_SIZE struct ht_lock { kmutex_t ht_lock; #ifdef _KERNEL unsigned char pad[(HT_LOCK_PAD - sizeof (kmutex_t))]; #endif }; #define BUF_LOCKS 256 typedef struct buf_hash_table { uint64_t ht_mask; arc_buf_hdr_t **ht_table; struct ht_lock ht_locks[BUF_LOCKS] __aligned(CACHE_LINE_SIZE); } buf_hash_table_t; static buf_hash_table_t buf_hash_table; #define BUF_HASH_INDEX(spa, dva, birth) \ (buf_hash(spa, dva, birth) & buf_hash_table.ht_mask) #define BUF_HASH_LOCK_NTRY(idx) (buf_hash_table.ht_locks[idx & (BUF_LOCKS-1)]) #define BUF_HASH_LOCK(idx) (&(BUF_HASH_LOCK_NTRY(idx).ht_lock)) #define HDR_LOCK(hdr) \ (BUF_HASH_LOCK(BUF_HASH_INDEX(hdr->b_spa, &hdr->b_dva, hdr->b_birth))) uint64_t zfs_crc64_table[256]; /* * Level 2 ARC */ #define L2ARC_WRITE_SIZE (8 * 1024 * 1024) /* initial write max */ #define L2ARC_HEADROOM 2 /* num of writes */ /* * If we discover during ARC scan any buffers to be compressed, we boost * our headroom for the next scanning cycle by this percentage multiple. */ #define L2ARC_HEADROOM_BOOST 200 #define L2ARC_FEED_SECS 1 /* caching interval secs */ #define L2ARC_FEED_MIN_MS 200 /* min caching interval ms */ #define l2arc_writes_sent ARCSTAT(arcstat_l2_writes_sent) #define l2arc_writes_done ARCSTAT(arcstat_l2_writes_done) /* L2ARC Performance Tunables */ uint64_t l2arc_write_max = L2ARC_WRITE_SIZE; /* default max write size */ uint64_t l2arc_write_boost = L2ARC_WRITE_SIZE; /* extra write during warmup */ uint64_t l2arc_headroom = L2ARC_HEADROOM; /* number of dev writes */ uint64_t l2arc_headroom_boost = L2ARC_HEADROOM_BOOST; uint64_t l2arc_feed_secs = L2ARC_FEED_SECS; /* interval seconds */ uint64_t l2arc_feed_min_ms = L2ARC_FEED_MIN_MS; /* min interval milliseconds */ boolean_t l2arc_noprefetch = B_TRUE; /* don't cache prefetch bufs */ boolean_t l2arc_feed_again = B_TRUE; /* turbo warmup */ boolean_t l2arc_norw = B_TRUE; /* no reads during writes */ SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, l2arc_write_max, CTLFLAG_RW, &l2arc_write_max, 0, "max write size"); SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, l2arc_write_boost, CTLFLAG_RW, &l2arc_write_boost, 0, "extra write during warmup"); SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, l2arc_headroom, CTLFLAG_RW, &l2arc_headroom, 0, "number of dev writes"); SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, l2arc_feed_secs, CTLFLAG_RW, &l2arc_feed_secs, 0, "interval seconds"); SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, l2arc_feed_min_ms, CTLFLAG_RW, &l2arc_feed_min_ms, 0, "min interval milliseconds"); SYSCTL_INT(_vfs_zfs, OID_AUTO, l2arc_noprefetch, CTLFLAG_RW, &l2arc_noprefetch, 0, "don't cache prefetch bufs"); SYSCTL_INT(_vfs_zfs, OID_AUTO, l2arc_feed_again, CTLFLAG_RW, &l2arc_feed_again, 0, "turbo warmup"); SYSCTL_INT(_vfs_zfs, OID_AUTO, l2arc_norw, CTLFLAG_RW, &l2arc_norw, 0, "no reads during writes"); SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, anon_size, CTLFLAG_RD, &ARC_anon.arcs_size, 0, "size of anonymous state"); SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, anon_metadata_lsize, CTLFLAG_RD, &ARC_anon.arcs_lsize[ARC_BUFC_METADATA], 0, "size of anonymous state"); SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, anon_data_lsize, CTLFLAG_RD, &ARC_anon.arcs_lsize[ARC_BUFC_DATA], 0, "size of anonymous state"); SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, mru_size, CTLFLAG_RD, &ARC_mru.arcs_size, 0, "size of mru state"); SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, mru_metadata_lsize, CTLFLAG_RD, &ARC_mru.arcs_lsize[ARC_BUFC_METADATA], 0, "size of metadata in mru state"); SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, mru_data_lsize, CTLFLAG_RD, &ARC_mru.arcs_lsize[ARC_BUFC_DATA], 0, "size of data in mru state"); SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, mru_ghost_size, CTLFLAG_RD, &ARC_mru_ghost.arcs_size, 0, "size of mru ghost state"); SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, mru_ghost_metadata_lsize, CTLFLAG_RD, &ARC_mru_ghost.arcs_lsize[ARC_BUFC_METADATA], 0, "size of metadata in mru ghost state"); SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, mru_ghost_data_lsize, CTLFLAG_RD, &ARC_mru_ghost.arcs_lsize[ARC_BUFC_DATA], 0, "size of data in mru ghost state"); SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, mfu_size, CTLFLAG_RD, &ARC_mfu.arcs_size, 0, "size of mfu state"); SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, mfu_metadata_lsize, CTLFLAG_RD, &ARC_mfu.arcs_lsize[ARC_BUFC_METADATA], 0, "size of metadata in mfu state"); SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, mfu_data_lsize, CTLFLAG_RD, &ARC_mfu.arcs_lsize[ARC_BUFC_DATA], 0, "size of data in mfu state"); SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, mfu_ghost_size, CTLFLAG_RD, &ARC_mfu_ghost.arcs_size, 0, "size of mfu ghost state"); SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, mfu_ghost_metadata_lsize, CTLFLAG_RD, &ARC_mfu_ghost.arcs_lsize[ARC_BUFC_METADATA], 0, "size of metadata in mfu ghost state"); SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, mfu_ghost_data_lsize, CTLFLAG_RD, &ARC_mfu_ghost.arcs_lsize[ARC_BUFC_DATA], 0, "size of data in mfu ghost state"); SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, l2c_only_size, CTLFLAG_RD, &ARC_l2c_only.arcs_size, 0, "size of mru state"); /* * L2ARC Internals */ typedef struct l2arc_dev { vdev_t *l2ad_vdev; /* vdev */ spa_t *l2ad_spa; /* spa */ uint64_t l2ad_hand; /* next write location */ uint64_t l2ad_start; /* first addr on device */ uint64_t l2ad_end; /* last addr on device */ uint64_t l2ad_evict; /* last addr eviction reached */ boolean_t l2ad_first; /* first sweep through */ boolean_t l2ad_writing; /* currently writing */ list_t *l2ad_buflist; /* buffer list */ list_node_t l2ad_node; /* device list node */ } l2arc_dev_t; static list_t L2ARC_dev_list; /* device list */ static list_t *l2arc_dev_list; /* device list pointer */ static kmutex_t l2arc_dev_mtx; /* device list mutex */ static l2arc_dev_t *l2arc_dev_last; /* last device used */ static kmutex_t l2arc_buflist_mtx; /* mutex for all buflists */ static list_t L2ARC_free_on_write; /* free after write buf list */ static list_t *l2arc_free_on_write; /* free after write list ptr */ static kmutex_t l2arc_free_on_write_mtx; /* mutex for list */ static uint64_t l2arc_ndev; /* number of devices */ typedef struct l2arc_read_callback { arc_buf_t *l2rcb_buf; /* read buffer */ spa_t *l2rcb_spa; /* spa */ blkptr_t l2rcb_bp; /* original blkptr */ zbookmark_phys_t l2rcb_zb; /* original bookmark */ int l2rcb_flags; /* original flags */ enum zio_compress l2rcb_compress; /* applied compress */ } l2arc_read_callback_t; typedef struct l2arc_write_callback { l2arc_dev_t *l2wcb_dev; /* device info */ arc_buf_hdr_t *l2wcb_head; /* head of write buflist */ } l2arc_write_callback_t; struct l2arc_buf_hdr { /* protected by arc_buf_hdr mutex */ l2arc_dev_t *b_dev; /* L2ARC device */ uint64_t b_daddr; /* disk address, offset byte */ /* compression applied to buffer data */ enum zio_compress b_compress; /* real alloc'd buffer size depending on b_compress applied */ int b_asize; /* temporary buffer holder for in-flight compressed data */ void *b_tmp_cdata; }; typedef struct l2arc_data_free { /* protected by l2arc_free_on_write_mtx */ void *l2df_data; size_t l2df_size; void (*l2df_func)(void *, size_t); list_node_t l2df_list_node; } l2arc_data_free_t; static kmutex_t l2arc_feed_thr_lock; static kcondvar_t l2arc_feed_thr_cv; static uint8_t l2arc_thread_exit; static void l2arc_read_done(zio_t *zio); static void l2arc_hdr_stat_add(void); static void l2arc_hdr_stat_remove(void); static boolean_t l2arc_compress_buf(l2arc_buf_hdr_t *l2hdr); static void l2arc_decompress_zio(zio_t *zio, arc_buf_hdr_t *hdr, enum zio_compress c); static void l2arc_release_cdata_buf(arc_buf_hdr_t *ab); static uint64_t buf_hash(uint64_t spa, const dva_t *dva, uint64_t birth) { uint8_t *vdva = (uint8_t *)dva; uint64_t crc = -1ULL; int i; ASSERT(zfs_crc64_table[128] == ZFS_CRC64_POLY); for (i = 0; i < sizeof (dva_t); i++) crc = (crc >> 8) ^ zfs_crc64_table[(crc ^ vdva[i]) & 0xFF]; crc ^= (spa>>8) ^ birth; return (crc); } #define BUF_EMPTY(buf) \ ((buf)->b_dva.dva_word[0] == 0 && \ (buf)->b_dva.dva_word[1] == 0 && \ (buf)->b_cksum0 == 0) #define BUF_EQUAL(spa, dva, birth, buf) \ ((buf)->b_dva.dva_word[0] == (dva)->dva_word[0]) && \ ((buf)->b_dva.dva_word[1] == (dva)->dva_word[1]) && \ ((buf)->b_birth == birth) && ((buf)->b_spa == spa) static void buf_discard_identity(arc_buf_hdr_t *hdr) { hdr->b_dva.dva_word[0] = 0; hdr->b_dva.dva_word[1] = 0; hdr->b_birth = 0; hdr->b_cksum0 = 0; } static arc_buf_hdr_t * buf_hash_find(uint64_t spa, const blkptr_t *bp, kmutex_t **lockp) { const dva_t *dva = BP_IDENTITY(bp); uint64_t birth = BP_PHYSICAL_BIRTH(bp); uint64_t idx = BUF_HASH_INDEX(spa, dva, birth); kmutex_t *hash_lock = BUF_HASH_LOCK(idx); arc_buf_hdr_t *buf; mutex_enter(hash_lock); for (buf = buf_hash_table.ht_table[idx]; buf != NULL; buf = buf->b_hash_next) { if (BUF_EQUAL(spa, dva, birth, buf)) { *lockp = hash_lock; return (buf); } } mutex_exit(hash_lock); *lockp = NULL; return (NULL); } /* * Insert an entry into the hash table. If there is already an element * equal to elem in the hash table, then the already existing element * will be returned and the new element will not be inserted. * Otherwise returns NULL. */ static arc_buf_hdr_t * buf_hash_insert(arc_buf_hdr_t *buf, kmutex_t **lockp) { uint64_t idx = BUF_HASH_INDEX(buf->b_spa, &buf->b_dva, buf->b_birth); kmutex_t *hash_lock = BUF_HASH_LOCK(idx); arc_buf_hdr_t *fbuf; uint32_t i; ASSERT(!DVA_IS_EMPTY(&buf->b_dva)); ASSERT(buf->b_birth != 0); ASSERT(!HDR_IN_HASH_TABLE(buf)); *lockp = hash_lock; mutex_enter(hash_lock); for (fbuf = buf_hash_table.ht_table[idx], i = 0; fbuf != NULL; fbuf = fbuf->b_hash_next, i++) { if (BUF_EQUAL(buf->b_spa, &buf->b_dva, buf->b_birth, fbuf)) return (fbuf); } buf->b_hash_next = buf_hash_table.ht_table[idx]; buf_hash_table.ht_table[idx] = buf; buf->b_flags |= ARC_IN_HASH_TABLE; /* collect some hash table performance data */ if (i > 0) { ARCSTAT_BUMP(arcstat_hash_collisions); if (i == 1) ARCSTAT_BUMP(arcstat_hash_chains); ARCSTAT_MAX(arcstat_hash_chain_max, i); } ARCSTAT_BUMP(arcstat_hash_elements); ARCSTAT_MAXSTAT(arcstat_hash_elements); return (NULL); } static void buf_hash_remove(arc_buf_hdr_t *buf) { arc_buf_hdr_t *fbuf, **bufp; uint64_t idx = BUF_HASH_INDEX(buf->b_spa, &buf->b_dva, buf->b_birth); ASSERT(MUTEX_HELD(BUF_HASH_LOCK(idx))); ASSERT(HDR_IN_HASH_TABLE(buf)); bufp = &buf_hash_table.ht_table[idx]; while ((fbuf = *bufp) != buf) { ASSERT(fbuf != NULL); bufp = &fbuf->b_hash_next; } *bufp = buf->b_hash_next; buf->b_hash_next = NULL; buf->b_flags &= ~ARC_IN_HASH_TABLE; /* collect some hash table performance data */ ARCSTAT_BUMPDOWN(arcstat_hash_elements); if (buf_hash_table.ht_table[idx] && buf_hash_table.ht_table[idx]->b_hash_next == NULL) ARCSTAT_BUMPDOWN(arcstat_hash_chains); } /* * Global data structures and functions for the buf kmem cache. */ static kmem_cache_t *hdr_cache; static kmem_cache_t *buf_cache; static void buf_fini(void) { int i; kmem_free(buf_hash_table.ht_table, (buf_hash_table.ht_mask + 1) * sizeof (void *)); for (i = 0; i < BUF_LOCKS; i++) mutex_destroy(&buf_hash_table.ht_locks[i].ht_lock); kmem_cache_destroy(hdr_cache); kmem_cache_destroy(buf_cache); } /* * Constructor callback - called when the cache is empty * and a new buf is requested. */ /* ARGSUSED */ static int hdr_cons(void *vbuf, void *unused, int kmflag) { arc_buf_hdr_t *buf = vbuf; bzero(buf, sizeof (arc_buf_hdr_t)); refcount_create(&buf->b_refcnt); cv_init(&buf->b_cv, NULL, CV_DEFAULT, NULL); mutex_init(&buf->b_freeze_lock, NULL, MUTEX_DEFAULT, NULL); arc_space_consume(sizeof (arc_buf_hdr_t), ARC_SPACE_HDRS); return (0); } /* ARGSUSED */ static int buf_cons(void *vbuf, void *unused, int kmflag) { arc_buf_t *buf = vbuf; bzero(buf, sizeof (arc_buf_t)); mutex_init(&buf->b_evict_lock, NULL, MUTEX_DEFAULT, NULL); arc_space_consume(sizeof (arc_buf_t), ARC_SPACE_HDRS); return (0); } /* * Destructor callback - called when a cached buf is * no longer required. */ /* ARGSUSED */ static void hdr_dest(void *vbuf, void *unused) { arc_buf_hdr_t *buf = vbuf; ASSERT(BUF_EMPTY(buf)); refcount_destroy(&buf->b_refcnt); cv_destroy(&buf->b_cv); mutex_destroy(&buf->b_freeze_lock); arc_space_return(sizeof (arc_buf_hdr_t), ARC_SPACE_HDRS); } /* ARGSUSED */ static void buf_dest(void *vbuf, void *unused) { arc_buf_t *buf = vbuf; mutex_destroy(&buf->b_evict_lock); arc_space_return(sizeof (arc_buf_t), ARC_SPACE_HDRS); } /* * Reclaim callback -- invoked when memory is low. */ /* ARGSUSED */ static void hdr_recl(void *unused) { dprintf("hdr_recl called\n"); /* * umem calls the reclaim func when we destroy the buf cache, * which is after we do arc_fini(). */ if (!arc_dead) cv_signal(&arc_reclaim_thr_cv); } static void buf_init(void) { uint64_t *ct; uint64_t hsize = 1ULL << 12; int i, j; /* * The hash table is big enough to fill all of physical memory * with an average block size of zfs_arc_average_blocksize (default 8K). * By default, the table will take up * totalmem * sizeof(void*) / 8K (1MB per GB with 8-byte pointers). */ while (hsize * zfs_arc_average_blocksize < (uint64_t)physmem * PAGESIZE) hsize <<= 1; retry: buf_hash_table.ht_mask = hsize - 1; buf_hash_table.ht_table = kmem_zalloc(hsize * sizeof (void*), KM_NOSLEEP); if (buf_hash_table.ht_table == NULL) { ASSERT(hsize > (1ULL << 8)); hsize >>= 1; goto retry; } hdr_cache = kmem_cache_create("arc_buf_hdr_t", sizeof (arc_buf_hdr_t), 0, hdr_cons, hdr_dest, hdr_recl, NULL, NULL, 0); buf_cache = kmem_cache_create("arc_buf_t", sizeof (arc_buf_t), 0, buf_cons, buf_dest, NULL, NULL, NULL, 0); for (i = 0; i < 256; i++) for (ct = zfs_crc64_table + i, *ct = i, j = 8; j > 0; j--) *ct = (*ct >> 1) ^ (-(*ct & 1) & ZFS_CRC64_POLY); for (i = 0; i < BUF_LOCKS; i++) { mutex_init(&buf_hash_table.ht_locks[i].ht_lock, NULL, MUTEX_DEFAULT, NULL); } } #define ARC_MINTIME (hz>>4) /* 62 ms */ static void arc_cksum_verify(arc_buf_t *buf) { zio_cksum_t zc; if (!(zfs_flags & ZFS_DEBUG_MODIFY)) return; mutex_enter(&buf->b_hdr->b_freeze_lock); if (buf->b_hdr->b_freeze_cksum == NULL || (buf->b_hdr->b_flags & ARC_IO_ERROR)) { mutex_exit(&buf->b_hdr->b_freeze_lock); return; } fletcher_2_native(buf->b_data, buf->b_hdr->b_size, &zc); if (!ZIO_CHECKSUM_EQUAL(*buf->b_hdr->b_freeze_cksum, zc)) panic("buffer modified while frozen!"); mutex_exit(&buf->b_hdr->b_freeze_lock); } static int arc_cksum_equal(arc_buf_t *buf) { zio_cksum_t zc; int equal; mutex_enter(&buf->b_hdr->b_freeze_lock); fletcher_2_native(buf->b_data, buf->b_hdr->b_size, &zc); equal = ZIO_CHECKSUM_EQUAL(*buf->b_hdr->b_freeze_cksum, zc); mutex_exit(&buf->b_hdr->b_freeze_lock); return (equal); } static void arc_cksum_compute(arc_buf_t *buf, boolean_t force) { if (!force && !(zfs_flags & ZFS_DEBUG_MODIFY)) return; mutex_enter(&buf->b_hdr->b_freeze_lock); if (buf->b_hdr->b_freeze_cksum != NULL) { mutex_exit(&buf->b_hdr->b_freeze_lock); return; } buf->b_hdr->b_freeze_cksum = kmem_alloc(sizeof (zio_cksum_t), KM_SLEEP); fletcher_2_native(buf->b_data, buf->b_hdr->b_size, buf->b_hdr->b_freeze_cksum); mutex_exit(&buf->b_hdr->b_freeze_lock); #ifdef illumos arc_buf_watch(buf); #endif /* illumos */ } #ifdef illumos #ifndef _KERNEL typedef struct procctl { long cmd; prwatch_t prwatch; } procctl_t; #endif /* ARGSUSED */ static void arc_buf_unwatch(arc_buf_t *buf) { #ifndef _KERNEL if (arc_watch) { int result; procctl_t ctl; ctl.cmd = PCWATCH; ctl.prwatch.pr_vaddr = (uintptr_t)buf->b_data; ctl.prwatch.pr_size = 0; ctl.prwatch.pr_wflags = 0; result = write(arc_procfd, &ctl, sizeof (ctl)); ASSERT3U(result, ==, sizeof (ctl)); } #endif } /* ARGSUSED */ static void arc_buf_watch(arc_buf_t *buf) { #ifndef _KERNEL if (arc_watch) { int result; procctl_t ctl; ctl.cmd = PCWATCH; ctl.prwatch.pr_vaddr = (uintptr_t)buf->b_data; ctl.prwatch.pr_size = buf->b_hdr->b_size; ctl.prwatch.pr_wflags = WA_WRITE; result = write(arc_procfd, &ctl, sizeof (ctl)); ASSERT3U(result, ==, sizeof (ctl)); } #endif } #endif /* illumos */ void arc_buf_thaw(arc_buf_t *buf) { if (zfs_flags & ZFS_DEBUG_MODIFY) { if (buf->b_hdr->b_state != arc_anon) panic("modifying non-anon buffer!"); if (buf->b_hdr->b_flags & ARC_IO_IN_PROGRESS) panic("modifying buffer while i/o in progress!"); arc_cksum_verify(buf); } mutex_enter(&buf->b_hdr->b_freeze_lock); if (buf->b_hdr->b_freeze_cksum != NULL) { kmem_free(buf->b_hdr->b_freeze_cksum, sizeof (zio_cksum_t)); buf->b_hdr->b_freeze_cksum = NULL; } if (zfs_flags & ZFS_DEBUG_MODIFY) { if (buf->b_hdr->b_thawed) kmem_free(buf->b_hdr->b_thawed, 1); buf->b_hdr->b_thawed = kmem_alloc(1, KM_SLEEP); } mutex_exit(&buf->b_hdr->b_freeze_lock); #ifdef illumos arc_buf_unwatch(buf); #endif /* illumos */ } void arc_buf_freeze(arc_buf_t *buf) { kmutex_t *hash_lock; if (!(zfs_flags & ZFS_DEBUG_MODIFY)) return; hash_lock = HDR_LOCK(buf->b_hdr); mutex_enter(hash_lock); ASSERT(buf->b_hdr->b_freeze_cksum != NULL || buf->b_hdr->b_state == arc_anon); arc_cksum_compute(buf, B_FALSE); mutex_exit(hash_lock); } static void get_buf_info(arc_buf_hdr_t *ab, arc_state_t *state, list_t **list, kmutex_t **lock) { uint64_t buf_hashid = buf_hash(ab->b_spa, &ab->b_dva, ab->b_birth); if (ab->b_type == ARC_BUFC_METADATA) buf_hashid &= (ARC_BUFC_NUMMETADATALISTS - 1); else { buf_hashid &= (ARC_BUFC_NUMDATALISTS - 1); buf_hashid += ARC_BUFC_NUMMETADATALISTS; } *list = &state->arcs_lists[buf_hashid]; *lock = ARCS_LOCK(state, buf_hashid); } static void add_reference(arc_buf_hdr_t *ab, kmutex_t *hash_lock, void *tag) { ASSERT(MUTEX_HELD(hash_lock)); if ((refcount_add(&ab->b_refcnt, tag) == 1) && (ab->b_state != arc_anon)) { uint64_t delta = ab->b_size * ab->b_datacnt; uint64_t *size = &ab->b_state->arcs_lsize[ab->b_type]; list_t *list; kmutex_t *lock; get_buf_info(ab, ab->b_state, &list, &lock); ASSERT(!MUTEX_HELD(lock)); mutex_enter(lock); ASSERT(list_link_active(&ab->b_arc_node)); list_remove(list, ab); if (GHOST_STATE(ab->b_state)) { ASSERT0(ab->b_datacnt); ASSERT3P(ab->b_buf, ==, NULL); delta = ab->b_size; } ASSERT(delta > 0); ASSERT3U(*size, >=, delta); atomic_add_64(size, -delta); mutex_exit(lock); /* remove the prefetch flag if we get a reference */ if (ab->b_flags & ARC_PREFETCH) ab->b_flags &= ~ARC_PREFETCH; } } static int remove_reference(arc_buf_hdr_t *ab, kmutex_t *hash_lock, void *tag) { int cnt; arc_state_t *state = ab->b_state; ASSERT(state == arc_anon || MUTEX_HELD(hash_lock)); ASSERT(!GHOST_STATE(state)); if (((cnt = refcount_remove(&ab->b_refcnt, tag)) == 0) && (state != arc_anon)) { uint64_t *size = &state->arcs_lsize[ab->b_type]; list_t *list; kmutex_t *lock; get_buf_info(ab, state, &list, &lock); ASSERT(!MUTEX_HELD(lock)); mutex_enter(lock); ASSERT(!list_link_active(&ab->b_arc_node)); list_insert_head(list, ab); ASSERT(ab->b_datacnt > 0); atomic_add_64(size, ab->b_size * ab->b_datacnt); mutex_exit(lock); } return (cnt); } /* * Move the supplied buffer to the indicated state. The mutex * for the buffer must be held by the caller. */ static void arc_change_state(arc_state_t *new_state, arc_buf_hdr_t *ab, kmutex_t *hash_lock) { arc_state_t *old_state = ab->b_state; int64_t refcnt = refcount_count(&ab->b_refcnt); uint64_t from_delta, to_delta; list_t *list; kmutex_t *lock; ASSERT(MUTEX_HELD(hash_lock)); ASSERT3P(new_state, !=, old_state); ASSERT(refcnt == 0 || ab->b_datacnt > 0); ASSERT(ab->b_datacnt == 0 || !GHOST_STATE(new_state)); ASSERT(ab->b_datacnt <= 1 || old_state != arc_anon); from_delta = to_delta = ab->b_datacnt * ab->b_size; /* * If this buffer is evictable, transfer it from the * old state list to the new state list. */ if (refcnt == 0) { if (old_state != arc_anon) { int use_mutex; uint64_t *size = &old_state->arcs_lsize[ab->b_type]; get_buf_info(ab, old_state, &list, &lock); use_mutex = !MUTEX_HELD(lock); if (use_mutex) mutex_enter(lock); ASSERT(list_link_active(&ab->b_arc_node)); list_remove(list, ab); /* * If prefetching out of the ghost cache, * we will have a non-zero datacnt. */ if (GHOST_STATE(old_state) && ab->b_datacnt == 0) { /* ghost elements have a ghost size */ ASSERT(ab->b_buf == NULL); from_delta = ab->b_size; } ASSERT3U(*size, >=, from_delta); atomic_add_64(size, -from_delta); if (use_mutex) mutex_exit(lock); } if (new_state != arc_anon) { int use_mutex; uint64_t *size = &new_state->arcs_lsize[ab->b_type]; get_buf_info(ab, new_state, &list, &lock); use_mutex = !MUTEX_HELD(lock); if (use_mutex) mutex_enter(lock); list_insert_head(list, ab); /* ghost elements have a ghost size */ if (GHOST_STATE(new_state)) { ASSERT(ab->b_datacnt == 0); ASSERT(ab->b_buf == NULL); to_delta = ab->b_size; } atomic_add_64(size, to_delta); if (use_mutex) mutex_exit(lock); } } ASSERT(!BUF_EMPTY(ab)); if (new_state == arc_anon && HDR_IN_HASH_TABLE(ab)) buf_hash_remove(ab); /* adjust state sizes */ if (to_delta) atomic_add_64(&new_state->arcs_size, to_delta); if (from_delta) { ASSERT3U(old_state->arcs_size, >=, from_delta); atomic_add_64(&old_state->arcs_size, -from_delta); } ab->b_state = new_state; /* adjust l2arc hdr stats */ if (new_state == arc_l2c_only) l2arc_hdr_stat_add(); else if (old_state == arc_l2c_only) l2arc_hdr_stat_remove(); } void arc_space_consume(uint64_t space, arc_space_type_t type) { ASSERT(type >= 0 && type < ARC_SPACE_NUMTYPES); switch (type) { case ARC_SPACE_DATA: ARCSTAT_INCR(arcstat_data_size, space); break; case ARC_SPACE_OTHER: ARCSTAT_INCR(arcstat_other_size, space); break; case ARC_SPACE_HDRS: ARCSTAT_INCR(arcstat_hdr_size, space); break; case ARC_SPACE_L2HDRS: ARCSTAT_INCR(arcstat_l2_hdr_size, space); break; } - atomic_add_64(&arc_meta_used, space); + ARCSTAT_INCR(arcstat_meta_used, space); atomic_add_64(&arc_size, space); } void arc_space_return(uint64_t space, arc_space_type_t type) { ASSERT(type >= 0 && type < ARC_SPACE_NUMTYPES); switch (type) { case ARC_SPACE_DATA: ARCSTAT_INCR(arcstat_data_size, -space); break; case ARC_SPACE_OTHER: ARCSTAT_INCR(arcstat_other_size, -space); break; case ARC_SPACE_HDRS: ARCSTAT_INCR(arcstat_hdr_size, -space); break; case ARC_SPACE_L2HDRS: ARCSTAT_INCR(arcstat_l2_hdr_size, -space); break; } ASSERT(arc_meta_used >= space); if (arc_meta_max < arc_meta_used) arc_meta_max = arc_meta_used; - atomic_add_64(&arc_meta_used, -space); + ARCSTAT_INCR(arcstat_meta_used, -space); ASSERT(arc_size >= space); atomic_add_64(&arc_size, -space); } arc_buf_t * arc_buf_alloc(spa_t *spa, int size, void *tag, arc_buf_contents_t type) { arc_buf_hdr_t *hdr; arc_buf_t *buf; ASSERT3U(size, >, 0); hdr = kmem_cache_alloc(hdr_cache, KM_PUSHPAGE); ASSERT(BUF_EMPTY(hdr)); hdr->b_size = size; hdr->b_type = type; hdr->b_spa = spa_load_guid(spa); hdr->b_state = arc_anon; hdr->b_arc_access = 0; buf = kmem_cache_alloc(buf_cache, KM_PUSHPAGE); buf->b_hdr = hdr; buf->b_data = NULL; buf->b_efunc = NULL; buf->b_private = NULL; buf->b_next = NULL; hdr->b_buf = buf; arc_get_data_buf(buf); hdr->b_datacnt = 1; hdr->b_flags = 0; ASSERT(refcount_is_zero(&hdr->b_refcnt)); (void) refcount_add(&hdr->b_refcnt, tag); return (buf); } static char *arc_onloan_tag = "onloan"; /* * Loan out an anonymous arc buffer. Loaned buffers are not counted as in * flight data by arc_tempreserve_space() until they are "returned". Loaned * buffers must be returned to the arc before they can be used by the DMU or * freed. */ arc_buf_t * arc_loan_buf(spa_t *spa, int size) { arc_buf_t *buf; buf = arc_buf_alloc(spa, size, arc_onloan_tag, ARC_BUFC_DATA); atomic_add_64(&arc_loaned_bytes, size); return (buf); } /* * Return a loaned arc buffer to the arc. */ void arc_return_buf(arc_buf_t *buf, void *tag) { arc_buf_hdr_t *hdr = buf->b_hdr; ASSERT(buf->b_data != NULL); (void) refcount_add(&hdr->b_refcnt, tag); (void) refcount_remove(&hdr->b_refcnt, arc_onloan_tag); atomic_add_64(&arc_loaned_bytes, -hdr->b_size); } /* Detach an arc_buf from a dbuf (tag) */ void arc_loan_inuse_buf(arc_buf_t *buf, void *tag) { arc_buf_hdr_t *hdr; ASSERT(buf->b_data != NULL); hdr = buf->b_hdr; (void) refcount_add(&hdr->b_refcnt, arc_onloan_tag); (void) refcount_remove(&hdr->b_refcnt, tag); buf->b_efunc = NULL; buf->b_private = NULL; atomic_add_64(&arc_loaned_bytes, hdr->b_size); } static arc_buf_t * arc_buf_clone(arc_buf_t *from) { arc_buf_t *buf; arc_buf_hdr_t *hdr = from->b_hdr; uint64_t size = hdr->b_size; ASSERT(hdr->b_state != arc_anon); buf = kmem_cache_alloc(buf_cache, KM_PUSHPAGE); buf->b_hdr = hdr; buf->b_data = NULL; buf->b_efunc = NULL; buf->b_private = NULL; buf->b_next = hdr->b_buf; hdr->b_buf = buf; arc_get_data_buf(buf); bcopy(from->b_data, buf->b_data, size); /* * This buffer already exists in the arc so create a duplicate * copy for the caller. If the buffer is associated with user data * then track the size and number of duplicates. These stats will be * updated as duplicate buffers are created and destroyed. */ if (hdr->b_type == ARC_BUFC_DATA) { ARCSTAT_BUMP(arcstat_duplicate_buffers); ARCSTAT_INCR(arcstat_duplicate_buffers_size, size); } hdr->b_datacnt += 1; return (buf); } void arc_buf_add_ref(arc_buf_t *buf, void* tag) { arc_buf_hdr_t *hdr; kmutex_t *hash_lock; /* * Check to see if this buffer is evicted. Callers * must verify b_data != NULL to know if the add_ref * was successful. */ mutex_enter(&buf->b_evict_lock); if (buf->b_data == NULL) { mutex_exit(&buf->b_evict_lock); return; } hash_lock = HDR_LOCK(buf->b_hdr); mutex_enter(hash_lock); hdr = buf->b_hdr; ASSERT3P(hash_lock, ==, HDR_LOCK(hdr)); mutex_exit(&buf->b_evict_lock); ASSERT(hdr->b_state == arc_mru || hdr->b_state == arc_mfu); add_reference(hdr, hash_lock, tag); DTRACE_PROBE1(arc__hit, arc_buf_hdr_t *, hdr); arc_access(hdr, hash_lock); mutex_exit(hash_lock); ARCSTAT_BUMP(arcstat_hits); ARCSTAT_CONDSTAT(!(hdr->b_flags & ARC_PREFETCH), demand, prefetch, hdr->b_type != ARC_BUFC_METADATA, data, metadata, hits); } static void arc_buf_free_on_write(void *data, size_t size, void (*free_func)(void *, size_t)) { l2arc_data_free_t *df; df = kmem_alloc(sizeof (l2arc_data_free_t), KM_SLEEP); df->l2df_data = data; df->l2df_size = size; df->l2df_func = free_func; mutex_enter(&l2arc_free_on_write_mtx); list_insert_head(l2arc_free_on_write, df); mutex_exit(&l2arc_free_on_write_mtx); } /* * Free the arc data buffer. If it is an l2arc write in progress, * the buffer is placed on l2arc_free_on_write to be freed later. */ static void arc_buf_data_free(arc_buf_t *buf, void (*free_func)(void *, size_t)) { arc_buf_hdr_t *hdr = buf->b_hdr; if (HDR_L2_WRITING(hdr)) { arc_buf_free_on_write(buf->b_data, hdr->b_size, free_func); ARCSTAT_BUMP(arcstat_l2_free_on_write); } else { free_func(buf->b_data, hdr->b_size); } } /* * Free up buf->b_data and if 'remove' is set, then pull the * arc_buf_t off of the the arc_buf_hdr_t's list and free it. */ static void arc_buf_l2_cdata_free(arc_buf_hdr_t *hdr) { l2arc_buf_hdr_t *l2hdr = hdr->b_l2hdr; ASSERT(MUTEX_HELD(&l2arc_buflist_mtx)); if (l2hdr->b_tmp_cdata == NULL) return; ASSERT(HDR_L2_WRITING(hdr)); arc_buf_free_on_write(l2hdr->b_tmp_cdata, hdr->b_size, zio_data_buf_free); ARCSTAT_BUMP(arcstat_l2_cdata_free_on_write); l2hdr->b_tmp_cdata = NULL; } static void arc_buf_destroy(arc_buf_t *buf, boolean_t recycle, boolean_t remove) { arc_buf_t **bufp; /* free up data associated with the buf */ if (buf->b_data) { arc_state_t *state = buf->b_hdr->b_state; uint64_t size = buf->b_hdr->b_size; arc_buf_contents_t type = buf->b_hdr->b_type; arc_cksum_verify(buf); #ifdef illumos arc_buf_unwatch(buf); #endif /* illumos */ if (!recycle) { if (type == ARC_BUFC_METADATA) { arc_buf_data_free(buf, zio_buf_free); arc_space_return(size, ARC_SPACE_DATA); } else { ASSERT(type == ARC_BUFC_DATA); arc_buf_data_free(buf, zio_data_buf_free); ARCSTAT_INCR(arcstat_data_size, -size); atomic_add_64(&arc_size, -size); } } if (list_link_active(&buf->b_hdr->b_arc_node)) { uint64_t *cnt = &state->arcs_lsize[type]; ASSERT(refcount_is_zero(&buf->b_hdr->b_refcnt)); ASSERT(state != arc_anon); ASSERT3U(*cnt, >=, size); atomic_add_64(cnt, -size); } ASSERT3U(state->arcs_size, >=, size); atomic_add_64(&state->arcs_size, -size); buf->b_data = NULL; /* * If we're destroying a duplicate buffer make sure * that the appropriate statistics are updated. */ if (buf->b_hdr->b_datacnt > 1 && buf->b_hdr->b_type == ARC_BUFC_DATA) { ARCSTAT_BUMPDOWN(arcstat_duplicate_buffers); ARCSTAT_INCR(arcstat_duplicate_buffers_size, -size); } ASSERT(buf->b_hdr->b_datacnt > 0); buf->b_hdr->b_datacnt -= 1; } /* only remove the buf if requested */ if (!remove) return; /* remove the buf from the hdr list */ for (bufp = &buf->b_hdr->b_buf; *bufp != buf; bufp = &(*bufp)->b_next) continue; *bufp = buf->b_next; buf->b_next = NULL; ASSERT(buf->b_efunc == NULL); /* clean up the buf */ buf->b_hdr = NULL; kmem_cache_free(buf_cache, buf); } static void arc_hdr_destroy(arc_buf_hdr_t *hdr) { ASSERT(refcount_is_zero(&hdr->b_refcnt)); ASSERT3P(hdr->b_state, ==, arc_anon); ASSERT(!HDR_IO_IN_PROGRESS(hdr)); l2arc_buf_hdr_t *l2hdr = hdr->b_l2hdr; if (l2hdr != NULL) { boolean_t buflist_held = MUTEX_HELD(&l2arc_buflist_mtx); /* * To prevent arc_free() and l2arc_evict() from * attempting to free the same buffer at the same time, * a FREE_IN_PROGRESS flag is given to arc_free() to * give it priority. l2arc_evict() can't destroy this * header while we are waiting on l2arc_buflist_mtx. * * The hdr may be removed from l2ad_buflist before we * grab l2arc_buflist_mtx, so b_l2hdr is rechecked. */ if (!buflist_held) { mutex_enter(&l2arc_buflist_mtx); l2hdr = hdr->b_l2hdr; } if (l2hdr != NULL) { trim_map_free(l2hdr->b_dev->l2ad_vdev, l2hdr->b_daddr, hdr->b_size, 0); list_remove(l2hdr->b_dev->l2ad_buflist, hdr); arc_buf_l2_cdata_free(hdr); ARCSTAT_INCR(arcstat_l2_size, -hdr->b_size); ARCSTAT_INCR(arcstat_l2_asize, -l2hdr->b_asize); vdev_space_update(l2hdr->b_dev->l2ad_vdev, -l2hdr->b_asize, 0, 0); kmem_free(l2hdr, sizeof (l2arc_buf_hdr_t)); if (hdr->b_state == arc_l2c_only) l2arc_hdr_stat_remove(); hdr->b_l2hdr = NULL; } if (!buflist_held) mutex_exit(&l2arc_buflist_mtx); } if (!BUF_EMPTY(hdr)) { ASSERT(!HDR_IN_HASH_TABLE(hdr)); buf_discard_identity(hdr); } while (hdr->b_buf) { arc_buf_t *buf = hdr->b_buf; if (buf->b_efunc) { mutex_enter(&arc_eviction_mtx); mutex_enter(&buf->b_evict_lock); ASSERT(buf->b_hdr != NULL); arc_buf_destroy(hdr->b_buf, FALSE, FALSE); hdr->b_buf = buf->b_next; buf->b_hdr = &arc_eviction_hdr; buf->b_next = arc_eviction_list; arc_eviction_list = buf; mutex_exit(&buf->b_evict_lock); mutex_exit(&arc_eviction_mtx); } else { arc_buf_destroy(hdr->b_buf, FALSE, TRUE); } } if (hdr->b_freeze_cksum != NULL) { kmem_free(hdr->b_freeze_cksum, sizeof (zio_cksum_t)); hdr->b_freeze_cksum = NULL; } if (hdr->b_thawed) { kmem_free(hdr->b_thawed, 1); hdr->b_thawed = NULL; } ASSERT(!list_link_active(&hdr->b_arc_node)); ASSERT3P(hdr->b_hash_next, ==, NULL); ASSERT3P(hdr->b_acb, ==, NULL); kmem_cache_free(hdr_cache, hdr); } void arc_buf_free(arc_buf_t *buf, void *tag) { arc_buf_hdr_t *hdr = buf->b_hdr; int hashed = hdr->b_state != arc_anon; ASSERT(buf->b_efunc == NULL); ASSERT(buf->b_data != NULL); if (hashed) { kmutex_t *hash_lock = HDR_LOCK(hdr); mutex_enter(hash_lock); hdr = buf->b_hdr; ASSERT3P(hash_lock, ==, HDR_LOCK(hdr)); (void) remove_reference(hdr, hash_lock, tag); if (hdr->b_datacnt > 1) { arc_buf_destroy(buf, FALSE, TRUE); } else { ASSERT(buf == hdr->b_buf); ASSERT(buf->b_efunc == NULL); hdr->b_flags |= ARC_BUF_AVAILABLE; } mutex_exit(hash_lock); } else if (HDR_IO_IN_PROGRESS(hdr)) { int destroy_hdr; /* * We are in the middle of an async write. Don't destroy * this buffer unless the write completes before we finish * decrementing the reference count. */ mutex_enter(&arc_eviction_mtx); (void) remove_reference(hdr, NULL, tag); ASSERT(refcount_is_zero(&hdr->b_refcnt)); destroy_hdr = !HDR_IO_IN_PROGRESS(hdr); mutex_exit(&arc_eviction_mtx); if (destroy_hdr) arc_hdr_destroy(hdr); } else { if (remove_reference(hdr, NULL, tag) > 0) arc_buf_destroy(buf, FALSE, TRUE); else arc_hdr_destroy(hdr); } } boolean_t arc_buf_remove_ref(arc_buf_t *buf, void* tag) { arc_buf_hdr_t *hdr = buf->b_hdr; kmutex_t *hash_lock = HDR_LOCK(hdr); boolean_t no_callback = (buf->b_efunc == NULL); if (hdr->b_state == arc_anon) { ASSERT(hdr->b_datacnt == 1); arc_buf_free(buf, tag); return (no_callback); } mutex_enter(hash_lock); hdr = buf->b_hdr; ASSERT3P(hash_lock, ==, HDR_LOCK(hdr)); ASSERT(hdr->b_state != arc_anon); ASSERT(buf->b_data != NULL); (void) remove_reference(hdr, hash_lock, tag); if (hdr->b_datacnt > 1) { if (no_callback) arc_buf_destroy(buf, FALSE, TRUE); } else if (no_callback) { ASSERT(hdr->b_buf == buf && buf->b_next == NULL); ASSERT(buf->b_efunc == NULL); hdr->b_flags |= ARC_BUF_AVAILABLE; } ASSERT(no_callback || hdr->b_datacnt > 1 || refcount_is_zero(&hdr->b_refcnt)); mutex_exit(hash_lock); return (no_callback); } int arc_buf_size(arc_buf_t *buf) { return (buf->b_hdr->b_size); } /* * Called from the DMU to determine if the current buffer should be * evicted. In order to ensure proper locking, the eviction must be initiated * from the DMU. Return true if the buffer is associated with user data and * duplicate buffers still exist. */ boolean_t arc_buf_eviction_needed(arc_buf_t *buf) { arc_buf_hdr_t *hdr; boolean_t evict_needed = B_FALSE; if (zfs_disable_dup_eviction) return (B_FALSE); mutex_enter(&buf->b_evict_lock); hdr = buf->b_hdr; if (hdr == NULL) { /* * We are in arc_do_user_evicts(); let that function * perform the eviction. */ ASSERT(buf->b_data == NULL); mutex_exit(&buf->b_evict_lock); return (B_FALSE); } else if (buf->b_data == NULL) { /* * We have already been added to the arc eviction list; * recommend eviction. */ ASSERT3P(hdr, ==, &arc_eviction_hdr); mutex_exit(&buf->b_evict_lock); return (B_TRUE); } if (hdr->b_datacnt > 1 && hdr->b_type == ARC_BUFC_DATA) evict_needed = B_TRUE; mutex_exit(&buf->b_evict_lock); return (evict_needed); } /* * Evict buffers from list until we've removed the specified number of * bytes. Move the removed buffers to the appropriate evict state. * If the recycle flag is set, then attempt to "recycle" a buffer: * - look for a buffer to evict that is `bytes' long. * - return the data block from this buffer rather than freeing it. * This flag is used by callers that are trying to make space for a * new buffer in a full arc cache. * * This function makes a "best effort". It skips over any buffers * it can't get a hash_lock on, and so may not catch all candidates. * It may also return without evicting as much space as requested. */ static void * arc_evict(arc_state_t *state, uint64_t spa, int64_t bytes, boolean_t recycle, arc_buf_contents_t type) { arc_state_t *evicted_state; uint64_t bytes_evicted = 0, skipped = 0, missed = 0; int64_t bytes_remaining; arc_buf_hdr_t *ab, *ab_prev = NULL; list_t *evicted_list, *list, *evicted_list_start, *list_start; kmutex_t *lock, *evicted_lock; kmutex_t *hash_lock; boolean_t have_lock; void *stolen = NULL; arc_buf_hdr_t marker = { 0 }; int count = 0; static int evict_metadata_offset, evict_data_offset; int i, idx, offset, list_count, lists; ASSERT(state == arc_mru || state == arc_mfu); evicted_state = (state == arc_mru) ? arc_mru_ghost : arc_mfu_ghost; if (type == ARC_BUFC_METADATA) { offset = 0; list_count = ARC_BUFC_NUMMETADATALISTS; list_start = &state->arcs_lists[0]; evicted_list_start = &evicted_state->arcs_lists[0]; idx = evict_metadata_offset; } else { offset = ARC_BUFC_NUMMETADATALISTS; list_start = &state->arcs_lists[offset]; evicted_list_start = &evicted_state->arcs_lists[offset]; list_count = ARC_BUFC_NUMDATALISTS; idx = evict_data_offset; } bytes_remaining = evicted_state->arcs_lsize[type]; lists = 0; evict_start: list = &list_start[idx]; evicted_list = &evicted_list_start[idx]; lock = ARCS_LOCK(state, (offset + idx)); evicted_lock = ARCS_LOCK(evicted_state, (offset + idx)); mutex_enter(lock); mutex_enter(evicted_lock); for (ab = list_tail(list); ab; ab = ab_prev) { ab_prev = list_prev(list, ab); bytes_remaining -= (ab->b_size * ab->b_datacnt); /* prefetch buffers have a minimum lifespan */ if (HDR_IO_IN_PROGRESS(ab) || (spa && ab->b_spa != spa) || (ab->b_flags & (ARC_PREFETCH|ARC_INDIRECT) && ddi_get_lbolt() - ab->b_arc_access < arc_min_prefetch_lifespan)) { skipped++; continue; } /* "lookahead" for better eviction candidate */ if (recycle && ab->b_size != bytes && ab_prev && ab_prev->b_size == bytes) continue; /* ignore markers */ if (ab->b_spa == 0) continue; /* * It may take a long time to evict all the bufs requested. * To avoid blocking all arc activity, periodically drop * the arcs_mtx and give other threads a chance to run * before reacquiring the lock. * * If we are looking for a buffer to recycle, we are in * the hot code path, so don't sleep. */ if (!recycle && count++ > arc_evict_iterations) { list_insert_after(list, ab, &marker); mutex_exit(evicted_lock); mutex_exit(lock); kpreempt(KPREEMPT_SYNC); mutex_enter(lock); mutex_enter(evicted_lock); ab_prev = list_prev(list, &marker); list_remove(list, &marker); count = 0; continue; } hash_lock = HDR_LOCK(ab); have_lock = MUTEX_HELD(hash_lock); if (have_lock || mutex_tryenter(hash_lock)) { ASSERT0(refcount_count(&ab->b_refcnt)); ASSERT(ab->b_datacnt > 0); while (ab->b_buf) { arc_buf_t *buf = ab->b_buf; if (!mutex_tryenter(&buf->b_evict_lock)) { missed += 1; break; } if (buf->b_data) { bytes_evicted += ab->b_size; if (recycle && ab->b_type == type && ab->b_size == bytes && !HDR_L2_WRITING(ab)) { stolen = buf->b_data; recycle = FALSE; } } if (buf->b_efunc) { mutex_enter(&arc_eviction_mtx); arc_buf_destroy(buf, buf->b_data == stolen, FALSE); ab->b_buf = buf->b_next; buf->b_hdr = &arc_eviction_hdr; buf->b_next = arc_eviction_list; arc_eviction_list = buf; mutex_exit(&arc_eviction_mtx); mutex_exit(&buf->b_evict_lock); } else { mutex_exit(&buf->b_evict_lock); arc_buf_destroy(buf, buf->b_data == stolen, TRUE); } } if (ab->b_l2hdr) { ARCSTAT_INCR(arcstat_evict_l2_cached, ab->b_size); } else { if (l2arc_write_eligible(ab->b_spa, ab)) { ARCSTAT_INCR(arcstat_evict_l2_eligible, ab->b_size); } else { ARCSTAT_INCR( arcstat_evict_l2_ineligible, ab->b_size); } } if (ab->b_datacnt == 0) { arc_change_state(evicted_state, ab, hash_lock); ASSERT(HDR_IN_HASH_TABLE(ab)); ab->b_flags |= ARC_IN_HASH_TABLE; ab->b_flags &= ~ARC_BUF_AVAILABLE; DTRACE_PROBE1(arc__evict, arc_buf_hdr_t *, ab); } if (!have_lock) mutex_exit(hash_lock); if (bytes >= 0 && bytes_evicted >= bytes) break; if (bytes_remaining > 0) { mutex_exit(evicted_lock); mutex_exit(lock); idx = ((idx + 1) & (list_count - 1)); lists++; goto evict_start; } } else { missed += 1; } } mutex_exit(evicted_lock); mutex_exit(lock); idx = ((idx + 1) & (list_count - 1)); lists++; if (bytes_evicted < bytes) { if (lists < list_count) goto evict_start; else dprintf("only evicted %lld bytes from %x", (longlong_t)bytes_evicted, state); } if (type == ARC_BUFC_METADATA) evict_metadata_offset = idx; else evict_data_offset = idx; if (skipped) ARCSTAT_INCR(arcstat_evict_skip, skipped); if (missed) ARCSTAT_INCR(arcstat_mutex_miss, missed); /* * Note: we have just evicted some data into the ghost state, * potentially putting the ghost size over the desired size. Rather * that evicting from the ghost list in this hot code path, leave * this chore to the arc_reclaim_thread(). */ if (stolen) ARCSTAT_BUMP(arcstat_stolen); return (stolen); } /* * Remove buffers from list until we've removed the specified number of * bytes. Destroy the buffers that are removed. */ static void arc_evict_ghost(arc_state_t *state, uint64_t spa, int64_t bytes) { arc_buf_hdr_t *ab, *ab_prev; arc_buf_hdr_t marker = { 0 }; list_t *list, *list_start; kmutex_t *hash_lock, *lock; uint64_t bytes_deleted = 0; uint64_t bufs_skipped = 0; int count = 0; static int evict_offset; int list_count, idx = evict_offset; int offset, lists = 0; ASSERT(GHOST_STATE(state)); /* * data lists come after metadata lists */ list_start = &state->arcs_lists[ARC_BUFC_NUMMETADATALISTS]; list_count = ARC_BUFC_NUMDATALISTS; offset = ARC_BUFC_NUMMETADATALISTS; evict_start: list = &list_start[idx]; lock = ARCS_LOCK(state, idx + offset); mutex_enter(lock); for (ab = list_tail(list); ab; ab = ab_prev) { ab_prev = list_prev(list, ab); if (ab->b_type > ARC_BUFC_NUMTYPES) panic("invalid ab=%p", (void *)ab); if (spa && ab->b_spa != spa) continue; /* ignore markers */ if (ab->b_spa == 0) continue; hash_lock = HDR_LOCK(ab); /* caller may be trying to modify this buffer, skip it */ if (MUTEX_HELD(hash_lock)) continue; /* * It may take a long time to evict all the bufs requested. * To avoid blocking all arc activity, periodically drop * the arcs_mtx and give other threads a chance to run * before reacquiring the lock. */ if (count++ > arc_evict_iterations) { list_insert_after(list, ab, &marker); mutex_exit(lock); kpreempt(KPREEMPT_SYNC); mutex_enter(lock); ab_prev = list_prev(list, &marker); list_remove(list, &marker); count = 0; continue; } if (mutex_tryenter(hash_lock)) { ASSERT(!HDR_IO_IN_PROGRESS(ab)); ASSERT(ab->b_buf == NULL); ARCSTAT_BUMP(arcstat_deleted); bytes_deleted += ab->b_size; if (ab->b_l2hdr != NULL) { /* * This buffer is cached on the 2nd Level ARC; * don't destroy the header. */ arc_change_state(arc_l2c_only, ab, hash_lock); mutex_exit(hash_lock); } else { arc_change_state(arc_anon, ab, hash_lock); mutex_exit(hash_lock); arc_hdr_destroy(ab); } DTRACE_PROBE1(arc__delete, arc_buf_hdr_t *, ab); if (bytes >= 0 && bytes_deleted >= bytes) break; } else if (bytes < 0) { /* * Insert a list marker and then wait for the * hash lock to become available. Once its * available, restart from where we left off. */ list_insert_after(list, ab, &marker); mutex_exit(lock); mutex_enter(hash_lock); mutex_exit(hash_lock); mutex_enter(lock); ab_prev = list_prev(list, &marker); list_remove(list, &marker); } else { bufs_skipped += 1; } } mutex_exit(lock); idx = ((idx + 1) & (ARC_BUFC_NUMDATALISTS - 1)); lists++; if (lists < list_count) goto evict_start; evict_offset = idx; if ((uintptr_t)list > (uintptr_t)&state->arcs_lists[ARC_BUFC_NUMMETADATALISTS] && (bytes < 0 || bytes_deleted < bytes)) { list_start = &state->arcs_lists[0]; list_count = ARC_BUFC_NUMMETADATALISTS; offset = lists = 0; goto evict_start; } if (bufs_skipped) { ARCSTAT_INCR(arcstat_mutex_miss, bufs_skipped); ASSERT(bytes >= 0); } if (bytes_deleted < bytes) dprintf("only deleted %lld bytes from %p", (longlong_t)bytes_deleted, state); } static void arc_adjust(void) { int64_t adjustment, delta; /* * Adjust MRU size */ adjustment = MIN((int64_t)(arc_size - arc_c), (int64_t)(arc_anon->arcs_size + arc_mru->arcs_size + arc_meta_used - arc_p)); if (adjustment > 0 && arc_mru->arcs_lsize[ARC_BUFC_DATA] > 0) { delta = MIN(arc_mru->arcs_lsize[ARC_BUFC_DATA], adjustment); (void) arc_evict(arc_mru, 0, delta, FALSE, ARC_BUFC_DATA); adjustment -= delta; } if (adjustment > 0 && arc_mru->arcs_lsize[ARC_BUFC_METADATA] > 0) { delta = MIN(arc_mru->arcs_lsize[ARC_BUFC_METADATA], adjustment); (void) arc_evict(arc_mru, 0, delta, FALSE, ARC_BUFC_METADATA); } /* * Adjust MFU size */ adjustment = arc_size - arc_c; if (adjustment > 0 && arc_mfu->arcs_lsize[ARC_BUFC_DATA] > 0) { delta = MIN(adjustment, arc_mfu->arcs_lsize[ARC_BUFC_DATA]); (void) arc_evict(arc_mfu, 0, delta, FALSE, ARC_BUFC_DATA); adjustment -= delta; } if (adjustment > 0 && arc_mfu->arcs_lsize[ARC_BUFC_METADATA] > 0) { int64_t delta = MIN(adjustment, arc_mfu->arcs_lsize[ARC_BUFC_METADATA]); (void) arc_evict(arc_mfu, 0, delta, FALSE, ARC_BUFC_METADATA); } /* * Adjust ghost lists */ adjustment = arc_mru->arcs_size + arc_mru_ghost->arcs_size - arc_c; if (adjustment > 0 && arc_mru_ghost->arcs_size > 0) { delta = MIN(arc_mru_ghost->arcs_size, adjustment); arc_evict_ghost(arc_mru_ghost, 0, delta); } adjustment = arc_mru_ghost->arcs_size + arc_mfu_ghost->arcs_size - arc_c; if (adjustment > 0 && arc_mfu_ghost->arcs_size > 0) { delta = MIN(arc_mfu_ghost->arcs_size, adjustment); arc_evict_ghost(arc_mfu_ghost, 0, delta); } } static void arc_do_user_evicts(void) { static arc_buf_t *tmp_arc_eviction_list; /* * Move list over to avoid LOR */ restart: mutex_enter(&arc_eviction_mtx); tmp_arc_eviction_list = arc_eviction_list; arc_eviction_list = NULL; mutex_exit(&arc_eviction_mtx); while (tmp_arc_eviction_list != NULL) { arc_buf_t *buf = tmp_arc_eviction_list; tmp_arc_eviction_list = buf->b_next; mutex_enter(&buf->b_evict_lock); buf->b_hdr = NULL; mutex_exit(&buf->b_evict_lock); if (buf->b_efunc != NULL) VERIFY0(buf->b_efunc(buf->b_private)); buf->b_efunc = NULL; buf->b_private = NULL; kmem_cache_free(buf_cache, buf); } if (arc_eviction_list != NULL) goto restart; } /* * Flush all *evictable* data from the cache for the given spa. * NOTE: this will not touch "active" (i.e. referenced) data. */ void arc_flush(spa_t *spa) { uint64_t guid = 0; if (spa) guid = spa_load_guid(spa); while (arc_mru->arcs_lsize[ARC_BUFC_DATA]) { (void) arc_evict(arc_mru, guid, -1, FALSE, ARC_BUFC_DATA); if (spa) break; } while (arc_mru->arcs_lsize[ARC_BUFC_METADATA]) { (void) arc_evict(arc_mru, guid, -1, FALSE, ARC_BUFC_METADATA); if (spa) break; } while (arc_mfu->arcs_lsize[ARC_BUFC_DATA]) { (void) arc_evict(arc_mfu, guid, -1, FALSE, ARC_BUFC_DATA); if (spa) break; } while (arc_mfu->arcs_lsize[ARC_BUFC_METADATA]) { (void) arc_evict(arc_mfu, guid, -1, FALSE, ARC_BUFC_METADATA); if (spa) break; } arc_evict_ghost(arc_mru_ghost, guid, -1); arc_evict_ghost(arc_mfu_ghost, guid, -1); mutex_enter(&arc_reclaim_thr_lock); arc_do_user_evicts(); mutex_exit(&arc_reclaim_thr_lock); ASSERT(spa || arc_eviction_list == NULL); } void arc_shrink(void) { if (arc_c > arc_c_min) { uint64_t to_free; DTRACE_PROBE4(arc__shrink, uint64_t, arc_c, uint64_t, arc_c_min, uint64_t, arc_p, uint64_t, to_free); #ifdef _KERNEL to_free = arc_c >> arc_shrink_shift; #else to_free = arc_c >> arc_shrink_shift; #endif if (arc_c > arc_c_min + to_free) atomic_add_64(&arc_c, -to_free); else arc_c = arc_c_min; atomic_add_64(&arc_p, -(arc_p >> arc_shrink_shift)); if (arc_c > arc_size) arc_c = MAX(arc_size, arc_c_min); if (arc_p > arc_c) arc_p = (arc_c >> 1); DTRACE_PROBE2(arc__shrunk, uint64_t, arc_c, uint64_t, arc_p); ASSERT(arc_c >= arc_c_min); ASSERT((int64_t)arc_p >= 0); } if (arc_size > arc_c) { DTRACE_PROBE2(arc__shrink_adjust, uint64_t, arc_size, uint64_t, arc_c); arc_adjust(); } } static int needfree = 0; static int arc_reclaim_needed(void) { #ifdef _KERNEL if (needfree) { DTRACE_PROBE(arc__reclaim_needfree); return (1); } /* * Cooperate with pagedaemon when it's time for it to scan * and reclaim some pages. */ if (freemem < zfs_arc_free_target) { DTRACE_PROBE2(arc__reclaim_freemem, uint64_t, freemem, uint64_t, zfs_arc_free_target); return (1); } #ifdef sun /* * take 'desfree' extra pages, so we reclaim sooner, rather than later */ extra = desfree; /* * check that we're out of range of the pageout scanner. It starts to * schedule paging if freemem is less than lotsfree and needfree. * lotsfree is the high-water mark for pageout, and needfree is the * number of needed free pages. We add extra pages here to make sure * the scanner doesn't start up while we're freeing memory. */ if (freemem < lotsfree + needfree + extra) return (1); /* * check to make sure that swapfs has enough space so that anon * reservations can still succeed. anon_resvmem() checks that the * availrmem is greater than swapfs_minfree, and the number of reserved * swap pages. We also add a bit of extra here just to prevent * circumstances from getting really dire. */ if (availrmem < swapfs_minfree + swapfs_reserve + extra) return (1); /* * Check that we have enough availrmem that memory locking (e.g., via * mlock(3C) or memcntl(2)) can still succeed. (pages_pp_maximum * stores the number of pages that cannot be locked; when availrmem * drops below pages_pp_maximum, page locking mechanisms such as * page_pp_lock() will fail.) */ if (availrmem <= pages_pp_maximum) return (1); #endif /* sun */ #if defined(__i386) || !defined(UMA_MD_SMALL_ALLOC) /* * If we're on an i386 platform, it's possible that we'll exhaust the * kernel heap space before we ever run out of available physical * memory. Most checks of the size of the heap_area compare against * tune.t_minarmem, which is the minimum available real memory that we * can have in the system. However, this is generally fixed at 25 pages * which is so low that it's useless. In this comparison, we seek to * calculate the total heap-size, and reclaim if more than 3/4ths of the * heap is allocated. (Or, in the calculation, if less than 1/4th is * free) */ if (vmem_size(heap_arena, VMEM_FREE) < (vmem_size(heap_arena, VMEM_FREE | VMEM_ALLOC) >> 2)) { DTRACE_PROBE2(arc__reclaim_used, uint64_t, vmem_size(heap_arena, VMEM_FREE), uint64_t, (vmem_size(heap_arena, VMEM_FREE | VMEM_ALLOC)) >> 2); return (1); } #endif #ifdef sun /* * If zio data pages are being allocated out of a separate heap segment, * then enforce that the size of available vmem for this arena remains * above about 1/16th free. * * Note: The 1/16th arena free requirement was put in place * to aggressively evict memory from the arc in order to avoid * memory fragmentation issues. */ if (zio_arena != NULL && vmem_size(zio_arena, VMEM_FREE) < (vmem_size(zio_arena, VMEM_ALLOC) >> 4)) return (1); #endif /* sun */ #else /* _KERNEL */ if (spa_get_random(100) == 0) return (1); #endif /* _KERNEL */ DTRACE_PROBE(arc__reclaim_no); return (0); } extern kmem_cache_t *zio_buf_cache[]; extern kmem_cache_t *zio_data_buf_cache[]; extern kmem_cache_t *range_seg_cache; static void __noinline arc_kmem_reap_now(arc_reclaim_strategy_t strat) { size_t i; kmem_cache_t *prev_cache = NULL; kmem_cache_t *prev_data_cache = NULL; DTRACE_PROBE(arc__kmem_reap_start); #ifdef _KERNEL if (arc_meta_used >= arc_meta_limit) { /* * We are exceeding our meta-data cache limit. * Purge some DNLC entries to release holds on meta-data. */ dnlc_reduce_cache((void *)(uintptr_t)arc_reduce_dnlc_percent); } #if defined(__i386) /* * Reclaim unused memory from all kmem caches. */ kmem_reap(); #endif #endif /* * An aggressive reclamation will shrink the cache size as well as * reap free buffers from the arc kmem caches. */ if (strat == ARC_RECLAIM_AGGR) arc_shrink(); for (i = 0; i < SPA_MAXBLOCKSIZE >> SPA_MINBLOCKSHIFT; i++) { if (zio_buf_cache[i] != prev_cache) { prev_cache = zio_buf_cache[i]; kmem_cache_reap_now(zio_buf_cache[i]); } if (zio_data_buf_cache[i] != prev_data_cache) { prev_data_cache = zio_data_buf_cache[i]; kmem_cache_reap_now(zio_data_buf_cache[i]); } } kmem_cache_reap_now(buf_cache); kmem_cache_reap_now(hdr_cache); kmem_cache_reap_now(range_seg_cache); #ifdef sun /* * Ask the vmem arena to reclaim unused memory from its * quantum caches. */ if (zio_arena != NULL && strat == ARC_RECLAIM_AGGR) vmem_qcache_reap(zio_arena); #endif DTRACE_PROBE(arc__kmem_reap_end); } static void arc_reclaim_thread(void *dummy __unused) { clock_t growtime = 0; arc_reclaim_strategy_t last_reclaim = ARC_RECLAIM_CONS; callb_cpr_t cpr; CALLB_CPR_INIT(&cpr, &arc_reclaim_thr_lock, callb_generic_cpr, FTAG); mutex_enter(&arc_reclaim_thr_lock); while (arc_thread_exit == 0) { if (arc_reclaim_needed()) { if (arc_no_grow) { if (last_reclaim == ARC_RECLAIM_CONS) { DTRACE_PROBE(arc__reclaim_aggr_no_grow); last_reclaim = ARC_RECLAIM_AGGR; } else { last_reclaim = ARC_RECLAIM_CONS; } } else { arc_no_grow = TRUE; last_reclaim = ARC_RECLAIM_AGGR; DTRACE_PROBE(arc__reclaim_aggr); membar_producer(); } /* reset the growth delay for every reclaim */ growtime = ddi_get_lbolt() + (arc_grow_retry * hz); if (needfree && last_reclaim == ARC_RECLAIM_CONS) { /* * If needfree is TRUE our vm_lowmem hook * was called and in that case we must free some * memory, so switch to aggressive mode. */ arc_no_grow = TRUE; last_reclaim = ARC_RECLAIM_AGGR; } arc_kmem_reap_now(last_reclaim); arc_warm = B_TRUE; } else if (arc_no_grow && ddi_get_lbolt() >= growtime) { arc_no_grow = FALSE; } arc_adjust(); if (arc_eviction_list != NULL) arc_do_user_evicts(); #ifdef _KERNEL if (needfree) { needfree = 0; wakeup(&needfree); } #endif /* block until needed, or one second, whichever is shorter */ CALLB_CPR_SAFE_BEGIN(&cpr); (void) cv_timedwait(&arc_reclaim_thr_cv, &arc_reclaim_thr_lock, hz); CALLB_CPR_SAFE_END(&cpr, &arc_reclaim_thr_lock); } arc_thread_exit = 0; cv_broadcast(&arc_reclaim_thr_cv); CALLB_CPR_EXIT(&cpr); /* drops arc_reclaim_thr_lock */ thread_exit(); } /* * Adapt arc info given the number of bytes we are trying to add and * the state that we are comming from. This function is only called * when we are adding new content to the cache. */ static void arc_adapt(int bytes, arc_state_t *state) { int mult; uint64_t arc_p_min = (arc_c >> arc_p_min_shift); if (state == arc_l2c_only) return; ASSERT(bytes > 0); /* * Adapt the target size of the MRU list: * - if we just hit in the MRU ghost list, then increase * the target size of the MRU list. * - if we just hit in the MFU ghost list, then increase * the target size of the MFU list by decreasing the * target size of the MRU list. */ if (state == arc_mru_ghost) { mult = ((arc_mru_ghost->arcs_size >= arc_mfu_ghost->arcs_size) ? 1 : (arc_mfu_ghost->arcs_size/arc_mru_ghost->arcs_size)); mult = MIN(mult, 10); /* avoid wild arc_p adjustment */ arc_p = MIN(arc_c - arc_p_min, arc_p + bytes * mult); } else if (state == arc_mfu_ghost) { uint64_t delta; mult = ((arc_mfu_ghost->arcs_size >= arc_mru_ghost->arcs_size) ? 1 : (arc_mru_ghost->arcs_size/arc_mfu_ghost->arcs_size)); mult = MIN(mult, 10); delta = MIN(bytes * mult, arc_p); arc_p = MAX(arc_p_min, arc_p - delta); } ASSERT((int64_t)arc_p >= 0); if (arc_reclaim_needed()) { cv_signal(&arc_reclaim_thr_cv); return; } if (arc_no_grow) return; if (arc_c >= arc_c_max) return; /* * If we're within (2 * maxblocksize) bytes of the target * cache size, increment the target cache size */ if (arc_size > arc_c - (2ULL << SPA_MAXBLOCKSHIFT)) { DTRACE_PROBE1(arc__inc_adapt, int, bytes); atomic_add_64(&arc_c, (int64_t)bytes); if (arc_c > arc_c_max) arc_c = arc_c_max; else if (state == arc_anon) atomic_add_64(&arc_p, (int64_t)bytes); if (arc_p > arc_c) arc_p = arc_c; } ASSERT((int64_t)arc_p >= 0); } /* * Check if the cache has reached its limits and eviction is required * prior to insert. */ static int arc_evict_needed(arc_buf_contents_t type) { if (type == ARC_BUFC_METADATA && arc_meta_used >= arc_meta_limit) return (1); if (arc_reclaim_needed()) return (1); return (arc_size > arc_c); } /* * The buffer, supplied as the first argument, needs a data block. * So, if we are at cache max, determine which cache should be victimized. * We have the following cases: * * 1. Insert for MRU, p > sizeof(arc_anon + arc_mru) -> * In this situation if we're out of space, but the resident size of the MFU is * under the limit, victimize the MFU cache to satisfy this insertion request. * * 2. Insert for MRU, p <= sizeof(arc_anon + arc_mru) -> * Here, we've used up all of the available space for the MRU, so we need to * evict from our own cache instead. Evict from the set of resident MRU * entries. * * 3. Insert for MFU (c - p) > sizeof(arc_mfu) -> * c minus p represents the MFU space in the cache, since p is the size of the * cache that is dedicated to the MRU. In this situation there's still space on * the MFU side, so the MRU side needs to be victimized. * * 4. Insert for MFU (c - p) < sizeof(arc_mfu) -> * MFU's resident set is consuming more space than it has been allotted. In * this situation, we must victimize our own cache, the MFU, for this insertion. */ static void arc_get_data_buf(arc_buf_t *buf) { arc_state_t *state = buf->b_hdr->b_state; uint64_t size = buf->b_hdr->b_size; arc_buf_contents_t type = buf->b_hdr->b_type; arc_adapt(size, state); /* * We have not yet reached cache maximum size, * just allocate a new buffer. */ if (!arc_evict_needed(type)) { if (type == ARC_BUFC_METADATA) { buf->b_data = zio_buf_alloc(size); arc_space_consume(size, ARC_SPACE_DATA); } else { ASSERT(type == ARC_BUFC_DATA); buf->b_data = zio_data_buf_alloc(size); ARCSTAT_INCR(arcstat_data_size, size); atomic_add_64(&arc_size, size); } goto out; } /* * If we are prefetching from the mfu ghost list, this buffer * will end up on the mru list; so steal space from there. */ if (state == arc_mfu_ghost) state = buf->b_hdr->b_flags & ARC_PREFETCH ? arc_mru : arc_mfu; else if (state == arc_mru_ghost) state = arc_mru; if (state == arc_mru || state == arc_anon) { uint64_t mru_used = arc_anon->arcs_size + arc_mru->arcs_size; state = (arc_mfu->arcs_lsize[type] >= size && arc_p > mru_used) ? arc_mfu : arc_mru; } else { /* MFU cases */ uint64_t mfu_space = arc_c - arc_p; state = (arc_mru->arcs_lsize[type] >= size && mfu_space > arc_mfu->arcs_size) ? arc_mru : arc_mfu; } if ((buf->b_data = arc_evict(state, 0, size, TRUE, type)) == NULL) { if (type == ARC_BUFC_METADATA) { buf->b_data = zio_buf_alloc(size); arc_space_consume(size, ARC_SPACE_DATA); } else { ASSERT(type == ARC_BUFC_DATA); buf->b_data = zio_data_buf_alloc(size); ARCSTAT_INCR(arcstat_data_size, size); atomic_add_64(&arc_size, size); } ARCSTAT_BUMP(arcstat_recycle_miss); } ASSERT(buf->b_data != NULL); out: /* * Update the state size. Note that ghost states have a * "ghost size" and so don't need to be updated. */ if (!GHOST_STATE(buf->b_hdr->b_state)) { arc_buf_hdr_t *hdr = buf->b_hdr; atomic_add_64(&hdr->b_state->arcs_size, size); if (list_link_active(&hdr->b_arc_node)) { ASSERT(refcount_is_zero(&hdr->b_refcnt)); atomic_add_64(&hdr->b_state->arcs_lsize[type], size); } /* * If we are growing the cache, and we are adding anonymous * data, and we have outgrown arc_p, update arc_p */ if (arc_size < arc_c && hdr->b_state == arc_anon && arc_anon->arcs_size + arc_mru->arcs_size > arc_p) arc_p = MIN(arc_c, arc_p + size); } ARCSTAT_BUMP(arcstat_allocated); } /* * This routine is called whenever a buffer is accessed. * NOTE: the hash lock is dropped in this function. */ static void arc_access(arc_buf_hdr_t *buf, kmutex_t *hash_lock) { clock_t now; ASSERT(MUTEX_HELD(hash_lock)); if (buf->b_state == arc_anon) { /* * This buffer is not in the cache, and does not * appear in our "ghost" list. Add the new buffer * to the MRU state. */ ASSERT(buf->b_arc_access == 0); buf->b_arc_access = ddi_get_lbolt(); DTRACE_PROBE1(new_state__mru, arc_buf_hdr_t *, buf); arc_change_state(arc_mru, buf, hash_lock); } else if (buf->b_state == arc_mru) { now = ddi_get_lbolt(); /* * If this buffer is here because of a prefetch, then either: * - clear the flag if this is a "referencing" read * (any subsequent access will bump this into the MFU state). * or * - move the buffer to the head of the list if this is * another prefetch (to make it less likely to be evicted). */ if ((buf->b_flags & ARC_PREFETCH) != 0) { if (refcount_count(&buf->b_refcnt) == 0) { ASSERT(list_link_active(&buf->b_arc_node)); } else { buf->b_flags &= ~ARC_PREFETCH; ARCSTAT_BUMP(arcstat_mru_hits); } buf->b_arc_access = now; return; } /* * This buffer has been "accessed" only once so far, * but it is still in the cache. Move it to the MFU * state. */ if (now > buf->b_arc_access + ARC_MINTIME) { /* * More than 125ms have passed since we * instantiated this buffer. Move it to the * most frequently used state. */ buf->b_arc_access = now; DTRACE_PROBE1(new_state__mfu, arc_buf_hdr_t *, buf); arc_change_state(arc_mfu, buf, hash_lock); } ARCSTAT_BUMP(arcstat_mru_hits); } else if (buf->b_state == arc_mru_ghost) { arc_state_t *new_state; /* * This buffer has been "accessed" recently, but * was evicted from the cache. Move it to the * MFU state. */ if (buf->b_flags & ARC_PREFETCH) { new_state = arc_mru; if (refcount_count(&buf->b_refcnt) > 0) buf->b_flags &= ~ARC_PREFETCH; DTRACE_PROBE1(new_state__mru, arc_buf_hdr_t *, buf); } else { new_state = arc_mfu; DTRACE_PROBE1(new_state__mfu, arc_buf_hdr_t *, buf); } buf->b_arc_access = ddi_get_lbolt(); arc_change_state(new_state, buf, hash_lock); ARCSTAT_BUMP(arcstat_mru_ghost_hits); } else if (buf->b_state == arc_mfu) { /* * This buffer has been accessed more than once and is * still in the cache. Keep it in the MFU state. * * NOTE: an add_reference() that occurred when we did * the arc_read() will have kicked this off the list. * If it was a prefetch, we will explicitly move it to * the head of the list now. */ if ((buf->b_flags & ARC_PREFETCH) != 0) { ASSERT(refcount_count(&buf->b_refcnt) == 0); ASSERT(list_link_active(&buf->b_arc_node)); } ARCSTAT_BUMP(arcstat_mfu_hits); buf->b_arc_access = ddi_get_lbolt(); } else if (buf->b_state == arc_mfu_ghost) { arc_state_t *new_state = arc_mfu; /* * This buffer has been accessed more than once but has * been evicted from the cache. Move it back to the * MFU state. */ if (buf->b_flags & ARC_PREFETCH) { /* * This is a prefetch access... * move this block back to the MRU state. */ ASSERT0(refcount_count(&buf->b_refcnt)); new_state = arc_mru; } buf->b_arc_access = ddi_get_lbolt(); DTRACE_PROBE1(new_state__mfu, arc_buf_hdr_t *, buf); arc_change_state(new_state, buf, hash_lock); ARCSTAT_BUMP(arcstat_mfu_ghost_hits); } else if (buf->b_state == arc_l2c_only) { /* * This buffer is on the 2nd Level ARC. */ buf->b_arc_access = ddi_get_lbolt(); DTRACE_PROBE1(new_state__mfu, arc_buf_hdr_t *, buf); arc_change_state(arc_mfu, buf, hash_lock); } else { ASSERT(!"invalid arc state"); } } /* a generic arc_done_func_t which you can use */ /* ARGSUSED */ void arc_bcopy_func(zio_t *zio, arc_buf_t *buf, void *arg) { if (zio == NULL || zio->io_error == 0) bcopy(buf->b_data, arg, buf->b_hdr->b_size); VERIFY(arc_buf_remove_ref(buf, arg)); } /* a generic arc_done_func_t */ void arc_getbuf_func(zio_t *zio, arc_buf_t *buf, void *arg) { arc_buf_t **bufp = arg; if (zio && zio->io_error) { VERIFY(arc_buf_remove_ref(buf, arg)); *bufp = NULL; } else { *bufp = buf; ASSERT(buf->b_data); } } static void arc_read_done(zio_t *zio) { arc_buf_hdr_t *hdr; arc_buf_t *buf; arc_buf_t *abuf; /* buffer we're assigning to callback */ kmutex_t *hash_lock = NULL; arc_callback_t *callback_list, *acb; int freeable = FALSE; buf = zio->io_private; hdr = buf->b_hdr; /* * The hdr was inserted into hash-table and removed from lists * prior to starting I/O. We should find this header, since * it's in the hash table, and it should be legit since it's * not possible to evict it during the I/O. The only possible * reason for it not to be found is if we were freed during the * read. */ if (HDR_IN_HASH_TABLE(hdr)) { ASSERT3U(hdr->b_birth, ==, BP_PHYSICAL_BIRTH(zio->io_bp)); ASSERT3U(hdr->b_dva.dva_word[0], ==, BP_IDENTITY(zio->io_bp)->dva_word[0]); ASSERT3U(hdr->b_dva.dva_word[1], ==, BP_IDENTITY(zio->io_bp)->dva_word[1]); arc_buf_hdr_t *found = buf_hash_find(hdr->b_spa, zio->io_bp, &hash_lock); ASSERT((found == NULL && HDR_FREED_IN_READ(hdr) && hash_lock == NULL) || (found == hdr && DVA_EQUAL(&hdr->b_dva, BP_IDENTITY(zio->io_bp))) || (found == hdr && HDR_L2_READING(hdr))); } hdr->b_flags &= ~ARC_L2_EVICTED; if (l2arc_noprefetch && (hdr->b_flags & ARC_PREFETCH)) hdr->b_flags &= ~ARC_L2CACHE; /* byteswap if necessary */ callback_list = hdr->b_acb; ASSERT(callback_list != NULL); if (BP_SHOULD_BYTESWAP(zio->io_bp) && zio->io_error == 0) { dmu_object_byteswap_t bswap = DMU_OT_BYTESWAP(BP_GET_TYPE(zio->io_bp)); arc_byteswap_func_t *func = BP_GET_LEVEL(zio->io_bp) > 0 ? byteswap_uint64_array : dmu_ot_byteswap[bswap].ob_func; func(buf->b_data, hdr->b_size); } arc_cksum_compute(buf, B_FALSE); #ifdef illumos arc_buf_watch(buf); #endif /* illumos */ if (hash_lock && zio->io_error == 0 && hdr->b_state == arc_anon) { /* * Only call arc_access on anonymous buffers. This is because * if we've issued an I/O for an evicted buffer, we've already * called arc_access (to prevent any simultaneous readers from * getting confused). */ arc_access(hdr, hash_lock); } /* create copies of the data buffer for the callers */ abuf = buf; for (acb = callback_list; acb; acb = acb->acb_next) { if (acb->acb_done) { if (abuf == NULL) { ARCSTAT_BUMP(arcstat_duplicate_reads); abuf = arc_buf_clone(buf); } acb->acb_buf = abuf; abuf = NULL; } } hdr->b_acb = NULL; hdr->b_flags &= ~ARC_IO_IN_PROGRESS; ASSERT(!HDR_BUF_AVAILABLE(hdr)); if (abuf == buf) { ASSERT(buf->b_efunc == NULL); ASSERT(hdr->b_datacnt == 1); hdr->b_flags |= ARC_BUF_AVAILABLE; } ASSERT(refcount_is_zero(&hdr->b_refcnt) || callback_list != NULL); if (zio->io_error != 0) { hdr->b_flags |= ARC_IO_ERROR; if (hdr->b_state != arc_anon) arc_change_state(arc_anon, hdr, hash_lock); if (HDR_IN_HASH_TABLE(hdr)) buf_hash_remove(hdr); freeable = refcount_is_zero(&hdr->b_refcnt); } /* * Broadcast before we drop the hash_lock to avoid the possibility * that the hdr (and hence the cv) might be freed before we get to * the cv_broadcast(). */ cv_broadcast(&hdr->b_cv); if (hash_lock) { mutex_exit(hash_lock); } else { /* * This block was freed while we waited for the read to * complete. It has been removed from the hash table and * moved to the anonymous state (so that it won't show up * in the cache). */ ASSERT3P(hdr->b_state, ==, arc_anon); freeable = refcount_is_zero(&hdr->b_refcnt); } /* execute each callback and free its structure */ while ((acb = callback_list) != NULL) { if (acb->acb_done) acb->acb_done(zio, acb->acb_buf, acb->acb_private); if (acb->acb_zio_dummy != NULL) { acb->acb_zio_dummy->io_error = zio->io_error; zio_nowait(acb->acb_zio_dummy); } callback_list = acb->acb_next; kmem_free(acb, sizeof (arc_callback_t)); } if (freeable) arc_hdr_destroy(hdr); } /* * "Read" the block block at the specified DVA (in bp) via the * cache. If the block is found in the cache, invoke the provided * callback immediately and return. Note that the `zio' parameter * in the callback will be NULL in this case, since no IO was * required. If the block is not in the cache pass the read request * on to the spa with a substitute callback function, so that the * requested block will be added to the cache. * * If a read request arrives for a block that has a read in-progress, * either wait for the in-progress read to complete (and return the * results); or, if this is a read with a "done" func, add a record * to the read to invoke the "done" func when the read completes, * and return; or just return. * * arc_read_done() will invoke all the requested "done" functions * for readers of this block. */ int arc_read(zio_t *pio, spa_t *spa, const blkptr_t *bp, arc_done_func_t *done, void *private, zio_priority_t priority, int zio_flags, uint32_t *arc_flags, const zbookmark_phys_t *zb) { arc_buf_hdr_t *hdr = NULL; arc_buf_t *buf = NULL; kmutex_t *hash_lock = NULL; zio_t *rzio; uint64_t guid = spa_load_guid(spa); ASSERT(!BP_IS_EMBEDDED(bp) || BPE_GET_ETYPE(bp) == BP_EMBEDDED_TYPE_DATA); top: if (!BP_IS_EMBEDDED(bp)) { /* * Embedded BP's have no DVA and require no I/O to "read". * Create an anonymous arc buf to back it. */ hdr = buf_hash_find(guid, bp, &hash_lock); } if (hdr != NULL && hdr->b_datacnt > 0) { *arc_flags |= ARC_CACHED; if (HDR_IO_IN_PROGRESS(hdr)) { if (*arc_flags & ARC_WAIT) { cv_wait(&hdr->b_cv, hash_lock); mutex_exit(hash_lock); goto top; } ASSERT(*arc_flags & ARC_NOWAIT); if (done) { arc_callback_t *acb = NULL; acb = kmem_zalloc(sizeof (arc_callback_t), KM_SLEEP); acb->acb_done = done; acb->acb_private = private; if (pio != NULL) acb->acb_zio_dummy = zio_null(pio, spa, NULL, NULL, NULL, zio_flags); ASSERT(acb->acb_done != NULL); acb->acb_next = hdr->b_acb; hdr->b_acb = acb; add_reference(hdr, hash_lock, private); mutex_exit(hash_lock); return (0); } mutex_exit(hash_lock); return (0); } ASSERT(hdr->b_state == arc_mru || hdr->b_state == arc_mfu); if (done) { add_reference(hdr, hash_lock, private); /* * If this block is already in use, create a new * copy of the data so that we will be guaranteed * that arc_release() will always succeed. */ buf = hdr->b_buf; ASSERT(buf); ASSERT(buf->b_data); if (HDR_BUF_AVAILABLE(hdr)) { ASSERT(buf->b_efunc == NULL); hdr->b_flags &= ~ARC_BUF_AVAILABLE; } else { buf = arc_buf_clone(buf); } } else if (*arc_flags & ARC_PREFETCH && refcount_count(&hdr->b_refcnt) == 0) { hdr->b_flags |= ARC_PREFETCH; } DTRACE_PROBE1(arc__hit, arc_buf_hdr_t *, hdr); arc_access(hdr, hash_lock); if (*arc_flags & ARC_L2CACHE) hdr->b_flags |= ARC_L2CACHE; if (*arc_flags & ARC_L2COMPRESS) hdr->b_flags |= ARC_L2COMPRESS; mutex_exit(hash_lock); ARCSTAT_BUMP(arcstat_hits); ARCSTAT_CONDSTAT(!(hdr->b_flags & ARC_PREFETCH), demand, prefetch, hdr->b_type != ARC_BUFC_METADATA, data, metadata, hits); if (done) done(NULL, buf, private); } else { uint64_t size = BP_GET_LSIZE(bp); arc_callback_t *acb; vdev_t *vd = NULL; uint64_t addr = 0; boolean_t devw = B_FALSE; enum zio_compress b_compress = ZIO_COMPRESS_OFF; uint64_t b_asize = 0; if (hdr == NULL) { /* this block is not in the cache */ arc_buf_hdr_t *exists = NULL; arc_buf_contents_t type = BP_GET_BUFC_TYPE(bp); buf = arc_buf_alloc(spa, size, private, type); hdr = buf->b_hdr; if (!BP_IS_EMBEDDED(bp)) { hdr->b_dva = *BP_IDENTITY(bp); hdr->b_birth = BP_PHYSICAL_BIRTH(bp); hdr->b_cksum0 = bp->blk_cksum.zc_word[0]; exists = buf_hash_insert(hdr, &hash_lock); } if (exists != NULL) { /* somebody beat us to the hash insert */ mutex_exit(hash_lock); buf_discard_identity(hdr); (void) arc_buf_remove_ref(buf, private); goto top; /* restart the IO request */ } /* if this is a prefetch, we don't have a reference */ if (*arc_flags & ARC_PREFETCH) { (void) remove_reference(hdr, hash_lock, private); hdr->b_flags |= ARC_PREFETCH; } if (*arc_flags & ARC_L2CACHE) hdr->b_flags |= ARC_L2CACHE; if (*arc_flags & ARC_L2COMPRESS) hdr->b_flags |= ARC_L2COMPRESS; if (BP_GET_LEVEL(bp) > 0) hdr->b_flags |= ARC_INDIRECT; } else { /* this block is in the ghost cache */ ASSERT(GHOST_STATE(hdr->b_state)); ASSERT(!HDR_IO_IN_PROGRESS(hdr)); ASSERT0(refcount_count(&hdr->b_refcnt)); ASSERT(hdr->b_buf == NULL); /* if this is a prefetch, we don't have a reference */ if (*arc_flags & ARC_PREFETCH) hdr->b_flags |= ARC_PREFETCH; else add_reference(hdr, hash_lock, private); if (*arc_flags & ARC_L2CACHE) hdr->b_flags |= ARC_L2CACHE; if (*arc_flags & ARC_L2COMPRESS) hdr->b_flags |= ARC_L2COMPRESS; buf = kmem_cache_alloc(buf_cache, KM_PUSHPAGE); buf->b_hdr = hdr; buf->b_data = NULL; buf->b_efunc = NULL; buf->b_private = NULL; buf->b_next = NULL; hdr->b_buf = buf; ASSERT(hdr->b_datacnt == 0); hdr->b_datacnt = 1; arc_get_data_buf(buf); arc_access(hdr, hash_lock); } ASSERT(!GHOST_STATE(hdr->b_state)); acb = kmem_zalloc(sizeof (arc_callback_t), KM_SLEEP); acb->acb_done = done; acb->acb_private = private; ASSERT(hdr->b_acb == NULL); hdr->b_acb = acb; hdr->b_flags |= ARC_IO_IN_PROGRESS; if (hdr->b_l2hdr != NULL && (vd = hdr->b_l2hdr->b_dev->l2ad_vdev) != NULL) { devw = hdr->b_l2hdr->b_dev->l2ad_writing; addr = hdr->b_l2hdr->b_daddr; b_compress = hdr->b_l2hdr->b_compress; b_asize = hdr->b_l2hdr->b_asize; /* * Lock out device removal. */ if (vdev_is_dead(vd) || !spa_config_tryenter(spa, SCL_L2ARC, vd, RW_READER)) vd = NULL; } if (hash_lock != NULL) mutex_exit(hash_lock); /* * At this point, we have a level 1 cache miss. Try again in * L2ARC if possible. */ ASSERT3U(hdr->b_size, ==, size); DTRACE_PROBE4(arc__miss, arc_buf_hdr_t *, hdr, blkptr_t *, bp, uint64_t, size, zbookmark_phys_t *, zb); ARCSTAT_BUMP(arcstat_misses); ARCSTAT_CONDSTAT(!(hdr->b_flags & ARC_PREFETCH), demand, prefetch, hdr->b_type != ARC_BUFC_METADATA, data, metadata, misses); #ifdef _KERNEL curthread->td_ru.ru_inblock++; #endif if (vd != NULL && l2arc_ndev != 0 && !(l2arc_norw && devw)) { /* * Read from the L2ARC if the following are true: * 1. The L2ARC vdev was previously cached. * 2. This buffer still has L2ARC metadata. * 3. This buffer isn't currently writing to the L2ARC. * 4. The L2ARC entry wasn't evicted, which may * also have invalidated the vdev. * 5. This isn't prefetch and l2arc_noprefetch is set. */ if (hdr->b_l2hdr != NULL && !HDR_L2_WRITING(hdr) && !HDR_L2_EVICTED(hdr) && !(l2arc_noprefetch && HDR_PREFETCH(hdr))) { l2arc_read_callback_t *cb; DTRACE_PROBE1(l2arc__hit, arc_buf_hdr_t *, hdr); ARCSTAT_BUMP(arcstat_l2_hits); cb = kmem_zalloc(sizeof (l2arc_read_callback_t), KM_SLEEP); cb->l2rcb_buf = buf; cb->l2rcb_spa = spa; cb->l2rcb_bp = *bp; cb->l2rcb_zb = *zb; cb->l2rcb_flags = zio_flags; cb->l2rcb_compress = b_compress; ASSERT(addr >= VDEV_LABEL_START_SIZE && addr + size < vd->vdev_psize - VDEV_LABEL_END_SIZE); /* * l2arc read. The SCL_L2ARC lock will be * released by l2arc_read_done(). * Issue a null zio if the underlying buffer * was squashed to zero size by compression. */ if (b_compress == ZIO_COMPRESS_EMPTY) { rzio = zio_null(pio, spa, vd, l2arc_read_done, cb, zio_flags | ZIO_FLAG_DONT_CACHE | ZIO_FLAG_CANFAIL | ZIO_FLAG_DONT_PROPAGATE | ZIO_FLAG_DONT_RETRY); } else { rzio = zio_read_phys(pio, vd, addr, b_asize, buf->b_data, ZIO_CHECKSUM_OFF, l2arc_read_done, cb, priority, zio_flags | ZIO_FLAG_DONT_CACHE | ZIO_FLAG_CANFAIL | ZIO_FLAG_DONT_PROPAGATE | ZIO_FLAG_DONT_RETRY, B_FALSE); } DTRACE_PROBE2(l2arc__read, vdev_t *, vd, zio_t *, rzio); ARCSTAT_INCR(arcstat_l2_read_bytes, b_asize); if (*arc_flags & ARC_NOWAIT) { zio_nowait(rzio); return (0); } ASSERT(*arc_flags & ARC_WAIT); if (zio_wait(rzio) == 0) return (0); /* l2arc read error; goto zio_read() */ } else { DTRACE_PROBE1(l2arc__miss, arc_buf_hdr_t *, hdr); ARCSTAT_BUMP(arcstat_l2_misses); if (HDR_L2_WRITING(hdr)) ARCSTAT_BUMP(arcstat_l2_rw_clash); spa_config_exit(spa, SCL_L2ARC, vd); } } else { if (vd != NULL) spa_config_exit(spa, SCL_L2ARC, vd); if (l2arc_ndev != 0) { DTRACE_PROBE1(l2arc__miss, arc_buf_hdr_t *, hdr); ARCSTAT_BUMP(arcstat_l2_misses); } } rzio = zio_read(pio, spa, bp, buf->b_data, size, arc_read_done, buf, priority, zio_flags, zb); if (*arc_flags & ARC_WAIT) return (zio_wait(rzio)); ASSERT(*arc_flags & ARC_NOWAIT); zio_nowait(rzio); } return (0); } void arc_set_callback(arc_buf_t *buf, arc_evict_func_t *func, void *private) { ASSERT(buf->b_hdr != NULL); ASSERT(buf->b_hdr->b_state != arc_anon); ASSERT(!refcount_is_zero(&buf->b_hdr->b_refcnt) || func == NULL); ASSERT(buf->b_efunc == NULL); ASSERT(!HDR_BUF_AVAILABLE(buf->b_hdr)); buf->b_efunc = func; buf->b_private = private; } /* * Notify the arc that a block was freed, and thus will never be used again. */ void arc_freed(spa_t *spa, const blkptr_t *bp) { arc_buf_hdr_t *hdr; kmutex_t *hash_lock; uint64_t guid = spa_load_guid(spa); ASSERT(!BP_IS_EMBEDDED(bp)); hdr = buf_hash_find(guid, bp, &hash_lock); if (hdr == NULL) return; if (HDR_BUF_AVAILABLE(hdr)) { arc_buf_t *buf = hdr->b_buf; add_reference(hdr, hash_lock, FTAG); hdr->b_flags &= ~ARC_BUF_AVAILABLE; mutex_exit(hash_lock); arc_release(buf, FTAG); (void) arc_buf_remove_ref(buf, FTAG); } else { mutex_exit(hash_lock); } } /* * Clear the user eviction callback set by arc_set_callback(), first calling * it if it exists. Because the presence of a callback keeps an arc_buf cached * clearing the callback may result in the arc_buf being destroyed. However, * it will not result in the *last* arc_buf being destroyed, hence the data * will remain cached in the ARC. We make a copy of the arc buffer here so * that we can process the callback without holding any locks. * * It's possible that the callback is already in the process of being cleared * by another thread. In this case we can not clear the callback. * * Returns B_TRUE if the callback was successfully called and cleared. */ boolean_t arc_clear_callback(arc_buf_t *buf) { arc_buf_hdr_t *hdr; kmutex_t *hash_lock; arc_evict_func_t *efunc = buf->b_efunc; void *private = buf->b_private; list_t *list, *evicted_list; kmutex_t *lock, *evicted_lock; mutex_enter(&buf->b_evict_lock); hdr = buf->b_hdr; if (hdr == NULL) { /* * We are in arc_do_user_evicts(). */ ASSERT(buf->b_data == NULL); mutex_exit(&buf->b_evict_lock); return (B_FALSE); } else if (buf->b_data == NULL) { /* * We are on the eviction list; process this buffer now * but let arc_do_user_evicts() do the reaping. */ buf->b_efunc = NULL; mutex_exit(&buf->b_evict_lock); VERIFY0(efunc(private)); return (B_TRUE); } hash_lock = HDR_LOCK(hdr); mutex_enter(hash_lock); hdr = buf->b_hdr; ASSERT3P(hash_lock, ==, HDR_LOCK(hdr)); ASSERT3U(refcount_count(&hdr->b_refcnt), <, hdr->b_datacnt); ASSERT(hdr->b_state == arc_mru || hdr->b_state == arc_mfu); buf->b_efunc = NULL; buf->b_private = NULL; if (hdr->b_datacnt > 1) { mutex_exit(&buf->b_evict_lock); arc_buf_destroy(buf, FALSE, TRUE); } else { ASSERT(buf == hdr->b_buf); hdr->b_flags |= ARC_BUF_AVAILABLE; mutex_exit(&buf->b_evict_lock); } mutex_exit(hash_lock); VERIFY0(efunc(private)); return (B_TRUE); } /* * Release this buffer from the cache, making it an anonymous buffer. This * must be done after a read and prior to modifying the buffer contents. * If the buffer has more than one reference, we must make * a new hdr for the buffer. */ void arc_release(arc_buf_t *buf, void *tag) { arc_buf_hdr_t *hdr; kmutex_t *hash_lock = NULL; l2arc_buf_hdr_t *l2hdr; uint64_t buf_size; /* * It would be nice to assert that if it's DMU metadata (level > * 0 || it's the dnode file), then it must be syncing context. * But we don't know that information at this level. */ mutex_enter(&buf->b_evict_lock); hdr = buf->b_hdr; /* this buffer is not on any list */ ASSERT(refcount_count(&hdr->b_refcnt) > 0); if (hdr->b_state == arc_anon) { /* this buffer is already released */ ASSERT(buf->b_efunc == NULL); } else { hash_lock = HDR_LOCK(hdr); mutex_enter(hash_lock); hdr = buf->b_hdr; ASSERT3P(hash_lock, ==, HDR_LOCK(hdr)); } l2hdr = hdr->b_l2hdr; if (l2hdr) { mutex_enter(&l2arc_buflist_mtx); arc_buf_l2_cdata_free(hdr); hdr->b_l2hdr = NULL; list_remove(l2hdr->b_dev->l2ad_buflist, hdr); } buf_size = hdr->b_size; /* * Do we have more than one buf? */ if (hdr->b_datacnt > 1) { arc_buf_hdr_t *nhdr; arc_buf_t **bufp; uint64_t blksz = hdr->b_size; uint64_t spa = hdr->b_spa; arc_buf_contents_t type = hdr->b_type; uint32_t flags = hdr->b_flags; ASSERT(hdr->b_buf != buf || buf->b_next != NULL); /* * Pull the data off of this hdr and attach it to * a new anonymous hdr. */ (void) remove_reference(hdr, hash_lock, tag); bufp = &hdr->b_buf; while (*bufp != buf) bufp = &(*bufp)->b_next; *bufp = buf->b_next; buf->b_next = NULL; ASSERT3U(hdr->b_state->arcs_size, >=, hdr->b_size); atomic_add_64(&hdr->b_state->arcs_size, -hdr->b_size); if (refcount_is_zero(&hdr->b_refcnt)) { uint64_t *size = &hdr->b_state->arcs_lsize[hdr->b_type]; ASSERT3U(*size, >=, hdr->b_size); atomic_add_64(size, -hdr->b_size); } /* * We're releasing a duplicate user data buffer, update * our statistics accordingly. */ if (hdr->b_type == ARC_BUFC_DATA) { ARCSTAT_BUMPDOWN(arcstat_duplicate_buffers); ARCSTAT_INCR(arcstat_duplicate_buffers_size, -hdr->b_size); } hdr->b_datacnt -= 1; arc_cksum_verify(buf); #ifdef illumos arc_buf_unwatch(buf); #endif /* illumos */ mutex_exit(hash_lock); nhdr = kmem_cache_alloc(hdr_cache, KM_PUSHPAGE); nhdr->b_size = blksz; nhdr->b_spa = spa; nhdr->b_type = type; nhdr->b_buf = buf; nhdr->b_state = arc_anon; nhdr->b_arc_access = 0; nhdr->b_flags = flags & ARC_L2_WRITING; nhdr->b_l2hdr = NULL; nhdr->b_datacnt = 1; nhdr->b_freeze_cksum = NULL; (void) refcount_add(&nhdr->b_refcnt, tag); buf->b_hdr = nhdr; mutex_exit(&buf->b_evict_lock); atomic_add_64(&arc_anon->arcs_size, blksz); } else { mutex_exit(&buf->b_evict_lock); ASSERT(refcount_count(&hdr->b_refcnt) == 1); ASSERT(!list_link_active(&hdr->b_arc_node)); ASSERT(!HDR_IO_IN_PROGRESS(hdr)); if (hdr->b_state != arc_anon) arc_change_state(arc_anon, hdr, hash_lock); hdr->b_arc_access = 0; if (hash_lock) mutex_exit(hash_lock); buf_discard_identity(hdr); arc_buf_thaw(buf); } buf->b_efunc = NULL; buf->b_private = NULL; if (l2hdr) { ARCSTAT_INCR(arcstat_l2_asize, -l2hdr->b_asize); vdev_space_update(l2hdr->b_dev->l2ad_vdev, -l2hdr->b_asize, 0, 0); trim_map_free(l2hdr->b_dev->l2ad_vdev, l2hdr->b_daddr, hdr->b_size, 0); kmem_free(l2hdr, sizeof (l2arc_buf_hdr_t)); ARCSTAT_INCR(arcstat_l2_size, -buf_size); mutex_exit(&l2arc_buflist_mtx); } } int arc_released(arc_buf_t *buf) { int released; mutex_enter(&buf->b_evict_lock); released = (buf->b_data != NULL && buf->b_hdr->b_state == arc_anon); mutex_exit(&buf->b_evict_lock); return (released); } #ifdef ZFS_DEBUG int arc_referenced(arc_buf_t *buf) { int referenced; mutex_enter(&buf->b_evict_lock); referenced = (refcount_count(&buf->b_hdr->b_refcnt)); mutex_exit(&buf->b_evict_lock); return (referenced); } #endif static void arc_write_ready(zio_t *zio) { arc_write_callback_t *callback = zio->io_private; arc_buf_t *buf = callback->awcb_buf; arc_buf_hdr_t *hdr = buf->b_hdr; ASSERT(!refcount_is_zero(&buf->b_hdr->b_refcnt)); callback->awcb_ready(zio, buf, callback->awcb_private); /* * If the IO is already in progress, then this is a re-write * attempt, so we need to thaw and re-compute the cksum. * It is the responsibility of the callback to handle the * accounting for any re-write attempt. */ if (HDR_IO_IN_PROGRESS(hdr)) { mutex_enter(&hdr->b_freeze_lock); if (hdr->b_freeze_cksum != NULL) { kmem_free(hdr->b_freeze_cksum, sizeof (zio_cksum_t)); hdr->b_freeze_cksum = NULL; } mutex_exit(&hdr->b_freeze_lock); } arc_cksum_compute(buf, B_FALSE); hdr->b_flags |= ARC_IO_IN_PROGRESS; } /* * The SPA calls this callback for each physical write that happens on behalf * of a logical write. See the comment in dbuf_write_physdone() for details. */ static void arc_write_physdone(zio_t *zio) { arc_write_callback_t *cb = zio->io_private; if (cb->awcb_physdone != NULL) cb->awcb_physdone(zio, cb->awcb_buf, cb->awcb_private); } static void arc_write_done(zio_t *zio) { arc_write_callback_t *callback = zio->io_private; arc_buf_t *buf = callback->awcb_buf; arc_buf_hdr_t *hdr = buf->b_hdr; ASSERT(hdr->b_acb == NULL); if (zio->io_error == 0) { if (BP_IS_HOLE(zio->io_bp) || BP_IS_EMBEDDED(zio->io_bp)) { buf_discard_identity(hdr); } else { hdr->b_dva = *BP_IDENTITY(zio->io_bp); hdr->b_birth = BP_PHYSICAL_BIRTH(zio->io_bp); hdr->b_cksum0 = zio->io_bp->blk_cksum.zc_word[0]; } } else { ASSERT(BUF_EMPTY(hdr)); } /* * If the block to be written was all-zero or compressed enough to be * embedded in the BP, no write was performed so there will be no * dva/birth/checksum. The buffer must therefore remain anonymous * (and uncached). */ if (!BUF_EMPTY(hdr)) { arc_buf_hdr_t *exists; kmutex_t *hash_lock; ASSERT(zio->io_error == 0); arc_cksum_verify(buf); exists = buf_hash_insert(hdr, &hash_lock); if (exists) { /* * This can only happen if we overwrite for * sync-to-convergence, because we remove * buffers from the hash table when we arc_free(). */ if (zio->io_flags & ZIO_FLAG_IO_REWRITE) { if (!BP_EQUAL(&zio->io_bp_orig, zio->io_bp)) panic("bad overwrite, hdr=%p exists=%p", (void *)hdr, (void *)exists); ASSERT(refcount_is_zero(&exists->b_refcnt)); arc_change_state(arc_anon, exists, hash_lock); mutex_exit(hash_lock); arc_hdr_destroy(exists); exists = buf_hash_insert(hdr, &hash_lock); ASSERT3P(exists, ==, NULL); } else if (zio->io_flags & ZIO_FLAG_NOPWRITE) { /* nopwrite */ ASSERT(zio->io_prop.zp_nopwrite); if (!BP_EQUAL(&zio->io_bp_orig, zio->io_bp)) panic("bad nopwrite, hdr=%p exists=%p", (void *)hdr, (void *)exists); } else { /* Dedup */ ASSERT(hdr->b_datacnt == 1); ASSERT(hdr->b_state == arc_anon); ASSERT(BP_GET_DEDUP(zio->io_bp)); ASSERT(BP_GET_LEVEL(zio->io_bp) == 0); } } hdr->b_flags &= ~ARC_IO_IN_PROGRESS; /* if it's not anon, we are doing a scrub */ if (!exists && hdr->b_state == arc_anon) arc_access(hdr, hash_lock); mutex_exit(hash_lock); } else { hdr->b_flags &= ~ARC_IO_IN_PROGRESS; } ASSERT(!refcount_is_zero(&hdr->b_refcnt)); callback->awcb_done(zio, buf, callback->awcb_private); kmem_free(callback, sizeof (arc_write_callback_t)); } zio_t * arc_write(zio_t *pio, spa_t *spa, uint64_t txg, blkptr_t *bp, arc_buf_t *buf, boolean_t l2arc, boolean_t l2arc_compress, const zio_prop_t *zp, arc_done_func_t *ready, arc_done_func_t *physdone, arc_done_func_t *done, void *private, zio_priority_t priority, int zio_flags, const zbookmark_phys_t *zb) { arc_buf_hdr_t *hdr = buf->b_hdr; arc_write_callback_t *callback; zio_t *zio; ASSERT(ready != NULL); ASSERT(done != NULL); ASSERT(!HDR_IO_ERROR(hdr)); ASSERT((hdr->b_flags & ARC_IO_IN_PROGRESS) == 0); ASSERT(hdr->b_acb == NULL); if (l2arc) hdr->b_flags |= ARC_L2CACHE; if (l2arc_compress) hdr->b_flags |= ARC_L2COMPRESS; callback = kmem_zalloc(sizeof (arc_write_callback_t), KM_SLEEP); callback->awcb_ready = ready; callback->awcb_physdone = physdone; callback->awcb_done = done; callback->awcb_private = private; callback->awcb_buf = buf; zio = zio_write(pio, spa, txg, bp, buf->b_data, hdr->b_size, zp, arc_write_ready, arc_write_physdone, arc_write_done, callback, priority, zio_flags, zb); return (zio); } static int arc_memory_throttle(uint64_t reserve, uint64_t txg) { #ifdef _KERNEL uint64_t available_memory = ptob(freemem); static uint64_t page_load = 0; static uint64_t last_txg = 0; #if defined(__i386) || !defined(UMA_MD_SMALL_ALLOC) available_memory = MIN(available_memory, ptob(vmem_size(heap_arena, VMEM_FREE))); #endif if (freemem > (uint64_t)physmem * arc_lotsfree_percent / 100) return (0); if (txg > last_txg) { last_txg = txg; page_load = 0; } /* * If we are in pageout, we know that memory is already tight, * the arc is already going to be evicting, so we just want to * continue to let page writes occur as quickly as possible. */ if (curproc == pageproc) { if (page_load > MAX(ptob(minfree), available_memory) / 4) return (SET_ERROR(ERESTART)); /* Note: reserve is inflated, so we deflate */ page_load += reserve / 8; return (0); } else if (page_load > 0 && arc_reclaim_needed()) { /* memory is low, delay before restarting */ ARCSTAT_INCR(arcstat_memory_throttle_count, 1); return (SET_ERROR(EAGAIN)); } page_load = 0; #endif return (0); } void arc_tempreserve_clear(uint64_t reserve) { atomic_add_64(&arc_tempreserve, -reserve); ASSERT((int64_t)arc_tempreserve >= 0); } int arc_tempreserve_space(uint64_t reserve, uint64_t txg) { int error; uint64_t anon_size; if (reserve > arc_c/4 && !arc_no_grow) { arc_c = MIN(arc_c_max, reserve * 4); DTRACE_PROBE1(arc__set_reserve, uint64_t, arc_c); } if (reserve > arc_c) return (SET_ERROR(ENOMEM)); /* * Don't count loaned bufs as in flight dirty data to prevent long * network delays from blocking transactions that are ready to be * assigned to a txg. */ anon_size = MAX((int64_t)(arc_anon->arcs_size - arc_loaned_bytes), 0); /* * Writes will, almost always, require additional memory allocations * in order to compress/encrypt/etc the data. We therefore need to * make sure that there is sufficient available memory for this. */ error = arc_memory_throttle(reserve, txg); if (error != 0) return (error); /* * Throttle writes when the amount of dirty data in the cache * gets too large. We try to keep the cache less than half full * of dirty blocks so that our sync times don't grow too large. * Note: if two requests come in concurrently, we might let them * both succeed, when one of them should fail. Not a huge deal. */ if (reserve + arc_tempreserve + anon_size > arc_c / 2 && anon_size > arc_c / 4) { dprintf("failing, arc_tempreserve=%lluK anon_meta=%lluK " "anon_data=%lluK tempreserve=%lluK arc_c=%lluK\n", arc_tempreserve>>10, arc_anon->arcs_lsize[ARC_BUFC_METADATA]>>10, arc_anon->arcs_lsize[ARC_BUFC_DATA]>>10, reserve>>10, arc_c>>10); return (SET_ERROR(ERESTART)); } atomic_add_64(&arc_tempreserve, reserve); return (0); } static kmutex_t arc_lowmem_lock; #ifdef _KERNEL static eventhandler_tag arc_event_lowmem = NULL; static void arc_lowmem(void *arg __unused, int howto __unused) { /* Serialize access via arc_lowmem_lock. */ mutex_enter(&arc_lowmem_lock); mutex_enter(&arc_reclaim_thr_lock); needfree = 1; DTRACE_PROBE(arc__needfree); cv_signal(&arc_reclaim_thr_cv); /* * It is unsafe to block here in arbitrary threads, because we can come * here from ARC itself and may hold ARC locks and thus risk a deadlock * with ARC reclaim thread. */ if (curproc == pageproc) { while (needfree) msleep(&needfree, &arc_reclaim_thr_lock, 0, "zfs:lowmem", 0); } mutex_exit(&arc_reclaim_thr_lock); mutex_exit(&arc_lowmem_lock); } #endif void arc_init(void) { int i, prefetch_tunable_set = 0; mutex_init(&arc_reclaim_thr_lock, NULL, MUTEX_DEFAULT, NULL); cv_init(&arc_reclaim_thr_cv, NULL, CV_DEFAULT, NULL); mutex_init(&arc_lowmem_lock, NULL, MUTEX_DEFAULT, NULL); /* Convert seconds to clock ticks */ arc_min_prefetch_lifespan = 1 * hz; /* Start out with 1/8 of all memory */ arc_c = kmem_size() / 8; #ifdef sun #ifdef _KERNEL /* * On architectures where the physical memory can be larger * than the addressable space (intel in 32-bit mode), we may * need to limit the cache to 1/8 of VM size. */ arc_c = MIN(arc_c, vmem_size(heap_arena, VMEM_ALLOC | VMEM_FREE) / 8); #endif #endif /* sun */ /* set min cache to 1/32 of all memory, or 16MB, whichever is more */ arc_c_min = MAX(arc_c / 4, 64<<18); /* set max to 1/2 of all memory, or all but 1GB, whichever is more */ if (arc_c * 8 >= 1<<30) arc_c_max = (arc_c * 8) - (1<<30); else arc_c_max = arc_c_min; arc_c_max = MAX(arc_c * 5, arc_c_max); #ifdef _KERNEL /* * Allow the tunables to override our calculations if they are * reasonable (ie. over 16MB) */ if (zfs_arc_max > 64<<18 && zfs_arc_max < kmem_size()) arc_c_max = zfs_arc_max; if (zfs_arc_min > 64<<18 && zfs_arc_min <= arc_c_max) arc_c_min = zfs_arc_min; #endif arc_c = arc_c_max; arc_p = (arc_c >> 1); /* limit meta-data to 1/4 of the arc capacity */ arc_meta_limit = arc_c_max / 4; /* Allow the tunable to override if it is reasonable */ if (zfs_arc_meta_limit > 0 && zfs_arc_meta_limit <= arc_c_max) arc_meta_limit = zfs_arc_meta_limit; if (arc_c_min < arc_meta_limit / 2 && zfs_arc_min == 0) arc_c_min = arc_meta_limit / 2; if (zfs_arc_grow_retry > 0) arc_grow_retry = zfs_arc_grow_retry; if (zfs_arc_shrink_shift > 0) arc_shrink_shift = zfs_arc_shrink_shift; if (zfs_arc_p_min_shift > 0) arc_p_min_shift = zfs_arc_p_min_shift; /* if kmem_flags are set, lets try to use less memory */ if (kmem_debugging()) arc_c = arc_c / 2; if (arc_c < arc_c_min) arc_c = arc_c_min; zfs_arc_min = arc_c_min; zfs_arc_max = arc_c_max; arc_anon = &ARC_anon; arc_mru = &ARC_mru; arc_mru_ghost = &ARC_mru_ghost; arc_mfu = &ARC_mfu; arc_mfu_ghost = &ARC_mfu_ghost; arc_l2c_only = &ARC_l2c_only; arc_size = 0; for (i = 0; i < ARC_BUFC_NUMLISTS; i++) { mutex_init(&arc_anon->arcs_locks[i].arcs_lock, NULL, MUTEX_DEFAULT, NULL); mutex_init(&arc_mru->arcs_locks[i].arcs_lock, NULL, MUTEX_DEFAULT, NULL); mutex_init(&arc_mru_ghost->arcs_locks[i].arcs_lock, NULL, MUTEX_DEFAULT, NULL); mutex_init(&arc_mfu->arcs_locks[i].arcs_lock, NULL, MUTEX_DEFAULT, NULL); mutex_init(&arc_mfu_ghost->arcs_locks[i].arcs_lock, NULL, MUTEX_DEFAULT, NULL); mutex_init(&arc_l2c_only->arcs_locks[i].arcs_lock, NULL, MUTEX_DEFAULT, NULL); list_create(&arc_mru->arcs_lists[i], sizeof (arc_buf_hdr_t), offsetof(arc_buf_hdr_t, b_arc_node)); list_create(&arc_mru_ghost->arcs_lists[i], sizeof (arc_buf_hdr_t), offsetof(arc_buf_hdr_t, b_arc_node)); list_create(&arc_mfu->arcs_lists[i], sizeof (arc_buf_hdr_t), offsetof(arc_buf_hdr_t, b_arc_node)); list_create(&arc_mfu_ghost->arcs_lists[i], sizeof (arc_buf_hdr_t), offsetof(arc_buf_hdr_t, b_arc_node)); list_create(&arc_mfu_ghost->arcs_lists[i], sizeof (arc_buf_hdr_t), offsetof(arc_buf_hdr_t, b_arc_node)); list_create(&arc_l2c_only->arcs_lists[i], sizeof (arc_buf_hdr_t), offsetof(arc_buf_hdr_t, b_arc_node)); } buf_init(); arc_thread_exit = 0; arc_eviction_list = NULL; mutex_init(&arc_eviction_mtx, NULL, MUTEX_DEFAULT, NULL); bzero(&arc_eviction_hdr, sizeof (arc_buf_hdr_t)); arc_ksp = kstat_create("zfs", 0, "arcstats", "misc", KSTAT_TYPE_NAMED, sizeof (arc_stats) / sizeof (kstat_named_t), KSTAT_FLAG_VIRTUAL); if (arc_ksp != NULL) { arc_ksp->ks_data = &arc_stats; kstat_install(arc_ksp); } (void) thread_create(NULL, 0, arc_reclaim_thread, NULL, 0, &p0, TS_RUN, minclsyspri); #ifdef _KERNEL arc_event_lowmem = EVENTHANDLER_REGISTER(vm_lowmem, arc_lowmem, NULL, EVENTHANDLER_PRI_FIRST); #endif arc_dead = FALSE; arc_warm = B_FALSE; /* * Calculate maximum amount of dirty data per pool. * * If it has been set by /etc/system, take that. * Otherwise, use a percentage of physical memory defined by * zfs_dirty_data_max_percent (default 10%) with a cap at * zfs_dirty_data_max_max (default 4GB). */ if (zfs_dirty_data_max == 0) { zfs_dirty_data_max = ptob(physmem) * zfs_dirty_data_max_percent / 100; zfs_dirty_data_max = MIN(zfs_dirty_data_max, zfs_dirty_data_max_max); } #ifdef _KERNEL if (TUNABLE_INT_FETCH("vfs.zfs.prefetch_disable", &zfs_prefetch_disable)) prefetch_tunable_set = 1; #ifdef __i386__ if (prefetch_tunable_set == 0) { printf("ZFS NOTICE: Prefetch is disabled by default on i386 " "-- to enable,\n"); printf(" add \"vfs.zfs.prefetch_disable=0\" " "to /boot/loader.conf.\n"); zfs_prefetch_disable = 1; } #else if ((((uint64_t)physmem * PAGESIZE) < (1ULL << 32)) && prefetch_tunable_set == 0) { printf("ZFS NOTICE: Prefetch is disabled by default if less " "than 4GB of RAM is present;\n" " to enable, add \"vfs.zfs.prefetch_disable=0\" " "to /boot/loader.conf.\n"); zfs_prefetch_disable = 1; } #endif /* Warn about ZFS memory and address space requirements. */ if (((uint64_t)physmem * PAGESIZE) < (256 + 128 + 64) * (1 << 20)) { printf("ZFS WARNING: Recommended minimum RAM size is 512MB; " "expect unstable behavior.\n"); } if (kmem_size() < 512 * (1 << 20)) { printf("ZFS WARNING: Recommended minimum kmem_size is 512MB; " "expect unstable behavior.\n"); printf(" Consider tuning vm.kmem_size and " "vm.kmem_size_max\n"); printf(" in /boot/loader.conf.\n"); } #endif } void arc_fini(void) { int i; mutex_enter(&arc_reclaim_thr_lock); arc_thread_exit = 1; cv_signal(&arc_reclaim_thr_cv); while (arc_thread_exit != 0) cv_wait(&arc_reclaim_thr_cv, &arc_reclaim_thr_lock); mutex_exit(&arc_reclaim_thr_lock); arc_flush(NULL); arc_dead = TRUE; if (arc_ksp != NULL) { kstat_delete(arc_ksp); arc_ksp = NULL; } mutex_destroy(&arc_eviction_mtx); mutex_destroy(&arc_reclaim_thr_lock); cv_destroy(&arc_reclaim_thr_cv); for (i = 0; i < ARC_BUFC_NUMLISTS; i++) { list_destroy(&arc_mru->arcs_lists[i]); list_destroy(&arc_mru_ghost->arcs_lists[i]); list_destroy(&arc_mfu->arcs_lists[i]); list_destroy(&arc_mfu_ghost->arcs_lists[i]); list_destroy(&arc_l2c_only->arcs_lists[i]); mutex_destroy(&arc_anon->arcs_locks[i].arcs_lock); mutex_destroy(&arc_mru->arcs_locks[i].arcs_lock); mutex_destroy(&arc_mru_ghost->arcs_locks[i].arcs_lock); mutex_destroy(&arc_mfu->arcs_locks[i].arcs_lock); mutex_destroy(&arc_mfu_ghost->arcs_locks[i].arcs_lock); mutex_destroy(&arc_l2c_only->arcs_locks[i].arcs_lock); } buf_fini(); ASSERT(arc_loaned_bytes == 0); mutex_destroy(&arc_lowmem_lock); #ifdef _KERNEL if (arc_event_lowmem != NULL) EVENTHANDLER_DEREGISTER(vm_lowmem, arc_event_lowmem); #endif } /* * Level 2 ARC * * The level 2 ARC (L2ARC) is a cache layer in-between main memory and disk. * It uses dedicated storage devices to hold cached data, which are populated * using large infrequent writes. The main role of this cache is to boost * the performance of random read workloads. The intended L2ARC devices * include short-stroked disks, solid state disks, and other media with * substantially faster read latency than disk. * * +-----------------------+ * | ARC | * +-----------------------+ * | ^ ^ * | | | * l2arc_feed_thread() arc_read() * | | | * | l2arc read | * V | | * +---------------+ | * | L2ARC | | * +---------------+ | * | ^ | * l2arc_write() | | * | | | * V | | * +-------+ +-------+ * | vdev | | vdev | * | cache | | cache | * +-------+ +-------+ * +=========+ .-----. * : L2ARC : |-_____-| * : devices : | Disks | * +=========+ `-_____-' * * Read requests are satisfied from the following sources, in order: * * 1) ARC * 2) vdev cache of L2ARC devices * 3) L2ARC devices * 4) vdev cache of disks * 5) disks * * Some L2ARC device types exhibit extremely slow write performance. * To accommodate for this there are some significant differences between * the L2ARC and traditional cache design: * * 1. There is no eviction path from the ARC to the L2ARC. Evictions from * the ARC behave as usual, freeing buffers and placing headers on ghost * lists. The ARC does not send buffers to the L2ARC during eviction as * this would add inflated write latencies for all ARC memory pressure. * * 2. The L2ARC attempts to cache data from the ARC before it is evicted. * It does this by periodically scanning buffers from the eviction-end of * the MFU and MRU ARC lists, copying them to the L2ARC devices if they are * not already there. It scans until a headroom of buffers is satisfied, * which itself is a buffer for ARC eviction. If a compressible buffer is * found during scanning and selected for writing to an L2ARC device, we * temporarily boost scanning headroom during the next scan cycle to make * sure we adapt to compression effects (which might significantly reduce * the data volume we write to L2ARC). The thread that does this is * l2arc_feed_thread(), illustrated below; example sizes are included to * provide a better sense of ratio than this diagram: * * head --> tail * +---------------------+----------+ * ARC_mfu |:::::#:::::::::::::::|o#o###o###|-->. # already on L2ARC * +---------------------+----------+ | o L2ARC eligible * ARC_mru |:#:::::::::::::::::::|#o#ooo####|-->| : ARC buffer * +---------------------+----------+ | * 15.9 Gbytes ^ 32 Mbytes | * headroom | * l2arc_feed_thread() * | * l2arc write hand <--[oooo]--' * | 8 Mbyte * | write max * V * +==============================+ * L2ARC dev |####|#|###|###| |####| ... | * +==============================+ * 32 Gbytes * * 3. If an ARC buffer is copied to the L2ARC but then hit instead of * evicted, then the L2ARC has cached a buffer much sooner than it probably * needed to, potentially wasting L2ARC device bandwidth and storage. It is * safe to say that this is an uncommon case, since buffers at the end of * the ARC lists have moved there due to inactivity. * * 4. If the ARC evicts faster than the L2ARC can maintain a headroom, * then the L2ARC simply misses copying some buffers. This serves as a * pressure valve to prevent heavy read workloads from both stalling the ARC * with waits and clogging the L2ARC with writes. This also helps prevent * the potential for the L2ARC to churn if it attempts to cache content too * quickly, such as during backups of the entire pool. * * 5. After system boot and before the ARC has filled main memory, there are * no evictions from the ARC and so the tails of the ARC_mfu and ARC_mru * lists can remain mostly static. Instead of searching from tail of these * lists as pictured, the l2arc_feed_thread() will search from the list heads * for eligible buffers, greatly increasing its chance of finding them. * * The L2ARC device write speed is also boosted during this time so that * the L2ARC warms up faster. Since there have been no ARC evictions yet, * there are no L2ARC reads, and no fear of degrading read performance * through increased writes. * * 6. Writes to the L2ARC devices are grouped and sent in-sequence, so that * the vdev queue can aggregate them into larger and fewer writes. Each * device is written to in a rotor fashion, sweeping writes through * available space then repeating. * * 7. The L2ARC does not store dirty content. It never needs to flush * write buffers back to disk based storage. * * 8. If an ARC buffer is written (and dirtied) which also exists in the * L2ARC, the now stale L2ARC buffer is immediately dropped. * * The performance of the L2ARC can be tweaked by a number of tunables, which * may be necessary for different workloads: * * l2arc_write_max max write bytes per interval * l2arc_write_boost extra write bytes during device warmup * l2arc_noprefetch skip caching prefetched buffers * l2arc_headroom number of max device writes to precache * l2arc_headroom_boost when we find compressed buffers during ARC * scanning, we multiply headroom by this * percentage factor for the next scan cycle, * since more compressed buffers are likely to * be present * l2arc_feed_secs seconds between L2ARC writing * * Tunables may be removed or added as future performance improvements are * integrated, and also may become zpool properties. * * There are three key functions that control how the L2ARC warms up: * * l2arc_write_eligible() check if a buffer is eligible to cache * l2arc_write_size() calculate how much to write * l2arc_write_interval() calculate sleep delay between writes * * These three functions determine what to write, how much, and how quickly * to send writes. */ static boolean_t l2arc_write_eligible(uint64_t spa_guid, arc_buf_hdr_t *ab) { /* * A buffer is *not* eligible for the L2ARC if it: * 1. belongs to a different spa. * 2. is already cached on the L2ARC. * 3. has an I/O in progress (it may be an incomplete read). * 4. is flagged not eligible (zfs property). */ if (ab->b_spa != spa_guid) { ARCSTAT_BUMP(arcstat_l2_write_spa_mismatch); return (B_FALSE); } if (ab->b_l2hdr != NULL) { ARCSTAT_BUMP(arcstat_l2_write_in_l2); return (B_FALSE); } if (HDR_IO_IN_PROGRESS(ab)) { ARCSTAT_BUMP(arcstat_l2_write_hdr_io_in_progress); return (B_FALSE); } if (!HDR_L2CACHE(ab)) { ARCSTAT_BUMP(arcstat_l2_write_not_cacheable); return (B_FALSE); } return (B_TRUE); } static uint64_t l2arc_write_size(void) { uint64_t size; /* * Make sure our globals have meaningful values in case the user * altered them. */ size = l2arc_write_max; if (size == 0) { cmn_err(CE_NOTE, "Bad value for l2arc_write_max, value must " "be greater than zero, resetting it to the default (%d)", L2ARC_WRITE_SIZE); size = l2arc_write_max = L2ARC_WRITE_SIZE; } if (arc_warm == B_FALSE) size += l2arc_write_boost; return (size); } static clock_t l2arc_write_interval(clock_t began, uint64_t wanted, uint64_t wrote) { clock_t interval, next, now; /* * If the ARC lists are busy, increase our write rate; if the * lists are stale, idle back. This is achieved by checking * how much we previously wrote - if it was more than half of * what we wanted, schedule the next write much sooner. */ if (l2arc_feed_again && wrote > (wanted / 2)) interval = (hz * l2arc_feed_min_ms) / 1000; else interval = hz * l2arc_feed_secs; now = ddi_get_lbolt(); next = MAX(now, MIN(now + interval, began + interval)); return (next); } static void l2arc_hdr_stat_add(void) { ARCSTAT_INCR(arcstat_l2_hdr_size, HDR_SIZE + L2HDR_SIZE); ARCSTAT_INCR(arcstat_hdr_size, -HDR_SIZE); } static void l2arc_hdr_stat_remove(void) { ARCSTAT_INCR(arcstat_l2_hdr_size, -(HDR_SIZE + L2HDR_SIZE)); ARCSTAT_INCR(arcstat_hdr_size, HDR_SIZE); } /* * Cycle through L2ARC devices. This is how L2ARC load balances. * If a device is returned, this also returns holding the spa config lock. */ static l2arc_dev_t * l2arc_dev_get_next(void) { l2arc_dev_t *first, *next = NULL; /* * Lock out the removal of spas (spa_namespace_lock), then removal * of cache devices (l2arc_dev_mtx). Once a device has been selected, * both locks will be dropped and a spa config lock held instead. */ mutex_enter(&spa_namespace_lock); mutex_enter(&l2arc_dev_mtx); /* if there are no vdevs, there is nothing to do */ if (l2arc_ndev == 0) goto out; first = NULL; next = l2arc_dev_last; do { /* loop around the list looking for a non-faulted vdev */ if (next == NULL) { next = list_head(l2arc_dev_list); } else { next = list_next(l2arc_dev_list, next); if (next == NULL) next = list_head(l2arc_dev_list); } /* if we have come back to the start, bail out */ if (first == NULL) first = next; else if (next == first) break; } while (vdev_is_dead(next->l2ad_vdev)); /* if we were unable to find any usable vdevs, return NULL */ if (vdev_is_dead(next->l2ad_vdev)) next = NULL; l2arc_dev_last = next; out: mutex_exit(&l2arc_dev_mtx); /* * Grab the config lock to prevent the 'next' device from being * removed while we are writing to it. */ if (next != NULL) spa_config_enter(next->l2ad_spa, SCL_L2ARC, next, RW_READER); mutex_exit(&spa_namespace_lock); return (next); } /* * Free buffers that were tagged for destruction. */ static void l2arc_do_free_on_write() { list_t *buflist; l2arc_data_free_t *df, *df_prev; mutex_enter(&l2arc_free_on_write_mtx); buflist = l2arc_free_on_write; for (df = list_tail(buflist); df; df = df_prev) { df_prev = list_prev(buflist, df); ASSERT(df->l2df_data != NULL); ASSERT(df->l2df_func != NULL); df->l2df_func(df->l2df_data, df->l2df_size); list_remove(buflist, df); kmem_free(df, sizeof (l2arc_data_free_t)); } mutex_exit(&l2arc_free_on_write_mtx); } /* * A write to a cache device has completed. Update all headers to allow * reads from these buffers to begin. */ static void l2arc_write_done(zio_t *zio) { l2arc_write_callback_t *cb; l2arc_dev_t *dev; list_t *buflist; arc_buf_hdr_t *head, *ab, *ab_prev; l2arc_buf_hdr_t *abl2; kmutex_t *hash_lock; int64_t bytes_dropped = 0; cb = zio->io_private; ASSERT(cb != NULL); dev = cb->l2wcb_dev; ASSERT(dev != NULL); head = cb->l2wcb_head; ASSERT(head != NULL); buflist = dev->l2ad_buflist; ASSERT(buflist != NULL); DTRACE_PROBE2(l2arc__iodone, zio_t *, zio, l2arc_write_callback_t *, cb); if (zio->io_error != 0) ARCSTAT_BUMP(arcstat_l2_writes_error); mutex_enter(&l2arc_buflist_mtx); /* * All writes completed, or an error was hit. */ for (ab = list_prev(buflist, head); ab; ab = ab_prev) { ab_prev = list_prev(buflist, ab); abl2 = ab->b_l2hdr; /* * Release the temporary compressed buffer as soon as possible. */ if (abl2->b_compress != ZIO_COMPRESS_OFF) l2arc_release_cdata_buf(ab); hash_lock = HDR_LOCK(ab); if (!mutex_tryenter(hash_lock)) { /* * This buffer misses out. It may be in a stage * of eviction. Its ARC_L2_WRITING flag will be * left set, denying reads to this buffer. */ ARCSTAT_BUMP(arcstat_l2_writes_hdr_miss); continue; } if (zio->io_error != 0) { /* * Error - drop L2ARC entry. */ list_remove(buflist, ab); ARCSTAT_INCR(arcstat_l2_asize, -abl2->b_asize); bytes_dropped += abl2->b_asize; ab->b_l2hdr = NULL; trim_map_free(abl2->b_dev->l2ad_vdev, abl2->b_daddr, ab->b_size, 0); kmem_free(abl2, sizeof (l2arc_buf_hdr_t)); ARCSTAT_INCR(arcstat_l2_size, -ab->b_size); } /* * Allow ARC to begin reads to this L2ARC entry. */ ab->b_flags &= ~ARC_L2_WRITING; mutex_exit(hash_lock); } atomic_inc_64(&l2arc_writes_done); list_remove(buflist, head); kmem_cache_free(hdr_cache, head); mutex_exit(&l2arc_buflist_mtx); vdev_space_update(dev->l2ad_vdev, -bytes_dropped, 0, 0); l2arc_do_free_on_write(); kmem_free(cb, sizeof (l2arc_write_callback_t)); } /* * A read to a cache device completed. Validate buffer contents before * handing over to the regular ARC routines. */ static void l2arc_read_done(zio_t *zio) { l2arc_read_callback_t *cb; arc_buf_hdr_t *hdr; arc_buf_t *buf; kmutex_t *hash_lock; int equal; ASSERT(zio->io_vd != NULL); ASSERT(zio->io_flags & ZIO_FLAG_DONT_PROPAGATE); spa_config_exit(zio->io_spa, SCL_L2ARC, zio->io_vd); cb = zio->io_private; ASSERT(cb != NULL); buf = cb->l2rcb_buf; ASSERT(buf != NULL); hash_lock = HDR_LOCK(buf->b_hdr); mutex_enter(hash_lock); hdr = buf->b_hdr; ASSERT3P(hash_lock, ==, HDR_LOCK(hdr)); /* * If the buffer was compressed, decompress it first. */ if (cb->l2rcb_compress != ZIO_COMPRESS_OFF) l2arc_decompress_zio(zio, hdr, cb->l2rcb_compress); ASSERT(zio->io_data != NULL); /* * Check this survived the L2ARC journey. */ equal = arc_cksum_equal(buf); if (equal && zio->io_error == 0 && !HDR_L2_EVICTED(hdr)) { mutex_exit(hash_lock); zio->io_private = buf; zio->io_bp_copy = cb->l2rcb_bp; /* XXX fix in L2ARC 2.0 */ zio->io_bp = &zio->io_bp_copy; /* XXX fix in L2ARC 2.0 */ arc_read_done(zio); } else { mutex_exit(hash_lock); /* * Buffer didn't survive caching. Increment stats and * reissue to the original storage device. */ if (zio->io_error != 0) { ARCSTAT_BUMP(arcstat_l2_io_error); } else { zio->io_error = SET_ERROR(EIO); } if (!equal) ARCSTAT_BUMP(arcstat_l2_cksum_bad); /* * If there's no waiter, issue an async i/o to the primary * storage now. If there *is* a waiter, the caller must * issue the i/o in a context where it's OK to block. */ if (zio->io_waiter == NULL) { zio_t *pio = zio_unique_parent(zio); ASSERT(!pio || pio->io_child_type == ZIO_CHILD_LOGICAL); zio_nowait(zio_read(pio, cb->l2rcb_spa, &cb->l2rcb_bp, buf->b_data, zio->io_size, arc_read_done, buf, zio->io_priority, cb->l2rcb_flags, &cb->l2rcb_zb)); } } kmem_free(cb, sizeof (l2arc_read_callback_t)); } /* * This is the list priority from which the L2ARC will search for pages to * cache. This is used within loops (0..3) to cycle through lists in the * desired order. This order can have a significant effect on cache * performance. * * Currently the metadata lists are hit first, MFU then MRU, followed by * the data lists. This function returns a locked list, and also returns * the lock pointer. */ static list_t * l2arc_list_locked(int list_num, kmutex_t **lock) { list_t *list = NULL; int idx; ASSERT(list_num >= 0 && list_num < 2 * ARC_BUFC_NUMLISTS); if (list_num < ARC_BUFC_NUMMETADATALISTS) { idx = list_num; list = &arc_mfu->arcs_lists[idx]; *lock = ARCS_LOCK(arc_mfu, idx); } else if (list_num < ARC_BUFC_NUMMETADATALISTS * 2) { idx = list_num - ARC_BUFC_NUMMETADATALISTS; list = &arc_mru->arcs_lists[idx]; *lock = ARCS_LOCK(arc_mru, idx); } else if (list_num < (ARC_BUFC_NUMMETADATALISTS * 2 + ARC_BUFC_NUMDATALISTS)) { idx = list_num - ARC_BUFC_NUMMETADATALISTS; list = &arc_mfu->arcs_lists[idx]; *lock = ARCS_LOCK(arc_mfu, idx); } else { idx = list_num - ARC_BUFC_NUMLISTS; list = &arc_mru->arcs_lists[idx]; *lock = ARCS_LOCK(arc_mru, idx); } ASSERT(!(MUTEX_HELD(*lock))); mutex_enter(*lock); return (list); } /* * Evict buffers from the device write hand to the distance specified in * bytes. This distance may span populated buffers, it may span nothing. * This is clearing a region on the L2ARC device ready for writing. * If the 'all' boolean is set, every buffer is evicted. */ static void l2arc_evict(l2arc_dev_t *dev, uint64_t distance, boolean_t all) { list_t *buflist; l2arc_buf_hdr_t *abl2; arc_buf_hdr_t *ab, *ab_prev; kmutex_t *hash_lock; uint64_t taddr; int64_t bytes_evicted = 0; buflist = dev->l2ad_buflist; if (buflist == NULL) return; if (!all && dev->l2ad_first) { /* * This is the first sweep through the device. There is * nothing to evict. */ return; } if (dev->l2ad_hand >= (dev->l2ad_end - (2 * distance))) { /* * When nearing the end of the device, evict to the end * before the device write hand jumps to the start. */ taddr = dev->l2ad_end; } else { taddr = dev->l2ad_hand + distance; } DTRACE_PROBE4(l2arc__evict, l2arc_dev_t *, dev, list_t *, buflist, uint64_t, taddr, boolean_t, all); top: mutex_enter(&l2arc_buflist_mtx); for (ab = list_tail(buflist); ab; ab = ab_prev) { ab_prev = list_prev(buflist, ab); hash_lock = HDR_LOCK(ab); if (!mutex_tryenter(hash_lock)) { /* * Missed the hash lock. Retry. */ ARCSTAT_BUMP(arcstat_l2_evict_lock_retry); mutex_exit(&l2arc_buflist_mtx); mutex_enter(hash_lock); mutex_exit(hash_lock); goto top; } if (HDR_L2_WRITE_HEAD(ab)) { /* * We hit a write head node. Leave it for * l2arc_write_done(). */ list_remove(buflist, ab); mutex_exit(hash_lock); continue; } if (!all && ab->b_l2hdr != NULL && (ab->b_l2hdr->b_daddr > taddr || ab->b_l2hdr->b_daddr < dev->l2ad_hand)) { /* * We've evicted to the target address, * or the end of the device. */ mutex_exit(hash_lock); break; } if (HDR_FREE_IN_PROGRESS(ab)) { /* * Already on the path to destruction. */ mutex_exit(hash_lock); continue; } if (ab->b_state == arc_l2c_only) { ASSERT(!HDR_L2_READING(ab)); /* * This doesn't exist in the ARC. Destroy. * arc_hdr_destroy() will call list_remove() * and decrement arcstat_l2_size. */ arc_change_state(arc_anon, ab, hash_lock); arc_hdr_destroy(ab); } else { /* * Invalidate issued or about to be issued * reads, since we may be about to write * over this location. */ if (HDR_L2_READING(ab)) { ARCSTAT_BUMP(arcstat_l2_evict_reading); ab->b_flags |= ARC_L2_EVICTED; } /* * Tell ARC this no longer exists in L2ARC. */ if (ab->b_l2hdr != NULL) { abl2 = ab->b_l2hdr; ARCSTAT_INCR(arcstat_l2_asize, -abl2->b_asize); bytes_evicted += abl2->b_asize; ab->b_l2hdr = NULL; /* * We are destroying l2hdr, so ensure that * its compressed buffer, if any, is not leaked. */ ASSERT(abl2->b_tmp_cdata == NULL); kmem_free(abl2, sizeof (l2arc_buf_hdr_t)); ARCSTAT_INCR(arcstat_l2_size, -ab->b_size); } list_remove(buflist, ab); /* * This may have been leftover after a * failed write. */ ab->b_flags &= ~ARC_L2_WRITING; } mutex_exit(hash_lock); } mutex_exit(&l2arc_buflist_mtx); vdev_space_update(dev->l2ad_vdev, -bytes_evicted, 0, 0); dev->l2ad_evict = taddr; } /* * Find and write ARC buffers to the L2ARC device. * * An ARC_L2_WRITING flag is set so that the L2ARC buffers are not valid * for reading until they have completed writing. * The headroom_boost is an in-out parameter used to maintain headroom boost * state between calls to this function. * * Returns the number of bytes actually written (which may be smaller than * the delta by which the device hand has changed due to alignment). */ static uint64_t l2arc_write_buffers(spa_t *spa, l2arc_dev_t *dev, uint64_t target_sz, boolean_t *headroom_boost) { arc_buf_hdr_t *ab, *ab_prev, *head; list_t *list; uint64_t write_asize, write_psize, write_sz, headroom, buf_compress_minsz; void *buf_data; kmutex_t *list_lock; boolean_t full; l2arc_write_callback_t *cb; zio_t *pio, *wzio; uint64_t guid = spa_load_guid(spa); const boolean_t do_headroom_boost = *headroom_boost; int try; ASSERT(dev->l2ad_vdev != NULL); /* Lower the flag now, we might want to raise it again later. */ *headroom_boost = B_FALSE; pio = NULL; write_sz = write_asize = write_psize = 0; full = B_FALSE; head = kmem_cache_alloc(hdr_cache, KM_PUSHPAGE); head->b_flags |= ARC_L2_WRITE_HEAD; ARCSTAT_BUMP(arcstat_l2_write_buffer_iter); /* * We will want to try to compress buffers that are at least 2x the * device sector size. */ buf_compress_minsz = 2 << dev->l2ad_vdev->vdev_ashift; /* * Copy buffers for L2ARC writing. */ mutex_enter(&l2arc_buflist_mtx); for (try = 0; try < 2 * ARC_BUFC_NUMLISTS; try++) { uint64_t passed_sz = 0; list = l2arc_list_locked(try, &list_lock); ARCSTAT_BUMP(arcstat_l2_write_buffer_list_iter); /* * L2ARC fast warmup. * * Until the ARC is warm and starts to evict, read from the * head of the ARC lists rather than the tail. */ if (arc_warm == B_FALSE) ab = list_head(list); else ab = list_tail(list); if (ab == NULL) ARCSTAT_BUMP(arcstat_l2_write_buffer_list_null_iter); headroom = target_sz * l2arc_headroom * 2 / ARC_BUFC_NUMLISTS; if (do_headroom_boost) headroom = (headroom * l2arc_headroom_boost) / 100; for (; ab; ab = ab_prev) { l2arc_buf_hdr_t *l2hdr; kmutex_t *hash_lock; uint64_t buf_sz; if (arc_warm == B_FALSE) ab_prev = list_next(list, ab); else ab_prev = list_prev(list, ab); ARCSTAT_INCR(arcstat_l2_write_buffer_bytes_scanned, ab->b_size); hash_lock = HDR_LOCK(ab); if (!mutex_tryenter(hash_lock)) { ARCSTAT_BUMP(arcstat_l2_write_trylock_fail); /* * Skip this buffer rather than waiting. */ continue; } passed_sz += ab->b_size; if (passed_sz > headroom) { /* * Searched too far. */ mutex_exit(hash_lock); ARCSTAT_BUMP(arcstat_l2_write_passed_headroom); break; } if (!l2arc_write_eligible(guid, ab)) { mutex_exit(hash_lock); continue; } if ((write_sz + ab->b_size) > target_sz) { full = B_TRUE; mutex_exit(hash_lock); ARCSTAT_BUMP(arcstat_l2_write_full); break; } if (pio == NULL) { /* * Insert a dummy header on the buflist so * l2arc_write_done() can find where the * write buffers begin without searching. */ list_insert_head(dev->l2ad_buflist, head); cb = kmem_alloc( sizeof (l2arc_write_callback_t), KM_SLEEP); cb->l2wcb_dev = dev; cb->l2wcb_head = head; pio = zio_root(spa, l2arc_write_done, cb, ZIO_FLAG_CANFAIL); ARCSTAT_BUMP(arcstat_l2_write_pios); } /* * Create and add a new L2ARC header. */ l2hdr = kmem_zalloc(sizeof (l2arc_buf_hdr_t), KM_SLEEP); l2hdr->b_dev = dev; ab->b_flags |= ARC_L2_WRITING; /* * Temporarily stash the data buffer in b_tmp_cdata. * The subsequent write step will pick it up from * there. This is because can't access ab->b_buf * without holding the hash_lock, which we in turn * can't access without holding the ARC list locks * (which we want to avoid during compression/writing). */ l2hdr->b_compress = ZIO_COMPRESS_OFF; l2hdr->b_asize = ab->b_size; l2hdr->b_tmp_cdata = ab->b_buf->b_data; buf_sz = ab->b_size; ab->b_l2hdr = l2hdr; list_insert_head(dev->l2ad_buflist, ab); /* * Compute and store the buffer cksum before * writing. On debug the cksum is verified first. */ arc_cksum_verify(ab->b_buf); arc_cksum_compute(ab->b_buf, B_TRUE); mutex_exit(hash_lock); write_sz += buf_sz; } mutex_exit(list_lock); if (full == B_TRUE) break; } /* No buffers selected for writing? */ if (pio == NULL) { ASSERT0(write_sz); mutex_exit(&l2arc_buflist_mtx); kmem_cache_free(hdr_cache, head); return (0); } /* * Now start writing the buffers. We're starting at the write head * and work backwards, retracing the course of the buffer selector * loop above. */ for (ab = list_prev(dev->l2ad_buflist, head); ab; ab = list_prev(dev->l2ad_buflist, ab)) { l2arc_buf_hdr_t *l2hdr; uint64_t buf_sz; /* * We shouldn't need to lock the buffer here, since we flagged * it as ARC_L2_WRITING in the previous step, but we must take * care to only access its L2 cache parameters. In particular, * ab->b_buf may be invalid by now due to ARC eviction. */ l2hdr = ab->b_l2hdr; l2hdr->b_daddr = dev->l2ad_hand; if ((ab->b_flags & ARC_L2COMPRESS) && l2hdr->b_asize >= buf_compress_minsz) { if (l2arc_compress_buf(l2hdr)) { /* * If compression succeeded, enable headroom * boost on the next scan cycle. */ *headroom_boost = B_TRUE; } } /* * Pick up the buffer data we had previously stashed away * (and now potentially also compressed). */ buf_data = l2hdr->b_tmp_cdata; buf_sz = l2hdr->b_asize; /* * If the data has not been compressed, then clear b_tmp_cdata * to make sure that it points only to a temporary compression * buffer. */ if (!L2ARC_IS_VALID_COMPRESS(l2hdr->b_compress)) l2hdr->b_tmp_cdata = NULL; /* Compression may have squashed the buffer to zero length. */ if (buf_sz != 0) { uint64_t buf_p_sz; wzio = zio_write_phys(pio, dev->l2ad_vdev, dev->l2ad_hand, buf_sz, buf_data, ZIO_CHECKSUM_OFF, NULL, NULL, ZIO_PRIORITY_ASYNC_WRITE, ZIO_FLAG_CANFAIL, B_FALSE); DTRACE_PROBE2(l2arc__write, vdev_t *, dev->l2ad_vdev, zio_t *, wzio); (void) zio_nowait(wzio); write_asize += buf_sz; /* * Keep the clock hand suitably device-aligned. */ buf_p_sz = vdev_psize_to_asize(dev->l2ad_vdev, buf_sz); write_psize += buf_p_sz; dev->l2ad_hand += buf_p_sz; } } mutex_exit(&l2arc_buflist_mtx); ASSERT3U(write_asize, <=, target_sz); ARCSTAT_BUMP(arcstat_l2_writes_sent); ARCSTAT_INCR(arcstat_l2_write_bytes, write_asize); ARCSTAT_INCR(arcstat_l2_size, write_sz); ARCSTAT_INCR(arcstat_l2_asize, write_asize); vdev_space_update(dev->l2ad_vdev, write_asize, 0, 0); /* * Bump device hand to the device start if it is approaching the end. * l2arc_evict() will already have evicted ahead for this case. */ if (dev->l2ad_hand >= (dev->l2ad_end - target_sz)) { dev->l2ad_hand = dev->l2ad_start; dev->l2ad_evict = dev->l2ad_start; dev->l2ad_first = B_FALSE; } dev->l2ad_writing = B_TRUE; (void) zio_wait(pio); dev->l2ad_writing = B_FALSE; return (write_asize); } /* * Compresses an L2ARC buffer. * The data to be compressed must be prefilled in l2hdr->b_tmp_cdata and its * size in l2hdr->b_asize. This routine tries to compress the data and * depending on the compression result there are three possible outcomes: * *) The buffer was incompressible. The original l2hdr contents were left * untouched and are ready for writing to an L2 device. * *) The buffer was all-zeros, so there is no need to write it to an L2 * device. To indicate this situation b_tmp_cdata is NULL'ed, b_asize is * set to zero and b_compress is set to ZIO_COMPRESS_EMPTY. * *) Compression succeeded and b_tmp_cdata was replaced with a temporary * data buffer which holds the compressed data to be written, and b_asize * tells us how much data there is. b_compress is set to the appropriate * compression algorithm. Once writing is done, invoke * l2arc_release_cdata_buf on this l2hdr to free this temporary buffer. * * Returns B_TRUE if compression succeeded, or B_FALSE if it didn't (the * buffer was incompressible). */ static boolean_t l2arc_compress_buf(l2arc_buf_hdr_t *l2hdr) { void *cdata; size_t csize, len, rounded; ASSERT(l2hdr->b_compress == ZIO_COMPRESS_OFF); ASSERT(l2hdr->b_tmp_cdata != NULL); len = l2hdr->b_asize; cdata = zio_data_buf_alloc(len); csize = zio_compress_data(ZIO_COMPRESS_LZ4, l2hdr->b_tmp_cdata, cdata, l2hdr->b_asize); if (csize == 0) { /* zero block, indicate that there's nothing to write */ zio_data_buf_free(cdata, len); l2hdr->b_compress = ZIO_COMPRESS_EMPTY; l2hdr->b_asize = 0; l2hdr->b_tmp_cdata = NULL; ARCSTAT_BUMP(arcstat_l2_compress_zeros); return (B_TRUE); } rounded = P2ROUNDUP(csize, (size_t)1 << l2hdr->b_dev->l2ad_vdev->vdev_ashift); if (rounded < len) { /* * Compression succeeded, we'll keep the cdata around for * writing and release it afterwards. */ if (rounded > csize) { bzero((char *)cdata + csize, rounded - csize); csize = rounded; } l2hdr->b_compress = ZIO_COMPRESS_LZ4; l2hdr->b_asize = csize; l2hdr->b_tmp_cdata = cdata; ARCSTAT_BUMP(arcstat_l2_compress_successes); return (B_TRUE); } else { /* * Compression failed, release the compressed buffer. * l2hdr will be left unmodified. */ zio_data_buf_free(cdata, len); ARCSTAT_BUMP(arcstat_l2_compress_failures); return (B_FALSE); } } /* * Decompresses a zio read back from an l2arc device. On success, the * underlying zio's io_data buffer is overwritten by the uncompressed * version. On decompression error (corrupt compressed stream), the * zio->io_error value is set to signal an I/O error. * * Please note that the compressed data stream is not checksummed, so * if the underlying device is experiencing data corruption, we may feed * corrupt data to the decompressor, so the decompressor needs to be * able to handle this situation (LZ4 does). */ static void l2arc_decompress_zio(zio_t *zio, arc_buf_hdr_t *hdr, enum zio_compress c) { ASSERT(L2ARC_IS_VALID_COMPRESS(c)); if (zio->io_error != 0) { /* * An io error has occured, just restore the original io * size in preparation for a main pool read. */ zio->io_orig_size = zio->io_size = hdr->b_size; return; } if (c == ZIO_COMPRESS_EMPTY) { /* * An empty buffer results in a null zio, which means we * need to fill its io_data after we're done restoring the * buffer's contents. */ ASSERT(hdr->b_buf != NULL); bzero(hdr->b_buf->b_data, hdr->b_size); zio->io_data = zio->io_orig_data = hdr->b_buf->b_data; } else { ASSERT(zio->io_data != NULL); /* * We copy the compressed data from the start of the arc buffer * (the zio_read will have pulled in only what we need, the * rest is garbage which we will overwrite at decompression) * and then decompress back to the ARC data buffer. This way we * can minimize copying by simply decompressing back over the * original compressed data (rather than decompressing to an * aux buffer and then copying back the uncompressed buffer, * which is likely to be much larger). */ uint64_t csize; void *cdata; csize = zio->io_size; cdata = zio_data_buf_alloc(csize); bcopy(zio->io_data, cdata, csize); if (zio_decompress_data(c, cdata, zio->io_data, csize, hdr->b_size) != 0) zio->io_error = EIO; zio_data_buf_free(cdata, csize); } /* Restore the expected uncompressed IO size. */ zio->io_orig_size = zio->io_size = hdr->b_size; } /* * Releases the temporary b_tmp_cdata buffer in an l2arc header structure. * This buffer serves as a temporary holder of compressed data while * the buffer entry is being written to an l2arc device. Once that is * done, we can dispose of it. */ static void l2arc_release_cdata_buf(arc_buf_hdr_t *ab) { l2arc_buf_hdr_t *l2hdr = ab->b_l2hdr; ASSERT(L2ARC_IS_VALID_COMPRESS(l2hdr->b_compress)); if (l2hdr->b_compress != ZIO_COMPRESS_EMPTY) { /* * If the data was compressed, then we've allocated a * temporary buffer for it, so now we need to release it. */ ASSERT(l2hdr->b_tmp_cdata != NULL); zio_data_buf_free(l2hdr->b_tmp_cdata, ab->b_size); l2hdr->b_tmp_cdata = NULL; } else { ASSERT(l2hdr->b_tmp_cdata == NULL); } } /* * This thread feeds the L2ARC at regular intervals. This is the beating * heart of the L2ARC. */ static void l2arc_feed_thread(void *dummy __unused) { callb_cpr_t cpr; l2arc_dev_t *dev; spa_t *spa; uint64_t size, wrote; clock_t begin, next = ddi_get_lbolt(); boolean_t headroom_boost = B_FALSE; CALLB_CPR_INIT(&cpr, &l2arc_feed_thr_lock, callb_generic_cpr, FTAG); mutex_enter(&l2arc_feed_thr_lock); while (l2arc_thread_exit == 0) { CALLB_CPR_SAFE_BEGIN(&cpr); (void) cv_timedwait(&l2arc_feed_thr_cv, &l2arc_feed_thr_lock, next - ddi_get_lbolt()); CALLB_CPR_SAFE_END(&cpr, &l2arc_feed_thr_lock); next = ddi_get_lbolt() + hz; /* * Quick check for L2ARC devices. */ mutex_enter(&l2arc_dev_mtx); if (l2arc_ndev == 0) { mutex_exit(&l2arc_dev_mtx); continue; } mutex_exit(&l2arc_dev_mtx); begin = ddi_get_lbolt(); /* * This selects the next l2arc device to write to, and in * doing so the next spa to feed from: dev->l2ad_spa. This * will return NULL if there are now no l2arc devices or if * they are all faulted. * * If a device is returned, its spa's config lock is also * held to prevent device removal. l2arc_dev_get_next() * will grab and release l2arc_dev_mtx. */ if ((dev = l2arc_dev_get_next()) == NULL) continue; spa = dev->l2ad_spa; ASSERT(spa != NULL); /* * If the pool is read-only then force the feed thread to * sleep a little longer. */ if (!spa_writeable(spa)) { next = ddi_get_lbolt() + 5 * l2arc_feed_secs * hz; spa_config_exit(spa, SCL_L2ARC, dev); continue; } /* * Avoid contributing to memory pressure. */ if (arc_reclaim_needed()) { ARCSTAT_BUMP(arcstat_l2_abort_lowmem); spa_config_exit(spa, SCL_L2ARC, dev); continue; } ARCSTAT_BUMP(arcstat_l2_feeds); size = l2arc_write_size(); /* * Evict L2ARC buffers that will be overwritten. */ l2arc_evict(dev, size, B_FALSE); /* * Write ARC buffers. */ wrote = l2arc_write_buffers(spa, dev, size, &headroom_boost); /* * Calculate interval between writes. */ next = l2arc_write_interval(begin, size, wrote); spa_config_exit(spa, SCL_L2ARC, dev); } l2arc_thread_exit = 0; cv_broadcast(&l2arc_feed_thr_cv); CALLB_CPR_EXIT(&cpr); /* drops l2arc_feed_thr_lock */ thread_exit(); } boolean_t l2arc_vdev_present(vdev_t *vd) { l2arc_dev_t *dev; mutex_enter(&l2arc_dev_mtx); for (dev = list_head(l2arc_dev_list); dev != NULL; dev = list_next(l2arc_dev_list, dev)) { if (dev->l2ad_vdev == vd) break; } mutex_exit(&l2arc_dev_mtx); return (dev != NULL); } /* * Add a vdev for use by the L2ARC. By this point the spa has already * validated the vdev and opened it. */ void l2arc_add_vdev(spa_t *spa, vdev_t *vd) { l2arc_dev_t *adddev; ASSERT(!l2arc_vdev_present(vd)); vdev_ashift_optimize(vd); /* * Create a new l2arc device entry. */ adddev = kmem_zalloc(sizeof (l2arc_dev_t), KM_SLEEP); adddev->l2ad_spa = spa; adddev->l2ad_vdev = vd; adddev->l2ad_start = VDEV_LABEL_START_SIZE; adddev->l2ad_end = VDEV_LABEL_START_SIZE + vdev_get_min_asize(vd); adddev->l2ad_hand = adddev->l2ad_start; adddev->l2ad_evict = adddev->l2ad_start; adddev->l2ad_first = B_TRUE; adddev->l2ad_writing = B_FALSE; /* * This is a list of all ARC buffers that are still valid on the * device. */ adddev->l2ad_buflist = kmem_zalloc(sizeof (list_t), KM_SLEEP); list_create(adddev->l2ad_buflist, sizeof (arc_buf_hdr_t), offsetof(arc_buf_hdr_t, b_l2node)); vdev_space_update(vd, 0, 0, adddev->l2ad_end - adddev->l2ad_hand); /* * Add device to global list */ mutex_enter(&l2arc_dev_mtx); list_insert_head(l2arc_dev_list, adddev); atomic_inc_64(&l2arc_ndev); mutex_exit(&l2arc_dev_mtx); } /* * Remove a vdev from the L2ARC. */ void l2arc_remove_vdev(vdev_t *vd) { l2arc_dev_t *dev, *nextdev, *remdev = NULL; /* * Find the device by vdev */ mutex_enter(&l2arc_dev_mtx); for (dev = list_head(l2arc_dev_list); dev; dev = nextdev) { nextdev = list_next(l2arc_dev_list, dev); if (vd == dev->l2ad_vdev) { remdev = dev; break; } } ASSERT(remdev != NULL); /* * Remove device from global list */ list_remove(l2arc_dev_list, remdev); l2arc_dev_last = NULL; /* may have been invalidated */ atomic_dec_64(&l2arc_ndev); mutex_exit(&l2arc_dev_mtx); /* * Clear all buflists and ARC references. L2ARC device flush. */ l2arc_evict(remdev, 0, B_TRUE); list_destroy(remdev->l2ad_buflist); kmem_free(remdev->l2ad_buflist, sizeof (list_t)); kmem_free(remdev, sizeof (l2arc_dev_t)); } void l2arc_init(void) { l2arc_thread_exit = 0; l2arc_ndev = 0; l2arc_writes_sent = 0; l2arc_writes_done = 0; mutex_init(&l2arc_feed_thr_lock, NULL, MUTEX_DEFAULT, NULL); cv_init(&l2arc_feed_thr_cv, NULL, CV_DEFAULT, NULL); mutex_init(&l2arc_dev_mtx, NULL, MUTEX_DEFAULT, NULL); mutex_init(&l2arc_buflist_mtx, NULL, MUTEX_DEFAULT, NULL); mutex_init(&l2arc_free_on_write_mtx, NULL, MUTEX_DEFAULT, NULL); l2arc_dev_list = &L2ARC_dev_list; l2arc_free_on_write = &L2ARC_free_on_write; list_create(l2arc_dev_list, sizeof (l2arc_dev_t), offsetof(l2arc_dev_t, l2ad_node)); list_create(l2arc_free_on_write, sizeof (l2arc_data_free_t), offsetof(l2arc_data_free_t, l2df_list_node)); } void l2arc_fini(void) { /* * This is called from dmu_fini(), which is called from spa_fini(); * Because of this, we can assume that all l2arc devices have * already been removed when the pools themselves were removed. */ l2arc_do_free_on_write(); mutex_destroy(&l2arc_feed_thr_lock); cv_destroy(&l2arc_feed_thr_cv); mutex_destroy(&l2arc_dev_mtx); mutex_destroy(&l2arc_buflist_mtx); mutex_destroy(&l2arc_free_on_write_mtx); list_destroy(l2arc_dev_list); list_destroy(l2arc_free_on_write); } void l2arc_start(void) { if (!(spa_mode_global & FWRITE)) return; (void) thread_create(NULL, 0, l2arc_feed_thread, NULL, 0, &p0, TS_RUN, minclsyspri); } void l2arc_stop(void) { if (!(spa_mode_global & FWRITE)) return; mutex_enter(&l2arc_feed_thr_lock); cv_signal(&l2arc_feed_thr_cv); /* kick thread out of startup */ l2arc_thread_exit = 1; while (l2arc_thread_exit != 0) cv_wait(&l2arc_feed_thr_cv, &l2arc_feed_thr_lock); mutex_exit(&l2arc_feed_thr_lock); } Index: projects/clang350-import/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c =================================================================== --- projects/clang350-import/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c (revision 275748) +++ projects/clang350-import/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c (revision 275749) @@ -1,648 +1,648 @@ /* * CDDL HEADER START * * The contents of this file are subject to the terms of the * Common Development and Distribution License (the "License"). * You may not use this file except in compliance with the License. * * You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE * or http://www.opensolaris.org/os/licensing. * See the License for the specific language governing permissions * and limitations under the License. * * When distributing Covered Code, include this CDDL HEADER in each * file and include the License file at usr/src/OPENSOLARIS.LICENSE. * If applicable, add the following below this CDDL HEADER, with the * fields enclosed by brackets "[]" replaced with your own identifying * information: Portions Copyright [yyyy] [name of copyright owner] * * CDDL HEADER END */ /* * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. * Copyright (c) 2012, 2014 by Delphix. All rights reserved. */ #include #include #include #include #include #include #include #include #include #include #include #include #include #include int zfs_pd_blks_max = 100; typedef struct prefetch_data { kmutex_t pd_mtx; kcondvar_t pd_cv; int pd_blks_max; int pd_blks_fetched; int pd_flags; boolean_t pd_cancel; boolean_t pd_exited; } prefetch_data_t; typedef struct traverse_data { spa_t *td_spa; uint64_t td_objset; blkptr_t *td_rootbp; uint64_t td_min_txg; zbookmark_phys_t *td_resume; int td_flags; prefetch_data_t *td_pfd; boolean_t td_paused; uint64_t td_hole_birth_enabled_txg; blkptr_cb_t *td_func; void *td_arg; } traverse_data_t; static int traverse_dnode(traverse_data_t *td, const dnode_phys_t *dnp, uint64_t objset, uint64_t object); static void prefetch_dnode_metadata(traverse_data_t *td, const dnode_phys_t *, uint64_t objset, uint64_t object); static int traverse_zil_block(zilog_t *zilog, blkptr_t *bp, void *arg, uint64_t claim_txg) { traverse_data_t *td = arg; zbookmark_phys_t zb; if (BP_IS_HOLE(bp)) return (0); if (claim_txg == 0 && bp->blk_birth >= spa_first_txg(td->td_spa)) return (0); SET_BOOKMARK(&zb, td->td_objset, ZB_ZIL_OBJECT, ZB_ZIL_LEVEL, bp->blk_cksum.zc_word[ZIL_ZC_SEQ]); (void) td->td_func(td->td_spa, zilog, bp, &zb, NULL, td->td_arg); return (0); } static int traverse_zil_record(zilog_t *zilog, lr_t *lrc, void *arg, uint64_t claim_txg) { traverse_data_t *td = arg; if (lrc->lrc_txtype == TX_WRITE) { lr_write_t *lr = (lr_write_t *)lrc; blkptr_t *bp = &lr->lr_blkptr; zbookmark_phys_t zb; if (BP_IS_HOLE(bp)) return (0); if (claim_txg == 0 || bp->blk_birth < claim_txg) return (0); SET_BOOKMARK(&zb, td->td_objset, lr->lr_foid, ZB_ZIL_LEVEL, lr->lr_offset / BP_GET_LSIZE(bp)); (void) td->td_func(td->td_spa, zilog, bp, &zb, NULL, td->td_arg); } return (0); } static void traverse_zil(traverse_data_t *td, zil_header_t *zh) { uint64_t claim_txg = zh->zh_claim_txg; zilog_t *zilog; /* * We only want to visit blocks that have been claimed but not yet * replayed; plus, in read-only mode, blocks that are already stable. */ if (claim_txg == 0 && spa_writeable(td->td_spa)) return; zilog = zil_alloc(spa_get_dsl(td->td_spa)->dp_meta_objset, zh); (void) zil_parse(zilog, traverse_zil_block, traverse_zil_record, td, claim_txg); zil_free(zilog); } typedef enum resume_skip { RESUME_SKIP_ALL, RESUME_SKIP_NONE, RESUME_SKIP_CHILDREN } resume_skip_t; /* * Returns RESUME_SKIP_ALL if td indicates that we are resuming a traversal and * the block indicated by zb does not need to be visited at all. Returns * RESUME_SKIP_CHILDREN if we are resuming a post traversal and we reach the * resume point. This indicates that this block should be visited but not its * children (since they must have been visited in a previous traversal). * Otherwise returns RESUME_SKIP_NONE. */ static resume_skip_t resume_skip_check(traverse_data_t *td, const dnode_phys_t *dnp, const zbookmark_phys_t *zb) { if (td->td_resume != NULL && !ZB_IS_ZERO(td->td_resume)) { /* * If we already visited this bp & everything below, * don't bother doing it again. */ if (zbookmark_is_before(dnp, zb, td->td_resume)) return (RESUME_SKIP_ALL); /* * If we found the block we're trying to resume from, zero * the bookmark out to indicate that we have resumed. */ if (bcmp(zb, td->td_resume, sizeof (*zb)) == 0) { bzero(td->td_resume, sizeof (*zb)); if (td->td_flags & TRAVERSE_POST) return (RESUME_SKIP_CHILDREN); } } return (RESUME_SKIP_NONE); } static void traverse_prefetch_metadata(traverse_data_t *td, const blkptr_t *bp, const zbookmark_phys_t *zb) { uint32_t flags = ARC_NOWAIT | ARC_PREFETCH; if (!(td->td_flags & TRAVERSE_PREFETCH_METADATA)) return; /* * If we are in the process of resuming, don't prefetch, because * some children will not be needed (and in fact may have already * been freed). */ if (td->td_resume != NULL && !ZB_IS_ZERO(td->td_resume)) return; if (BP_IS_HOLE(bp) || bp->blk_birth <= td->td_min_txg) return; if (BP_GET_LEVEL(bp) == 0 && BP_GET_TYPE(bp) != DMU_OT_DNODE) return; (void) arc_read(NULL, td->td_spa, bp, NULL, NULL, ZIO_PRIORITY_ASYNC_READ, ZIO_FLAG_CANFAIL, &flags, zb); } static boolean_t prefetch_needed(prefetch_data_t *pfd, const blkptr_t *bp) { ASSERT(pfd->pd_flags & TRAVERSE_PREFETCH_DATA); if (BP_IS_HOLE(bp) || BP_IS_EMBEDDED(bp) || BP_GET_TYPE(bp) == DMU_OT_INTENT_LOG) return (B_FALSE); return (B_TRUE); } static int traverse_visitbp(traverse_data_t *td, const dnode_phys_t *dnp, const blkptr_t *bp, const zbookmark_phys_t *zb) { zbookmark_phys_t czb; int err = 0; arc_buf_t *buf = NULL; prefetch_data_t *pd = td->td_pfd; boolean_t hard = td->td_flags & TRAVERSE_HARD; switch (resume_skip_check(td, dnp, zb)) { case RESUME_SKIP_ALL: return (0); case RESUME_SKIP_CHILDREN: goto post; case RESUME_SKIP_NONE: break; default: ASSERT(0); } if (bp->blk_birth == 0) { /* * Since this block has a birth time of 0 it must be a * hole created before the SPA_FEATURE_HOLE_BIRTH * feature was enabled. If SPA_FEATURE_HOLE_BIRTH * was enabled before the min_txg for this traveral we * know the hole must have been created before the * min_txg for this traveral, so we can skip it. If * SPA_FEATURE_HOLE_BIRTH was enabled after the min_txg * for this traveral we cannot tell if the hole was * created before or after the min_txg for this * traversal, so we cannot skip it. */ if (td->td_hole_birth_enabled_txg < td->td_min_txg) return (0); } else if (bp->blk_birth <= td->td_min_txg) { return (0); } if (pd != NULL && !pd->pd_exited && prefetch_needed(pd, bp)) { mutex_enter(&pd->pd_mtx); ASSERT(pd->pd_blks_fetched >= 0); while (pd->pd_blks_fetched == 0 && !pd->pd_exited) cv_wait(&pd->pd_cv, &pd->pd_mtx); pd->pd_blks_fetched--; cv_broadcast(&pd->pd_cv); mutex_exit(&pd->pd_mtx); } if (BP_IS_HOLE(bp)) { err = td->td_func(td->td_spa, NULL, bp, zb, dnp, td->td_arg); if (err != 0) goto post; return (0); } if (td->td_flags & TRAVERSE_PRE) { err = td->td_func(td->td_spa, NULL, bp, zb, dnp, td->td_arg); if (err == TRAVERSE_VISIT_NO_CHILDREN) return (0); if (err != 0) goto post; } if (BP_GET_LEVEL(bp) > 0) { uint32_t flags = ARC_WAIT; int i; blkptr_t *cbp; int epb = BP_GET_LSIZE(bp) >> SPA_BLKPTRSHIFT; err = arc_read(NULL, td->td_spa, bp, arc_getbuf_func, &buf, ZIO_PRIORITY_ASYNC_READ, ZIO_FLAG_CANFAIL, &flags, zb); if (err != 0) goto post; cbp = buf->b_data; for (i = 0; i < epb; i++) { SET_BOOKMARK(&czb, zb->zb_objset, zb->zb_object, zb->zb_level - 1, zb->zb_blkid * epb + i); traverse_prefetch_metadata(td, &cbp[i], &czb); } /* recursively visitbp() blocks below this */ for (i = 0; i < epb; i++) { SET_BOOKMARK(&czb, zb->zb_objset, zb->zb_object, zb->zb_level - 1, zb->zb_blkid * epb + i); err = traverse_visitbp(td, dnp, &cbp[i], &czb); if (err != 0) break; } } else if (BP_GET_TYPE(bp) == DMU_OT_DNODE) { uint32_t flags = ARC_WAIT; int i; int epb = BP_GET_LSIZE(bp) >> DNODE_SHIFT; err = arc_read(NULL, td->td_spa, bp, arc_getbuf_func, &buf, ZIO_PRIORITY_ASYNC_READ, ZIO_FLAG_CANFAIL, &flags, zb); if (err != 0) goto post; dnp = buf->b_data; for (i = 0; i < epb; i++) { prefetch_dnode_metadata(td, &dnp[i], zb->zb_objset, zb->zb_blkid * epb + i); } /* recursively visitbp() blocks below this */ for (i = 0; i < epb; i++) { err = traverse_dnode(td, &dnp[i], zb->zb_objset, zb->zb_blkid * epb + i); if (err != 0) break; } } else if (BP_GET_TYPE(bp) == DMU_OT_OBJSET) { uint32_t flags = ARC_WAIT; objset_phys_t *osp; dnode_phys_t *dnp; err = arc_read(NULL, td->td_spa, bp, arc_getbuf_func, &buf, ZIO_PRIORITY_ASYNC_READ, ZIO_FLAG_CANFAIL, &flags, zb); if (err != 0) goto post; osp = buf->b_data; dnp = &osp->os_meta_dnode; prefetch_dnode_metadata(td, dnp, zb->zb_objset, DMU_META_DNODE_OBJECT); if (arc_buf_size(buf) >= sizeof (objset_phys_t)) { prefetch_dnode_metadata(td, &osp->os_groupused_dnode, zb->zb_objset, DMU_GROUPUSED_OBJECT); prefetch_dnode_metadata(td, &osp->os_userused_dnode, zb->zb_objset, DMU_USERUSED_OBJECT); } err = traverse_dnode(td, dnp, zb->zb_objset, DMU_META_DNODE_OBJECT); if (err == 0 && arc_buf_size(buf) >= sizeof (objset_phys_t)) { dnp = &osp->os_groupused_dnode; err = traverse_dnode(td, dnp, zb->zb_objset, DMU_GROUPUSED_OBJECT); } if (err == 0 && arc_buf_size(buf) >= sizeof (objset_phys_t)) { dnp = &osp->os_userused_dnode; err = traverse_dnode(td, dnp, zb->zb_objset, DMU_USERUSED_OBJECT); } } if (buf) (void) arc_buf_remove_ref(buf, &buf); post: if (err == 0 && (td->td_flags & TRAVERSE_POST)) err = td->td_func(td->td_spa, NULL, bp, zb, dnp, td->td_arg); if (hard && (err == EIO || err == ECKSUM)) { /* * Ignore this disk error as requested by the HARD flag, * and continue traversal. */ err = 0; } /* * If we are stopping here, set td_resume. */ if (td->td_resume != NULL && err != 0 && !td->td_paused) { td->td_resume->zb_objset = zb->zb_objset; td->td_resume->zb_object = zb->zb_object; td->td_resume->zb_level = 0; /* * If we have stopped on an indirect block (e.g. due to * i/o error), we have not visited anything below it. * Set the bookmark to the first level-0 block that we need * to visit. This way, the resuming code does not need to * deal with resuming from indirect blocks. */ td->td_resume->zb_blkid = zb->zb_blkid << (zb->zb_level * (dnp->dn_indblkshift - SPA_BLKPTRSHIFT)); td->td_paused = B_TRUE; } return (err); } static void prefetch_dnode_metadata(traverse_data_t *td, const dnode_phys_t *dnp, uint64_t objset, uint64_t object) { int j; zbookmark_phys_t czb; for (j = 0; j < dnp->dn_nblkptr; j++) { SET_BOOKMARK(&czb, objset, object, dnp->dn_nlevels - 1, j); traverse_prefetch_metadata(td, &dnp->dn_blkptr[j], &czb); } if (dnp->dn_flags & DNODE_FLAG_SPILL_BLKPTR) { SET_BOOKMARK(&czb, objset, object, 0, DMU_SPILL_BLKID); traverse_prefetch_metadata(td, &dnp->dn_spill, &czb); } } static int traverse_dnode(traverse_data_t *td, const dnode_phys_t *dnp, uint64_t objset, uint64_t object) { int j, err = 0; zbookmark_phys_t czb; for (j = 0; j < dnp->dn_nblkptr; j++) { SET_BOOKMARK(&czb, objset, object, dnp->dn_nlevels - 1, j); err = traverse_visitbp(td, dnp, &dnp->dn_blkptr[j], &czb); if (err != 0) break; } - if (dnp->dn_flags & DNODE_FLAG_SPILL_BLKPTR) { + if (err == 0 && dnp->dn_flags & DNODE_FLAG_SPILL_BLKPTR) { SET_BOOKMARK(&czb, objset, object, 0, DMU_SPILL_BLKID); err = traverse_visitbp(td, dnp, &dnp->dn_spill, &czb); } return (err); } /* ARGSUSED */ static int traverse_prefetcher(spa_t *spa, zilog_t *zilog, const blkptr_t *bp, const zbookmark_phys_t *zb, const dnode_phys_t *dnp, void *arg) { prefetch_data_t *pfd = arg; uint32_t aflags = ARC_NOWAIT | ARC_PREFETCH; ASSERT(pfd->pd_blks_fetched >= 0); if (pfd->pd_cancel) return (SET_ERROR(EINTR)); if (!prefetch_needed(pfd, bp)) return (0); mutex_enter(&pfd->pd_mtx); while (!pfd->pd_cancel && pfd->pd_blks_fetched >= pfd->pd_blks_max) cv_wait(&pfd->pd_cv, &pfd->pd_mtx); pfd->pd_blks_fetched++; cv_broadcast(&pfd->pd_cv); mutex_exit(&pfd->pd_mtx); (void) arc_read(NULL, spa, bp, NULL, NULL, ZIO_PRIORITY_ASYNC_READ, ZIO_FLAG_CANFAIL | ZIO_FLAG_SPECULATIVE, &aflags, zb); return (0); } static void traverse_prefetch_thread(void *arg) { traverse_data_t *td_main = arg; traverse_data_t td = *td_main; zbookmark_phys_t czb; td.td_func = traverse_prefetcher; td.td_arg = td_main->td_pfd; td.td_pfd = NULL; SET_BOOKMARK(&czb, td.td_objset, ZB_ROOT_OBJECT, ZB_ROOT_LEVEL, ZB_ROOT_BLKID); (void) traverse_visitbp(&td, NULL, td.td_rootbp, &czb); mutex_enter(&td_main->td_pfd->pd_mtx); td_main->td_pfd->pd_exited = B_TRUE; cv_broadcast(&td_main->td_pfd->pd_cv); mutex_exit(&td_main->td_pfd->pd_mtx); } /* * NB: dataset must not be changing on-disk (eg, is a snapshot or we are * in syncing context). */ static int traverse_impl(spa_t *spa, dsl_dataset_t *ds, uint64_t objset, blkptr_t *rootbp, uint64_t txg_start, zbookmark_phys_t *resume, int flags, blkptr_cb_t func, void *arg) { traverse_data_t td; prefetch_data_t pd = { 0 }; zbookmark_phys_t czb; int err; ASSERT(ds == NULL || objset == ds->ds_object); ASSERT(!(flags & TRAVERSE_PRE) || !(flags & TRAVERSE_POST)); /* * The data prefetching mechanism (the prefetch thread) is incompatible * with resuming from a bookmark. */ ASSERT(resume == NULL || !(flags & TRAVERSE_PREFETCH_DATA)); td.td_spa = spa; td.td_objset = objset; td.td_rootbp = rootbp; td.td_min_txg = txg_start; td.td_resume = resume; td.td_func = func; td.td_arg = arg; td.td_pfd = &pd; td.td_flags = flags; td.td_paused = B_FALSE; if (spa_feature_is_active(spa, SPA_FEATURE_HOLE_BIRTH)) { VERIFY(spa_feature_enabled_txg(spa, SPA_FEATURE_HOLE_BIRTH, &td.td_hole_birth_enabled_txg)); } else { td.td_hole_birth_enabled_txg = 0; } pd.pd_blks_max = zfs_pd_blks_max; pd.pd_flags = flags; mutex_init(&pd.pd_mtx, NULL, MUTEX_DEFAULT, NULL); cv_init(&pd.pd_cv, NULL, CV_DEFAULT, NULL); /* See comment on ZIL traversal in dsl_scan_visitds. */ if (ds != NULL && !dsl_dataset_is_snapshot(ds) && !BP_IS_HOLE(rootbp)) { uint32_t flags = ARC_WAIT; objset_phys_t *osp; arc_buf_t *buf; err = arc_read(NULL, td.td_spa, rootbp, arc_getbuf_func, &buf, ZIO_PRIORITY_ASYNC_READ, ZIO_FLAG_CANFAIL, &flags, NULL); if (err != 0) return (err); osp = buf->b_data; traverse_zil(&td, &osp->os_zil_header); (void) arc_buf_remove_ref(buf, &buf); } if (!(flags & TRAVERSE_PREFETCH_DATA) || 0 == taskq_dispatch(system_taskq, traverse_prefetch_thread, &td, TQ_NOQUEUE)) pd.pd_exited = B_TRUE; SET_BOOKMARK(&czb, td.td_objset, ZB_ROOT_OBJECT, ZB_ROOT_LEVEL, ZB_ROOT_BLKID); err = traverse_visitbp(&td, NULL, rootbp, &czb); mutex_enter(&pd.pd_mtx); pd.pd_cancel = B_TRUE; cv_broadcast(&pd.pd_cv); while (!pd.pd_exited) cv_wait(&pd.pd_cv, &pd.pd_mtx); mutex_exit(&pd.pd_mtx); mutex_destroy(&pd.pd_mtx); cv_destroy(&pd.pd_cv); return (err); } /* * NB: dataset must not be changing on-disk (eg, is a snapshot or we are * in syncing context). */ int traverse_dataset(dsl_dataset_t *ds, uint64_t txg_start, int flags, blkptr_cb_t func, void *arg) { return (traverse_impl(ds->ds_dir->dd_pool->dp_spa, ds, ds->ds_object, &ds->ds_phys->ds_bp, txg_start, NULL, flags, func, arg)); } int traverse_dataset_destroyed(spa_t *spa, blkptr_t *blkptr, uint64_t txg_start, zbookmark_phys_t *resume, int flags, blkptr_cb_t func, void *arg) { return (traverse_impl(spa, NULL, ZB_DESTROYED_OBJSET, blkptr, txg_start, resume, flags, func, arg)); } /* * NB: pool must not be changing on-disk (eg, from zdb or sync context). */ int traverse_pool(spa_t *spa, uint64_t txg_start, int flags, blkptr_cb_t func, void *arg) { int err; uint64_t obj; dsl_pool_t *dp = spa_get_dsl(spa); objset_t *mos = dp->dp_meta_objset; boolean_t hard = (flags & TRAVERSE_HARD); /* visit the MOS */ err = traverse_impl(spa, NULL, 0, spa_get_rootblkptr(spa), txg_start, NULL, flags, func, arg); if (err != 0) return (err); /* visit each dataset */ for (obj = 1; err == 0; err = dmu_object_next(mos, &obj, FALSE, txg_start)) { dmu_object_info_t doi; err = dmu_object_info(mos, obj, &doi); if (err != 0) { if (hard) continue; break; } if (doi.doi_bonus_type == DMU_OT_DSL_DATASET) { dsl_dataset_t *ds; uint64_t txg = txg_start; dsl_pool_config_enter(dp, FTAG); err = dsl_dataset_hold_obj(dp, obj, FTAG, &ds); dsl_pool_config_exit(dp, FTAG); if (err != 0) { if (hard) continue; break; } if (ds->ds_phys->ds_prev_snap_txg > txg) txg = ds->ds_phys->ds_prev_snap_txg; err = traverse_dataset(ds, txg, flags, func, arg); dsl_dataset_rele(ds, FTAG); if (err != 0) break; } } if (err == ESRCH) err = 0; return (err); } Index: projects/clang350-import/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dnode_sync.c =================================================================== --- projects/clang350-import/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dnode_sync.c (revision 275748) +++ projects/clang350-import/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dnode_sync.c (revision 275749) @@ -1,728 +1,727 @@ /* * CDDL HEADER START * * The contents of this file are subject to the terms of the * Common Development and Distribution License (the "License"). * You may not use this file except in compliance with the License. * * You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE * or http://www.opensolaris.org/os/licensing. * See the License for the specific language governing permissions * and limitations under the License. * * When distributing Covered Code, include this CDDL HEADER in each * file and include the License file at usr/src/OPENSOLARIS.LICENSE. * If applicable, add the following below this CDDL HEADER, with the * fields enclosed by brackets "[]" replaced with your own identifying * information: Portions Copyright [yyyy] [name of copyright owner] * * CDDL HEADER END */ /* * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. * Copyright (c) 2012, 2014 by Delphix. All rights reserved. */ #include #include #include #include #include #include #include #include #include #include static void dnode_increase_indirection(dnode_t *dn, dmu_tx_t *tx) { dmu_buf_impl_t *db; int txgoff = tx->tx_txg & TXG_MASK; int nblkptr = dn->dn_phys->dn_nblkptr; int old_toplvl = dn->dn_phys->dn_nlevels - 1; int new_level = dn->dn_next_nlevels[txgoff]; int i; rw_enter(&dn->dn_struct_rwlock, RW_WRITER); /* this dnode can't be paged out because it's dirty */ ASSERT(dn->dn_phys->dn_type != DMU_OT_NONE); ASSERT(RW_WRITE_HELD(&dn->dn_struct_rwlock)); ASSERT(new_level > 1 && dn->dn_phys->dn_nlevels > 0); db = dbuf_hold_level(dn, dn->dn_phys->dn_nlevels, 0, FTAG); ASSERT(db != NULL); dn->dn_phys->dn_nlevels = new_level; dprintf("os=%p obj=%llu, increase to %d\n", dn->dn_objset, dn->dn_object, dn->dn_phys->dn_nlevels); /* check for existing blkptrs in the dnode */ for (i = 0; i < nblkptr; i++) if (!BP_IS_HOLE(&dn->dn_phys->dn_blkptr[i])) break; if (i != nblkptr) { /* transfer dnode's block pointers to new indirect block */ (void) dbuf_read(db, NULL, DB_RF_MUST_SUCCEED|DB_RF_HAVESTRUCT); ASSERT(db->db.db_data); ASSERT(arc_released(db->db_buf)); ASSERT3U(sizeof (blkptr_t) * nblkptr, <=, db->db.db_size); bcopy(dn->dn_phys->dn_blkptr, db->db.db_data, sizeof (blkptr_t) * nblkptr); arc_buf_freeze(db->db_buf); } /* set dbuf's parent pointers to new indirect buf */ for (i = 0; i < nblkptr; i++) { dmu_buf_impl_t *child = dbuf_find(dn, old_toplvl, i); if (child == NULL) continue; #ifdef DEBUG DB_DNODE_ENTER(child); ASSERT3P(DB_DNODE(child), ==, dn); DB_DNODE_EXIT(child); #endif /* DEBUG */ if (child->db_parent && child->db_parent != dn->dn_dbuf) { ASSERT(child->db_parent->db_level == db->db_level); ASSERT(child->db_blkptr != &dn->dn_phys->dn_blkptr[child->db_blkid]); mutex_exit(&child->db_mtx); continue; } ASSERT(child->db_parent == NULL || child->db_parent == dn->dn_dbuf); child->db_parent = db; dbuf_add_ref(db, child); if (db->db.db_data) child->db_blkptr = (blkptr_t *)db->db.db_data + i; else child->db_blkptr = NULL; dprintf_dbuf_bp(child, child->db_blkptr, "changed db_blkptr to new indirect %s", ""); mutex_exit(&child->db_mtx); } bzero(dn->dn_phys->dn_blkptr, sizeof (blkptr_t) * nblkptr); dbuf_rele(db, FTAG); rw_exit(&dn->dn_struct_rwlock); } static void free_blocks(dnode_t *dn, blkptr_t *bp, int num, dmu_tx_t *tx) { dsl_dataset_t *ds = dn->dn_objset->os_dsl_dataset; uint64_t bytesfreed = 0; dprintf("ds=%p obj=%llx num=%d\n", ds, dn->dn_object, num); for (int i = 0; i < num; i++, bp++) { if (BP_IS_HOLE(bp)) continue; bytesfreed += dsl_dataset_block_kill(ds, bp, tx, B_FALSE); ASSERT3U(bytesfreed, <=, DN_USED_BYTES(dn->dn_phys)); /* * Save some useful information on the holes being * punched, including logical size, type, and indirection * level. Retaining birth time enables detection of when * holes are punched for reducing the number of free * records transmitted during a zfs send. */ uint64_t lsize = BP_GET_LSIZE(bp); dmu_object_type_t type = BP_GET_TYPE(bp); uint64_t lvl = BP_GET_LEVEL(bp); bzero(bp, sizeof (blkptr_t)); if (spa_feature_is_active(dn->dn_objset->os_spa, SPA_FEATURE_HOLE_BIRTH)) { BP_SET_LSIZE(bp, lsize); BP_SET_TYPE(bp, type); BP_SET_LEVEL(bp, lvl); BP_SET_BIRTH(bp, dmu_tx_get_txg(tx), 0); } } dnode_diduse_space(dn, -bytesfreed); } #ifdef ZFS_DEBUG static void free_verify(dmu_buf_impl_t *db, uint64_t start, uint64_t end, dmu_tx_t *tx) { int off, num; int i, err, epbs; uint64_t txg = tx->tx_txg; dnode_t *dn; DB_DNODE_ENTER(db); dn = DB_DNODE(db); epbs = dn->dn_phys->dn_indblkshift - SPA_BLKPTRSHIFT; off = start - (db->db_blkid * 1<=, 0); ASSERT3U(num, >=, 0); ASSERT3U(db->db_level, >, 0); ASSERT3U(db->db.db_size, ==, 1 << dn->dn_phys->dn_indblkshift); ASSERT3U(off+num, <=, db->db.db_size >> SPA_BLKPTRSHIFT); ASSERT(db->db_blkptr != NULL); for (i = off; i < off+num; i++) { uint64_t *buf; dmu_buf_impl_t *child; dbuf_dirty_record_t *dr; int j; ASSERT(db->db_level == 1); rw_enter(&dn->dn_struct_rwlock, RW_READER); err = dbuf_hold_impl(dn, db->db_level-1, (db->db_blkid << epbs) + i, TRUE, FTAG, &child); rw_exit(&dn->dn_struct_rwlock); if (err == ENOENT) continue; ASSERT(err == 0); ASSERT(child->db_level == 0); dr = child->db_last_dirty; while (dr && dr->dr_txg > txg) dr = dr->dr_next; ASSERT(dr == NULL || dr->dr_txg == txg); /* data_old better be zeroed */ if (dr) { buf = dr->dt.dl.dr_data->b_data; for (j = 0; j < child->db.db_size >> 3; j++) { if (buf[j] != 0) { panic("freed data not zero: " "child=%p i=%d off=%d num=%d\n", (void *)child, i, off, num); } } } /* * db_data better be zeroed unless it's dirty in a * future txg. */ mutex_enter(&child->db_mtx); buf = child->db.db_data; if (buf != NULL && child->db_state != DB_FILL && child->db_last_dirty == NULL) { for (j = 0; j < child->db.db_size >> 3; j++) { if (buf[j] != 0) { panic("freed data not zero: " "child=%p i=%d off=%d num=%d\n", (void *)child, i, off, num); } } } mutex_exit(&child->db_mtx); dbuf_rele(child, FTAG); } DB_DNODE_EXIT(db); } #endif static void free_children(dmu_buf_impl_t *db, uint64_t blkid, uint64_t nblks, dmu_tx_t *tx) { dnode_t *dn; blkptr_t *bp; dmu_buf_impl_t *subdb; uint64_t start, end, dbstart, dbend, i; int epbs, shift; /* * There is a small possibility that this block will not be cached: * 1 - if level > 1 and there are no children with level <= 1 * 2 - if this block was evicted since we read it from * dmu_tx_hold_free(). */ if (db->db_state != DB_CACHED) (void) dbuf_read(db, NULL, DB_RF_MUST_SUCCEED); dbuf_release_bp(db); bp = db->db.db_data; DB_DNODE_ENTER(db); dn = DB_DNODE(db); epbs = dn->dn_phys->dn_indblkshift - SPA_BLKPTRSHIFT; shift = (db->db_level - 1) * epbs; dbstart = db->db_blkid << epbs; start = blkid >> shift; if (dbstart < start) { bp += start - dbstart; } else { start = dbstart; } dbend = ((db->db_blkid + 1) << epbs) - 1; end = (blkid + nblks - 1) >> shift; if (dbend <= end) end = dbend; ASSERT3U(start, <=, end); if (db->db_level == 1) { FREE_VERIFY(db, start, end, tx); free_blocks(dn, bp, end-start+1, tx); } else { for (i = start; i <= end; i++, bp++) { if (BP_IS_HOLE(bp)) continue; rw_enter(&dn->dn_struct_rwlock, RW_READER); VERIFY0(dbuf_hold_impl(dn, db->db_level - 1, i, B_TRUE, FTAG, &subdb)); rw_exit(&dn->dn_struct_rwlock); ASSERT3P(bp, ==, subdb->db_blkptr); free_children(subdb, blkid, nblks, tx); dbuf_rele(subdb, FTAG); } } /* If this whole block is free, free ourself too. */ for (i = 0, bp = db->db.db_data; i < 1 << epbs; i++, bp++) { if (!BP_IS_HOLE(bp)) break; } if (i == 1 << epbs) { /* didn't find any non-holes */ bzero(db->db.db_data, db->db.db_size); free_blocks(dn, db->db_blkptr, 1, tx); } else { /* * Partial block free; must be marked dirty so that it * will be written out. */ ASSERT(db->db_dirtycnt > 0); } DB_DNODE_EXIT(db); arc_buf_freeze(db->db_buf); } /* * Traverse the indicated range of the provided file * and "free" all the blocks contained there. */ static void dnode_sync_free_range_impl(dnode_t *dn, uint64_t blkid, uint64_t nblks, dmu_tx_t *tx) { blkptr_t *bp = dn->dn_phys->dn_blkptr; int dnlevel = dn->dn_phys->dn_nlevels; boolean_t trunc = B_FALSE; if (blkid > dn->dn_phys->dn_maxblkid) return; ASSERT(dn->dn_phys->dn_maxblkid < UINT64_MAX); if (blkid + nblks > dn->dn_phys->dn_maxblkid) { nblks = dn->dn_phys->dn_maxblkid - blkid + 1; trunc = B_TRUE; } /* There are no indirect blocks in the object */ if (dnlevel == 1) { if (blkid >= dn->dn_phys->dn_nblkptr) { /* this range was never made persistent */ return; } ASSERT3U(blkid + nblks, <=, dn->dn_phys->dn_nblkptr); free_blocks(dn, bp + blkid, nblks, tx); } else { int shift = (dnlevel - 1) * (dn->dn_phys->dn_indblkshift - SPA_BLKPTRSHIFT); int start = blkid >> shift; int end = (blkid + nblks - 1) >> shift; dmu_buf_impl_t *db; ASSERT(start < dn->dn_phys->dn_nblkptr); bp += start; for (int i = start; i <= end; i++, bp++) { if (BP_IS_HOLE(bp)) continue; rw_enter(&dn->dn_struct_rwlock, RW_READER); VERIFY0(dbuf_hold_impl(dn, dnlevel - 1, i, TRUE, FTAG, &db)); rw_exit(&dn->dn_struct_rwlock); free_children(db, blkid, nblks, tx); dbuf_rele(db, FTAG); } } if (trunc) { dn->dn_phys->dn_maxblkid = blkid == 0 ? 0 : blkid - 1; uint64_t off = (dn->dn_phys->dn_maxblkid + 1) * (dn->dn_phys->dn_datablkszsec << SPA_MINBLOCKSHIFT); ASSERT(off < dn->dn_phys->dn_maxblkid || dn->dn_phys->dn_maxblkid == 0 || dnode_next_offset(dn, 0, &off, 1, 1, 0) != 0); } } typedef struct dnode_sync_free_range_arg { dnode_t *dsfra_dnode; dmu_tx_t *dsfra_tx; } dnode_sync_free_range_arg_t; static void dnode_sync_free_range(void *arg, uint64_t blkid, uint64_t nblks) { dnode_sync_free_range_arg_t *dsfra = arg; dnode_t *dn = dsfra->dsfra_dnode; mutex_exit(&dn->dn_mtx); dnode_sync_free_range_impl(dn, blkid, nblks, dsfra->dsfra_tx); mutex_enter(&dn->dn_mtx); } /* * Try to kick all the dnode's dbufs out of the cache... */ void dnode_evict_dbufs(dnode_t *dn) { int progress; int pass = 0; do { dmu_buf_impl_t *db, *db_next; int evicting = FALSE; progress = FALSE; mutex_enter(&dn->dn_dbufs_mtx); for (db = avl_first(&dn->dn_dbufs); db != NULL; db = db_next) { db_next = AVL_NEXT(&dn->dn_dbufs, db); #ifdef DEBUG DB_DNODE_ENTER(db); ASSERT3P(DB_DNODE(db), ==, dn); DB_DNODE_EXIT(db); #endif /* DEBUG */ mutex_enter(&db->db_mtx); if (db->db_state == DB_EVICTING) { progress = TRUE; evicting = TRUE; mutex_exit(&db->db_mtx); } else if (refcount_is_zero(&db->db_holds)) { progress = TRUE; dbuf_clear(db); /* exits db_mtx for us */ } else { mutex_exit(&db->db_mtx); } } /* * NB: we need to drop dn_dbufs_mtx between passes so * that any DB_EVICTING dbufs can make progress. * Ideally, we would have some cv we could wait on, but * since we don't, just wait a bit to give the other * thread a chance to run. */ mutex_exit(&dn->dn_dbufs_mtx); if (evicting) delay(1); pass++; ASSERT(pass < 100); /* sanity check */ } while (progress); rw_enter(&dn->dn_struct_rwlock, RW_WRITER); if (dn->dn_bonus && refcount_is_zero(&dn->dn_bonus->db_holds)) { mutex_enter(&dn->dn_bonus->db_mtx); dbuf_evict(dn->dn_bonus); dn->dn_bonus = NULL; } rw_exit(&dn->dn_struct_rwlock); } static void dnode_undirty_dbufs(list_t *list) { dbuf_dirty_record_t *dr; while (dr = list_head(list)) { dmu_buf_impl_t *db = dr->dr_dbuf; uint64_t txg = dr->dr_txg; if (db->db_level != 0) dnode_undirty_dbufs(&dr->dt.di.dr_children); mutex_enter(&db->db_mtx); /* XXX - use dbuf_undirty()? */ list_remove(list, dr); ASSERT(db->db_last_dirty == dr); db->db_last_dirty = NULL; db->db_dirtycnt -= 1; if (db->db_level == 0) { ASSERT(db->db_blkid == DMU_BONUS_BLKID || dr->dt.dl.dr_data == db->db_buf); dbuf_unoverride(dr); } else { mutex_destroy(&dr->dt.di.dr_mtx); list_destroy(&dr->dt.di.dr_children); } kmem_free(dr, sizeof (dbuf_dirty_record_t)); dbuf_rele_and_unlock(db, (void *)(uintptr_t)txg); } } static void dnode_sync_free(dnode_t *dn, dmu_tx_t *tx) { int txgoff = tx->tx_txg & TXG_MASK; ASSERT(dmu_tx_is_syncing(tx)); /* * Our contents should have been freed in dnode_sync() by the * free range record inserted by the caller of dnode_free(). */ ASSERT0(DN_USED_BYTES(dn->dn_phys)); ASSERT(BP_IS_HOLE(dn->dn_phys->dn_blkptr)); dnode_undirty_dbufs(&dn->dn_dirty_records[txgoff]); dnode_evict_dbufs(dn); ASSERT(avl_is_empty(&dn->dn_dbufs)); ASSERT3P(dn->dn_bonus, ==, NULL); /* * XXX - It would be nice to assert this, but we may still * have residual holds from async evictions from the arc... * * zfs_obj_to_path() also depends on this being * commented out. * * ASSERT3U(refcount_count(&dn->dn_holds), ==, 1); */ /* Undirty next bits */ dn->dn_next_nlevels[txgoff] = 0; dn->dn_next_indblkshift[txgoff] = 0; dn->dn_next_blksz[txgoff] = 0; /* ASSERT(blkptrs are zero); */ ASSERT(dn->dn_phys->dn_type != DMU_OT_NONE); ASSERT(dn->dn_type != DMU_OT_NONE); ASSERT(dn->dn_free_txg > 0); if (dn->dn_allocated_txg != dn->dn_free_txg) dmu_buf_will_dirty(&dn->dn_dbuf->db, tx); bzero(dn->dn_phys, sizeof (dnode_phys_t)); mutex_enter(&dn->dn_mtx); dn->dn_type = DMU_OT_NONE; dn->dn_maxblkid = 0; dn->dn_allocated_txg = 0; dn->dn_free_txg = 0; dn->dn_have_spill = B_FALSE; mutex_exit(&dn->dn_mtx); ASSERT(dn->dn_object != DMU_META_DNODE_OBJECT); dnode_rele(dn, (void *)(uintptr_t)tx->tx_txg); /* * Now that we've released our hold, the dnode may * be evicted, so we musn't access it. */ } /* * Write out the dnode's dirty buffers. */ void dnode_sync(dnode_t *dn, dmu_tx_t *tx) { dnode_phys_t *dnp = dn->dn_phys; int txgoff = tx->tx_txg & TXG_MASK; list_t *list = &dn->dn_dirty_records[txgoff]; static const dnode_phys_t zerodn = { 0 }; boolean_t kill_spill = B_FALSE; ASSERT(dmu_tx_is_syncing(tx)); ASSERT(dnp->dn_type != DMU_OT_NONE || dn->dn_allocated_txg); ASSERT(dnp->dn_type != DMU_OT_NONE || bcmp(dnp, &zerodn, DNODE_SIZE) == 0); DNODE_VERIFY(dn); ASSERT(dn->dn_dbuf == NULL || arc_released(dn->dn_dbuf->db_buf)); if (dmu_objset_userused_enabled(dn->dn_objset) && !DMU_OBJECT_IS_SPECIAL(dn->dn_object)) { mutex_enter(&dn->dn_mtx); dn->dn_oldused = DN_USED_BYTES(dn->dn_phys); dn->dn_oldflags = dn->dn_phys->dn_flags; dn->dn_phys->dn_flags |= DNODE_FLAG_USERUSED_ACCOUNTED; mutex_exit(&dn->dn_mtx); dmu_objset_userquota_get_ids(dn, B_FALSE, tx); } else { /* Once we account for it, we should always account for it. */ ASSERT(!(dn->dn_phys->dn_flags & DNODE_FLAG_USERUSED_ACCOUNTED)); } mutex_enter(&dn->dn_mtx); if (dn->dn_allocated_txg == tx->tx_txg) { /* The dnode is newly allocated or reallocated */ if (dnp->dn_type == DMU_OT_NONE) { /* this is a first alloc, not a realloc */ dnp->dn_nlevels = 1; dnp->dn_nblkptr = dn->dn_nblkptr; } dnp->dn_type = dn->dn_type; dnp->dn_bonustype = dn->dn_bonustype; dnp->dn_bonuslen = dn->dn_bonuslen; } ASSERT(dnp->dn_nlevels > 1 || BP_IS_HOLE(&dnp->dn_blkptr[0]) || BP_IS_EMBEDDED(&dnp->dn_blkptr[0]) || BP_GET_LSIZE(&dnp->dn_blkptr[0]) == dnp->dn_datablkszsec << SPA_MINBLOCKSHIFT); ASSERT(dnp->dn_nlevels < 2 || BP_IS_HOLE(&dnp->dn_blkptr[0]) || BP_GET_LSIZE(&dnp->dn_blkptr[0]) == 1 << dnp->dn_indblkshift); if (dn->dn_next_type[txgoff] != 0) { dnp->dn_type = dn->dn_type; dn->dn_next_type[txgoff] = 0; } if (dn->dn_next_blksz[txgoff] != 0) { ASSERT(P2PHASE(dn->dn_next_blksz[txgoff], SPA_MINBLOCKSIZE) == 0); ASSERT(BP_IS_HOLE(&dnp->dn_blkptr[0]) || dn->dn_maxblkid == 0 || list_head(list) != NULL || dn->dn_next_blksz[txgoff] >> SPA_MINBLOCKSHIFT == dnp->dn_datablkszsec || range_tree_space(dn->dn_free_ranges[txgoff]) != 0); dnp->dn_datablkszsec = dn->dn_next_blksz[txgoff] >> SPA_MINBLOCKSHIFT; dn->dn_next_blksz[txgoff] = 0; } if (dn->dn_next_bonuslen[txgoff] != 0) { if (dn->dn_next_bonuslen[txgoff] == DN_ZERO_BONUSLEN) dnp->dn_bonuslen = 0; else dnp->dn_bonuslen = dn->dn_next_bonuslen[txgoff]; ASSERT(dnp->dn_bonuslen <= DN_MAX_BONUSLEN); dn->dn_next_bonuslen[txgoff] = 0; } if (dn->dn_next_bonustype[txgoff] != 0) { ASSERT(DMU_OT_IS_VALID(dn->dn_next_bonustype[txgoff])); dnp->dn_bonustype = dn->dn_next_bonustype[txgoff]; dn->dn_next_bonustype[txgoff] = 0; } boolean_t freeing_dnode = dn->dn_free_txg > 0 && dn->dn_free_txg <= tx->tx_txg; /* - * We will either remove a spill block when a file is being removed - * or we have been asked to remove it. + * Remove the spill block if we have been explicitly asked to + * remove it, or if the object is being removed. */ - if (dn->dn_rm_spillblk[txgoff] || - ((dnp->dn_flags & DNODE_FLAG_SPILL_BLKPTR) && freeing_dnode)) { - if ((dnp->dn_flags & DNODE_FLAG_SPILL_BLKPTR)) + if (dn->dn_rm_spillblk[txgoff] || freeing_dnode) { + if (dnp->dn_flags & DNODE_FLAG_SPILL_BLKPTR) kill_spill = B_TRUE; dn->dn_rm_spillblk[txgoff] = 0; } if (dn->dn_next_indblkshift[txgoff] != 0) { ASSERT(dnp->dn_nlevels == 1); dnp->dn_indblkshift = dn->dn_next_indblkshift[txgoff]; dn->dn_next_indblkshift[txgoff] = 0; } /* * Just take the live (open-context) values for checksum and compress. * Strictly speaking it's a future leak, but nothing bad happens if we * start using the new checksum or compress algorithm a little early. */ dnp->dn_checksum = dn->dn_checksum; dnp->dn_compress = dn->dn_compress; mutex_exit(&dn->dn_mtx); if (kill_spill) { free_blocks(dn, &dn->dn_phys->dn_spill, 1, tx); mutex_enter(&dn->dn_mtx); dnp->dn_flags &= ~DNODE_FLAG_SPILL_BLKPTR; mutex_exit(&dn->dn_mtx); } /* process all the "freed" ranges in the file */ if (dn->dn_free_ranges[txgoff] != NULL) { dnode_sync_free_range_arg_t dsfra; dsfra.dsfra_dnode = dn; dsfra.dsfra_tx = tx; mutex_enter(&dn->dn_mtx); range_tree_vacate(dn->dn_free_ranges[txgoff], dnode_sync_free_range, &dsfra); range_tree_destroy(dn->dn_free_ranges[txgoff]); dn->dn_free_ranges[txgoff] = NULL; mutex_exit(&dn->dn_mtx); } if (freeing_dnode) { dnode_sync_free(dn, tx); return; } if (dn->dn_next_nlevels[txgoff]) { dnode_increase_indirection(dn, tx); dn->dn_next_nlevels[txgoff] = 0; } if (dn->dn_next_nblkptr[txgoff]) { /* this should only happen on a realloc */ ASSERT(dn->dn_allocated_txg == tx->tx_txg); if (dn->dn_next_nblkptr[txgoff] > dnp->dn_nblkptr) { /* zero the new blkptrs we are gaining */ bzero(dnp->dn_blkptr + dnp->dn_nblkptr, sizeof (blkptr_t) * (dn->dn_next_nblkptr[txgoff] - dnp->dn_nblkptr)); #ifdef ZFS_DEBUG } else { int i; ASSERT(dn->dn_next_nblkptr[txgoff] < dnp->dn_nblkptr); /* the blkptrs we are losing better be unallocated */ for (i = dn->dn_next_nblkptr[txgoff]; i < dnp->dn_nblkptr; i++) ASSERT(BP_IS_HOLE(&dnp->dn_blkptr[i])); #endif } mutex_enter(&dn->dn_mtx); dnp->dn_nblkptr = dn->dn_next_nblkptr[txgoff]; dn->dn_next_nblkptr[txgoff] = 0; mutex_exit(&dn->dn_mtx); } dbuf_sync_list(list, tx); if (!DMU_OBJECT_IS_SPECIAL(dn->dn_object)) { ASSERT3P(list_head(list), ==, NULL); dnode_rele(dn, (void *)(uintptr_t)tx->tx_txg); } /* * Although we have dropped our reference to the dnode, it * can't be evicted until its written, and we haven't yet * initiated the IO for the dnode's dbuf. */ } Index: projects/clang350-import/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_dataset.c =================================================================== --- projects/clang350-import/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_dataset.c (revision 275748) +++ projects/clang350-import/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_dataset.c (revision 275749) @@ -1,3390 +1,3388 @@ /* * CDDL HEADER START * * The contents of this file are subject to the terms of the * Common Development and Distribution License (the "License"). * You may not use this file except in compliance with the License. * * You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE * or http://www.opensolaris.org/os/licensing. * See the License for the specific language governing permissions * and limitations under the License. * * When distributing Covered Code, include this CDDL HEADER in each * file and include the License file at usr/src/OPENSOLARIS.LICENSE. * If applicable, add the following below this CDDL HEADER, with the * fields enclosed by brackets "[]" replaced with your own identifying * information: Portions Copyright [yyyy] [name of copyright owner] * * CDDL HEADER END */ /* * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. * Portions Copyright (c) 2011 Martin Matuska * Copyright (c) 2011, 2014 by Delphix. All rights reserved. * Copyright (c) 2014, Joyent, Inc. All rights reserved. * Copyright (c) 2014 RackTop Systems. */ #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include SYSCTL_DECL(_vfs_zfs); /* * The SPA supports block sizes up to 16MB. However, very large blocks * can have an impact on i/o latency (e.g. tying up a spinning disk for * ~300ms), and also potentially on the memory allocator. Therefore, * we do not allow the recordsize to be set larger than zfs_max_recordsize * (default 1MB). Larger blocks can be created by changing this tunable, * and pools with larger blocks can always be imported and used, regardless * of this setting. */ int zfs_max_recordsize = 1 * 1024 * 1024; SYSCTL_INT(_vfs_zfs, OID_AUTO, max_recordsize, CTLFLAG_RWTUN, &zfs_max_recordsize, 0, "Maximum block size. Expect dragons when tuning this."); #define SWITCH64(x, y) \ { \ uint64_t __tmp = (x); \ (x) = (y); \ (y) = __tmp; \ } #define DS_REF_MAX (1ULL << 62) /* * Figure out how much of this delta should be propogated to the dsl_dir * layer. If there's a refreservation, that space has already been * partially accounted for in our ancestors. */ static int64_t parent_delta(dsl_dataset_t *ds, int64_t delta) { uint64_t old_bytes, new_bytes; if (ds->ds_reserved == 0) return (delta); old_bytes = MAX(ds->ds_phys->ds_unique_bytes, ds->ds_reserved); new_bytes = MAX(ds->ds_phys->ds_unique_bytes + delta, ds->ds_reserved); ASSERT3U(ABS((int64_t)(new_bytes - old_bytes)), <=, ABS(delta)); return (new_bytes - old_bytes); } void dsl_dataset_block_born(dsl_dataset_t *ds, const blkptr_t *bp, dmu_tx_t *tx) { int used = bp_get_dsize_sync(tx->tx_pool->dp_spa, bp); int compressed = BP_GET_PSIZE(bp); int uncompressed = BP_GET_UCSIZE(bp); int64_t delta; dprintf_bp(bp, "ds=%p", ds); ASSERT(dmu_tx_is_syncing(tx)); /* It could have been compressed away to nothing */ if (BP_IS_HOLE(bp)) return; ASSERT(BP_GET_TYPE(bp) != DMU_OT_NONE); ASSERT(DMU_OT_IS_VALID(BP_GET_TYPE(bp))); if (ds == NULL) { dsl_pool_mos_diduse_space(tx->tx_pool, used, compressed, uncompressed); return; } dmu_buf_will_dirty(ds->ds_dbuf, tx); mutex_enter(&ds->ds_lock); delta = parent_delta(ds, used); ds->ds_phys->ds_referenced_bytes += used; ds->ds_phys->ds_compressed_bytes += compressed; ds->ds_phys->ds_uncompressed_bytes += uncompressed; ds->ds_phys->ds_unique_bytes += used; if (BP_GET_LSIZE(bp) > SPA_OLD_MAXBLOCKSIZE) ds->ds_need_large_blocks = B_TRUE; mutex_exit(&ds->ds_lock); dsl_dir_diduse_space(ds->ds_dir, DD_USED_HEAD, delta, compressed, uncompressed, tx); dsl_dir_transfer_space(ds->ds_dir, used - delta, DD_USED_REFRSRV, DD_USED_HEAD, tx); } int dsl_dataset_block_kill(dsl_dataset_t *ds, const blkptr_t *bp, dmu_tx_t *tx, boolean_t async) { int used = bp_get_dsize_sync(tx->tx_pool->dp_spa, bp); int compressed = BP_GET_PSIZE(bp); int uncompressed = BP_GET_UCSIZE(bp); if (BP_IS_HOLE(bp)) return (0); ASSERT(dmu_tx_is_syncing(tx)); ASSERT(bp->blk_birth <= tx->tx_txg); if (ds == NULL) { dsl_free(tx->tx_pool, tx->tx_txg, bp); dsl_pool_mos_diduse_space(tx->tx_pool, -used, -compressed, -uncompressed); return (used); } ASSERT3P(tx->tx_pool, ==, ds->ds_dir->dd_pool); ASSERT(!dsl_dataset_is_snapshot(ds)); dmu_buf_will_dirty(ds->ds_dbuf, tx); if (bp->blk_birth > ds->ds_phys->ds_prev_snap_txg) { int64_t delta; dprintf_bp(bp, "freeing ds=%llu", ds->ds_object); dsl_free(tx->tx_pool, tx->tx_txg, bp); mutex_enter(&ds->ds_lock); ASSERT(ds->ds_phys->ds_unique_bytes >= used || !DS_UNIQUE_IS_ACCURATE(ds)); delta = parent_delta(ds, -used); ds->ds_phys->ds_unique_bytes -= used; mutex_exit(&ds->ds_lock); dsl_dir_diduse_space(ds->ds_dir, DD_USED_HEAD, delta, -compressed, -uncompressed, tx); dsl_dir_transfer_space(ds->ds_dir, -used - delta, DD_USED_REFRSRV, DD_USED_HEAD, tx); } else { dprintf_bp(bp, "putting on dead list: %s", ""); if (async) { /* * We are here as part of zio's write done callback, * which means we're a zio interrupt thread. We can't * call dsl_deadlist_insert() now because it may block * waiting for I/O. Instead, put bp on the deferred * queue and let dsl_pool_sync() finish the job. */ bplist_append(&ds->ds_pending_deadlist, bp); } else { dsl_deadlist_insert(&ds->ds_deadlist, bp, tx); } ASSERT3U(ds->ds_prev->ds_object, ==, ds->ds_phys->ds_prev_snap_obj); ASSERT(ds->ds_prev->ds_phys->ds_num_children > 0); /* if (bp->blk_birth > prev prev snap txg) prev unique += bs */ if (ds->ds_prev->ds_phys->ds_next_snap_obj == ds->ds_object && bp->blk_birth > ds->ds_prev->ds_phys->ds_prev_snap_txg) { dmu_buf_will_dirty(ds->ds_prev->ds_dbuf, tx); mutex_enter(&ds->ds_prev->ds_lock); ds->ds_prev->ds_phys->ds_unique_bytes += used; mutex_exit(&ds->ds_prev->ds_lock); } if (bp->blk_birth > ds->ds_dir->dd_origin_txg) { dsl_dir_transfer_space(ds->ds_dir, used, DD_USED_HEAD, DD_USED_SNAP, tx); } } mutex_enter(&ds->ds_lock); ASSERT3U(ds->ds_phys->ds_referenced_bytes, >=, used); ds->ds_phys->ds_referenced_bytes -= used; ASSERT3U(ds->ds_phys->ds_compressed_bytes, >=, compressed); ds->ds_phys->ds_compressed_bytes -= compressed; ASSERT3U(ds->ds_phys->ds_uncompressed_bytes, >=, uncompressed); ds->ds_phys->ds_uncompressed_bytes -= uncompressed; mutex_exit(&ds->ds_lock); return (used); } uint64_t dsl_dataset_prev_snap_txg(dsl_dataset_t *ds) { uint64_t trysnap = 0; if (ds == NULL) return (0); /* * The snapshot creation could fail, but that would cause an * incorrect FALSE return, which would only result in an * overestimation of the amount of space that an operation would * consume, which is OK. * * There's also a small window where we could miss a pending * snapshot, because we could set the sync task in the quiescing * phase. So this should only be used as a guess. */ if (ds->ds_trysnap_txg > spa_last_synced_txg(ds->ds_dir->dd_pool->dp_spa)) trysnap = ds->ds_trysnap_txg; return (MAX(ds->ds_phys->ds_prev_snap_txg, trysnap)); } boolean_t dsl_dataset_block_freeable(dsl_dataset_t *ds, const blkptr_t *bp, uint64_t blk_birth) { if (blk_birth <= dsl_dataset_prev_snap_txg(ds) || (bp != NULL && BP_IS_HOLE(bp))) return (B_FALSE); ddt_prefetch(dsl_dataset_get_spa(ds), bp); return (B_TRUE); } /* ARGSUSED */ static void dsl_dataset_evict(dmu_buf_t *db, void *dsv) { dsl_dataset_t *ds = dsv; ASSERT(ds->ds_owner == NULL); unique_remove(ds->ds_fsid_guid); if (ds->ds_objset != NULL) dmu_objset_evict(ds->ds_objset); if (ds->ds_prev) { dsl_dataset_rele(ds->ds_prev, ds); ds->ds_prev = NULL; } bplist_destroy(&ds->ds_pending_deadlist); if (ds->ds_phys->ds_deadlist_obj != 0) dsl_deadlist_close(&ds->ds_deadlist); if (ds->ds_dir) dsl_dir_rele(ds->ds_dir, ds); ASSERT(!list_link_active(&ds->ds_synced_link)); if (mutex_owned(&ds->ds_lock)) mutex_exit(&ds->ds_lock); mutex_destroy(&ds->ds_lock); if (mutex_owned(&ds->ds_opening_lock)) mutex_exit(&ds->ds_opening_lock); mutex_destroy(&ds->ds_opening_lock); mutex_destroy(&ds->ds_sendstream_lock); refcount_destroy(&ds->ds_longholds); kmem_free(ds, sizeof (dsl_dataset_t)); } int dsl_dataset_get_snapname(dsl_dataset_t *ds) { dsl_dataset_phys_t *headphys; int err; dmu_buf_t *headdbuf; dsl_pool_t *dp = ds->ds_dir->dd_pool; objset_t *mos = dp->dp_meta_objset; if (ds->ds_snapname[0]) return (0); if (ds->ds_phys->ds_next_snap_obj == 0) return (0); err = dmu_bonus_hold(mos, ds->ds_dir->dd_phys->dd_head_dataset_obj, FTAG, &headdbuf); if (err != 0) return (err); headphys = headdbuf->db_data; err = zap_value_search(dp->dp_meta_objset, headphys->ds_snapnames_zapobj, ds->ds_object, 0, ds->ds_snapname); dmu_buf_rele(headdbuf, FTAG); return (err); } int dsl_dataset_snap_lookup(dsl_dataset_t *ds, const char *name, uint64_t *value) { objset_t *mos = ds->ds_dir->dd_pool->dp_meta_objset; uint64_t snapobj = ds->ds_phys->ds_snapnames_zapobj; matchtype_t mt; int err; if (ds->ds_phys->ds_flags & DS_FLAG_CI_DATASET) mt = MT_FIRST; else mt = MT_EXACT; err = zap_lookup_norm(mos, snapobj, name, 8, 1, value, mt, NULL, 0, NULL); if (err == ENOTSUP && mt == MT_FIRST) err = zap_lookup(mos, snapobj, name, 8, 1, value); return (err); } int dsl_dataset_snap_remove(dsl_dataset_t *ds, const char *name, dmu_tx_t *tx, boolean_t adj_cnt) { objset_t *mos = ds->ds_dir->dd_pool->dp_meta_objset; uint64_t snapobj = ds->ds_phys->ds_snapnames_zapobj; matchtype_t mt; int err; dsl_dir_snap_cmtime_update(ds->ds_dir); if (ds->ds_phys->ds_flags & DS_FLAG_CI_DATASET) mt = MT_FIRST; else mt = MT_EXACT; err = zap_remove_norm(mos, snapobj, name, mt, tx); if (err == ENOTSUP && mt == MT_FIRST) err = zap_remove(mos, snapobj, name, tx); if (err == 0 && adj_cnt) dsl_fs_ss_count_adjust(ds->ds_dir, -1, DD_FIELD_SNAPSHOT_COUNT, tx); return (err); } int dsl_dataset_hold_obj(dsl_pool_t *dp, uint64_t dsobj, void *tag, dsl_dataset_t **dsp) { objset_t *mos = dp->dp_meta_objset; dmu_buf_t *dbuf; dsl_dataset_t *ds; int err; dmu_object_info_t doi; ASSERT(dsl_pool_config_held(dp)); err = dmu_bonus_hold(mos, dsobj, tag, &dbuf); if (err != 0) return (err); /* Make sure dsobj has the correct object type. */ dmu_object_info_from_db(dbuf, &doi); if (doi.doi_bonus_type != DMU_OT_DSL_DATASET) { dmu_buf_rele(dbuf, tag); return (SET_ERROR(EINVAL)); } ds = dmu_buf_get_user(dbuf); if (ds == NULL) { dsl_dataset_t *winner = NULL; ds = kmem_zalloc(sizeof (dsl_dataset_t), KM_SLEEP); ds->ds_dbuf = dbuf; ds->ds_object = dsobj; ds->ds_phys = dbuf->db_data; mutex_init(&ds->ds_lock, NULL, MUTEX_DEFAULT, NULL); mutex_init(&ds->ds_opening_lock, NULL, MUTEX_DEFAULT, NULL); mutex_init(&ds->ds_sendstream_lock, NULL, MUTEX_DEFAULT, NULL); refcount_create(&ds->ds_longholds); bplist_create(&ds->ds_pending_deadlist); dsl_deadlist_open(&ds->ds_deadlist, mos, ds->ds_phys->ds_deadlist_obj); list_create(&ds->ds_sendstreams, sizeof (dmu_sendarg_t), offsetof(dmu_sendarg_t, dsa_link)); if (doi.doi_type == DMU_OTN_ZAP_METADATA) { int zaperr = zap_contains(mos, dsobj, DS_FIELD_LARGE_BLOCKS); if (zaperr != ENOENT) { VERIFY0(zaperr); ds->ds_large_blocks = B_TRUE; } } if (err == 0) { err = dsl_dir_hold_obj(dp, ds->ds_phys->ds_dir_obj, NULL, ds, &ds->ds_dir); } if (err != 0) { mutex_destroy(&ds->ds_lock); mutex_destroy(&ds->ds_opening_lock); mutex_destroy(&ds->ds_sendstream_lock); refcount_destroy(&ds->ds_longholds); bplist_destroy(&ds->ds_pending_deadlist); dsl_deadlist_close(&ds->ds_deadlist); kmem_free(ds, sizeof (dsl_dataset_t)); dmu_buf_rele(dbuf, tag); return (err); } if (!dsl_dataset_is_snapshot(ds)) { ds->ds_snapname[0] = '\0'; if (ds->ds_phys->ds_prev_snap_obj != 0) { err = dsl_dataset_hold_obj(dp, ds->ds_phys->ds_prev_snap_obj, ds, &ds->ds_prev); } if (doi.doi_type == DMU_OTN_ZAP_METADATA) { int zaperr = zap_lookup(mos, ds->ds_object, DS_FIELD_BOOKMARK_NAMES, sizeof (ds->ds_bookmarks), 1, &ds->ds_bookmarks); if (zaperr != ENOENT) VERIFY0(zaperr); } } else { if (zfs_flags & ZFS_DEBUG_SNAPNAMES) err = dsl_dataset_get_snapname(ds); if (err == 0 && ds->ds_phys->ds_userrefs_obj != 0) { err = zap_count( ds->ds_dir->dd_pool->dp_meta_objset, ds->ds_phys->ds_userrefs_obj, &ds->ds_userrefs); } } if (err == 0 && !dsl_dataset_is_snapshot(ds)) { err = dsl_prop_get_int_ds(ds, zfs_prop_to_name(ZFS_PROP_REFRESERVATION), &ds->ds_reserved); if (err == 0) { err = dsl_prop_get_int_ds(ds, zfs_prop_to_name(ZFS_PROP_REFQUOTA), &ds->ds_quota); } } else { ds->ds_reserved = ds->ds_quota = 0; } if (err != 0 || (winner = dmu_buf_set_user_ie(dbuf, ds, &ds->ds_phys, dsl_dataset_evict)) != NULL) { bplist_destroy(&ds->ds_pending_deadlist); dsl_deadlist_close(&ds->ds_deadlist); if (ds->ds_prev) dsl_dataset_rele(ds->ds_prev, ds); dsl_dir_rele(ds->ds_dir, ds); mutex_destroy(&ds->ds_lock); mutex_destroy(&ds->ds_opening_lock); mutex_destroy(&ds->ds_sendstream_lock); refcount_destroy(&ds->ds_longholds); kmem_free(ds, sizeof (dsl_dataset_t)); if (err != 0) { dmu_buf_rele(dbuf, tag); return (err); } ds = winner; } else { ds->ds_fsid_guid = unique_insert(ds->ds_phys->ds_fsid_guid); } } ASSERT3P(ds->ds_dbuf, ==, dbuf); ASSERT3P(ds->ds_phys, ==, dbuf->db_data); ASSERT(ds->ds_phys->ds_prev_snap_obj != 0 || spa_version(dp->dp_spa) < SPA_VERSION_ORIGIN || dp->dp_origin_snap == NULL || ds == dp->dp_origin_snap); *dsp = ds; return (0); } int dsl_dataset_hold(dsl_pool_t *dp, const char *name, void *tag, dsl_dataset_t **dsp) { dsl_dir_t *dd; const char *snapname; uint64_t obj; int err = 0; err = dsl_dir_hold(dp, name, FTAG, &dd, &snapname); if (err != 0) return (err); ASSERT(dsl_pool_config_held(dp)); obj = dd->dd_phys->dd_head_dataset_obj; if (obj != 0) err = dsl_dataset_hold_obj(dp, obj, tag, dsp); else err = SET_ERROR(ENOENT); /* we may be looking for a snapshot */ if (err == 0 && snapname != NULL) { dsl_dataset_t *ds; if (*snapname++ != '@') { dsl_dataset_rele(*dsp, tag); dsl_dir_rele(dd, FTAG); return (SET_ERROR(ENOENT)); } dprintf("looking for snapshot '%s'\n", snapname); err = dsl_dataset_snap_lookup(*dsp, snapname, &obj); if (err == 0) err = dsl_dataset_hold_obj(dp, obj, tag, &ds); dsl_dataset_rele(*dsp, tag); if (err == 0) { mutex_enter(&ds->ds_lock); if (ds->ds_snapname[0] == 0) (void) strlcpy(ds->ds_snapname, snapname, sizeof (ds->ds_snapname)); mutex_exit(&ds->ds_lock); *dsp = ds; } } dsl_dir_rele(dd, FTAG); return (err); } int dsl_dataset_own_obj(dsl_pool_t *dp, uint64_t dsobj, void *tag, dsl_dataset_t **dsp) { int err = dsl_dataset_hold_obj(dp, dsobj, tag, dsp); if (err != 0) return (err); if (!dsl_dataset_tryown(*dsp, tag)) { dsl_dataset_rele(*dsp, tag); *dsp = NULL; return (SET_ERROR(EBUSY)); } return (0); } int dsl_dataset_own(dsl_pool_t *dp, const char *name, void *tag, dsl_dataset_t **dsp) { int err = dsl_dataset_hold(dp, name, tag, dsp); if (err != 0) return (err); if (!dsl_dataset_tryown(*dsp, tag)) { dsl_dataset_rele(*dsp, tag); return (SET_ERROR(EBUSY)); } return (0); } /* * See the comment above dsl_pool_hold() for details. In summary, a long * hold is used to prevent destruction of a dataset while the pool hold * is dropped, allowing other concurrent operations (e.g. spa_sync()). * * The dataset and pool must be held when this function is called. After it * is called, the pool hold may be released while the dataset is still held * and accessed. */ void dsl_dataset_long_hold(dsl_dataset_t *ds, void *tag) { ASSERT(dsl_pool_config_held(ds->ds_dir->dd_pool)); (void) refcount_add(&ds->ds_longholds, tag); } void dsl_dataset_long_rele(dsl_dataset_t *ds, void *tag) { (void) refcount_remove(&ds->ds_longholds, tag); } /* Return B_TRUE if there are any long holds on this dataset. */ boolean_t dsl_dataset_long_held(dsl_dataset_t *ds) { return (!refcount_is_zero(&ds->ds_longholds)); } void dsl_dataset_name(dsl_dataset_t *ds, char *name) { if (ds == NULL) { (void) strcpy(name, "mos"); } else { dsl_dir_name(ds->ds_dir, name); VERIFY0(dsl_dataset_get_snapname(ds)); if (ds->ds_snapname[0]) { (void) strcat(name, "@"); /* * We use a "recursive" mutex so that we * can call dprintf_ds() with ds_lock held. */ if (!MUTEX_HELD(&ds->ds_lock)) { mutex_enter(&ds->ds_lock); (void) strcat(name, ds->ds_snapname); mutex_exit(&ds->ds_lock); } else { (void) strcat(name, ds->ds_snapname); } } } } void dsl_dataset_rele(dsl_dataset_t *ds, void *tag) { dmu_buf_rele(ds->ds_dbuf, tag); } void dsl_dataset_disown(dsl_dataset_t *ds, void *tag) { - ASSERT(ds->ds_owner == tag && ds->ds_dbuf != NULL); + ASSERT3P(ds->ds_owner, ==, tag); + ASSERT(ds->ds_dbuf != NULL); mutex_enter(&ds->ds_lock); ds->ds_owner = NULL; mutex_exit(&ds->ds_lock); dsl_dataset_long_rele(ds, tag); - if (ds->ds_dbuf != NULL) - dsl_dataset_rele(ds, tag); - else - dsl_dataset_evict(NULL, ds); + dsl_dataset_rele(ds, tag); } boolean_t dsl_dataset_tryown(dsl_dataset_t *ds, void *tag) { boolean_t gotit = FALSE; mutex_enter(&ds->ds_lock); if (ds->ds_owner == NULL && !DS_IS_INCONSISTENT(ds)) { ds->ds_owner = tag; dsl_dataset_long_hold(ds, tag); gotit = TRUE; } mutex_exit(&ds->ds_lock); return (gotit); } uint64_t dsl_dataset_create_sync_dd(dsl_dir_t *dd, dsl_dataset_t *origin, uint64_t flags, dmu_tx_t *tx) { dsl_pool_t *dp = dd->dd_pool; dmu_buf_t *dbuf; dsl_dataset_phys_t *dsphys; uint64_t dsobj; objset_t *mos = dp->dp_meta_objset; if (origin == NULL) origin = dp->dp_origin_snap; ASSERT(origin == NULL || origin->ds_dir->dd_pool == dp); ASSERT(origin == NULL || origin->ds_phys->ds_num_children > 0); ASSERT(dmu_tx_is_syncing(tx)); ASSERT(dd->dd_phys->dd_head_dataset_obj == 0); dsobj = dmu_object_alloc(mos, DMU_OT_DSL_DATASET, 0, DMU_OT_DSL_DATASET, sizeof (dsl_dataset_phys_t), tx); VERIFY0(dmu_bonus_hold(mos, dsobj, FTAG, &dbuf)); dmu_buf_will_dirty(dbuf, tx); dsphys = dbuf->db_data; bzero(dsphys, sizeof (dsl_dataset_phys_t)); dsphys->ds_dir_obj = dd->dd_object; dsphys->ds_flags = flags; dsphys->ds_fsid_guid = unique_create(); do { (void) random_get_pseudo_bytes((void*)&dsphys->ds_guid, sizeof (dsphys->ds_guid)); } while (dsphys->ds_guid == 0); dsphys->ds_snapnames_zapobj = zap_create_norm(mos, U8_TEXTPREP_TOUPPER, DMU_OT_DSL_DS_SNAP_MAP, DMU_OT_NONE, 0, tx); dsphys->ds_creation_time = gethrestime_sec(); dsphys->ds_creation_txg = tx->tx_txg == TXG_INITIAL ? 1 : tx->tx_txg; if (origin == NULL) { dsphys->ds_deadlist_obj = dsl_deadlist_alloc(mos, tx); } else { dsl_dataset_t *ohds; /* head of the origin snapshot */ dsphys->ds_prev_snap_obj = origin->ds_object; dsphys->ds_prev_snap_txg = origin->ds_phys->ds_creation_txg; dsphys->ds_referenced_bytes = origin->ds_phys->ds_referenced_bytes; dsphys->ds_compressed_bytes = origin->ds_phys->ds_compressed_bytes; dsphys->ds_uncompressed_bytes = origin->ds_phys->ds_uncompressed_bytes; dsphys->ds_bp = origin->ds_phys->ds_bp; /* * Inherit flags that describe the dataset's contents * (INCONSISTENT) or properties (Case Insensitive). */ dsphys->ds_flags |= origin->ds_phys->ds_flags & (DS_FLAG_INCONSISTENT | DS_FLAG_CI_DATASET); if (origin->ds_large_blocks) dsl_dataset_activate_large_blocks_sync_impl(dsobj, tx); dmu_buf_will_dirty(origin->ds_dbuf, tx); origin->ds_phys->ds_num_children++; VERIFY0(dsl_dataset_hold_obj(dp, origin->ds_dir->dd_phys->dd_head_dataset_obj, FTAG, &ohds)); dsphys->ds_deadlist_obj = dsl_deadlist_clone(&ohds->ds_deadlist, dsphys->ds_prev_snap_txg, dsphys->ds_prev_snap_obj, tx); dsl_dataset_rele(ohds, FTAG); if (spa_version(dp->dp_spa) >= SPA_VERSION_NEXT_CLONES) { if (origin->ds_phys->ds_next_clones_obj == 0) { origin->ds_phys->ds_next_clones_obj = zap_create(mos, DMU_OT_NEXT_CLONES, DMU_OT_NONE, 0, tx); } VERIFY0(zap_add_int(mos, origin->ds_phys->ds_next_clones_obj, dsobj, tx)); } dmu_buf_will_dirty(dd->dd_dbuf, tx); dd->dd_phys->dd_origin_obj = origin->ds_object; if (spa_version(dp->dp_spa) >= SPA_VERSION_DIR_CLONES) { if (origin->ds_dir->dd_phys->dd_clones == 0) { dmu_buf_will_dirty(origin->ds_dir->dd_dbuf, tx); origin->ds_dir->dd_phys->dd_clones = zap_create(mos, DMU_OT_DSL_CLONES, DMU_OT_NONE, 0, tx); } VERIFY0(zap_add_int(mos, origin->ds_dir->dd_phys->dd_clones, dsobj, tx)); } } if (spa_version(dp->dp_spa) >= SPA_VERSION_UNIQUE_ACCURATE) dsphys->ds_flags |= DS_FLAG_UNIQUE_ACCURATE; dmu_buf_rele(dbuf, FTAG); dmu_buf_will_dirty(dd->dd_dbuf, tx); dd->dd_phys->dd_head_dataset_obj = dsobj; return (dsobj); } static void dsl_dataset_zero_zil(dsl_dataset_t *ds, dmu_tx_t *tx) { objset_t *os; VERIFY0(dmu_objset_from_ds(ds, &os)); bzero(&os->os_zil_header, sizeof (os->os_zil_header)); dsl_dataset_dirty(ds, tx); } uint64_t dsl_dataset_create_sync(dsl_dir_t *pdd, const char *lastname, dsl_dataset_t *origin, uint64_t flags, cred_t *cr, dmu_tx_t *tx) { dsl_pool_t *dp = pdd->dd_pool; uint64_t dsobj, ddobj; dsl_dir_t *dd; ASSERT(dmu_tx_is_syncing(tx)); ASSERT(lastname[0] != '@'); ddobj = dsl_dir_create_sync(dp, pdd, lastname, tx); VERIFY0(dsl_dir_hold_obj(dp, ddobj, lastname, FTAG, &dd)); dsobj = dsl_dataset_create_sync_dd(dd, origin, flags & ~DS_CREATE_FLAG_NODIRTY, tx); dsl_deleg_set_create_perms(dd, tx, cr); /* * Since we're creating a new node we know it's a leaf, so we can * initialize the counts if the limit feature is active. */ if (spa_feature_is_active(dp->dp_spa, SPA_FEATURE_FS_SS_LIMIT)) { uint64_t cnt = 0; objset_t *os = dd->dd_pool->dp_meta_objset; dsl_dir_zapify(dd, tx); VERIFY0(zap_add(os, dd->dd_object, DD_FIELD_FILESYSTEM_COUNT, sizeof (cnt), 1, &cnt, tx)); VERIFY0(zap_add(os, dd->dd_object, DD_FIELD_SNAPSHOT_COUNT, sizeof (cnt), 1, &cnt, tx)); } dsl_dir_rele(dd, FTAG); /* * If we are creating a clone, make sure we zero out any stale * data from the origin snapshots zil header. */ if (origin != NULL && !(flags & DS_CREATE_FLAG_NODIRTY)) { dsl_dataset_t *ds; VERIFY0(dsl_dataset_hold_obj(dp, dsobj, FTAG, &ds)); dsl_dataset_zero_zil(ds, tx); dsl_dataset_rele(ds, FTAG); } return (dsobj); } #ifdef __FreeBSD__ /* FreeBSD ioctl compat begin */ struct destroyarg { nvlist_t *nvl; const char *snapname; }; static int dsl_check_snap_cb(const char *name, void *arg) { struct destroyarg *da = arg; dsl_dataset_t *ds; char *dsname; dsname = kmem_asprintf("%s@%s", name, da->snapname); fnvlist_add_boolean(da->nvl, dsname); kmem_free(dsname, strlen(dsname) + 1); return (0); } int dmu_get_recursive_snaps_nvl(char *fsname, const char *snapname, nvlist_t *snaps) { struct destroyarg *da; int err; da = kmem_zalloc(sizeof (struct destroyarg), KM_SLEEP); da->nvl = snaps; da->snapname = snapname; err = dmu_objset_find(fsname, dsl_check_snap_cb, da, DS_FIND_CHILDREN); kmem_free(da, sizeof (struct destroyarg)); return (err); } /* FreeBSD ioctl compat end */ #endif /* __FreeBSD__ */ /* * The unique space in the head dataset can be calculated by subtracting * the space used in the most recent snapshot, that is still being used * in this file system, from the space currently in use. To figure out * the space in the most recent snapshot still in use, we need to take * the total space used in the snapshot and subtract out the space that * has been freed up since the snapshot was taken. */ void dsl_dataset_recalc_head_uniq(dsl_dataset_t *ds) { uint64_t mrs_used; uint64_t dlused, dlcomp, dluncomp; ASSERT(!dsl_dataset_is_snapshot(ds)); if (ds->ds_phys->ds_prev_snap_obj != 0) mrs_used = ds->ds_prev->ds_phys->ds_referenced_bytes; else mrs_used = 0; dsl_deadlist_space(&ds->ds_deadlist, &dlused, &dlcomp, &dluncomp); ASSERT3U(dlused, <=, mrs_used); ds->ds_phys->ds_unique_bytes = ds->ds_phys->ds_referenced_bytes - (mrs_used - dlused); if (spa_version(ds->ds_dir->dd_pool->dp_spa) >= SPA_VERSION_UNIQUE_ACCURATE) ds->ds_phys->ds_flags |= DS_FLAG_UNIQUE_ACCURATE; } void dsl_dataset_remove_from_next_clones(dsl_dataset_t *ds, uint64_t obj, dmu_tx_t *tx) { objset_t *mos = ds->ds_dir->dd_pool->dp_meta_objset; uint64_t count; int err; ASSERT(ds->ds_phys->ds_num_children >= 2); err = zap_remove_int(mos, ds->ds_phys->ds_next_clones_obj, obj, tx); /* * The err should not be ENOENT, but a bug in a previous version * of the code could cause upgrade_clones_cb() to not set * ds_next_snap_obj when it should, leading to a missing entry. * If we knew that the pool was created after * SPA_VERSION_NEXT_CLONES, we could assert that it isn't * ENOENT. However, at least we can check that we don't have * too many entries in the next_clones_obj even after failing to * remove this one. */ if (err != ENOENT) VERIFY0(err); ASSERT0(zap_count(mos, ds->ds_phys->ds_next_clones_obj, &count)); ASSERT3U(count, <=, ds->ds_phys->ds_num_children - 2); } blkptr_t * dsl_dataset_get_blkptr(dsl_dataset_t *ds) { return (&ds->ds_phys->ds_bp); } void dsl_dataset_set_blkptr(dsl_dataset_t *ds, blkptr_t *bp, dmu_tx_t *tx) { ASSERT(dmu_tx_is_syncing(tx)); /* If it's the meta-objset, set dp_meta_rootbp */ if (ds == NULL) { tx->tx_pool->dp_meta_rootbp = *bp; } else { dmu_buf_will_dirty(ds->ds_dbuf, tx); ds->ds_phys->ds_bp = *bp; } } spa_t * dsl_dataset_get_spa(dsl_dataset_t *ds) { return (ds->ds_dir->dd_pool->dp_spa); } void dsl_dataset_dirty(dsl_dataset_t *ds, dmu_tx_t *tx) { dsl_pool_t *dp; if (ds == NULL) /* this is the meta-objset */ return; ASSERT(ds->ds_objset != NULL); if (ds->ds_phys->ds_next_snap_obj != 0) panic("dirtying snapshot!"); dp = ds->ds_dir->dd_pool; if (txg_list_add(&dp->dp_dirty_datasets, ds, tx->tx_txg)) { /* up the hold count until we can be written out */ dmu_buf_add_ref(ds->ds_dbuf, ds); } } boolean_t dsl_dataset_is_dirty(dsl_dataset_t *ds) { for (int t = 0; t < TXG_SIZE; t++) { if (txg_list_member(&ds->ds_dir->dd_pool->dp_dirty_datasets, ds, t)) return (B_TRUE); } return (B_FALSE); } static int dsl_dataset_snapshot_reserve_space(dsl_dataset_t *ds, dmu_tx_t *tx) { uint64_t asize; if (!dmu_tx_is_syncing(tx)) return (0); /* * If there's an fs-only reservation, any blocks that might become * owned by the snapshot dataset must be accommodated by space * outside of the reservation. */ ASSERT(ds->ds_reserved == 0 || DS_UNIQUE_IS_ACCURATE(ds)); asize = MIN(ds->ds_phys->ds_unique_bytes, ds->ds_reserved); if (asize > dsl_dir_space_available(ds->ds_dir, NULL, 0, TRUE)) return (SET_ERROR(ENOSPC)); /* * Propagate any reserved space for this snapshot to other * snapshot checks in this sync group. */ if (asize > 0) dsl_dir_willuse_space(ds->ds_dir, asize, tx); return (0); } typedef struct dsl_dataset_snapshot_arg { nvlist_t *ddsa_snaps; nvlist_t *ddsa_props; nvlist_t *ddsa_errors; cred_t *ddsa_cr; } dsl_dataset_snapshot_arg_t; int dsl_dataset_snapshot_check_impl(dsl_dataset_t *ds, const char *snapname, dmu_tx_t *tx, boolean_t recv, uint64_t cnt, cred_t *cr) { int error; uint64_t value; ds->ds_trysnap_txg = tx->tx_txg; if (!dmu_tx_is_syncing(tx)) return (0); /* * We don't allow multiple snapshots of the same txg. If there * is already one, try again. */ if (ds->ds_phys->ds_prev_snap_txg >= tx->tx_txg) return (SET_ERROR(EAGAIN)); /* * Check for conflicting snapshot name. */ error = dsl_dataset_snap_lookup(ds, snapname, &value); if (error == 0) return (SET_ERROR(EEXIST)); if (error != ENOENT) return (error); /* * We don't allow taking snapshots of inconsistent datasets, such as * those into which we are currently receiving. However, if we are * creating this snapshot as part of a receive, this check will be * executed atomically with respect to the completion of the receive * itself but prior to the clearing of DS_FLAG_INCONSISTENT; in this * case we ignore this, knowing it will be fixed up for us shortly in * dmu_recv_end_sync(). */ if (!recv && DS_IS_INCONSISTENT(ds)) return (SET_ERROR(EBUSY)); /* * Skip the check for temporary snapshots or if we have already checked * the counts in dsl_dataset_snapshot_check. This means we really only * check the count here when we're receiving a stream. */ if (cnt != 0 && cr != NULL) { error = dsl_fs_ss_limit_check(ds->ds_dir, cnt, ZFS_PROP_SNAPSHOT_LIMIT, NULL, cr); if (error != 0) return (error); } error = dsl_dataset_snapshot_reserve_space(ds, tx); if (error != 0) return (error); return (0); } static int dsl_dataset_snapshot_check(void *arg, dmu_tx_t *tx) { dsl_dataset_snapshot_arg_t *ddsa = arg; dsl_pool_t *dp = dmu_tx_pool(tx); nvpair_t *pair; int rv = 0; /* * Pre-compute how many total new snapshots will be created for each * level in the tree and below. This is needed for validating the * snapshot limit when either taking a recursive snapshot or when * taking multiple snapshots. * * The problem is that the counts are not actually adjusted when * we are checking, only when we finally sync. For a single snapshot, * this is easy, the count will increase by 1 at each node up the tree, * but its more complicated for the recursive/multiple snapshot case. * * The dsl_fs_ss_limit_check function does recursively check the count * at each level up the tree but since it is validating each snapshot * independently we need to be sure that we are validating the complete * count for the entire set of snapshots. We do this by rolling up the * counts for each component of the name into an nvlist and then * checking each of those cases with the aggregated count. * * This approach properly handles not only the recursive snapshot * case (where we get all of those on the ddsa_snaps list) but also * the sibling case (e.g. snapshot a/b and a/c so that we will also * validate the limit on 'a' using a count of 2). * * We validate the snapshot names in the third loop and only report * name errors once. */ if (dmu_tx_is_syncing(tx)) { nvlist_t *cnt_track = NULL; cnt_track = fnvlist_alloc(); /* Rollup aggregated counts into the cnt_track list */ for (pair = nvlist_next_nvpair(ddsa->ddsa_snaps, NULL); pair != NULL; pair = nvlist_next_nvpair(ddsa->ddsa_snaps, pair)) { char *pdelim; uint64_t val; char nm[MAXPATHLEN]; (void) strlcpy(nm, nvpair_name(pair), sizeof (nm)); pdelim = strchr(nm, '@'); if (pdelim == NULL) continue; *pdelim = '\0'; do { if (nvlist_lookup_uint64(cnt_track, nm, &val) == 0) { /* update existing entry */ fnvlist_add_uint64(cnt_track, nm, val + 1); } else { /* add to list */ fnvlist_add_uint64(cnt_track, nm, 1); } pdelim = strrchr(nm, '/'); if (pdelim != NULL) *pdelim = '\0'; } while (pdelim != NULL); } /* Check aggregated counts at each level */ for (pair = nvlist_next_nvpair(cnt_track, NULL); pair != NULL; pair = nvlist_next_nvpair(cnt_track, pair)) { int error = 0; char *name; uint64_t cnt = 0; dsl_dataset_t *ds; name = nvpair_name(pair); cnt = fnvpair_value_uint64(pair); ASSERT(cnt > 0); error = dsl_dataset_hold(dp, name, FTAG, &ds); if (error == 0) { error = dsl_fs_ss_limit_check(ds->ds_dir, cnt, ZFS_PROP_SNAPSHOT_LIMIT, NULL, ddsa->ddsa_cr); dsl_dataset_rele(ds, FTAG); } if (error != 0) { if (ddsa->ddsa_errors != NULL) fnvlist_add_int32(ddsa->ddsa_errors, name, error); rv = error; /* only report one error for this check */ break; } } nvlist_free(cnt_track); } for (pair = nvlist_next_nvpair(ddsa->ddsa_snaps, NULL); pair != NULL; pair = nvlist_next_nvpair(ddsa->ddsa_snaps, pair)) { int error = 0; dsl_dataset_t *ds; char *name, *atp; char dsname[MAXNAMELEN]; name = nvpair_name(pair); if (strlen(name) >= MAXNAMELEN) error = SET_ERROR(ENAMETOOLONG); if (error == 0) { atp = strchr(name, '@'); if (atp == NULL) error = SET_ERROR(EINVAL); if (error == 0) (void) strlcpy(dsname, name, atp - name + 1); } if (error == 0) error = dsl_dataset_hold(dp, dsname, FTAG, &ds); if (error == 0) { /* passing 0/NULL skips dsl_fs_ss_limit_check */ error = dsl_dataset_snapshot_check_impl(ds, atp + 1, tx, B_FALSE, 0, NULL); dsl_dataset_rele(ds, FTAG); } if (error != 0) { if (ddsa->ddsa_errors != NULL) { fnvlist_add_int32(ddsa->ddsa_errors, name, error); } rv = error; } } return (rv); } void dsl_dataset_snapshot_sync_impl(dsl_dataset_t *ds, const char *snapname, dmu_tx_t *tx) { static zil_header_t zero_zil; dsl_pool_t *dp = ds->ds_dir->dd_pool; dmu_buf_t *dbuf; dsl_dataset_phys_t *dsphys; uint64_t dsobj, crtxg; objset_t *mos = dp->dp_meta_objset; objset_t *os; ASSERT(RRW_WRITE_HELD(&dp->dp_config_rwlock)); /* * If we are on an old pool, the zil must not be active, in which * case it will be zeroed. Usually zil_suspend() accomplishes this. */ ASSERT(spa_version(dmu_tx_pool(tx)->dp_spa) >= SPA_VERSION_FAST_SNAP || dmu_objset_from_ds(ds, &os) != 0 || bcmp(&os->os_phys->os_zil_header, &zero_zil, sizeof (zero_zil)) == 0); dsl_fs_ss_count_adjust(ds->ds_dir, 1, DD_FIELD_SNAPSHOT_COUNT, tx); /* * The origin's ds_creation_txg has to be < TXG_INITIAL */ if (strcmp(snapname, ORIGIN_DIR_NAME) == 0) crtxg = 1; else crtxg = tx->tx_txg; dsobj = dmu_object_alloc(mos, DMU_OT_DSL_DATASET, 0, DMU_OT_DSL_DATASET, sizeof (dsl_dataset_phys_t), tx); VERIFY0(dmu_bonus_hold(mos, dsobj, FTAG, &dbuf)); dmu_buf_will_dirty(dbuf, tx); dsphys = dbuf->db_data; bzero(dsphys, sizeof (dsl_dataset_phys_t)); dsphys->ds_dir_obj = ds->ds_dir->dd_object; dsphys->ds_fsid_guid = unique_create(); do { (void) random_get_pseudo_bytes((void*)&dsphys->ds_guid, sizeof (dsphys->ds_guid)); } while (dsphys->ds_guid == 0); dsphys->ds_prev_snap_obj = ds->ds_phys->ds_prev_snap_obj; dsphys->ds_prev_snap_txg = ds->ds_phys->ds_prev_snap_txg; dsphys->ds_next_snap_obj = ds->ds_object; dsphys->ds_num_children = 1; dsphys->ds_creation_time = gethrestime_sec(); dsphys->ds_creation_txg = crtxg; dsphys->ds_deadlist_obj = ds->ds_phys->ds_deadlist_obj; dsphys->ds_referenced_bytes = ds->ds_phys->ds_referenced_bytes; dsphys->ds_compressed_bytes = ds->ds_phys->ds_compressed_bytes; dsphys->ds_uncompressed_bytes = ds->ds_phys->ds_uncompressed_bytes; dsphys->ds_flags = ds->ds_phys->ds_flags; dsphys->ds_bp = ds->ds_phys->ds_bp; dmu_buf_rele(dbuf, FTAG); if (ds->ds_large_blocks) dsl_dataset_activate_large_blocks_sync_impl(dsobj, tx); ASSERT3U(ds->ds_prev != 0, ==, ds->ds_phys->ds_prev_snap_obj != 0); if (ds->ds_prev) { uint64_t next_clones_obj = ds->ds_prev->ds_phys->ds_next_clones_obj; ASSERT(ds->ds_prev->ds_phys->ds_next_snap_obj == ds->ds_object || ds->ds_prev->ds_phys->ds_num_children > 1); if (ds->ds_prev->ds_phys->ds_next_snap_obj == ds->ds_object) { dmu_buf_will_dirty(ds->ds_prev->ds_dbuf, tx); ASSERT3U(ds->ds_phys->ds_prev_snap_txg, ==, ds->ds_prev->ds_phys->ds_creation_txg); ds->ds_prev->ds_phys->ds_next_snap_obj = dsobj; } else if (next_clones_obj != 0) { dsl_dataset_remove_from_next_clones(ds->ds_prev, dsphys->ds_next_snap_obj, tx); VERIFY0(zap_add_int(mos, next_clones_obj, dsobj, tx)); } } /* * If we have a reference-reservation on this dataset, we will * need to increase the amount of refreservation being charged * since our unique space is going to zero. */ if (ds->ds_reserved) { int64_t delta; ASSERT(DS_UNIQUE_IS_ACCURATE(ds)); delta = MIN(ds->ds_phys->ds_unique_bytes, ds->ds_reserved); dsl_dir_diduse_space(ds->ds_dir, DD_USED_REFRSRV, delta, 0, 0, tx); } dmu_buf_will_dirty(ds->ds_dbuf, tx); ds->ds_phys->ds_deadlist_obj = dsl_deadlist_clone(&ds->ds_deadlist, UINT64_MAX, ds->ds_phys->ds_prev_snap_obj, tx); dsl_deadlist_close(&ds->ds_deadlist); dsl_deadlist_open(&ds->ds_deadlist, mos, ds->ds_phys->ds_deadlist_obj); dsl_deadlist_add_key(&ds->ds_deadlist, ds->ds_phys->ds_prev_snap_txg, tx); ASSERT3U(ds->ds_phys->ds_prev_snap_txg, <, tx->tx_txg); ds->ds_phys->ds_prev_snap_obj = dsobj; ds->ds_phys->ds_prev_snap_txg = crtxg; ds->ds_phys->ds_unique_bytes = 0; if (spa_version(dp->dp_spa) >= SPA_VERSION_UNIQUE_ACCURATE) ds->ds_phys->ds_flags |= DS_FLAG_UNIQUE_ACCURATE; VERIFY0(zap_add(mos, ds->ds_phys->ds_snapnames_zapobj, snapname, 8, 1, &dsobj, tx)); if (ds->ds_prev) dsl_dataset_rele(ds->ds_prev, ds); VERIFY0(dsl_dataset_hold_obj(dp, ds->ds_phys->ds_prev_snap_obj, ds, &ds->ds_prev)); dsl_scan_ds_snapshotted(ds, tx); dsl_dir_snap_cmtime_update(ds->ds_dir); spa_history_log_internal_ds(ds->ds_prev, "snapshot", tx, ""); } static void dsl_dataset_snapshot_sync(void *arg, dmu_tx_t *tx) { dsl_dataset_snapshot_arg_t *ddsa = arg; dsl_pool_t *dp = dmu_tx_pool(tx); nvpair_t *pair; for (pair = nvlist_next_nvpair(ddsa->ddsa_snaps, NULL); pair != NULL; pair = nvlist_next_nvpair(ddsa->ddsa_snaps, pair)) { dsl_dataset_t *ds; char *name, *atp; char dsname[MAXNAMELEN]; name = nvpair_name(pair); atp = strchr(name, '@'); (void) strlcpy(dsname, name, atp - name + 1); VERIFY0(dsl_dataset_hold(dp, dsname, FTAG, &ds)); dsl_dataset_snapshot_sync_impl(ds, atp + 1, tx); if (ddsa->ddsa_props != NULL) { dsl_props_set_sync_impl(ds->ds_prev, ZPROP_SRC_LOCAL, ddsa->ddsa_props, tx); } dsl_dataset_rele(ds, FTAG); } } /* * The snapshots must all be in the same pool. * All-or-nothing: if there are any failures, nothing will be modified. */ int dsl_dataset_snapshot(nvlist_t *snaps, nvlist_t *props, nvlist_t *errors) { dsl_dataset_snapshot_arg_t ddsa; nvpair_t *pair; boolean_t needsuspend; int error; spa_t *spa; char *firstname; nvlist_t *suspended = NULL; pair = nvlist_next_nvpair(snaps, NULL); if (pair == NULL) return (0); firstname = nvpair_name(pair); error = spa_open(firstname, &spa, FTAG); if (error != 0) return (error); needsuspend = (spa_version(spa) < SPA_VERSION_FAST_SNAP); spa_close(spa, FTAG); if (needsuspend) { suspended = fnvlist_alloc(); for (pair = nvlist_next_nvpair(snaps, NULL); pair != NULL; pair = nvlist_next_nvpair(snaps, pair)) { char fsname[MAXNAMELEN]; char *snapname = nvpair_name(pair); char *atp; void *cookie; atp = strchr(snapname, '@'); if (atp == NULL) { error = SET_ERROR(EINVAL); break; } (void) strlcpy(fsname, snapname, atp - snapname + 1); error = zil_suspend(fsname, &cookie); if (error != 0) break; fnvlist_add_uint64(suspended, fsname, (uintptr_t)cookie); } } ddsa.ddsa_snaps = snaps; ddsa.ddsa_props = props; ddsa.ddsa_errors = errors; ddsa.ddsa_cr = CRED(); if (error == 0) { error = dsl_sync_task(firstname, dsl_dataset_snapshot_check, dsl_dataset_snapshot_sync, &ddsa, fnvlist_num_pairs(snaps) * 3, ZFS_SPACE_CHECK_NORMAL); } if (suspended != NULL) { for (pair = nvlist_next_nvpair(suspended, NULL); pair != NULL; pair = nvlist_next_nvpair(suspended, pair)) { zil_resume((void *)(uintptr_t) fnvpair_value_uint64(pair)); } fnvlist_free(suspended); } #ifdef __FreeBSD__ #ifdef _KERNEL if (error == 0) { for (pair = nvlist_next_nvpair(snaps, NULL); pair != NULL; pair = nvlist_next_nvpair(snaps, pair)) { char *snapname = nvpair_name(pair); zvol_create_minors(snapname); } } #endif #endif return (error); } typedef struct dsl_dataset_snapshot_tmp_arg { const char *ddsta_fsname; const char *ddsta_snapname; minor_t ddsta_cleanup_minor; const char *ddsta_htag; } dsl_dataset_snapshot_tmp_arg_t; static int dsl_dataset_snapshot_tmp_check(void *arg, dmu_tx_t *tx) { dsl_dataset_snapshot_tmp_arg_t *ddsta = arg; dsl_pool_t *dp = dmu_tx_pool(tx); dsl_dataset_t *ds; int error; error = dsl_dataset_hold(dp, ddsta->ddsta_fsname, FTAG, &ds); if (error != 0) return (error); /* NULL cred means no limit check for tmp snapshot */ error = dsl_dataset_snapshot_check_impl(ds, ddsta->ddsta_snapname, tx, B_FALSE, 0, NULL); if (error != 0) { dsl_dataset_rele(ds, FTAG); return (error); } if (spa_version(dp->dp_spa) < SPA_VERSION_USERREFS) { dsl_dataset_rele(ds, FTAG); return (SET_ERROR(ENOTSUP)); } error = dsl_dataset_user_hold_check_one(NULL, ddsta->ddsta_htag, B_TRUE, tx); if (error != 0) { dsl_dataset_rele(ds, FTAG); return (error); } dsl_dataset_rele(ds, FTAG); return (0); } static void dsl_dataset_snapshot_tmp_sync(void *arg, dmu_tx_t *tx) { dsl_dataset_snapshot_tmp_arg_t *ddsta = arg; dsl_pool_t *dp = dmu_tx_pool(tx); dsl_dataset_t *ds; VERIFY0(dsl_dataset_hold(dp, ddsta->ddsta_fsname, FTAG, &ds)); dsl_dataset_snapshot_sync_impl(ds, ddsta->ddsta_snapname, tx); dsl_dataset_user_hold_sync_one(ds->ds_prev, ddsta->ddsta_htag, ddsta->ddsta_cleanup_minor, gethrestime_sec(), tx); dsl_destroy_snapshot_sync_impl(ds->ds_prev, B_TRUE, tx); dsl_dataset_rele(ds, FTAG); } int dsl_dataset_snapshot_tmp(const char *fsname, const char *snapname, minor_t cleanup_minor, const char *htag) { dsl_dataset_snapshot_tmp_arg_t ddsta; int error; spa_t *spa; boolean_t needsuspend; void *cookie; ddsta.ddsta_fsname = fsname; ddsta.ddsta_snapname = snapname; ddsta.ddsta_cleanup_minor = cleanup_minor; ddsta.ddsta_htag = htag; error = spa_open(fsname, &spa, FTAG); if (error != 0) return (error); needsuspend = (spa_version(spa) < SPA_VERSION_FAST_SNAP); spa_close(spa, FTAG); if (needsuspend) { error = zil_suspend(fsname, &cookie); if (error != 0) return (error); } error = dsl_sync_task(fsname, dsl_dataset_snapshot_tmp_check, dsl_dataset_snapshot_tmp_sync, &ddsta, 3, ZFS_SPACE_CHECK_RESERVED); if (needsuspend) zil_resume(cookie); return (error); } void dsl_dataset_sync(dsl_dataset_t *ds, zio_t *zio, dmu_tx_t *tx) { ASSERT(dmu_tx_is_syncing(tx)); ASSERT(ds->ds_objset != NULL); ASSERT(ds->ds_phys->ds_next_snap_obj == 0); /* * in case we had to change ds_fsid_guid when we opened it, * sync it out now. */ dmu_buf_will_dirty(ds->ds_dbuf, tx); ds->ds_phys->ds_fsid_guid = ds->ds_fsid_guid; dmu_objset_sync(ds->ds_objset, zio, tx); if (ds->ds_need_large_blocks && !ds->ds_large_blocks) { dsl_dataset_activate_large_blocks_sync_impl(ds->ds_object, tx); ds->ds_large_blocks = B_TRUE; } } static void get_clones_stat(dsl_dataset_t *ds, nvlist_t *nv) { uint64_t count = 0; objset_t *mos = ds->ds_dir->dd_pool->dp_meta_objset; zap_cursor_t zc; zap_attribute_t za; nvlist_t *propval = fnvlist_alloc(); nvlist_t *val = fnvlist_alloc(); ASSERT(dsl_pool_config_held(ds->ds_dir->dd_pool)); /* * There may be missing entries in ds_next_clones_obj * due to a bug in a previous version of the code. * Only trust it if it has the right number of entries. */ if (ds->ds_phys->ds_next_clones_obj != 0) { VERIFY0(zap_count(mos, ds->ds_phys->ds_next_clones_obj, &count)); } if (count != ds->ds_phys->ds_num_children - 1) goto fail; for (zap_cursor_init(&zc, mos, ds->ds_phys->ds_next_clones_obj); zap_cursor_retrieve(&zc, &za) == 0; zap_cursor_advance(&zc)) { dsl_dataset_t *clone; char buf[ZFS_MAXNAMELEN]; VERIFY0(dsl_dataset_hold_obj(ds->ds_dir->dd_pool, za.za_first_integer, FTAG, &clone)); dsl_dir_name(clone->ds_dir, buf); fnvlist_add_boolean(val, buf); dsl_dataset_rele(clone, FTAG); } zap_cursor_fini(&zc); fnvlist_add_nvlist(propval, ZPROP_VALUE, val); fnvlist_add_nvlist(nv, zfs_prop_to_name(ZFS_PROP_CLONES), propval); fail: nvlist_free(val); nvlist_free(propval); } void dsl_dataset_stats(dsl_dataset_t *ds, nvlist_t *nv) { dsl_pool_t *dp = ds->ds_dir->dd_pool; uint64_t refd, avail, uobjs, aobjs, ratio; ASSERT(dsl_pool_config_held(dp)); ratio = ds->ds_phys->ds_compressed_bytes == 0 ? 100 : (ds->ds_phys->ds_uncompressed_bytes * 100 / ds->ds_phys->ds_compressed_bytes); dsl_prop_nvlist_add_uint64(nv, ZFS_PROP_REFRATIO, ratio); dsl_prop_nvlist_add_uint64(nv, ZFS_PROP_LOGICALREFERENCED, ds->ds_phys->ds_uncompressed_bytes); if (dsl_dataset_is_snapshot(ds)) { dsl_prop_nvlist_add_uint64(nv, ZFS_PROP_COMPRESSRATIO, ratio); dsl_prop_nvlist_add_uint64(nv, ZFS_PROP_USED, ds->ds_phys->ds_unique_bytes); get_clones_stat(ds, nv); } else { if (ds->ds_prev != NULL && ds->ds_prev != dp->dp_origin_snap) { char buf[MAXNAMELEN]; dsl_dataset_name(ds->ds_prev, buf); dsl_prop_nvlist_add_string(nv, ZFS_PROP_PREV_SNAP, buf); } dsl_dir_stats(ds->ds_dir, nv); } dsl_dataset_space(ds, &refd, &avail, &uobjs, &aobjs); dsl_prop_nvlist_add_uint64(nv, ZFS_PROP_AVAILABLE, avail); dsl_prop_nvlist_add_uint64(nv, ZFS_PROP_REFERENCED, refd); dsl_prop_nvlist_add_uint64(nv, ZFS_PROP_CREATION, ds->ds_phys->ds_creation_time); dsl_prop_nvlist_add_uint64(nv, ZFS_PROP_CREATETXG, ds->ds_phys->ds_creation_txg); dsl_prop_nvlist_add_uint64(nv, ZFS_PROP_REFQUOTA, ds->ds_quota); dsl_prop_nvlist_add_uint64(nv, ZFS_PROP_REFRESERVATION, ds->ds_reserved); dsl_prop_nvlist_add_uint64(nv, ZFS_PROP_GUID, ds->ds_phys->ds_guid); dsl_prop_nvlist_add_uint64(nv, ZFS_PROP_UNIQUE, ds->ds_phys->ds_unique_bytes); dsl_prop_nvlist_add_uint64(nv, ZFS_PROP_OBJSETID, ds->ds_object); dsl_prop_nvlist_add_uint64(nv, ZFS_PROP_USERREFS, ds->ds_userrefs); dsl_prop_nvlist_add_uint64(nv, ZFS_PROP_DEFER_DESTROY, DS_IS_DEFER_DESTROY(ds) ? 1 : 0); if (ds->ds_phys->ds_prev_snap_obj != 0) { uint64_t written, comp, uncomp; dsl_pool_t *dp = ds->ds_dir->dd_pool; dsl_dataset_t *prev; int err = dsl_dataset_hold_obj(dp, ds->ds_phys->ds_prev_snap_obj, FTAG, &prev); if (err == 0) { err = dsl_dataset_space_written(prev, ds, &written, &comp, &uncomp); dsl_dataset_rele(prev, FTAG); if (err == 0) { dsl_prop_nvlist_add_uint64(nv, ZFS_PROP_WRITTEN, written); } } } } void dsl_dataset_fast_stat(dsl_dataset_t *ds, dmu_objset_stats_t *stat) { dsl_pool_t *dp = ds->ds_dir->dd_pool; ASSERT(dsl_pool_config_held(dp)); stat->dds_creation_txg = ds->ds_phys->ds_creation_txg; stat->dds_inconsistent = ds->ds_phys->ds_flags & DS_FLAG_INCONSISTENT; stat->dds_guid = ds->ds_phys->ds_guid; stat->dds_origin[0] = '\0'; if (dsl_dataset_is_snapshot(ds)) { stat->dds_is_snapshot = B_TRUE; stat->dds_num_clones = ds->ds_phys->ds_num_children - 1; } else { stat->dds_is_snapshot = B_FALSE; stat->dds_num_clones = 0; if (dsl_dir_is_clone(ds->ds_dir)) { dsl_dataset_t *ods; VERIFY0(dsl_dataset_hold_obj(dp, ds->ds_dir->dd_phys->dd_origin_obj, FTAG, &ods)); dsl_dataset_name(ods, stat->dds_origin); dsl_dataset_rele(ods, FTAG); } } } uint64_t dsl_dataset_fsid_guid(dsl_dataset_t *ds) { return (ds->ds_fsid_guid); } void dsl_dataset_space(dsl_dataset_t *ds, uint64_t *refdbytesp, uint64_t *availbytesp, uint64_t *usedobjsp, uint64_t *availobjsp) { *refdbytesp = ds->ds_phys->ds_referenced_bytes; *availbytesp = dsl_dir_space_available(ds->ds_dir, NULL, 0, TRUE); if (ds->ds_reserved > ds->ds_phys->ds_unique_bytes) *availbytesp += ds->ds_reserved - ds->ds_phys->ds_unique_bytes; if (ds->ds_quota != 0) { /* * Adjust available bytes according to refquota */ if (*refdbytesp < ds->ds_quota) *availbytesp = MIN(*availbytesp, ds->ds_quota - *refdbytesp); else *availbytesp = 0; } *usedobjsp = BP_GET_FILL(&ds->ds_phys->ds_bp); *availobjsp = DN_MAX_OBJECT - *usedobjsp; } boolean_t dsl_dataset_modified_since_snap(dsl_dataset_t *ds, dsl_dataset_t *snap) { dsl_pool_t *dp = ds->ds_dir->dd_pool; ASSERT(dsl_pool_config_held(dp)); if (snap == NULL) return (B_FALSE); if (ds->ds_phys->ds_bp.blk_birth > snap->ds_phys->ds_creation_txg) { objset_t *os, *os_snap; /* * It may be that only the ZIL differs, because it was * reset in the head. Don't count that as being * modified. */ if (dmu_objset_from_ds(ds, &os) != 0) return (B_TRUE); if (dmu_objset_from_ds(snap, &os_snap) != 0) return (B_TRUE); return (bcmp(&os->os_phys->os_meta_dnode, &os_snap->os_phys->os_meta_dnode, sizeof (os->os_phys->os_meta_dnode)) != 0); } return (B_FALSE); } typedef struct dsl_dataset_rename_snapshot_arg { const char *ddrsa_fsname; const char *ddrsa_oldsnapname; const char *ddrsa_newsnapname; boolean_t ddrsa_recursive; dmu_tx_t *ddrsa_tx; } dsl_dataset_rename_snapshot_arg_t; /* ARGSUSED */ static int dsl_dataset_rename_snapshot_check_impl(dsl_pool_t *dp, dsl_dataset_t *hds, void *arg) { dsl_dataset_rename_snapshot_arg_t *ddrsa = arg; int error; uint64_t val; error = dsl_dataset_snap_lookup(hds, ddrsa->ddrsa_oldsnapname, &val); if (error != 0) { /* ignore nonexistent snapshots */ return (error == ENOENT ? 0 : error); } /* new name should not exist */ error = dsl_dataset_snap_lookup(hds, ddrsa->ddrsa_newsnapname, &val); if (error == 0) error = SET_ERROR(EEXIST); else if (error == ENOENT) error = 0; /* dataset name + 1 for the "@" + the new snapshot name must fit */ if (dsl_dir_namelen(hds->ds_dir) + 1 + strlen(ddrsa->ddrsa_newsnapname) >= MAXNAMELEN) error = SET_ERROR(ENAMETOOLONG); return (error); } static int dsl_dataset_rename_snapshot_check(void *arg, dmu_tx_t *tx) { dsl_dataset_rename_snapshot_arg_t *ddrsa = arg; dsl_pool_t *dp = dmu_tx_pool(tx); dsl_dataset_t *hds; int error; error = dsl_dataset_hold(dp, ddrsa->ddrsa_fsname, FTAG, &hds); if (error != 0) return (error); if (ddrsa->ddrsa_recursive) { error = dmu_objset_find_dp(dp, hds->ds_dir->dd_object, dsl_dataset_rename_snapshot_check_impl, ddrsa, DS_FIND_CHILDREN); } else { error = dsl_dataset_rename_snapshot_check_impl(dp, hds, ddrsa); } dsl_dataset_rele(hds, FTAG); return (error); } static int dsl_dataset_rename_snapshot_sync_impl(dsl_pool_t *dp, dsl_dataset_t *hds, void *arg) { #ifdef __FreeBSD__ #ifdef _KERNEL char *oldname, *newname; #endif #endif dsl_dataset_rename_snapshot_arg_t *ddrsa = arg; dsl_dataset_t *ds; uint64_t val; dmu_tx_t *tx = ddrsa->ddrsa_tx; int error; error = dsl_dataset_snap_lookup(hds, ddrsa->ddrsa_oldsnapname, &val); ASSERT(error == 0 || error == ENOENT); if (error == ENOENT) { /* ignore nonexistent snapshots */ return (0); } VERIFY0(dsl_dataset_hold_obj(dp, val, FTAG, &ds)); /* log before we change the name */ spa_history_log_internal_ds(ds, "rename", tx, "-> @%s", ddrsa->ddrsa_newsnapname); VERIFY0(dsl_dataset_snap_remove(hds, ddrsa->ddrsa_oldsnapname, tx, B_FALSE)); mutex_enter(&ds->ds_lock); (void) strcpy(ds->ds_snapname, ddrsa->ddrsa_newsnapname); mutex_exit(&ds->ds_lock); VERIFY0(zap_add(dp->dp_meta_objset, hds->ds_phys->ds_snapnames_zapobj, ds->ds_snapname, 8, 1, &ds->ds_object, tx)); #ifdef __FreeBSD__ #ifdef _KERNEL oldname = kmem_alloc(MAXPATHLEN, KM_SLEEP); newname = kmem_alloc(MAXPATHLEN, KM_SLEEP); snprintf(oldname, MAXPATHLEN, "%s@%s", ddrsa->ddrsa_fsname, ddrsa->ddrsa_oldsnapname); snprintf(newname, MAXPATHLEN, "%s@%s", ddrsa->ddrsa_fsname, ddrsa->ddrsa_newsnapname); zfsvfs_update_fromname(oldname, newname); zvol_rename_minors(oldname, newname); kmem_free(newname, MAXPATHLEN); kmem_free(oldname, MAXPATHLEN); #endif #endif dsl_dataset_rele(ds, FTAG); return (0); } static void dsl_dataset_rename_snapshot_sync(void *arg, dmu_tx_t *tx) { dsl_dataset_rename_snapshot_arg_t *ddrsa = arg; dsl_pool_t *dp = dmu_tx_pool(tx); dsl_dataset_t *hds; VERIFY0(dsl_dataset_hold(dp, ddrsa->ddrsa_fsname, FTAG, &hds)); ddrsa->ddrsa_tx = tx; if (ddrsa->ddrsa_recursive) { VERIFY0(dmu_objset_find_dp(dp, hds->ds_dir->dd_object, dsl_dataset_rename_snapshot_sync_impl, ddrsa, DS_FIND_CHILDREN)); } else { VERIFY0(dsl_dataset_rename_snapshot_sync_impl(dp, hds, ddrsa)); } dsl_dataset_rele(hds, FTAG); } int dsl_dataset_rename_snapshot(const char *fsname, const char *oldsnapname, const char *newsnapname, boolean_t recursive) { dsl_dataset_rename_snapshot_arg_t ddrsa; ddrsa.ddrsa_fsname = fsname; ddrsa.ddrsa_oldsnapname = oldsnapname; ddrsa.ddrsa_newsnapname = newsnapname; ddrsa.ddrsa_recursive = recursive; return (dsl_sync_task(fsname, dsl_dataset_rename_snapshot_check, dsl_dataset_rename_snapshot_sync, &ddrsa, 1, ZFS_SPACE_CHECK_RESERVED)); } /* * If we're doing an ownership handoff, we need to make sure that there is * only one long hold on the dataset. We're not allowed to change anything here * so we don't permanently release the long hold or regular hold here. We want * to do this only when syncing to avoid the dataset unexpectedly going away * when we release the long hold. */ static int dsl_dataset_handoff_check(dsl_dataset_t *ds, void *owner, dmu_tx_t *tx) { boolean_t held; if (!dmu_tx_is_syncing(tx)) return (0); if (owner != NULL) { VERIFY3P(ds->ds_owner, ==, owner); dsl_dataset_long_rele(ds, owner); } held = dsl_dataset_long_held(ds); if (owner != NULL) dsl_dataset_long_hold(ds, owner); if (held) return (SET_ERROR(EBUSY)); return (0); } typedef struct dsl_dataset_rollback_arg { const char *ddra_fsname; void *ddra_owner; nvlist_t *ddra_result; } dsl_dataset_rollback_arg_t; static int dsl_dataset_rollback_check(void *arg, dmu_tx_t *tx) { dsl_dataset_rollback_arg_t *ddra = arg; dsl_pool_t *dp = dmu_tx_pool(tx); dsl_dataset_t *ds; int64_t unused_refres_delta; int error; error = dsl_dataset_hold(dp, ddra->ddra_fsname, FTAG, &ds); if (error != 0) return (error); /* must not be a snapshot */ if (dsl_dataset_is_snapshot(ds)) { dsl_dataset_rele(ds, FTAG); return (SET_ERROR(EINVAL)); } /* must have a most recent snapshot */ if (ds->ds_phys->ds_prev_snap_txg < TXG_INITIAL) { dsl_dataset_rele(ds, FTAG); return (SET_ERROR(EINVAL)); } /* must not have any bookmarks after the most recent snapshot */ nvlist_t *proprequest = fnvlist_alloc(); fnvlist_add_boolean(proprequest, zfs_prop_to_name(ZFS_PROP_CREATETXG)); nvlist_t *bookmarks = fnvlist_alloc(); error = dsl_get_bookmarks_impl(ds, proprequest, bookmarks); fnvlist_free(proprequest); if (error != 0) return (error); for (nvpair_t *pair = nvlist_next_nvpair(bookmarks, NULL); pair != NULL; pair = nvlist_next_nvpair(bookmarks, pair)) { nvlist_t *valuenv = fnvlist_lookup_nvlist(fnvpair_value_nvlist(pair), zfs_prop_to_name(ZFS_PROP_CREATETXG)); uint64_t createtxg = fnvlist_lookup_uint64(valuenv, "value"); if (createtxg > ds->ds_phys->ds_prev_snap_txg) { fnvlist_free(bookmarks); dsl_dataset_rele(ds, FTAG); return (SET_ERROR(EEXIST)); } } fnvlist_free(bookmarks); error = dsl_dataset_handoff_check(ds, ddra->ddra_owner, tx); if (error != 0) { dsl_dataset_rele(ds, FTAG); return (error); } /* * Check if the snap we are rolling back to uses more than * the refquota. */ if (ds->ds_quota != 0 && ds->ds_prev->ds_phys->ds_referenced_bytes > ds->ds_quota) { dsl_dataset_rele(ds, FTAG); return (SET_ERROR(EDQUOT)); } /* * When we do the clone swap, we will temporarily use more space * due to the refreservation (the head will no longer have any * unique space, so the entire amount of the refreservation will need * to be free). We will immediately destroy the clone, freeing * this space, but the freeing happens over many txg's. */ unused_refres_delta = (int64_t)MIN(ds->ds_reserved, ds->ds_phys->ds_unique_bytes); if (unused_refres_delta > 0 && unused_refres_delta > dsl_dir_space_available(ds->ds_dir, NULL, 0, TRUE)) { dsl_dataset_rele(ds, FTAG); return (SET_ERROR(ENOSPC)); } dsl_dataset_rele(ds, FTAG); return (0); } static void dsl_dataset_rollback_sync(void *arg, dmu_tx_t *tx) { dsl_dataset_rollback_arg_t *ddra = arg; dsl_pool_t *dp = dmu_tx_pool(tx); dsl_dataset_t *ds, *clone; uint64_t cloneobj; char namebuf[ZFS_MAXNAMELEN]; VERIFY0(dsl_dataset_hold(dp, ddra->ddra_fsname, FTAG, &ds)); dsl_dataset_name(ds->ds_prev, namebuf); fnvlist_add_string(ddra->ddra_result, "target", namebuf); cloneobj = dsl_dataset_create_sync(ds->ds_dir, "%rollback", ds->ds_prev, DS_CREATE_FLAG_NODIRTY, kcred, tx); VERIFY0(dsl_dataset_hold_obj(dp, cloneobj, FTAG, &clone)); dsl_dataset_clone_swap_sync_impl(clone, ds, tx); dsl_dataset_zero_zil(ds, tx); dsl_destroy_head_sync_impl(clone, tx); dsl_dataset_rele(clone, FTAG); dsl_dataset_rele(ds, FTAG); } /* * Rolls back the given filesystem or volume to the most recent snapshot. * The name of the most recent snapshot will be returned under key "target" * in the result nvlist. * * If owner != NULL: * - The existing dataset MUST be owned by the specified owner at entry * - Upon return, dataset will still be held by the same owner, whether we * succeed or not. * * This mode is required any time the existing filesystem is mounted. See * notes above zfs_suspend_fs() for further details. */ int dsl_dataset_rollback(const char *fsname, void *owner, nvlist_t *result) { dsl_dataset_rollback_arg_t ddra; ddra.ddra_fsname = fsname; ddra.ddra_owner = owner; ddra.ddra_result = result; return (dsl_sync_task(fsname, dsl_dataset_rollback_check, dsl_dataset_rollback_sync, &ddra, 1, ZFS_SPACE_CHECK_RESERVED)); } struct promotenode { list_node_t link; dsl_dataset_t *ds; }; typedef struct dsl_dataset_promote_arg { const char *ddpa_clonename; dsl_dataset_t *ddpa_clone; list_t shared_snaps, origin_snaps, clone_snaps; dsl_dataset_t *origin_origin; /* origin of the origin */ uint64_t used, comp, uncomp, unique, cloneusedsnap, originusedsnap; char *err_ds; cred_t *cr; } dsl_dataset_promote_arg_t; static int snaplist_space(list_t *l, uint64_t mintxg, uint64_t *spacep); static int promote_hold(dsl_dataset_promote_arg_t *ddpa, dsl_pool_t *dp, void *tag); static void promote_rele(dsl_dataset_promote_arg_t *ddpa, void *tag); static int dsl_dataset_promote_check(void *arg, dmu_tx_t *tx) { dsl_dataset_promote_arg_t *ddpa = arg; dsl_pool_t *dp = dmu_tx_pool(tx); dsl_dataset_t *hds; struct promotenode *snap; dsl_dataset_t *origin_ds; int err; uint64_t unused; uint64_t ss_mv_cnt; err = promote_hold(ddpa, dp, FTAG); if (err != 0) return (err); hds = ddpa->ddpa_clone; if (hds->ds_phys->ds_flags & DS_FLAG_NOPROMOTE) { promote_rele(ddpa, FTAG); return (SET_ERROR(EXDEV)); } /* * Compute and check the amount of space to transfer. Since this is * so expensive, don't do the preliminary check. */ if (!dmu_tx_is_syncing(tx)) { promote_rele(ddpa, FTAG); return (0); } snap = list_head(&ddpa->shared_snaps); origin_ds = snap->ds; /* compute origin's new unique space */ snap = list_tail(&ddpa->clone_snaps); ASSERT3U(snap->ds->ds_phys->ds_prev_snap_obj, ==, origin_ds->ds_object); dsl_deadlist_space_range(&snap->ds->ds_deadlist, origin_ds->ds_phys->ds_prev_snap_txg, UINT64_MAX, &ddpa->unique, &unused, &unused); /* * Walk the snapshots that we are moving * * Compute space to transfer. Consider the incremental changes * to used by each snapshot: * (my used) = (prev's used) + (blocks born) - (blocks killed) * So each snapshot gave birth to: * (blocks born) = (my used) - (prev's used) + (blocks killed) * So a sequence would look like: * (uN - u(N-1) + kN) + ... + (u1 - u0 + k1) + (u0 - 0 + k0) * Which simplifies to: * uN + kN + kN-1 + ... + k1 + k0 * Note however, if we stop before we reach the ORIGIN we get: * uN + kN + kN-1 + ... + kM - uM-1 */ ss_mv_cnt = 0; ddpa->used = origin_ds->ds_phys->ds_referenced_bytes; ddpa->comp = origin_ds->ds_phys->ds_compressed_bytes; ddpa->uncomp = origin_ds->ds_phys->ds_uncompressed_bytes; for (snap = list_head(&ddpa->shared_snaps); snap; snap = list_next(&ddpa->shared_snaps, snap)) { uint64_t val, dlused, dlcomp, dluncomp; dsl_dataset_t *ds = snap->ds; ss_mv_cnt++; /* * If there are long holds, we won't be able to evict * the objset. */ if (dsl_dataset_long_held(ds)) { err = SET_ERROR(EBUSY); goto out; } /* Check that the snapshot name does not conflict */ VERIFY0(dsl_dataset_get_snapname(ds)); err = dsl_dataset_snap_lookup(hds, ds->ds_snapname, &val); if (err == 0) { (void) strcpy(ddpa->err_ds, snap->ds->ds_snapname); err = SET_ERROR(EEXIST); goto out; } if (err != ENOENT) goto out; /* The very first snapshot does not have a deadlist */ if (ds->ds_phys->ds_prev_snap_obj == 0) continue; dsl_deadlist_space(&ds->ds_deadlist, &dlused, &dlcomp, &dluncomp); ddpa->used += dlused; ddpa->comp += dlcomp; ddpa->uncomp += dluncomp; } /* * If we are a clone of a clone then we never reached ORIGIN, * so we need to subtract out the clone origin's used space. */ if (ddpa->origin_origin) { ddpa->used -= ddpa->origin_origin->ds_phys->ds_referenced_bytes; ddpa->comp -= ddpa->origin_origin->ds_phys->ds_compressed_bytes; ddpa->uncomp -= ddpa->origin_origin->ds_phys->ds_uncompressed_bytes; } /* Check that there is enough space and limit headroom here */ err = dsl_dir_transfer_possible(origin_ds->ds_dir, hds->ds_dir, 0, ss_mv_cnt, ddpa->used, ddpa->cr); if (err != 0) goto out; /* * Compute the amounts of space that will be used by snapshots * after the promotion (for both origin and clone). For each, * it is the amount of space that will be on all of their * deadlists (that was not born before their new origin). */ if (hds->ds_dir->dd_phys->dd_flags & DD_FLAG_USED_BREAKDOWN) { uint64_t space; /* * Note, typically this will not be a clone of a clone, * so dd_origin_txg will be < TXG_INITIAL, so * these snaplist_space() -> dsl_deadlist_space_range() * calls will be fast because they do not have to * iterate over all bps. */ snap = list_head(&ddpa->origin_snaps); err = snaplist_space(&ddpa->shared_snaps, snap->ds->ds_dir->dd_origin_txg, &ddpa->cloneusedsnap); if (err != 0) goto out; err = snaplist_space(&ddpa->clone_snaps, snap->ds->ds_dir->dd_origin_txg, &space); if (err != 0) goto out; ddpa->cloneusedsnap += space; } if (origin_ds->ds_dir->dd_phys->dd_flags & DD_FLAG_USED_BREAKDOWN) { err = snaplist_space(&ddpa->origin_snaps, origin_ds->ds_phys->ds_creation_txg, &ddpa->originusedsnap); if (err != 0) goto out; } out: promote_rele(ddpa, FTAG); return (err); } static void dsl_dataset_promote_sync(void *arg, dmu_tx_t *tx) { dsl_dataset_promote_arg_t *ddpa = arg; dsl_pool_t *dp = dmu_tx_pool(tx); dsl_dataset_t *hds; struct promotenode *snap; dsl_dataset_t *origin_ds; dsl_dataset_t *origin_head; dsl_dir_t *dd; dsl_dir_t *odd = NULL; uint64_t oldnext_obj; int64_t delta; #if defined(__FreeBSD__) && defined(_KERNEL) char *oldname, *newname; #endif VERIFY0(promote_hold(ddpa, dp, FTAG)); hds = ddpa->ddpa_clone; ASSERT0(hds->ds_phys->ds_flags & DS_FLAG_NOPROMOTE); snap = list_head(&ddpa->shared_snaps); origin_ds = snap->ds; dd = hds->ds_dir; snap = list_head(&ddpa->origin_snaps); origin_head = snap->ds; /* * We need to explicitly open odd, since origin_ds's dd will be * changing. */ VERIFY0(dsl_dir_hold_obj(dp, origin_ds->ds_dir->dd_object, NULL, FTAG, &odd)); /* change origin's next snap */ dmu_buf_will_dirty(origin_ds->ds_dbuf, tx); oldnext_obj = origin_ds->ds_phys->ds_next_snap_obj; snap = list_tail(&ddpa->clone_snaps); ASSERT3U(snap->ds->ds_phys->ds_prev_snap_obj, ==, origin_ds->ds_object); origin_ds->ds_phys->ds_next_snap_obj = snap->ds->ds_object; /* change the origin's next clone */ if (origin_ds->ds_phys->ds_next_clones_obj) { dsl_dataset_remove_from_next_clones(origin_ds, snap->ds->ds_object, tx); VERIFY0(zap_add_int(dp->dp_meta_objset, origin_ds->ds_phys->ds_next_clones_obj, oldnext_obj, tx)); } /* change origin */ dmu_buf_will_dirty(dd->dd_dbuf, tx); ASSERT3U(dd->dd_phys->dd_origin_obj, ==, origin_ds->ds_object); dd->dd_phys->dd_origin_obj = odd->dd_phys->dd_origin_obj; dd->dd_origin_txg = origin_head->ds_dir->dd_origin_txg; dmu_buf_will_dirty(odd->dd_dbuf, tx); odd->dd_phys->dd_origin_obj = origin_ds->ds_object; origin_head->ds_dir->dd_origin_txg = origin_ds->ds_phys->ds_creation_txg; /* change dd_clone entries */ if (spa_version(dp->dp_spa) >= SPA_VERSION_DIR_CLONES) { VERIFY0(zap_remove_int(dp->dp_meta_objset, odd->dd_phys->dd_clones, hds->ds_object, tx)); VERIFY0(zap_add_int(dp->dp_meta_objset, ddpa->origin_origin->ds_dir->dd_phys->dd_clones, hds->ds_object, tx)); VERIFY0(zap_remove_int(dp->dp_meta_objset, ddpa->origin_origin->ds_dir->dd_phys->dd_clones, origin_head->ds_object, tx)); if (dd->dd_phys->dd_clones == 0) { dd->dd_phys->dd_clones = zap_create(dp->dp_meta_objset, DMU_OT_DSL_CLONES, DMU_OT_NONE, 0, tx); } VERIFY0(zap_add_int(dp->dp_meta_objset, dd->dd_phys->dd_clones, origin_head->ds_object, tx)); } #if defined(__FreeBSD__) && defined(_KERNEL) /* Take the spa_namespace_lock early so zvol renames don't deadlock. */ mutex_enter(&spa_namespace_lock); oldname = kmem_alloc(MAXPATHLEN, KM_SLEEP); newname = kmem_alloc(MAXPATHLEN, KM_SLEEP); #endif /* move snapshots to this dir */ for (snap = list_head(&ddpa->shared_snaps); snap; snap = list_next(&ddpa->shared_snaps, snap)) { dsl_dataset_t *ds = snap->ds; /* * Property callbacks are registered to a particular * dsl_dir. Since ours is changing, evict the objset * so that they will be unregistered from the old dsl_dir. */ if (ds->ds_objset) { dmu_objset_evict(ds->ds_objset); ds->ds_objset = NULL; } /* move snap name entry */ VERIFY0(dsl_dataset_get_snapname(ds)); VERIFY0(dsl_dataset_snap_remove(origin_head, ds->ds_snapname, tx, B_TRUE)); VERIFY0(zap_add(dp->dp_meta_objset, hds->ds_phys->ds_snapnames_zapobj, ds->ds_snapname, 8, 1, &ds->ds_object, tx)); dsl_fs_ss_count_adjust(hds->ds_dir, 1, DD_FIELD_SNAPSHOT_COUNT, tx); /* change containing dsl_dir */ dmu_buf_will_dirty(ds->ds_dbuf, tx); ASSERT3U(ds->ds_phys->ds_dir_obj, ==, odd->dd_object); ds->ds_phys->ds_dir_obj = dd->dd_object; ASSERT3P(ds->ds_dir, ==, odd); dsl_dir_rele(ds->ds_dir, ds); VERIFY0(dsl_dir_hold_obj(dp, dd->dd_object, NULL, ds, &ds->ds_dir)); #if defined(__FreeBSD__) && defined(_KERNEL) dsl_dataset_name(ds, newname); zfsvfs_update_fromname(oldname, newname); zvol_rename_minors(oldname, newname); #endif /* move any clone references */ if (ds->ds_phys->ds_next_clones_obj && spa_version(dp->dp_spa) >= SPA_VERSION_DIR_CLONES) { zap_cursor_t zc; zap_attribute_t za; for (zap_cursor_init(&zc, dp->dp_meta_objset, ds->ds_phys->ds_next_clones_obj); zap_cursor_retrieve(&zc, &za) == 0; zap_cursor_advance(&zc)) { dsl_dataset_t *cnds; uint64_t o; if (za.za_first_integer == oldnext_obj) { /* * We've already moved the * origin's reference. */ continue; } VERIFY0(dsl_dataset_hold_obj(dp, za.za_first_integer, FTAG, &cnds)); o = cnds->ds_dir->dd_phys->dd_head_dataset_obj; VERIFY0(zap_remove_int(dp->dp_meta_objset, odd->dd_phys->dd_clones, o, tx)); VERIFY0(zap_add_int(dp->dp_meta_objset, dd->dd_phys->dd_clones, o, tx)); dsl_dataset_rele(cnds, FTAG); } zap_cursor_fini(&zc); } ASSERT(!dsl_prop_hascb(ds)); } #if defined(__FreeBSD__) && defined(_KERNEL) mutex_exit(&spa_namespace_lock); kmem_free(newname, MAXPATHLEN); kmem_free(oldname, MAXPATHLEN); #endif /* * Change space accounting. * Note, pa->*usedsnap and dd_used_breakdown[SNAP] will either * both be valid, or both be 0 (resulting in delta == 0). This * is true for each of {clone,origin} independently. */ delta = ddpa->cloneusedsnap - dd->dd_phys->dd_used_breakdown[DD_USED_SNAP]; ASSERT3S(delta, >=, 0); ASSERT3U(ddpa->used, >=, delta); dsl_dir_diduse_space(dd, DD_USED_SNAP, delta, 0, 0, tx); dsl_dir_diduse_space(dd, DD_USED_HEAD, ddpa->used - delta, ddpa->comp, ddpa->uncomp, tx); delta = ddpa->originusedsnap - odd->dd_phys->dd_used_breakdown[DD_USED_SNAP]; ASSERT3S(delta, <=, 0); ASSERT3U(ddpa->used, >=, -delta); dsl_dir_diduse_space(odd, DD_USED_SNAP, delta, 0, 0, tx); dsl_dir_diduse_space(odd, DD_USED_HEAD, -ddpa->used - delta, -ddpa->comp, -ddpa->uncomp, tx); origin_ds->ds_phys->ds_unique_bytes = ddpa->unique; /* log history record */ spa_history_log_internal_ds(hds, "promote", tx, ""); dsl_dir_rele(odd, FTAG); promote_rele(ddpa, FTAG); } /* * Make a list of dsl_dataset_t's for the snapshots between first_obj * (exclusive) and last_obj (inclusive). The list will be in reverse * order (last_obj will be the list_head()). If first_obj == 0, do all * snapshots back to this dataset's origin. */ static int snaplist_make(dsl_pool_t *dp, uint64_t first_obj, uint64_t last_obj, list_t *l, void *tag) { uint64_t obj = last_obj; list_create(l, sizeof (struct promotenode), offsetof(struct promotenode, link)); while (obj != first_obj) { dsl_dataset_t *ds; struct promotenode *snap; int err; err = dsl_dataset_hold_obj(dp, obj, tag, &ds); ASSERT(err != ENOENT); if (err != 0) return (err); if (first_obj == 0) first_obj = ds->ds_dir->dd_phys->dd_origin_obj; snap = kmem_alloc(sizeof (*snap), KM_SLEEP); snap->ds = ds; list_insert_tail(l, snap); obj = ds->ds_phys->ds_prev_snap_obj; } return (0); } static int snaplist_space(list_t *l, uint64_t mintxg, uint64_t *spacep) { struct promotenode *snap; *spacep = 0; for (snap = list_head(l); snap; snap = list_next(l, snap)) { uint64_t used, comp, uncomp; dsl_deadlist_space_range(&snap->ds->ds_deadlist, mintxg, UINT64_MAX, &used, &comp, &uncomp); *spacep += used; } return (0); } static void snaplist_destroy(list_t *l, void *tag) { struct promotenode *snap; if (l == NULL || !list_link_active(&l->list_head)) return; while ((snap = list_tail(l)) != NULL) { list_remove(l, snap); dsl_dataset_rele(snap->ds, tag); kmem_free(snap, sizeof (*snap)); } list_destroy(l); } static int promote_hold(dsl_dataset_promote_arg_t *ddpa, dsl_pool_t *dp, void *tag) { int error; dsl_dir_t *dd; struct promotenode *snap; error = dsl_dataset_hold(dp, ddpa->ddpa_clonename, tag, &ddpa->ddpa_clone); if (error != 0) return (error); dd = ddpa->ddpa_clone->ds_dir; if (dsl_dataset_is_snapshot(ddpa->ddpa_clone) || !dsl_dir_is_clone(dd)) { dsl_dataset_rele(ddpa->ddpa_clone, tag); return (SET_ERROR(EINVAL)); } error = snaplist_make(dp, 0, dd->dd_phys->dd_origin_obj, &ddpa->shared_snaps, tag); if (error != 0) goto out; error = snaplist_make(dp, 0, ddpa->ddpa_clone->ds_object, &ddpa->clone_snaps, tag); if (error != 0) goto out; snap = list_head(&ddpa->shared_snaps); ASSERT3U(snap->ds->ds_object, ==, dd->dd_phys->dd_origin_obj); error = snaplist_make(dp, dd->dd_phys->dd_origin_obj, snap->ds->ds_dir->dd_phys->dd_head_dataset_obj, &ddpa->origin_snaps, tag); if (error != 0) goto out; if (snap->ds->ds_dir->dd_phys->dd_origin_obj != 0) { error = dsl_dataset_hold_obj(dp, snap->ds->ds_dir->dd_phys->dd_origin_obj, tag, &ddpa->origin_origin); if (error != 0) goto out; } out: if (error != 0) promote_rele(ddpa, tag); return (error); } static void promote_rele(dsl_dataset_promote_arg_t *ddpa, void *tag) { snaplist_destroy(&ddpa->shared_snaps, tag); snaplist_destroy(&ddpa->clone_snaps, tag); snaplist_destroy(&ddpa->origin_snaps, tag); if (ddpa->origin_origin != NULL) dsl_dataset_rele(ddpa->origin_origin, tag); dsl_dataset_rele(ddpa->ddpa_clone, tag); } /* * Promote a clone. * * If it fails due to a conflicting snapshot name, "conflsnap" will be filled * in with the name. (It must be at least MAXNAMELEN bytes long.) */ int dsl_dataset_promote(const char *name, char *conflsnap) { dsl_dataset_promote_arg_t ddpa = { 0 }; uint64_t numsnaps; int error; objset_t *os; /* * We will modify space proportional to the number of * snapshots. Compute numsnaps. */ error = dmu_objset_hold(name, FTAG, &os); if (error != 0) return (error); error = zap_count(dmu_objset_pool(os)->dp_meta_objset, dmu_objset_ds(os)->ds_phys->ds_snapnames_zapobj, &numsnaps); dmu_objset_rele(os, FTAG); if (error != 0) return (error); ddpa.ddpa_clonename = name; ddpa.err_ds = conflsnap; ddpa.cr = CRED(); return (dsl_sync_task(name, dsl_dataset_promote_check, dsl_dataset_promote_sync, &ddpa, 2 + numsnaps, ZFS_SPACE_CHECK_RESERVED)); } int dsl_dataset_clone_swap_check_impl(dsl_dataset_t *clone, dsl_dataset_t *origin_head, boolean_t force, void *owner, dmu_tx_t *tx) { int64_t unused_refres_delta; /* they should both be heads */ if (dsl_dataset_is_snapshot(clone) || dsl_dataset_is_snapshot(origin_head)) return (SET_ERROR(EINVAL)); /* if we are not forcing, the branch point should be just before them */ if (!force && clone->ds_prev != origin_head->ds_prev) return (SET_ERROR(EINVAL)); /* clone should be the clone (unless they are unrelated) */ if (clone->ds_prev != NULL && clone->ds_prev != clone->ds_dir->dd_pool->dp_origin_snap && origin_head->ds_dir != clone->ds_prev->ds_dir) return (SET_ERROR(EINVAL)); /* the clone should be a child of the origin */ if (clone->ds_dir->dd_parent != origin_head->ds_dir) return (SET_ERROR(EINVAL)); /* origin_head shouldn't be modified unless 'force' */ if (!force && dsl_dataset_modified_since_snap(origin_head, origin_head->ds_prev)) return (SET_ERROR(ETXTBSY)); /* origin_head should have no long holds (e.g. is not mounted) */ if (dsl_dataset_handoff_check(origin_head, owner, tx)) return (SET_ERROR(EBUSY)); /* check amount of any unconsumed refreservation */ unused_refres_delta = (int64_t)MIN(origin_head->ds_reserved, origin_head->ds_phys->ds_unique_bytes) - (int64_t)MIN(origin_head->ds_reserved, clone->ds_phys->ds_unique_bytes); if (unused_refres_delta > 0 && unused_refres_delta > dsl_dir_space_available(origin_head->ds_dir, NULL, 0, TRUE)) return (SET_ERROR(ENOSPC)); /* clone can't be over the head's refquota */ if (origin_head->ds_quota != 0 && clone->ds_phys->ds_referenced_bytes > origin_head->ds_quota) return (SET_ERROR(EDQUOT)); return (0); } void dsl_dataset_clone_swap_sync_impl(dsl_dataset_t *clone, dsl_dataset_t *origin_head, dmu_tx_t *tx) { dsl_pool_t *dp = dmu_tx_pool(tx); int64_t unused_refres_delta; ASSERT(clone->ds_reserved == 0); ASSERT(origin_head->ds_quota == 0 || clone->ds_phys->ds_unique_bytes <= origin_head->ds_quota); ASSERT3P(clone->ds_prev, ==, origin_head->ds_prev); dmu_buf_will_dirty(clone->ds_dbuf, tx); dmu_buf_will_dirty(origin_head->ds_dbuf, tx); if (clone->ds_objset != NULL) { dmu_objset_evict(clone->ds_objset); clone->ds_objset = NULL; } if (origin_head->ds_objset != NULL) { dmu_objset_evict(origin_head->ds_objset); origin_head->ds_objset = NULL; } unused_refres_delta = (int64_t)MIN(origin_head->ds_reserved, origin_head->ds_phys->ds_unique_bytes) - (int64_t)MIN(origin_head->ds_reserved, clone->ds_phys->ds_unique_bytes); /* * Reset origin's unique bytes, if it exists. */ if (clone->ds_prev) { dsl_dataset_t *origin = clone->ds_prev; uint64_t comp, uncomp; dmu_buf_will_dirty(origin->ds_dbuf, tx); dsl_deadlist_space_range(&clone->ds_deadlist, origin->ds_phys->ds_prev_snap_txg, UINT64_MAX, &origin->ds_phys->ds_unique_bytes, &comp, &uncomp); } /* swap blkptrs */ { blkptr_t tmp; tmp = origin_head->ds_phys->ds_bp; origin_head->ds_phys->ds_bp = clone->ds_phys->ds_bp; clone->ds_phys->ds_bp = tmp; } /* set dd_*_bytes */ { int64_t dused, dcomp, duncomp; uint64_t cdl_used, cdl_comp, cdl_uncomp; uint64_t odl_used, odl_comp, odl_uncomp; ASSERT3U(clone->ds_dir->dd_phys-> dd_used_breakdown[DD_USED_SNAP], ==, 0); dsl_deadlist_space(&clone->ds_deadlist, &cdl_used, &cdl_comp, &cdl_uncomp); dsl_deadlist_space(&origin_head->ds_deadlist, &odl_used, &odl_comp, &odl_uncomp); dused = clone->ds_phys->ds_referenced_bytes + cdl_used - (origin_head->ds_phys->ds_referenced_bytes + odl_used); dcomp = clone->ds_phys->ds_compressed_bytes + cdl_comp - (origin_head->ds_phys->ds_compressed_bytes + odl_comp); duncomp = clone->ds_phys->ds_uncompressed_bytes + cdl_uncomp - (origin_head->ds_phys->ds_uncompressed_bytes + odl_uncomp); dsl_dir_diduse_space(origin_head->ds_dir, DD_USED_HEAD, dused, dcomp, duncomp, tx); dsl_dir_diduse_space(clone->ds_dir, DD_USED_HEAD, -dused, -dcomp, -duncomp, tx); /* * The difference in the space used by snapshots is the * difference in snapshot space due to the head's * deadlist (since that's the only thing that's * changing that affects the snapused). */ dsl_deadlist_space_range(&clone->ds_deadlist, origin_head->ds_dir->dd_origin_txg, UINT64_MAX, &cdl_used, &cdl_comp, &cdl_uncomp); dsl_deadlist_space_range(&origin_head->ds_deadlist, origin_head->ds_dir->dd_origin_txg, UINT64_MAX, &odl_used, &odl_comp, &odl_uncomp); dsl_dir_transfer_space(origin_head->ds_dir, cdl_used - odl_used, DD_USED_HEAD, DD_USED_SNAP, tx); } /* swap ds_*_bytes */ SWITCH64(origin_head->ds_phys->ds_referenced_bytes, clone->ds_phys->ds_referenced_bytes); SWITCH64(origin_head->ds_phys->ds_compressed_bytes, clone->ds_phys->ds_compressed_bytes); SWITCH64(origin_head->ds_phys->ds_uncompressed_bytes, clone->ds_phys->ds_uncompressed_bytes); SWITCH64(origin_head->ds_phys->ds_unique_bytes, clone->ds_phys->ds_unique_bytes); /* apply any parent delta for change in unconsumed refreservation */ dsl_dir_diduse_space(origin_head->ds_dir, DD_USED_REFRSRV, unused_refres_delta, 0, 0, tx); /* * Swap deadlists. */ dsl_deadlist_close(&clone->ds_deadlist); dsl_deadlist_close(&origin_head->ds_deadlist); SWITCH64(origin_head->ds_phys->ds_deadlist_obj, clone->ds_phys->ds_deadlist_obj); dsl_deadlist_open(&clone->ds_deadlist, dp->dp_meta_objset, clone->ds_phys->ds_deadlist_obj); dsl_deadlist_open(&origin_head->ds_deadlist, dp->dp_meta_objset, origin_head->ds_phys->ds_deadlist_obj); dsl_scan_ds_clone_swapped(origin_head, clone, tx); spa_history_log_internal_ds(clone, "clone swap", tx, "parent=%s", origin_head->ds_dir->dd_myname); } /* * Given a pool name and a dataset object number in that pool, * return the name of that dataset. */ int dsl_dsobj_to_dsname(char *pname, uint64_t obj, char *buf) { dsl_pool_t *dp; dsl_dataset_t *ds; int error; error = dsl_pool_hold(pname, FTAG, &dp); if (error != 0) return (error); error = dsl_dataset_hold_obj(dp, obj, FTAG, &ds); if (error == 0) { dsl_dataset_name(ds, buf); dsl_dataset_rele(ds, FTAG); } dsl_pool_rele(dp, FTAG); return (error); } int dsl_dataset_check_quota(dsl_dataset_t *ds, boolean_t check_quota, uint64_t asize, uint64_t inflight, uint64_t *used, uint64_t *ref_rsrv) { int error = 0; ASSERT3S(asize, >, 0); /* * *ref_rsrv is the portion of asize that will come from any * unconsumed refreservation space. */ *ref_rsrv = 0; mutex_enter(&ds->ds_lock); /* * Make a space adjustment for reserved bytes. */ if (ds->ds_reserved > ds->ds_phys->ds_unique_bytes) { ASSERT3U(*used, >=, ds->ds_reserved - ds->ds_phys->ds_unique_bytes); *used -= (ds->ds_reserved - ds->ds_phys->ds_unique_bytes); *ref_rsrv = asize - MIN(asize, parent_delta(ds, asize + inflight)); } if (!check_quota || ds->ds_quota == 0) { mutex_exit(&ds->ds_lock); return (0); } /* * If they are requesting more space, and our current estimate * is over quota, they get to try again unless the actual * on-disk is over quota and there are no pending changes (which * may free up space for us). */ if (ds->ds_phys->ds_referenced_bytes + inflight >= ds->ds_quota) { if (inflight > 0 || ds->ds_phys->ds_referenced_bytes < ds->ds_quota) error = SET_ERROR(ERESTART); else error = SET_ERROR(EDQUOT); } mutex_exit(&ds->ds_lock); return (error); } typedef struct dsl_dataset_set_qr_arg { const char *ddsqra_name; zprop_source_t ddsqra_source; uint64_t ddsqra_value; } dsl_dataset_set_qr_arg_t; /* ARGSUSED */ static int dsl_dataset_set_refquota_check(void *arg, dmu_tx_t *tx) { dsl_dataset_set_qr_arg_t *ddsqra = arg; dsl_pool_t *dp = dmu_tx_pool(tx); dsl_dataset_t *ds; int error; uint64_t newval; if (spa_version(dp->dp_spa) < SPA_VERSION_REFQUOTA) return (SET_ERROR(ENOTSUP)); error = dsl_dataset_hold(dp, ddsqra->ddsqra_name, FTAG, &ds); if (error != 0) return (error); if (dsl_dataset_is_snapshot(ds)) { dsl_dataset_rele(ds, FTAG); return (SET_ERROR(EINVAL)); } error = dsl_prop_predict(ds->ds_dir, zfs_prop_to_name(ZFS_PROP_REFQUOTA), ddsqra->ddsqra_source, ddsqra->ddsqra_value, &newval); if (error != 0) { dsl_dataset_rele(ds, FTAG); return (error); } if (newval == 0) { dsl_dataset_rele(ds, FTAG); return (0); } if (newval < ds->ds_phys->ds_referenced_bytes || newval < ds->ds_reserved) { dsl_dataset_rele(ds, FTAG); return (SET_ERROR(ENOSPC)); } dsl_dataset_rele(ds, FTAG); return (0); } static void dsl_dataset_set_refquota_sync(void *arg, dmu_tx_t *tx) { dsl_dataset_set_qr_arg_t *ddsqra = arg; dsl_pool_t *dp = dmu_tx_pool(tx); dsl_dataset_t *ds; uint64_t newval; VERIFY0(dsl_dataset_hold(dp, ddsqra->ddsqra_name, FTAG, &ds)); dsl_prop_set_sync_impl(ds, zfs_prop_to_name(ZFS_PROP_REFQUOTA), ddsqra->ddsqra_source, sizeof (ddsqra->ddsqra_value), 1, &ddsqra->ddsqra_value, tx); VERIFY0(dsl_prop_get_int_ds(ds, zfs_prop_to_name(ZFS_PROP_REFQUOTA), &newval)); if (ds->ds_quota != newval) { dmu_buf_will_dirty(ds->ds_dbuf, tx); ds->ds_quota = newval; } dsl_dataset_rele(ds, FTAG); } int dsl_dataset_set_refquota(const char *dsname, zprop_source_t source, uint64_t refquota) { dsl_dataset_set_qr_arg_t ddsqra; ddsqra.ddsqra_name = dsname; ddsqra.ddsqra_source = source; ddsqra.ddsqra_value = refquota; return (dsl_sync_task(dsname, dsl_dataset_set_refquota_check, dsl_dataset_set_refquota_sync, &ddsqra, 0, ZFS_SPACE_CHECK_NONE)); } static int dsl_dataset_set_refreservation_check(void *arg, dmu_tx_t *tx) { dsl_dataset_set_qr_arg_t *ddsqra = arg; dsl_pool_t *dp = dmu_tx_pool(tx); dsl_dataset_t *ds; int error; uint64_t newval, unique; if (spa_version(dp->dp_spa) < SPA_VERSION_REFRESERVATION) return (SET_ERROR(ENOTSUP)); error = dsl_dataset_hold(dp, ddsqra->ddsqra_name, FTAG, &ds); if (error != 0) return (error); if (dsl_dataset_is_snapshot(ds)) { dsl_dataset_rele(ds, FTAG); return (SET_ERROR(EINVAL)); } error = dsl_prop_predict(ds->ds_dir, zfs_prop_to_name(ZFS_PROP_REFRESERVATION), ddsqra->ddsqra_source, ddsqra->ddsqra_value, &newval); if (error != 0) { dsl_dataset_rele(ds, FTAG); return (error); } /* * If we are doing the preliminary check in open context, the * space estimates may be inaccurate. */ if (!dmu_tx_is_syncing(tx)) { dsl_dataset_rele(ds, FTAG); return (0); } mutex_enter(&ds->ds_lock); if (!DS_UNIQUE_IS_ACCURATE(ds)) dsl_dataset_recalc_head_uniq(ds); unique = ds->ds_phys->ds_unique_bytes; mutex_exit(&ds->ds_lock); if (MAX(unique, newval) > MAX(unique, ds->ds_reserved)) { uint64_t delta = MAX(unique, newval) - MAX(unique, ds->ds_reserved); if (delta > dsl_dir_space_available(ds->ds_dir, NULL, 0, B_TRUE) || (ds->ds_quota > 0 && newval > ds->ds_quota)) { dsl_dataset_rele(ds, FTAG); return (SET_ERROR(ENOSPC)); } } dsl_dataset_rele(ds, FTAG); return (0); } void dsl_dataset_set_refreservation_sync_impl(dsl_dataset_t *ds, zprop_source_t source, uint64_t value, dmu_tx_t *tx) { uint64_t newval; uint64_t unique; int64_t delta; dsl_prop_set_sync_impl(ds, zfs_prop_to_name(ZFS_PROP_REFRESERVATION), source, sizeof (value), 1, &value, tx); VERIFY0(dsl_prop_get_int_ds(ds, zfs_prop_to_name(ZFS_PROP_REFRESERVATION), &newval)); dmu_buf_will_dirty(ds->ds_dbuf, tx); mutex_enter(&ds->ds_dir->dd_lock); mutex_enter(&ds->ds_lock); ASSERT(DS_UNIQUE_IS_ACCURATE(ds)); unique = ds->ds_phys->ds_unique_bytes; delta = MAX(0, (int64_t)(newval - unique)) - MAX(0, (int64_t)(ds->ds_reserved - unique)); ds->ds_reserved = newval; mutex_exit(&ds->ds_lock); dsl_dir_diduse_space(ds->ds_dir, DD_USED_REFRSRV, delta, 0, 0, tx); mutex_exit(&ds->ds_dir->dd_lock); } static void dsl_dataset_set_refreservation_sync(void *arg, dmu_tx_t *tx) { dsl_dataset_set_qr_arg_t *ddsqra = arg; dsl_pool_t *dp = dmu_tx_pool(tx); dsl_dataset_t *ds; VERIFY0(dsl_dataset_hold(dp, ddsqra->ddsqra_name, FTAG, &ds)); dsl_dataset_set_refreservation_sync_impl(ds, ddsqra->ddsqra_source, ddsqra->ddsqra_value, tx); dsl_dataset_rele(ds, FTAG); } int dsl_dataset_set_refreservation(const char *dsname, zprop_source_t source, uint64_t refreservation) { dsl_dataset_set_qr_arg_t ddsqra; ddsqra.ddsqra_name = dsname; ddsqra.ddsqra_source = source; ddsqra.ddsqra_value = refreservation; return (dsl_sync_task(dsname, dsl_dataset_set_refreservation_check, dsl_dataset_set_refreservation_sync, &ddsqra, 0, ZFS_SPACE_CHECK_NONE)); } /* * Return (in *usedp) the amount of space written in new that is not * present in oldsnap. New may be a snapshot or the head. Old must be * a snapshot before new, in new's filesystem (or its origin). If not then * fail and return EINVAL. * * The written space is calculated by considering two components: First, we * ignore any freed space, and calculate the written as new's used space * minus old's used space. Next, we add in the amount of space that was freed * between the two snapshots, thus reducing new's used space relative to old's. * Specifically, this is the space that was born before old->ds_creation_txg, * and freed before new (ie. on new's deadlist or a previous deadlist). * * space freed [---------------------] * snapshots ---O-------O--------O-------O------ * oldsnap new */ int dsl_dataset_space_written(dsl_dataset_t *oldsnap, dsl_dataset_t *new, uint64_t *usedp, uint64_t *compp, uint64_t *uncompp) { int err = 0; uint64_t snapobj; dsl_pool_t *dp = new->ds_dir->dd_pool; ASSERT(dsl_pool_config_held(dp)); *usedp = 0; *usedp += new->ds_phys->ds_referenced_bytes; *usedp -= oldsnap->ds_phys->ds_referenced_bytes; *compp = 0; *compp += new->ds_phys->ds_compressed_bytes; *compp -= oldsnap->ds_phys->ds_compressed_bytes; *uncompp = 0; *uncompp += new->ds_phys->ds_uncompressed_bytes; *uncompp -= oldsnap->ds_phys->ds_uncompressed_bytes; snapobj = new->ds_object; while (snapobj != oldsnap->ds_object) { dsl_dataset_t *snap; uint64_t used, comp, uncomp; if (snapobj == new->ds_object) { snap = new; } else { err = dsl_dataset_hold_obj(dp, snapobj, FTAG, &snap); if (err != 0) break; } if (snap->ds_phys->ds_prev_snap_txg == oldsnap->ds_phys->ds_creation_txg) { /* * The blocks in the deadlist can not be born after * ds_prev_snap_txg, so get the whole deadlist space, * which is more efficient (especially for old-format * deadlists). Unfortunately the deadlist code * doesn't have enough information to make this * optimization itself. */ dsl_deadlist_space(&snap->ds_deadlist, &used, &comp, &uncomp); } else { dsl_deadlist_space_range(&snap->ds_deadlist, 0, oldsnap->ds_phys->ds_creation_txg, &used, &comp, &uncomp); } *usedp += used; *compp += comp; *uncompp += uncomp; /* * If we get to the beginning of the chain of snapshots * (ds_prev_snap_obj == 0) before oldsnap, then oldsnap * was not a snapshot of/before new. */ snapobj = snap->ds_phys->ds_prev_snap_obj; if (snap != new) dsl_dataset_rele(snap, FTAG); if (snapobj == 0) { err = SET_ERROR(EINVAL); break; } } return (err); } /* * Return (in *usedp) the amount of space that will be reclaimed if firstsnap, * lastsnap, and all snapshots in between are deleted. * * blocks that would be freed [---------------------------] * snapshots ---O-------O--------O-------O--------O * firstsnap lastsnap * * This is the set of blocks that were born after the snap before firstsnap, * (birth > firstsnap->prev_snap_txg) and died before the snap after the * last snap (ie, is on lastsnap->ds_next->ds_deadlist or an earlier deadlist). * We calculate this by iterating over the relevant deadlists (from the snap * after lastsnap, backward to the snap after firstsnap), summing up the * space on the deadlist that was born after the snap before firstsnap. */ int dsl_dataset_space_wouldfree(dsl_dataset_t *firstsnap, dsl_dataset_t *lastsnap, uint64_t *usedp, uint64_t *compp, uint64_t *uncompp) { int err = 0; uint64_t snapobj; dsl_pool_t *dp = firstsnap->ds_dir->dd_pool; ASSERT(dsl_dataset_is_snapshot(firstsnap)); ASSERT(dsl_dataset_is_snapshot(lastsnap)); /* * Check that the snapshots are in the same dsl_dir, and firstsnap * is before lastsnap. */ if (firstsnap->ds_dir != lastsnap->ds_dir || firstsnap->ds_phys->ds_creation_txg > lastsnap->ds_phys->ds_creation_txg) return (SET_ERROR(EINVAL)); *usedp = *compp = *uncompp = 0; snapobj = lastsnap->ds_phys->ds_next_snap_obj; while (snapobj != firstsnap->ds_object) { dsl_dataset_t *ds; uint64_t used, comp, uncomp; err = dsl_dataset_hold_obj(dp, snapobj, FTAG, &ds); if (err != 0) break; dsl_deadlist_space_range(&ds->ds_deadlist, firstsnap->ds_phys->ds_prev_snap_txg, UINT64_MAX, &used, &comp, &uncomp); *usedp += used; *compp += comp; *uncompp += uncomp; snapobj = ds->ds_phys->ds_prev_snap_obj; ASSERT3U(snapobj, !=, 0); dsl_dataset_rele(ds, FTAG); } return (err); } static int dsl_dataset_activate_large_blocks_check(void *arg, dmu_tx_t *tx) { const char *dsname = arg; dsl_dataset_t *ds; dsl_pool_t *dp = dmu_tx_pool(tx); int error = 0; if (!spa_feature_is_enabled(dp->dp_spa, SPA_FEATURE_LARGE_BLOCKS)) return (SET_ERROR(ENOTSUP)); ASSERT(spa_feature_is_enabled(dp->dp_spa, SPA_FEATURE_EXTENSIBLE_DATASET)); error = dsl_dataset_hold(dp, dsname, FTAG, &ds); if (error != 0) return (error); if (ds->ds_large_blocks) error = EALREADY; dsl_dataset_rele(ds, FTAG); return (error); } void dsl_dataset_activate_large_blocks_sync_impl(uint64_t dsobj, dmu_tx_t *tx) { spa_t *spa = dmu_tx_pool(tx)->dp_spa; objset_t *mos = dmu_tx_pool(tx)->dp_meta_objset; uint64_t zero = 0; spa_feature_incr(spa, SPA_FEATURE_LARGE_BLOCKS, tx); dmu_object_zapify(mos, dsobj, DMU_OT_DSL_DATASET, tx); VERIFY0(zap_add(mos, dsobj, DS_FIELD_LARGE_BLOCKS, sizeof (zero), 1, &zero, tx)); } static void dsl_dataset_activate_large_blocks_sync(void *arg, dmu_tx_t *tx) { const char *dsname = arg; dsl_dataset_t *ds; VERIFY0(dsl_dataset_hold(dmu_tx_pool(tx), dsname, FTAG, &ds)); dsl_dataset_activate_large_blocks_sync_impl(ds->ds_object, tx); ASSERT(!ds->ds_large_blocks); ds->ds_large_blocks = B_TRUE; dsl_dataset_rele(ds, FTAG); } int dsl_dataset_activate_large_blocks(const char *dsname) { int error; error = dsl_sync_task(dsname, dsl_dataset_activate_large_blocks_check, dsl_dataset_activate_large_blocks_sync, (void *)dsname, 1, ZFS_SPACE_CHECK_RESERVED); /* * EALREADY indicates that this dataset already supports large blocks. */ if (error == EALREADY) error = 0; return (error); } /* * Return TRUE if 'earlier' is an earlier snapshot in 'later's timeline. * For example, they could both be snapshots of the same filesystem, and * 'earlier' is before 'later'. Or 'earlier' could be the origin of * 'later's filesystem. Or 'earlier' could be an older snapshot in the origin's * filesystem. Or 'earlier' could be the origin's origin. * * If non-zero, earlier_txg is used instead of earlier's ds_creation_txg. */ boolean_t dsl_dataset_is_before(dsl_dataset_t *later, dsl_dataset_t *earlier, uint64_t earlier_txg) { dsl_pool_t *dp = later->ds_dir->dd_pool; int error; boolean_t ret; ASSERT(dsl_pool_config_held(dp)); ASSERT(dsl_dataset_is_snapshot(earlier) || earlier_txg != 0); if (earlier_txg == 0) earlier_txg = earlier->ds_phys->ds_creation_txg; if (dsl_dataset_is_snapshot(later) && earlier_txg >= later->ds_phys->ds_creation_txg) return (B_FALSE); if (later->ds_dir == earlier->ds_dir) return (B_TRUE); if (!dsl_dir_is_clone(later->ds_dir)) return (B_FALSE); if (later->ds_dir->dd_phys->dd_origin_obj == earlier->ds_object) return (B_TRUE); dsl_dataset_t *origin; error = dsl_dataset_hold_obj(dp, later->ds_dir->dd_phys->dd_origin_obj, FTAG, &origin); if (error != 0) return (B_FALSE); ret = dsl_dataset_is_before(origin, earlier, earlier_txg); dsl_dataset_rele(origin, FTAG); return (ret); } void dsl_dataset_zapify(dsl_dataset_t *ds, dmu_tx_t *tx) { objset_t *mos = ds->ds_dir->dd_pool->dp_meta_objset; dmu_object_zapify(mos, ds->ds_object, DMU_OT_DSL_DATASET, tx); } Index: projects/clang350-import/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_prop.c =================================================================== --- projects/clang350-import/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_prop.c (revision 275748) +++ projects/clang350-import/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_prop.c (revision 275749) @@ -1,1125 +1,1125 @@ /* * CDDL HEADER START * * The contents of this file are subject to the terms of the * Common Development and Distribution License (the "License"). * You may not use this file except in compliance with the License. * * You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE * or http://www.opensolaris.org/os/licensing. * See the License for the specific language governing permissions * and limitations under the License. * * When distributing Covered Code, include this CDDL HEADER in each * file and include the License file at usr/src/OPENSOLARIS.LICENSE. * If applicable, add the following below this CDDL HEADER, with the * fields enclosed by brackets "[]" replaced with your own identifying * information: Portions Copyright [yyyy] [name of copyright owner] * * CDDL HEADER END */ /* * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. * Copyright (c) 2012, 2014 by Delphix. All rights reserved. */ #include #include #include #include #include #include #include #include #include #include #include #include "zfs_prop.h" #define ZPROP_INHERIT_SUFFIX "$inherit" #define ZPROP_RECVD_SUFFIX "$recvd" static int dodefault(const char *propname, int intsz, int numints, void *buf) { zfs_prop_t prop; /* * The setonce properties are read-only, BUT they still * have a default value that can be used as the initial * value. */ if ((prop = zfs_name_to_prop(propname)) == ZPROP_INVAL || (zfs_prop_readonly(prop) && !zfs_prop_setonce(prop))) return (SET_ERROR(ENOENT)); if (zfs_prop_get_type(prop) == PROP_TYPE_STRING) { if (intsz != 1) return (SET_ERROR(EOVERFLOW)); (void) strncpy(buf, zfs_prop_default_string(prop), numints); } else { if (intsz != 8 || numints < 1) return (SET_ERROR(EOVERFLOW)); *(uint64_t *)buf = zfs_prop_default_numeric(prop); } return (0); } int dsl_prop_get_dd(dsl_dir_t *dd, const char *propname, int intsz, int numints, void *buf, char *setpoint, boolean_t snapshot) { int err = ENOENT; dsl_dir_t *target = dd; objset_t *mos = dd->dd_pool->dp_meta_objset; zfs_prop_t prop; boolean_t inheritable; boolean_t inheriting = B_FALSE; char *inheritstr; char *recvdstr; ASSERT(dsl_pool_config_held(dd->dd_pool)); if (setpoint) setpoint[0] = '\0'; prop = zfs_name_to_prop(propname); inheritable = (prop == ZPROP_INVAL || zfs_prop_inheritable(prop)); inheritstr = kmem_asprintf("%s%s", propname, ZPROP_INHERIT_SUFFIX); recvdstr = kmem_asprintf("%s%s", propname, ZPROP_RECVD_SUFFIX); /* * Note: dd may become NULL, therefore we shouldn't dereference it * after this loop. */ for (; dd != NULL; dd = dd->dd_parent) { if (dd != target || snapshot) { if (!inheritable) break; inheriting = B_TRUE; } /* Check for a local value. */ err = zap_lookup(mos, dd->dd_phys->dd_props_zapobj, propname, intsz, numints, buf); if (err != ENOENT) { if (setpoint != NULL && err == 0) dsl_dir_name(dd, setpoint); break; } /* * Skip the check for a received value if there is an explicit * inheritance entry. */ err = zap_contains(mos, dd->dd_phys->dd_props_zapobj, inheritstr); if (err != 0 && err != ENOENT) break; if (err == ENOENT) { /* Check for a received value. */ err = zap_lookup(mos, dd->dd_phys->dd_props_zapobj, recvdstr, intsz, numints, buf); if (err != ENOENT) { if (setpoint != NULL && err == 0) { if (inheriting) { dsl_dir_name(dd, setpoint); } else { (void) strcpy(setpoint, ZPROP_SOURCE_VAL_RECVD); } } break; } } /* * If we found an explicit inheritance entry, err is zero even * though we haven't yet found the value, so reinitializing err * at the end of the loop (instead of at the beginning) ensures * that err has a valid post-loop value. */ err = SET_ERROR(ENOENT); } if (err == ENOENT) err = dodefault(propname, intsz, numints, buf); strfree(inheritstr); strfree(recvdstr); return (err); } int dsl_prop_get_ds(dsl_dataset_t *ds, const char *propname, int intsz, int numints, void *buf, char *setpoint) { zfs_prop_t prop = zfs_name_to_prop(propname); boolean_t inheritable; boolean_t snapshot; uint64_t zapobj; ASSERT(dsl_pool_config_held(ds->ds_dir->dd_pool)); inheritable = (prop == ZPROP_INVAL || zfs_prop_inheritable(prop)); - snapshot = (ds->ds_phys != NULL && dsl_dataset_is_snapshot(ds)); - zapobj = (ds->ds_phys == NULL ? 0 : ds->ds_phys->ds_props_obj); + snapshot = dsl_dataset_is_snapshot(ds); + zapobj = ds->ds_phys->ds_props_obj; if (zapobj != 0) { objset_t *mos = ds->ds_dir->dd_pool->dp_meta_objset; int err; ASSERT(snapshot); /* Check for a local value. */ err = zap_lookup(mos, zapobj, propname, intsz, numints, buf); if (err != ENOENT) { if (setpoint != NULL && err == 0) dsl_dataset_name(ds, setpoint); return (err); } /* * Skip the check for a received value if there is an explicit * inheritance entry. */ if (inheritable) { char *inheritstr = kmem_asprintf("%s%s", propname, ZPROP_INHERIT_SUFFIX); err = zap_contains(mos, zapobj, inheritstr); strfree(inheritstr); if (err != 0 && err != ENOENT) return (err); } if (err == ENOENT) { /* Check for a received value. */ char *recvdstr = kmem_asprintf("%s%s", propname, ZPROP_RECVD_SUFFIX); err = zap_lookup(mos, zapobj, recvdstr, intsz, numints, buf); strfree(recvdstr); if (err != ENOENT) { if (setpoint != NULL && err == 0) (void) strcpy(setpoint, ZPROP_SOURCE_VAL_RECVD); return (err); } } } return (dsl_prop_get_dd(ds->ds_dir, propname, intsz, numints, buf, setpoint, snapshot)); } /* * Register interest in the named property. We'll call the callback * once to notify it of the current property value, and again each time * the property changes, until this callback is unregistered. * * Return 0 on success, errno if the prop is not an integer value. */ int dsl_prop_register(dsl_dataset_t *ds, const char *propname, dsl_prop_changed_cb_t *callback, void *cbarg) { dsl_dir_t *dd = ds->ds_dir; dsl_pool_t *dp = dd->dd_pool; uint64_t value; dsl_prop_cb_record_t *cbr; int err; ASSERT(dsl_pool_config_held(dp)); err = dsl_prop_get_int_ds(ds, propname, &value); if (err != 0) return (err); cbr = kmem_alloc(sizeof (dsl_prop_cb_record_t), KM_SLEEP); cbr->cbr_ds = ds; cbr->cbr_propname = kmem_alloc(strlen(propname)+1, KM_SLEEP); (void) strcpy((char *)cbr->cbr_propname, propname); cbr->cbr_func = callback; cbr->cbr_arg = cbarg; mutex_enter(&dd->dd_lock); list_insert_head(&dd->dd_prop_cbs, cbr); mutex_exit(&dd->dd_lock); cbr->cbr_func(cbr->cbr_arg, value); return (0); } int dsl_prop_get(const char *dsname, const char *propname, int intsz, int numints, void *buf, char *setpoint) { objset_t *os; int error; error = dmu_objset_hold(dsname, FTAG, &os); if (error != 0) return (error); error = dsl_prop_get_ds(dmu_objset_ds(os), propname, intsz, numints, buf, setpoint); dmu_objset_rele(os, FTAG); return (error); } /* * Get the current property value. It may have changed by the time this * function returns, so it is NOT safe to follow up with * dsl_prop_register() and assume that the value has not changed in * between. * * Return 0 on success, ENOENT if ddname is invalid. */ int dsl_prop_get_integer(const char *ddname, const char *propname, uint64_t *valuep, char *setpoint) { return (dsl_prop_get(ddname, propname, 8, 1, valuep, setpoint)); } int dsl_prop_get_int_ds(dsl_dataset_t *ds, const char *propname, uint64_t *valuep) { return (dsl_prop_get_ds(ds, propname, 8, 1, valuep, NULL)); } /* * Predict the effective value of the given special property if it were set with * the given value and source. This is not a general purpose function. It exists * only to handle the special requirements of the quota and reservation * properties. The fact that these properties are non-inheritable greatly * simplifies the prediction logic. * * Returns 0 on success, a positive error code on failure, or -1 if called with * a property not handled by this function. */ int dsl_prop_predict(dsl_dir_t *dd, const char *propname, zprop_source_t source, uint64_t value, uint64_t *newvalp) { zfs_prop_t prop = zfs_name_to_prop(propname); objset_t *mos; uint64_t zapobj; uint64_t version; char *recvdstr; int err = 0; switch (prop) { case ZFS_PROP_QUOTA: case ZFS_PROP_RESERVATION: case ZFS_PROP_REFQUOTA: case ZFS_PROP_REFRESERVATION: break; default: return (-1); } mos = dd->dd_pool->dp_meta_objset; zapobj = dd->dd_phys->dd_props_zapobj; recvdstr = kmem_asprintf("%s%s", propname, ZPROP_RECVD_SUFFIX); version = spa_version(dd->dd_pool->dp_spa); if (version < SPA_VERSION_RECVD_PROPS) { if (source & ZPROP_SRC_NONE) source = ZPROP_SRC_NONE; else if (source & ZPROP_SRC_RECEIVED) source = ZPROP_SRC_LOCAL; } switch (source) { case ZPROP_SRC_NONE: /* Revert to the received value, if any. */ err = zap_lookup(mos, zapobj, recvdstr, 8, 1, newvalp); if (err == ENOENT) *newvalp = 0; break; case ZPROP_SRC_LOCAL: *newvalp = value; break; case ZPROP_SRC_RECEIVED: /* * If there's no local setting, then the new received value will * be the effective value. */ err = zap_lookup(mos, zapobj, propname, 8, 1, newvalp); if (err == ENOENT) *newvalp = value; break; case (ZPROP_SRC_NONE | ZPROP_SRC_RECEIVED): /* * We're clearing the received value, so the local setting (if * it exists) remains the effective value. */ err = zap_lookup(mos, zapobj, propname, 8, 1, newvalp); if (err == ENOENT) *newvalp = 0; break; default: panic("unexpected property source: %d", source); } strfree(recvdstr); if (err == ENOENT) return (0); return (err); } /* * Unregister this callback. Return 0 on success, ENOENT if ddname is * invalid, or ENOMSG if no matching callback registered. */ int dsl_prop_unregister(dsl_dataset_t *ds, const char *propname, dsl_prop_changed_cb_t *callback, void *cbarg) { dsl_dir_t *dd = ds->ds_dir; dsl_prop_cb_record_t *cbr; mutex_enter(&dd->dd_lock); for (cbr = list_head(&dd->dd_prop_cbs); cbr; cbr = list_next(&dd->dd_prop_cbs, cbr)) { if (cbr->cbr_ds == ds && cbr->cbr_func == callback && cbr->cbr_arg == cbarg && strcmp(cbr->cbr_propname, propname) == 0) break; } if (cbr == NULL) { mutex_exit(&dd->dd_lock); return (SET_ERROR(ENOMSG)); } list_remove(&dd->dd_prop_cbs, cbr); mutex_exit(&dd->dd_lock); kmem_free((void*)cbr->cbr_propname, strlen(cbr->cbr_propname)+1); kmem_free(cbr, sizeof (dsl_prop_cb_record_t)); return (0); } boolean_t dsl_prop_hascb(dsl_dataset_t *ds) { dsl_dir_t *dd = ds->ds_dir; boolean_t rv = B_FALSE; dsl_prop_cb_record_t *cbr; mutex_enter(&dd->dd_lock); for (cbr = list_head(&dd->dd_prop_cbs); cbr; cbr = list_next(&dd->dd_prop_cbs, cbr)) { if (cbr->cbr_ds == ds) { rv = B_TRUE; break; } } mutex_exit(&dd->dd_lock); return (rv); } /* ARGSUSED */ static int dsl_prop_notify_all_cb(dsl_pool_t *dp, dsl_dataset_t *ds, void *arg) { dsl_dir_t *dd = ds->ds_dir; dsl_prop_cb_record_t *cbr; mutex_enter(&dd->dd_lock); for (cbr = list_head(&dd->dd_prop_cbs); cbr; cbr = list_next(&dd->dd_prop_cbs, cbr)) { uint64_t value; if (dsl_prop_get_ds(cbr->cbr_ds, cbr->cbr_propname, sizeof (value), 1, &value, NULL) == 0) cbr->cbr_func(cbr->cbr_arg, value); } mutex_exit(&dd->dd_lock); return (0); } /* * Update all property values for ddobj & its descendants. This is used * when renaming the dir. */ void dsl_prop_notify_all(dsl_dir_t *dd) { dsl_pool_t *dp = dd->dd_pool; ASSERT(RRW_WRITE_HELD(&dp->dp_config_rwlock)); (void) dmu_objset_find_dp(dp, dd->dd_object, dsl_prop_notify_all_cb, NULL, DS_FIND_CHILDREN); } static void dsl_prop_changed_notify(dsl_pool_t *dp, uint64_t ddobj, const char *propname, uint64_t value, int first) { dsl_dir_t *dd; dsl_prop_cb_record_t *cbr; objset_t *mos = dp->dp_meta_objset; zap_cursor_t zc; zap_attribute_t *za; int err; ASSERT(RRW_WRITE_HELD(&dp->dp_config_rwlock)); err = dsl_dir_hold_obj(dp, ddobj, NULL, FTAG, &dd); if (err) return; if (!first) { /* * If the prop is set here, then this change is not * being inherited here or below; stop the recursion. */ err = zap_contains(mos, dd->dd_phys->dd_props_zapobj, propname); if (err == 0) { dsl_dir_rele(dd, FTAG); return; } ASSERT3U(err, ==, ENOENT); } mutex_enter(&dd->dd_lock); for (cbr = list_head(&dd->dd_prop_cbs); cbr; cbr = list_next(&dd->dd_prop_cbs, cbr)) { uint64_t propobj = cbr->cbr_ds->ds_phys->ds_props_obj; if (strcmp(cbr->cbr_propname, propname) != 0) continue; /* * If the property is set on this ds, then it is not * inherited here; don't call the callback. */ if (propobj && 0 == zap_contains(mos, propobj, propname)) continue; cbr->cbr_func(cbr->cbr_arg, value); } mutex_exit(&dd->dd_lock); za = kmem_alloc(sizeof (zap_attribute_t), KM_SLEEP); for (zap_cursor_init(&zc, mos, dd->dd_phys->dd_child_dir_zapobj); zap_cursor_retrieve(&zc, za) == 0; zap_cursor_advance(&zc)) { dsl_prop_changed_notify(dp, za->za_first_integer, propname, value, FALSE); } kmem_free(za, sizeof (zap_attribute_t)); zap_cursor_fini(&zc); dsl_dir_rele(dd, FTAG); } void dsl_prop_set_sync_impl(dsl_dataset_t *ds, const char *propname, zprop_source_t source, int intsz, int numints, const void *value, dmu_tx_t *tx) { objset_t *mos = ds->ds_dir->dd_pool->dp_meta_objset; uint64_t zapobj, intval, dummy; int isint; char valbuf[32]; const char *valstr = NULL; char *inheritstr; char *recvdstr; char *tbuf = NULL; int err; uint64_t version = spa_version(ds->ds_dir->dd_pool->dp_spa); isint = (dodefault(propname, 8, 1, &intval) == 0); - if (ds->ds_phys != NULL && dsl_dataset_is_snapshot(ds)) { + if (dsl_dataset_is_snapshot(ds)) { ASSERT(version >= SPA_VERSION_SNAP_PROPS); if (ds->ds_phys->ds_props_obj == 0) { dmu_buf_will_dirty(ds->ds_dbuf, tx); ds->ds_phys->ds_props_obj = zap_create(mos, DMU_OT_DSL_PROPS, DMU_OT_NONE, 0, tx); } zapobj = ds->ds_phys->ds_props_obj; } else { zapobj = ds->ds_dir->dd_phys->dd_props_zapobj; } if (version < SPA_VERSION_RECVD_PROPS) { if (source & ZPROP_SRC_NONE) source = ZPROP_SRC_NONE; else if (source & ZPROP_SRC_RECEIVED) source = ZPROP_SRC_LOCAL; } inheritstr = kmem_asprintf("%s%s", propname, ZPROP_INHERIT_SUFFIX); recvdstr = kmem_asprintf("%s%s", propname, ZPROP_RECVD_SUFFIX); switch (source) { case ZPROP_SRC_NONE: /* * revert to received value, if any (inherit -S) * - remove propname * - remove propname$inherit */ err = zap_remove(mos, zapobj, propname, tx); ASSERT(err == 0 || err == ENOENT); err = zap_remove(mos, zapobj, inheritstr, tx); ASSERT(err == 0 || err == ENOENT); break; case ZPROP_SRC_LOCAL: /* * remove propname$inherit * set propname -> value */ err = zap_remove(mos, zapobj, inheritstr, tx); ASSERT(err == 0 || err == ENOENT); VERIFY0(zap_update(mos, zapobj, propname, intsz, numints, value, tx)); break; case ZPROP_SRC_INHERITED: /* * explicitly inherit * - remove propname * - set propname$inherit */ err = zap_remove(mos, zapobj, propname, tx); ASSERT(err == 0 || err == ENOENT); if (version >= SPA_VERSION_RECVD_PROPS && dsl_prop_get_int_ds(ds, ZPROP_HAS_RECVD, &dummy) == 0) { dummy = 0; VERIFY0(zap_update(mos, zapobj, inheritstr, 8, 1, &dummy, tx)); } break; case ZPROP_SRC_RECEIVED: /* * set propname$recvd -> value */ err = zap_update(mos, zapobj, recvdstr, intsz, numints, value, tx); ASSERT(err == 0); break; case (ZPROP_SRC_NONE | ZPROP_SRC_LOCAL | ZPROP_SRC_RECEIVED): /* * clear local and received settings * - remove propname * - remove propname$inherit * - remove propname$recvd */ err = zap_remove(mos, zapobj, propname, tx); ASSERT(err == 0 || err == ENOENT); err = zap_remove(mos, zapobj, inheritstr, tx); ASSERT(err == 0 || err == ENOENT); /* FALLTHRU */ case (ZPROP_SRC_NONE | ZPROP_SRC_RECEIVED): /* * remove propname$recvd */ err = zap_remove(mos, zapobj, recvdstr, tx); ASSERT(err == 0 || err == ENOENT); break; default: cmn_err(CE_PANIC, "unexpected property source: %d", source); } strfree(inheritstr); strfree(recvdstr); if (isint) { VERIFY0(dsl_prop_get_int_ds(ds, propname, &intval)); - if (ds->ds_phys != NULL && dsl_dataset_is_snapshot(ds)) { + if (dsl_dataset_is_snapshot(ds)) { dsl_prop_cb_record_t *cbr; /* * It's a snapshot; nothing can inherit this * property, so just look for callbacks on this * ds here. */ mutex_enter(&ds->ds_dir->dd_lock); for (cbr = list_head(&ds->ds_dir->dd_prop_cbs); cbr; cbr = list_next(&ds->ds_dir->dd_prop_cbs, cbr)) { if (cbr->cbr_ds == ds && strcmp(cbr->cbr_propname, propname) == 0) cbr->cbr_func(cbr->cbr_arg, intval); } mutex_exit(&ds->ds_dir->dd_lock); } else { dsl_prop_changed_notify(ds->ds_dir->dd_pool, ds->ds_dir->dd_object, propname, intval, TRUE); } (void) snprintf(valbuf, sizeof (valbuf), "%lld", (longlong_t)intval); valstr = valbuf; } else { if (source == ZPROP_SRC_LOCAL) { valstr = value; } else { tbuf = kmem_alloc(ZAP_MAXVALUELEN, KM_SLEEP); if (dsl_prop_get_ds(ds, propname, 1, ZAP_MAXVALUELEN, tbuf, NULL) == 0) valstr = tbuf; } } spa_history_log_internal_ds(ds, (source == ZPROP_SRC_NONE || source == ZPROP_SRC_INHERITED) ? "inherit" : "set", tx, "%s=%s", propname, (valstr == NULL ? "" : valstr)); if (tbuf != NULL) kmem_free(tbuf, ZAP_MAXVALUELEN); } int dsl_prop_set_int(const char *dsname, const char *propname, zprop_source_t source, uint64_t value) { nvlist_t *nvl = fnvlist_alloc(); int error; fnvlist_add_uint64(nvl, propname, value); error = dsl_props_set(dsname, source, nvl); fnvlist_free(nvl); return (error); } int dsl_prop_set_string(const char *dsname, const char *propname, zprop_source_t source, const char *value) { nvlist_t *nvl = fnvlist_alloc(); int error; fnvlist_add_string(nvl, propname, value); error = dsl_props_set(dsname, source, nvl); fnvlist_free(nvl); return (error); } int dsl_prop_inherit(const char *dsname, const char *propname, zprop_source_t source) { nvlist_t *nvl = fnvlist_alloc(); int error; fnvlist_add_boolean(nvl, propname); error = dsl_props_set(dsname, source, nvl); fnvlist_free(nvl); return (error); } typedef struct dsl_props_set_arg { const char *dpsa_dsname; zprop_source_t dpsa_source; nvlist_t *dpsa_props; } dsl_props_set_arg_t; static int dsl_props_set_check(void *arg, dmu_tx_t *tx) { dsl_props_set_arg_t *dpsa = arg; dsl_pool_t *dp = dmu_tx_pool(tx); dsl_dataset_t *ds; uint64_t version; nvpair_t *elem = NULL; int err; err = dsl_dataset_hold(dp, dpsa->dpsa_dsname, FTAG, &ds); if (err != 0) return (err); version = spa_version(ds->ds_dir->dd_pool->dp_spa); while ((elem = nvlist_next_nvpair(dpsa->dpsa_props, elem)) != NULL) { if (strlen(nvpair_name(elem)) >= ZAP_MAXNAMELEN) { dsl_dataset_rele(ds, FTAG); return (SET_ERROR(ENAMETOOLONG)); } if (nvpair_type(elem) == DATA_TYPE_STRING) { char *valstr = fnvpair_value_string(elem); if (strlen(valstr) >= (version < SPA_VERSION_STMF_PROP ? ZAP_OLDMAXVALUELEN : ZAP_MAXVALUELEN)) { dsl_dataset_rele(ds, FTAG); return (E2BIG); } } } if (dsl_dataset_is_snapshot(ds) && version < SPA_VERSION_SNAP_PROPS) { dsl_dataset_rele(ds, FTAG); return (SET_ERROR(ENOTSUP)); } dsl_dataset_rele(ds, FTAG); return (0); } void dsl_props_set_sync_impl(dsl_dataset_t *ds, zprop_source_t source, nvlist_t *props, dmu_tx_t *tx) { nvpair_t *elem = NULL; while ((elem = nvlist_next_nvpair(props, elem)) != NULL) { nvpair_t *pair = elem; if (nvpair_type(pair) == DATA_TYPE_NVLIST) { /* * dsl_prop_get_all_impl() returns properties in this * format. */ nvlist_t *attrs = fnvpair_value_nvlist(pair); pair = fnvlist_lookup_nvpair(attrs, ZPROP_VALUE); } if (nvpair_type(pair) == DATA_TYPE_STRING) { const char *value = fnvpair_value_string(pair); dsl_prop_set_sync_impl(ds, nvpair_name(pair), source, 1, strlen(value) + 1, value, tx); } else if (nvpair_type(pair) == DATA_TYPE_UINT64) { uint64_t intval = fnvpair_value_uint64(pair); dsl_prop_set_sync_impl(ds, nvpair_name(pair), source, sizeof (intval), 1, &intval, tx); } else if (nvpair_type(pair) == DATA_TYPE_BOOLEAN) { dsl_prop_set_sync_impl(ds, nvpair_name(pair), source, 0, 0, NULL, tx); } else { panic("invalid nvpair type"); } } } static void dsl_props_set_sync(void *arg, dmu_tx_t *tx) { dsl_props_set_arg_t *dpsa = arg; dsl_pool_t *dp = dmu_tx_pool(tx); dsl_dataset_t *ds; VERIFY0(dsl_dataset_hold(dp, dpsa->dpsa_dsname, FTAG, &ds)); dsl_props_set_sync_impl(ds, dpsa->dpsa_source, dpsa->dpsa_props, tx); dsl_dataset_rele(ds, FTAG); } /* * All-or-nothing; if any prop can't be set, nothing will be modified. */ int dsl_props_set(const char *dsname, zprop_source_t source, nvlist_t *props) { dsl_props_set_arg_t dpsa; int nblks = 0; dpsa.dpsa_dsname = dsname; dpsa.dpsa_source = source; dpsa.dpsa_props = props; /* * If the source includes NONE, then we will only be removing entries * from the ZAP object. In that case don't check for ENOSPC. */ if ((source & ZPROP_SRC_NONE) == 0) nblks = 2 * fnvlist_num_pairs(props); return (dsl_sync_task(dsname, dsl_props_set_check, dsl_props_set_sync, &dpsa, nblks, ZFS_SPACE_CHECK_RESERVED)); } typedef enum dsl_prop_getflags { DSL_PROP_GET_INHERITING = 0x1, /* searching parent of target ds */ DSL_PROP_GET_SNAPSHOT = 0x2, /* snapshot dataset */ DSL_PROP_GET_LOCAL = 0x4, /* local properties */ DSL_PROP_GET_RECEIVED = 0x8 /* received properties */ } dsl_prop_getflags_t; static int dsl_prop_get_all_impl(objset_t *mos, uint64_t propobj, const char *setpoint, dsl_prop_getflags_t flags, nvlist_t *nv) { zap_cursor_t zc; zap_attribute_t za; int err = 0; for (zap_cursor_init(&zc, mos, propobj); (err = zap_cursor_retrieve(&zc, &za)) == 0; zap_cursor_advance(&zc)) { nvlist_t *propval; zfs_prop_t prop; char buf[ZAP_MAXNAMELEN]; char *valstr; const char *suffix; const char *propname; const char *source; suffix = strchr(za.za_name, '$'); if (suffix == NULL) { /* * Skip local properties if we only want received * properties. */ if (flags & DSL_PROP_GET_RECEIVED) continue; propname = za.za_name; source = setpoint; } else if (strcmp(suffix, ZPROP_INHERIT_SUFFIX) == 0) { /* Skip explicitly inherited entries. */ continue; } else if (strcmp(suffix, ZPROP_RECVD_SUFFIX) == 0) { if (flags & DSL_PROP_GET_LOCAL) continue; (void) strncpy(buf, za.za_name, (suffix - za.za_name)); buf[suffix - za.za_name] = '\0'; propname = buf; if (!(flags & DSL_PROP_GET_RECEIVED)) { /* Skip if locally overridden. */ err = zap_contains(mos, propobj, propname); if (err == 0) continue; if (err != ENOENT) break; /* Skip if explicitly inherited. */ valstr = kmem_asprintf("%s%s", propname, ZPROP_INHERIT_SUFFIX); err = zap_contains(mos, propobj, valstr); strfree(valstr); if (err == 0) continue; if (err != ENOENT) break; } source = ((flags & DSL_PROP_GET_INHERITING) ? setpoint : ZPROP_SOURCE_VAL_RECVD); } else { /* * For backward compatibility, skip suffixes we don't * recognize. */ continue; } prop = zfs_name_to_prop(propname); /* Skip non-inheritable properties. */ if ((flags & DSL_PROP_GET_INHERITING) && prop != ZPROP_INVAL && !zfs_prop_inheritable(prop)) continue; /* Skip properties not valid for this type. */ if ((flags & DSL_PROP_GET_SNAPSHOT) && prop != ZPROP_INVAL && !zfs_prop_valid_for_type(prop, ZFS_TYPE_SNAPSHOT)) continue; /* Skip properties already defined. */ if (nvlist_exists(nv, propname)) continue; VERIFY(nvlist_alloc(&propval, NV_UNIQUE_NAME, KM_SLEEP) == 0); if (za.za_integer_length == 1) { /* * String property */ char *tmp = kmem_alloc(za.za_num_integers, KM_SLEEP); err = zap_lookup(mos, propobj, za.za_name, 1, za.za_num_integers, tmp); if (err != 0) { kmem_free(tmp, za.za_num_integers); break; } VERIFY(nvlist_add_string(propval, ZPROP_VALUE, tmp) == 0); kmem_free(tmp, za.za_num_integers); } else { /* * Integer property */ ASSERT(za.za_integer_length == 8); (void) nvlist_add_uint64(propval, ZPROP_VALUE, za.za_first_integer); } VERIFY(nvlist_add_string(propval, ZPROP_SOURCE, source) == 0); VERIFY(nvlist_add_nvlist(nv, propname, propval) == 0); nvlist_free(propval); } zap_cursor_fini(&zc); if (err == ENOENT) err = 0; return (err); } /* * Iterate over all properties for this dataset and return them in an nvlist. */ static int dsl_prop_get_all_ds(dsl_dataset_t *ds, nvlist_t **nvp, dsl_prop_getflags_t flags) { dsl_dir_t *dd = ds->ds_dir; dsl_pool_t *dp = dd->dd_pool; objset_t *mos = dp->dp_meta_objset; int err = 0; char setpoint[MAXNAMELEN]; VERIFY(nvlist_alloc(nvp, NV_UNIQUE_NAME, KM_SLEEP) == 0); if (dsl_dataset_is_snapshot(ds)) flags |= DSL_PROP_GET_SNAPSHOT; ASSERT(dsl_pool_config_held(dp)); if (ds->ds_phys->ds_props_obj != 0) { ASSERT(flags & DSL_PROP_GET_SNAPSHOT); dsl_dataset_name(ds, setpoint); err = dsl_prop_get_all_impl(mos, ds->ds_phys->ds_props_obj, setpoint, flags, *nvp); if (err) goto out; } for (; dd != NULL; dd = dd->dd_parent) { if (dd != ds->ds_dir || (flags & DSL_PROP_GET_SNAPSHOT)) { if (flags & (DSL_PROP_GET_LOCAL | DSL_PROP_GET_RECEIVED)) break; flags |= DSL_PROP_GET_INHERITING; } dsl_dir_name(dd, setpoint); err = dsl_prop_get_all_impl(mos, dd->dd_phys->dd_props_zapobj, setpoint, flags, *nvp); if (err) break; } out: return (err); } boolean_t dsl_prop_get_hasrecvd(const char *dsname) { uint64_t dummy; return (0 == dsl_prop_get_integer(dsname, ZPROP_HAS_RECVD, &dummy, NULL)); } static int dsl_prop_set_hasrecvd_impl(const char *dsname, zprop_source_t source) { uint64_t version; spa_t *spa; int error = 0; VERIFY0(spa_open(dsname, &spa, FTAG)); version = spa_version(spa); spa_close(spa, FTAG); if (version >= SPA_VERSION_RECVD_PROPS) error = dsl_prop_set_int(dsname, ZPROP_HAS_RECVD, source, 0); return (error); } /* * Call after successfully receiving properties to ensure that only the first * receive on or after SPA_VERSION_RECVD_PROPS blows away local properties. */ int dsl_prop_set_hasrecvd(const char *dsname) { int error = 0; if (!dsl_prop_get_hasrecvd(dsname)) error = dsl_prop_set_hasrecvd_impl(dsname, ZPROP_SRC_LOCAL); return (error); } void dsl_prop_unset_hasrecvd(const char *dsname) { VERIFY0(dsl_prop_set_hasrecvd_impl(dsname, ZPROP_SRC_NONE)); } int dsl_prop_get_all(objset_t *os, nvlist_t **nvp) { return (dsl_prop_get_all_ds(os->os_dsl_dataset, nvp, 0)); } int dsl_prop_get_received(const char *dsname, nvlist_t **nvp) { objset_t *os; int error; /* * Received properties are not distinguishable from local properties * until the dataset has received properties on or after * SPA_VERSION_RECVD_PROPS. */ dsl_prop_getflags_t flags = (dsl_prop_get_hasrecvd(dsname) ? DSL_PROP_GET_RECEIVED : DSL_PROP_GET_LOCAL); error = dmu_objset_hold(dsname, FTAG, &os); if (error != 0) return (error); error = dsl_prop_get_all_ds(os->os_dsl_dataset, nvp, flags); dmu_objset_rele(os, FTAG); return (error); } void dsl_prop_nvlist_add_uint64(nvlist_t *nv, zfs_prop_t prop, uint64_t value) { nvlist_t *propval; const char *propname = zfs_prop_to_name(prop); uint64_t default_value; if (nvlist_lookup_nvlist(nv, propname, &propval) == 0) { VERIFY(nvlist_add_uint64(propval, ZPROP_VALUE, value) == 0); return; } VERIFY(nvlist_alloc(&propval, NV_UNIQUE_NAME, KM_SLEEP) == 0); VERIFY(nvlist_add_uint64(propval, ZPROP_VALUE, value) == 0); /* Indicate the default source if we can. */ if (dodefault(propname, 8, 1, &default_value) == 0 && value == default_value) { VERIFY(nvlist_add_string(propval, ZPROP_SOURCE, "") == 0); } VERIFY(nvlist_add_nvlist(nv, propname, propval) == 0); nvlist_free(propval); } void dsl_prop_nvlist_add_string(nvlist_t *nv, zfs_prop_t prop, const char *value) { nvlist_t *propval; const char *propname = zfs_prop_to_name(prop); if (nvlist_lookup_nvlist(nv, propname, &propval) == 0) { VERIFY(nvlist_add_string(propval, ZPROP_VALUE, value) == 0); return; } VERIFY(nvlist_alloc(&propval, NV_UNIQUE_NAME, KM_SLEEP) == 0); VERIFY(nvlist_add_string(propval, ZPROP_VALUE, value) == 0); VERIFY(nvlist_add_nvlist(nv, propname, propval) == 0); nvlist_free(propval); } Index: projects/clang350-import/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_scan.c =================================================================== --- projects/clang350-import/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_scan.c (revision 275748) +++ projects/clang350-import/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_scan.c (revision 275749) @@ -1,1812 +1,1827 @@ /* * CDDL HEADER START * * The contents of this file are subject to the terms of the * Common Development and Distribution License (the "License"). * You may not use this file except in compliance with the License. * * You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE * or http://www.opensolaris.org/os/licensing. * See the License for the specific language governing permissions * and limitations under the License. * * When distributing Covered Code, include this CDDL HEADER in each * file and include the License file at usr/src/OPENSOLARIS.LICENSE. * If applicable, add the following below this CDDL HEADER, with the * fields enclosed by brackets "[]" replaced with your own identifying * information: Portions Copyright [yyyy] [name of copyright owner] * * CDDL HEADER END */ /* * Copyright (c) 2008, 2010, Oracle and/or its affiliates. All rights reserved. * Copyright (c) 2011, 2014 by Delphix. All rights reserved. */ #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #ifdef _KERNEL #include #endif typedef int (scan_cb_t)(dsl_pool_t *, const blkptr_t *, const zbookmark_phys_t *); static scan_cb_t dsl_scan_scrub_cb; static void dsl_scan_cancel_sync(void *, dmu_tx_t *); static void dsl_scan_sync_state(dsl_scan_t *, dmu_tx_t *tx); unsigned int zfs_top_maxinflight = 32; /* maximum I/Os per top-level */ unsigned int zfs_resilver_delay = 2; /* number of ticks to delay resilver */ unsigned int zfs_scrub_delay = 4; /* number of ticks to delay scrub */ unsigned int zfs_scan_idle = 50; /* idle window in clock ticks */ unsigned int zfs_scan_min_time_ms = 1000; /* min millisecs to scrub per txg */ unsigned int zfs_free_min_time_ms = 1000; /* min millisecs to free per txg */ unsigned int zfs_resilver_min_time_ms = 3000; /* min millisecs to resilver per txg */ boolean_t zfs_no_scrub_io = B_FALSE; /* set to disable scrub i/o */ boolean_t zfs_no_scrub_prefetch = B_FALSE; /* set to disable scrub prefetch */ SYSCTL_DECL(_vfs_zfs); SYSCTL_UINT(_vfs_zfs, OID_AUTO, top_maxinflight, CTLFLAG_RWTUN, &zfs_top_maxinflight, 0, "Maximum I/Os per top-level vdev"); SYSCTL_UINT(_vfs_zfs, OID_AUTO, resilver_delay, CTLFLAG_RWTUN, &zfs_resilver_delay, 0, "Number of ticks to delay resilver"); SYSCTL_UINT(_vfs_zfs, OID_AUTO, scrub_delay, CTLFLAG_RWTUN, &zfs_scrub_delay, 0, "Number of ticks to delay scrub"); SYSCTL_UINT(_vfs_zfs, OID_AUTO, scan_idle, CTLFLAG_RWTUN, &zfs_scan_idle, 0, "Idle scan window in clock ticks"); SYSCTL_UINT(_vfs_zfs, OID_AUTO, scan_min_time_ms, CTLFLAG_RWTUN, &zfs_scan_min_time_ms, 0, "Min millisecs to scrub per txg"); SYSCTL_UINT(_vfs_zfs, OID_AUTO, free_min_time_ms, CTLFLAG_RWTUN, &zfs_free_min_time_ms, 0, "Min millisecs to free per txg"); SYSCTL_UINT(_vfs_zfs, OID_AUTO, resilver_min_time_ms, CTLFLAG_RWTUN, &zfs_resilver_min_time_ms, 0, "Min millisecs to resilver per txg"); SYSCTL_INT(_vfs_zfs, OID_AUTO, no_scrub_io, CTLFLAG_RWTUN, &zfs_no_scrub_io, 0, "Disable scrub I/O"); SYSCTL_INT(_vfs_zfs, OID_AUTO, no_scrub_prefetch, CTLFLAG_RWTUN, &zfs_no_scrub_prefetch, 0, "Disable scrub prefetching"); enum ddt_class zfs_scrub_ddt_class_max = DDT_CLASS_DUPLICATE; /* max number of blocks to free in a single TXG */ uint64_t zfs_free_max_blocks = UINT64_MAX; SYSCTL_UQUAD(_vfs_zfs, OID_AUTO, free_max_blocks, CTLFLAG_RWTUN, &zfs_free_max_blocks, 0, "Maximum number of blocks to free in one TXG"); #define DSL_SCAN_IS_SCRUB_RESILVER(scn) \ ((scn)->scn_phys.scn_func == POOL_SCAN_SCRUB || \ (scn)->scn_phys.scn_func == POOL_SCAN_RESILVER) extern int zfs_txg_timeout; /* the order has to match pool_scan_type */ static scan_cb_t *scan_funcs[POOL_SCAN_FUNCS] = { NULL, dsl_scan_scrub_cb, /* POOL_SCAN_SCRUB */ dsl_scan_scrub_cb, /* POOL_SCAN_RESILVER */ }; int dsl_scan_init(dsl_pool_t *dp, uint64_t txg) { int err; dsl_scan_t *scn; spa_t *spa = dp->dp_spa; uint64_t f; scn = dp->dp_scan = kmem_zalloc(sizeof (dsl_scan_t), KM_SLEEP); scn->scn_dp = dp; /* * It's possible that we're resuming a scan after a reboot so * make sure that the scan_async_destroying flag is initialized * appropriately. */ ASSERT(!scn->scn_async_destroying); scn->scn_async_destroying = spa_feature_is_active(dp->dp_spa, SPA_FEATURE_ASYNC_DESTROY); err = zap_lookup(dp->dp_meta_objset, DMU_POOL_DIRECTORY_OBJECT, "scrub_func", sizeof (uint64_t), 1, &f); if (err == 0) { /* * There was an old-style scrub in progress. Restart a * new-style scrub from the beginning. */ scn->scn_restart_txg = txg; zfs_dbgmsg("old-style scrub was in progress; " "restarting new-style scrub in txg %llu", scn->scn_restart_txg); /* * Load the queue obj from the old location so that it * can be freed by dsl_scan_done(). */ (void) zap_lookup(dp->dp_meta_objset, DMU_POOL_DIRECTORY_OBJECT, "scrub_queue", sizeof (uint64_t), 1, &scn->scn_phys.scn_queue_obj); } else { err = zap_lookup(dp->dp_meta_objset, DMU_POOL_DIRECTORY_OBJECT, DMU_POOL_SCAN, sizeof (uint64_t), SCAN_PHYS_NUMINTS, &scn->scn_phys); if (err == ENOENT) return (0); else if (err) return (err); if (scn->scn_phys.scn_state == DSS_SCANNING && spa_prev_software_version(dp->dp_spa) < SPA_VERSION_SCAN) { /* * A new-type scrub was in progress on an old * pool, and the pool was accessed by old * software. Restart from the beginning, since * the old software may have changed the pool in * the meantime. */ scn->scn_restart_txg = txg; zfs_dbgmsg("new-style scrub was modified " "by old software; restarting in txg %llu", scn->scn_restart_txg); } } spa_scan_stat_init(spa); return (0); } void dsl_scan_fini(dsl_pool_t *dp) { if (dp->dp_scan) { kmem_free(dp->dp_scan, sizeof (dsl_scan_t)); dp->dp_scan = NULL; } } /* ARGSUSED */ static int dsl_scan_setup_check(void *arg, dmu_tx_t *tx) { dsl_scan_t *scn = dmu_tx_pool(tx)->dp_scan; if (scn->scn_phys.scn_state == DSS_SCANNING) return (SET_ERROR(EBUSY)); return (0); } static void dsl_scan_setup_sync(void *arg, dmu_tx_t *tx) { dsl_scan_t *scn = dmu_tx_pool(tx)->dp_scan; pool_scan_func_t *funcp = arg; dmu_object_type_t ot = 0; dsl_pool_t *dp = scn->scn_dp; spa_t *spa = dp->dp_spa; ASSERT(scn->scn_phys.scn_state != DSS_SCANNING); ASSERT(*funcp > POOL_SCAN_NONE && *funcp < POOL_SCAN_FUNCS); bzero(&scn->scn_phys, sizeof (scn->scn_phys)); scn->scn_phys.scn_func = *funcp; scn->scn_phys.scn_state = DSS_SCANNING; scn->scn_phys.scn_min_txg = 0; scn->scn_phys.scn_max_txg = tx->tx_txg; scn->scn_phys.scn_ddt_class_max = DDT_CLASSES - 1; /* the entire DDT */ scn->scn_phys.scn_start_time = gethrestime_sec(); scn->scn_phys.scn_errors = 0; scn->scn_phys.scn_to_examine = spa->spa_root_vdev->vdev_stat.vs_alloc; scn->scn_restart_txg = 0; scn->scn_done_txg = 0; spa_scan_stat_init(spa); if (DSL_SCAN_IS_SCRUB_RESILVER(scn)) { scn->scn_phys.scn_ddt_class_max = zfs_scrub_ddt_class_max; /* rewrite all disk labels */ vdev_config_dirty(spa->spa_root_vdev); if (vdev_resilver_needed(spa->spa_root_vdev, &scn->scn_phys.scn_min_txg, &scn->scn_phys.scn_max_txg)) { spa_event_notify(spa, NULL, ESC_ZFS_RESILVER_START); } else { spa_event_notify(spa, NULL, ESC_ZFS_SCRUB_START); } spa->spa_scrub_started = B_TRUE; /* * If this is an incremental scrub, limit the DDT scrub phase * to just the auto-ditto class (for correctness); the rest * of the scrub should go faster using top-down pruning. */ if (scn->scn_phys.scn_min_txg > TXG_INITIAL) scn->scn_phys.scn_ddt_class_max = DDT_CLASS_DITTO; } /* back to the generic stuff */ if (dp->dp_blkstats == NULL) { dp->dp_blkstats = kmem_alloc(sizeof (zfs_all_blkstats_t), KM_SLEEP); } bzero(dp->dp_blkstats, sizeof (zfs_all_blkstats_t)); if (spa_version(spa) < SPA_VERSION_DSL_SCRUB) ot = DMU_OT_ZAP_OTHER; scn->scn_phys.scn_queue_obj = zap_create(dp->dp_meta_objset, ot ? ot : DMU_OT_SCAN_QUEUE, DMU_OT_NONE, 0, tx); dsl_scan_sync_state(scn, tx); spa_history_log_internal(spa, "scan setup", tx, "func=%u mintxg=%llu maxtxg=%llu", *funcp, scn->scn_phys.scn_min_txg, scn->scn_phys.scn_max_txg); } /* ARGSUSED */ static void dsl_scan_done(dsl_scan_t *scn, boolean_t complete, dmu_tx_t *tx) { static const char *old_names[] = { "scrub_bookmark", "scrub_ddt_bookmark", "scrub_ddt_class_max", "scrub_queue", "scrub_min_txg", "scrub_max_txg", "scrub_func", "scrub_errors", NULL }; dsl_pool_t *dp = scn->scn_dp; spa_t *spa = dp->dp_spa; int i; /* Remove any remnants of an old-style scrub. */ for (i = 0; old_names[i]; i++) { (void) zap_remove(dp->dp_meta_objset, DMU_POOL_DIRECTORY_OBJECT, old_names[i], tx); } if (scn->scn_phys.scn_queue_obj != 0) { VERIFY(0 == dmu_object_free(dp->dp_meta_objset, scn->scn_phys.scn_queue_obj, tx)); scn->scn_phys.scn_queue_obj = 0; } /* * If we were "restarted" from a stopped state, don't bother * with anything else. */ if (scn->scn_phys.scn_state != DSS_SCANNING) return; if (complete) scn->scn_phys.scn_state = DSS_FINISHED; else scn->scn_phys.scn_state = DSS_CANCELED; spa_history_log_internal(spa, "scan done", tx, "complete=%u", complete); if (DSL_SCAN_IS_SCRUB_RESILVER(scn)) { mutex_enter(&spa->spa_scrub_lock); while (spa->spa_scrub_inflight > 0) { cv_wait(&spa->spa_scrub_io_cv, &spa->spa_scrub_lock); } mutex_exit(&spa->spa_scrub_lock); spa->spa_scrub_started = B_FALSE; spa->spa_scrub_active = B_FALSE; /* * If the scrub/resilver completed, update all DTLs to * reflect this. Whether it succeeded or not, vacate * all temporary scrub DTLs. */ vdev_dtl_reassess(spa->spa_root_vdev, tx->tx_txg, complete ? scn->scn_phys.scn_max_txg : 0, B_TRUE); if (complete) { spa_event_notify(spa, NULL, scn->scn_phys.scn_min_txg ? ESC_ZFS_RESILVER_FINISH : ESC_ZFS_SCRUB_FINISH); } spa_errlog_rotate(spa); /* * We may have finished replacing a device. * Let the async thread assess this and handle the detach. */ spa_async_request(spa, SPA_ASYNC_RESILVER_DONE); } scn->scn_phys.scn_end_time = gethrestime_sec(); } /* ARGSUSED */ static int dsl_scan_cancel_check(void *arg, dmu_tx_t *tx) { dsl_scan_t *scn = dmu_tx_pool(tx)->dp_scan; if (scn->scn_phys.scn_state != DSS_SCANNING) return (SET_ERROR(ENOENT)); return (0); } /* ARGSUSED */ static void dsl_scan_cancel_sync(void *arg, dmu_tx_t *tx) { dsl_scan_t *scn = dmu_tx_pool(tx)->dp_scan; dsl_scan_done(scn, B_FALSE, tx); dsl_scan_sync_state(scn, tx); } int dsl_scan_cancel(dsl_pool_t *dp) { return (dsl_sync_task(spa_name(dp->dp_spa), dsl_scan_cancel_check, dsl_scan_cancel_sync, NULL, 3, ZFS_SPACE_CHECK_RESERVED)); } static void dsl_scan_visitbp(blkptr_t *bp, const zbookmark_phys_t *zb, dnode_phys_t *dnp, dsl_dataset_t *ds, dsl_scan_t *scn, dmu_objset_type_t ostype, dmu_tx_t *tx); static void dsl_scan_visitdnode(dsl_scan_t *, dsl_dataset_t *ds, dmu_objset_type_t ostype, dnode_phys_t *dnp, uint64_t object, dmu_tx_t *tx); void dsl_free(dsl_pool_t *dp, uint64_t txg, const blkptr_t *bp) { zio_free(dp->dp_spa, txg, bp); } void dsl_free_sync(zio_t *pio, dsl_pool_t *dp, uint64_t txg, const blkptr_t *bpp) { ASSERT(dsl_pool_sync_context(dp)); zio_nowait(zio_free_sync(pio, dp->dp_spa, txg, bpp, BP_GET_PSIZE(bpp), pio->io_flags)); } static uint64_t dsl_scan_ds_maxtxg(dsl_dataset_t *ds) { uint64_t smt = ds->ds_dir->dd_pool->dp_scan->scn_phys.scn_max_txg; if (dsl_dataset_is_snapshot(ds)) return (MIN(smt, ds->ds_phys->ds_creation_txg)); return (smt); } static void dsl_scan_sync_state(dsl_scan_t *scn, dmu_tx_t *tx) { VERIFY0(zap_update(scn->scn_dp->dp_meta_objset, DMU_POOL_DIRECTORY_OBJECT, DMU_POOL_SCAN, sizeof (uint64_t), SCAN_PHYS_NUMINTS, &scn->scn_phys, tx)); } +extern int zfs_vdev_async_write_active_min_dirty_percent; + static boolean_t dsl_scan_check_pause(dsl_scan_t *scn, const zbookmark_phys_t *zb) { - uint64_t elapsed_nanosecs; - unsigned int mintime; - /* we never skip user/group accounting objects */ if (zb && (int64_t)zb->zb_object < 0) return (B_FALSE); if (scn->scn_pausing) return (B_TRUE); /* we're already pausing */ if (!ZB_IS_ZERO(&scn->scn_phys.scn_bookmark)) return (B_FALSE); /* we're resuming */ /* We only know how to resume from level-0 blocks. */ if (zb && zb->zb_level != 0) return (B_FALSE); - mintime = (scn->scn_phys.scn_func == POOL_SCAN_RESILVER) ? + /* + * We pause if: + * - we have scanned for the maximum time: an entire txg + * timeout (default 5 sec) + * or + * - we have scanned for at least the minimum time (default 1 sec + * for scrub, 3 sec for resilver), and either we have sufficient + * dirty data that we are starting to write more quickly + * (default 30%), or someone is explicitly waiting for this txg + * to complete. + * or + * - the spa is shutting down because this pool is being exported + * or the machine is rebooting. + */ + int mintime = (scn->scn_phys.scn_func == POOL_SCAN_RESILVER) ? zfs_resilver_min_time_ms : zfs_scan_min_time_ms; - elapsed_nanosecs = gethrtime() - scn->scn_sync_start_time; - if (elapsed_nanosecs / NANOSEC > zfs_txg_timeout || + uint64_t elapsed_nanosecs = gethrtime() - scn->scn_sync_start_time; + int dirty_pct = scn->scn_dp->dp_dirty_total * 100 / zfs_dirty_data_max; + if (elapsed_nanosecs / NANOSEC >= zfs_txg_timeout || (NSEC2MSEC(elapsed_nanosecs) > mintime && - txg_sync_waiting(scn->scn_dp)) || + (txg_sync_waiting(scn->scn_dp) || + dirty_pct >= zfs_vdev_async_write_active_min_dirty_percent)) || spa_shutting_down(scn->scn_dp->dp_spa)) { if (zb) { dprintf("pausing at bookmark %llx/%llx/%llx/%llx\n", (longlong_t)zb->zb_objset, (longlong_t)zb->zb_object, (longlong_t)zb->zb_level, (longlong_t)zb->zb_blkid); scn->scn_phys.scn_bookmark = *zb; } dprintf("pausing at DDT bookmark %llx/%llx/%llx/%llx\n", (longlong_t)scn->scn_phys.scn_ddt_bookmark.ddb_class, (longlong_t)scn->scn_phys.scn_ddt_bookmark.ddb_type, (longlong_t)scn->scn_phys.scn_ddt_bookmark.ddb_checksum, (longlong_t)scn->scn_phys.scn_ddt_bookmark.ddb_cursor); scn->scn_pausing = B_TRUE; return (B_TRUE); } return (B_FALSE); } typedef struct zil_scan_arg { dsl_pool_t *zsa_dp; zil_header_t *zsa_zh; } zil_scan_arg_t; /* ARGSUSED */ static int dsl_scan_zil_block(zilog_t *zilog, blkptr_t *bp, void *arg, uint64_t claim_txg) { zil_scan_arg_t *zsa = arg; dsl_pool_t *dp = zsa->zsa_dp; dsl_scan_t *scn = dp->dp_scan; zil_header_t *zh = zsa->zsa_zh; zbookmark_phys_t zb; if (BP_IS_HOLE(bp) || bp->blk_birth <= scn->scn_phys.scn_cur_min_txg) return (0); /* * One block ("stubby") can be allocated a long time ago; we * want to visit that one because it has been allocated * (on-disk) even if it hasn't been claimed (even though for * scrub there's nothing to do to it). */ if (claim_txg == 0 && bp->blk_birth >= spa_first_txg(dp->dp_spa)) return (0); SET_BOOKMARK(&zb, zh->zh_log.blk_cksum.zc_word[ZIL_ZC_OBJSET], ZB_ZIL_OBJECT, ZB_ZIL_LEVEL, bp->blk_cksum.zc_word[ZIL_ZC_SEQ]); VERIFY(0 == scan_funcs[scn->scn_phys.scn_func](dp, bp, &zb)); return (0); } /* ARGSUSED */ static int dsl_scan_zil_record(zilog_t *zilog, lr_t *lrc, void *arg, uint64_t claim_txg) { if (lrc->lrc_txtype == TX_WRITE) { zil_scan_arg_t *zsa = arg; dsl_pool_t *dp = zsa->zsa_dp; dsl_scan_t *scn = dp->dp_scan; zil_header_t *zh = zsa->zsa_zh; lr_write_t *lr = (lr_write_t *)lrc; blkptr_t *bp = &lr->lr_blkptr; zbookmark_phys_t zb; if (BP_IS_HOLE(bp) || bp->blk_birth <= scn->scn_phys.scn_cur_min_txg) return (0); /* * birth can be < claim_txg if this record's txg is * already txg sync'ed (but this log block contains * other records that are not synced) */ if (claim_txg == 0 || bp->blk_birth < claim_txg) return (0); SET_BOOKMARK(&zb, zh->zh_log.blk_cksum.zc_word[ZIL_ZC_OBJSET], lr->lr_foid, ZB_ZIL_LEVEL, lr->lr_offset / BP_GET_LSIZE(bp)); VERIFY(0 == scan_funcs[scn->scn_phys.scn_func](dp, bp, &zb)); } return (0); } static void dsl_scan_zil(dsl_pool_t *dp, zil_header_t *zh) { uint64_t claim_txg = zh->zh_claim_txg; zil_scan_arg_t zsa = { dp, zh }; zilog_t *zilog; /* * We only want to visit blocks that have been claimed but not yet * replayed (or, in read-only mode, blocks that *would* be claimed). */ if (claim_txg == 0 && spa_writeable(dp->dp_spa)) return; zilog = zil_alloc(dp->dp_meta_objset, zh); (void) zil_parse(zilog, dsl_scan_zil_block, dsl_scan_zil_record, &zsa, claim_txg); zil_free(zilog); } /* ARGSUSED */ static void dsl_scan_prefetch(dsl_scan_t *scn, arc_buf_t *buf, blkptr_t *bp, uint64_t objset, uint64_t object, uint64_t blkid) { zbookmark_phys_t czb; uint32_t flags = ARC_NOWAIT | ARC_PREFETCH; if (zfs_no_scrub_prefetch) return; if (BP_IS_HOLE(bp) || bp->blk_birth <= scn->scn_phys.scn_min_txg || (BP_GET_LEVEL(bp) == 0 && BP_GET_TYPE(bp) != DMU_OT_DNODE)) return; SET_BOOKMARK(&czb, objset, object, BP_GET_LEVEL(bp), blkid); (void) arc_read(scn->scn_zio_root, scn->scn_dp->dp_spa, bp, NULL, NULL, ZIO_PRIORITY_ASYNC_READ, ZIO_FLAG_CANFAIL | ZIO_FLAG_SCAN_THREAD, &flags, &czb); } static boolean_t dsl_scan_check_resume(dsl_scan_t *scn, const dnode_phys_t *dnp, const zbookmark_phys_t *zb) { /* * We never skip over user/group accounting objects (obj<0) */ if (!ZB_IS_ZERO(&scn->scn_phys.scn_bookmark) && (int64_t)zb->zb_object >= 0) { /* * If we already visited this bp & everything below (in * a prior txg sync), don't bother doing it again. */ if (zbookmark_is_before(dnp, zb, &scn->scn_phys.scn_bookmark)) return (B_TRUE); /* * If we found the block we're trying to resume from, or * we went past it to a different object, zero it out to * indicate that it's OK to start checking for pausing * again. */ if (bcmp(zb, &scn->scn_phys.scn_bookmark, sizeof (*zb)) == 0 || zb->zb_object > scn->scn_phys.scn_bookmark.zb_object) { dprintf("resuming at %llx/%llx/%llx/%llx\n", (longlong_t)zb->zb_objset, (longlong_t)zb->zb_object, (longlong_t)zb->zb_level, (longlong_t)zb->zb_blkid); bzero(&scn->scn_phys.scn_bookmark, sizeof (*zb)); } } return (B_FALSE); } /* * Return nonzero on i/o error. * Return new buf to write out in *bufp. */ static int dsl_scan_recurse(dsl_scan_t *scn, dsl_dataset_t *ds, dmu_objset_type_t ostype, dnode_phys_t *dnp, const blkptr_t *bp, const zbookmark_phys_t *zb, dmu_tx_t *tx) { dsl_pool_t *dp = scn->scn_dp; int zio_flags = ZIO_FLAG_CANFAIL | ZIO_FLAG_SCAN_THREAD; int err; if (BP_GET_LEVEL(bp) > 0) { uint32_t flags = ARC_WAIT; int i; blkptr_t *cbp; int epb = BP_GET_LSIZE(bp) >> SPA_BLKPTRSHIFT; arc_buf_t *buf; err = arc_read(NULL, dp->dp_spa, bp, arc_getbuf_func, &buf, ZIO_PRIORITY_ASYNC_READ, zio_flags, &flags, zb); if (err) { scn->scn_phys.scn_errors++; return (err); } for (i = 0, cbp = buf->b_data; i < epb; i++, cbp++) { dsl_scan_prefetch(scn, buf, cbp, zb->zb_objset, zb->zb_object, zb->zb_blkid * epb + i); } for (i = 0, cbp = buf->b_data; i < epb; i++, cbp++) { zbookmark_phys_t czb; SET_BOOKMARK(&czb, zb->zb_objset, zb->zb_object, zb->zb_level - 1, zb->zb_blkid * epb + i); dsl_scan_visitbp(cbp, &czb, dnp, ds, scn, ostype, tx); } (void) arc_buf_remove_ref(buf, &buf); } else if (BP_GET_TYPE(bp) == DMU_OT_DNODE) { uint32_t flags = ARC_WAIT; dnode_phys_t *cdnp; int i, j; int epb = BP_GET_LSIZE(bp) >> DNODE_SHIFT; arc_buf_t *buf; err = arc_read(NULL, dp->dp_spa, bp, arc_getbuf_func, &buf, ZIO_PRIORITY_ASYNC_READ, zio_flags, &flags, zb); if (err) { scn->scn_phys.scn_errors++; return (err); } for (i = 0, cdnp = buf->b_data; i < epb; i++, cdnp++) { for (j = 0; j < cdnp->dn_nblkptr; j++) { blkptr_t *cbp = &cdnp->dn_blkptr[j]; dsl_scan_prefetch(scn, buf, cbp, zb->zb_objset, zb->zb_blkid * epb + i, j); } } for (i = 0, cdnp = buf->b_data; i < epb; i++, cdnp++) { dsl_scan_visitdnode(scn, ds, ostype, cdnp, zb->zb_blkid * epb + i, tx); } (void) arc_buf_remove_ref(buf, &buf); } else if (BP_GET_TYPE(bp) == DMU_OT_OBJSET) { uint32_t flags = ARC_WAIT; objset_phys_t *osp; arc_buf_t *buf; err = arc_read(NULL, dp->dp_spa, bp, arc_getbuf_func, &buf, ZIO_PRIORITY_ASYNC_READ, zio_flags, &flags, zb); if (err) { scn->scn_phys.scn_errors++; return (err); } osp = buf->b_data; dsl_scan_visitdnode(scn, ds, osp->os_type, &osp->os_meta_dnode, DMU_META_DNODE_OBJECT, tx); if (OBJSET_BUF_HAS_USERUSED(buf)) { /* * We also always visit user/group accounting * objects, and never skip them, even if we are * pausing. This is necessary so that the space * deltas from this txg get integrated. */ dsl_scan_visitdnode(scn, ds, osp->os_type, &osp->os_groupused_dnode, DMU_GROUPUSED_OBJECT, tx); dsl_scan_visitdnode(scn, ds, osp->os_type, &osp->os_userused_dnode, DMU_USERUSED_OBJECT, tx); } (void) arc_buf_remove_ref(buf, &buf); } return (0); } static void dsl_scan_visitdnode(dsl_scan_t *scn, dsl_dataset_t *ds, dmu_objset_type_t ostype, dnode_phys_t *dnp, uint64_t object, dmu_tx_t *tx) { int j; for (j = 0; j < dnp->dn_nblkptr; j++) { zbookmark_phys_t czb; SET_BOOKMARK(&czb, ds ? ds->ds_object : 0, object, dnp->dn_nlevels - 1, j); dsl_scan_visitbp(&dnp->dn_blkptr[j], &czb, dnp, ds, scn, ostype, tx); } if (dnp->dn_flags & DNODE_FLAG_SPILL_BLKPTR) { zbookmark_phys_t czb; SET_BOOKMARK(&czb, ds ? ds->ds_object : 0, object, 0, DMU_SPILL_BLKID); dsl_scan_visitbp(&dnp->dn_spill, &czb, dnp, ds, scn, ostype, tx); } } /* * The arguments are in this order because mdb can only print the * first 5; we want them to be useful. */ static void dsl_scan_visitbp(blkptr_t *bp, const zbookmark_phys_t *zb, dnode_phys_t *dnp, dsl_dataset_t *ds, dsl_scan_t *scn, dmu_objset_type_t ostype, dmu_tx_t *tx) { dsl_pool_t *dp = scn->scn_dp; arc_buf_t *buf = NULL; blkptr_t bp_toread = *bp; /* ASSERT(pbuf == NULL || arc_released(pbuf)); */ if (dsl_scan_check_pause(scn, zb)) return; if (dsl_scan_check_resume(scn, dnp, zb)) return; if (BP_IS_HOLE(bp)) return; scn->scn_visited_this_txg++; dprintf_bp(bp, "visiting ds=%p/%llu zb=%llx/%llx/%llx/%llx bp=%p", ds, ds ? ds->ds_object : 0, zb->zb_objset, zb->zb_object, zb->zb_level, zb->zb_blkid, bp); if (bp->blk_birth <= scn->scn_phys.scn_cur_min_txg) return; if (dsl_scan_recurse(scn, ds, ostype, dnp, &bp_toread, zb, tx) != 0) return; /* * If dsl_scan_ddt() has aready visited this block, it will have * already done any translations or scrubbing, so don't call the * callback again. */ if (ddt_class_contains(dp->dp_spa, scn->scn_phys.scn_ddt_class_max, bp)) { ASSERT(buf == NULL); return; } /* * If this block is from the future (after cur_max_txg), then we * are doing this on behalf of a deleted snapshot, and we will * revisit the future block on the next pass of this dataset. * Don't scan it now unless we need to because something * under it was modified. */ if (BP_PHYSICAL_BIRTH(bp) <= scn->scn_phys.scn_cur_max_txg) { scan_funcs[scn->scn_phys.scn_func](dp, bp, zb); } } static void dsl_scan_visit_rootbp(dsl_scan_t *scn, dsl_dataset_t *ds, blkptr_t *bp, dmu_tx_t *tx) { zbookmark_phys_t zb; SET_BOOKMARK(&zb, ds ? ds->ds_object : DMU_META_OBJSET, ZB_ROOT_OBJECT, ZB_ROOT_LEVEL, ZB_ROOT_BLKID); dsl_scan_visitbp(bp, &zb, NULL, ds, scn, DMU_OST_NONE, tx); dprintf_ds(ds, "finished scan%s", ""); } void dsl_scan_ds_destroyed(dsl_dataset_t *ds, dmu_tx_t *tx) { dsl_pool_t *dp = ds->ds_dir->dd_pool; dsl_scan_t *scn = dp->dp_scan; uint64_t mintxg; if (scn->scn_phys.scn_state != DSS_SCANNING) return; if (scn->scn_phys.scn_bookmark.zb_objset == ds->ds_object) { if (dsl_dataset_is_snapshot(ds)) { /* Note, scn_cur_{min,max}_txg stays the same. */ scn->scn_phys.scn_bookmark.zb_objset = ds->ds_phys->ds_next_snap_obj; zfs_dbgmsg("destroying ds %llu; currently traversing; " "reset zb_objset to %llu", (u_longlong_t)ds->ds_object, (u_longlong_t)ds->ds_phys->ds_next_snap_obj); scn->scn_phys.scn_flags |= DSF_VISIT_DS_AGAIN; } else { SET_BOOKMARK(&scn->scn_phys.scn_bookmark, ZB_DESTROYED_OBJSET, 0, 0, 0); zfs_dbgmsg("destroying ds %llu; currently traversing; " "reset bookmark to -1,0,0,0", (u_longlong_t)ds->ds_object); } } else if (zap_lookup_int_key(dp->dp_meta_objset, scn->scn_phys.scn_queue_obj, ds->ds_object, &mintxg) == 0) { ASSERT3U(ds->ds_phys->ds_num_children, <=, 1); VERIFY3U(0, ==, zap_remove_int(dp->dp_meta_objset, scn->scn_phys.scn_queue_obj, ds->ds_object, tx)); if (dsl_dataset_is_snapshot(ds)) { /* * We keep the same mintxg; it could be > * ds_creation_txg if the previous snapshot was * deleted too. */ VERIFY(zap_add_int_key(dp->dp_meta_objset, scn->scn_phys.scn_queue_obj, ds->ds_phys->ds_next_snap_obj, mintxg, tx) == 0); zfs_dbgmsg("destroying ds %llu; in queue; " "replacing with %llu", (u_longlong_t)ds->ds_object, (u_longlong_t)ds->ds_phys->ds_next_snap_obj); } else { zfs_dbgmsg("destroying ds %llu; in queue; removing", (u_longlong_t)ds->ds_object); } } else { zfs_dbgmsg("destroying ds %llu; ignoring", (u_longlong_t)ds->ds_object); } /* * dsl_scan_sync() should be called after this, and should sync * out our changed state, but just to be safe, do it here. */ dsl_scan_sync_state(scn, tx); } void dsl_scan_ds_snapshotted(dsl_dataset_t *ds, dmu_tx_t *tx) { dsl_pool_t *dp = ds->ds_dir->dd_pool; dsl_scan_t *scn = dp->dp_scan; uint64_t mintxg; if (scn->scn_phys.scn_state != DSS_SCANNING) return; ASSERT(ds->ds_phys->ds_prev_snap_obj != 0); if (scn->scn_phys.scn_bookmark.zb_objset == ds->ds_object) { scn->scn_phys.scn_bookmark.zb_objset = ds->ds_phys->ds_prev_snap_obj; zfs_dbgmsg("snapshotting ds %llu; currently traversing; " "reset zb_objset to %llu", (u_longlong_t)ds->ds_object, (u_longlong_t)ds->ds_phys->ds_prev_snap_obj); } else if (zap_lookup_int_key(dp->dp_meta_objset, scn->scn_phys.scn_queue_obj, ds->ds_object, &mintxg) == 0) { VERIFY3U(0, ==, zap_remove_int(dp->dp_meta_objset, scn->scn_phys.scn_queue_obj, ds->ds_object, tx)); VERIFY(zap_add_int_key(dp->dp_meta_objset, scn->scn_phys.scn_queue_obj, ds->ds_phys->ds_prev_snap_obj, mintxg, tx) == 0); zfs_dbgmsg("snapshotting ds %llu; in queue; " "replacing with %llu", (u_longlong_t)ds->ds_object, (u_longlong_t)ds->ds_phys->ds_prev_snap_obj); } dsl_scan_sync_state(scn, tx); } void dsl_scan_ds_clone_swapped(dsl_dataset_t *ds1, dsl_dataset_t *ds2, dmu_tx_t *tx) { dsl_pool_t *dp = ds1->ds_dir->dd_pool; dsl_scan_t *scn = dp->dp_scan; uint64_t mintxg; if (scn->scn_phys.scn_state != DSS_SCANNING) return; if (scn->scn_phys.scn_bookmark.zb_objset == ds1->ds_object) { scn->scn_phys.scn_bookmark.zb_objset = ds2->ds_object; zfs_dbgmsg("clone_swap ds %llu; currently traversing; " "reset zb_objset to %llu", (u_longlong_t)ds1->ds_object, (u_longlong_t)ds2->ds_object); } else if (scn->scn_phys.scn_bookmark.zb_objset == ds2->ds_object) { scn->scn_phys.scn_bookmark.zb_objset = ds1->ds_object; zfs_dbgmsg("clone_swap ds %llu; currently traversing; " "reset zb_objset to %llu", (u_longlong_t)ds2->ds_object, (u_longlong_t)ds1->ds_object); } if (zap_lookup_int_key(dp->dp_meta_objset, scn->scn_phys.scn_queue_obj, ds1->ds_object, &mintxg) == 0) { int err; ASSERT3U(mintxg, ==, ds1->ds_phys->ds_prev_snap_txg); ASSERT3U(mintxg, ==, ds2->ds_phys->ds_prev_snap_txg); VERIFY3U(0, ==, zap_remove_int(dp->dp_meta_objset, scn->scn_phys.scn_queue_obj, ds1->ds_object, tx)); err = zap_add_int_key(dp->dp_meta_objset, scn->scn_phys.scn_queue_obj, ds2->ds_object, mintxg, tx); VERIFY(err == 0 || err == EEXIST); if (err == EEXIST) { /* Both were there to begin with */ VERIFY(0 == zap_add_int_key(dp->dp_meta_objset, scn->scn_phys.scn_queue_obj, ds1->ds_object, mintxg, tx)); } zfs_dbgmsg("clone_swap ds %llu; in queue; " "replacing with %llu", (u_longlong_t)ds1->ds_object, (u_longlong_t)ds2->ds_object); } else if (zap_lookup_int_key(dp->dp_meta_objset, scn->scn_phys.scn_queue_obj, ds2->ds_object, &mintxg) == 0) { ASSERT3U(mintxg, ==, ds1->ds_phys->ds_prev_snap_txg); ASSERT3U(mintxg, ==, ds2->ds_phys->ds_prev_snap_txg); VERIFY3U(0, ==, zap_remove_int(dp->dp_meta_objset, scn->scn_phys.scn_queue_obj, ds2->ds_object, tx)); VERIFY(0 == zap_add_int_key(dp->dp_meta_objset, scn->scn_phys.scn_queue_obj, ds1->ds_object, mintxg, tx)); zfs_dbgmsg("clone_swap ds %llu; in queue; " "replacing with %llu", (u_longlong_t)ds2->ds_object, (u_longlong_t)ds1->ds_object); } dsl_scan_sync_state(scn, tx); } struct enqueue_clones_arg { dmu_tx_t *tx; uint64_t originobj; }; /* ARGSUSED */ static int enqueue_clones_cb(dsl_pool_t *dp, dsl_dataset_t *hds, void *arg) { struct enqueue_clones_arg *eca = arg; dsl_dataset_t *ds; int err; dsl_scan_t *scn = dp->dp_scan; if (hds->ds_dir->dd_phys->dd_origin_obj != eca->originobj) return (0); err = dsl_dataset_hold_obj(dp, hds->ds_object, FTAG, &ds); if (err) return (err); while (ds->ds_phys->ds_prev_snap_obj != eca->originobj) { dsl_dataset_t *prev; err = dsl_dataset_hold_obj(dp, ds->ds_phys->ds_prev_snap_obj, FTAG, &prev); dsl_dataset_rele(ds, FTAG); if (err) return (err); ds = prev; } VERIFY(zap_add_int_key(dp->dp_meta_objset, scn->scn_phys.scn_queue_obj, ds->ds_object, ds->ds_phys->ds_prev_snap_txg, eca->tx) == 0); dsl_dataset_rele(ds, FTAG); return (0); } static void dsl_scan_visitds(dsl_scan_t *scn, uint64_t dsobj, dmu_tx_t *tx) { dsl_pool_t *dp = scn->scn_dp; dsl_dataset_t *ds; objset_t *os; VERIFY3U(0, ==, dsl_dataset_hold_obj(dp, dsobj, FTAG, &ds)); if (dmu_objset_from_ds(ds, &os)) goto out; /* * Only the ZIL in the head (non-snapshot) is valid. Even though * snapshots can have ZIL block pointers (which may be the same * BP as in the head), they must be ignored. So we traverse the * ZIL here, rather than in scan_recurse(), because the regular * snapshot block-sharing rules don't apply to it. */ if (DSL_SCAN_IS_SCRUB_RESILVER(scn) && !dsl_dataset_is_snapshot(ds)) dsl_scan_zil(dp, &os->os_zil_header); /* * Iterate over the bps in this ds. */ dmu_buf_will_dirty(ds->ds_dbuf, tx); dsl_scan_visit_rootbp(scn, ds, &ds->ds_phys->ds_bp, tx); char *dsname = kmem_alloc(ZFS_MAXNAMELEN, KM_SLEEP); dsl_dataset_name(ds, dsname); zfs_dbgmsg("scanned dataset %llu (%s) with min=%llu max=%llu; " "pausing=%u", (longlong_t)dsobj, dsname, (longlong_t)scn->scn_phys.scn_cur_min_txg, (longlong_t)scn->scn_phys.scn_cur_max_txg, (int)scn->scn_pausing); kmem_free(dsname, ZFS_MAXNAMELEN); if (scn->scn_pausing) goto out; /* * We've finished this pass over this dataset. */ /* * If we did not completely visit this dataset, do another pass. */ if (scn->scn_phys.scn_flags & DSF_VISIT_DS_AGAIN) { zfs_dbgmsg("incomplete pass; visiting again"); scn->scn_phys.scn_flags &= ~DSF_VISIT_DS_AGAIN; VERIFY(zap_add_int_key(dp->dp_meta_objset, scn->scn_phys.scn_queue_obj, ds->ds_object, scn->scn_phys.scn_cur_max_txg, tx) == 0); goto out; } /* * Add descendent datasets to work queue. */ if (ds->ds_phys->ds_next_snap_obj != 0) { VERIFY(zap_add_int_key(dp->dp_meta_objset, scn->scn_phys.scn_queue_obj, ds->ds_phys->ds_next_snap_obj, ds->ds_phys->ds_creation_txg, tx) == 0); } if (ds->ds_phys->ds_num_children > 1) { boolean_t usenext = B_FALSE; if (ds->ds_phys->ds_next_clones_obj != 0) { uint64_t count; /* * A bug in a previous version of the code could * cause upgrade_clones_cb() to not set * ds_next_snap_obj when it should, leading to a * missing entry. Therefore we can only use the * next_clones_obj when its count is correct. */ int err = zap_count(dp->dp_meta_objset, ds->ds_phys->ds_next_clones_obj, &count); if (err == 0 && count == ds->ds_phys->ds_num_children - 1) usenext = B_TRUE; } if (usenext) { VERIFY0(zap_join_key(dp->dp_meta_objset, ds->ds_phys->ds_next_clones_obj, scn->scn_phys.scn_queue_obj, ds->ds_phys->ds_creation_txg, tx)); } else { struct enqueue_clones_arg eca; eca.tx = tx; eca.originobj = ds->ds_object; VERIFY0(dmu_objset_find_dp(dp, dp->dp_root_dir_obj, enqueue_clones_cb, &eca, DS_FIND_CHILDREN)); } } out: dsl_dataset_rele(ds, FTAG); } /* ARGSUSED */ static int enqueue_cb(dsl_pool_t *dp, dsl_dataset_t *hds, void *arg) { dmu_tx_t *tx = arg; dsl_dataset_t *ds; int err; dsl_scan_t *scn = dp->dp_scan; err = dsl_dataset_hold_obj(dp, hds->ds_object, FTAG, &ds); if (err) return (err); while (ds->ds_phys->ds_prev_snap_obj != 0) { dsl_dataset_t *prev; err = dsl_dataset_hold_obj(dp, ds->ds_phys->ds_prev_snap_obj, FTAG, &prev); if (err) { dsl_dataset_rele(ds, FTAG); return (err); } /* * If this is a clone, we don't need to worry about it for now. */ if (prev->ds_phys->ds_next_snap_obj != ds->ds_object) { dsl_dataset_rele(ds, FTAG); dsl_dataset_rele(prev, FTAG); return (0); } dsl_dataset_rele(ds, FTAG); ds = prev; } VERIFY(zap_add_int_key(dp->dp_meta_objset, scn->scn_phys.scn_queue_obj, ds->ds_object, ds->ds_phys->ds_prev_snap_txg, tx) == 0); dsl_dataset_rele(ds, FTAG); return (0); } /* * Scrub/dedup interaction. * * If there are N references to a deduped block, we don't want to scrub it * N times -- ideally, we should scrub it exactly once. * * We leverage the fact that the dde's replication class (enum ddt_class) * is ordered from highest replication class (DDT_CLASS_DITTO) to lowest * (DDT_CLASS_UNIQUE) so that we may walk the DDT in that order. * * To prevent excess scrubbing, the scrub begins by walking the DDT * to find all blocks with refcnt > 1, and scrubs each of these once. * Since there are two replication classes which contain blocks with * refcnt > 1, we scrub the highest replication class (DDT_CLASS_DITTO) first. * Finally the top-down scrub begins, only visiting blocks with refcnt == 1. * * There would be nothing more to say if a block's refcnt couldn't change * during a scrub, but of course it can so we must account for changes * in a block's replication class. * * Here's an example of what can occur: * * If a block has refcnt > 1 during the DDT scrub phase, but has refcnt == 1 * when visited during the top-down scrub phase, it will be scrubbed twice. * This negates our scrub optimization, but is otherwise harmless. * * If a block has refcnt == 1 during the DDT scrub phase, but has refcnt > 1 * on each visit during the top-down scrub phase, it will never be scrubbed. * To catch this, ddt_sync_entry() notifies the scrub code whenever a block's * reference class transitions to a higher level (i.e DDT_CLASS_UNIQUE to * DDT_CLASS_DUPLICATE); if it transitions from refcnt == 1 to refcnt > 1 * while a scrub is in progress, it scrubs the block right then. */ static void dsl_scan_ddt(dsl_scan_t *scn, dmu_tx_t *tx) { ddt_bookmark_t *ddb = &scn->scn_phys.scn_ddt_bookmark; ddt_entry_t dde = { 0 }; int error; uint64_t n = 0; while ((error = ddt_walk(scn->scn_dp->dp_spa, ddb, &dde)) == 0) { ddt_t *ddt; if (ddb->ddb_class > scn->scn_phys.scn_ddt_class_max) break; dprintf("visiting ddb=%llu/%llu/%llu/%llx\n", (longlong_t)ddb->ddb_class, (longlong_t)ddb->ddb_type, (longlong_t)ddb->ddb_checksum, (longlong_t)ddb->ddb_cursor); /* There should be no pending changes to the dedup table */ ddt = scn->scn_dp->dp_spa->spa_ddt[ddb->ddb_checksum]; ASSERT(avl_first(&ddt->ddt_tree) == NULL); dsl_scan_ddt_entry(scn, ddb->ddb_checksum, &dde, tx); n++; if (dsl_scan_check_pause(scn, NULL)) break; } zfs_dbgmsg("scanned %llu ddt entries with class_max = %u; pausing=%u", (longlong_t)n, (int)scn->scn_phys.scn_ddt_class_max, (int)scn->scn_pausing); ASSERT(error == 0 || error == ENOENT); ASSERT(error != ENOENT || ddb->ddb_class > scn->scn_phys.scn_ddt_class_max); } /* ARGSUSED */ void dsl_scan_ddt_entry(dsl_scan_t *scn, enum zio_checksum checksum, ddt_entry_t *dde, dmu_tx_t *tx) { const ddt_key_t *ddk = &dde->dde_key; ddt_phys_t *ddp = dde->dde_phys; blkptr_t bp; zbookmark_phys_t zb = { 0 }; if (scn->scn_phys.scn_state != DSS_SCANNING) return; for (int p = 0; p < DDT_PHYS_TYPES; p++, ddp++) { if (ddp->ddp_phys_birth == 0 || ddp->ddp_phys_birth > scn->scn_phys.scn_max_txg) continue; ddt_bp_create(checksum, ddk, ddp, &bp); scn->scn_visited_this_txg++; scan_funcs[scn->scn_phys.scn_func](scn->scn_dp, &bp, &zb); } } static void dsl_scan_visit(dsl_scan_t *scn, dmu_tx_t *tx) { dsl_pool_t *dp = scn->scn_dp; zap_cursor_t zc; zap_attribute_t za; if (scn->scn_phys.scn_ddt_bookmark.ddb_class <= scn->scn_phys.scn_ddt_class_max) { scn->scn_phys.scn_cur_min_txg = scn->scn_phys.scn_min_txg; scn->scn_phys.scn_cur_max_txg = scn->scn_phys.scn_max_txg; dsl_scan_ddt(scn, tx); if (scn->scn_pausing) return; } if (scn->scn_phys.scn_bookmark.zb_objset == DMU_META_OBJSET) { /* First do the MOS & ORIGIN */ scn->scn_phys.scn_cur_min_txg = scn->scn_phys.scn_min_txg; scn->scn_phys.scn_cur_max_txg = scn->scn_phys.scn_max_txg; dsl_scan_visit_rootbp(scn, NULL, &dp->dp_meta_rootbp, tx); spa_set_rootblkptr(dp->dp_spa, &dp->dp_meta_rootbp); if (scn->scn_pausing) return; if (spa_version(dp->dp_spa) < SPA_VERSION_DSL_SCRUB) { VERIFY0(dmu_objset_find_dp(dp, dp->dp_root_dir_obj, enqueue_cb, tx, DS_FIND_CHILDREN)); } else { dsl_scan_visitds(scn, dp->dp_origin_snap->ds_object, tx); } ASSERT(!scn->scn_pausing); } else if (scn->scn_phys.scn_bookmark.zb_objset != ZB_DESTROYED_OBJSET) { /* * If we were paused, continue from here. Note if the * ds we were paused on was deleted, the zb_objset may * be -1, so we will skip this and find a new objset * below. */ dsl_scan_visitds(scn, scn->scn_phys.scn_bookmark.zb_objset, tx); if (scn->scn_pausing) return; } /* * In case we were paused right at the end of the ds, zero the * bookmark so we don't think that we're still trying to resume. */ bzero(&scn->scn_phys.scn_bookmark, sizeof (zbookmark_phys_t)); /* keep pulling things out of the zap-object-as-queue */ while (zap_cursor_init(&zc, dp->dp_meta_objset, scn->scn_phys.scn_queue_obj), zap_cursor_retrieve(&zc, &za) == 0) { dsl_dataset_t *ds; uint64_t dsobj; dsobj = strtonum(za.za_name, NULL); VERIFY3U(0, ==, zap_remove_int(dp->dp_meta_objset, scn->scn_phys.scn_queue_obj, dsobj, tx)); /* Set up min/max txg */ VERIFY3U(0, ==, dsl_dataset_hold_obj(dp, dsobj, FTAG, &ds)); if (za.za_first_integer != 0) { scn->scn_phys.scn_cur_min_txg = MAX(scn->scn_phys.scn_min_txg, za.za_first_integer); } else { scn->scn_phys.scn_cur_min_txg = MAX(scn->scn_phys.scn_min_txg, ds->ds_phys->ds_prev_snap_txg); } scn->scn_phys.scn_cur_max_txg = dsl_scan_ds_maxtxg(ds); dsl_dataset_rele(ds, FTAG); dsl_scan_visitds(scn, dsobj, tx); zap_cursor_fini(&zc); if (scn->scn_pausing) return; } zap_cursor_fini(&zc); } static boolean_t dsl_scan_free_should_pause(dsl_scan_t *scn) { uint64_t elapsed_nanosecs; if (zfs_recover) return (B_FALSE); if (scn->scn_visited_this_txg >= zfs_free_max_blocks) return (B_TRUE); elapsed_nanosecs = gethrtime() - scn->scn_sync_start_time; return (elapsed_nanosecs / NANOSEC > zfs_txg_timeout || (NSEC2MSEC(elapsed_nanosecs) > zfs_free_min_time_ms && txg_sync_waiting(scn->scn_dp)) || spa_shutting_down(scn->scn_dp->dp_spa)); } static int dsl_scan_free_block_cb(void *arg, const blkptr_t *bp, dmu_tx_t *tx) { dsl_scan_t *scn = arg; if (!scn->scn_is_bptree || (BP_GET_LEVEL(bp) == 0 && BP_GET_TYPE(bp) != DMU_OT_OBJSET)) { if (dsl_scan_free_should_pause(scn)) return (SET_ERROR(ERESTART)); } zio_nowait(zio_free_sync(scn->scn_zio_root, scn->scn_dp->dp_spa, dmu_tx_get_txg(tx), bp, BP_GET_PSIZE(bp), 0)); dsl_dir_diduse_space(tx->tx_pool->dp_free_dir, DD_USED_HEAD, -bp_get_dsize_sync(scn->scn_dp->dp_spa, bp), -BP_GET_PSIZE(bp), -BP_GET_UCSIZE(bp), tx); scn->scn_visited_this_txg++; return (0); } boolean_t dsl_scan_active(dsl_scan_t *scn) { spa_t *spa = scn->scn_dp->dp_spa; uint64_t used = 0, comp, uncomp; if (spa->spa_load_state != SPA_LOAD_NONE) return (B_FALSE); if (spa_shutting_down(spa)) return (B_FALSE); if (scn->scn_phys.scn_state == DSS_SCANNING || (scn->scn_async_destroying && !scn->scn_async_stalled)) return (B_TRUE); if (spa_version(scn->scn_dp->dp_spa) >= SPA_VERSION_DEADLISTS) { (void) bpobj_space(&scn->scn_dp->dp_free_bpobj, &used, &comp, &uncomp); } return (used != 0); } void dsl_scan_sync(dsl_pool_t *dp, dmu_tx_t *tx) { dsl_scan_t *scn = dp->dp_scan; spa_t *spa = dp->dp_spa; int err = 0; /* * Check for scn_restart_txg before checking spa_load_state, so * that we can restart an old-style scan while the pool is being * imported (see dsl_scan_init). */ if (scn->scn_restart_txg != 0 && scn->scn_restart_txg <= tx->tx_txg) { pool_scan_func_t func = POOL_SCAN_SCRUB; dsl_scan_done(scn, B_FALSE, tx); if (vdev_resilver_needed(spa->spa_root_vdev, NULL, NULL)) func = POOL_SCAN_RESILVER; zfs_dbgmsg("restarting scan func=%u txg=%llu", func, tx->tx_txg); dsl_scan_setup_sync(&func, tx); } /* * If the scan is inactive due to a stalled async destroy, try again. */ if ((!scn->scn_async_stalled && !dsl_scan_active(scn)) || spa_sync_pass(dp->dp_spa) > 1) return; scn->scn_visited_this_txg = 0; scn->scn_pausing = B_FALSE; scn->scn_sync_start_time = gethrtime(); spa->spa_scrub_active = B_TRUE; /* * First process the async destroys. If we pause, don't do * any scrubbing or resilvering. This ensures that there are no * async destroys while we are scanning, so the scan code doesn't * have to worry about traversing it. It is also faster to free the * blocks than to scrub them. */ if (spa_version(dp->dp_spa) >= SPA_VERSION_DEADLISTS) { scn->scn_is_bptree = B_FALSE; scn->scn_zio_root = zio_root(dp->dp_spa, NULL, NULL, ZIO_FLAG_MUSTSUCCEED); err = bpobj_iterate(&dp->dp_free_bpobj, dsl_scan_free_block_cb, scn, tx); VERIFY3U(0, ==, zio_wait(scn->scn_zio_root)); if (err != 0 && err != ERESTART) zfs_panic_recover("error %u from bpobj_iterate()", err); } if (err == 0 && spa_feature_is_active(spa, SPA_FEATURE_ASYNC_DESTROY)) { ASSERT(scn->scn_async_destroying); scn->scn_is_bptree = B_TRUE; scn->scn_zio_root = zio_root(dp->dp_spa, NULL, NULL, ZIO_FLAG_MUSTSUCCEED); err = bptree_iterate(dp->dp_meta_objset, dp->dp_bptree_obj, B_TRUE, dsl_scan_free_block_cb, scn, tx); VERIFY0(zio_wait(scn->scn_zio_root)); if (err == EIO || err == ECKSUM) { err = 0; } else if (err != 0 && err != ERESTART) { zfs_panic_recover("error %u from " "traverse_dataset_destroyed()", err); } if (bptree_is_empty(dp->dp_meta_objset, dp->dp_bptree_obj)) { /* finished; deactivate async destroy feature */ spa_feature_decr(spa, SPA_FEATURE_ASYNC_DESTROY, tx); ASSERT(!spa_feature_is_active(spa, SPA_FEATURE_ASYNC_DESTROY)); VERIFY0(zap_remove(dp->dp_meta_objset, DMU_POOL_DIRECTORY_OBJECT, DMU_POOL_BPTREE_OBJ, tx)); VERIFY0(bptree_free(dp->dp_meta_objset, dp->dp_bptree_obj, tx)); dp->dp_bptree_obj = 0; scn->scn_async_destroying = B_FALSE; } else { /* * If we didn't make progress, mark the async destroy as * stalled, so that we will not initiate a spa_sync() on * its behalf. */ scn->scn_async_stalled = (scn->scn_visited_this_txg == 0); } } if (scn->scn_visited_this_txg) { zfs_dbgmsg("freed %llu blocks in %llums from " "free_bpobj/bptree txg %llu; err=%d", (longlong_t)scn->scn_visited_this_txg, (longlong_t) NSEC2MSEC(gethrtime() - scn->scn_sync_start_time), (longlong_t)tx->tx_txg, err); scn->scn_visited_this_txg = 0; /* * Write out changes to the DDT that may be required as a * result of the blocks freed. This ensures that the DDT * is clean when a scrub/resilver runs. */ ddt_sync(spa, tx->tx_txg); } if (err != 0) return; if (!scn->scn_async_destroying && zfs_free_leak_on_eio && (dp->dp_free_dir->dd_phys->dd_used_bytes != 0 || dp->dp_free_dir->dd_phys->dd_compressed_bytes != 0 || dp->dp_free_dir->dd_phys->dd_uncompressed_bytes != 0)) { /* * We have finished background destroying, but there is still * some space left in the dp_free_dir. Transfer this leaked * space to the dp_leak_dir. */ if (dp->dp_leak_dir == NULL) { rrw_enter(&dp->dp_config_rwlock, RW_WRITER, FTAG); (void) dsl_dir_create_sync(dp, dp->dp_root_dir, LEAK_DIR_NAME, tx); VERIFY0(dsl_pool_open_special_dir(dp, LEAK_DIR_NAME, &dp->dp_leak_dir)); rrw_exit(&dp->dp_config_rwlock, FTAG); } dsl_dir_diduse_space(dp->dp_leak_dir, DD_USED_HEAD, dp->dp_free_dir->dd_phys->dd_used_bytes, dp->dp_free_dir->dd_phys->dd_compressed_bytes, dp->dp_free_dir->dd_phys->dd_uncompressed_bytes, tx); dsl_dir_diduse_space(dp->dp_free_dir, DD_USED_HEAD, -dp->dp_free_dir->dd_phys->dd_used_bytes, -dp->dp_free_dir->dd_phys->dd_compressed_bytes, -dp->dp_free_dir->dd_phys->dd_uncompressed_bytes, tx); } if (!scn->scn_async_destroying) { /* finished; verify that space accounting went to zero */ ASSERT0(dp->dp_free_dir->dd_phys->dd_used_bytes); ASSERT0(dp->dp_free_dir->dd_phys->dd_compressed_bytes); ASSERT0(dp->dp_free_dir->dd_phys->dd_uncompressed_bytes); } if (scn->scn_phys.scn_state != DSS_SCANNING) return; if (scn->scn_done_txg == tx->tx_txg) { ASSERT(!scn->scn_pausing); /* finished with scan. */ zfs_dbgmsg("txg %llu scan complete", tx->tx_txg); dsl_scan_done(scn, B_TRUE, tx); ASSERT3U(spa->spa_scrub_inflight, ==, 0); dsl_scan_sync_state(scn, tx); return; } if (scn->scn_phys.scn_ddt_bookmark.ddb_class <= scn->scn_phys.scn_ddt_class_max) { zfs_dbgmsg("doing scan sync txg %llu; " "ddt bm=%llu/%llu/%llu/%llx", (longlong_t)tx->tx_txg, (longlong_t)scn->scn_phys.scn_ddt_bookmark.ddb_class, (longlong_t)scn->scn_phys.scn_ddt_bookmark.ddb_type, (longlong_t)scn->scn_phys.scn_ddt_bookmark.ddb_checksum, (longlong_t)scn->scn_phys.scn_ddt_bookmark.ddb_cursor); ASSERT(scn->scn_phys.scn_bookmark.zb_objset == 0); ASSERT(scn->scn_phys.scn_bookmark.zb_object == 0); ASSERT(scn->scn_phys.scn_bookmark.zb_level == 0); ASSERT(scn->scn_phys.scn_bookmark.zb_blkid == 0); } else { zfs_dbgmsg("doing scan sync txg %llu; bm=%llu/%llu/%llu/%llu", (longlong_t)tx->tx_txg, (longlong_t)scn->scn_phys.scn_bookmark.zb_objset, (longlong_t)scn->scn_phys.scn_bookmark.zb_object, (longlong_t)scn->scn_phys.scn_bookmark.zb_level, (longlong_t)scn->scn_phys.scn_bookmark.zb_blkid); } scn->scn_zio_root = zio_root(dp->dp_spa, NULL, NULL, ZIO_FLAG_CANFAIL); dsl_pool_config_enter(dp, FTAG); dsl_scan_visit(scn, tx); dsl_pool_config_exit(dp, FTAG); (void) zio_wait(scn->scn_zio_root); scn->scn_zio_root = NULL; zfs_dbgmsg("visited %llu blocks in %llums", (longlong_t)scn->scn_visited_this_txg, (longlong_t)NSEC2MSEC(gethrtime() - scn->scn_sync_start_time)); if (!scn->scn_pausing) { scn->scn_done_txg = tx->tx_txg + 1; zfs_dbgmsg("txg %llu traversal complete, waiting till txg %llu", tx->tx_txg, scn->scn_done_txg); } if (DSL_SCAN_IS_SCRUB_RESILVER(scn)) { mutex_enter(&spa->spa_scrub_lock); while (spa->spa_scrub_inflight > 0) { cv_wait(&spa->spa_scrub_io_cv, &spa->spa_scrub_lock); } mutex_exit(&spa->spa_scrub_lock); } dsl_scan_sync_state(scn, tx); } /* * This will start a new scan, or restart an existing one. */ void dsl_resilver_restart(dsl_pool_t *dp, uint64_t txg) { if (txg == 0) { dmu_tx_t *tx; tx = dmu_tx_create_dd(dp->dp_mos_dir); VERIFY(0 == dmu_tx_assign(tx, TXG_WAIT)); txg = dmu_tx_get_txg(tx); dp->dp_scan->scn_restart_txg = txg; dmu_tx_commit(tx); } else { dp->dp_scan->scn_restart_txg = txg; } zfs_dbgmsg("restarting resilver txg=%llu", txg); } boolean_t dsl_scan_resilvering(dsl_pool_t *dp) { return (dp->dp_scan->scn_phys.scn_state == DSS_SCANNING && dp->dp_scan->scn_phys.scn_func == POOL_SCAN_RESILVER); } /* * scrub consumers */ static void count_block(zfs_all_blkstats_t *zab, const blkptr_t *bp) { int i; /* * If we resume after a reboot, zab will be NULL; don't record * incomplete stats in that case. */ if (zab == NULL) return; for (i = 0; i < 4; i++) { int l = (i < 2) ? BP_GET_LEVEL(bp) : DN_MAX_LEVELS; int t = (i & 1) ? BP_GET_TYPE(bp) : DMU_OT_TOTAL; if (t & DMU_OT_NEWTYPE) t = DMU_OT_OTHER; zfs_blkstat_t *zb = &zab->zab_type[l][t]; int equal; zb->zb_count++; zb->zb_asize += BP_GET_ASIZE(bp); zb->zb_lsize += BP_GET_LSIZE(bp); zb->zb_psize += BP_GET_PSIZE(bp); zb->zb_gangs += BP_COUNT_GANG(bp); switch (BP_GET_NDVAS(bp)) { case 2: if (DVA_GET_VDEV(&bp->blk_dva[0]) == DVA_GET_VDEV(&bp->blk_dva[1])) zb->zb_ditto_2_of_2_samevdev++; break; case 3: equal = (DVA_GET_VDEV(&bp->blk_dva[0]) == DVA_GET_VDEV(&bp->blk_dva[1])) + (DVA_GET_VDEV(&bp->blk_dva[0]) == DVA_GET_VDEV(&bp->blk_dva[2])) + (DVA_GET_VDEV(&bp->blk_dva[1]) == DVA_GET_VDEV(&bp->blk_dva[2])); if (equal == 1) zb->zb_ditto_2_of_3_samevdev++; else if (equal == 3) zb->zb_ditto_3_of_3_samevdev++; break; } } } static void dsl_scan_scrub_done(zio_t *zio) { spa_t *spa = zio->io_spa; zio_data_buf_free(zio->io_data, zio->io_size); mutex_enter(&spa->spa_scrub_lock); spa->spa_scrub_inflight--; cv_broadcast(&spa->spa_scrub_io_cv); if (zio->io_error && (zio->io_error != ECKSUM || !(zio->io_flags & ZIO_FLAG_SPECULATIVE))) { spa->spa_dsl_pool->dp_scan->scn_phys.scn_errors++; } mutex_exit(&spa->spa_scrub_lock); } static int dsl_scan_scrub_cb(dsl_pool_t *dp, const blkptr_t *bp, const zbookmark_phys_t *zb) { dsl_scan_t *scn = dp->dp_scan; size_t size = BP_GET_PSIZE(bp); spa_t *spa = dp->dp_spa; uint64_t phys_birth = BP_PHYSICAL_BIRTH(bp); boolean_t needs_io; int zio_flags = ZIO_FLAG_SCAN_THREAD | ZIO_FLAG_RAW | ZIO_FLAG_CANFAIL; unsigned int scan_delay = 0; if (phys_birth <= scn->scn_phys.scn_min_txg || phys_birth >= scn->scn_phys.scn_max_txg) return (0); count_block(dp->dp_blkstats, bp); if (BP_IS_EMBEDDED(bp)) return (0); ASSERT(DSL_SCAN_IS_SCRUB_RESILVER(scn)); if (scn->scn_phys.scn_func == POOL_SCAN_SCRUB) { zio_flags |= ZIO_FLAG_SCRUB; needs_io = B_TRUE; scan_delay = zfs_scrub_delay; } else { ASSERT3U(scn->scn_phys.scn_func, ==, POOL_SCAN_RESILVER); zio_flags |= ZIO_FLAG_RESILVER; needs_io = B_FALSE; scan_delay = zfs_resilver_delay; } /* If it's an intent log block, failure is expected. */ if (zb->zb_level == ZB_ZIL_LEVEL) zio_flags |= ZIO_FLAG_SPECULATIVE; for (int d = 0; d < BP_GET_NDVAS(bp); d++) { vdev_t *vd = vdev_lookup_top(spa, DVA_GET_VDEV(&bp->blk_dva[d])); /* * Keep track of how much data we've examined so that * zpool(1M) status can make useful progress reports. */ scn->scn_phys.scn_examined += DVA_GET_ASIZE(&bp->blk_dva[d]); spa->spa_scan_pass_exam += DVA_GET_ASIZE(&bp->blk_dva[d]); /* if it's a resilver, this may not be in the target range */ if (!needs_io) { if (DVA_GET_GANG(&bp->blk_dva[d])) { /* * Gang members may be spread across multiple * vdevs, so the best estimate we have is the * scrub range, which has already been checked. * XXX -- it would be better to change our * allocation policy to ensure that all * gang members reside on the same vdev. */ needs_io = B_TRUE; } else { needs_io = vdev_dtl_contains(vd, DTL_PARTIAL, phys_birth, 1); } } } if (needs_io && !zfs_no_scrub_io) { vdev_t *rvd = spa->spa_root_vdev; uint64_t maxinflight = rvd->vdev_children * MAX(zfs_top_maxinflight, 1); void *data = zio_data_buf_alloc(size); mutex_enter(&spa->spa_scrub_lock); while (spa->spa_scrub_inflight >= maxinflight) cv_wait(&spa->spa_scrub_io_cv, &spa->spa_scrub_lock); spa->spa_scrub_inflight++; mutex_exit(&spa->spa_scrub_lock); /* * If we're seeing recent (zfs_scan_idle) "important" I/Os * then throttle our workload to limit the impact of a scan. */ if (ddi_get_lbolt64() - spa->spa_last_io <= zfs_scan_idle) delay(MAX((int)scan_delay, 0)); zio_nowait(zio_read(NULL, spa, bp, data, size, dsl_scan_scrub_done, NULL, ZIO_PRIORITY_SCRUB, zio_flags, zb)); } /* do not relocate this block */ return (0); } int dsl_scan(dsl_pool_t *dp, pool_scan_func_t func) { spa_t *spa = dp->dp_spa; /* * Purge all vdev caches and probe all devices. We do this here * rather than in sync context because this requires a writer lock * on the spa_config lock, which we can't do from sync context. The * spa_scrub_reopen flag indicates that vdev_open() should not * attempt to start another scrub. */ spa_vdev_state_enter(spa, SCL_NONE); spa->spa_scrub_reopen = B_TRUE; vdev_reopen(spa->spa_root_vdev); spa->spa_scrub_reopen = B_FALSE; (void) spa_vdev_state_exit(spa, NULL, 0); return (dsl_sync_task(spa_name(spa), dsl_scan_setup_check, dsl_scan_setup_sync, &func, 0, ZFS_SPACE_CHECK_NONE)); } Index: projects/clang350-import/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/spa.h =================================================================== --- projects/clang350-import/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/spa.h (revision 275748) +++ projects/clang350-import/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/sys/spa.h (revision 275749) @@ -1,893 +1,894 @@ /* * CDDL HEADER START * * The contents of this file are subject to the terms of the * Common Development and Distribution License (the "License"). * You may not use this file except in compliance with the License. * * You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE * or http://www.opensolaris.org/os/licensing. * See the License for the specific language governing permissions * and limitations under the License. * * When distributing Covered Code, include this CDDL HEADER in each * file and include the License file at usr/src/OPENSOLARIS.LICENSE. * If applicable, add the following below this CDDL HEADER, with the * fields enclosed by brackets "[]" replaced with your own identifying * information: Portions Copyright [yyyy] [name of copyright owner] * * CDDL HEADER END */ /* * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. * Copyright (c) 2011, 2014 by Delphix. All rights reserved. * Copyright 2011 Nexenta Systems, Inc. All rights reserved. */ #ifndef _SYS_SPA_H #define _SYS_SPA_H #include #include #include #include #include #include #ifdef __cplusplus extern "C" { #endif /* * Forward references that lots of things need. */ typedef struct spa spa_t; typedef struct vdev vdev_t; typedef struct metaslab metaslab_t; typedef struct metaslab_group metaslab_group_t; typedef struct metaslab_class metaslab_class_t; typedef struct zio zio_t; typedef struct zilog zilog_t; typedef struct spa_aux_vdev spa_aux_vdev_t; typedef struct ddt ddt_t; typedef struct ddt_entry ddt_entry_t; struct dsl_pool; struct dsl_dataset; /* * General-purpose 32-bit and 64-bit bitfield encodings. */ #define BF32_DECODE(x, low, len) P2PHASE((x) >> (low), 1U << (len)) #define BF64_DECODE(x, low, len) P2PHASE((x) >> (low), 1ULL << (len)) #define BF32_ENCODE(x, low, len) (P2PHASE((x), 1U << (len)) << (low)) #define BF64_ENCODE(x, low, len) (P2PHASE((x), 1ULL << (len)) << (low)) #define BF32_GET(x, low, len) BF32_DECODE(x, low, len) #define BF64_GET(x, low, len) BF64_DECODE(x, low, len) #define BF32_SET(x, low, len, val) do { \ ASSERT3U(val, <, 1U << (len)); \ ASSERT3U(low + len, <=, 32); \ (x) ^= BF32_ENCODE((x >> low) ^ (val), low, len); \ _NOTE(CONSTCOND) } while (0) #define BF64_SET(x, low, len, val) do { \ ASSERT3U(val, <, 1ULL << (len)); \ ASSERT3U(low + len, <=, 64); \ ((x) ^= BF64_ENCODE((x >> low) ^ (val), low, len)); \ _NOTE(CONSTCOND) } while (0) #define BF32_GET_SB(x, low, len, shift, bias) \ ((BF32_GET(x, low, len) + (bias)) << (shift)) #define BF64_GET_SB(x, low, len, shift, bias) \ ((BF64_GET(x, low, len) + (bias)) << (shift)) #define BF32_SET_SB(x, low, len, shift, bias, val) do { \ ASSERT(IS_P2ALIGNED(val, 1U << shift)); \ ASSERT3S((val) >> (shift), >=, bias); \ BF32_SET(x, low, len, ((val) >> (shift)) - (bias)); \ _NOTE(CONSTCOND) } while (0) #define BF64_SET_SB(x, low, len, shift, bias, val) do { \ ASSERT(IS_P2ALIGNED(val, 1ULL << shift)); \ ASSERT3S((val) >> (shift), >=, bias); \ BF64_SET(x, low, len, ((val) >> (shift)) - (bias)); \ _NOTE(CONSTCOND) } while (0) /* * We currently support block sizes from 512 bytes to 16MB. * The benefits of larger blocks, and thus larger IO, need to be weighed * against the cost of COWing a giant block to modify one byte, and the * large latency of reading or writing a large block. * * Note that although blocks up to 16MB are supported, the recordsize * property can not be set larger than zfs_max_recordsize (default 1MB). * See the comment near zfs_max_recordsize in dsl_dataset.c for details. * * Note that although the LSIZE field of the blkptr_t can store sizes up * to 32MB, the dnode's dn_datablkszsec can only store sizes up to * 32MB - 512 bytes. Therefore, we limit SPA_MAXBLOCKSIZE to 16MB. */ #define SPA_MINBLOCKSHIFT 9 #define SPA_OLD_MAXBLOCKSHIFT 17 #define SPA_MAXBLOCKSHIFT 24 #define SPA_MINBLOCKSIZE (1ULL << SPA_MINBLOCKSHIFT) #define SPA_OLD_MAXBLOCKSIZE (1ULL << SPA_OLD_MAXBLOCKSHIFT) #define SPA_MAXBLOCKSIZE (1ULL << SPA_MAXBLOCKSHIFT) /* * Default maximum supported logical ashift. * * The current 8k allocation block size limit is due to the 8k * aligned/sized operations performed by vdev_probe() on * vdev_label->vl_pad2. Using another "safe region" for these tests * would allow the limit to be raised to 16k, at the expense of * only having 8 available uberblocks in the label area. */ #define SPA_MAXASHIFT 13 /* * Default minimum supported logical ashift. */ #define SPA_MINASHIFT SPA_MINBLOCKSHIFT /* * Size of block to hold the configuration data (a packed nvlist) */ #define SPA_CONFIG_BLOCKSIZE (1ULL << 14) /* * The DVA size encodings for LSIZE and PSIZE support blocks up to 32MB. * The ASIZE encoding should be at least 64 times larger (6 more bits) * to support up to 4-way RAID-Z mirror mode with worst-case gang block * overhead, three DVAs per bp, plus one more bit in case we do anything * else that expands the ASIZE. */ #define SPA_LSIZEBITS 16 /* LSIZE up to 32M (2^16 * 512) */ #define SPA_PSIZEBITS 16 /* PSIZE up to 32M (2^16 * 512) */ #define SPA_ASIZEBITS 24 /* ASIZE up to 64 times larger */ /* * All SPA data is represented by 128-bit data virtual addresses (DVAs). * The members of the dva_t should be considered opaque outside the SPA. */ typedef struct dva { uint64_t dva_word[2]; } dva_t; /* * Each block has a 256-bit checksum -- strong enough for cryptographic hashes. */ typedef struct zio_cksum { uint64_t zc_word[4]; } zio_cksum_t; /* * Each block is described by its DVAs, time of birth, checksum, etc. * The word-by-word, bit-by-bit layout of the blkptr is as follows: * * 64 56 48 40 32 24 16 8 0 * +-------+-------+-------+-------+-------+-------+-------+-------+ * 0 | vdev1 | GRID | ASIZE | * +-------+-------+-------+-------+-------+-------+-------+-------+ * 1 |G| offset1 | * +-------+-------+-------+-------+-------+-------+-------+-------+ * 2 | vdev2 | GRID | ASIZE | * +-------+-------+-------+-------+-------+-------+-------+-------+ * 3 |G| offset2 | * +-------+-------+-------+-------+-------+-------+-------+-------+ * 4 | vdev3 | GRID | ASIZE | * +-------+-------+-------+-------+-------+-------+-------+-------+ * 5 |G| offset3 | * +-------+-------+-------+-------+-------+-------+-------+-------+ * 6 |BDX|lvl| type | cksum |E| comp| PSIZE | LSIZE | * +-------+-------+-------+-------+-------+-------+-------+-------+ * 7 | padding | * +-------+-------+-------+-------+-------+-------+-------+-------+ * 8 | padding | * +-------+-------+-------+-------+-------+-------+-------+-------+ * 9 | physical birth txg | * +-------+-------+-------+-------+-------+-------+-------+-------+ * a | logical birth txg | * +-------+-------+-------+-------+-------+-------+-------+-------+ * b | fill count | * +-------+-------+-------+-------+-------+-------+-------+-------+ * c | checksum[0] | * +-------+-------+-------+-------+-------+-------+-------+-------+ * d | checksum[1] | * +-------+-------+-------+-------+-------+-------+-------+-------+ * e | checksum[2] | * +-------+-------+-------+-------+-------+-------+-------+-------+ * f | checksum[3] | * +-------+-------+-------+-------+-------+-------+-------+-------+ * * Legend: * * vdev virtual device ID * offset offset into virtual device * LSIZE logical size * PSIZE physical size (after compression) * ASIZE allocated size (including RAID-Z parity and gang block headers) * GRID RAID-Z layout information (reserved for future use) * cksum checksum function * comp compression function * G gang block indicator * B byteorder (endianness) * D dedup * X encryption (on version 30, which is not supported) * E blkptr_t contains embedded data (see below) * lvl level of indirection * type DMU object type * phys birth txg of block allocation; zero if same as logical birth txg * log. birth transaction group in which the block was logically born * fill count number of non-zero blocks under this bp * checksum[4] 256-bit checksum of the data this bp describes */ /* * "Embedded" blkptr_t's don't actually point to a block, instead they * have a data payload embedded in the blkptr_t itself. See the comment * in blkptr.c for more details. * * The blkptr_t is laid out as follows: * * 64 56 48 40 32 24 16 8 0 * +-------+-------+-------+-------+-------+-------+-------+-------+ * 0 | payload | * 1 | payload | * 2 | payload | * 3 | payload | * 4 | payload | * 5 | payload | * +-------+-------+-------+-------+-------+-------+-------+-------+ * 6 |BDX|lvl| type | etype |E| comp| PSIZE| LSIZE | * +-------+-------+-------+-------+-------+-------+-------+-------+ * 7 | payload | * 8 | payload | * 9 | payload | * +-------+-------+-------+-------+-------+-------+-------+-------+ * a | logical birth txg | * +-------+-------+-------+-------+-------+-------+-------+-------+ * b | payload | * c | payload | * d | payload | * e | payload | * f | payload | * +-------+-------+-------+-------+-------+-------+-------+-------+ * * Legend: * * payload contains the embedded data * B (byteorder) byteorder (endianness) * D (dedup) padding (set to zero) * X encryption (set to zero; see above) * E (embedded) set to one * lvl indirection level * type DMU object type * etype how to interpret embedded data (BP_EMBEDDED_TYPE_*) * comp compression function of payload * PSIZE size of payload after compression, in bytes * LSIZE logical size of payload, in bytes * note that 25 bits is enough to store the largest * "normal" BP's LSIZE (2^16 * 2^9) in bytes * log. birth transaction group in which the block was logically born * * Note that LSIZE and PSIZE are stored in bytes, whereas for non-embedded * bp's they are stored in units of SPA_MINBLOCKSHIFT. * Generally, the generic BP_GET_*() macros can be used on embedded BP's. * The B, D, X, lvl, type, and comp fields are stored the same as with normal * BP's so the BP_SET_* macros can be used with them. etype, PSIZE, LSIZE must * be set with the BPE_SET_* macros. BP_SET_EMBEDDED() should be called before * other macros, as they assert that they are only used on BP's of the correct * "embedded-ness". */ #define BPE_GET_ETYPE(bp) \ (ASSERT(BP_IS_EMBEDDED(bp)), \ BF64_GET((bp)->blk_prop, 40, 8)) #define BPE_SET_ETYPE(bp, t) do { \ ASSERT(BP_IS_EMBEDDED(bp)); \ BF64_SET((bp)->blk_prop, 40, 8, t); \ _NOTE(CONSTCOND) } while (0) #define BPE_GET_LSIZE(bp) \ (ASSERT(BP_IS_EMBEDDED(bp)), \ BF64_GET_SB((bp)->blk_prop, 0, 25, 0, 1)) #define BPE_SET_LSIZE(bp, x) do { \ ASSERT(BP_IS_EMBEDDED(bp)); \ BF64_SET_SB((bp)->blk_prop, 0, 25, 0, 1, x); \ _NOTE(CONSTCOND) } while (0) #define BPE_GET_PSIZE(bp) \ (ASSERT(BP_IS_EMBEDDED(bp)), \ BF64_GET_SB((bp)->blk_prop, 25, 7, 0, 1)) #define BPE_SET_PSIZE(bp, x) do { \ ASSERT(BP_IS_EMBEDDED(bp)); \ BF64_SET_SB((bp)->blk_prop, 25, 7, 0, 1, x); \ _NOTE(CONSTCOND) } while (0) typedef enum bp_embedded_type { BP_EMBEDDED_TYPE_DATA, BP_EMBEDDED_TYPE_RESERVED, /* Reserved for an unintegrated feature. */ NUM_BP_EMBEDDED_TYPES = BP_EMBEDDED_TYPE_RESERVED } bp_embedded_type_t; #define BPE_NUM_WORDS 14 #define BPE_PAYLOAD_SIZE (BPE_NUM_WORDS * sizeof (uint64_t)) #define BPE_IS_PAYLOADWORD(bp, wp) \ ((wp) != &(bp)->blk_prop && (wp) != &(bp)->blk_birth) #define SPA_BLKPTRSHIFT 7 /* blkptr_t is 128 bytes */ #define SPA_DVAS_PER_BP 3 /* Number of DVAs in a bp */ /* * A block is a hole when it has either 1) never been written to, or * 2) is zero-filled. In both cases, ZFS can return all zeroes for all reads * without physically allocating disk space. Holes are represented in the * blkptr_t structure by zeroed blk_dva. Correct checking for holes is * done through the BP_IS_HOLE macro. For holes, the logical size, level, * DMU object type, and birth times are all also stored for holes that * were written to at some point (i.e. were punched after having been filled). */ typedef struct blkptr { dva_t blk_dva[SPA_DVAS_PER_BP]; /* Data Virtual Addresses */ uint64_t blk_prop; /* size, compression, type, etc */ uint64_t blk_pad[2]; /* Extra space for the future */ uint64_t blk_phys_birth; /* txg when block was allocated */ uint64_t blk_birth; /* transaction group at birth */ uint64_t blk_fill; /* fill count */ zio_cksum_t blk_cksum; /* 256-bit checksum */ } blkptr_t; /* * Macros to get and set fields in a bp or DVA. */ #define DVA_GET_ASIZE(dva) \ BF64_GET_SB((dva)->dva_word[0], 0, SPA_ASIZEBITS, SPA_MINBLOCKSHIFT, 0) #define DVA_SET_ASIZE(dva, x) \ BF64_SET_SB((dva)->dva_word[0], 0, SPA_ASIZEBITS, \ SPA_MINBLOCKSHIFT, 0, x) #define DVA_GET_GRID(dva) BF64_GET((dva)->dva_word[0], 24, 8) #define DVA_SET_GRID(dva, x) BF64_SET((dva)->dva_word[0], 24, 8, x) #define DVA_GET_VDEV(dva) BF64_GET((dva)->dva_word[0], 32, 32) #define DVA_SET_VDEV(dva, x) BF64_SET((dva)->dva_word[0], 32, 32, x) #define DVA_GET_OFFSET(dva) \ BF64_GET_SB((dva)->dva_word[1], 0, 63, SPA_MINBLOCKSHIFT, 0) #define DVA_SET_OFFSET(dva, x) \ BF64_SET_SB((dva)->dva_word[1], 0, 63, SPA_MINBLOCKSHIFT, 0, x) #define DVA_GET_GANG(dva) BF64_GET((dva)->dva_word[1], 63, 1) #define DVA_SET_GANG(dva, x) BF64_SET((dva)->dva_word[1], 63, 1, x) #define BP_GET_LSIZE(bp) \ (BP_IS_EMBEDDED(bp) ? \ (BPE_GET_ETYPE(bp) == BP_EMBEDDED_TYPE_DATA ? BPE_GET_LSIZE(bp) : 0): \ BF64_GET_SB((bp)->blk_prop, 0, SPA_LSIZEBITS, SPA_MINBLOCKSHIFT, 1)) #define BP_SET_LSIZE(bp, x) do { \ ASSERT(!BP_IS_EMBEDDED(bp)); \ BF64_SET_SB((bp)->blk_prop, \ 0, SPA_LSIZEBITS, SPA_MINBLOCKSHIFT, 1, x); \ _NOTE(CONSTCOND) } while (0) #define BP_GET_PSIZE(bp) \ (BP_IS_EMBEDDED(bp) ? 0 : \ BF64_GET_SB((bp)->blk_prop, 16, SPA_PSIZEBITS, SPA_MINBLOCKSHIFT, 1)) #define BP_SET_PSIZE(bp, x) do { \ ASSERT(!BP_IS_EMBEDDED(bp)); \ BF64_SET_SB((bp)->blk_prop, \ 16, SPA_PSIZEBITS, SPA_MINBLOCKSHIFT, 1, x); \ _NOTE(CONSTCOND) } while (0) #define BP_GET_COMPRESS(bp) BF64_GET((bp)->blk_prop, 32, 7) #define BP_SET_COMPRESS(bp, x) BF64_SET((bp)->blk_prop, 32, 7, x) #define BP_IS_EMBEDDED(bp) BF64_GET((bp)->blk_prop, 39, 1) #define BP_SET_EMBEDDED(bp, x) BF64_SET((bp)->blk_prop, 39, 1, x) #define BP_GET_CHECKSUM(bp) \ (BP_IS_EMBEDDED(bp) ? ZIO_CHECKSUM_OFF : \ BF64_GET((bp)->blk_prop, 40, 8)) #define BP_SET_CHECKSUM(bp, x) do { \ ASSERT(!BP_IS_EMBEDDED(bp)); \ BF64_SET((bp)->blk_prop, 40, 8, x); \ _NOTE(CONSTCOND) } while (0) #define BP_GET_TYPE(bp) BF64_GET((bp)->blk_prop, 48, 8) #define BP_SET_TYPE(bp, x) BF64_SET((bp)->blk_prop, 48, 8, x) #define BP_GET_LEVEL(bp) BF64_GET((bp)->blk_prop, 56, 5) #define BP_SET_LEVEL(bp, x) BF64_SET((bp)->blk_prop, 56, 5, x) #define BP_GET_DEDUP(bp) BF64_GET((bp)->blk_prop, 62, 1) #define BP_SET_DEDUP(bp, x) BF64_SET((bp)->blk_prop, 62, 1, x) #define BP_GET_BYTEORDER(bp) BF64_GET((bp)->blk_prop, 63, 1) #define BP_SET_BYTEORDER(bp, x) BF64_SET((bp)->blk_prop, 63, 1, x) #define BP_PHYSICAL_BIRTH(bp) \ (BP_IS_EMBEDDED(bp) ? 0 : \ (bp)->blk_phys_birth ? (bp)->blk_phys_birth : (bp)->blk_birth) #define BP_SET_BIRTH(bp, logical, physical) \ { \ ASSERT(!BP_IS_EMBEDDED(bp)); \ (bp)->blk_birth = (logical); \ (bp)->blk_phys_birth = ((logical) == (physical) ? 0 : (physical)); \ } #define BP_GET_FILL(bp) (BP_IS_EMBEDDED(bp) ? 1 : (bp)->blk_fill) #define BP_GET_ASIZE(bp) \ (BP_IS_EMBEDDED(bp) ? 0 : \ DVA_GET_ASIZE(&(bp)->blk_dva[0]) + \ DVA_GET_ASIZE(&(bp)->blk_dva[1]) + \ DVA_GET_ASIZE(&(bp)->blk_dva[2])) #define BP_GET_UCSIZE(bp) \ ((BP_GET_LEVEL(bp) > 0 || DMU_OT_IS_METADATA(BP_GET_TYPE(bp))) ? \ BP_GET_PSIZE(bp) : BP_GET_LSIZE(bp)) #define BP_GET_NDVAS(bp) \ (BP_IS_EMBEDDED(bp) ? 0 : \ !!DVA_GET_ASIZE(&(bp)->blk_dva[0]) + \ !!DVA_GET_ASIZE(&(bp)->blk_dva[1]) + \ !!DVA_GET_ASIZE(&(bp)->blk_dva[2])) #define BP_COUNT_GANG(bp) \ (BP_IS_EMBEDDED(bp) ? 0 : \ (DVA_GET_GANG(&(bp)->blk_dva[0]) + \ DVA_GET_GANG(&(bp)->blk_dva[1]) + \ DVA_GET_GANG(&(bp)->blk_dva[2]))) #define DVA_EQUAL(dva1, dva2) \ ((dva1)->dva_word[1] == (dva2)->dva_word[1] && \ (dva1)->dva_word[0] == (dva2)->dva_word[0]) #define BP_EQUAL(bp1, bp2) \ (BP_PHYSICAL_BIRTH(bp1) == BP_PHYSICAL_BIRTH(bp2) && \ (bp1)->blk_birth == (bp2)->blk_birth && \ DVA_EQUAL(&(bp1)->blk_dva[0], &(bp2)->blk_dva[0]) && \ DVA_EQUAL(&(bp1)->blk_dva[1], &(bp2)->blk_dva[1]) && \ DVA_EQUAL(&(bp1)->blk_dva[2], &(bp2)->blk_dva[2])) #define ZIO_CHECKSUM_EQUAL(zc1, zc2) \ (0 == (((zc1).zc_word[0] - (zc2).zc_word[0]) | \ ((zc1).zc_word[1] - (zc2).zc_word[1]) | \ ((zc1).zc_word[2] - (zc2).zc_word[2]) | \ ((zc1).zc_word[3] - (zc2).zc_word[3]))) #define DVA_IS_VALID(dva) (DVA_GET_ASIZE(dva) != 0) #define ZIO_SET_CHECKSUM(zcp, w0, w1, w2, w3) \ { \ (zcp)->zc_word[0] = w0; \ (zcp)->zc_word[1] = w1; \ (zcp)->zc_word[2] = w2; \ (zcp)->zc_word[3] = w3; \ } #define BP_IDENTITY(bp) (ASSERT(!BP_IS_EMBEDDED(bp)), &(bp)->blk_dva[0]) #define BP_IS_GANG(bp) \ (BP_IS_EMBEDDED(bp) ? B_FALSE : DVA_GET_GANG(BP_IDENTITY(bp))) #define DVA_IS_EMPTY(dva) ((dva)->dva_word[0] == 0ULL && \ (dva)->dva_word[1] == 0ULL) #define BP_IS_HOLE(bp) \ (!BP_IS_EMBEDDED(bp) && DVA_IS_EMPTY(BP_IDENTITY(bp))) /* BP_IS_RAIDZ(bp) assumes no block compression */ #define BP_IS_RAIDZ(bp) (DVA_GET_ASIZE(&(bp)->blk_dva[0]) > \ BP_GET_PSIZE(bp)) #define BP_ZERO(bp) \ { \ (bp)->blk_dva[0].dva_word[0] = 0; \ (bp)->blk_dva[0].dva_word[1] = 0; \ (bp)->blk_dva[1].dva_word[0] = 0; \ (bp)->blk_dva[1].dva_word[1] = 0; \ (bp)->blk_dva[2].dva_word[0] = 0; \ (bp)->blk_dva[2].dva_word[1] = 0; \ (bp)->blk_prop = 0; \ (bp)->blk_pad[0] = 0; \ (bp)->blk_pad[1] = 0; \ (bp)->blk_phys_birth = 0; \ (bp)->blk_birth = 0; \ (bp)->blk_fill = 0; \ ZIO_SET_CHECKSUM(&(bp)->blk_cksum, 0, 0, 0, 0); \ } #if BYTE_ORDER == _BIG_ENDIAN #define ZFS_HOST_BYTEORDER (0ULL) #else #define ZFS_HOST_BYTEORDER (1ULL) #endif #define BP_SHOULD_BYTESWAP(bp) (BP_GET_BYTEORDER(bp) != ZFS_HOST_BYTEORDER) #define BP_SPRINTF_LEN 320 /* * This macro allows code sharing between zfs, libzpool, and mdb. * 'func' is either snprintf() or mdb_snprintf(). * 'ws' (whitespace) can be ' ' for single-line format, '\n' for multi-line. */ #define SNPRINTF_BLKPTR(func, ws, buf, size, bp, type, checksum, compress) \ { \ static const char *copyname[] = \ { "zero", "single", "double", "triple" }; \ int len = 0; \ int copies = 0; \ \ if (bp == NULL) { \ len += func(buf + len, size - len, ""); \ } else if (BP_IS_HOLE(bp)) { \ len += func(buf + len, size - len, ""); \ if (bp->blk_birth > 0) { \ len += func(buf + len, size - len, \ " birth=%lluL", \ (u_longlong_t)bp->blk_birth); \ } \ } else if (BP_IS_EMBEDDED(bp)) { \ len = func(buf + len, size - len, \ "EMBEDDED [L%llu %s] et=%u %s " \ "size=%llxL/%llxP birth=%lluL", \ (u_longlong_t)BP_GET_LEVEL(bp), \ type, \ (int)BPE_GET_ETYPE(bp), \ compress, \ (u_longlong_t)BPE_GET_LSIZE(bp), \ (u_longlong_t)BPE_GET_PSIZE(bp), \ (u_longlong_t)bp->blk_birth); \ } else { \ for (int d = 0; d < BP_GET_NDVAS(bp); d++) { \ const dva_t *dva = &bp->blk_dva[d]; \ if (DVA_IS_VALID(dva)) \ copies++; \ len += func(buf + len, size - len, \ "DVA[%d]=<%llu:%llx:%llx>%c", d, \ (u_longlong_t)DVA_GET_VDEV(dva), \ (u_longlong_t)DVA_GET_OFFSET(dva), \ (u_longlong_t)DVA_GET_ASIZE(dva), \ ws); \ } \ if (BP_IS_GANG(bp) && \ DVA_GET_ASIZE(&bp->blk_dva[2]) <= \ DVA_GET_ASIZE(&bp->blk_dva[1]) / 2) \ copies--; \ len += func(buf + len, size - len, \ "[L%llu %s] %s %s %s %s %s %s%c" \ "size=%llxL/%llxP birth=%lluL/%lluP fill=%llu%c" \ "cksum=%llx:%llx:%llx:%llx", \ (u_longlong_t)BP_GET_LEVEL(bp), \ type, \ checksum, \ compress, \ BP_GET_BYTEORDER(bp) == 0 ? "BE" : "LE", \ BP_IS_GANG(bp) ? "gang" : "contiguous", \ BP_GET_DEDUP(bp) ? "dedup" : "unique", \ copyname[copies], \ ws, \ (u_longlong_t)BP_GET_LSIZE(bp), \ (u_longlong_t)BP_GET_PSIZE(bp), \ (u_longlong_t)bp->blk_birth, \ (u_longlong_t)BP_PHYSICAL_BIRTH(bp), \ (u_longlong_t)BP_GET_FILL(bp), \ ws, \ (u_longlong_t)bp->blk_cksum.zc_word[0], \ (u_longlong_t)bp->blk_cksum.zc_word[1], \ (u_longlong_t)bp->blk_cksum.zc_word[2], \ (u_longlong_t)bp->blk_cksum.zc_word[3]); \ } \ ASSERT(len < size); \ } #include #define BP_GET_BUFC_TYPE(bp) \ (((BP_GET_LEVEL(bp) > 0) || (DMU_OT_IS_METADATA(BP_GET_TYPE(bp)))) ? \ ARC_BUFC_METADATA : ARC_BUFC_DATA) typedef enum spa_import_type { SPA_IMPORT_EXISTING, SPA_IMPORT_ASSEMBLE } spa_import_type_t; /* state manipulation functions */ extern int spa_open(const char *pool, spa_t **, void *tag); extern int spa_open_rewind(const char *pool, spa_t **, void *tag, nvlist_t *policy, nvlist_t **config); extern int spa_get_stats(const char *pool, nvlist_t **config, char *altroot, size_t buflen); extern int spa_create(const char *pool, nvlist_t *config, nvlist_t *props, nvlist_t *zplprops); #if defined(sun) extern int spa_import_rootpool(char *devpath, char *devid); #else extern int spa_import_rootpool(const char *name); #endif extern int spa_import(const char *pool, nvlist_t *config, nvlist_t *props, uint64_t flags); extern nvlist_t *spa_tryimport(nvlist_t *tryconfig); extern int spa_destroy(char *pool); extern int spa_export(char *pool, nvlist_t **oldconfig, boolean_t force, boolean_t hardforce); extern int spa_reset(char *pool); extern void spa_async_request(spa_t *spa, int flag); extern void spa_async_unrequest(spa_t *spa, int flag); extern void spa_async_suspend(spa_t *spa); extern void spa_async_resume(spa_t *spa); extern spa_t *spa_inject_addref(char *pool); extern void spa_inject_delref(spa_t *spa); extern void spa_scan_stat_init(spa_t *spa); extern int spa_scan_get_stats(spa_t *spa, pool_scan_stat_t *ps); #define SPA_ASYNC_CONFIG_UPDATE 0x01 #define SPA_ASYNC_REMOVE 0x02 #define SPA_ASYNC_PROBE 0x04 #define SPA_ASYNC_RESILVER_DONE 0x08 #define SPA_ASYNC_RESILVER 0x10 #define SPA_ASYNC_AUTOEXPAND 0x20 #define SPA_ASYNC_REMOVE_DONE 0x40 #define SPA_ASYNC_REMOVE_STOP 0x80 /* * Controls the behavior of spa_vdev_remove(). */ #define SPA_REMOVE_UNSPARE 0x01 #define SPA_REMOVE_DONE 0x02 /* device manipulation */ extern int spa_vdev_add(spa_t *spa, nvlist_t *nvroot); extern int spa_vdev_attach(spa_t *spa, uint64_t guid, nvlist_t *nvroot, int replacing); extern int spa_vdev_detach(spa_t *spa, uint64_t guid, uint64_t pguid, int replace_done); extern int spa_vdev_remove(spa_t *spa, uint64_t guid, boolean_t unspare); extern boolean_t spa_vdev_remove_active(spa_t *spa); extern int spa_vdev_setpath(spa_t *spa, uint64_t guid, const char *newpath); extern int spa_vdev_setfru(spa_t *spa, uint64_t guid, const char *newfru); extern int spa_vdev_split_mirror(spa_t *spa, char *newname, nvlist_t *config, nvlist_t *props, boolean_t exp); /* spare state (which is global across all pools) */ extern void spa_spare_add(vdev_t *vd); extern void spa_spare_remove(vdev_t *vd); extern boolean_t spa_spare_exists(uint64_t guid, uint64_t *pool, int *refcnt); extern void spa_spare_activate(vdev_t *vd); /* L2ARC state (which is global across all pools) */ extern void spa_l2cache_add(vdev_t *vd); extern void spa_l2cache_remove(vdev_t *vd); extern boolean_t spa_l2cache_exists(uint64_t guid, uint64_t *pool); extern void spa_l2cache_activate(vdev_t *vd); extern void spa_l2cache_drop(spa_t *spa); /* scanning */ extern int spa_scan(spa_t *spa, pool_scan_func_t func); extern int spa_scan_stop(spa_t *spa); /* spa syncing */ extern void spa_sync(spa_t *spa, uint64_t txg); /* only for DMU use */ extern void spa_sync_allpools(void); /* spa namespace global mutex */ extern kmutex_t spa_namespace_lock; /* * SPA configuration functions in spa_config.c */ #define SPA_CONFIG_UPDATE_POOL 0 #define SPA_CONFIG_UPDATE_VDEVS 1 extern void spa_config_sync(spa_t *, boolean_t, boolean_t); extern void spa_config_load(void); extern nvlist_t *spa_all_configs(uint64_t *); extern void spa_config_set(spa_t *spa, nvlist_t *config); extern nvlist_t *spa_config_generate(spa_t *spa, vdev_t *vd, uint64_t txg, int getstats); extern void spa_config_update(spa_t *spa, int what); /* * Miscellaneous SPA routines in spa_misc.c */ /* Namespace manipulation */ extern spa_t *spa_lookup(const char *name); extern spa_t *spa_add(const char *name, nvlist_t *config, const char *altroot); extern void spa_remove(spa_t *spa); extern spa_t *spa_next(spa_t *prev); /* Refcount functions */ extern void spa_open_ref(spa_t *spa, void *tag); extern void spa_close(spa_t *spa, void *tag); extern boolean_t spa_refcount_zero(spa_t *spa); #define SCL_NONE 0x00 #define SCL_CONFIG 0x01 #define SCL_STATE 0x02 #define SCL_L2ARC 0x04 /* hack until L2ARC 2.0 */ #define SCL_ALLOC 0x08 #define SCL_ZIO 0x10 #define SCL_FREE 0x20 #define SCL_VDEV 0x40 #define SCL_LOCKS 7 #define SCL_ALL ((1 << SCL_LOCKS) - 1) #define SCL_STATE_ALL (SCL_STATE | SCL_L2ARC | SCL_ZIO) /* Pool configuration locks */ extern int spa_config_tryenter(spa_t *spa, int locks, void *tag, krw_t rw); extern void spa_config_enter(spa_t *spa, int locks, void *tag, krw_t rw); extern void spa_config_exit(spa_t *spa, int locks, void *tag); extern int spa_config_held(spa_t *spa, int locks, krw_t rw); /* Pool vdev add/remove lock */ extern uint64_t spa_vdev_enter(spa_t *spa); extern uint64_t spa_vdev_config_enter(spa_t *spa); extern void spa_vdev_config_exit(spa_t *spa, vdev_t *vd, uint64_t txg, int error, char *tag); extern int spa_vdev_exit(spa_t *spa, vdev_t *vd, uint64_t txg, int error); /* Pool vdev state change lock */ extern void spa_vdev_state_enter(spa_t *spa, int oplock); extern int spa_vdev_state_exit(spa_t *spa, vdev_t *vd, int error); /* Log state */ typedef enum spa_log_state { SPA_LOG_UNKNOWN = 0, /* unknown log state */ SPA_LOG_MISSING, /* missing log(s) */ SPA_LOG_CLEAR, /* clear the log(s) */ SPA_LOG_GOOD, /* log(s) are good */ } spa_log_state_t; extern spa_log_state_t spa_get_log_state(spa_t *spa); extern void spa_set_log_state(spa_t *spa, spa_log_state_t state); extern int spa_offline_log(spa_t *spa); /* Log claim callback */ extern void spa_claim_notify(zio_t *zio); /* Accessor functions */ extern boolean_t spa_shutting_down(spa_t *spa); extern struct dsl_pool *spa_get_dsl(spa_t *spa); extern boolean_t spa_is_initializing(spa_t *spa); extern blkptr_t *spa_get_rootblkptr(spa_t *spa); extern void spa_set_rootblkptr(spa_t *spa, const blkptr_t *bp); extern void spa_altroot(spa_t *, char *, size_t); extern int spa_sync_pass(spa_t *spa); extern char *spa_name(spa_t *spa); extern uint64_t spa_guid(spa_t *spa); extern uint64_t spa_load_guid(spa_t *spa); extern uint64_t spa_last_synced_txg(spa_t *spa); extern uint64_t spa_first_txg(spa_t *spa); extern uint64_t spa_syncing_txg(spa_t *spa); extern uint64_t spa_version(spa_t *spa); extern pool_state_t spa_state(spa_t *spa); extern spa_load_state_t spa_load_state(spa_t *spa); extern uint64_t spa_freeze_txg(spa_t *spa); extern uint64_t spa_get_asize(spa_t *spa, uint64_t lsize); extern uint64_t spa_get_dspace(spa_t *spa); extern uint64_t spa_get_slop_space(spa_t *spa); extern void spa_update_dspace(spa_t *spa); extern uint64_t spa_version(spa_t *spa); extern boolean_t spa_deflate(spa_t *spa); extern metaslab_class_t *spa_normal_class(spa_t *spa); extern metaslab_class_t *spa_log_class(spa_t *spa); extern int spa_max_replication(spa_t *spa); extern int spa_prev_software_version(spa_t *spa); extern int spa_busy(void); extern uint8_t spa_get_failmode(spa_t *spa); extern boolean_t spa_suspended(spa_t *spa); extern uint64_t spa_bootfs(spa_t *spa); extern uint64_t spa_delegation(spa_t *spa); extern objset_t *spa_meta_objset(spa_t *spa); extern uint64_t spa_deadman_synctime(spa_t *spa); /* Miscellaneous support routines */ extern void spa_activate_mos_feature(spa_t *spa, const char *feature, dmu_tx_t *tx); extern void spa_deactivate_mos_feature(spa_t *spa, const char *feature); extern int spa_rename(const char *oldname, const char *newname); extern spa_t *spa_by_guid(uint64_t pool_guid, uint64_t device_guid); extern boolean_t spa_guid_exists(uint64_t pool_guid, uint64_t device_guid); extern char *spa_strdup(const char *); extern void spa_strfree(char *); extern uint64_t spa_get_random(uint64_t range); extern uint64_t spa_generate_guid(spa_t *spa); extern void snprintf_blkptr(char *buf, size_t buflen, const blkptr_t *bp); extern void spa_freeze(spa_t *spa); extern int spa_change_guid(spa_t *spa); extern void spa_upgrade(spa_t *spa, uint64_t version); extern void spa_evict_all(void); extern vdev_t *spa_lookup_by_guid(spa_t *spa, uint64_t guid, boolean_t l2cache); extern boolean_t spa_has_spare(spa_t *, uint64_t guid); extern uint64_t dva_get_dsize_sync(spa_t *spa, const dva_t *dva); extern uint64_t bp_get_dsize_sync(spa_t *spa, const blkptr_t *bp); extern uint64_t bp_get_dsize(spa_t *spa, const blkptr_t *bp); extern boolean_t spa_has_slogs(spa_t *spa); extern boolean_t spa_is_root(spa_t *spa); extern boolean_t spa_writeable(spa_t *spa); extern boolean_t spa_has_pending_synctask(spa_t *spa); extern int spa_maxblocksize(spa_t *spa); +extern void zfs_blkptr_verify(spa_t *spa, const blkptr_t *bp); extern int spa_mode(spa_t *spa); extern uint64_t zfs_strtonum(const char *str, char **nptr); #define strtonum(str, nptr) zfs_strtonum((str), (nptr)) extern char *spa_his_ievent_table[]; extern void spa_history_create_obj(spa_t *spa, dmu_tx_t *tx); extern int spa_history_get(spa_t *spa, uint64_t *offset, uint64_t *len_read, char *his_buf); extern int spa_history_log(spa_t *spa, const char *his_buf); extern int spa_history_log_nvl(spa_t *spa, nvlist_t *nvl); extern void spa_history_log_version(spa_t *spa, const char *operation); extern void spa_history_log_internal(spa_t *spa, const char *operation, dmu_tx_t *tx, const char *fmt, ...); extern void spa_history_log_internal_ds(struct dsl_dataset *ds, const char *op, dmu_tx_t *tx, const char *fmt, ...); extern void spa_history_log_internal_dd(dsl_dir_t *dd, const char *operation, dmu_tx_t *tx, const char *fmt, ...); /* error handling */ struct zbookmark_phys; extern void spa_log_error(spa_t *spa, zio_t *zio); extern void zfs_ereport_post(const char *cls, spa_t *spa, vdev_t *vd, zio_t *zio, uint64_t stateoroffset, uint64_t length); extern void zfs_post_remove(spa_t *spa, vdev_t *vd); extern void zfs_post_state_change(spa_t *spa, vdev_t *vd); extern void zfs_post_autoreplace(spa_t *spa, vdev_t *vd); extern uint64_t spa_get_errlog_size(spa_t *spa); extern int spa_get_errlog(spa_t *spa, void *uaddr, size_t *count); extern void spa_errlog_rotate(spa_t *spa); extern void spa_errlog_drain(spa_t *spa); extern void spa_errlog_sync(spa_t *spa, uint64_t txg); extern void spa_get_errlists(spa_t *spa, avl_tree_t *last, avl_tree_t *scrub); /* vdev cache */ extern void vdev_cache_stat_init(void); extern void vdev_cache_stat_fini(void); /* Initialization and termination */ extern void spa_init(int flags); extern void spa_fini(void); extern void spa_boot_init(); /* properties */ extern int spa_prop_set(spa_t *spa, nvlist_t *nvp); extern int spa_prop_get(spa_t *spa, nvlist_t **nvp); extern void spa_prop_clear_bootfs(spa_t *spa, uint64_t obj, dmu_tx_t *tx); extern void spa_configfile_set(spa_t *, nvlist_t *, boolean_t); /* asynchronous event notification */ extern void spa_event_notify(spa_t *spa, vdev_t *vdev, const char *name); #ifdef ZFS_DEBUG #define dprintf_bp(bp, fmt, ...) do { \ if (zfs_flags & ZFS_DEBUG_DPRINTF) { \ char *__blkbuf = kmem_alloc(BP_SPRINTF_LEN, KM_SLEEP); \ snprintf_blkptr(__blkbuf, BP_SPRINTF_LEN, (bp)); \ dprintf(fmt " %s\n", __VA_ARGS__, __blkbuf); \ kmem_free(__blkbuf, BP_SPRINTF_LEN); \ } \ _NOTE(CONSTCOND) } while (0) #else #define dprintf_bp(bp, fmt, ...) #endif extern boolean_t spa_debug_enabled(spa_t *spa); #define spa_dbgmsg(spa, ...) \ { \ if (spa_debug_enabled(spa)) \ zfs_dbgmsg(__VA_ARGS__); \ } extern int spa_mode_global; /* mode, e.g. FREAD | FWRITE */ #ifdef __cplusplus } #endif #endif /* _SYS_SPA_H */ Index: projects/clang350-import/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c =================================================================== --- projects/clang350-import/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c (revision 275748) +++ projects/clang350-import/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c (revision 275749) @@ -1,3410 +1,3493 @@ /* * CDDL HEADER START * * The contents of this file are subject to the terms of the * Common Development and Distribution License (the "License"). * You may not use this file except in compliance with the License. * * You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE * or http://www.opensolaris.org/os/licensing. * See the License for the specific language governing permissions * and limitations under the License. * * When distributing Covered Code, include this CDDL HEADER in each * file and include the License file at usr/src/OPENSOLARIS.LICENSE. * If applicable, add the following below this CDDL HEADER, with the * fields enclosed by brackets "[]" replaced with your own identifying * information: Portions Copyright [yyyy] [name of copyright owner] * * CDDL HEADER END */ /* * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. * Copyright (c) 2011, 2014 by Delphix. All rights reserved. * Copyright (c) 2011 Nexenta Systems, Inc. All rights reserved. */ #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include SYSCTL_DECL(_vfs_zfs); SYSCTL_NODE(_vfs_zfs, OID_AUTO, zio, CTLFLAG_RW, 0, "ZFS ZIO"); #if defined(__amd64__) static int zio_use_uma = 1; #else static int zio_use_uma = 0; #endif SYSCTL_INT(_vfs_zfs_zio, OID_AUTO, use_uma, CTLFLAG_RDTUN, &zio_use_uma, 0, "Use uma(9) for ZIO allocations"); static int zio_exclude_metadata = 0; SYSCTL_INT(_vfs_zfs_zio, OID_AUTO, exclude_metadata, CTLFLAG_RDTUN, &zio_exclude_metadata, 0, "Exclude metadata buffers from dumps as well"); zio_trim_stats_t zio_trim_stats = { { "bytes", KSTAT_DATA_UINT64, "Number of bytes successfully TRIMmed" }, { "success", KSTAT_DATA_UINT64, "Number of successful TRIM requests" }, { "unsupported", KSTAT_DATA_UINT64, "Number of TRIM requests that failed because TRIM is not supported" }, { "failed", KSTAT_DATA_UINT64, "Number of TRIM requests that failed for reasons other than not supported" }, }; static kstat_t *zio_trim_ksp; /* * ========================================================================== * I/O type descriptions * ========================================================================== */ const char *zio_type_name[ZIO_TYPES] = { "zio_null", "zio_read", "zio_write", "zio_free", "zio_claim", "zio_ioctl" }; /* * ========================================================================== * I/O kmem caches * ========================================================================== */ kmem_cache_t *zio_cache; kmem_cache_t *zio_link_cache; kmem_cache_t *zio_buf_cache[SPA_MAXBLOCKSIZE >> SPA_MINBLOCKSHIFT]; kmem_cache_t *zio_data_buf_cache[SPA_MAXBLOCKSIZE >> SPA_MINBLOCKSHIFT]; #ifdef _KERNEL extern vmem_t *zio_alloc_arena; #endif #define ZIO_PIPELINE_CONTINUE 0x100 #define ZIO_PIPELINE_STOP 0x101 /* * The following actions directly effect the spa's sync-to-convergence logic. * The values below define the sync pass when we start performing the action. * Care should be taken when changing these values as they directly impact * spa_sync() performance. Tuning these values may introduce subtle performance * pathologies and should only be done in the context of performance analysis. * These tunables will eventually be removed and replaced with #defines once * enough analysis has been done to determine optimal values. * * The 'zfs_sync_pass_deferred_free' pass must be greater than 1 to ensure that * regular blocks are not deferred. */ int zfs_sync_pass_deferred_free = 2; /* defer frees starting in this pass */ SYSCTL_INT(_vfs_zfs, OID_AUTO, sync_pass_deferred_free, CTLFLAG_RDTUN, &zfs_sync_pass_deferred_free, 0, "defer frees starting in this pass"); int zfs_sync_pass_dont_compress = 5; /* don't compress starting in this pass */ SYSCTL_INT(_vfs_zfs, OID_AUTO, sync_pass_dont_compress, CTLFLAG_RDTUN, &zfs_sync_pass_dont_compress, 0, "don't compress starting in this pass"); int zfs_sync_pass_rewrite = 2; /* rewrite new bps starting in this pass */ SYSCTL_INT(_vfs_zfs, OID_AUTO, sync_pass_rewrite, CTLFLAG_RDTUN, &zfs_sync_pass_rewrite, 0, "rewrite new bps starting in this pass"); /* * An allocating zio is one that either currently has the DVA allocate * stage set or will have it later in its lifetime. */ #define IO_IS_ALLOCATING(zio) ((zio)->io_orig_pipeline & ZIO_STAGE_DVA_ALLOCATE) boolean_t zio_requeue_io_start_cut_in_line = B_TRUE; #ifdef ZFS_DEBUG int zio_buf_debug_limit = 16384; #else int zio_buf_debug_limit = 0; #endif void zio_init(void) { size_t c; zio_cache = kmem_cache_create("zio_cache", sizeof (zio_t), 0, NULL, NULL, NULL, NULL, NULL, 0); zio_link_cache = kmem_cache_create("zio_link_cache", sizeof (zio_link_t), 0, NULL, NULL, NULL, NULL, NULL, 0); if (!zio_use_uma) goto out; /* * For small buffers, we want a cache for each multiple of * SPA_MINBLOCKSIZE. For larger buffers, we want a cache * for each quarter-power of 2. */ for (c = 0; c < SPA_MAXBLOCKSIZE >> SPA_MINBLOCKSHIFT; c++) { size_t size = (c + 1) << SPA_MINBLOCKSHIFT; size_t p2 = size; size_t align = 0; size_t cflags = (size > zio_buf_debug_limit) ? KMC_NODEBUG : 0; while (!ISP2(p2)) p2 &= p2 - 1; #ifdef illumos #ifndef _KERNEL /* * If we are using watchpoints, put each buffer on its own page, * to eliminate the performance overhead of trapping to the * kernel when modifying a non-watched buffer that shares the * page with a watched buffer. */ if (arc_watch && !IS_P2ALIGNED(size, PAGESIZE)) continue; #endif #endif /* illumos */ if (size <= 4 * SPA_MINBLOCKSIZE) { align = SPA_MINBLOCKSIZE; } else if (IS_P2ALIGNED(size, p2 >> 2)) { align = MIN(p2 >> 2, PAGESIZE); } if (align != 0) { char name[36]; (void) sprintf(name, "zio_buf_%lu", (ulong_t)size); zio_buf_cache[c] = kmem_cache_create(name, size, align, NULL, NULL, NULL, NULL, NULL, cflags); /* * Since zio_data bufs do not appear in crash dumps, we * pass KMC_NOTOUCH so that no allocator metadata is * stored with the buffers. */ (void) sprintf(name, "zio_data_buf_%lu", (ulong_t)size); zio_data_buf_cache[c] = kmem_cache_create(name, size, align, NULL, NULL, NULL, NULL, NULL, cflags | KMC_NOTOUCH | KMC_NODEBUG); } } while (--c != 0) { ASSERT(zio_buf_cache[c] != NULL); if (zio_buf_cache[c - 1] == NULL) zio_buf_cache[c - 1] = zio_buf_cache[c]; ASSERT(zio_data_buf_cache[c] != NULL); if (zio_data_buf_cache[c - 1] == NULL) zio_data_buf_cache[c - 1] = zio_data_buf_cache[c]; } out: zio_inject_init(); zio_trim_ksp = kstat_create("zfs", 0, "zio_trim", "misc", KSTAT_TYPE_NAMED, sizeof(zio_trim_stats) / sizeof(kstat_named_t), KSTAT_FLAG_VIRTUAL); if (zio_trim_ksp != NULL) { zio_trim_ksp->ks_data = &zio_trim_stats; kstat_install(zio_trim_ksp); } } void zio_fini(void) { size_t c; kmem_cache_t *last_cache = NULL; kmem_cache_t *last_data_cache = NULL; for (c = 0; c < SPA_MAXBLOCKSIZE >> SPA_MINBLOCKSHIFT; c++) { if (zio_buf_cache[c] != last_cache) { last_cache = zio_buf_cache[c]; kmem_cache_destroy(zio_buf_cache[c]); } zio_buf_cache[c] = NULL; if (zio_data_buf_cache[c] != last_data_cache) { last_data_cache = zio_data_buf_cache[c]; kmem_cache_destroy(zio_data_buf_cache[c]); } zio_data_buf_cache[c] = NULL; } kmem_cache_destroy(zio_link_cache); kmem_cache_destroy(zio_cache); zio_inject_fini(); if (zio_trim_ksp != NULL) { kstat_delete(zio_trim_ksp); zio_trim_ksp = NULL; } } /* * ========================================================================== * Allocate and free I/O buffers * ========================================================================== */ /* * Use zio_buf_alloc to allocate ZFS metadata. This data will appear in a * crashdump if the kernel panics, so use it judiciously. Obviously, it's * useful to inspect ZFS metadata, but if possible, we should avoid keeping * excess / transient data in-core during a crashdump. */ void * zio_buf_alloc(size_t size) { size_t c = (size - 1) >> SPA_MINBLOCKSHIFT; int flags = zio_exclude_metadata ? KM_NODEBUG : 0; - ASSERT3U(c, <, SPA_MAXBLOCKSIZE >> SPA_MINBLOCKSHIFT); + VERIFY3U(c, <, SPA_MAXBLOCKSIZE >> SPA_MINBLOCKSHIFT); if (zio_use_uma) return (kmem_cache_alloc(zio_buf_cache[c], KM_PUSHPAGE)); else return (kmem_alloc(size, KM_SLEEP|flags)); } /* * Use zio_data_buf_alloc to allocate data. The data will not appear in a * crashdump if the kernel panics. This exists so that we will limit the amount * of ZFS data that shows up in a kernel crashdump. (Thus reducing the amount * of kernel heap dumped to disk when the kernel panics) */ void * zio_data_buf_alloc(size_t size) { size_t c = (size - 1) >> SPA_MINBLOCKSHIFT; - ASSERT(c < SPA_MAXBLOCKSIZE >> SPA_MINBLOCKSHIFT); + VERIFY3U(c, <, SPA_MAXBLOCKSIZE >> SPA_MINBLOCKSHIFT); if (zio_use_uma) return (kmem_cache_alloc(zio_data_buf_cache[c], KM_PUSHPAGE)); else return (kmem_alloc(size, KM_SLEEP | KM_NODEBUG)); } void zio_buf_free(void *buf, size_t size) { size_t c = (size - 1) >> SPA_MINBLOCKSHIFT; - ASSERT(c < SPA_MAXBLOCKSIZE >> SPA_MINBLOCKSHIFT); + VERIFY3U(c, <, SPA_MAXBLOCKSIZE >> SPA_MINBLOCKSHIFT); if (zio_use_uma) kmem_cache_free(zio_buf_cache[c], buf); else kmem_free(buf, size); } void zio_data_buf_free(void *buf, size_t size) { size_t c = (size - 1) >> SPA_MINBLOCKSHIFT; - ASSERT(c < SPA_MAXBLOCKSIZE >> SPA_MINBLOCKSHIFT); + VERIFY3U(c, <, SPA_MAXBLOCKSIZE >> SPA_MINBLOCKSHIFT); if (zio_use_uma) kmem_cache_free(zio_data_buf_cache[c], buf); else kmem_free(buf, size); } /* * ========================================================================== * Push and pop I/O transform buffers * ========================================================================== */ static void zio_push_transform(zio_t *zio, void *data, uint64_t size, uint64_t bufsize, zio_transform_func_t *transform) { zio_transform_t *zt = kmem_alloc(sizeof (zio_transform_t), KM_SLEEP); zt->zt_orig_data = zio->io_data; zt->zt_orig_size = zio->io_size; zt->zt_bufsize = bufsize; zt->zt_transform = transform; zt->zt_next = zio->io_transform_stack; zio->io_transform_stack = zt; zio->io_data = data; zio->io_size = size; } static void zio_pop_transforms(zio_t *zio) { zio_transform_t *zt; while ((zt = zio->io_transform_stack) != NULL) { if (zt->zt_transform != NULL) zt->zt_transform(zio, zt->zt_orig_data, zt->zt_orig_size); if (zt->zt_bufsize != 0) zio_buf_free(zio->io_data, zt->zt_bufsize); zio->io_data = zt->zt_orig_data; zio->io_size = zt->zt_orig_size; zio->io_transform_stack = zt->zt_next; kmem_free(zt, sizeof (zio_transform_t)); } } /* * ========================================================================== * I/O transform callbacks for subblocks and decompression * ========================================================================== */ static void zio_subblock(zio_t *zio, void *data, uint64_t size) { ASSERT(zio->io_size > size); if (zio->io_type == ZIO_TYPE_READ) bcopy(zio->io_data, data, size); } static void zio_decompress(zio_t *zio, void *data, uint64_t size) { if (zio->io_error == 0 && zio_decompress_data(BP_GET_COMPRESS(zio->io_bp), zio->io_data, data, zio->io_size, size) != 0) zio->io_error = SET_ERROR(EIO); } /* * ========================================================================== * I/O parent/child relationships and pipeline interlocks * ========================================================================== */ /* * NOTE - Callers to zio_walk_parents() and zio_walk_children must * continue calling these functions until they return NULL. * Otherwise, the next caller will pick up the list walk in * some indeterminate state. (Otherwise every caller would * have to pass in a cookie to keep the state represented by * io_walk_link, which gets annoying.) */ zio_t * zio_walk_parents(zio_t *cio) { zio_link_t *zl = cio->io_walk_link; list_t *pl = &cio->io_parent_list; zl = (zl == NULL) ? list_head(pl) : list_next(pl, zl); cio->io_walk_link = zl; if (zl == NULL) return (NULL); ASSERT(zl->zl_child == cio); return (zl->zl_parent); } zio_t * zio_walk_children(zio_t *pio) { zio_link_t *zl = pio->io_walk_link; list_t *cl = &pio->io_child_list; zl = (zl == NULL) ? list_head(cl) : list_next(cl, zl); pio->io_walk_link = zl; if (zl == NULL) return (NULL); ASSERT(zl->zl_parent == pio); return (zl->zl_child); } zio_t * zio_unique_parent(zio_t *cio) { zio_t *pio = zio_walk_parents(cio); VERIFY(zio_walk_parents(cio) == NULL); return (pio); } void zio_add_child(zio_t *pio, zio_t *cio) { zio_link_t *zl = kmem_cache_alloc(zio_link_cache, KM_SLEEP); /* * Logical I/Os can have logical, gang, or vdev children. * Gang I/Os can have gang or vdev children. * Vdev I/Os can only have vdev children. * The following ASSERT captures all of these constraints. */ ASSERT(cio->io_child_type <= pio->io_child_type); zl->zl_parent = pio; zl->zl_child = cio; mutex_enter(&cio->io_lock); mutex_enter(&pio->io_lock); ASSERT(pio->io_state[ZIO_WAIT_DONE] == 0); for (int w = 0; w < ZIO_WAIT_TYPES; w++) pio->io_children[cio->io_child_type][w] += !cio->io_state[w]; list_insert_head(&pio->io_child_list, zl); list_insert_head(&cio->io_parent_list, zl); pio->io_child_count++; cio->io_parent_count++; mutex_exit(&pio->io_lock); mutex_exit(&cio->io_lock); } static void zio_remove_child(zio_t *pio, zio_t *cio, zio_link_t *zl) { ASSERT(zl->zl_parent == pio); ASSERT(zl->zl_child == cio); mutex_enter(&cio->io_lock); mutex_enter(&pio->io_lock); list_remove(&pio->io_child_list, zl); list_remove(&cio->io_parent_list, zl); pio->io_child_count--; cio->io_parent_count--; mutex_exit(&pio->io_lock); mutex_exit(&cio->io_lock); kmem_cache_free(zio_link_cache, zl); } static boolean_t zio_wait_for_children(zio_t *zio, enum zio_child child, enum zio_wait_type wait) { uint64_t *countp = &zio->io_children[child][wait]; boolean_t waiting = B_FALSE; mutex_enter(&zio->io_lock); ASSERT(zio->io_stall == NULL); if (*countp != 0) { zio->io_stage >>= 1; zio->io_stall = countp; waiting = B_TRUE; } mutex_exit(&zio->io_lock); return (waiting); } static void zio_notify_parent(zio_t *pio, zio_t *zio, enum zio_wait_type wait) { uint64_t *countp = &pio->io_children[zio->io_child_type][wait]; int *errorp = &pio->io_child_error[zio->io_child_type]; mutex_enter(&pio->io_lock); if (zio->io_error && !(zio->io_flags & ZIO_FLAG_DONT_PROPAGATE)) *errorp = zio_worst_error(*errorp, zio->io_error); pio->io_reexecute |= zio->io_reexecute; ASSERT3U(*countp, >, 0); (*countp)--; if (*countp == 0 && pio->io_stall == countp) { pio->io_stall = NULL; mutex_exit(&pio->io_lock); zio_execute(pio); } else { mutex_exit(&pio->io_lock); } } static void zio_inherit_child_errors(zio_t *zio, enum zio_child c) { if (zio->io_child_error[c] != 0 && zio->io_error == 0) zio->io_error = zio->io_child_error[c]; } /* * ========================================================================== * Create the various types of I/O (read, write, free, etc) * ========================================================================== */ static zio_t * zio_create(zio_t *pio, spa_t *spa, uint64_t txg, const blkptr_t *bp, void *data, uint64_t size, zio_done_func_t *done, void *private, zio_type_t type, zio_priority_t priority, enum zio_flag flags, vdev_t *vd, uint64_t offset, const zbookmark_phys_t *zb, enum zio_stage stage, enum zio_stage pipeline) { zio_t *zio; ASSERT3U(type == ZIO_TYPE_FREE || size, <=, SPA_MAXBLOCKSIZE); ASSERT(P2PHASE(size, SPA_MINBLOCKSIZE) == 0); ASSERT(P2PHASE(offset, SPA_MINBLOCKSIZE) == 0); ASSERT(!vd || spa_config_held(spa, SCL_STATE_ALL, RW_READER)); ASSERT(!bp || !(flags & ZIO_FLAG_CONFIG_WRITER)); ASSERT(vd || stage == ZIO_STAGE_OPEN); zio = kmem_cache_alloc(zio_cache, KM_SLEEP); bzero(zio, sizeof (zio_t)); mutex_init(&zio->io_lock, NULL, MUTEX_DEFAULT, NULL); cv_init(&zio->io_cv, NULL, CV_DEFAULT, NULL); list_create(&zio->io_parent_list, sizeof (zio_link_t), offsetof(zio_link_t, zl_parent_node)); list_create(&zio->io_child_list, sizeof (zio_link_t), offsetof(zio_link_t, zl_child_node)); if (vd != NULL) zio->io_child_type = ZIO_CHILD_VDEV; else if (flags & ZIO_FLAG_GANG_CHILD) zio->io_child_type = ZIO_CHILD_GANG; else if (flags & ZIO_FLAG_DDT_CHILD) zio->io_child_type = ZIO_CHILD_DDT; else zio->io_child_type = ZIO_CHILD_LOGICAL; if (bp != NULL) { zio->io_bp = (blkptr_t *)bp; zio->io_bp_copy = *bp; zio->io_bp_orig = *bp; if (type != ZIO_TYPE_WRITE || zio->io_child_type == ZIO_CHILD_DDT) zio->io_bp = &zio->io_bp_copy; /* so caller can free */ if (zio->io_child_type == ZIO_CHILD_LOGICAL) zio->io_logical = zio; if (zio->io_child_type > ZIO_CHILD_GANG && BP_IS_GANG(bp)) pipeline |= ZIO_GANG_STAGES; } zio->io_spa = spa; zio->io_txg = txg; zio->io_done = done; zio->io_private = private; zio->io_type = type; zio->io_priority = priority; zio->io_vd = vd; zio->io_offset = offset; zio->io_orig_data = zio->io_data = data; zio->io_orig_size = zio->io_size = size; zio->io_orig_flags = zio->io_flags = flags; zio->io_orig_stage = zio->io_stage = stage; zio->io_orig_pipeline = zio->io_pipeline = pipeline; zio->io_state[ZIO_WAIT_READY] = (stage >= ZIO_STAGE_READY); zio->io_state[ZIO_WAIT_DONE] = (stage >= ZIO_STAGE_DONE); if (zb != NULL) zio->io_bookmark = *zb; if (pio != NULL) { if (zio->io_logical == NULL) zio->io_logical = pio->io_logical; if (zio->io_child_type == ZIO_CHILD_GANG) zio->io_gang_leader = pio->io_gang_leader; zio_add_child(pio, zio); } return (zio); } static void zio_destroy(zio_t *zio) { list_destroy(&zio->io_parent_list); list_destroy(&zio->io_child_list); mutex_destroy(&zio->io_lock); cv_destroy(&zio->io_cv); kmem_cache_free(zio_cache, zio); } zio_t * zio_null(zio_t *pio, spa_t *spa, vdev_t *vd, zio_done_func_t *done, void *private, enum zio_flag flags) { zio_t *zio; zio = zio_create(pio, spa, 0, NULL, NULL, 0, done, private, ZIO_TYPE_NULL, ZIO_PRIORITY_NOW, flags, vd, 0, NULL, ZIO_STAGE_OPEN, ZIO_INTERLOCK_PIPELINE); return (zio); } zio_t * zio_root(spa_t *spa, zio_done_func_t *done, void *private, enum zio_flag flags) { return (zio_null(NULL, spa, NULL, done, private, flags)); } +void +zfs_blkptr_verify(spa_t *spa, const blkptr_t *bp) +{ + if (!DMU_OT_IS_VALID(BP_GET_TYPE(bp))) { + zfs_panic_recover("blkptr at %p has invalid TYPE %llu", + bp, (longlong_t)BP_GET_TYPE(bp)); + } + if (BP_GET_CHECKSUM(bp) >= ZIO_CHECKSUM_FUNCTIONS || + BP_GET_CHECKSUM(bp) <= ZIO_CHECKSUM_ON) { + zfs_panic_recover("blkptr at %p has invalid CHECKSUM %llu", + bp, (longlong_t)BP_GET_CHECKSUM(bp)); + } + if (BP_GET_COMPRESS(bp) >= ZIO_COMPRESS_FUNCTIONS || + BP_GET_COMPRESS(bp) <= ZIO_COMPRESS_ON) { + zfs_panic_recover("blkptr at %p has invalid COMPRESS %llu", + bp, (longlong_t)BP_GET_COMPRESS(bp)); + } + if (BP_GET_LSIZE(bp) > SPA_MAXBLOCKSIZE) { + zfs_panic_recover("blkptr at %p has invalid LSIZE %llu", + bp, (longlong_t)BP_GET_LSIZE(bp)); + } + if (BP_GET_PSIZE(bp) > SPA_MAXBLOCKSIZE) { + zfs_panic_recover("blkptr at %p has invalid PSIZE %llu", + bp, (longlong_t)BP_GET_PSIZE(bp)); + } + + if (BP_IS_EMBEDDED(bp)) { + if (BPE_GET_ETYPE(bp) > NUM_BP_EMBEDDED_TYPES) { + zfs_panic_recover("blkptr at %p has invalid ETYPE %llu", + bp, (longlong_t)BPE_GET_ETYPE(bp)); + } + } + + /* + * Pool-specific checks. + * + * Note: it would be nice to verify that the blk_birth and + * BP_PHYSICAL_BIRTH() are not too large. However, spa_freeze() + * allows the birth time of log blocks (and dmu_sync()-ed blocks + * that are in the log) to be arbitrarily large. + */ + for (int i = 0; i < BP_GET_NDVAS(bp); i++) { + uint64_t vdevid = DVA_GET_VDEV(&bp->blk_dva[i]); + if (vdevid >= spa->spa_root_vdev->vdev_children) { + zfs_panic_recover("blkptr at %p DVA %u has invalid " + "VDEV %llu", + bp, i, (longlong_t)vdevid); + } + vdev_t *vd = spa->spa_root_vdev->vdev_child[vdevid]; + if (vd == NULL) { + zfs_panic_recover("blkptr at %p DVA %u has invalid " + "VDEV %llu", + bp, i, (longlong_t)vdevid); + } + if (vd->vdev_ops == &vdev_hole_ops) { + zfs_panic_recover("blkptr at %p DVA %u has hole " + "VDEV %llu", + bp, i, (longlong_t)vdevid); + + } + if (vd->vdev_ops == &vdev_missing_ops) { + /* + * "missing" vdevs are valid during import, but we + * don't have their detailed info (e.g. asize), so + * we can't perform any more checks on them. + */ + continue; + } + uint64_t offset = DVA_GET_OFFSET(&bp->blk_dva[i]); + uint64_t asize = DVA_GET_ASIZE(&bp->blk_dva[i]); + if (BP_IS_GANG(bp)) + asize = vdev_psize_to_asize(vd, SPA_GANGBLOCKSIZE); + if (offset + asize > vd->vdev_asize) { + zfs_panic_recover("blkptr at %p DVA %u has invalid " + "OFFSET %llu", + bp, i, (longlong_t)offset); + } + } +} + zio_t * zio_read(zio_t *pio, spa_t *spa, const blkptr_t *bp, void *data, uint64_t size, zio_done_func_t *done, void *private, zio_priority_t priority, enum zio_flag flags, const zbookmark_phys_t *zb) { zio_t *zio; + zfs_blkptr_verify(spa, bp); + zio = zio_create(pio, spa, BP_PHYSICAL_BIRTH(bp), bp, data, size, done, private, ZIO_TYPE_READ, priority, flags, NULL, 0, zb, ZIO_STAGE_OPEN, (flags & ZIO_FLAG_DDT_CHILD) ? ZIO_DDT_CHILD_READ_PIPELINE : ZIO_READ_PIPELINE); return (zio); } zio_t * zio_write(zio_t *pio, spa_t *spa, uint64_t txg, blkptr_t *bp, void *data, uint64_t size, const zio_prop_t *zp, zio_done_func_t *ready, zio_done_func_t *physdone, zio_done_func_t *done, void *private, zio_priority_t priority, enum zio_flag flags, const zbookmark_phys_t *zb) { zio_t *zio; ASSERT(zp->zp_checksum >= ZIO_CHECKSUM_OFF && zp->zp_checksum < ZIO_CHECKSUM_FUNCTIONS && zp->zp_compress >= ZIO_COMPRESS_OFF && zp->zp_compress < ZIO_COMPRESS_FUNCTIONS && DMU_OT_IS_VALID(zp->zp_type) && zp->zp_level < 32 && zp->zp_copies > 0 && zp->zp_copies <= spa_max_replication(spa)); zio = zio_create(pio, spa, txg, bp, data, size, done, private, ZIO_TYPE_WRITE, priority, flags, NULL, 0, zb, ZIO_STAGE_OPEN, (flags & ZIO_FLAG_DDT_CHILD) ? ZIO_DDT_CHILD_WRITE_PIPELINE : ZIO_WRITE_PIPELINE); zio->io_ready = ready; zio->io_physdone = physdone; zio->io_prop = *zp; /* * Data can be NULL if we are going to call zio_write_override() to * provide the already-allocated BP. But we may need the data to * verify a dedup hit (if requested). In this case, don't try to * dedup (just take the already-allocated BP verbatim). */ if (data == NULL && zio->io_prop.zp_dedup_verify) { zio->io_prop.zp_dedup = zio->io_prop.zp_dedup_verify = B_FALSE; } return (zio); } zio_t * zio_rewrite(zio_t *pio, spa_t *spa, uint64_t txg, blkptr_t *bp, void *data, uint64_t size, zio_done_func_t *done, void *private, zio_priority_t priority, enum zio_flag flags, zbookmark_phys_t *zb) { zio_t *zio; zio = zio_create(pio, spa, txg, bp, data, size, done, private, ZIO_TYPE_WRITE, priority, flags, NULL, 0, zb, ZIO_STAGE_OPEN, ZIO_REWRITE_PIPELINE); return (zio); } void zio_write_override(zio_t *zio, blkptr_t *bp, int copies, boolean_t nopwrite) { ASSERT(zio->io_type == ZIO_TYPE_WRITE); ASSERT(zio->io_child_type == ZIO_CHILD_LOGICAL); ASSERT(zio->io_stage == ZIO_STAGE_OPEN); ASSERT(zio->io_txg == spa_syncing_txg(zio->io_spa)); /* * We must reset the io_prop to match the values that existed * when the bp was first written by dmu_sync() keeping in mind * that nopwrite and dedup are mutually exclusive. */ zio->io_prop.zp_dedup = nopwrite ? B_FALSE : zio->io_prop.zp_dedup; zio->io_prop.zp_nopwrite = nopwrite; zio->io_prop.zp_copies = copies; zio->io_bp_override = bp; } void zio_free(spa_t *spa, uint64_t txg, const blkptr_t *bp) { /* * The check for EMBEDDED is a performance optimization. We * process the free here (by ignoring it) rather than * putting it on the list and then processing it in zio_free_sync(). */ if (BP_IS_EMBEDDED(bp)) return; metaslab_check_free(spa, bp); /* * Frees that are for the currently-syncing txg, are not going to be * deferred, and which will not need to do a read (i.e. not GANG or * DEDUP), can be processed immediately. Otherwise, put them on the * in-memory list for later processing. */ if (zfs_trim_enabled || BP_IS_GANG(bp) || BP_GET_DEDUP(bp) || txg != spa->spa_syncing_txg || spa_sync_pass(spa) >= zfs_sync_pass_deferred_free) { bplist_append(&spa->spa_free_bplist[txg & TXG_MASK], bp); } else { VERIFY0(zio_wait(zio_free_sync(NULL, spa, txg, bp, BP_GET_PSIZE(bp), 0))); } } zio_t * zio_free_sync(zio_t *pio, spa_t *spa, uint64_t txg, const blkptr_t *bp, uint64_t size, enum zio_flag flags) { zio_t *zio; enum zio_stage stage = ZIO_FREE_PIPELINE; ASSERT(!BP_IS_HOLE(bp)); ASSERT(spa_syncing_txg(spa) == txg); ASSERT(spa_sync_pass(spa) < zfs_sync_pass_deferred_free); if (BP_IS_EMBEDDED(bp)) return (zio_null(pio, spa, NULL, NULL, NULL, 0)); metaslab_check_free(spa, bp); arc_freed(spa, bp); if (zfs_trim_enabled) stage |= ZIO_STAGE_ISSUE_ASYNC | ZIO_STAGE_VDEV_IO_START | ZIO_STAGE_VDEV_IO_ASSESS; /* * GANG and DEDUP blocks can induce a read (for the gang block header, * or the DDT), so issue them asynchronously so that this thread is * not tied up. */ else if (BP_IS_GANG(bp) || BP_GET_DEDUP(bp)) stage |= ZIO_STAGE_ISSUE_ASYNC; flags |= ZIO_FLAG_DONT_QUEUE; zio = zio_create(pio, spa, txg, bp, NULL, size, NULL, NULL, ZIO_TYPE_FREE, ZIO_PRIORITY_NOW, flags, NULL, 0, NULL, ZIO_STAGE_OPEN, stage); return (zio); } zio_t * zio_claim(zio_t *pio, spa_t *spa, uint64_t txg, const blkptr_t *bp, zio_done_func_t *done, void *private, enum zio_flag flags) { zio_t *zio; dprintf_bp(bp, "claiming in txg %llu", txg); if (BP_IS_EMBEDDED(bp)) return (zio_null(pio, spa, NULL, NULL, NULL, 0)); /* * A claim is an allocation of a specific block. Claims are needed * to support immediate writes in the intent log. The issue is that * immediate writes contain committed data, but in a txg that was * *not* committed. Upon opening the pool after an unclean shutdown, * the intent log claims all blocks that contain immediate write data * so that the SPA knows they're in use. * * All claims *must* be resolved in the first txg -- before the SPA * starts allocating blocks -- so that nothing is allocated twice. * If txg == 0 we just verify that the block is claimable. */ ASSERT3U(spa->spa_uberblock.ub_rootbp.blk_birth, <, spa_first_txg(spa)); ASSERT(txg == spa_first_txg(spa) || txg == 0); ASSERT(!BP_GET_DEDUP(bp) || !spa_writeable(spa)); /* zdb(1M) */ zio = zio_create(pio, spa, txg, bp, NULL, BP_GET_PSIZE(bp), done, private, ZIO_TYPE_CLAIM, ZIO_PRIORITY_NOW, flags, NULL, 0, NULL, ZIO_STAGE_OPEN, ZIO_CLAIM_PIPELINE); return (zio); } zio_t * zio_ioctl(zio_t *pio, spa_t *spa, vdev_t *vd, int cmd, uint64_t offset, uint64_t size, zio_done_func_t *done, void *private, zio_priority_t priority, enum zio_flag flags) { zio_t *zio; int c; if (vd->vdev_children == 0) { zio = zio_create(pio, spa, 0, NULL, NULL, size, done, private, ZIO_TYPE_IOCTL, priority, flags, vd, offset, NULL, ZIO_STAGE_OPEN, ZIO_IOCTL_PIPELINE); zio->io_cmd = cmd; } else { zio = zio_null(pio, spa, NULL, NULL, NULL, flags); for (c = 0; c < vd->vdev_children; c++) zio_nowait(zio_ioctl(zio, spa, vd->vdev_child[c], cmd, offset, size, done, private, priority, flags)); } return (zio); } zio_t * zio_read_phys(zio_t *pio, vdev_t *vd, uint64_t offset, uint64_t size, void *data, int checksum, zio_done_func_t *done, void *private, zio_priority_t priority, enum zio_flag flags, boolean_t labels) { zio_t *zio; ASSERT(vd->vdev_children == 0); ASSERT(!labels || offset + size <= VDEV_LABEL_START_SIZE || offset >= vd->vdev_psize - VDEV_LABEL_END_SIZE); ASSERT3U(offset + size, <=, vd->vdev_psize); zio = zio_create(pio, vd->vdev_spa, 0, NULL, data, size, done, private, ZIO_TYPE_READ, priority, flags | ZIO_FLAG_PHYSICAL, vd, offset, NULL, ZIO_STAGE_OPEN, ZIO_READ_PHYS_PIPELINE); zio->io_prop.zp_checksum = checksum; return (zio); } zio_t * zio_write_phys(zio_t *pio, vdev_t *vd, uint64_t offset, uint64_t size, void *data, int checksum, zio_done_func_t *done, void *private, zio_priority_t priority, enum zio_flag flags, boolean_t labels) { zio_t *zio; ASSERT(vd->vdev_children == 0); ASSERT(!labels || offset + size <= VDEV_LABEL_START_SIZE || offset >= vd->vdev_psize - VDEV_LABEL_END_SIZE); ASSERT3U(offset + size, <=, vd->vdev_psize); zio = zio_create(pio, vd->vdev_spa, 0, NULL, data, size, done, private, ZIO_TYPE_WRITE, priority, flags | ZIO_FLAG_PHYSICAL, vd, offset, NULL, ZIO_STAGE_OPEN, ZIO_WRITE_PHYS_PIPELINE); zio->io_prop.zp_checksum = checksum; if (zio_checksum_table[checksum].ci_eck) { /* * zec checksums are necessarily destructive -- they modify * the end of the write buffer to hold the verifier/checksum. * Therefore, we must make a local copy in case the data is * being written to multiple places in parallel. */ void *wbuf = zio_buf_alloc(size); bcopy(data, wbuf, size); zio_push_transform(zio, wbuf, size, size, NULL); } return (zio); } /* * Create a child I/O to do some work for us. */ zio_t * zio_vdev_child_io(zio_t *pio, blkptr_t *bp, vdev_t *vd, uint64_t offset, void *data, uint64_t size, int type, zio_priority_t priority, enum zio_flag flags, zio_done_func_t *done, void *private) { enum zio_stage pipeline = ZIO_VDEV_CHILD_PIPELINE; zio_t *zio; ASSERT(vd->vdev_parent == (pio->io_vd ? pio->io_vd : pio->io_spa->spa_root_vdev)); if (type == ZIO_TYPE_READ && bp != NULL) { /* * If we have the bp, then the child should perform the * checksum and the parent need not. This pushes error * detection as close to the leaves as possible and * eliminates redundant checksums in the interior nodes. */ pipeline |= ZIO_STAGE_CHECKSUM_VERIFY; pio->io_pipeline &= ~ZIO_STAGE_CHECKSUM_VERIFY; } /* Not all IO types require vdev io done stage e.g. free */ if (!(pio->io_pipeline & ZIO_STAGE_VDEV_IO_DONE)) pipeline &= ~ZIO_STAGE_VDEV_IO_DONE; if (vd->vdev_children == 0) offset += VDEV_LABEL_START_SIZE; flags |= ZIO_VDEV_CHILD_FLAGS(pio) | ZIO_FLAG_DONT_PROPAGATE; /* * If we've decided to do a repair, the write is not speculative -- * even if the original read was. */ if (flags & ZIO_FLAG_IO_REPAIR) flags &= ~ZIO_FLAG_SPECULATIVE; zio = zio_create(pio, pio->io_spa, pio->io_txg, bp, data, size, done, private, type, priority, flags, vd, offset, &pio->io_bookmark, ZIO_STAGE_VDEV_IO_START >> 1, pipeline); zio->io_physdone = pio->io_physdone; if (vd->vdev_ops->vdev_op_leaf && zio->io_logical != NULL) zio->io_logical->io_phys_children++; return (zio); } zio_t * zio_vdev_delegated_io(vdev_t *vd, uint64_t offset, void *data, uint64_t size, int type, zio_priority_t priority, enum zio_flag flags, zio_done_func_t *done, void *private) { zio_t *zio; ASSERT(vd->vdev_ops->vdev_op_leaf); zio = zio_create(NULL, vd->vdev_spa, 0, NULL, data, size, done, private, type, priority, flags | ZIO_FLAG_CANFAIL | ZIO_FLAG_DONT_RETRY | ZIO_FLAG_DELEGATED, vd, offset, NULL, ZIO_STAGE_VDEV_IO_START >> 1, ZIO_VDEV_CHILD_PIPELINE); return (zio); } void zio_flush(zio_t *zio, vdev_t *vd) { zio_nowait(zio_ioctl(zio, zio->io_spa, vd, DKIOCFLUSHWRITECACHE, 0, 0, NULL, NULL, ZIO_PRIORITY_NOW, ZIO_FLAG_CANFAIL | ZIO_FLAG_DONT_PROPAGATE | ZIO_FLAG_DONT_RETRY)); } zio_t * zio_trim(zio_t *zio, spa_t *spa, vdev_t *vd, uint64_t offset, uint64_t size) { ASSERT(vd->vdev_ops->vdev_op_leaf); return (zio_create(zio, spa, 0, NULL, NULL, size, NULL, NULL, ZIO_TYPE_FREE, ZIO_PRIORITY_TRIM, ZIO_FLAG_DONT_AGGREGATE | ZIO_FLAG_CANFAIL | ZIO_FLAG_DONT_PROPAGATE | ZIO_FLAG_DONT_RETRY, vd, offset, NULL, ZIO_STAGE_OPEN, ZIO_FREE_PHYS_PIPELINE)); } void zio_shrink(zio_t *zio, uint64_t size) { ASSERT(zio->io_executor == NULL); ASSERT(zio->io_orig_size == zio->io_size); ASSERT(size <= zio->io_size); /* * We don't shrink for raidz because of problems with the * reconstruction when reading back less than the block size. * Note, BP_IS_RAIDZ() assumes no compression. */ ASSERT(BP_GET_COMPRESS(zio->io_bp) == ZIO_COMPRESS_OFF); if (!BP_IS_RAIDZ(zio->io_bp)) zio->io_orig_size = zio->io_size = size; } /* * ========================================================================== * Prepare to read and write logical blocks * ========================================================================== */ static int zio_read_bp_init(zio_t *zio) { blkptr_t *bp = zio->io_bp; if (BP_GET_COMPRESS(bp) != ZIO_COMPRESS_OFF && zio->io_child_type == ZIO_CHILD_LOGICAL && !(zio->io_flags & ZIO_FLAG_RAW)) { uint64_t psize = BP_IS_EMBEDDED(bp) ? BPE_GET_PSIZE(bp) : BP_GET_PSIZE(bp); void *cbuf = zio_buf_alloc(psize); zio_push_transform(zio, cbuf, psize, psize, zio_decompress); } if (BP_IS_EMBEDDED(bp) && BPE_GET_ETYPE(bp) == BP_EMBEDDED_TYPE_DATA) { zio->io_pipeline = ZIO_INTERLOCK_PIPELINE; decode_embedded_bp_compressed(bp, zio->io_data); } else { ASSERT(!BP_IS_EMBEDDED(bp)); } if (!DMU_OT_IS_METADATA(BP_GET_TYPE(bp)) && BP_GET_LEVEL(bp) == 0) zio->io_flags |= ZIO_FLAG_DONT_CACHE; if (BP_GET_TYPE(bp) == DMU_OT_DDT_ZAP) zio->io_flags |= ZIO_FLAG_DONT_CACHE; if (BP_GET_DEDUP(bp) && zio->io_child_type == ZIO_CHILD_LOGICAL) zio->io_pipeline = ZIO_DDT_READ_PIPELINE; return (ZIO_PIPELINE_CONTINUE); } static int zio_write_bp_init(zio_t *zio) { spa_t *spa = zio->io_spa; zio_prop_t *zp = &zio->io_prop; enum zio_compress compress = zp->zp_compress; blkptr_t *bp = zio->io_bp; uint64_t lsize = zio->io_size; uint64_t psize = lsize; int pass = 1; /* * If our children haven't all reached the ready stage, * wait for them and then repeat this pipeline stage. */ if (zio_wait_for_children(zio, ZIO_CHILD_GANG, ZIO_WAIT_READY) || zio_wait_for_children(zio, ZIO_CHILD_LOGICAL, ZIO_WAIT_READY)) return (ZIO_PIPELINE_STOP); if (!IO_IS_ALLOCATING(zio)) return (ZIO_PIPELINE_CONTINUE); ASSERT(zio->io_child_type != ZIO_CHILD_DDT); if (zio->io_bp_override) { ASSERT(bp->blk_birth != zio->io_txg); ASSERT(BP_GET_DEDUP(zio->io_bp_override) == 0); *bp = *zio->io_bp_override; zio->io_pipeline = ZIO_INTERLOCK_PIPELINE; if (BP_IS_EMBEDDED(bp)) return (ZIO_PIPELINE_CONTINUE); /* * If we've been overridden and nopwrite is set then * set the flag accordingly to indicate that a nopwrite * has already occurred. */ if (!BP_IS_HOLE(bp) && zp->zp_nopwrite) { ASSERT(!zp->zp_dedup); zio->io_flags |= ZIO_FLAG_NOPWRITE; return (ZIO_PIPELINE_CONTINUE); } ASSERT(!zp->zp_nopwrite); if (BP_IS_HOLE(bp) || !zp->zp_dedup) return (ZIO_PIPELINE_CONTINUE); ASSERT(zio_checksum_table[zp->zp_checksum].ci_dedup || zp->zp_dedup_verify); if (BP_GET_CHECKSUM(bp) == zp->zp_checksum) { BP_SET_DEDUP(bp, 1); zio->io_pipeline |= ZIO_STAGE_DDT_WRITE; return (ZIO_PIPELINE_CONTINUE); } zio->io_bp_override = NULL; BP_ZERO(bp); } if (!BP_IS_HOLE(bp) && bp->blk_birth == zio->io_txg) { /* * We're rewriting an existing block, which means we're * working on behalf of spa_sync(). For spa_sync() to * converge, it must eventually be the case that we don't * have to allocate new blocks. But compression changes * the blocksize, which forces a reallocate, and makes * convergence take longer. Therefore, after the first * few passes, stop compressing to ensure convergence. */ pass = spa_sync_pass(spa); ASSERT(zio->io_txg == spa_syncing_txg(spa)); ASSERT(zio->io_child_type == ZIO_CHILD_LOGICAL); ASSERT(!BP_GET_DEDUP(bp)); if (pass >= zfs_sync_pass_dont_compress) compress = ZIO_COMPRESS_OFF; /* Make sure someone doesn't change their mind on overwrites */ ASSERT(BP_IS_EMBEDDED(bp) || MIN(zp->zp_copies + BP_IS_GANG(bp), spa_max_replication(spa)) == BP_GET_NDVAS(bp)); } if (compress != ZIO_COMPRESS_OFF) { void *cbuf = zio_buf_alloc(lsize); psize = zio_compress_data(compress, zio->io_data, cbuf, lsize); if (psize == 0 || psize == lsize) { compress = ZIO_COMPRESS_OFF; zio_buf_free(cbuf, lsize); } else if (!zp->zp_dedup && psize <= BPE_PAYLOAD_SIZE && zp->zp_level == 0 && !DMU_OT_HAS_FILL(zp->zp_type) && spa_feature_is_enabled(spa, SPA_FEATURE_EMBEDDED_DATA)) { encode_embedded_bp_compressed(bp, cbuf, compress, lsize, psize); BPE_SET_ETYPE(bp, BP_EMBEDDED_TYPE_DATA); BP_SET_TYPE(bp, zio->io_prop.zp_type); BP_SET_LEVEL(bp, zio->io_prop.zp_level); zio_buf_free(cbuf, lsize); bp->blk_birth = zio->io_txg; zio->io_pipeline = ZIO_INTERLOCK_PIPELINE; ASSERT(spa_feature_is_active(spa, SPA_FEATURE_EMBEDDED_DATA)); return (ZIO_PIPELINE_CONTINUE); } else { /* * Round up compressed size to MINBLOCKSIZE and * zero the tail. */ size_t rounded = P2ROUNDUP(psize, (size_t)SPA_MINBLOCKSIZE); if (rounded > psize) { bzero((char *)cbuf + psize, rounded - psize); psize = rounded; } if (psize == lsize) { compress = ZIO_COMPRESS_OFF; zio_buf_free(cbuf, lsize); } else { zio_push_transform(zio, cbuf, psize, lsize, NULL); } } } /* * The final pass of spa_sync() must be all rewrites, but the first * few passes offer a trade-off: allocating blocks defers convergence, * but newly allocated blocks are sequential, so they can be written * to disk faster. Therefore, we allow the first few passes of * spa_sync() to allocate new blocks, but force rewrites after that. * There should only be a handful of blocks after pass 1 in any case. */ if (!BP_IS_HOLE(bp) && bp->blk_birth == zio->io_txg && BP_GET_PSIZE(bp) == psize && pass >= zfs_sync_pass_rewrite) { ASSERT(psize != 0); enum zio_stage gang_stages = zio->io_pipeline & ZIO_GANG_STAGES; zio->io_pipeline = ZIO_REWRITE_PIPELINE | gang_stages; zio->io_flags |= ZIO_FLAG_IO_REWRITE; } else { BP_ZERO(bp); zio->io_pipeline = ZIO_WRITE_PIPELINE; } if (psize == 0) { if (zio->io_bp_orig.blk_birth != 0 && spa_feature_is_active(spa, SPA_FEATURE_HOLE_BIRTH)) { BP_SET_LSIZE(bp, lsize); BP_SET_TYPE(bp, zp->zp_type); BP_SET_LEVEL(bp, zp->zp_level); BP_SET_BIRTH(bp, zio->io_txg, 0); } zio->io_pipeline = ZIO_INTERLOCK_PIPELINE; } else { ASSERT(zp->zp_checksum != ZIO_CHECKSUM_GANG_HEADER); BP_SET_LSIZE(bp, lsize); BP_SET_TYPE(bp, zp->zp_type); BP_SET_LEVEL(bp, zp->zp_level); BP_SET_PSIZE(bp, psize); BP_SET_COMPRESS(bp, compress); BP_SET_CHECKSUM(bp, zp->zp_checksum); BP_SET_DEDUP(bp, zp->zp_dedup); BP_SET_BYTEORDER(bp, ZFS_HOST_BYTEORDER); if (zp->zp_dedup) { ASSERT(zio->io_child_type == ZIO_CHILD_LOGICAL); ASSERT(!(zio->io_flags & ZIO_FLAG_IO_REWRITE)); zio->io_pipeline = ZIO_DDT_WRITE_PIPELINE; } if (zp->zp_nopwrite) { ASSERT(zio->io_child_type == ZIO_CHILD_LOGICAL); ASSERT(!(zio->io_flags & ZIO_FLAG_IO_REWRITE)); zio->io_pipeline |= ZIO_STAGE_NOP_WRITE; } } return (ZIO_PIPELINE_CONTINUE); } static int zio_free_bp_init(zio_t *zio) { blkptr_t *bp = zio->io_bp; if (zio->io_child_type == ZIO_CHILD_LOGICAL) { if (BP_GET_DEDUP(bp)) zio->io_pipeline = ZIO_DDT_FREE_PIPELINE; } return (ZIO_PIPELINE_CONTINUE); } /* * ========================================================================== * Execute the I/O pipeline * ========================================================================== */ static void zio_taskq_dispatch(zio_t *zio, zio_taskq_type_t q, boolean_t cutinline) { spa_t *spa = zio->io_spa; zio_type_t t = zio->io_type; int flags = (cutinline ? TQ_FRONT : 0); ASSERT(q == ZIO_TASKQ_ISSUE || q == ZIO_TASKQ_INTERRUPT); /* * If we're a config writer or a probe, the normal issue and * interrupt threads may all be blocked waiting for the config lock. * In this case, select the otherwise-unused taskq for ZIO_TYPE_NULL. */ if (zio->io_flags & (ZIO_FLAG_CONFIG_WRITER | ZIO_FLAG_PROBE)) t = ZIO_TYPE_NULL; /* * A similar issue exists for the L2ARC write thread until L2ARC 2.0. */ if (t == ZIO_TYPE_WRITE && zio->io_vd && zio->io_vd->vdev_aux) t = ZIO_TYPE_NULL; /* * If this is a high priority I/O, then use the high priority taskq if * available. */ if (zio->io_priority == ZIO_PRIORITY_NOW && spa->spa_zio_taskq[t][q + 1].stqs_count != 0) q++; ASSERT3U(q, <, ZIO_TASKQ_TYPES); /* * NB: We are assuming that the zio can only be dispatched * to a single taskq at a time. It would be a grievous error * to dispatch the zio to another taskq at the same time. */ #if defined(illumos) || !defined(_KERNEL) ASSERT(zio->io_tqent.tqent_next == NULL); #else ASSERT(zio->io_tqent.tqent_task.ta_pending == 0); #endif spa_taskq_dispatch_ent(spa, t, q, (task_func_t *)zio_execute, zio, flags, &zio->io_tqent); } static boolean_t zio_taskq_member(zio_t *zio, zio_taskq_type_t q) { kthread_t *executor = zio->io_executor; spa_t *spa = zio->io_spa; for (zio_type_t t = 0; t < ZIO_TYPES; t++) { spa_taskqs_t *tqs = &spa->spa_zio_taskq[t][q]; uint_t i; for (i = 0; i < tqs->stqs_count; i++) { if (taskq_member(tqs->stqs_taskq[i], executor)) return (B_TRUE); } } return (B_FALSE); } static int zio_issue_async(zio_t *zio) { zio_taskq_dispatch(zio, ZIO_TASKQ_ISSUE, B_FALSE); return (ZIO_PIPELINE_STOP); } void zio_interrupt(zio_t *zio) { zio_taskq_dispatch(zio, ZIO_TASKQ_INTERRUPT, B_FALSE); } /* * Execute the I/O pipeline until one of the following occurs: * * (1) the I/O completes * (2) the pipeline stalls waiting for dependent child I/Os * (3) the I/O issues, so we're waiting for an I/O completion interrupt * (4) the I/O is delegated by vdev-level caching or aggregation * (5) the I/O is deferred due to vdev-level queueing * (6) the I/O is handed off to another thread. * * In all cases, the pipeline stops whenever there's no CPU work; it never * burns a thread in cv_wait(). * * There's no locking on io_stage because there's no legitimate way * for multiple threads to be attempting to process the same I/O. */ static zio_pipe_stage_t *zio_pipeline[]; void zio_execute(zio_t *zio) { zio->io_executor = curthread; while (zio->io_stage < ZIO_STAGE_DONE) { enum zio_stage pipeline = zio->io_pipeline; enum zio_stage stage = zio->io_stage; int rv; ASSERT(!MUTEX_HELD(&zio->io_lock)); ASSERT(ISP2(stage)); ASSERT(zio->io_stall == NULL); do { stage <<= 1; } while ((stage & pipeline) == 0); ASSERT(stage <= ZIO_STAGE_DONE); /* * If we are in interrupt context and this pipeline stage * will grab a config lock that is held across I/O, * or may wait for an I/O that needs an interrupt thread * to complete, issue async to avoid deadlock. * * For VDEV_IO_START, we cut in line so that the io will * be sent to disk promptly. */ if ((stage & ZIO_BLOCKING_STAGES) && zio->io_vd == NULL && zio_taskq_member(zio, ZIO_TASKQ_INTERRUPT)) { boolean_t cut = (stage == ZIO_STAGE_VDEV_IO_START) ? zio_requeue_io_start_cut_in_line : B_FALSE; zio_taskq_dispatch(zio, ZIO_TASKQ_ISSUE, cut); return; } zio->io_stage = stage; rv = zio_pipeline[highbit64(stage) - 1](zio); if (rv == ZIO_PIPELINE_STOP) return; ASSERT(rv == ZIO_PIPELINE_CONTINUE); } } /* * ========================================================================== * Initiate I/O, either sync or async * ========================================================================== */ int zio_wait(zio_t *zio) { int error; ASSERT(zio->io_stage == ZIO_STAGE_OPEN); ASSERT(zio->io_executor == NULL); zio->io_waiter = curthread; zio_execute(zio); mutex_enter(&zio->io_lock); while (zio->io_executor != NULL) cv_wait(&zio->io_cv, &zio->io_lock); mutex_exit(&zio->io_lock); error = zio->io_error; zio_destroy(zio); return (error); } void zio_nowait(zio_t *zio) { ASSERT(zio->io_executor == NULL); if (zio->io_child_type == ZIO_CHILD_LOGICAL && zio_unique_parent(zio) == NULL) { /* * This is a logical async I/O with no parent to wait for it. * We add it to the spa_async_root_zio "Godfather" I/O which * will ensure they complete prior to unloading the pool. */ spa_t *spa = zio->io_spa; zio_add_child(spa->spa_async_zio_root[CPU_SEQID], zio); } zio_execute(zio); } /* * ========================================================================== * Reexecute or suspend/resume failed I/O * ========================================================================== */ static void zio_reexecute(zio_t *pio) { zio_t *cio, *cio_next; ASSERT(pio->io_child_type == ZIO_CHILD_LOGICAL); ASSERT(pio->io_orig_stage == ZIO_STAGE_OPEN); ASSERT(pio->io_gang_leader == NULL); ASSERT(pio->io_gang_tree == NULL); pio->io_flags = pio->io_orig_flags; pio->io_stage = pio->io_orig_stage; pio->io_pipeline = pio->io_orig_pipeline; pio->io_reexecute = 0; pio->io_flags |= ZIO_FLAG_REEXECUTED; pio->io_error = 0; for (int w = 0; w < ZIO_WAIT_TYPES; w++) pio->io_state[w] = 0; for (int c = 0; c < ZIO_CHILD_TYPES; c++) pio->io_child_error[c] = 0; if (IO_IS_ALLOCATING(pio)) BP_ZERO(pio->io_bp); /* * As we reexecute pio's children, new children could be created. * New children go to the head of pio's io_child_list, however, * so we will (correctly) not reexecute them. The key is that * the remainder of pio's io_child_list, from 'cio_next' onward, * cannot be affected by any side effects of reexecuting 'cio'. */ for (cio = zio_walk_children(pio); cio != NULL; cio = cio_next) { cio_next = zio_walk_children(pio); mutex_enter(&pio->io_lock); for (int w = 0; w < ZIO_WAIT_TYPES; w++) pio->io_children[cio->io_child_type][w]++; mutex_exit(&pio->io_lock); zio_reexecute(cio); } /* * Now that all children have been reexecuted, execute the parent. * We don't reexecute "The Godfather" I/O here as it's the * responsibility of the caller to wait on him. */ if (!(pio->io_flags & ZIO_FLAG_GODFATHER)) zio_execute(pio); } void zio_suspend(spa_t *spa, zio_t *zio) { if (spa_get_failmode(spa) == ZIO_FAILURE_MODE_PANIC) fm_panic("Pool '%s' has encountered an uncorrectable I/O " "failure and the failure mode property for this pool " "is set to panic.", spa_name(spa)); zfs_ereport_post(FM_EREPORT_ZFS_IO_FAILURE, spa, NULL, NULL, 0, 0); mutex_enter(&spa->spa_suspend_lock); if (spa->spa_suspend_zio_root == NULL) spa->spa_suspend_zio_root = zio_root(spa, NULL, NULL, ZIO_FLAG_CANFAIL | ZIO_FLAG_SPECULATIVE | ZIO_FLAG_GODFATHER); spa->spa_suspended = B_TRUE; if (zio != NULL) { ASSERT(!(zio->io_flags & ZIO_FLAG_GODFATHER)); ASSERT(zio != spa->spa_suspend_zio_root); ASSERT(zio->io_child_type == ZIO_CHILD_LOGICAL); ASSERT(zio_unique_parent(zio) == NULL); ASSERT(zio->io_stage == ZIO_STAGE_DONE); zio_add_child(spa->spa_suspend_zio_root, zio); } mutex_exit(&spa->spa_suspend_lock); } int zio_resume(spa_t *spa) { zio_t *pio; /* * Reexecute all previously suspended i/o. */ mutex_enter(&spa->spa_suspend_lock); spa->spa_suspended = B_FALSE; cv_broadcast(&spa->spa_suspend_cv); pio = spa->spa_suspend_zio_root; spa->spa_suspend_zio_root = NULL; mutex_exit(&spa->spa_suspend_lock); if (pio == NULL) return (0); zio_reexecute(pio); return (zio_wait(pio)); } void zio_resume_wait(spa_t *spa) { mutex_enter(&spa->spa_suspend_lock); while (spa_suspended(spa)) cv_wait(&spa->spa_suspend_cv, &spa->spa_suspend_lock); mutex_exit(&spa->spa_suspend_lock); } /* * ========================================================================== * Gang blocks. * * A gang block is a collection of small blocks that looks to the DMU * like one large block. When zio_dva_allocate() cannot find a block * of the requested size, due to either severe fragmentation or the pool * being nearly full, it calls zio_write_gang_block() to construct the * block from smaller fragments. * * A gang block consists of a gang header (zio_gbh_phys_t) and up to * three (SPA_GBH_NBLKPTRS) gang members. The gang header is just like * an indirect block: it's an array of block pointers. It consumes * only one sector and hence is allocatable regardless of fragmentation. * The gang header's bps point to its gang members, which hold the data. * * Gang blocks are self-checksumming, using the bp's * as the verifier to ensure uniqueness of the SHA256 checksum. * Critically, the gang block bp's blk_cksum is the checksum of the data, * not the gang header. This ensures that data block signatures (needed for * deduplication) are independent of how the block is physically stored. * * Gang blocks can be nested: a gang member may itself be a gang block. * Thus every gang block is a tree in which root and all interior nodes are * gang headers, and the leaves are normal blocks that contain user data. * The root of the gang tree is called the gang leader. * * To perform any operation (read, rewrite, free, claim) on a gang block, * zio_gang_assemble() first assembles the gang tree (minus data leaves) * in the io_gang_tree field of the original logical i/o by recursively * reading the gang leader and all gang headers below it. This yields * an in-core tree containing the contents of every gang header and the * bps for every constituent of the gang block. * * With the gang tree now assembled, zio_gang_issue() just walks the gang tree * and invokes a callback on each bp. To free a gang block, zio_gang_issue() * calls zio_free_gang() -- a trivial wrapper around zio_free() -- for each bp. * zio_claim_gang() provides a similarly trivial wrapper for zio_claim(). * zio_read_gang() is a wrapper around zio_read() that omits reading gang * headers, since we already have those in io_gang_tree. zio_rewrite_gang() * performs a zio_rewrite() of the data or, for gang headers, a zio_rewrite() * of the gang header plus zio_checksum_compute() of the data to update the * gang header's blk_cksum as described above. * * The two-phase assemble/issue model solves the problem of partial failure -- * what if you'd freed part of a gang block but then couldn't read the * gang header for another part? Assembling the entire gang tree first * ensures that all the necessary gang header I/O has succeeded before * starting the actual work of free, claim, or write. Once the gang tree * is assembled, free and claim are in-memory operations that cannot fail. * * In the event that a gang write fails, zio_dva_unallocate() walks the * gang tree to immediately free (i.e. insert back into the space map) * everything we've allocated. This ensures that we don't get ENOSPC * errors during repeated suspend/resume cycles due to a flaky device. * * Gang rewrites only happen during sync-to-convergence. If we can't assemble * the gang tree, we won't modify the block, so we can safely defer the free * (knowing that the block is still intact). If we *can* assemble the gang * tree, then even if some of the rewrites fail, zio_dva_unallocate() will free * each constituent bp and we can allocate a new block on the next sync pass. * * In all cases, the gang tree allows complete recovery from partial failure. * ========================================================================== */ static zio_t * zio_read_gang(zio_t *pio, blkptr_t *bp, zio_gang_node_t *gn, void *data) { if (gn != NULL) return (pio); return (zio_read(pio, pio->io_spa, bp, data, BP_GET_PSIZE(bp), NULL, NULL, pio->io_priority, ZIO_GANG_CHILD_FLAGS(pio), &pio->io_bookmark)); } zio_t * zio_rewrite_gang(zio_t *pio, blkptr_t *bp, zio_gang_node_t *gn, void *data) { zio_t *zio; if (gn != NULL) { zio = zio_rewrite(pio, pio->io_spa, pio->io_txg, bp, gn->gn_gbh, SPA_GANGBLOCKSIZE, NULL, NULL, pio->io_priority, ZIO_GANG_CHILD_FLAGS(pio), &pio->io_bookmark); /* * As we rewrite each gang header, the pipeline will compute * a new gang block header checksum for it; but no one will * compute a new data checksum, so we do that here. The one * exception is the gang leader: the pipeline already computed * its data checksum because that stage precedes gang assembly. * (Presently, nothing actually uses interior data checksums; * this is just good hygiene.) */ if (gn != pio->io_gang_leader->io_gang_tree) { zio_checksum_compute(zio, BP_GET_CHECKSUM(bp), data, BP_GET_PSIZE(bp)); } /* * If we are here to damage data for testing purposes, * leave the GBH alone so that we can detect the damage. */ if (pio->io_gang_leader->io_flags & ZIO_FLAG_INDUCE_DAMAGE) zio->io_pipeline &= ~ZIO_VDEV_IO_STAGES; } else { zio = zio_rewrite(pio, pio->io_spa, pio->io_txg, bp, data, BP_GET_PSIZE(bp), NULL, NULL, pio->io_priority, ZIO_GANG_CHILD_FLAGS(pio), &pio->io_bookmark); } return (zio); } /* ARGSUSED */ zio_t * zio_free_gang(zio_t *pio, blkptr_t *bp, zio_gang_node_t *gn, void *data) { return (zio_free_sync(pio, pio->io_spa, pio->io_txg, bp, BP_IS_GANG(bp) ? SPA_GANGBLOCKSIZE : BP_GET_PSIZE(bp), ZIO_GANG_CHILD_FLAGS(pio))); } /* ARGSUSED */ zio_t * zio_claim_gang(zio_t *pio, blkptr_t *bp, zio_gang_node_t *gn, void *data) { return (zio_claim(pio, pio->io_spa, pio->io_txg, bp, NULL, NULL, ZIO_GANG_CHILD_FLAGS(pio))); } static zio_gang_issue_func_t *zio_gang_issue_func[ZIO_TYPES] = { NULL, zio_read_gang, zio_rewrite_gang, zio_free_gang, zio_claim_gang, NULL }; static void zio_gang_tree_assemble_done(zio_t *zio); static zio_gang_node_t * zio_gang_node_alloc(zio_gang_node_t **gnpp) { zio_gang_node_t *gn; ASSERT(*gnpp == NULL); gn = kmem_zalloc(sizeof (*gn), KM_SLEEP); gn->gn_gbh = zio_buf_alloc(SPA_GANGBLOCKSIZE); *gnpp = gn; return (gn); } static void zio_gang_node_free(zio_gang_node_t **gnpp) { zio_gang_node_t *gn = *gnpp; for (int g = 0; g < SPA_GBH_NBLKPTRS; g++) ASSERT(gn->gn_child[g] == NULL); zio_buf_free(gn->gn_gbh, SPA_GANGBLOCKSIZE); kmem_free(gn, sizeof (*gn)); *gnpp = NULL; } static void zio_gang_tree_free(zio_gang_node_t **gnpp) { zio_gang_node_t *gn = *gnpp; if (gn == NULL) return; for (int g = 0; g < SPA_GBH_NBLKPTRS; g++) zio_gang_tree_free(&gn->gn_child[g]); zio_gang_node_free(gnpp); } static void zio_gang_tree_assemble(zio_t *gio, blkptr_t *bp, zio_gang_node_t **gnpp) { zio_gang_node_t *gn = zio_gang_node_alloc(gnpp); ASSERT(gio->io_gang_leader == gio); ASSERT(BP_IS_GANG(bp)); zio_nowait(zio_read(gio, gio->io_spa, bp, gn->gn_gbh, SPA_GANGBLOCKSIZE, zio_gang_tree_assemble_done, gn, gio->io_priority, ZIO_GANG_CHILD_FLAGS(gio), &gio->io_bookmark)); } static void zio_gang_tree_assemble_done(zio_t *zio) { zio_t *gio = zio->io_gang_leader; zio_gang_node_t *gn = zio->io_private; blkptr_t *bp = zio->io_bp; ASSERT(gio == zio_unique_parent(zio)); ASSERT(zio->io_child_count == 0); if (zio->io_error) return; if (BP_SHOULD_BYTESWAP(bp)) byteswap_uint64_array(zio->io_data, zio->io_size); ASSERT(zio->io_data == gn->gn_gbh); ASSERT(zio->io_size == SPA_GANGBLOCKSIZE); ASSERT(gn->gn_gbh->zg_tail.zec_magic == ZEC_MAGIC); for (int g = 0; g < SPA_GBH_NBLKPTRS; g++) { blkptr_t *gbp = &gn->gn_gbh->zg_blkptr[g]; if (!BP_IS_GANG(gbp)) continue; zio_gang_tree_assemble(gio, gbp, &gn->gn_child[g]); } } static void zio_gang_tree_issue(zio_t *pio, zio_gang_node_t *gn, blkptr_t *bp, void *data) { zio_t *gio = pio->io_gang_leader; zio_t *zio; ASSERT(BP_IS_GANG(bp) == !!gn); ASSERT(BP_GET_CHECKSUM(bp) == BP_GET_CHECKSUM(gio->io_bp)); ASSERT(BP_GET_LSIZE(bp) == BP_GET_PSIZE(bp) || gn == gio->io_gang_tree); /* * If you're a gang header, your data is in gn->gn_gbh. * If you're a gang member, your data is in 'data' and gn == NULL. */ zio = zio_gang_issue_func[gio->io_type](pio, bp, gn, data); if (gn != NULL) { ASSERT(gn->gn_gbh->zg_tail.zec_magic == ZEC_MAGIC); for (int g = 0; g < SPA_GBH_NBLKPTRS; g++) { blkptr_t *gbp = &gn->gn_gbh->zg_blkptr[g]; if (BP_IS_HOLE(gbp)) continue; zio_gang_tree_issue(zio, gn->gn_child[g], gbp, data); data = (char *)data + BP_GET_PSIZE(gbp); } } if (gn == gio->io_gang_tree && gio->io_data != NULL) ASSERT3P((char *)gio->io_data + gio->io_size, ==, data); if (zio != pio) zio_nowait(zio); } static int zio_gang_assemble(zio_t *zio) { blkptr_t *bp = zio->io_bp; ASSERT(BP_IS_GANG(bp) && zio->io_gang_leader == NULL); ASSERT(zio->io_child_type > ZIO_CHILD_GANG); zio->io_gang_leader = zio; zio_gang_tree_assemble(zio, bp, &zio->io_gang_tree); return (ZIO_PIPELINE_CONTINUE); } static int zio_gang_issue(zio_t *zio) { blkptr_t *bp = zio->io_bp; if (zio_wait_for_children(zio, ZIO_CHILD_GANG, ZIO_WAIT_DONE)) return (ZIO_PIPELINE_STOP); ASSERT(BP_IS_GANG(bp) && zio->io_gang_leader == zio); ASSERT(zio->io_child_type > ZIO_CHILD_GANG); if (zio->io_child_error[ZIO_CHILD_GANG] == 0) zio_gang_tree_issue(zio, zio->io_gang_tree, bp, zio->io_data); else zio_gang_tree_free(&zio->io_gang_tree); zio->io_pipeline = ZIO_INTERLOCK_PIPELINE; return (ZIO_PIPELINE_CONTINUE); } static void zio_write_gang_member_ready(zio_t *zio) { zio_t *pio = zio_unique_parent(zio); zio_t *gio = zio->io_gang_leader; dva_t *cdva = zio->io_bp->blk_dva; dva_t *pdva = pio->io_bp->blk_dva; uint64_t asize; if (BP_IS_HOLE(zio->io_bp)) return; ASSERT(BP_IS_HOLE(&zio->io_bp_orig)); ASSERT(zio->io_child_type == ZIO_CHILD_GANG); ASSERT3U(zio->io_prop.zp_copies, ==, gio->io_prop.zp_copies); ASSERT3U(zio->io_prop.zp_copies, <=, BP_GET_NDVAS(zio->io_bp)); ASSERT3U(pio->io_prop.zp_copies, <=, BP_GET_NDVAS(pio->io_bp)); ASSERT3U(BP_GET_NDVAS(zio->io_bp), <=, BP_GET_NDVAS(pio->io_bp)); mutex_enter(&pio->io_lock); for (int d = 0; d < BP_GET_NDVAS(zio->io_bp); d++) { ASSERT(DVA_GET_GANG(&pdva[d])); asize = DVA_GET_ASIZE(&pdva[d]); asize += DVA_GET_ASIZE(&cdva[d]); DVA_SET_ASIZE(&pdva[d], asize); } mutex_exit(&pio->io_lock); } static int zio_write_gang_block(zio_t *pio) { spa_t *spa = pio->io_spa; blkptr_t *bp = pio->io_bp; zio_t *gio = pio->io_gang_leader; zio_t *zio; zio_gang_node_t *gn, **gnpp; zio_gbh_phys_t *gbh; uint64_t txg = pio->io_txg; uint64_t resid = pio->io_size; uint64_t lsize; int copies = gio->io_prop.zp_copies; int gbh_copies = MIN(copies + 1, spa_max_replication(spa)); zio_prop_t zp; int error; error = metaslab_alloc(spa, spa_normal_class(spa), SPA_GANGBLOCKSIZE, bp, gbh_copies, txg, pio == gio ? NULL : gio->io_bp, METASLAB_HINTBP_FAVOR | METASLAB_GANG_HEADER); if (error) { pio->io_error = error; return (ZIO_PIPELINE_CONTINUE); } if (pio == gio) { gnpp = &gio->io_gang_tree; } else { gnpp = pio->io_private; ASSERT(pio->io_ready == zio_write_gang_member_ready); } gn = zio_gang_node_alloc(gnpp); gbh = gn->gn_gbh; bzero(gbh, SPA_GANGBLOCKSIZE); /* * Create the gang header. */ zio = zio_rewrite(pio, spa, txg, bp, gbh, SPA_GANGBLOCKSIZE, NULL, NULL, pio->io_priority, ZIO_GANG_CHILD_FLAGS(pio), &pio->io_bookmark); /* * Create and nowait the gang children. */ for (int g = 0; resid != 0; resid -= lsize, g++) { lsize = P2ROUNDUP(resid / (SPA_GBH_NBLKPTRS - g), SPA_MINBLOCKSIZE); ASSERT(lsize >= SPA_MINBLOCKSIZE && lsize <= resid); zp.zp_checksum = gio->io_prop.zp_checksum; zp.zp_compress = ZIO_COMPRESS_OFF; zp.zp_type = DMU_OT_NONE; zp.zp_level = 0; zp.zp_copies = gio->io_prop.zp_copies; zp.zp_dedup = B_FALSE; zp.zp_dedup_verify = B_FALSE; zp.zp_nopwrite = B_FALSE; zio_nowait(zio_write(zio, spa, txg, &gbh->zg_blkptr[g], (char *)pio->io_data + (pio->io_size - resid), lsize, &zp, zio_write_gang_member_ready, NULL, NULL, &gn->gn_child[g], pio->io_priority, ZIO_GANG_CHILD_FLAGS(pio), &pio->io_bookmark)); } /* * Set pio's pipeline to just wait for zio to finish. */ pio->io_pipeline = ZIO_INTERLOCK_PIPELINE; zio_nowait(zio); return (ZIO_PIPELINE_CONTINUE); } /* * The zio_nop_write stage in the pipeline determines if allocating * a new bp is necessary. By leveraging a cryptographically secure checksum, * such as SHA256, we can compare the checksums of the new data and the old * to determine if allocating a new block is required. The nopwrite * feature can handle writes in either syncing or open context (i.e. zil * writes) and as a result is mutually exclusive with dedup. */ static int zio_nop_write(zio_t *zio) { blkptr_t *bp = zio->io_bp; blkptr_t *bp_orig = &zio->io_bp_orig; zio_prop_t *zp = &zio->io_prop; ASSERT(BP_GET_LEVEL(bp) == 0); ASSERT(!(zio->io_flags & ZIO_FLAG_IO_REWRITE)); ASSERT(zp->zp_nopwrite); ASSERT(!zp->zp_dedup); ASSERT(zio->io_bp_override == NULL); ASSERT(IO_IS_ALLOCATING(zio)); /* * Check to see if the original bp and the new bp have matching * characteristics (i.e. same checksum, compression algorithms, etc). * If they don't then just continue with the pipeline which will * allocate a new bp. */ if (BP_IS_HOLE(bp_orig) || !zio_checksum_table[BP_GET_CHECKSUM(bp)].ci_dedup || BP_GET_CHECKSUM(bp) != BP_GET_CHECKSUM(bp_orig) || BP_GET_COMPRESS(bp) != BP_GET_COMPRESS(bp_orig) || BP_GET_DEDUP(bp) != BP_GET_DEDUP(bp_orig) || zp->zp_copies != BP_GET_NDVAS(bp_orig)) return (ZIO_PIPELINE_CONTINUE); /* * If the checksums match then reset the pipeline so that we * avoid allocating a new bp and issuing any I/O. */ if (ZIO_CHECKSUM_EQUAL(bp->blk_cksum, bp_orig->blk_cksum)) { ASSERT(zio_checksum_table[zp->zp_checksum].ci_dedup); ASSERT3U(BP_GET_PSIZE(bp), ==, BP_GET_PSIZE(bp_orig)); ASSERT3U(BP_GET_LSIZE(bp), ==, BP_GET_LSIZE(bp_orig)); ASSERT(zp->zp_compress != ZIO_COMPRESS_OFF); ASSERT(bcmp(&bp->blk_prop, &bp_orig->blk_prop, sizeof (uint64_t)) == 0); *bp = *bp_orig; zio->io_pipeline = ZIO_INTERLOCK_PIPELINE; zio->io_flags |= ZIO_FLAG_NOPWRITE; } return (ZIO_PIPELINE_CONTINUE); } /* * ========================================================================== * Dedup * ========================================================================== */ static void zio_ddt_child_read_done(zio_t *zio) { blkptr_t *bp = zio->io_bp; ddt_entry_t *dde = zio->io_private; ddt_phys_t *ddp; zio_t *pio = zio_unique_parent(zio); mutex_enter(&pio->io_lock); ddp = ddt_phys_select(dde, bp); if (zio->io_error == 0) ddt_phys_clear(ddp); /* this ddp doesn't need repair */ if (zio->io_error == 0 && dde->dde_repair_data == NULL) dde->dde_repair_data = zio->io_data; else zio_buf_free(zio->io_data, zio->io_size); mutex_exit(&pio->io_lock); } static int zio_ddt_read_start(zio_t *zio) { blkptr_t *bp = zio->io_bp; ASSERT(BP_GET_DEDUP(bp)); ASSERT(BP_GET_PSIZE(bp) == zio->io_size); ASSERT(zio->io_child_type == ZIO_CHILD_LOGICAL); if (zio->io_child_error[ZIO_CHILD_DDT]) { ddt_t *ddt = ddt_select(zio->io_spa, bp); ddt_entry_t *dde = ddt_repair_start(ddt, bp); ddt_phys_t *ddp = dde->dde_phys; ddt_phys_t *ddp_self = ddt_phys_select(dde, bp); blkptr_t blk; ASSERT(zio->io_vsd == NULL); zio->io_vsd = dde; if (ddp_self == NULL) return (ZIO_PIPELINE_CONTINUE); for (int p = 0; p < DDT_PHYS_TYPES; p++, ddp++) { if (ddp->ddp_phys_birth == 0 || ddp == ddp_self) continue; ddt_bp_create(ddt->ddt_checksum, &dde->dde_key, ddp, &blk); zio_nowait(zio_read(zio, zio->io_spa, &blk, zio_buf_alloc(zio->io_size), zio->io_size, zio_ddt_child_read_done, dde, zio->io_priority, ZIO_DDT_CHILD_FLAGS(zio) | ZIO_FLAG_DONT_PROPAGATE, &zio->io_bookmark)); } return (ZIO_PIPELINE_CONTINUE); } zio_nowait(zio_read(zio, zio->io_spa, bp, zio->io_data, zio->io_size, NULL, NULL, zio->io_priority, ZIO_DDT_CHILD_FLAGS(zio), &zio->io_bookmark)); return (ZIO_PIPELINE_CONTINUE); } static int zio_ddt_read_done(zio_t *zio) { blkptr_t *bp = zio->io_bp; if (zio_wait_for_children(zio, ZIO_CHILD_DDT, ZIO_WAIT_DONE)) return (ZIO_PIPELINE_STOP); ASSERT(BP_GET_DEDUP(bp)); ASSERT(BP_GET_PSIZE(bp) == zio->io_size); ASSERT(zio->io_child_type == ZIO_CHILD_LOGICAL); if (zio->io_child_error[ZIO_CHILD_DDT]) { ddt_t *ddt = ddt_select(zio->io_spa, bp); ddt_entry_t *dde = zio->io_vsd; if (ddt == NULL) { ASSERT(spa_load_state(zio->io_spa) != SPA_LOAD_NONE); return (ZIO_PIPELINE_CONTINUE); } if (dde == NULL) { zio->io_stage = ZIO_STAGE_DDT_READ_START >> 1; zio_taskq_dispatch(zio, ZIO_TASKQ_ISSUE, B_FALSE); return (ZIO_PIPELINE_STOP); } if (dde->dde_repair_data != NULL) { bcopy(dde->dde_repair_data, zio->io_data, zio->io_size); zio->io_child_error[ZIO_CHILD_DDT] = 0; } ddt_repair_done(ddt, dde); zio->io_vsd = NULL; } ASSERT(zio->io_vsd == NULL); return (ZIO_PIPELINE_CONTINUE); } static boolean_t zio_ddt_collision(zio_t *zio, ddt_t *ddt, ddt_entry_t *dde) { spa_t *spa = zio->io_spa; /* * Note: we compare the original data, not the transformed data, * because when zio->io_bp is an override bp, we will not have * pushed the I/O transforms. That's an important optimization * because otherwise we'd compress/encrypt all dmu_sync() data twice. */ for (int p = DDT_PHYS_SINGLE; p <= DDT_PHYS_TRIPLE; p++) { zio_t *lio = dde->dde_lead_zio[p]; if (lio != NULL) { return (lio->io_orig_size != zio->io_orig_size || bcmp(zio->io_orig_data, lio->io_orig_data, zio->io_orig_size) != 0); } } for (int p = DDT_PHYS_SINGLE; p <= DDT_PHYS_TRIPLE; p++) { ddt_phys_t *ddp = &dde->dde_phys[p]; if (ddp->ddp_phys_birth != 0) { arc_buf_t *abuf = NULL; uint32_t aflags = ARC_WAIT; blkptr_t blk = *zio->io_bp; int error; ddt_bp_fill(ddp, &blk, ddp->ddp_phys_birth); ddt_exit(ddt); error = arc_read(NULL, spa, &blk, arc_getbuf_func, &abuf, ZIO_PRIORITY_SYNC_READ, ZIO_FLAG_CANFAIL | ZIO_FLAG_SPECULATIVE, &aflags, &zio->io_bookmark); if (error == 0) { if (arc_buf_size(abuf) != zio->io_orig_size || bcmp(abuf->b_data, zio->io_orig_data, zio->io_orig_size) != 0) error = SET_ERROR(EEXIST); VERIFY(arc_buf_remove_ref(abuf, &abuf)); } ddt_enter(ddt); return (error != 0); } } return (B_FALSE); } static void zio_ddt_child_write_ready(zio_t *zio) { int p = zio->io_prop.zp_copies; ddt_t *ddt = ddt_select(zio->io_spa, zio->io_bp); ddt_entry_t *dde = zio->io_private; ddt_phys_t *ddp = &dde->dde_phys[p]; zio_t *pio; if (zio->io_error) return; ddt_enter(ddt); ASSERT(dde->dde_lead_zio[p] == zio); ddt_phys_fill(ddp, zio->io_bp); while ((pio = zio_walk_parents(zio)) != NULL) ddt_bp_fill(ddp, pio->io_bp, zio->io_txg); ddt_exit(ddt); } static void zio_ddt_child_write_done(zio_t *zio) { int p = zio->io_prop.zp_copies; ddt_t *ddt = ddt_select(zio->io_spa, zio->io_bp); ddt_entry_t *dde = zio->io_private; ddt_phys_t *ddp = &dde->dde_phys[p]; ddt_enter(ddt); ASSERT(ddp->ddp_refcnt == 0); ASSERT(dde->dde_lead_zio[p] == zio); dde->dde_lead_zio[p] = NULL; if (zio->io_error == 0) { while (zio_walk_parents(zio) != NULL) ddt_phys_addref(ddp); } else { ddt_phys_clear(ddp); } ddt_exit(ddt); } static void zio_ddt_ditto_write_done(zio_t *zio) { int p = DDT_PHYS_DITTO; zio_prop_t *zp = &zio->io_prop; blkptr_t *bp = zio->io_bp; ddt_t *ddt = ddt_select(zio->io_spa, bp); ddt_entry_t *dde = zio->io_private; ddt_phys_t *ddp = &dde->dde_phys[p]; ddt_key_t *ddk = &dde->dde_key; ddt_enter(ddt); ASSERT(ddp->ddp_refcnt == 0); ASSERT(dde->dde_lead_zio[p] == zio); dde->dde_lead_zio[p] = NULL; if (zio->io_error == 0) { ASSERT(ZIO_CHECKSUM_EQUAL(bp->blk_cksum, ddk->ddk_cksum)); ASSERT(zp->zp_copies < SPA_DVAS_PER_BP); ASSERT(zp->zp_copies == BP_GET_NDVAS(bp) - BP_IS_GANG(bp)); if (ddp->ddp_phys_birth != 0) ddt_phys_free(ddt, ddk, ddp, zio->io_txg); ddt_phys_fill(ddp, bp); } ddt_exit(ddt); } static int zio_ddt_write(zio_t *zio) { spa_t *spa = zio->io_spa; blkptr_t *bp = zio->io_bp; uint64_t txg = zio->io_txg; zio_prop_t *zp = &zio->io_prop; int p = zp->zp_copies; int ditto_copies; zio_t *cio = NULL; zio_t *dio = NULL; ddt_t *ddt = ddt_select(spa, bp); ddt_entry_t *dde; ddt_phys_t *ddp; ASSERT(BP_GET_DEDUP(bp)); ASSERT(BP_GET_CHECKSUM(bp) == zp->zp_checksum); ASSERT(BP_IS_HOLE(bp) || zio->io_bp_override); ddt_enter(ddt); dde = ddt_lookup(ddt, bp, B_TRUE); ddp = &dde->dde_phys[p]; if (zp->zp_dedup_verify && zio_ddt_collision(zio, ddt, dde)) { /* * If we're using a weak checksum, upgrade to a strong checksum * and try again. If we're already using a strong checksum, * we can't resolve it, so just convert to an ordinary write. * (And automatically e-mail a paper to Nature?) */ if (!zio_checksum_table[zp->zp_checksum].ci_dedup) { zp->zp_checksum = spa_dedup_checksum(spa); zio_pop_transforms(zio); zio->io_stage = ZIO_STAGE_OPEN; BP_ZERO(bp); } else { zp->zp_dedup = B_FALSE; } zio->io_pipeline = ZIO_WRITE_PIPELINE; ddt_exit(ddt); return (ZIO_PIPELINE_CONTINUE); } ditto_copies = ddt_ditto_copies_needed(ddt, dde, ddp); ASSERT(ditto_copies < SPA_DVAS_PER_BP); if (ditto_copies > ddt_ditto_copies_present(dde) && dde->dde_lead_zio[DDT_PHYS_DITTO] == NULL) { zio_prop_t czp = *zp; czp.zp_copies = ditto_copies; /* * If we arrived here with an override bp, we won't have run * the transform stack, so we won't have the data we need to * generate a child i/o. So, toss the override bp and restart. * This is safe, because using the override bp is just an * optimization; and it's rare, so the cost doesn't matter. */ if (zio->io_bp_override) { zio_pop_transforms(zio); zio->io_stage = ZIO_STAGE_OPEN; zio->io_pipeline = ZIO_WRITE_PIPELINE; zio->io_bp_override = NULL; BP_ZERO(bp); ddt_exit(ddt); return (ZIO_PIPELINE_CONTINUE); } dio = zio_write(zio, spa, txg, bp, zio->io_orig_data, zio->io_orig_size, &czp, NULL, NULL, zio_ddt_ditto_write_done, dde, zio->io_priority, ZIO_DDT_CHILD_FLAGS(zio), &zio->io_bookmark); zio_push_transform(dio, zio->io_data, zio->io_size, 0, NULL); dde->dde_lead_zio[DDT_PHYS_DITTO] = dio; } if (ddp->ddp_phys_birth != 0 || dde->dde_lead_zio[p] != NULL) { if (ddp->ddp_phys_birth != 0) ddt_bp_fill(ddp, bp, txg); if (dde->dde_lead_zio[p] != NULL) zio_add_child(zio, dde->dde_lead_zio[p]); else ddt_phys_addref(ddp); } else if (zio->io_bp_override) { ASSERT(bp->blk_birth == txg); ASSERT(BP_EQUAL(bp, zio->io_bp_override)); ddt_phys_fill(ddp, bp); ddt_phys_addref(ddp); } else { cio = zio_write(zio, spa, txg, bp, zio->io_orig_data, zio->io_orig_size, zp, zio_ddt_child_write_ready, NULL, zio_ddt_child_write_done, dde, zio->io_priority, ZIO_DDT_CHILD_FLAGS(zio), &zio->io_bookmark); zio_push_transform(cio, zio->io_data, zio->io_size, 0, NULL); dde->dde_lead_zio[p] = cio; } ddt_exit(ddt); if (cio) zio_nowait(cio); if (dio) zio_nowait(dio); return (ZIO_PIPELINE_CONTINUE); } ddt_entry_t *freedde; /* for debugging */ static int zio_ddt_free(zio_t *zio) { spa_t *spa = zio->io_spa; blkptr_t *bp = zio->io_bp; ddt_t *ddt = ddt_select(spa, bp); ddt_entry_t *dde; ddt_phys_t *ddp; ASSERT(BP_GET_DEDUP(bp)); ASSERT(zio->io_child_type == ZIO_CHILD_LOGICAL); ddt_enter(ddt); freedde = dde = ddt_lookup(ddt, bp, B_TRUE); ddp = ddt_phys_select(dde, bp); ddt_phys_decref(ddp); ddt_exit(ddt); return (ZIO_PIPELINE_CONTINUE); } /* * ========================================================================== * Allocate and free blocks * ========================================================================== */ static int zio_dva_allocate(zio_t *zio) { spa_t *spa = zio->io_spa; metaslab_class_t *mc = spa_normal_class(spa); blkptr_t *bp = zio->io_bp; int error; int flags = 0; if (zio->io_gang_leader == NULL) { ASSERT(zio->io_child_type > ZIO_CHILD_GANG); zio->io_gang_leader = zio; } ASSERT(BP_IS_HOLE(bp)); ASSERT0(BP_GET_NDVAS(bp)); ASSERT3U(zio->io_prop.zp_copies, >, 0); ASSERT3U(zio->io_prop.zp_copies, <=, spa_max_replication(spa)); ASSERT3U(zio->io_size, ==, BP_GET_PSIZE(bp)); /* * The dump device does not support gang blocks so allocation on * behalf of the dump device (i.e. ZIO_FLAG_NODATA) must avoid * the "fast" gang feature. */ flags |= (zio->io_flags & ZIO_FLAG_NODATA) ? METASLAB_GANG_AVOID : 0; flags |= (zio->io_flags & ZIO_FLAG_GANG_CHILD) ? METASLAB_GANG_CHILD : 0; error = metaslab_alloc(spa, mc, zio->io_size, bp, zio->io_prop.zp_copies, zio->io_txg, NULL, flags); if (error) { spa_dbgmsg(spa, "%s: metaslab allocation failure: zio %p, " "size %llu, error %d", spa_name(spa), zio, zio->io_size, error); if (error == ENOSPC && zio->io_size > SPA_MINBLOCKSIZE) return (zio_write_gang_block(zio)); zio->io_error = error; } return (ZIO_PIPELINE_CONTINUE); } static int zio_dva_free(zio_t *zio) { metaslab_free(zio->io_spa, zio->io_bp, zio->io_txg, B_FALSE); return (ZIO_PIPELINE_CONTINUE); } static int zio_dva_claim(zio_t *zio) { int error; error = metaslab_claim(zio->io_spa, zio->io_bp, zio->io_txg); if (error) zio->io_error = error; return (ZIO_PIPELINE_CONTINUE); } /* * Undo an allocation. This is used by zio_done() when an I/O fails * and we want to give back the block we just allocated. * This handles both normal blocks and gang blocks. */ static void zio_dva_unallocate(zio_t *zio, zio_gang_node_t *gn, blkptr_t *bp) { ASSERT(bp->blk_birth == zio->io_txg || BP_IS_HOLE(bp)); ASSERT(zio->io_bp_override == NULL); if (!BP_IS_HOLE(bp)) metaslab_free(zio->io_spa, bp, bp->blk_birth, B_TRUE); if (gn != NULL) { for (int g = 0; g < SPA_GBH_NBLKPTRS; g++) { zio_dva_unallocate(zio, gn->gn_child[g], &gn->gn_gbh->zg_blkptr[g]); } } } /* * Try to allocate an intent log block. Return 0 on success, errno on failure. */ int zio_alloc_zil(spa_t *spa, uint64_t txg, blkptr_t *new_bp, blkptr_t *old_bp, uint64_t size, boolean_t use_slog) { int error = 1; ASSERT(txg > spa_syncing_txg(spa)); /* * ZIL blocks are always contiguous (i.e. not gang blocks) so we * set the METASLAB_GANG_AVOID flag so that they don't "fast gang" * when allocating them. */ if (use_slog) { error = metaslab_alloc(spa, spa_log_class(spa), size, new_bp, 1, txg, old_bp, METASLAB_HINTBP_AVOID | METASLAB_GANG_AVOID); } if (error) { error = metaslab_alloc(spa, spa_normal_class(spa), size, new_bp, 1, txg, old_bp, METASLAB_HINTBP_AVOID); } if (error == 0) { BP_SET_LSIZE(new_bp, size); BP_SET_PSIZE(new_bp, size); BP_SET_COMPRESS(new_bp, ZIO_COMPRESS_OFF); BP_SET_CHECKSUM(new_bp, spa_version(spa) >= SPA_VERSION_SLIM_ZIL ? ZIO_CHECKSUM_ZILOG2 : ZIO_CHECKSUM_ZILOG); BP_SET_TYPE(new_bp, DMU_OT_INTENT_LOG); BP_SET_LEVEL(new_bp, 0); BP_SET_DEDUP(new_bp, 0); BP_SET_BYTEORDER(new_bp, ZFS_HOST_BYTEORDER); } return (error); } /* * Free an intent log block. */ void zio_free_zil(spa_t *spa, uint64_t txg, blkptr_t *bp) { ASSERT(BP_GET_TYPE(bp) == DMU_OT_INTENT_LOG); ASSERT(!BP_IS_GANG(bp)); zio_free(spa, txg, bp); } /* * ========================================================================== * Read, write and delete to physical devices * ========================================================================== */ /* * Issue an I/O to the underlying vdev. Typically the issue pipeline * stops after this stage and will resume upon I/O completion. * However, there are instances where the vdev layer may need to * continue the pipeline when an I/O was not issued. Since the I/O * that was sent to the vdev layer might be different than the one * currently active in the pipeline (see vdev_queue_io()), we explicitly * force the underlying vdev layers to call either zio_execute() or * zio_interrupt() to ensure that the pipeline continues with the correct I/O. */ static int zio_vdev_io_start(zio_t *zio) { vdev_t *vd = zio->io_vd; uint64_t align; spa_t *spa = zio->io_spa; int ret; ASSERT(zio->io_error == 0); ASSERT(zio->io_child_error[ZIO_CHILD_VDEV] == 0); if (vd == NULL) { if (!(zio->io_flags & ZIO_FLAG_CONFIG_WRITER)) spa_config_enter(spa, SCL_ZIO, zio, RW_READER); /* * The mirror_ops handle multiple DVAs in a single BP. */ vdev_mirror_ops.vdev_op_io_start(zio); return (ZIO_PIPELINE_STOP); } if (vd->vdev_ops->vdev_op_leaf && zio->io_type == ZIO_TYPE_FREE && zio->io_priority == ZIO_PRIORITY_NOW) { trim_map_free(vd, zio->io_offset, zio->io_size, zio->io_txg); return (ZIO_PIPELINE_CONTINUE); } /* * We keep track of time-sensitive I/Os so that the scan thread * can quickly react to certain workloads. In particular, we care * about non-scrubbing, top-level reads and writes with the following * characteristics: * - synchronous writes of user data to non-slog devices * - any reads of user data * When these conditions are met, adjust the timestamp of spa_last_io * which allows the scan thread to adjust its workload accordingly. */ if (!(zio->io_flags & ZIO_FLAG_SCAN_THREAD) && zio->io_bp != NULL && vd == vd->vdev_top && !vd->vdev_islog && zio->io_bookmark.zb_objset != DMU_META_OBJSET && zio->io_txg != spa_syncing_txg(spa)) { uint64_t old = spa->spa_last_io; uint64_t new = ddi_get_lbolt64(); if (old != new) (void) atomic_cas_64(&spa->spa_last_io, old, new); } align = 1ULL << vd->vdev_top->vdev_ashift; if (!(zio->io_flags & ZIO_FLAG_PHYSICAL) && P2PHASE(zio->io_size, align) != 0) { /* Transform logical writes to be a full physical block size. */ uint64_t asize = P2ROUNDUP(zio->io_size, align); char *abuf = NULL; if (zio->io_type == ZIO_TYPE_READ || zio->io_type == ZIO_TYPE_WRITE) abuf = zio_buf_alloc(asize); ASSERT(vd == vd->vdev_top); if (zio->io_type == ZIO_TYPE_WRITE) { bcopy(zio->io_data, abuf, zio->io_size); bzero(abuf + zio->io_size, asize - zio->io_size); } zio_push_transform(zio, abuf, asize, abuf ? asize : 0, zio_subblock); } /* * If this is not a physical io, make sure that it is properly aligned * before proceeding. */ if (!(zio->io_flags & ZIO_FLAG_PHYSICAL)) { ASSERT0(P2PHASE(zio->io_offset, align)); ASSERT0(P2PHASE(zio->io_size, align)); } else { /* * For physical writes, we allow 512b aligned writes and assume * the device will perform a read-modify-write as necessary. */ ASSERT0(P2PHASE(zio->io_offset, SPA_MINBLOCKSIZE)); ASSERT0(P2PHASE(zio->io_size, SPA_MINBLOCKSIZE)); } VERIFY(zio->io_type == ZIO_TYPE_READ || spa_writeable(spa)); /* * If this is a repair I/O, and there's no self-healing involved -- * that is, we're just resilvering what we expect to resilver -- * then don't do the I/O unless zio's txg is actually in vd's DTL. * This prevents spurious resilvering with nested replication. * For example, given a mirror of mirrors, (A+B)+(C+D), if only * A is out of date, we'll read from C+D, then use the data to * resilver A+B -- but we don't actually want to resilver B, just A. * The top-level mirror has no way to know this, so instead we just * discard unnecessary repairs as we work our way down the vdev tree. * The same logic applies to any form of nested replication: * ditto + mirror, RAID-Z + replacing, etc. This covers them all. */ if ((zio->io_flags & ZIO_FLAG_IO_REPAIR) && !(zio->io_flags & ZIO_FLAG_SELF_HEAL) && zio->io_txg != 0 && /* not a delegated i/o */ !vdev_dtl_contains(vd, DTL_PARTIAL, zio->io_txg, 1)) { ASSERT(zio->io_type == ZIO_TYPE_WRITE); zio_vdev_io_bypass(zio); return (ZIO_PIPELINE_CONTINUE); } if (vd->vdev_ops->vdev_op_leaf) { switch (zio->io_type) { case ZIO_TYPE_READ: if (vdev_cache_read(zio)) return (ZIO_PIPELINE_CONTINUE); /* FALLTHROUGH */ case ZIO_TYPE_WRITE: case ZIO_TYPE_FREE: if ((zio = vdev_queue_io(zio)) == NULL) return (ZIO_PIPELINE_STOP); if (!vdev_accessible(vd, zio)) { zio->io_error = SET_ERROR(ENXIO); zio_interrupt(zio); return (ZIO_PIPELINE_STOP); } break; } /* * Note that we ignore repair writes for TRIM because they can * conflict with normal writes. This isn't an issue because, by * definition, we only repair blocks that aren't freed. */ if (zio->io_type == ZIO_TYPE_WRITE && !(zio->io_flags & ZIO_FLAG_IO_REPAIR) && !trim_map_write_start(zio)) return (ZIO_PIPELINE_STOP); } vd->vdev_ops->vdev_op_io_start(zio); return (ZIO_PIPELINE_STOP); } static int zio_vdev_io_done(zio_t *zio) { vdev_t *vd = zio->io_vd; vdev_ops_t *ops = vd ? vd->vdev_ops : &vdev_mirror_ops; boolean_t unexpected_error = B_FALSE; if (zio_wait_for_children(zio, ZIO_CHILD_VDEV, ZIO_WAIT_DONE)) return (ZIO_PIPELINE_STOP); ASSERT(zio->io_type == ZIO_TYPE_READ || zio->io_type == ZIO_TYPE_WRITE || zio->io_type == ZIO_TYPE_FREE); if (vd != NULL && vd->vdev_ops->vdev_op_leaf && (zio->io_type == ZIO_TYPE_READ || zio->io_type == ZIO_TYPE_WRITE || zio->io_type == ZIO_TYPE_FREE)) { if (zio->io_type == ZIO_TYPE_WRITE && !(zio->io_flags & ZIO_FLAG_IO_REPAIR)) trim_map_write_done(zio); vdev_queue_io_done(zio); if (zio->io_type == ZIO_TYPE_WRITE) vdev_cache_write(zio); if (zio_injection_enabled && zio->io_error == 0) zio->io_error = zio_handle_device_injection(vd, zio, EIO); if (zio_injection_enabled && zio->io_error == 0) zio->io_error = zio_handle_label_injection(zio, EIO); if (zio->io_error) { if (zio->io_error == ENOTSUP && zio->io_type == ZIO_TYPE_FREE) { /* Not all devices support TRIM. */ } else if (!vdev_accessible(vd, zio)) { zio->io_error = SET_ERROR(ENXIO); } else { unexpected_error = B_TRUE; } } } ops->vdev_op_io_done(zio); if (unexpected_error) VERIFY(vdev_probe(vd, zio) == NULL); return (ZIO_PIPELINE_CONTINUE); } /* * For non-raidz ZIOs, we can just copy aside the bad data read from the * disk, and use that to finish the checksum ereport later. */ static void zio_vsd_default_cksum_finish(zio_cksum_report_t *zcr, const void *good_buf) { /* no processing needed */ zfs_ereport_finish_checksum(zcr, good_buf, zcr->zcr_cbdata, B_FALSE); } /*ARGSUSED*/ void zio_vsd_default_cksum_report(zio_t *zio, zio_cksum_report_t *zcr, void *ignored) { void *buf = zio_buf_alloc(zio->io_size); bcopy(zio->io_data, buf, zio->io_size); zcr->zcr_cbinfo = zio->io_size; zcr->zcr_cbdata = buf; zcr->zcr_finish = zio_vsd_default_cksum_finish; zcr->zcr_free = zio_buf_free; } static int zio_vdev_io_assess(zio_t *zio) { vdev_t *vd = zio->io_vd; if (zio_wait_for_children(zio, ZIO_CHILD_VDEV, ZIO_WAIT_DONE)) return (ZIO_PIPELINE_STOP); if (vd == NULL && !(zio->io_flags & ZIO_FLAG_CONFIG_WRITER)) spa_config_exit(zio->io_spa, SCL_ZIO, zio); if (zio->io_vsd != NULL) { zio->io_vsd_ops->vsd_free(zio); zio->io_vsd = NULL; } if (zio_injection_enabled && zio->io_error == 0) zio->io_error = zio_handle_fault_injection(zio, EIO); if (zio->io_type == ZIO_TYPE_FREE && zio->io_priority != ZIO_PRIORITY_NOW) { switch (zio->io_error) { case 0: ZIO_TRIM_STAT_INCR(bytes, zio->io_size); ZIO_TRIM_STAT_BUMP(success); break; case EOPNOTSUPP: ZIO_TRIM_STAT_BUMP(unsupported); break; default: ZIO_TRIM_STAT_BUMP(failed); break; } } /* * If the I/O failed, determine whether we should attempt to retry it. * * On retry, we cut in line in the issue queue, since we don't want * compression/checksumming/etc. work to prevent our (cheap) IO reissue. */ if (zio->io_error && vd == NULL && !(zio->io_flags & (ZIO_FLAG_DONT_RETRY | ZIO_FLAG_IO_RETRY))) { ASSERT(!(zio->io_flags & ZIO_FLAG_DONT_QUEUE)); /* not a leaf */ ASSERT(!(zio->io_flags & ZIO_FLAG_IO_BYPASS)); /* not a leaf */ zio->io_error = 0; zio->io_flags |= ZIO_FLAG_IO_RETRY | ZIO_FLAG_DONT_CACHE | ZIO_FLAG_DONT_AGGREGATE; zio->io_stage = ZIO_STAGE_VDEV_IO_START >> 1; zio_taskq_dispatch(zio, ZIO_TASKQ_ISSUE, zio_requeue_io_start_cut_in_line); return (ZIO_PIPELINE_STOP); } /* * If we got an error on a leaf device, convert it to ENXIO * if the device is not accessible at all. */ if (zio->io_error && vd != NULL && vd->vdev_ops->vdev_op_leaf && !vdev_accessible(vd, zio)) zio->io_error = SET_ERROR(ENXIO); /* * If we can't write to an interior vdev (mirror or RAID-Z), * set vdev_cant_write so that we stop trying to allocate from it. */ if (zio->io_error == ENXIO && zio->io_type == ZIO_TYPE_WRITE && vd != NULL && !vd->vdev_ops->vdev_op_leaf) { vd->vdev_cant_write = B_TRUE; } if (zio->io_error) zio->io_pipeline = ZIO_INTERLOCK_PIPELINE; if (vd != NULL && vd->vdev_ops->vdev_op_leaf && zio->io_physdone != NULL) { ASSERT(!(zio->io_flags & ZIO_FLAG_DELEGATED)); ASSERT(zio->io_child_type == ZIO_CHILD_VDEV); zio->io_physdone(zio->io_logical); } return (ZIO_PIPELINE_CONTINUE); } void zio_vdev_io_reissue(zio_t *zio) { ASSERT(zio->io_stage == ZIO_STAGE_VDEV_IO_START); ASSERT(zio->io_error == 0); zio->io_stage >>= 1; } void zio_vdev_io_redone(zio_t *zio) { ASSERT(zio->io_stage == ZIO_STAGE_VDEV_IO_DONE); zio->io_stage >>= 1; } void zio_vdev_io_bypass(zio_t *zio) { ASSERT(zio->io_stage == ZIO_STAGE_VDEV_IO_START); ASSERT(zio->io_error == 0); zio->io_flags |= ZIO_FLAG_IO_BYPASS; zio->io_stage = ZIO_STAGE_VDEV_IO_ASSESS >> 1; } /* * ========================================================================== * Generate and verify checksums * ========================================================================== */ static int zio_checksum_generate(zio_t *zio) { blkptr_t *bp = zio->io_bp; enum zio_checksum checksum; if (bp == NULL) { /* * This is zio_write_phys(). * We're either generating a label checksum, or none at all. */ checksum = zio->io_prop.zp_checksum; if (checksum == ZIO_CHECKSUM_OFF) return (ZIO_PIPELINE_CONTINUE); ASSERT(checksum == ZIO_CHECKSUM_LABEL); } else { if (BP_IS_GANG(bp) && zio->io_child_type == ZIO_CHILD_GANG) { ASSERT(!IO_IS_ALLOCATING(zio)); checksum = ZIO_CHECKSUM_GANG_HEADER; } else { checksum = BP_GET_CHECKSUM(bp); } } zio_checksum_compute(zio, checksum, zio->io_data, zio->io_size); return (ZIO_PIPELINE_CONTINUE); } static int zio_checksum_verify(zio_t *zio) { zio_bad_cksum_t info; blkptr_t *bp = zio->io_bp; int error; ASSERT(zio->io_vd != NULL); if (bp == NULL) { /* * This is zio_read_phys(). * We're either verifying a label checksum, or nothing at all. */ if (zio->io_prop.zp_checksum == ZIO_CHECKSUM_OFF) return (ZIO_PIPELINE_CONTINUE); ASSERT(zio->io_prop.zp_checksum == ZIO_CHECKSUM_LABEL); } if ((error = zio_checksum_error(zio, &info)) != 0) { zio->io_error = error; - if (!(zio->io_flags & ZIO_FLAG_SPECULATIVE)) { + if (error == ECKSUM && + !(zio->io_flags & ZIO_FLAG_SPECULATIVE)) { zfs_ereport_start_checksum(zio->io_spa, zio->io_vd, zio, zio->io_offset, zio->io_size, NULL, &info); } } return (ZIO_PIPELINE_CONTINUE); } /* * Called by RAID-Z to ensure we don't compute the checksum twice. */ void zio_checksum_verified(zio_t *zio) { zio->io_pipeline &= ~ZIO_STAGE_CHECKSUM_VERIFY; } /* * ========================================================================== * Error rank. Error are ranked in the order 0, ENXIO, ECKSUM, EIO, other. * An error of 0 indicates success. ENXIO indicates whole-device failure, * which may be transient (e.g. unplugged) or permament. ECKSUM and EIO * indicate errors that are specific to one I/O, and most likely permanent. * Any other error is presumed to be worse because we weren't expecting it. * ========================================================================== */ int zio_worst_error(int e1, int e2) { static int zio_error_rank[] = { 0, ENXIO, ECKSUM, EIO }; int r1, r2; for (r1 = 0; r1 < sizeof (zio_error_rank) / sizeof (int); r1++) if (e1 == zio_error_rank[r1]) break; for (r2 = 0; r2 < sizeof (zio_error_rank) / sizeof (int); r2++) if (e2 == zio_error_rank[r2]) break; return (r1 > r2 ? e1 : e2); } /* * ========================================================================== * I/O completion * ========================================================================== */ static int zio_ready(zio_t *zio) { blkptr_t *bp = zio->io_bp; zio_t *pio, *pio_next; if (zio_wait_for_children(zio, ZIO_CHILD_GANG, ZIO_WAIT_READY) || zio_wait_for_children(zio, ZIO_CHILD_DDT, ZIO_WAIT_READY)) return (ZIO_PIPELINE_STOP); if (zio->io_ready) { ASSERT(IO_IS_ALLOCATING(zio)); ASSERT(bp->blk_birth == zio->io_txg || BP_IS_HOLE(bp) || (zio->io_flags & ZIO_FLAG_NOPWRITE)); ASSERT(zio->io_children[ZIO_CHILD_GANG][ZIO_WAIT_READY] == 0); zio->io_ready(zio); } if (bp != NULL && bp != &zio->io_bp_copy) zio->io_bp_copy = *bp; if (zio->io_error) zio->io_pipeline = ZIO_INTERLOCK_PIPELINE; mutex_enter(&zio->io_lock); zio->io_state[ZIO_WAIT_READY] = 1; pio = zio_walk_parents(zio); mutex_exit(&zio->io_lock); /* * As we notify zio's parents, new parents could be added. * New parents go to the head of zio's io_parent_list, however, * so we will (correctly) not notify them. The remainder of zio's * io_parent_list, from 'pio_next' onward, cannot change because * all parents must wait for us to be done before they can be done. */ for (; pio != NULL; pio = pio_next) { pio_next = zio_walk_parents(zio); zio_notify_parent(pio, zio, ZIO_WAIT_READY); } if (zio->io_flags & ZIO_FLAG_NODATA) { if (BP_IS_GANG(bp)) { zio->io_flags &= ~ZIO_FLAG_NODATA; } else { ASSERT((uintptr_t)zio->io_data < SPA_MAXBLOCKSIZE); zio->io_pipeline &= ~ZIO_VDEV_IO_STAGES; } } if (zio_injection_enabled && zio->io_spa->spa_syncing_txg == zio->io_txg) zio_handle_ignored_writes(zio); return (ZIO_PIPELINE_CONTINUE); } static int zio_done(zio_t *zio) { spa_t *spa = zio->io_spa; zio_t *lio = zio->io_logical; blkptr_t *bp = zio->io_bp; vdev_t *vd = zio->io_vd; uint64_t psize = zio->io_size; zio_t *pio, *pio_next; /* * If our children haven't all completed, * wait for them and then repeat this pipeline stage. */ if (zio_wait_for_children(zio, ZIO_CHILD_VDEV, ZIO_WAIT_DONE) || zio_wait_for_children(zio, ZIO_CHILD_GANG, ZIO_WAIT_DONE) || zio_wait_for_children(zio, ZIO_CHILD_DDT, ZIO_WAIT_DONE) || zio_wait_for_children(zio, ZIO_CHILD_LOGICAL, ZIO_WAIT_DONE)) return (ZIO_PIPELINE_STOP); for (int c = 0; c < ZIO_CHILD_TYPES; c++) for (int w = 0; w < ZIO_WAIT_TYPES; w++) ASSERT(zio->io_children[c][w] == 0); if (bp != NULL && !BP_IS_EMBEDDED(bp)) { ASSERT(bp->blk_pad[0] == 0); ASSERT(bp->blk_pad[1] == 0); ASSERT(bcmp(bp, &zio->io_bp_copy, sizeof (blkptr_t)) == 0 || (bp == zio_unique_parent(zio)->io_bp)); if (zio->io_type == ZIO_TYPE_WRITE && !BP_IS_HOLE(bp) && zio->io_bp_override == NULL && !(zio->io_flags & ZIO_FLAG_IO_REPAIR)) { ASSERT(!BP_SHOULD_BYTESWAP(bp)); ASSERT3U(zio->io_prop.zp_copies, <=, BP_GET_NDVAS(bp)); ASSERT(BP_COUNT_GANG(bp) == 0 || (BP_COUNT_GANG(bp) == BP_GET_NDVAS(bp))); } if (zio->io_flags & ZIO_FLAG_NOPWRITE) VERIFY(BP_EQUAL(bp, &zio->io_bp_orig)); } /* * If there were child vdev/gang/ddt errors, they apply to us now. */ zio_inherit_child_errors(zio, ZIO_CHILD_VDEV); zio_inherit_child_errors(zio, ZIO_CHILD_GANG); zio_inherit_child_errors(zio, ZIO_CHILD_DDT); /* * If the I/O on the transformed data was successful, generate any * checksum reports now while we still have the transformed data. */ if (zio->io_error == 0) { while (zio->io_cksum_report != NULL) { zio_cksum_report_t *zcr = zio->io_cksum_report; uint64_t align = zcr->zcr_align; uint64_t asize = P2ROUNDUP(psize, align); char *abuf = zio->io_data; if (asize != psize) { abuf = zio_buf_alloc(asize); bcopy(zio->io_data, abuf, psize); bzero(abuf + psize, asize - psize); } zio->io_cksum_report = zcr->zcr_next; zcr->zcr_next = NULL; zcr->zcr_finish(zcr, abuf); zfs_ereport_free_checksum(zcr); if (asize != psize) zio_buf_free(abuf, asize); } } zio_pop_transforms(zio); /* note: may set zio->io_error */ vdev_stat_update(zio, psize); if (zio->io_error) { /* * If this I/O is attached to a particular vdev, * generate an error message describing the I/O failure * at the block level. We ignore these errors if the * device is currently unavailable. */ if (zio->io_error != ECKSUM && vd != NULL && !vdev_is_dead(vd)) zfs_ereport_post(FM_EREPORT_ZFS_IO, spa, vd, zio, 0, 0); if ((zio->io_error == EIO || !(zio->io_flags & (ZIO_FLAG_SPECULATIVE | ZIO_FLAG_DONT_PROPAGATE))) && zio == lio) { /* * For logical I/O requests, tell the SPA to log the * error and generate a logical data ereport. */ spa_log_error(spa, zio); zfs_ereport_post(FM_EREPORT_ZFS_DATA, spa, NULL, zio, 0, 0); } } if (zio->io_error && zio == lio) { /* * Determine whether zio should be reexecuted. This will * propagate all the way to the root via zio_notify_parent(). */ ASSERT(vd == NULL && bp != NULL); ASSERT(zio->io_child_type == ZIO_CHILD_LOGICAL); if (IO_IS_ALLOCATING(zio) && !(zio->io_flags & ZIO_FLAG_CANFAIL)) { if (zio->io_error != ENOSPC) zio->io_reexecute |= ZIO_REEXECUTE_NOW; else zio->io_reexecute |= ZIO_REEXECUTE_SUSPEND; } if ((zio->io_type == ZIO_TYPE_READ || zio->io_type == ZIO_TYPE_FREE) && !(zio->io_flags & ZIO_FLAG_SCAN_THREAD) && zio->io_error == ENXIO && spa_load_state(spa) == SPA_LOAD_NONE && spa_get_failmode(spa) != ZIO_FAILURE_MODE_CONTINUE) zio->io_reexecute |= ZIO_REEXECUTE_SUSPEND; if (!(zio->io_flags & ZIO_FLAG_CANFAIL) && !zio->io_reexecute) zio->io_reexecute |= ZIO_REEXECUTE_SUSPEND; /* * Here is a possibly good place to attempt to do * either combinatorial reconstruction or error correction * based on checksums. It also might be a good place * to send out preliminary ereports before we suspend * processing. */ } /* * If there were logical child errors, they apply to us now. * We defer this until now to avoid conflating logical child * errors with errors that happened to the zio itself when * updating vdev stats and reporting FMA events above. */ zio_inherit_child_errors(zio, ZIO_CHILD_LOGICAL); if ((zio->io_error || zio->io_reexecute) && IO_IS_ALLOCATING(zio) && zio->io_gang_leader == zio && !(zio->io_flags & (ZIO_FLAG_IO_REWRITE | ZIO_FLAG_NOPWRITE))) zio_dva_unallocate(zio, zio->io_gang_tree, bp); zio_gang_tree_free(&zio->io_gang_tree); /* * Godfather I/Os should never suspend. */ if ((zio->io_flags & ZIO_FLAG_GODFATHER) && (zio->io_reexecute & ZIO_REEXECUTE_SUSPEND)) zio->io_reexecute = 0; if (zio->io_reexecute) { /* * This is a logical I/O that wants to reexecute. * * Reexecute is top-down. When an i/o fails, if it's not * the root, it simply notifies its parent and sticks around. * The parent, seeing that it still has children in zio_done(), * does the same. This percolates all the way up to the root. * The root i/o will reexecute or suspend the entire tree. * * This approach ensures that zio_reexecute() honors * all the original i/o dependency relationships, e.g. * parents not executing until children are ready. */ ASSERT(zio->io_child_type == ZIO_CHILD_LOGICAL); zio->io_gang_leader = NULL; mutex_enter(&zio->io_lock); zio->io_state[ZIO_WAIT_DONE] = 1; mutex_exit(&zio->io_lock); /* * "The Godfather" I/O monitors its children but is * not a true parent to them. It will track them through * the pipeline but severs its ties whenever they get into * trouble (e.g. suspended). This allows "The Godfather" * I/O to return status without blocking. */ for (pio = zio_walk_parents(zio); pio != NULL; pio = pio_next) { zio_link_t *zl = zio->io_walk_link; pio_next = zio_walk_parents(zio); if ((pio->io_flags & ZIO_FLAG_GODFATHER) && (zio->io_reexecute & ZIO_REEXECUTE_SUSPEND)) { zio_remove_child(pio, zio, zl); zio_notify_parent(pio, zio, ZIO_WAIT_DONE); } } if ((pio = zio_unique_parent(zio)) != NULL) { /* * We're not a root i/o, so there's nothing to do * but notify our parent. Don't propagate errors * upward since we haven't permanently failed yet. */ ASSERT(!(zio->io_flags & ZIO_FLAG_GODFATHER)); zio->io_flags |= ZIO_FLAG_DONT_PROPAGATE; zio_notify_parent(pio, zio, ZIO_WAIT_DONE); } else if (zio->io_reexecute & ZIO_REEXECUTE_SUSPEND) { /* * We'd fail again if we reexecuted now, so suspend * until conditions improve (e.g. device comes online). */ zio_suspend(spa, zio); } else { /* * Reexecution is potentially a huge amount of work. * Hand it off to the otherwise-unused claim taskq. */ #if defined(illumos) || !defined(_KERNEL) ASSERT(zio->io_tqent.tqent_next == NULL); #else ASSERT(zio->io_tqent.tqent_task.ta_pending == 0); #endif spa_taskq_dispatch_ent(spa, ZIO_TYPE_CLAIM, ZIO_TASKQ_ISSUE, (task_func_t *)zio_reexecute, zio, 0, &zio->io_tqent); } return (ZIO_PIPELINE_STOP); } ASSERT(zio->io_child_count == 0); ASSERT(zio->io_reexecute == 0); ASSERT(zio->io_error == 0 || (zio->io_flags & ZIO_FLAG_CANFAIL)); /* * Report any checksum errors, since the I/O is complete. */ while (zio->io_cksum_report != NULL) { zio_cksum_report_t *zcr = zio->io_cksum_report; zio->io_cksum_report = zcr->zcr_next; zcr->zcr_next = NULL; zcr->zcr_finish(zcr, NULL); zfs_ereport_free_checksum(zcr); } /* * It is the responsibility of the done callback to ensure that this * particular zio is no longer discoverable for adoption, and as * such, cannot acquire any new parents. */ if (zio->io_done) zio->io_done(zio); mutex_enter(&zio->io_lock); zio->io_state[ZIO_WAIT_DONE] = 1; mutex_exit(&zio->io_lock); for (pio = zio_walk_parents(zio); pio != NULL; pio = pio_next) { zio_link_t *zl = zio->io_walk_link; pio_next = zio_walk_parents(zio); zio_remove_child(pio, zio, zl); zio_notify_parent(pio, zio, ZIO_WAIT_DONE); } if (zio->io_waiter != NULL) { mutex_enter(&zio->io_lock); zio->io_executor = NULL; cv_broadcast(&zio->io_cv); mutex_exit(&zio->io_lock); } else { zio_destroy(zio); } return (ZIO_PIPELINE_STOP); } /* * ========================================================================== * I/O pipeline definition * ========================================================================== */ static zio_pipe_stage_t *zio_pipeline[] = { NULL, zio_read_bp_init, zio_free_bp_init, zio_issue_async, zio_write_bp_init, zio_checksum_generate, zio_nop_write, zio_ddt_read_start, zio_ddt_read_done, zio_ddt_write, zio_ddt_free, zio_gang_assemble, zio_gang_issue, zio_dva_allocate, zio_dva_free, zio_dva_claim, zio_ready, zio_vdev_io_start, zio_vdev_io_done, zio_vdev_io_assess, zio_checksum_verify, zio_done }; /* dnp is the dnode for zb1->zb_object */ boolean_t zbookmark_is_before(const dnode_phys_t *dnp, const zbookmark_phys_t *zb1, const zbookmark_phys_t *zb2) { uint64_t zb1nextL0, zb2thisobj; ASSERT(zb1->zb_objset == zb2->zb_objset); ASSERT(zb2->zb_level == 0); /* The objset_phys_t isn't before anything. */ if (dnp == NULL) return (B_FALSE); zb1nextL0 = (zb1->zb_blkid + 1) << ((zb1->zb_level) * (dnp->dn_indblkshift - SPA_BLKPTRSHIFT)); zb2thisobj = zb2->zb_object ? zb2->zb_object : zb2->zb_blkid << (DNODE_BLOCK_SHIFT - DNODE_SHIFT); if (zb1->zb_object == DMU_META_DNODE_OBJECT) { uint64_t nextobj = zb1nextL0 * (dnp->dn_datablkszsec << SPA_MINBLOCKSHIFT) >> DNODE_SHIFT; return (nextobj <= zb2thisobj); } if (zb1->zb_object < zb2thisobj) return (B_TRUE); if (zb1->zb_object > zb2thisobj) return (B_FALSE); if (zb2->zb_object == DMU_META_DNODE_OBJECT) return (B_FALSE); return (zb1nextL0 <= zb2->zb_blkid); } Index: projects/clang350-import/sys/cddl/contrib/opensolaris =================================================================== --- projects/clang350-import/sys/cddl/contrib/opensolaris (revision 275748) +++ projects/clang350-import/sys/cddl/contrib/opensolaris (revision 275749) Property changes on: projects/clang350-import/sys/cddl/contrib/opensolaris ___________________________________________________________________ Modified: svn:mergeinfo ## -0,0 +0,2 ## Merged /head/sys/cddl/contrib/opensolaris:r275623-275748 Merged /vendor-sys/illumos/dist:r247174,275542-275546,275548 Index: projects/clang350-import/sys/conf/files =================================================================== --- projects/clang350-import/sys/conf/files (revision 275748) +++ projects/clang350-import/sys/conf/files (revision 275749) @@ -1,4016 +1,4019 @@ # $FreeBSD$ # # The long compile-with and dependency lines are required because of # limitations in config: backslash-newline doesn't work in strings, and # dependency lines other than the first are silently ignored. # acpi_quirks.h optional acpi \ dependency "$S/tools/acpi_quirks2h.awk $S/dev/acpica/acpi_quirks" \ compile-with "${AWK} -f $S/tools/acpi_quirks2h.awk $S/dev/acpica/acpi_quirks" \ no-obj no-implicit-rule before-depend \ clean "acpi_quirks.h" # # The 'fdt_dtb_file' target covers an actual DTB file name, which is derived # from the specified source (DTS) file: .dts -> .dtb # fdt_dtb_file optional fdt fdt_dtb_static \ compile-with "sh -c 'MACHINE=${MACHINE} $S/tools/fdt/make_dtb.sh $S ${FDT_DTS_FILE} ${.CURDIR}'" \ no-obj no-implicit-rule before-depend \ clean "${FDT_DTS_FILE:R}.dtb" fdt_static_dtb.h optional fdt fdt_dtb_static \ compile-with "sh -c 'MACHINE=${MACHINE} $S/tools/fdt/make_dtbh.sh ${FDT_DTS_FILE} ${.CURDIR}'" \ dependency "fdt_dtb_file" \ no-obj no-implicit-rule before-depend \ clean "fdt_static_dtb.h" feeder_eq_gen.h optional sound \ dependency "$S/tools/sound/feeder_eq_mkfilter.awk" \ compile-with "${AWK} -f $S/tools/sound/feeder_eq_mkfilter.awk -- ${FEEDER_EQ_PRESETS} > feeder_eq_gen.h" \ no-obj no-implicit-rule before-depend \ clean "feeder_eq_gen.h" feeder_rate_gen.h optional sound \ dependency "$S/tools/sound/feeder_rate_mkfilter.awk" \ compile-with "${AWK} -f $S/tools/sound/feeder_rate_mkfilter.awk -- ${FEEDER_RATE_PRESETS} > feeder_rate_gen.h" \ no-obj no-implicit-rule before-depend \ clean "feeder_rate_gen.h" snd_fxdiv_gen.h optional sound \ dependency "$S/tools/sound/snd_fxdiv_gen.awk" \ compile-with "${AWK} -f $S/tools/sound/snd_fxdiv_gen.awk -- > snd_fxdiv_gen.h" \ no-obj no-implicit-rule before-depend \ clean "snd_fxdiv_gen.h" miidevs.h optional miibus | mii \ dependency "$S/tools/miidevs2h.awk $S/dev/mii/miidevs" \ compile-with "${AWK} -f $S/tools/miidevs2h.awk $S/dev/mii/miidevs" \ no-obj no-implicit-rule before-depend \ clean "miidevs.h" pccarddevs.h standard \ dependency "$S/tools/pccarddevs2h.awk $S/dev/pccard/pccarddevs" \ compile-with "${AWK} -f $S/tools/pccarddevs2h.awk $S/dev/pccard/pccarddevs" \ no-obj no-implicit-rule before-depend \ clean "pccarddevs.h" teken_state.h optional sc | vt \ dependency "$S/teken/gensequences $S/teken/sequences" \ compile-with "${AWK} -f $S/teken/gensequences $S/teken/sequences > teken_state.h" \ no-obj no-implicit-rule before-depend \ clean "teken_state.h" usbdevs.h optional usb \ dependency "$S/tools/usbdevs2h.awk $S/dev/usb/usbdevs" \ compile-with "${AWK} -f $S/tools/usbdevs2h.awk $S/dev/usb/usbdevs -h" \ no-obj no-implicit-rule before-depend \ clean "usbdevs.h" usbdevs_data.h optional usb \ dependency "$S/tools/usbdevs2h.awk $S/dev/usb/usbdevs" \ compile-with "${AWK} -f $S/tools/usbdevs2h.awk $S/dev/usb/usbdevs -d" \ no-obj no-implicit-rule before-depend \ clean "usbdevs_data.h" cam/cam.c optional scbus cam/cam_compat.c optional scbus cam/cam_periph.c optional scbus cam/cam_queue.c optional scbus cam/cam_sim.c optional scbus cam/cam_xpt.c optional scbus cam/ata/ata_all.c optional scbus cam/ata/ata_xpt.c optional scbus cam/ata/ata_pmp.c optional scbus cam/scsi/scsi_xpt.c optional scbus cam/scsi/scsi_all.c optional scbus cam/scsi/scsi_cd.c optional cd cam/scsi/scsi_ch.c optional ch cam/ata/ata_da.c optional ada | da cam/ctl/ctl.c optional ctl cam/ctl/ctl_backend.c optional ctl cam/ctl/ctl_backend_block.c optional ctl cam/ctl/ctl_backend_ramdisk.c optional ctl cam/ctl/ctl_cmd_table.c optional ctl cam/ctl/ctl_frontend.c optional ctl cam/ctl/ctl_frontend_cam_sim.c optional ctl cam/ctl/ctl_frontend_internal.c optional ctl cam/ctl/ctl_frontend_iscsi.c optional ctl cam/ctl/ctl_scsi_all.c optional ctl cam/ctl/ctl_tpc.c optional ctl cam/ctl/ctl_tpc_local.c optional ctl cam/ctl/ctl_error.c optional ctl cam/ctl/ctl_util.c optional ctl cam/ctl/scsi_ctl.c optional ctl cam/scsi/scsi_da.c optional da cam/scsi/scsi_low.c optional ct | ncv | nsp | stg cam/scsi/scsi_pass.c optional pass cam/scsi/scsi_pt.c optional pt cam/scsi/scsi_sa.c optional sa cam/scsi/scsi_enc.c optional ses cam/scsi/scsi_enc_ses.c optional ses cam/scsi/scsi_enc_safte.c optional ses cam/scsi/scsi_sg.c optional sg cam/scsi/scsi_targ_bh.c optional targbh cam/scsi/scsi_target.c optional targ cam/scsi/smp_all.c optional scbus # shared between zfs and dtrace cddl/compat/opensolaris/kern/opensolaris.c optional zfs compile-with "${ZFS_C}" cddl/compat/opensolaris/kern/opensolaris_cmn_err.c optional zfs compile-with "${ZFS_C}" cddl/compat/opensolaris/kern/opensolaris_kmem.c optional zfs compile-with "${ZFS_C}" cddl/compat/opensolaris/kern/opensolaris_misc.c optional zfs compile-with "${ZFS_C}" cddl/compat/opensolaris/kern/opensolaris_sunddi.c optional zfs compile-with "${ZFS_C}" # zfs specific cddl/compat/opensolaris/kern/opensolaris_acl.c optional zfs compile-with "${ZFS_C}" cddl/compat/opensolaris/kern/opensolaris_dtrace.c optional zfs compile-with "${ZFS_C}" cddl/compat/opensolaris/kern/opensolaris_kobj.c optional zfs compile-with "${ZFS_C}" cddl/compat/opensolaris/kern/opensolaris_kstat.c optional zfs compile-with "${ZFS_C}" cddl/compat/opensolaris/kern/opensolaris_lookup.c optional zfs compile-with "${ZFS_C}" cddl/compat/opensolaris/kern/opensolaris_policy.c optional zfs compile-with "${ZFS_C}" cddl/compat/opensolaris/kern/opensolaris_string.c optional zfs compile-with "${ZFS_C}" cddl/compat/opensolaris/kern/opensolaris_sysevent.c optional zfs compile-with "${ZFS_C}" cddl/compat/opensolaris/kern/opensolaris_taskq.c optional zfs compile-with "${ZFS_C}" cddl/compat/opensolaris/kern/opensolaris_uio.c optional zfs compile-with "${ZFS_C}" cddl/compat/opensolaris/kern/opensolaris_vfs.c optional zfs compile-with "${ZFS_C}" cddl/compat/opensolaris/kern/opensolaris_vm.c optional zfs compile-with "${ZFS_C}" cddl/compat/opensolaris/kern/opensolaris_zone.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/common/acl/acl_common.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/common/avl/avl.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/common/nvpair/fnvpair.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/common/nvpair/nvpair.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/common/nvpair/nvpair_alloc_fixed.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/common/unicode/u8_textprep.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/common/zfs/zfeature_common.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/common/zfs/zfs_comutil.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/common/zfs/zfs_deleg.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/common/zfs/zfs_fletcher.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/common/zfs/zfs_ioctl_compat.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/common/zfs/zfs_namecheck.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/common/zfs/zfs_prop.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/common/zfs/zpool_prop.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/common/zfs/zprop_common.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/gfs.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/vnode.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/blkptr.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/bplist.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/bpobj.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/bptree.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/ddt.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/ddt_zap.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/dmu.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_diff.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_object.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_objset.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_send.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_traverse.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_tx.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_zfetch.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/dnode.c optional zfs compile-with "${ZFS_C}" \ warning "kernel contains CDDL licensed ZFS filesystem" cddl/contrib/opensolaris/uts/common/fs/zfs/dnode_sync.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_bookmark.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_dataset.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_deadlist.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_deleg.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_destroy.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_dir.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_pool.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_prop.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_scan.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_userhold.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_synctask.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/gzip.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/lz4.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/lzjb.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/metaslab.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/range_tree.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/refcount.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/rrwlock.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/sa.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/sha256.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/spa_config.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/spa_errlog.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/spa_history.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/spa_misc.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/space_map.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/space_reftree.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/trim_map.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/txg.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/uberblock.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/unique.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/vdev.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_cache.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_file.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_geom.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_label.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_mirror.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_missing.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_raidz.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_root.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/zap.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/zap_leaf.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/zap_micro.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/zfeature.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_acl.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_byteswap.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ctldir.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_debug.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_dir.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_fm.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_fuid.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_log.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_onexit.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_replay.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_rlock.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_sa.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vfsops.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_znode.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/zil.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/zio_checksum.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/zio_compress.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/zio_inject.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/zle.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/zrlock.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/fs/zfs/zvol.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/os/callb.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/os/fm.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/os/list.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/os/nvpair_alloc_system.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/zmod/adler32.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/zmod/deflate.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/zmod/inffast.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/zmod/inflate.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/zmod/inftrees.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/zmod/opensolaris_crc32.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/zmod/trees.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/zmod/zmod.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/zmod/zmod_subr.c optional zfs compile-with "${ZFS_C}" cddl/contrib/opensolaris/uts/common/zmod/zutil.c optional zfs compile-with "${ZFS_C}" compat/freebsd32/freebsd32_capability.c optional compat_freebsd32 compat/freebsd32/freebsd32_ioctl.c optional compat_freebsd32 compat/freebsd32/freebsd32_misc.c optional compat_freebsd32 compat/freebsd32/freebsd32_syscalls.c optional compat_freebsd32 compat/freebsd32/freebsd32_sysent.c optional compat_freebsd32 contrib/altq/altq/altq_cbq.c optional altq contrib/altq/altq/altq_cdnr.c optional altq contrib/altq/altq/altq_hfsc.c optional altq contrib/altq/altq/altq_priq.c optional altq contrib/altq/altq/altq_red.c optional altq contrib/altq/altq/altq_rio.c optional altq contrib/altq/altq/altq_rmclass.c optional altq contrib/altq/altq/altq_subr.c optional altq contrib/dev/acpica/common/ahids.c optional acpi acpi_debug contrib/dev/acpica/common/ahuuids.c optional acpi acpi_debug contrib/dev/acpica/components/debugger/dbcmds.c optional acpi acpi_debug contrib/dev/acpica/components/debugger/dbconvert.c optional acpi acpi_debug contrib/dev/acpica/components/debugger/dbdisply.c optional acpi acpi_debug contrib/dev/acpica/components/debugger/dbexec.c optional acpi acpi_debug contrib/dev/acpica/components/debugger/dbfileio.c optional acpi acpi_debug contrib/dev/acpica/components/debugger/dbhistry.c optional acpi acpi_debug contrib/dev/acpica/components/debugger/dbinput.c optional acpi acpi_debug contrib/dev/acpica/components/debugger/dbmethod.c optional acpi acpi_debug contrib/dev/acpica/components/debugger/dbnames.c optional acpi acpi_debug contrib/dev/acpica/components/debugger/dbstats.c optional acpi acpi_debug contrib/dev/acpica/components/debugger/dbtest.c optional acpi acpi_debug contrib/dev/acpica/components/debugger/dbutils.c optional acpi acpi_debug contrib/dev/acpica/components/debugger/dbxface.c optional acpi acpi_debug contrib/dev/acpica/components/disassembler/dmbuffer.c optional acpi acpi_debug contrib/dev/acpica/components/disassembler/dmdeferred.c optional acpi acpi_debug contrib/dev/acpica/components/disassembler/dmnames.c optional acpi acpi_debug contrib/dev/acpica/components/disassembler/dmopcode.c optional acpi acpi_debug contrib/dev/acpica/components/disassembler/dmobject.c optional acpi acpi_debug contrib/dev/acpica/components/disassembler/dmresrc.c optional acpi acpi_debug contrib/dev/acpica/components/disassembler/dmresrcl.c optional acpi acpi_debug contrib/dev/acpica/components/disassembler/dmresrcl2.c optional acpi acpi_debug contrib/dev/acpica/components/disassembler/dmresrcs.c optional acpi acpi_debug contrib/dev/acpica/components/disassembler/dmutils.c optional acpi acpi_debug contrib/dev/acpica/components/disassembler/dmwalk.c optional acpi acpi_debug contrib/dev/acpica/components/dispatcher/dsargs.c optional acpi contrib/dev/acpica/components/dispatcher/dscontrol.c optional acpi contrib/dev/acpica/components/dispatcher/dsfield.c optional acpi contrib/dev/acpica/components/dispatcher/dsinit.c optional acpi contrib/dev/acpica/components/dispatcher/dsmethod.c optional acpi contrib/dev/acpica/components/dispatcher/dsmthdat.c optional acpi contrib/dev/acpica/components/dispatcher/dsobject.c optional acpi contrib/dev/acpica/components/dispatcher/dsopcode.c optional acpi contrib/dev/acpica/components/dispatcher/dsutils.c optional acpi contrib/dev/acpica/components/dispatcher/dswexec.c optional acpi contrib/dev/acpica/components/dispatcher/dswload.c optional acpi contrib/dev/acpica/components/dispatcher/dswload2.c optional acpi contrib/dev/acpica/components/dispatcher/dswscope.c optional acpi contrib/dev/acpica/components/dispatcher/dswstate.c optional acpi contrib/dev/acpica/components/events/evevent.c optional acpi contrib/dev/acpica/components/events/evglock.c optional acpi contrib/dev/acpica/components/events/evgpe.c optional acpi contrib/dev/acpica/components/events/evgpeblk.c optional acpi contrib/dev/acpica/components/events/evgpeinit.c optional acpi contrib/dev/acpica/components/events/evgpeutil.c optional acpi contrib/dev/acpica/components/events/evhandler.c optional acpi contrib/dev/acpica/components/events/evmisc.c optional acpi contrib/dev/acpica/components/events/evregion.c optional acpi contrib/dev/acpica/components/events/evrgnini.c optional acpi contrib/dev/acpica/components/events/evsci.c optional acpi contrib/dev/acpica/components/events/evxface.c optional acpi contrib/dev/acpica/components/events/evxfevnt.c optional acpi contrib/dev/acpica/components/events/evxfgpe.c optional acpi contrib/dev/acpica/components/events/evxfregn.c optional acpi contrib/dev/acpica/components/executer/exconfig.c optional acpi contrib/dev/acpica/components/executer/exconvrt.c optional acpi contrib/dev/acpica/components/executer/excreate.c optional acpi contrib/dev/acpica/components/executer/exdebug.c optional acpi contrib/dev/acpica/components/executer/exdump.c optional acpi contrib/dev/acpica/components/executer/exfield.c optional acpi contrib/dev/acpica/components/executer/exfldio.c optional acpi contrib/dev/acpica/components/executer/exmisc.c optional acpi contrib/dev/acpica/components/executer/exmutex.c optional acpi contrib/dev/acpica/components/executer/exnames.c optional acpi contrib/dev/acpica/components/executer/exoparg1.c optional acpi contrib/dev/acpica/components/executer/exoparg2.c optional acpi contrib/dev/acpica/components/executer/exoparg3.c optional acpi contrib/dev/acpica/components/executer/exoparg6.c optional acpi contrib/dev/acpica/components/executer/exprep.c optional acpi contrib/dev/acpica/components/executer/exregion.c optional acpi contrib/dev/acpica/components/executer/exresnte.c optional acpi contrib/dev/acpica/components/executer/exresolv.c optional acpi contrib/dev/acpica/components/executer/exresop.c optional acpi contrib/dev/acpica/components/executer/exstore.c optional acpi contrib/dev/acpica/components/executer/exstoren.c optional acpi contrib/dev/acpica/components/executer/exstorob.c optional acpi contrib/dev/acpica/components/executer/exsystem.c optional acpi contrib/dev/acpica/components/executer/exutils.c optional acpi contrib/dev/acpica/components/hardware/hwacpi.c optional acpi contrib/dev/acpica/components/hardware/hwesleep.c optional acpi contrib/dev/acpica/components/hardware/hwgpe.c optional acpi contrib/dev/acpica/components/hardware/hwpci.c optional acpi contrib/dev/acpica/components/hardware/hwregs.c optional acpi contrib/dev/acpica/components/hardware/hwsleep.c optional acpi contrib/dev/acpica/components/hardware/hwtimer.c optional acpi contrib/dev/acpica/components/hardware/hwvalid.c optional acpi contrib/dev/acpica/components/hardware/hwxface.c optional acpi contrib/dev/acpica/components/hardware/hwxfsleep.c optional acpi contrib/dev/acpica/components/namespace/nsaccess.c optional acpi contrib/dev/acpica/components/namespace/nsalloc.c optional acpi contrib/dev/acpica/components/namespace/nsarguments.c optional acpi contrib/dev/acpica/components/namespace/nsconvert.c optional acpi contrib/dev/acpica/components/namespace/nsdump.c optional acpi contrib/dev/acpica/components/namespace/nseval.c optional acpi contrib/dev/acpica/components/namespace/nsinit.c optional acpi contrib/dev/acpica/components/namespace/nsload.c optional acpi contrib/dev/acpica/components/namespace/nsnames.c optional acpi contrib/dev/acpica/components/namespace/nsobject.c optional acpi contrib/dev/acpica/components/namespace/nsparse.c optional acpi contrib/dev/acpica/components/namespace/nspredef.c optional acpi contrib/dev/acpica/components/namespace/nsprepkg.c optional acpi contrib/dev/acpica/components/namespace/nsrepair.c optional acpi contrib/dev/acpica/components/namespace/nsrepair2.c optional acpi contrib/dev/acpica/components/namespace/nssearch.c optional acpi contrib/dev/acpica/components/namespace/nsutils.c optional acpi contrib/dev/acpica/components/namespace/nswalk.c optional acpi contrib/dev/acpica/components/namespace/nsxfeval.c optional acpi contrib/dev/acpica/components/namespace/nsxfname.c optional acpi contrib/dev/acpica/components/namespace/nsxfobj.c optional acpi contrib/dev/acpica/components/parser/psargs.c optional acpi contrib/dev/acpica/components/parser/psloop.c optional acpi contrib/dev/acpica/components/parser/psobject.c optional acpi contrib/dev/acpica/components/parser/psopcode.c optional acpi contrib/dev/acpica/components/parser/psopinfo.c optional acpi contrib/dev/acpica/components/parser/psparse.c optional acpi contrib/dev/acpica/components/parser/psscope.c optional acpi contrib/dev/acpica/components/parser/pstree.c optional acpi contrib/dev/acpica/components/parser/psutils.c optional acpi contrib/dev/acpica/components/parser/pswalk.c optional acpi contrib/dev/acpica/components/parser/psxface.c optional acpi contrib/dev/acpica/components/resources/rsaddr.c optional acpi contrib/dev/acpica/components/resources/rscalc.c optional acpi contrib/dev/acpica/components/resources/rscreate.c optional acpi contrib/dev/acpica/components/resources/rsdump.c optional acpi contrib/dev/acpica/components/resources/rsdumpinfo.c optional acpi contrib/dev/acpica/components/resources/rsinfo.c optional acpi contrib/dev/acpica/components/resources/rsio.c optional acpi contrib/dev/acpica/components/resources/rsirq.c optional acpi contrib/dev/acpica/components/resources/rslist.c optional acpi contrib/dev/acpica/components/resources/rsmemory.c optional acpi contrib/dev/acpica/components/resources/rsmisc.c optional acpi contrib/dev/acpica/components/resources/rsserial.c optional acpi contrib/dev/acpica/components/resources/rsutils.c optional acpi contrib/dev/acpica/components/resources/rsxface.c optional acpi contrib/dev/acpica/components/tables/tbdata.c optional acpi contrib/dev/acpica/components/tables/tbfadt.c optional acpi contrib/dev/acpica/components/tables/tbfind.c optional acpi contrib/dev/acpica/components/tables/tbinstal.c optional acpi contrib/dev/acpica/components/tables/tbprint.c optional acpi contrib/dev/acpica/components/tables/tbutils.c optional acpi contrib/dev/acpica/components/tables/tbxface.c optional acpi contrib/dev/acpica/components/tables/tbxfload.c optional acpi contrib/dev/acpica/components/tables/tbxfroot.c optional acpi contrib/dev/acpica/components/utilities/utaddress.c optional acpi contrib/dev/acpica/components/utilities/utalloc.c optional acpi contrib/dev/acpica/components/utilities/utbuffer.c optional acpi contrib/dev/acpica/components/utilities/utcache.c optional acpi contrib/dev/acpica/components/utilities/utcopy.c optional acpi contrib/dev/acpica/components/utilities/utdebug.c optional acpi contrib/dev/acpica/components/utilities/utdecode.c optional acpi contrib/dev/acpica/components/utilities/utdelete.c optional acpi contrib/dev/acpica/components/utilities/uterror.c optional acpi contrib/dev/acpica/components/utilities/uteval.c optional acpi contrib/dev/acpica/components/utilities/utexcep.c optional acpi contrib/dev/acpica/components/utilities/utglobal.c optional acpi contrib/dev/acpica/components/utilities/uthex.c optional acpi contrib/dev/acpica/components/utilities/utids.c optional acpi contrib/dev/acpica/components/utilities/utinit.c optional acpi contrib/dev/acpica/components/utilities/utlock.c optional acpi contrib/dev/acpica/components/utilities/utmath.c optional acpi contrib/dev/acpica/components/utilities/utmisc.c optional acpi contrib/dev/acpica/components/utilities/utmutex.c optional acpi contrib/dev/acpica/components/utilities/utobject.c optional acpi contrib/dev/acpica/components/utilities/utosi.c optional acpi contrib/dev/acpica/components/utilities/utownerid.c optional acpi contrib/dev/acpica/components/utilities/utpredef.c optional acpi contrib/dev/acpica/components/utilities/utresrc.c optional acpi contrib/dev/acpica/components/utilities/utstate.c optional acpi contrib/dev/acpica/components/utilities/utstring.c optional acpi contrib/dev/acpica/components/utilities/utuuid.c optional acpi acpi_debug contrib/dev/acpica/components/utilities/utxface.c optional acpi contrib/dev/acpica/components/utilities/utxferror.c optional acpi contrib/dev/acpica/components/utilities/utxfinit.c optional acpi #contrib/dev/acpica/components/utilities/utxfmutex.c optional acpi contrib/ipfilter/netinet/fil.c optional ipfilter inet \ compile-with "${NORMAL_C} ${NO_WSELF_ASSIGN} -Wno-unused -I$S/contrib/ipfilter" contrib/ipfilter/netinet/ip_auth.c optional ipfilter inet \ compile-with "${NORMAL_C} -Wno-unused -I$S/contrib/ipfilter" contrib/ipfilter/netinet/ip_fil_freebsd.c optional ipfilter inet \ compile-with "${NORMAL_C} -Wno-unused -I$S/contrib/ipfilter" contrib/ipfilter/netinet/ip_frag.c optional ipfilter inet \ compile-with "${NORMAL_C} -Wno-unused -I$S/contrib/ipfilter" contrib/ipfilter/netinet/ip_log.c optional ipfilter inet \ compile-with "${NORMAL_C} -I$S/contrib/ipfilter" contrib/ipfilter/netinet/ip_nat.c optional ipfilter inet \ compile-with "${NORMAL_C} -Wno-unused -I$S/contrib/ipfilter" contrib/ipfilter/netinet/ip_proxy.c optional ipfilter inet \ compile-with "${NORMAL_C} ${NO_WSELF_ASSIGN} -Wno-unused -I$S/contrib/ipfilter" contrib/ipfilter/netinet/ip_state.c optional ipfilter inet \ compile-with "${NORMAL_C} -Wno-unused -I$S/contrib/ipfilter" contrib/ipfilter/netinet/ip_lookup.c optional ipfilter inet \ compile-with "${NORMAL_C} ${NO_WSELF_ASSIGN} -Wno-unused -Wno-error -I$S/contrib/ipfilter" contrib/ipfilter/netinet/ip_pool.c optional ipfilter inet \ compile-with "${NORMAL_C} -Wno-unused -I$S/contrib/ipfilter" contrib/ipfilter/netinet/ip_htable.c optional ipfilter inet \ compile-with "${NORMAL_C} -Wno-unused -I$S/contrib/ipfilter" contrib/ipfilter/netinet/ip_sync.c optional ipfilter inet \ compile-with "${NORMAL_C} -Wno-unused -I$S/contrib/ipfilter" contrib/ipfilter/netinet/mlfk_ipl.c optional ipfilter inet \ compile-with "${NORMAL_C} -I$S/contrib/ipfilter" contrib/ipfilter/netinet/ip_nat6.c optional ipfilter inet \ compile-with "${NORMAL_C} -Wno-unused -I$S/contrib/ipfilter" contrib/ipfilter/netinet/ip_rules.c optional ipfilter inet \ compile-with "${NORMAL_C} -I$S/contrib/ipfilter" contrib/ipfilter/netinet/ip_scan.c optional ipfilter inet \ compile-with "${NORMAL_C} -Wno-unused -I$S/contrib/ipfilter" contrib/ipfilter/netinet/ip_dstlist.c optional ipfilter inet \ compile-with "${NORMAL_C} -Wno-unused -I$S/contrib/ipfilter" contrib/ipfilter/netinet/radix_ipf.c optional ipfilter inet \ compile-with "${NORMAL_C} -I$S/contrib/ipfilter" contrib/libfdt/fdt.c optional fdt contrib/libfdt/fdt_ro.c optional fdt contrib/libfdt/fdt_rw.c optional fdt contrib/libfdt/fdt_strerror.c optional fdt contrib/libfdt/fdt_sw.c optional fdt contrib/libfdt/fdt_wip.c optional fdt contrib/ngatm/netnatm/api/cc_conn.c optional ngatm_ccatm \ compile-with "${NORMAL_C_NOWERROR} -I$S/contrib/ngatm" contrib/ngatm/netnatm/api/cc_data.c optional ngatm_ccatm \ compile-with "${NORMAL_C} -I$S/contrib/ngatm" contrib/ngatm/netnatm/api/cc_dump.c optional ngatm_ccatm \ compile-with "${NORMAL_C} -I$S/contrib/ngatm" contrib/ngatm/netnatm/api/cc_port.c optional ngatm_ccatm \ compile-with "${NORMAL_C} -I$S/contrib/ngatm" contrib/ngatm/netnatm/api/cc_sig.c optional ngatm_ccatm \ compile-with "${NORMAL_C} -I$S/contrib/ngatm" contrib/ngatm/netnatm/api/cc_user.c optional ngatm_ccatm \ compile-with "${NORMAL_C} -I$S/contrib/ngatm" contrib/ngatm/netnatm/api/unisap.c optional ngatm_ccatm \ compile-with "${NORMAL_C} -I$S/contrib/ngatm" contrib/ngatm/netnatm/misc/straddr.c optional ngatm_atmbase \ compile-with "${NORMAL_C} -I$S/contrib/ngatm" contrib/ngatm/netnatm/misc/unimsg_common.c optional ngatm_atmbase \ compile-with "${NORMAL_C} -I$S/contrib/ngatm" contrib/ngatm/netnatm/msg/traffic.c optional ngatm_atmbase \ compile-with "${NORMAL_C} -I$S/contrib/ngatm" contrib/ngatm/netnatm/msg/uni_ie.c optional ngatm_atmbase \ compile-with "${NORMAL_C} -I$S/contrib/ngatm" contrib/ngatm/netnatm/msg/uni_msg.c optional ngatm_atmbase \ compile-with "${NORMAL_C} -I$S/contrib/ngatm" contrib/ngatm/netnatm/saal/saal_sscfu.c optional ngatm_sscfu \ compile-with "${NORMAL_C} -I$S/contrib/ngatm" contrib/ngatm/netnatm/saal/saal_sscop.c optional ngatm_sscop \ compile-with "${NORMAL_C} -I$S/contrib/ngatm" contrib/ngatm/netnatm/sig/sig_call.c optional ngatm_uni \ compile-with "${NORMAL_C} -I$S/contrib/ngatm" contrib/ngatm/netnatm/sig/sig_coord.c optional ngatm_uni \ compile-with "${NORMAL_C} -I$S/contrib/ngatm" contrib/ngatm/netnatm/sig/sig_party.c optional ngatm_uni \ compile-with "${NORMAL_C} -I$S/contrib/ngatm" contrib/ngatm/netnatm/sig/sig_print.c optional ngatm_uni \ compile-with "${NORMAL_C} -I$S/contrib/ngatm" contrib/ngatm/netnatm/sig/sig_reset.c optional ngatm_uni \ compile-with "${NORMAL_C} -I$S/contrib/ngatm" contrib/ngatm/netnatm/sig/sig_uni.c optional ngatm_uni \ compile-with "${NORMAL_C} -I$S/contrib/ngatm" contrib/ngatm/netnatm/sig/sig_unimsgcpy.c optional ngatm_uni \ compile-with "${NORMAL_C} -I$S/contrib/ngatm" contrib/ngatm/netnatm/sig/sig_verify.c optional ngatm_uni \ compile-with "${NORMAL_C} -I$S/contrib/ngatm" crypto/blowfish/bf_ecb.c optional ipsec crypto/blowfish/bf_skey.c optional crypto | ipsec crypto/camellia/camellia.c optional crypto | ipsec crypto/camellia/camellia-api.c optional crypto | ipsec crypto/des/des_ecb.c optional crypto | ipsec | netsmb crypto/des/des_setkey.c optional crypto | ipsec | netsmb crypto/rc4/rc4.c optional netgraph_mppc_encryption | kgssapi crypto/rijndael/rijndael-alg-fst.c optional crypto | geom_bde | \ ipsec | random | wlan_ccmp crypto/rijndael/rijndael-api-fst.c optional geom_bde | random crypto/rijndael/rijndael-api.c optional crypto | ipsec | wlan_ccmp crypto/sha1.c optional carp | crypto | ipsec | \ netgraph_mppc_encryption | sctp crypto/sha2/sha2.c optional crypto | geom_bde | ipsec | random | \ sctp | zfs crypto/sha2/sha256c.c optional crypto | geom_bde | ipsec | random | \ sctp | zfs crypto/siphash/siphash.c optional inet | inet6 crypto/siphash/siphash_test.c optional inet | inet6 ddb/db_access.c optional ddb ddb/db_break.c optional ddb ddb/db_capture.c optional ddb ddb/db_command.c optional ddb ddb/db_examine.c optional ddb ddb/db_expr.c optional ddb ddb/db_input.c optional ddb ddb/db_lex.c optional ddb ddb/db_main.c optional ddb ddb/db_output.c optional ddb ddb/db_print.c optional ddb ddb/db_ps.c optional ddb ddb/db_run.c optional ddb ddb/db_script.c optional ddb ddb/db_sym.c optional ddb ddb/db_thread.c optional ddb ddb/db_textdump.c optional ddb ddb/db_variables.c optional ddb ddb/db_watch.c optional ddb ddb/db_write_cmd.c optional ddb dev/aac/aac.c optional aac dev/aac/aac_cam.c optional aacp aac dev/aac/aac_debug.c optional aac dev/aac/aac_disk.c optional aac dev/aac/aac_linux.c optional aac compat_linux dev/aac/aac_pci.c optional aac pci dev/aacraid/aacraid.c optional aacraid dev/aacraid/aacraid_cam.c optional aacraid scbus dev/aacraid/aacraid_debug.c optional aacraid dev/aacraid/aacraid_linux.c optional aacraid compat_linux dev/aacraid/aacraid_pci.c optional aacraid pci dev/acpi_support/acpi_wmi.c optional acpi_wmi acpi dev/acpi_support/acpi_asus.c optional acpi_asus acpi dev/acpi_support/acpi_asus_wmi.c optional acpi_asus_wmi acpi dev/acpi_support/acpi_fujitsu.c optional acpi_fujitsu acpi dev/acpi_support/acpi_hp.c optional acpi_hp acpi dev/acpi_support/acpi_ibm.c optional acpi_ibm acpi dev/acpi_support/acpi_panasonic.c optional acpi_panasonic acpi dev/acpi_support/acpi_sony.c optional acpi_sony acpi dev/acpi_support/acpi_toshiba.c optional acpi_toshiba acpi dev/acpi_support/atk0110.c optional aibs acpi dev/acpica/Osd/OsdDebug.c optional acpi dev/acpica/Osd/OsdHardware.c optional acpi dev/acpica/Osd/OsdInterrupt.c optional acpi dev/acpica/Osd/OsdMemory.c optional acpi dev/acpica/Osd/OsdSchedule.c optional acpi dev/acpica/Osd/OsdStream.c optional acpi dev/acpica/Osd/OsdSynch.c optional acpi dev/acpica/Osd/OsdTable.c optional acpi dev/acpica/acpi.c optional acpi dev/acpica/acpi_acad.c optional acpi dev/acpica/acpi_battery.c optional acpi dev/acpica/acpi_button.c optional acpi dev/acpica/acpi_cmbat.c optional acpi dev/acpica/acpi_cpu.c optional acpi dev/acpica/acpi_ec.c optional acpi dev/acpica/acpi_hpet.c optional acpi dev/acpica/acpi_isab.c optional acpi isa dev/acpica/acpi_lid.c optional acpi dev/acpica/acpi_package.c optional acpi dev/acpica/acpi_pci.c optional acpi pci dev/acpica/acpi_pci_link.c optional acpi pci dev/acpica/acpi_pcib.c optional acpi pci dev/acpica/acpi_pcib_acpi.c optional acpi pci dev/acpica/acpi_pcib_pci.c optional acpi pci dev/acpica/acpi_perf.c optional acpi dev/acpica/acpi_powerres.c optional acpi dev/acpica/acpi_quirk.c optional acpi dev/acpica/acpi_resource.c optional acpi dev/acpica/acpi_smbat.c optional acpi dev/acpica/acpi_thermal.c optional acpi dev/acpica/acpi_throttle.c optional acpi dev/acpica/acpi_timer.c optional acpi dev/acpica/acpi_video.c optional acpi_video acpi dev/acpica/acpi_dock.c optional acpi_dock acpi dev/adlink/adlink.c optional adlink dev/advansys/adv_eisa.c optional adv eisa dev/advansys/adv_pci.c optional adv pci dev/advansys/advansys.c optional adv dev/advansys/advlib.c optional adv dev/advansys/advmcode.c optional adv dev/advansys/adw_pci.c optional adw pci dev/advansys/adwcam.c optional adw dev/advansys/adwlib.c optional adw dev/advansys/adwmcode.c optional adw dev/ae/if_ae.c optional ae pci dev/age/if_age.c optional age pci dev/agp/agp.c optional agp pci dev/agp/agp_if.m optional agp pci dev/aha/aha.c optional aha dev/aha/aha_isa.c optional aha isa dev/aha/aha_mca.c optional aha mca dev/ahb/ahb.c optional ahb eisa dev/ahci/ahci.c optional ahci dev/ahci/ahciem.c optional ahci dev/ahci/ahci_pci.c optional ahci pci dev/aic/aic.c optional aic dev/aic/aic_pccard.c optional aic pccard dev/aic7xxx/ahc_eisa.c optional ahc eisa dev/aic7xxx/ahc_isa.c optional ahc isa dev/aic7xxx/ahc_pci.c optional ahc pci \ compile-with "${NORMAL_C} ${NO_WCONSTANT_CONVERSION}" dev/aic7xxx/ahd_pci.c optional ahd pci \ compile-with "${NORMAL_C} ${NO_WCONSTANT_CONVERSION}" dev/aic7xxx/aic7770.c optional ahc dev/aic7xxx/aic79xx.c optional ahd pci dev/aic7xxx/aic79xx_osm.c optional ahd pci dev/aic7xxx/aic79xx_pci.c optional ahd pci dev/aic7xxx/aic79xx_reg_print.c optional ahd pci ahd_reg_pretty_print dev/aic7xxx/aic7xxx.c optional ahc dev/aic7xxx/aic7xxx_93cx6.c optional ahc dev/aic7xxx/aic7xxx_osm.c optional ahc dev/aic7xxx/aic7xxx_pci.c optional ahc pci dev/aic7xxx/aic7xxx_reg_print.c optional ahc ahc_reg_pretty_print dev/alc/if_alc.c optional alc pci dev/ale/if_ale.c optional ale pci dev/alpm/alpm.c optional alpm pci dev/altera/avgen/altera_avgen.c optional altera_avgen dev/altera/avgen/altera_avgen_fdt.c optional altera_avgen fdt dev/altera/avgen/altera_avgen_nexus.c optional altera_avgen dev/altera/sdcard/altera_sdcard.c optional altera_sdcard dev/altera/sdcard/altera_sdcard_disk.c optional altera_sdcard dev/altera/sdcard/altera_sdcard_io.c optional altera_sdcard dev/altera/sdcard/altera_sdcard_fdt.c optional altera_sdcard fdt dev/altera/sdcard/altera_sdcard_nexus.c optional altera_sdcard dev/altera/pio/pio.c optional altera_pio dev/altera/pio/pio_if.m optional altera_pio dev/amdpm/amdpm.c optional amdpm pci | nfpm pci dev/amdsmb/amdsmb.c optional amdsmb pci dev/amr/amr.c optional amr dev/amr/amr_cam.c optional amrp amr dev/amr/amr_disk.c optional amr dev/amr/amr_linux.c optional amr compat_linux dev/amr/amr_pci.c optional amr pci dev/an/if_an.c optional an dev/an/if_an_isa.c optional an isa dev/an/if_an_pccard.c optional an pccard dev/an/if_an_pci.c optional an pci dev/asr/asr.c optional asr pci \ compile-with "${NORMAL_C} ${NO_WARRAY_BOUNDS}" # dev/ata/ata_if.m optional ata | atacore dev/ata/ata-all.c optional ata | atacore dev/ata/ata-dma.c optional ata | atacore dev/ata/ata-lowlevel.c optional ata | atacore dev/ata/ata-sata.c optional ata | atacore dev/ata/ata-card.c optional ata pccard | atapccard dev/ata/ata-cbus.c optional ata pc98 | atapc98 dev/ata/ata-isa.c optional ata isa | ataisa dev/ata/ata-pci.c optional ata pci | atapci dev/ata/chipsets/ata-ahci.c optional ata pci | ataahci | ataacerlabs | \ ataati | ataintel | atajmicron | \ atavia | atanvidia dev/ata/chipsets/ata-acard.c optional ata pci | ataacard dev/ata/chipsets/ata-acerlabs.c optional ata pci | ataacerlabs dev/ata/chipsets/ata-adaptec.c optional ata pci | ataadaptec dev/ata/chipsets/ata-amd.c optional ata pci | ataamd dev/ata/chipsets/ata-ati.c optional ata pci | ataati dev/ata/chipsets/ata-cenatek.c optional ata pci | atacenatek dev/ata/chipsets/ata-cypress.c optional ata pci | atacypress dev/ata/chipsets/ata-cyrix.c optional ata pci | atacyrix dev/ata/chipsets/ata-highpoint.c optional ata pci | atahighpoint dev/ata/chipsets/ata-intel.c optional ata pci | ataintel dev/ata/chipsets/ata-ite.c optional ata pci | ataite dev/ata/chipsets/ata-jmicron.c optional ata pci | atajmicron dev/ata/chipsets/ata-marvell.c optional ata pci | atamarvell | ataadaptec dev/ata/chipsets/ata-micron.c optional ata pci | atamicron dev/ata/chipsets/ata-national.c optional ata pci | atanational dev/ata/chipsets/ata-netcell.c optional ata pci | atanetcell dev/ata/chipsets/ata-nvidia.c optional ata pci | atanvidia dev/ata/chipsets/ata-promise.c optional ata pci | atapromise dev/ata/chipsets/ata-serverworks.c optional ata pci | ataserverworks dev/ata/chipsets/ata-siliconimage.c optional ata pci | atasiliconimage | ataati dev/ata/chipsets/ata-sis.c optional ata pci | atasis dev/ata/chipsets/ata-via.c optional ata pci | atavia # dev/ath/if_ath_pci.c optional ath_pci pci \ compile-with "${NORMAL_C} -I$S/dev/ath" # dev/ath/if_ath_ahb.c optional ath_ahb \ compile-with "${NORMAL_C} -I$S/dev/ath" # dev/ath/if_ath.c optional ath \ compile-with "${NORMAL_C} -I$S/dev/ath" dev/ath/if_ath_alq.c optional ath \ compile-with "${NORMAL_C} -I$S/dev/ath" dev/ath/if_ath_beacon.c optional ath \ compile-with "${NORMAL_C} -I$S/dev/ath" dev/ath/if_ath_btcoex.c optional ath \ compile-with "${NORMAL_C} -I$S/dev/ath" dev/ath/if_ath_debug.c optional ath \ compile-with "${NORMAL_C} -I$S/dev/ath" dev/ath/if_ath_keycache.c optional ath \ compile-with "${NORMAL_C} -I$S/dev/ath" dev/ath/if_ath_led.c optional ath \ compile-with "${NORMAL_C} -I$S/dev/ath" dev/ath/if_ath_lna_div.c optional ath \ compile-with "${NORMAL_C} -I$S/dev/ath" dev/ath/if_ath_tx.c optional ath \ compile-with "${NORMAL_C} -I$S/dev/ath" dev/ath/if_ath_tx_edma.c optional ath \ compile-with "${NORMAL_C} -I$S/dev/ath" dev/ath/if_ath_tx_ht.c optional ath \ compile-with "${NORMAL_C} -I$S/dev/ath" dev/ath/if_ath_tdma.c optional ath \ compile-with "${NORMAL_C} -I$S/dev/ath" dev/ath/if_ath_sysctl.c optional ath \ compile-with "${NORMAL_C} -I$S/dev/ath" dev/ath/if_ath_rx.c optional ath \ compile-with "${NORMAL_C} -I$S/dev/ath" dev/ath/if_ath_rx_edma.c optional ath \ compile-with "${NORMAL_C} -I$S/dev/ath" dev/ath/if_ath_spectral.c optional ath \ compile-with "${NORMAL_C} -I$S/dev/ath" dev/ath/ah_osdep.c optional ath \ compile-with "${NORMAL_C} -I$S/dev/ath" # dev/ath/ath_hal/ah.c optional ath \ compile-with "${NORMAL_C} -I$S/dev/ath" dev/ath/ath_hal/ah_eeprom_v1.c optional ath_hal | ath_ar5210 \ compile-with "${NORMAL_C} -I$S/dev/ath" dev/ath/ath_hal/ah_eeprom_v3.c optional ath_hal | ath_ar5211 | ath_ar5212 \ compile-with "${NORMAL_C} -I$S/dev/ath" dev/ath/ath_hal/ah_eeprom_v14.c \ optional ath_hal | ath_ar5416 | ath_ar9160 | ath_ar9280 \ compile-with "${NORMAL_C} -I$S/dev/ath" dev/ath/ath_hal/ah_eeprom_v4k.c \ optional ath_hal | ath_ar9285 \ compile-with "${NORMAL_C} -I$S/dev/ath" dev/ath/ath_hal/ah_eeprom_9287.c \ optional ath_hal | ath_ar9287 \ compile-with "${NORMAL_C} -I$S/dev/ath" dev/ath/ath_hal/ah_regdomain.c optional ath \ compile-with "${NORMAL_C} ${NO_WSHIFT_COUNT_NEGATIVE} ${NO_WSHIFT_COUNT_OVERFLOW} -I$S/dev/ath" # ar5210 dev/ath/ath_hal/ar5210/ar5210_attach.c optional ath_hal | ath_ar5210 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar5210/ar5210_beacon.c optional ath_hal | ath_ar5210 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar5210/ar5210_interrupts.c optional ath_hal | ath_ar5210 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar5210/ar5210_keycache.c optional ath_hal | ath_ar5210 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar5210/ar5210_misc.c optional ath_hal | ath_ar5210 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar5210/ar5210_phy.c optional ath_hal | ath_ar5210 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar5210/ar5210_power.c optional ath_hal | ath_ar5210 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar5210/ar5210_recv.c optional ath_hal | ath_ar5210 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar5210/ar5210_reset.c optional ath_hal | ath_ar5210 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar5210/ar5210_xmit.c optional ath_hal | ath_ar5210 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" # ar5211 dev/ath/ath_hal/ar5211/ar5211_attach.c optional ath_hal | ath_ar5211 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar5211/ar5211_beacon.c optional ath_hal | ath_ar5211 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar5211/ar5211_interrupts.c optional ath_hal | ath_ar5211 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar5211/ar5211_keycache.c optional ath_hal | ath_ar5211 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar5211/ar5211_misc.c optional ath_hal | ath_ar5211 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar5211/ar5211_phy.c optional ath_hal | ath_ar5211 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar5211/ar5211_power.c optional ath_hal | ath_ar5211 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar5211/ar5211_recv.c optional ath_hal | ath_ar5211 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar5211/ar5211_reset.c optional ath_hal | ath_ar5211 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar5211/ar5211_xmit.c optional ath_hal | ath_ar5211 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" # ar5212 dev/ath/ath_hal/ar5212/ar5212_ani.c \ optional ath_hal | ath_ar5212 | ath_ar5416 | ath_ar9160 | ath_ar9280 | \ ath_ar9285 ath_ar9287 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar5212/ar5212_attach.c \ optional ath_hal | ath_ar5212 | ath_ar5416 | ath_ar9160 | ath_ar9280 | \ ath_ar9285 ath_ar9287 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar5212/ar5212_beacon.c \ optional ath_hal | ath_ar5212 | ath_ar5416 | ath_ar9160 | ath_ar9280 | \ ath_ar9285 ath_ar9287 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar5212/ar5212_eeprom.c \ optional ath_hal | ath_ar5212 | ath_ar5416 | ath_ar9160 | ath_ar9280 | \ ath_ar9285 ath_ar9287 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar5212/ar5212_gpio.c \ optional ath_hal | ath_ar5212 | ath_ar5416 | ath_ar9160 | ath_ar9280 | \ ath_ar9285 ath_ar9287 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar5212/ar5212_interrupts.c \ optional ath_hal | ath_ar5212 | ath_ar5416 | ath_ar9160 | ath_ar9280 | \ ath_ar9285 ath_ar9287 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar5212/ar5212_keycache.c \ optional ath_hal | ath_ar5212 | ath_ar5416 | ath_ar9160 | ath_ar9280 | \ ath_ar9285 ath_ar9287 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar5212/ar5212_misc.c \ optional ath_hal | ath_ar5212 | ath_ar5416 | ath_ar9160 | ath_ar9280 | \ ath_ar9285 ath_ar9287 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar5212/ar5212_phy.c \ optional ath_hal | ath_ar5212 | ath_ar5416 | ath_ar9160 | ath_ar9280 | \ ath_ar9285 ath_ar9287 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar5212/ar5212_power.c \ optional ath_hal | ath_ar5212 | ath_ar5416 | ath_ar9160 | ath_ar9280 | \ ath_ar9285 ath_ar9287 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar5212/ar5212_recv.c \ optional ath_hal | ath_ar5212 | ath_ar5416 | ath_ar9160 | ath_ar9280 | \ ath_ar9285 ath_ar9287 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar5212/ar5212_reset.c \ optional ath_hal | ath_ar5212 | ath_ar5416 | ath_ar9160 | ath_ar9280 | \ ath_ar9285 ath_ar9287 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar5212/ar5212_rfgain.c \ optional ath_hal | ath_ar5212 | ath_ar5416 | ath_ar9160 | ath_ar9280 | \ ath_ar9285 ath_ar9287 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar5212/ar5212_xmit.c \ optional ath_hal | ath_ar5212 | ath_ar5416 | ath_ar9160 | ath_ar9280 | \ ath_ar9285 ath_ar9287 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" # ar5416 (depends on ar5212) dev/ath/ath_hal/ar5416/ar5416_ani.c \ optional ath_hal | ath_ar5416 | ath_ar9160 | ath_ar9280 | ath_ar9285 | \ ath_ar9287 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar5416/ar5416_attach.c \ optional ath_hal | ath_ar5416 | ath_ar9160 | ath_ar9280 | ath_ar9285 | \ ath_ar9287 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar5416/ar5416_beacon.c \ optional ath_hal | ath_ar5416 | ath_ar9160 | ath_ar9280 | ath_ar9285 | \ ath_ar9287 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar5416/ar5416_btcoex.c \ optional ath_hal | ath_ar5416 | ath_ar9160 | ath_ar9280 | ath_ar9285 | \ ath_ar9287 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar5416/ar5416_cal.c \ optional ath_hal | ath_ar5416 | ath_ar9160 | ath_ar9280 | ath_ar9285 | \ ath_ar9287 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar5416/ar5416_cal_iq.c \ optional ath_hal | ath_ar5416 | ath_ar9160 | ath_ar9280 | ath_ar9285 | \ ath_ar9287 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar5416/ar5416_cal_adcgain.c \ optional ath_hal | ath_ar5416 | ath_ar9160 | ath_ar9280 | ath_ar9285 | \ ath_ar9287 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar5416/ar5416_cal_adcdc.c \ optional ath_hal | ath_ar5416 | ath_ar9160 | ath_ar9280 | ath_ar9285 | \ ath_ar9287 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar5416/ar5416_eeprom.c \ optional ath_hal | ath_ar5416 | ath_ar9160 | ath_ar9280 | ath_ar9285 | \ ath_ar9287 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar5416/ar5416_gpio.c \ optional ath_hal | ath_ar5416 | ath_ar9160 | ath_ar9280 | ath_ar9285 | \ ath_ar9287 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar5416/ar5416_interrupts.c \ optional ath_hal | ath_ar5416 | ath_ar9160 | ath_ar9280 | ath_ar9285 | \ ath_ar9287 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar5416/ar5416_keycache.c \ optional ath_hal | ath_ar5416 | ath_ar9160 | ath_ar9280 | ath_ar9285 | \ ath_ar9287 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar5416/ar5416_misc.c \ optional ath_hal | ath_ar5416 | ath_ar9160 | ath_ar9280 | ath_ar9285 | \ ath_ar9287 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar5416/ar5416_phy.c \ optional ath_hal | ath_ar5416 | ath_ar9160 | ath_ar9280 | ath_ar9285 | \ ath_ar9287 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar5416/ar5416_power.c \ optional ath_hal | ath_ar5416 | ath_ar9160 | ath_ar9280 | ath_ar9285 | \ ath_ar9287 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar5416/ar5416_radar.c \ optional ath_hal | ath_ar5416 | ath_ar9160 | ath_ar9280 | ath_ar9285 | \ ath_ar9287 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar5416/ar5416_recv.c \ optional ath_hal | ath_ar5416 | ath_ar9160 | ath_ar9280 | ath_ar9285 | \ ath_ar9287 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar5416/ar5416_reset.c \ optional ath_hal | ath_ar5416 | ath_ar9160 | ath_ar9280 | ath_ar9285 | \ ath_ar9287 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar5416/ar5416_spectral.c \ optional ath_hal | ath_ar5416 | ath_ar9160 | ath_ar9280 | ath_ar9285 | \ ath_ar9287 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar5416/ar5416_xmit.c \ optional ath_hal | ath_ar5416 | ath_ar9160 | ath_ar9280 | ath_ar9285 | \ ath_ar9287 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" # ar9130 (depends upon ar5416) - also requires AH_SUPPORT_AR9130 # # Since this is an embedded MAC SoC, there's no need to compile it into the # default HAL. dev/ath/ath_hal/ar9001/ar9130_attach.c optional ath_ar9130 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar9001/ar9130_phy.c optional ath_ar9130 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar9001/ar9130_eeprom.c optional ath_ar9130 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" # ar9160 (depends on ar5416) dev/ath/ath_hal/ar9001/ar9160_attach.c optional ath_hal | ath_ar9160 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" # ar9280 (depends on ar5416) dev/ath/ath_hal/ar9002/ar9280_attach.c optional ath_hal | ath_ar9280 | \ ath_ar9285 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar9002/ar9280_olc.c optional ath_hal | ath_ar9280 | \ ath_ar9285 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" # ar9285 (depends on ar5416 and ar9280) dev/ath/ath_hal/ar9002/ar9285_attach.c optional ath_hal | ath_ar9285 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar9002/ar9285_btcoex.c optional ath_hal | ath_ar9285 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar9002/ar9285_reset.c optional ath_hal | ath_ar9285 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar9002/ar9285_cal.c optional ath_hal | ath_ar9285 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar9002/ar9285_phy.c optional ath_hal | ath_ar9285 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar9002/ar9285_diversity.c optional ath_hal | ath_ar9285 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" # ar9287 (depends on ar5416) dev/ath/ath_hal/ar9002/ar9287_attach.c optional ath_hal | ath_ar9287 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar9002/ar9287_reset.c optional ath_hal | ath_ar9287 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar9002/ar9287_cal.c optional ath_hal | ath_ar9287 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar9002/ar9287_olc.c optional ath_hal | ath_ar9287 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" # ar9300 contrib/dev/ath/ath_hal/ar9300/ar9300_ani.c optional ath_hal | ath_ar9300 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal -I$S/contrib/dev/ath/ath_hal" contrib/dev/ath/ath_hal/ar9300/ar9300_attach.c optional ath_hal | ath_ar9300 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal -I$S/contrib/dev/ath/ath_hal" contrib/dev/ath/ath_hal/ar9300/ar9300_beacon.c optional ath_hal | ath_ar9300 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal -I$S/contrib/dev/ath/ath_hal" contrib/dev/ath/ath_hal/ar9300/ar9300_eeprom.c optional ath_hal | ath_ar9300 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal -I$S/contrib/dev/ath/ath_hal ${NO_WCONSTANT_CONVERSION}" contrib/dev/ath/ath_hal/ar9300/ar9300_freebsd.c optional ath_hal | ath_ar9300 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal -I$S/contrib/dev/ath/ath_hal" contrib/dev/ath/ath_hal/ar9300/ar9300_gpio.c optional ath_hal | ath_ar9300 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal -I$S/contrib/dev/ath/ath_hal" contrib/dev/ath/ath_hal/ar9300/ar9300_interrupts.c optional ath_hal | ath_ar9300 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal -I$S/contrib/dev/ath/ath_hal" contrib/dev/ath/ath_hal/ar9300/ar9300_keycache.c optional ath_hal | ath_ar9300 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal -I$S/contrib/dev/ath/ath_hal" contrib/dev/ath/ath_hal/ar9300/ar9300_mci.c optional ath_hal | ath_ar9300 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal -I$S/contrib/dev/ath/ath_hal" contrib/dev/ath/ath_hal/ar9300/ar9300_misc.c optional ath_hal | ath_ar9300 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal -I$S/contrib/dev/ath/ath_hal" contrib/dev/ath/ath_hal/ar9300/ar9300_paprd.c optional ath_hal | ath_ar9300 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal -I$S/contrib/dev/ath/ath_hal" contrib/dev/ath/ath_hal/ar9300/ar9300_phy.c optional ath_hal | ath_ar9300 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal -I$S/contrib/dev/ath/ath_hal" contrib/dev/ath/ath_hal/ar9300/ar9300_power.c optional ath_hal | ath_ar9300 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal -I$S/contrib/dev/ath/ath_hal" contrib/dev/ath/ath_hal/ar9300/ar9300_radar.c optional ath_hal | ath_ar9300 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal -I$S/contrib/dev/ath/ath_hal" contrib/dev/ath/ath_hal/ar9300/ar9300_radio.c optional ath_hal | ath_ar9300 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal -I$S/contrib/dev/ath/ath_hal" contrib/dev/ath/ath_hal/ar9300/ar9300_recv.c optional ath_hal | ath_ar9300 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal -I$S/contrib/dev/ath/ath_hal" contrib/dev/ath/ath_hal/ar9300/ar9300_recv_ds.c optional ath_hal | ath_ar9300 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal -I$S/contrib/dev/ath/ath_hal" contrib/dev/ath/ath_hal/ar9300/ar9300_reset.c optional ath_hal | ath_ar9300 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal -I$S/contrib/dev/ath/ath_hal ${NO_WSOMETIMES_UNINITIALIZED} -Wno-unused-function" contrib/dev/ath/ath_hal/ar9300/ar9300_stub.c optional ath_hal | ath_ar9300 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal -I$S/contrib/dev/ath/ath_hal" contrib/dev/ath/ath_hal/ar9300/ar9300_stub_funcs.c optional ath_hal | ath_ar9300 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal -I$S/contrib/dev/ath/ath_hal" contrib/dev/ath/ath_hal/ar9300/ar9300_timer.c optional ath_hal | ath_ar9300 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal -I$S/contrib/dev/ath/ath_hal" contrib/dev/ath/ath_hal/ar9300/ar9300_xmit.c optional ath_hal | ath_ar9300 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal -I$S/contrib/dev/ath/ath_hal" contrib/dev/ath/ath_hal/ar9300/ar9300_xmit_ds.c optional ath_hal | ath_ar9300 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal -I$S/contrib/dev/ath/ath_hal" # rf backends dev/ath/ath_hal/ar5212/ar2316.c optional ath_rf2316 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar5212/ar2317.c optional ath_rf2317 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar5212/ar2413.c optional ath_hal | ath_rf2413 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar5212/ar2425.c optional ath_hal | ath_rf2425 | ath_rf2417 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar5212/ar5111.c optional ath_hal | ath_rf5111 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar5212/ar5112.c optional ath_hal | ath_rf5112 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar5212/ar5413.c optional ath_hal | ath_rf5413 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar5416/ar2133.c optional ath_hal | ath_ar5416 | \ ath_ar9130 | ath_ar9160 | ath_ar9280 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar9002/ar9280.c optional ath_hal | ath_ar9280 | ath_ar9285 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar9002/ar9285.c optional ath_hal | ath_ar9285 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" dev/ath/ath_hal/ar9002/ar9287.c optional ath_hal | ath_ar9287 \ compile-with "${NORMAL_C} -I$S/dev/ath -I$S/dev/ath/ath_hal" # ath rate control algorithms dev/ath/ath_rate/amrr/amrr.c optional ath_rate_amrr \ compile-with "${NORMAL_C} -I$S/dev/ath" dev/ath/ath_rate/onoe/onoe.c optional ath_rate_onoe \ compile-with "${NORMAL_C} -I$S/dev/ath" dev/ath/ath_rate/sample/sample.c optional ath_rate_sample \ compile-with "${NORMAL_C} -I$S/dev/ath" # ath DFS modules dev/ath/ath_dfs/null/dfs_null.c optional ath \ compile-with "${NORMAL_C} -I$S/dev/ath" # dev/bce/if_bce.c optional bce dev/bfe/if_bfe.c optional bfe dev/bge/if_bge.c optional bge dev/bktr/bktr_audio.c optional bktr pci dev/bktr/bktr_card.c optional bktr pci dev/bktr/bktr_core.c optional bktr pci dev/bktr/bktr_i2c.c optional bktr pci smbus dev/bktr/bktr_os.c optional bktr pci dev/bktr/bktr_tuner.c optional bktr pci dev/bktr/msp34xx.c optional bktr pci dev/buslogic/bt.c optional bt dev/buslogic/bt_eisa.c optional bt eisa dev/buslogic/bt_isa.c optional bt isa dev/buslogic/bt_mca.c optional bt mca dev/buslogic/bt_pci.c optional bt pci dev/bwi/bwimac.c optional bwi dev/bwi/bwiphy.c optional bwi dev/bwi/bwirf.c optional bwi dev/bwi/if_bwi.c optional bwi dev/bwi/if_bwi_pci.c optional bwi pci # XXX Work around clang warning, until maintainer approves fix. dev/bwn/if_bwn.c optional bwn siba_bwn \ compile-with "${NORMAL_C} ${NO_WSOMETIMES_UNINITIALIZED}" dev/cardbus/cardbus.c optional cardbus dev/cardbus/cardbus_cis.c optional cardbus dev/cardbus/cardbus_device.c optional cardbus dev/cas/if_cas.c optional cas dev/cfi/cfi_bus_fdt.c optional cfi fdt dev/cfi/cfi_bus_nexus.c optional cfi dev/cfi/cfi_core.c optional cfi dev/cfi/cfi_dev.c optional cfi dev/cfi/cfi_disk.c optional cfid dev/ciss/ciss.c optional ciss dev/cm/smc90cx6.c optional cm dev/cmx/cmx.c optional cmx dev/cmx/cmx_pccard.c optional cmx pccard dev/cpufreq/ichss.c optional cpufreq dev/cs/if_cs.c optional cs dev/cs/if_cs_isa.c optional cs isa dev/cs/if_cs_pccard.c optional cs pccard dev/cxgb/cxgb_main.c optional cxgb pci \ compile-with "${NORMAL_C} -I$S/dev/cxgb" dev/cxgb/cxgb_sge.c optional cxgb pci \ compile-with "${NORMAL_C} -I$S/dev/cxgb" dev/cxgb/common/cxgb_mc5.c optional cxgb pci \ compile-with "${NORMAL_C} -I$S/dev/cxgb" dev/cxgb/common/cxgb_vsc7323.c optional cxgb pci \ compile-with "${NORMAL_C} -I$S/dev/cxgb" dev/cxgb/common/cxgb_vsc8211.c optional cxgb pci \ compile-with "${NORMAL_C} -I$S/dev/cxgb" dev/cxgb/common/cxgb_ael1002.c optional cxgb pci \ compile-with "${NORMAL_C} -I$S/dev/cxgb" dev/cxgb/common/cxgb_aq100x.c optional cxgb pci \ compile-with "${NORMAL_C} -I$S/dev/cxgb" dev/cxgb/common/cxgb_mv88e1xxx.c optional cxgb pci \ compile-with "${NORMAL_C} -I$S/dev/cxgb" dev/cxgb/common/cxgb_xgmac.c optional cxgb pci \ compile-with "${NORMAL_C} -I$S/dev/cxgb" dev/cxgb/common/cxgb_t3_hw.c optional cxgb pci \ compile-with "${NORMAL_C} -I$S/dev/cxgb" dev/cxgb/common/cxgb_tn1010.c optional cxgb pci \ compile-with "${NORMAL_C} -I$S/dev/cxgb" dev/cxgb/sys/uipc_mvec.c optional cxgb pci \ compile-with "${NORMAL_C} -I$S/dev/cxgb" dev/cxgb/cxgb_t3fw.c optional cxgb cxgb_t3fw \ compile-with "${NORMAL_C} -I$S/dev/cxgb" dev/cxgbe/t4_main.c optional cxgbe pci \ compile-with "${NORMAL_C} -I$S/dev/cxgbe" dev/cxgbe/t4_netmap.c optional cxgbe pci \ compile-with "${NORMAL_C} -I$S/dev/cxgbe" dev/cxgbe/t4_sge.c optional cxgbe pci \ compile-with "${NORMAL_C} -I$S/dev/cxgbe" dev/cxgbe/t4_l2t.c optional cxgbe pci \ compile-with "${NORMAL_C} -I$S/dev/cxgbe" dev/cxgbe/t4_tracer.c optional cxgbe pci \ compile-with "${NORMAL_C} -I$S/dev/cxgbe" dev/cxgbe/common/t4_hw.c optional cxgbe pci \ compile-with "${NORMAL_C} -I$S/dev/cxgbe" t4fw_cfg.c optional cxgbe \ compile-with "${AWK} -f $S/tools/fw_stub.awk t4fw_cfg.fw:t4fw_cfg t4fw_cfg_uwire.fw:t4fw_cfg_uwire t4fw.fw:t4fw -mt4fw_cfg -c${.TARGET}" \ no-implicit-rule before-depend local \ clean "t4fw_cfg.c" t4fw_cfg.fwo optional cxgbe \ dependency "t4fw_cfg.fw" \ compile-with "${NORMAL_FWO}" \ no-implicit-rule \ clean "t4fw_cfg.fwo" t4fw_cfg.fw optional cxgbe \ dependency "$S/dev/cxgbe/firmware/t4fw_cfg.txt" \ compile-with "${CP} ${.ALLSRC} ${.TARGET}" \ no-obj no-implicit-rule \ clean "t4fw_cfg.fw" t4fw_cfg_uwire.fwo optional cxgbe \ dependency "t4fw_cfg_uwire.fw" \ compile-with "${NORMAL_FWO}" \ no-implicit-rule \ clean "t4fw_cfg_uwire.fwo" t4fw_cfg_uwire.fw optional cxgbe \ dependency "$S/dev/cxgbe/firmware/t4fw_cfg_uwire.txt" \ compile-with "${CP} ${.ALLSRC} ${.TARGET}" \ no-obj no-implicit-rule \ clean "t4fw_cfg_uwire.fw" t4fw.fwo optional cxgbe \ dependency "t4fw.fw" \ compile-with "${NORMAL_FWO}" \ no-implicit-rule \ clean "t4fw.fwo" t4fw.fw optional cxgbe \ dependency "$S/dev/cxgbe/firmware/t4fw-1.11.27.0.bin.uu" \ compile-with "${NORMAL_FW}" \ no-obj no-implicit-rule \ clean "t4fw.fw" t5fw_cfg.c optional cxgbe \ compile-with "${AWK} -f $S/tools/fw_stub.awk t5fw_cfg.fw:t5fw_cfg t5fw.fw:t5fw -mt5fw_cfg -c${.TARGET}" \ no-implicit-rule before-depend local \ clean "t5fw_cfg.c" t5fw_cfg.fwo optional cxgbe \ dependency "t5fw_cfg.fw" \ compile-with "${NORMAL_FWO}" \ no-implicit-rule \ clean "t5fw_cfg.fwo" t5fw_cfg.fw optional cxgbe \ dependency "$S/dev/cxgbe/firmware/t5fw_cfg.txt" \ compile-with "${CP} ${.ALLSRC} ${.TARGET}" \ no-obj no-implicit-rule \ clean "t5fw_cfg.fw" t5fw.fwo optional cxgbe \ dependency "t5fw.fw" \ compile-with "${NORMAL_FWO}" \ no-implicit-rule \ clean "t5fw.fwo" t5fw.fw optional cxgbe \ dependency "$S/dev/cxgbe/firmware/t5fw-1.11.27.0.bin.uu" \ compile-with "${NORMAL_FW}" \ no-obj no-implicit-rule \ clean "t5fw.fw" dev/cy/cy.c optional cy dev/cy/cy_isa.c optional cy isa dev/cy/cy_pci.c optional cy pci dev/dc/if_dc.c optional dc pci dev/dc/dcphy.c optional dc pci dev/dc/pnphy.c optional dc pci dev/dcons/dcons.c optional dcons dev/dcons/dcons_crom.c optional dcons_crom dev/dcons/dcons_os.c optional dcons dev/de/if_de.c optional de pci dev/digi/CX.c optional digi_CX dev/digi/CX_PCI.c optional digi_CX_PCI dev/digi/EPCX.c optional digi_EPCX dev/digi/EPCX_PCI.c optional digi_EPCX_PCI dev/digi/Xe.c optional digi_Xe dev/digi/Xem.c optional digi_Xem dev/digi/Xr.c optional digi_Xr dev/digi/digi.c optional digi dev/digi/digi_isa.c optional digi isa dev/digi/digi_pci.c optional digi pci dev/dpt/dpt_eisa.c optional dpt eisa dev/dpt/dpt_pci.c optional dpt pci dev/dpt/dpt_scsi.c optional dpt dev/drm/ati_pcigart.c optional drm dev/drm/drm_agpsupport.c optional drm dev/drm/drm_auth.c optional drm dev/drm/drm_bufs.c optional drm dev/drm/drm_context.c optional drm dev/drm/drm_dma.c optional drm dev/drm/drm_drawable.c optional drm dev/drm/drm_drv.c optional drm dev/drm/drm_fops.c optional drm dev/drm/drm_hashtab.c optional drm dev/drm/drm_ioctl.c optional drm dev/drm/drm_irq.c optional drm dev/drm/drm_lock.c optional drm dev/drm/drm_memory.c optional drm dev/drm/drm_mm.c optional drm dev/drm/drm_pci.c optional drm dev/drm/drm_scatter.c optional drm dev/drm/drm_sman.c optional drm dev/drm/drm_sysctl.c optional drm dev/drm/drm_vm.c optional drm dev/drm/i915_dma.c optional i915drm dev/drm/i915_drv.c optional i915drm dev/drm/i915_irq.c optional i915drm dev/drm/i915_mem.c optional i915drm dev/drm/i915_suspend.c optional i915drm dev/drm/mach64_dma.c optional mach64drm dev/drm/mach64_drv.c optional mach64drm dev/drm/mach64_irq.c optional mach64drm dev/drm/mach64_state.c optional mach64drm dev/drm/mga_dma.c optional mgadrm dev/drm/mga_drv.c optional mgadrm dev/drm/mga_irq.c optional mgadrm dev/drm/mga_state.c optional mgadrm dev/drm/mga_warp.c optional mgadrm dev/drm/r128_cce.c optional r128drm \ compile-with "${NORMAL_C} ${NO_WUNUSED_VALUE} ${NO_WCONSTANT_CONVERSION}" dev/drm/r128_drv.c optional r128drm dev/drm/r128_irq.c optional r128drm dev/drm/r128_state.c optional r128drm \ compile-with "${NORMAL_C} ${NO_WUNUSED_VALUE}" dev/drm/r300_cmdbuf.c optional radeondrm dev/drm/r600_blit.c optional radeondrm dev/drm/r600_cp.c optional radeondrm \ compile-with "${NORMAL_C} ${NO_WUNUSED_VALUE} ${NO_WCONSTANT_CONVERSION}" dev/drm/radeon_cp.c optional radeondrm \ compile-with "${NORMAL_C} ${NO_WUNUSED_VALUE} ${NO_WCONSTANT_CONVERSION}" dev/drm/radeon_cs.c optional radeondrm dev/drm/radeon_drv.c optional radeondrm dev/drm/radeon_irq.c optional radeondrm dev/drm/radeon_mem.c optional radeondrm dev/drm/radeon_state.c optional radeondrm dev/drm/savage_bci.c optional savagedrm dev/drm/savage_drv.c optional savagedrm dev/drm/savage_state.c optional savagedrm dev/drm/sis_drv.c optional sisdrm dev/drm/sis_ds.c optional sisdrm dev/drm/sis_mm.c optional sisdrm dev/drm/tdfx_drv.c optional tdfxdrm dev/drm/via_dma.c optional viadrm dev/drm/via_dmablit.c optional viadrm dev/drm/via_drv.c optional viadrm dev/drm/via_irq.c optional viadrm dev/drm/via_map.c optional viadrm dev/drm/via_mm.c optional viadrm dev/drm/via_verifier.c optional viadrm dev/drm/via_video.c optional viadrm dev/ed/if_ed.c optional ed dev/ed/if_ed_novell.c optional ed dev/ed/if_ed_rtl80x9.c optional ed dev/ed/if_ed_pccard.c optional ed pccard dev/ed/if_ed_pci.c optional ed pci dev/eisa/eisa_if.m standard dev/eisa/eisaconf.c optional eisa dev/e1000/if_em.c optional em \ compile-with "${NORMAL_C} -I$S/dev/e1000" dev/e1000/if_lem.c optional em \ compile-with "${NORMAL_C} -I$S/dev/e1000" dev/e1000/if_igb.c optional igb \ compile-with "${NORMAL_C} -I$S/dev/e1000" dev/e1000/e1000_80003es2lan.c optional em | igb \ compile-with "${NORMAL_C} -I$S/dev/e1000" dev/e1000/e1000_82540.c optional em | igb \ compile-with "${NORMAL_C} -I$S/dev/e1000" dev/e1000/e1000_82541.c optional em | igb \ compile-with "${NORMAL_C} -I$S/dev/e1000" dev/e1000/e1000_82542.c optional em | igb \ compile-with "${NORMAL_C} -I$S/dev/e1000" dev/e1000/e1000_82543.c optional em | igb \ compile-with "${NORMAL_C} -I$S/dev/e1000" dev/e1000/e1000_82571.c optional em | igb \ compile-with "${NORMAL_C} -I$S/dev/e1000" dev/e1000/e1000_82575.c optional em | igb \ compile-with "${NORMAL_C} -I$S/dev/e1000" dev/e1000/e1000_ich8lan.c optional em | igb \ compile-with "${NORMAL_C} -I$S/dev/e1000" dev/e1000/e1000_i210.c optional em | igb \ compile-with "${NORMAL_C} -I$S/dev/e1000" dev/e1000/e1000_api.c optional em | igb \ compile-with "${NORMAL_C} -I$S/dev/e1000" dev/e1000/e1000_mac.c optional em | igb \ compile-with "${NORMAL_C} -I$S/dev/e1000" dev/e1000/e1000_manage.c optional em | igb \ compile-with "${NORMAL_C} -I$S/dev/e1000" dev/e1000/e1000_nvm.c optional em | igb \ compile-with "${NORMAL_C} -I$S/dev/e1000" dev/e1000/e1000_phy.c optional em | igb \ compile-with "${NORMAL_C} -I$S/dev/e1000" dev/e1000/e1000_vf.c optional em | igb \ compile-with "${NORMAL_C} -I$S/dev/e1000" dev/e1000/e1000_mbx.c optional em | igb \ compile-with "${NORMAL_C} -I$S/dev/e1000" dev/e1000/e1000_osdep.c optional em | igb \ compile-with "${NORMAL_C} -I$S/dev/e1000" dev/et/if_et.c optional et dev/en/if_en_pci.c optional en pci dev/en/midway.c optional en dev/ep/if_ep.c optional ep dev/ep/if_ep_eisa.c optional ep eisa dev/ep/if_ep_isa.c optional ep isa dev/ep/if_ep_mca.c optional ep mca dev/ep/if_ep_pccard.c optional ep pccard dev/esp/esp_pci.c optional esp pci dev/esp/ncr53c9x.c optional esp dev/etherswitch/arswitch/arswitch.c optional arswitch dev/etherswitch/arswitch/arswitch_reg.c optional arswitch dev/etherswitch/arswitch/arswitch_phy.c optional arswitch dev/etherswitch/arswitch/arswitch_8216.c optional arswitch dev/etherswitch/arswitch/arswitch_8226.c optional arswitch dev/etherswitch/arswitch/arswitch_8316.c optional arswitch dev/etherswitch/arswitch/arswitch_8327.c optional arswitch dev/etherswitch/arswitch/arswitch_7240.c optional arswitch dev/etherswitch/arswitch/arswitch_9340.c optional arswitch dev/etherswitch/arswitch/arswitch_vlans.c optional arswitch dev/etherswitch/etherswitch.c optional etherswitch dev/etherswitch/etherswitch_if.m optional etherswitch dev/etherswitch/ip17x/ip17x.c optional ip17x dev/etherswitch/ip17x/ip175c.c optional ip17x dev/etherswitch/ip17x/ip175d.c optional ip17x dev/etherswitch/ip17x/ip17x_phy.c optional ip17x dev/etherswitch/ip17x/ip17x_vlans.c optional ip17x dev/etherswitch/mdio_if.m optional miiproxy dev/etherswitch/mdio.c optional miiproxy dev/etherswitch/miiproxy.c optional miiproxy dev/etherswitch/rtl8366/rtl8366rb.c optional rtl8366rb dev/etherswitch/ukswitch/ukswitch.c optional ukswitch dev/ex/if_ex.c optional ex dev/ex/if_ex_isa.c optional ex isa dev/ex/if_ex_pccard.c optional ex pccard dev/exca/exca.c optional cbb dev/fatm/if_fatm.c optional fatm pci dev/fb/fbd.c optional fbd | vt dev/fb/fb_if.m standard dev/fb/splash.c optional sc splash dev/fdt/fdt_clock.c optional fdt fdt_clock dev/fdt/fdt_clock_if.m optional fdt fdt_clock dev/fdt/fdt_common.c optional fdt dev/fdt/fdt_pinctrl.c optional fdt fdt_pinctrl dev/fdt/fdt_pinctrl_if.m optional fdt fdt_pinctrl dev/fdt/fdt_slicer.c optional fdt cfi | fdt nand dev/fdt/fdt_static_dtb.S optional fdt fdt_dtb_static \ dependency "$S/boot/fdt/dts/${MACHINE}/${FDT_DTS_FILE}" dev/fdt/simplebus.c optional fdt dev/fe/if_fe.c optional fe dev/fe/if_fe_pccard.c optional fe pccard dev/filemon/filemon.c optional filemon dev/firewire/firewire.c optional firewire dev/firewire/fwcrom.c optional firewire dev/firewire/fwdev.c optional firewire dev/firewire/fwdma.c optional firewire dev/firewire/fwmem.c optional firewire dev/firewire/fwohci.c optional firewire dev/firewire/fwohci_pci.c optional firewire pci dev/firewire/if_fwe.c optional fwe dev/firewire/if_fwip.c optional fwip dev/firewire/sbp.c optional sbp dev/firewire/sbp_targ.c optional sbp_targ dev/flash/at45d.c optional at45d dev/flash/mx25l.c optional mx25l dev/fxp/if_fxp.c optional fxp dev/fxp/inphy.c optional fxp dev/gem/if_gem.c optional gem dev/gem/if_gem_pci.c optional gem pci dev/gem/if_gem_sbus.c optional gem sbus dev/gpio/gpiobus.c optional gpio \ dependency "gpiobus_if.h" dev/gpio/gpioc.c optional gpio \ dependency "gpio_if.h" dev/gpio/gpioiic.c optional gpioiic dev/gpio/gpioled.c optional gpioled dev/gpio/gpio_if.m optional gpio dev/gpio/gpiobus_if.m optional gpio dev/gpio/ofw_gpiobus.c optional fdt gpio dev/hatm/if_hatm.c optional hatm pci dev/hatm/if_hatm_intr.c optional hatm pci dev/hatm/if_hatm_ioctl.c optional hatm pci dev/hatm/if_hatm_rx.c optional hatm pci dev/hatm/if_hatm_tx.c optional hatm pci dev/hifn/hifn7751.c optional hifn dev/hme/if_hme.c optional hme dev/hme/if_hme_pci.c optional hme pci dev/hme/if_hme_sbus.c optional hme sbus dev/hptiop/hptiop.c optional hptiop scbus dev/hwpmc/hwpmc_logging.c optional hwpmc dev/hwpmc/hwpmc_mod.c optional hwpmc dev/hwpmc/hwpmc_soft.c optional hwpmc dev/ichsmb/ichsmb.c optional ichsmb dev/ichsmb/ichsmb_pci.c optional ichsmb pci dev/ida/ida.c optional ida dev/ida/ida_disk.c optional ida dev/ida/ida_eisa.c optional ida eisa dev/ida/ida_pci.c optional ida pci dev/ie/if_ie.c optional ie isa nowerror dev/ie/if_ie_isa.c optional ie isa dev/ieee488/ibfoo.c optional pcii | tnt4882 dev/ieee488/pcii.c optional pcii dev/ieee488/tnt4882.c optional tnt4882 dev/ieee488/upd7210.c optional pcii | tnt4882 dev/iicbus/ad7418.c optional ad7418 dev/iicbus/ds133x.c optional ds133x dev/iicbus/ds1374.c optional ds1374 dev/iicbus/ds1672.c optional ds1672 dev/iicbus/icee.c optional icee dev/iicbus/if_ic.c optional ic dev/iicbus/iic.c optional iic dev/iicbus/iicbb.c optional iicbb dev/iicbus/iicbb_if.m optional iicbb dev/iicbus/iicbus.c optional iicbus dev/iicbus/iicbus_if.m optional iicbus dev/iicbus/iiconf.c optional iicbus dev/iicbus/iicsmb.c optional iicsmb \ dependency "iicbus_if.h" dev/iicbus/iicoc.c optional iicoc dev/iicbus/lm75.c optional lm75 dev/iicbus/pcf8563.c optional pcf8563 dev/iicbus/s35390a.c optional s35390a dev/iir/iir.c optional iir dev/iir/iir_ctrl.c optional iir dev/iir/iir_pci.c optional iir pci dev/intpm/intpm.c optional intpm pci # XXX Work around clang warning, until maintainer approves fix. dev/ips/ips.c optional ips \ compile-with "${NORMAL_C} ${NO_WSOMETIMES_UNINITIALIZED}" dev/ips/ips_commands.c optional ips dev/ips/ips_disk.c optional ips dev/ips/ips_ioctl.c optional ips dev/ips/ips_pci.c optional ips pci dev/ipw/if_ipw.c optional ipw ipwbssfw.c optional ipwbssfw | ipwfw \ compile-with "${AWK} -f $S/tools/fw_stub.awk ipw_bss.fw:ipw_bss:130 -lintel_ipw -mipw_bss -c${.TARGET}" \ no-implicit-rule before-depend local \ clean "ipwbssfw.c" ipw_bss.fwo optional ipwbssfw | ipwfw \ dependency "ipw_bss.fw" \ compile-with "${NORMAL_FWO}" \ no-implicit-rule \ clean "ipw_bss.fwo" ipw_bss.fw optional ipwbssfw | ipwfw \ dependency "$S/contrib/dev/ipw/ipw2100-1.3.fw.uu" \ compile-with "${NORMAL_FW}" \ no-obj no-implicit-rule \ clean "ipw_bss.fw" ipwibssfw.c optional ipwibssfw | ipwfw \ compile-with "${AWK} -f $S/tools/fw_stub.awk ipw_ibss.fw:ipw_ibss:130 -lintel_ipw -mipw_ibss -c${.TARGET}" \ no-implicit-rule before-depend local \ clean "ipwibssfw.c" ipw_ibss.fwo optional ipwibssfw | ipwfw \ dependency "ipw_ibss.fw" \ compile-with "${NORMAL_FWO}" \ no-implicit-rule \ clean "ipw_ibss.fwo" ipw_ibss.fw optional ipwibssfw | ipwfw \ dependency "$S/contrib/dev/ipw/ipw2100-1.3-i.fw.uu" \ compile-with "${NORMAL_FW}" \ no-obj no-implicit-rule \ clean "ipw_ibss.fw" ipwmonitorfw.c optional ipwmonitorfw | ipwfw \ compile-with "${AWK} -f $S/tools/fw_stub.awk ipw_monitor.fw:ipw_monitor:130 -lintel_ipw -mipw_monitor -c${.TARGET}" \ no-implicit-rule before-depend local \ clean "ipwmonitorfw.c" ipw_monitor.fwo optional ipwmonitorfw | ipwfw \ dependency "ipw_monitor.fw" \ compile-with "${NORMAL_FWO}" \ no-implicit-rule \ clean "ipw_monitor.fwo" ipw_monitor.fw optional ipwmonitorfw | ipwfw \ dependency "$S/contrib/dev/ipw/ipw2100-1.3-p.fw.uu" \ compile-with "${NORMAL_FW}" \ no-obj no-implicit-rule \ clean "ipw_monitor.fw" dev/iscsi/icl.c optional iscsi | ctl dev/iscsi/icl_proxy.c optional iscsi | ctl dev/iscsi/iscsi.c optional iscsi scbus dev/iscsi_initiator/iscsi.c optional iscsi_initiator scbus dev/iscsi_initiator/iscsi_subr.c optional iscsi_initiator scbus dev/iscsi_initiator/isc_cam.c optional iscsi_initiator scbus dev/iscsi_initiator/isc_soc.c optional iscsi_initiator scbus dev/iscsi_initiator/isc_sm.c optional iscsi_initiator scbus dev/iscsi_initiator/isc_subr.c optional iscsi_initiator scbus dev/ismt/ismt.c optional ismt dev/isp/isp.c optional isp dev/isp/isp_freebsd.c optional isp dev/isp/isp_library.c optional isp dev/isp/isp_pci.c optional isp pci dev/isp/isp_sbus.c optional isp sbus dev/isp/isp_target.c optional isp dev/ispfw/ispfw.c optional ispfw dev/iwi/if_iwi.c optional iwi iwibssfw.c optional iwibssfw | iwifw \ compile-with "${AWK} -f $S/tools/fw_stub.awk iwi_bss.fw:iwi_bss:300 -lintel_iwi -miwi_bss -c${.TARGET}" \ no-implicit-rule before-depend local \ clean "iwibssfw.c" iwi_bss.fwo optional iwibssfw | iwifw \ dependency "iwi_bss.fw" \ compile-with "${NORMAL_FWO}" \ no-implicit-rule \ clean "iwi_bss.fwo" iwi_bss.fw optional iwibssfw | iwifw \ dependency "$S/contrib/dev/iwi/ipw2200-bss.fw.uu" \ compile-with "${NORMAL_FW}" \ no-obj no-implicit-rule \ clean "iwi_bss.fw" iwiibssfw.c optional iwiibssfw | iwifw \ compile-with "${AWK} -f $S/tools/fw_stub.awk iwi_ibss.fw:iwi_ibss:300 -lintel_iwi -miwi_ibss -c${.TARGET}" \ no-implicit-rule before-depend local \ clean "iwiibssfw.c" iwi_ibss.fwo optional iwiibssfw | iwifw \ dependency "iwi_ibss.fw" \ compile-with "${NORMAL_FWO}" \ no-implicit-rule \ clean "iwi_ibss.fwo" iwi_ibss.fw optional iwiibssfw | iwifw \ dependency "$S/contrib/dev/iwi/ipw2200-ibss.fw.uu" \ compile-with "${NORMAL_FW}" \ no-obj no-implicit-rule \ clean "iwi_ibss.fw" iwimonitorfw.c optional iwimonitorfw | iwifw \ compile-with "${AWK} -f $S/tools/fw_stub.awk iwi_monitor.fw:iwi_monitor:300 -lintel_iwi -miwi_monitor -c${.TARGET}" \ no-implicit-rule before-depend local \ clean "iwimonitorfw.c" iwi_monitor.fwo optional iwimonitorfw | iwifw \ dependency "iwi_monitor.fw" \ compile-with "${NORMAL_FWO}" \ no-implicit-rule \ clean "iwi_monitor.fwo" iwi_monitor.fw optional iwimonitorfw | iwifw \ dependency "$S/contrib/dev/iwi/ipw2200-sniffer.fw.uu" \ compile-with "${NORMAL_FW}" \ no-obj no-implicit-rule \ clean "iwi_monitor.fw" dev/iwn/if_iwn.c optional iwn iwn1000fw.c optional iwn1000fw | iwnfw \ compile-with "${AWK} -f $S/tools/fw_stub.awk iwn1000.fw:iwn1000fw -miwn1000fw -c${.TARGET}" \ no-implicit-rule before-depend local \ clean "iwn1000fw.c" iwn1000fw.fwo optional iwn1000fw | iwnfw \ dependency "iwn1000.fw" \ compile-with "${NORMAL_FWO}" \ no-implicit-rule \ clean "iwn1000fw.fwo" iwn1000.fw optional iwn1000fw | iwnfw \ dependency "$S/contrib/dev/iwn/iwlwifi-1000-39.31.5.1.fw.uu" \ compile-with "${NORMAL_FW}" \ no-obj no-implicit-rule \ clean "iwn1000.fw" iwn100fw.c optional iwn100fw | iwnfw \ compile-with "${AWK} -f $S/tools/fw_stub.awk iwn100.fw:iwn100fw -miwn100fw -c${.TARGET}" \ no-implicit-rule before-depend local \ clean "iwn100fw.c" iwn100fw.fwo optional iwn100fw | iwnfw \ dependency "iwn100.fw" \ compile-with "${NORMAL_FWO}" \ no-implicit-rule \ clean "iwn100fw.fwo" iwn100.fw optional iwn100fw | iwnfw \ dependency "$S/contrib/dev/iwn/iwlwifi-100-39.31.5.1.fw.uu" \ compile-with "${NORMAL_FW}" \ no-obj no-implicit-rule \ clean "iwn100.fw" iwn105fw.c optional iwn105fw | iwnfw \ compile-with "${AWK} -f $S/tools/fw_stub.awk iwn105.fw:iwn105fw -miwn105fw -c${.TARGET}" \ no-implicit-rule before-depend local \ clean "iwn105fw.c" iwn105fw.fwo optional iwn105fw | iwnfw \ dependency "iwn105.fw" \ compile-with "${NORMAL_FWO}" \ no-implicit-rule \ clean "iwn105fw.fwo" iwn105.fw optional iwn105fw | iwnfw \ dependency "$S/contrib/dev/iwn/iwlwifi-105-6-18.168.6.1.fw.uu" \ compile-with "${NORMAL_FW}" \ no-obj no-implicit-rule \ clean "iwn105.fw" iwn135fw.c optional iwn135fw | iwnfw \ compile-with "${AWK} -f $S/tools/fw_stub.awk iwn135.fw:iwn135fw -miwn135fw -c${.TARGET}" \ no-implicit-rule before-depend local \ clean "iwn135fw.c" iwn135fw.fwo optional iwn135fw | iwnfw \ dependency "iwn135.fw" \ compile-with "${NORMAL_FWO}" \ no-implicit-rule \ clean "iwn135fw.fwo" iwn135.fw optional iwn135fw | iwnfw \ dependency "$S/contrib/dev/iwn/iwlwifi-135-6-18.168.6.1.fw.uu" \ compile-with "${NORMAL_FW}" \ no-obj no-implicit-rule \ clean "iwn135.fw" iwn2000fw.c optional iwn2000fw | iwnfw \ compile-with "${AWK} -f $S/tools/fw_stub.awk iwn2000.fw:iwn2000fw -miwn2000fw -c${.TARGET}" \ no-implicit-rule before-depend local \ clean "iwn2000fw.c" iwn2000fw.fwo optional iwn2000fw | iwnfw \ dependency "iwn2000.fw" \ compile-with "${NORMAL_FWO}" \ no-implicit-rule \ clean "iwn2000fw.fwo" iwn2000.fw optional iwn2000fw | iwnfw \ dependency "$S/contrib/dev/iwn/iwlwifi-2000-18.168.6.1.fw.uu" \ compile-with "${NORMAL_FW}" \ no-obj no-implicit-rule \ clean "iwn2000.fw" iwn2030fw.c optional iwn2030fw | iwnfw \ compile-with "${AWK} -f $S/tools/fw_stub.awk iwn2030.fw:iwn2030fw -miwn2030fw -c${.TARGET}" \ no-implicit-rule before-depend local \ clean "iwn2030fw.c" iwn2030fw.fwo optional iwn2030fw | iwnfw \ dependency "iwn2030.fw" \ compile-with "${NORMAL_FWO}" \ no-implicit-rule \ clean "iwn2030fw.fwo" iwn2030.fw optional iwn2030fw | iwnfw \ dependency "$S/contrib/dev/iwn/iwnwifi-2030-18.168.6.1.fw.uu" \ compile-with "${NORMAL_FW}" \ no-obj no-implicit-rule \ clean "iwn2030.fw" iwn4965fw.c optional iwn4965fw | iwnfw \ compile-with "${AWK} -f $S/tools/fw_stub.awk iwn4965.fw:iwn4965fw -miwn4965fw -c${.TARGET}" \ no-implicit-rule before-depend local \ clean "iwn4965fw.c" iwn4965fw.fwo optional iwn4965fw | iwnfw \ dependency "iwn4965.fw" \ compile-with "${NORMAL_FWO}" \ no-implicit-rule \ clean "iwn4965fw.fwo" iwn4965.fw optional iwn4965fw | iwnfw \ dependency "$S/contrib/dev/iwn/iwlwifi-4965-228.61.2.24.fw.uu" \ compile-with "${NORMAL_FW}" \ no-obj no-implicit-rule \ clean "iwn4965.fw" iwn5000fw.c optional iwn5000fw | iwnfw \ compile-with "${AWK} -f $S/tools/fw_stub.awk iwn5000.fw:iwn5000fw -miwn5000fw -c${.TARGET}" \ no-implicit-rule before-depend local \ clean "iwn5000fw.c" iwn5000fw.fwo optional iwn5000fw | iwnfw \ dependency "iwn5000.fw" \ compile-with "${NORMAL_FWO}" \ no-implicit-rule \ clean "iwn5000fw.fwo" iwn5000.fw optional iwn5000fw | iwnfw \ dependency "$S/contrib/dev/iwn/iwlwifi-5000-8.83.5.1.fw.uu" \ compile-with "${NORMAL_FW}" \ no-obj no-implicit-rule \ clean "iwn5000.fw" iwn5150fw.c optional iwn5150fw | iwnfw \ compile-with "${AWK} -f $S/tools/fw_stub.awk iwn5150.fw:iwn5150fw -miwn5150fw -c${.TARGET}" \ no-implicit-rule before-depend local \ clean "iwn5150fw.c" iwn5150fw.fwo optional iwn5150fw | iwnfw \ dependency "iwn5150.fw" \ compile-with "${NORMAL_FWO}" \ no-implicit-rule \ clean "iwn5150fw.fwo" iwn5150.fw optional iwn5150fw | iwnfw \ dependency "$S/contrib/dev/iwn/iwlwifi-5150-8.24.2.2.fw.uu"\ compile-with "${NORMAL_FW}" \ no-obj no-implicit-rule \ clean "iwn5150.fw" iwn6000fw.c optional iwn6000fw | iwnfw \ compile-with "${AWK} -f $S/tools/fw_stub.awk iwn6000.fw:iwn6000fw -miwn6000fw -c${.TARGET}" \ no-implicit-rule before-depend local \ clean "iwn6000fw.c" iwn6000fw.fwo optional iwn6000fw | iwnfw \ dependency "iwn6000.fw" \ compile-with "${NORMAL_FWO}" \ no-implicit-rule \ clean "iwn6000fw.fwo" iwn6000.fw optional iwn6000fw | iwnfw \ dependency "$S/contrib/dev/iwn/iwlwifi-6000-9.221.4.1.fw.uu" \ compile-with "${NORMAL_FW}" \ no-obj no-implicit-rule \ clean "iwn6000.fw" iwn6000g2afw.c optional iwn6000g2afw | iwnfw \ compile-with "${AWK} -f $S/tools/fw_stub.awk iwn6000g2a.fw:iwn6000g2afw -miwn6000g2afw -c${.TARGET}" \ no-implicit-rule before-depend local \ clean "iwn6000g2afw.c" iwn6000g2afw.fwo optional iwn6000g2afw | iwnfw \ dependency "iwn6000g2a.fw" \ compile-with "${NORMAL_FWO}" \ no-implicit-rule \ clean "iwn6000g2afw.fwo" iwn6000g2a.fw optional iwn6000g2afw | iwnfw \ dependency "$S/contrib/dev/iwn/iwlwifi-6000g2a-17.168.5.2.fw.uu" \ compile-with "${NORMAL_FW}" \ no-obj no-implicit-rule \ clean "iwn6000g2a.fw" iwn6000g2bfw.c optional iwn6000g2bfw | iwnfw \ compile-with "${AWK} -f $S/tools/fw_stub.awk iwn6000g2b.fw:iwn6000g2bfw -miwn6000g2bfw -c${.TARGET}" \ no-implicit-rule before-depend local \ clean "iwn6000g2bfw.c" iwn6000g2bfw.fwo optional iwn6000g2bfw | iwnfw \ dependency "iwn6000g2b.fw" \ compile-with "${NORMAL_FWO}" \ no-implicit-rule \ clean "iwn6000g2bfw.fwo" iwn6000g2b.fw optional iwn6000g2bfw | iwnfw \ dependency "$S/contrib/dev/iwn/iwlwifi-6000g2b-17.168.5.2.fw.uu" \ compile-with "${NORMAL_FW}" \ no-obj no-implicit-rule \ clean "iwn6000g2b.fw" iwn6050fw.c optional iwn6050fw | iwnfw \ compile-with "${AWK} -f $S/tools/fw_stub.awk iwn6050.fw:iwn6050fw -miwn6050fw -c${.TARGET}" \ no-implicit-rule before-depend local \ clean "iwn6050fw.c" iwn6050fw.fwo optional iwn6050fw | iwnfw \ dependency "iwn6050.fw" \ compile-with "${NORMAL_FWO}" \ no-implicit-rule \ clean "iwn6050fw.fwo" iwn6050.fw optional iwn6050fw | iwnfw \ dependency "$S/contrib/dev/iwn/iwlwifi-6050-41.28.5.1.fw.uu" \ compile-with "${NORMAL_FW}" \ no-obj no-implicit-rule \ clean "iwn6050.fw" dev/ixgb/if_ixgb.c optional ixgb dev/ixgb/ixgb_ee.c optional ixgb dev/ixgb/ixgb_hw.c optional ixgb dev/ixgbe/ixgbe.c optional ixgbe inet \ compile-with "${NORMAL_C} -I$S/dev/ixgbe -DSMP" dev/ixgbe/ixv.c optional ixgbe inet \ compile-with "${NORMAL_C} -I$S/dev/ixgbe" dev/ixgbe/ixgbe_phy.c optional ixgbe inet \ compile-with "${NORMAL_C} -I$S/dev/ixgbe" dev/ixgbe/ixgbe_api.c optional ixgbe inet \ compile-with "${NORMAL_C} -I$S/dev/ixgbe" dev/ixgbe/ixgbe_common.c optional ixgbe inet \ compile-with "${NORMAL_C} -I$S/dev/ixgbe" dev/ixgbe/ixgbe_mbx.c optional ixgbe inet \ compile-with "${NORMAL_C} -I$S/dev/ixgbe" dev/ixgbe/ixgbe_vf.c optional ixgbe inet \ compile-with "${NORMAL_C} -I$S/dev/ixgbe" dev/ixgbe/ixgbe_82598.c optional ixgbe inet \ compile-with "${NORMAL_C} -I$S/dev/ixgbe" dev/ixgbe/ixgbe_82599.c optional ixgbe inet \ compile-with "${NORMAL_C} -I$S/dev/ixgbe" dev/ixgbe/ixgbe_x540.c optional ixgbe inet \ compile-with "${NORMAL_C} -I$S/dev/ixgbe" dev/ixgbe/ixgbe_dcb.c optional ixgbe inet \ compile-with "${NORMAL_C} -I$S/dev/ixgbe" dev/ixgbe/ixgbe_dcb_82598.c optional ixgbe inet \ compile-with "${NORMAL_C} -I$S/dev/ixgbe" dev/ixgbe/ixgbe_dcb_82599.c optional ixgbe inet \ compile-with "${NORMAL_C} -I$S/dev/ixgbe" dev/jme/if_jme.c optional jme pci dev/joy/joy.c optional joy dev/joy/joy_isa.c optional joy isa dev/kbdmux/kbdmux.c optional kbdmux dev/ksyms/ksyms.c optional ksyms dev/le/am7990.c optional le dev/le/am79900.c optional le dev/le/if_le_pci.c optional le pci dev/le/lance.c optional le dev/led/led.c standard dev/lge/if_lge.c optional lge dev/lmc/if_lmc.c optional lmc dev/malo/if_malo.c optional malo dev/malo/if_malohal.c optional malo dev/malo/if_malo_pci.c optional malo pci dev/mc146818/mc146818.c optional mc146818 dev/mca/mca_bus.c optional mca dev/mcd/mcd.c optional mcd isa nowerror dev/mcd/mcd_isa.c optional mcd isa nowerror dev/md/md.c optional md dev/mem/memdev.c optional mem dev/mem/memutil.c optional mem dev/mfi/mfi.c optional mfi dev/mfi/mfi_debug.c optional mfi dev/mfi/mfi_pci.c optional mfi pci dev/mfi/mfi_disk.c optional mfi dev/mfi/mfi_syspd.c optional mfi dev/mfi/mfi_tbolt.c optional mfi dev/mfi/mfi_linux.c optional mfi compat_linux dev/mfi/mfi_cam.c optional mfip scbus dev/mii/acphy.c optional miibus | acphy dev/mii/amphy.c optional miibus | amphy dev/mii/atphy.c optional miibus | atphy dev/mii/axphy.c optional miibus | axphy dev/mii/bmtphy.c optional miibus | bmtphy dev/mii/brgphy.c optional miibus | brgphy dev/mii/ciphy.c optional miibus | ciphy dev/mii/e1000phy.c optional miibus | e1000phy dev/mii/gentbi.c optional miibus | gentbi dev/mii/icsphy.c optional miibus | icsphy dev/mii/ip1000phy.c optional miibus | ip1000phy dev/mii/jmphy.c optional miibus | jmphy dev/mii/lxtphy.c optional miibus | lxtphy dev/mii/mii.c optional miibus | mii dev/mii/mii_bitbang.c optional miibus | mii_bitbang dev/mii/mii_physubr.c optional miibus | mii dev/mii/miibus_if.m optional miibus | mii dev/mii/mlphy.c optional miibus | mlphy dev/mii/nsgphy.c optional miibus | nsgphy dev/mii/nsphy.c optional miibus | nsphy dev/mii/nsphyter.c optional miibus | nsphyter dev/mii/pnaphy.c optional miibus | pnaphy dev/mii/qsphy.c optional miibus | qsphy dev/mii/rdcphy.c optional miibus | rdcphy dev/mii/rgephy.c optional miibus | rgephy dev/mii/rlphy.c optional miibus | rlphy dev/mii/rlswitch.c optional rlswitch dev/mii/smcphy.c optional miibus | smcphy dev/mii/smscphy.c optional miibus | smscphy dev/mii/tdkphy.c optional miibus | tdkphy dev/mii/tlphy.c optional miibus | tlphy dev/mii/truephy.c optional miibus | truephy dev/mii/ukphy.c optional miibus | mii dev/mii/ukphy_subr.c optional miibus | mii dev/mii/xmphy.c optional miibus | xmphy dev/mk48txx/mk48txx.c optional mk48txx dev/mlx/mlx.c optional mlx dev/mlx/mlx_disk.c optional mlx dev/mlx/mlx_pci.c optional mlx pci dev/mly/mly.c optional mly dev/mmc/mmc.c optional mmc dev/mmc/mmcbr_if.m standard dev/mmc/mmcbus_if.m standard dev/mmc/mmcsd.c optional mmcsd dev/mn/if_mn.c optional mn pci dev/mpr/mpr.c optional mpr dev/mpr/mpr_config.c optional mpr # XXX Work around clang warning, until maintainer approves fix. dev/mpr/mpr_mapping.c optional mpr \ compile-with "${NORMAL_C} ${NO_WSOMETIMES_UNINITIALIZED}" dev/mpr/mpr_pci.c optional mpr pci dev/mpr/mpr_sas.c optional mpr \ compile-with "${NORMAL_C} ${NO_WUNNEEDED_INTERNAL_DECL}" dev/mpr/mpr_sas_lsi.c optional mpr dev/mpr/mpr_table.c optional mpr dev/mpr/mpr_user.c optional mpr dev/mps/mps.c optional mps dev/mps/mps_config.c optional mps # XXX Work around clang warning, until maintainer approves fix. dev/mps/mps_mapping.c optional mps \ compile-with "${NORMAL_C} ${NO_WSOMETIMES_UNINITIALIZED}" dev/mps/mps_pci.c optional mps pci dev/mps/mps_sas.c optional mps \ compile-with "${NORMAL_C} ${NO_WUNNEEDED_INTERNAL_DECL}" dev/mps/mps_sas_lsi.c optional mps dev/mps/mps_table.c optional mps dev/mps/mps_user.c optional mps dev/mpt/mpt.c optional mpt dev/mpt/mpt_cam.c optional mpt dev/mpt/mpt_debug.c optional mpt dev/mpt/mpt_pci.c optional mpt pci dev/mpt/mpt_raid.c optional mpt dev/mpt/mpt_user.c optional mpt dev/mrsas/mrsas.c optional mrsas dev/mrsas/mrsas_cam.c optional mrsas dev/mrsas/mrsas_ioctl.c optional mrsas dev/mrsas/mrsas_fp.c optional mrsas dev/msk/if_msk.c optional msk dev/mvs/mvs.c optional mvs dev/mvs/mvs_if.m optional mvs dev/mvs/mvs_pci.c optional mvs pci dev/mwl/if_mwl.c optional mwl dev/mwl/if_mwl_pci.c optional mwl pci dev/mwl/mwlhal.c optional mwl mwlfw.c optional mwlfw \ compile-with "${AWK} -f $S/tools/fw_stub.awk mw88W8363.fw:mw88W8363fw mwlboot.fw:mwlboot -mmwl -c${.TARGET}" \ no-implicit-rule before-depend local \ clean "mwlfw.c" mw88W8363.fwo optional mwlfw \ dependency "mw88W8363.fw" \ compile-with "${NORMAL_FWO}" \ no-implicit-rule \ clean "mw88W8363.fwo" mw88W8363.fw optional mwlfw \ dependency "$S/contrib/dev/mwl/mw88W8363.fw.uu" \ compile-with "${NORMAL_FW}" \ no-obj no-implicit-rule \ clean "mw88W8363.fw" mwlboot.fwo optional mwlfw \ dependency "mwlboot.fw" \ compile-with "${NORMAL_FWO}" \ no-implicit-rule \ clean "mwlboot.fwo" mwlboot.fw optional mwlfw \ dependency "$S/contrib/dev/mwl/mwlboot.fw.uu" \ compile-with "${NORMAL_FW}" \ no-obj no-implicit-rule \ clean "mwlboot.fw" dev/mxge/if_mxge.c optional mxge pci dev/mxge/mxge_eth_z8e.c optional mxge pci dev/mxge/mxge_ethp_z8e.c optional mxge pci dev/mxge/mxge_rss_eth_z8e.c optional mxge pci dev/mxge/mxge_rss_ethp_z8e.c optional mxge pci dev/my/if_my.c optional my dev/nand/nand.c optional nand dev/nand/nand_bbt.c optional nand dev/nand/nand_cdev.c optional nand dev/nand/nand_generic.c optional nand dev/nand/nand_geom.c optional nand dev/nand/nand_id.c optional nand dev/nand/nandbus.c optional nand dev/nand/nandbus_if.m optional nand dev/nand/nand_if.m optional nand dev/nand/nandsim.c optional nandsim nand dev/nand/nandsim_chip.c optional nandsim nand dev/nand/nandsim_ctrl.c optional nandsim nand dev/nand/nandsim_log.c optional nandsim nand dev/nand/nandsim_swap.c optional nandsim nand dev/nand/nfc_if.m optional nand dev/ncr/ncr.c optional ncr pci dev/ncv/ncr53c500.c optional ncv dev/ncv/ncr53c500_pccard.c optional ncv pccard dev/netmap/netmap.c optional netmap dev/netmap/netmap_freebsd.c optional netmap dev/netmap/netmap_generic.c optional netmap dev/netmap/netmap_mbq.c optional netmap dev/netmap/netmap_mem2.c optional netmap dev/netmap/netmap_monitor.c optional netmap dev/netmap/netmap_offloadings.c optional netmap dev/netmap/netmap_pipe.c optional netmap dev/netmap/netmap_vale.c optional netmap # compile-with "${NORMAL_C} -Wconversion -Wextra" dev/nfsmb/nfsmb.c optional nfsmb pci dev/nge/if_nge.c optional nge dev/nxge/if_nxge.c optional nxge \ compile-with "${NORMAL_C} ${NO_WSELF_ASSIGN}" dev/nxge/xgehal/xgehal-device.c optional nxge \ compile-with "${NORMAL_C} ${NO_WSELF_ASSIGN}" dev/nxge/xgehal/xgehal-mm.c optional nxge dev/nxge/xgehal/xge-queue.c optional nxge dev/nxge/xgehal/xgehal-driver.c optional nxge \ compile-with "${NORMAL_C} ${NO_WSELF_ASSIGN}" dev/nxge/xgehal/xgehal-ring.c optional nxge \ compile-with "${NORMAL_C} ${NO_WSELF_ASSIGN}" dev/nxge/xgehal/xgehal-channel.c optional nxge \ compile-with "${NORMAL_C} ${NO_WSELF_ASSIGN}" dev/nxge/xgehal/xgehal-fifo.c optional nxge \ compile-with "${NORMAL_C} ${NO_WSELF_ASSIGN}" dev/nxge/xgehal/xgehal-stats.c optional nxge \ compile-with "${NORMAL_C} ${NO_WSELF_ASSIGN}" dev/nxge/xgehal/xgehal-config.c optional nxge dev/nxge/xgehal/xgehal-mgmt.c optional nxge \ compile-with "${NORMAL_C} ${NO_WSELF_ASSIGN}" dev/nmdm/nmdm.c optional nmdm dev/nsp/nsp.c optional nsp dev/nsp/nsp_pccard.c optional nsp pccard dev/null/null.c standard dev/oce/oce_hw.c optional oce pci dev/oce/oce_if.c optional oce pci dev/oce/oce_mbox.c optional oce pci dev/oce/oce_queue.c optional oce pci dev/oce/oce_sysctl.c optional oce pci dev/oce/oce_util.c optional oce pci dev/ofw/ofw_bus_if.m optional fdt dev/ofw/ofw_bus_subr.c optional fdt dev/ofw/ofw_fdt.c optional fdt dev/ofw/ofw_if.m optional fdt dev/ofw/ofw_iicbus.c optional fdt iicbus dev/ofw/ofwbus.c optional fdt dev/ofw/openfirm.c optional fdt dev/ofw/openfirmio.c optional fdt dev/patm/if_patm.c optional patm pci dev/patm/if_patm_attach.c optional patm pci dev/patm/if_patm_intr.c optional patm pci dev/patm/if_patm_ioctl.c optional patm pci dev/patm/if_patm_rtables.c optional patm pci dev/patm/if_patm_rx.c optional patm pci dev/patm/if_patm_tx.c optional patm pci dev/pbio/pbio.c optional pbio isa dev/pccard/card_if.m standard dev/pccard/pccard.c optional pccard dev/pccard/pccard_cis.c optional pccard dev/pccard/pccard_cis_quirks.c optional pccard dev/pccard/pccard_device.c optional pccard dev/pccard/power_if.m standard dev/pccbb/pccbb.c optional cbb dev/pccbb/pccbb_isa.c optional cbb isa dev/pccbb/pccbb_pci.c optional cbb pci dev/pcf/pcf.c optional pcf dev/pci/eisa_pci.c optional pci eisa dev/pci/fixup_pci.c optional pci dev/pci/hostb_pci.c optional pci dev/pci/ignore_pci.c optional pci dev/pci/isa_pci.c optional pci isa dev/pci/pci.c optional pci dev/pci/pci_if.m standard dev/pci/pci_pci.c optional pci dev/pci/pci_subr.c optional pci dev/pci/pci_user.c optional pci dev/pci/pcib_if.m standard dev/pci/pcib_support.c standard dev/pci/vga_pci.c optional pci dev/pcn/if_pcn.c optional pcn pci dev/pdq/if_fea.c optional fea eisa dev/pdq/if_fpa.c optional fpa pci dev/pdq/pdq.c optional nowerror fea eisa | fpa pci dev/pdq/pdq_ifsubr.c optional nowerror fea eisa | fpa pci dev/ppbus/if_plip.c optional plip dev/ppbus/immio.c optional vpo dev/ppbus/lpbb.c optional lpbb dev/ppbus/lpt.c optional lpt dev/ppbus/pcfclock.c optional pcfclock dev/ppbus/ppb_1284.c optional ppbus dev/ppbus/ppb_base.c optional ppbus dev/ppbus/ppb_msq.c optional ppbus dev/ppbus/ppbconf.c optional ppbus dev/ppbus/ppbus_if.m optional ppbus dev/ppbus/ppi.c optional ppi dev/ppbus/pps.c optional pps dev/ppbus/vpo.c optional vpo dev/ppbus/vpoio.c optional vpo dev/ppc/ppc.c optional ppc dev/ppc/ppc_acpi.c optional ppc acpi dev/ppc/ppc_isa.c optional ppc isa dev/ppc/ppc_pci.c optional ppc pci dev/ppc/ppc_puc.c optional ppc puc dev/pst/pst-iop.c optional pst dev/pst/pst-pci.c optional pst pci dev/pst/pst-raid.c optional pst dev/pty/pty.c optional pty dev/puc/puc.c optional puc dev/puc/puc_cfg.c optional puc dev/puc/puc_pccard.c optional puc pccard dev/puc/puc_pci.c optional puc pci dev/puc/pucdata.c optional puc pci dev/quicc/quicc_core.c optional quicc dev/ral/rt2560.c optional ral dev/ral/rt2661.c optional ral dev/ral/rt2860.c optional ral dev/ral/if_ral_pci.c optional ral pci rt2561fw.c optional rt2561fw | ralfw \ compile-with "${AWK} -f $S/tools/fw_stub.awk rt2561.fw:rt2561fw -mrt2561 -c${.TARGET}" \ no-implicit-rule before-depend local \ clean "rt2561fw.c" rt2561fw.fwo optional rt2561fw | ralfw \ dependency "rt2561.fw" \ compile-with "${NORMAL_FWO}" \ no-implicit-rule \ clean "rt2561fw.fwo" rt2561.fw optional rt2561fw | ralfw \ dependency "$S/contrib/dev/ral/rt2561.fw.uu" \ compile-with "${NORMAL_FW}" \ no-obj no-implicit-rule \ clean "rt2561.fw" rt2561sfw.c optional rt2561sfw | ralfw \ compile-with "${AWK} -f $S/tools/fw_stub.awk rt2561s.fw:rt2561sfw -mrt2561s -c${.TARGET}" \ no-implicit-rule before-depend local \ clean "rt2561sfw.c" rt2561sfw.fwo optional rt2561sfw | ralfw \ dependency "rt2561s.fw" \ compile-with "${NORMAL_FWO}" \ no-implicit-rule \ clean "rt2561sfw.fwo" rt2561s.fw optional rt2561sfw | ralfw \ dependency "$S/contrib/dev/ral/rt2561s.fw.uu" \ compile-with "${NORMAL_FW}" \ no-obj no-implicit-rule \ clean "rt2561s.fw" rt2661fw.c optional rt2661fw | ralfw \ compile-with "${AWK} -f $S/tools/fw_stub.awk rt2661.fw:rt2661fw -mrt2661 -c${.TARGET}" \ no-implicit-rule before-depend local \ clean "rt2661fw.c" rt2661fw.fwo optional rt2661fw | ralfw \ dependency "rt2661.fw" \ compile-with "${NORMAL_FWO}" \ no-implicit-rule \ clean "rt2661fw.fwo" rt2661.fw optional rt2661fw | ralfw \ dependency "$S/contrib/dev/ral/rt2661.fw.uu" \ compile-with "${NORMAL_FW}" \ no-obj no-implicit-rule \ clean "rt2661.fw" rt2860fw.c optional rt2860fw | ralfw \ compile-with "${AWK} -f $S/tools/fw_stub.awk rt2860.fw:rt2860fw -mrt2860 -c${.TARGET}" \ no-implicit-rule before-depend local \ clean "rt2860fw.c" rt2860fw.fwo optional rt2860fw | ralfw \ dependency "rt2860.fw" \ compile-with "${NORMAL_FWO}" \ no-implicit-rule \ clean "rt2860fw.fwo" rt2860.fw optional rt2860fw | ralfw \ dependency "$S/contrib/dev/ral/rt2860.fw.uu" \ compile-with "${NORMAL_FW}" \ no-obj no-implicit-rule \ clean "rt2860.fw" dev/random/randomdev.c standard dev/random/random_adaptors.c standard dev/random/dummy_rng.c standard dev/random/live_entropy_sources.c standard dev/random/random_harvestq.c standard dev/random/randomdev_soft.c optional random dev/random/yarrow.c optional random dev/random/fortuna.c optional random dev/random/hash.c optional random dev/rc/rc.c optional rc dev/re/if_re.c optional re dev/rl/if_rl.c optional rl pci dev/rndtest/rndtest.c optional rndtest dev/rp/rp.c optional rp dev/rp/rp_isa.c optional rp isa dev/rp/rp_pci.c optional rp pci dev/safe/safe.c optional safe dev/scc/scc_if.m optional scc dev/scc/scc_bfe_ebus.c optional scc ebus dev/scc/scc_bfe_quicc.c optional scc quicc dev/scc/scc_bfe_sbus.c optional scc fhc | scc sbus dev/scc/scc_core.c optional scc dev/scc/scc_dev_quicc.c optional scc quicc dev/scc/scc_dev_sab82532.c optional scc dev/scc/scc_dev_z8530.c optional scc dev/scd/scd.c optional scd isa dev/scd/scd_isa.c optional scd isa dev/sdhci/sdhci.c optional sdhci dev/sdhci/sdhci_if.m optional sdhci dev/sdhci/sdhci_pci.c optional sdhci pci dev/sf/if_sf.c optional sf pci dev/sge/if_sge.c optional sge pci dev/si/si.c optional si dev/si/si2_z280.c optional si dev/si/si3_t225.c optional si dev/si/si_eisa.c optional si eisa dev/si/si_isa.c optional si isa dev/si/si_pci.c optional si pci dev/siba/siba.c optional siba dev/siba/siba_bwn.c optional siba_bwn pci dev/siba/siba_cc.c optional siba dev/siba/siba_core.c optional siba | siba_bwn pci dev/siba/siba_pcib.c optional siba pci dev/siis/siis.c optional siis pci dev/sis/if_sis.c optional sis pci dev/sk/if_sk.c optional sk pci dev/smbus/smb.c optional smb dev/smbus/smbconf.c optional smbus dev/smbus/smbus.c optional smbus dev/smbus/smbus_if.m optional smbus dev/smc/if_smc.c optional smc dev/smc/if_smc_fdt.c optional smc fdt dev/sn/if_sn.c optional sn dev/sn/if_sn_isa.c optional sn isa dev/sn/if_sn_pccard.c optional sn pccard dev/snp/snp.c optional snp dev/sound/clone.c optional sound dev/sound/unit.c optional sound dev/sound/isa/ad1816.c optional snd_ad1816 isa dev/sound/isa/ess.c optional snd_ess isa dev/sound/isa/gusc.c optional snd_gusc isa dev/sound/isa/mss.c optional snd_mss isa dev/sound/isa/sb16.c optional snd_sb16 isa dev/sound/isa/sb8.c optional snd_sb8 isa dev/sound/isa/sbc.c optional snd_sbc isa dev/sound/isa/sndbuf_dma.c optional sound isa dev/sound/pci/als4000.c optional snd_als4000 pci dev/sound/pci/atiixp.c optional snd_atiixp pci dev/sound/pci/cmi.c optional snd_cmi pci dev/sound/pci/cs4281.c optional snd_cs4281 pci dev/sound/pci/csa.c optional snd_csa pci dev/sound/pci/csapcm.c optional snd_csa pci dev/sound/pci/ds1.c optional snd_ds1 pci dev/sound/pci/emu10k1.c optional snd_emu10k1 pci dev/sound/pci/emu10kx.c optional snd_emu10kx pci dev/sound/pci/emu10kx-pcm.c optional snd_emu10kx pci dev/sound/pci/emu10kx-midi.c optional snd_emu10kx pci dev/sound/pci/envy24.c optional snd_envy24 pci dev/sound/pci/envy24ht.c optional snd_envy24ht pci dev/sound/pci/es137x.c optional snd_es137x pci dev/sound/pci/fm801.c optional snd_fm801 pci dev/sound/pci/ich.c optional snd_ich pci dev/sound/pci/maestro.c optional snd_maestro pci dev/sound/pci/maestro3.c optional snd_maestro3 pci dev/sound/pci/neomagic.c optional snd_neomagic pci dev/sound/pci/solo.c optional snd_solo pci dev/sound/pci/spicds.c optional snd_spicds pci dev/sound/pci/t4dwave.c optional snd_t4dwave pci dev/sound/pci/via8233.c optional snd_via8233 pci dev/sound/pci/via82c686.c optional snd_via82c686 pci dev/sound/pci/vibes.c optional snd_vibes pci dev/sound/pci/hda/hdaa.c optional snd_hda pci dev/sound/pci/hda/hdaa_patches.c optional snd_hda pci dev/sound/pci/hda/hdac.c optional snd_hda pci dev/sound/pci/hda/hdac_if.m optional snd_hda pci dev/sound/pci/hda/hdacc.c optional snd_hda pci dev/sound/pci/hdspe.c optional snd_hdspe pci dev/sound/pci/hdspe-pcm.c optional snd_hdspe pci dev/sound/pcm/ac97.c optional sound dev/sound/pcm/ac97_if.m optional sound dev/sound/pcm/ac97_patch.c optional sound dev/sound/pcm/buffer.c optional sound \ dependency "snd_fxdiv_gen.h" dev/sound/pcm/channel.c optional sound dev/sound/pcm/channel_if.m optional sound dev/sound/pcm/dsp.c optional sound dev/sound/pcm/feeder.c optional sound dev/sound/pcm/feeder_chain.c optional sound dev/sound/pcm/feeder_eq.c optional sound \ dependency "feeder_eq_gen.h" \ dependency "snd_fxdiv_gen.h" dev/sound/pcm/feeder_if.m optional sound dev/sound/pcm/feeder_format.c optional sound \ dependency "snd_fxdiv_gen.h" dev/sound/pcm/feeder_matrix.c optional sound \ dependency "snd_fxdiv_gen.h" dev/sound/pcm/feeder_mixer.c optional sound \ dependency "snd_fxdiv_gen.h" dev/sound/pcm/feeder_rate.c optional sound \ dependency "feeder_rate_gen.h" \ dependency "snd_fxdiv_gen.h" dev/sound/pcm/feeder_volume.c optional sound \ dependency "snd_fxdiv_gen.h" dev/sound/pcm/mixer.c optional sound dev/sound/pcm/mixer_if.m optional sound dev/sound/pcm/sndstat.c optional sound dev/sound/pcm/sound.c optional sound dev/sound/pcm/vchan.c optional sound dev/sound/usb/uaudio.c optional snd_uaudio usb dev/sound/usb/uaudio_pcm.c optional snd_uaudio usb dev/sound/midi/midi.c optional sound dev/sound/midi/mpu401.c optional sound dev/sound/midi/mpu_if.m optional sound dev/sound/midi/mpufoi_if.m optional sound dev/sound/midi/sequencer.c optional sound dev/sound/midi/synth_if.m optional sound dev/spibus/ofw_spibus.c optional fdt spibus dev/spibus/spibus.c optional spibus \ dependency "spibus_if.h" dev/spibus/spibus_if.m optional spibus dev/ste/if_ste.c optional ste pci dev/stg/tmc18c30.c optional stg dev/stg/tmc18c30_isa.c optional stg isa dev/stg/tmc18c30_pccard.c optional stg pccard dev/stg/tmc18c30_pci.c optional stg pci dev/stg/tmc18c30_subr.c optional stg dev/stge/if_stge.c optional stge dev/streams/streams.c optional streams dev/sym/sym_hipd.c optional sym \ dependency "$S/dev/sym/sym_{conf,defs}.h" dev/syscons/blank/blank_saver.c optional blank_saver dev/syscons/daemon/daemon_saver.c optional daemon_saver dev/syscons/dragon/dragon_saver.c optional dragon_saver dev/syscons/fade/fade_saver.c optional fade_saver dev/syscons/fire/fire_saver.c optional fire_saver dev/syscons/green/green_saver.c optional green_saver dev/syscons/logo/logo.c optional logo_saver dev/syscons/logo/logo_saver.c optional logo_saver dev/syscons/rain/rain_saver.c optional rain_saver dev/syscons/schistory.c optional sc dev/syscons/scmouse.c optional sc dev/syscons/scterm.c optional sc dev/syscons/scvidctl.c optional sc dev/syscons/snake/snake_saver.c optional snake_saver dev/syscons/star/star_saver.c optional star_saver dev/syscons/syscons.c optional sc dev/syscons/sysmouse.c optional sc dev/syscons/warp/warp_saver.c optional warp_saver dev/tdfx/tdfx_linux.c optional tdfx_linux tdfx compat_linux dev/tdfx/tdfx_pci.c optional tdfx pci dev/ti/if_ti.c optional ti pci dev/tl/if_tl.c optional tl pci dev/trm/trm.c optional trm dev/twa/tw_cl_init.c optional twa \ compile-with "${NORMAL_C} -I$S/dev/twa" dev/twa/tw_cl_intr.c optional twa \ compile-with "${NORMAL_C} -I$S/dev/twa" dev/twa/tw_cl_io.c optional twa \ compile-with "${NORMAL_C} -I$S/dev/twa" dev/twa/tw_cl_misc.c optional twa \ compile-with "${NORMAL_C} -I$S/dev/twa" dev/twa/tw_osl_cam.c optional twa \ compile-with "${NORMAL_C} -I$S/dev/twa" dev/twa/tw_osl_freebsd.c optional twa \ compile-with "${NORMAL_C} -I$S/dev/twa" dev/twe/twe.c optional twe dev/twe/twe_freebsd.c optional twe dev/tws/tws.c optional tws dev/tws/tws_cam.c optional tws dev/tws/tws_hdm.c optional tws dev/tws/tws_services.c optional tws dev/tws/tws_user.c optional tws dev/tx/if_tx.c optional tx dev/txp/if_txp.c optional txp dev/uart/uart_bus_acpi.c optional uart acpi #dev/uart/uart_bus_cbus.c optional uart cbus dev/uart/uart_bus_ebus.c optional uart ebus dev/uart/uart_bus_fdt.c optional uart fdt dev/uart/uart_bus_isa.c optional uart isa dev/uart/uart_bus_pccard.c optional uart pccard dev/uart/uart_bus_pci.c optional uart pci dev/uart/uart_bus_puc.c optional uart puc dev/uart/uart_bus_scc.c optional uart scc dev/uart/uart_core.c optional uart dev/uart/uart_dbg.c optional uart gdb dev/uart/uart_dev_ns8250.c optional uart uart_ns8250 dev/uart/uart_dev_pl011.c optional uart pl011 dev/uart/uart_dev_quicc.c optional uart quicc dev/uart/uart_dev_sab82532.c optional uart uart_sab82532 dev/uart/uart_dev_sab82532.c optional uart scc dev/uart/uart_dev_z8530.c optional uart uart_z8530 dev/uart/uart_dev_z8530.c optional uart scc dev/uart/uart_if.m optional uart dev/uart/uart_subr.c optional uart dev/uart/uart_tty.c optional uart dev/ubsec/ubsec.c optional ubsec # # USB controller drivers # dev/usb/controller/at91dci.c optional at91dci dev/usb/controller/at91dci_atmelarm.c optional at91dci at91rm9200 dev/usb/controller/musb_otg.c optional musb dev/usb/controller/musb_otg_atmelarm.c optional musb at91rm9200 dev/usb/controller/dwc_otg.c optional dwcotg dev/usb/controller/dwc_otg_fdt.c optional dwcotg fdt dev/usb/controller/ehci.c optional ehci dev/usb/controller/ehci_pci.c optional ehci pci dev/usb/controller/ohci.c optional ohci dev/usb/controller/ohci_atmelarm.c optional ohci at91rm9200 dev/usb/controller/ohci_pci.c optional ohci pci dev/usb/controller/uhci.c optional uhci dev/usb/controller/uhci_pci.c optional uhci pci dev/usb/controller/xhci.c optional xhci dev/usb/controller/xhci_pci.c optional xhci pci dev/usb/controller/saf1761_otg.c optional saf1761otg dev/usb/controller/saf1761_otg_fdt.c optional saf1761otg fdt dev/usb/controller/uss820dci.c optional uss820dci dev/usb/controller/uss820dci_atmelarm.c optional uss820dci at91rm9200 dev/usb/controller/usb_controller.c optional usb # # USB storage drivers # dev/usb/storage/umass.c optional umass dev/usb/storage/urio.c optional urio dev/usb/storage/ustorage_fs.c optional usfs # # USB core # dev/usb/usb_busdma.c optional usb dev/usb/usb_compat_linux.c optional usb dev/usb/usb_core.c optional usb dev/usb/usb_debug.c optional usb dev/usb/usb_dev.c optional usb dev/usb/usb_device.c optional usb dev/usb/usb_dynamic.c optional usb dev/usb/usb_error.c optional usb dev/usb/usb_generic.c optional usb dev/usb/usb_handle_request.c optional usb dev/usb/usb_hid.c optional usb dev/usb/usb_hub.c optional usb dev/usb/usb_if.m optional usb dev/usb/usb_lookup.c optional usb dev/usb/usb_mbuf.c optional usb dev/usb/usb_msctest.c optional usb dev/usb/usb_parse.c optional usb dev/usb/usb_pf.c optional usb dev/usb/usb_process.c optional usb dev/usb/usb_request.c optional usb dev/usb/usb_transfer.c optional usb dev/usb/usb_util.c optional usb # # USB network drivers # dev/usb/net/if_aue.c optional aue dev/usb/net/if_axe.c optional axe dev/usb/net/if_axge.c optional axge dev/usb/net/if_cdce.c optional cdce dev/usb/net/if_cue.c optional cue dev/usb/net/if_ipheth.c optional ipheth dev/usb/net/if_kue.c optional kue dev/usb/net/if_mos.c optional mos dev/usb/net/if_rue.c optional rue dev/usb/net/if_smsc.c optional smsc dev/usb/net/if_udav.c optional udav dev/usb/net/if_usie.c optional usie dev/usb/net/if_urndis.c optional urndis dev/usb/net/ruephy.c optional rue dev/usb/net/usb_ethernet.c optional aue | axe | axge | cdce | cue | kue | \ mos | rue | smsc | udav | ipheth | \ urndis dev/usb/net/uhso.c optional uhso # # USB WLAN drivers # dev/usb/wlan/if_rsu.c optional rsu rsu-rtl8712fw.c optional rsu-rtl8712fw | rsufw \ compile-with "${AWK} -f $S/tools/fw_stub.awk rsu-rtl8712fw.fw:rsu-rtl8712fw:120 -mrsu-rtl8712fw -c${.TARGET}" \ no-implicit-rule before-depend local \ clean "rsu-rtl8712fw.c" rsu-rtl8712fw.fwo optional rsu-rtl8712fw | rsufw \ dependency "rsu-rtl8712fw.fw" \ compile-with "${NORMAL_FWO}" \ no-implicit-rule \ clean "rsu-rtl8712fw.fwo" rsu-rtl8712fw.fw optional rsu-rtl8712.fw | rsufw \ dependency "$S/contrib/dev/rsu/rsu-rtl8712fw.fw.uu" \ compile-with "${NORMAL_FW}" \ no-obj no-implicit-rule \ clean "rsu-rtl8712fw.fw" dev/usb/wlan/if_rum.c optional rum dev/usb/wlan/if_run.c optional run runfw.c optional runfw \ compile-with "${AWK} -f $S/tools/fw_stub.awk run.fw:runfw -mrunfw -c${.TARGET}" \ no-implicit-rule before-depend local \ clean "runfw.c" runfw.fwo optional runfw \ dependency "run.fw" \ compile-with "${NORMAL_FWO}" \ no-implicit-rule \ clean "runfw.fwo" run.fw optional runfw \ dependency "$S/contrib/dev/run/rt2870.fw.uu" \ compile-with "${NORMAL_FW}" \ no-obj no-implicit-rule \ clean "run.fw" dev/usb/wlan/if_uath.c optional uath dev/usb/wlan/if_upgt.c optional upgt dev/usb/wlan/if_ural.c optional ural dev/usb/wlan/if_urtw.c optional urtw dev/usb/wlan/if_urtwn.c optional urtwn urtwn-rtl8188eufw.c optional urtwn-rtl8188eufw | urtwnfw \ compile-with "${AWK} -f $S/tools/fw_stub.awk urtwn-rtl8188eufw.fw:urtwn-rtl8188eufw:111 -murtwn-rtl8188eufw -c${.TARGET}" \ no-implicit-rule before-depend local \ clean "urtwn-rtl8188eufw.c" urtwn-rtl8188eufw.fwo optional urtwn-rtl8188eufw | urtwnfw \ dependency "urtwn-rtl8188eufw.fw" \ compile-with "${NORMAL_FWO}" \ no-implicit-rule \ clean "urtwn-rtl8188eufw.fwo" urtwn-rtl8188eufw.fw optional urtwn-rtl8188eufw | urtwnfw \ dependency "$S/contrib/dev/urtwn/urtwn-rtl8188eufw.fw.uu" \ compile-with "${NORMAL_FW}" \ no-obj no-implicit-rule \ clean "urtwn-rtl8188eufw.fw" urtwn-rtl8192cfwT.c optional urtwn-rtl8192cfwT | urtwnfw \ compile-with "${AWK} -f $S/tools/fw_stub.awk urtwn-rtl8192cfwT.fw:urtwn-rtl8192cfwT:111 -murtwn-rtl8192cfwT -c${.TARGET}" \ no-implicit-rule before-depend local \ clean "urtwn-rtl8192cfwT.c" urtwn-rtl8192cfwT.fwo optional urtwn-rtl8192cfwT | urtwnfw \ dependency "urtwn-rtl8192cfwT.fw" \ compile-with "${NORMAL_FWO}" \ no-implicit-rule \ clean "urtwn-rtl8192cfwT.fwo" urtwn-rtl8192cfwT.fw optional urtwn-rtl8192cfwT | urtwnfw \ dependency "$S/contrib/dev/urtwn/urtwn-rtl8192cfwT.fw.uu" \ compile-with "${NORMAL_FW}" \ no-obj no-implicit-rule \ clean "urtwn-rtl8192cfwT.fw" urtwn-rtl8192cfwU.c optional urtwn-rtl8192cfwU | urtwnfw \ compile-with "${AWK} -f $S/tools/fw_stub.awk urtwn-rtl8192cfwU.fw:urtwn-rtl8192cfwU:111 -murtwn-rtl8192cfwU -c${.TARGET}" \ no-implicit-rule before-depend local \ clean "urtwn-rtl8192cfwU.c" urtwn-rtl8192cfwU.fwo optional urtwn-rtl8192cfwU | urtwnfw \ dependency "urtwn-rtl8192cfwU.fw" \ compile-with "${NORMAL_FWO}" \ no-implicit-rule \ clean "urtwn-rtl8192cfwU.fwo" urtwn-rtl8192cfwU.fw optional urtwn-rtl8192cfwU | urtwnfw \ dependency "$S/contrib/dev/urtwn/urtwn-rtl8192cfwU.fw.uu" \ compile-with "${NORMAL_FW}" \ no-obj no-implicit-rule \ clean "urtwn-rtl8192cfwU.fw" dev/usb/wlan/if_zyd.c optional zyd # # USB serial and parallel port drivers # dev/usb/serial/u3g.c optional u3g dev/usb/serial/uark.c optional uark dev/usb/serial/ubsa.c optional ubsa dev/usb/serial/ubser.c optional ubser dev/usb/serial/uchcom.c optional uchcom dev/usb/serial/ucycom.c optional ucycom dev/usb/serial/ufoma.c optional ufoma dev/usb/serial/uftdi.c optional uftdi dev/usb/serial/ugensa.c optional ugensa dev/usb/serial/uipaq.c optional uipaq dev/usb/serial/ulpt.c optional ulpt dev/usb/serial/umcs.c optional umcs dev/usb/serial/umct.c optional umct dev/usb/serial/umodem.c optional umodem dev/usb/serial/umoscom.c optional umoscom dev/usb/serial/uplcom.c optional uplcom dev/usb/serial/uslcom.c optional uslcom dev/usb/serial/uvisor.c optional uvisor dev/usb/serial/uvscom.c optional uvscom dev/usb/serial/usb_serial.c optional ucom | u3g | uark | ubsa | ubser | \ uchcom | ucycom | ufoma | uftdi | \ ugensa | uipaq | umcs | umct | \ umodem | umoscom | uplcom | usie | \ uslcom | uvisor | uvscom # # USB misc drivers # dev/usb/misc/ufm.c optional ufm dev/usb/misc/udbp.c optional udbp dev/usb/misc/uled.c optional uled # # USB input drivers # dev/usb/input/atp.c optional atp dev/usb/input/uep.c optional uep dev/usb/input/uhid.c optional uhid dev/usb/input/ukbd.c optional ukbd dev/usb/input/ums.c optional ums dev/usb/input/wsp.c optional wsp # # USB quirks # dev/usb/quirk/usb_quirk.c optional usb # # USB templates # dev/usb/template/usb_template.c optional usb_template dev/usb/template/usb_template_audio.c optional usb_template dev/usb/template/usb_template_cdce.c optional usb_template dev/usb/template/usb_template_kbd.c optional usb_template dev/usb/template/usb_template_modem.c optional usb_template dev/usb/template/usb_template_mouse.c optional usb_template dev/usb/template/usb_template_msc.c optional usb_template dev/usb/template/usb_template_mtp.c optional usb_template dev/usb/template/usb_template_phone.c optional usb_template # # USB END # dev/utopia/idtphy.c optional utopia dev/utopia/suni.c optional utopia dev/utopia/utopia.c optional utopia dev/vge/if_vge.c optional vge dev/viapm/viapm.c optional viapm pci dev/virtio/virtio.c optional virtio dev/virtio/virtqueue.c optional virtio dev/virtio/virtio_bus_if.m optional virtio dev/virtio/virtio_if.m optional virtio dev/virtio/pci/virtio_pci.c optional virtio_pci dev/virtio/mmio/virtio_mmio.c optional virtio_mmio dev/virtio/mmio/virtio_mmio_if.m optional virtio_mmio dev/virtio/network/if_vtnet.c optional vtnet dev/virtio/block/virtio_blk.c optional virtio_blk dev/virtio/balloon/virtio_balloon.c optional virtio_balloon dev/virtio/scsi/virtio_scsi.c optional virtio_scsi dev/virtio/random/virtio_random.c optional virtio_random dev/virtio/console/virtio_console.c optional virtio_console dev/vkbd/vkbd.c optional vkbd dev/vr/if_vr.c optional vr pci dev/vt/colors/vt_termcolors.c optional vt dev/vt/font/vt_font_default.c optional vt dev/vt/font/vt_mouse_cursor.c optional vt dev/vt/hw/efifb/efifb.c optional vt_efifb dev/vt/hw/fb/vt_fb.c optional vt dev/vt/hw/vga/vt_vga.c optional vt vt_vga dev/vt/logo/logo_freebsd.c optional vt splash dev/vt/vt_buf.c optional vt dev/vt/vt_consolectl.c optional vt dev/vt/vt_core.c optional vt dev/vt/vt_font.c optional vt dev/vt/vt_sysmouse.c optional vt dev/vte/if_vte.c optional vte pci dev/vx/if_vx.c optional vx dev/vx/if_vx_eisa.c optional vx eisa dev/vx/if_vx_pci.c optional vx pci dev/vxge/vxge.c optional vxge dev/vxge/vxgehal/vxgehal-ifmsg.c optional vxge dev/vxge/vxgehal/vxgehal-mrpcim.c optional vxge dev/vxge/vxgehal/vxge-queue.c optional vxge dev/vxge/vxgehal/vxgehal-ring.c optional vxge dev/vxge/vxgehal/vxgehal-swapper.c optional vxge dev/vxge/vxgehal/vxgehal-mgmt.c optional vxge dev/vxge/vxgehal/vxgehal-srpcim.c optional vxge dev/vxge/vxgehal/vxgehal-config.c optional vxge dev/vxge/vxgehal/vxgehal-blockpool.c optional vxge dev/vxge/vxgehal/vxgehal-doorbells.c optional vxge dev/vxge/vxgehal/vxgehal-mgmtaux.c optional vxge dev/vxge/vxgehal/vxgehal-device.c optional vxge dev/vxge/vxgehal/vxgehal-mm.c optional vxge dev/vxge/vxgehal/vxgehal-driver.c optional vxge dev/vxge/vxgehal/vxgehal-virtualpath.c optional vxge dev/vxge/vxgehal/vxgehal-channel.c optional vxge dev/vxge/vxgehal/vxgehal-fifo.c optional vxge dev/watchdog/watchdog.c standard dev/wb/if_wb.c optional wb pci dev/wds/wd7000.c optional wds isa dev/wi/if_wi.c optional wi dev/wi/if_wi_pccard.c optional wi pccard dev/wi/if_wi_pci.c optional wi pci dev/wl/if_wl.c optional wl isa dev/wpi/if_wpi.c optional wpi pci wpifw.c optional wpifw \ compile-with "${AWK} -f $S/tools/fw_stub.awk wpi.fw:wpifw:153229 -mwpi -c${.TARGET}" \ no-implicit-rule before-depend local \ clean "wpifw.c" wpifw.fwo optional wpifw \ dependency "wpi.fw" \ compile-with "${NORMAL_FWO}" \ no-implicit-rule \ clean "wpifw.fwo" wpi.fw optional wpifw \ dependency "$S/contrib/dev/wpi/iwlwifi-3945-15.32.2.9.fw.uu" \ compile-with "${NORMAL_FW}" \ no-obj no-implicit-rule \ clean "wpi.fw" dev/xe/if_xe.c optional xe dev/xe/if_xe_pccard.c optional xe pccard dev/xen/balloon/balloon.c optional xen | xenhvm dev/xen/blkfront/blkfront.c optional xen | xenhvm dev/xen/blkback/blkback.c optional xen | xenhvm dev/xen/console/console.c optional xen | xenhvm dev/xen/console/xencons_ring.c optional xen | xenhvm dev/xen/control/control.c optional xen | xenhvm dev/xen/grant_table/grant_table.c optional xen | xenhvm dev/xen/netback/netback.c optional xen | xenhvm dev/xen/netfront/netfront.c optional xen | xenhvm dev/xen/xenpci/xenpci.c optional xenpci dev/xen/timer/timer.c optional xen | xenhvm dev/xen/pvcpu/pvcpu.c optional xen | xenhvm dev/xen/xenstore/xenstore.c optional xen | xenhvm dev/xen/xenstore/xenstore_dev.c optional xen | xenhvm dev/xen/xenstore/xenstored_dev.c optional xen | xenhvm dev/xen/evtchn/evtchn_dev.c optional xen | xenhvm dev/xen/privcmd/privcmd.c optional xen | xenhvm dev/xl/if_xl.c optional xl pci dev/xl/xlphy.c optional xl pci fs/autofs/autofs.c optional autofs fs/autofs/autofs_vfsops.c optional autofs fs/autofs/autofs_vnops.c optional autofs fs/deadfs/dead_vnops.c standard fs/devfs/devfs_devs.c standard fs/devfs/devfs_dir.c standard fs/devfs/devfs_rule.c standard fs/devfs/devfs_vfsops.c standard fs/devfs/devfs_vnops.c standard fs/fdescfs/fdesc_vfsops.c optional fdescfs fs/fdescfs/fdesc_vnops.c optional fdescfs fs/fifofs/fifo_vnops.c standard fs/cuse/cuse.c optional cuse fs/fuse/fuse_device.c optional fuse fs/fuse/fuse_file.c optional fuse fs/fuse/fuse_internal.c optional fuse fs/fuse/fuse_io.c optional fuse fs/fuse/fuse_ipc.c optional fuse fs/fuse/fuse_main.c optional fuse fs/fuse/fuse_node.c optional fuse fs/fuse/fuse_vfsops.c optional fuse fs/fuse/fuse_vnops.c optional fuse fs/msdosfs/msdosfs_conv.c optional msdosfs fs/msdosfs/msdosfs_denode.c optional msdosfs fs/msdosfs/msdosfs_fat.c optional msdosfs fs/msdosfs/msdosfs_fileno.c optional msdosfs fs/msdosfs/msdosfs_iconv.c optional msdosfs_iconv fs/msdosfs/msdosfs_lookup.c optional msdosfs fs/msdosfs/msdosfs_vfsops.c optional msdosfs fs/msdosfs/msdosfs_vnops.c optional msdosfs fs/nandfs/bmap.c optional nandfs fs/nandfs/nandfs_alloc.c optional nandfs fs/nandfs/nandfs_bmap.c optional nandfs fs/nandfs/nandfs_buffer.c optional nandfs fs/nandfs/nandfs_cleaner.c optional nandfs fs/nandfs/nandfs_cpfile.c optional nandfs fs/nandfs/nandfs_dat.c optional nandfs fs/nandfs/nandfs_dir.c optional nandfs fs/nandfs/nandfs_ifile.c optional nandfs fs/nandfs/nandfs_segment.c optional nandfs fs/nandfs/nandfs_subr.c optional nandfs fs/nandfs/nandfs_sufile.c optional nandfs fs/nandfs/nandfs_vfsops.c optional nandfs fs/nandfs/nandfs_vnops.c optional nandfs fs/nfs/nfs_commonkrpc.c optional nfscl | nfsd fs/nfs/nfs_commonsubs.c optional nfscl | nfsd fs/nfs/nfs_commonport.c optional nfscl | nfsd fs/nfs/nfs_commonacl.c optional nfscl | nfsd fs/nfsclient/nfs_clcomsubs.c optional nfscl fs/nfsclient/nfs_clsubs.c optional nfscl fs/nfsclient/nfs_clstate.c optional nfscl fs/nfsclient/nfs_clkrpc.c optional nfscl fs/nfsclient/nfs_clrpcops.c optional nfscl fs/nfsclient/nfs_clvnops.c optional nfscl fs/nfsclient/nfs_clnode.c optional nfscl fs/nfsclient/nfs_clvfsops.c optional nfscl fs/nfsclient/nfs_clport.c optional nfscl fs/nfsclient/nfs_clbio.c optional nfscl fs/nfsclient/nfs_clnfsiod.c optional nfscl fs/nfsserver/nfs_fha_new.c optional nfsd inet fs/nfsserver/nfs_nfsdsocket.c optional nfsd inet fs/nfsserver/nfs_nfsdsubs.c optional nfsd inet fs/nfsserver/nfs_nfsdstate.c optional nfsd inet fs/nfsserver/nfs_nfsdkrpc.c optional nfsd inet fs/nfsserver/nfs_nfsdserv.c optional nfsd inet fs/nfsserver/nfs_nfsdport.c optional nfsd inet fs/nfsserver/nfs_nfsdcache.c optional nfsd inet fs/nullfs/null_subr.c optional nullfs fs/nullfs/null_vfsops.c optional nullfs fs/nullfs/null_vnops.c optional nullfs fs/procfs/procfs.c optional procfs fs/procfs/procfs_ctl.c optional procfs fs/procfs/procfs_dbregs.c optional procfs fs/procfs/procfs_fpregs.c optional procfs fs/procfs/procfs_ioctl.c optional procfs fs/procfs/procfs_map.c optional procfs fs/procfs/procfs_mem.c optional procfs fs/procfs/procfs_note.c optional procfs fs/procfs/procfs_osrel.c optional procfs fs/procfs/procfs_regs.c optional procfs fs/procfs/procfs_rlimit.c optional procfs fs/procfs/procfs_status.c optional procfs fs/procfs/procfs_type.c optional procfs fs/pseudofs/pseudofs.c optional pseudofs fs/pseudofs/pseudofs_fileno.c optional pseudofs fs/pseudofs/pseudofs_vncache.c optional pseudofs fs/pseudofs/pseudofs_vnops.c optional pseudofs fs/smbfs/smbfs_io.c optional smbfs fs/smbfs/smbfs_node.c optional smbfs fs/smbfs/smbfs_smb.c optional smbfs fs/smbfs/smbfs_subr.c optional smbfs fs/smbfs/smbfs_vfsops.c optional smbfs fs/smbfs/smbfs_vnops.c optional smbfs fs/udf/osta.c optional udf fs/udf/udf_iconv.c optional udf_iconv fs/udf/udf_vfsops.c optional udf fs/udf/udf_vnops.c optional udf fs/unionfs/union_subr.c optional unionfs fs/unionfs/union_vfsops.c optional unionfs fs/unionfs/union_vnops.c optional unionfs fs/tmpfs/tmpfs_vnops.c optional tmpfs fs/tmpfs/tmpfs_fifoops.c optional tmpfs fs/tmpfs/tmpfs_vfsops.c optional tmpfs fs/tmpfs/tmpfs_subr.c optional tmpfs gdb/gdb_cons.c optional gdb gdb/gdb_main.c optional gdb gdb/gdb_packet.c optional gdb geom/bde/g_bde.c optional geom_bde geom/bde/g_bde_crypt.c optional geom_bde geom/bde/g_bde_lock.c optional geom_bde geom/bde/g_bde_work.c optional geom_bde geom/cache/g_cache.c optional geom_cache geom/concat/g_concat.c optional geom_concat geom/eli/g_eli.c optional geom_eli geom/eli/g_eli_crypto.c optional geom_eli geom/eli/g_eli_ctl.c optional geom_eli geom/eli/g_eli_integrity.c optional geom_eli geom/eli/g_eli_key.c optional geom_eli geom/eli/g_eli_key_cache.c optional geom_eli geom/eli/g_eli_privacy.c optional geom_eli geom/eli/pkcs5v2.c optional geom_eli geom/gate/g_gate.c optional geom_gate geom/geom_aes.c optional geom_aes geom/geom_bsd.c optional geom_bsd geom/geom_bsd_enc.c optional geom_bsd | geom_part_bsd geom/geom_ccd.c optional ccd | geom_ccd geom/geom_ctl.c standard geom/geom_dev.c standard geom/geom_disk.c standard geom/geom_dump.c standard geom/geom_event.c standard geom/geom_fox.c optional geom_fox geom/geom_flashmap.c optional fdt cfi | fdt nand geom/geom_io.c standard geom/geom_kern.c standard geom/geom_map.c optional geom_map geom/geom_mbr.c optional geom_mbr geom/geom_mbr_enc.c optional geom_mbr geom/geom_pc98.c optional geom_pc98 geom/geom_pc98_enc.c optional geom_pc98 geom/geom_redboot.c optional geom_redboot geom/geom_slice.c standard geom/geom_subr.c standard geom/geom_sunlabel.c optional geom_sunlabel geom/geom_sunlabel_enc.c optional geom_sunlabel geom/geom_vfs.c standard geom/geom_vol_ffs.c optional geom_vol geom/journal/g_journal.c optional geom_journal geom/journal/g_journal_ufs.c optional geom_journal geom/label/g_label.c optional geom_label | geom_label_gpt geom/label/g_label_ext2fs.c optional geom_label geom/label/g_label_iso9660.c optional geom_label geom/label/g_label_msdosfs.c optional geom_label geom/label/g_label_ntfs.c optional geom_label geom/label/g_label_reiserfs.c optional geom_label geom/label/g_label_ufs.c optional geom_label geom/label/g_label_gpt.c optional geom_label | geom_label_gpt geom/label/g_label_disk_ident.c optional geom_label geom/linux_lvm/g_linux_lvm.c optional geom_linux_lvm geom/mirror/g_mirror.c optional geom_mirror geom/mirror/g_mirror_ctl.c optional geom_mirror geom/mountver/g_mountver.c optional geom_mountver geom/multipath/g_multipath.c optional geom_multipath geom/nop/g_nop.c optional geom_nop geom/part/g_part.c standard geom/part/g_part_if.m standard geom/part/g_part_apm.c optional geom_part_apm geom/part/g_part_bsd.c optional geom_part_bsd geom/part/g_part_bsd64.c optional geom_part_bsd64 geom/part/g_part_ebr.c optional geom_part_ebr geom/part/g_part_gpt.c optional geom_part_gpt geom/part/g_part_ldm.c optional geom_part_ldm geom/part/g_part_mbr.c optional geom_part_mbr geom/part/g_part_pc98.c optional geom_part_pc98 geom/part/g_part_vtoc8.c optional geom_part_vtoc8 geom/raid/g_raid.c optional geom_raid geom/raid/g_raid_ctl.c optional geom_raid geom/raid/g_raid_md_if.m optional geom_raid geom/raid/g_raid_tr_if.m optional geom_raid geom/raid/md_ddf.c optional geom_raid geom/raid/md_intel.c optional geom_raid geom/raid/md_jmicron.c optional geom_raid geom/raid/md_nvidia.c optional geom_raid geom/raid/md_promise.c optional geom_raid geom/raid/md_sii.c optional geom_raid geom/raid/tr_concat.c optional geom_raid geom/raid/tr_raid0.c optional geom_raid geom/raid/tr_raid1.c optional geom_raid geom/raid/tr_raid1e.c optional geom_raid geom/raid/tr_raid5.c optional geom_raid geom/raid3/g_raid3.c optional geom_raid3 geom/raid3/g_raid3_ctl.c optional geom_raid3 geom/shsec/g_shsec.c optional geom_shsec geom/stripe/g_stripe.c optional geom_stripe geom/uncompress/g_uncompress.c optional geom_uncompress contrib/xz-embedded/freebsd/xz_malloc.c \ optional xz_embedded | geom_uncompress \ compile-with "${NORMAL_C} -I$S/contrib/xz-embedded/freebsd/ -I$S/contrib/xz-embedded/linux/lib/xz/ -I$S/contrib/xz-embedded/linux/include/linux/" contrib/xz-embedded/linux/lib/xz/xz_crc32.c \ optional xz_embedded | geom_uncompress \ compile-with "${NORMAL_C} -I$S/contrib/xz-embedded/freebsd/ -I$S/contrib/xz-embedded/linux/lib/xz/ -I$S/contrib/xz-embedded/linux/include/linux/" contrib/xz-embedded/linux/lib/xz/xz_dec_bcj.c \ optional xz_embedded | geom_uncompress \ compile-with "${NORMAL_C} -I$S/contrib/xz-embedded/freebsd/ -I$S/contrib/xz-embedded/linux/lib/xz/ -I$S/contrib/xz-embedded/linux/include/linux/" contrib/xz-embedded/linux/lib/xz/xz_dec_lzma2.c \ optional xz_embedded | geom_uncompress \ compile-with "${NORMAL_C} -I$S/contrib/xz-embedded/freebsd/ -I$S/contrib/xz-embedded/linux/lib/xz/ -I$S/contrib/xz-embedded/linux/include/linux/" contrib/xz-embedded/linux/lib/xz/xz_dec_stream.c \ optional xz_embedded | geom_uncompress \ compile-with "${NORMAL_C} -I$S/contrib/xz-embedded/freebsd/ -I$S/contrib/xz-embedded/linux/lib/xz/ -I$S/contrib/xz-embedded/linux/include/linux/" geom/uzip/g_uzip.c optional geom_uzip geom/vinum/geom_vinum.c optional geom_vinum geom/vinum/geom_vinum_create.c optional geom_vinum geom/vinum/geom_vinum_drive.c optional geom_vinum geom/vinum/geom_vinum_plex.c optional geom_vinum geom/vinum/geom_vinum_volume.c optional geom_vinum geom/vinum/geom_vinum_subr.c optional geom_vinum geom/vinum/geom_vinum_raid5.c optional geom_vinum geom/vinum/geom_vinum_share.c optional geom_vinum geom/vinum/geom_vinum_list.c optional geom_vinum geom/vinum/geom_vinum_rm.c optional geom_vinum geom/vinum/geom_vinum_init.c optional geom_vinum geom/vinum/geom_vinum_state.c optional geom_vinum geom/vinum/geom_vinum_rename.c optional geom_vinum geom/vinum/geom_vinum_move.c optional geom_vinum geom/vinum/geom_vinum_events.c optional geom_vinum geom/virstor/binstream.c optional geom_virstor geom/virstor/g_virstor.c optional geom_virstor geom/virstor/g_virstor_md.c optional geom_virstor geom/zero/g_zero.c optional geom_zero fs/ext2fs/ext2_alloc.c optional ext2fs fs/ext2fs/ext2_balloc.c optional ext2fs fs/ext2fs/ext2_bmap.c optional ext2fs fs/ext2fs/ext2_extents.c optional ext2fs fs/ext2fs/ext2_inode.c optional ext2fs fs/ext2fs/ext2_inode_cnv.c optional ext2fs fs/ext2fs/ext2_hash.c optional ext2fs fs/ext2fs/ext2_htree.c optional ext2fs fs/ext2fs/ext2_lookup.c optional ext2fs fs/ext2fs/ext2_subr.c optional ext2fs fs/ext2fs/ext2_vfsops.c optional ext2fs fs/ext2fs/ext2_vnops.c optional ext2fs gnu/fs/reiserfs/reiserfs_hashes.c optional reiserfs \ warning "kernel contains GPL contaminated ReiserFS filesystem" gnu/fs/reiserfs/reiserfs_inode.c optional reiserfs gnu/fs/reiserfs/reiserfs_item_ops.c optional reiserfs gnu/fs/reiserfs/reiserfs_namei.c optional reiserfs gnu/fs/reiserfs/reiserfs_prints.c optional reiserfs gnu/fs/reiserfs/reiserfs_stree.c optional reiserfs gnu/fs/reiserfs/reiserfs_vfsops.c optional reiserfs gnu/fs/reiserfs/reiserfs_vnops.c optional reiserfs # isa/isa_if.m standard isa/isa_common.c optional isa isa/isahint.c optional isa isa/pnp.c optional isa isapnp isa/pnpparse.c optional isa isapnp fs/cd9660/cd9660_bmap.c optional cd9660 fs/cd9660/cd9660_lookup.c optional cd9660 fs/cd9660/cd9660_node.c optional cd9660 fs/cd9660/cd9660_rrip.c optional cd9660 fs/cd9660/cd9660_util.c optional cd9660 fs/cd9660/cd9660_vfsops.c optional cd9660 fs/cd9660/cd9660_vnops.c optional cd9660 fs/cd9660/cd9660_iconv.c optional cd9660_iconv kern/bus_if.m standard kern/clock_if.m standard kern/cpufreq_if.m standard kern/device_if.m standard kern/imgact_binmisc.c optional imagact_binmisc kern/imgact_elf.c standard kern/imgact_elf32.c optional compat_freebsd32 kern/imgact_shell.c standard kern/inflate.c optional gzip kern/init_main.c standard kern/init_sysent.c standard kern/ksched.c optional _kposix_priority_scheduling kern/kern_acct.c standard kern/kern_alq.c optional alq kern/kern_clock.c standard kern/kern_condvar.c standard kern/kern_conf.c standard kern/kern_cons.c standard kern/kern_cpu.c standard kern/kern_cpuset.c standard kern/kern_context.c standard kern/kern_descrip.c standard kern/kern_dtrace.c optional kdtrace_hooks kern/kern_environment.c standard kern/kern_et.c standard kern/kern_event.c standard kern/kern_exec.c standard kern/kern_exit.c standard kern/kern_fail.c standard kern/kern_ffclock.c standard kern/kern_fork.c standard kern/kern_gzio.c optional gzio kern/kern_hhook.c standard kern/kern_idle.c standard kern/kern_intr.c standard kern/kern_jail.c standard kern/kern_khelp.c standard kern/kern_kthread.c standard kern/kern_ktr.c optional ktr kern/kern_ktrace.c standard kern/kern_linker.c standard kern/kern_lock.c standard kern/kern_lockf.c standard kern/kern_lockstat.c optional kdtrace_hooks kern/kern_loginclass.c standard kern/kern_malloc.c standard kern/kern_mbuf.c standard kern/kern_mib.c standard kern/kern_module.c standard kern/kern_mtxpool.c standard kern/kern_mutex.c standard kern/kern_ntptime.c standard kern/kern_osd.c standard kern/kern_physio.c standard kern/kern_pmc.c standard kern/kern_poll.c optional device_polling kern/kern_priv.c standard kern/kern_proc.c standard kern/kern_prot.c standard kern/kern_racct.c standard kern/kern_rangelock.c standard kern/kern_rctl.c standard kern/kern_resource.c standard kern/kern_rmlock.c standard kern/kern_rwlock.c standard kern/kern_sdt.c optional kdtrace_hooks kern/kern_sema.c standard kern/kern_sharedpage.c standard kern/kern_shutdown.c standard kern/kern_sig.c standard kern/kern_switch.c standard kern/kern_sx.c standard kern/kern_synch.c standard kern/kern_syscalls.c standard kern/kern_sysctl.c standard kern/kern_tc.c standard kern/kern_thr.c standard kern/kern_thread.c standard kern/kern_time.c standard kern/kern_timeout.c standard kern/kern_umtx.c standard kern/kern_uuid.c standard kern/kern_xxx.c standard kern/link_elf.c standard kern/linker_if.m standard kern/md4c.c optional netsmb kern/md5c.c standard kern/p1003_1b.c standard kern/posix4_mib.c standard kern/sched_4bsd.c optional sched_4bsd kern/sched_ule.c optional sched_ule kern/serdev_if.m standard kern/stack_protector.c standard \ compile-with "${NORMAL_C:N-fstack-protector*}" kern/subr_acl_nfs4.c optional ufs_acl | zfs kern/subr_acl_posix1e.c optional ufs_acl kern/subr_autoconf.c standard kern/subr_blist.c standard kern/subr_bus.c standard kern/subr_bus_dma.c standard kern/subr_bufring.c standard kern/subr_capability.c standard kern/subr_clock.c standard kern/subr_counter.c standard kern/subr_devstat.c standard kern/subr_disk.c standard kern/subr_eventhandler.c standard kern/subr_fattime.c standard kern/subr_firmware.c optional firmware kern/subr_hash.c standard kern/subr_hints.c standard kern/subr_kdb.c standard kern/subr_kobj.c standard kern/subr_lock.c standard kern/subr_log.c standard kern/subr_mbpool.c optional libmbpool kern/subr_mchain.c optional libmchain kern/subr_module.c standard kern/subr_msgbuf.c standard kern/subr_param.c standard kern/subr_pcpu.c standard kern/subr_pctrie.c standard kern/subr_power.c standard kern/subr_prf.c standard kern/subr_prof.c standard kern/subr_rman.c standard kern/subr_rtc.c standard kern/subr_sbuf.c standard kern/subr_scanf.c standard kern/subr_sglist.c standard kern/subr_sleepqueue.c standard kern/subr_smp.c standard kern/subr_stack.c optional ddb | stack | ktr kern/subr_taskqueue.c standard kern/subr_terminal.c optional vt kern/subr_trap.c standard kern/subr_turnstile.c standard kern/subr_uio.c standard kern/subr_unit.c standard kern/subr_vmem.c standard kern/subr_witness.c optional witness kern/sys_capability.c standard kern/sys_generic.c standard kern/sys_pipe.c standard kern/sys_procdesc.c standard kern/sys_process.c standard kern/sys_socket.c standard kern/syscalls.c standard kern/sysv_ipc.c standard kern/sysv_msg.c optional sysvmsg kern/sysv_sem.c optional sysvsem kern/sysv_shm.c optional sysvshm kern/tty.c standard kern/tty_compat.c optional compat_43tty kern/tty_info.c standard kern/tty_inq.c standard kern/tty_outq.c standard kern/tty_pts.c standard kern/tty_tty.c standard kern/tty_ttydisc.c standard kern/uipc_accf.c standard kern/uipc_debug.c optional ddb kern/uipc_domain.c standard kern/uipc_mbuf.c standard kern/uipc_mbuf2.c standard kern/uipc_mqueue.c optional p1003_1b_mqueue kern/uipc_sem.c optional p1003_1b_semaphores kern/uipc_shm.c standard kern/uipc_sockbuf.c standard kern/uipc_socket.c standard kern/uipc_syscalls.c standard kern/uipc_usrreq.c standard kern/vfs_acl.c standard kern/vfs_aio.c optional vfs_aio kern/vfs_bio.c standard kern/vfs_cache.c standard kern/vfs_cluster.c standard kern/vfs_default.c standard kern/vfs_export.c standard kern/vfs_extattr.c standard kern/vfs_hash.c standard kern/vfs_init.c standard kern/vfs_lookup.c standard kern/vfs_mount.c standard kern/vfs_mountroot.c standard kern/vfs_subr.c standard kern/vfs_syscalls.c standard kern/vfs_vnops.c standard # # Kernel GSS-API # gssd.h optional kgssapi \ dependency "$S/kgssapi/gssd.x" \ compile-with "RPCGEN_CPP='${CPP}' rpcgen -hM $S/kgssapi/gssd.x | grep -v pthread.h > gssd.h" \ no-obj no-implicit-rule before-depend local \ clean "gssd.h" gssd_xdr.c optional kgssapi \ dependency "$S/kgssapi/gssd.x gssd.h" \ compile-with "RPCGEN_CPP='${CPP}' rpcgen -c $S/kgssapi/gssd.x -o gssd_xdr.c" \ no-implicit-rule before-depend local \ clean "gssd_xdr.c" gssd_clnt.c optional kgssapi \ dependency "$S/kgssapi/gssd.x gssd.h" \ compile-with "RPCGEN_CPP='${CPP}' rpcgen -lM $S/kgssapi/gssd.x | grep -v string.h > gssd_clnt.c" \ no-implicit-rule before-depend local \ clean "gssd_clnt.c" kgssapi/gss_accept_sec_context.c optional kgssapi kgssapi/gss_add_oid_set_member.c optional kgssapi kgssapi/gss_acquire_cred.c optional kgssapi kgssapi/gss_canonicalize_name.c optional kgssapi kgssapi/gss_create_empty_oid_set.c optional kgssapi kgssapi/gss_delete_sec_context.c optional kgssapi kgssapi/gss_display_status.c optional kgssapi kgssapi/gss_export_name.c optional kgssapi kgssapi/gss_get_mic.c optional kgssapi kgssapi/gss_init_sec_context.c optional kgssapi kgssapi/gss_impl.c optional kgssapi kgssapi/gss_import_name.c optional kgssapi kgssapi/gss_names.c optional kgssapi kgssapi/gss_pname_to_uid.c optional kgssapi kgssapi/gss_release_buffer.c optional kgssapi kgssapi/gss_release_cred.c optional kgssapi kgssapi/gss_release_name.c optional kgssapi kgssapi/gss_release_oid_set.c optional kgssapi kgssapi/gss_set_cred_option.c optional kgssapi kgssapi/gss_test_oid_set_member.c optional kgssapi kgssapi/gss_unwrap.c optional kgssapi kgssapi/gss_verify_mic.c optional kgssapi kgssapi/gss_wrap.c optional kgssapi kgssapi/gss_wrap_size_limit.c optional kgssapi kgssapi/gssd_prot.c optional kgssapi kgssapi/krb5/krb5_mech.c optional kgssapi kgssapi/krb5/kcrypto.c optional kgssapi kgssapi/krb5/kcrypto_aes.c optional kgssapi kgssapi/krb5/kcrypto_arcfour.c optional kgssapi kgssapi/krb5/kcrypto_des.c optional kgssapi kgssapi/krb5/kcrypto_des3.c optional kgssapi kgssapi/kgss_if.m optional kgssapi kgssapi/gsstest.c optional kgssapi_debug # These files in libkern/ are those needed by all architectures. Some # of the files in libkern/ are only needed on some architectures, e.g., # libkern/divdi3.c is needed by i386 but not alpha. Also, some of these # routines may be optimized for a particular platform. In either case, # the file should be moved to conf/files. from here. # libkern/arc4random.c standard libkern/bcd.c standard libkern/bsearch.c standard libkern/crc32.c standard libkern/explicit_bzero.c standard libkern/fnmatch.c standard libkern/iconv.c optional libiconv libkern/iconv_converter_if.m optional libiconv libkern/iconv_ucs.c optional libiconv libkern/iconv_xlat.c optional libiconv libkern/iconv_xlat16.c optional libiconv libkern/inet_aton.c standard libkern/inet_ntoa.c standard libkern/inet_ntop.c standard libkern/inet_pton.c standard libkern/jenkins_hash.c standard libkern/murmur3_32.c standard libkern/mcount.c optional profiling-routine libkern/memcchr.c standard libkern/memchr.c optional fdt | gdb libkern/memcmp.c standard libkern/memmem.c optional gdb libkern/qsort.c standard libkern/qsort_r.c standard libkern/random.c standard libkern/scanc.c standard libkern/strcasecmp.c standard libkern/strcat.c standard libkern/strchr.c standard libkern/strcmp.c standard libkern/strcpy.c standard libkern/strcspn.c standard libkern/strdup.c standard libkern/strndup.c standard libkern/strlcat.c standard libkern/strlcpy.c standard libkern/strlen.c standard libkern/strncmp.c standard libkern/strncpy.c standard libkern/strnlen.c standard libkern/strrchr.c standard libkern/strsep.c standard libkern/strspn.c standard libkern/strstr.c standard libkern/strtol.c standard libkern/strtoq.c standard libkern/strtoul.c standard libkern/strtouq.c standard libkern/strvalid.c standard +libkern/timingsafe_bcmp.c standard net/bpf.c standard net/bpf_buffer.c optional bpf net/bpf_jitter.c optional bpf_jitter net/bpf_filter.c optional bpf | netgraph_bpf net/bpf_zerocopy.c optional bpf net/bridgestp.c optional bridge | if_bridge net/flowtable.c optional flowtable inet | flowtable inet6 net/ieee8023ad_lacp.c optional lagg net/if.c standard net/if_arcsubr.c optional arcnet net/if_atmsubr.c optional atm net/if_bridge.c optional bridge inet | if_bridge inet net/if_clone.c standard net/if_dead.c standard net/if_debug.c optional ddb net/if_disc.c optional disc net/if_edsc.c optional edsc net/if_enc.c optional enc ipsec inet | enc ipsec inet6 net/if_epair.c optional epair net/if_ethersubr.c optional ether net/if_fddisubr.c optional fddi net/if_fwsubr.c optional fwip net/if_gif.c optional gif inet | gif inet6 | \ netgraph_gif inet | netgraph_gif inet6 net/if_gre.c optional gre inet | gre inet6 net/if_iso88025subr.c optional token net/if_lagg.c optional lagg net/if_loop.c optional loop net/if_llatbl.c standard net/if_me.c optional me inet net/if_media.c standard net/if_mib.c standard net/if_spppfr.c optional sppp | netgraph_sppp net/if_spppsubr.c optional sppp | netgraph_sppp net/if_stf.c optional stf inet inet6 net/if_tun.c optional tun net/if_tap.c optional tap net/if_vlan.c optional vlan net/if_vxlan.c optional vxlan inet | vxlan inet6 net/mppcc.c optional netgraph_mppc_compression net/mppcd.c optional netgraph_mppc_compression net/netisr.c standard net/pfil.c optional ether | inet net/radix.c standard net/radix_mpath.c standard net/raw_cb.c standard net/raw_usrreq.c standard net/route.c standard net/rtsock.c standard net/slcompress.c optional netgraph_vjc | sppp | \ netgraph_sppp net/vnet.c optional vimage net/zlib.c optional crypto | geom_uzip | ipsec | \ mxge | netgraph_deflate | \ ddb_ctf | gzio | geom_uncompress net80211/ieee80211.c optional wlan net80211/ieee80211_acl.c optional wlan wlan_acl net80211/ieee80211_action.c optional wlan net80211/ieee80211_ageq.c optional wlan net80211/ieee80211_adhoc.c optional wlan \ compile-with "${NORMAL_C} -Wno-unused-function" net80211/ieee80211_ageq.c optional wlan net80211/ieee80211_amrr.c optional wlan | wlan_amrr net80211/ieee80211_crypto.c optional wlan \ compile-with "${NORMAL_C} -Wno-unused-function" net80211/ieee80211_crypto_ccmp.c optional wlan wlan_ccmp net80211/ieee80211_crypto_none.c optional wlan net80211/ieee80211_crypto_tkip.c optional wlan wlan_tkip net80211/ieee80211_crypto_wep.c optional wlan wlan_wep net80211/ieee80211_ddb.c optional wlan ddb net80211/ieee80211_dfs.c optional wlan net80211/ieee80211_freebsd.c optional wlan net80211/ieee80211_hostap.c optional wlan \ compile-with "${NORMAL_C} -Wno-unused-function" net80211/ieee80211_ht.c optional wlan net80211/ieee80211_hwmp.c optional wlan ieee80211_support_mesh net80211/ieee80211_input.c optional wlan net80211/ieee80211_ioctl.c optional wlan net80211/ieee80211_mesh.c optional wlan ieee80211_support_mesh \ compile-with "${NORMAL_C} -Wno-unused-function" net80211/ieee80211_monitor.c optional wlan net80211/ieee80211_node.c optional wlan net80211/ieee80211_output.c optional wlan net80211/ieee80211_phy.c optional wlan net80211/ieee80211_power.c optional wlan net80211/ieee80211_proto.c optional wlan net80211/ieee80211_radiotap.c optional wlan net80211/ieee80211_ratectl.c optional wlan net80211/ieee80211_ratectl_none.c optional wlan net80211/ieee80211_regdomain.c optional wlan net80211/ieee80211_rssadapt.c optional wlan wlan_rssadapt net80211/ieee80211_scan.c optional wlan net80211/ieee80211_scan_sta.c optional wlan net80211/ieee80211_sta.c optional wlan \ compile-with "${NORMAL_C} -Wno-unused-function" net80211/ieee80211_superg.c optional wlan ieee80211_support_superg net80211/ieee80211_tdma.c optional wlan ieee80211_support_tdma net80211/ieee80211_wds.c optional wlan net80211/ieee80211_xauth.c optional wlan wlan_xauth net80211/ieee80211_alq.c optional wlan ieee80211_alq netgraph/atm/ccatm/ng_ccatm.c optional ngatm_ccatm \ compile-with "${NORMAL_C} -I$S/contrib/ngatm" netgraph/atm/ng_atm.c optional ngatm_atm netgraph/atm/ngatmbase.c optional ngatm_atmbase \ compile-with "${NORMAL_C} -I$S/contrib/ngatm" netgraph/atm/sscfu/ng_sscfu.c optional ngatm_sscfu \ compile-with "${NORMAL_C} -I$S/contrib/ngatm" netgraph/atm/sscop/ng_sscop.c optional ngatm_sscop \ compile-with "${NORMAL_C} -I$S/contrib/ngatm" netgraph/atm/uni/ng_uni.c optional ngatm_uni \ compile-with "${NORMAL_C} -I$S/contrib/ngatm" netgraph/bluetooth/common/ng_bluetooth.c optional netgraph_bluetooth netgraph/bluetooth/drivers/bt3c/ng_bt3c_pccard.c optional netgraph_bluetooth_bt3c netgraph/bluetooth/drivers/h4/ng_h4.c optional netgraph_bluetooth_h4 netgraph/bluetooth/drivers/ubt/ng_ubt.c optional netgraph_bluetooth_ubt usb netgraph/bluetooth/drivers/ubtbcmfw/ubtbcmfw.c optional netgraph_bluetooth_ubtbcmfw usb netgraph/bluetooth/hci/ng_hci_cmds.c optional netgraph_bluetooth_hci netgraph/bluetooth/hci/ng_hci_evnt.c optional netgraph_bluetooth_hci netgraph/bluetooth/hci/ng_hci_main.c optional netgraph_bluetooth_hci netgraph/bluetooth/hci/ng_hci_misc.c optional netgraph_bluetooth_hci netgraph/bluetooth/hci/ng_hci_ulpi.c optional netgraph_bluetooth_hci netgraph/bluetooth/l2cap/ng_l2cap_cmds.c optional netgraph_bluetooth_l2cap netgraph/bluetooth/l2cap/ng_l2cap_evnt.c optional netgraph_bluetooth_l2cap netgraph/bluetooth/l2cap/ng_l2cap_llpi.c optional netgraph_bluetooth_l2cap netgraph/bluetooth/l2cap/ng_l2cap_main.c optional netgraph_bluetooth_l2cap netgraph/bluetooth/l2cap/ng_l2cap_misc.c optional netgraph_bluetooth_l2cap netgraph/bluetooth/l2cap/ng_l2cap_ulpi.c optional netgraph_bluetooth_l2cap netgraph/bluetooth/socket/ng_btsocket.c optional netgraph_bluetooth_socket netgraph/bluetooth/socket/ng_btsocket_hci_raw.c optional netgraph_bluetooth_socket netgraph/bluetooth/socket/ng_btsocket_l2cap.c optional netgraph_bluetooth_socket netgraph/bluetooth/socket/ng_btsocket_l2cap_raw.c optional netgraph_bluetooth_socket netgraph/bluetooth/socket/ng_btsocket_rfcomm.c optional netgraph_bluetooth_socket netgraph/bluetooth/socket/ng_btsocket_sco.c optional netgraph_bluetooth_socket netgraph/netflow/netflow.c optional netgraph_netflow netgraph/netflow/netflow_v9.c optional netgraph_netflow netgraph/netflow/ng_netflow.c optional netgraph_netflow netgraph/ng_UI.c optional netgraph_UI netgraph/ng_async.c optional netgraph_async netgraph/ng_atmllc.c optional netgraph_atmllc netgraph/ng_base.c optional netgraph netgraph/ng_bpf.c optional netgraph_bpf netgraph/ng_bridge.c optional netgraph_bridge netgraph/ng_car.c optional netgraph_car netgraph/ng_cisco.c optional netgraph_cisco netgraph/ng_deflate.c optional netgraph_deflate netgraph/ng_device.c optional netgraph_device netgraph/ng_echo.c optional netgraph_echo netgraph/ng_eiface.c optional netgraph_eiface netgraph/ng_ether.c optional netgraph_ether netgraph/ng_ether_echo.c optional netgraph_ether_echo netgraph/ng_frame_relay.c optional netgraph_frame_relay netgraph/ng_gif.c optional netgraph_gif inet6 | netgraph_gif inet netgraph/ng_gif_demux.c optional netgraph_gif_demux netgraph/ng_hole.c optional netgraph_hole netgraph/ng_iface.c optional netgraph_iface netgraph/ng_ip_input.c optional netgraph_ip_input netgraph/ng_ipfw.c optional netgraph_ipfw inet ipfirewall netgraph/ng_ksocket.c optional netgraph_ksocket netgraph/ng_l2tp.c optional netgraph_l2tp netgraph/ng_lmi.c optional netgraph_lmi netgraph/ng_mppc.c optional netgraph_mppc_compression | \ netgraph_mppc_encryption netgraph/ng_nat.c optional netgraph_nat inet libalias netgraph/ng_one2many.c optional netgraph_one2many netgraph/ng_parse.c optional netgraph netgraph/ng_patch.c optional netgraph_patch netgraph/ng_pipe.c optional netgraph_pipe netgraph/ng_ppp.c optional netgraph_ppp netgraph/ng_pppoe.c optional netgraph_pppoe netgraph/ng_pptpgre.c optional netgraph_pptpgre netgraph/ng_pred1.c optional netgraph_pred1 netgraph/ng_rfc1490.c optional netgraph_rfc1490 netgraph/ng_socket.c optional netgraph_socket netgraph/ng_split.c optional netgraph_split netgraph/ng_sppp.c optional netgraph_sppp netgraph/ng_tag.c optional netgraph_tag netgraph/ng_tcpmss.c optional netgraph_tcpmss netgraph/ng_tee.c optional netgraph_tee netgraph/ng_tty.c optional netgraph_tty netgraph/ng_vjc.c optional netgraph_vjc netgraph/ng_vlan.c optional netgraph_vlan netinet/accf_data.c optional accept_filter_data inet netinet/accf_dns.c optional accept_filter_dns inet netinet/accf_http.c optional accept_filter_http inet netinet/if_atm.c optional atm netinet/if_ether.c optional inet ether netinet/igmp.c optional inet netinet/in.c optional inet netinet/in_debug.c optional inet ddb netinet/in_kdtrace.c optional inet | inet6 netinet/ip_carp.c optional inet carp | inet6 carp netinet/in_gif.c optional gif inet | netgraph_gif inet netinet/ip_gre.c optional gre inet netinet/ip_id.c optional inet netinet/in_mcast.c optional inet netinet/in_pcb.c optional inet | inet6 netinet/in_pcbgroup.c optional inet pcbgroup | inet6 pcbgroup netinet/in_proto.c optional inet | inet6 netinet/in_rmx.c optional inet netinet/in_rss.c optional inet rss | inet6 rss netinet/ip_divert.c optional inet ipdivert ipfirewall netinet/ip_ecn.c optional inet | inet6 netinet/ip_encap.c optional inet | inet6 netinet/ip_fastfwd.c optional inet netinet/ip_icmp.c optional inet | inet6 netinet/ip_input.c optional inet netinet/ip_ipsec.c optional inet ipsec netinet/ip_mroute.c optional mrouting inet netinet/ip_options.c optional inet netinet/ip_output.c optional inet netinet/raw_ip.c optional inet | inet6 netinet/cc/cc.c optional inet | inet6 netinet/cc/cc_newreno.c optional inet | inet6 netinet/sctp_asconf.c optional inet sctp | inet6 sctp netinet/sctp_auth.c optional inet sctp | inet6 sctp netinet/sctp_bsd_addr.c optional inet sctp | inet6 sctp netinet/sctp_cc_functions.c optional inet sctp | inet6 sctp netinet/sctp_crc32.c optional inet sctp | inet6 sctp netinet/sctp_indata.c optional inet sctp | inet6 sctp netinet/sctp_input.c optional inet sctp | inet6 sctp netinet/sctp_output.c optional inet sctp | inet6 sctp netinet/sctp_pcb.c optional inet sctp | inet6 sctp netinet/sctp_peeloff.c optional inet sctp | inet6 sctp netinet/sctp_ss_functions.c optional inet sctp | inet6 sctp netinet/sctp_syscalls.c optional inet sctp | inet6 sctp netinet/sctp_sysctl.c optional inet sctp | inet6 sctp netinet/sctp_timer.c optional inet sctp | inet6 sctp netinet/sctp_usrreq.c optional inet sctp | inet6 sctp netinet/sctputil.c optional inet sctp | inet6 sctp netinet/tcp_debug.c optional tcpdebug netinet/tcp_hostcache.c optional inet | inet6 netinet/tcp_input.c optional inet | inet6 netinet/tcp_lro.c optional inet | inet6 netinet/tcp_output.c optional inet | inet6 netinet/tcp_offload.c optional tcp_offload inet | tcp_offload inet6 netinet/tcp_reass.c optional inet | inet6 netinet/tcp_sack.c optional inet | inet6 netinet/tcp_subr.c optional inet | inet6 netinet/tcp_syncache.c optional inet | inet6 netinet/tcp_timer.c optional inet | inet6 netinet/tcp_timewait.c optional inet | inet6 netinet/tcp_usrreq.c optional inet | inet6 netinet/toeplitz.c optional inet rss | inet6 rss netinet/udp_usrreq.c optional inet | inet6 netinet/libalias/alias.c optional libalias inet | netgraph_nat inet netinet/libalias/alias_db.c optional libalias inet | netgraph_nat inet netinet/libalias/alias_mod.c optional libalias | netgraph_nat netinet/libalias/alias_proxy.c optional libalias inet | netgraph_nat inet netinet/libalias/alias_util.c optional libalias inet | netgraph_nat inet netinet/libalias/alias_sctp.c optional libalias inet | netgraph_nat inet netinet6/dest6.c optional inet6 netinet6/frag6.c optional inet6 netinet6/icmp6.c optional inet6 netinet6/in6.c optional inet6 netinet6/in6_cksum.c optional inet6 netinet6/in6_gif.c optional gif inet6 | netgraph_gif inet6 netinet6/in6_ifattach.c optional inet6 netinet6/in6_mcast.c optional inet6 netinet6/in6_pcb.c optional inet6 netinet6/in6_pcbgroup.c optional inet6 pcbgroup netinet6/in6_proto.c optional inet6 netinet6/in6_rmx.c optional inet6 netinet6/in6_src.c optional inet6 netinet6/ip6_forward.c optional inet6 netinet6/ip6_gre.c optional gre inet6 netinet6/ip6_id.c optional inet6 netinet6/ip6_input.c optional inet6 netinet6/ip6_mroute.c optional mrouting inet6 netinet6/ip6_output.c optional inet6 netinet6/ip6_ipsec.c optional inet6 ipsec netinet6/mld6.c optional inet6 netinet6/nd6.c optional inet6 netinet6/nd6_nbr.c optional inet6 netinet6/nd6_rtr.c optional inet6 netinet6/raw_ip6.c optional inet6 netinet6/route6.c optional inet6 netinet6/scope6.c optional inet6 netinet6/sctp6_usrreq.c optional inet6 sctp netinet6/udp6_usrreq.c optional inet6 netipsec/ipsec.c optional ipsec inet | ipsec inet6 netipsec/ipsec_input.c optional ipsec inet | ipsec inet6 netipsec/ipsec_mbuf.c optional ipsec inet | ipsec inet6 netipsec/ipsec_output.c optional ipsec inet | ipsec inet6 netipsec/key.c optional ipsec inet | ipsec inet6 netipsec/key_debug.c optional ipsec inet | ipsec inet6 netipsec/keysock.c optional ipsec inet | ipsec inet6 netipsec/xform_ah.c optional ipsec inet | ipsec inet6 netipsec/xform_esp.c optional ipsec inet | ipsec inet6 netipsec/xform_ipcomp.c optional ipsec inet | ipsec inet6 netipsec/xform_ipip.c optional ipsec inet | ipsec inet6 netipsec/xform_tcp.c optional ipsec inet tcp_signature | \ ipsec inet6 tcp_signature netnatm/natm.c optional natm netnatm/natm_pcb.c optional natm netnatm/natm_proto.c optional natm netpfil/ipfw/dn_heap.c optional inet dummynet netpfil/ipfw/dn_sched_fifo.c optional inet dummynet netpfil/ipfw/dn_sched_prio.c optional inet dummynet netpfil/ipfw/dn_sched_qfq.c optional inet dummynet netpfil/ipfw/dn_sched_rr.c optional inet dummynet netpfil/ipfw/dn_sched_wf2q.c optional inet dummynet netpfil/ipfw/ip_dummynet.c optional inet dummynet netpfil/ipfw/ip_dn_io.c optional inet dummynet netpfil/ipfw/ip_dn_glue.c optional inet dummynet netpfil/ipfw/ip_fw2.c optional inet ipfirewall netpfil/ipfw/ip_fw_dynamic.c optional inet ipfirewall netpfil/ipfw/ip_fw_log.c optional inet ipfirewall netpfil/ipfw/ip_fw_pfil.c optional inet ipfirewall netpfil/ipfw/ip_fw_sockopt.c optional inet ipfirewall netpfil/ipfw/ip_fw_table.c optional inet ipfirewall netpfil/ipfw/ip_fw_table_algo.c optional inet ipfirewall netpfil/ipfw/ip_fw_table_value.c optional inet ipfirewall netpfil/ipfw/ip_fw_iface.c optional inet ipfirewall netpfil/ipfw/ip_fw_nat.c optional inet ipfirewall_nat netpfil/pf/if_pflog.c optional pflog pf inet netpfil/pf/if_pfsync.c optional pfsync pf inet netpfil/pf/pf.c optional pf inet netpfil/pf/pf_if.c optional pf inet netpfil/pf/pf_ioctl.c optional pf inet netpfil/pf/pf_lb.c optional pf inet netpfil/pf/pf_norm.c optional pf inet netpfil/pf/pf_osfp.c optional pf inet netpfil/pf/pf_ruleset.c optional pf inet netpfil/pf/pf_table.c optional pf inet netpfil/pf/in4_cksum.c optional pf inet netsmb/smb_conn.c optional netsmb netsmb/smb_crypt.c optional netsmb netsmb/smb_dev.c optional netsmb netsmb/smb_iod.c optional netsmb netsmb/smb_rq.c optional netsmb netsmb/smb_smb.c optional netsmb netsmb/smb_subr.c optional netsmb netsmb/smb_trantcp.c optional netsmb netsmb/smb_usr.c optional netsmb nfs/bootp_subr.c optional bootp nfsclient | bootp nfscl nfs/krpc_subr.c optional bootp nfsclient | bootp nfscl nfs/nfs_common.c optional nfsclient | nfsserver nfs/nfs_diskless.c optional nfsclient nfs_root | nfscl nfs_root nfs/nfs_fha.c optional nfsserver | nfsd nfs/nfs_lock.c optional nfsclient | nfscl | nfslockd | nfsd nfsclient/nfs_bio.c optional nfsclient nfsclient/nfs_node.c optional nfsclient nfsclient/nfs_krpc.c optional nfsclient nfsclient/nfs_subs.c optional nfsclient nfsclient/nfs_nfsiod.c optional nfsclient nfsclient/nfs_vfsops.c optional nfsclient nfsclient/nfs_vnops.c optional nfsclient nfsserver/nfs_fha_old.c optional nfsserver nfsserver/nfs_serv.c optional nfsserver nfsserver/nfs_srvkrpc.c optional nfsserver nfsserver/nfs_srvsubs.c optional nfsserver nfs/nfs_nfssvc.c optional nfsserver | nfscl | nfsd nlm/nlm_advlock.c optional nfslockd | nfsd nlm/nlm_prot_clnt.c optional nfslockd | nfsd nlm/nlm_prot_impl.c optional nfslockd | nfsd nlm/nlm_prot_server.c optional nfslockd | nfsd nlm/nlm_prot_svc.c optional nfslockd | nfsd nlm/nlm_prot_xdr.c optional nfslockd | nfsd nlm/sm_inter_xdr.c optional nfslockd | nfsd # OpenFabrics Enterprise Distribution (Infiniband) ofed/include/linux/linux_compat.c optional ofed \ no-depend compile-with "${OFED_C}" ofed/include/linux/linux_idr.c optional ofed \ no-depend compile-with "${OFED_C}" ofed/include/linux/linux_radix.c optional ofed \ no-depend compile-with "${OFED_C}" ofed/drivers/infiniband/core/addr.c optional ofed \ no-depend \ compile-with "${OFED_C} -I$S/ofed/drivers/infiniband/core/" ofed/drivers/infiniband/core/agent.c optional ofed \ no-depend \ compile-with "${OFED_C} -I$S/ofed/drivers/infiniband/core/" ofed/drivers/infiniband/core/cache.c optional ofed \ no-depend \ compile-with "${OFED_C} -I$S/ofed/drivers/infiniband/core/" # XXX Mad.c must be ordered before cm.c for sysinit sets to occur in # the correct order. ofed/drivers/infiniband/core/mad.c optional ofed \ no-depend \ compile-with "${OFED_C} -I$S/ofed/drivers/infiniband/core/" ofed/drivers/infiniband/core/cm.c optional ofed \ no-depend \ compile-with "${OFED_C} -I$S/ofed/drivers/infiniband/core/ -Wno-unused-function" ofed/drivers/infiniband/core/cma.c optional ofed \ no-depend \ compile-with "${OFED_C} -I$S/ofed/drivers/infiniband/core/" ofed/drivers/infiniband/core/device.c optional ofed \ no-depend \ compile-with "${OFED_C} -I$S/ofed/drivers/infiniband/core/" ofed/drivers/infiniband/core/fmr_pool.c optional ofed \ no-depend \ compile-with "${OFED_C} -I$S/ofed/drivers/infiniband/core/" ofed/drivers/infiniband/core/iwcm.c optional ofed \ no-depend \ compile-with "${OFED_C} -I$S/ofed/drivers/infiniband/core/" ofed/drivers/infiniband/core/local_sa.c optional ofed \ no-depend \ compile-with "${OFED_C} -I$S/ofed/drivers/infiniband/core/" ofed/drivers/infiniband/core/mad_rmpp.c optional ofed \ no-depend \ compile-with "${OFED_C} -I$S/ofed/drivers/infiniband/core/" ofed/drivers/infiniband/core/multicast.c optional ofed \ no-depend \ compile-with "${OFED_C} -I$S/ofed/drivers/infiniband/core/" ofed/drivers/infiniband/core/notice.c optional ofed \ no-depend \ compile-with "${OFED_C} -I$S/ofed/drivers/infiniband/core/" ofed/drivers/infiniband/core/packer.c optional ofed \ no-depend \ compile-with "${OFED_C} -I$S/ofed/drivers/infiniband/core/" ofed/drivers/infiniband/core/sa_query.c optional ofed \ no-depend \ compile-with "${OFED_C} -I$S/ofed/drivers/infiniband/core/" ofed/drivers/infiniband/core/smi.c optional ofed \ no-depend \ compile-with "${OFED_C} -I$S/ofed/drivers/infiniband/core/" ofed/drivers/infiniband/core/sysfs.c optional ofed \ no-depend \ compile-with "${OFED_C} -I$S/ofed/drivers/infiniband/core/" ofed/drivers/infiniband/core/ucm.c optional ofed \ no-depend \ compile-with "${OFED_C} -I$S/ofed/drivers/infiniband/core/" ofed/drivers/infiniband/core/ucma.c optional ofed \ no-depend \ compile-with "${OFED_C} -I$S/ofed/drivers/infiniband/core/" ofed/drivers/infiniband/core/ud_header.c optional ofed \ no-depend \ compile-with "${OFED_C} -I$S/ofed/drivers/infiniband/core/" ofed/drivers/infiniband/core/umem.c optional ofed \ no-depend \ compile-with "${OFED_C} -I$S/ofed/drivers/infiniband/core/" ofed/drivers/infiniband/core/user_mad.c optional ofed \ no-depend \ compile-with "${OFED_C} -I$S/ofed/drivers/infiniband/core/" ofed/drivers/infiniband/core/uverbs_cmd.c optional ofed \ no-depend \ compile-with "${OFED_C} -I$S/ofed/drivers/infiniband/core/" ofed/drivers/infiniband/core/uverbs_main.c optional ofed \ no-depend \ compile-with "${OFED_C} -I$S/ofed/drivers/infiniband/core/" ofed/drivers/infiniband/core/uverbs_marshall.c optional ofed \ no-depend \ compile-with "${OFED_C} -I$S/ofed/drivers/infiniband/core/" ofed/drivers/infiniband/core/verbs.c optional ofed \ no-depend \ compile-with "${OFED_C} -I$S/ofed/drivers/infiniband/core/" ofed/drivers/infiniband/ulp/ipoib/ipoib_cm.c optional ipoib \ no-depend \ compile-with "${OFED_C} -I$S/ofed/drivers/infiniband/ulp/ipoib/" #ofed/drivers/infiniband/ulp/ipoib/ipoib_fs.c optional ipoib \ # no-depend \ # compile-with "${OFED_C} -I$S/ofed/drivers/infiniband/ulp/ipoib/" ofed/drivers/infiniband/ulp/ipoib/ipoib_ib.c optional ipoib \ no-depend \ compile-with "${OFED_C} -I$S/ofed/drivers/infiniband/ulp/ipoib/" ofed/drivers/infiniband/ulp/ipoib/ipoib_main.c optional ipoib \ no-depend \ compile-with "${OFED_C} -I$S/ofed/drivers/infiniband/ulp/ipoib/" ofed/drivers/infiniband/ulp/ipoib/ipoib_multicast.c optional ipoib \ no-depend \ compile-with "${OFED_C} -I$S/ofed/drivers/infiniband/ulp/ipoib/" ofed/drivers/infiniband/ulp/ipoib/ipoib_verbs.c optional ipoib \ no-depend \ compile-with "${OFED_C} -I$S/ofed/drivers/infiniband/ulp/ipoib/" #ofed/drivers/infiniband/ulp/ipoib/ipoib_vlan.c optional ipoib \ # no-depend \ # compile-with "${OFED_C} -I$S/ofed/drivers/infiniband/ulp/ipoib/" ofed/drivers/infiniband/ulp/sdp/sdp_bcopy.c optional sdp inet \ no-depend \ compile-with "${OFED_C} -I$S/ofed/drivers/infiniband/ulp/sdp/" ofed/drivers/infiniband/ulp/sdp/sdp_main.c optional sdp inet \ no-depend \ compile-with "${OFED_C} -I$S/ofed/drivers/infiniband/ulp/sdp/" ofed/drivers/infiniband/ulp/sdp/sdp_rx.c optional sdp inet \ no-depend \ compile-with "${OFED_C} -I$S/ofed/drivers/infiniband/ulp/sdp/" ofed/drivers/infiniband/ulp/sdp/sdp_cma.c optional sdp inet \ no-depend \ compile-with "${OFED_C} -I$S/ofed/drivers/infiniband/ulp/sdp/" ofed/drivers/infiniband/ulp/sdp/sdp_tx.c optional sdp inet \ no-depend \ compile-with "${OFED_C} -I$S/ofed/drivers/infiniband/ulp/sdp/" ofed/drivers/infiniband/hw/mlx4/alias_GUID.c optional mlx4ib \ no-depend obj-prefix "mlx4ib_" \ compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/infiniband/hw/mlx4/" ofed/drivers/infiniband/hw/mlx4/mcg.c optional mlx4ib \ no-depend obj-prefix "mlx4ib_" \ compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/infiniband/hw/mlx4/" ofed/drivers/infiniband/hw/mlx4/sysfs.c optional mlx4ib \ no-depend obj-prefix "mlx4ib_" \ compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/infiniband/hw/mlx4/" ofed/drivers/infiniband/hw/mlx4/cm.c optional mlx4ib \ no-depend obj-prefix "mlx4ib_" \ compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/infiniband/hw/mlx4/" ofed/drivers/infiniband/hw/mlx4/ah.c optional mlx4ib \ no-depend obj-prefix "mlx4ib_" \ compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/infiniband/hw/mlx4/" ofed/drivers/infiniband/hw/mlx4/cq.c optional mlx4ib \ no-depend obj-prefix "mlx4ib_" \ compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/infiniband/hw/mlx4/" ofed/drivers/infiniband/hw/mlx4/doorbell.c optional mlx4ib \ no-depend obj-prefix "mlx4ib_" \ compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/infiniband/hw/mlx4/" ofed/drivers/infiniband/hw/mlx4/mad.c optional mlx4ib \ no-depend obj-prefix "mlx4ib_" \ compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/infiniband/hw/mlx4/" ofed/drivers/infiniband/hw/mlx4/main.c optional mlx4ib \ no-depend obj-prefix "mlx4ib_" \ compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/infiniband/hw/mlx4/" ofed/drivers/infiniband/hw/mlx4/mr.c optional mlx4ib \ no-depend obj-prefix "mlx4ib_" \ compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/infiniband/hw/mlx4/" ofed/drivers/infiniband/hw/mlx4/qp.c optional mlx4ib \ no-depend obj-prefix "mlx4ib_" \ compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/infiniband/hw/mlx4/" ofed/drivers/infiniband/hw/mlx4/srq.c optional mlx4ib \ no-depend obj-prefix "mlx4ib_" \ compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/infiniband/hw/mlx4/" ofed/drivers/infiniband/hw/mlx4/wc.c optional mlx4ib \ no-depend obj-prefix "mlx4ib_" \ compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/infiniband/hw/mlx4/" ofed/drivers/net/mlx4/alloc.c optional mlx4ib | mlxen \ no-depend obj-prefix "mlx4_" \ compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/net/mlx4/" ofed/drivers/net/mlx4/catas.c optional mlx4ib | mlxen \ no-depend obj-prefix "mlx4_" \ compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/net/mlx4/" ofed/drivers/net/mlx4/cmd.c optional mlx4ib | mlxen \ no-depend obj-prefix "mlx4_" \ compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/net/mlx4/" ofed/drivers/net/mlx4/cq.c optional mlx4ib | mlxen \ no-depend obj-prefix "mlx4_" \ compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/net/mlx4/" ofed/drivers/net/mlx4/eq.c optional mlx4ib | mlxen \ no-depend obj-prefix "mlx4_" \ compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/net/mlx4/" ofed/drivers/net/mlx4/fw.c optional mlx4ib | mlxen \ no-depend obj-prefix "mlx4_" \ compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/net/mlx4/" ofed/drivers/net/mlx4/icm.c optional mlx4ib | mlxen \ no-depend obj-prefix "mlx4_" \ compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/net/mlx4/" ofed/drivers/net/mlx4/intf.c optional mlx4ib | mlxen \ no-depend obj-prefix "mlx4_" \ compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/net/mlx4/" ofed/drivers/net/mlx4/main.c optional mlx4ib | mlxen \ no-depend obj-prefix "mlx4_" \ compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/net/mlx4/" ofed/drivers/net/mlx4/mcg.c optional mlx4ib | mlxen \ no-depend obj-prefix "mlx4_" \ compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/net/mlx4/ -Wno-unused" ofed/drivers/net/mlx4/mr.c optional mlx4ib | mlxen \ no-depend obj-prefix "mlx4_" \ compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/net/mlx4/" ofed/drivers/net/mlx4/pd.c optional mlx4ib | mlxen \ no-depend obj-prefix "mlx4_" \ compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/net/mlx4/" ofed/drivers/net/mlx4/port.c optional mlx4ib | mlxen \ no-depend obj-prefix "mlx4_" \ compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/net/mlx4/" ofed/drivers/net/mlx4/profile.c optional mlx4ib | mlxen \ no-depend obj-prefix "mlx4_" \ compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/net/mlx4/" ofed/drivers/net/mlx4/qp.c optional mlx4ib | mlxen \ no-depend obj-prefix "mlx4_" \ compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/net/mlx4/" ofed/drivers/net/mlx4/reset.c optional mlx4ib | mlxen \ no-depend obj-prefix "mlx4_" \ compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/net/mlx4/" ofed/drivers/net/mlx4/sense.c optional mlx4ib | mlxen \ no-depend obj-prefix "mlx4_" \ compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/net/mlx4/" ofed/drivers/net/mlx4/srq.c optional mlx4ib | mlxen \ no-depend obj-prefix "mlx4_" \ compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/net/mlx4/" ofed/drivers/net/mlx4/resource_tracker.c optional mlx4ib | mlxen \ no-depend obj-prefix "mlx4_" \ compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/net/mlx4/" ofed/drivers/net/mlx4/sys_tune.c optional mlx4ib | mlxen \ no-depend obj-prefix "mlx4_" \ compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/net/mlx4/" ofed/drivers/net/mlx4/en_cq.c optional mlxen \ no-depend obj-prefix "mlx4_" \ compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/net/mlx4/" ofed/drivers/net/mlx4/utils.c optional mlxen \ no-depend obj-prefix "mlx4_" \ compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/net/mlx4/" ofed/drivers/net/mlx4/en_main.c optional mlxen \ no-depend obj-prefix "mlx4_" \ compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/net/mlx4/" ofed/drivers/net/mlx4/en_netdev.c optional mlxen \ no-depend obj-prefix "mlx4_" \ compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/net/mlx4/" ofed/drivers/net/mlx4/en_port.c optional mlxen \ no-depend obj-prefix "mlx4_" \ compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/net/mlx4/" ofed/drivers/net/mlx4/en_resources.c optional mlxen \ no-depend obj-prefix "mlx4_" \ compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/net/mlx4/" ofed/drivers/net/mlx4/en_rx.c optional mlxen \ no-depend obj-prefix "mlx4_" \ compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/net/mlx4/" ofed/drivers/net/mlx4/en_tx.c optional mlxen \ no-depend obj-prefix "mlx4_" \ compile-with "${OFED_C_NOIMP} -I$S/ofed/drivers/net/mlx4/" ofed/drivers/infiniband/hw/mthca/mthca_allocator.c optional mthca \ no-depend compile-with "${OFED_C}" ofed/drivers/infiniband/hw/mthca/mthca_av.c optional mthca \ no-depend compile-with "${OFED_C}" ofed/drivers/infiniband/hw/mthca/mthca_catas.c optional mthca \ no-depend compile-with "${OFED_C}" ofed/drivers/infiniband/hw/mthca/mthca_cmd.c optional mthca \ no-depend compile-with "${OFED_C}" ofed/drivers/infiniband/hw/mthca/mthca_cq.c optional mthca \ no-depend compile-with "${OFED_C}" ofed/drivers/infiniband/hw/mthca/mthca_eq.c optional mthca \ no-depend compile-with "${OFED_C}" ofed/drivers/infiniband/hw/mthca/mthca_mad.c optional mthca \ no-depend compile-with "${OFED_C}" ofed/drivers/infiniband/hw/mthca/mthca_main.c optional mthca \ no-depend compile-with "${OFED_C}" ofed/drivers/infiniband/hw/mthca/mthca_mcg.c optional mthca \ no-depend compile-with "${OFED_C}" ofed/drivers/infiniband/hw/mthca/mthca_memfree.c optional mthca \ no-depend compile-with "${OFED_C}" ofed/drivers/infiniband/hw/mthca/mthca_mr.c optional mthca \ no-depend compile-with "${OFED_C}" ofed/drivers/infiniband/hw/mthca/mthca_pd.c optional mthca \ no-depend compile-with "${OFED_C}" ofed/drivers/infiniband/hw/mthca/mthca_profile.c optional mthca \ no-depend compile-with "${OFED_C}" ofed/drivers/infiniband/hw/mthca/mthca_provider.c optional mthca \ no-depend compile-with "${OFED_C}" ofed/drivers/infiniband/hw/mthca/mthca_qp.c optional mthca \ no-depend compile-with "${OFED_C}" ofed/drivers/infiniband/hw/mthca/mthca_reset.c optional mthca \ no-depend compile-with "${OFED_C}" ofed/drivers/infiniband/hw/mthca/mthca_srq.c optional mthca \ no-depend compile-with "${OFED_C}" ofed/drivers/infiniband/hw/mthca/mthca_uar.c optional mthca \ no-depend compile-with "${OFED_C}" # crypto support opencrypto/cast.c optional crypto | ipsec opencrypto/criov.c optional crypto opencrypto/crypto.c optional crypto opencrypto/cryptodev.c optional cryptodev opencrypto/cryptodev_if.m optional crypto opencrypto/cryptosoft.c optional crypto opencrypto/cryptodeflate.c optional crypto +opencrypto/gmac.c optional crypto +opencrypto/gfmult.c optional crypto opencrypto/rmd160.c optional crypto | ipsec opencrypto/skipjack.c optional crypto opencrypto/xform.c optional crypto rpc/auth_none.c optional krpc | nfslockd | nfsclient | nfsserver | nfscl | nfsd rpc/auth_unix.c optional krpc | nfslockd | nfsclient | nfscl | nfsd rpc/authunix_prot.c optional krpc | nfslockd | nfsclient | nfsserver | nfscl | nfsd rpc/clnt_bck.c optional krpc | nfslockd | nfsserver | nfscl | nfsd rpc/clnt_dg.c optional krpc | nfslockd | nfsclient | nfscl | nfsd rpc/clnt_rc.c optional krpc | nfslockd | nfsclient | nfscl | nfsd rpc/clnt_vc.c optional krpc | nfslockd | nfsclient | nfsserver | nfscl | nfsd rpc/getnetconfig.c optional krpc | nfslockd | nfsclient | nfsserver | nfscl | nfsd rpc/replay.c optional krpc | nfslockd | nfsclient | nfsserver | nfscl | nfsd rpc/rpc_callmsg.c optional krpc | nfslockd | nfsclient | nfsserver | nfscl | nfsd rpc/rpc_generic.c optional krpc | nfslockd | nfsclient | nfsserver | nfscl | nfsd rpc/rpc_prot.c optional krpc | nfslockd | nfsclient | nfsserver | nfscl | nfsd rpc/rpcb_clnt.c optional krpc | nfslockd | nfsclient | nfsserver | nfscl | nfsd rpc/rpcb_prot.c optional krpc | nfslockd | nfsclient | nfsserver | nfscl | nfsd rpc/svc.c optional krpc | nfslockd | nfsclient | nfsserver | nfscl | nfsd rpc/svc_auth.c optional krpc | nfslockd | nfsclient | nfsserver | nfscl | nfsd rpc/svc_auth_unix.c optional krpc | nfslockd | nfsclient | nfsserver | nfscl | nfsd rpc/svc_dg.c optional krpc | nfslockd | nfsserver | nfscl | nfsd rpc/svc_generic.c optional krpc | nfslockd | nfsserver | nfscl | nfsd rpc/svc_vc.c optional krpc | nfslockd | nfsserver | nfscl | nfsd rpc/rpcsec_gss/rpcsec_gss.c optional krpc kgssapi | nfslockd kgssapi | nfscl kgssapi | nfsd kgssapi rpc/rpcsec_gss/rpcsec_gss_conf.c optional krpc kgssapi | nfslockd kgssapi | nfscl kgssapi | nfsd kgssapi rpc/rpcsec_gss/rpcsec_gss_misc.c optional krpc kgssapi | nfslockd kgssapi | nfscl kgssapi | nfsd kgssapi rpc/rpcsec_gss/rpcsec_gss_prot.c optional krpc kgssapi | nfslockd kgssapi | nfscl kgssapi | nfsd kgssapi rpc/rpcsec_gss/svc_rpcsec_gss.c optional krpc kgssapi | nfslockd kgssapi | nfscl kgssapi | nfsd kgssapi security/audit/audit.c optional audit security/audit/audit_arg.c optional audit security/audit/audit_bsm.c optional audit security/audit/audit_bsm_klib.c optional audit security/audit/audit_pipe.c optional audit security/audit/audit_syscalls.c standard security/audit/audit_trigger.c optional audit security/audit/audit_worker.c optional audit security/audit/bsm_domain.c optional audit security/audit/bsm_errno.c optional audit security/audit/bsm_fcntl.c optional audit security/audit/bsm_socket_type.c optional audit security/audit/bsm_token.c optional audit security/mac/mac_audit.c optional mac audit security/mac/mac_cred.c optional mac security/mac/mac_framework.c optional mac security/mac/mac_inet.c optional mac inet | mac inet6 security/mac/mac_inet6.c optional mac inet6 security/mac/mac_label.c optional mac security/mac/mac_net.c optional mac security/mac/mac_pipe.c optional mac security/mac/mac_posix_sem.c optional mac security/mac/mac_posix_shm.c optional mac security/mac/mac_priv.c optional mac security/mac/mac_process.c optional mac security/mac/mac_socket.c optional mac security/mac/mac_syscalls.c standard security/mac/mac_system.c optional mac security/mac/mac_sysv_msg.c optional mac security/mac/mac_sysv_sem.c optional mac security/mac/mac_sysv_shm.c optional mac security/mac/mac_vfs.c optional mac security/mac_biba/mac_biba.c optional mac_biba security/mac_bsdextended/mac_bsdextended.c optional mac_bsdextended security/mac_bsdextended/ugidfw_system.c optional mac_bsdextended security/mac_bsdextended/ugidfw_vnode.c optional mac_bsdextended security/mac_ifoff/mac_ifoff.c optional mac_ifoff security/mac_lomac/mac_lomac.c optional mac_lomac security/mac_mls/mac_mls.c optional mac_mls security/mac_none/mac_none.c optional mac_none security/mac_partition/mac_partition.c optional mac_partition security/mac_portacl/mac_portacl.c optional mac_portacl security/mac_seeotheruids/mac_seeotheruids.c optional mac_seeotheruids security/mac_stub/mac_stub.c optional mac_stub security/mac_test/mac_test.c optional mac_test teken/teken.c optional sc | vt ufs/ffs/ffs_alloc.c optional ffs ufs/ffs/ffs_balloc.c optional ffs ufs/ffs/ffs_inode.c optional ffs ufs/ffs/ffs_snapshot.c optional ffs ufs/ffs/ffs_softdep.c optional ffs ufs/ffs/ffs_subr.c optional ffs ufs/ffs/ffs_tables.c optional ffs ufs/ffs/ffs_vfsops.c optional ffs ufs/ffs/ffs_vnops.c optional ffs ufs/ffs/ffs_rawread.c optional ffs directio ufs/ffs/ffs_suspend.c optional ffs ufs/ufs/ufs_acl.c optional ffs ufs/ufs/ufs_bmap.c optional ffs ufs/ufs/ufs_dirhash.c optional ffs ufs/ufs/ufs_extattr.c optional ffs ufs/ufs/ufs_gjournal.c optional ffs UFS_GJOURNAL ufs/ufs/ufs_inode.c optional ffs ufs/ufs/ufs_lookup.c optional ffs ufs/ufs/ufs_quota.c optional ffs ufs/ufs/ufs_vfsops.c optional ffs ufs/ufs/ufs_vnops.c optional ffs vm/default_pager.c standard vm/device_pager.c standard vm/phys_pager.c standard vm/redzone.c optional DEBUG_REDZONE vm/sg_pager.c standard vm/swap_pager.c standard vm/uma_core.c standard vm/uma_dbg.c standard vm/memguard.c optional DEBUG_MEMGUARD vm/vm_fault.c standard vm/vm_glue.c standard vm/vm_init.c standard vm/vm_kern.c standard vm/vm_map.c standard vm/vm_meter.c standard vm/vm_mmap.c standard vm/vm_object.c standard vm/vm_page.c standard vm/vm_pageout.c standard vm/vm_pager.c standard vm/vm_phys.c standard vm/vm_radix.c standard vm/vm_reserv.c standard vm/vm_unix.c standard vm/vm_zeroidle.c standard vm/vnode_pager.c standard xen/features.c optional xen | xenhvm xen/xenbus/xenbus_if.m optional xen | xenhvm xen/xenbus/xenbus.c optional xen | xenhvm xen/xenbus/xenbusb_if.m optional xen | xenhvm xen/xenbus/xenbusb.c optional xen | xenhvm xen/xenbus/xenbusb_front.c optional xen | xenhvm xen/xenbus/xenbusb_back.c optional xen | xenhvm xdr/xdr.c optional krpc | nfslockd | nfsclient | nfsserver | nfscl | nfsd xdr/xdr_array.c optional krpc | nfslockd | nfsclient | nfsserver | nfscl | nfsd xdr/xdr_mbuf.c optional krpc | nfslockd | nfsclient | nfsserver | nfscl | nfsd xdr/xdr_mem.c optional krpc | nfslockd | nfsclient | nfsserver | nfscl | nfsd xdr/xdr_reference.c optional krpc | nfslockd | nfsclient | nfsserver | nfscl | nfsd xdr/xdr_sizeof.c optional krpc | nfslockd | nfsclient | nfsserver | nfscl | nfsd Index: projects/clang350-import/sys/conf/files.amd64 =================================================================== --- projects/clang350-import/sys/conf/files.amd64 (revision 275748) +++ projects/clang350-import/sys/conf/files.amd64 (revision 275749) @@ -1,577 +1,582 @@ # This file tells config what files go into building a kernel, # files marked standard are always included. # # $FreeBSD$ # # The long compile-with and dependency lines are required because of # limitations in config: backslash-newline doesn't work in strings, and # dependency lines other than the first are silently ignored. # # linux32_genassym.o optional compat_linux32 \ dependency "$S/amd64/linux32/linux32_genassym.c" \ compile-with "${CC} ${CFLAGS:N-fno-common} -c ${.IMPSRC}" \ no-obj no-implicit-rule \ clean "linux32_genassym.o" # linux32_assym.h optional compat_linux32 \ dependency "$S/kern/genassym.sh linux32_genassym.o" \ compile-with "sh $S/kern/genassym.sh linux32_genassym.o > ${.TARGET}" \ no-obj no-implicit-rule before-depend \ clean "linux32_assym.h" # ia32_genassym.o standard \ dependency "$S/compat/ia32/ia32_genassym.c" \ compile-with "${CC} ${CFLAGS:N-fno-common} -c ${.IMPSRC}" \ no-obj no-implicit-rule \ clean "ia32_genassym.o" # ia32_assym.h standard \ dependency "$S/kern/genassym.sh ia32_genassym.o" \ compile-with "env NM='${NM}' sh $S/kern/genassym.sh ia32_genassym.o > ${.TARGET}" \ no-obj no-implicit-rule before-depend \ clean "ia32_assym.h" # font.h optional sc_dflt_font \ compile-with "uudecode < /usr/share/syscons/fonts/${SC_DFLT_FONT}-8x16.fnt && file2c 'static u_char dflt_font_16[16*256] = {' '};' < ${SC_DFLT_FONT}-8x16 > font.h && uudecode < /usr/share/syscons/fonts/${SC_DFLT_FONT}-8x14.fnt && file2c 'static u_char dflt_font_14[14*256] = {' '};' < ${SC_DFLT_FONT}-8x14 >> font.h && uudecode < /usr/share/syscons/fonts/${SC_DFLT_FONT}-8x8.fnt && file2c 'static u_char dflt_font_8[8*256] = {' '};' < ${SC_DFLT_FONT}-8x8 >> font.h" \ no-obj no-implicit-rule before-depend \ clean "font.h ${SC_DFLT_FONT}-8x14 ${SC_DFLT_FONT}-8x16 ${SC_DFLT_FONT}-8x8" # atkbdmap.h optional atkbd_dflt_keymap \ compile-with "/usr/sbin/kbdcontrol -L ${ATKBD_DFLT_KEYMAP} | sed -e 's/^static keymap_t.* = /static keymap_t key_map = /' -e 's/^static accentmap_t.* = /static accentmap_t accent_map = /' > atkbdmap.h" \ no-obj no-implicit-rule before-depend \ clean "atkbdmap.h" # ukbdmap.h optional ukbd_dflt_keymap \ compile-with "/usr/sbin/kbdcontrol -L ${UKBD_DFLT_KEYMAP} | sed -e 's/^static keymap_t.* = /static keymap_t key_map = /' -e 's/^static accentmap_t.* = /static accentmap_t accent_map = /' > ukbdmap.h" \ no-obj no-implicit-rule before-depend \ clean "ukbdmap.h" # hpt27xx_lib.o optional hpt27xx \ dependency "$S/dev/hpt27xx/amd64-elf.hpt27xx_lib.o.uu" \ compile-with "uudecode < $S/dev/hpt27xx/amd64-elf.hpt27xx_lib.o.uu" \ no-implicit-rule # hptmvraid.o optional hptmv \ dependency "$S/dev/hptmv/amd64-elf.raid.o.uu" \ compile-with "uudecode < $S/dev/hptmv/amd64-elf.raid.o.uu" \ no-implicit-rule # hptnr_lib.o optional hptnr \ dependency "$S/dev/hptnr/amd64-elf.hptnr_lib.o.uu" \ compile-with "uudecode < $S/dev/hptnr/amd64-elf.hptnr_lib.o.uu" \ no-implicit-rule # hptrr_lib.o optional hptrr \ dependency "$S/dev/hptrr/amd64-elf.hptrr_lib.o.uu" \ compile-with "uudecode < $S/dev/hptrr/amd64-elf.hptrr_lib.o.uu" \ no-implicit-rule # amd64/acpica/acpi_machdep.c optional acpi acpi_wakecode.o optional acpi \ dependency "$S/amd64/acpica/acpi_wakecode.S assym.s" \ compile-with "${NORMAL_S}" \ no-obj no-implicit-rule before-depend \ clean "acpi_wakecode.o" acpi_wakecode.bin optional acpi \ dependency "acpi_wakecode.o" \ compile-with "${OBJCOPY} -S -O binary acpi_wakecode.o ${.TARGET}" \ no-obj no-implicit-rule before-depend \ clean "acpi_wakecode.bin" acpi_wakecode.h optional acpi \ dependency "acpi_wakecode.bin" \ compile-with "file2c -sx 'static char wakecode[] = {' '};' < acpi_wakecode.bin > ${.TARGET}" \ no-obj no-implicit-rule before-depend \ clean "acpi_wakecode.h" acpi_wakedata.h optional acpi \ dependency "acpi_wakecode.o" \ compile-with '${NM} -n --defined-only acpi_wakecode.o | while read offset dummy what; do echo "#define $${what} 0x$${offset}"; done > ${.TARGET}' \ no-obj no-implicit-rule before-depend \ clean "acpi_wakedata.h" # amd64/amd64/amd64_mem.c optional mem #amd64/amd64/apic_vector.S standard amd64/amd64/atomic.c standard amd64/amd64/autoconf.c standard amd64/amd64/bios.c standard amd64/amd64/bpf_jit_machdep.c optional bpf_jitter amd64/amd64/cpu_switch.S standard amd64/amd64/db_disasm.c optional ddb amd64/amd64/db_interface.c optional ddb amd64/amd64/db_trace.c optional ddb amd64/amd64/elf_machdep.c standard amd64/amd64/exception.S standard amd64/amd64/fpu.c standard amd64/amd64/gdb_machdep.c optional gdb amd64/amd64/in_cksum.c optional inet | inet6 amd64/amd64/initcpu.c standard amd64/amd64/io.c optional io amd64/amd64/locore.S standard no-obj amd64/amd64/xen-locore.S optional xenhvm amd64/amd64/machdep.c standard amd64/amd64/mem.c optional mem amd64/amd64/minidump_machdep.c standard amd64/amd64/mp_machdep.c optional smp amd64/amd64/mp_watchdog.c optional mp_watchdog smp amd64/amd64/mpboot.S optional smp amd64/amd64/pmap.c standard amd64/amd64/prof_machdep.c optional profiling-routine amd64/amd64/ptrace_machdep.c standard amd64/amd64/sigtramp.S standard amd64/amd64/stack_machdep.c optional ddb | stack amd64/amd64/support.S standard amd64/amd64/sys_machdep.c standard amd64/amd64/trap.c standard amd64/amd64/uio_machdep.c standard amd64/amd64/uma_machdep.c standard amd64/amd64/vm_machdep.c standard amd64/pci/pci_cfgreg.c optional pci cddl/contrib/opensolaris/common/atomic/amd64/opensolaris_atomic.S optional zfs compile-with "${ZFS_S}" crypto/aesni/aeskeys_amd64.S optional aesni crypto/aesni/aesni.c optional aesni +aesni_ghash.o optional aesni \ + dependency "$S/crypto/aesni/aesni_ghash.c" \ + compile-with "${CC} -c ${CFLAGS:C/^-O2$/-O3/:N-nostdinc} ${WERROR} ${PROF} -mmmx -msse -msse4 -maes -mpclmul ${.IMPSRC}" \ + no-implicit-rule \ + clean "aesni_ghash.o" aesni_wrap.o optional aesni \ dependency "$S/crypto/aesni/aesni_wrap.c" \ - compile-with "${CC} -c ${CFLAGS:C/^-O2$/-O3/:N-nostdinc} ${WERROR} ${PROF} -mmmx -msse -maes ${.IMPSRC}" \ + compile-with "${CC} -c ${CFLAGS:C/^-O2$/-O3/:N-nostdinc} ${WERROR} ${PROF} -mmmx -msse -msse4 -maes ${.IMPSRC}" \ no-implicit-rule \ clean "aesni_wrap.o" crypto/blowfish/bf_enc.c optional crypto | ipsec crypto/des/des_enc.c optional crypto | ipsec | netsmb crypto/via/padlock.c optional padlock crypto/via/padlock_cipher.c optional padlock crypto/via/padlock_hash.c optional padlock dev/acpica/acpi_if.m standard dev/acpi_support/acpi_wmi_if.m standard dev/agp/agp_amd64.c optional agp dev/agp/agp_i810.c optional agp dev/agp/agp_via.c optional agp dev/amdsbwd/amdsbwd.c optional amdsbwd dev/amdtemp/amdtemp.c optional amdtemp dev/arcmsr/arcmsr.c optional arcmsr pci dev/asmc/asmc.c optional asmc isa dev/atkbdc/atkbd.c optional atkbd atkbdc dev/atkbdc/atkbd_atkbdc.c optional atkbd atkbdc dev/atkbdc/atkbdc.c optional atkbdc dev/atkbdc/atkbdc_isa.c optional atkbdc isa dev/atkbdc/atkbdc_subr.c optional atkbdc dev/atkbdc/psm.c optional psm atkbdc dev/bxe/bxe.c optional bxe pci dev/bxe/bxe_stats.c optional bxe pci dev/bxe/bxe_debug.c optional bxe pci dev/bxe/ecore_sp.c optional bxe pci dev/bxe/bxe_elink.c optional bxe pci dev/bxe/57710_init_values.c optional bxe pci dev/bxe/57711_init_values.c optional bxe pci dev/bxe/57712_init_values.c optional bxe pci dev/coretemp/coretemp.c optional coretemp dev/cpuctl/cpuctl.c optional cpuctl dev/dpms/dpms.c optional dpms # There are no systems with isa slots, so all ed isa entries should go.. dev/ed/if_ed_3c503.c optional ed isa ed_3c503 dev/ed/if_ed_isa.c optional ed isa dev/ed/if_ed_wd80x3.c optional ed isa dev/ed/if_ed_hpp.c optional ed isa ed_hpp dev/ed/if_ed_sic.c optional ed isa ed_sic dev/fb/fb.c optional fb | vga dev/fb/s3_pci.c optional s3pci dev/fb/vesa.c optional vga vesa dev/fb/vga.c optional vga dev/ichwd/ichwd.c optional ichwd dev/if_ndis/if_ndis.c optional ndis dev/if_ndis/if_ndis_pccard.c optional ndis pccard dev/if_ndis/if_ndis_pci.c optional ndis cardbus | ndis pci dev/if_ndis/if_ndis_usb.c optional ndis usb dev/io/iodev.c optional io dev/ipmi/ipmi.c optional ipmi dev/ipmi/ipmi_acpi.c optional ipmi acpi dev/ipmi/ipmi_isa.c optional ipmi isa dev/ipmi/ipmi_kcs.c optional ipmi dev/ipmi/ipmi_smic.c optional ipmi dev/ipmi/ipmi_smbus.c optional ipmi smbus dev/ipmi/ipmi_smbios.c optional ipmi dev/ipmi/ipmi_ssif.c optional ipmi smbus dev/ipmi/ipmi_pci.c optional ipmi pci dev/ipmi/ipmi_linux.c optional ipmi compat_linux32 dev/ixl/if_ixl.c optional ixl pci \ compile-with "${NORMAL_C} -I$S/dev/ixl" dev/ixl/if_ixlv.c optional ixlv pci \ compile-with "${NORMAL_C} -I$S/dev/ixl" dev/ixl/ixlvc.c optional ixlv pci \ compile-with "${NORMAL_C} -I$S/dev/ixl" dev/ixl/ixl_txrx.c optional ixl pci | ixlv pci \ compile-with "${NORMAL_C} -I$S/dev/ixl" dev/ixl/i40e_osdep.c optional ixl pci | ixlv pci \ compile-with "${NORMAL_C} -I$S/dev/ixl" dev/ixl/i40e_lan_hmc.c optional ixl pci | ixlv pci \ compile-with "${NORMAL_C} -I$S/dev/ixl" dev/ixl/i40e_hmc.c optional ixl pci | ixlv pci \ compile-with "${NORMAL_C} -I$S/dev/ixl" dev/ixl/i40e_common.c optional ixl pci | ixlv pci \ compile-with "${NORMAL_C} -I$S/dev/ixl" dev/ixl/i40e_nvm.c optional ixl pci | ixlv pci \ compile-with "${NORMAL_C} -I$S/dev/ixl" dev/ixl/i40e_adminq.c optional ixl pci | ixlv pci \ compile-with "${NORMAL_C} -I$S/dev/ixl" dev/fdc/fdc.c optional fdc dev/fdc/fdc_acpi.c optional fdc dev/fdc/fdc_isa.c optional fdc isa dev/fdc/fdc_pccard.c optional fdc pccard dev/fdt/fdt_x86.c optional fdt dev/hpt27xx/hpt27xx_os_bsd.c optional hpt27xx dev/hpt27xx/hpt27xx_osm_bsd.c optional hpt27xx dev/hpt27xx/hpt27xx_config.c optional hpt27xx dev/hptmv/entry.c optional hptmv dev/hptmv/mv.c optional hptmv dev/hptmv/gui_lib.c optional hptmv dev/hptmv/hptproc.c optional hptmv dev/hptmv/ioctl.c optional hptmv dev/hptnr/hptnr_os_bsd.c optional hptnr dev/hptnr/hptnr_osm_bsd.c optional hptnr dev/hptnr/hptnr_config.c optional hptnr dev/hptrr/hptrr_os_bsd.c optional hptrr dev/hptrr/hptrr_osm_bsd.c optional hptrr dev/hptrr/hptrr_config.c optional hptrr dev/hwpmc/hwpmc_amd.c optional hwpmc dev/hwpmc/hwpmc_intel.c optional hwpmc dev/hwpmc/hwpmc_core.c optional hwpmc dev/hwpmc/hwpmc_uncore.c optional hwpmc dev/hwpmc/hwpmc_piv.c optional hwpmc dev/hwpmc/hwpmc_tsc.c optional hwpmc dev/hwpmc/hwpmc_x86.c optional hwpmc dev/hyperv/netvsc/hv_net_vsc.c optional hyperv dev/hyperv/netvsc/hv_netvsc_drv_freebsd.c optional hyperv dev/hyperv/netvsc/hv_rndis_filter.c optional hyperv dev/hyperv/stordisengage/hv_ata_pci_disengage.c optional hyperv dev/hyperv/storvsc/hv_storvsc_drv_freebsd.c optional hyperv dev/hyperv/utilities/hv_kvp.c optional hyperv dev/hyperv/utilities/hv_util.c optional hyperv dev/hyperv/vmbus/hv_channel.c optional hyperv dev/hyperv/vmbus/hv_channel_mgmt.c optional hyperv dev/hyperv/vmbus/hv_connection.c optional hyperv dev/hyperv/vmbus/hv_hv.c optional hyperv dev/hyperv/vmbus/hv_ring_buffer.c optional hyperv dev/hyperv/vmbus/hv_vmbus_drv_freebsd.c optional hyperv dev/kbd/kbd.c optional atkbd | sc | ukbd | vt dev/nfe/if_nfe.c optional nfe pci dev/ntb/if_ntb/if_ntb.c optional if_ntb dev/ntb/ntb_hw/ntb_hw.c optional if_ntb ntb_hw dev/nvd/nvd.c optional nvd nvme dev/nvme/nvme.c optional nvme dev/nvme/nvme_ctrlr.c optional nvme dev/nvme/nvme_ctrlr_cmd.c optional nvme dev/nvme/nvme_ns.c optional nvme dev/nvme/nvme_ns_cmd.c optional nvme dev/nvme/nvme_qpair.c optional nvme dev/nvme/nvme_sysctl.c optional nvme dev/nvme/nvme_test.c optional nvme dev/nvme/nvme_util.c optional nvme dev/nvram/nvram.c optional nvram isa dev/random/ivy.c optional rdrand_rng dev/random/nehemiah.c optional padlock_rng dev/qlxge/qls_dbg.c optional qlxge pci dev/qlxge/qls_dump.c optional qlxge pci dev/qlxge/qls_hw.c optional qlxge pci dev/qlxge/qls_ioctl.c optional qlxge pci dev/qlxge/qls_isr.c optional qlxge pci dev/qlxge/qls_os.c optional qlxge pci dev/qlxgb/qla_dbg.c optional qlxgb pci dev/qlxgb/qla_hw.c optional qlxgb pci dev/qlxgb/qla_ioctl.c optional qlxgb pci dev/qlxgb/qla_isr.c optional qlxgb pci dev/qlxgb/qla_misc.c optional qlxgb pci dev/qlxgb/qla_os.c optional qlxgb pci dev/qlxgbe/ql_dbg.c optional qlxgbe pci dev/qlxgbe/ql_hw.c optional qlxgbe pci dev/qlxgbe/ql_ioctl.c optional qlxgbe pci dev/qlxgbe/ql_isr.c optional qlxgbe pci dev/qlxgbe/ql_misc.c optional qlxgbe pci dev/qlxgbe/ql_os.c optional qlxgbe pci dev/qlxgbe/ql_reset.c optional qlxgbe pci dev/sfxge/common/efx_bootcfg.c optional sfxge inet pci dev/sfxge/common/efx_ev.c optional sfxge inet pci dev/sfxge/common/efx_filter.c optional sfxge inet pci dev/sfxge/common/efx_intr.c optional sfxge inet pci dev/sfxge/common/efx_mac.c optional sfxge inet pci dev/sfxge/common/efx_mcdi.c optional sfxge inet pci dev/sfxge/common/efx_mon.c optional sfxge inet pci dev/sfxge/common/efx_nic.c optional sfxge inet pci dev/sfxge/common/efx_nvram.c optional sfxge inet pci dev/sfxge/common/efx_phy.c optional sfxge inet pci dev/sfxge/common/efx_port.c optional sfxge inet pci dev/sfxge/common/efx_rx.c optional sfxge inet pci dev/sfxge/common/efx_sram.c optional sfxge inet pci dev/sfxge/common/efx_tx.c optional sfxge inet pci dev/sfxge/common/efx_vpd.c optional sfxge inet pci dev/sfxge/common/efx_wol.c optional sfxge inet pci dev/sfxge/common/siena_mac.c optional sfxge inet pci dev/sfxge/common/siena_mon.c optional sfxge inet pci dev/sfxge/common/siena_nic.c optional sfxge inet pci dev/sfxge/common/siena_nvram.c optional sfxge inet pci dev/sfxge/common/siena_phy.c optional sfxge inet pci dev/sfxge/common/siena_sram.c optional sfxge inet pci dev/sfxge/common/siena_vpd.c optional sfxge inet pci dev/sfxge/sfxge.c optional sfxge inet pci dev/sfxge/sfxge_dma.c optional sfxge inet pci dev/sfxge/sfxge_ev.c optional sfxge inet pci dev/sfxge/sfxge_intr.c optional sfxge inet pci dev/sfxge/sfxge_mcdi.c optional sfxge inet pci dev/sfxge/sfxge_port.c optional sfxge inet pci dev/sfxge/sfxge_rx.c optional sfxge inet pci dev/sfxge/sfxge_tx.c optional sfxge inet pci dev/sio/sio.c optional sio dev/sio/sio_isa.c optional sio isa dev/sio/sio_pccard.c optional sio pccard dev/sio/sio_pci.c optional sio pci dev/sio/sio_puc.c optional sio puc dev/speaker/spkr.c optional speaker dev/syscons/apm/apm_saver.c optional apm_saver apm dev/syscons/scterm-teken.c optional sc dev/syscons/scvesactl.c optional sc vga vesa dev/syscons/scvgarndr.c optional sc vga dev/syscons/scvtb.c optional sc dev/tpm/tpm.c optional tpm dev/tpm/tpm_acpi.c optional tpm acpi dev/tpm/tpm_isa.c optional tpm isa dev/uart/uart_cpu_x86.c optional uart dev/viawd/viawd.c optional viawd dev/vmware/vmxnet3/if_vmx.c optional vmx dev/wbwd/wbwd.c optional wbwd dev/wpi/if_wpi.c optional wpi dev/xen/pci/xen_acpi_pci.c optional xenhvm dev/xen/pci/xen_pci.c optional xenhvm dev/isci/isci.c optional isci dev/isci/isci_controller.c optional isci dev/isci/isci_domain.c optional isci dev/isci/isci_interrupt.c optional isci dev/isci/isci_io_request.c optional isci dev/isci/isci_logger.c optional isci dev/isci/isci_oem_parameters.c optional isci dev/isci/isci_remote_device.c optional isci dev/isci/isci_sysctl.c optional isci dev/isci/isci_task_request.c optional isci dev/isci/isci_timer.c optional isci dev/isci/scil/sati.c optional isci dev/isci/scil/sati_abort_task_set.c optional isci dev/isci/scil/sati_atapi.c optional isci dev/isci/scil/sati_device.c optional isci dev/isci/scil/sati_inquiry.c optional isci dev/isci/scil/sati_log_sense.c optional isci dev/isci/scil/sati_lun_reset.c optional isci dev/isci/scil/sati_mode_pages.c optional isci dev/isci/scil/sati_mode_select.c optional isci dev/isci/scil/sati_mode_sense.c optional isci dev/isci/scil/sati_mode_sense_10.c optional isci dev/isci/scil/sati_mode_sense_6.c optional isci dev/isci/scil/sati_move.c optional isci dev/isci/scil/sati_passthrough.c optional isci dev/isci/scil/sati_read.c optional isci dev/isci/scil/sati_read_buffer.c optional isci dev/isci/scil/sati_read_capacity.c optional isci dev/isci/scil/sati_reassign_blocks.c optional isci dev/isci/scil/sati_report_luns.c optional isci dev/isci/scil/sati_request_sense.c optional isci dev/isci/scil/sati_start_stop_unit.c optional isci dev/isci/scil/sati_synchronize_cache.c optional isci dev/isci/scil/sati_test_unit_ready.c optional isci dev/isci/scil/sati_unmap.c optional isci dev/isci/scil/sati_util.c optional isci dev/isci/scil/sati_verify.c optional isci dev/isci/scil/sati_write.c optional isci dev/isci/scil/sati_write_and_verify.c optional isci dev/isci/scil/sati_write_buffer.c optional isci dev/isci/scil/sati_write_long.c optional isci dev/isci/scil/sci_abstract_list.c optional isci dev/isci/scil/sci_base_controller.c optional isci dev/isci/scil/sci_base_domain.c optional isci dev/isci/scil/sci_base_iterator.c optional isci dev/isci/scil/sci_base_library.c optional isci dev/isci/scil/sci_base_logger.c optional isci dev/isci/scil/sci_base_memory_descriptor_list.c optional isci dev/isci/scil/sci_base_memory_descriptor_list_decorator.c optional isci dev/isci/scil/sci_base_object.c optional isci dev/isci/scil/sci_base_observer.c optional isci dev/isci/scil/sci_base_phy.c optional isci dev/isci/scil/sci_base_port.c optional isci dev/isci/scil/sci_base_remote_device.c optional isci dev/isci/scil/sci_base_request.c optional isci dev/isci/scil/sci_base_state_machine.c optional isci dev/isci/scil/sci_base_state_machine_logger.c optional isci dev/isci/scil/sci_base_state_machine_observer.c optional isci dev/isci/scil/sci_base_subject.c optional isci dev/isci/scil/sci_util.c optional isci dev/isci/scil/scic_sds_controller.c optional isci dev/isci/scil/scic_sds_library.c optional isci dev/isci/scil/scic_sds_pci.c optional isci dev/isci/scil/scic_sds_phy.c optional isci dev/isci/scil/scic_sds_port.c optional isci dev/isci/scil/scic_sds_port_configuration_agent.c optional isci dev/isci/scil/scic_sds_remote_device.c optional isci dev/isci/scil/scic_sds_remote_node_context.c optional isci dev/isci/scil/scic_sds_remote_node_table.c optional isci dev/isci/scil/scic_sds_request.c optional isci dev/isci/scil/scic_sds_sgpio.c optional isci dev/isci/scil/scic_sds_smp_remote_device.c optional isci dev/isci/scil/scic_sds_smp_request.c optional isci dev/isci/scil/scic_sds_ssp_request.c optional isci dev/isci/scil/scic_sds_stp_packet_request.c optional isci dev/isci/scil/scic_sds_stp_remote_device.c optional isci dev/isci/scil/scic_sds_stp_request.c optional isci dev/isci/scil/scic_sds_unsolicited_frame_control.c optional isci dev/isci/scil/scif_sas_controller.c optional isci dev/isci/scil/scif_sas_controller_state_handlers.c optional isci dev/isci/scil/scif_sas_controller_states.c optional isci dev/isci/scil/scif_sas_domain.c optional isci dev/isci/scil/scif_sas_domain_state_handlers.c optional isci dev/isci/scil/scif_sas_domain_states.c optional isci dev/isci/scil/scif_sas_high_priority_request_queue.c optional isci dev/isci/scil/scif_sas_internal_io_request.c optional isci dev/isci/scil/scif_sas_io_request.c optional isci dev/isci/scil/scif_sas_io_request_state_handlers.c optional isci dev/isci/scil/scif_sas_io_request_states.c optional isci dev/isci/scil/scif_sas_library.c optional isci dev/isci/scil/scif_sas_remote_device.c optional isci dev/isci/scil/scif_sas_remote_device_ready_substate_handlers.c optional isci dev/isci/scil/scif_sas_remote_device_ready_substates.c optional isci dev/isci/scil/scif_sas_remote_device_starting_substate_handlers.c optional isci dev/isci/scil/scif_sas_remote_device_starting_substates.c optional isci dev/isci/scil/scif_sas_remote_device_state_handlers.c optional isci dev/isci/scil/scif_sas_remote_device_states.c optional isci dev/isci/scil/scif_sas_request.c optional isci dev/isci/scil/scif_sas_smp_activity_clear_affiliation.c optional isci dev/isci/scil/scif_sas_smp_io_request.c optional isci dev/isci/scil/scif_sas_smp_phy.c optional isci dev/isci/scil/scif_sas_smp_remote_device.c optional isci dev/isci/scil/scif_sas_stp_io_request.c optional isci dev/isci/scil/scif_sas_stp_remote_device.c optional isci dev/isci/scil/scif_sas_stp_task_request.c optional isci dev/isci/scil/scif_sas_task_request.c optional isci dev/isci/scil/scif_sas_task_request_state_handlers.c optional isci dev/isci/scil/scif_sas_task_request_states.c optional isci dev/isci/scil/scif_sas_timer.c optional isci isa/syscons_isa.c optional sc isa/vga_isa.c optional vga kern/kern_clocksource.c standard kern/link_elf_obj.c standard # # IA32 binary support # #amd64/ia32/ia32_exception.S optional compat_freebsd32 amd64/ia32/ia32_reg.c optional compat_freebsd32 amd64/ia32/ia32_signal.c optional compat_freebsd32 amd64/ia32/ia32_sigtramp.S optional compat_freebsd32 amd64/ia32/ia32_syscall.c optional compat_freebsd32 amd64/ia32/ia32_misc.c optional compat_freebsd32 compat/ia32/ia32_sysvec.c optional compat_freebsd32 compat/linprocfs/linprocfs.c optional linprocfs compat/linsysfs/linsysfs.c optional linsysfs # # Linux/i386 binary support # amd64/linux32/linux32_dummy.c optional compat_linux32 amd64/linux32/linux32_locore.s optional compat_linux32 \ dependency "linux32_assym.h" amd64/linux32/linux32_machdep.c optional compat_linux32 amd64/linux32/linux32_support.s optional compat_linux32 \ dependency "linux32_assym.h" amd64/linux32/linux32_sysent.c optional compat_linux32 amd64/linux32/linux32_sysvec.c optional compat_linux32 compat/linux/linux_emul.c optional compat_linux32 compat/linux/linux_file.c optional compat_linux32 compat/linux/linux_fork.c optional compat_linux32 compat/linux/linux_futex.c optional compat_linux32 compat/linux/linux_getcwd.c optional compat_linux32 compat/linux/linux_ioctl.c optional compat_linux32 compat/linux/linux_ipc.c optional compat_linux32 compat/linux/linux_mib.c optional compat_linux32 compat/linux/linux_misc.c optional compat_linux32 compat/linux/linux_signal.c optional compat_linux32 compat/linux/linux_socket.c optional compat_linux32 compat/linux/linux_stats.c optional compat_linux32 compat/linux/linux_sysctl.c optional compat_linux32 compat/linux/linux_time.c optional compat_linux32 compat/linux/linux_timer.c optional compat_linux32 compat/linux/linux_uid16.c optional compat_linux32 compat/linux/linux_util.c optional compat_linux32 dev/amr/amr_linux.c optional compat_linux32 amr dev/mfi/mfi_linux.c optional compat_linux32 mfi # # Windows NDIS driver support # compat/ndis/kern_ndis.c optional ndisapi pci compat/ndis/kern_windrv.c optional ndisapi pci compat/ndis/subr_hal.c optional ndisapi pci compat/ndis/subr_ndis.c optional ndisapi pci compat/ndis/subr_ntoskrnl.c optional ndisapi pci compat/ndis/subr_pe.c optional ndisapi pci compat/ndis/subr_usbd.c optional ndisapi pci compat/ndis/winx64_wrap.S optional ndisapi pci # libkern/memmove.c standard libkern/memset.c standard # # x86 real mode BIOS emulator, required by atkbdc/dpms/vesa # compat/x86bios/x86bios.c optional x86bios | atkbd | dpms | vesa contrib/x86emu/x86emu.c optional x86bios | atkbd | dpms | vesa # # bvm console # dev/bvm/bvm_console.c optional bvmconsole dev/bvm/bvm_dbg.c optional bvmdebug # # x86 shared code between IA32, AMD64 and PC98 architectures # x86/acpica/OsdEnvironment.c optional acpi x86/acpica/acpi_apm.c optional acpi x86/acpica/acpi_wakeup.c optional acpi x86/acpica/madt.c optional acpi x86/acpica/srat.c optional acpi x86/bios/smbios.c optional smbios x86/bios/vpd.c optional vpd x86/cpufreq/powernow.c optional cpufreq x86/cpufreq/est.c optional cpufreq x86/cpufreq/hwpstate.c optional cpufreq x86/cpufreq/p4tcc.c optional cpufreq x86/iommu/busdma_dmar.c optional acpi acpi_dmar pci x86/iommu/intel_ctx.c optional acpi acpi_dmar pci x86/iommu/intel_drv.c optional acpi acpi_dmar pci x86/iommu/intel_fault.c optional acpi acpi_dmar pci x86/iommu/intel_gas.c optional acpi acpi_dmar pci x86/iommu/intel_idpgtbl.c optional acpi acpi_dmar pci x86/iommu/intel_qi.c optional acpi acpi_dmar pci x86/iommu/intel_quirks.c optional acpi acpi_dmar pci x86/iommu/intel_utils.c optional acpi acpi_dmar pci x86/isa/atpic.c optional atpic isa x86/isa/atrtc.c standard x86/isa/clock.c standard x86/isa/elcr.c optional atpic isa | mptable x86/isa/isa.c standard x86/isa/isa_dma.c standard x86/isa/nmi.c standard x86/isa/orm.c optional isa x86/pci/pci_bus.c optional pci x86/pci/qpi.c optional pci x86/x86/busdma_bounce.c standard x86/x86/busdma_machdep.c standard x86/x86/dump_machdep.c standard x86/x86/fdt_machdep.c optional fdt x86/x86/identcpu.c standard x86/x86/intr_machdep.c standard x86/x86/io_apic.c standard x86/x86/legacy.c standard x86/x86/local_apic.c standard x86/x86/mca.c standard x86/x86/mptable.c optional mptable x86/x86/mptable_pci.c optional mptable pci x86/x86/msi.c optional pci x86/x86/nexus.c standard x86/x86/tsc.c standard x86/x86/delay.c standard x86/xen/hvm.c optional xenhvm x86/xen/xen_intr.c optional xen | xenhvm x86/xen/pv.c optional xenhvm x86/xen/pvcpu_enum.c optional xenhvm x86/xen/xen_apic.c optional xenhvm x86/xen/xenpv.c optional xenhvm x86/xen/xen_nexus.c optional xenhvm x86/xen/xen_msi.c optional xenhvm x86/xen/xen_pci_bus.c optional xenhvm Index: projects/clang350-import/sys/conf/files.i386 =================================================================== --- projects/clang350-import/sys/conf/files.i386 (revision 275748) +++ projects/clang350-import/sys/conf/files.i386 (revision 275749) @@ -1,592 +1,597 @@ # This file tells config what files go into building a kernel, # files marked standard are always included. # # $FreeBSD$ # # The long compile-with and dependency lines are required because of # limitations in config: backslash-newline doesn't work in strings, and # dependency lines other than the first are silently ignored. # linux_genassym.o optional compat_linux \ dependency "$S/i386/linux/linux_genassym.c" \ compile-with "${CC} ${CFLAGS:N-fno-common} -c ${.IMPSRC}" \ no-obj no-implicit-rule \ clean "linux_genassym.o" # linux_assym.h optional compat_linux \ dependency "$S/kern/genassym.sh linux_genassym.o" \ compile-with "sh $S/kern/genassym.sh linux_genassym.o > ${.TARGET}" \ no-obj no-implicit-rule before-depend \ clean "linux_assym.h" # svr4_genassym.o optional compat_svr4 \ dependency "$S/i386/svr4/svr4_genassym.c" \ compile-with "${CC} ${CFLAGS:N-fno-common} -c ${.IMPSRC}" \ no-obj no-implicit-rule \ clean "svr4_genassym.o" # svr4_assym.h optional compat_svr4 \ dependency "$S/kern/genassym.sh svr4_genassym.o" \ compile-with "sh $S/kern/genassym.sh svr4_genassym.o > ${.TARGET}" \ no-obj no-implicit-rule before-depend \ clean "svr4_assym.h" # font.h optional sc_dflt_font \ compile-with "uudecode < /usr/share/syscons/fonts/${SC_DFLT_FONT}-8x16.fnt && file2c 'static u_char dflt_font_16[16*256] = {' '};' < ${SC_DFLT_FONT}-8x16 > font.h && uudecode < /usr/share/syscons/fonts/${SC_DFLT_FONT}-8x14.fnt && file2c 'static u_char dflt_font_14[14*256] = {' '};' < ${SC_DFLT_FONT}-8x14 >> font.h && uudecode < /usr/share/syscons/fonts/${SC_DFLT_FONT}-8x8.fnt && file2c 'static u_char dflt_font_8[8*256] = {' '};' < ${SC_DFLT_FONT}-8x8 >> font.h" \ no-obj no-implicit-rule before-depend \ clean "font.h ${SC_DFLT_FONT}-8x14 ${SC_DFLT_FONT}-8x16 ${SC_DFLT_FONT}-8x8" # atkbdmap.h optional atkbd_dflt_keymap \ compile-with "/usr/sbin/kbdcontrol -L ${ATKBD_DFLT_KEYMAP} | sed -e 's/^static keymap_t.* = /static keymap_t key_map = /' -e 's/^static accentmap_t.* = /static accentmap_t accent_map = /' > atkbdmap.h" \ no-obj no-implicit-rule before-depend \ clean "atkbdmap.h" # ukbdmap.h optional ukbd_dflt_keymap \ compile-with "/usr/sbin/kbdcontrol -L ${UKBD_DFLT_KEYMAP} | sed -e 's/^static keymap_t.* = /static keymap_t key_map = /' -e 's/^static accentmap_t.* = /static accentmap_t accent_map = /' > ukbdmap.h" \ no-obj no-implicit-rule before-depend \ clean "ukbdmap.h" # hpt27xx_lib.o optional hpt27xx \ dependency "$S/dev/hpt27xx/i386-elf.hpt27xx_lib.o.uu" \ compile-with "uudecode < $S/dev/hpt27xx/i386-elf.hpt27xx_lib.o.uu" \ no-implicit-rule # hptmvraid.o optional hptmv \ dependency "$S/dev/hptmv/i386-elf.raid.o.uu" \ compile-with "uudecode < $S/dev/hptmv/i386-elf.raid.o.uu" \ no-implicit-rule # hptnr_lib.o optional hptnr \ dependency "$S/dev/hptnr/i386-elf.hptnr_lib.o.uu" \ compile-with "uudecode < $S/dev/hptnr/i386-elf.hptnr_lib.o.uu" \ no-implicit-rule # hptrr_lib.o optional hptrr \ dependency "$S/dev/hptrr/i386-elf.hptrr_lib.o.uu" \ compile-with "uudecode < $S/dev/hptrr/i386-elf.hptrr_lib.o.uu" \ no-implicit-rule # cddl/contrib/opensolaris/common/atomic/i386/opensolaris_atomic.S optional zfs compile-with "${ZFS_S}" compat/linprocfs/linprocfs.c optional linprocfs compat/linsysfs/linsysfs.c optional linsysfs compat/linux/linux_emul.c optional compat_linux compat/linux/linux_file.c optional compat_linux compat/linux/linux_fork.c optional compat_linux compat/linux/linux_futex.c optional compat_linux compat/linux/linux_getcwd.c optional compat_linux compat/linux/linux_ioctl.c optional compat_linux compat/linux/linux_ipc.c optional compat_linux compat/linux/linux_mib.c optional compat_linux compat/linux/linux_misc.c optional compat_linux compat/linux/linux_signal.c optional compat_linux compat/linux/linux_socket.c optional compat_linux compat/linux/linux_stats.c optional compat_linux compat/linux/linux_sysctl.c optional compat_linux compat/linux/linux_time.c optional compat_linux compat/linux/linux_timer.c optional compat_linux compat/linux/linux_uid16.c optional compat_linux compat/linux/linux_util.c optional compat_linux compat/ndis/kern_ndis.c optional ndisapi pci compat/ndis/kern_windrv.c optional ndisapi pci compat/ndis/subr_hal.c optional ndisapi pci compat/ndis/subr_ndis.c optional ndisapi pci compat/ndis/subr_ntoskrnl.c optional ndisapi pci compat/ndis/subr_pe.c optional ndisapi pci compat/ndis/subr_usbd.c optional ndisapi pci compat/ndis/winx32_wrap.S optional ndisapi pci compat/svr4/imgact_svr4.c optional compat_svr4 compat/svr4/svr4_fcntl.c optional compat_svr4 compat/svr4/svr4_filio.c optional compat_svr4 compat/svr4/svr4_ioctl.c optional compat_svr4 compat/svr4/svr4_ipc.c optional compat_svr4 compat/svr4/svr4_misc.c optional compat_svr4 compat/svr4/svr4_resource.c optional compat_svr4 compat/svr4/svr4_signal.c optional compat_svr4 compat/svr4/svr4_socket.c optional compat_svr4 compat/svr4/svr4_sockio.c optional compat_svr4 compat/svr4/svr4_stat.c optional compat_svr4 compat/svr4/svr4_stream.c optional compat_svr4 compat/svr4/svr4_syscallnames.c optional compat_svr4 compat/svr4/svr4_sysent.c optional compat_svr4 compat/svr4/svr4_sysvec.c optional compat_svr4 compat/svr4/svr4_termios.c optional compat_svr4 bf_enc.o optional crypto | ipsec \ dependency "$S/crypto/blowfish/arch/i386/bf_enc.S $S/crypto/blowfish/arch/i386/bf_enc_586.S $S/crypto/blowfish/arch/i386/bf_enc_686.S" \ compile-with "${CC} -c -I$S/crypto/blowfish/arch/i386 ${ASM_CFLAGS} ${WERROR} ${.IMPSRC}" \ no-implicit-rule crypto/aesni/aeskeys_i386.S optional aesni crypto/aesni/aesni.c optional aesni +aesni_ghash.o optional aesni \ + dependency "$S/crypto/aesni/aesni_ghash.c" \ + compile-with "${CC} -c ${CFLAGS:C/^-O2$/-O3/:N-nostdinc} ${WERROR} ${PROF} -mmmx -msse -msse4 -maes -mpclmul ${.IMPSRC}" \ + no-implicit-rule \ + clean "aesni_ghash.o" aesni_wrap.o optional aesni \ dependency "$S/crypto/aesni/aesni_wrap.c" \ - compile-with "${CC} -c ${CFLAGS:C/^-O2$/-O3/:N-nostdinc} ${WERROR} ${PROF} -mmmx -msse -maes ${.IMPSRC}" \ + compile-with "${CC} -c ${CFLAGS:C/^-O2$/-O3/:N-nostdinc} ${WERROR} ${PROF} -mmmx -msse -msse4 -maes ${.IMPSRC}" \ no-implicit-rule \ clean "aesni_wrap.o" crypto/des/arch/i386/des_enc.S optional crypto | ipsec | netsmb crypto/via/padlock.c optional padlock crypto/via/padlock_cipher.c optional padlock crypto/via/padlock_hash.c optional padlock dev/advansys/adv_isa.c optional adv isa dev/agp/agp_ali.c optional agp dev/agp/agp_amd.c optional agp dev/agp/agp_amd64.c optional agp dev/agp/agp_ati.c optional agp dev/agp/agp_i810.c optional agp dev/agp/agp_intel.c optional agp dev/agp/agp_nvidia.c optional agp dev/agp/agp_sis.c optional agp dev/agp/agp_via.c optional agp dev/aic/aic_isa.c optional aic isa dev/amdsbwd/amdsbwd.c optional amdsbwd dev/amdtemp/amdtemp.c optional amdtemp dev/arcmsr/arcmsr.c optional arcmsr pci dev/asmc/asmc.c optional asmc isa dev/atkbdc/atkbd.c optional atkbd atkbdc dev/atkbdc/atkbd_atkbdc.c optional atkbd atkbdc dev/atkbdc/atkbdc.c optional atkbdc dev/atkbdc/atkbdc_isa.c optional atkbdc isa dev/atkbdc/atkbdc_subr.c optional atkbdc dev/atkbdc/psm.c optional psm atkbdc dev/bxe/bxe.c optional bxe pci dev/bxe/bxe_stats.c optional bxe pci dev/bxe/bxe_debug.c optional bxe pci dev/bxe/ecore_sp.c optional bxe pci dev/bxe/bxe_elink.c optional bxe pci dev/bxe/57710_init_values.c optional bxe pci dev/bxe/57711_init_values.c optional bxe pci dev/bxe/57712_init_values.c optional bxe pci dev/ce/ceddk.c optional ce dev/ce/if_ce.c optional ce dev/ce/tau32-ddk.c optional ce \ compile-with "${NORMAL_C} ${NO_WCONSTANT_CONVERSION}" dev/cm/if_cm_isa.c optional cm isa dev/coretemp/coretemp.c optional coretemp dev/cp/cpddk.c optional cp dev/cp/if_cp.c optional cp dev/cpuctl/cpuctl.c optional cpuctl dev/ctau/ctau.c optional ctau dev/ctau/ctddk.c optional ctau dev/ctau/if_ct.c optional ctau dev/cx/csigma.c optional cx dev/cx/cxddk.c optional cx dev/cx/if_cx.c optional cx dev/dpms/dpms.c optional dpms dev/ed/if_ed_3c503.c optional ed isa ed_3c503 dev/ed/if_ed_isa.c optional ed isa dev/ed/if_ed_wd80x3.c optional ed isa dev/ed/if_ed_hpp.c optional ed isa ed_hpp dev/ed/if_ed_sic.c optional ed isa ed_sic dev/fb/fb.c optional fb | vga dev/fb/s3_pci.c optional s3pci dev/fb/vesa.c optional vga vesa dev/fb/vga.c optional vga dev/fdc/fdc.c optional fdc dev/fdc/fdc_acpi.c optional fdc dev/fdc/fdc_isa.c optional fdc isa dev/fdc/fdc_pccard.c optional fdc pccard dev/fdt/fdt_x86.c optional fdt dev/fe/if_fe_isa.c optional fe isa dev/glxiic/glxiic.c optional glxiic dev/glxsb/glxsb.c optional glxsb dev/glxsb/glxsb_hash.c optional glxsb dev/hpt27xx/hpt27xx_os_bsd.c optional hpt27xx dev/hpt27xx/hpt27xx_osm_bsd.c optional hpt27xx dev/hpt27xx/hpt27xx_config.c optional hpt27xx dev/hptmv/entry.c optional hptmv dev/hptmv/mv.c optional hptmv dev/hptmv/gui_lib.c optional hptmv dev/hptmv/hptproc.c optional hptmv dev/hptmv/ioctl.c optional hptmv dev/hptnr/hptnr_os_bsd.c optional hptnr dev/hptnr/hptnr_osm_bsd.c optional hptnr dev/hptnr/hptnr_config.c optional hptnr dev/hptrr/hptrr_os_bsd.c optional hptrr dev/hptrr/hptrr_osm_bsd.c optional hptrr dev/hptrr/hptrr_config.c optional hptrr dev/hwpmc/hwpmc_amd.c optional hwpmc dev/hwpmc/hwpmc_intel.c optional hwpmc dev/hwpmc/hwpmc_core.c optional hwpmc dev/hwpmc/hwpmc_uncore.c optional hwpmc dev/hwpmc/hwpmc_pentium.c optional hwpmc dev/hwpmc/hwpmc_piv.c optional hwpmc dev/hwpmc/hwpmc_ppro.c optional hwpmc dev/hwpmc/hwpmc_tsc.c optional hwpmc dev/hwpmc/hwpmc_x86.c optional hwpmc dev/hyperv/netvsc/hv_net_vsc.c optional hyperv dev/hyperv/netvsc/hv_netvsc_drv_freebsd.c optional hyperv dev/hyperv/netvsc/hv_rndis_filter.c optional hyperv dev/hyperv/stordisengage/hv_ata_pci_disengage.c optional hyperv dev/hyperv/storvsc/hv_storvsc_drv_freebsd.c optional hyperv dev/hyperv/utilities/hv_kvp.c optional hyperv dev/hyperv/utilities/hv_util.c optional hyperv dev/hyperv/vmbus/hv_channel.c optional hyperv dev/hyperv/vmbus/hv_channel_mgmt.c optional hyperv dev/hyperv/vmbus/hv_connection.c optional hyperv dev/hyperv/vmbus/hv_hv.c optional hyperv dev/hyperv/vmbus/hv_ring_buffer.c optional hyperv dev/hyperv/vmbus/hv_vmbus_drv_freebsd.c optional hyperv dev/ichwd/ichwd.c optional ichwd dev/if_ndis/if_ndis.c optional ndis dev/if_ndis/if_ndis_pccard.c optional ndis pccard dev/if_ndis/if_ndis_pci.c optional ndis cardbus | ndis pci dev/if_ndis/if_ndis_usb.c optional ndis usb dev/io/iodev.c optional io dev/ipmi/ipmi.c optional ipmi dev/ipmi/ipmi_acpi.c optional ipmi acpi dev/ipmi/ipmi_isa.c optional ipmi isa dev/ipmi/ipmi_kcs.c optional ipmi dev/ipmi/ipmi_smic.c optional ipmi dev/ipmi/ipmi_smbus.c optional ipmi smbus dev/ipmi/ipmi_smbios.c optional ipmi dev/ipmi/ipmi_ssif.c optional ipmi smbus dev/ipmi/ipmi_pci.c optional ipmi pci dev/ipmi/ipmi_linux.c optional ipmi compat_linux dev/kbd/kbd.c optional atkbd | sc | ukbd | vt dev/le/if_le_isa.c optional le isa dev/mse/mse.c optional mse dev/mse/mse_isa.c optional mse isa dev/nfe/if_nfe.c optional nfe pci dev/nvd/nvd.c optional nvd nvme dev/nvme/nvme.c optional nvme dev/nvme/nvme_ctrlr.c optional nvme dev/nvme/nvme_ctrlr_cmd.c optional nvme dev/nvme/nvme_ns.c optional nvme dev/nvme/nvme_ns_cmd.c optional nvme dev/nvme/nvme_qpair.c optional nvme dev/nvme/nvme_sysctl.c optional nvme dev/nvme/nvme_test.c optional nvme dev/nvme/nvme_util.c optional nvme dev/nvram/nvram.c optional nvram isa dev/pcf/pcf_isa.c optional pcf dev/random/ivy.c optional rdrand_rng dev/random/nehemiah.c optional padlock_rng dev/sbni/if_sbni.c optional sbni dev/sbni/if_sbni_isa.c optional sbni isa dev/sbni/if_sbni_pci.c optional sbni pci dev/sio/sio.c optional sio dev/sio/sio_isa.c optional sio isa dev/sio/sio_pccard.c optional sio pccard dev/sio/sio_pci.c optional sio pci dev/sio/sio_puc.c optional sio puc dev/speaker/spkr.c optional speaker dev/syscons/apm/apm_saver.c optional apm_saver apm dev/syscons/scterm-teken.c optional sc dev/syscons/scvesactl.c optional sc vga vesa dev/syscons/scvgarndr.c optional sc vga dev/syscons/scvtb.c optional sc dev/tpm/tpm.c optional tpm dev/tpm/tpm_acpi.c optional tpm acpi dev/tpm/tpm_isa.c optional tpm isa dev/uart/uart_cpu_x86.c optional uart dev/viawd/viawd.c optional viawd dev/vmware/vmxnet3/if_vmx.c optional vmx dev/acpica/acpi_if.m standard dev/acpi_support/acpi_wmi_if.m standard dev/wbwd/wbwd.c optional wbwd dev/wpi/if_wpi.c optional wpi dev/isci/isci.c optional isci dev/isci/isci_controller.c optional isci dev/isci/isci_domain.c optional isci dev/isci/isci_interrupt.c optional isci dev/isci/isci_io_request.c optional isci dev/isci/isci_logger.c optional isci dev/isci/isci_oem_parameters.c optional isci dev/isci/isci_remote_device.c optional isci dev/isci/isci_sysctl.c optional isci dev/isci/isci_task_request.c optional isci dev/isci/isci_timer.c optional isci dev/isci/scil/sati.c optional isci dev/isci/scil/sati_abort_task_set.c optional isci dev/isci/scil/sati_atapi.c optional isci dev/isci/scil/sati_device.c optional isci dev/isci/scil/sati_inquiry.c optional isci dev/isci/scil/sati_log_sense.c optional isci dev/isci/scil/sati_lun_reset.c optional isci dev/isci/scil/sati_mode_pages.c optional isci dev/isci/scil/sati_mode_select.c optional isci dev/isci/scil/sati_mode_sense.c optional isci dev/isci/scil/sati_mode_sense_10.c optional isci dev/isci/scil/sati_mode_sense_6.c optional isci dev/isci/scil/sati_move.c optional isci dev/isci/scil/sati_passthrough.c optional isci dev/isci/scil/sati_read.c optional isci dev/isci/scil/sati_read_buffer.c optional isci dev/isci/scil/sati_read_capacity.c optional isci dev/isci/scil/sati_reassign_blocks.c optional isci dev/isci/scil/sati_report_luns.c optional isci dev/isci/scil/sati_request_sense.c optional isci dev/isci/scil/sati_start_stop_unit.c optional isci dev/isci/scil/sati_synchronize_cache.c optional isci dev/isci/scil/sati_test_unit_ready.c optional isci dev/isci/scil/sati_unmap.c optional isci dev/isci/scil/sati_util.c optional isci dev/isci/scil/sati_verify.c optional isci dev/isci/scil/sati_write.c optional isci dev/isci/scil/sati_write_and_verify.c optional isci dev/isci/scil/sati_write_buffer.c optional isci dev/isci/scil/sati_write_long.c optional isci dev/isci/scil/sci_abstract_list.c optional isci dev/isci/scil/sci_base_controller.c optional isci dev/isci/scil/sci_base_domain.c optional isci dev/isci/scil/sci_base_iterator.c optional isci dev/isci/scil/sci_base_library.c optional isci dev/isci/scil/sci_base_logger.c optional isci dev/isci/scil/sci_base_memory_descriptor_list.c optional isci dev/isci/scil/sci_base_memory_descriptor_list_decorator.c optional isci dev/isci/scil/sci_base_object.c optional isci dev/isci/scil/sci_base_observer.c optional isci dev/isci/scil/sci_base_phy.c optional isci dev/isci/scil/sci_base_port.c optional isci dev/isci/scil/sci_base_remote_device.c optional isci dev/isci/scil/sci_base_request.c optional isci dev/isci/scil/sci_base_state_machine.c optional isci dev/isci/scil/sci_base_state_machine_logger.c optional isci dev/isci/scil/sci_base_state_machine_observer.c optional isci dev/isci/scil/sci_base_subject.c optional isci dev/isci/scil/sci_util.c optional isci dev/isci/scil/scic_sds_controller.c optional isci dev/isci/scil/scic_sds_library.c optional isci dev/isci/scil/scic_sds_pci.c optional isci dev/isci/scil/scic_sds_phy.c optional isci dev/isci/scil/scic_sds_port.c optional isci dev/isci/scil/scic_sds_port_configuration_agent.c optional isci dev/isci/scil/scic_sds_remote_device.c optional isci dev/isci/scil/scic_sds_remote_node_context.c optional isci dev/isci/scil/scic_sds_remote_node_table.c optional isci dev/isci/scil/scic_sds_request.c optional isci dev/isci/scil/scic_sds_sgpio.c optional isci dev/isci/scil/scic_sds_smp_remote_device.c optional isci dev/isci/scil/scic_sds_smp_request.c optional isci dev/isci/scil/scic_sds_ssp_request.c optional isci dev/isci/scil/scic_sds_stp_packet_request.c optional isci dev/isci/scil/scic_sds_stp_remote_device.c optional isci dev/isci/scil/scic_sds_stp_request.c optional isci dev/isci/scil/scic_sds_unsolicited_frame_control.c optional isci dev/isci/scil/scif_sas_controller.c optional isci dev/isci/scil/scif_sas_controller_state_handlers.c optional isci dev/isci/scil/scif_sas_controller_states.c optional isci dev/isci/scil/scif_sas_domain.c optional isci dev/isci/scil/scif_sas_domain_state_handlers.c optional isci dev/isci/scil/scif_sas_domain_states.c optional isci dev/isci/scil/scif_sas_high_priority_request_queue.c optional isci dev/isci/scil/scif_sas_internal_io_request.c optional isci dev/isci/scil/scif_sas_io_request.c optional isci dev/isci/scil/scif_sas_io_request_state_handlers.c optional isci dev/isci/scil/scif_sas_io_request_states.c optional isci dev/isci/scil/scif_sas_library.c optional isci dev/isci/scil/scif_sas_remote_device.c optional isci dev/isci/scil/scif_sas_remote_device_ready_substate_handlers.c optional isci dev/isci/scil/scif_sas_remote_device_ready_substates.c optional isci dev/isci/scil/scif_sas_remote_device_starting_substate_handlers.c optional isci dev/isci/scil/scif_sas_remote_device_starting_substates.c optional isci dev/isci/scil/scif_sas_remote_device_state_handlers.c optional isci dev/isci/scil/scif_sas_remote_device_states.c optional isci dev/isci/scil/scif_sas_request.c optional isci dev/isci/scil/scif_sas_smp_activity_clear_affiliation.c optional isci dev/isci/scil/scif_sas_smp_io_request.c optional isci dev/isci/scil/scif_sas_smp_phy.c optional isci dev/isci/scil/scif_sas_smp_remote_device.c optional isci dev/isci/scil/scif_sas_stp_io_request.c optional isci dev/isci/scil/scif_sas_stp_remote_device.c optional isci dev/isci/scil/scif_sas_stp_task_request.c optional isci dev/isci/scil/scif_sas_task_request.c optional isci dev/isci/scil/scif_sas_task_request_state_handlers.c optional isci dev/isci/scil/scif_sas_task_request_states.c optional isci dev/isci/scil/scif_sas_timer.c optional isci i386/acpica/acpi_machdep.c optional acpi acpi_wakecode.o optional acpi \ dependency "$S/i386/acpica/acpi_wakecode.S assym.s" \ compile-with "${NORMAL_S}" \ no-obj no-implicit-rule before-depend \ clean "acpi_wakecode.o" acpi_wakecode.bin optional acpi \ dependency "acpi_wakecode.o" \ compile-with "${OBJCOPY} -S -O binary acpi_wakecode.o ${.TARGET}" \ no-obj no-implicit-rule before-depend \ clean "acpi_wakecode.bin" acpi_wakecode.h optional acpi \ dependency "acpi_wakecode.bin" \ compile-with "file2c -sx 'static char wakecode[] = {' '};' < acpi_wakecode.bin > ${.TARGET}" \ no-obj no-implicit-rule before-depend \ clean "acpi_wakecode.h" acpi_wakedata.h optional acpi \ dependency "acpi_wakecode.o" \ compile-with '${NM} -n --defined-only acpi_wakecode.o | while read offset dummy what; do echo "#define $${what} 0x$${offset}"; done > ${.TARGET}' \ no-obj no-implicit-rule before-depend \ clean "acpi_wakedata.h" # i386/bios/apm.c optional apm i386/bios/mca_machdep.c optional mca i386/bios/smapi.c optional smapi i386/bios/smapi_bios.S optional smapi #i386/i386/apic_vector.s optional apic i386/i386/atomic.c standard \ compile-with "${CC} -c ${CFLAGS} ${DEFINED_PROF:S/^$/-fomit-frame-pointer/} ${.IMPSRC}" i386/i386/autoconf.c standard i386/i386/bios.c optional native i386/i386/bioscall.s optional native i386/i386/bpf_jit_machdep.c optional bpf_jitter i386/i386/db_disasm.c optional ddb i386/i386/db_interface.c optional ddb i386/i386/db_trace.c optional ddb i386/i386/elan-mmcr.c optional cpu_elan | cpu_soekris i386/i386/elf_machdep.c standard i386/i386/exception.s optional native i386/xen/exception.s optional xen i386/i386/gdb_machdep.c optional gdb i386/i386/geode.c optional cpu_geode i386/i386/i686_mem.c optional mem i386/i386/in_cksum.c optional inet | inet6 i386/i386/initcpu.c standard i386/i386/io.c optional io i386/i386/k6_mem.c optional mem i386/i386/locore.s optional native no-obj i386/xen/locore.s optional xen no-obj i386/i386/longrun.c optional cpu_enable_longrun i386/i386/machdep.c standard i386/xen/xen_machdep.c optional xen i386/i386/mem.c optional mem i386/i386/minidump_machdep.c standard i386/i386/mp_clock.c optional smp i386/i386/mp_machdep.c optional native smp i386/xen/mp_machdep.c optional xen smp i386/i386/mp_watchdog.c optional mp_watchdog smp i386/i386/mpboot.s optional smp native i386/xen/mptable.c optional apic xen i386/i386/perfmon.c optional perfmon i386/i386/pmap.c optional native i386/xen/pmap.c optional xen i386/i386/ptrace_machdep.c standard i386/i386/stack_machdep.c optional ddb | stack i386/i386/support.s standard i386/i386/swtch.s standard i386/i386/sys_machdep.c standard i386/i386/trap.c standard i386/i386/uio_machdep.c standard i386/i386/vm86.c standard i386/i386/vm_machdep.c standard i386/ibcs2/ibcs2_errno.c optional ibcs2 i386/ibcs2/ibcs2_fcntl.c optional ibcs2 i386/ibcs2/ibcs2_ioctl.c optional ibcs2 i386/ibcs2/ibcs2_ipc.c optional ibcs2 i386/ibcs2/ibcs2_isc.c optional ibcs2 i386/ibcs2/ibcs2_isc_sysent.c optional ibcs2 i386/ibcs2/ibcs2_misc.c optional ibcs2 i386/ibcs2/ibcs2_msg.c optional ibcs2 i386/ibcs2/ibcs2_other.c optional ibcs2 i386/ibcs2/ibcs2_signal.c optional ibcs2 i386/ibcs2/ibcs2_socksys.c optional ibcs2 i386/ibcs2/ibcs2_stat.c optional ibcs2 i386/ibcs2/ibcs2_sysent.c optional ibcs2 i386/ibcs2/ibcs2_sysi86.c optional ibcs2 i386/ibcs2/ibcs2_sysvec.c optional ibcs2 i386/ibcs2/ibcs2_util.c optional ibcs2 i386/ibcs2/ibcs2_xenix.c optional ibcs2 i386/ibcs2/ibcs2_xenix_sysent.c optional ibcs2 i386/ibcs2/imgact_coff.c optional ibcs2 i386/xen/clock.c optional xen i386/isa/elink.c optional ep | ie i386/isa/npx.c optional npx i386/isa/pmtimer.c optional pmtimer i386/isa/prof_machdep.c optional profiling-routine i386/isa/spic.c optional spic i386/linux/imgact_linux.c optional compat_linux i386/linux/linux_dummy.c optional compat_linux i386/linux/linux_locore.s optional compat_linux \ dependency "linux_assym.h" i386/linux/linux_machdep.c optional compat_linux i386/linux/linux_ptrace.c optional compat_linux i386/linux/linux_support.s optional compat_linux \ dependency "linux_assym.h" i386/linux/linux_sysent.c optional compat_linux i386/linux/linux_sysvec.c optional compat_linux i386/pci/pci_cfgreg.c optional pci i386/pci/pci_pir.c optional pci i386/svr4/svr4_locore.s optional compat_svr4 \ dependency "svr4_assym.h" \ warning "COMPAT_SVR4 is broken and should be avoided" i386/svr4/svr4_machdep.c optional compat_svr4 # isa/syscons_isa.c optional sc isa/vga_isa.c optional vga kern/kern_clocksource.c standard kern/imgact_aout.c optional compat_aout kern/imgact_gzip.c optional gzip kern/subr_sfbuf.c standard libkern/divdi3.c standard libkern/flsll.c standard libkern/memmove.c standard libkern/memset.c standard libkern/moddi3.c standard libkern/qdivrem.c standard libkern/ucmpdi2.c standard libkern/udivdi3.c standard libkern/umoddi3.c standard i386/xbox/xbox.c optional xbox i386/xbox/xboxfb.c optional xboxfb dev/fb/boot_font.c optional xboxfb i386/xbox/pic16l.s optional xbox # # x86 real mode BIOS support, required by atkbdc/dpms/vesa # compat/x86bios/x86bios.c optional x86bios | atkbd | dpms | vesa # # bvm console # dev/bvm/bvm_console.c optional bvmconsole dev/bvm/bvm_dbg.c optional bvmdebug # # x86 shared code between IA32, AMD64 and PC98 architectures # x86/acpica/OsdEnvironment.c optional acpi x86/acpica/acpi_apm.c optional acpi x86/acpica/acpi_wakeup.c optional acpi x86/acpica/madt.c optional acpi apic x86/acpica/srat.c optional acpi x86/bios/smbios.c optional smbios x86/bios/vpd.c optional vpd x86/cpufreq/est.c optional cpufreq x86/cpufreq/hwpstate.c optional cpufreq x86/cpufreq/p4tcc.c optional cpufreq x86/cpufreq/powernow.c optional cpufreq x86/cpufreq/smist.c optional cpufreq x86/iommu/busdma_dmar.c optional acpi acpi_dmar pci x86/iommu/intel_ctx.c optional acpi acpi_dmar pci x86/iommu/intel_drv.c optional acpi acpi_dmar pci x86/iommu/intel_fault.c optional acpi acpi_dmar pci x86/iommu/intel_gas.c optional acpi acpi_dmar pci x86/iommu/intel_idpgtbl.c optional acpi acpi_dmar pci x86/iommu/intel_qi.c optional acpi acpi_dmar pci x86/iommu/intel_quirks.c optional acpi acpi_dmar pci x86/iommu/intel_utils.c optional acpi acpi_dmar pci x86/isa/atpic.c optional atpic x86/isa/atrtc.c optional native x86/isa/clock.c optional native x86/isa/elcr.c optional atpic | apic native x86/isa/isa.c optional isa x86/isa/isa_dma.c optional isa x86/isa/nmi.c standard x86/isa/orm.c optional isa x86/pci/pci_bus.c optional pci x86/pci/qpi.c optional pci x86/x86/busdma_bounce.c standard x86/x86/busdma_machdep.c standard x86/x86/dump_machdep.c standard x86/x86/fdt_machdep.c optional fdt x86/x86/identcpu.c standard x86/x86/intr_machdep.c standard x86/x86/io_apic.c optional apic x86/x86/legacy.c optional native x86/x86/local_apic.c optional apic x86/x86/mca.c standard x86/x86/mptable.c optional apic native x86/x86/mptable_pci.c optional apic native pci x86/x86/msi.c optional apic pci x86/x86/nexus.c standard x86/x86/tsc.c standard x86/x86/delay.c standard x86/xen/hvm.c optional xenhvm x86/xen/xen_intr.c optional xen | xenhvm x86/xen/xen_apic.c optional xenhvm x86/xen/xenpv.c optional xen | xenhvm x86/xen/xen_nexus.c optional xen | xenhvm x86/xen/xen_msi.c optional xen | xenhvm Index: projects/clang350-import/sys/conf =================================================================== --- projects/clang350-import/sys/conf (revision 275748) +++ projects/clang350-import/sys/conf (revision 275749) Property changes on: projects/clang350-import/sys/conf ___________________________________________________________________ Modified: svn:mergeinfo ## -0,0 +0,1 ## Merged /head/sys/conf:r275685-275748 Index: projects/clang350-import/sys/crypto/aesni/aesni.c =================================================================== --- projects/clang350-import/sys/crypto/aesni/aesni.c (revision 275748) +++ projects/clang350-import/sys/crypto/aesni/aesni.c (revision 275749) @@ -1,448 +1,589 @@ /*- * Copyright (c) 2005-2008 Pawel Jakub Dawidek * Copyright (c) 2010 Konstantin Belousov + * Copyright (c) 2014 The FreeBSD Foundation * All rights reserved. * + * Portions of this software were developed by John-Mark Gurney + * under sponsorship of the FreeBSD Foundation and + * Rubicon Communications, LLC (Netgate). + * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHORS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #include __FBSDID("$FreeBSD$"); #include #include #include #include #include #include #include #include #include #include #include +#include #include #include +#include struct aesni_softc { int32_t cid; uint32_t sid; TAILQ_HEAD(aesni_sessions_head, aesni_session) sessions; struct rwlock lock; }; static int aesni_newsession(device_t, uint32_t *sidp, struct cryptoini *cri); static int aesni_freesession(device_t, uint64_t tid); static void aesni_freesession_locked(struct aesni_softc *sc, struct aesni_session *ses); static int aesni_cipher_setup(struct aesni_session *ses, struct cryptoini *encini); static int aesni_cipher_process(struct aesni_session *ses, - struct cryptodesc *enccrd, struct cryptop *crp); + struct cryptodesc *enccrd, struct cryptodesc *authcrd, struct cryptop *crp); MALLOC_DEFINE(M_AESNI, "aesni_data", "AESNI Data"); static void aesni_identify(driver_t *drv, device_t parent) { /* NB: order 10 is so we get attached after h/w devices */ if (device_find_child(parent, "aesni", -1) == NULL && BUS_ADD_CHILD(parent, 10, "aesni", -1) == 0) panic("aesni: could not attach"); } static int aesni_probe(device_t dev) { if ((cpu_feature2 & CPUID2_AESNI) == 0) { device_printf(dev, "No AESNI support.\n"); return (EINVAL); } - if ((cpu_feature & CPUID_SSE2) == 0) { - device_printf(dev, "No SSE2 support but AESNI!?!\n"); + if ((cpu_feature2 & CPUID2_SSE41) == 0) { + device_printf(dev, "No SSE4.1 support.\n"); return (EINVAL); } - device_set_desc_copy(dev, "AES-CBC,AES-XTS"); + device_set_desc_copy(dev, "AES-CBC,AES-XTS,AES-GCM,AES-ICM"); return (0); } static int aesni_attach(device_t dev) { struct aesni_softc *sc; sc = device_get_softc(dev); TAILQ_INIT(&sc->sessions); sc->sid = 1; sc->cid = crypto_get_driverid(dev, CRYPTOCAP_F_HARDWARE | CRYPTOCAP_F_SYNC); if (sc->cid < 0) { device_printf(dev, "Could not get crypto driver id.\n"); return (ENOMEM); } rw_init(&sc->lock, "aesni_lock"); crypto_register(sc->cid, CRYPTO_AES_CBC, 0, 0); + crypto_register(sc->cid, CRYPTO_AES_ICM, 0, 0); + crypto_register(sc->cid, CRYPTO_AES_NIST_GCM_16, 0, 0); + crypto_register(sc->cid, CRYPTO_AES_128_NIST_GMAC, 0, 0); + crypto_register(sc->cid, CRYPTO_AES_192_NIST_GMAC, 0, 0); + crypto_register(sc->cid, CRYPTO_AES_256_NIST_GMAC, 0, 0); crypto_register(sc->cid, CRYPTO_AES_XTS, 0, 0); return (0); } static int aesni_detach(device_t dev) { struct aesni_softc *sc; struct aesni_session *ses; sc = device_get_softc(dev); rw_wlock(&sc->lock); TAILQ_FOREACH(ses, &sc->sessions, next) { if (ses->used) { rw_wunlock(&sc->lock); device_printf(dev, "Cannot detach, sessions still active.\n"); return (EBUSY); } } while ((ses = TAILQ_FIRST(&sc->sessions)) != NULL) { TAILQ_REMOVE(&sc->sessions, ses, next); fpu_kern_free_ctx(ses->fpu_ctx); free(ses, M_AESNI); } rw_wunlock(&sc->lock); rw_destroy(&sc->lock); crypto_unregister_all(sc->cid); return (0); } static int aesni_newsession(device_t dev, uint32_t *sidp, struct cryptoini *cri) { struct aesni_softc *sc; struct aesni_session *ses; struct cryptoini *encini; int error; - if (sidp == NULL || cri == NULL) + if (sidp == NULL || cri == NULL) { + CRYPTDEB("no sidp or cri"); return (EINVAL); + } sc = device_get_softc(dev); ses = NULL; encini = NULL; for (; cri != NULL; cri = cri->cri_next) { switch (cri->cri_alg) { case CRYPTO_AES_CBC: + case CRYPTO_AES_ICM: case CRYPTO_AES_XTS: - if (encini != NULL) + case CRYPTO_AES_NIST_GCM_16: + if (encini != NULL) { + CRYPTDEB("encini already set"); return (EINVAL); + } encini = cri; break; + case CRYPTO_AES_128_NIST_GMAC: + case CRYPTO_AES_192_NIST_GMAC: + case CRYPTO_AES_256_NIST_GMAC: + /* + * nothing to do here, maybe in the future cache some + * values for GHASH + */ + break; default: + CRYPTDEB("unhandled algorithm"); return (EINVAL); } } - if (encini == NULL) + if (encini == NULL) { + CRYPTDEB("no cipher"); return (EINVAL); + } rw_wlock(&sc->lock); /* * Free sessions goes first, so if first session is used, we need to * allocate one. */ ses = TAILQ_FIRST(&sc->sessions); if (ses == NULL || ses->used) { ses = malloc(sizeof(*ses), M_AESNI, M_NOWAIT | M_ZERO); if (ses == NULL) { rw_wunlock(&sc->lock); return (ENOMEM); } ses->fpu_ctx = fpu_kern_alloc_ctx(FPU_KERN_NORMAL | FPU_KERN_NOWAIT); if (ses->fpu_ctx == NULL) { free(ses, M_AESNI); rw_wunlock(&sc->lock); return (ENOMEM); } ses->id = sc->sid++; } else { TAILQ_REMOVE(&sc->sessions, ses, next); } ses->used = 1; TAILQ_INSERT_TAIL(&sc->sessions, ses, next); rw_wunlock(&sc->lock); ses->algo = encini->cri_alg; error = aesni_cipher_setup(ses, encini); if (error != 0) { + CRYPTDEB("setup failed"); rw_wlock(&sc->lock); aesni_freesession_locked(sc, ses); rw_wunlock(&sc->lock); return (error); } *sidp = ses->id; return (0); } static void aesni_freesession_locked(struct aesni_softc *sc, struct aesni_session *ses) { struct fpu_kern_ctx *ctx; uint32_t sid; sid = ses->id; TAILQ_REMOVE(&sc->sessions, ses, next); ctx = ses->fpu_ctx; - bzero(ses, sizeof(*ses)); + *ses = (struct aesni_session){}; ses->id = sid; ses->fpu_ctx = ctx; TAILQ_INSERT_HEAD(&sc->sessions, ses, next); } static int aesni_freesession(device_t dev, uint64_t tid) { struct aesni_softc *sc; struct aesni_session *ses; uint32_t sid; sc = device_get_softc(dev); sid = ((uint32_t)tid) & 0xffffffff; rw_wlock(&sc->lock); TAILQ_FOREACH_REVERSE(ses, &sc->sessions, aesni_sessions_head, next) { if (ses->id == sid) break; } if (ses == NULL) { rw_wunlock(&sc->lock); return (EINVAL); } aesni_freesession_locked(sc, ses); rw_wunlock(&sc->lock); return (0); } static int aesni_process(device_t dev, struct cryptop *crp, int hint __unused) { struct aesni_softc *sc = device_get_softc(dev); struct aesni_session *ses = NULL; - struct cryptodesc *crd, *enccrd; - int error; + struct cryptodesc *crd, *enccrd, *authcrd; + int error, needauth; error = 0; enccrd = NULL; + authcrd = NULL; + needauth = 0; /* Sanity check. */ if (crp == NULL) return (EINVAL); if (crp->crp_callback == NULL || crp->crp_desc == NULL) { error = EINVAL; goto out; } for (crd = crp->crp_desc; crd != NULL; crd = crd->crd_next) { switch (crd->crd_alg) { case CRYPTO_AES_CBC: + case CRYPTO_AES_ICM: case CRYPTO_AES_XTS: if (enccrd != NULL) { error = EINVAL; goto out; } enccrd = crd; break; + + case CRYPTO_AES_NIST_GCM_16: + if (enccrd != NULL) { + error = EINVAL; + goto out; + } + enccrd = crd; + needauth = 1; + break; + + case CRYPTO_AES_128_NIST_GMAC: + case CRYPTO_AES_192_NIST_GMAC: + case CRYPTO_AES_256_NIST_GMAC: + if (authcrd != NULL) { + error = EINVAL; + goto out; + } + authcrd = crd; + needauth = 1; + break; + default: - return (EINVAL); + error = EINVAL; + goto out; } } - if (enccrd == NULL || (enccrd->crd_len % AES_BLOCK_LEN) != 0) { + + if (enccrd == NULL || (needauth && authcrd == NULL)) { error = EINVAL; goto out; } + /* CBC & XTS can only handle full blocks for now */ + if ((enccrd->crd_alg == CRYPTO_AES_CBC || enccrd->crd_alg == + CRYPTO_AES_XTS) && (enccrd->crd_len % AES_BLOCK_LEN) != 0) { + error = EINVAL; + goto out; + } + rw_rlock(&sc->lock); TAILQ_FOREACH_REVERSE(ses, &sc->sessions, aesni_sessions_head, next) { if (ses->id == (crp->crp_sid & 0xffffffff)) break; } rw_runlock(&sc->lock); if (ses == NULL) { error = EINVAL; goto out; } - error = aesni_cipher_process(ses, enccrd, crp); + error = aesni_cipher_process(ses, enccrd, authcrd, crp); if (error != 0) goto out; out: crp->crp_etype = error; crypto_done(crp); return (error); } uint8_t * aesni_cipher_alloc(struct cryptodesc *enccrd, struct cryptop *crp, int *allocated) { + struct mbuf *m; struct uio *uio; struct iovec *iov; uint8_t *addr; - if (crp->crp_flags & CRYPTO_F_IMBUF) - goto alloc; - else if (crp->crp_flags & CRYPTO_F_IOV) { + if (crp->crp_flags & CRYPTO_F_IMBUF) { + m = (struct mbuf *)crp->crp_buf; + if (m->m_next != NULL) + goto alloc; + addr = mtod(m, uint8_t *); + } else if (crp->crp_flags & CRYPTO_F_IOV) { uio = (struct uio *)crp->crp_buf; if (uio->uio_iovcnt != 1) goto alloc; iov = uio->uio_iov; - addr = (u_char *)iov->iov_base + enccrd->crd_skip; + addr = (uint8_t *)iov->iov_base; } else - addr = (u_char *)crp->crp_buf; + addr = (uint8_t *)crp->crp_buf; *allocated = 0; + addr += enccrd->crd_skip; return (addr); alloc: addr = malloc(enccrd->crd_len, M_AESNI, M_NOWAIT); if (addr != NULL) { *allocated = 1; crypto_copydata(crp->crp_flags, crp->crp_buf, enccrd->crd_skip, enccrd->crd_len, addr); } else *allocated = 0; return (addr); } static device_method_t aesni_methods[] = { DEVMETHOD(device_identify, aesni_identify), DEVMETHOD(device_probe, aesni_probe), DEVMETHOD(device_attach, aesni_attach), DEVMETHOD(device_detach, aesni_detach), DEVMETHOD(cryptodev_newsession, aesni_newsession), DEVMETHOD(cryptodev_freesession, aesni_freesession), DEVMETHOD(cryptodev_process, aesni_process), {0, 0}, }; static driver_t aesni_driver = { "aesni", aesni_methods, sizeof(struct aesni_softc), }; static devclass_t aesni_devclass; DRIVER_MODULE(aesni, nexus, aesni_driver, aesni_devclass, 0, 0); MODULE_VERSION(aesni, 1); MODULE_DEPEND(aesni, crypto, 1, 1, 1); static int aesni_cipher_setup(struct aesni_session *ses, struct cryptoini *encini) { struct thread *td; int error; td = curthread; error = fpu_kern_enter(td, ses->fpu_ctx, FPU_KERN_NORMAL | FPU_KERN_KTHR); if (error != 0) return (error); error = aesni_cipher_setup_common(ses, encini->cri_key, encini->cri_klen); fpu_kern_leave(td, ses->fpu_ctx); return (error); } +/* + * authcrd contains the associated date. + */ static int aesni_cipher_process(struct aesni_session *ses, struct cryptodesc *enccrd, - struct cryptop *crp) + struct cryptodesc *authcrd, struct cryptop *crp) { + uint8_t tag[GMAC_DIGEST_LEN]; struct thread *td; - uint8_t *buf; - int error, allocated; + uint8_t *buf, *authbuf; + int error, allocated, authallocated; + int ivlen, encflag; + encflag = (enccrd->crd_flags & CRD_F_ENCRYPT) == CRD_F_ENCRYPT; + + if ((enccrd->crd_alg == CRYPTO_AES_ICM || + enccrd->crd_alg == CRYPTO_AES_NIST_GCM_16) && + (enccrd->crd_flags & CRD_F_IV_EXPLICIT) == 0) + return (EINVAL); + buf = aesni_cipher_alloc(enccrd, crp, &allocated); if (buf == NULL) return (ENOMEM); + authbuf = NULL; + authallocated = 0; + if (authcrd != NULL) { + authbuf = aesni_cipher_alloc(authcrd, crp, &authallocated); + if (authbuf == NULL) { + error = ENOMEM; + goto out1; + } + } + td = curthread; error = fpu_kern_enter(td, ses->fpu_ctx, FPU_KERN_NORMAL | FPU_KERN_KTHR); if (error != 0) goto out1; if ((enccrd->crd_flags & CRD_F_KEY_EXPLICIT) != 0) { error = aesni_cipher_setup_common(ses, enccrd->crd_key, enccrd->crd_klen); if (error != 0) goto out; } - if ((enccrd->crd_flags & CRD_F_ENCRYPT) != 0) { - if ((enccrd->crd_flags & CRD_F_IV_EXPLICIT) != 0) - bcopy(enccrd->crd_iv, ses->iv, AES_BLOCK_LEN); - if ((enccrd->crd_flags & CRD_F_IV_PRESENT) == 0) - crypto_copyback(crp->crp_flags, crp->crp_buf, - enccrd->crd_inject, AES_BLOCK_LEN, ses->iv); - if (ses->algo == CRYPTO_AES_CBC) { + /* XXX - validate that enccrd and authcrd have/use same key? */ + switch (enccrd->crd_alg) { + case CRYPTO_AES_CBC: + case CRYPTO_AES_ICM: + ivlen = AES_BLOCK_LEN; + break; + case CRYPTO_AES_XTS: + ivlen = 8; + break; + case CRYPTO_AES_NIST_GCM_16: + ivlen = 12; /* should support arbitarily larger */ + break; + } + + /* Setup ses->iv */ + bzero(ses->iv, sizeof ses->iv); + if ((enccrd->crd_flags & CRD_F_IV_EXPLICIT) != 0) + bcopy(enccrd->crd_iv, ses->iv, ivlen); + else if (encflag && ((enccrd->crd_flags & CRD_F_IV_PRESENT) != 0)) + arc4rand(ses->iv, ivlen, 0); + else + crypto_copydata(crp->crp_flags, crp->crp_buf, + enccrd->crd_inject, ivlen, ses->iv); + + if (authcrd != NULL && !encflag) + crypto_copydata(crp->crp_flags, crp->crp_buf, + authcrd->crd_inject, GMAC_DIGEST_LEN, tag); + else + bzero(tag, sizeof tag); + + /* Do work */ + switch (ses->algo) { + case CRYPTO_AES_CBC: + if (encflag) aesni_encrypt_cbc(ses->rounds, ses->enc_schedule, enccrd->crd_len, buf, buf, ses->iv); - } else /* if (ses->algo == CRYPTO_AES_XTS) */ { + else + aesni_decrypt_cbc(ses->rounds, ses->dec_schedule, + enccrd->crd_len, buf, ses->iv); + break; + case CRYPTO_AES_ICM: + /* encryption & decryption are the same */ + aesni_encrypt_icm(ses->rounds, ses->enc_schedule, + enccrd->crd_len, buf, buf, ses->iv); + break; + case CRYPTO_AES_XTS: + if (encflag) aesni_encrypt_xts(ses->rounds, ses->enc_schedule, ses->xts_schedule, enccrd->crd_len, buf, buf, ses->iv); - } - } else { - if ((enccrd->crd_flags & CRD_F_IV_EXPLICIT) != 0) - bcopy(enccrd->crd_iv, ses->iv, AES_BLOCK_LEN); else - crypto_copydata(crp->crp_flags, crp->crp_buf, - enccrd->crd_inject, AES_BLOCK_LEN, ses->iv); - if (ses->algo == CRYPTO_AES_CBC) { - aesni_decrypt_cbc(ses->rounds, ses->dec_schedule, - enccrd->crd_len, buf, ses->iv); - } else /* if (ses->algo == CRYPTO_AES_XTS) */ { aesni_decrypt_xts(ses->rounds, ses->dec_schedule, ses->xts_schedule, enccrd->crd_len, buf, buf, ses->iv); + break; + case CRYPTO_AES_NIST_GCM_16: + if (encflag) + AES_GCM_encrypt(buf, buf, authbuf, ses->iv, tag, + enccrd->crd_len, authcrd->crd_len, ivlen, + ses->enc_schedule, ses->rounds); + else { + if (!AES_GCM_decrypt(buf, buf, authbuf, ses->iv, tag, + enccrd->crd_len, authcrd->crd_len, ivlen, + ses->enc_schedule, ses->rounds)) + error = EBADMSG; } + break; } + if (allocated) crypto_copyback(crp->crp_flags, crp->crp_buf, enccrd->crd_skip, enccrd->crd_len, buf); - if ((enccrd->crd_flags & CRD_F_ENCRYPT) != 0) - crypto_copydata(crp->crp_flags, crp->crp_buf, - enccrd->crd_skip + enccrd->crd_len - AES_BLOCK_LEN, - AES_BLOCK_LEN, ses->iv); + + /* + * OpenBSD doesn't copy this back. This primes the IV for the next + * chain. Why do we not do it for decrypt? + */ + if (encflag && enccrd->crd_alg == CRYPTO_AES_CBC) + bcopy(buf + enccrd->crd_len - AES_BLOCK_LEN, ses->iv, AES_BLOCK_LEN); + + if (!error && authcrd != NULL) { + crypto_copyback(crp->crp_flags, crp->crp_buf, + authcrd->crd_inject, GMAC_DIGEST_LEN, tag); + } + out: fpu_kern_leave(td, ses->fpu_ctx); out1: if (allocated) { bzero(buf, enccrd->crd_len); free(buf, M_AESNI); } + if (authallocated) + free(authbuf, M_AESNI); return (error); } Index: projects/clang350-import/sys/crypto/aesni/aesni.h =================================================================== --- projects/clang350-import/sys/crypto/aesni/aesni.h (revision 275748) +++ projects/clang350-import/sys/crypto/aesni/aesni.h (revision 275749) @@ -1,104 +1,117 @@ /*- * Copyright (c) 2010 Konstantin Belousov * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHORS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * $FreeBSD$ */ #ifndef _AESNI_H_ #define _AESNI_H_ #include #include #include #include #if defined(__amd64__) || (defined(__i386__) && !defined(PC98)) #include #include #include #include #endif #if defined(__i386__) #include #elif defined(__amd64__) #include #endif #define AES128_ROUNDS 10 #define AES192_ROUNDS 12 #define AES256_ROUNDS 14 #define AES_SCHED_LEN ((AES256_ROUNDS + 1) * AES_BLOCK_LEN) struct aesni_session { uint8_t enc_schedule[AES_SCHED_LEN] __aligned(16); uint8_t dec_schedule[AES_SCHED_LEN] __aligned(16); uint8_t xts_schedule[AES_SCHED_LEN] __aligned(16); uint8_t iv[AES_BLOCK_LEN]; int algo; int rounds; /* uint8_t *ses_ictx; */ /* uint8_t *ses_octx; */ /* int ses_mlen; */ int used; uint32_t id; TAILQ_ENTRY(aesni_session) next; struct fpu_kern_ctx *fpu_ctx; }; /* * Internal functions, implemented in assembler. */ void aesni_set_enckey(const uint8_t *userkey, uint8_t *encrypt_schedule /*__aligned(16)*/, int number_of_rounds); void aesni_set_deckey(const uint8_t *encrypt_schedule /*__aligned(16)*/, uint8_t *decrypt_schedule /*__aligned(16)*/, int number_of_rounds); /* * Slightly more public interfaces. */ void aesni_encrypt_cbc(int rounds, const void *key_schedule /*__aligned(16)*/, size_t len, const uint8_t *from, uint8_t *to, const uint8_t iv[AES_BLOCK_LEN]); void aesni_decrypt_cbc(int rounds, const void *key_schedule /*__aligned(16)*/, size_t len, uint8_t *buf, const uint8_t iv[AES_BLOCK_LEN]); void aesni_encrypt_ecb(int rounds, const void *key_schedule /*__aligned(16)*/, size_t len, const uint8_t *from, uint8_t *to); void aesni_decrypt_ecb(int rounds, const void *key_schedule /*__aligned(16)*/, size_t len, const uint8_t *from, uint8_t *to); +void aesni_encrypt_icm(int rounds, const void *key_schedule /*__aligned(16)*/, + size_t len, const uint8_t *from, uint8_t *to, + const uint8_t iv[AES_BLOCK_LEN]); void aesni_encrypt_xts(int rounds, const void *data_schedule /*__aligned(16)*/, const void *tweak_schedule /*__aligned(16)*/, size_t len, const uint8_t *from, uint8_t *to, const uint8_t iv[AES_BLOCK_LEN]); void aesni_decrypt_xts(int rounds, const void *data_schedule /*__aligned(16)*/, const void *tweak_schedule /*__aligned(16)*/, size_t len, const uint8_t *from, uint8_t *to, const uint8_t iv[AES_BLOCK_LEN]); + +/* GCM & GHASH functions */ +void AES_GCM_encrypt(const unsigned char *in, unsigned char *out, + const unsigned char *addt, const unsigned char *ivec, + unsigned char *tag, uint32_t nbytes, uint32_t abytes, int ibytes, + const unsigned char *key, int nr); +int AES_GCM_decrypt(const unsigned char *in, unsigned char *out, + const unsigned char *addt, const unsigned char *ivec, + unsigned char *tag, uint32_t nbytes, uint32_t abytes, int ibytes, + const unsigned char *key, int nr); int aesni_cipher_setup_common(struct aesni_session *ses, const uint8_t *key, int keylen); uint8_t *aesni_cipher_alloc(struct cryptodesc *enccrd, struct cryptop *crp, int *allocated); #endif /* _AESNI_H_ */ Index: projects/clang350-import/sys/crypto/aesni/aesni_ghash.c =================================================================== --- projects/clang350-import/sys/crypto/aesni/aesni_ghash.c (nonexistent) +++ projects/clang350-import/sys/crypto/aesni/aesni_ghash.c (revision 275749) @@ -0,0 +1,803 @@ +/*- + * Copyright (c) 2014 The FreeBSD Foundation + * All rights reserved. + * + * This software was developed by John-Mark Gurney under + * the sponsorship of the FreeBSD Foundation and + * Rubicon Communications, LLC (Netgate). + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * + * $FreeBSD$ + * + */ + +/* + * Figure 5, 8 and 12 are copied from the Intel white paper: + * Intel® Carry-Less Multiplication Instruction and its Usage for + * Computing the GCM Mode + * + * and as such are: + * Copyright © 2010 Intel Corporation. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * * Neither the name of Intel Corporation nor the + * names of its contributors may be used to endorse or promote products + * derived from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#ifdef _KERNEL +#include +#else +#include +#endif + +#include +#include +#include + +static inline int +m128icmp(__m128i a, __m128i b) +{ + __m128i cmp; + + cmp = _mm_cmpeq_epi32(a, b); + + return _mm_movemask_epi8(cmp) == 0xffff; +} + +#ifdef __i386__ +static inline __m128i +_mm_insert_epi64(__m128i a, int64_t b, const int ndx) +{ + + if (!ndx) { + a = _mm_insert_epi32(a, b, 0); + a = _mm_insert_epi32(a, b >> 32, 1); + } else { + a = _mm_insert_epi32(a, b, 2); + a = _mm_insert_epi32(a, b >> 32, 3); + } + + return a; +} +#endif + +/* some code from carry-less-multiplication-instruction-in-gcm-mode-paper.pdf */ + +/* Figure 5. Code Sample - Performing Ghash Using Algorithms 1 and 5 (C) */ +static void +gfmul(__m128i a, __m128i b, __m128i *res) +{ + __m128i tmp2, tmp3, tmp4, tmp5, tmp6, tmp7, tmp8, tmp9; + + tmp3 = _mm_clmulepi64_si128(a, b, 0x00); + tmp4 = _mm_clmulepi64_si128(a, b, 0x10); + tmp5 = _mm_clmulepi64_si128(a, b, 0x01); + tmp6 = _mm_clmulepi64_si128(a, b, 0x11); + + tmp4 = _mm_xor_si128(tmp4, tmp5); + tmp5 = _mm_slli_si128(tmp4, 8); + tmp4 = _mm_srli_si128(tmp4, 8); + tmp3 = _mm_xor_si128(tmp3, tmp5); + tmp6 = _mm_xor_si128(tmp6, tmp4); + + tmp7 = _mm_srli_epi32(tmp3, 31); + tmp8 = _mm_srli_epi32(tmp6, 31); + tmp3 = _mm_slli_epi32(tmp3, 1); + tmp6 = _mm_slli_epi32(tmp6, 1); + + tmp9 = _mm_srli_si128(tmp7, 12); + tmp8 = _mm_slli_si128(tmp8, 4); + tmp7 = _mm_slli_si128(tmp7, 4); + tmp3 = _mm_or_si128(tmp3, tmp7); + tmp6 = _mm_or_si128(tmp6, tmp8); + tmp6 = _mm_or_si128(tmp6, tmp9); + + tmp7 = _mm_slli_epi32(tmp3, 31); + tmp8 = _mm_slli_epi32(tmp3, 30); + tmp9 = _mm_slli_epi32(tmp3, 25); + + tmp7 = _mm_xor_si128(tmp7, tmp8); + tmp7 = _mm_xor_si128(tmp7, tmp9); + tmp8 = _mm_srli_si128(tmp7, 4); + tmp7 = _mm_slli_si128(tmp7, 12); + tmp3 = _mm_xor_si128(tmp3, tmp7); + + tmp2 = _mm_srli_epi32(tmp3, 1); + tmp4 = _mm_srli_epi32(tmp3, 2); + tmp5 = _mm_srli_epi32(tmp3, 7); + tmp2 = _mm_xor_si128(tmp2, tmp4); + tmp2 = _mm_xor_si128(tmp2, tmp5); + tmp2 = _mm_xor_si128(tmp2, tmp8); + tmp3 = _mm_xor_si128(tmp3, tmp2); + tmp6 = _mm_xor_si128(tmp6, tmp3); + + *res = tmp6; +} + +/* + * Figure 8. Code Sample - Performing Ghash Using an Aggregated Reduction + * Method */ +static void +reduce4(__m128i H1, __m128i H2, __m128i H3, __m128i H4, + __m128i X1, __m128i X2, __m128i X3, __m128i X4, __m128i *res) +{ + /*algorithm by Krzysztof Jankowski, Pierre Laurent - Intel*/ + __m128i H1_X1_lo, H1_X1_hi, H2_X2_lo, H2_X2_hi, H3_X3_lo, + H3_X3_hi, H4_X4_lo, H4_X4_hi, lo, hi; + __m128i tmp0, tmp1, tmp2, tmp3; + __m128i tmp4, tmp5, tmp6, tmp7; + __m128i tmp8, tmp9; + + H1_X1_lo = _mm_clmulepi64_si128(H1, X1, 0x00); + H2_X2_lo = _mm_clmulepi64_si128(H2, X2, 0x00); + H3_X3_lo = _mm_clmulepi64_si128(H3, X3, 0x00); + H4_X4_lo = _mm_clmulepi64_si128(H4, X4, 0x00); + + lo = _mm_xor_si128(H1_X1_lo, H2_X2_lo); + lo = _mm_xor_si128(lo, H3_X3_lo); + lo = _mm_xor_si128(lo, H4_X4_lo); + + H1_X1_hi = _mm_clmulepi64_si128(H1, X1, 0x11); + H2_X2_hi = _mm_clmulepi64_si128(H2, X2, 0x11); + H3_X3_hi = _mm_clmulepi64_si128(H3, X3, 0x11); + H4_X4_hi = _mm_clmulepi64_si128(H4, X4, 0x11); + + hi = _mm_xor_si128(H1_X1_hi, H2_X2_hi); + hi = _mm_xor_si128(hi, H3_X3_hi); + hi = _mm_xor_si128(hi, H4_X4_hi); + + tmp0 = _mm_shuffle_epi32(H1, 78); + tmp4 = _mm_shuffle_epi32(X1, 78); + tmp0 = _mm_xor_si128(tmp0, H1); + tmp4 = _mm_xor_si128(tmp4, X1); + tmp1 = _mm_shuffle_epi32(H2, 78); + tmp5 = _mm_shuffle_epi32(X2, 78); + tmp1 = _mm_xor_si128(tmp1, H2); + tmp5 = _mm_xor_si128(tmp5, X2); + tmp2 = _mm_shuffle_epi32(H3, 78); + tmp6 = _mm_shuffle_epi32(X3, 78); + tmp2 = _mm_xor_si128(tmp2, H3); + tmp6 = _mm_xor_si128(tmp6, X3); + tmp3 = _mm_shuffle_epi32(H4, 78); + tmp7 = _mm_shuffle_epi32(X4, 78); + tmp3 = _mm_xor_si128(tmp3, H4); + tmp7 = _mm_xor_si128(tmp7, X4); + + tmp0 = _mm_clmulepi64_si128(tmp0, tmp4, 0x00); + tmp1 = _mm_clmulepi64_si128(tmp1, tmp5, 0x00); + tmp2 = _mm_clmulepi64_si128(tmp2, tmp6, 0x00); + tmp3 = _mm_clmulepi64_si128(tmp3, tmp7, 0x00); + + tmp0 = _mm_xor_si128(tmp0, lo); + tmp0 = _mm_xor_si128(tmp0, hi); + tmp0 = _mm_xor_si128(tmp1, tmp0); + tmp0 = _mm_xor_si128(tmp2, tmp0); + tmp0 = _mm_xor_si128(tmp3, tmp0); + + tmp4 = _mm_slli_si128(tmp0, 8); + tmp0 = _mm_srli_si128(tmp0, 8); + + lo = _mm_xor_si128(tmp4, lo); + hi = _mm_xor_si128(tmp0, hi); + + tmp3 = lo; + tmp6 = hi; + + tmp7 = _mm_srli_epi32(tmp3, 31); + tmp8 = _mm_srli_epi32(tmp6, 31); + tmp3 = _mm_slli_epi32(tmp3, 1); + tmp6 = _mm_slli_epi32(tmp6, 1); + + tmp9 = _mm_srli_si128(tmp7, 12); + tmp8 = _mm_slli_si128(tmp8, 4); + tmp7 = _mm_slli_si128(tmp7, 4); + tmp3 = _mm_or_si128(tmp3, tmp7); + tmp6 = _mm_or_si128(tmp6, tmp8); + tmp6 = _mm_or_si128(tmp6, tmp9); + + tmp7 = _mm_slli_epi32(tmp3, 31); + tmp8 = _mm_slli_epi32(tmp3, 30); + tmp9 = _mm_slli_epi32(tmp3, 25); + + tmp7 = _mm_xor_si128(tmp7, tmp8); + tmp7 = _mm_xor_si128(tmp7, tmp9); + tmp8 = _mm_srli_si128(tmp7, 4); + tmp7 = _mm_slli_si128(tmp7, 12); + tmp3 = _mm_xor_si128(tmp3, tmp7); + + tmp2 = _mm_srli_epi32(tmp3, 1); + tmp4 = _mm_srli_epi32(tmp3, 2); + tmp5 = _mm_srli_epi32(tmp3, 7); + tmp2 = _mm_xor_si128(tmp2, tmp4); + tmp2 = _mm_xor_si128(tmp2, tmp5); + tmp2 = _mm_xor_si128(tmp2, tmp8); + tmp3 = _mm_xor_si128(tmp3, tmp2); + tmp6 = _mm_xor_si128(tmp6, tmp3); + + *res = tmp6; +} + +/* + * Figure 12. AES-GCM: Processing Four Blocks in Parallel with Aggregated + * Every Four Blocks + */ +/* + * per NIST SP-800-38D, 5.2.1.1, len(p) <= 2^39-256 (in bits), or + * 2^32-256*8*16 bytes. + */ +void +AES_GCM_encrypt(const unsigned char *in, unsigned char *out, + const unsigned char *addt, const unsigned char *ivec, + unsigned char *tag, uint32_t nbytes, uint32_t abytes, int ibytes, + const unsigned char *key, int nr) +{ + int i, j ,k; + __m128i tmp1, tmp2, tmp3, tmp4; + __m128i tmp5, tmp6, tmp7, tmp8; + __m128i H, H2, H3, H4, Y, T; + __m128i *KEY = (__m128i*)key; + __m128i ctr1, ctr2, ctr3, ctr4; + __m128i ctr5, ctr6, ctr7, ctr8; + __m128i last_block = _mm_setzero_si128(); + __m128i ONE = _mm_set_epi32(0, 1, 0, 0); + __m128i EIGHT = _mm_set_epi32(0, 8, 0, 0); + __m128i BSWAP_EPI64 = _mm_set_epi8(8,9,10,11,12,13,14,15,0,1,2,3,4,5,6, + 7); + __m128i BSWAP_MASK = _mm_set_epi8(0,1,2,3,4,5,6,7,8,9,10,11,12,13,14, + 15); + __m128i X = _mm_setzero_si128(); + + if (ibytes == 96/8) { + Y = _mm_loadu_si128((__m128i*)ivec); + Y = _mm_insert_epi32(Y, 0x1000000, 3); + /*(Compute E[ZERO, KS] and E[Y0, KS] together*/ + tmp1 = _mm_xor_si128(X, KEY[0]); + tmp2 = _mm_xor_si128(Y, KEY[0]); + for (j=1; j < nr-1; j+=2) { + tmp1 = _mm_aesenc_si128(tmp1, KEY[j]); + tmp2 = _mm_aesenc_si128(tmp2, KEY[j]); + + tmp1 = _mm_aesenc_si128(tmp1, KEY[j+1]); + tmp2 = _mm_aesenc_si128(tmp2, KEY[j+1]); + } + tmp1 = _mm_aesenc_si128(tmp1, KEY[nr-1]); + tmp2 = _mm_aesenc_si128(tmp2, KEY[nr-1]); + + H = _mm_aesenclast_si128(tmp1, KEY[nr]); + T = _mm_aesenclast_si128(tmp2, KEY[nr]); + + H = _mm_shuffle_epi8(H, BSWAP_MASK); + } else { + tmp1 = _mm_xor_si128(X, KEY[0]); + for (j=1; j * Copyright (c) 2010 Konstantin Belousov * Copyright (c) 2010-2011 Pawel Jakub Dawidek * Copyright 2012-2013 John-Mark Gurney + * Copyright (c) 2014 The FreeBSD Foundation * All rights reserved. * + * Portions of this software were developed by John-Mark Gurney + * under sponsorship of the FreeBSD Foundation and + * Rubicon Communications, LLC (Netgate). + * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHORS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #include __FBSDID("$FreeBSD$"); - + #include #include #include #include #include #include - + +#include + #include "aesencdec.h" +#include MALLOC_DECLARE(M_AESNI); struct blocks8 { __m128i blk[8]; } __packed; void aesni_encrypt_cbc(int rounds, const void *key_schedule, size_t len, const uint8_t *from, uint8_t *to, const uint8_t iv[AES_BLOCK_LEN]) { __m128i tot, ivreg; size_t i; len /= AES_BLOCK_LEN; ivreg = _mm_loadu_si128((const __m128i *)iv); for (i = 0; i < len; i++) { tot = aesni_enc(rounds - 1, key_schedule, _mm_loadu_si128((const __m128i *)from) ^ ivreg); ivreg = tot; _mm_storeu_si128((__m128i *)to, tot); from += AES_BLOCK_LEN; to += AES_BLOCK_LEN; } } void aesni_decrypt_cbc(int rounds, const void *key_schedule, size_t len, uint8_t *buf, const uint8_t iv[AES_BLOCK_LEN]) { __m128i blocks[8]; struct blocks8 *blks; __m128i ivreg, nextiv; size_t i, j, cnt; ivreg = _mm_loadu_si128((const __m128i *)iv); cnt = len / AES_BLOCK_LEN / 8; for (i = 0; i < cnt; i++) { blks = (struct blocks8 *)buf; aesni_dec8(rounds - 1, key_schedule, blks->blk[0], blks->blk[1], blks->blk[2], blks->blk[3], blks->blk[4], blks->blk[5], blks->blk[6], blks->blk[7], &blocks[0]); for (j = 0; j < 8; j++) { nextiv = blks->blk[j]; blks->blk[j] = blocks[j] ^ ivreg; ivreg = nextiv; } buf += AES_BLOCK_LEN * 8; } i *= 8; cnt = len / AES_BLOCK_LEN; for (; i < cnt; i++) { nextiv = _mm_loadu_si128((void *)buf); _mm_storeu_si128((void *)buf, aesni_dec(rounds - 1, key_schedule, nextiv) ^ ivreg); ivreg = nextiv; buf += AES_BLOCK_LEN; } } void aesni_encrypt_ecb(int rounds, const void *key_schedule, size_t len, const uint8_t *from, uint8_t *to) { __m128i tot; __m128i tout[8]; struct blocks8 *top; const struct blocks8 *blks; size_t i, cnt; cnt = len / AES_BLOCK_LEN / 8; for (i = 0; i < cnt; i++) { blks = (const struct blocks8 *)from; top = (struct blocks8 *)to; aesni_enc8(rounds - 1, key_schedule, blks->blk[0], blks->blk[1], blks->blk[2], blks->blk[3], blks->blk[4], blks->blk[5], blks->blk[6], blks->blk[7], tout); top->blk[0] = tout[0]; top->blk[1] = tout[1]; top->blk[2] = tout[2]; top->blk[3] = tout[3]; top->blk[4] = tout[4]; top->blk[5] = tout[5]; top->blk[6] = tout[6]; top->blk[7] = tout[7]; from += AES_BLOCK_LEN * 8; to += AES_BLOCK_LEN * 8; } i *= 8; cnt = len / AES_BLOCK_LEN; for (; i < cnt; i++) { tot = aesni_enc(rounds - 1, key_schedule, _mm_loadu_si128((const __m128i *)from)); _mm_storeu_si128((__m128i *)to, tot); from += AES_BLOCK_LEN; to += AES_BLOCK_LEN; } } void aesni_decrypt_ecb(int rounds, const void *key_schedule, size_t len, const uint8_t from[AES_BLOCK_LEN], uint8_t to[AES_BLOCK_LEN]) { __m128i tot; __m128i tout[8]; const struct blocks8 *blks; struct blocks8 *top; size_t i, cnt; cnt = len / AES_BLOCK_LEN / 8; for (i = 0; i < cnt; i++) { blks = (const struct blocks8 *)from; top = (struct blocks8 *)to; aesni_dec8(rounds - 1, key_schedule, blks->blk[0], blks->blk[1], blks->blk[2], blks->blk[3], blks->blk[4], blks->blk[5], blks->blk[6], blks->blk[7], tout); top->blk[0] = tout[0]; top->blk[1] = tout[1]; top->blk[2] = tout[2]; top->blk[3] = tout[3]; top->blk[4] = tout[4]; top->blk[5] = tout[5]; top->blk[6] = tout[6]; top->blk[7] = tout[7]; from += AES_BLOCK_LEN * 8; to += AES_BLOCK_LEN * 8; } i *= 8; cnt = len / AES_BLOCK_LEN; for (; i < cnt; i++) { tot = aesni_dec(rounds - 1, key_schedule, _mm_loadu_si128((const __m128i *)from)); _mm_storeu_si128((__m128i *)to, tot); from += AES_BLOCK_LEN; to += AES_BLOCK_LEN; } } +/* + * mixed endian increment, low 64bits stored in hi word to be compatible + * with _icm's BSWAP. + */ +static inline __m128i +nextc(__m128i x) +{ + const __m128i ONE = _mm_setr_epi32(0, 0, 1, 0); + const __m128i ZERO = _mm_setzero_si128(); + + x = _mm_add_epi64(x, ONE); + __m128i t = _mm_cmpeq_epi64(x, ZERO); + t = _mm_unpackhi_epi64(t, ZERO); + x = _mm_sub_epi64(x, t); + + return x; +} + +void +aesni_encrypt_icm(int rounds, const void *key_schedule, size_t len, + const uint8_t *from, uint8_t *to, const uint8_t iv[AES_BLOCK_LEN]) +{ + __m128i tot; + __m128i tmp1, tmp2, tmp3, tmp4; + __m128i tmp5, tmp6, tmp7, tmp8; + __m128i ctr1, ctr2, ctr3, ctr4; + __m128i ctr5, ctr6, ctr7, ctr8; + __m128i BSWAP_EPI64; + __m128i tout[8]; + struct blocks8 *top; + const struct blocks8 *blks; + size_t i, cnt; + + BSWAP_EPI64 = _mm_set_epi8(8,9,10,11,12,13,14,15,0,1,2,3,4,5,6,7); + + ctr1 = _mm_loadu_si128((__m128i*)iv); + ctr1 = _mm_shuffle_epi8(ctr1, BSWAP_EPI64); + + cnt = len / AES_BLOCK_LEN / 8; + for (i = 0; i < cnt; i++) { + tmp1 = _mm_shuffle_epi8(ctr1, BSWAP_EPI64); + ctr2 = nextc(ctr1); + tmp2 = _mm_shuffle_epi8(ctr2, BSWAP_EPI64); + ctr3 = nextc(ctr2); + tmp3 = _mm_shuffle_epi8(ctr3, BSWAP_EPI64); + ctr4 = nextc(ctr3); + tmp4 = _mm_shuffle_epi8(ctr4, BSWAP_EPI64); + ctr5 = nextc(ctr4); + tmp5 = _mm_shuffle_epi8(ctr5, BSWAP_EPI64); + ctr6 = nextc(ctr5); + tmp6 = _mm_shuffle_epi8(ctr6, BSWAP_EPI64); + ctr7 = nextc(ctr6); + tmp7 = _mm_shuffle_epi8(ctr7, BSWAP_EPI64); + ctr8 = nextc(ctr7); + tmp8 = _mm_shuffle_epi8(ctr8, BSWAP_EPI64); + ctr1 = nextc(ctr8); + + blks = (const struct blocks8 *)from; + top = (struct blocks8 *)to; + aesni_enc8(rounds - 1, key_schedule, tmp1, tmp2, tmp3, tmp4, + tmp5, tmp6, tmp7, tmp8, tout); + + top->blk[0] = blks->blk[0] ^ tout[0]; + top->blk[1] = blks->blk[1] ^ tout[1]; + top->blk[2] = blks->blk[2] ^ tout[2]; + top->blk[3] = blks->blk[3] ^ tout[3]; + top->blk[4] = blks->blk[4] ^ tout[4]; + top->blk[5] = blks->blk[5] ^ tout[5]; + top->blk[6] = blks->blk[6] ^ tout[6]; + top->blk[7] = blks->blk[7] ^ tout[7]; + + from += AES_BLOCK_LEN * 8; + to += AES_BLOCK_LEN * 8; + } + i *= 8; + cnt = len / AES_BLOCK_LEN; + for (; i < cnt; i++) { + tmp1 = _mm_shuffle_epi8(ctr1, BSWAP_EPI64); + ctr1 = nextc(ctr1); + + tot = aesni_enc(rounds - 1, key_schedule, tmp1); + + tot = tot ^ _mm_loadu_si128((const __m128i *)from); + _mm_storeu_si128((__m128i *)to, tot); + + from += AES_BLOCK_LEN; + to += AES_BLOCK_LEN; + } + + /* handle remaining partial round */ + if (len % AES_BLOCK_LEN != 0) { + tmp1 = _mm_shuffle_epi8(ctr1, BSWAP_EPI64); + tot = aesni_enc(rounds - 1, key_schedule, tmp1); + tot = tot ^ _mm_loadu_si128((const __m128i *)from); + memcpy(to, &tot, len % AES_BLOCK_LEN); + } +} + #define AES_XTS_BLOCKSIZE 16 #define AES_XTS_IVSIZE 8 #define AES_XTS_ALPHA 0x87 /* GF(2^128) generator polynomial */ static inline __m128i xts_crank_lfsr(__m128i inp) { const __m128i alphamask = _mm_set_epi32(1, 1, 1, AES_XTS_ALPHA); __m128i xtweak, ret; /* set up xor mask */ xtweak = _mm_shuffle_epi32(inp, 0x93); xtweak = _mm_srai_epi32(xtweak, 31); xtweak &= alphamask; /* next term */ ret = _mm_slli_epi32(inp, 1); ret ^= xtweak; return ret; } static void aesni_crypt_xts_block(int rounds, const __m128i *key_schedule, __m128i *tweak, const uint8_t *from, uint8_t *to, int do_encrypt) { __m128i block; block = _mm_loadu_si128((const __m128i *)from) ^ *tweak; if (do_encrypt) block = aesni_enc(rounds - 1, key_schedule, block); else block = aesni_dec(rounds - 1, key_schedule, block); _mm_storeu_si128((__m128i *)to, block ^ *tweak); *tweak = xts_crank_lfsr(*tweak); } static void aesni_crypt_xts_block8(int rounds, const __m128i *key_schedule, __m128i *tweak, const uint8_t *from, uint8_t *to, int do_encrypt) { __m128i tmptweak; __m128i a, b, c, d, e, f, g, h; __m128i tweaks[8]; __m128i tmp[8]; __m128i *top; const __m128i *fromp; tmptweak = *tweak; /* * unroll the loop. This lets gcc put values directly in the * register and saves memory accesses. */ fromp = (const __m128i *)from; #define PREPINP(v, pos) \ do { \ tweaks[(pos)] = tmptweak; \ (v) = _mm_loadu_si128(&fromp[pos]) ^ \ tmptweak; \ tmptweak = xts_crank_lfsr(tmptweak); \ } while (0) PREPINP(a, 0); PREPINP(b, 1); PREPINP(c, 2); PREPINP(d, 3); PREPINP(e, 4); PREPINP(f, 5); PREPINP(g, 6); PREPINP(h, 7); *tweak = tmptweak; if (do_encrypt) aesni_enc8(rounds - 1, key_schedule, a, b, c, d, e, f, g, h, tmp); else aesni_dec8(rounds - 1, key_schedule, a, b, c, d, e, f, g, h, tmp); top = (__m128i *)to; _mm_storeu_si128(&top[0], tmp[0] ^ tweaks[0]); _mm_storeu_si128(&top[1], tmp[1] ^ tweaks[1]); _mm_storeu_si128(&top[2], tmp[2] ^ tweaks[2]); _mm_storeu_si128(&top[3], tmp[3] ^ tweaks[3]); _mm_storeu_si128(&top[4], tmp[4] ^ tweaks[4]); _mm_storeu_si128(&top[5], tmp[5] ^ tweaks[5]); _mm_storeu_si128(&top[6], tmp[6] ^ tweaks[6]); _mm_storeu_si128(&top[7], tmp[7] ^ tweaks[7]); } static void aesni_crypt_xts(int rounds, const __m128i *data_schedule, const __m128i *tweak_schedule, size_t len, const uint8_t *from, uint8_t *to, const uint8_t iv[AES_BLOCK_LEN], int do_encrypt) { __m128i tweakreg; uint8_t tweak[AES_XTS_BLOCKSIZE] __aligned(16); size_t i, cnt; /* * Prepare tweak as E_k2(IV). IV is specified as LE representation * of a 64-bit block number which we allow to be passed in directly. */ #if BYTE_ORDER == LITTLE_ENDIAN bcopy(iv, tweak, AES_XTS_IVSIZE); /* Last 64 bits of IV are always zero. */ bzero(tweak + AES_XTS_IVSIZE, AES_XTS_IVSIZE); #else #error Only LITTLE_ENDIAN architectures are supported. #endif tweakreg = _mm_loadu_si128((__m128i *)&tweak[0]); tweakreg = aesni_enc(rounds - 1, tweak_schedule, tweakreg); cnt = len / AES_XTS_BLOCKSIZE / 8; for (i = 0; i < cnt; i++) { aesni_crypt_xts_block8(rounds, data_schedule, &tweakreg, from, to, do_encrypt); from += AES_XTS_BLOCKSIZE * 8; to += AES_XTS_BLOCKSIZE * 8; } i *= 8; cnt = len / AES_XTS_BLOCKSIZE; for (; i < cnt; i++) { aesni_crypt_xts_block(rounds, data_schedule, &tweakreg, from, to, do_encrypt); from += AES_XTS_BLOCKSIZE; to += AES_XTS_BLOCKSIZE; } } void aesni_encrypt_xts(int rounds, const void *data_schedule, const void *tweak_schedule, size_t len, const uint8_t *from, uint8_t *to, const uint8_t iv[AES_BLOCK_LEN]) { aesni_crypt_xts(rounds, data_schedule, tweak_schedule, len, from, to, iv, 1); } void aesni_decrypt_xts(int rounds, const void *data_schedule, const void *tweak_schedule, size_t len, const uint8_t *from, uint8_t *to, const uint8_t iv[AES_BLOCK_LEN]) { aesni_crypt_xts(rounds, data_schedule, tweak_schedule, len, from, to, iv, 0); } int aesni_cipher_setup_common(struct aesni_session *ses, const uint8_t *key, int keylen) { + int decsched; + decsched = 1; + switch (ses->algo) { + case CRYPTO_AES_ICM: + case CRYPTO_AES_NIST_GCM_16: + decsched = 0; + /* FALLTHROUGH */ case CRYPTO_AES_CBC: switch (keylen) { case 128: ses->rounds = AES128_ROUNDS; break; case 192: ses->rounds = AES192_ROUNDS; break; case 256: ses->rounds = AES256_ROUNDS; break; default: + CRYPTDEB("invalid CBC/ICM/GCM key length"); return (EINVAL); } break; case CRYPTO_AES_XTS: switch (keylen) { case 256: ses->rounds = AES128_ROUNDS; break; case 512: ses->rounds = AES256_ROUNDS; break; default: + CRYPTDEB("invalid XTS key length"); return (EINVAL); } break; default: return (EINVAL); } aesni_set_enckey(key, ses->enc_schedule, ses->rounds); - aesni_set_deckey(ses->enc_schedule, ses->dec_schedule, ses->rounds); - if (ses->algo == CRYPTO_AES_CBC) - arc4rand(ses->iv, sizeof(ses->iv), 0); - else /* if (ses->algo == CRYPTO_AES_XTS) */ { + if (decsched) + aesni_set_deckey(ses->enc_schedule, ses->dec_schedule, + ses->rounds); + + if (ses->algo == CRYPTO_AES_XTS) aesni_set_enckey(key + keylen / 16, ses->xts_schedule, ses->rounds); - } return (0); } Index: projects/clang350-import/sys/crypto/via/padlock_hash.c =================================================================== --- projects/clang350-import/sys/crypto/via/padlock_hash.c (revision 275748) +++ projects/clang350-import/sys/crypto/via/padlock_hash.c (revision 275749) @@ -1,400 +1,400 @@ /*- * Copyright (c) 2006 Pawel Jakub Dawidek * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHORS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #include __FBSDID("$FreeBSD$"); #include #include #include #include #include #include #include #include #if defined(__amd64__) || (defined(__i386__) && !defined(PC98)) #include #include #include #include #endif #include #include #include /* for hmac_ipad_buffer and hmac_opad_buffer */ #include #include /* * Implementation notes. * * Some VIA CPUs provides SHA1 and SHA256 acceleration. * We implement all HMAC algorithms provided by crypto(9) framework, but we do * the crypto work in software unless this is HMAC/SHA1 or HMAC/SHA256 and * our CPU can accelerate it. * * Additional CPU instructions, which preform SHA1 and SHA256 are one-shot * functions - we have only one chance to give the data, CPU itself will add * the padding and calculate hash automatically. * This means, it is not possible to implement common init(), update(), final() * methods. * The way I've choosen is to keep adding data to the buffer on update() * (reallocating the buffer if necessary) and call XSHA{1,256} instruction on * final(). */ struct padlock_sha_ctx { uint8_t *psc_buf; int psc_offset; int psc_size; }; CTASSERT(sizeof(struct padlock_sha_ctx) <= sizeof(union authctx)); static void padlock_sha_init(struct padlock_sha_ctx *ctx); -static int padlock_sha_update(struct padlock_sha_ctx *ctx, uint8_t *buf, +static int padlock_sha_update(struct padlock_sha_ctx *ctx, const uint8_t *buf, uint16_t bufsize); static void padlock_sha1_final(uint8_t *hash, struct padlock_sha_ctx *ctx); static void padlock_sha256_final(uint8_t *hash, struct padlock_sha_ctx *ctx); static struct auth_hash padlock_hmac_sha1 = { CRYPTO_SHA1_HMAC, "HMAC-SHA1", 20, SHA1_HASH_LEN, SHA1_HMAC_BLOCK_LEN, sizeof(struct padlock_sha_ctx), - (void (*)(void *))padlock_sha_init, - (int (*)(void *, uint8_t *, uint16_t))padlock_sha_update, + (void (*)(void *))padlock_sha_init, NULL, NULL, + (int (*)(void *, const uint8_t *, uint16_t))padlock_sha_update, (void (*)(uint8_t *, void *))padlock_sha1_final }; static struct auth_hash padlock_hmac_sha256 = { CRYPTO_SHA2_256_HMAC, "HMAC-SHA2-256", 32, SHA2_256_HASH_LEN, SHA2_256_HMAC_BLOCK_LEN, sizeof(struct padlock_sha_ctx), - (void (*)(void *))padlock_sha_init, - (int (*)(void *, uint8_t *, uint16_t))padlock_sha_update, + (void (*)(void *))padlock_sha_init, NULL, NULL, + (int (*)(void *, const uint8_t *, uint16_t))padlock_sha_update, (void (*)(uint8_t *, void *))padlock_sha256_final }; MALLOC_DECLARE(M_PADLOCK); static __inline void padlock_output_block(uint32_t *src, uint32_t *dst, size_t count) { while (count-- > 0) *dst++ = bswap32(*src++); } static void padlock_do_sha1(const u_char *in, u_char *out, int count) { u_char buf[128+16]; /* PadLock needs at least 128 bytes buffer. */ u_char *result = PADLOCK_ALIGN(buf); ((uint32_t *)result)[0] = 0x67452301; ((uint32_t *)result)[1] = 0xEFCDAB89; ((uint32_t *)result)[2] = 0x98BADCFE; ((uint32_t *)result)[3] = 0x10325476; ((uint32_t *)result)[4] = 0xC3D2E1F0; #ifdef __GNUCLIKE_ASM __asm __volatile( ".byte 0xf3, 0x0f, 0xa6, 0xc8" /* rep xsha1 */ : "+S"(in), "+D"(result) : "c"(count), "a"(0) ); #endif padlock_output_block((uint32_t *)result, (uint32_t *)out, SHA1_HASH_LEN / sizeof(uint32_t)); } static void padlock_do_sha256(const char *in, char *out, int count) { char buf[128+16]; /* PadLock needs at least 128 bytes buffer. */ char *result = PADLOCK_ALIGN(buf); ((uint32_t *)result)[0] = 0x6A09E667; ((uint32_t *)result)[1] = 0xBB67AE85; ((uint32_t *)result)[2] = 0x3C6EF372; ((uint32_t *)result)[3] = 0xA54FF53A; ((uint32_t *)result)[4] = 0x510E527F; ((uint32_t *)result)[5] = 0x9B05688C; ((uint32_t *)result)[6] = 0x1F83D9AB; ((uint32_t *)result)[7] = 0x5BE0CD19; #ifdef __GNUCLIKE_ASM __asm __volatile( ".byte 0xf3, 0x0f, 0xa6, 0xd0" /* rep xsha256 */ : "+S"(in), "+D"(result) : "c"(count), "a"(0) ); #endif padlock_output_block((uint32_t *)result, (uint32_t *)out, SHA2_256_HASH_LEN / sizeof(uint32_t)); } static void padlock_sha_init(struct padlock_sha_ctx *ctx) { ctx->psc_buf = NULL; ctx->psc_offset = 0; ctx->psc_size = 0; } static int -padlock_sha_update(struct padlock_sha_ctx *ctx, uint8_t *buf, uint16_t bufsize) +padlock_sha_update(struct padlock_sha_ctx *ctx, const uint8_t *buf, uint16_t bufsize) { if (ctx->psc_size - ctx->psc_offset < bufsize) { ctx->psc_size = MAX(ctx->psc_size * 2, ctx->psc_size + bufsize); ctx->psc_buf = realloc(ctx->psc_buf, ctx->psc_size, M_PADLOCK, M_NOWAIT); if(ctx->psc_buf == NULL) return (ENOMEM); } bcopy(buf, ctx->psc_buf + ctx->psc_offset, bufsize); ctx->psc_offset += bufsize; return (0); } static void padlock_sha_free(struct padlock_sha_ctx *ctx) { if (ctx->psc_buf != NULL) { //bzero(ctx->psc_buf, ctx->psc_size); free(ctx->psc_buf, M_PADLOCK); ctx->psc_buf = NULL; ctx->psc_offset = 0; ctx->psc_size = 0; } } static void padlock_sha1_final(uint8_t *hash, struct padlock_sha_ctx *ctx) { padlock_do_sha1(ctx->psc_buf, hash, ctx->psc_offset); padlock_sha_free(ctx); } static void padlock_sha256_final(uint8_t *hash, struct padlock_sha_ctx *ctx) { padlock_do_sha256(ctx->psc_buf, hash, ctx->psc_offset); padlock_sha_free(ctx); } static void padlock_copy_ctx(struct auth_hash *axf, void *sctx, void *dctx) { if ((via_feature_xcrypt & VIA_HAS_SHA) != 0 && (axf->type == CRYPTO_SHA1_HMAC || axf->type == CRYPTO_SHA2_256_HMAC)) { struct padlock_sha_ctx *spctx = sctx, *dpctx = dctx; dpctx->psc_offset = spctx->psc_offset; dpctx->psc_size = spctx->psc_size; dpctx->psc_buf = malloc(dpctx->psc_size, M_PADLOCK, M_WAITOK); bcopy(spctx->psc_buf, dpctx->psc_buf, dpctx->psc_size); } else { bcopy(sctx, dctx, axf->ctxsize); } } static void padlock_free_ctx(struct auth_hash *axf, void *ctx) { if ((via_feature_xcrypt & VIA_HAS_SHA) != 0 && (axf->type == CRYPTO_SHA1_HMAC || axf->type == CRYPTO_SHA2_256_HMAC)) { padlock_sha_free(ctx); } } static void padlock_hash_key_setup(struct padlock_session *ses, caddr_t key, int klen) { struct auth_hash *axf; int i; klen /= 8; axf = ses->ses_axf; /* * Try to free contexts before using them, because * padlock_hash_key_setup() can be called twice - once from * padlock_newsession() and again from padlock_process(). */ padlock_free_ctx(axf, ses->ses_ictx); padlock_free_ctx(axf, ses->ses_octx); for (i = 0; i < klen; i++) key[i] ^= HMAC_IPAD_VAL; axf->Init(ses->ses_ictx); axf->Update(ses->ses_ictx, key, klen); axf->Update(ses->ses_ictx, hmac_ipad_buffer, axf->blocksize - klen); for (i = 0; i < klen; i++) key[i] ^= (HMAC_IPAD_VAL ^ HMAC_OPAD_VAL); axf->Init(ses->ses_octx); axf->Update(ses->ses_octx, key, klen); axf->Update(ses->ses_octx, hmac_opad_buffer, axf->blocksize - klen); for (i = 0; i < klen; i++) key[i] ^= HMAC_OPAD_VAL; } /* * Compute keyed-hash authenticator. */ static int padlock_authcompute(struct padlock_session *ses, struct cryptodesc *crd, caddr_t buf, int flags) { u_char hash[HASH_MAX_LEN]; struct auth_hash *axf; union authctx ctx; int error; axf = ses->ses_axf; padlock_copy_ctx(axf, ses->ses_ictx, &ctx); error = crypto_apply(flags, buf, crd->crd_skip, crd->crd_len, (int (*)(void *, void *, unsigned int))axf->Update, (caddr_t)&ctx); if (error != 0) { padlock_free_ctx(axf, &ctx); return (error); } axf->Final(hash, &ctx); padlock_copy_ctx(axf, ses->ses_octx, &ctx); axf->Update(&ctx, hash, axf->hashsize); axf->Final(hash, &ctx); /* Inject the authentication data */ crypto_copyback(flags, buf, crd->crd_inject, ses->ses_mlen == 0 ? axf->hashsize : ses->ses_mlen, hash); return (0); } int padlock_hash_setup(struct padlock_session *ses, struct cryptoini *macini) { ses->ses_mlen = macini->cri_mlen; /* Find software structure which describes HMAC algorithm. */ switch (macini->cri_alg) { case CRYPTO_NULL_HMAC: ses->ses_axf = &auth_hash_null; break; case CRYPTO_MD5_HMAC: ses->ses_axf = &auth_hash_hmac_md5; break; case CRYPTO_SHA1_HMAC: if ((via_feature_xcrypt & VIA_HAS_SHA) != 0) ses->ses_axf = &padlock_hmac_sha1; else ses->ses_axf = &auth_hash_hmac_sha1; break; case CRYPTO_RIPEMD160_HMAC: ses->ses_axf = &auth_hash_hmac_ripemd_160; break; case CRYPTO_SHA2_256_HMAC: if ((via_feature_xcrypt & VIA_HAS_SHA) != 0) ses->ses_axf = &padlock_hmac_sha256; else ses->ses_axf = &auth_hash_hmac_sha2_256; break; case CRYPTO_SHA2_384_HMAC: ses->ses_axf = &auth_hash_hmac_sha2_384; break; case CRYPTO_SHA2_512_HMAC: ses->ses_axf = &auth_hash_hmac_sha2_512; break; } /* Allocate memory for HMAC inner and outer contexts. */ ses->ses_ictx = malloc(ses->ses_axf->ctxsize, M_PADLOCK, M_ZERO | M_NOWAIT); ses->ses_octx = malloc(ses->ses_axf->ctxsize, M_PADLOCK, M_ZERO | M_NOWAIT); if (ses->ses_ictx == NULL || ses->ses_octx == NULL) return (ENOMEM); /* Setup key if given. */ if (macini->cri_key != NULL) { padlock_hash_key_setup(ses, macini->cri_key, macini->cri_klen); } return (0); } int padlock_hash_process(struct padlock_session *ses, struct cryptodesc *maccrd, struct cryptop *crp) { struct thread *td; int error; td = curthread; error = fpu_kern_enter(td, ses->ses_fpu_ctx, FPU_KERN_NORMAL | FPU_KERN_KTHR); if (error != 0) return (error); if ((maccrd->crd_flags & CRD_F_KEY_EXPLICIT) != 0) padlock_hash_key_setup(ses, maccrd->crd_key, maccrd->crd_klen); error = padlock_authcompute(ses, maccrd, crp->crp_buf, crp->crp_flags); fpu_kern_leave(td, ses->ses_fpu_ctx); return (error); } void padlock_hash_free(struct padlock_session *ses) { if (ses->ses_ictx != NULL) { padlock_free_ctx(ses->ses_axf, ses->ses_ictx); bzero(ses->ses_ictx, ses->ses_axf->ctxsize); free(ses->ses_ictx, M_PADLOCK); ses->ses_ictx = NULL; } if (ses->ses_octx != NULL) { padlock_free_ctx(ses->ses_axf, ses->ses_octx); bzero(ses->ses_octx, ses->ses_axf->ctxsize); free(ses->ses_octx, M_PADLOCK); ses->ses_octx = NULL; } } Index: projects/clang350-import/sys/dev/beri/virtio/virtio_mmio_platform.c =================================================================== --- projects/clang350-import/sys/dev/beri/virtio/virtio_mmio_platform.c (revision 275748) +++ projects/clang350-import/sys/dev/beri/virtio/virtio_mmio_platform.c (revision 275749) @@ -1,256 +1,266 @@ /*- * Copyright (c) 2014 Ruslan Bukin * All rights reserved. * * This software was developed by SRI International and the University of * Cambridge Computer Laboratory under DARPA/AFRL contract (FA8750-10-C-0237) * ("CTSRD"), as part of the DARPA CRASH research programme. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ /* * BERI interface for Virtio MMIO bus. * * This driver provides interrupt-engine for software-implemented * Virtio MMIO backend. */ #include __FBSDID("$FreeBSD$"); #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include "virtio_mmio_if.h" #include "pio_if.h" static void platform_intr(void *arg); struct virtio_mmio_platform_softc { struct resource *res[1]; bus_space_tag_t bst; bus_space_handle_t bsh; device_t dev; void (*intr_handler)(void *); void *ih_user; device_t pio_recv; device_t pio_send; }; static int setup_pio(struct virtio_mmio_platform_softc *sc, char *name, device_t *dev) { phandle_t pio_node; struct fdt_ic *ic; phandle_t xref; phandle_t node; if ((node = ofw_bus_get_node(sc->dev)) == -1) return (ENXIO); if (OF_searchencprop(node, name, &xref, sizeof(xref)) == -1) { return (ENXIO); } pio_node = OF_node_from_xref(xref); SLIST_FOREACH(ic, &fdt_ic_list_head, fdt_ics) { if (ic->iph == pio_node) { *dev = ic->dev; PIO_CONFIGURE(*dev, PIO_OUT_ALL, PIO_UNMASK_ALL); return (0); } } return (ENXIO); } static int virtio_mmio_platform_probe(device_t dev) { if (!ofw_bus_status_okay(dev)) return (ENXIO); if (!ofw_bus_is_compatible(dev, "beri,virtio_mmio_platform")) return (ENXIO); device_set_desc(dev, "Virtio MMIO platform"); return (BUS_PROBE_DEFAULT); } static int virtio_mmio_platform_attach(device_t dev) { struct virtio_mmio_platform_softc *sc; struct fdt_ic *fic; phandle_t node; sc = device_get_softc(dev); sc->dev = dev; if (setup_pio(sc, "pio-send", &sc->pio_send) != 0) return (ENXIO); if (setup_pio(sc, "pio-recv", &sc->pio_recv) != 0) return (ENXIO); if ((node = ofw_bus_get_node(sc->dev)) == -1) return (ENXIO); fic = malloc(sizeof(*fic), M_DEVBUF, M_WAITOK|M_ZERO); fic->iph = node; fic->dev = dev; SLIST_INSERT_HEAD(&fdt_ic_list_head, fic, fdt_ics); return (0); } static int platform_note(device_t dev, size_t offset, int val) { struct virtio_mmio_platform_softc *sc; int note; int i; sc = device_get_softc(dev); switch (offset) { case (VIRTIO_MMIO_QUEUE_NOTIFY): if (val == 0) note = Q_NOTIFY; else if (val == 1) note = Q_NOTIFY1; break; case (VIRTIO_MMIO_QUEUE_PFN): note = Q_PFN; break; case (VIRTIO_MMIO_QUEUE_SEL): note = Q_SEL; break; default: note = 0; } if (note) { mips_dcache_wbinv_all(); PIO_SET(sc->pio_send, note, 1); /* * Wait until host ack the request. * Usually done within few cycles. * TODO: bad */ for (i = 100; i > 0; i--) { if (PIO_READ(sc->pio_send) == 0) break; } if (i == 0) device_printf(sc->dev, "Warning: host busy\n"); } return (0); } static void platform_intr(void *arg) { struct virtio_mmio_platform_softc *sc; int reg; sc = arg; /* Read pending */ reg = PIO_READ(sc->pio_recv); /* Ack */ PIO_SET(sc->pio_recv, reg, 0); /* Writeback, invalidate cache */ mips_dcache_wbinv_all(); if (sc->intr_handler != NULL) sc->intr_handler(sc->ih_user); } static int platform_setup_intr(device_t dev, device_t mmio_dev, void *intr_handler, void *ih_user) { struct virtio_mmio_platform_softc *sc; sc = device_get_softc(dev); sc->intr_handler = intr_handler; sc->ih_user = ih_user; PIO_SETUP_IRQ(sc->pio_recv, platform_intr, sc); return (0); } +static int +platform_poll(device_t dev) +{ + + mips_dcache_wbinv_all(); + + return (0); +} + static device_method_t virtio_mmio_platform_methods[] = { DEVMETHOD(device_probe, virtio_mmio_platform_probe), DEVMETHOD(device_attach, virtio_mmio_platform_attach), /* virtio_mmio_if.h */ DEVMETHOD(virtio_mmio_note, platform_note), + DEVMETHOD(virtio_mmio_poll, platform_poll), DEVMETHOD(virtio_mmio_setup_intr, platform_setup_intr), DEVMETHOD_END }; static driver_t virtio_mmio_platform_driver = { "virtio_mmio_platform", virtio_mmio_platform_methods, sizeof(struct virtio_mmio_platform_softc), }; static devclass_t virtio_mmio_platform_devclass; DRIVER_MODULE(virtio_mmio_platform, simplebus, virtio_mmio_platform_driver, virtio_mmio_platform_devclass, 0, 0); Index: projects/clang350-import/sys/dev/cxgbe/adapter.h =================================================================== --- projects/clang350-import/sys/dev/cxgbe/adapter.h (revision 275748) +++ projects/clang350-import/sys/dev/cxgbe/adapter.h (revision 275749) @@ -1,1083 +1,1084 @@ /*- * Copyright (c) 2011 Chelsio Communications, Inc. * All rights reserved. * Written by: Navdeep Parhar * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * $FreeBSD$ * */ #ifndef __T4_ADAPTER_H__ #define __T4_ADAPTER_H__ #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include "offload.h" #include "common/t4_msg.h" #include "firmware/t4fw_interface.h" +#define KTR_CXGBE KTR_SPARE3 MALLOC_DECLARE(M_CXGBE); #define CXGBE_UNIMPLEMENTED(s) \ panic("%s (%s, line %d) not implemented yet.", s, __FILE__, __LINE__) #if defined(__i386__) || defined(__amd64__) static __inline void prefetch(void *x) { __asm volatile("prefetcht0 %0" :: "m" (*(unsigned long *)x)); } #else #define prefetch(x) #endif #ifndef SYSCTL_ADD_UQUAD #define SYSCTL_ADD_UQUAD SYSCTL_ADD_QUAD #define sysctl_handle_64 sysctl_handle_quad #define CTLTYPE_U64 CTLTYPE_QUAD #endif #if (__FreeBSD_version >= 900030) || \ ((__FreeBSD_version >= 802507) && (__FreeBSD_version < 900000)) #define SBUF_DRAIN 1 #endif #ifdef __amd64__ /* XXX: need systemwide bus_space_read_8/bus_space_write_8 */ static __inline uint64_t t4_bus_space_read_8(bus_space_tag_t tag, bus_space_handle_t handle, bus_size_t offset) { KASSERT(tag == X86_BUS_SPACE_MEM, ("%s: can only handle mem space", __func__)); return (*(volatile uint64_t *)(handle + offset)); } static __inline void t4_bus_space_write_8(bus_space_tag_t tag, bus_space_handle_t bsh, bus_size_t offset, uint64_t value) { KASSERT(tag == X86_BUS_SPACE_MEM, ("%s: can only handle mem space", __func__)); *(volatile uint64_t *)(bsh + offset) = value; } #else static __inline uint64_t t4_bus_space_read_8(bus_space_tag_t tag, bus_space_handle_t handle, bus_size_t offset) { return (uint64_t)bus_space_read_4(tag, handle, offset) + ((uint64_t)bus_space_read_4(tag, handle, offset + 4) << 32); } static __inline void t4_bus_space_write_8(bus_space_tag_t tag, bus_space_handle_t bsh, bus_size_t offset, uint64_t value) { bus_space_write_4(tag, bsh, offset, value); bus_space_write_4(tag, bsh, offset + 4, value >> 32); } #endif struct adapter; typedef struct adapter adapter_t; enum { /* * All ingress queues use this entry size. Note that the firmware event * queue and any iq expecting CPL_RX_PKT in the descriptor needs this to * be at least 64. */ IQ_ESIZE = 64, /* Default queue sizes for all kinds of ingress queues */ FW_IQ_QSIZE = 256, RX_IQ_QSIZE = 1024, /* All egress queues use this entry size */ EQ_ESIZE = 64, /* Default queue sizes for all kinds of egress queues */ CTRL_EQ_QSIZE = 128, TX_EQ_QSIZE = 1024, #if MJUMPAGESIZE != MCLBYTES SW_ZONE_SIZES = 4, /* cluster, jumbop, jumbo9k, jumbo16k */ #else SW_ZONE_SIZES = 3, /* cluster, jumbo9k, jumbo16k */ #endif CL_METADATA_SIZE = CACHE_LINE_SIZE, SGE_MAX_WR_NDESC = SGE_MAX_WR_LEN / EQ_ESIZE, /* max WR size in desc */ TX_SGL_SEGS = 36, TX_WR_FLITS = SGE_MAX_WR_LEN / 8 }; enum { /* adapter intr_type */ INTR_INTX = (1 << 0), INTR_MSI = (1 << 1), INTR_MSIX = (1 << 2) }; enum { XGMAC_MTU = (1 << 0), XGMAC_PROMISC = (1 << 1), XGMAC_ALLMULTI = (1 << 2), XGMAC_VLANEX = (1 << 3), XGMAC_UCADDR = (1 << 4), XGMAC_MCADDRS = (1 << 5), XGMAC_ALL = 0xffff }; enum { /* flags understood by begin_synchronized_op */ HOLD_LOCK = (1 << 0), SLEEP_OK = (1 << 1), INTR_OK = (1 << 2), /* flags understood by end_synchronized_op */ LOCK_HELD = HOLD_LOCK, }; enum { /* adapter flags */ FULL_INIT_DONE = (1 << 0), FW_OK = (1 << 1), /* INTR_DIRECT = (1 << 2), No longer used. */ MASTER_PF = (1 << 3), ADAP_SYSCTL_CTX = (1 << 4), TOM_INIT_DONE = (1 << 5), BUF_PACKING_OK = (1 << 6), CXGBE_BUSY = (1 << 9), /* port flags */ DOOMED = (1 << 0), PORT_INIT_DONE = (1 << 1), PORT_SYSCTL_CTX = (1 << 2), HAS_TRACEQ = (1 << 3), INTR_RXQ = (1 << 4), /* All NIC rxq's take interrupts */ INTR_OFLD_RXQ = (1 << 5), /* All TOE rxq's take interrupts */ INTR_NM_RXQ = (1 << 6), /* All netmap rxq's take interrupts */ INTR_ALL = (INTR_RXQ | INTR_OFLD_RXQ | INTR_NM_RXQ), }; #define IS_DOOMED(pi) ((pi)->flags & DOOMED) #define SET_DOOMED(pi) do {(pi)->flags |= DOOMED;} while (0) #define IS_BUSY(sc) ((sc)->flags & CXGBE_BUSY) #define SET_BUSY(sc) do {(sc)->flags |= CXGBE_BUSY;} while (0) #define CLR_BUSY(sc) do {(sc)->flags &= ~CXGBE_BUSY;} while (0) struct port_info { device_t dev; struct adapter *adapter; struct ifnet *ifp; struct ifmedia media; struct mtx pi_lock; char lockname[16]; unsigned long flags; int if_flags; uint16_t *rss; uint16_t viid; int16_t xact_addr_filt;/* index of exact MAC address filter */ uint16_t rss_size; /* size of VI's RSS table slice */ uint8_t lport; /* associated offload logical port */ int8_t mdio_addr; uint8_t port_type; uint8_t mod_type; uint8_t port_id; uint8_t tx_chan; uint8_t rx_chan_map; /* rx MPS channel bitmap */ /* These need to be int as they are used in sysctl */ int ntxq; /* # of tx queues */ int first_txq; /* index of first tx queue */ int rsrv_noflowq; /* Reserve queue 0 for non-flowid packets */ int nrxq; /* # of rx queues */ int first_rxq; /* index of first rx queue */ #ifdef TCP_OFFLOAD int nofldtxq; /* # of offload tx queues */ int first_ofld_txq; /* index of first offload tx queue */ int nofldrxq; /* # of offload rx queues */ int first_ofld_rxq; /* index of first offload rx queue */ #endif #ifdef DEV_NETMAP int nnmtxq; /* # of netmap tx queues */ int first_nm_txq; /* index of first netmap tx queue */ int nnmrxq; /* # of netmap rx queues */ int first_nm_rxq; /* index of first netmap rx queue */ struct ifnet *nm_ifp; struct ifmedia nm_media; int nmif_flags; uint16_t nm_viid; int16_t nm_xact_addr_filt; uint16_t nm_rss_size; /* size of netmap VI's RSS table slice */ #endif int tmr_idx; int pktc_idx; int qsize_rxq; int qsize_txq; int linkdnrc; struct link_config link_cfg; struct timeval last_refreshed; struct port_stats stats; u_int tnl_cong_drops; eventhandler_tag vlan_c; struct callout tick; struct sysctl_ctx_list ctx; /* from ifconfig up to driver detach */ uint8_t hw_addr[ETHER_ADDR_LEN]; /* factory MAC address, won't change */ }; /* Where the cluster came from, how it has been carved up. */ struct cluster_layout { int8_t zidx; int8_t hwidx; uint16_t region1; /* mbufs laid out within this region */ /* region2 is the DMA region */ uint16_t region3; /* cluster_metadata within this region */ }; struct cluster_metadata { u_int refcount; #ifdef INVARIANTS struct fl_sdesc *sd; /* For debug only. Could easily be stale */ #endif }; struct fl_sdesc { caddr_t cl; uint16_t nmbuf; /* # of driver originated mbufs with ref on cluster */ struct cluster_layout cll; }; struct tx_desc { __be64 flit[8]; }; struct tx_map { struct mbuf *m; bus_dmamap_t map; }; /* DMA maps used for tx */ struct tx_maps { struct tx_map *maps; uint32_t map_total; /* # of DMA maps */ uint32_t map_pidx; /* next map to be used */ uint32_t map_cidx; /* reclaimed up to this index */ uint32_t map_avail; /* # of available maps */ }; struct tx_sdesc { uint8_t desc_used; /* # of hardware descriptors used by the WR */ uint8_t credits; /* NIC txq: # of frames sent out in the WR */ }; #define IQ_PAD (IQ_ESIZE - sizeof(struct rsp_ctrl) - sizeof(struct rss_header)) struct iq_desc { struct rss_header rss; uint8_t cpl[IQ_PAD]; struct rsp_ctrl rsp; }; #undef IQ_PAD CTASSERT(sizeof(struct iq_desc) == IQ_ESIZE); enum { /* iq flags */ IQ_ALLOCATED = (1 << 0), /* firmware resources allocated */ IQ_HAS_FL = (1 << 1), /* iq associated with a freelist */ IQ_INTR = (1 << 2), /* iq takes direct interrupt */ IQ_LRO_ENABLED = (1 << 3), /* iq is an eth rxq with LRO enabled */ /* iq state */ IQS_DISABLED = 0, IQS_BUSY = 1, IQS_IDLE = 2, }; /* * Ingress Queue: T4 is producer, driver is consumer. */ struct sge_iq { uint32_t flags; volatile int state; struct adapter *adapter; struct iq_desc *desc; /* KVA of descriptor ring */ int8_t intr_pktc_idx; /* packet count threshold index */ uint8_t gen; /* generation bit */ uint8_t intr_params; /* interrupt holdoff parameters */ uint8_t intr_next; /* XXX: holdoff for next interrupt */ uint16_t qsize; /* size (# of entries) of the queue */ uint16_t sidx; /* index of the entry with the status page */ uint16_t cidx; /* consumer index */ uint16_t cntxt_id; /* SGE context id for the iq */ uint16_t abs_id; /* absolute SGE id for the iq */ STAILQ_ENTRY(sge_iq) link; bus_dma_tag_t desc_tag; bus_dmamap_t desc_map; bus_addr_t ba; /* bus address of descriptor ring */ }; enum { EQ_CTRL = 1, EQ_ETH = 2, #ifdef TCP_OFFLOAD EQ_OFLD = 3, #endif /* eq flags */ EQ_TYPEMASK = 7, /* 3 lsbits hold the type */ EQ_ALLOCATED = (1 << 3), /* firmware resources allocated */ EQ_DOOMED = (1 << 4), /* about to be destroyed */ EQ_CRFLUSHED = (1 << 5), /* expecting an update from SGE */ EQ_STALLED = (1 << 6), /* out of hw descriptors or dmamaps */ }; /* Listed in order of preference. Update t4_sysctls too if you change these */ enum {DOORBELL_UDB, DOORBELL_WCWR, DOORBELL_UDBWC, DOORBELL_KDB}; /* * Egress Queue: driver is producer, T4 is consumer. * * Note: A free list is an egress queue (driver produces the buffers and T4 * consumes them) but it's special enough to have its own struct (see sge_fl). */ struct sge_eq { unsigned int flags; /* MUST be first */ unsigned int cntxt_id; /* SGE context id for the eq */ bus_dma_tag_t desc_tag; bus_dmamap_t desc_map; char lockname[16]; struct mtx eq_lock; struct tx_desc *desc; /* KVA of descriptor ring */ bus_addr_t ba; /* bus address of descriptor ring */ struct sge_qstat *spg; /* status page, for convenience */ uint16_t doorbells; volatile uint32_t *udb; /* KVA of doorbell (lies within BAR2) */ u_int udb_qid; /* relative qid within the doorbell page */ uint16_t cap; /* max # of desc, for convenience */ uint16_t avail; /* available descriptors, for convenience */ uint16_t qsize; /* size (# of entries) of the queue */ uint16_t cidx; /* consumer idx (desc idx) */ uint16_t pidx; /* producer idx (desc idx) */ uint16_t pending; /* # of descriptors used since last doorbell */ uint16_t iqid; /* iq that gets egr_update for the eq */ uint8_t tx_chan; /* tx channel used by the eq */ struct task tx_task; struct callout tx_callout; /* stats */ uint32_t egr_update; /* # of SGE_EGR_UPDATE notifications for eq */ uint32_t unstalled; /* recovered from stall */ }; struct sw_zone_info { uma_zone_t zone; /* zone that this cluster comes from */ int size; /* size of cluster: 2K, 4K, 9K, 16K, etc. */ int type; /* EXT_xxx type of the cluster */ int8_t head_hwidx; int8_t tail_hwidx; }; struct hw_buf_info { int8_t zidx; /* backpointer to zone; -ve means unused */ int8_t next; /* next hwidx for this zone; -1 means no more */ int size; }; enum { FL_STARVING = (1 << 0), /* on the adapter's list of starving fl's */ FL_DOOMED = (1 << 1), /* about to be destroyed */ FL_BUF_PACKING = (1 << 2), /* buffer packing enabled */ FL_BUF_RESUME = (1 << 3), /* resume from the middle of the frame */ }; #define FL_RUNNING_LOW(fl) \ (IDXDIFF(fl->dbidx * 8, fl->cidx, fl->sidx * 8) <= fl->lowat) #define FL_NOT_RUNNING_LOW(fl) \ (IDXDIFF(fl->dbidx * 8, fl->cidx, fl->sidx * 8) >= 2 * fl->lowat) struct sge_fl { struct mtx fl_lock; __be64 *desc; /* KVA of descriptor ring, ptr to addresses */ struct fl_sdesc *sdesc; /* KVA of software descriptor ring */ struct cluster_layout cll_def; /* default refill zone, layout */ uint16_t lowat; /* # of buffers <= this means fl needs help */ int flags; uint16_t buf_boundary; /* The 16b idx all deal with hw descriptors */ uint16_t dbidx; /* hw pidx after last doorbell */ uint16_t sidx; /* index of status page */ volatile uint16_t hw_cidx; /* The 32b idx are all buffer idx, not hardware descriptor idx */ uint32_t cidx; /* consumer index */ uint32_t pidx; /* producer index */ uint32_t dbval; u_int rx_offset; /* offset in fl buf (when buffer packing) */ volatile uint32_t *udb; uint64_t mbuf_allocated;/* # of mbuf allocated from zone_mbuf */ uint64_t mbuf_inlined; /* # of mbuf created within clusters */ uint64_t cl_allocated; /* # of clusters allocated */ uint64_t cl_recycled; /* # of clusters recycled */ uint64_t cl_fast_recycled; /* # of clusters recycled (fast) */ /* These 3 are valid when FL_BUF_RESUME is set, stale otherwise. */ struct mbuf *m0; struct mbuf **pnext; u_int remaining; uint16_t qsize; /* # of hw descriptors (status page included) */ uint16_t cntxt_id; /* SGE context id for the freelist */ TAILQ_ENTRY(sge_fl) link; /* All starving freelists */ bus_dma_tag_t desc_tag; bus_dmamap_t desc_map; char lockname[16]; bus_addr_t ba; /* bus address of descriptor ring */ struct cluster_layout cll_alt; /* alternate refill zone, layout */ }; /* txq: SGE egress queue + what's needed for Ethernet NIC */ struct sge_txq { struct sge_eq eq; /* MUST be first */ struct ifnet *ifp; /* the interface this txq belongs to */ bus_dma_tag_t tx_tag; /* tag for transmit buffers */ struct buf_ring *br; /* tx buffer ring */ struct tx_sdesc *sdesc; /* KVA of software descriptor ring */ struct mbuf *m; /* held up due to temporary resource shortage */ struct tx_maps txmaps; /* stats for common events first */ uint64_t txcsum; /* # of times hardware assisted with checksum */ uint64_t tso_wrs; /* # of TSO work requests */ uint64_t vlan_insertion;/* # of times VLAN tag was inserted */ uint64_t imm_wrs; /* # of work requests with immediate data */ uint64_t sgl_wrs; /* # of work requests with direct SGL */ uint64_t txpkt_wrs; /* # of txpkt work requests (not coalesced) */ uint64_t txpkts_wrs; /* # of coalesced tx work requests */ uint64_t txpkts_pkts; /* # of frames in coalesced tx work requests */ /* stats for not-that-common events */ uint32_t no_dmamap; /* no DMA map to load the mbuf */ uint32_t no_desc; /* out of hardware descriptors */ } __aligned(CACHE_LINE_SIZE); /* rxq: SGE ingress queue + SGE free list + miscellaneous items */ struct sge_rxq { struct sge_iq iq; /* MUST be first */ struct sge_fl fl; /* MUST follow iq */ struct ifnet *ifp; /* the interface this rxq belongs to */ #if defined(INET) || defined(INET6) struct lro_ctrl lro; /* LRO state */ #endif /* stats for common events first */ uint64_t rxcsum; /* # of times hardware assisted with checksum */ uint64_t vlan_extraction;/* # of times VLAN tag was extracted */ /* stats for not-that-common events */ } __aligned(CACHE_LINE_SIZE); static inline struct sge_rxq * iq_to_rxq(struct sge_iq *iq) { return (__containerof(iq, struct sge_rxq, iq)); } #ifdef TCP_OFFLOAD /* ofld_rxq: SGE ingress queue + SGE free list + miscellaneous items */ struct sge_ofld_rxq { struct sge_iq iq; /* MUST be first */ struct sge_fl fl; /* MUST follow iq */ } __aligned(CACHE_LINE_SIZE); static inline struct sge_ofld_rxq * iq_to_ofld_rxq(struct sge_iq *iq) { return (__containerof(iq, struct sge_ofld_rxq, iq)); } #endif struct wrqe { STAILQ_ENTRY(wrqe) link; struct sge_wrq *wrq; int wr_len; uint64_t wr[] __aligned(16); }; /* * wrq: SGE egress queue that is given prebuilt work requests. Both the control * and offload tx queues are of this type. */ struct sge_wrq { struct sge_eq eq; /* MUST be first */ struct adapter *adapter; /* List of WRs held up due to lack of tx descriptors */ STAILQ_HEAD(, wrqe) wr_list; /* stats for common events first */ uint64_t tx_wrs; /* # of tx work requests */ /* stats for not-that-common events */ uint32_t no_desc; /* out of hardware descriptors */ } __aligned(CACHE_LINE_SIZE); #ifdef DEV_NETMAP struct sge_nm_rxq { struct port_info *pi; struct iq_desc *iq_desc; uint16_t iq_abs_id; uint16_t iq_cntxt_id; uint16_t iq_cidx; uint16_t iq_sidx; uint8_t iq_gen; __be64 *fl_desc; uint16_t fl_cntxt_id; uint32_t fl_cidx; uint32_t fl_pidx; uint32_t fl_sidx; uint32_t fl_db_val; u_int fl_hwidx:4; u_int nid; /* netmap ring # for this queue */ /* infrequently used items after this */ bus_dma_tag_t iq_desc_tag; bus_dmamap_t iq_desc_map; bus_addr_t iq_ba; int intr_idx; bus_dma_tag_t fl_desc_tag; bus_dmamap_t fl_desc_map; bus_addr_t fl_ba; } __aligned(CACHE_LINE_SIZE); struct sge_nm_txq { struct tx_desc *desc; uint16_t cidx; uint16_t pidx; uint16_t sidx; uint16_t equiqidx; /* EQUIQ last requested at this pidx */ uint16_t equeqidx; /* EQUEQ last requested at this pidx */ uint16_t dbidx; /* pidx of the most recent doorbell */ uint16_t doorbells; volatile uint32_t *udb; u_int udb_qid; u_int cntxt_id; __be32 cpl_ctrl0; /* for convenience */ u_int nid; /* netmap ring # for this queue */ /* infrequently used items after this */ bus_dma_tag_t desc_tag; bus_dmamap_t desc_map; bus_addr_t ba; int iqidx; } __aligned(CACHE_LINE_SIZE); #endif struct sge { int timer_val[SGE_NTIMERS]; int counter_val[SGE_NCOUNTERS]; int fl_starve_threshold; int fl_starve_threshold2; int eq_s_qpp; int iq_s_qpp; int nrxq; /* total # of Ethernet rx queues */ int ntxq; /* total # of Ethernet tx tx queues */ #ifdef TCP_OFFLOAD int nofldrxq; /* total # of TOE rx queues */ int nofldtxq; /* total # of TOE tx queues */ #endif #ifdef DEV_NETMAP int nnmrxq; /* total # of netmap rx queues */ int nnmtxq; /* total # of netmap tx queues */ #endif int niq; /* total # of ingress queues */ int neq; /* total # of egress queues */ struct sge_iq fwq; /* Firmware event queue */ struct sge_wrq mgmtq; /* Management queue (control queue) */ struct sge_wrq *ctrlq; /* Control queues */ struct sge_txq *txq; /* NIC tx queues */ struct sge_rxq *rxq; /* NIC rx queues */ #ifdef TCP_OFFLOAD struct sge_wrq *ofld_txq; /* TOE tx queues */ struct sge_ofld_rxq *ofld_rxq; /* TOE rx queues */ #endif #ifdef DEV_NETMAP struct sge_nm_txq *nm_txq; /* netmap tx queues */ struct sge_nm_rxq *nm_rxq; /* netmap rx queues */ #endif uint16_t iq_start; int eq_start; struct sge_iq **iqmap; /* iq->cntxt_id to iq mapping */ struct sge_eq **eqmap; /* eq->cntxt_id to eq mapping */ int pad_boundary; int pack_boundary; int8_t safe_hwidx1; /* may not have room for metadata */ int8_t safe_hwidx2; /* with room for metadata and maybe more */ struct sw_zone_info sw_zone_info[SW_ZONE_SIZES]; struct hw_buf_info hw_buf_info[SGE_FLBUF_SIZES]; }; struct rss_header; typedef int (*cpl_handler_t)(struct sge_iq *, const struct rss_header *, struct mbuf *); typedef int (*an_handler_t)(struct sge_iq *, const struct rsp_ctrl *); typedef int (*fw_msg_handler_t)(struct adapter *, const __be64 *); struct adapter { SLIST_ENTRY(adapter) link; device_t dev; struct cdev *cdev; /* PCIe register resources */ int regs_rid; struct resource *regs_res; int msix_rid; struct resource *msix_res; bus_space_handle_t bh; bus_space_tag_t bt; bus_size_t mmio_len; int udbs_rid; struct resource *udbs_res; volatile uint8_t *udbs_base; unsigned int pf; unsigned int mbox; /* Interrupt information */ int intr_type; int intr_count; struct irq { struct resource *res; int rid; void *tag; } *irq; bus_dma_tag_t dmat; /* Parent DMA tag */ struct sge sge; int lro_timeout; struct taskqueue *tq[NCHAN]; /* taskqueues that flush data out */ struct port_info *port[MAX_NPORTS]; uint8_t chan_map[NCHAN]; #ifdef TCP_OFFLOAD void *tom_softc; /* (struct tom_data *) */ struct tom_tunables tt; void *iwarp_softc; /* (struct c4iw_dev *) */ void *iscsi_softc; #endif struct l2t_data *l2t; /* L2 table */ struct tid_info tids; uint16_t doorbells; int open_device_map; #ifdef TCP_OFFLOAD int offload_map; #endif int flags; char ifp_lockname[16]; struct mtx ifp_lock; struct ifnet *ifp; /* tracer ifp */ struct ifmedia media; int traceq; /* iq used by all tracers, -1 if none */ int tracer_valid; /* bitmap of valid tracers */ int tracer_enabled; /* bitmap of enabled tracers */ char fw_version[32]; char cfg_file[32]; u_int cfcsum; struct adapter_params params; struct t4_virt_res vres; uint16_t linkcaps; uint16_t niccaps; uint16_t toecaps; uint16_t rdmacaps; uint16_t iscsicaps; uint16_t fcoecaps; struct sysctl_ctx_list ctx; /* from adapter_full_init to full_uninit */ struct mtx sc_lock; char lockname[16]; /* Starving free lists */ struct mtx sfl_lock; /* same cache-line as sc_lock? but that's ok */ TAILQ_HEAD(, sge_fl) sfl; struct callout sfl_callout; struct mtx regwin_lock; /* for indirect reads and memory windows */ an_handler_t an_handler __aligned(CACHE_LINE_SIZE); fw_msg_handler_t fw_msg_handler[5]; /* NUM_FW6_TYPES */ cpl_handler_t cpl_handler[0xef]; /* NUM_CPL_CMDS */ #ifdef INVARIANTS const char *last_op; const void *last_op_thr; #endif int sc_do_rxcopy; }; #define ADAPTER_LOCK(sc) mtx_lock(&(sc)->sc_lock) #define ADAPTER_UNLOCK(sc) mtx_unlock(&(sc)->sc_lock) #define ADAPTER_LOCK_ASSERT_OWNED(sc) mtx_assert(&(sc)->sc_lock, MA_OWNED) #define ADAPTER_LOCK_ASSERT_NOTOWNED(sc) mtx_assert(&(sc)->sc_lock, MA_NOTOWNED) /* XXX: not bulletproof, but much better than nothing */ #define ASSERT_SYNCHRONIZED_OP(sc) \ KASSERT(IS_BUSY(sc) && \ (mtx_owned(&(sc)->sc_lock) || sc->last_op_thr == curthread), \ ("%s: operation not synchronized.", __func__)) #define PORT_LOCK(pi) mtx_lock(&(pi)->pi_lock) #define PORT_UNLOCK(pi) mtx_unlock(&(pi)->pi_lock) #define PORT_LOCK_ASSERT_OWNED(pi) mtx_assert(&(pi)->pi_lock, MA_OWNED) #define PORT_LOCK_ASSERT_NOTOWNED(pi) mtx_assert(&(pi)->pi_lock, MA_NOTOWNED) #define FL_LOCK(fl) mtx_lock(&(fl)->fl_lock) #define FL_TRYLOCK(fl) mtx_trylock(&(fl)->fl_lock) #define FL_UNLOCK(fl) mtx_unlock(&(fl)->fl_lock) #define FL_LOCK_ASSERT_OWNED(fl) mtx_assert(&(fl)->fl_lock, MA_OWNED) #define FL_LOCK_ASSERT_NOTOWNED(fl) mtx_assert(&(fl)->fl_lock, MA_NOTOWNED) #define RXQ_FL_LOCK(rxq) FL_LOCK(&(rxq)->fl) #define RXQ_FL_UNLOCK(rxq) FL_UNLOCK(&(rxq)->fl) #define RXQ_FL_LOCK_ASSERT_OWNED(rxq) FL_LOCK_ASSERT_OWNED(&(rxq)->fl) #define RXQ_FL_LOCK_ASSERT_NOTOWNED(rxq) FL_LOCK_ASSERT_NOTOWNED(&(rxq)->fl) #define EQ_LOCK(eq) mtx_lock(&(eq)->eq_lock) #define EQ_TRYLOCK(eq) mtx_trylock(&(eq)->eq_lock) #define EQ_UNLOCK(eq) mtx_unlock(&(eq)->eq_lock) #define EQ_LOCK_ASSERT_OWNED(eq) mtx_assert(&(eq)->eq_lock, MA_OWNED) #define EQ_LOCK_ASSERT_NOTOWNED(eq) mtx_assert(&(eq)->eq_lock, MA_NOTOWNED) #define TXQ_LOCK(txq) EQ_LOCK(&(txq)->eq) #define TXQ_TRYLOCK(txq) EQ_TRYLOCK(&(txq)->eq) #define TXQ_UNLOCK(txq) EQ_UNLOCK(&(txq)->eq) #define TXQ_LOCK_ASSERT_OWNED(txq) EQ_LOCK_ASSERT_OWNED(&(txq)->eq) #define TXQ_LOCK_ASSERT_NOTOWNED(txq) EQ_LOCK_ASSERT_NOTOWNED(&(txq)->eq) #define for_each_txq(pi, iter, q) \ for (q = &pi->adapter->sge.txq[pi->first_txq], iter = 0; \ iter < pi->ntxq; ++iter, ++q) #define for_each_rxq(pi, iter, q) \ for (q = &pi->adapter->sge.rxq[pi->first_rxq], iter = 0; \ iter < pi->nrxq; ++iter, ++q) #define for_each_ofld_txq(pi, iter, q) \ for (q = &pi->adapter->sge.ofld_txq[pi->first_ofld_txq], iter = 0; \ iter < pi->nofldtxq; ++iter, ++q) #define for_each_ofld_rxq(pi, iter, q) \ for (q = &pi->adapter->sge.ofld_rxq[pi->first_ofld_rxq], iter = 0; \ iter < pi->nofldrxq; ++iter, ++q) #define for_each_nm_txq(pi, iter, q) \ for (q = &pi->adapter->sge.nm_txq[pi->first_nm_txq], iter = 0; \ iter < pi->nnmtxq; ++iter, ++q) #define for_each_nm_rxq(pi, iter, q) \ for (q = &pi->adapter->sge.nm_rxq[pi->first_nm_rxq], iter = 0; \ iter < pi->nnmrxq; ++iter, ++q) #define IDXINCR(idx, incr, wrap) do { \ idx = wrap - idx > incr ? idx + incr : incr - (wrap - idx); \ } while (0) #define IDXDIFF(head, tail, wrap) \ ((head) >= (tail) ? (head) - (tail) : (wrap) - (tail) + (head)) /* One for errors, one for firmware events */ #define T4_EXTRA_INTR 2 static inline uint32_t t4_read_reg(struct adapter *sc, uint32_t reg) { return bus_space_read_4(sc->bt, sc->bh, reg); } static inline void t4_write_reg(struct adapter *sc, uint32_t reg, uint32_t val) { bus_space_write_4(sc->bt, sc->bh, reg, val); } static inline uint64_t t4_read_reg64(struct adapter *sc, uint32_t reg) { return t4_bus_space_read_8(sc->bt, sc->bh, reg); } static inline void t4_write_reg64(struct adapter *sc, uint32_t reg, uint64_t val) { t4_bus_space_write_8(sc->bt, sc->bh, reg, val); } static inline void t4_os_pci_read_cfg1(struct adapter *sc, int reg, uint8_t *val) { *val = pci_read_config(sc->dev, reg, 1); } static inline void t4_os_pci_write_cfg1(struct adapter *sc, int reg, uint8_t val) { pci_write_config(sc->dev, reg, val, 1); } static inline void t4_os_pci_read_cfg2(struct adapter *sc, int reg, uint16_t *val) { *val = pci_read_config(sc->dev, reg, 2); } static inline void t4_os_pci_write_cfg2(struct adapter *sc, int reg, uint16_t val) { pci_write_config(sc->dev, reg, val, 2); } static inline void t4_os_pci_read_cfg4(struct adapter *sc, int reg, uint32_t *val) { *val = pci_read_config(sc->dev, reg, 4); } static inline void t4_os_pci_write_cfg4(struct adapter *sc, int reg, uint32_t val) { pci_write_config(sc->dev, reg, val, 4); } static inline struct port_info * adap2pinfo(struct adapter *sc, int idx) { return (sc->port[idx]); } static inline void t4_os_set_hw_addr(struct adapter *sc, int idx, uint8_t hw_addr[]) { bcopy(hw_addr, sc->port[idx]->hw_addr, ETHER_ADDR_LEN); } static inline bool is_10G_port(const struct port_info *pi) { return ((pi->link_cfg.supported & FW_PORT_CAP_SPEED_10G) != 0); } static inline bool is_40G_port(const struct port_info *pi) { return ((pi->link_cfg.supported & FW_PORT_CAP_SPEED_40G) != 0); } static inline int tx_resume_threshold(struct sge_eq *eq) { return (eq->qsize / 4); } /* t4_main.c */ void t4_tx_task(void *, int); void t4_tx_callout(void *); int t4_os_find_pci_capability(struct adapter *, int); int t4_os_pci_save_state(struct adapter *); int t4_os_pci_restore_state(struct adapter *); void t4_os_portmod_changed(const struct adapter *, int); void t4_os_link_changed(struct adapter *, int, int, int); void t4_iterate(void (*)(struct adapter *, void *), void *); int t4_register_cpl_handler(struct adapter *, int, cpl_handler_t); int t4_register_an_handler(struct adapter *, an_handler_t); int t4_register_fw_msg_handler(struct adapter *, int, fw_msg_handler_t); int t4_filter_rpl(struct sge_iq *, const struct rss_header *, struct mbuf *); int begin_synchronized_op(struct adapter *, struct port_info *, int, char *); void end_synchronized_op(struct adapter *, int); int update_mac_settings(struct ifnet *, int); int adapter_full_init(struct adapter *); int adapter_full_uninit(struct adapter *); int port_full_init(struct port_info *); int port_full_uninit(struct port_info *); #ifdef DEV_NETMAP /* t4_netmap.c */ int create_netmap_ifnet(struct port_info *); int destroy_netmap_ifnet(struct port_info *); void t4_nm_intr(void *); #endif /* t4_sge.c */ void t4_sge_modload(void); void t4_sge_modunload(void); uint64_t t4_sge_extfree_refs(void); void t4_init_sge_cpl_handlers(struct adapter *); void t4_tweak_chip_settings(struct adapter *); int t4_read_chip_settings(struct adapter *); int t4_create_dma_tag(struct adapter *); void t4_sge_sysctls(struct adapter *, struct sysctl_ctx_list *, struct sysctl_oid_list *); int t4_destroy_dma_tag(struct adapter *); int t4_setup_adapter_queues(struct adapter *); int t4_teardown_adapter_queues(struct adapter *); int t4_setup_port_queues(struct port_info *); int t4_teardown_port_queues(struct port_info *); int t4_alloc_tx_maps(struct tx_maps *, bus_dma_tag_t, int, int); void t4_free_tx_maps(struct tx_maps *, bus_dma_tag_t); void t4_intr_all(void *); void t4_intr(void *); void t4_intr_err(void *); void t4_intr_evt(void *); void t4_wrq_tx_locked(struct adapter *, struct sge_wrq *, struct wrqe *); int t4_eth_tx(struct ifnet *, struct sge_txq *, struct mbuf *); void t4_update_fl_bufsize(struct ifnet *); int can_resume_tx(struct sge_eq *); /* t4_tracer.c */ struct t4_tracer; void t4_tracer_modload(void); void t4_tracer_modunload(void); void t4_tracer_port_detach(struct adapter *); int t4_get_tracer(struct adapter *, struct t4_tracer *); int t4_set_tracer(struct adapter *, struct t4_tracer *); int t4_trace_pkt(struct sge_iq *, const struct rss_header *, struct mbuf *); int t5_trace_pkt(struct sge_iq *, const struct rss_header *, struct mbuf *); static inline struct wrqe * alloc_wrqe(int wr_len, struct sge_wrq *wrq) { int len = offsetof(struct wrqe, wr) + wr_len; struct wrqe *wr; wr = malloc(len, M_CXGBE, M_NOWAIT); if (__predict_false(wr == NULL)) return (NULL); wr->wr_len = wr_len; wr->wrq = wrq; return (wr); } static inline void * wrtod(struct wrqe *wr) { return (&wr->wr[0]); } static inline void free_wrqe(struct wrqe *wr) { free(wr, M_CXGBE); } static inline void t4_wrq_tx(struct adapter *sc, struct wrqe *wr) { struct sge_wrq *wrq = wr->wrq; TXQ_LOCK(wrq); t4_wrq_tx_locked(sc, wrq, wr); TXQ_UNLOCK(wrq); } #endif Index: projects/clang350-import/sys/dev/cxgbe/tom/t4_tom.h =================================================================== --- projects/clang350-import/sys/dev/cxgbe/tom/t4_tom.h (revision 275748) +++ projects/clang350-import/sys/dev/cxgbe/tom/t4_tom.h (revision 275749) @@ -1,315 +1,314 @@ /*- * Copyright (c) 2012 Chelsio Communications, Inc. * All rights reserved. * Written by: Navdeep Parhar * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * $FreeBSD$ * */ #ifndef __T4_TOM_H__ #define __T4_TOM_H__ -#define KTR_CXGBE KTR_SPARE3 #define LISTEN_HASH_SIZE 32 /* * Min receive window. We want it to be large enough to accommodate receive * coalescing, handle jumbo frames, and not trigger sender SWS avoidance. */ #define MIN_RCV_WND (24 * 1024U) /* * Max receive window supported by HW in bytes. Only a small part of it can * be set through option0, the rest needs to be set through RX_DATA_ACK. */ #define MAX_RCV_WND ((1U << 27) - 1) #define DDP_RSVD_WIN (16 * 1024U) #define SB_DDP_INDICATE SB_IN_TOE /* soreceive must respond to indicate */ #define USE_DDP_RX_FLOW_CONTROL /* TOE PCB flags */ enum { TPF_ATTACHED = (1 << 0), /* a tcpcb refers to this toepcb */ TPF_FLOWC_WR_SENT = (1 << 1), /* firmware flow context WR sent */ TPF_TX_DATA_SENT = (1 << 2), /* some data sent */ TPF_TX_SUSPENDED = (1 << 3), /* tx suspended for lack of resources */ TPF_SEND_FIN = (1 << 4), /* send FIN after all pending data */ TPF_FIN_SENT = (1 << 5), /* FIN has been sent */ TPF_ABORT_SHUTDOWN = (1 << 6), /* connection abort is in progress */ TPF_CPL_PENDING = (1 << 7), /* haven't received the last CPL */ TPF_SYNQE = (1 << 8), /* synq_entry, not really a toepcb */ TPF_SYNQE_NEEDFREE = (1 << 9), /* synq_entry was malloc'd separately */ TPF_SYNQE_TCPDDP = (1 << 10), /* ulp_mode TCPDDP in toepcb */ TPF_SYNQE_EXPANDED = (1 << 11), /* toepcb ready, tid context updated */ TPF_SYNQE_HAS_L2TE = (1 << 12), /* we've replied to PASS_ACCEPT_REQ */ }; enum { DDP_OK = (1 << 0), /* OK to turn on DDP */ DDP_SC_REQ = (1 << 1), /* state change (on/off) requested */ DDP_ON = (1 << 2), /* DDP is turned on */ DDP_BUF0_ACTIVE = (1 << 3), /* buffer 0 in use (not invalidated) */ DDP_BUF1_ACTIVE = (1 << 4), /* buffer 1 in use (not invalidated) */ }; struct ofld_tx_sdesc { uint32_t plen; /* payload length */ uint8_t tx_credits; /* firmware tx credits (unit is 16B) */ }; struct ppod_region { TAILQ_ENTRY(ppod_region) link; int used; /* # of pods used by this region */ int free; /* # of contiguous pods free right after this region */ }; struct ddp_buffer { uint32_t tag; /* includes color, page pod addr, and DDP page size */ int nppods; int offset; int len; struct ppod_region ppod_region; int npages; vm_page_t *pages; }; struct toepcb { TAILQ_ENTRY(toepcb) link; /* toep_list */ u_int flags; /* miscellaneous flags */ struct tom_data *td; struct inpcb *inp; /* backpointer to host stack's PCB */ struct port_info *port; /* physical port */ struct sge_wrq *ofld_txq; struct sge_ofld_rxq *ofld_rxq; struct sge_wrq *ctrlq; struct l2t_entry *l2te; /* L2 table entry used by this connection */ struct clip_entry *ce; /* CLIP table entry used by this tid */ int tid; /* Connection identifier */ /* tx credit handling */ u_int tx_total; /* total tx WR credits (in 16B units) */ u_int tx_credits; /* tx WR credits (in 16B units) available */ u_int tx_nocompl; /* tx WR credits since last compl request */ u_int plen_nocompl; /* payload since last compl request */ /* rx credit handling */ u_int sb_cc; /* last noted value of so_rcv->sb_cc */ int rx_credits; /* rx credits (in bytes) to be returned to hw */ u_int ulp_mode; /* ULP mode */ u_int ddp_flags; struct ddp_buffer *db[2]; time_t ddp_disabled; uint8_t ddp_score; /* Tx software descriptor */ uint8_t txsd_total; uint8_t txsd_pidx; uint8_t txsd_cidx; uint8_t txsd_avail; struct ofld_tx_sdesc txsd[]; }; struct flowc_tx_params { uint32_t snd_nxt; uint32_t rcv_nxt; unsigned int snd_space; unsigned int mss; }; #define DDP_RETRY_WAIT 5 /* seconds to wait before re-enabling DDP */ #define DDP_LOW_SCORE 1 #define DDP_HIGH_SCORE 3 /* * Compressed state for embryonic connections for a listener. Barely fits in * 64B, try not to grow it further. */ struct synq_entry { TAILQ_ENTRY(synq_entry) link; /* listen_ctx's synq link */ int flags; /* same as toepcb's tp_flags */ int tid; struct listen_ctx *lctx; /* backpointer to listen ctx */ struct mbuf *syn; uint32_t iss; uint32_t ts; volatile uintptr_t wr; volatile u_int refcnt; uint16_t l2e_idx; uint16_t rcv_bufsize; }; /* listen_ctx flags */ #define LCTX_RPL_PENDING 1 /* waiting for a CPL_PASS_OPEN_RPL */ struct listen_ctx { LIST_ENTRY(listen_ctx) link; /* listen hash linkage */ volatile int refcount; int stid; struct stid_region stid_region; int flags; struct inpcb *inp; /* listening socket's inp */ struct sge_wrq *ctrlq; struct sge_ofld_rxq *ofld_rxq; struct clip_entry *ce; TAILQ_HEAD(, synq_entry) synq; }; TAILQ_HEAD(ppod_head, ppod_region); struct clip_entry { TAILQ_ENTRY(clip_entry) link; struct in6_addr lip; /* local IPv6 address */ u_int refcount; }; TAILQ_HEAD(clip_head, clip_entry); struct tom_data { struct toedev tod; /* toepcb's associated with this TOE device */ struct mtx toep_list_lock; TAILQ_HEAD(, toepcb) toep_list; struct mtx lctx_hash_lock; LIST_HEAD(, listen_ctx) *listen_hash; u_long listen_mask; int lctx_count; /* # of lctx in the hash table */ struct mtx ppod_lock; int nppods; int nppods_free; /* # of available ppods */ int nppods_free_head; /* # of available ppods at the begining */ struct ppod_head ppods; struct mtx clip_table_lock; struct clip_head clip_table; int clip_gen; /* WRs that will not be sent to the chip because L2 resolution failed */ struct mtx unsent_wr_lock; STAILQ_HEAD(, wrqe) unsent_wr_list; struct task reclaim_wr_resources; }; static inline struct tom_data * tod_td(struct toedev *tod) { return (__containerof(tod, struct tom_data, tod)); } static inline struct adapter * td_adapter(struct tom_data *td) { return (td->tod.tod_softc); } /* t4_tom.c */ struct toepcb *alloc_toepcb(struct port_info *, int, int, int); void free_toepcb(struct toepcb *); void offload_socket(struct socket *, struct toepcb *); void undo_offload_socket(struct socket *); void final_cpl_received(struct toepcb *); void insert_tid(struct adapter *, int, void *); void *lookup_tid(struct adapter *, int); void update_tid(struct adapter *, int, void *); void remove_tid(struct adapter *, int); void release_tid(struct adapter *, int, struct sge_wrq *); int find_best_mtu_idx(struct adapter *, struct in_conninfo *, int); u_long select_rcv_wnd(struct socket *); int select_rcv_wscale(void); uint64_t calc_opt0(struct socket *, struct port_info *, struct l2t_entry *, int, int, int, int); uint64_t select_ntuple(struct port_info *, struct l2t_entry *); void set_tcpddp_ulp_mode(struct toepcb *); int negative_advice(int); struct clip_entry *hold_lip(struct tom_data *, struct in6_addr *); void release_lip(struct tom_data *, struct clip_entry *); /* t4_connect.c */ void t4_init_connect_cpl_handlers(struct adapter *); int t4_connect(struct toedev *, struct socket *, struct rtentry *, struct sockaddr *); void act_open_failure_cleanup(struct adapter *, u_int, u_int); /* t4_listen.c */ void t4_init_listen_cpl_handlers(struct adapter *); int t4_listen_start(struct toedev *, struct tcpcb *); int t4_listen_stop(struct toedev *, struct tcpcb *); void t4_syncache_added(struct toedev *, void *); void t4_syncache_removed(struct toedev *, void *); int t4_syncache_respond(struct toedev *, void *, struct mbuf *); int do_abort_req_synqe(struct sge_iq *, const struct rss_header *, struct mbuf *); int do_abort_rpl_synqe(struct sge_iq *, const struct rss_header *, struct mbuf *); void t4_offload_socket(struct toedev *, void *, struct socket *); /* t4_cpl_io.c */ void t4_init_cpl_io_handlers(struct adapter *); void t4_uninit_cpl_io_handlers(struct adapter *); void send_abort_rpl(struct adapter *, struct sge_wrq *, int , int); void send_flowc_wr(struct toepcb *, struct flowc_tx_params *); void send_reset(struct adapter *, struct toepcb *, uint32_t); void make_established(struct toepcb *, uint32_t, uint32_t, uint16_t); void t4_rcvd(struct toedev *, struct tcpcb *); int t4_tod_output(struct toedev *, struct tcpcb *); int t4_send_fin(struct toedev *, struct tcpcb *); int t4_send_rst(struct toedev *, struct tcpcb *); void t4_set_tcb_field(struct adapter *, struct toepcb *, int, uint16_t, uint64_t, uint64_t); void t4_push_frames(struct adapter *sc, struct toepcb *toep, int drop); /* t4_ddp.c */ void t4_init_ddp(struct adapter *, struct tom_data *); void t4_uninit_ddp(struct adapter *, struct tom_data *); int t4_soreceive_ddp(struct socket *, struct sockaddr **, struct uio *, struct mbuf **, struct mbuf **, int *); struct mbuf *get_ddp_mbuf(int); void enable_ddp(struct adapter *, struct toepcb *toep); void release_ddp_resources(struct toepcb *toep); void insert_ddp_data(struct toepcb *, uint32_t); /* ULP related */ #define CXGBE_ISCSI_MBUF_TAG 50 int t4tom_cpl_handler_registered(struct adapter *, unsigned int); void t4tom_register_cpl_iscsi_callback(void (*fp)(struct tom_data *, struct socket *, void *, unsigned int)); void t4tom_register_queue_iscsi_callback(struct mbuf *(*fp)(struct socket *, unsigned int, int *)); void t4_ulp_push_frames(struct adapter *sc, struct toepcb *toep, int); int t4_cpl_iscsi_callback(struct tom_data *, struct toepcb *, void *, uint32_t); struct mbuf *t4_queue_iscsi_callback(struct socket *, struct toepcb *, uint32_t, int *); extern void (*tom_cpl_iscsi_callback)(struct tom_data *, struct socket *, void *, unsigned int); extern struct mbuf *(*tom_queue_iscsi_callback)(struct socket*, unsigned int, int *); #endif Index: projects/clang350-import/sys/dev/virtio/mmio/virtio_mmio.c =================================================================== --- projects/clang350-import/sys/dev/virtio/mmio/virtio_mmio.c (revision 275748) +++ projects/clang350-import/sys/dev/virtio/mmio/virtio_mmio.c (revision 275749) @@ -1,840 +1,853 @@ /*- * Copyright (c) 2014 Ruslan Bukin * Copyright (c) 2014 The FreeBSD Foundation * All rights reserved. * * This software was developed by SRI International and the University of * Cambridge Computer Laboratory under DARPA/AFRL contract (FA8750-10-C-0237) * ("CTSRD"), as part of the DARPA CRASH research programme. * * Portions of this software were developed by Andrew Turner * under sponsorship from the FreeBSD Foundation. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ /* * VirtIO MMIO interface. * This driver is heavily based on VirtIO PCI interface driver. */ /* * FDT example: * virtio_block@1000 { * compatible = "virtio,mmio"; * reg = <0x1000 0x100>; * interrupts = <63>; * interrupt-parent = <&GIC>; * }; */ #include __FBSDID("$FreeBSD$"); #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include "virtio_mmio_if.h" #include "virtio_bus_if.h" #include "virtio_if.h" #define PAGE_SHIFT 12 struct vtmmio_virtqueue { struct virtqueue *vtv_vq; int vtv_no_intr; }; struct vtmmio_softc { device_t dev; device_t platform; struct resource *res[2]; uint64_t vtmmio_features; uint32_t vtmmio_flags; /* This "bus" will only ever have one child. */ device_t vtmmio_child_dev; struct virtio_feature_desc *vtmmio_child_feat_desc; int vtmmio_nvqs; struct vtmmio_virtqueue *vtmmio_vqs; void *ih; }; static int vtmmio_probe(device_t); static int vtmmio_attach(device_t); static int vtmmio_detach(device_t); static int vtmmio_suspend(device_t); static int vtmmio_resume(device_t); static int vtmmio_shutdown(device_t); static void vtmmio_driver_added(device_t, driver_t *); static void vtmmio_child_detached(device_t, device_t); static int vtmmio_read_ivar(device_t, device_t, int, uintptr_t *); static int vtmmio_write_ivar(device_t, device_t, int, uintptr_t); static uint64_t vtmmio_negotiate_features(device_t, uint64_t); static int vtmmio_with_feature(device_t, uint64_t); static int vtmmio_alloc_virtqueues(device_t, int, int, struct vq_alloc_info *); static int vtmmio_setup_intr(device_t, enum intr_type); static void vtmmio_stop(device_t); +static void vtmmio_poll(device_t); static int vtmmio_reinit(device_t, uint64_t); static void vtmmio_reinit_complete(device_t); static void vtmmio_notify_virtqueue(device_t, uint16_t); static uint8_t vtmmio_get_status(device_t); static void vtmmio_set_status(device_t, uint8_t); static void vtmmio_read_dev_config(device_t, bus_size_t, void *, int); static void vtmmio_write_dev_config(device_t, bus_size_t, void *, int); static void vtmmio_describe_features(struct vtmmio_softc *, const char *, uint64_t); static void vtmmio_probe_and_attach_child(struct vtmmio_softc *); static int vtmmio_reinit_virtqueue(struct vtmmio_softc *, int); static void vtmmio_free_interrupts(struct vtmmio_softc *); static void vtmmio_free_virtqueues(struct vtmmio_softc *); static void vtmmio_release_child_resources(struct vtmmio_softc *); static void vtmmio_reset(struct vtmmio_softc *); static void vtmmio_select_virtqueue(struct vtmmio_softc *, int); static void vtmmio_vq_intr(void *); /* * I/O port read/write wrappers. */ #define vtmmio_write_config_1(sc, o, v) \ do { \ bus_write_1((sc)->res[0], (o), (v)); \ if (sc->platform != NULL) \ VIRTIO_MMIO_NOTE(sc->platform, (o), (v)); \ } while (0) #define vtmmio_write_config_2(sc, o, v) \ do { \ bus_write_2((sc)->res[0], (o), (v)); \ if (sc->platform != NULL) \ VIRTIO_MMIO_NOTE(sc->platform, (o), (v)); \ } while (0) #define vtmmio_write_config_4(sc, o, v) \ do { \ bus_write_4((sc)->res[0], (o), (v)); \ if (sc->platform != NULL) \ VIRTIO_MMIO_NOTE(sc->platform, (o), (v)); \ } while (0) #define vtmmio_read_config_1(sc, o) \ bus_read_1((sc)->res[0], (o)) #define vtmmio_read_config_2(sc, o) \ bus_read_2((sc)->res[0], (o)) #define vtmmio_read_config_4(sc, o) \ bus_read_4((sc)->res[0], (o)) static device_method_t vtmmio_methods[] = { /* Device interface. */ DEVMETHOD(device_probe, vtmmio_probe), DEVMETHOD(device_attach, vtmmio_attach), DEVMETHOD(device_detach, vtmmio_detach), DEVMETHOD(device_suspend, vtmmio_suspend), DEVMETHOD(device_resume, vtmmio_resume), DEVMETHOD(device_shutdown, vtmmio_shutdown), /* Bus interface. */ DEVMETHOD(bus_driver_added, vtmmio_driver_added), DEVMETHOD(bus_child_detached, vtmmio_child_detached), DEVMETHOD(bus_read_ivar, vtmmio_read_ivar), DEVMETHOD(bus_write_ivar, vtmmio_write_ivar), /* VirtIO bus interface. */ DEVMETHOD(virtio_bus_negotiate_features, vtmmio_negotiate_features), DEVMETHOD(virtio_bus_with_feature, vtmmio_with_feature), DEVMETHOD(virtio_bus_alloc_virtqueues, vtmmio_alloc_virtqueues), DEVMETHOD(virtio_bus_setup_intr, vtmmio_setup_intr), DEVMETHOD(virtio_bus_stop, vtmmio_stop), + DEVMETHOD(virtio_bus_poll, vtmmio_poll), DEVMETHOD(virtio_bus_reinit, vtmmio_reinit), DEVMETHOD(virtio_bus_reinit_complete, vtmmio_reinit_complete), DEVMETHOD(virtio_bus_notify_vq, vtmmio_notify_virtqueue), DEVMETHOD(virtio_bus_read_device_config, vtmmio_read_dev_config), DEVMETHOD(virtio_bus_write_device_config, vtmmio_write_dev_config), DEVMETHOD_END }; static driver_t vtmmio_driver = { "virtio_mmio", vtmmio_methods, sizeof(struct vtmmio_softc) }; devclass_t vtmmio_devclass; DRIVER_MODULE(virtio_mmio, simplebus, vtmmio_driver, vtmmio_devclass, 0, 0); MODULE_VERSION(virtio_mmio, 1); MODULE_DEPEND(virtio_mmio, simplebus, 1, 1, 1); MODULE_DEPEND(virtio_mmio, virtio, 1, 1, 1); static int vtmmio_setup_intr(device_t dev, enum intr_type type) { struct vtmmio_softc *sc; int rid; int err; sc = device_get_softc(dev); if (sc->platform != NULL) { err = VIRTIO_MMIO_SETUP_INTR(sc->platform, sc->dev, vtmmio_vq_intr, sc); if (err == 0) { /* Okay we have backend-specific interrupts */ return (0); } } rid = 0; sc->res[1] = bus_alloc_resource_any(dev, SYS_RES_IRQ, &rid, RF_ACTIVE); if (!sc->res[1]) { device_printf(dev, "Can't allocate interrupt\n"); return (ENXIO); } if (bus_setup_intr(dev, sc->res[1], INTR_TYPE_MISC | INTR_MPSAFE, NULL, vtmmio_vq_intr, sc, &sc->ih)) { device_printf(dev, "Can't setup the interrupt\n"); return (ENXIO); } return (0); } static int vtmmio_probe(device_t dev) { if (!ofw_bus_status_okay(dev)) return (ENXIO); if (!ofw_bus_is_compatible(dev, "virtio,mmio")) return (ENXIO); device_set_desc(dev, "VirtIO MMIO adapter"); return (BUS_PROBE_DEFAULT); } static int vtmmio_setup_platform(struct vtmmio_softc *sc) { phandle_t platform_node; struct fdt_ic *ic; phandle_t xref; phandle_t node; sc->platform = NULL; if ((node = ofw_bus_get_node(sc->dev)) == -1) return (ENXIO); if (OF_searchencprop(node, "platform", &xref, sizeof(xref)) == -1) { return (ENXIO); } platform_node = OF_node_from_xref(xref); SLIST_FOREACH(ic, &fdt_ic_list_head, fdt_ics) { if (ic->iph == platform_node) { sc->platform = ic->dev; break; } } if (sc->platform == NULL) { /* No platform-specific device. Ignore it. */ } return (0); } static int vtmmio_attach(device_t dev) { struct vtmmio_softc *sc; device_t child; int rid; sc = device_get_softc(dev); sc->dev = dev; vtmmio_setup_platform(sc); rid = 0; sc->res[0] = bus_alloc_resource_any(dev, SYS_RES_MEMORY, &rid, RF_ACTIVE); if (!sc->res[0]) { device_printf(dev, "Cannot allocate memory window.\n"); return (ENXIO); } vtmmio_reset(sc); /* Tell the host we've noticed this device. */ vtmmio_set_status(dev, VIRTIO_CONFIG_STATUS_ACK); if ((child = device_add_child(dev, NULL, -1)) == NULL) { device_printf(dev, "Cannot create child device.\n"); vtmmio_set_status(dev, VIRTIO_CONFIG_STATUS_FAILED); vtmmio_detach(dev); return (ENOMEM); } sc->vtmmio_child_dev = child; vtmmio_probe_and_attach_child(sc); return (0); } static int vtmmio_detach(device_t dev) { struct vtmmio_softc *sc; device_t child; int error; sc = device_get_softc(dev); if ((child = sc->vtmmio_child_dev) != NULL) { error = device_delete_child(dev, child); if (error) return (error); sc->vtmmio_child_dev = NULL; } vtmmio_reset(sc); if (sc->res[0] != NULL) { bus_release_resource(dev, SYS_RES_MEMORY, 0, sc->res[0]); sc->res[0] = NULL; } return (0); } static int vtmmio_suspend(device_t dev) { return (bus_generic_suspend(dev)); } static int vtmmio_resume(device_t dev) { return (bus_generic_resume(dev)); } static int vtmmio_shutdown(device_t dev) { (void) bus_generic_shutdown(dev); /* Forcibly stop the host device. */ vtmmio_stop(dev); return (0); } static void vtmmio_driver_added(device_t dev, driver_t *driver) { struct vtmmio_softc *sc; sc = device_get_softc(dev); vtmmio_probe_and_attach_child(sc); } static void vtmmio_child_detached(device_t dev, device_t child) { struct vtmmio_softc *sc; sc = device_get_softc(dev); vtmmio_reset(sc); vtmmio_release_child_resources(sc); } static int vtmmio_read_ivar(device_t dev, device_t child, int index, uintptr_t *result) { struct vtmmio_softc *sc; sc = device_get_softc(dev); if (sc->vtmmio_child_dev != child) return (ENOENT); switch (index) { case VIRTIO_IVAR_DEVTYPE: case VIRTIO_IVAR_SUBDEVICE: *result = vtmmio_read_config_4(sc, VIRTIO_MMIO_DEVICE_ID); break; case VIRTIO_IVAR_VENDOR: *result = vtmmio_read_config_4(sc, VIRTIO_MMIO_VENDOR_ID); break; default: return (ENOENT); } return (0); } static int vtmmio_write_ivar(device_t dev, device_t child, int index, uintptr_t value) { struct vtmmio_softc *sc; sc = device_get_softc(dev); if (sc->vtmmio_child_dev != child) return (ENOENT); switch (index) { case VIRTIO_IVAR_FEATURE_DESC: sc->vtmmio_child_feat_desc = (void *) value; break; default: return (ENOENT); } return (0); } static uint64_t vtmmio_negotiate_features(device_t dev, uint64_t child_features) { struct vtmmio_softc *sc; uint64_t host_features, features; sc = device_get_softc(dev); host_features = vtmmio_read_config_4(sc, VIRTIO_MMIO_HOST_FEATURES); vtmmio_describe_features(sc, "host", host_features); /* * Limit negotiated features to what the driver, virtqueue, and * host all support. */ features = host_features & child_features; features = virtqueue_filter_features(features); sc->vtmmio_features = features; vtmmio_describe_features(sc, "negotiated", features); vtmmio_write_config_4(sc, VIRTIO_MMIO_GUEST_FEATURES, features); return (features); } static int vtmmio_with_feature(device_t dev, uint64_t feature) { struct vtmmio_softc *sc; sc = device_get_softc(dev); return ((sc->vtmmio_features & feature) != 0); } static int vtmmio_alloc_virtqueues(device_t dev, int flags, int nvqs, struct vq_alloc_info *vq_info) { struct vtmmio_virtqueue *vqx; struct vq_alloc_info *info; struct vtmmio_softc *sc; struct virtqueue *vq; uint32_t size; int idx, error; sc = device_get_softc(dev); if (sc->vtmmio_nvqs != 0) return (EALREADY); if (nvqs <= 0) return (EINVAL); sc->vtmmio_vqs = malloc(nvqs * sizeof(struct vtmmio_virtqueue), M_DEVBUF, M_NOWAIT | M_ZERO); if (sc->vtmmio_vqs == NULL) return (ENOMEM); for (idx = 0; idx < nvqs; idx++) { vqx = &sc->vtmmio_vqs[idx]; info = &vq_info[idx]; vtmmio_write_config_4(sc, VIRTIO_MMIO_QUEUE_SEL, idx); vtmmio_select_virtqueue(sc, idx); size = vtmmio_read_config_4(sc, VIRTIO_MMIO_QUEUE_NUM_MAX); error = virtqueue_alloc(dev, idx, size, VIRTIO_MMIO_VRING_ALIGN, 0xFFFFFFFFUL, info, &vq); if (error) { device_printf(dev, "cannot allocate virtqueue %d: %d\n", idx, error); break; } vtmmio_write_config_4(sc, VIRTIO_MMIO_QUEUE_NUM, size); vtmmio_write_config_4(sc, VIRTIO_MMIO_QUEUE_ALIGN, VIRTIO_MMIO_VRING_ALIGN); #if 0 device_printf(dev, "virtqueue paddr 0x%08lx\n", (uint64_t)virtqueue_paddr(vq)); #endif vtmmio_write_config_4(sc, VIRTIO_MMIO_QUEUE_PFN, virtqueue_paddr(vq) >> PAGE_SHIFT); vqx->vtv_vq = *info->vqai_vq = vq; vqx->vtv_no_intr = info->vqai_intr == NULL; sc->vtmmio_nvqs++; } if (error) vtmmio_free_virtqueues(sc); return (error); } static void vtmmio_stop(device_t dev) { vtmmio_reset(device_get_softc(dev)); +} + +static void +vtmmio_poll(device_t dev) +{ + struct vtmmio_softc *sc; + + sc = device_get_softc(dev); + + if (sc->platform != NULL) + VIRTIO_MMIO_POLL(sc->platform); } static int vtmmio_reinit(device_t dev, uint64_t features) { struct vtmmio_softc *sc; int idx, error; sc = device_get_softc(dev); if (vtmmio_get_status(dev) != VIRTIO_CONFIG_STATUS_RESET) vtmmio_stop(dev); /* * Quickly drive the status through ACK and DRIVER. The device * does not become usable again until vtmmio_reinit_complete(). */ vtmmio_set_status(dev, VIRTIO_CONFIG_STATUS_ACK); vtmmio_set_status(dev, VIRTIO_CONFIG_STATUS_DRIVER); vtmmio_negotiate_features(dev, features); for (idx = 0; idx < sc->vtmmio_nvqs; idx++) { error = vtmmio_reinit_virtqueue(sc, idx); if (error) return (error); } return (0); } static void vtmmio_reinit_complete(device_t dev) { vtmmio_set_status(dev, VIRTIO_CONFIG_STATUS_DRIVER_OK); } static void vtmmio_notify_virtqueue(device_t dev, uint16_t queue) { struct vtmmio_softc *sc; sc = device_get_softc(dev); vtmmio_write_config_4(sc, VIRTIO_MMIO_QUEUE_NOTIFY, queue); } static uint8_t vtmmio_get_status(device_t dev) { struct vtmmio_softc *sc; sc = device_get_softc(dev); return (vtmmio_read_config_4(sc, VIRTIO_MMIO_STATUS)); } static void vtmmio_set_status(device_t dev, uint8_t status) { struct vtmmio_softc *sc; sc = device_get_softc(dev); if (status != VIRTIO_CONFIG_STATUS_RESET) status |= vtmmio_get_status(dev); vtmmio_write_config_4(sc, VIRTIO_MMIO_STATUS, status); } static void vtmmio_read_dev_config(device_t dev, bus_size_t offset, void *dst, int length) { struct vtmmio_softc *sc; bus_size_t off; uint8_t *d; int size; sc = device_get_softc(dev); off = VIRTIO_MMIO_CONFIG + offset; for (d = dst; length > 0; d += size, off += size, length -= size) { #ifdef ALLOW_WORD_ALIGNED_ACCESS if (length >= 4) { size = 4; *(uint32_t *)d = vtmmio_read_config_4(sc, off); } else if (length >= 2) { size = 2; *(uint16_t *)d = vtmmio_read_config_2(sc, off); } else #endif { size = 1; *d = vtmmio_read_config_1(sc, off); } } } static void vtmmio_write_dev_config(device_t dev, bus_size_t offset, void *src, int length) { struct vtmmio_softc *sc; bus_size_t off; uint8_t *s; int size; sc = device_get_softc(dev); off = VIRTIO_MMIO_CONFIG + offset; for (s = src; length > 0; s += size, off += size, length -= size) { #ifdef ALLOW_WORD_ALIGNED_ACCESS if (length >= 4) { size = 4; vtmmio_write_config_4(sc, off, *(uint32_t *)s); } else if (length >= 2) { size = 2; vtmmio_write_config_2(sc, off, *(uint16_t *)s); } else #endif { size = 1; vtmmio_write_config_1(sc, off, *s); } } } static void vtmmio_describe_features(struct vtmmio_softc *sc, const char *msg, uint64_t features) { device_t dev, child; dev = sc->dev; child = sc->vtmmio_child_dev; if (device_is_attached(child) && bootverbose == 0) return; virtio_describe(dev, msg, features, sc->vtmmio_child_feat_desc); } static void vtmmio_probe_and_attach_child(struct vtmmio_softc *sc) { device_t dev, child; dev = sc->dev; child = sc->vtmmio_child_dev; if (child == NULL) return; if (device_get_state(child) != DS_NOTPRESENT) { return; } if (device_probe(child) != 0) { return; } vtmmio_set_status(dev, VIRTIO_CONFIG_STATUS_DRIVER); if (device_attach(child) != 0) { vtmmio_set_status(dev, VIRTIO_CONFIG_STATUS_FAILED); vtmmio_reset(sc); vtmmio_release_child_resources(sc); /* Reset status for future attempt. */ vtmmio_set_status(dev, VIRTIO_CONFIG_STATUS_ACK); } else { vtmmio_set_status(dev, VIRTIO_CONFIG_STATUS_DRIVER_OK); VIRTIO_ATTACH_COMPLETED(child); } } static int vtmmio_reinit_virtqueue(struct vtmmio_softc *sc, int idx) { struct vtmmio_virtqueue *vqx; struct virtqueue *vq; int error; uint16_t size; vqx = &sc->vtmmio_vqs[idx]; vq = vqx->vtv_vq; KASSERT(vq != NULL, ("%s: vq %d not allocated", __func__, idx)); vtmmio_select_virtqueue(sc, idx); size = vtmmio_read_config_4(sc, VIRTIO_MMIO_QUEUE_NUM_MAX); error = virtqueue_reinit(vq, size); if (error) return (error); vtmmio_write_config_4(sc, VIRTIO_MMIO_QUEUE_PFN, virtqueue_paddr(vq) >> PAGE_SHIFT); return (0); } static void vtmmio_free_interrupts(struct vtmmio_softc *sc) { if (sc->ih != NULL) bus_teardown_intr(sc->dev, sc->res[1], sc->ih); if (sc->res[1] != NULL) bus_release_resource(sc->dev, SYS_RES_IRQ, 0, sc->res[1]); } static void vtmmio_free_virtqueues(struct vtmmio_softc *sc) { struct vtmmio_virtqueue *vqx; int idx; for (idx = 0; idx < sc->vtmmio_nvqs; idx++) { vqx = &sc->vtmmio_vqs[idx]; vtmmio_select_virtqueue(sc, idx); vtmmio_write_config_4(sc, VIRTIO_MMIO_QUEUE_PFN, 0); virtqueue_free(vqx->vtv_vq); vqx->vtv_vq = NULL; } free(sc->vtmmio_vqs, M_DEVBUF); sc->vtmmio_vqs = NULL; sc->vtmmio_nvqs = 0; } static void vtmmio_release_child_resources(struct vtmmio_softc *sc) { vtmmio_free_interrupts(sc); vtmmio_free_virtqueues(sc); } static void vtmmio_reset(struct vtmmio_softc *sc) { /* * Setting the status to RESET sets the host device to * the original, uninitialized state. */ vtmmio_set_status(sc->dev, VIRTIO_CONFIG_STATUS_RESET); } static void vtmmio_select_virtqueue(struct vtmmio_softc *sc, int idx) { vtmmio_write_config_4(sc, VIRTIO_MMIO_QUEUE_SEL, idx); } static void vtmmio_vq_intr(void *arg) { struct vtmmio_virtqueue *vqx; struct vtmmio_softc *sc; struct virtqueue *vq; uint32_t status; int idx; sc = arg; status = vtmmio_read_config_4(sc, VIRTIO_MMIO_INTERRUPT_STATUS); vtmmio_write_config_4(sc, VIRTIO_MMIO_INTERRUPT_ACK, status); /* The config changed */ if (status & VIRTIO_MMIO_INT_CONFIG) if (sc->vtmmio_child_dev != NULL) VIRTIO_CONFIG_CHANGE(sc->vtmmio_child_dev); /* Notify all virtqueues. */ if (status & VIRTIO_MMIO_INT_VRING) { for (idx = 0; idx < sc->vtmmio_nvqs; idx++) { vqx = &sc->vtmmio_vqs[idx]; if (vqx->vtv_no_intr == 0) { vq = vqx->vtv_vq; virtqueue_intr(vq); } } } } Index: projects/clang350-import/sys/dev/virtio/mmio/virtio_mmio_if.m =================================================================== --- projects/clang350-import/sys/dev/virtio/mmio/virtio_mmio_if.m (revision 275748) +++ projects/clang350-import/sys/dev/virtio/mmio/virtio_mmio_if.m (revision 275749) @@ -1,77 +1,84 @@ #- # Copyright (c) 2014 Ruslan Bukin # All rights reserved. # # This software was developed by SRI International and the University of # Cambridge Computer Laboratory under DARPA/AFRL contract (FA8750-10-C-0237) # ("CTSRD"), as part of the DARPA CRASH research programme. # # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions # are met: # 1. Redistributions of source code must retain the above copyright # notice, this list of conditions and the following disclaimer. # 2. Redistributions in binary form must reproduce the above copyright # notice, this list of conditions and the following disclaimer in the # documentation and/or other materials provided with the distribution. # # THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND # ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE # IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE # ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE # FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL # DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS # OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) # HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT # LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY # OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF # SUCH DAMAGE. # # $FreeBSD$ # #include # # This is optional interface to virtio mmio backend. # Useful when backend is implemented not by the hardware but software, e.g. # by using another cpu core. # INTERFACE virtio_mmio; CODE { static int virtio_mmio_note(device_t dev, size_t offset, int val) { return (1); } static int virtio_mmio_setup_intr(device_t dev, device_t mmio_dev, void *handler, void *ih_user) { return (1); } }; # # Inform backend we have data wrotten to offset. # METHOD int note { device_t dev; size_t offset; int val; } DEFAULT virtio_mmio_note; # +# Inform backend we are going to poll virtqueue. +# +METHOD int poll { + device_t dev; +}; + +# # Setup backend-specific interrupts. # METHOD int setup_intr { device_t dev; device_t mmio_dev; void *handler; void *ih_user; } DEFAULT virtio_mmio_setup_intr; Index: projects/clang350-import/sys/dev/virtio/virtio_bus_if.m =================================================================== --- projects/clang350-import/sys/dev/virtio/virtio_bus_if.m (revision 275748) +++ projects/clang350-import/sys/dev/virtio/virtio_bus_if.m (revision 275749) @@ -1,89 +1,94 @@ #- # Copyright (c) 2011, Bryan Venteicher # All rights reserved. # # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions # are met: # 1. Redistributions of source code must retain the above copyright # notice, this list of conditions and the following disclaimer. # 2. Redistributions in binary form must reproduce the above copyright # notice, this list of conditions and the following disclaimer in the # documentation and/or other materials provided with the distribution. # # THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND # ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE # IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE # ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE # FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL # DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS # OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) # HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT # LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY # OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF # SUCH DAMAGE. # # $FreeBSD$ #include #include INTERFACE virtio_bus; HEADER { struct vq_alloc_info; }; METHOD uint64_t negotiate_features { device_t dev; uint64_t child_features; }; METHOD int with_feature { device_t dev; uint64_t feature; }; METHOD int alloc_virtqueues { device_t dev; int flags; int nvqs; struct vq_alloc_info *info; }; METHOD int setup_intr { device_t dev; enum intr_type type; }; METHOD void stop { device_t dev; }; METHOD int reinit { device_t dev; uint64_t features; }; METHOD void reinit_complete { device_t dev; }; METHOD void notify_vq { device_t dev; uint16_t queue; }; METHOD void read_device_config { device_t dev; bus_size_t offset; void *dst; int len; }; METHOD void write_device_config { device_t dev; bus_size_t offset; void *src; int len; }; + +METHOD void poll { + device_t dev; +}; + Index: projects/clang350-import/sys/dev/virtio/virtqueue.c =================================================================== --- projects/clang350-import/sys/dev/virtio/virtqueue.c (revision 275748) +++ projects/clang350-import/sys/dev/virtio/virtqueue.c (revision 275749) @@ -1,828 +1,831 @@ /*- * Copyright (c) 2011, Bryan Venteicher * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice unmodified, this list of conditions, and the following * disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. * IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. */ /* * Implements the virtqueue interface as basically described * in the original VirtIO paper. */ #include __FBSDID("$FreeBSD$"); #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include "virtio_bus_if.h" struct virtqueue { device_t vq_dev; char vq_name[VIRTQUEUE_MAX_NAME_SZ]; uint16_t vq_queue_index; uint16_t vq_nentries; uint32_t vq_flags; #define VIRTQUEUE_FLAG_INDIRECT 0x0001 #define VIRTQUEUE_FLAG_EVENT_IDX 0x0002 int vq_alignment; int vq_ring_size; void *vq_ring_mem; int vq_max_indirect_size; int vq_indirect_mem_size; virtqueue_intr_t *vq_intrhand; void *vq_intrhand_arg; struct vring vq_ring; uint16_t vq_free_cnt; uint16_t vq_queued_cnt; /* * Head of the free chain in the descriptor table. If * there are no free descriptors, this will be set to * VQ_RING_DESC_CHAIN_END. */ uint16_t vq_desc_head_idx; /* * Last consumed descriptor in the used table, * trails vq_ring.used->idx. */ uint16_t vq_used_cons_idx; struct vq_desc_extra { void *cookie; struct vring_desc *indirect; vm_paddr_t indirect_paddr; uint16_t ndescs; } vq_descx[0]; }; /* * The maximum virtqueue size is 2^15. Use that value as the end of * descriptor chain terminator since it will never be a valid index * in the descriptor table. This is used to verify we are correctly * handling vq_free_cnt. */ #define VQ_RING_DESC_CHAIN_END 32768 #define VQASSERT(_vq, _exp, _msg, ...) \ KASSERT((_exp),("%s: %s - "_msg, __func__, (_vq)->vq_name, \ ##__VA_ARGS__)) #define VQ_RING_ASSERT_VALID_IDX(_vq, _idx) \ VQASSERT((_vq), (_idx) < (_vq)->vq_nentries, \ "invalid ring index: %d, max: %d", (_idx), \ (_vq)->vq_nentries) #define VQ_RING_ASSERT_CHAIN_TERM(_vq) \ VQASSERT((_vq), (_vq)->vq_desc_head_idx == \ VQ_RING_DESC_CHAIN_END, "full ring terminated " \ "incorrectly: head idx: %d", (_vq)->vq_desc_head_idx) static int virtqueue_init_indirect(struct virtqueue *vq, int); static void virtqueue_free_indirect(struct virtqueue *vq); static void virtqueue_init_indirect_list(struct virtqueue *, struct vring_desc *); static void vq_ring_init(struct virtqueue *); static void vq_ring_update_avail(struct virtqueue *, uint16_t); static uint16_t vq_ring_enqueue_segments(struct virtqueue *, struct vring_desc *, uint16_t, struct sglist *, int, int); static int vq_ring_use_indirect(struct virtqueue *, int); static void vq_ring_enqueue_indirect(struct virtqueue *, void *, struct sglist *, int, int); static int vq_ring_enable_interrupt(struct virtqueue *, uint16_t); static int vq_ring_must_notify_host(struct virtqueue *); static void vq_ring_notify_host(struct virtqueue *); static void vq_ring_free_chain(struct virtqueue *, uint16_t); uint64_t virtqueue_filter_features(uint64_t features) { uint64_t mask; mask = (1 << VIRTIO_TRANSPORT_F_START) - 1; mask |= VIRTIO_RING_F_INDIRECT_DESC; mask |= VIRTIO_RING_F_EVENT_IDX; return (features & mask); } int virtqueue_alloc(device_t dev, uint16_t queue, uint16_t size, int align, vm_paddr_t highaddr, struct vq_alloc_info *info, struct virtqueue **vqp) { struct virtqueue *vq; int error; *vqp = NULL; error = 0; if (size == 0) { device_printf(dev, "virtqueue %d (%s) does not exist (size is zero)\n", queue, info->vqai_name); return (ENODEV); } else if (!powerof2(size)) { device_printf(dev, "virtqueue %d (%s) size is not a power of 2: %d\n", queue, info->vqai_name, size); return (ENXIO); } else if (info->vqai_maxindirsz > VIRTIO_MAX_INDIRECT) { device_printf(dev, "virtqueue %d (%s) requested too many " "indirect descriptors: %d, max %d\n", queue, info->vqai_name, info->vqai_maxindirsz, VIRTIO_MAX_INDIRECT); return (EINVAL); } vq = malloc(sizeof(struct virtqueue) + size * sizeof(struct vq_desc_extra), M_DEVBUF, M_NOWAIT | M_ZERO); if (vq == NULL) { device_printf(dev, "cannot allocate virtqueue\n"); return (ENOMEM); } vq->vq_dev = dev; strlcpy(vq->vq_name, info->vqai_name, sizeof(vq->vq_name)); vq->vq_queue_index = queue; vq->vq_alignment = align; vq->vq_nentries = size; vq->vq_free_cnt = size; vq->vq_intrhand = info->vqai_intr; vq->vq_intrhand_arg = info->vqai_intr_arg; if (VIRTIO_BUS_WITH_FEATURE(dev, VIRTIO_RING_F_EVENT_IDX) != 0) vq->vq_flags |= VIRTQUEUE_FLAG_EVENT_IDX; if (info->vqai_maxindirsz > 1) { error = virtqueue_init_indirect(vq, info->vqai_maxindirsz); if (error) goto fail; } vq->vq_ring_size = round_page(vring_size(size, align)); vq->vq_ring_mem = contigmalloc(vq->vq_ring_size, M_DEVBUF, M_NOWAIT | M_ZERO, 0, highaddr, PAGE_SIZE, 0); if (vq->vq_ring_mem == NULL) { device_printf(dev, "cannot allocate memory for virtqueue ring\n"); error = ENOMEM; goto fail; } vq_ring_init(vq); virtqueue_disable_intr(vq); *vqp = vq; fail: if (error) virtqueue_free(vq); return (error); } static int virtqueue_init_indirect(struct virtqueue *vq, int indirect_size) { device_t dev; struct vq_desc_extra *dxp; int i, size; dev = vq->vq_dev; if (VIRTIO_BUS_WITH_FEATURE(dev, VIRTIO_RING_F_INDIRECT_DESC) == 0) { /* * Indirect descriptors requested by the driver but not * negotiated. Return zero to keep the initialization * going: we'll run fine without. */ if (bootverbose) device_printf(dev, "virtqueue %d (%s) requested " "indirect descriptors but not negotiated\n", vq->vq_queue_index, vq->vq_name); return (0); } size = indirect_size * sizeof(struct vring_desc); vq->vq_max_indirect_size = indirect_size; vq->vq_indirect_mem_size = size; vq->vq_flags |= VIRTQUEUE_FLAG_INDIRECT; for (i = 0; i < vq->vq_nentries; i++) { dxp = &vq->vq_descx[i]; dxp->indirect = malloc(size, M_DEVBUF, M_NOWAIT); if (dxp->indirect == NULL) { device_printf(dev, "cannot allocate indirect list\n"); return (ENOMEM); } dxp->indirect_paddr = vtophys(dxp->indirect); virtqueue_init_indirect_list(vq, dxp->indirect); } return (0); } static void virtqueue_free_indirect(struct virtqueue *vq) { struct vq_desc_extra *dxp; int i; for (i = 0; i < vq->vq_nentries; i++) { dxp = &vq->vq_descx[i]; if (dxp->indirect == NULL) break; free(dxp->indirect, M_DEVBUF); dxp->indirect = NULL; dxp->indirect_paddr = 0; } vq->vq_flags &= ~VIRTQUEUE_FLAG_INDIRECT; vq->vq_indirect_mem_size = 0; } static void virtqueue_init_indirect_list(struct virtqueue *vq, struct vring_desc *indirect) { int i; bzero(indirect, vq->vq_indirect_mem_size); for (i = 0; i < vq->vq_max_indirect_size - 1; i++) indirect[i].next = i + 1; indirect[i].next = VQ_RING_DESC_CHAIN_END; } int virtqueue_reinit(struct virtqueue *vq, uint16_t size) { struct vq_desc_extra *dxp; int i; if (vq->vq_nentries != size) { device_printf(vq->vq_dev, "%s: '%s' changed size; old=%hu, new=%hu\n", __func__, vq->vq_name, vq->vq_nentries, size); return (EINVAL); } /* Warn if the virtqueue was not properly cleaned up. */ if (vq->vq_free_cnt != vq->vq_nentries) { device_printf(vq->vq_dev, "%s: warning '%s' virtqueue not empty, " "leaking %d entries\n", __func__, vq->vq_name, vq->vq_nentries - vq->vq_free_cnt); } vq->vq_desc_head_idx = 0; vq->vq_used_cons_idx = 0; vq->vq_queued_cnt = 0; vq->vq_free_cnt = vq->vq_nentries; /* To be safe, reset all our allocated memory. */ bzero(vq->vq_ring_mem, vq->vq_ring_size); for (i = 0; i < vq->vq_nentries; i++) { dxp = &vq->vq_descx[i]; dxp->cookie = NULL; dxp->ndescs = 0; if (vq->vq_flags & VIRTQUEUE_FLAG_INDIRECT) virtqueue_init_indirect_list(vq, dxp->indirect); } vq_ring_init(vq); virtqueue_disable_intr(vq); return (0); } void virtqueue_free(struct virtqueue *vq) { if (vq->vq_free_cnt != vq->vq_nentries) { device_printf(vq->vq_dev, "%s: freeing non-empty virtqueue, " "leaking %d entries\n", vq->vq_name, vq->vq_nentries - vq->vq_free_cnt); } if (vq->vq_flags & VIRTQUEUE_FLAG_INDIRECT) virtqueue_free_indirect(vq); if (vq->vq_ring_mem != NULL) { contigfree(vq->vq_ring_mem, vq->vq_ring_size, M_DEVBUF); vq->vq_ring_size = 0; vq->vq_ring_mem = NULL; } free(vq, M_DEVBUF); } vm_paddr_t virtqueue_paddr(struct virtqueue *vq) { return (vtophys(vq->vq_ring_mem)); } int virtqueue_size(struct virtqueue *vq) { return (vq->vq_nentries); } int virtqueue_nfree(struct virtqueue *vq) { return (vq->vq_free_cnt); } int virtqueue_empty(struct virtqueue *vq) { return (vq->vq_nentries == vq->vq_free_cnt); } int virtqueue_full(struct virtqueue *vq) { return (vq->vq_free_cnt == 0); } void virtqueue_notify(struct virtqueue *vq) { /* Ensure updated avail->idx is visible to host. */ mb(); if (vq_ring_must_notify_host(vq)) vq_ring_notify_host(vq); vq->vq_queued_cnt = 0; } int virtqueue_nused(struct virtqueue *vq) { uint16_t used_idx, nused; used_idx = vq->vq_ring.used->idx; nused = (uint16_t)(used_idx - vq->vq_used_cons_idx); VQASSERT(vq, nused <= vq->vq_nentries, "used more than available"); return (nused); } int virtqueue_intr_filter(struct virtqueue *vq) { if (vq->vq_used_cons_idx == vq->vq_ring.used->idx) return (0); virtqueue_disable_intr(vq); return (1); } void virtqueue_intr(struct virtqueue *vq) { vq->vq_intrhand(vq->vq_intrhand_arg); } int virtqueue_enable_intr(struct virtqueue *vq) { return (vq_ring_enable_interrupt(vq, 0)); } int virtqueue_postpone_intr(struct virtqueue *vq, vq_postpone_t hint) { uint16_t ndesc, avail_idx; avail_idx = vq->vq_ring.avail->idx; ndesc = (uint16_t)(avail_idx - vq->vq_used_cons_idx); switch (hint) { case VQ_POSTPONE_SHORT: ndesc = ndesc / 4; break; case VQ_POSTPONE_LONG: ndesc = (ndesc * 3) / 4; break; case VQ_POSTPONE_EMPTIED: break; } return (vq_ring_enable_interrupt(vq, ndesc)); } /* * Note this is only considered a hint to the host. */ void virtqueue_disable_intr(struct virtqueue *vq) { if (vq->vq_flags & VIRTQUEUE_FLAG_EVENT_IDX) { vring_used_event(&vq->vq_ring) = vq->vq_used_cons_idx - vq->vq_nentries - 1; } else vq->vq_ring.avail->flags |= VRING_AVAIL_F_NO_INTERRUPT; } int virtqueue_enqueue(struct virtqueue *vq, void *cookie, struct sglist *sg, int readable, int writable) { struct vq_desc_extra *dxp; int needed; uint16_t head_idx, idx; needed = readable + writable; VQASSERT(vq, cookie != NULL, "enqueuing with no cookie"); VQASSERT(vq, needed == sg->sg_nseg, "segment count mismatch, %d, %d", needed, sg->sg_nseg); VQASSERT(vq, needed <= vq->vq_nentries || needed <= vq->vq_max_indirect_size, "too many segments to enqueue: %d, %d/%d", needed, vq->vq_nentries, vq->vq_max_indirect_size); if (needed < 1) return (EINVAL); if (vq->vq_free_cnt == 0) return (ENOSPC); if (vq_ring_use_indirect(vq, needed)) { vq_ring_enqueue_indirect(vq, cookie, sg, readable, writable); return (0); } else if (vq->vq_free_cnt < needed) return (EMSGSIZE); head_idx = vq->vq_desc_head_idx; VQ_RING_ASSERT_VALID_IDX(vq, head_idx); dxp = &vq->vq_descx[head_idx]; VQASSERT(vq, dxp->cookie == NULL, "cookie already exists for index %d", head_idx); dxp->cookie = cookie; dxp->ndescs = needed; idx = vq_ring_enqueue_segments(vq, vq->vq_ring.desc, head_idx, sg, readable, writable); vq->vq_desc_head_idx = idx; vq->vq_free_cnt -= needed; if (vq->vq_free_cnt == 0) VQ_RING_ASSERT_CHAIN_TERM(vq); else VQ_RING_ASSERT_VALID_IDX(vq, idx); vq_ring_update_avail(vq, head_idx); return (0); } void * virtqueue_dequeue(struct virtqueue *vq, uint32_t *len) { struct vring_used_elem *uep; void *cookie; uint16_t used_idx, desc_idx; if (vq->vq_used_cons_idx == vq->vq_ring.used->idx) return (NULL); used_idx = vq->vq_used_cons_idx++ & (vq->vq_nentries - 1); uep = &vq->vq_ring.used->ring[used_idx]; rmb(); desc_idx = (uint16_t) uep->id; if (len != NULL) *len = uep->len; vq_ring_free_chain(vq, desc_idx); cookie = vq->vq_descx[desc_idx].cookie; VQASSERT(vq, cookie != NULL, "no cookie for index %d", desc_idx); vq->vq_descx[desc_idx].cookie = NULL; return (cookie); } void * virtqueue_poll(struct virtqueue *vq, uint32_t *len) { void *cookie; - while ((cookie = virtqueue_dequeue(vq, len)) == NULL) + VIRTIO_BUS_POLL(vq->vq_dev); + while ((cookie = virtqueue_dequeue(vq, len)) == NULL) { cpu_spinwait(); + VIRTIO_BUS_POLL(vq->vq_dev); + } return (cookie); } void * virtqueue_drain(struct virtqueue *vq, int *last) { void *cookie; int idx; cookie = NULL; idx = *last; while (idx < vq->vq_nentries && cookie == NULL) { if ((cookie = vq->vq_descx[idx].cookie) != NULL) { vq->vq_descx[idx].cookie = NULL; /* Free chain to keep free count consistent. */ vq_ring_free_chain(vq, idx); } idx++; } *last = idx; return (cookie); } void virtqueue_dump(struct virtqueue *vq) { if (vq == NULL) return; printf("VQ: %s - size=%d; free=%d; used=%d; queued=%d; " "desc_head_idx=%d; avail.idx=%d; used_cons_idx=%d; " "used.idx=%d; used_event_idx=%d; avail.flags=0x%x; used.flags=0x%x\n", vq->vq_name, vq->vq_nentries, vq->vq_free_cnt, virtqueue_nused(vq), vq->vq_queued_cnt, vq->vq_desc_head_idx, vq->vq_ring.avail->idx, vq->vq_used_cons_idx, vq->vq_ring.used->idx, vring_used_event(&vq->vq_ring), vq->vq_ring.avail->flags, vq->vq_ring.used->flags); } static void vq_ring_init(struct virtqueue *vq) { struct vring *vr; char *ring_mem; int i, size; ring_mem = vq->vq_ring_mem; size = vq->vq_nentries; vr = &vq->vq_ring; vring_init(vr, size, ring_mem, vq->vq_alignment); for (i = 0; i < size - 1; i++) vr->desc[i].next = i + 1; vr->desc[i].next = VQ_RING_DESC_CHAIN_END; } static void vq_ring_update_avail(struct virtqueue *vq, uint16_t desc_idx) { uint16_t avail_idx; /* * Place the head of the descriptor chain into the next slot and make * it usable to the host. The chain is made available now rather than * deferring to virtqueue_notify() in the hopes that if the host is * currently running on another CPU, we can keep it processing the new * descriptor. */ avail_idx = vq->vq_ring.avail->idx & (vq->vq_nentries - 1); vq->vq_ring.avail->ring[avail_idx] = desc_idx; wmb(); vq->vq_ring.avail->idx++; /* Keep pending count until virtqueue_notify(). */ vq->vq_queued_cnt++; } static uint16_t vq_ring_enqueue_segments(struct virtqueue *vq, struct vring_desc *desc, uint16_t head_idx, struct sglist *sg, int readable, int writable) { struct sglist_seg *seg; struct vring_desc *dp; int i, needed; uint16_t idx; needed = readable + writable; for (i = 0, idx = head_idx, seg = sg->sg_segs; i < needed; i++, idx = dp->next, seg++) { VQASSERT(vq, idx != VQ_RING_DESC_CHAIN_END, "premature end of free desc chain"); dp = &desc[idx]; dp->addr = seg->ss_paddr; dp->len = seg->ss_len; dp->flags = 0; if (i < needed - 1) dp->flags |= VRING_DESC_F_NEXT; if (i >= readable) dp->flags |= VRING_DESC_F_WRITE; } return (idx); } static int vq_ring_use_indirect(struct virtqueue *vq, int needed) { if ((vq->vq_flags & VIRTQUEUE_FLAG_INDIRECT) == 0) return (0); if (vq->vq_max_indirect_size < needed) return (0); if (needed < 2) return (0); return (1); } static void vq_ring_enqueue_indirect(struct virtqueue *vq, void *cookie, struct sglist *sg, int readable, int writable) { struct vring_desc *dp; struct vq_desc_extra *dxp; int needed; uint16_t head_idx; needed = readable + writable; VQASSERT(vq, needed <= vq->vq_max_indirect_size, "enqueuing too many indirect descriptors"); head_idx = vq->vq_desc_head_idx; VQ_RING_ASSERT_VALID_IDX(vq, head_idx); dp = &vq->vq_ring.desc[head_idx]; dxp = &vq->vq_descx[head_idx]; VQASSERT(vq, dxp->cookie == NULL, "cookie already exists for index %d", head_idx); dxp->cookie = cookie; dxp->ndescs = 1; dp->addr = dxp->indirect_paddr; dp->len = needed * sizeof(struct vring_desc); dp->flags = VRING_DESC_F_INDIRECT; vq_ring_enqueue_segments(vq, dxp->indirect, 0, sg, readable, writable); vq->vq_desc_head_idx = dp->next; vq->vq_free_cnt--; if (vq->vq_free_cnt == 0) VQ_RING_ASSERT_CHAIN_TERM(vq); else VQ_RING_ASSERT_VALID_IDX(vq, vq->vq_desc_head_idx); vq_ring_update_avail(vq, head_idx); } static int vq_ring_enable_interrupt(struct virtqueue *vq, uint16_t ndesc) { /* * Enable interrupts, making sure we get the latest index of * what's already been consumed. */ if (vq->vq_flags & VIRTQUEUE_FLAG_EVENT_IDX) vring_used_event(&vq->vq_ring) = vq->vq_used_cons_idx + ndesc; else vq->vq_ring.avail->flags &= ~VRING_AVAIL_F_NO_INTERRUPT; mb(); /* * Enough items may have already been consumed to meet our threshold * since we last checked. Let our caller know so it processes the new * entries. */ if (virtqueue_nused(vq) > ndesc) return (1); return (0); } static int vq_ring_must_notify_host(struct virtqueue *vq) { uint16_t new_idx, prev_idx, event_idx; if (vq->vq_flags & VIRTQUEUE_FLAG_EVENT_IDX) { new_idx = vq->vq_ring.avail->idx; prev_idx = new_idx - vq->vq_queued_cnt; event_idx = vring_avail_event(&vq->vq_ring); return (vring_need_event(event_idx, new_idx, prev_idx) != 0); } return ((vq->vq_ring.used->flags & VRING_USED_F_NO_NOTIFY) == 0); } static void vq_ring_notify_host(struct virtqueue *vq) { VIRTIO_BUS_NOTIFY_VQ(vq->vq_dev, vq->vq_queue_index); } static void vq_ring_free_chain(struct virtqueue *vq, uint16_t desc_idx) { struct vring_desc *dp; struct vq_desc_extra *dxp; VQ_RING_ASSERT_VALID_IDX(vq, desc_idx); dp = &vq->vq_ring.desc[desc_idx]; dxp = &vq->vq_descx[desc_idx]; if (vq->vq_free_cnt == 0) VQ_RING_ASSERT_CHAIN_TERM(vq); vq->vq_free_cnt += dxp->ndescs; dxp->ndescs--; if ((dp->flags & VRING_DESC_F_INDIRECT) == 0) { while (dp->flags & VRING_DESC_F_NEXT) { VQ_RING_ASSERT_VALID_IDX(vq, dp->next); dp = &vq->vq_ring.desc[dp->next]; dxp->ndescs--; } } VQASSERT(vq, dxp->ndescs == 0, "failed to free entire desc chain, remaining: %d", dxp->ndescs); /* * We must append the existing free chain, if any, to the end of * newly freed chain. If the virtqueue was completely used, then * head would be VQ_RING_DESC_CHAIN_END (ASSERTed above). */ dp->next = vq->vq_desc_head_idx; vq->vq_desc_head_idx = desc_idx; } Index: projects/clang350-import/sys/geom/eli/g_eli_crypto.c =================================================================== --- projects/clang350-import/sys/geom/eli/g_eli_crypto.c (revision 275748) +++ projects/clang350-import/sys/geom/eli/g_eli_crypto.c (revision 275749) @@ -1,295 +1,295 @@ /*- * Copyright (c) 2005-2010 Pawel Jakub Dawidek * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHORS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #include __FBSDID("$FreeBSD$"); #include #ifdef _KERNEL #include #include #include #else #include #include #include #include #include #include #define _OpenSSL_ #endif #include #ifdef _KERNEL MALLOC_DECLARE(M_ELI); static int g_eli_crypto_done(struct cryptop *crp) { crp->crp_opaque = (void *)crp; wakeup(crp); return (0); } static int g_eli_crypto_cipher(u_int algo, int enc, u_char *data, size_t datasize, const u_char *key, size_t keysize) { struct cryptoini cri; struct cryptop *crp; struct cryptodesc *crd; uint64_t sid; u_char *p; int error; KASSERT(algo != CRYPTO_AES_XTS, ("%s: CRYPTO_AES_XTS unexpected here", __func__)); bzero(&cri, sizeof(cri)); cri.cri_alg = algo; cri.cri_key = __DECONST(void *, key); cri.cri_klen = keysize; error = crypto_newsession(&sid, &cri, CRYPTOCAP_F_SOFTWARE); if (error != 0) return (error); p = malloc(sizeof(*crp) + sizeof(*crd), M_ELI, M_NOWAIT | M_ZERO); if (p == NULL) { crypto_freesession(sid); return (ENOMEM); } crp = (struct cryptop *)p; p += sizeof(*crp); crd = (struct cryptodesc *)p; p += sizeof(*crd); crd->crd_skip = 0; crd->crd_len = datasize; crd->crd_flags = CRD_F_IV_EXPLICIT | CRD_F_IV_PRESENT; if (enc) crd->crd_flags |= CRD_F_ENCRYPT; crd->crd_alg = algo; crd->crd_key = __DECONST(void *, key); crd->crd_klen = keysize; bzero(crd->crd_iv, sizeof(crd->crd_iv)); crd->crd_next = NULL; crp->crp_sid = sid; crp->crp_ilen = datasize; crp->crp_olen = datasize; crp->crp_opaque = NULL; crp->crp_callback = g_eli_crypto_done; crp->crp_buf = (void *)data; - crp->crp_flags = CRYPTO_F_CBIFSYNC | CRYPTO_F_REL; + crp->crp_flags = CRYPTO_F_CBIFSYNC; crp->crp_desc = crd; error = crypto_dispatch(crp); if (error == 0) { while (crp->crp_opaque == NULL) tsleep(crp, PRIBIO, "geli", hz / 5); error = crp->crp_etype; } free(crp, M_ELI); crypto_freesession(sid); return (error); } #else /* !_KERNEL */ static int g_eli_crypto_cipher(u_int algo, int enc, u_char *data, size_t datasize, const u_char *key, size_t keysize) { EVP_CIPHER_CTX ctx; const EVP_CIPHER *type; u_char iv[keysize]; int outsize; assert(algo != CRYPTO_AES_XTS); switch (algo) { case CRYPTO_NULL_CBC: type = EVP_enc_null(); break; case CRYPTO_AES_CBC: switch (keysize) { case 128: type = EVP_aes_128_cbc(); break; case 192: type = EVP_aes_192_cbc(); break; case 256: type = EVP_aes_256_cbc(); break; default: return (EINVAL); } break; case CRYPTO_BLF_CBC: type = EVP_bf_cbc(); break; #ifndef OPENSSL_NO_CAMELLIA case CRYPTO_CAMELLIA_CBC: switch (keysize) { case 128: type = EVP_camellia_128_cbc(); break; case 192: type = EVP_camellia_192_cbc(); break; case 256: type = EVP_camellia_256_cbc(); break; default: return (EINVAL); } break; #endif case CRYPTO_3DES_CBC: type = EVP_des_ede3_cbc(); break; default: return (EINVAL); } EVP_CIPHER_CTX_init(&ctx); EVP_CipherInit_ex(&ctx, type, NULL, NULL, NULL, enc); EVP_CIPHER_CTX_set_key_length(&ctx, keysize / 8); EVP_CIPHER_CTX_set_padding(&ctx, 0); bzero(iv, sizeof(iv)); EVP_CipherInit_ex(&ctx, NULL, NULL, key, iv, enc); if (EVP_CipherUpdate(&ctx, data, &outsize, data, datasize) == 0) { EVP_CIPHER_CTX_cleanup(&ctx); return (EINVAL); } assert(outsize == (int)datasize); if (EVP_CipherFinal_ex(&ctx, data + outsize, &outsize) == 0) { EVP_CIPHER_CTX_cleanup(&ctx); return (EINVAL); } assert(outsize == 0); EVP_CIPHER_CTX_cleanup(&ctx); return (0); } #endif /* !_KERNEL */ int g_eli_crypto_encrypt(u_int algo, u_char *data, size_t datasize, const u_char *key, size_t keysize) { /* We prefer AES-CBC for metadata protection. */ if (algo == CRYPTO_AES_XTS) algo = CRYPTO_AES_CBC; return (g_eli_crypto_cipher(algo, 1, data, datasize, key, keysize)); } int g_eli_crypto_decrypt(u_int algo, u_char *data, size_t datasize, const u_char *key, size_t keysize) { /* We prefer AES-CBC for metadata protection. */ if (algo == CRYPTO_AES_XTS) algo = CRYPTO_AES_CBC; return (g_eli_crypto_cipher(algo, 0, data, datasize, key, keysize)); } void g_eli_crypto_hmac_init(struct hmac_ctx *ctx, const uint8_t *hkey, size_t hkeylen) { u_char k_ipad[128], key[128]; SHA512_CTX lctx; u_int i; bzero(key, sizeof(key)); if (hkeylen == 0) ; /* do nothing */ else if (hkeylen <= 128) bcopy(hkey, key, hkeylen); else { /* If key is longer than 128 bytes reset it to key = SHA512(key). */ SHA512_Init(&lctx); SHA512_Update(&lctx, hkey, hkeylen); SHA512_Final(key, &lctx); } /* XOR key with ipad and opad values. */ for (i = 0; i < sizeof(key); i++) { k_ipad[i] = key[i] ^ 0x36; ctx->k_opad[i] = key[i] ^ 0x5c; } bzero(key, sizeof(key)); /* Perform inner SHA512. */ SHA512_Init(&ctx->shactx); SHA512_Update(&ctx->shactx, k_ipad, sizeof(k_ipad)); bzero(k_ipad, sizeof(k_ipad)); } void g_eli_crypto_hmac_update(struct hmac_ctx *ctx, const uint8_t *data, size_t datasize) { SHA512_Update(&ctx->shactx, data, datasize); } void g_eli_crypto_hmac_final(struct hmac_ctx *ctx, uint8_t *md, size_t mdsize) { u_char digest[SHA512_MDLEN]; SHA512_CTX lctx; SHA512_Final(digest, &ctx->shactx); /* Perform outer SHA512. */ SHA512_Init(&lctx); SHA512_Update(&lctx, ctx->k_opad, sizeof(ctx->k_opad)); bzero(ctx, sizeof(*ctx)); SHA512_Update(&lctx, digest, sizeof(digest)); SHA512_Final(digest, &lctx); bzero(&lctx, sizeof(lctx)); /* mdsize == 0 means "Give me the whole hash!" */ if (mdsize == 0) mdsize = SHA512_MDLEN; bcopy(digest, md, mdsize); bzero(digest, sizeof(digest)); } void g_eli_crypto_hmac(const uint8_t *hkey, size_t hkeysize, const uint8_t *data, size_t datasize, uint8_t *md, size_t mdsize) { struct hmac_ctx ctx; g_eli_crypto_hmac_init(&ctx, hkey, hkeysize); g_eli_crypto_hmac_update(&ctx, data, datasize); g_eli_crypto_hmac_final(&ctx, md, mdsize); } Index: projects/clang350-import/sys/geom/eli/g_eli_integrity.c =================================================================== --- projects/clang350-import/sys/geom/eli/g_eli_integrity.c (revision 275748) +++ projects/clang350-import/sys/geom/eli/g_eli_integrity.c (revision 275749) @@ -1,528 +1,528 @@ /*- * Copyright (c) 2005-2011 Pawel Jakub Dawidek * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHORS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #include __FBSDID("$FreeBSD$"); #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include /* * The data layout description when integrity verification is configured. * * One of the most important assumption here is that authenticated data and its * HMAC has to be stored in the same place (namely in the same sector) to make * it work reliable. * The problem is that file systems work only with sectors that are multiple of * 512 bytes and a power of two number. * My idea to implement it is as follows. * Let's store HMAC in sector. This is a must. This leaves us 480 bytes for * data. We can't use that directly (ie. we can't create provider with 480 bytes * sector size). We need another sector from where we take only 32 bytes of data * and we store HMAC of this data as well. This takes two sectors from the * original provider at the input and leaves us one sector of authenticated data * at the output. Not very efficient, but you got the idea. * Now, let's assume, we want to create provider with 4096 bytes sector. * To output 4096 bytes of authenticated data we need 8x480 plus 1x256, so we * need nine 512-bytes sectors at the input to get one 4096-bytes sector at the * output. That's better. With 4096 bytes sector we can use 89% of size of the * original provider. I find it as an acceptable cost. * The reliability comes from the fact, that every HMAC stored inside the sector * is calculated only for the data in the same sector, so its impossible to * write new data and leave old HMAC or vice versa. * * And here is the picture: * * da0: +----+----+ +----+----+ +----+----+ +----+----+ +----+----+ +----+----+ +----+----+ +----+----+ +----+-----+ * |32b |480b| |32b |480b| |32b |480b| |32b |480b| |32b |480b| |32b |480b| |32b |480b| |32b |480b| |32b |256b | * |HMAC|Data| |HMAC|Data| |HMAC|Data| |HMAC|Data| |HMAC|Data| |HMAC|Data| |HMAC|Data| |HMAC|Data| |HMAC|Data | * +----+----+ +----+----+ +----+----+ +----+----+ +----+----+ +----+----+ +----+----+ +----+----+ +----+-----+ * |512 bytes| |512 bytes| |512 bytes| |512 bytes| |512 bytes| |512 bytes| |512 bytes| |512 bytes| |288 bytes | * +---------+ +---------+ +---------+ +---------+ +---------+ +---------+ +---------+ +---------+ |224 unused| * +----------+ * da0.eli: +----+----+----+----+----+----+----+----+----+ * |480b|480b|480b|480b|480b|480b|480b|480b|256b| * +----+----+----+----+----+----+----+----+----+ * | 4096 bytes | * +--------------------------------------------+ * * PS. You can use any sector size with geli(8). My example is using 4kB, * because it's most efficient. For 8kB sectors you need 2 extra sectors, * so the cost is the same as for 4kB sectors. */ /* * Code paths: * BIO_READ: * g_eli_start -> g_eli_auth_read -> g_io_request -> g_eli_read_done -> g_eli_auth_run -> g_eli_auth_read_done -> g_io_deliver * BIO_WRITE: * g_eli_start -> g_eli_auth_run -> g_eli_auth_write_done -> g_io_request -> g_eli_write_done -> g_io_deliver */ MALLOC_DECLARE(M_ELI); /* * Here we generate key for HMAC. Every sector has its own HMAC key, so it is * not possible to copy sectors. * We cannot depend on fact, that every sector has its own IV, because different * IV doesn't change HMAC, when we use encrypt-then-authenticate method. */ static void g_eli_auth_keygen(struct g_eli_softc *sc, off_t offset, u_char *key) { SHA256_CTX ctx; /* Copy precalculated SHA256 context. */ bcopy(&sc->sc_akeyctx, &ctx, sizeof(ctx)); SHA256_Update(&ctx, (uint8_t *)&offset, sizeof(offset)); SHA256_Final(key, &ctx); } /* * The function is called after we read and decrypt data. * * g_eli_start -> g_eli_auth_read -> g_io_request -> g_eli_read_done -> g_eli_auth_run -> G_ELI_AUTH_READ_DONE -> g_io_deliver */ static int g_eli_auth_read_done(struct cryptop *crp) { struct g_eli_softc *sc; struct bio *bp; if (crp->crp_etype == EAGAIN) { if (g_eli_crypto_rerun(crp) == 0) return (0); } bp = (struct bio *)crp->crp_opaque; bp->bio_inbed++; if (crp->crp_etype == 0) { bp->bio_completed += crp->crp_olen; G_ELI_DEBUG(3, "Crypto READ request done (%d/%d) (add=%jd completed=%jd).", bp->bio_inbed, bp->bio_children, (intmax_t)crp->crp_olen, (intmax_t)bp->bio_completed); } else { G_ELI_DEBUG(1, "Crypto READ request failed (%d/%d) error=%d.", bp->bio_inbed, bp->bio_children, crp->crp_etype); if (bp->bio_error == 0) bp->bio_error = crp->crp_etype; } sc = bp->bio_to->geom->softc; g_eli_key_drop(sc, crp->crp_desc->crd_next->crd_key); /* * Do we have all sectors already? */ if (bp->bio_inbed < bp->bio_children) return (0); if (bp->bio_error == 0) { u_int i, lsec, nsec, data_secsize, decr_secsize, encr_secsize; u_char *srcdata, *dstdata, *auth; off_t coroff, corsize; /* * Verify data integrity based on calculated and read HMACs. */ /* Sectorsize of decrypted provider eg. 4096. */ decr_secsize = bp->bio_to->sectorsize; /* The real sectorsize of encrypted provider, eg. 512. */ encr_secsize = LIST_FIRST(&sc->sc_geom->consumer)->provider->sectorsize; /* Number of data bytes in one encrypted sector, eg. 480. */ data_secsize = sc->sc_data_per_sector; /* Number of sectors from decrypted provider, eg. 2. */ nsec = bp->bio_length / decr_secsize; /* Number of sectors from encrypted provider, eg. 18. */ nsec = (nsec * sc->sc_bytes_per_sector) / encr_secsize; /* Last sector number in every big sector, eg. 9. */ lsec = sc->sc_bytes_per_sector / encr_secsize; srcdata = bp->bio_driver2; dstdata = bp->bio_data; auth = srcdata + encr_secsize * nsec; coroff = -1; corsize = 0; for (i = 1; i <= nsec; i++) { data_secsize = sc->sc_data_per_sector; if ((i % lsec) == 0) data_secsize = decr_secsize % data_secsize; if (bcmp(srcdata, auth, sc->sc_alen) != 0) { /* * Curruption detected, remember the offset if * this is the first corrupted sector and * increase size. */ if (bp->bio_error == 0) bp->bio_error = -1; if (coroff == -1) { coroff = bp->bio_offset + (dstdata - (u_char *)bp->bio_data); } corsize += data_secsize; } else { /* * No curruption, good. * Report previous corruption if there was one. */ if (coroff != -1) { G_ELI_DEBUG(0, "%s: Failed to authenticate %jd " "bytes of data at offset %jd.", sc->sc_name, (intmax_t)corsize, (intmax_t)coroff); coroff = -1; corsize = 0; } bcopy(srcdata + sc->sc_alen, dstdata, data_secsize); } srcdata += encr_secsize; dstdata += data_secsize; auth += sc->sc_alen; } /* Report previous corruption if there was one. */ if (coroff != -1) { G_ELI_DEBUG(0, "%s: Failed to authenticate %jd " "bytes of data at offset %jd.", sc->sc_name, (intmax_t)corsize, (intmax_t)coroff); } } free(bp->bio_driver2, M_ELI); bp->bio_driver2 = NULL; if (bp->bio_error != 0) { if (bp->bio_error == -1) bp->bio_error = EINVAL; else { G_ELI_LOGREQ(0, bp, "Crypto READ request failed (error=%d).", bp->bio_error); } bp->bio_completed = 0; } /* * Read is finished, send it up. */ g_io_deliver(bp, bp->bio_error); atomic_subtract_int(&sc->sc_inflight, 1); return (0); } /* * The function is called after data encryption. * * g_eli_start -> g_eli_auth_run -> G_ELI_AUTH_WRITE_DONE -> g_io_request -> g_eli_write_done -> g_io_deliver */ static int g_eli_auth_write_done(struct cryptop *crp) { struct g_eli_softc *sc; struct g_consumer *cp; struct bio *bp, *cbp, *cbp2; u_int nsec; if (crp->crp_etype == EAGAIN) { if (g_eli_crypto_rerun(crp) == 0) return (0); } bp = (struct bio *)crp->crp_opaque; bp->bio_inbed++; if (crp->crp_etype == 0) { G_ELI_DEBUG(3, "Crypto WRITE request done (%d/%d).", bp->bio_inbed, bp->bio_children); } else { G_ELI_DEBUG(1, "Crypto WRITE request failed (%d/%d) error=%d.", bp->bio_inbed, bp->bio_children, crp->crp_etype); if (bp->bio_error == 0) bp->bio_error = crp->crp_etype; } sc = bp->bio_to->geom->softc; g_eli_key_drop(sc, crp->crp_desc->crd_key); /* * All sectors are already encrypted? */ if (bp->bio_inbed < bp->bio_children) return (0); if (bp->bio_error != 0) { G_ELI_LOGREQ(0, bp, "Crypto WRITE request failed (error=%d).", bp->bio_error); free(bp->bio_driver2, M_ELI); bp->bio_driver2 = NULL; cbp = bp->bio_driver1; bp->bio_driver1 = NULL; g_destroy_bio(cbp); g_io_deliver(bp, bp->bio_error); atomic_subtract_int(&sc->sc_inflight, 1); return (0); } cp = LIST_FIRST(&sc->sc_geom->consumer); cbp = bp->bio_driver1; bp->bio_driver1 = NULL; cbp->bio_to = cp->provider; cbp->bio_done = g_eli_write_done; /* Number of sectors from decrypted provider, eg. 1. */ nsec = bp->bio_length / bp->bio_to->sectorsize; /* Number of sectors from encrypted provider, eg. 9. */ nsec = (nsec * sc->sc_bytes_per_sector) / cp->provider->sectorsize; cbp->bio_length = cp->provider->sectorsize * nsec; cbp->bio_offset = (bp->bio_offset / bp->bio_to->sectorsize) * sc->sc_bytes_per_sector; cbp->bio_data = bp->bio_driver2; /* * We write more than what is requested, so we have to be ready to write * more than MAXPHYS. */ cbp2 = NULL; if (cbp->bio_length > MAXPHYS) { cbp2 = g_duplicate_bio(bp); cbp2->bio_length = cbp->bio_length - MAXPHYS; cbp2->bio_data = cbp->bio_data + MAXPHYS; cbp2->bio_offset = cbp->bio_offset + MAXPHYS; cbp2->bio_to = cp->provider; cbp2->bio_done = g_eli_write_done; cbp->bio_length = MAXPHYS; } /* * Send encrypted data to the provider. */ G_ELI_LOGREQ(2, cbp, "Sending request."); bp->bio_inbed = 0; bp->bio_children = (cbp2 != NULL ? 2 : 1); g_io_request(cbp, cp); if (cbp2 != NULL) { G_ELI_LOGREQ(2, cbp2, "Sending request."); g_io_request(cbp2, cp); } return (0); } void g_eli_auth_read(struct g_eli_softc *sc, struct bio *bp) { struct g_consumer *cp; struct bio *cbp, *cbp2; size_t size; off_t nsec; bp->bio_pflags = 0; cp = LIST_FIRST(&sc->sc_geom->consumer); cbp = bp->bio_driver1; bp->bio_driver1 = NULL; cbp->bio_to = cp->provider; cbp->bio_done = g_eli_read_done; /* Number of sectors from decrypted provider, eg. 1. */ nsec = bp->bio_length / bp->bio_to->sectorsize; /* Number of sectors from encrypted provider, eg. 9. */ nsec = (nsec * sc->sc_bytes_per_sector) / cp->provider->sectorsize; cbp->bio_length = cp->provider->sectorsize * nsec; size = cbp->bio_length; size += sc->sc_alen * nsec; size += sizeof(struct cryptop) * nsec; size += sizeof(struct cryptodesc) * nsec * 2; size += G_ELI_AUTH_SECKEYLEN * nsec; cbp->bio_offset = (bp->bio_offset / bp->bio_to->sectorsize) * sc->sc_bytes_per_sector; bp->bio_driver2 = malloc(size, M_ELI, M_WAITOK); cbp->bio_data = bp->bio_driver2; /* * We read more than what is requested, so we have to be ready to read * more than MAXPHYS. */ cbp2 = NULL; if (cbp->bio_length > MAXPHYS) { cbp2 = g_duplicate_bio(bp); cbp2->bio_length = cbp->bio_length - MAXPHYS; cbp2->bio_data = cbp->bio_data + MAXPHYS; cbp2->bio_offset = cbp->bio_offset + MAXPHYS; cbp2->bio_to = cp->provider; cbp2->bio_done = g_eli_read_done; cbp->bio_length = MAXPHYS; } /* * Read encrypted data from provider. */ G_ELI_LOGREQ(2, cbp, "Sending request."); g_io_request(cbp, cp); if (cbp2 != NULL) { G_ELI_LOGREQ(2, cbp2, "Sending request."); g_io_request(cbp2, cp); } } /* * This is the main function responsible for cryptography (ie. communication * with crypto(9) subsystem). * * BIO_READ: * g_eli_start -> g_eli_auth_read -> g_io_request -> g_eli_read_done -> G_ELI_AUTH_RUN -> g_eli_auth_read_done -> g_io_deliver * BIO_WRITE: * g_eli_start -> G_ELI_AUTH_RUN -> g_eli_auth_write_done -> g_io_request -> g_eli_write_done -> g_io_deliver */ void g_eli_auth_run(struct g_eli_worker *wr, struct bio *bp) { struct g_eli_softc *sc; struct cryptop *crp; struct cryptodesc *crde, *crda; u_int i, lsec, nsec, data_secsize, decr_secsize, encr_secsize; off_t dstoff; int err, error; u_char *p, *data, *auth, *authkey, *plaindata; G_ELI_LOGREQ(3, bp, "%s", __func__); bp->bio_pflags = wr->w_number; sc = wr->w_softc; /* Sectorsize of decrypted provider eg. 4096. */ decr_secsize = bp->bio_to->sectorsize; /* The real sectorsize of encrypted provider, eg. 512. */ encr_secsize = LIST_FIRST(&sc->sc_geom->consumer)->provider->sectorsize; /* Number of data bytes in one encrypted sector, eg. 480. */ data_secsize = sc->sc_data_per_sector; /* Number of sectors from decrypted provider, eg. 2. */ nsec = bp->bio_length / decr_secsize; /* Number of sectors from encrypted provider, eg. 18. */ nsec = (nsec * sc->sc_bytes_per_sector) / encr_secsize; /* Last sector number in every big sector, eg. 9. */ lsec = sc->sc_bytes_per_sector / encr_secsize; /* Destination offset, used for IV generation. */ dstoff = (bp->bio_offset / bp->bio_to->sectorsize) * sc->sc_bytes_per_sector; auth = NULL; /* Silence compiler warning. */ plaindata = bp->bio_data; if (bp->bio_cmd == BIO_READ) { data = bp->bio_driver2; auth = data + encr_secsize * nsec; p = auth + sc->sc_alen * nsec; } else { size_t size; size = encr_secsize * nsec; size += sizeof(*crp) * nsec; size += sizeof(*crde) * nsec; size += sizeof(*crda) * nsec; size += G_ELI_AUTH_SECKEYLEN * nsec; data = malloc(size, M_ELI, M_WAITOK); bp->bio_driver2 = data; p = data + encr_secsize * nsec; } bp->bio_inbed = 0; bp->bio_children = nsec; error = 0; for (i = 1; i <= nsec; i++, dstoff += encr_secsize) { crp = (struct cryptop *)p; p += sizeof(*crp); crde = (struct cryptodesc *)p; p += sizeof(*crde); crda = (struct cryptodesc *)p; p += sizeof(*crda); authkey = (u_char *)p; p += G_ELI_AUTH_SECKEYLEN; data_secsize = sc->sc_data_per_sector; if ((i % lsec) == 0) data_secsize = decr_secsize % data_secsize; if (bp->bio_cmd == BIO_READ) { /* Remember read HMAC. */ bcopy(data, auth, sc->sc_alen); auth += sc->sc_alen; /* TODO: bzero(9) can be commented out later. */ bzero(data, sc->sc_alen); } else { bcopy(plaindata, data + sc->sc_alen, data_secsize); plaindata += data_secsize; } crp->crp_sid = wr->w_sid; crp->crp_ilen = sc->sc_alen + data_secsize; crp->crp_olen = data_secsize; crp->crp_opaque = (void *)bp; crp->crp_buf = (void *)data; data += encr_secsize; - crp->crp_flags = CRYPTO_F_CBIFSYNC | CRYPTO_F_REL; + crp->crp_flags = CRYPTO_F_CBIFSYNC; if (g_eli_batch) crp->crp_flags |= CRYPTO_F_BATCH; if (bp->bio_cmd == BIO_WRITE) { crp->crp_callback = g_eli_auth_write_done; crp->crp_desc = crde; crde->crd_next = crda; crda->crd_next = NULL; } else { crp->crp_callback = g_eli_auth_read_done; crp->crp_desc = crda; crda->crd_next = crde; crde->crd_next = NULL; } crde->crd_skip = sc->sc_alen; crde->crd_len = data_secsize; crde->crd_flags = CRD_F_IV_EXPLICIT | CRD_F_IV_PRESENT; if ((sc->sc_flags & G_ELI_FLAG_FIRST_KEY) == 0) crde->crd_flags |= CRD_F_KEY_EXPLICIT; if (bp->bio_cmd == BIO_WRITE) crde->crd_flags |= CRD_F_ENCRYPT; crde->crd_alg = sc->sc_ealgo; crde->crd_key = g_eli_key_hold(sc, dstoff, encr_secsize); crde->crd_klen = sc->sc_ekeylen; if (sc->sc_ealgo == CRYPTO_AES_XTS) crde->crd_klen <<= 1; g_eli_crypto_ivgen(sc, dstoff, crde->crd_iv, sizeof(crde->crd_iv)); crda->crd_skip = sc->sc_alen; crda->crd_len = data_secsize; crda->crd_inject = 0; crda->crd_flags = CRD_F_KEY_EXPLICIT; crda->crd_alg = sc->sc_aalgo; g_eli_auth_keygen(sc, dstoff, authkey); crda->crd_key = authkey; crda->crd_klen = G_ELI_AUTH_SECKEYLEN * 8; crp->crp_etype = 0; err = crypto_dispatch(crp); if (err != 0 && error == 0) error = err; } if (bp->bio_error == 0) bp->bio_error = error; } Index: projects/clang350-import/sys/geom/eli/g_eli_privacy.c =================================================================== --- projects/clang350-import/sys/geom/eli/g_eli_privacy.c (revision 275748) +++ projects/clang350-import/sys/geom/eli/g_eli_privacy.c (revision 275749) @@ -1,317 +1,317 @@ /*- * Copyright (c) 2005-2011 Pawel Jakub Dawidek * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHORS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #include __FBSDID("$FreeBSD$"); #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include /* * Code paths: * BIO_READ: * g_eli_start -> g_eli_crypto_read -> g_io_request -> g_eli_read_done -> g_eli_crypto_run -> g_eli_crypto_read_done -> g_io_deliver * BIO_WRITE: * g_eli_start -> g_eli_crypto_run -> g_eli_crypto_write_done -> g_io_request -> g_eli_write_done -> g_io_deliver */ MALLOC_DECLARE(M_ELI); /* * The function is called after we read and decrypt data. * * g_eli_start -> g_eli_crypto_read -> g_io_request -> g_eli_read_done -> g_eli_crypto_run -> G_ELI_CRYPTO_READ_DONE -> g_io_deliver */ static int g_eli_crypto_read_done(struct cryptop *crp) { struct g_eli_softc *sc; struct bio *bp; if (crp->crp_etype == EAGAIN) { if (g_eli_crypto_rerun(crp) == 0) return (0); } bp = (struct bio *)crp->crp_opaque; bp->bio_inbed++; if (crp->crp_etype == 0) { G_ELI_DEBUG(3, "Crypto READ request done (%d/%d).", bp->bio_inbed, bp->bio_children); bp->bio_completed += crp->crp_olen; } else { G_ELI_DEBUG(1, "Crypto READ request failed (%d/%d) error=%d.", bp->bio_inbed, bp->bio_children, crp->crp_etype); if (bp->bio_error == 0) bp->bio_error = crp->crp_etype; } sc = bp->bio_to->geom->softc; g_eli_key_drop(sc, crp->crp_desc->crd_key); /* * Do we have all sectors already? */ if (bp->bio_inbed < bp->bio_children) return (0); free(bp->bio_driver2, M_ELI); bp->bio_driver2 = NULL; if (bp->bio_error != 0) { G_ELI_LOGREQ(0, bp, "Crypto READ request failed (error=%d).", bp->bio_error); bp->bio_completed = 0; } /* * Read is finished, send it up. */ g_io_deliver(bp, bp->bio_error); atomic_subtract_int(&sc->sc_inflight, 1); return (0); } /* * The function is called after data encryption. * * g_eli_start -> g_eli_crypto_run -> G_ELI_CRYPTO_WRITE_DONE -> g_io_request -> g_eli_write_done -> g_io_deliver */ static int g_eli_crypto_write_done(struct cryptop *crp) { struct g_eli_softc *sc; struct g_geom *gp; struct g_consumer *cp; struct bio *bp, *cbp; if (crp->crp_etype == EAGAIN) { if (g_eli_crypto_rerun(crp) == 0) return (0); } bp = (struct bio *)crp->crp_opaque; bp->bio_inbed++; if (crp->crp_etype == 0) { G_ELI_DEBUG(3, "Crypto WRITE request done (%d/%d).", bp->bio_inbed, bp->bio_children); } else { G_ELI_DEBUG(1, "Crypto WRITE request failed (%d/%d) error=%d.", bp->bio_inbed, bp->bio_children, crp->crp_etype); if (bp->bio_error == 0) bp->bio_error = crp->crp_etype; } gp = bp->bio_to->geom; sc = gp->softc; g_eli_key_drop(sc, crp->crp_desc->crd_key); /* * All sectors are already encrypted? */ if (bp->bio_inbed < bp->bio_children) return (0); bp->bio_inbed = 0; bp->bio_children = 1; cbp = bp->bio_driver1; bp->bio_driver1 = NULL; if (bp->bio_error != 0) { G_ELI_LOGREQ(0, bp, "Crypto WRITE request failed (error=%d).", bp->bio_error); free(bp->bio_driver2, M_ELI); bp->bio_driver2 = NULL; g_destroy_bio(cbp); g_io_deliver(bp, bp->bio_error); atomic_subtract_int(&sc->sc_inflight, 1); return (0); } cbp->bio_data = bp->bio_driver2; cbp->bio_done = g_eli_write_done; cp = LIST_FIRST(&gp->consumer); cbp->bio_to = cp->provider; G_ELI_LOGREQ(2, cbp, "Sending request."); /* * Send encrypted data to the provider. */ g_io_request(cbp, cp); return (0); } /* * The function is called to read encrypted data. * * g_eli_start -> G_ELI_CRYPTO_READ -> g_io_request -> g_eli_read_done -> g_eli_crypto_run -> g_eli_crypto_read_done -> g_io_deliver */ void g_eli_crypto_read(struct g_eli_softc *sc, struct bio *bp, boolean_t fromworker) { struct g_consumer *cp; struct bio *cbp; if (!fromworker) { /* * We are not called from the worker thread, so check if * device is suspended. */ mtx_lock(&sc->sc_queue_mtx); if (sc->sc_flags & G_ELI_FLAG_SUSPEND) { /* * If device is suspended, we place the request onto * the queue, so it can be handled after resume. */ G_ELI_DEBUG(0, "device suspended, move onto queue"); bioq_insert_tail(&sc->sc_queue, bp); mtx_unlock(&sc->sc_queue_mtx); wakeup(sc); return; } atomic_add_int(&sc->sc_inflight, 1); mtx_unlock(&sc->sc_queue_mtx); } bp->bio_pflags = 0; bp->bio_driver2 = NULL; cbp = bp->bio_driver1; cbp->bio_done = g_eli_read_done; cp = LIST_FIRST(&sc->sc_geom->consumer); cbp->bio_to = cp->provider; G_ELI_LOGREQ(2, cbp, "Sending request."); /* * Read encrypted data from provider. */ g_io_request(cbp, cp); } /* * This is the main function responsible for cryptography (ie. communication * with crypto(9) subsystem). * * BIO_READ: * g_eli_start -> g_eli_crypto_read -> g_io_request -> g_eli_read_done -> G_ELI_CRYPTO_RUN -> g_eli_crypto_read_done -> g_io_deliver * BIO_WRITE: * g_eli_start -> G_ELI_CRYPTO_RUN -> g_eli_crypto_write_done -> g_io_request -> g_eli_write_done -> g_io_deliver */ void g_eli_crypto_run(struct g_eli_worker *wr, struct bio *bp) { struct g_eli_softc *sc; struct cryptop *crp; struct cryptodesc *crd; u_int i, nsec, secsize; int err, error; off_t dstoff; size_t size; u_char *p, *data; G_ELI_LOGREQ(3, bp, "%s", __func__); bp->bio_pflags = wr->w_number; sc = wr->w_softc; secsize = LIST_FIRST(&sc->sc_geom->provider)->sectorsize; nsec = bp->bio_length / secsize; /* * Calculate how much memory do we need. * We need separate crypto operation for every single sector. * It is much faster to calculate total amount of needed memory here and * do the allocation once instead of allocating memory in pieces (many, * many pieces). */ size = sizeof(*crp) * nsec; size += sizeof(*crd) * nsec; /* * If we write the data we cannot destroy current bio_data content, * so we need to allocate more memory for encrypted data. */ if (bp->bio_cmd == BIO_WRITE) size += bp->bio_length; p = malloc(size, M_ELI, M_WAITOK); bp->bio_inbed = 0; bp->bio_children = nsec; bp->bio_driver2 = p; if (bp->bio_cmd == BIO_READ) data = bp->bio_data; else { data = p; p += bp->bio_length; bcopy(bp->bio_data, data, bp->bio_length); } error = 0; for (i = 0, dstoff = bp->bio_offset; i < nsec; i++, dstoff += secsize) { crp = (struct cryptop *)p; p += sizeof(*crp); crd = (struct cryptodesc *)p; p += sizeof(*crd); crp->crp_sid = wr->w_sid; crp->crp_ilen = secsize; crp->crp_olen = secsize; crp->crp_opaque = (void *)bp; crp->crp_buf = (void *)data; data += secsize; if (bp->bio_cmd == BIO_WRITE) crp->crp_callback = g_eli_crypto_write_done; else /* if (bp->bio_cmd == BIO_READ) */ crp->crp_callback = g_eli_crypto_read_done; - crp->crp_flags = CRYPTO_F_CBIFSYNC | CRYPTO_F_REL; + crp->crp_flags = CRYPTO_F_CBIFSYNC; if (g_eli_batch) crp->crp_flags |= CRYPTO_F_BATCH; crp->crp_desc = crd; crd->crd_skip = 0; crd->crd_len = secsize; crd->crd_flags = CRD_F_IV_EXPLICIT | CRD_F_IV_PRESENT; if ((sc->sc_flags & G_ELI_FLAG_SINGLE_KEY) == 0) crd->crd_flags |= CRD_F_KEY_EXPLICIT; if (bp->bio_cmd == BIO_WRITE) crd->crd_flags |= CRD_F_ENCRYPT; crd->crd_alg = sc->sc_ealgo; crd->crd_key = g_eli_key_hold(sc, dstoff, secsize); crd->crd_klen = sc->sc_ekeylen; if (sc->sc_ealgo == CRYPTO_AES_XTS) crd->crd_klen <<= 1; g_eli_crypto_ivgen(sc, dstoff, crd->crd_iv, sizeof(crd->crd_iv)); crd->crd_next = NULL; crp->crp_etype = 0; err = crypto_dispatch(crp); if (error == 0) error = err; } if (bp->bio_error == 0) bp->bio_error = error; } Index: projects/clang350-import/sys/kern/kern_exec.c =================================================================== --- projects/clang350-import/sys/kern/kern_exec.c (revision 275748) +++ projects/clang350-import/sys/kern/kern_exec.c (revision 275749) @@ -1,1499 +1,1499 @@ /*- * Copyright (c) 1993, David Greenman * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #include __FBSDID("$FreeBSD$"); #include "opt_capsicum.h" #include "opt_hwpmc_hooks.h" #include "opt_ktrace.h" #include "opt_vm.h" #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #ifdef KTRACE #include #endif #include #include #include #include #include #include #include #include #include #ifdef HWPMC_HOOKS #include #endif #include #include #include #ifdef KDTRACE_HOOKS #include dtrace_execexit_func_t dtrace_fasttrap_exec; #endif SDT_PROVIDER_DECLARE(proc); SDT_PROBE_DEFINE1(proc, kernel, , exec, "char *"); SDT_PROBE_DEFINE1(proc, kernel, , exec__failure, "int"); SDT_PROBE_DEFINE1(proc, kernel, , exec__success, "char *"); MALLOC_DEFINE(M_PARGS, "proc-args", "Process arguments"); static int sysctl_kern_ps_strings(SYSCTL_HANDLER_ARGS); static int sysctl_kern_usrstack(SYSCTL_HANDLER_ARGS); static int sysctl_kern_stackprot(SYSCTL_HANDLER_ARGS); static int do_execve(struct thread *td, struct image_args *args, struct mac *mac_p); /* XXX This should be vm_size_t. */ SYSCTL_PROC(_kern, KERN_PS_STRINGS, ps_strings, CTLTYPE_ULONG|CTLFLAG_RD, NULL, 0, sysctl_kern_ps_strings, "LU", ""); /* XXX This should be vm_size_t. */ SYSCTL_PROC(_kern, KERN_USRSTACK, usrstack, CTLTYPE_ULONG|CTLFLAG_RD| CTLFLAG_CAPRD, NULL, 0, sysctl_kern_usrstack, "LU", ""); SYSCTL_PROC(_kern, OID_AUTO, stackprot, CTLTYPE_INT|CTLFLAG_RD, NULL, 0, sysctl_kern_stackprot, "I", ""); u_long ps_arg_cache_limit = PAGE_SIZE / 16; SYSCTL_ULONG(_kern, OID_AUTO, ps_arg_cache_limit, CTLFLAG_RW, &ps_arg_cache_limit, 0, ""); static int disallow_high_osrel; SYSCTL_INT(_kern, OID_AUTO, disallow_high_osrel, CTLFLAG_RW, &disallow_high_osrel, 0, "Disallow execution of binaries built for higher version of the world"); static int map_at_zero = 0; SYSCTL_INT(_security_bsd, OID_AUTO, map_at_zero, CTLFLAG_RWTUN, &map_at_zero, 0, "Permit processes to map an object at virtual address 0."); static int sysctl_kern_ps_strings(SYSCTL_HANDLER_ARGS) { struct proc *p; int error; p = curproc; #ifdef SCTL_MASK32 if (req->flags & SCTL_MASK32) { unsigned int val; val = (unsigned int)p->p_sysent->sv_psstrings; error = SYSCTL_OUT(req, &val, sizeof(val)); } else #endif error = SYSCTL_OUT(req, &p->p_sysent->sv_psstrings, sizeof(p->p_sysent->sv_psstrings)); return error; } static int sysctl_kern_usrstack(SYSCTL_HANDLER_ARGS) { struct proc *p; int error; p = curproc; #ifdef SCTL_MASK32 if (req->flags & SCTL_MASK32) { unsigned int val; val = (unsigned int)p->p_sysent->sv_usrstack; error = SYSCTL_OUT(req, &val, sizeof(val)); } else #endif error = SYSCTL_OUT(req, &p->p_sysent->sv_usrstack, sizeof(p->p_sysent->sv_usrstack)); return error; } static int sysctl_kern_stackprot(SYSCTL_HANDLER_ARGS) { struct proc *p; p = curproc; return (SYSCTL_OUT(req, &p->p_sysent->sv_stackprot, sizeof(p->p_sysent->sv_stackprot))); } /* * Each of the items is a pointer to a `const struct execsw', hence the * double pointer here. */ static const struct execsw **execsw; #ifndef _SYS_SYSPROTO_H_ struct execve_args { char *fname; char **argv; char **envv; }; #endif int sys_execve(td, uap) struct thread *td; struct execve_args /* { char *fname; char **argv; char **envv; } */ *uap; { int error; struct image_args args; error = exec_copyin_args(&args, uap->fname, UIO_USERSPACE, uap->argv, uap->envv); if (error == 0) error = kern_execve(td, &args, NULL); return (error); } #ifndef _SYS_SYSPROTO_H_ struct fexecve_args { int fd; char **argv; char **envv; } #endif int sys_fexecve(struct thread *td, struct fexecve_args *uap) { int error; struct image_args args; error = exec_copyin_args(&args, NULL, UIO_SYSSPACE, uap->argv, uap->envv); if (error == 0) { args.fd = uap->fd; error = kern_execve(td, &args, NULL); } return (error); } #ifndef _SYS_SYSPROTO_H_ struct __mac_execve_args { char *fname; char **argv; char **envv; struct mac *mac_p; }; #endif int sys___mac_execve(td, uap) struct thread *td; struct __mac_execve_args /* { char *fname; char **argv; char **envv; struct mac *mac_p; } */ *uap; { #ifdef MAC int error; struct image_args args; error = exec_copyin_args(&args, uap->fname, UIO_USERSPACE, uap->argv, uap->envv); if (error == 0) error = kern_execve(td, &args, uap->mac_p); return (error); #else return (ENOSYS); #endif } /* * XXX: kern_execve has the astonishing property of not always returning to * the caller. If sufficiently bad things happen during the call to * do_execve(), it can end up calling exit1(); as a result, callers must * avoid doing anything which they might need to undo (e.g., allocating * memory). */ int kern_execve(td, args, mac_p) struct thread *td; struct image_args *args; struct mac *mac_p; { struct proc *p = td->td_proc; struct vmspace *oldvmspace; int error; AUDIT_ARG_ARGV(args->begin_argv, args->argc, args->begin_envv - args->begin_argv); AUDIT_ARG_ENVV(args->begin_envv, args->envc, args->endp - args->begin_envv); if (p->p_flag & P_HADTHREADS) { PROC_LOCK(p); - if (thread_single(SINGLE_BOUNDARY)) { + if (thread_single(p, SINGLE_BOUNDARY)) { PROC_UNLOCK(p); exec_free_args(args); return (ERESTART); /* Try again later. */ } PROC_UNLOCK(p); } KASSERT((td->td_pflags & TDP_EXECVMSPC) == 0, ("nested execve")); oldvmspace = td->td_proc->p_vmspace; error = do_execve(td, args, mac_p); if (p->p_flag & P_HADTHREADS) { PROC_LOCK(p); /* * If success, we upgrade to SINGLE_EXIT state to * force other threads to suicide. */ if (error == 0) - thread_single(SINGLE_EXIT); + thread_single(p, SINGLE_EXIT); else - thread_single_end(); + thread_single_end(p, SINGLE_BOUNDARY); PROC_UNLOCK(p); } if ((td->td_pflags & TDP_EXECVMSPC) != 0) { KASSERT(td->td_proc->p_vmspace != oldvmspace, ("oldvmspace still used")); vmspace_free(oldvmspace); td->td_pflags &= ~TDP_EXECVMSPC; } return (error); } /* * In-kernel implementation of execve(). All arguments are assumed to be * userspace pointers from the passed thread. */ static int do_execve(td, args, mac_p) struct thread *td; struct image_args *args; struct mac *mac_p; { struct proc *p = td->td_proc; struct nameidata nd; struct ucred *newcred = NULL, *oldcred; struct uidinfo *euip = NULL; register_t *stack_base; int error, i; struct image_params image_params, *imgp; struct vattr attr; int (*img_first)(struct image_params *); struct pargs *oldargs = NULL, *newargs = NULL; struct sigacts *oldsigacts, *newsigacts; #ifdef KTRACE struct vnode *tracevp = NULL; struct ucred *tracecred = NULL; #endif struct vnode *textvp = NULL, *binvp; cap_rights_t rights; int credential_changing; int textset; #ifdef MAC struct label *interpvplabel = NULL; int will_transition; #endif #ifdef HWPMC_HOOKS struct pmckern_procexec pe; #endif static const char fexecv_proc_title[] = "(fexecv)"; imgp = &image_params; /* * Lock the process and set the P_INEXEC flag to indicate that * it should be left alone until we're done here. This is * necessary to avoid race conditions - e.g. in ptrace() - * that might allow a local user to illicitly obtain elevated * privileges. */ PROC_LOCK(p); KASSERT((p->p_flag & P_INEXEC) == 0, ("%s(): process already has P_INEXEC flag", __func__)); p->p_flag |= P_INEXEC; PROC_UNLOCK(p); /* * Initialize part of the common data */ bzero(imgp, sizeof(*imgp)); imgp->proc = p; imgp->attr = &attr; imgp->args = args; #ifdef MAC error = mac_execve_enter(imgp, mac_p); if (error) goto exec_fail; #endif /* * Translate the file name. namei() returns a vnode pointer * in ni_vp amoung other things. * * XXXAUDIT: It would be desirable to also audit the name of the * interpreter if this is an interpreted binary. */ if (args->fname != NULL) { NDINIT(&nd, LOOKUP, ISOPEN | LOCKLEAF | FOLLOW | SAVENAME | AUDITVNODE1, UIO_SYSSPACE, args->fname, td); } SDT_PROBE(proc, kernel, , exec, args->fname, 0, 0, 0, 0 ); interpret: if (args->fname != NULL) { #ifdef CAPABILITY_MODE /* * While capability mode can't reach this point via direct * path arguments to execve(), we also don't allow * interpreters to be used in capability mode (for now). * Catch indirect lookups and return a permissions error. */ if (IN_CAPABILITY_MODE(td)) { error = ECAPMODE; goto exec_fail; } #endif error = namei(&nd); if (error) goto exec_fail; binvp = nd.ni_vp; imgp->vp = binvp; } else { AUDIT_ARG_FD(args->fd); /* * Descriptors opened only with O_EXEC or O_RDONLY are allowed. */ error = fgetvp_exec(td, args->fd, cap_rights_init(&rights, CAP_FEXECVE), &binvp); if (error) goto exec_fail; vn_lock(binvp, LK_EXCLUSIVE | LK_RETRY); AUDIT_ARG_VNODE1(binvp); imgp->vp = binvp; } /* * Check file permissions (also 'opens' file) */ error = exec_check_permissions(imgp); if (error) goto exec_fail_dealloc; imgp->object = imgp->vp->v_object; if (imgp->object != NULL) vm_object_reference(imgp->object); /* * Set VV_TEXT now so no one can write to the executable while we're * activating it. * * Remember if this was set before and unset it in case this is not * actually an executable image. */ textset = VOP_IS_TEXT(imgp->vp); VOP_SET_TEXT(imgp->vp); error = exec_map_first_page(imgp); if (error) goto exec_fail_dealloc; imgp->proc->p_osrel = 0; /* * If the current process has a special image activator it * wants to try first, call it. For example, emulating shell * scripts differently. */ error = -1; if ((img_first = imgp->proc->p_sysent->sv_imgact_try) != NULL) error = img_first(imgp); /* * Loop through the list of image activators, calling each one. * An activator returns -1 if there is no match, 0 on success, * and an error otherwise. */ for (i = 0; error == -1 && execsw[i]; ++i) { if (execsw[i]->ex_imgact == NULL || execsw[i]->ex_imgact == img_first) { continue; } error = (*execsw[i]->ex_imgact)(imgp); } if (error) { if (error == -1) { if (textset == 0) VOP_UNSET_TEXT(imgp->vp); error = ENOEXEC; } goto exec_fail_dealloc; } /* * Special interpreter operation, cleanup and loop up to try to * activate the interpreter. */ if (imgp->interpreted) { exec_unmap_first_page(imgp); /* * VV_TEXT needs to be unset for scripts. There is a short * period before we determine that something is a script where * VV_TEXT will be set. The vnode lock is held over this * entire period so nothing should illegitimately be blocked. */ VOP_UNSET_TEXT(imgp->vp); /* free name buffer and old vnode */ if (args->fname != NULL) NDFREE(&nd, NDF_ONLY_PNBUF); #ifdef MAC mac_execve_interpreter_enter(binvp, &interpvplabel); #endif if (imgp->opened) { VOP_CLOSE(binvp, FREAD, td->td_ucred, td); imgp->opened = 0; } vput(binvp); vm_object_deallocate(imgp->object); imgp->object = NULL; /* set new name to that of the interpreter */ NDINIT(&nd, LOOKUP, LOCKLEAF | FOLLOW | SAVENAME, UIO_SYSSPACE, imgp->interpreter_name, td); args->fname = imgp->interpreter_name; goto interpret; } /* * NB: We unlock the vnode here because it is believed that none * of the sv_copyout_strings/sv_fixup operations require the vnode. */ VOP_UNLOCK(imgp->vp, 0); /* * Do the best to calculate the full path to the image file. */ if (imgp->auxargs != NULL && ((args->fname != NULL && args->fname[0] == '/') || vn_fullpath(td, imgp->vp, &imgp->execpath, &imgp->freepath) != 0)) imgp->execpath = args->fname; if (disallow_high_osrel && P_OSREL_MAJOR(p->p_osrel) > P_OSREL_MAJOR(__FreeBSD_version)) { error = ENOEXEC; uprintf("Osrel %d for image %s too high\n", p->p_osrel, imgp->execpath != NULL ? imgp->execpath : ""); vn_lock(imgp->vp, LK_SHARED | LK_RETRY); goto exec_fail_dealloc; } /* * Copy out strings (args and env) and initialize stack base */ if (p->p_sysent->sv_copyout_strings) stack_base = (*p->p_sysent->sv_copyout_strings)(imgp); else stack_base = exec_copyout_strings(imgp); /* * If custom stack fixup routine present for this process * let it do the stack setup. * Else stuff argument count as first item on stack */ if (p->p_sysent->sv_fixup != NULL) (*p->p_sysent->sv_fixup)(&stack_base, imgp); else suword(--stack_base, imgp->args->argc); /* * For security and other reasons, the file descriptor table cannot * be shared after an exec. */ fdunshare(td); /* close files on exec */ fdcloseexec(td); /* * Malloc things before we need locks. */ i = imgp->args->begin_envv - imgp->args->begin_argv; /* Cache arguments if they fit inside our allowance */ if (ps_arg_cache_limit >= i + sizeof(struct pargs)) { newargs = pargs_alloc(i); bcopy(imgp->args->begin_argv, newargs->ar_args, i); } vn_lock(imgp->vp, LK_SHARED | LK_RETRY); /* Get a reference to the vnode prior to locking the proc */ VREF(binvp); /* * For security and other reasons, signal handlers cannot * be shared after an exec. The new process gets a copy of the old * handlers. In execsigs(), the new process will have its signals * reset. */ if (sigacts_shared(p->p_sigacts)) { oldsigacts = p->p_sigacts; newsigacts = sigacts_alloc(); sigacts_copy(newsigacts, oldsigacts); } else { oldsigacts = NULL; newsigacts = NULL; /* satisfy gcc */ } PROC_LOCK(p); if (oldsigacts) p->p_sigacts = newsigacts; oldcred = p->p_ucred; /* Stop profiling */ stopprofclock(p); /* reset caught signals */ execsigs(p); /* name this process - nameiexec(p, ndp) */ bzero(p->p_comm, sizeof(p->p_comm)); if (args->fname) bcopy(nd.ni_cnd.cn_nameptr, p->p_comm, min(nd.ni_cnd.cn_namelen, MAXCOMLEN)); else if (vn_commname(binvp, p->p_comm, sizeof(p->p_comm)) != 0) bcopy(fexecv_proc_title, p->p_comm, sizeof(fexecv_proc_title)); bcopy(p->p_comm, td->td_name, sizeof(td->td_name)); #ifdef KTR sched_clear_tdname(td); #endif /* * mark as execed, wakeup the process that vforked (if any) and tell * it that it now has its own resources back */ p->p_flag |= P_EXEC; if (p->p_flag & P_PPWAIT) { p->p_flag &= ~(P_PPWAIT | P_PPTRACE); cv_broadcast(&p->p_pwait); } /* * Implement image setuid/setgid. * * Don't honor setuid/setgid if the filesystem prohibits it or if * the process is being traced. * * We disable setuid/setgid/etc in compatibility mode on the basis * that most setugid applications are not written with that * environment in mind, and will therefore almost certainly operate * incorrectly. In principle there's no reason that setugid * applications might not be useful in capability mode, so we may want * to reconsider this conservative design choice in the future. * * XXXMAC: For the time being, use NOSUID to also prohibit * transitions on the file system. */ credential_changing = 0; credential_changing |= (attr.va_mode & S_ISUID) && oldcred->cr_uid != attr.va_uid; credential_changing |= (attr.va_mode & S_ISGID) && oldcred->cr_gid != attr.va_gid; #ifdef MAC will_transition = mac_vnode_execve_will_transition(oldcred, imgp->vp, interpvplabel, imgp); credential_changing |= will_transition; #endif if (credential_changing && #ifdef CAPABILITY_MODE ((oldcred->cr_flags & CRED_FLAG_CAPMODE) == 0) && #endif (imgp->vp->v_mount->mnt_flag & MNT_NOSUID) == 0 && (p->p_flag & P_TRACED) == 0) { /* * Turn off syscall tracing for set-id programs, except for * root. Record any set-id flags first to make sure that * we do not regain any tracing during a possible block. */ setsugid(p); #ifdef KTRACE if (p->p_tracecred != NULL && priv_check_cred(p->p_tracecred, PRIV_DEBUG_DIFFCRED, 0)) ktrprocexec(p, &tracecred, &tracevp); #endif /* * Close any file descriptors 0..2 that reference procfs, * then make sure file descriptors 0..2 are in use. * * Both fdsetugidsafety() and fdcheckstd() may call functions * taking sleepable locks, so temporarily drop our locks. */ PROC_UNLOCK(p); VOP_UNLOCK(imgp->vp, 0); fdsetugidsafety(td); error = fdcheckstd(td); if (error != 0) goto done1; newcred = crdup(oldcred); euip = uifind(attr.va_uid); vn_lock(imgp->vp, LK_SHARED | LK_RETRY); PROC_LOCK(p); /* * Set the new credentials. */ if (attr.va_mode & S_ISUID) change_euid(newcred, euip); if (attr.va_mode & S_ISGID) change_egid(newcred, attr.va_gid); #ifdef MAC if (will_transition) { mac_vnode_execve_transition(oldcred, newcred, imgp->vp, interpvplabel, imgp); } #endif /* * Implement correct POSIX saved-id behavior. * * XXXMAC: Note that the current logic will save the * uid and gid if a MAC domain transition occurs, even * though maybe it shouldn't. */ change_svuid(newcred, newcred->cr_uid); change_svgid(newcred, newcred->cr_gid); p->p_ucred = newcred; } else { if (oldcred->cr_uid == oldcred->cr_ruid && oldcred->cr_gid == oldcred->cr_rgid) p->p_flag &= ~P_SUGID; /* * Implement correct POSIX saved-id behavior. * * XXX: It's not clear that the existing behavior is * POSIX-compliant. A number of sources indicate that the * saved uid/gid should only be updated if the new ruid is * not equal to the old ruid, or the new euid is not equal * to the old euid and the new euid is not equal to the old * ruid. The FreeBSD code always updates the saved uid/gid. * Also, this code uses the new (replaced) euid and egid as * the source, which may or may not be the right ones to use. */ if (oldcred->cr_svuid != oldcred->cr_uid || oldcred->cr_svgid != oldcred->cr_gid) { PROC_UNLOCK(p); VOP_UNLOCK(imgp->vp, 0); newcred = crdup(oldcred); vn_lock(imgp->vp, LK_SHARED | LK_RETRY); PROC_LOCK(p); change_svuid(newcred, newcred->cr_uid); change_svgid(newcred, newcred->cr_gid); p->p_ucred = newcred; } } /* * Store the vp for use in procfs. This vnode was referenced prior * to locking the proc lock. */ textvp = p->p_textvp; p->p_textvp = binvp; #ifdef KDTRACE_HOOKS /* * Tell the DTrace fasttrap provider about the exec if it * has declared an interest. */ if (dtrace_fasttrap_exec) dtrace_fasttrap_exec(p); #endif /* * Notify others that we exec'd, and clear the P_INEXEC flag * as we're now a bona fide freshly-execed process. */ KNOTE_LOCKED(&p->p_klist, NOTE_EXEC); p->p_flag &= ~P_INEXEC; /* clear "fork but no exec" flag, as we _are_ execing */ p->p_acflag &= ~AFORK; /* * Free any previous argument cache and replace it with * the new argument cache, if any. */ oldargs = p->p_args; p->p_args = newargs; newargs = NULL; #ifdef HWPMC_HOOKS /* * Check if system-wide sampling is in effect or if the * current process is using PMCs. If so, do exec() time * processing. This processing needs to happen AFTER the * P_INEXEC flag is cleared. * * The proc lock needs to be released before taking the PMC * SX. */ if (PMC_SYSTEM_SAMPLING_ACTIVE() || PMC_PROC_IS_USING_PMCS(p)) { PROC_UNLOCK(p); VOP_UNLOCK(imgp->vp, 0); pe.pm_credentialschanged = credential_changing; pe.pm_entryaddr = imgp->entry_addr; PMC_CALL_HOOK_X(td, PMC_FN_PROCESS_EXEC, (void *) &pe); vn_lock(imgp->vp, LK_SHARED | LK_RETRY); } else PROC_UNLOCK(p); #else /* !HWPMC_HOOKS */ PROC_UNLOCK(p); #endif /* Set values passed into the program in registers. */ if (p->p_sysent->sv_setregs) (*p->p_sysent->sv_setregs)(td, imgp, (u_long)(uintptr_t)stack_base); else exec_setregs(td, imgp, (u_long)(uintptr_t)stack_base); vfs_mark_atime(imgp->vp, td->td_ucred); SDT_PROBE(proc, kernel, , exec__success, args->fname, 0, 0, 0, 0); VOP_UNLOCK(imgp->vp, 0); done1: /* * Free any resources malloc'd earlier that we didn't use. */ if (euip != NULL) uifree(euip); if (newcred != NULL) crfree(oldcred); /* * Handle deferred decrement of ref counts. */ if (textvp != NULL) vrele(textvp); if (error != 0) vrele(binvp); #ifdef KTRACE if (tracevp != NULL) vrele(tracevp); if (tracecred != NULL) crfree(tracecred); #endif vn_lock(imgp->vp, LK_SHARED | LK_RETRY); pargs_drop(oldargs); pargs_drop(newargs); if (oldsigacts != NULL) sigacts_free(oldsigacts); exec_fail_dealloc: /* * free various allocated resources */ if (imgp->firstpage != NULL) exec_unmap_first_page(imgp); if (imgp->vp != NULL) { if (args->fname) NDFREE(&nd, NDF_ONLY_PNBUF); if (imgp->opened) VOP_CLOSE(imgp->vp, FREAD, td->td_ucred, td); vput(imgp->vp); } if (imgp->object != NULL) vm_object_deallocate(imgp->object); free(imgp->freepath, M_TEMP); if (error == 0) { PROC_LOCK(p); td->td_dbgflags |= TDB_EXEC; PROC_UNLOCK(p); /* * Stop the process here if its stop event mask has * the S_EXEC bit set. */ STOPEVENT(p, S_EXEC, 0); goto done2; } exec_fail: /* we're done here, clear P_INEXEC */ PROC_LOCK(p); p->p_flag &= ~P_INEXEC; PROC_UNLOCK(p); SDT_PROBE(proc, kernel, , exec__failure, error, 0, 0, 0, 0); done2: #ifdef MAC mac_execve_exit(imgp); mac_execve_interpreter_exit(interpvplabel); #endif exec_free_args(args); if (error && imgp->vmspace_destroyed) { /* sorry, no more process anymore. exit gracefully */ exit1(td, W_EXITCODE(0, SIGABRT)); /* NOT REACHED */ } #ifdef KTRACE if (error == 0) ktrprocctor(p); #endif return (error); } int exec_map_first_page(imgp) struct image_params *imgp; { int rv, i; int initial_pagein; vm_page_t ma[VM_INITIAL_PAGEIN]; vm_object_t object; if (imgp->firstpage != NULL) exec_unmap_first_page(imgp); object = imgp->vp->v_object; if (object == NULL) return (EACCES); VM_OBJECT_WLOCK(object); #if VM_NRESERVLEVEL > 0 if ((object->flags & OBJ_COLORED) == 0) { object->flags |= OBJ_COLORED; object->pg_color = 0; } #endif ma[0] = vm_page_grab(object, 0, VM_ALLOC_NORMAL); if (ma[0]->valid != VM_PAGE_BITS_ALL) { initial_pagein = VM_INITIAL_PAGEIN; if (initial_pagein > object->size) initial_pagein = object->size; for (i = 1; i < initial_pagein; i++) { if ((ma[i] = vm_page_next(ma[i - 1])) != NULL) { if (ma[i]->valid) break; if (vm_page_tryxbusy(ma[i])) break; } else { ma[i] = vm_page_alloc(object, i, VM_ALLOC_NORMAL | VM_ALLOC_IFNOTCACHED); if (ma[i] == NULL) break; } } initial_pagein = i; rv = vm_pager_get_pages(object, ma, initial_pagein, 0); ma[0] = vm_page_lookup(object, 0); if ((rv != VM_PAGER_OK) || (ma[0] == NULL)) { if (ma[0] != NULL) { vm_page_lock(ma[0]); vm_page_free(ma[0]); vm_page_unlock(ma[0]); } VM_OBJECT_WUNLOCK(object); return (EIO); } } vm_page_xunbusy(ma[0]); vm_page_lock(ma[0]); vm_page_hold(ma[0]); vm_page_activate(ma[0]); vm_page_unlock(ma[0]); VM_OBJECT_WUNLOCK(object); imgp->firstpage = sf_buf_alloc(ma[0], 0); imgp->image_header = (char *)sf_buf_kva(imgp->firstpage); return (0); } void exec_unmap_first_page(imgp) struct image_params *imgp; { vm_page_t m; if (imgp->firstpage != NULL) { m = sf_buf_page(imgp->firstpage); sf_buf_free(imgp->firstpage); imgp->firstpage = NULL; vm_page_lock(m); vm_page_unhold(m); vm_page_unlock(m); } } /* * Destroy old address space, and allocate a new stack * The new stack is only SGROWSIZ large because it is grown * automatically in trap.c. */ int exec_new_vmspace(imgp, sv) struct image_params *imgp; struct sysentvec *sv; { int error; struct proc *p = imgp->proc; struct vmspace *vmspace = p->p_vmspace; vm_object_t obj; vm_offset_t sv_minuser, stack_addr; vm_map_t map; u_long ssiz; imgp->vmspace_destroyed = 1; imgp->sysent = sv; /* May be called with Giant held */ EVENTHANDLER_INVOKE(process_exec, p, imgp); /* * Blow away entire process VM, if address space not shared, * otherwise, create a new VM space so that other threads are * not disrupted */ map = &vmspace->vm_map; if (map_at_zero) sv_minuser = sv->sv_minuser; else sv_minuser = MAX(sv->sv_minuser, PAGE_SIZE); if (vmspace->vm_refcnt == 1 && vm_map_min(map) == sv_minuser && vm_map_max(map) == sv->sv_maxuser) { shmexit(vmspace); pmap_remove_pages(vmspace_pmap(vmspace)); vm_map_remove(map, vm_map_min(map), vm_map_max(map)); } else { error = vmspace_exec(p, sv_minuser, sv->sv_maxuser); if (error) return (error); vmspace = p->p_vmspace; map = &vmspace->vm_map; } /* Map a shared page */ obj = sv->sv_shared_page_obj; if (obj != NULL) { vm_object_reference(obj); error = vm_map_fixed(map, obj, 0, sv->sv_shared_page_base, sv->sv_shared_page_len, VM_PROT_READ | VM_PROT_EXECUTE, VM_PROT_READ | VM_PROT_EXECUTE, MAP_INHERIT_SHARE | MAP_ACC_NO_CHARGE); if (error) { vm_object_deallocate(obj); return (error); } } /* Allocate a new stack */ if (sv->sv_maxssiz != NULL) ssiz = *sv->sv_maxssiz; else ssiz = maxssiz; stack_addr = sv->sv_usrstack - ssiz; error = vm_map_stack(map, stack_addr, (vm_size_t)ssiz, obj != NULL && imgp->stack_prot != 0 ? imgp->stack_prot : sv->sv_stackprot, VM_PROT_ALL, MAP_STACK_GROWS_DOWN); if (error) return (error); /* * vm_ssize and vm_maxsaddr are somewhat antiquated concepts, but they * are still used to enforce the stack rlimit on the process stack. */ vmspace->vm_ssize = sgrowsiz >> PAGE_SHIFT; vmspace->vm_maxsaddr = (char *)sv->sv_usrstack - ssiz; return (0); } /* * Copy out argument and environment strings from the old process address * space into the temporary string buffer. */ int exec_copyin_args(struct image_args *args, char *fname, enum uio_seg segflg, char **argv, char **envv) { u_long argp, envp; int error; size_t length; bzero(args, sizeof(*args)); if (argv == NULL) return (EFAULT); /* * Allocate demand-paged memory for the file name, argument, and * environment strings. */ error = exec_alloc_args(args); if (error != 0) return (error); /* * Copy the file name. */ if (fname != NULL) { args->fname = args->buf; error = (segflg == UIO_SYSSPACE) ? copystr(fname, args->fname, PATH_MAX, &length) : copyinstr(fname, args->fname, PATH_MAX, &length); if (error != 0) goto err_exit; } else length = 0; args->begin_argv = args->buf + length; args->endp = args->begin_argv; args->stringspace = ARG_MAX; /* * extract arguments first */ for (;;) { error = fueword(argv++, &argp); if (error == -1) { error = EFAULT; goto err_exit; } if (argp == 0) break; error = copyinstr((void *)(uintptr_t)argp, args->endp, args->stringspace, &length); if (error != 0) { if (error == ENAMETOOLONG) error = E2BIG; goto err_exit; } args->stringspace -= length; args->endp += length; args->argc++; } args->begin_envv = args->endp; /* * extract environment strings */ if (envv) { for (;;) { error = fueword(envv++, &envp); if (error == -1) { error = EFAULT; goto err_exit; } if (envp == 0) break; error = copyinstr((void *)(uintptr_t)envp, args->endp, args->stringspace, &length); if (error != 0) { if (error == ENAMETOOLONG) error = E2BIG; goto err_exit; } args->stringspace -= length; args->endp += length; args->envc++; } } return (0); err_exit: exec_free_args(args); return (error); } /* * Allocate temporary demand-paged, zero-filled memory for the file name, * argument, and environment strings. Returns zero if the allocation succeeds * and ENOMEM otherwise. */ int exec_alloc_args(struct image_args *args) { args->buf = (char *)kmap_alloc_wait(exec_map, PATH_MAX + ARG_MAX); return (args->buf != NULL ? 0 : ENOMEM); } void exec_free_args(struct image_args *args) { if (args->buf != NULL) { kmap_free_wakeup(exec_map, (vm_offset_t)args->buf, PATH_MAX + ARG_MAX); args->buf = NULL; } if (args->fname_buf != NULL) { free(args->fname_buf, M_TEMP); args->fname_buf = NULL; } } /* * Copy strings out to the new process address space, constructing new arg * and env vector tables. Return a pointer to the base so that it can be used * as the initial stack pointer. */ register_t * exec_copyout_strings(imgp) struct image_params *imgp; { int argc, envc; char **vectp; char *stringp; uintptr_t destp; register_t *stack_base; struct ps_strings *arginfo; struct proc *p; size_t execpath_len; int szsigcode, szps; char canary[sizeof(long) * 8]; szps = sizeof(pagesizes[0]) * MAXPAGESIZES; /* * Calculate string base and vector table pointers. * Also deal with signal trampoline code for this exec type. */ if (imgp->execpath != NULL && imgp->auxargs != NULL) execpath_len = strlen(imgp->execpath) + 1; else execpath_len = 0; p = imgp->proc; szsigcode = 0; arginfo = (struct ps_strings *)p->p_sysent->sv_psstrings; if (p->p_sysent->sv_sigcode_base == 0) { if (p->p_sysent->sv_szsigcode != NULL) szsigcode = *(p->p_sysent->sv_szsigcode); } destp = (uintptr_t)arginfo; /* * install sigcode */ if (szsigcode != 0) { destp -= szsigcode; destp = rounddown2(destp, sizeof(void *)); copyout(p->p_sysent->sv_sigcode, (void *)destp, szsigcode); } /* * Copy the image path for the rtld. */ if (execpath_len != 0) { destp -= execpath_len; imgp->execpathp = destp; copyout(imgp->execpath, (void *)destp, execpath_len); } /* * Prepare the canary for SSP. */ arc4rand(canary, sizeof(canary), 0); destp -= sizeof(canary); imgp->canary = destp; copyout(canary, (void *)destp, sizeof(canary)); imgp->canarylen = sizeof(canary); /* * Prepare the pagesizes array. */ destp -= szps; destp = rounddown2(destp, sizeof(void *)); imgp->pagesizes = destp; copyout(pagesizes, (void *)destp, szps); imgp->pagesizeslen = szps; destp -= ARG_MAX - imgp->args->stringspace; destp = rounddown2(destp, sizeof(void *)); /* * If we have a valid auxargs ptr, prepare some room * on the stack. */ if (imgp->auxargs) { /* * 'AT_COUNT*2' is size for the ELF Auxargs data. This is for * lower compatibility. */ imgp->auxarg_size = (imgp->auxarg_size) ? imgp->auxarg_size : (AT_COUNT * 2); /* * The '+ 2' is for the null pointers at the end of each of * the arg and env vector sets,and imgp->auxarg_size is room * for argument of Runtime loader. */ vectp = (char **)(destp - (imgp->args->argc + imgp->args->envc + 2 + imgp->auxarg_size) * sizeof(char *)); } else { /* * The '+ 2' is for the null pointers at the end of each of * the arg and env vector sets */ vectp = (char **)(destp - (imgp->args->argc + imgp->args->envc + 2) * sizeof(char *)); } /* * vectp also becomes our initial stack base */ stack_base = (register_t *)vectp; stringp = imgp->args->begin_argv; argc = imgp->args->argc; envc = imgp->args->envc; /* * Copy out strings - arguments and environment. */ copyout(stringp, (void *)destp, ARG_MAX - imgp->args->stringspace); /* * Fill in "ps_strings" struct for ps, w, etc. */ suword(&arginfo->ps_argvstr, (long)(intptr_t)vectp); suword32(&arginfo->ps_nargvstr, argc); /* * Fill in argument portion of vector table. */ for (; argc > 0; --argc) { suword(vectp++, (long)(intptr_t)destp); while (*stringp++ != 0) destp++; destp++; } /* a null vector table pointer separates the argp's from the envp's */ suword(vectp++, 0); suword(&arginfo->ps_envstr, (long)(intptr_t)vectp); suword32(&arginfo->ps_nenvstr, envc); /* * Fill in environment portion of vector table. */ for (; envc > 0; --envc) { suword(vectp++, (long)(intptr_t)destp); while (*stringp++ != 0) destp++; destp++; } /* end of vector table is a null pointer */ suword(vectp, 0); return (stack_base); } /* * Check permissions of file to execute. * Called with imgp->vp locked. * Return 0 for success or error code on failure. */ int exec_check_permissions(imgp) struct image_params *imgp; { struct vnode *vp = imgp->vp; struct vattr *attr = imgp->attr; struct thread *td; int error, writecount; td = curthread; /* Get file attributes */ error = VOP_GETATTR(vp, attr, td->td_ucred); if (error) return (error); #ifdef MAC error = mac_vnode_check_exec(td->td_ucred, imgp->vp, imgp); if (error) return (error); #endif /* * 1) Check if file execution is disabled for the filesystem that * this file resides on. * 2) Ensure that at least one execute bit is on. Otherwise, a * privileged user will always succeed, and we don't want this * to happen unless the file really is executable. * 3) Ensure that the file is a regular file. */ if ((vp->v_mount->mnt_flag & MNT_NOEXEC) || (attr->va_mode & (S_IXUSR | S_IXGRP | S_IXOTH)) == 0 || (attr->va_type != VREG)) return (EACCES); /* * Zero length files can't be exec'd */ if (attr->va_size == 0) return (ENOEXEC); /* * Check for execute permission to file based on current credentials. */ error = VOP_ACCESS(vp, VEXEC, td->td_ucred, td); if (error) return (error); /* * Check number of open-for-writes on the file and deny execution * if there are any. */ error = VOP_GET_WRITECOUNT(vp, &writecount); if (error != 0) return (error); if (writecount != 0) return (ETXTBSY); /* * Call filesystem specific open routine (which does nothing in the * general case). */ error = VOP_OPEN(vp, FREAD, td->td_ucred, td, NULL); if (error == 0) imgp->opened = 1; return (error); } /* * Exec handler registration */ int exec_register(execsw_arg) const struct execsw *execsw_arg; { const struct execsw **es, **xs, **newexecsw; int count = 2; /* New slot and trailing NULL */ if (execsw) for (es = execsw; *es; es++) count++; newexecsw = malloc(count * sizeof(*es), M_TEMP, M_WAITOK); if (newexecsw == NULL) return (ENOMEM); xs = newexecsw; if (execsw) for (es = execsw; *es; es++) *xs++ = *es; *xs++ = execsw_arg; *xs = NULL; if (execsw) free(execsw, M_TEMP); execsw = newexecsw; return (0); } int exec_unregister(execsw_arg) const struct execsw *execsw_arg; { const struct execsw **es, **xs, **newexecsw; int count = 1; if (execsw == NULL) panic("unregister with no handlers left?\n"); for (es = execsw; *es; es++) { if (*es == execsw_arg) break; } if (*es == NULL) return (ENOENT); for (es = execsw; *es; es++) if (*es != execsw_arg) count++; newexecsw = malloc(count * sizeof(*es), M_TEMP, M_WAITOK); if (newexecsw == NULL) return (ENOMEM); xs = newexecsw; for (es = execsw; *es; es++) if (*es != execsw_arg) *xs++ = *es; *xs = NULL; if (execsw) free(execsw, M_TEMP); execsw = newexecsw; return (0); } Index: projects/clang350-import/sys/kern/kern_exit.c =================================================================== --- projects/clang350-import/sys/kern/kern_exit.c (revision 275748) +++ projects/clang350-import/sys/kern/kern_exit.c (revision 275749) @@ -1,1314 +1,1314 @@ /*- * Copyright (c) 1982, 1986, 1989, 1991, 1993 * The Regents of the University of California. All rights reserved. * (c) UNIX System Laboratories, Inc. * All or some portions of this file are derived from material licensed * to the University of California by American Telephone and Telegraph * Co. or Unix System Laboratories, Inc. and are reproduced herein with * the permission of UNIX System Laboratories, Inc. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 4. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * @(#)kern_exit.c 8.7 (Berkeley) 2/12/94 */ #include __FBSDID("$FreeBSD$"); #include "opt_compat.h" #include "opt_ktrace.h" #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include /* for acct_process() function prototype */ #include #include #include #include #ifdef KTRACE #include #endif #include #include #include #include #include #include #include #include #include #ifdef KDTRACE_HOOKS #include dtrace_execexit_func_t dtrace_fasttrap_exit; #endif SDT_PROVIDER_DECLARE(proc); SDT_PROBE_DEFINE1(proc, kernel, , exit, "int"); /* Hook for NFS teardown procedure. */ void (*nlminfo_release_p)(struct proc *p); struct proc * proc_realparent(struct proc *child) { struct proc *p, *parent; sx_assert(&proctree_lock, SX_LOCKED); if ((child->p_treeflag & P_TREE_ORPHANED) == 0) { if (child->p_oppid == 0 || child->p_pptr->p_pid == child->p_oppid) parent = child->p_pptr; else parent = initproc; return (parent); } for (p = child; (p->p_treeflag & P_TREE_FIRST_ORPHAN) == 0;) { /* Cannot use LIST_PREV(), since the list head is not known. */ p = __containerof(p->p_orphan.le_prev, struct proc, p_orphan.le_next); KASSERT((p->p_treeflag & P_TREE_ORPHANED) != 0, ("missing P_ORPHAN %p", p)); } parent = __containerof(p->p_orphan.le_prev, struct proc, p_orphans.lh_first); return (parent); } static void clear_orphan(struct proc *p) { struct proc *p1; sx_assert(&proctree_lock, SA_XLOCKED); if ((p->p_treeflag & P_TREE_ORPHANED) == 0) return; if ((p->p_treeflag & P_TREE_FIRST_ORPHAN) != 0) { p1 = LIST_NEXT(p, p_orphan); if (p1 != NULL) p1->p_treeflag |= P_TREE_FIRST_ORPHAN; p->p_treeflag &= ~P_TREE_FIRST_ORPHAN; } LIST_REMOVE(p, p_orphan); p->p_treeflag &= ~P_TREE_ORPHANED; } /* * exit -- death of process. */ void sys_sys_exit(struct thread *td, struct sys_exit_args *uap) { exit1(td, W_EXITCODE(uap->rval, 0)); /* NOTREACHED */ } /* * Exit: deallocate address space and other resources, change proc state to * zombie, and unlink proc from allproc and parent's lists. Save exit status * and rusage for wait(). Check for child processes and orphan them. */ void exit1(struct thread *td, int rv) { struct proc *p, *nq, *q, *t; struct thread *tdt; struct vnode *ttyvp = NULL; mtx_assert(&Giant, MA_NOTOWNED); p = td->td_proc; /* * XXX in case we're rebooting we just let init die in order to * work around an unsolved stack overflow seen very late during * shutdown on sparc64 when the gmirror worker process exists. */ if (p == initproc && rebooting == 0) { printf("init died (signal %d, exit %d)\n", WTERMSIG(rv), WEXITSTATUS(rv)); panic("Going nowhere without my init!"); } /* * MUST abort all other threads before proceeding past here. */ PROC_LOCK(p); /* * First check if some other thread or external request got * here before us. If so, act appropriately: exit or suspend. * We must ensure that stop requests are handled before we set * P_WEXIT. */ thread_suspend_check(0); while (p->p_flag & P_HADTHREADS) { /* * Kill off the other threads. This requires * some co-operation from other parts of the kernel * so it may not be instantaneous. With this state set * any thread entering the kernel from userspace will * thread_exit() in trap(). Any thread attempting to * sleep will return immediately with EINTR or EWOULDBLOCK * which will hopefully force them to back out to userland * freeing resources as they go. Any thread attempting * to return to userland will thread_exit() from userret(). * thread_exit() will unsuspend us when the last of the * other threads exits. * If there is already a thread singler after resumption, * calling thread_single will fail; in that case, we just * re-check all suspension request, the thread should * either be suspended there or exit. */ - if (!thread_single(SINGLE_EXIT)) + if (!thread_single(p, SINGLE_EXIT)) /* * All other activity in this process is now * stopped. Threading support has been turned * off. */ break; /* * Recheck for new stop or suspend requests which * might appear while process lock was dropped in * thread_single(). */ thread_suspend_check(0); } KASSERT(p->p_numthreads == 1, ("exit1: proc %p exiting with %d threads", p, p->p_numthreads)); racct_sub(p, RACCT_NTHR, 1); /* * Wakeup anyone in procfs' PIOCWAIT. They should have a hold * on our vmspace, so we should block below until they have * released their reference to us. Note that if they have * requested S_EXIT stops we will block here until they ack * via PIOCCONT. */ _STOPEVENT(p, S_EXIT, rv); /* * Ignore any pending request to stop due to a stop signal. * Once P_WEXIT is set, future requests will be ignored as * well. */ p->p_flag &= ~P_STOPPED_SIG; KASSERT(!P_SHOULDSTOP(p), ("exiting process is stopped")); /* * Note that we are exiting and do another wakeup of anyone in * PIOCWAIT in case they aren't listening for S_EXIT stops or * decided to wait again after we told them we are exiting. */ p->p_flag |= P_WEXIT; wakeup(&p->p_stype); /* * Wait for any processes that have a hold on our vmspace to * release their reference. */ while (p->p_lock > 0) msleep(&p->p_lock, &p->p_mtx, PWAIT, "exithold", 0); p->p_xstat = rv; /* Let event handler change exit status */ PROC_UNLOCK(p); /* Drain the limit callout while we don't have the proc locked */ callout_drain(&p->p_limco); #ifdef AUDIT /* * The Sun BSM exit token contains two components: an exit status as * passed to exit(), and a return value to indicate what sort of exit * it was. The exit status is WEXITSTATUS(rv), but it's not clear * what the return value is. */ AUDIT_ARG_EXIT(WEXITSTATUS(rv), 0); AUDIT_SYSCALL_EXIT(0, td); #endif /* Are we a task leader with peers? */ if (p->p_peers != NULL && p == p->p_leader) { mtx_lock(&ppeers_lock); q = p->p_peers; while (q != NULL) { PROC_LOCK(q); kern_psignal(q, SIGKILL); PROC_UNLOCK(q); q = q->p_peers; } while (p->p_peers != NULL) msleep(p, &ppeers_lock, PWAIT, "exit1", 0); mtx_unlock(&ppeers_lock); } /* * Check if any loadable modules need anything done at process exit. * E.g. SYSV IPC stuff * XXX what if one of these generates an error? */ EVENTHANDLER_INVOKE(process_exit, p); /* * If parent is waiting for us to exit or exec, * P_PPWAIT is set; we will wakeup the parent below. */ PROC_LOCK(p); rv = p->p_xstat; /* Event handler could change exit status */ stopprofclock(p); p->p_flag &= ~(P_TRACED | P_PPWAIT | P_PPTRACE); /* * Stop the real interval timer. If the handler is currently * executing, prevent it from rearming itself and let it finish. */ if (timevalisset(&p->p_realtimer.it_value) && callout_stop(&p->p_itcallout) == 0) { timevalclear(&p->p_realtimer.it_interval); msleep(&p->p_itcallout, &p->p_mtx, PWAIT, "ritwait", 0); KASSERT(!timevalisset(&p->p_realtimer.it_value), ("realtime timer is still armed")); } PROC_UNLOCK(p); /* * Reset any sigio structures pointing to us as a result of * F_SETOWN with our pid. */ funsetownlst(&p->p_sigiolst); /* * If this process has an nlminfo data area (for lockd), release it */ if (nlminfo_release_p != NULL && p->p_nlminfo != NULL) (*nlminfo_release_p)(p); /* * Close open files and release open-file table. * This may block! */ fdescfree(td); /* * If this thread tickled GEOM, we need to wait for the giggling to * stop before we return to userland */ if (td->td_pflags & TDP_GEOM) g_waitidle(); /* * Remove ourself from our leader's peer list and wake our leader. */ if (p->p_leader->p_peers != NULL) { mtx_lock(&ppeers_lock); if (p->p_leader->p_peers != NULL) { q = p->p_leader; while (q->p_peers != p) q = q->p_peers; q->p_peers = p->p_peers; wakeup(p->p_leader); } mtx_unlock(&ppeers_lock); } vmspace_exit(td); sx_xlock(&proctree_lock); if (SESS_LEADER(p)) { struct session *sp = p->p_session; struct tty *tp; /* * s_ttyp is not zero'd; we use this to indicate that * the session once had a controlling terminal. (for * logging and informational purposes) */ SESS_LOCK(sp); ttyvp = sp->s_ttyvp; tp = sp->s_ttyp; sp->s_ttyvp = NULL; sp->s_ttydp = NULL; sp->s_leader = NULL; SESS_UNLOCK(sp); /* * Signal foreground pgrp and revoke access to * controlling terminal if it has not been revoked * already. * * Because the TTY may have been revoked in the mean * time and could already have a new session associated * with it, make sure we don't send a SIGHUP to a * foreground process group that does not belong to this * session. */ if (tp != NULL) { tty_lock(tp); if (tp->t_session == sp) tty_signal_pgrp(tp, SIGHUP); tty_unlock(tp); } if (ttyvp != NULL) { sx_xunlock(&proctree_lock); if (vn_lock(ttyvp, LK_EXCLUSIVE) == 0) { VOP_REVOKE(ttyvp, REVOKEALL); VOP_UNLOCK(ttyvp, 0); } sx_xlock(&proctree_lock); } } fixjobc(p, p->p_pgrp, 0); sx_xunlock(&proctree_lock); (void)acct_process(td); /* Release the TTY now we've unlocked everything. */ if (ttyvp != NULL) vrele(ttyvp); #ifdef KTRACE ktrprocexit(td); #endif /* * Release reference to text vnode */ if (p->p_textvp != NULL) { vrele(p->p_textvp); p->p_textvp = NULL; } /* * Release our limits structure. */ lim_free(p->p_limit); p->p_limit = NULL; tidhash_remove(td); /* * Remove proc from allproc queue and pidhash chain. * Place onto zombproc. Unlink from parent's child list. */ sx_xlock(&allproc_lock); LIST_REMOVE(p, p_list); LIST_INSERT_HEAD(&zombproc, p, p_list); LIST_REMOVE(p, p_hash); sx_xunlock(&allproc_lock); /* * Call machine-dependent code to release any * machine-dependent resources other than the address space. * The address space is released by "vmspace_exitfree(p)" in * vm_waitproc(). */ cpu_exit(td); WITNESS_WARN(WARN_PANIC, NULL, "process (pid %d) exiting", p->p_pid); /* * Reparent all children processes: * - traced ones to the original parent (or init if we are that parent) * - the rest to init */ sx_xlock(&proctree_lock); q = LIST_FIRST(&p->p_children); if (q != NULL) /* only need this if any child is S_ZOMB */ wakeup(initproc); for (; q != NULL; q = nq) { nq = LIST_NEXT(q, p_sibling); PROC_LOCK(q); q->p_sigparent = SIGCHLD; if (!(q->p_flag & P_TRACED)) { proc_reparent(q, initproc); } else { /* * Traced processes are killed since their existence * means someone is screwing up. */ t = proc_realparent(q); if (t == p) { proc_reparent(q, initproc); } else { PROC_LOCK(t); proc_reparent(q, t); PROC_UNLOCK(t); } /* * Since q was found on our children list, the * proc_reparent() call moved q to the orphan * list due to present P_TRACED flag. Clear * orphan link for q now while q is locked. */ clear_orphan(q); q->p_flag &= ~(P_TRACED | P_STOPPED_TRACE); FOREACH_THREAD_IN_PROC(q, tdt) tdt->td_dbgflags &= ~TDB_SUSPEND; kern_psignal(q, SIGKILL); } PROC_UNLOCK(q); } /* * Also get rid of our orphans. */ while ((q = LIST_FIRST(&p->p_orphans)) != NULL) { PROC_LOCK(q); clear_orphan(q); PROC_UNLOCK(q); } /* Save exit status. */ PROC_LOCK(p); p->p_xthread = td; /* Tell the prison that we are gone. */ prison_proc_free(p->p_ucred->cr_prison); #ifdef KDTRACE_HOOKS /* * Tell the DTrace fasttrap provider about the exit if it * has declared an interest. */ if (dtrace_fasttrap_exit) dtrace_fasttrap_exit(p); #endif /* * Notify interested parties of our demise. */ KNOTE_LOCKED(&p->p_klist, NOTE_EXIT); #ifdef KDTRACE_HOOKS int reason = CLD_EXITED; if (WCOREDUMP(rv)) reason = CLD_DUMPED; else if (WIFSIGNALED(rv)) reason = CLD_KILLED; SDT_PROBE(proc, kernel, , exit, reason, 0, 0, 0, 0); #endif /* * Just delete all entries in the p_klist. At this point we won't * report any more events, and there are nasty race conditions that * can beat us if we don't. */ knlist_clear(&p->p_klist, 1); /* * If this is a process with a descriptor, we may not need to deliver * a signal to the parent. proctree_lock is held over * procdesc_exit() to serialize concurrent calls to close() and * exit(). */ if (p->p_procdesc == NULL || procdesc_exit(p)) { /* * Notify parent that we're gone. If parent has the * PS_NOCLDWAIT flag set, or if the handler is set to SIG_IGN, * notify process 1 instead (and hope it will handle this * situation). */ PROC_LOCK(p->p_pptr); mtx_lock(&p->p_pptr->p_sigacts->ps_mtx); if (p->p_pptr->p_sigacts->ps_flag & (PS_NOCLDWAIT | PS_CLDSIGIGN)) { struct proc *pp; mtx_unlock(&p->p_pptr->p_sigacts->ps_mtx); pp = p->p_pptr; PROC_UNLOCK(pp); proc_reparent(p, initproc); p->p_sigparent = SIGCHLD; PROC_LOCK(p->p_pptr); /* * Notify parent, so in case he was wait(2)ing or * executing waitpid(2) with our pid, he will * continue. */ wakeup(pp); } else mtx_unlock(&p->p_pptr->p_sigacts->ps_mtx); if (p->p_pptr == initproc) kern_psignal(p->p_pptr, SIGCHLD); else if (p->p_sigparent != 0) { if (p->p_sigparent == SIGCHLD) childproc_exited(p); else /* LINUX thread */ kern_psignal(p->p_pptr, p->p_sigparent); } } else PROC_LOCK(p->p_pptr); sx_xunlock(&proctree_lock); /* * The state PRS_ZOMBIE prevents other proesses from sending * signal to the process, to avoid memory leak, we free memory * for signal queue at the time when the state is set. */ sigqueue_flush(&p->p_sigqueue); sigqueue_flush(&td->td_sigqueue); /* * We have to wait until after acquiring all locks before * changing p_state. We need to avoid all possible context * switches (including ones from blocking on a mutex) while * marked as a zombie. We also have to set the zombie state * before we release the parent process' proc lock to avoid * a lost wakeup. So, we first call wakeup, then we grab the * sched lock, update the state, and release the parent process' * proc lock. */ wakeup(p->p_pptr); cv_broadcast(&p->p_pwait); sched_exit(p->p_pptr, td); PROC_SLOCK(p); p->p_state = PRS_ZOMBIE; PROC_UNLOCK(p->p_pptr); /* * Hopefully no one will try to deliver a signal to the process this * late in the game. */ knlist_destroy(&p->p_klist); /* * Save our children's rusage information in our exit rusage. */ PROC_STATLOCK(p); ruadd(&p->p_ru, &p->p_rux, &p->p_stats->p_cru, &p->p_crux); PROC_STATUNLOCK(p); /* * Make sure the scheduler takes this thread out of its tables etc. * This will also release this thread's reference to the ucred. * Other thread parts to release include pcb bits and such. */ thread_exit(); } #ifndef _SYS_SYSPROTO_H_ struct abort2_args { char *why; int nargs; void **args; }; #endif int sys_abort2(struct thread *td, struct abort2_args *uap) { struct proc *p = td->td_proc; struct sbuf *sb; void *uargs[16]; int error, i, sig; /* * Do it right now so we can log either proper call of abort2(), or * note, that invalid argument was passed. 512 is big enough to * handle 16 arguments' descriptions with additional comments. */ sb = sbuf_new(NULL, NULL, 512, SBUF_FIXEDLEN); sbuf_clear(sb); sbuf_printf(sb, "%s(pid %d uid %d) aborted: ", p->p_comm, p->p_pid, td->td_ucred->cr_uid); /* * Since we can't return from abort2(), send SIGKILL in cases, where * abort2() was called improperly */ sig = SIGKILL; /* Prevent from DoSes from user-space. */ if (uap->nargs < 0 || uap->nargs > 16) goto out; if (uap->nargs > 0) { if (uap->args == NULL) goto out; error = copyin(uap->args, uargs, uap->nargs * sizeof(void *)); if (error != 0) goto out; } /* * Limit size of 'reason' string to 128. Will fit even when * maximal number of arguments was chosen to be logged. */ if (uap->why != NULL) { error = sbuf_copyin(sb, uap->why, 128); if (error < 0) goto out; } else { sbuf_printf(sb, "(null)"); } if (uap->nargs > 0) { sbuf_printf(sb, "("); for (i = 0;i < uap->nargs; i++) sbuf_printf(sb, "%s%p", i == 0 ? "" : ", ", uargs[i]); sbuf_printf(sb, ")"); } /* * Final stage: arguments were proper, string has been * successfully copied from userspace, and copying pointers * from user-space succeed. */ sig = SIGABRT; out: if (sig == SIGKILL) { sbuf_trim(sb); sbuf_printf(sb, " (Reason text inaccessible)"); } sbuf_cat(sb, "\n"); sbuf_finish(sb); log(LOG_INFO, "%s", sbuf_data(sb)); sbuf_delete(sb); exit1(td, W_EXITCODE(0, sig)); return (0); } #ifdef COMPAT_43 /* * The dirty work is handled by kern_wait(). */ int owait(struct thread *td, struct owait_args *uap __unused) { int error, status; error = kern_wait(td, WAIT_ANY, &status, 0, NULL); if (error == 0) td->td_retval[1] = status; return (error); } #endif /* COMPAT_43 */ /* * The dirty work is handled by kern_wait(). */ int sys_wait4(struct thread *td, struct wait4_args *uap) { struct rusage ru, *rup; int error, status; if (uap->rusage != NULL) rup = &ru; else rup = NULL; error = kern_wait(td, uap->pid, &status, uap->options, rup); if (uap->status != NULL && error == 0) error = copyout(&status, uap->status, sizeof(status)); if (uap->rusage != NULL && error == 0) error = copyout(&ru, uap->rusage, sizeof(struct rusage)); return (error); } int sys_wait6(struct thread *td, struct wait6_args *uap) { struct __wrusage wru, *wrup; siginfo_t si, *sip; idtype_t idtype; id_t id; int error, status; idtype = uap->idtype; id = uap->id; if (uap->wrusage != NULL) wrup = &wru; else wrup = NULL; if (uap->info != NULL) { sip = &si; bzero(sip, sizeof(*sip)); } else sip = NULL; /* * We expect all callers of wait6() to know about WEXITED and * WTRAPPED. */ error = kern_wait6(td, idtype, id, &status, uap->options, wrup, sip); if (uap->status != NULL && error == 0) error = copyout(&status, uap->status, sizeof(status)); if (uap->wrusage != NULL && error == 0) error = copyout(&wru, uap->wrusage, sizeof(wru)); if (uap->info != NULL && error == 0) error = copyout(&si, uap->info, sizeof(si)); return (error); } /* * Reap the remains of a zombie process and optionally return status and * rusage. Asserts and will release both the proctree_lock and the process * lock as part of its work. */ void proc_reap(struct thread *td, struct proc *p, int *status, int options) { struct proc *q, *t; sx_assert(&proctree_lock, SA_XLOCKED); PROC_LOCK_ASSERT(p, MA_OWNED); PROC_SLOCK_ASSERT(p, MA_OWNED); KASSERT(p->p_state == PRS_ZOMBIE, ("proc_reap: !PRS_ZOMBIE")); q = td->td_proc; PROC_SUNLOCK(p); td->td_retval[0] = p->p_pid; if (status) *status = p->p_xstat; /* convert to int */ if (options & WNOWAIT) { /* * Only poll, returning the status. Caller does not wish to * release the proc struct just yet. */ PROC_UNLOCK(p); sx_xunlock(&proctree_lock); return; } PROC_LOCK(q); sigqueue_take(p->p_ksi); PROC_UNLOCK(q); PROC_UNLOCK(p); /* * If we got the child via a ptrace 'attach', we need to give it back * to the old parent. */ if (p->p_oppid != 0) { t = proc_realparent(p); PROC_LOCK(t); PROC_LOCK(p); proc_reparent(p, t); p->p_oppid = 0; PROC_UNLOCK(p); pksignal(t, SIGCHLD, p->p_ksi); wakeup(t); cv_broadcast(&p->p_pwait); PROC_UNLOCK(t); sx_xunlock(&proctree_lock); return; } /* * Remove other references to this process to ensure we have an * exclusive reference. */ sx_xlock(&allproc_lock); LIST_REMOVE(p, p_list); /* off zombproc */ sx_xunlock(&allproc_lock); LIST_REMOVE(p, p_sibling); PROC_LOCK(p); clear_orphan(p); PROC_UNLOCK(p); leavepgrp(p); if (p->p_procdesc != NULL) procdesc_reap(p); sx_xunlock(&proctree_lock); /* * As a side effect of this lock, we know that all other writes to * this proc are visible now, so no more locking is needed for p. */ PROC_LOCK(p); p->p_xstat = 0; /* XXX: why? */ PROC_UNLOCK(p); PROC_LOCK(q); ruadd(&q->p_stats->p_cru, &q->p_crux, &p->p_ru, &p->p_rux); PROC_UNLOCK(q); /* * Decrement the count of procs running with this uid. */ (void)chgproccnt(p->p_ucred->cr_ruidinfo, -1, 0); /* * Destroy resource accounting information associated with the process. */ #ifdef RACCT PROC_LOCK(p); racct_sub(p, RACCT_NPROC, 1); PROC_UNLOCK(p); #endif racct_proc_exit(p); /* * Free credentials, arguments, and sigacts. */ crfree(p->p_ucred); p->p_ucred = NULL; pargs_drop(p->p_args); p->p_args = NULL; sigacts_free(p->p_sigacts); p->p_sigacts = NULL; /* * Do any thread-system specific cleanups. */ thread_wait(p); /* * Give vm and machine-dependent layer a chance to free anything that * cpu_exit couldn't release while still running in process context. */ vm_waitproc(p); #ifdef MAC mac_proc_destroy(p); #endif KASSERT(FIRST_THREAD_IN_PROC(p), ("proc_reap: no residual thread!")); uma_zfree(proc_zone, p); sx_xlock(&allproc_lock); nprocs--; sx_xunlock(&allproc_lock); } static int proc_to_reap(struct thread *td, struct proc *p, idtype_t idtype, id_t id, int *status, int options, struct __wrusage *wrusage, siginfo_t *siginfo) { struct proc *q; struct rusage *rup; sx_assert(&proctree_lock, SA_XLOCKED); q = td->td_proc; PROC_LOCK(p); switch (idtype) { case P_ALL: break; case P_PID: if (p->p_pid != (pid_t)id) { PROC_UNLOCK(p); return (0); } break; case P_PGID: if (p->p_pgid != (pid_t)id) { PROC_UNLOCK(p); return (0); } break; case P_SID: if (p->p_session->s_sid != (pid_t)id) { PROC_UNLOCK(p); return (0); } break; case P_UID: if (p->p_ucred->cr_uid != (uid_t)id) { PROC_UNLOCK(p); return (0); } break; case P_GID: if (p->p_ucred->cr_gid != (gid_t)id) { PROC_UNLOCK(p); return (0); } break; case P_JAILID: if (p->p_ucred->cr_prison->pr_id != (int)id) { PROC_UNLOCK(p); return (0); } break; /* * It seems that the thread structures get zeroed out * at process exit. This makes it impossible to * support P_SETID, P_CID or P_CPUID. */ default: PROC_UNLOCK(p); return (0); } if (p_canwait(td, p)) { PROC_UNLOCK(p); return (0); } if (((options & WEXITED) == 0) && (p->p_state == PRS_ZOMBIE)) { PROC_UNLOCK(p); return (0); } /* * This special case handles a kthread spawned by linux_clone * (see linux_misc.c). The linux_wait4 and linux_waitpid * functions need to be able to distinguish between waiting * on a process and waiting on a thread. It is a thread if * p_sigparent is not SIGCHLD, and the WLINUXCLONE option * signifies we want to wait for threads and not processes. */ if ((p->p_sigparent != SIGCHLD) ^ ((options & WLINUXCLONE) != 0)) { PROC_UNLOCK(p); return (0); } if (siginfo != NULL) { bzero(siginfo, sizeof(*siginfo)); siginfo->si_errno = 0; /* * SUSv4 requires that the si_signo value is always * SIGCHLD. Obey it despite the rfork(2) interface * allows to request other signal for child exit * notification. */ siginfo->si_signo = SIGCHLD; /* * This is still a rough estimate. We will fix the * cases TRAPPED, STOPPED, and CONTINUED later. */ if (WCOREDUMP(p->p_xstat)) { siginfo->si_code = CLD_DUMPED; siginfo->si_status = WTERMSIG(p->p_xstat); } else if (WIFSIGNALED(p->p_xstat)) { siginfo->si_code = CLD_KILLED; siginfo->si_status = WTERMSIG(p->p_xstat); } else { siginfo->si_code = CLD_EXITED; siginfo->si_status = WEXITSTATUS(p->p_xstat); } siginfo->si_pid = p->p_pid; siginfo->si_uid = p->p_ucred->cr_uid; /* * The si_addr field would be useful additional * detail, but apparently the PC value may be lost * when we reach this point. bzero() above sets * siginfo->si_addr to NULL. */ } /* * There should be no reason to limit resources usage info to * exited processes only. A snapshot about any resources used * by a stopped process may be exactly what is needed. */ if (wrusage != NULL) { rup = &wrusage->wru_self; *rup = p->p_ru; PROC_STATLOCK(p); calcru(p, &rup->ru_utime, &rup->ru_stime); PROC_STATUNLOCK(p); rup = &wrusage->wru_children; *rup = p->p_stats->p_cru; calccru(p, &rup->ru_utime, &rup->ru_stime); } if (p->p_state == PRS_ZOMBIE) { PROC_SLOCK(p); proc_reap(td, p, status, options); return (-1); } PROC_UNLOCK(p); return (1); } int kern_wait(struct thread *td, pid_t pid, int *status, int options, struct rusage *rusage) { struct __wrusage wru, *wrup; idtype_t idtype; id_t id; int ret; /* * Translate the special pid values into the (idtype, pid) * pair for kern_wait6. The WAIT_MYPGRP case is handled by * kern_wait6() on its own. */ if (pid == WAIT_ANY) { idtype = P_ALL; id = 0; } else if (pid < 0) { idtype = P_PGID; id = (id_t)-pid; } else { idtype = P_PID; id = (id_t)pid; } if (rusage != NULL) wrup = &wru; else wrup = NULL; /* * For backward compatibility we implicitly add flags WEXITED * and WTRAPPED here. */ options |= WEXITED | WTRAPPED; ret = kern_wait6(td, idtype, id, status, options, wrup, NULL); if (rusage != NULL) *rusage = wru.wru_self; return (ret); } int kern_wait6(struct thread *td, idtype_t idtype, id_t id, int *status, int options, struct __wrusage *wrusage, siginfo_t *siginfo) { struct proc *p, *q; int error, nfound, ret; AUDIT_ARG_VALUE((int)idtype); /* XXX - This is likely wrong! */ AUDIT_ARG_PID((pid_t)id); /* XXX - This may be wrong! */ AUDIT_ARG_VALUE(options); q = td->td_proc; if ((pid_t)id == WAIT_MYPGRP && (idtype == P_PID || idtype == P_PGID)) { PROC_LOCK(q); id = (id_t)q->p_pgid; PROC_UNLOCK(q); idtype = P_PGID; } /* If we don't know the option, just return. */ if ((options & ~(WUNTRACED | WNOHANG | WCONTINUED | WNOWAIT | WEXITED | WTRAPPED | WLINUXCLONE)) != 0) return (EINVAL); if ((options & (WEXITED | WUNTRACED | WCONTINUED | WTRAPPED)) == 0) { /* * We will be unable to find any matching processes, * because there are no known events to look for. * Prefer to return error instead of blocking * indefinitely. */ return (EINVAL); } loop: if (q->p_flag & P_STATCHILD) { PROC_LOCK(q); q->p_flag &= ~P_STATCHILD; PROC_UNLOCK(q); } nfound = 0; sx_xlock(&proctree_lock); LIST_FOREACH(p, &q->p_children, p_sibling) { ret = proc_to_reap(td, p, idtype, id, status, options, wrusage, siginfo); if (ret == 0) continue; else if (ret == 1) nfound++; else return (0); PROC_LOCK(p); PROC_SLOCK(p); if ((options & WTRAPPED) != 0 && (p->p_flag & P_TRACED) != 0 && (p->p_flag & (P_STOPPED_TRACE | P_STOPPED_SIG)) != 0 && (p->p_suspcount == p->p_numthreads) && ((p->p_flag & P_WAITED) == 0)) { PROC_SUNLOCK(p); if ((options & WNOWAIT) == 0) p->p_flag |= P_WAITED; sx_xunlock(&proctree_lock); td->td_retval[0] = p->p_pid; if (status != NULL) *status = W_STOPCODE(p->p_xstat); if (siginfo != NULL) { siginfo->si_status = p->p_xstat; siginfo->si_code = CLD_TRAPPED; } if ((options & WNOWAIT) == 0) { PROC_LOCK(q); sigqueue_take(p->p_ksi); PROC_UNLOCK(q); } PROC_UNLOCK(p); return (0); } if ((options & WUNTRACED) != 0 && (p->p_flag & P_STOPPED_SIG) != 0 && (p->p_suspcount == p->p_numthreads) && ((p->p_flag & P_WAITED) == 0)) { PROC_SUNLOCK(p); if ((options & WNOWAIT) == 0) p->p_flag |= P_WAITED; sx_xunlock(&proctree_lock); td->td_retval[0] = p->p_pid; if (status != NULL) *status = W_STOPCODE(p->p_xstat); if (siginfo != NULL) { siginfo->si_status = p->p_xstat; siginfo->si_code = CLD_STOPPED; } if ((options & WNOWAIT) == 0) { PROC_LOCK(q); sigqueue_take(p->p_ksi); PROC_UNLOCK(q); } PROC_UNLOCK(p); return (0); } PROC_SUNLOCK(p); if ((options & WCONTINUED) != 0 && (p->p_flag & P_CONTINUED) != 0) { sx_xunlock(&proctree_lock); td->td_retval[0] = p->p_pid; if ((options & WNOWAIT) == 0) { p->p_flag &= ~P_CONTINUED; PROC_LOCK(q); sigqueue_take(p->p_ksi); PROC_UNLOCK(q); } PROC_UNLOCK(p); if (status != NULL) *status = SIGCONT; if (siginfo != NULL) { siginfo->si_status = SIGCONT; siginfo->si_code = CLD_CONTINUED; } return (0); } PROC_UNLOCK(p); } /* * Look in the orphans list too, to allow the parent to * collect it's child exit status even if child is being * debugged. * * Debugger detaches from the parent upon successful * switch-over from parent to child. At this point due to * re-parenting the parent loses the child to debugger and a * wait4(2) call would report that it has no children to wait * for. By maintaining a list of orphans we allow the parent * to successfully wait until the child becomes a zombie. */ LIST_FOREACH(p, &q->p_orphans, p_orphan) { ret = proc_to_reap(td, p, idtype, id, status, options, wrusage, siginfo); if (ret == 0) continue; else if (ret == 1) nfound++; else return (0); } if (nfound == 0) { sx_xunlock(&proctree_lock); return (ECHILD); } if (options & WNOHANG) { sx_xunlock(&proctree_lock); td->td_retval[0] = 0; return (0); } PROC_LOCK(q); sx_xunlock(&proctree_lock); if (q->p_flag & P_STATCHILD) { q->p_flag &= ~P_STATCHILD; error = 0; } else error = msleep(q, &q->p_mtx, PWAIT | PCATCH, "wait", 0); PROC_UNLOCK(q); if (error) return (error); goto loop; } /* * Make process 'parent' the new parent of process 'child'. * Must be called with an exclusive hold of proctree lock. */ void proc_reparent(struct proc *child, struct proc *parent) { sx_assert(&proctree_lock, SX_XLOCKED); PROC_LOCK_ASSERT(child, MA_OWNED); if (child->p_pptr == parent) return; PROC_LOCK(child->p_pptr); sigqueue_take(child->p_ksi); PROC_UNLOCK(child->p_pptr); LIST_REMOVE(child, p_sibling); LIST_INSERT_HEAD(&parent->p_children, child, p_sibling); clear_orphan(child); if (child->p_flag & P_TRACED) { if (LIST_EMPTY(&child->p_pptr->p_orphans)) { child->p_treeflag |= P_TREE_FIRST_ORPHAN; LIST_INSERT_HEAD(&child->p_pptr->p_orphans, child, p_orphan); } else { LIST_INSERT_AFTER(LIST_FIRST(&child->p_pptr->p_orphans), child, p_orphan); } child->p_treeflag |= P_TREE_ORPHANED; } child->p_pptr = parent; } Index: projects/clang350-import/sys/kern/kern_fork.c =================================================================== --- projects/clang350-import/sys/kern/kern_fork.c (revision 275748) +++ projects/clang350-import/sys/kern/kern_fork.c (revision 275749) @@ -1,1040 +1,1041 @@ /*- * Copyright (c) 1982, 1986, 1989, 1991, 1993 * The Regents of the University of California. All rights reserved. * (c) UNIX System Laboratories, Inc. * All or some portions of this file are derived from material licensed * to the University of California by American Telephone and Telegraph * Co. or Unix System Laboratories, Inc. and are reproduced herein with * the permission of UNIX System Laboratories, Inc. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 4. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * @(#)kern_fork.c 8.6 (Berkeley) 4/8/94 */ #include __FBSDID("$FreeBSD$"); #include "opt_ktrace.h" #include "opt_kstack_pages.h" #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #ifdef KDTRACE_HOOKS #include dtrace_fork_func_t dtrace_fasttrap_fork; #endif SDT_PROVIDER_DECLARE(proc); SDT_PROBE_DEFINE3(proc, kernel, , create, "struct proc *", "struct proc *", "int"); #ifndef _SYS_SYSPROTO_H_ struct fork_args { int dummy; }; #endif /* ARGSUSED */ int sys_fork(struct thread *td, struct fork_args *uap) { int error; struct proc *p2; error = fork1(td, RFFDG | RFPROC, 0, &p2, NULL, 0); if (error == 0) { td->td_retval[0] = p2->p_pid; td->td_retval[1] = 0; } return (error); } /* ARGUSED */ int sys_pdfork(td, uap) struct thread *td; struct pdfork_args *uap; { int error, fd; struct proc *p2; /* * It is necessary to return fd by reference because 0 is a valid file * descriptor number, and the child needs to be able to distinguish * itself from the parent using the return value. */ error = fork1(td, RFFDG | RFPROC | RFPROCDESC, 0, &p2, &fd, uap->flags); if (error == 0) { td->td_retval[0] = p2->p_pid; td->td_retval[1] = 0; error = copyout(&fd, uap->fdp, sizeof(fd)); } return (error); } /* ARGSUSED */ int sys_vfork(struct thread *td, struct vfork_args *uap) { int error, flags; struct proc *p2; flags = RFFDG | RFPROC | RFPPWAIT | RFMEM; error = fork1(td, flags, 0, &p2, NULL, 0); if (error == 0) { td->td_retval[0] = p2->p_pid; td->td_retval[1] = 0; } return (error); } int sys_rfork(struct thread *td, struct rfork_args *uap) { struct proc *p2; int error; /* Don't allow kernel-only flags. */ if ((uap->flags & RFKERNELONLY) != 0) return (EINVAL); AUDIT_ARG_FFLAGS(uap->flags); error = fork1(td, uap->flags, 0, &p2, NULL, 0); if (error == 0) { td->td_retval[0] = p2 ? p2->p_pid : 0; td->td_retval[1] = 0; } return (error); } int nprocs = 1; /* process 0 */ int lastpid = 0; SYSCTL_INT(_kern, OID_AUTO, lastpid, CTLFLAG_RD, &lastpid, 0, "Last used PID"); /* * Random component to lastpid generation. We mix in a random factor to make * it a little harder to predict. We sanity check the modulus value to avoid * doing it in critical paths. Don't let it be too small or we pointlessly * waste randomness entropy, and don't let it be impossibly large. Using a * modulus that is too big causes a LOT more process table scans and slows * down fork processing as the pidchecked caching is defeated. */ static int randompid = 0; static int sysctl_kern_randompid(SYSCTL_HANDLER_ARGS) { int error, pid; error = sysctl_wire_old_buffer(req, sizeof(int)); if (error != 0) return(error); sx_xlock(&allproc_lock); pid = randompid; error = sysctl_handle_int(oidp, &pid, 0, req); if (error == 0 && req->newptr != NULL) { if (pid < 0 || pid > pid_max - 100) /* out of range */ pid = pid_max - 100; else if (pid < 2) /* NOP */ pid = 0; else if (pid < 100) /* Make it reasonable */ pid = 100; randompid = pid; } sx_xunlock(&allproc_lock); return (error); } SYSCTL_PROC(_kern, OID_AUTO, randompid, CTLTYPE_INT|CTLFLAG_RW, 0, 0, sysctl_kern_randompid, "I", "Random PID modulus"); static int fork_findpid(int flags) { struct proc *p; int trypid; static int pidchecked = 0; /* * Requires allproc_lock in order to iterate over the list * of processes, and proctree_lock to access p_pgrp. */ sx_assert(&allproc_lock, SX_LOCKED); sx_assert(&proctree_lock, SX_LOCKED); /* * Find an unused process ID. We remember a range of unused IDs * ready to use (from lastpid+1 through pidchecked-1). * * If RFHIGHPID is set (used during system boot), do not allocate * low-numbered pids. */ trypid = lastpid + 1; if (flags & RFHIGHPID) { if (trypid < 10) trypid = 10; } else { if (randompid) trypid += arc4random() % randompid; } retry: /* * If the process ID prototype has wrapped around, * restart somewhat above 0, as the low-numbered procs * tend to include daemons that don't exit. */ if (trypid >= pid_max) { trypid = trypid % pid_max; if (trypid < 100) trypid += 100; pidchecked = 0; } if (trypid >= pidchecked) { int doingzomb = 0; pidchecked = PID_MAX; /* * Scan the active and zombie procs to check whether this pid * is in use. Remember the lowest pid that's greater * than trypid, so we can avoid checking for a while. */ p = LIST_FIRST(&allproc); again: for (; p != NULL; p = LIST_NEXT(p, p_list)) { while (p->p_pid == trypid || (p->p_pgrp != NULL && (p->p_pgrp->pg_id == trypid || (p->p_session != NULL && p->p_session->s_sid == trypid)))) { trypid++; if (trypid >= pidchecked) goto retry; } if (p->p_pid > trypid && pidchecked > p->p_pid) pidchecked = p->p_pid; if (p->p_pgrp != NULL) { if (p->p_pgrp->pg_id > trypid && pidchecked > p->p_pgrp->pg_id) pidchecked = p->p_pgrp->pg_id; if (p->p_session != NULL && p->p_session->s_sid > trypid && pidchecked > p->p_session->s_sid) pidchecked = p->p_session->s_sid; } } if (!doingzomb) { doingzomb = 1; p = LIST_FIRST(&zombproc); goto again; } } /* * RFHIGHPID does not mess with the lastpid counter during boot. */ if (flags & RFHIGHPID) pidchecked = 0; else lastpid = trypid; return (trypid); } static int fork_norfproc(struct thread *td, int flags) { int error; struct proc *p1; KASSERT((flags & RFPROC) == 0, ("fork_norfproc called with RFPROC set")); p1 = td->td_proc; if (((p1->p_flag & (P_HADTHREADS|P_SYSTEM)) == P_HADTHREADS) && (flags & (RFCFDG | RFFDG))) { PROC_LOCK(p1); - if (thread_single(SINGLE_BOUNDARY)) { + if (thread_single(p1, SINGLE_BOUNDARY)) { PROC_UNLOCK(p1); return (ERESTART); } PROC_UNLOCK(p1); } error = vm_forkproc(td, NULL, NULL, NULL, flags); if (error) goto fail; /* * Close all file descriptors. */ if (flags & RFCFDG) { struct filedesc *fdtmp; fdtmp = fdinit(td->td_proc->p_fd, false); fdescfree(td); p1->p_fd = fdtmp; } /* * Unshare file descriptors (from parent). */ if (flags & RFFDG) fdunshare(td); fail: if (((p1->p_flag & (P_HADTHREADS|P_SYSTEM)) == P_HADTHREADS) && (flags & (RFCFDG | RFFDG))) { PROC_LOCK(p1); - thread_single_end(); + thread_single_end(p1, SINGLE_BOUNDARY); PROC_UNLOCK(p1); } return (error); } static void do_fork(struct thread *td, int flags, struct proc *p2, struct thread *td2, struct vmspace *vm2, int pdflags) { struct proc *p1, *pptr; int p2_held, trypid; struct filedesc *fd; struct filedesc_to_leader *fdtol; struct sigacts *newsigacts; sx_assert(&proctree_lock, SX_SLOCKED); sx_assert(&allproc_lock, SX_XLOCKED); p2_held = 0; p1 = td->td_proc; /* * Increment the nprocs resource before blocking can occur. There * are hard-limits as to the number of processes that can run. */ nprocs++; trypid = fork_findpid(flags); sx_sunlock(&proctree_lock); p2->p_state = PRS_NEW; /* protect against others */ p2->p_pid = trypid; AUDIT_ARG_PID(p2->p_pid); LIST_INSERT_HEAD(&allproc, p2, p_list); + allproc_gen++; LIST_INSERT_HEAD(PIDHASH(p2->p_pid), p2, p_hash); tidhash_add(td2); PROC_LOCK(p2); PROC_LOCK(p1); sx_xunlock(&allproc_lock); bcopy(&p1->p_startcopy, &p2->p_startcopy, __rangeof(struct proc, p_startcopy, p_endcopy)); pargs_hold(p2->p_args); PROC_UNLOCK(p1); bzero(&p2->p_startzero, __rangeof(struct proc, p_startzero, p_endzero)); p2->p_ucred = crhold(td->td_ucred); /* Tell the prison that we exist. */ prison_proc_hold(p2->p_ucred->cr_prison); PROC_UNLOCK(p2); /* * Malloc things while we don't hold any locks. */ if (flags & RFSIGSHARE) newsigacts = NULL; else newsigacts = sigacts_alloc(); /* * Copy filedesc. */ if (flags & RFCFDG) { fd = fdinit(p1->p_fd, false); fdtol = NULL; } else if (flags & RFFDG) { fd = fdcopy(p1->p_fd); fdtol = NULL; } else { fd = fdshare(p1->p_fd); if (p1->p_fdtol == NULL) p1->p_fdtol = filedesc_to_leader_alloc(NULL, NULL, p1->p_leader); if ((flags & RFTHREAD) != 0) { /* * Shared file descriptor table, and shared * process leaders. */ fdtol = p1->p_fdtol; FILEDESC_XLOCK(p1->p_fd); fdtol->fdl_refcount++; FILEDESC_XUNLOCK(p1->p_fd); } else { /* * Shared file descriptor table, and different * process leaders. */ fdtol = filedesc_to_leader_alloc(p1->p_fdtol, p1->p_fd, p2); } } /* * Make a proc table entry for the new process. * Start by zeroing the section of proc that is zero-initialized, * then copy the section that is copied directly from the parent. */ PROC_LOCK(p2); PROC_LOCK(p1); bzero(&td2->td_startzero, __rangeof(struct thread, td_startzero, td_endzero)); bcopy(&td->td_startcopy, &td2->td_startcopy, __rangeof(struct thread, td_startcopy, td_endcopy)); bcopy(&p2->p_comm, &td2->td_name, sizeof(td2->td_name)); td2->td_sigstk = td->td_sigstk; td2->td_flags = TDF_INMEM; td2->td_lend_user_pri = PRI_MAX; #ifdef VIMAGE td2->td_vnet = NULL; td2->td_vnet_lpush = NULL; #endif /* * Allow the scheduler to initialize the child. */ thread_lock(td); sched_fork(td, td2); thread_unlock(td); /* * Duplicate sub-structures as needed. * Increase reference counts on shared objects. */ p2->p_flag = P_INMEM; p2->p_flag2 = 0; p2->p_swtick = ticks; if (p1->p_flag & P_PROFIL) startprofclock(p2); td2->td_ucred = crhold(p2->p_ucred); if (flags & RFSIGSHARE) { p2->p_sigacts = sigacts_hold(p1->p_sigacts); } else { sigacts_copy(newsigacts, p1->p_sigacts); p2->p_sigacts = newsigacts; } if (flags & RFTSIGZMB) p2->p_sigparent = RFTSIGNUM(flags); else if (flags & RFLINUXTHPN) p2->p_sigparent = SIGUSR1; else p2->p_sigparent = SIGCHLD; p2->p_textvp = p1->p_textvp; p2->p_fd = fd; p2->p_fdtol = fdtol; if (p1->p_flag2 & P2_INHERIT_PROTECTED) { p2->p_flag |= P_PROTECTED; p2->p_flag2 |= P2_INHERIT_PROTECTED; } /* * p_limit is copy-on-write. Bump its refcount. */ lim_fork(p1, p2); pstats_fork(p1->p_stats, p2->p_stats); PROC_UNLOCK(p1); PROC_UNLOCK(p2); /* Bump references to the text vnode (for procfs). */ if (p2->p_textvp) vref(p2->p_textvp); /* * Set up linkage for kernel based threading. */ if ((flags & RFTHREAD) != 0) { mtx_lock(&ppeers_lock); p2->p_peers = p1->p_peers; p1->p_peers = p2; p2->p_leader = p1->p_leader; mtx_unlock(&ppeers_lock); PROC_LOCK(p1->p_leader); if ((p1->p_leader->p_flag & P_WEXIT) != 0) { PROC_UNLOCK(p1->p_leader); /* * The task leader is exiting, so process p1 is * going to be killed shortly. Since p1 obviously * isn't dead yet, we know that the leader is either * sending SIGKILL's to all the processes in this * task or is sleeping waiting for all the peers to * exit. We let p1 complete the fork, but we need * to go ahead and kill the new process p2 since * the task leader may not get a chance to send * SIGKILL to it. We leave it on the list so that * the task leader will wait for this new process * to commit suicide. */ PROC_LOCK(p2); kern_psignal(p2, SIGKILL); PROC_UNLOCK(p2); } else PROC_UNLOCK(p1->p_leader); } else { p2->p_peers = NULL; p2->p_leader = p2; } sx_xlock(&proctree_lock); PGRP_LOCK(p1->p_pgrp); PROC_LOCK(p2); PROC_LOCK(p1); /* * Preserve some more flags in subprocess. P_PROFIL has already * been preserved. */ p2->p_flag |= p1->p_flag & P_SUGID; td2->td_pflags |= td->td_pflags & TDP_ALTSTACK; SESS_LOCK(p1->p_session); if (p1->p_session->s_ttyvp != NULL && p1->p_flag & P_CONTROLT) p2->p_flag |= P_CONTROLT; SESS_UNLOCK(p1->p_session); if (flags & RFPPWAIT) p2->p_flag |= P_PPWAIT; p2->p_pgrp = p1->p_pgrp; LIST_INSERT_AFTER(p1, p2, p_pglist); PGRP_UNLOCK(p1->p_pgrp); LIST_INIT(&p2->p_children); LIST_INIT(&p2->p_orphans); callout_init_mtx(&p2->p_itcallout, &p2->p_mtx, 0); /* * If PF_FORK is set, the child process inherits the * procfs ioctl flags from its parent. */ if (p1->p_pfsflags & PF_FORK) { p2->p_stops = p1->p_stops; p2->p_pfsflags = p1->p_pfsflags; } /* * This begins the section where we must prevent the parent * from being swapped. */ _PHOLD(p1); PROC_UNLOCK(p1); /* * Attach the new process to its parent. * * If RFNOWAIT is set, the newly created process becomes a child * of init. This effectively disassociates the child from the * parent. */ if (flags & RFNOWAIT) pptr = initproc; else pptr = p1; p2->p_pptr = pptr; LIST_INSERT_HEAD(&pptr->p_children, p2, p_sibling); sx_xunlock(&proctree_lock); /* Inform accounting that we have forked. */ p2->p_acflag = AFORK; PROC_UNLOCK(p2); #ifdef KTRACE ktrprocfork(p1, p2); #endif /* * Finish creating the child process. It will return via a different * execution path later. (ie: directly into user mode) */ vm_forkproc(td, p2, td2, vm2, flags); if (flags == (RFFDG | RFPROC)) { PCPU_INC(cnt.v_forks); PCPU_ADD(cnt.v_forkpages, p2->p_vmspace->vm_dsize + p2->p_vmspace->vm_ssize); } else if (flags == (RFFDG | RFPROC | RFPPWAIT | RFMEM)) { PCPU_INC(cnt.v_vforks); PCPU_ADD(cnt.v_vforkpages, p2->p_vmspace->vm_dsize + p2->p_vmspace->vm_ssize); } else if (p1 == &proc0) { PCPU_INC(cnt.v_kthreads); PCPU_ADD(cnt.v_kthreadpages, p2->p_vmspace->vm_dsize + p2->p_vmspace->vm_ssize); } else { PCPU_INC(cnt.v_rforks); PCPU_ADD(cnt.v_rforkpages, p2->p_vmspace->vm_dsize + p2->p_vmspace->vm_ssize); } /* * Associate the process descriptor with the process before anything * can happen that might cause that process to need the descriptor. * However, don't do this until after fork(2) can no longer fail. */ if (flags & RFPROCDESC) procdesc_new(p2, pdflags); /* * Both processes are set up, now check if any loadable modules want * to adjust anything. */ EVENTHANDLER_INVOKE(process_fork, p1, p2, flags); /* * Set the child start time and mark the process as being complete. */ PROC_LOCK(p2); PROC_LOCK(p1); microuptime(&p2->p_stats->p_start); PROC_SLOCK(p2); p2->p_state = PRS_NORMAL; PROC_SUNLOCK(p2); #ifdef KDTRACE_HOOKS /* * Tell the DTrace fasttrap provider about the new process so that any * tracepoints inherited from the parent can be removed. We have to do * this only after p_state is PRS_NORMAL since the fasttrap module will * use pfind() later on. */ if ((flags & RFMEM) == 0 && dtrace_fasttrap_fork) dtrace_fasttrap_fork(p1, p2); #endif if ((p1->p_flag & (P_TRACED | P_FOLLOWFORK)) == (P_TRACED | P_FOLLOWFORK)) { /* * Arrange for debugger to receive the fork event. * * We can report PL_FLAG_FORKED regardless of * P_FOLLOWFORK settings, but it does not make a sense * for runaway child. */ td->td_dbgflags |= TDB_FORK; td->td_dbg_forked = p2->p_pid; td2->td_dbgflags |= TDB_STOPATFORK; _PHOLD(p2); p2_held = 1; } if (flags & RFPPWAIT) { td->td_pflags |= TDP_RFPPWAIT; td->td_rfppwait_p = p2; } PROC_UNLOCK(p2); if ((flags & RFSTOPPED) == 0) { /* * If RFSTOPPED not requested, make child runnable and * add to run queue. */ thread_lock(td2); TD_SET_CAN_RUN(td2); sched_add(td2, SRQ_BORING); thread_unlock(td2); } /* * Now can be swapped. */ _PRELE(p1); PROC_UNLOCK(p1); /* * Tell any interested parties about the new process. */ knote_fork(&p1->p_klist, p2->p_pid); SDT_PROBE(proc, kernel, , create, p2, p1, flags, 0, 0); /* * Wait until debugger is attached to child. */ PROC_LOCK(p2); while ((td2->td_dbgflags & TDB_STOPATFORK) != 0) cv_wait(&p2->p_dbgwait, &p2->p_mtx); if (p2_held) _PRELE(p2); PROC_UNLOCK(p2); } int fork1(struct thread *td, int flags, int pages, struct proc **procp, int *procdescp, int pdflags) { struct proc *p1; struct proc *newproc; int ok; struct thread *td2; struct vmspace *vm2; vm_ooffset_t mem_charged; int error; static int curfail; static struct timeval lastfail; struct file *fp_procdesc = NULL; /* Check for the undefined or unimplemented flags. */ if ((flags & ~(RFFLAGS | RFTSIGFLAGS(RFTSIGMASK))) != 0) return (EINVAL); /* Signal value requires RFTSIGZMB. */ if ((flags & RFTSIGFLAGS(RFTSIGMASK)) != 0 && (flags & RFTSIGZMB) == 0) return (EINVAL); /* Can't copy and clear. */ if ((flags & (RFFDG|RFCFDG)) == (RFFDG|RFCFDG)) return (EINVAL); /* Check the validity of the signal number. */ if ((flags & RFTSIGZMB) != 0 && (u_int)RFTSIGNUM(flags) > _SIG_MAXSIG) return (EINVAL); if ((flags & RFPROCDESC) != 0) { /* Can't not create a process yet get a process descriptor. */ if ((flags & RFPROC) == 0) return (EINVAL); /* Must provide a place to put a procdesc if creating one. */ if (procdescp == NULL) return (EINVAL); } p1 = td->td_proc; /* * Here we don't create a new process, but we divorce * certain parts of a process from itself. */ if ((flags & RFPROC) == 0) { *procp = NULL; return (fork_norfproc(td, flags)); } /* * If required, create a process descriptor in the parent first; we * will abandon it if something goes wrong. We don't finit() until * later. */ if (flags & RFPROCDESC) { error = falloc(td, &fp_procdesc, procdescp, 0); if (error != 0) return (error); } mem_charged = 0; vm2 = NULL; if (pages == 0) pages = KSTACK_PAGES; /* Allocate new proc. */ newproc = uma_zalloc(proc_zone, M_WAITOK); td2 = FIRST_THREAD_IN_PROC(newproc); if (td2 == NULL) { td2 = thread_alloc(pages); if (td2 == NULL) { error = ENOMEM; goto fail1; } proc_linkup(newproc, td2); } else { if (td2->td_kstack == 0 || td2->td_kstack_pages != pages) { if (td2->td_kstack != 0) vm_thread_dispose(td2); if (!thread_alloc_stack(td2, pages)) { error = ENOMEM; goto fail1; } } } if ((flags & RFMEM) == 0) { vm2 = vmspace_fork(p1->p_vmspace, &mem_charged); if (vm2 == NULL) { error = ENOMEM; goto fail1; } if (!swap_reserve(mem_charged)) { /* * The swap reservation failed. The accounting * from the entries of the copied vm2 will be * substracted in vmspace_free(), so force the * reservation there. */ swap_reserve_force(mem_charged); error = ENOMEM; goto fail1; } } else vm2 = NULL; /* * XXX: This is ugly; when we copy resource usage, we need to bump * per-cred resource counters. */ newproc->p_ucred = p1->p_ucred; /* * Initialize resource accounting for the child process. */ error = racct_proc_fork(p1, newproc); if (error != 0) { error = EAGAIN; goto fail1; } #ifdef MAC mac_proc_init(newproc); #endif knlist_init_mtx(&newproc->p_klist, &newproc->p_mtx); STAILQ_INIT(&newproc->p_ktr); /* We have to lock the process tree while we look for a pid. */ sx_slock(&proctree_lock); /* * Although process entries are dynamically created, we still keep * a global limit on the maximum number we will create. Don't allow * a nonprivileged user to use the last ten processes; don't let root * exceed the limit. The variable nprocs is the current number of * processes, maxproc is the limit. */ sx_xlock(&allproc_lock); if ((nprocs >= maxproc - 10 && priv_check_cred(td->td_ucred, PRIV_MAXPROC, 0) != 0) || nprocs >= maxproc) { error = EAGAIN; goto fail; } /* * Increment the count of procs running with this uid. Don't allow * a nonprivileged user to exceed their current limit. * * XXXRW: Can we avoid privilege here if it's not needed? */ error = priv_check_cred(td->td_ucred, PRIV_PROC_LIMIT, 0); if (error == 0) ok = chgproccnt(td->td_ucred->cr_ruidinfo, 1, 0); else { PROC_LOCK(p1); ok = chgproccnt(td->td_ucred->cr_ruidinfo, 1, lim_cur(p1, RLIMIT_NPROC)); PROC_UNLOCK(p1); } if (ok) { do_fork(td, flags, newproc, td2, vm2, pdflags); /* * Return child proc pointer to parent. */ *procp = newproc; if (flags & RFPROCDESC) { procdesc_finit(newproc->p_procdesc, fp_procdesc); fdrop(fp_procdesc, td); } racct_proc_fork_done(newproc); return (0); } error = EAGAIN; fail: sx_sunlock(&proctree_lock); if (ppsratecheck(&lastfail, &curfail, 1)) printf("maxproc limit exceeded by uid %u (pid %d); see tuning(7) and login.conf(5)\n", td->td_ucred->cr_ruid, p1->p_pid); sx_xunlock(&allproc_lock); #ifdef MAC mac_proc_destroy(newproc); #endif racct_proc_exit(newproc); fail1: if (vm2 != NULL) vmspace_free(vm2); uma_zfree(proc_zone, newproc); if ((flags & RFPROCDESC) != 0 && fp_procdesc != NULL) { fdclose(td->td_proc->p_fd, fp_procdesc, *procdescp, td); fdrop(fp_procdesc, td); } pause("fork", hz / 2); return (error); } /* * Handle the return of a child process from fork1(). This function * is called from the MD fork_trampoline() entry point. */ void fork_exit(void (*callout)(void *, struct trapframe *), void *arg, struct trapframe *frame) { struct proc *p; struct thread *td; struct thread *dtd; td = curthread; p = td->td_proc; KASSERT(p->p_state == PRS_NORMAL, ("executing process is still new")); CTR4(KTR_PROC, "fork_exit: new thread %p (td_sched %p, pid %d, %s)", td, td->td_sched, p->p_pid, td->td_name); sched_fork_exit(td); /* * Processes normally resume in mi_switch() after being * cpu_switch()'ed to, but when children start up they arrive here * instead, so we must do much the same things as mi_switch() would. */ if ((dtd = PCPU_GET(deadthread))) { PCPU_SET(deadthread, NULL); thread_stash(dtd); } thread_unlock(td); /* * cpu_set_fork_handler intercepts this function call to * have this call a non-return function to stay in kernel mode. * initproc has its own fork handler, but it does return. */ KASSERT(callout != NULL, ("NULL callout in fork_exit")); callout(arg, frame); /* * Check if a kernel thread misbehaved and returned from its main * function. */ if (p->p_flag & P_KTHREAD) { printf("Kernel thread \"%s\" (pid %d) exited prematurely.\n", td->td_name, p->p_pid); kproc_exit(0); } mtx_assert(&Giant, MA_NOTOWNED); if (p->p_sysent->sv_schedtail != NULL) (p->p_sysent->sv_schedtail)(td); } /* * Simplified back end of syscall(), used when returning from fork() * directly into user mode. Giant is not held on entry, and must not * be held on return. This function is passed in to fork_exit() as the * first parameter and is called when returning to a new userland process. */ void fork_return(struct thread *td, struct trapframe *frame) { struct proc *p, *dbg; if (td->td_dbgflags & TDB_STOPATFORK) { p = td->td_proc; sx_xlock(&proctree_lock); PROC_LOCK(p); if ((p->p_pptr->p_flag & (P_TRACED | P_FOLLOWFORK)) == (P_TRACED | P_FOLLOWFORK)) { /* * If debugger still wants auto-attach for the * parent's children, do it now. */ dbg = p->p_pptr->p_pptr; p->p_flag |= P_TRACED; p->p_oppid = p->p_pptr->p_pid; proc_reparent(p, dbg); sx_xunlock(&proctree_lock); td->td_dbgflags |= TDB_CHILD; ptracestop(td, SIGSTOP); td->td_dbgflags &= ~TDB_CHILD; } else { /* * ... otherwise clear the request. */ sx_xunlock(&proctree_lock); td->td_dbgflags &= ~TDB_STOPATFORK; cv_broadcast(&p->p_dbgwait); } PROC_UNLOCK(p); } userret(td, frame); #ifdef KTRACE if (KTRPOINT(td, KTR_SYSRET)) ktrsysret(SYS_fork, 0, 0); #endif } Index: projects/clang350-import/sys/kern/kern_proc.c =================================================================== --- projects/clang350-import/sys/kern/kern_proc.c (revision 275748) +++ projects/clang350-import/sys/kern/kern_proc.c (revision 275749) @@ -1,2898 +1,3036 @@ /*- * Copyright (c) 1982, 1986, 1989, 1991, 1993 * The Regents of the University of California. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 4. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * @(#)kern_proc.c 8.7 (Berkeley) 2/14/95 */ #include __FBSDID("$FreeBSD$"); #include "opt_compat.h" #include "opt_ddb.h" #include "opt_ktrace.h" #include "opt_kstack_pages.h" #include "opt_stack.h" #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #ifdef DDB #include #endif #include #include #include #include #include #include #include #include #ifdef COMPAT_FREEBSD32 #include #include #endif SDT_PROVIDER_DEFINE(proc); SDT_PROBE_DEFINE4(proc, kernel, ctor, entry, "struct proc *", "int", "void *", "int"); SDT_PROBE_DEFINE4(proc, kernel, ctor, return, "struct proc *", "int", "void *", "int"); SDT_PROBE_DEFINE4(proc, kernel, dtor, entry, "struct proc *", "int", "void *", "struct thread *"); SDT_PROBE_DEFINE3(proc, kernel, dtor, return, "struct proc *", "int", "void *"); SDT_PROBE_DEFINE3(proc, kernel, init, entry, "struct proc *", "int", "int"); SDT_PROBE_DEFINE3(proc, kernel, init, return, "struct proc *", "int", "int"); MALLOC_DEFINE(M_PGRP, "pgrp", "process group header"); MALLOC_DEFINE(M_SESSION, "session", "session header"); static MALLOC_DEFINE(M_PROC, "proc", "Proc structures"); MALLOC_DEFINE(M_SUBPROC, "subproc", "Proc sub-structures"); static void doenterpgrp(struct proc *, struct pgrp *); static void orphanpg(struct pgrp *pg); static void fill_kinfo_aggregate(struct proc *p, struct kinfo_proc *kp); static void fill_kinfo_proc_only(struct proc *p, struct kinfo_proc *kp); static void fill_kinfo_thread(struct thread *td, struct kinfo_proc *kp, int preferthread); static void pgadjustjobc(struct pgrp *pgrp, int entering); static void pgdelete(struct pgrp *); static int proc_ctor(void *mem, int size, void *arg, int flags); static void proc_dtor(void *mem, int size, void *arg); static int proc_init(void *mem, int size, int flags); static void proc_fini(void *mem, int size); static void pargs_free(struct pargs *pa); static struct proc *zpfind_locked(pid_t pid); /* * Other process lists */ struct pidhashhead *pidhashtbl; u_long pidhash; struct pgrphashhead *pgrphashtbl; u_long pgrphash; struct proclist allproc; struct proclist zombproc; struct sx allproc_lock; struct sx proctree_lock; struct mtx ppeers_lock; uma_zone_t proc_zone; int kstack_pages = KSTACK_PAGES; SYSCTL_INT(_kern, OID_AUTO, kstack_pages, CTLFLAG_RD, &kstack_pages, 0, "Kernel stack size in pages"); static int vmmap_skip_res_cnt = 0; SYSCTL_INT(_kern, OID_AUTO, proc_vmmap_skip_resident_count, CTLFLAG_RW, &vmmap_skip_res_cnt, 0, "Skip calculation of the pages resident count in kern.proc.vmmap"); CTASSERT(sizeof(struct kinfo_proc) == KINFO_PROC_SIZE); #ifdef COMPAT_FREEBSD32 CTASSERT(sizeof(struct kinfo_proc32) == KINFO_PROC32_SIZE); #endif /* * Initialize global process hashing structures. */ void procinit() { sx_init(&allproc_lock, "allproc"); sx_init(&proctree_lock, "proctree"); mtx_init(&ppeers_lock, "p_peers", NULL, MTX_DEF); LIST_INIT(&allproc); LIST_INIT(&zombproc); pidhashtbl = hashinit(maxproc / 4, M_PROC, &pidhash); pgrphashtbl = hashinit(maxproc / 4, M_PROC, &pgrphash); proc_zone = uma_zcreate("PROC", sched_sizeof_proc(), proc_ctor, proc_dtor, proc_init, proc_fini, UMA_ALIGN_PTR, UMA_ZONE_NOFREE); uihashinit(); } /* * Prepare a proc for use. */ static int proc_ctor(void *mem, int size, void *arg, int flags) { struct proc *p; p = (struct proc *)mem; SDT_PROBE(proc, kernel, ctor , entry, p, size, arg, flags, 0); EVENTHANDLER_INVOKE(process_ctor, p); SDT_PROBE(proc, kernel, ctor , return, p, size, arg, flags, 0); return (0); } /* * Reclaim a proc after use. */ static void proc_dtor(void *mem, int size, void *arg) { struct proc *p; struct thread *td; /* INVARIANTS checks go here */ p = (struct proc *)mem; td = FIRST_THREAD_IN_PROC(p); SDT_PROBE(proc, kernel, dtor, entry, p, size, arg, td, 0); if (td != NULL) { #ifdef INVARIANTS KASSERT((p->p_numthreads == 1), ("bad number of threads in exiting process")); KASSERT(STAILQ_EMPTY(&p->p_ktr), ("proc_dtor: non-empty p_ktr")); #endif /* Free all OSD associated to this thread. */ osd_thread_exit(td); } EVENTHANDLER_INVOKE(process_dtor, p); if (p->p_ksi != NULL) KASSERT(! KSI_ONQ(p->p_ksi), ("SIGCHLD queue")); SDT_PROBE(proc, kernel, dtor, return, p, size, arg, 0, 0); } /* * Initialize type-stable parts of a proc (when newly created). */ static int proc_init(void *mem, int size, int flags) { struct proc *p; p = (struct proc *)mem; SDT_PROBE(proc, kernel, init, entry, p, size, flags, 0, 0); p->p_sched = (struct p_sched *)&p[1]; bzero(&p->p_mtx, sizeof(struct mtx)); mtx_init(&p->p_mtx, "process lock", NULL, MTX_DEF | MTX_DUPOK); mtx_init(&p->p_slock, "process slock", NULL, MTX_SPIN); mtx_init(&p->p_statmtx, "pstatl", NULL, MTX_SPIN); mtx_init(&p->p_itimmtx, "pitiml", NULL, MTX_SPIN); mtx_init(&p->p_profmtx, "pprofl", NULL, MTX_SPIN); cv_init(&p->p_pwait, "ppwait"); cv_init(&p->p_dbgwait, "dbgwait"); TAILQ_INIT(&p->p_threads); /* all threads in proc */ EVENTHANDLER_INVOKE(process_init, p); p->p_stats = pstats_alloc(); SDT_PROBE(proc, kernel, init, return, p, size, flags, 0, 0); return (0); } /* * UMA should ensure that this function is never called. * Freeing a proc structure would violate type stability. */ static void proc_fini(void *mem, int size) { #ifdef notnow struct proc *p; p = (struct proc *)mem; EVENTHANDLER_INVOKE(process_fini, p); pstats_free(p->p_stats); thread_free(FIRST_THREAD_IN_PROC(p)); mtx_destroy(&p->p_mtx); if (p->p_ksi != NULL) ksiginfo_free(p->p_ksi); #else panic("proc reclaimed"); #endif } /* * Is p an inferior of the current process? */ int inferior(struct proc *p) { sx_assert(&proctree_lock, SX_LOCKED); PROC_LOCK_ASSERT(p, MA_OWNED); for (; p != curproc; p = proc_realparent(p)) { if (p->p_pid == 0) return (0); } return (1); } struct proc * pfind_locked(pid_t pid) { struct proc *p; sx_assert(&allproc_lock, SX_LOCKED); LIST_FOREACH(p, PIDHASH(pid), p_hash) { if (p->p_pid == pid) { PROC_LOCK(p); if (p->p_state == PRS_NEW) { PROC_UNLOCK(p); p = NULL; } break; } } return (p); } /* * Locate a process by number; return only "live" processes -- i.e., neither * zombies nor newly born but incompletely initialized processes. By not * returning processes in the PRS_NEW state, we allow callers to avoid * testing for that condition to avoid dereferencing p_ucred, et al. */ struct proc * pfind(pid_t pid) { struct proc *p; sx_slock(&allproc_lock); p = pfind_locked(pid); sx_sunlock(&allproc_lock); return (p); } static struct proc * pfind_tid_locked(pid_t tid) { struct proc *p; struct thread *td; sx_assert(&allproc_lock, SX_LOCKED); FOREACH_PROC_IN_SYSTEM(p) { PROC_LOCK(p); if (p->p_state == PRS_NEW) { PROC_UNLOCK(p); continue; } FOREACH_THREAD_IN_PROC(p, td) { if (td->td_tid == tid) goto found; } PROC_UNLOCK(p); } found: return (p); } /* * Locate a process group by number. * The caller must hold proctree_lock. */ struct pgrp * pgfind(pgid) register pid_t pgid; { register struct pgrp *pgrp; sx_assert(&proctree_lock, SX_LOCKED); LIST_FOREACH(pgrp, PGRPHASH(pgid), pg_hash) { if (pgrp->pg_id == pgid) { PGRP_LOCK(pgrp); return (pgrp); } } return (NULL); } /* * Locate process and do additional manipulations, depending on flags. */ int pget(pid_t pid, int flags, struct proc **pp) { struct proc *p; int error; sx_slock(&allproc_lock); if (pid <= PID_MAX) { p = pfind_locked(pid); if (p == NULL && (flags & PGET_NOTWEXIT) == 0) p = zpfind_locked(pid); } else if ((flags & PGET_NOTID) == 0) { p = pfind_tid_locked(pid); } else { p = NULL; } sx_sunlock(&allproc_lock); if (p == NULL) return (ESRCH); if ((flags & PGET_CANSEE) != 0) { error = p_cansee(curthread, p); if (error != 0) goto errout; } if ((flags & PGET_CANDEBUG) != 0) { error = p_candebug(curthread, p); if (error != 0) goto errout; } if ((flags & PGET_ISCURRENT) != 0 && curproc != p) { error = EPERM; goto errout; } if ((flags & PGET_NOTWEXIT) != 0 && (p->p_flag & P_WEXIT) != 0) { error = ESRCH; goto errout; } if ((flags & PGET_NOTINEXEC) != 0 && (p->p_flag & P_INEXEC) != 0) { /* * XXXRW: Not clear ESRCH is the right error during proc * execve(). */ error = ESRCH; goto errout; } if ((flags & PGET_HOLD) != 0) { _PHOLD(p); PROC_UNLOCK(p); } *pp = p; return (0); errout: PROC_UNLOCK(p); return (error); } /* * Create a new process group. * pgid must be equal to the pid of p. * Begin a new session if required. */ int enterpgrp(p, pgid, pgrp, sess) register struct proc *p; pid_t pgid; struct pgrp *pgrp; struct session *sess; { sx_assert(&proctree_lock, SX_XLOCKED); KASSERT(pgrp != NULL, ("enterpgrp: pgrp == NULL")); KASSERT(p->p_pid == pgid, ("enterpgrp: new pgrp and pid != pgid")); KASSERT(pgfind(pgid) == NULL, ("enterpgrp: pgrp with pgid exists")); KASSERT(!SESS_LEADER(p), ("enterpgrp: session leader attempted setpgrp")); mtx_init(&pgrp->pg_mtx, "process group", NULL, MTX_DEF | MTX_DUPOK); if (sess != NULL) { /* * new session */ mtx_init(&sess->s_mtx, "session", NULL, MTX_DEF); PROC_LOCK(p); p->p_flag &= ~P_CONTROLT; PROC_UNLOCK(p); PGRP_LOCK(pgrp); sess->s_leader = p; sess->s_sid = p->p_pid; refcount_init(&sess->s_count, 1); sess->s_ttyvp = NULL; sess->s_ttydp = NULL; sess->s_ttyp = NULL; bcopy(p->p_session->s_login, sess->s_login, sizeof(sess->s_login)); pgrp->pg_session = sess; KASSERT(p == curproc, ("enterpgrp: mksession and p != curproc")); } else { pgrp->pg_session = p->p_session; sess_hold(pgrp->pg_session); PGRP_LOCK(pgrp); } pgrp->pg_id = pgid; LIST_INIT(&pgrp->pg_members); /* * As we have an exclusive lock of proctree_lock, * this should not deadlock. */ LIST_INSERT_HEAD(PGRPHASH(pgid), pgrp, pg_hash); pgrp->pg_jobc = 0; SLIST_INIT(&pgrp->pg_sigiolst); PGRP_UNLOCK(pgrp); doenterpgrp(p, pgrp); return (0); } /* * Move p to an existing process group */ int enterthispgrp(p, pgrp) register struct proc *p; struct pgrp *pgrp; { sx_assert(&proctree_lock, SX_XLOCKED); PROC_LOCK_ASSERT(p, MA_NOTOWNED); PGRP_LOCK_ASSERT(pgrp, MA_NOTOWNED); PGRP_LOCK_ASSERT(p->p_pgrp, MA_NOTOWNED); SESS_LOCK_ASSERT(p->p_session, MA_NOTOWNED); KASSERT(pgrp->pg_session == p->p_session, ("%s: pgrp's session %p, p->p_session %p.\n", __func__, pgrp->pg_session, p->p_session)); KASSERT(pgrp != p->p_pgrp, ("%s: p belongs to pgrp.", __func__)); doenterpgrp(p, pgrp); return (0); } /* * Move p to a process group */ static void doenterpgrp(p, pgrp) struct proc *p; struct pgrp *pgrp; { struct pgrp *savepgrp; sx_assert(&proctree_lock, SX_XLOCKED); PROC_LOCK_ASSERT(p, MA_NOTOWNED); PGRP_LOCK_ASSERT(pgrp, MA_NOTOWNED); PGRP_LOCK_ASSERT(p->p_pgrp, MA_NOTOWNED); SESS_LOCK_ASSERT(p->p_session, MA_NOTOWNED); savepgrp = p->p_pgrp; /* * Adjust eligibility of affected pgrps to participate in job control. * Increment eligibility counts before decrementing, otherwise we * could reach 0 spuriously during the first call. */ fixjobc(p, pgrp, 1); fixjobc(p, p->p_pgrp, 0); PGRP_LOCK(pgrp); PGRP_LOCK(savepgrp); PROC_LOCK(p); LIST_REMOVE(p, p_pglist); p->p_pgrp = pgrp; PROC_UNLOCK(p); LIST_INSERT_HEAD(&pgrp->pg_members, p, p_pglist); PGRP_UNLOCK(savepgrp); PGRP_UNLOCK(pgrp); if (LIST_EMPTY(&savepgrp->pg_members)) pgdelete(savepgrp); } /* * remove process from process group */ int leavepgrp(p) register struct proc *p; { struct pgrp *savepgrp; sx_assert(&proctree_lock, SX_XLOCKED); savepgrp = p->p_pgrp; PGRP_LOCK(savepgrp); PROC_LOCK(p); LIST_REMOVE(p, p_pglist); p->p_pgrp = NULL; PROC_UNLOCK(p); PGRP_UNLOCK(savepgrp); if (LIST_EMPTY(&savepgrp->pg_members)) pgdelete(savepgrp); return (0); } /* * delete a process group */ static void pgdelete(pgrp) register struct pgrp *pgrp; { struct session *savesess; struct tty *tp; sx_assert(&proctree_lock, SX_XLOCKED); PGRP_LOCK_ASSERT(pgrp, MA_NOTOWNED); SESS_LOCK_ASSERT(pgrp->pg_session, MA_NOTOWNED); /* * Reset any sigio structures pointing to us as a result of * F_SETOWN with our pgid. */ funsetownlst(&pgrp->pg_sigiolst); PGRP_LOCK(pgrp); tp = pgrp->pg_session->s_ttyp; LIST_REMOVE(pgrp, pg_hash); savesess = pgrp->pg_session; PGRP_UNLOCK(pgrp); /* Remove the reference to the pgrp before deallocating it. */ if (tp != NULL) { tty_lock(tp); tty_rel_pgrp(tp, pgrp); } mtx_destroy(&pgrp->pg_mtx); free(pgrp, M_PGRP); sess_release(savesess); } static void pgadjustjobc(pgrp, entering) struct pgrp *pgrp; int entering; { PGRP_LOCK(pgrp); if (entering) pgrp->pg_jobc++; else { --pgrp->pg_jobc; if (pgrp->pg_jobc == 0) orphanpg(pgrp); } PGRP_UNLOCK(pgrp); } /* * Adjust pgrp jobc counters when specified process changes process group. * We count the number of processes in each process group that "qualify" * the group for terminal job control (those with a parent in a different * process group of the same session). If that count reaches zero, the * process group becomes orphaned. Check both the specified process' * process group and that of its children. * entering == 0 => p is leaving specified group. * entering == 1 => p is entering specified group. */ void fixjobc(p, pgrp, entering) register struct proc *p; register struct pgrp *pgrp; int entering; { register struct pgrp *hispgrp; register struct session *mysession; sx_assert(&proctree_lock, SX_LOCKED); PROC_LOCK_ASSERT(p, MA_NOTOWNED); PGRP_LOCK_ASSERT(pgrp, MA_NOTOWNED); SESS_LOCK_ASSERT(pgrp->pg_session, MA_NOTOWNED); /* * Check p's parent to see whether p qualifies its own process * group; if so, adjust count for p's process group. */ mysession = pgrp->pg_session; if ((hispgrp = p->p_pptr->p_pgrp) != pgrp && hispgrp->pg_session == mysession) pgadjustjobc(pgrp, entering); /* * Check this process' children to see whether they qualify * their process groups; if so, adjust counts for children's * process groups. */ LIST_FOREACH(p, &p->p_children, p_sibling) { hispgrp = p->p_pgrp; if (hispgrp == pgrp || hispgrp->pg_session != mysession) continue; PROC_LOCK(p); if (p->p_state == PRS_ZOMBIE) { PROC_UNLOCK(p); continue; } PROC_UNLOCK(p); pgadjustjobc(hispgrp, entering); } } /* * A process group has become orphaned; * if there are any stopped processes in the group, * hang-up all process in that group. */ static void orphanpg(pg) struct pgrp *pg; { register struct proc *p; PGRP_LOCK_ASSERT(pg, MA_OWNED); LIST_FOREACH(p, &pg->pg_members, p_pglist) { PROC_LOCK(p); if (P_SHOULDSTOP(p)) { PROC_UNLOCK(p); LIST_FOREACH(p, &pg->pg_members, p_pglist) { PROC_LOCK(p); kern_psignal(p, SIGHUP); kern_psignal(p, SIGCONT); PROC_UNLOCK(p); } return; } PROC_UNLOCK(p); } } void sess_hold(struct session *s) { refcount_acquire(&s->s_count); } void sess_release(struct session *s) { if (refcount_release(&s->s_count)) { if (s->s_ttyp != NULL) { tty_lock(s->s_ttyp); tty_rel_sess(s->s_ttyp, s); } mtx_destroy(&s->s_mtx); free(s, M_SESSION); } } #ifdef DDB DB_SHOW_COMMAND(pgrpdump, pgrpdump) { register struct pgrp *pgrp; register struct proc *p; register int i; for (i = 0; i <= pgrphash; i++) { if (!LIST_EMPTY(&pgrphashtbl[i])) { printf("\tindx %d\n", i); LIST_FOREACH(pgrp, &pgrphashtbl[i], pg_hash) { printf( "\tpgrp %p, pgid %ld, sess %p, sesscnt %d, mem %p\n", (void *)pgrp, (long)pgrp->pg_id, (void *)pgrp->pg_session, pgrp->pg_session->s_count, (void *)LIST_FIRST(&pgrp->pg_members)); LIST_FOREACH(p, &pgrp->pg_members, p_pglist) { printf("\t\tpid %ld addr %p pgrp %p\n", (long)p->p_pid, (void *)p, (void *)p->p_pgrp); } } } } } #endif /* DDB */ /* * Calculate the kinfo_proc members which contain process-wide * informations. * Must be called with the target process locked. */ static void fill_kinfo_aggregate(struct proc *p, struct kinfo_proc *kp) { struct thread *td; PROC_LOCK_ASSERT(p, MA_OWNED); kp->ki_estcpu = 0; kp->ki_pctcpu = 0; FOREACH_THREAD_IN_PROC(p, td) { thread_lock(td); kp->ki_pctcpu += sched_pctcpu(td); kp->ki_estcpu += td->td_estcpu; thread_unlock(td); } } /* * Clear kinfo_proc and fill in any information that is common * to all threads in the process. * Must be called with the target process locked. */ static void fill_kinfo_proc_only(struct proc *p, struct kinfo_proc *kp) { struct thread *td0; struct tty *tp; struct session *sp; struct ucred *cred; struct sigacts *ps; /* For proc_realparent. */ sx_assert(&proctree_lock, SX_LOCKED); PROC_LOCK_ASSERT(p, MA_OWNED); bzero(kp, sizeof(*kp)); kp->ki_structsize = sizeof(*kp); kp->ki_paddr = p; kp->ki_addr =/* p->p_addr; */0; /* XXX */ kp->ki_args = p->p_args; kp->ki_textvp = p->p_textvp; #ifdef KTRACE kp->ki_tracep = p->p_tracevp; kp->ki_traceflag = p->p_traceflag; #endif kp->ki_fd = p->p_fd; kp->ki_vmspace = p->p_vmspace; kp->ki_flag = p->p_flag; kp->ki_flag2 = p->p_flag2; cred = p->p_ucred; if (cred) { kp->ki_uid = cred->cr_uid; kp->ki_ruid = cred->cr_ruid; kp->ki_svuid = cred->cr_svuid; kp->ki_cr_flags = 0; if (cred->cr_flags & CRED_FLAG_CAPMODE) kp->ki_cr_flags |= KI_CRF_CAPABILITY_MODE; /* XXX bde doesn't like KI_NGROUPS */ if (cred->cr_ngroups > KI_NGROUPS) { kp->ki_ngroups = KI_NGROUPS; kp->ki_cr_flags |= KI_CRF_GRP_OVERFLOW; } else kp->ki_ngroups = cred->cr_ngroups; bcopy(cred->cr_groups, kp->ki_groups, kp->ki_ngroups * sizeof(gid_t)); kp->ki_rgid = cred->cr_rgid; kp->ki_svgid = cred->cr_svgid; /* If jailed(cred), emulate the old P_JAILED flag. */ if (jailed(cred)) { kp->ki_flag |= P_JAILED; /* If inside the jail, use 0 as a jail ID. */ if (cred->cr_prison != curthread->td_ucred->cr_prison) kp->ki_jid = cred->cr_prison->pr_id; } strlcpy(kp->ki_loginclass, cred->cr_loginclass->lc_name, sizeof(kp->ki_loginclass)); } ps = p->p_sigacts; if (ps) { mtx_lock(&ps->ps_mtx); kp->ki_sigignore = ps->ps_sigignore; kp->ki_sigcatch = ps->ps_sigcatch; mtx_unlock(&ps->ps_mtx); } if (p->p_state != PRS_NEW && p->p_state != PRS_ZOMBIE && p->p_vmspace != NULL) { struct vmspace *vm = p->p_vmspace; kp->ki_size = vm->vm_map.size; kp->ki_rssize = vmspace_resident_count(vm); /*XXX*/ FOREACH_THREAD_IN_PROC(p, td0) { if (!TD_IS_SWAPPED(td0)) kp->ki_rssize += td0->td_kstack_pages; } kp->ki_swrss = vm->vm_swrss; kp->ki_tsize = vm->vm_tsize; kp->ki_dsize = vm->vm_dsize; kp->ki_ssize = vm->vm_ssize; } else if (p->p_state == PRS_ZOMBIE) kp->ki_stat = SZOMB; if (kp->ki_flag & P_INMEM) kp->ki_sflag = PS_INMEM; else kp->ki_sflag = 0; /* Calculate legacy swtime as seconds since 'swtick'. */ kp->ki_swtime = (ticks - p->p_swtick) / hz; kp->ki_pid = p->p_pid; kp->ki_nice = p->p_nice; kp->ki_fibnum = p->p_fibnum; kp->ki_start = p->p_stats->p_start; timevaladd(&kp->ki_start, &boottime); PROC_STATLOCK(p); rufetch(p, &kp->ki_rusage); kp->ki_runtime = cputick2usec(p->p_rux.rux_runtime); calcru(p, &kp->ki_rusage.ru_utime, &kp->ki_rusage.ru_stime); PROC_STATUNLOCK(p); calccru(p, &kp->ki_childutime, &kp->ki_childstime); /* Some callers want child times in a single value. */ kp->ki_childtime = kp->ki_childstime; timevaladd(&kp->ki_childtime, &kp->ki_childutime); FOREACH_THREAD_IN_PROC(p, td0) kp->ki_cow += td0->td_cow; tp = NULL; if (p->p_pgrp) { kp->ki_pgid = p->p_pgrp->pg_id; kp->ki_jobc = p->p_pgrp->pg_jobc; sp = p->p_pgrp->pg_session; if (sp != NULL) { kp->ki_sid = sp->s_sid; SESS_LOCK(sp); strlcpy(kp->ki_login, sp->s_login, sizeof(kp->ki_login)); if (sp->s_ttyvp) kp->ki_kiflag |= KI_CTTY; if (SESS_LEADER(p)) kp->ki_kiflag |= KI_SLEADER; /* XXX proctree_lock */ tp = sp->s_ttyp; SESS_UNLOCK(sp); } } if ((p->p_flag & P_CONTROLT) && tp != NULL) { kp->ki_tdev = tty_udev(tp); kp->ki_tpgid = tp->t_pgrp ? tp->t_pgrp->pg_id : NO_PID; if (tp->t_session) kp->ki_tsid = tp->t_session->s_sid; } else kp->ki_tdev = NODEV; if (p->p_comm[0] != '\0') strlcpy(kp->ki_comm, p->p_comm, sizeof(kp->ki_comm)); if (p->p_sysent && p->p_sysent->sv_name != NULL && p->p_sysent->sv_name[0] != '\0') strlcpy(kp->ki_emul, p->p_sysent->sv_name, sizeof(kp->ki_emul)); kp->ki_siglist = p->p_siglist; kp->ki_xstat = p->p_xstat; kp->ki_acflag = p->p_acflag; kp->ki_lock = p->p_lock; if (p->p_pptr) { kp->ki_ppid = proc_realparent(p)->p_pid; if (p->p_flag & P_TRACED) kp->ki_tracer = p->p_pptr->p_pid; } } /* * Fill in information that is thread specific. Must be called with * target process locked. If 'preferthread' is set, overwrite certain * process-related fields that are maintained for both threads and * processes. */ static void fill_kinfo_thread(struct thread *td, struct kinfo_proc *kp, int preferthread) { struct proc *p; p = td->td_proc; kp->ki_tdaddr = td; PROC_LOCK_ASSERT(p, MA_OWNED); if (preferthread) PROC_STATLOCK(p); thread_lock(td); if (td->td_wmesg != NULL) strlcpy(kp->ki_wmesg, td->td_wmesg, sizeof(kp->ki_wmesg)); else bzero(kp->ki_wmesg, sizeof(kp->ki_wmesg)); strlcpy(kp->ki_tdname, td->td_name, sizeof(kp->ki_tdname)); if (TD_ON_LOCK(td)) { kp->ki_kiflag |= KI_LOCKBLOCK; strlcpy(kp->ki_lockname, td->td_lockname, sizeof(kp->ki_lockname)); } else { kp->ki_kiflag &= ~KI_LOCKBLOCK; bzero(kp->ki_lockname, sizeof(kp->ki_lockname)); } if (p->p_state == PRS_NORMAL) { /* approximate. */ if (TD_ON_RUNQ(td) || TD_CAN_RUN(td) || TD_IS_RUNNING(td)) { kp->ki_stat = SRUN; } else if (P_SHOULDSTOP(p)) { kp->ki_stat = SSTOP; } else if (TD_IS_SLEEPING(td)) { kp->ki_stat = SSLEEP; } else if (TD_ON_LOCK(td)) { kp->ki_stat = SLOCK; } else { kp->ki_stat = SWAIT; } } else if (p->p_state == PRS_ZOMBIE) { kp->ki_stat = SZOMB; } else { kp->ki_stat = SIDL; } /* Things in the thread */ kp->ki_wchan = td->td_wchan; kp->ki_pri.pri_level = td->td_priority; kp->ki_pri.pri_native = td->td_base_pri; /* * Note: legacy fields; clamp at the old NOCPU value and/or * the maximum u_char CPU value. */ if (td->td_lastcpu == NOCPU) kp->ki_lastcpu_old = NOCPU_OLD; else if (td->td_lastcpu > MAXCPU_OLD) kp->ki_lastcpu_old = MAXCPU_OLD; else kp->ki_lastcpu_old = td->td_lastcpu; if (td->td_oncpu == NOCPU) kp->ki_oncpu_old = NOCPU_OLD; else if (td->td_oncpu > MAXCPU_OLD) kp->ki_oncpu_old = MAXCPU_OLD; else kp->ki_oncpu_old = td->td_oncpu; kp->ki_lastcpu = td->td_lastcpu; kp->ki_oncpu = td->td_oncpu; kp->ki_tdflags = td->td_flags; kp->ki_tid = td->td_tid; kp->ki_numthreads = p->p_numthreads; kp->ki_pcb = td->td_pcb; kp->ki_kstack = (void *)td->td_kstack; kp->ki_slptime = (ticks - td->td_slptick) / hz; kp->ki_pri.pri_class = td->td_pri_class; kp->ki_pri.pri_user = td->td_user_pri; if (preferthread) { rufetchtd(td, &kp->ki_rusage); kp->ki_runtime = cputick2usec(td->td_rux.rux_runtime); kp->ki_pctcpu = sched_pctcpu(td); kp->ki_estcpu = td->td_estcpu; kp->ki_cow = td->td_cow; } /* We can't get this anymore but ps etc never used it anyway. */ kp->ki_rqindex = 0; if (preferthread) kp->ki_siglist = td->td_siglist; kp->ki_sigmask = td->td_sigmask; thread_unlock(td); if (preferthread) PROC_STATUNLOCK(p); } /* * Fill in a kinfo_proc structure for the specified process. * Must be called with the target process locked. */ void fill_kinfo_proc(struct proc *p, struct kinfo_proc *kp) { MPASS(FIRST_THREAD_IN_PROC(p) != NULL); fill_kinfo_proc_only(p, kp); fill_kinfo_thread(FIRST_THREAD_IN_PROC(p), kp, 0); fill_kinfo_aggregate(p, kp); } struct pstats * pstats_alloc(void) { return (malloc(sizeof(struct pstats), M_SUBPROC, M_ZERO|M_WAITOK)); } /* * Copy parts of p_stats; zero the rest of p_stats (statistics). */ void pstats_fork(struct pstats *src, struct pstats *dst) { bzero(&dst->pstat_startzero, __rangeof(struct pstats, pstat_startzero, pstat_endzero)); bcopy(&src->pstat_startcopy, &dst->pstat_startcopy, __rangeof(struct pstats, pstat_startcopy, pstat_endcopy)); } void pstats_free(struct pstats *ps) { free(ps, M_SUBPROC); } static struct proc * zpfind_locked(pid_t pid) { struct proc *p; sx_assert(&allproc_lock, SX_LOCKED); LIST_FOREACH(p, &zombproc, p_list) { if (p->p_pid == pid) { PROC_LOCK(p); break; } } return (p); } /* * Locate a zombie process by number */ struct proc * zpfind(pid_t pid) { struct proc *p; sx_slock(&allproc_lock); p = zpfind_locked(pid); sx_sunlock(&allproc_lock); return (p); } #ifdef COMPAT_FREEBSD32 /* * This function is typically used to copy out the kernel address, so * it can be replaced by assignment of zero. */ static inline uint32_t ptr32_trim(void *ptr) { uintptr_t uptr; uptr = (uintptr_t)ptr; return ((uptr > UINT_MAX) ? 0 : uptr); } #define PTRTRIM_CP(src,dst,fld) \ do { (dst).fld = ptr32_trim((src).fld); } while (0) static void freebsd32_kinfo_proc_out(const struct kinfo_proc *ki, struct kinfo_proc32 *ki32) { int i; bzero(ki32, sizeof(struct kinfo_proc32)); ki32->ki_structsize = sizeof(struct kinfo_proc32); CP(*ki, *ki32, ki_layout); PTRTRIM_CP(*ki, *ki32, ki_args); PTRTRIM_CP(*ki, *ki32, ki_paddr); PTRTRIM_CP(*ki, *ki32, ki_addr); PTRTRIM_CP(*ki, *ki32, ki_tracep); PTRTRIM_CP(*ki, *ki32, ki_textvp); PTRTRIM_CP(*ki, *ki32, ki_fd); PTRTRIM_CP(*ki, *ki32, ki_vmspace); PTRTRIM_CP(*ki, *ki32, ki_wchan); CP(*ki, *ki32, ki_pid); CP(*ki, *ki32, ki_ppid); CP(*ki, *ki32, ki_pgid); CP(*ki, *ki32, ki_tpgid); CP(*ki, *ki32, ki_sid); CP(*ki, *ki32, ki_tsid); CP(*ki, *ki32, ki_jobc); CP(*ki, *ki32, ki_tdev); CP(*ki, *ki32, ki_siglist); CP(*ki, *ki32, ki_sigmask); CP(*ki, *ki32, ki_sigignore); CP(*ki, *ki32, ki_sigcatch); CP(*ki, *ki32, ki_uid); CP(*ki, *ki32, ki_ruid); CP(*ki, *ki32, ki_svuid); CP(*ki, *ki32, ki_rgid); CP(*ki, *ki32, ki_svgid); CP(*ki, *ki32, ki_ngroups); for (i = 0; i < KI_NGROUPS; i++) CP(*ki, *ki32, ki_groups[i]); CP(*ki, *ki32, ki_size); CP(*ki, *ki32, ki_rssize); CP(*ki, *ki32, ki_swrss); CP(*ki, *ki32, ki_tsize); CP(*ki, *ki32, ki_dsize); CP(*ki, *ki32, ki_ssize); CP(*ki, *ki32, ki_xstat); CP(*ki, *ki32, ki_acflag); CP(*ki, *ki32, ki_pctcpu); CP(*ki, *ki32, ki_estcpu); CP(*ki, *ki32, ki_slptime); CP(*ki, *ki32, ki_swtime); CP(*ki, *ki32, ki_cow); CP(*ki, *ki32, ki_runtime); TV_CP(*ki, *ki32, ki_start); TV_CP(*ki, *ki32, ki_childtime); CP(*ki, *ki32, ki_flag); CP(*ki, *ki32, ki_kiflag); CP(*ki, *ki32, ki_traceflag); CP(*ki, *ki32, ki_stat); CP(*ki, *ki32, ki_nice); CP(*ki, *ki32, ki_lock); CP(*ki, *ki32, ki_rqindex); CP(*ki, *ki32, ki_oncpu); CP(*ki, *ki32, ki_lastcpu); /* XXX TODO: wrap cpu value as appropriate */ CP(*ki, *ki32, ki_oncpu_old); CP(*ki, *ki32, ki_lastcpu_old); bcopy(ki->ki_tdname, ki32->ki_tdname, TDNAMLEN + 1); bcopy(ki->ki_wmesg, ki32->ki_wmesg, WMESGLEN + 1); bcopy(ki->ki_login, ki32->ki_login, LOGNAMELEN + 1); bcopy(ki->ki_lockname, ki32->ki_lockname, LOCKNAMELEN + 1); bcopy(ki->ki_comm, ki32->ki_comm, COMMLEN + 1); bcopy(ki->ki_emul, ki32->ki_emul, KI_EMULNAMELEN + 1); bcopy(ki->ki_loginclass, ki32->ki_loginclass, LOGINCLASSLEN + 1); CP(*ki, *ki32, ki_tracer); CP(*ki, *ki32, ki_flag2); CP(*ki, *ki32, ki_fibnum); CP(*ki, *ki32, ki_cr_flags); CP(*ki, *ki32, ki_jid); CP(*ki, *ki32, ki_numthreads); CP(*ki, *ki32, ki_tid); CP(*ki, *ki32, ki_pri); freebsd32_rusage_out(&ki->ki_rusage, &ki32->ki_rusage); freebsd32_rusage_out(&ki->ki_rusage_ch, &ki32->ki_rusage_ch); PTRTRIM_CP(*ki, *ki32, ki_pcb); PTRTRIM_CP(*ki, *ki32, ki_kstack); PTRTRIM_CP(*ki, *ki32, ki_udata); CP(*ki, *ki32, ki_sflag); CP(*ki, *ki32, ki_tdflags); } #endif int kern_proc_out(struct proc *p, struct sbuf *sb, int flags) { struct thread *td; struct kinfo_proc ki; #ifdef COMPAT_FREEBSD32 struct kinfo_proc32 ki32; #endif int error; PROC_LOCK_ASSERT(p, MA_OWNED); MPASS(FIRST_THREAD_IN_PROC(p) != NULL); error = 0; fill_kinfo_proc(p, &ki); if ((flags & KERN_PROC_NOTHREADS) != 0) { #ifdef COMPAT_FREEBSD32 if ((flags & KERN_PROC_MASK32) != 0) { freebsd32_kinfo_proc_out(&ki, &ki32); if (sbuf_bcat(sb, &ki32, sizeof(ki32)) != 0) error = ENOMEM; } else #endif if (sbuf_bcat(sb, &ki, sizeof(ki)) != 0) error = ENOMEM; } else { FOREACH_THREAD_IN_PROC(p, td) { fill_kinfo_thread(td, &ki, 1); #ifdef COMPAT_FREEBSD32 if ((flags & KERN_PROC_MASK32) != 0) { freebsd32_kinfo_proc_out(&ki, &ki32); if (sbuf_bcat(sb, &ki32, sizeof(ki32)) != 0) error = ENOMEM; } else #endif if (sbuf_bcat(sb, &ki, sizeof(ki)) != 0) error = ENOMEM; if (error != 0) break; } } PROC_UNLOCK(p); return (error); } static int sysctl_out_proc(struct proc *p, struct sysctl_req *req, int flags, int doingzomb) { struct sbuf sb; struct kinfo_proc ki; struct proc *np; int error, error2; pid_t pid; pid = p->p_pid; sbuf_new_for_sysctl(&sb, (char *)&ki, sizeof(ki), req); error = kern_proc_out(p, &sb, flags); error2 = sbuf_finish(&sb); sbuf_delete(&sb); if (error != 0) return (error); else if (error2 != 0) return (error2); if (doingzomb) np = zpfind(pid); else { if (pid == 0) return (0); np = pfind(pid); } if (np == NULL) return (ESRCH); if (np != p) { PROC_UNLOCK(np); return (ESRCH); } PROC_UNLOCK(np); return (0); } static int sysctl_kern_proc(SYSCTL_HANDLER_ARGS) { int *name = (int *)arg1; u_int namelen = arg2; struct proc *p; int flags, doingzomb, oid_number; int error = 0; oid_number = oidp->oid_number; if (oid_number != KERN_PROC_ALL && (oid_number & KERN_PROC_INC_THREAD) == 0) flags = KERN_PROC_NOTHREADS; else { flags = 0; oid_number &= ~KERN_PROC_INC_THREAD; } #ifdef COMPAT_FREEBSD32 if (req->flags & SCTL_MASK32) flags |= KERN_PROC_MASK32; #endif if (oid_number == KERN_PROC_PID) { if (namelen != 1) return (EINVAL); error = sysctl_wire_old_buffer(req, 0); if (error) return (error); sx_slock(&proctree_lock); error = pget((pid_t)name[0], PGET_CANSEE, &p); if (error == 0) error = sysctl_out_proc(p, req, flags, 0); sx_sunlock(&proctree_lock); return (error); } switch (oid_number) { case KERN_PROC_ALL: if (namelen != 0) return (EINVAL); break; case KERN_PROC_PROC: if (namelen != 0 && namelen != 1) return (EINVAL); break; default: if (namelen != 1) return (EINVAL); break; } if (!req->oldptr) { /* overestimate by 5 procs */ error = SYSCTL_OUT(req, 0, sizeof (struct kinfo_proc) * 5); if (error) return (error); } error = sysctl_wire_old_buffer(req, 0); if (error != 0) return (error); sx_slock(&proctree_lock); sx_slock(&allproc_lock); for (doingzomb=0 ; doingzomb < 2 ; doingzomb++) { if (!doingzomb) p = LIST_FIRST(&allproc); else p = LIST_FIRST(&zombproc); for (; p != 0; p = LIST_NEXT(p, p_list)) { /* * Skip embryonic processes. */ PROC_LOCK(p); if (p->p_state == PRS_NEW) { PROC_UNLOCK(p); continue; } KASSERT(p->p_ucred != NULL, ("process credential is NULL for non-NEW proc")); /* * Show a user only appropriate processes. */ if (p_cansee(curthread, p)) { PROC_UNLOCK(p); continue; } /* * TODO - make more efficient (see notes below). * do by session. */ switch (oid_number) { case KERN_PROC_GID: if (p->p_ucred->cr_gid != (gid_t)name[0]) { PROC_UNLOCK(p); continue; } break; case KERN_PROC_PGRP: /* could do this by traversing pgrp */ if (p->p_pgrp == NULL || p->p_pgrp->pg_id != (pid_t)name[0]) { PROC_UNLOCK(p); continue; } break; case KERN_PROC_RGID: if (p->p_ucred->cr_rgid != (gid_t)name[0]) { PROC_UNLOCK(p); continue; } break; case KERN_PROC_SESSION: if (p->p_session == NULL || p->p_session->s_sid != (pid_t)name[0]) { PROC_UNLOCK(p); continue; } break; case KERN_PROC_TTY: if ((p->p_flag & P_CONTROLT) == 0 || p->p_session == NULL) { PROC_UNLOCK(p); continue; } /* XXX proctree_lock */ SESS_LOCK(p->p_session); if (p->p_session->s_ttyp == NULL || tty_udev(p->p_session->s_ttyp) != (dev_t)name[0]) { SESS_UNLOCK(p->p_session); PROC_UNLOCK(p); continue; } SESS_UNLOCK(p->p_session); break; case KERN_PROC_UID: if (p->p_ucred->cr_uid != (uid_t)name[0]) { PROC_UNLOCK(p); continue; } break; case KERN_PROC_RUID: if (p->p_ucred->cr_ruid != (uid_t)name[0]) { PROC_UNLOCK(p); continue; } break; case KERN_PROC_PROC: break; default: break; } error = sysctl_out_proc(p, req, flags, doingzomb); if (error) { sx_sunlock(&allproc_lock); sx_sunlock(&proctree_lock); return (error); } } } sx_sunlock(&allproc_lock); sx_sunlock(&proctree_lock); return (0); } struct pargs * pargs_alloc(int len) { struct pargs *pa; pa = malloc(sizeof(struct pargs) + len, M_PARGS, M_WAITOK); refcount_init(&pa->ar_ref, 1); pa->ar_length = len; return (pa); } static void pargs_free(struct pargs *pa) { free(pa, M_PARGS); } void pargs_hold(struct pargs *pa) { if (pa == NULL) return; refcount_acquire(&pa->ar_ref); } void pargs_drop(struct pargs *pa) { if (pa == NULL) return; if (refcount_release(&pa->ar_ref)) pargs_free(pa); } static int proc_read_mem(struct thread *td, struct proc *p, vm_offset_t offset, void* buf, size_t len) { struct iovec iov; struct uio uio; iov.iov_base = (caddr_t)buf; iov.iov_len = len; uio.uio_iov = &iov; uio.uio_iovcnt = 1; uio.uio_offset = offset; uio.uio_resid = (ssize_t)len; uio.uio_segflg = UIO_SYSSPACE; uio.uio_rw = UIO_READ; uio.uio_td = td; return (proc_rwmem(p, &uio)); } static int proc_read_string(struct thread *td, struct proc *p, const char *sptr, char *buf, size_t len) { size_t i; int error; error = proc_read_mem(td, p, (vm_offset_t)sptr, buf, len); /* * Reading the chunk may validly return EFAULT if the string is shorter * than the chunk and is aligned at the end of the page, assuming the * next page is not mapped. So if EFAULT is returned do a fallback to * one byte read loop. */ if (error == EFAULT) { for (i = 0; i < len; i++, buf++, sptr++) { error = proc_read_mem(td, p, (vm_offset_t)sptr, buf, 1); if (error != 0) return (error); if (*buf == '\0') break; } error = 0; } return (error); } #define PROC_AUXV_MAX 256 /* Safety limit on auxv size. */ enum proc_vector_type { PROC_ARG, PROC_ENV, PROC_AUX, }; #ifdef COMPAT_FREEBSD32 static int get_proc_vector32(struct thread *td, struct proc *p, char ***proc_vectorp, size_t *vsizep, enum proc_vector_type type) { struct freebsd32_ps_strings pss; Elf32_Auxinfo aux; vm_offset_t vptr, ptr; uint32_t *proc_vector32; char **proc_vector; size_t vsize, size; int i, error; error = proc_read_mem(td, p, (vm_offset_t)(p->p_sysent->sv_psstrings), &pss, sizeof(pss)); if (error != 0) return (error); switch (type) { case PROC_ARG: vptr = (vm_offset_t)PTRIN(pss.ps_argvstr); vsize = pss.ps_nargvstr; if (vsize > ARG_MAX) return (ENOEXEC); size = vsize * sizeof(int32_t); break; case PROC_ENV: vptr = (vm_offset_t)PTRIN(pss.ps_envstr); vsize = pss.ps_nenvstr; if (vsize > ARG_MAX) return (ENOEXEC); size = vsize * sizeof(int32_t); break; case PROC_AUX: vptr = (vm_offset_t)PTRIN(pss.ps_envstr) + (pss.ps_nenvstr + 1) * sizeof(int32_t); if (vptr % 4 != 0) return (ENOEXEC); for (ptr = vptr, i = 0; i < PROC_AUXV_MAX; i++) { error = proc_read_mem(td, p, ptr, &aux, sizeof(aux)); if (error != 0) return (error); if (aux.a_type == AT_NULL) break; ptr += sizeof(aux); } if (aux.a_type != AT_NULL) return (ENOEXEC); vsize = i + 1; size = vsize * sizeof(aux); break; default: KASSERT(0, ("Wrong proc vector type: %d", type)); return (EINVAL); } proc_vector32 = malloc(size, M_TEMP, M_WAITOK); error = proc_read_mem(td, p, vptr, proc_vector32, size); if (error != 0) goto done; if (type == PROC_AUX) { *proc_vectorp = (char **)proc_vector32; *vsizep = vsize; return (0); } proc_vector = malloc(vsize * sizeof(char *), M_TEMP, M_WAITOK); for (i = 0; i < (int)vsize; i++) proc_vector[i] = PTRIN(proc_vector32[i]); *proc_vectorp = proc_vector; *vsizep = vsize; done: free(proc_vector32, M_TEMP); return (error); } #endif static int get_proc_vector(struct thread *td, struct proc *p, char ***proc_vectorp, size_t *vsizep, enum proc_vector_type type) { struct ps_strings pss; Elf_Auxinfo aux; vm_offset_t vptr, ptr; char **proc_vector; size_t vsize, size; int error, i; #ifdef COMPAT_FREEBSD32 if (SV_PROC_FLAG(p, SV_ILP32) != 0) return (get_proc_vector32(td, p, proc_vectorp, vsizep, type)); #endif error = proc_read_mem(td, p, (vm_offset_t)(p->p_sysent->sv_psstrings), &pss, sizeof(pss)); if (error != 0) return (error); switch (type) { case PROC_ARG: vptr = (vm_offset_t)pss.ps_argvstr; vsize = pss.ps_nargvstr; if (vsize > ARG_MAX) return (ENOEXEC); size = vsize * sizeof(char *); break; case PROC_ENV: vptr = (vm_offset_t)pss.ps_envstr; vsize = pss.ps_nenvstr; if (vsize > ARG_MAX) return (ENOEXEC); size = vsize * sizeof(char *); break; case PROC_AUX: /* * The aux array is just above env array on the stack. Check * that the address is naturally aligned. */ vptr = (vm_offset_t)pss.ps_envstr + (pss.ps_nenvstr + 1) * sizeof(char *); #if __ELF_WORD_SIZE == 64 if (vptr % sizeof(uint64_t) != 0) #else if (vptr % sizeof(uint32_t) != 0) #endif return (ENOEXEC); /* * We count the array size reading the aux vectors from the * stack until AT_NULL vector is returned. So (to keep the code * simple) we read the process stack twice: the first time here * to find the size and the second time when copying the vectors * to the allocated proc_vector. */ for (ptr = vptr, i = 0; i < PROC_AUXV_MAX; i++) { error = proc_read_mem(td, p, ptr, &aux, sizeof(aux)); if (error != 0) return (error); if (aux.a_type == AT_NULL) break; ptr += sizeof(aux); } /* * If the PROC_AUXV_MAX entries are iterated over, and we have * not reached AT_NULL, it is most likely we are reading wrong * data: either the process doesn't have auxv array or data has * been modified. Return the error in this case. */ if (aux.a_type != AT_NULL) return (ENOEXEC); vsize = i + 1; size = vsize * sizeof(aux); break; default: KASSERT(0, ("Wrong proc vector type: %d", type)); return (EINVAL); /* In case we are built without INVARIANTS. */ } proc_vector = malloc(size, M_TEMP, M_WAITOK); if (proc_vector == NULL) return (ENOMEM); error = proc_read_mem(td, p, vptr, proc_vector, size); if (error != 0) { free(proc_vector, M_TEMP); return (error); } *proc_vectorp = proc_vector; *vsizep = vsize; return (0); } #define GET_PS_STRINGS_CHUNK_SZ 256 /* Chunk size (bytes) for ps_strings operations. */ static int get_ps_strings(struct thread *td, struct proc *p, struct sbuf *sb, enum proc_vector_type type) { size_t done, len, nchr, vsize; int error, i; char **proc_vector, *sptr; char pss_string[GET_PS_STRINGS_CHUNK_SZ]; PROC_ASSERT_HELD(p); /* * We are not going to read more than 2 * (PATH_MAX + ARG_MAX) bytes. */ nchr = 2 * (PATH_MAX + ARG_MAX); error = get_proc_vector(td, p, &proc_vector, &vsize, type); if (error != 0) return (error); for (done = 0, i = 0; i < (int)vsize && done < nchr; i++) { /* * The program may have scribbled into its argv array, e.g. to * remove some arguments. If that has happened, break out * before trying to read from NULL. */ if (proc_vector[i] == NULL) break; for (sptr = proc_vector[i]; ; sptr += GET_PS_STRINGS_CHUNK_SZ) { error = proc_read_string(td, p, sptr, pss_string, sizeof(pss_string)); if (error != 0) goto done; len = strnlen(pss_string, GET_PS_STRINGS_CHUNK_SZ); if (done + len >= nchr) len = nchr - done - 1; sbuf_bcat(sb, pss_string, len); if (len != GET_PS_STRINGS_CHUNK_SZ) break; done += GET_PS_STRINGS_CHUNK_SZ; } sbuf_bcat(sb, "", 1); done += len + 1; } done: free(proc_vector, M_TEMP); return (error); } int proc_getargv(struct thread *td, struct proc *p, struct sbuf *sb) { return (get_ps_strings(curthread, p, sb, PROC_ARG)); } int proc_getenvv(struct thread *td, struct proc *p, struct sbuf *sb) { return (get_ps_strings(curthread, p, sb, PROC_ENV)); } int proc_getauxv(struct thread *td, struct proc *p, struct sbuf *sb) { size_t vsize, size; char **auxv; int error; error = get_proc_vector(td, p, &auxv, &vsize, PROC_AUX); if (error == 0) { #ifdef COMPAT_FREEBSD32 if (SV_PROC_FLAG(p, SV_ILP32) != 0) size = vsize * sizeof(Elf32_Auxinfo); else #endif size = vsize * sizeof(Elf_Auxinfo); if (sbuf_bcat(sb, auxv, size) != 0) error = ENOMEM; free(auxv, M_TEMP); } return (error); } /* * This sysctl allows a process to retrieve the argument list or process * title for another process without groping around in the address space * of the other process. It also allow a process to set its own "process * title to a string of its own choice. */ static int sysctl_kern_proc_args(SYSCTL_HANDLER_ARGS) { int *name = (int *)arg1; u_int namelen = arg2; struct pargs *newpa, *pa; struct proc *p; struct sbuf sb; int flags, error = 0, error2; if (namelen != 1) return (EINVAL); flags = PGET_CANSEE; if (req->newptr != NULL) flags |= PGET_ISCURRENT; error = pget((pid_t)name[0], flags, &p); if (error) return (error); pa = p->p_args; if (pa != NULL) { pargs_hold(pa); PROC_UNLOCK(p); error = SYSCTL_OUT(req, pa->ar_args, pa->ar_length); pargs_drop(pa); } else if ((p->p_flag & (P_WEXIT | P_SYSTEM)) == 0) { _PHOLD(p); PROC_UNLOCK(p); sbuf_new_for_sysctl(&sb, NULL, GET_PS_STRINGS_CHUNK_SZ, req); error = proc_getargv(curthread, p, &sb); error2 = sbuf_finish(&sb); PRELE(p); sbuf_delete(&sb); if (error == 0 && error2 != 0) error = error2; } else { PROC_UNLOCK(p); } if (error != 0 || req->newptr == NULL) return (error); if (req->newlen + sizeof(struct pargs) > ps_arg_cache_limit) return (ENOMEM); newpa = pargs_alloc(req->newlen); error = SYSCTL_IN(req, newpa->ar_args, req->newlen); if (error != 0) { pargs_free(newpa); return (error); } PROC_LOCK(p); pa = p->p_args; p->p_args = newpa; PROC_UNLOCK(p); pargs_drop(pa); return (0); } /* * This sysctl allows a process to retrieve environment of another process. */ static int sysctl_kern_proc_env(SYSCTL_HANDLER_ARGS) { int *name = (int *)arg1; u_int namelen = arg2; struct proc *p; struct sbuf sb; int error, error2; if (namelen != 1) return (EINVAL); error = pget((pid_t)name[0], PGET_WANTREAD, &p); if (error != 0) return (error); if ((p->p_flag & P_SYSTEM) != 0) { PRELE(p); return (0); } sbuf_new_for_sysctl(&sb, NULL, GET_PS_STRINGS_CHUNK_SZ, req); error = proc_getenvv(curthread, p, &sb); error2 = sbuf_finish(&sb); PRELE(p); sbuf_delete(&sb); return (error != 0 ? error : error2); } /* * This sysctl allows a process to retrieve ELF auxiliary vector of * another process. */ static int sysctl_kern_proc_auxv(SYSCTL_HANDLER_ARGS) { int *name = (int *)arg1; u_int namelen = arg2; struct proc *p; struct sbuf sb; int error, error2; if (namelen != 1) return (EINVAL); error = pget((pid_t)name[0], PGET_WANTREAD, &p); if (error != 0) return (error); if ((p->p_flag & P_SYSTEM) != 0) { PRELE(p); return (0); } sbuf_new_for_sysctl(&sb, NULL, GET_PS_STRINGS_CHUNK_SZ, req); error = proc_getauxv(curthread, p, &sb); error2 = sbuf_finish(&sb); PRELE(p); sbuf_delete(&sb); return (error != 0 ? error : error2); } /* * This sysctl allows a process to retrieve the path of the executable for * itself or another process. */ static int sysctl_kern_proc_pathname(SYSCTL_HANDLER_ARGS) { pid_t *pidp = (pid_t *)arg1; unsigned int arglen = arg2; struct proc *p; struct vnode *vp; char *retbuf, *freebuf; int error; if (arglen != 1) return (EINVAL); if (*pidp == -1) { /* -1 means this process */ p = req->td->td_proc; } else { error = pget(*pidp, PGET_CANSEE, &p); if (error != 0) return (error); } vp = p->p_textvp; if (vp == NULL) { if (*pidp != -1) PROC_UNLOCK(p); return (0); } vref(vp); if (*pidp != -1) PROC_UNLOCK(p); error = vn_fullpath(req->td, vp, &retbuf, &freebuf); vrele(vp); if (error) return (error); error = SYSCTL_OUT(req, retbuf, strlen(retbuf) + 1); free(freebuf, M_TEMP); return (error); } static int sysctl_kern_proc_sv_name(SYSCTL_HANDLER_ARGS) { struct proc *p; char *sv_name; int *name; int namelen; int error; namelen = arg2; if (namelen != 1) return (EINVAL); name = (int *)arg1; error = pget((pid_t)name[0], PGET_CANSEE, &p); if (error != 0) return (error); sv_name = p->p_sysent->sv_name; PROC_UNLOCK(p); return (sysctl_handle_string(oidp, sv_name, 0, req)); } #ifdef KINFO_OVMENTRY_SIZE CTASSERT(sizeof(struct kinfo_ovmentry) == KINFO_OVMENTRY_SIZE); #endif #ifdef COMPAT_FREEBSD7 static int sysctl_kern_proc_ovmmap(SYSCTL_HANDLER_ARGS) { vm_map_entry_t entry, tmp_entry; unsigned int last_timestamp; char *fullpath, *freepath; struct kinfo_ovmentry *kve; struct vattr va; struct ucred *cred; int error, *name; struct vnode *vp; struct proc *p; vm_map_t map; struct vmspace *vm; name = (int *)arg1; error = pget((pid_t)name[0], PGET_WANTREAD, &p); if (error != 0) return (error); vm = vmspace_acquire_ref(p); if (vm == NULL) { PRELE(p); return (ESRCH); } kve = malloc(sizeof(*kve), M_TEMP, M_WAITOK); map = &vm->vm_map; vm_map_lock_read(map); for (entry = map->header.next; entry != &map->header; entry = entry->next) { vm_object_t obj, tobj, lobj; vm_offset_t addr; if (entry->eflags & MAP_ENTRY_IS_SUB_MAP) continue; bzero(kve, sizeof(*kve)); kve->kve_structsize = sizeof(*kve); kve->kve_private_resident = 0; obj = entry->object.vm_object; if (obj != NULL) { VM_OBJECT_RLOCK(obj); if (obj->shadow_count == 1) kve->kve_private_resident = obj->resident_page_count; } kve->kve_resident = 0; addr = entry->start; while (addr < entry->end) { if (pmap_extract(map->pmap, addr)) kve->kve_resident++; addr += PAGE_SIZE; } for (lobj = tobj = obj; tobj; tobj = tobj->backing_object) { if (tobj != obj) VM_OBJECT_RLOCK(tobj); if (lobj != obj) VM_OBJECT_RUNLOCK(lobj); lobj = tobj; } kve->kve_start = (void*)entry->start; kve->kve_end = (void*)entry->end; kve->kve_offset = (off_t)entry->offset; if (entry->protection & VM_PROT_READ) kve->kve_protection |= KVME_PROT_READ; if (entry->protection & VM_PROT_WRITE) kve->kve_protection |= KVME_PROT_WRITE; if (entry->protection & VM_PROT_EXECUTE) kve->kve_protection |= KVME_PROT_EXEC; if (entry->eflags & MAP_ENTRY_COW) kve->kve_flags |= KVME_FLAG_COW; if (entry->eflags & MAP_ENTRY_NEEDS_COPY) kve->kve_flags |= KVME_FLAG_NEEDS_COPY; if (entry->eflags & MAP_ENTRY_NOCOREDUMP) kve->kve_flags |= KVME_FLAG_NOCOREDUMP; last_timestamp = map->timestamp; vm_map_unlock_read(map); kve->kve_fileid = 0; kve->kve_fsid = 0; freepath = NULL; fullpath = ""; if (lobj) { vp = NULL; switch (lobj->type) { case OBJT_DEFAULT: kve->kve_type = KVME_TYPE_DEFAULT; break; case OBJT_VNODE: kve->kve_type = KVME_TYPE_VNODE; vp = lobj->handle; vref(vp); break; case OBJT_SWAP: kve->kve_type = KVME_TYPE_SWAP; break; case OBJT_DEVICE: kve->kve_type = KVME_TYPE_DEVICE; break; case OBJT_PHYS: kve->kve_type = KVME_TYPE_PHYS; break; case OBJT_DEAD: kve->kve_type = KVME_TYPE_DEAD; break; case OBJT_SG: kve->kve_type = KVME_TYPE_SG; break; default: kve->kve_type = KVME_TYPE_UNKNOWN; break; } if (lobj != obj) VM_OBJECT_RUNLOCK(lobj); kve->kve_ref_count = obj->ref_count; kve->kve_shadow_count = obj->shadow_count; VM_OBJECT_RUNLOCK(obj); if (vp != NULL) { vn_fullpath(curthread, vp, &fullpath, &freepath); cred = curthread->td_ucred; vn_lock(vp, LK_SHARED | LK_RETRY); if (VOP_GETATTR(vp, &va, cred) == 0) { kve->kve_fileid = va.va_fileid; kve->kve_fsid = va.va_fsid; } vput(vp); } } else { kve->kve_type = KVME_TYPE_NONE; kve->kve_ref_count = 0; kve->kve_shadow_count = 0; } strlcpy(kve->kve_path, fullpath, sizeof(kve->kve_path)); if (freepath != NULL) free(freepath, M_TEMP); error = SYSCTL_OUT(req, kve, sizeof(*kve)); vm_map_lock_read(map); if (error) break; if (last_timestamp != map->timestamp) { vm_map_lookup_entry(map, addr - 1, &tmp_entry); entry = tmp_entry; } } vm_map_unlock_read(map); vmspace_free(vm); PRELE(p); free(kve, M_TEMP); return (error); } #endif /* COMPAT_FREEBSD7 */ #ifdef KINFO_VMENTRY_SIZE CTASSERT(sizeof(struct kinfo_vmentry) == KINFO_VMENTRY_SIZE); #endif static void kern_proc_vmmap_resident(vm_map_t map, vm_map_entry_t entry, struct kinfo_vmentry *kve) { vm_object_t obj, tobj; vm_page_t m, m_adv; vm_offset_t addr; vm_paddr_t locked_pa; vm_pindex_t pi, pi_adv, pindex; locked_pa = 0; obj = entry->object.vm_object; addr = entry->start; m_adv = NULL; pi = OFF_TO_IDX(entry->offset); for (; addr < entry->end; addr += IDX_TO_OFF(pi_adv), pi += pi_adv) { if (m_adv != NULL) { m = m_adv; } else { pi_adv = OFF_TO_IDX(entry->end - addr); pindex = pi; for (tobj = obj;; tobj = tobj->backing_object) { m = vm_page_find_least(tobj, pindex); if (m != NULL) { if (m->pindex == pindex) break; if (pi_adv > m->pindex - pindex) { pi_adv = m->pindex - pindex; m_adv = m; } } if (tobj->backing_object == NULL) goto next; pindex += OFF_TO_IDX(tobj-> backing_object_offset); } } m_adv = NULL; if (m->psind != 0 && addr + pagesizes[1] <= entry->end && (addr & (pagesizes[1] - 1)) == 0 && (pmap_mincore(map->pmap, addr, &locked_pa) & MINCORE_SUPER) != 0) { kve->kve_flags |= KVME_FLAG_SUPER; pi_adv = OFF_TO_IDX(pagesizes[1]); } else { /* * We do not test the found page on validity. * Either the page is busy and being paged in, * or it was invalidated. The first case * should be counted as resident, the second * is not so clear; we do account both. */ pi_adv = 1; } kve->kve_resident += pi_adv; next:; } PA_UNLOCK_COND(locked_pa); } /* * Must be called with the process locked and will return unlocked. */ int kern_proc_vmmap_out(struct proc *p, struct sbuf *sb) { vm_map_entry_t entry, tmp_entry; struct vattr va; vm_map_t map; vm_object_t obj, tobj, lobj; char *fullpath, *freepath; struct kinfo_vmentry *kve; struct ucred *cred; struct vnode *vp; struct vmspace *vm; vm_offset_t addr; unsigned int last_timestamp; int error; PROC_LOCK_ASSERT(p, MA_OWNED); _PHOLD(p); PROC_UNLOCK(p); vm = vmspace_acquire_ref(p); if (vm == NULL) { PRELE(p); return (ESRCH); } kve = malloc(sizeof(*kve), M_TEMP, M_WAITOK); error = 0; map = &vm->vm_map; vm_map_lock_read(map); for (entry = map->header.next; entry != &map->header; entry = entry->next) { if (entry->eflags & MAP_ENTRY_IS_SUB_MAP) continue; addr = entry->end; bzero(kve, sizeof(*kve)); obj = entry->object.vm_object; if (obj != NULL) { for (tobj = obj; tobj != NULL; tobj = tobj->backing_object) { VM_OBJECT_RLOCK(tobj); lobj = tobj; } if (obj->backing_object == NULL) kve->kve_private_resident = obj->resident_page_count; if (!vmmap_skip_res_cnt) kern_proc_vmmap_resident(map, entry, kve); for (tobj = obj; tobj != NULL; tobj = tobj->backing_object) { if (tobj != obj && tobj != lobj) VM_OBJECT_RUNLOCK(tobj); } } else { lobj = NULL; } kve->kve_start = entry->start; kve->kve_end = entry->end; kve->kve_offset = entry->offset; if (entry->protection & VM_PROT_READ) kve->kve_protection |= KVME_PROT_READ; if (entry->protection & VM_PROT_WRITE) kve->kve_protection |= KVME_PROT_WRITE; if (entry->protection & VM_PROT_EXECUTE) kve->kve_protection |= KVME_PROT_EXEC; if (entry->eflags & MAP_ENTRY_COW) kve->kve_flags |= KVME_FLAG_COW; if (entry->eflags & MAP_ENTRY_NEEDS_COPY) kve->kve_flags |= KVME_FLAG_NEEDS_COPY; if (entry->eflags & MAP_ENTRY_NOCOREDUMP) kve->kve_flags |= KVME_FLAG_NOCOREDUMP; if (entry->eflags & MAP_ENTRY_GROWS_UP) kve->kve_flags |= KVME_FLAG_GROWS_UP; if (entry->eflags & MAP_ENTRY_GROWS_DOWN) kve->kve_flags |= KVME_FLAG_GROWS_DOWN; last_timestamp = map->timestamp; vm_map_unlock_read(map); freepath = NULL; fullpath = ""; if (lobj != NULL) { vp = NULL; switch (lobj->type) { case OBJT_DEFAULT: kve->kve_type = KVME_TYPE_DEFAULT; break; case OBJT_VNODE: kve->kve_type = KVME_TYPE_VNODE; vp = lobj->handle; vref(vp); break; case OBJT_SWAP: kve->kve_type = KVME_TYPE_SWAP; break; case OBJT_DEVICE: kve->kve_type = KVME_TYPE_DEVICE; break; case OBJT_PHYS: kve->kve_type = KVME_TYPE_PHYS; break; case OBJT_DEAD: kve->kve_type = KVME_TYPE_DEAD; break; case OBJT_SG: kve->kve_type = KVME_TYPE_SG; break; case OBJT_MGTDEVICE: kve->kve_type = KVME_TYPE_MGTDEVICE; break; default: kve->kve_type = KVME_TYPE_UNKNOWN; break; } if (lobj != obj) VM_OBJECT_RUNLOCK(lobj); kve->kve_ref_count = obj->ref_count; kve->kve_shadow_count = obj->shadow_count; VM_OBJECT_RUNLOCK(obj); if (vp != NULL) { vn_fullpath(curthread, vp, &fullpath, &freepath); kve->kve_vn_type = vntype_to_kinfo(vp->v_type); cred = curthread->td_ucred; vn_lock(vp, LK_SHARED | LK_RETRY); if (VOP_GETATTR(vp, &va, cred) == 0) { kve->kve_vn_fileid = va.va_fileid; kve->kve_vn_fsid = va.va_fsid; kve->kve_vn_mode = MAKEIMODE(va.va_type, va.va_mode); kve->kve_vn_size = va.va_size; kve->kve_vn_rdev = va.va_rdev; kve->kve_status = KF_ATTR_VALID; } vput(vp); } } else { kve->kve_type = KVME_TYPE_NONE; kve->kve_ref_count = 0; kve->kve_shadow_count = 0; } strlcpy(kve->kve_path, fullpath, sizeof(kve->kve_path)); if (freepath != NULL) free(freepath, M_TEMP); /* Pack record size down */ kve->kve_structsize = offsetof(struct kinfo_vmentry, kve_path) + strlen(kve->kve_path) + 1; kve->kve_structsize = roundup(kve->kve_structsize, sizeof(uint64_t)); if (sbuf_bcat(sb, kve, kve->kve_structsize) != 0) error = ENOMEM; vm_map_lock_read(map); if (error != 0) break; if (last_timestamp != map->timestamp) { vm_map_lookup_entry(map, addr - 1, &tmp_entry); entry = tmp_entry; } } vm_map_unlock_read(map); vmspace_free(vm); PRELE(p); free(kve, M_TEMP); return (error); } static int sysctl_kern_proc_vmmap(SYSCTL_HANDLER_ARGS) { struct proc *p; struct sbuf sb; int error, error2, *name; name = (int *)arg1; sbuf_new_for_sysctl(&sb, NULL, sizeof(struct kinfo_vmentry), req); error = pget((pid_t)name[0], PGET_CANDEBUG | PGET_NOTWEXIT, &p); if (error != 0) { sbuf_delete(&sb); return (error); } error = kern_proc_vmmap_out(p, &sb); error2 = sbuf_finish(&sb); sbuf_delete(&sb); return (error != 0 ? error : error2); } #if defined(STACK) || defined(DDB) static int sysctl_kern_proc_kstack(SYSCTL_HANDLER_ARGS) { struct kinfo_kstack *kkstp; int error, i, *name, numthreads; lwpid_t *lwpidarray; struct thread *td; struct stack *st; struct sbuf sb; struct proc *p; name = (int *)arg1; error = pget((pid_t)name[0], PGET_NOTINEXEC | PGET_WANTREAD, &p); if (error != 0) return (error); kkstp = malloc(sizeof(*kkstp), M_TEMP, M_WAITOK); st = stack_create(); lwpidarray = NULL; numthreads = 0; PROC_LOCK(p); repeat: if (numthreads < p->p_numthreads) { if (lwpidarray != NULL) { free(lwpidarray, M_TEMP); lwpidarray = NULL; } numthreads = p->p_numthreads; PROC_UNLOCK(p); lwpidarray = malloc(sizeof(*lwpidarray) * numthreads, M_TEMP, M_WAITOK | M_ZERO); PROC_LOCK(p); goto repeat; } i = 0; /* * XXXRW: During the below loop, execve(2) and countless other sorts * of changes could have taken place. Should we check to see if the * vmspace has been replaced, or the like, in order to prevent * giving a snapshot that spans, say, execve(2), with some threads * before and some after? Among other things, the credentials could * have changed, in which case the right to extract debug info might * no longer be assured. */ FOREACH_THREAD_IN_PROC(p, td) { KASSERT(i < numthreads, ("sysctl_kern_proc_kstack: numthreads")); lwpidarray[i] = td->td_tid; i++; } numthreads = i; for (i = 0; i < numthreads; i++) { td = thread_find(p, lwpidarray[i]); if (td == NULL) { continue; } bzero(kkstp, sizeof(*kkstp)); (void)sbuf_new(&sb, kkstp->kkst_trace, sizeof(kkstp->kkst_trace), SBUF_FIXEDLEN); thread_lock(td); kkstp->kkst_tid = td->td_tid; if (TD_IS_SWAPPED(td)) kkstp->kkst_state = KKST_STATE_SWAPPED; else if (TD_IS_RUNNING(td)) kkstp->kkst_state = KKST_STATE_RUNNING; else { kkstp->kkst_state = KKST_STATE_STACKOK; stack_save_td(st, td); } thread_unlock(td); PROC_UNLOCK(p); stack_sbuf_print(&sb, st); sbuf_finish(&sb); sbuf_delete(&sb); error = SYSCTL_OUT(req, kkstp, sizeof(*kkstp)); PROC_LOCK(p); if (error) break; } _PRELE(p); PROC_UNLOCK(p); if (lwpidarray != NULL) free(lwpidarray, M_TEMP); stack_destroy(st); free(kkstp, M_TEMP); return (error); } #endif /* * This sysctl allows a process to retrieve the full list of groups from * itself or another process. */ static int sysctl_kern_proc_groups(SYSCTL_HANDLER_ARGS) { pid_t *pidp = (pid_t *)arg1; unsigned int arglen = arg2; struct proc *p; struct ucred *cred; int error; if (arglen != 1) return (EINVAL); if (*pidp == -1) { /* -1 means this process */ p = req->td->td_proc; PROC_LOCK(p); } else { error = pget(*pidp, PGET_CANSEE, &p); if (error != 0) return (error); } cred = crhold(p->p_ucred); PROC_UNLOCK(p); error = SYSCTL_OUT(req, cred->cr_groups, cred->cr_ngroups * sizeof(gid_t)); crfree(cred); return (error); } /* * This sysctl allows a process to retrieve or/and set the resource limit for * another process. */ static int sysctl_kern_proc_rlimit(SYSCTL_HANDLER_ARGS) { int *name = (int *)arg1; u_int namelen = arg2; struct rlimit rlim; struct proc *p; u_int which; int flags, error; if (namelen != 2) return (EINVAL); which = (u_int)name[1]; if (which >= RLIM_NLIMITS) return (EINVAL); if (req->newptr != NULL && req->newlen != sizeof(rlim)) return (EINVAL); flags = PGET_HOLD | PGET_NOTWEXIT; if (req->newptr != NULL) flags |= PGET_CANDEBUG; else flags |= PGET_CANSEE; error = pget((pid_t)name[0], flags, &p); if (error != 0) return (error); /* * Retrieve limit. */ if (req->oldptr != NULL) { PROC_LOCK(p); lim_rlimit(p, which, &rlim); PROC_UNLOCK(p); } error = SYSCTL_OUT(req, &rlim, sizeof(rlim)); if (error != 0) goto errout; /* * Set limit. */ if (req->newptr != NULL) { error = SYSCTL_IN(req, &rlim, sizeof(rlim)); if (error == 0) error = kern_proc_setrlimit(curthread, p, which, &rlim); } errout: PRELE(p); return (error); } /* * This sysctl allows a process to retrieve ps_strings structure location of * another process. */ static int sysctl_kern_proc_ps_strings(SYSCTL_HANDLER_ARGS) { int *name = (int *)arg1; u_int namelen = arg2; struct proc *p; vm_offset_t ps_strings; int error; #ifdef COMPAT_FREEBSD32 uint32_t ps_strings32; #endif if (namelen != 1) return (EINVAL); error = pget((pid_t)name[0], PGET_CANDEBUG, &p); if (error != 0) return (error); #ifdef COMPAT_FREEBSD32 if ((req->flags & SCTL_MASK32) != 0) { /* * We return 0 if the 32 bit emulation request is for a 64 bit * process. */ ps_strings32 = SV_PROC_FLAG(p, SV_ILP32) != 0 ? PTROUT(p->p_sysent->sv_psstrings) : 0; PROC_UNLOCK(p); error = SYSCTL_OUT(req, &ps_strings32, sizeof(ps_strings32)); return (error); } #endif ps_strings = p->p_sysent->sv_psstrings; PROC_UNLOCK(p); error = SYSCTL_OUT(req, &ps_strings, sizeof(ps_strings)); return (error); } /* * This sysctl allows a process to retrieve umask of another process. */ static int sysctl_kern_proc_umask(SYSCTL_HANDLER_ARGS) { int *name = (int *)arg1; u_int namelen = arg2; struct proc *p; int error; u_short fd_cmask; if (namelen != 1) return (EINVAL); error = pget((pid_t)name[0], PGET_WANTREAD, &p); if (error != 0) return (error); FILEDESC_SLOCK(p->p_fd); fd_cmask = p->p_fd->fd_cmask; FILEDESC_SUNLOCK(p->p_fd); PRELE(p); error = SYSCTL_OUT(req, &fd_cmask, sizeof(fd_cmask)); return (error); } /* * This sysctl allows a process to set and retrieve binary osreldate of * another process. */ static int sysctl_kern_proc_osrel(SYSCTL_HANDLER_ARGS) { int *name = (int *)arg1; u_int namelen = arg2; struct proc *p; int flags, error, osrel; if (namelen != 1) return (EINVAL); if (req->newptr != NULL && req->newlen != sizeof(osrel)) return (EINVAL); flags = PGET_HOLD | PGET_NOTWEXIT; if (req->newptr != NULL) flags |= PGET_CANDEBUG; else flags |= PGET_CANSEE; error = pget((pid_t)name[0], flags, &p); if (error != 0) return (error); error = SYSCTL_OUT(req, &p->p_osrel, sizeof(p->p_osrel)); if (error != 0) goto errout; if (req->newptr != NULL) { error = SYSCTL_IN(req, &osrel, sizeof(osrel)); if (error != 0) goto errout; if (osrel < 0) { error = EINVAL; goto errout; } p->p_osrel = osrel; } errout: PRELE(p); return (error); } static int sysctl_kern_proc_sigtramp(SYSCTL_HANDLER_ARGS) { int *name = (int *)arg1; u_int namelen = arg2; struct proc *p; struct kinfo_sigtramp kst; const struct sysentvec *sv; int error; #ifdef COMPAT_FREEBSD32 struct kinfo_sigtramp32 kst32; #endif if (namelen != 1) return (EINVAL); error = pget((pid_t)name[0], PGET_CANDEBUG, &p); if (error != 0) return (error); sv = p->p_sysent; #ifdef COMPAT_FREEBSD32 if ((req->flags & SCTL_MASK32) != 0) { bzero(&kst32, sizeof(kst32)); if (SV_PROC_FLAG(p, SV_ILP32)) { if (sv->sv_sigcode_base != 0) { kst32.ksigtramp_start = sv->sv_sigcode_base; kst32.ksigtramp_end = sv->sv_sigcode_base + *sv->sv_szsigcode; } else { kst32.ksigtramp_start = sv->sv_psstrings - *sv->sv_szsigcode; kst32.ksigtramp_end = sv->sv_psstrings; } } PROC_UNLOCK(p); error = SYSCTL_OUT(req, &kst32, sizeof(kst32)); return (error); } #endif bzero(&kst, sizeof(kst)); if (sv->sv_sigcode_base != 0) { kst.ksigtramp_start = (char *)sv->sv_sigcode_base; kst.ksigtramp_end = (char *)sv->sv_sigcode_base + *sv->sv_szsigcode; } else { kst.ksigtramp_start = (char *)sv->sv_psstrings - *sv->sv_szsigcode; kst.ksigtramp_end = (char *)sv->sv_psstrings; } PROC_UNLOCK(p); error = SYSCTL_OUT(req, &kst, sizeof(kst)); return (error); } SYSCTL_NODE(_kern, KERN_PROC, proc, CTLFLAG_RD, 0, "Process table"); SYSCTL_PROC(_kern_proc, KERN_PROC_ALL, all, CTLFLAG_RD|CTLTYPE_STRUCT| CTLFLAG_MPSAFE, 0, 0, sysctl_kern_proc, "S,proc", "Return entire process table"); static SYSCTL_NODE(_kern_proc, KERN_PROC_GID, gid, CTLFLAG_RD | CTLFLAG_MPSAFE, sysctl_kern_proc, "Process table"); static SYSCTL_NODE(_kern_proc, KERN_PROC_PGRP, pgrp, CTLFLAG_RD | CTLFLAG_MPSAFE, sysctl_kern_proc, "Process table"); static SYSCTL_NODE(_kern_proc, KERN_PROC_RGID, rgid, CTLFLAG_RD | CTLFLAG_MPSAFE, sysctl_kern_proc, "Process table"); static SYSCTL_NODE(_kern_proc, KERN_PROC_SESSION, sid, CTLFLAG_RD | CTLFLAG_MPSAFE, sysctl_kern_proc, "Process table"); static SYSCTL_NODE(_kern_proc, KERN_PROC_TTY, tty, CTLFLAG_RD | CTLFLAG_MPSAFE, sysctl_kern_proc, "Process table"); static SYSCTL_NODE(_kern_proc, KERN_PROC_UID, uid, CTLFLAG_RD | CTLFLAG_MPSAFE, sysctl_kern_proc, "Process table"); static SYSCTL_NODE(_kern_proc, KERN_PROC_RUID, ruid, CTLFLAG_RD | CTLFLAG_MPSAFE, sysctl_kern_proc, "Process table"); static SYSCTL_NODE(_kern_proc, KERN_PROC_PID, pid, CTLFLAG_RD | CTLFLAG_MPSAFE, sysctl_kern_proc, "Process table"); static SYSCTL_NODE(_kern_proc, KERN_PROC_PROC, proc, CTLFLAG_RD | CTLFLAG_MPSAFE, sysctl_kern_proc, "Return process table, no threads"); static SYSCTL_NODE(_kern_proc, KERN_PROC_ARGS, args, CTLFLAG_RW | CTLFLAG_ANYBODY | CTLFLAG_MPSAFE, sysctl_kern_proc_args, "Process argument list"); static SYSCTL_NODE(_kern_proc, KERN_PROC_ENV, env, CTLFLAG_RD | CTLFLAG_MPSAFE, sysctl_kern_proc_env, "Process environment"); static SYSCTL_NODE(_kern_proc, KERN_PROC_AUXV, auxv, CTLFLAG_RD | CTLFLAG_MPSAFE, sysctl_kern_proc_auxv, "Process ELF auxiliary vector"); static SYSCTL_NODE(_kern_proc, KERN_PROC_PATHNAME, pathname, CTLFLAG_RD | CTLFLAG_MPSAFE, sysctl_kern_proc_pathname, "Process executable path"); static SYSCTL_NODE(_kern_proc, KERN_PROC_SV_NAME, sv_name, CTLFLAG_RD | CTLFLAG_MPSAFE, sysctl_kern_proc_sv_name, "Process syscall vector name (ABI type)"); static SYSCTL_NODE(_kern_proc, (KERN_PROC_GID | KERN_PROC_INC_THREAD), gid_td, CTLFLAG_RD | CTLFLAG_MPSAFE, sysctl_kern_proc, "Process table"); static SYSCTL_NODE(_kern_proc, (KERN_PROC_PGRP | KERN_PROC_INC_THREAD), pgrp_td, CTLFLAG_RD | CTLFLAG_MPSAFE, sysctl_kern_proc, "Process table"); static SYSCTL_NODE(_kern_proc, (KERN_PROC_RGID | KERN_PROC_INC_THREAD), rgid_td, CTLFLAG_RD | CTLFLAG_MPSAFE, sysctl_kern_proc, "Process table"); static SYSCTL_NODE(_kern_proc, (KERN_PROC_SESSION | KERN_PROC_INC_THREAD), sid_td, CTLFLAG_RD | CTLFLAG_MPSAFE, sysctl_kern_proc, "Process table"); static SYSCTL_NODE(_kern_proc, (KERN_PROC_TTY | KERN_PROC_INC_THREAD), tty_td, CTLFLAG_RD | CTLFLAG_MPSAFE, sysctl_kern_proc, "Process table"); static SYSCTL_NODE(_kern_proc, (KERN_PROC_UID | KERN_PROC_INC_THREAD), uid_td, CTLFLAG_RD | CTLFLAG_MPSAFE, sysctl_kern_proc, "Process table"); static SYSCTL_NODE(_kern_proc, (KERN_PROC_RUID | KERN_PROC_INC_THREAD), ruid_td, CTLFLAG_RD | CTLFLAG_MPSAFE, sysctl_kern_proc, "Process table"); static SYSCTL_NODE(_kern_proc, (KERN_PROC_PID | KERN_PROC_INC_THREAD), pid_td, CTLFLAG_RD | CTLFLAG_MPSAFE, sysctl_kern_proc, "Process table"); static SYSCTL_NODE(_kern_proc, (KERN_PROC_PROC | KERN_PROC_INC_THREAD), proc_td, CTLFLAG_RD | CTLFLAG_MPSAFE, sysctl_kern_proc, "Return process table, no threads"); #ifdef COMPAT_FREEBSD7 static SYSCTL_NODE(_kern_proc, KERN_PROC_OVMMAP, ovmmap, CTLFLAG_RD | CTLFLAG_MPSAFE, sysctl_kern_proc_ovmmap, "Old Process vm map entries"); #endif static SYSCTL_NODE(_kern_proc, KERN_PROC_VMMAP, vmmap, CTLFLAG_RD | CTLFLAG_MPSAFE, sysctl_kern_proc_vmmap, "Process vm map entries"); #if defined(STACK) || defined(DDB) static SYSCTL_NODE(_kern_proc, KERN_PROC_KSTACK, kstack, CTLFLAG_RD | CTLFLAG_MPSAFE, sysctl_kern_proc_kstack, "Process kernel stacks"); #endif static SYSCTL_NODE(_kern_proc, KERN_PROC_GROUPS, groups, CTLFLAG_RD | CTLFLAG_MPSAFE, sysctl_kern_proc_groups, "Process groups"); static SYSCTL_NODE(_kern_proc, KERN_PROC_RLIMIT, rlimit, CTLFLAG_RW | CTLFLAG_ANYBODY | CTLFLAG_MPSAFE, sysctl_kern_proc_rlimit, "Process resource limits"); static SYSCTL_NODE(_kern_proc, KERN_PROC_PS_STRINGS, ps_strings, CTLFLAG_RD | CTLFLAG_MPSAFE, sysctl_kern_proc_ps_strings, "Process ps_strings location"); static SYSCTL_NODE(_kern_proc, KERN_PROC_UMASK, umask, CTLFLAG_RD | CTLFLAG_MPSAFE, sysctl_kern_proc_umask, "Process umask"); static SYSCTL_NODE(_kern_proc, KERN_PROC_OSREL, osrel, CTLFLAG_RW | CTLFLAG_ANYBODY | CTLFLAG_MPSAFE, sysctl_kern_proc_osrel, "Process binary osreldate"); static SYSCTL_NODE(_kern_proc, KERN_PROC_SIGTRAMP, sigtramp, CTLFLAG_RD | CTLFLAG_MPSAFE, sysctl_kern_proc_sigtramp, "Process signal trampoline location"); + +int allproc_gen; + +void +stop_all_proc(void) +{ + struct proc *cp, *p; + int r, gen; + bool restart, seen_stopped, seen_exiting, stopped_some; + + cp = curproc; + /* + * stop_all_proc() assumes that all process which have + * usermode must be stopped, except current process, for + * obvious reasons. Since other threads in the process + * establishing global stop could unstop something, disable + * calls from multithreaded processes as precaution. The + * service must not be user-callable anyway. + */ + KASSERT((cp->p_flag & P_HADTHREADS) == 0 || + (cp->p_flag & P_KTHREAD) != 0, ("mt stop_all_proc")); + +allproc_loop: + sx_xlock(&allproc_lock); + gen = allproc_gen; + seen_exiting = seen_stopped = stopped_some = restart = false; + LIST_REMOVE(cp, p_list); + LIST_INSERT_HEAD(&allproc, cp, p_list); + for (;;) { + p = LIST_NEXT(cp, p_list); + if (p == NULL) + break; + LIST_REMOVE(cp, p_list); + LIST_INSERT_AFTER(p, cp, p_list); + PROC_LOCK(p); + if ((p->p_flag & (P_KTHREAD | P_SYSTEM | + P_TOTAL_STOP)) != 0) { + PROC_UNLOCK(p); + continue; + } + if ((p->p_flag & P_WEXIT) != 0) { + seen_exiting = true; + PROC_UNLOCK(p); + continue; + } + if (P_SHOULDSTOP(p) == P_STOPPED_SINGLE) { + /* + * Stopped processes are tolerated when there + * are no other processes which might continue + * them. P_STOPPED_SINGLE but not + * P_TOTAL_STOP process still has at least one + * thread running. + */ + seen_stopped = true; + PROC_UNLOCK(p); + continue; + } + _PHOLD(p); + sx_xunlock(&allproc_lock); + r = thread_single(p, SINGLE_ALLPROC); + if (r != 0) + restart = true; + else + stopped_some = true; + _PRELE(p); + PROC_UNLOCK(p); + sx_xlock(&allproc_lock); + } + /* Catch forked children we did not see in iteration. */ + if (gen != allproc_gen) + restart = true; + sx_xunlock(&allproc_lock); + if (restart || stopped_some || seen_exiting || seen_stopped) { + kern_yield(PRI_USER); + goto allproc_loop; + } +} + +void +resume_all_proc(void) +{ + struct proc *cp, *p; + + cp = curproc; + sx_xlock(&allproc_lock); + LIST_REMOVE(cp, p_list); + LIST_INSERT_HEAD(&allproc, cp, p_list); + for (;;) { + p = LIST_NEXT(cp, p_list); + if (p == NULL) + break; + LIST_REMOVE(cp, p_list); + LIST_INSERT_AFTER(p, cp, p_list); + PROC_LOCK(p); + if ((p->p_flag & P_TOTAL_STOP) != 0) { + sx_xunlock(&allproc_lock); + _PHOLD(p); + thread_single_end(p, SINGLE_ALLPROC); + _PRELE(p); + PROC_UNLOCK(p); + sx_xlock(&allproc_lock); + } else { + PROC_UNLOCK(p); + } + } + sx_xunlock(&allproc_lock); +} + +#define TOTAL_STOP_DEBUG 1 +#ifdef TOTAL_STOP_DEBUG +volatile static int ap_resume; +#include + +static int +sysctl_debug_stop_all_proc(SYSCTL_HANDLER_ARGS) +{ + int error, val; + + val = 0; + ap_resume = 0; + error = sysctl_handle_int(oidp, &val, 0, req); + if (error != 0 || req->newptr == NULL) + return (error); + if (val != 0) { + stop_all_proc(); + syncer_suspend(); + while (ap_resume == 0) + ; + syncer_resume(); + resume_all_proc(); + } + return (0); +} + +SYSCTL_PROC(_debug, OID_AUTO, stop_all_proc, CTLTYPE_INT | CTLFLAG_RW | + CTLFLAG_MPSAFE, (void *)&ap_resume, 0, sysctl_debug_stop_all_proc, "I", + ""); +#endif Index: projects/clang350-import/sys/kern/kern_sig.c =================================================================== --- projects/clang350-import/sys/kern/kern_sig.c (revision 275748) +++ projects/clang350-import/sys/kern/kern_sig.c (revision 275749) @@ -1,3477 +1,3477 @@ /*- * Copyright (c) 1982, 1986, 1989, 1991, 1993 * The Regents of the University of California. All rights reserved. * (c) UNIX System Laboratories, Inc. * All or some portions of this file are derived from material licensed * to the University of California by American Telephone and Telegraph * Co. or Unix System Laboratories, Inc. and are reproduced herein with * the permission of UNIX System Laboratories, Inc. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 4. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * @(#)kern_sig.c 8.7 (Berkeley) 4/18/94 */ #include __FBSDID("$FreeBSD$"); #include "opt_compat.h" #include "opt_ktrace.h" #include "opt_core.h" #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #define ONSIG 32 /* NSIG for osig* syscalls. XXX. */ SDT_PROVIDER_DECLARE(proc); SDT_PROBE_DEFINE3(proc, kernel, , signal__send, "struct thread *", "struct proc *", "int"); SDT_PROBE_DEFINE2(proc, kernel, , signal__clear, "int", "ksiginfo_t *"); SDT_PROBE_DEFINE3(proc, kernel, , signal__discard, "struct thread *", "struct proc *", "int"); static int coredump(struct thread *); static int killpg1(struct thread *td, int sig, int pgid, int all, ksiginfo_t *ksi); static int issignal(struct thread *td); static int sigprop(int sig); static void tdsigwakeup(struct thread *, int, sig_t, int); static void sig_suspend_threads(struct thread *, struct proc *, int); static int filt_sigattach(struct knote *kn); static void filt_sigdetach(struct knote *kn); static int filt_signal(struct knote *kn, long hint); static struct thread *sigtd(struct proc *p, int sig, int prop); static void sigqueue_start(void); static uma_zone_t ksiginfo_zone = NULL; struct filterops sig_filtops = { .f_isfd = 0, .f_attach = filt_sigattach, .f_detach = filt_sigdetach, .f_event = filt_signal, }; static int kern_logsigexit = 1; SYSCTL_INT(_kern, KERN_LOGSIGEXIT, logsigexit, CTLFLAG_RW, &kern_logsigexit, 0, "Log processes quitting on abnormal signals to syslog(3)"); static int kern_forcesigexit = 1; SYSCTL_INT(_kern, OID_AUTO, forcesigexit, CTLFLAG_RW, &kern_forcesigexit, 0, "Force trap signal to be handled"); static SYSCTL_NODE(_kern, OID_AUTO, sigqueue, CTLFLAG_RW, 0, "POSIX real time signal"); static int max_pending_per_proc = 128; SYSCTL_INT(_kern_sigqueue, OID_AUTO, max_pending_per_proc, CTLFLAG_RW, &max_pending_per_proc, 0, "Max pending signals per proc"); static int preallocate_siginfo = 1024; SYSCTL_INT(_kern_sigqueue, OID_AUTO, preallocate, CTLFLAG_RDTUN, &preallocate_siginfo, 0, "Preallocated signal memory size"); static int signal_overflow = 0; SYSCTL_INT(_kern_sigqueue, OID_AUTO, overflow, CTLFLAG_RD, &signal_overflow, 0, "Number of signals overflew"); static int signal_alloc_fail = 0; SYSCTL_INT(_kern_sigqueue, OID_AUTO, alloc_fail, CTLFLAG_RD, &signal_alloc_fail, 0, "signals failed to be allocated"); SYSINIT(signal, SI_SUB_P1003_1B, SI_ORDER_FIRST+3, sigqueue_start, NULL); /* * Policy -- Can ucred cr1 send SIGIO to process cr2? * Should use cr_cansignal() once cr_cansignal() allows SIGIO and SIGURG * in the right situations. */ #define CANSIGIO(cr1, cr2) \ ((cr1)->cr_uid == 0 || \ (cr1)->cr_ruid == (cr2)->cr_ruid || \ (cr1)->cr_uid == (cr2)->cr_ruid || \ (cr1)->cr_ruid == (cr2)->cr_uid || \ (cr1)->cr_uid == (cr2)->cr_uid) static int sugid_coredump; SYSCTL_INT(_kern, OID_AUTO, sugid_coredump, CTLFLAG_RWTUN, &sugid_coredump, 0, "Allow setuid and setgid processes to dump core"); static int capmode_coredump; SYSCTL_INT(_kern, OID_AUTO, capmode_coredump, CTLFLAG_RWTUN, &capmode_coredump, 0, "Allow processes in capability mode to dump core"); static int do_coredump = 1; SYSCTL_INT(_kern, OID_AUTO, coredump, CTLFLAG_RW, &do_coredump, 0, "Enable/Disable coredumps"); static int set_core_nodump_flag = 0; SYSCTL_INT(_kern, OID_AUTO, nodump_coredump, CTLFLAG_RW, &set_core_nodump_flag, 0, "Enable setting the NODUMP flag on coredump files"); /* * Signal properties and actions. * The array below categorizes the signals and their default actions * according to the following properties: */ #define SA_KILL 0x01 /* terminates process by default */ #define SA_CORE 0x02 /* ditto and coredumps */ #define SA_STOP 0x04 /* suspend process */ #define SA_TTYSTOP 0x08 /* ditto, from tty */ #define SA_IGNORE 0x10 /* ignore by default */ #define SA_CONT 0x20 /* continue if suspended */ #define SA_CANTMASK 0x40 /* non-maskable, catchable */ static int sigproptbl[NSIG] = { SA_KILL, /* SIGHUP */ SA_KILL, /* SIGINT */ SA_KILL|SA_CORE, /* SIGQUIT */ SA_KILL|SA_CORE, /* SIGILL */ SA_KILL|SA_CORE, /* SIGTRAP */ SA_KILL|SA_CORE, /* SIGABRT */ SA_KILL|SA_CORE, /* SIGEMT */ SA_KILL|SA_CORE, /* SIGFPE */ SA_KILL, /* SIGKILL */ SA_KILL|SA_CORE, /* SIGBUS */ SA_KILL|SA_CORE, /* SIGSEGV */ SA_KILL|SA_CORE, /* SIGSYS */ SA_KILL, /* SIGPIPE */ SA_KILL, /* SIGALRM */ SA_KILL, /* SIGTERM */ SA_IGNORE, /* SIGURG */ SA_STOP, /* SIGSTOP */ SA_STOP|SA_TTYSTOP, /* SIGTSTP */ SA_IGNORE|SA_CONT, /* SIGCONT */ SA_IGNORE, /* SIGCHLD */ SA_STOP|SA_TTYSTOP, /* SIGTTIN */ SA_STOP|SA_TTYSTOP, /* SIGTTOU */ SA_IGNORE, /* SIGIO */ SA_KILL, /* SIGXCPU */ SA_KILL, /* SIGXFSZ */ SA_KILL, /* SIGVTALRM */ SA_KILL, /* SIGPROF */ SA_IGNORE, /* SIGWINCH */ SA_IGNORE, /* SIGINFO */ SA_KILL, /* SIGUSR1 */ SA_KILL, /* SIGUSR2 */ }; static void reschedule_signals(struct proc *p, sigset_t block, int flags); static void sigqueue_start(void) { ksiginfo_zone = uma_zcreate("ksiginfo", sizeof(ksiginfo_t), NULL, NULL, NULL, NULL, UMA_ALIGN_PTR, 0); uma_prealloc(ksiginfo_zone, preallocate_siginfo); p31b_setcfg(CTL_P1003_1B_REALTIME_SIGNALS, _POSIX_REALTIME_SIGNALS); p31b_setcfg(CTL_P1003_1B_RTSIG_MAX, SIGRTMAX - SIGRTMIN + 1); p31b_setcfg(CTL_P1003_1B_SIGQUEUE_MAX, max_pending_per_proc); } ksiginfo_t * ksiginfo_alloc(int wait) { int flags; flags = M_ZERO; if (! wait) flags |= M_NOWAIT; if (ksiginfo_zone != NULL) return ((ksiginfo_t *)uma_zalloc(ksiginfo_zone, flags)); return (NULL); } void ksiginfo_free(ksiginfo_t *ksi) { uma_zfree(ksiginfo_zone, ksi); } static __inline int ksiginfo_tryfree(ksiginfo_t *ksi) { if (!(ksi->ksi_flags & KSI_EXT)) { uma_zfree(ksiginfo_zone, ksi); return (1); } return (0); } void sigqueue_init(sigqueue_t *list, struct proc *p) { SIGEMPTYSET(list->sq_signals); SIGEMPTYSET(list->sq_kill); TAILQ_INIT(&list->sq_list); list->sq_proc = p; list->sq_flags = SQ_INIT; } /* * Get a signal's ksiginfo. * Return: * 0 - signal not found * others - signal number */ static int sigqueue_get(sigqueue_t *sq, int signo, ksiginfo_t *si) { struct proc *p = sq->sq_proc; struct ksiginfo *ksi, *next; int count = 0; KASSERT(sq->sq_flags & SQ_INIT, ("sigqueue not inited")); if (!SIGISMEMBER(sq->sq_signals, signo)) return (0); if (SIGISMEMBER(sq->sq_kill, signo)) { count++; SIGDELSET(sq->sq_kill, signo); } TAILQ_FOREACH_SAFE(ksi, &sq->sq_list, ksi_link, next) { if (ksi->ksi_signo == signo) { if (count == 0) { TAILQ_REMOVE(&sq->sq_list, ksi, ksi_link); ksi->ksi_sigq = NULL; ksiginfo_copy(ksi, si); if (ksiginfo_tryfree(ksi) && p != NULL) p->p_pendingcnt--; } if (++count > 1) break; } } if (count <= 1) SIGDELSET(sq->sq_signals, signo); si->ksi_signo = signo; return (signo); } void sigqueue_take(ksiginfo_t *ksi) { struct ksiginfo *kp; struct proc *p; sigqueue_t *sq; if (ksi == NULL || (sq = ksi->ksi_sigq) == NULL) return; p = sq->sq_proc; TAILQ_REMOVE(&sq->sq_list, ksi, ksi_link); ksi->ksi_sigq = NULL; if (!(ksi->ksi_flags & KSI_EXT) && p != NULL) p->p_pendingcnt--; for (kp = TAILQ_FIRST(&sq->sq_list); kp != NULL; kp = TAILQ_NEXT(kp, ksi_link)) { if (kp->ksi_signo == ksi->ksi_signo) break; } if (kp == NULL && !SIGISMEMBER(sq->sq_kill, ksi->ksi_signo)) SIGDELSET(sq->sq_signals, ksi->ksi_signo); } static int sigqueue_add(sigqueue_t *sq, int signo, ksiginfo_t *si) { struct proc *p = sq->sq_proc; struct ksiginfo *ksi; int ret = 0; KASSERT(sq->sq_flags & SQ_INIT, ("sigqueue not inited")); if (signo == SIGKILL || signo == SIGSTOP || si == NULL) { SIGADDSET(sq->sq_kill, signo); goto out_set_bit; } /* directly insert the ksi, don't copy it */ if (si->ksi_flags & KSI_INS) { if (si->ksi_flags & KSI_HEAD) TAILQ_INSERT_HEAD(&sq->sq_list, si, ksi_link); else TAILQ_INSERT_TAIL(&sq->sq_list, si, ksi_link); si->ksi_sigq = sq; goto out_set_bit; } if (__predict_false(ksiginfo_zone == NULL)) { SIGADDSET(sq->sq_kill, signo); goto out_set_bit; } if (p != NULL && p->p_pendingcnt >= max_pending_per_proc) { signal_overflow++; ret = EAGAIN; } else if ((ksi = ksiginfo_alloc(0)) == NULL) { signal_alloc_fail++; ret = EAGAIN; } else { if (p != NULL) p->p_pendingcnt++; ksiginfo_copy(si, ksi); ksi->ksi_signo = signo; if (si->ksi_flags & KSI_HEAD) TAILQ_INSERT_HEAD(&sq->sq_list, ksi, ksi_link); else TAILQ_INSERT_TAIL(&sq->sq_list, ksi, ksi_link); ksi->ksi_sigq = sq; } if ((si->ksi_flags & KSI_TRAP) != 0 || (si->ksi_flags & KSI_SIGQ) == 0) { if (ret != 0) SIGADDSET(sq->sq_kill, signo); ret = 0; goto out_set_bit; } if (ret != 0) return (ret); out_set_bit: SIGADDSET(sq->sq_signals, signo); return (ret); } void sigqueue_flush(sigqueue_t *sq) { struct proc *p = sq->sq_proc; ksiginfo_t *ksi; KASSERT(sq->sq_flags & SQ_INIT, ("sigqueue not inited")); if (p != NULL) PROC_LOCK_ASSERT(p, MA_OWNED); while ((ksi = TAILQ_FIRST(&sq->sq_list)) != NULL) { TAILQ_REMOVE(&sq->sq_list, ksi, ksi_link); ksi->ksi_sigq = NULL; if (ksiginfo_tryfree(ksi) && p != NULL) p->p_pendingcnt--; } SIGEMPTYSET(sq->sq_signals); SIGEMPTYSET(sq->sq_kill); } static void sigqueue_move_set(sigqueue_t *src, sigqueue_t *dst, const sigset_t *set) { sigset_t tmp; struct proc *p1, *p2; ksiginfo_t *ksi, *next; KASSERT(src->sq_flags & SQ_INIT, ("src sigqueue not inited")); KASSERT(dst->sq_flags & SQ_INIT, ("dst sigqueue not inited")); p1 = src->sq_proc; p2 = dst->sq_proc; /* Move siginfo to target list */ TAILQ_FOREACH_SAFE(ksi, &src->sq_list, ksi_link, next) { if (SIGISMEMBER(*set, ksi->ksi_signo)) { TAILQ_REMOVE(&src->sq_list, ksi, ksi_link); if (p1 != NULL) p1->p_pendingcnt--; TAILQ_INSERT_TAIL(&dst->sq_list, ksi, ksi_link); ksi->ksi_sigq = dst; if (p2 != NULL) p2->p_pendingcnt++; } } /* Move pending bits to target list */ tmp = src->sq_kill; SIGSETAND(tmp, *set); SIGSETOR(dst->sq_kill, tmp); SIGSETNAND(src->sq_kill, tmp); tmp = src->sq_signals; SIGSETAND(tmp, *set); SIGSETOR(dst->sq_signals, tmp); SIGSETNAND(src->sq_signals, tmp); } #if 0 static void sigqueue_move(sigqueue_t *src, sigqueue_t *dst, int signo) { sigset_t set; SIGEMPTYSET(set); SIGADDSET(set, signo); sigqueue_move_set(src, dst, &set); } #endif static void sigqueue_delete_set(sigqueue_t *sq, const sigset_t *set) { struct proc *p = sq->sq_proc; ksiginfo_t *ksi, *next; KASSERT(sq->sq_flags & SQ_INIT, ("src sigqueue not inited")); /* Remove siginfo queue */ TAILQ_FOREACH_SAFE(ksi, &sq->sq_list, ksi_link, next) { if (SIGISMEMBER(*set, ksi->ksi_signo)) { TAILQ_REMOVE(&sq->sq_list, ksi, ksi_link); ksi->ksi_sigq = NULL; if (ksiginfo_tryfree(ksi) && p != NULL) p->p_pendingcnt--; } } SIGSETNAND(sq->sq_kill, *set); SIGSETNAND(sq->sq_signals, *set); } void sigqueue_delete(sigqueue_t *sq, int signo) { sigset_t set; SIGEMPTYSET(set); SIGADDSET(set, signo); sigqueue_delete_set(sq, &set); } /* Remove a set of signals for a process */ static void sigqueue_delete_set_proc(struct proc *p, const sigset_t *set) { sigqueue_t worklist; struct thread *td0; PROC_LOCK_ASSERT(p, MA_OWNED); sigqueue_init(&worklist, NULL); sigqueue_move_set(&p->p_sigqueue, &worklist, set); FOREACH_THREAD_IN_PROC(p, td0) sigqueue_move_set(&td0->td_sigqueue, &worklist, set); sigqueue_flush(&worklist); } void sigqueue_delete_proc(struct proc *p, int signo) { sigset_t set; SIGEMPTYSET(set); SIGADDSET(set, signo); sigqueue_delete_set_proc(p, &set); } static void sigqueue_delete_stopmask_proc(struct proc *p) { sigset_t set; SIGEMPTYSET(set); SIGADDSET(set, SIGSTOP); SIGADDSET(set, SIGTSTP); SIGADDSET(set, SIGTTIN); SIGADDSET(set, SIGTTOU); sigqueue_delete_set_proc(p, &set); } /* * Determine signal that should be delivered to thread td, the current * thread, 0 if none. If there is a pending stop signal with default * action, the process stops in issignal(). */ int cursig(struct thread *td) { PROC_LOCK_ASSERT(td->td_proc, MA_OWNED); mtx_assert(&td->td_proc->p_sigacts->ps_mtx, MA_OWNED); THREAD_LOCK_ASSERT(td, MA_NOTOWNED); return (SIGPENDING(td) ? issignal(td) : 0); } /* * Arrange for ast() to handle unmasked pending signals on return to user * mode. This must be called whenever a signal is added to td_sigqueue or * unmasked in td_sigmask. */ void signotify(struct thread *td) { struct proc *p; p = td->td_proc; PROC_LOCK_ASSERT(p, MA_OWNED); if (SIGPENDING(td)) { thread_lock(td); td->td_flags |= TDF_NEEDSIGCHK | TDF_ASTPENDING; thread_unlock(td); } } int sigonstack(size_t sp) { struct thread *td = curthread; return ((td->td_pflags & TDP_ALTSTACK) ? #if defined(COMPAT_43) ((td->td_sigstk.ss_size == 0) ? (td->td_sigstk.ss_flags & SS_ONSTACK) : ((sp - (size_t)td->td_sigstk.ss_sp) < td->td_sigstk.ss_size)) #else ((sp - (size_t)td->td_sigstk.ss_sp) < td->td_sigstk.ss_size) #endif : 0); } static __inline int sigprop(int sig) { if (sig > 0 && sig < NSIG) return (sigproptbl[_SIG_IDX(sig)]); return (0); } int sig_ffs(sigset_t *set) { int i; for (i = 0; i < _SIG_WORDS; i++) if (set->__bits[i]) return (ffs(set->__bits[i]) + (i * 32)); return (0); } static bool sigact_flag_test(struct sigaction *act, int flag) { /* * SA_SIGINFO is reset when signal disposition is set to * ignore or default. Other flags are kept according to user * settings. */ return ((act->sa_flags & flag) != 0 && (flag != SA_SIGINFO || ((__sighandler_t *)act->sa_sigaction != SIG_IGN && (__sighandler_t *)act->sa_sigaction != SIG_DFL))); } /* * kern_sigaction * sigaction * freebsd4_sigaction * osigaction */ int kern_sigaction(td, sig, act, oact, flags) struct thread *td; register int sig; struct sigaction *act, *oact; int flags; { struct sigacts *ps; struct proc *p = td->td_proc; if (!_SIG_VALID(sig)) return (EINVAL); if (act != NULL && (act->sa_flags & ~(SA_ONSTACK | SA_RESTART | SA_RESETHAND | SA_NOCLDSTOP | SA_NODEFER | SA_NOCLDWAIT | SA_SIGINFO)) != 0) return (EINVAL); PROC_LOCK(p); ps = p->p_sigacts; mtx_lock(&ps->ps_mtx); if (oact) { oact->sa_mask = ps->ps_catchmask[_SIG_IDX(sig)]; oact->sa_flags = 0; if (SIGISMEMBER(ps->ps_sigonstack, sig)) oact->sa_flags |= SA_ONSTACK; if (!SIGISMEMBER(ps->ps_sigintr, sig)) oact->sa_flags |= SA_RESTART; if (SIGISMEMBER(ps->ps_sigreset, sig)) oact->sa_flags |= SA_RESETHAND; if (SIGISMEMBER(ps->ps_signodefer, sig)) oact->sa_flags |= SA_NODEFER; if (SIGISMEMBER(ps->ps_siginfo, sig)) { oact->sa_flags |= SA_SIGINFO; oact->sa_sigaction = (__siginfohandler_t *)ps->ps_sigact[_SIG_IDX(sig)]; } else oact->sa_handler = ps->ps_sigact[_SIG_IDX(sig)]; if (sig == SIGCHLD && ps->ps_flag & PS_NOCLDSTOP) oact->sa_flags |= SA_NOCLDSTOP; if (sig == SIGCHLD && ps->ps_flag & PS_NOCLDWAIT) oact->sa_flags |= SA_NOCLDWAIT; } if (act) { if ((sig == SIGKILL || sig == SIGSTOP) && act->sa_handler != SIG_DFL) { mtx_unlock(&ps->ps_mtx); PROC_UNLOCK(p); return (EINVAL); } /* * Change setting atomically. */ ps->ps_catchmask[_SIG_IDX(sig)] = act->sa_mask; SIG_CANTMASK(ps->ps_catchmask[_SIG_IDX(sig)]); if (sigact_flag_test(act, SA_SIGINFO)) { ps->ps_sigact[_SIG_IDX(sig)] = (__sighandler_t *)act->sa_sigaction; SIGADDSET(ps->ps_siginfo, sig); } else { ps->ps_sigact[_SIG_IDX(sig)] = act->sa_handler; SIGDELSET(ps->ps_siginfo, sig); } if (!sigact_flag_test(act, SA_RESTART)) SIGADDSET(ps->ps_sigintr, sig); else SIGDELSET(ps->ps_sigintr, sig); if (sigact_flag_test(act, SA_ONSTACK)) SIGADDSET(ps->ps_sigonstack, sig); else SIGDELSET(ps->ps_sigonstack, sig); if (sigact_flag_test(act, SA_RESETHAND)) SIGADDSET(ps->ps_sigreset, sig); else SIGDELSET(ps->ps_sigreset, sig); if (sigact_flag_test(act, SA_NODEFER)) SIGADDSET(ps->ps_signodefer, sig); else SIGDELSET(ps->ps_signodefer, sig); if (sig == SIGCHLD) { if (act->sa_flags & SA_NOCLDSTOP) ps->ps_flag |= PS_NOCLDSTOP; else ps->ps_flag &= ~PS_NOCLDSTOP; if (act->sa_flags & SA_NOCLDWAIT) { /* * Paranoia: since SA_NOCLDWAIT is implemented * by reparenting the dying child to PID 1 (and * trust it to reap the zombie), PID 1 itself * is forbidden to set SA_NOCLDWAIT. */ if (p->p_pid == 1) ps->ps_flag &= ~PS_NOCLDWAIT; else ps->ps_flag |= PS_NOCLDWAIT; } else ps->ps_flag &= ~PS_NOCLDWAIT; if (ps->ps_sigact[_SIG_IDX(SIGCHLD)] == SIG_IGN) ps->ps_flag |= PS_CLDSIGIGN; else ps->ps_flag &= ~PS_CLDSIGIGN; } /* * Set bit in ps_sigignore for signals that are set to SIG_IGN, * and for signals set to SIG_DFL where the default is to * ignore. However, don't put SIGCONT in ps_sigignore, as we * have to restart the process. */ if (ps->ps_sigact[_SIG_IDX(sig)] == SIG_IGN || (sigprop(sig) & SA_IGNORE && ps->ps_sigact[_SIG_IDX(sig)] == SIG_DFL)) { /* never to be seen again */ sigqueue_delete_proc(p, sig); if (sig != SIGCONT) /* easier in psignal */ SIGADDSET(ps->ps_sigignore, sig); SIGDELSET(ps->ps_sigcatch, sig); } else { SIGDELSET(ps->ps_sigignore, sig); if (ps->ps_sigact[_SIG_IDX(sig)] == SIG_DFL) SIGDELSET(ps->ps_sigcatch, sig); else SIGADDSET(ps->ps_sigcatch, sig); } #ifdef COMPAT_FREEBSD4 if (ps->ps_sigact[_SIG_IDX(sig)] == SIG_IGN || ps->ps_sigact[_SIG_IDX(sig)] == SIG_DFL || (flags & KSA_FREEBSD4) == 0) SIGDELSET(ps->ps_freebsd4, sig); else SIGADDSET(ps->ps_freebsd4, sig); #endif #ifdef COMPAT_43 if (ps->ps_sigact[_SIG_IDX(sig)] == SIG_IGN || ps->ps_sigact[_SIG_IDX(sig)] == SIG_DFL || (flags & KSA_OSIGSET) == 0) SIGDELSET(ps->ps_osigset, sig); else SIGADDSET(ps->ps_osigset, sig); #endif } mtx_unlock(&ps->ps_mtx); PROC_UNLOCK(p); return (0); } #ifndef _SYS_SYSPROTO_H_ struct sigaction_args { int sig; struct sigaction *act; struct sigaction *oact; }; #endif int sys_sigaction(td, uap) struct thread *td; register struct sigaction_args *uap; { struct sigaction act, oact; register struct sigaction *actp, *oactp; int error; actp = (uap->act != NULL) ? &act : NULL; oactp = (uap->oact != NULL) ? &oact : NULL; if (actp) { error = copyin(uap->act, actp, sizeof(act)); if (error) return (error); } error = kern_sigaction(td, uap->sig, actp, oactp, 0); if (oactp && !error) error = copyout(oactp, uap->oact, sizeof(oact)); return (error); } #ifdef COMPAT_FREEBSD4 #ifndef _SYS_SYSPROTO_H_ struct freebsd4_sigaction_args { int sig; struct sigaction *act; struct sigaction *oact; }; #endif int freebsd4_sigaction(td, uap) struct thread *td; register struct freebsd4_sigaction_args *uap; { struct sigaction act, oact; register struct sigaction *actp, *oactp; int error; actp = (uap->act != NULL) ? &act : NULL; oactp = (uap->oact != NULL) ? &oact : NULL; if (actp) { error = copyin(uap->act, actp, sizeof(act)); if (error) return (error); } error = kern_sigaction(td, uap->sig, actp, oactp, KSA_FREEBSD4); if (oactp && !error) error = copyout(oactp, uap->oact, sizeof(oact)); return (error); } #endif /* COMAPT_FREEBSD4 */ #ifdef COMPAT_43 /* XXX - COMPAT_FBSD3 */ #ifndef _SYS_SYSPROTO_H_ struct osigaction_args { int signum; struct osigaction *nsa; struct osigaction *osa; }; #endif int osigaction(td, uap) struct thread *td; register struct osigaction_args *uap; { struct osigaction sa; struct sigaction nsa, osa; register struct sigaction *nsap, *osap; int error; if (uap->signum <= 0 || uap->signum >= ONSIG) return (EINVAL); nsap = (uap->nsa != NULL) ? &nsa : NULL; osap = (uap->osa != NULL) ? &osa : NULL; if (nsap) { error = copyin(uap->nsa, &sa, sizeof(sa)); if (error) return (error); nsap->sa_handler = sa.sa_handler; nsap->sa_flags = sa.sa_flags; OSIG2SIG(sa.sa_mask, nsap->sa_mask); } error = kern_sigaction(td, uap->signum, nsap, osap, KSA_OSIGSET); if (osap && !error) { sa.sa_handler = osap->sa_handler; sa.sa_flags = osap->sa_flags; SIG2OSIG(osap->sa_mask, sa.sa_mask); error = copyout(&sa, uap->osa, sizeof(sa)); } return (error); } #if !defined(__i386__) /* Avoid replicating the same stub everywhere */ int osigreturn(td, uap) struct thread *td; struct osigreturn_args *uap; { return (nosys(td, (struct nosys_args *)uap)); } #endif #endif /* COMPAT_43 */ /* * Initialize signal state for process 0; * set to ignore signals that are ignored by default. */ void siginit(p) struct proc *p; { register int i; struct sigacts *ps; PROC_LOCK(p); ps = p->p_sigacts; mtx_lock(&ps->ps_mtx); for (i = 1; i <= NSIG; i++) { if (sigprop(i) & SA_IGNORE && i != SIGCONT) { SIGADDSET(ps->ps_sigignore, i); } } mtx_unlock(&ps->ps_mtx); PROC_UNLOCK(p); } /* * Reset specified signal to the default disposition. */ static void sigdflt(struct sigacts *ps, int sig) { mtx_assert(&ps->ps_mtx, MA_OWNED); SIGDELSET(ps->ps_sigcatch, sig); if ((sigprop(sig) & SA_IGNORE) != 0 && sig != SIGCONT) SIGADDSET(ps->ps_sigignore, sig); ps->ps_sigact[_SIG_IDX(sig)] = SIG_DFL; SIGDELSET(ps->ps_siginfo, sig); } /* * Reset signals for an exec of the specified process. */ void execsigs(struct proc *p) { struct sigacts *ps; int sig; struct thread *td; /* * Reset caught signals. Held signals remain held * through td_sigmask (unless they were caught, * and are now ignored by default). */ PROC_LOCK_ASSERT(p, MA_OWNED); td = FIRST_THREAD_IN_PROC(p); ps = p->p_sigacts; mtx_lock(&ps->ps_mtx); while (SIGNOTEMPTY(ps->ps_sigcatch)) { sig = sig_ffs(&ps->ps_sigcatch); sigdflt(ps, sig); if ((sigprop(sig) & SA_IGNORE) != 0) sigqueue_delete_proc(p, sig); } /* * Reset stack state to the user stack. * Clear set of signals caught on the signal stack. */ td->td_sigstk.ss_flags = SS_DISABLE; td->td_sigstk.ss_size = 0; td->td_sigstk.ss_sp = 0; td->td_pflags &= ~TDP_ALTSTACK; /* * Reset no zombies if child dies flag as Solaris does. */ ps->ps_flag &= ~(PS_NOCLDWAIT | PS_CLDSIGIGN); if (ps->ps_sigact[_SIG_IDX(SIGCHLD)] == SIG_IGN) ps->ps_sigact[_SIG_IDX(SIGCHLD)] = SIG_DFL; mtx_unlock(&ps->ps_mtx); } /* * kern_sigprocmask() * * Manipulate signal mask. */ int kern_sigprocmask(struct thread *td, int how, sigset_t *set, sigset_t *oset, int flags) { sigset_t new_block, oset1; struct proc *p; int error; p = td->td_proc; if ((flags & SIGPROCMASK_PROC_LOCKED) != 0) PROC_LOCK_ASSERT(p, MA_OWNED); else PROC_LOCK(p); mtx_assert(&p->p_sigacts->ps_mtx, (flags & SIGPROCMASK_PS_LOCKED) != 0 ? MA_OWNED : MA_NOTOWNED); if (oset != NULL) *oset = td->td_sigmask; error = 0; if (set != NULL) { switch (how) { case SIG_BLOCK: SIG_CANTMASK(*set); oset1 = td->td_sigmask; SIGSETOR(td->td_sigmask, *set); new_block = td->td_sigmask; SIGSETNAND(new_block, oset1); break; case SIG_UNBLOCK: SIGSETNAND(td->td_sigmask, *set); signotify(td); goto out; case SIG_SETMASK: SIG_CANTMASK(*set); oset1 = td->td_sigmask; if (flags & SIGPROCMASK_OLD) SIGSETLO(td->td_sigmask, *set); else td->td_sigmask = *set; new_block = td->td_sigmask; SIGSETNAND(new_block, oset1); signotify(td); break; default: error = EINVAL; goto out; } /* * The new_block set contains signals that were not previously * blocked, but are blocked now. * * In case we block any signal that was not previously blocked * for td, and process has the signal pending, try to schedule * signal delivery to some thread that does not block the * signal, possibly waking it up. */ if (p->p_numthreads != 1) reschedule_signals(p, new_block, flags); } out: if (!(flags & SIGPROCMASK_PROC_LOCKED)) PROC_UNLOCK(p); return (error); } #ifndef _SYS_SYSPROTO_H_ struct sigprocmask_args { int how; const sigset_t *set; sigset_t *oset; }; #endif int sys_sigprocmask(td, uap) register struct thread *td; struct sigprocmask_args *uap; { sigset_t set, oset; sigset_t *setp, *osetp; int error; setp = (uap->set != NULL) ? &set : NULL; osetp = (uap->oset != NULL) ? &oset : NULL; if (setp) { error = copyin(uap->set, setp, sizeof(set)); if (error) return (error); } error = kern_sigprocmask(td, uap->how, setp, osetp, 0); if (osetp && !error) { error = copyout(osetp, uap->oset, sizeof(oset)); } return (error); } #ifdef COMPAT_43 /* XXX - COMPAT_FBSD3 */ #ifndef _SYS_SYSPROTO_H_ struct osigprocmask_args { int how; osigset_t mask; }; #endif int osigprocmask(td, uap) register struct thread *td; struct osigprocmask_args *uap; { sigset_t set, oset; int error; OSIG2SIG(uap->mask, set); error = kern_sigprocmask(td, uap->how, &set, &oset, 1); SIG2OSIG(oset, td->td_retval[0]); return (error); } #endif /* COMPAT_43 */ int sys_sigwait(struct thread *td, struct sigwait_args *uap) { ksiginfo_t ksi; sigset_t set; int error; error = copyin(uap->set, &set, sizeof(set)); if (error) { td->td_retval[0] = error; return (0); } error = kern_sigtimedwait(td, set, &ksi, NULL); if (error) { if (error == EINTR && td->td_proc->p_osrel < P_OSREL_SIGWAIT) error = ERESTART; if (error == ERESTART) return (error); td->td_retval[0] = error; return (0); } error = copyout(&ksi.ksi_signo, uap->sig, sizeof(ksi.ksi_signo)); td->td_retval[0] = error; return (0); } int sys_sigtimedwait(struct thread *td, struct sigtimedwait_args *uap) { struct timespec ts; struct timespec *timeout; sigset_t set; ksiginfo_t ksi; int error; if (uap->timeout) { error = copyin(uap->timeout, &ts, sizeof(ts)); if (error) return (error); timeout = &ts; } else timeout = NULL; error = copyin(uap->set, &set, sizeof(set)); if (error) return (error); error = kern_sigtimedwait(td, set, &ksi, timeout); if (error) return (error); if (uap->info) error = copyout(&ksi.ksi_info, uap->info, sizeof(siginfo_t)); if (error == 0) td->td_retval[0] = ksi.ksi_signo; return (error); } int sys_sigwaitinfo(struct thread *td, struct sigwaitinfo_args *uap) { ksiginfo_t ksi; sigset_t set; int error; error = copyin(uap->set, &set, sizeof(set)); if (error) return (error); error = kern_sigtimedwait(td, set, &ksi, NULL); if (error) return (error); if (uap->info) error = copyout(&ksi.ksi_info, uap->info, sizeof(siginfo_t)); if (error == 0) td->td_retval[0] = ksi.ksi_signo; return (error); } int kern_sigtimedwait(struct thread *td, sigset_t waitset, ksiginfo_t *ksi, struct timespec *timeout) { struct sigacts *ps; sigset_t saved_mask, new_block; struct proc *p; int error, sig, timo, timevalid = 0; struct timespec rts, ets, ts; struct timeval tv; p = td->td_proc; error = 0; ets.tv_sec = 0; ets.tv_nsec = 0; if (timeout != NULL) { if (timeout->tv_nsec >= 0 && timeout->tv_nsec < 1000000000) { timevalid = 1; getnanouptime(&rts); ets = rts; timespecadd(&ets, timeout); } } ksiginfo_init(ksi); /* Some signals can not be waited for. */ SIG_CANTMASK(waitset); ps = p->p_sigacts; PROC_LOCK(p); saved_mask = td->td_sigmask; SIGSETNAND(td->td_sigmask, waitset); for (;;) { mtx_lock(&ps->ps_mtx); sig = cursig(td); mtx_unlock(&ps->ps_mtx); if (sig != 0 && SIGISMEMBER(waitset, sig)) { if (sigqueue_get(&td->td_sigqueue, sig, ksi) != 0 || sigqueue_get(&p->p_sigqueue, sig, ksi) != 0) { error = 0; break; } } if (error != 0) break; /* * POSIX says this must be checked after looking for pending * signals. */ if (timeout != NULL) { if (!timevalid) { error = EINVAL; break; } getnanouptime(&rts); if (timespeccmp(&rts, &ets, >=)) { error = EAGAIN; break; } ts = ets; timespecsub(&ts, &rts); TIMESPEC_TO_TIMEVAL(&tv, &ts); timo = tvtohz(&tv); } else { timo = 0; } error = msleep(ps, &p->p_mtx, PPAUSE|PCATCH, "sigwait", timo); if (timeout != NULL) { if (error == ERESTART) { /* Timeout can not be restarted. */ error = EINTR; } else if (error == EAGAIN) { /* We will calculate timeout by ourself. */ error = 0; } } } new_block = saved_mask; SIGSETNAND(new_block, td->td_sigmask); td->td_sigmask = saved_mask; /* * Fewer signals can be delivered to us, reschedule signal * notification. */ if (p->p_numthreads != 1) reschedule_signals(p, new_block, 0); if (error == 0) { SDT_PROBE(proc, kernel, , signal__clear, sig, ksi, 0, 0, 0); if (ksi->ksi_code == SI_TIMER) itimer_accept(p, ksi->ksi_timerid, ksi); #ifdef KTRACE if (KTRPOINT(td, KTR_PSIG)) { sig_t action; mtx_lock(&ps->ps_mtx); action = ps->ps_sigact[_SIG_IDX(sig)]; mtx_unlock(&ps->ps_mtx); ktrpsig(sig, action, &td->td_sigmask, ksi->ksi_code); } #endif if (sig == SIGKILL) sigexit(td, sig); } PROC_UNLOCK(p); return (error); } #ifndef _SYS_SYSPROTO_H_ struct sigpending_args { sigset_t *set; }; #endif int sys_sigpending(td, uap) struct thread *td; struct sigpending_args *uap; { struct proc *p = td->td_proc; sigset_t pending; PROC_LOCK(p); pending = p->p_sigqueue.sq_signals; SIGSETOR(pending, td->td_sigqueue.sq_signals); PROC_UNLOCK(p); return (copyout(&pending, uap->set, sizeof(sigset_t))); } #ifdef COMPAT_43 /* XXX - COMPAT_FBSD3 */ #ifndef _SYS_SYSPROTO_H_ struct osigpending_args { int dummy; }; #endif int osigpending(td, uap) struct thread *td; struct osigpending_args *uap; { struct proc *p = td->td_proc; sigset_t pending; PROC_LOCK(p); pending = p->p_sigqueue.sq_signals; SIGSETOR(pending, td->td_sigqueue.sq_signals); PROC_UNLOCK(p); SIG2OSIG(pending, td->td_retval[0]); return (0); } #endif /* COMPAT_43 */ #if defined(COMPAT_43) /* * Generalized interface signal handler, 4.3-compatible. */ #ifndef _SYS_SYSPROTO_H_ struct osigvec_args { int signum; struct sigvec *nsv; struct sigvec *osv; }; #endif /* ARGSUSED */ int osigvec(td, uap) struct thread *td; register struct osigvec_args *uap; { struct sigvec vec; struct sigaction nsa, osa; register struct sigaction *nsap, *osap; int error; if (uap->signum <= 0 || uap->signum >= ONSIG) return (EINVAL); nsap = (uap->nsv != NULL) ? &nsa : NULL; osap = (uap->osv != NULL) ? &osa : NULL; if (nsap) { error = copyin(uap->nsv, &vec, sizeof(vec)); if (error) return (error); nsap->sa_handler = vec.sv_handler; OSIG2SIG(vec.sv_mask, nsap->sa_mask); nsap->sa_flags = vec.sv_flags; nsap->sa_flags ^= SA_RESTART; /* opposite of SV_INTERRUPT */ } error = kern_sigaction(td, uap->signum, nsap, osap, KSA_OSIGSET); if (osap && !error) { vec.sv_handler = osap->sa_handler; SIG2OSIG(osap->sa_mask, vec.sv_mask); vec.sv_flags = osap->sa_flags; vec.sv_flags &= ~SA_NOCLDWAIT; vec.sv_flags ^= SA_RESTART; error = copyout(&vec, uap->osv, sizeof(vec)); } return (error); } #ifndef _SYS_SYSPROTO_H_ struct osigblock_args { int mask; }; #endif int osigblock(td, uap) register struct thread *td; struct osigblock_args *uap; { sigset_t set, oset; OSIG2SIG(uap->mask, set); kern_sigprocmask(td, SIG_BLOCK, &set, &oset, 0); SIG2OSIG(oset, td->td_retval[0]); return (0); } #ifndef _SYS_SYSPROTO_H_ struct osigsetmask_args { int mask; }; #endif int osigsetmask(td, uap) struct thread *td; struct osigsetmask_args *uap; { sigset_t set, oset; OSIG2SIG(uap->mask, set); kern_sigprocmask(td, SIG_SETMASK, &set, &oset, 0); SIG2OSIG(oset, td->td_retval[0]); return (0); } #endif /* COMPAT_43 */ /* * Suspend calling thread until signal, providing mask to be set in the * meantime. */ #ifndef _SYS_SYSPROTO_H_ struct sigsuspend_args { const sigset_t *sigmask; }; #endif /* ARGSUSED */ int sys_sigsuspend(td, uap) struct thread *td; struct sigsuspend_args *uap; { sigset_t mask; int error; error = copyin(uap->sigmask, &mask, sizeof(mask)); if (error) return (error); return (kern_sigsuspend(td, mask)); } int kern_sigsuspend(struct thread *td, sigset_t mask) { struct proc *p = td->td_proc; int has_sig, sig; /* * When returning from sigsuspend, we want * the old mask to be restored after the * signal handler has finished. Thus, we * save it here and mark the sigacts structure * to indicate this. */ PROC_LOCK(p); kern_sigprocmask(td, SIG_SETMASK, &mask, &td->td_oldsigmask, SIGPROCMASK_PROC_LOCKED); td->td_pflags |= TDP_OLDMASK; /* * Process signals now. Otherwise, we can get spurious wakeup * due to signal entered process queue, but delivered to other * thread. But sigsuspend should return only on signal * delivery. */ (p->p_sysent->sv_set_syscall_retval)(td, EINTR); for (has_sig = 0; !has_sig;) { while (msleep(&p->p_sigacts, &p->p_mtx, PPAUSE|PCATCH, "pause", 0) == 0) /* void */; thread_suspend_check(0); mtx_lock(&p->p_sigacts->ps_mtx); while ((sig = cursig(td)) != 0) has_sig += postsig(sig); mtx_unlock(&p->p_sigacts->ps_mtx); } PROC_UNLOCK(p); td->td_errno = EINTR; td->td_pflags |= TDP_NERRNO; return (EJUSTRETURN); } #ifdef COMPAT_43 /* XXX - COMPAT_FBSD3 */ /* * Compatibility sigsuspend call for old binaries. Note nonstandard calling * convention: libc stub passes mask, not pointer, to save a copyin. */ #ifndef _SYS_SYSPROTO_H_ struct osigsuspend_args { osigset_t mask; }; #endif /* ARGSUSED */ int osigsuspend(td, uap) struct thread *td; struct osigsuspend_args *uap; { sigset_t mask; OSIG2SIG(uap->mask, mask); return (kern_sigsuspend(td, mask)); } #endif /* COMPAT_43 */ #if defined(COMPAT_43) #ifndef _SYS_SYSPROTO_H_ struct osigstack_args { struct sigstack *nss; struct sigstack *oss; }; #endif /* ARGSUSED */ int osigstack(td, uap) struct thread *td; register struct osigstack_args *uap; { struct sigstack nss, oss; int error = 0; if (uap->nss != NULL) { error = copyin(uap->nss, &nss, sizeof(nss)); if (error) return (error); } oss.ss_sp = td->td_sigstk.ss_sp; oss.ss_onstack = sigonstack(cpu_getstack(td)); if (uap->nss != NULL) { td->td_sigstk.ss_sp = nss.ss_sp; td->td_sigstk.ss_size = 0; td->td_sigstk.ss_flags |= nss.ss_onstack & SS_ONSTACK; td->td_pflags |= TDP_ALTSTACK; } if (uap->oss != NULL) error = copyout(&oss, uap->oss, sizeof(oss)); return (error); } #endif /* COMPAT_43 */ #ifndef _SYS_SYSPROTO_H_ struct sigaltstack_args { stack_t *ss; stack_t *oss; }; #endif /* ARGSUSED */ int sys_sigaltstack(td, uap) struct thread *td; register struct sigaltstack_args *uap; { stack_t ss, oss; int error; if (uap->ss != NULL) { error = copyin(uap->ss, &ss, sizeof(ss)); if (error) return (error); } error = kern_sigaltstack(td, (uap->ss != NULL) ? &ss : NULL, (uap->oss != NULL) ? &oss : NULL); if (error) return (error); if (uap->oss != NULL) error = copyout(&oss, uap->oss, sizeof(stack_t)); return (error); } int kern_sigaltstack(struct thread *td, stack_t *ss, stack_t *oss) { struct proc *p = td->td_proc; int oonstack; oonstack = sigonstack(cpu_getstack(td)); if (oss != NULL) { *oss = td->td_sigstk; oss->ss_flags = (td->td_pflags & TDP_ALTSTACK) ? ((oonstack) ? SS_ONSTACK : 0) : SS_DISABLE; } if (ss != NULL) { if (oonstack) return (EPERM); if ((ss->ss_flags & ~SS_DISABLE) != 0) return (EINVAL); if (!(ss->ss_flags & SS_DISABLE)) { if (ss->ss_size < p->p_sysent->sv_minsigstksz) return (ENOMEM); td->td_sigstk = *ss; td->td_pflags |= TDP_ALTSTACK; } else { td->td_pflags &= ~TDP_ALTSTACK; } } return (0); } /* * Common code for kill process group/broadcast kill. * cp is calling process. */ static int killpg1(struct thread *td, int sig, int pgid, int all, ksiginfo_t *ksi) { struct proc *p; struct pgrp *pgrp; int err; int ret; ret = ESRCH; if (all) { /* * broadcast */ sx_slock(&allproc_lock); FOREACH_PROC_IN_SYSTEM(p) { PROC_LOCK(p); if (p->p_pid <= 1 || p->p_flag & P_SYSTEM || p == td->td_proc || p->p_state == PRS_NEW) { PROC_UNLOCK(p); continue; } err = p_cansignal(td, p, sig); if (err == 0) { if (sig) pksignal(p, sig, ksi); ret = err; } else if (ret == ESRCH) ret = err; PROC_UNLOCK(p); } sx_sunlock(&allproc_lock); } else { sx_slock(&proctree_lock); if (pgid == 0) { /* * zero pgid means send to my process group. */ pgrp = td->td_proc->p_pgrp; PGRP_LOCK(pgrp); } else { pgrp = pgfind(pgid); if (pgrp == NULL) { sx_sunlock(&proctree_lock); return (ESRCH); } } sx_sunlock(&proctree_lock); LIST_FOREACH(p, &pgrp->pg_members, p_pglist) { PROC_LOCK(p); if (p->p_pid <= 1 || p->p_flag & P_SYSTEM || p->p_state == PRS_NEW) { PROC_UNLOCK(p); continue; } err = p_cansignal(td, p, sig); if (err == 0) { if (sig) pksignal(p, sig, ksi); ret = err; } else if (ret == ESRCH) ret = err; PROC_UNLOCK(p); } PGRP_UNLOCK(pgrp); } return (ret); } #ifndef _SYS_SYSPROTO_H_ struct kill_args { int pid; int signum; }; #endif /* ARGSUSED */ int sys_kill(struct thread *td, struct kill_args *uap) { ksiginfo_t ksi; struct proc *p; int error; /* * A process in capability mode can send signals only to himself. * The main rationale behind this is that abort(3) is implemented as * kill(getpid(), SIGABRT). */ if (IN_CAPABILITY_MODE(td) && uap->pid != td->td_proc->p_pid) return (ECAPMODE); AUDIT_ARG_SIGNUM(uap->signum); AUDIT_ARG_PID(uap->pid); if ((u_int)uap->signum > _SIG_MAXSIG) return (EINVAL); ksiginfo_init(&ksi); ksi.ksi_signo = uap->signum; ksi.ksi_code = SI_USER; ksi.ksi_pid = td->td_proc->p_pid; ksi.ksi_uid = td->td_ucred->cr_ruid; if (uap->pid > 0) { /* kill single process */ if ((p = pfind(uap->pid)) == NULL) { if ((p = zpfind(uap->pid)) == NULL) return (ESRCH); } AUDIT_ARG_PROCESS(p); error = p_cansignal(td, p, uap->signum); if (error == 0 && uap->signum) pksignal(p, uap->signum, &ksi); PROC_UNLOCK(p); return (error); } switch (uap->pid) { case -1: /* broadcast signal */ return (killpg1(td, uap->signum, 0, 1, &ksi)); case 0: /* signal own process group */ return (killpg1(td, uap->signum, 0, 0, &ksi)); default: /* negative explicit process group */ return (killpg1(td, uap->signum, -uap->pid, 0, &ksi)); } /* NOTREACHED */ } int sys_pdkill(td, uap) struct thread *td; struct pdkill_args *uap; { struct proc *p; cap_rights_t rights; int error; AUDIT_ARG_SIGNUM(uap->signum); AUDIT_ARG_FD(uap->fd); if ((u_int)uap->signum > _SIG_MAXSIG) return (EINVAL); error = procdesc_find(td, uap->fd, cap_rights_init(&rights, CAP_PDKILL), &p); if (error) return (error); AUDIT_ARG_PROCESS(p); error = p_cansignal(td, p, uap->signum); if (error == 0 && uap->signum) kern_psignal(p, uap->signum); PROC_UNLOCK(p); return (error); } #if defined(COMPAT_43) #ifndef _SYS_SYSPROTO_H_ struct okillpg_args { int pgid; int signum; }; #endif /* ARGSUSED */ int okillpg(struct thread *td, struct okillpg_args *uap) { ksiginfo_t ksi; AUDIT_ARG_SIGNUM(uap->signum); AUDIT_ARG_PID(uap->pgid); if ((u_int)uap->signum > _SIG_MAXSIG) return (EINVAL); ksiginfo_init(&ksi); ksi.ksi_signo = uap->signum; ksi.ksi_code = SI_USER; ksi.ksi_pid = td->td_proc->p_pid; ksi.ksi_uid = td->td_ucred->cr_ruid; return (killpg1(td, uap->signum, uap->pgid, 0, &ksi)); } #endif /* COMPAT_43 */ #ifndef _SYS_SYSPROTO_H_ struct sigqueue_args { pid_t pid; int signum; /* union sigval */ void *value; }; #endif int sys_sigqueue(struct thread *td, struct sigqueue_args *uap) { ksiginfo_t ksi; struct proc *p; int error; if ((u_int)uap->signum > _SIG_MAXSIG) return (EINVAL); /* * Specification says sigqueue can only send signal to * single process. */ if (uap->pid <= 0) return (EINVAL); if ((p = pfind(uap->pid)) == NULL) { if ((p = zpfind(uap->pid)) == NULL) return (ESRCH); } error = p_cansignal(td, p, uap->signum); if (error == 0 && uap->signum != 0) { ksiginfo_init(&ksi); ksi.ksi_flags = KSI_SIGQ; ksi.ksi_signo = uap->signum; ksi.ksi_code = SI_QUEUE; ksi.ksi_pid = td->td_proc->p_pid; ksi.ksi_uid = td->td_ucred->cr_ruid; ksi.ksi_value.sival_ptr = uap->value; error = pksignal(p, ksi.ksi_signo, &ksi); } PROC_UNLOCK(p); return (error); } /* * Send a signal to a process group. */ void gsignal(int pgid, int sig, ksiginfo_t *ksi) { struct pgrp *pgrp; if (pgid != 0) { sx_slock(&proctree_lock); pgrp = pgfind(pgid); sx_sunlock(&proctree_lock); if (pgrp != NULL) { pgsignal(pgrp, sig, 0, ksi); PGRP_UNLOCK(pgrp); } } } /* * Send a signal to a process group. If checktty is 1, * limit to members which have a controlling terminal. */ void pgsignal(struct pgrp *pgrp, int sig, int checkctty, ksiginfo_t *ksi) { struct proc *p; if (pgrp) { PGRP_LOCK_ASSERT(pgrp, MA_OWNED); LIST_FOREACH(p, &pgrp->pg_members, p_pglist) { PROC_LOCK(p); if (p->p_state == PRS_NORMAL && (checkctty == 0 || p->p_flag & P_CONTROLT)) pksignal(p, sig, ksi); PROC_UNLOCK(p); } } } /* * Recalculate the signal mask and reset the signal disposition after * usermode frame for delivery is formed. Should be called after * mach-specific routine, because sysent->sv_sendsig() needs correct * ps_siginfo and signal mask. */ static void postsig_done(int sig, struct thread *td, struct sigacts *ps) { sigset_t mask; mtx_assert(&ps->ps_mtx, MA_OWNED); td->td_ru.ru_nsignals++; mask = ps->ps_catchmask[_SIG_IDX(sig)]; if (!SIGISMEMBER(ps->ps_signodefer, sig)) SIGADDSET(mask, sig); kern_sigprocmask(td, SIG_BLOCK, &mask, NULL, SIGPROCMASK_PROC_LOCKED | SIGPROCMASK_PS_LOCKED); if (SIGISMEMBER(ps->ps_sigreset, sig)) sigdflt(ps, sig); } /* * Send a signal caused by a trap to the current thread. If it will be * caught immediately, deliver it with correct code. Otherwise, post it * normally. */ void trapsignal(struct thread *td, ksiginfo_t *ksi) { struct sigacts *ps; struct proc *p; int sig; int code; p = td->td_proc; sig = ksi->ksi_signo; code = ksi->ksi_code; KASSERT(_SIG_VALID(sig), ("invalid signal")); PROC_LOCK(p); ps = p->p_sigacts; mtx_lock(&ps->ps_mtx); if ((p->p_flag & P_TRACED) == 0 && SIGISMEMBER(ps->ps_sigcatch, sig) && !SIGISMEMBER(td->td_sigmask, sig)) { #ifdef KTRACE if (KTRPOINT(curthread, KTR_PSIG)) ktrpsig(sig, ps->ps_sigact[_SIG_IDX(sig)], &td->td_sigmask, code); #endif (*p->p_sysent->sv_sendsig)(ps->ps_sigact[_SIG_IDX(sig)], ksi, &td->td_sigmask); postsig_done(sig, td, ps); mtx_unlock(&ps->ps_mtx); } else { /* * Avoid a possible infinite loop if the thread * masking the signal or process is ignoring the * signal. */ if (kern_forcesigexit && (SIGISMEMBER(td->td_sigmask, sig) || ps->ps_sigact[_SIG_IDX(sig)] == SIG_IGN)) { SIGDELSET(td->td_sigmask, sig); SIGDELSET(ps->ps_sigcatch, sig); SIGDELSET(ps->ps_sigignore, sig); ps->ps_sigact[_SIG_IDX(sig)] = SIG_DFL; } mtx_unlock(&ps->ps_mtx); p->p_code = code; /* XXX for core dump/debugger */ p->p_sig = sig; /* XXX to verify code */ tdsendsignal(p, td, sig, ksi); } PROC_UNLOCK(p); } static struct thread * sigtd(struct proc *p, int sig, int prop) { struct thread *td, *signal_td; PROC_LOCK_ASSERT(p, MA_OWNED); /* * Check if current thread can handle the signal without * switching context to another thread. */ if (curproc == p && !SIGISMEMBER(curthread->td_sigmask, sig)) return (curthread); signal_td = NULL; FOREACH_THREAD_IN_PROC(p, td) { if (!SIGISMEMBER(td->td_sigmask, sig)) { signal_td = td; break; } } if (signal_td == NULL) signal_td = FIRST_THREAD_IN_PROC(p); return (signal_td); } /* * Send the signal to the process. If the signal has an action, the action * is usually performed by the target process rather than the caller; we add * the signal to the set of pending signals for the process. * * Exceptions: * o When a stop signal is sent to a sleeping process that takes the * default action, the process is stopped without awakening it. * o SIGCONT restarts stopped processes (or puts them back to sleep) * regardless of the signal action (eg, blocked or ignored). * * Other ignored signals are discarded immediately. * * NB: This function may be entered from the debugger via the "kill" DDB * command. There is little that can be done to mitigate the possibly messy * side effects of this unwise possibility. */ void kern_psignal(struct proc *p, int sig) { ksiginfo_t ksi; ksiginfo_init(&ksi); ksi.ksi_signo = sig; ksi.ksi_code = SI_KERNEL; (void) tdsendsignal(p, NULL, sig, &ksi); } int pksignal(struct proc *p, int sig, ksiginfo_t *ksi) { return (tdsendsignal(p, NULL, sig, ksi)); } /* Utility function for finding a thread to send signal event to. */ int sigev_findtd(struct proc *p ,struct sigevent *sigev, struct thread **ttd) { struct thread *td; if (sigev->sigev_notify == SIGEV_THREAD_ID) { td = tdfind(sigev->sigev_notify_thread_id, p->p_pid); if (td == NULL) return (ESRCH); *ttd = td; } else { *ttd = NULL; PROC_LOCK(p); } return (0); } void tdsignal(struct thread *td, int sig) { ksiginfo_t ksi; ksiginfo_init(&ksi); ksi.ksi_signo = sig; ksi.ksi_code = SI_KERNEL; (void) tdsendsignal(td->td_proc, td, sig, &ksi); } void tdksignal(struct thread *td, int sig, ksiginfo_t *ksi) { (void) tdsendsignal(td->td_proc, td, sig, ksi); } int tdsendsignal(struct proc *p, struct thread *td, int sig, ksiginfo_t *ksi) { sig_t action; sigqueue_t *sigqueue; int prop; struct sigacts *ps; int intrval; int ret = 0; int wakeup_swapper; MPASS(td == NULL || p == td->td_proc); PROC_LOCK_ASSERT(p, MA_OWNED); if (!_SIG_VALID(sig)) panic("%s(): invalid signal %d", __func__, sig); KASSERT(ksi == NULL || !KSI_ONQ(ksi), ("%s: ksi on queue", __func__)); /* * IEEE Std 1003.1-2001: return success when killing a zombie. */ if (p->p_state == PRS_ZOMBIE) { if (ksi && (ksi->ksi_flags & KSI_INS)) ksiginfo_tryfree(ksi); return (ret); } ps = p->p_sigacts; KNOTE_LOCKED(&p->p_klist, NOTE_SIGNAL | sig); prop = sigprop(sig); if (td == NULL) { td = sigtd(p, sig, prop); sigqueue = &p->p_sigqueue; } else sigqueue = &td->td_sigqueue; SDT_PROBE(proc, kernel, , signal__send, td, p, sig, 0, 0 ); /* * If the signal is being ignored, * then we forget about it immediately. * (Note: we don't set SIGCONT in ps_sigignore, * and if it is set to SIG_IGN, * action will be SIG_DFL here.) */ mtx_lock(&ps->ps_mtx); if (SIGISMEMBER(ps->ps_sigignore, sig)) { SDT_PROBE(proc, kernel, , signal__discard, td, p, sig, 0, 0 ); mtx_unlock(&ps->ps_mtx); if (ksi && (ksi->ksi_flags & KSI_INS)) ksiginfo_tryfree(ksi); return (ret); } if (SIGISMEMBER(td->td_sigmask, sig)) action = SIG_HOLD; else if (SIGISMEMBER(ps->ps_sigcatch, sig)) action = SIG_CATCH; else action = SIG_DFL; if (SIGISMEMBER(ps->ps_sigintr, sig)) intrval = EINTR; else intrval = ERESTART; mtx_unlock(&ps->ps_mtx); if (prop & SA_CONT) sigqueue_delete_stopmask_proc(p); else if (prop & SA_STOP) { /* * If sending a tty stop signal to a member of an orphaned * process group, discard the signal here if the action * is default; don't stop the process below if sleeping, * and don't clear any pending SIGCONT. */ if ((prop & SA_TTYSTOP) && (p->p_pgrp->pg_jobc == 0) && (action == SIG_DFL)) { if (ksi && (ksi->ksi_flags & KSI_INS)) ksiginfo_tryfree(ksi); return (ret); } sigqueue_delete_proc(p, SIGCONT); if (p->p_flag & P_CONTINUED) { p->p_flag &= ~P_CONTINUED; PROC_LOCK(p->p_pptr); sigqueue_take(p->p_ksi); PROC_UNLOCK(p->p_pptr); } } ret = sigqueue_add(sigqueue, sig, ksi); if (ret != 0) return (ret); signotify(td); /* * Defer further processing for signals which are held, * except that stopped processes must be continued by SIGCONT. */ if (action == SIG_HOLD && !((prop & SA_CONT) && (p->p_flag & P_STOPPED_SIG))) return (ret); /* * SIGKILL: Remove procfs STOPEVENTs. */ if (sig == SIGKILL) { /* from procfs_ioctl.c: PIOCBIC */ p->p_stops = 0; /* from procfs_ioctl.c: PIOCCONT */ p->p_step = 0; wakeup(&p->p_step); } /* * Some signals have a process-wide effect and a per-thread * component. Most processing occurs when the process next * tries to cross the user boundary, however there are some * times when processing needs to be done immediately, such as * waking up threads so that they can cross the user boundary. * We try to do the per-process part here. */ if (P_SHOULDSTOP(p)) { KASSERT(!(p->p_flag & P_WEXIT), ("signal to stopped but exiting process")); if (sig == SIGKILL) { /* * If traced process is already stopped, * then no further action is necessary. */ if (p->p_flag & P_TRACED) goto out; /* * SIGKILL sets process running. * It will die elsewhere. * All threads must be restarted. */ p->p_flag &= ~P_STOPPED_SIG; goto runfast; } if (prop & SA_CONT) { /* * If traced process is already stopped, * then no further action is necessary. */ if (p->p_flag & P_TRACED) goto out; /* * If SIGCONT is default (or ignored), we continue the * process but don't leave the signal in sigqueue as * it has no further action. If SIGCONT is held, we * continue the process and leave the signal in * sigqueue. If the process catches SIGCONT, let it * handle the signal itself. If it isn't waiting on * an event, it goes back to run state. * Otherwise, process goes back to sleep state. */ p->p_flag &= ~P_STOPPED_SIG; PROC_SLOCK(p); if (p->p_numthreads == p->p_suspcount) { PROC_SUNLOCK(p); p->p_flag |= P_CONTINUED; p->p_xstat = SIGCONT; PROC_LOCK(p->p_pptr); childproc_continued(p); PROC_UNLOCK(p->p_pptr); PROC_SLOCK(p); } if (action == SIG_DFL) { thread_unsuspend(p); PROC_SUNLOCK(p); sigqueue_delete(sigqueue, sig); goto out; } if (action == SIG_CATCH) { /* * The process wants to catch it so it needs * to run at least one thread, but which one? */ PROC_SUNLOCK(p); goto runfast; } /* * The signal is not ignored or caught. */ thread_unsuspend(p); PROC_SUNLOCK(p); goto out; } if (prop & SA_STOP) { /* * If traced process is already stopped, * then no further action is necessary. */ if (p->p_flag & P_TRACED) goto out; /* * Already stopped, don't need to stop again * (If we did the shell could get confused). * Just make sure the signal STOP bit set. */ p->p_flag |= P_STOPPED_SIG; sigqueue_delete(sigqueue, sig); goto out; } /* * All other kinds of signals: * If a thread is sleeping interruptibly, simulate a * wakeup so that when it is continued it will be made * runnable and can look at the signal. However, don't make * the PROCESS runnable, leave it stopped. * It may run a bit until it hits a thread_suspend_check(). */ wakeup_swapper = 0; PROC_SLOCK(p); thread_lock(td); if (TD_ON_SLEEPQ(td) && (td->td_flags & TDF_SINTR)) wakeup_swapper = sleepq_abort(td, intrval); thread_unlock(td); PROC_SUNLOCK(p); if (wakeup_swapper) kick_proc0(); goto out; /* * Mutexes are short lived. Threads waiting on them will * hit thread_suspend_check() soon. */ } else if (p->p_state == PRS_NORMAL) { if (p->p_flag & P_TRACED || action == SIG_CATCH) { tdsigwakeup(td, sig, action, intrval); goto out; } MPASS(action == SIG_DFL); if (prop & SA_STOP) { if (p->p_flag & (P_PPWAIT|P_WEXIT)) goto out; p->p_flag |= P_STOPPED_SIG; p->p_xstat = sig; PROC_SLOCK(p); sig_suspend_threads(td, p, 1); if (p->p_numthreads == p->p_suspcount) { /* * only thread sending signal to another * process can reach here, if thread is sending * signal to its process, because thread does * not suspend itself here, p_numthreads * should never be equal to p_suspcount. */ thread_stopped(p); PROC_SUNLOCK(p); sigqueue_delete_proc(p, p->p_xstat); } else PROC_SUNLOCK(p); goto out; } } else { /* Not in "NORMAL" state. discard the signal. */ sigqueue_delete(sigqueue, sig); goto out; } /* * The process is not stopped so we need to apply the signal to all the * running threads. */ runfast: tdsigwakeup(td, sig, action, intrval); PROC_SLOCK(p); thread_unsuspend(p); PROC_SUNLOCK(p); out: /* If we jump here, proc slock should not be owned. */ PROC_SLOCK_ASSERT(p, MA_NOTOWNED); return (ret); } /* * The force of a signal has been directed against a single * thread. We need to see what we can do about knocking it * out of any sleep it may be in etc. */ static void tdsigwakeup(struct thread *td, int sig, sig_t action, int intrval) { struct proc *p = td->td_proc; register int prop; int wakeup_swapper; wakeup_swapper = 0; PROC_LOCK_ASSERT(p, MA_OWNED); prop = sigprop(sig); PROC_SLOCK(p); thread_lock(td); /* * Bring the priority of a thread up if we want it to get * killed in this lifetime. */ if (action == SIG_DFL && (prop & SA_KILL) && td->td_priority > PUSER) sched_prio(td, PUSER); if (TD_ON_SLEEPQ(td)) { /* * If thread is sleeping uninterruptibly * we can't interrupt the sleep... the signal will * be noticed when the process returns through * trap() or syscall(). */ if ((td->td_flags & TDF_SINTR) == 0) goto out; /* * If SIGCONT is default (or ignored) and process is * asleep, we are finished; the process should not * be awakened. */ if ((prop & SA_CONT) && action == SIG_DFL) { thread_unlock(td); PROC_SUNLOCK(p); sigqueue_delete(&p->p_sigqueue, sig); /* * It may be on either list in this state. * Remove from both for now. */ sigqueue_delete(&td->td_sigqueue, sig); return; } /* * Don't awaken a sleeping thread for SIGSTOP if the * STOP signal is deferred. */ if ((prop & SA_STOP) && (td->td_flags & TDF_SBDRY)) goto out; /* * Give low priority threads a better chance to run. */ if (td->td_priority > PUSER) sched_prio(td, PUSER); wakeup_swapper = sleepq_abort(td, intrval); } else { /* * Other states do nothing with the signal immediately, * other than kicking ourselves if we are running. * It will either never be noticed, or noticed very soon. */ #ifdef SMP if (TD_IS_RUNNING(td) && td != curthread) forward_signal(td); #endif } out: PROC_SUNLOCK(p); thread_unlock(td); if (wakeup_swapper) kick_proc0(); } static void sig_suspend_threads(struct thread *td, struct proc *p, int sending) { struct thread *td2; PROC_LOCK_ASSERT(p, MA_OWNED); PROC_SLOCK_ASSERT(p, MA_OWNED); FOREACH_THREAD_IN_PROC(p, td2) { thread_lock(td2); td2->td_flags |= TDF_ASTPENDING | TDF_NEEDSUSPCHK; if ((TD_IS_SLEEPING(td2) || TD_IS_SWAPPED(td2)) && (td2->td_flags & TDF_SINTR)) { if (td2->td_flags & TDF_SBDRY) { /* * Once a thread is asleep with * TDF_SBDRY set, it should never * become suspended due to this check. */ KASSERT(!TD_IS_SUSPENDED(td2), ("thread with deferred stops suspended")); } else if (!TD_IS_SUSPENDED(td2)) { thread_suspend_one(td2); } } else if (!TD_IS_SUSPENDED(td2)) { if (sending || td != td2) td2->td_flags |= TDF_ASTPENDING; #ifdef SMP if (TD_IS_RUNNING(td2) && td2 != td) forward_signal(td2); #endif } thread_unlock(td2); } } int ptracestop(struct thread *td, int sig) { struct proc *p = td->td_proc; PROC_LOCK_ASSERT(p, MA_OWNED); KASSERT(!(p->p_flag & P_WEXIT), ("Stopping exiting process")); WITNESS_WARN(WARN_GIANTOK | WARN_SLEEPOK, &p->p_mtx.lock_object, "Stopping for traced signal"); td->td_dbgflags |= TDB_XSIG; td->td_xsig = sig; PROC_SLOCK(p); while ((p->p_flag & P_TRACED) && (td->td_dbgflags & TDB_XSIG)) { if (p->p_flag & P_SINGLE_EXIT) { td->td_dbgflags &= ~TDB_XSIG; PROC_SUNLOCK(p); return (sig); } /* * Just make wait() to work, the last stopped thread * will win. */ p->p_xstat = sig; p->p_xthread = td; p->p_flag |= (P_STOPPED_SIG|P_STOPPED_TRACE); sig_suspend_threads(td, p, 0); if ((td->td_dbgflags & TDB_STOPATFORK) != 0) { td->td_dbgflags &= ~TDB_STOPATFORK; cv_broadcast(&p->p_dbgwait); } stopme: - thread_suspend_switch(td); + thread_suspend_switch(td, p); if (p->p_xthread == td) p->p_xthread = NULL; if (!(p->p_flag & P_TRACED)) break; if (td->td_dbgflags & TDB_SUSPEND) { if (p->p_flag & P_SINGLE_EXIT) break; goto stopme; } } PROC_SUNLOCK(p); return (td->td_xsig); } static void reschedule_signals(struct proc *p, sigset_t block, int flags) { struct sigacts *ps; struct thread *td; int sig; PROC_LOCK_ASSERT(p, MA_OWNED); ps = p->p_sigacts; mtx_assert(&ps->ps_mtx, (flags & SIGPROCMASK_PS_LOCKED) != 0 ? MA_OWNED : MA_NOTOWNED); if (SIGISEMPTY(p->p_siglist)) return; SIGSETAND(block, p->p_siglist); while ((sig = sig_ffs(&block)) != 0) { SIGDELSET(block, sig); td = sigtd(p, sig, 0); signotify(td); if (!(flags & SIGPROCMASK_PS_LOCKED)) mtx_lock(&ps->ps_mtx); if (p->p_flag & P_TRACED || SIGISMEMBER(ps->ps_sigcatch, sig)) tdsigwakeup(td, sig, SIG_CATCH, (SIGISMEMBER(ps->ps_sigintr, sig) ? EINTR : ERESTART)); if (!(flags & SIGPROCMASK_PS_LOCKED)) mtx_unlock(&ps->ps_mtx); } } void tdsigcleanup(struct thread *td) { struct proc *p; sigset_t unblocked; p = td->td_proc; PROC_LOCK_ASSERT(p, MA_OWNED); sigqueue_flush(&td->td_sigqueue); if (p->p_numthreads == 1) return; /* * Since we cannot handle signals, notify signal post code * about this by filling the sigmask. * * Also, if needed, wake up thread(s) that do not block the * same signals as the exiting thread, since the thread might * have been selected for delivery and woken up. */ SIGFILLSET(unblocked); SIGSETNAND(unblocked, td->td_sigmask); SIGFILLSET(td->td_sigmask); reschedule_signals(p, unblocked, 0); } /* * Defer the delivery of SIGSTOP for the current thread. Returns true * if stops were deferred and false if they were already deferred. */ int sigdeferstop(void) { struct thread *td; td = curthread; if (td->td_flags & TDF_SBDRY) return (0); thread_lock(td); td->td_flags |= TDF_SBDRY; thread_unlock(td); return (1); } /* * Permit the delivery of SIGSTOP for the current thread. This does * not immediately suspend if a stop was posted. Instead, the thread * will suspend either via ast() or a subsequent interruptible sleep. */ void sigallowstop() { struct thread *td; td = curthread; thread_lock(td); td->td_flags &= ~TDF_SBDRY; thread_unlock(td); } /* * If the current process has received a signal (should be caught or cause * termination, should interrupt current syscall), return the signal number. * Stop signals with default action are processed immediately, then cleared; * they aren't returned. This is checked after each entry to the system for * a syscall or trap (though this can usually be done without calling issignal * by checking the pending signal masks in cursig.) The normal call * sequence is * * while (sig = cursig(curthread)) * postsig(sig); */ static int issignal(struct thread *td) { struct proc *p; struct sigacts *ps; struct sigqueue *queue; sigset_t sigpending; int sig, prop, newsig; p = td->td_proc; ps = p->p_sigacts; mtx_assert(&ps->ps_mtx, MA_OWNED); PROC_LOCK_ASSERT(p, MA_OWNED); for (;;) { int traced = (p->p_flag & P_TRACED) || (p->p_stops & S_SIG); sigpending = td->td_sigqueue.sq_signals; SIGSETOR(sigpending, p->p_sigqueue.sq_signals); SIGSETNAND(sigpending, td->td_sigmask); if (p->p_flag & P_PPWAIT || td->td_flags & TDF_SBDRY) SIG_STOPSIGMASK(sigpending); if (SIGISEMPTY(sigpending)) /* no signal to send */ return (0); sig = sig_ffs(&sigpending); if (p->p_stops & S_SIG) { mtx_unlock(&ps->ps_mtx); stopevent(p, S_SIG, sig); mtx_lock(&ps->ps_mtx); } /* * We should see pending but ignored signals * only if P_TRACED was on when they were posted. */ if (SIGISMEMBER(ps->ps_sigignore, sig) && (traced == 0)) { sigqueue_delete(&td->td_sigqueue, sig); sigqueue_delete(&p->p_sigqueue, sig); continue; } if (p->p_flag & P_TRACED && (p->p_flag & P_PPTRACE) == 0) { /* * If traced, always stop. * Remove old signal from queue before the stop. * XXX shrug off debugger, it causes siginfo to * be thrown away. */ queue = &td->td_sigqueue; td->td_dbgksi.ksi_signo = 0; if (sigqueue_get(queue, sig, &td->td_dbgksi) == 0) { queue = &p->p_sigqueue; sigqueue_get(queue, sig, &td->td_dbgksi); } mtx_unlock(&ps->ps_mtx); newsig = ptracestop(td, sig); mtx_lock(&ps->ps_mtx); if (sig != newsig) { /* * If parent wants us to take the signal, * then it will leave it in p->p_xstat; * otherwise we just look for signals again. */ if (newsig == 0) continue; sig = newsig; /* * Put the new signal into td_sigqueue. If the * signal is being masked, look for other * signals. */ sigqueue_add(queue, sig, NULL); if (SIGISMEMBER(td->td_sigmask, sig)) continue; signotify(td); } else { if (td->td_dbgksi.ksi_signo != 0) { td->td_dbgksi.ksi_flags |= KSI_HEAD; if (sigqueue_add(&td->td_sigqueue, sig, &td->td_dbgksi) != 0) td->td_dbgksi.ksi_signo = 0; } if (td->td_dbgksi.ksi_signo == 0) sigqueue_add(&td->td_sigqueue, sig, NULL); } /* * If the traced bit got turned off, go back up * to the top to rescan signals. This ensures * that p_sig* and p_sigact are consistent. */ if ((p->p_flag & P_TRACED) == 0) continue; } prop = sigprop(sig); /* * Decide whether the signal should be returned. * Return the signal's number, or fall through * to clear it from the pending mask. */ switch ((intptr_t)p->p_sigacts->ps_sigact[_SIG_IDX(sig)]) { case (intptr_t)SIG_DFL: /* * Don't take default actions on system processes. */ if (p->p_pid <= 1) { #ifdef DIAGNOSTIC /* * Are you sure you want to ignore SIGSEGV * in init? XXX */ printf("Process (pid %lu) got signal %d\n", (u_long)p->p_pid, sig); #endif break; /* == ignore */ } /* * If there is a pending stop signal to process * with default action, stop here, * then clear the signal. However, * if process is member of an orphaned * process group, ignore tty stop signals. */ if (prop & SA_STOP) { if (p->p_flag & (P_TRACED|P_WEXIT) || (p->p_pgrp->pg_jobc == 0 && prop & SA_TTYSTOP)) break; /* == ignore */ mtx_unlock(&ps->ps_mtx); WITNESS_WARN(WARN_GIANTOK | WARN_SLEEPOK, &p->p_mtx.lock_object, "Catching SIGSTOP"); p->p_flag |= P_STOPPED_SIG; p->p_xstat = sig; PROC_SLOCK(p); sig_suspend_threads(td, p, 0); - thread_suspend_switch(td); + thread_suspend_switch(td, p); PROC_SUNLOCK(p); mtx_lock(&ps->ps_mtx); break; } else if (prop & SA_IGNORE) { /* * Except for SIGCONT, shouldn't get here. * Default action is to ignore; drop it. */ break; /* == ignore */ } else return (sig); /*NOTREACHED*/ case (intptr_t)SIG_IGN: /* * Masking above should prevent us ever trying * to take action on an ignored signal other * than SIGCONT, unless process is traced. */ if ((prop & SA_CONT) == 0 && (p->p_flag & P_TRACED) == 0) printf("issignal\n"); break; /* == ignore */ default: /* * This signal has an action, let * postsig() process it. */ return (sig); } sigqueue_delete(&td->td_sigqueue, sig); /* take the signal! */ sigqueue_delete(&p->p_sigqueue, sig); } /* NOTREACHED */ } void thread_stopped(struct proc *p) { int n; PROC_LOCK_ASSERT(p, MA_OWNED); PROC_SLOCK_ASSERT(p, MA_OWNED); n = p->p_suspcount; if (p == curproc) n++; if ((p->p_flag & P_STOPPED_SIG) && (n == p->p_numthreads)) { PROC_SUNLOCK(p); p->p_flag &= ~P_WAITED; PROC_LOCK(p->p_pptr); childproc_stopped(p, (p->p_flag & P_TRACED) ? CLD_TRAPPED : CLD_STOPPED); PROC_UNLOCK(p->p_pptr); PROC_SLOCK(p); } } /* * Take the action for the specified signal * from the current set of pending signals. */ int postsig(sig) register int sig; { struct thread *td = curthread; register struct proc *p = td->td_proc; struct sigacts *ps; sig_t action; ksiginfo_t ksi; sigset_t returnmask; KASSERT(sig != 0, ("postsig")); PROC_LOCK_ASSERT(p, MA_OWNED); ps = p->p_sigacts; mtx_assert(&ps->ps_mtx, MA_OWNED); ksiginfo_init(&ksi); if (sigqueue_get(&td->td_sigqueue, sig, &ksi) == 0 && sigqueue_get(&p->p_sigqueue, sig, &ksi) == 0) return (0); ksi.ksi_signo = sig; if (ksi.ksi_code == SI_TIMER) itimer_accept(p, ksi.ksi_timerid, &ksi); action = ps->ps_sigact[_SIG_IDX(sig)]; #ifdef KTRACE if (KTRPOINT(td, KTR_PSIG)) ktrpsig(sig, action, td->td_pflags & TDP_OLDMASK ? &td->td_oldsigmask : &td->td_sigmask, ksi.ksi_code); #endif if (p->p_stops & S_SIG) { mtx_unlock(&ps->ps_mtx); stopevent(p, S_SIG, sig); mtx_lock(&ps->ps_mtx); } if (action == SIG_DFL) { /* * Default action, where the default is to kill * the process. (Other cases were ignored above.) */ mtx_unlock(&ps->ps_mtx); sigexit(td, sig); /* NOTREACHED */ } else { /* * If we get here, the signal must be caught. */ KASSERT(action != SIG_IGN && !SIGISMEMBER(td->td_sigmask, sig), ("postsig action")); /* * Set the new mask value and also defer further * occurrences of this signal. * * Special case: user has done a sigsuspend. Here the * current mask is not of interest, but rather the * mask from before the sigsuspend is what we want * restored after the signal processing is completed. */ if (td->td_pflags & TDP_OLDMASK) { returnmask = td->td_oldsigmask; td->td_pflags &= ~TDP_OLDMASK; } else returnmask = td->td_sigmask; if (p->p_sig == sig) { p->p_code = 0; p->p_sig = 0; } (*p->p_sysent->sv_sendsig)(action, &ksi, &returnmask); postsig_done(sig, td, ps); } return (1); } /* * Kill the current process for stated reason. */ void killproc(p, why) struct proc *p; char *why; { PROC_LOCK_ASSERT(p, MA_OWNED); CTR3(KTR_PROC, "killproc: proc %p (pid %d, %s)", p, p->p_pid, p->p_comm); log(LOG_ERR, "pid %d (%s), uid %d, was killed: %s\n", p->p_pid, p->p_comm, p->p_ucred ? p->p_ucred->cr_uid : -1, why); p->p_flag |= P_WKILLED; kern_psignal(p, SIGKILL); } /* * Force the current process to exit with the specified signal, dumping core * if appropriate. We bypass the normal tests for masked and caught signals, * allowing unrecoverable failures to terminate the process without changing * signal state. Mark the accounting record with the signal termination. * If dumping core, save the signal number for the debugger. Calls exit and * does not return. */ void sigexit(td, sig) struct thread *td; int sig; { struct proc *p = td->td_proc; PROC_LOCK_ASSERT(p, MA_OWNED); p->p_acflag |= AXSIG; /* * We must be single-threading to generate a core dump. This * ensures that the registers in the core file are up-to-date. * Also, the ELF dump handler assumes that the thread list doesn't * change out from under it. * * XXX If another thread attempts to single-thread before us * (e.g. via fork()), we won't get a dump at all. */ - if ((sigprop(sig) & SA_CORE) && (thread_single(SINGLE_NO_EXIT) == 0)) { + if ((sigprop(sig) & SA_CORE) && thread_single(p, SINGLE_NO_EXIT) == 0) { p->p_sig = sig; /* * Log signals which would cause core dumps * (Log as LOG_INFO to appease those who don't want * these messages.) * XXX : Todo, as well as euid, write out ruid too * Note that coredump() drops proc lock. */ if (coredump(td) == 0) sig |= WCOREFLAG; if (kern_logsigexit) log(LOG_INFO, "pid %d (%s), uid %d: exited on signal %d%s\n", p->p_pid, p->p_comm, td->td_ucred ? td->td_ucred->cr_uid : -1, sig &~ WCOREFLAG, sig & WCOREFLAG ? " (core dumped)" : ""); } else PROC_UNLOCK(p); exit1(td, W_EXITCODE(0, sig)); /* NOTREACHED */ } /* * Send queued SIGCHLD to parent when child process's state * is changed. */ static void sigparent(struct proc *p, int reason, int status) { PROC_LOCK_ASSERT(p, MA_OWNED); PROC_LOCK_ASSERT(p->p_pptr, MA_OWNED); if (p->p_ksi != NULL) { p->p_ksi->ksi_signo = SIGCHLD; p->p_ksi->ksi_code = reason; p->p_ksi->ksi_status = status; p->p_ksi->ksi_pid = p->p_pid; p->p_ksi->ksi_uid = p->p_ucred->cr_ruid; if (KSI_ONQ(p->p_ksi)) return; } pksignal(p->p_pptr, SIGCHLD, p->p_ksi); } static void childproc_jobstate(struct proc *p, int reason, int sig) { struct sigacts *ps; PROC_LOCK_ASSERT(p, MA_OWNED); PROC_LOCK_ASSERT(p->p_pptr, MA_OWNED); /* * Wake up parent sleeping in kern_wait(), also send * SIGCHLD to parent, but SIGCHLD does not guarantee * that parent will awake, because parent may masked * the signal. */ p->p_pptr->p_flag |= P_STATCHILD; wakeup(p->p_pptr); ps = p->p_pptr->p_sigacts; mtx_lock(&ps->ps_mtx); if ((ps->ps_flag & PS_NOCLDSTOP) == 0) { mtx_unlock(&ps->ps_mtx); sigparent(p, reason, sig); } else mtx_unlock(&ps->ps_mtx); } void childproc_stopped(struct proc *p, int reason) { /* p_xstat is a plain signal number, not a full wait() status here. */ childproc_jobstate(p, reason, p->p_xstat); } void childproc_continued(struct proc *p) { childproc_jobstate(p, CLD_CONTINUED, SIGCONT); } void childproc_exited(struct proc *p) { int reason; int xstat = p->p_xstat; /* convert to int */ int status; if (WCOREDUMP(xstat)) reason = CLD_DUMPED, status = WTERMSIG(xstat); else if (WIFSIGNALED(xstat)) reason = CLD_KILLED, status = WTERMSIG(xstat); else reason = CLD_EXITED, status = WEXITSTATUS(xstat); /* * XXX avoid calling wakeup(p->p_pptr), the work is * done in exit1(). */ sigparent(p, reason, status); } /* * We only have 1 character for the core count in the format * string, so the range will be 0-9 */ #define MAX_NUM_CORES 10 static int num_cores = 5; static int sysctl_debug_num_cores_check (SYSCTL_HANDLER_ARGS) { int error; int new_val; new_val = num_cores; error = sysctl_handle_int(oidp, &new_val, 0, req); if (error != 0 || req->newptr == NULL) return (error); if (new_val > MAX_NUM_CORES) new_val = MAX_NUM_CORES; if (new_val < 0) new_val = 0; num_cores = new_val; return (0); } SYSCTL_PROC(_debug, OID_AUTO, ncores, CTLTYPE_INT|CTLFLAG_RW, 0, sizeof(int), sysctl_debug_num_cores_check, "I", ""); #if defined(COMPRESS_USER_CORES) int compress_user_cores = 1; SYSCTL_INT(_kern, OID_AUTO, compress_user_cores, CTLFLAG_RW, &compress_user_cores, 0, "Compression of user corefiles"); int compress_user_cores_gzlevel = -1; /* default level */ SYSCTL_INT(_kern, OID_AUTO, compress_user_cores_gzlevel, CTLFLAG_RW, &compress_user_cores_gzlevel, -1, "Corefile gzip compression level"); #define GZ_SUFFIX ".gz" #define GZ_SUFFIX_LEN 3 #endif static char corefilename[MAXPATHLEN] = {"%N.core"}; SYSCTL_STRING(_kern, OID_AUTO, corefile, CTLFLAG_RWTUN, corefilename, sizeof(corefilename), "Process corefile name format string"); /* * corefile_open(comm, uid, pid, td, compress, vpp, namep) * Expand the name described in corefilename, using name, uid, and pid * and open/create core file. * corefilename is a printf-like string, with three format specifiers: * %N name of process ("name") * %P process id (pid) * %U user id (uid) * For example, "%N.core" is the default; they can be disabled completely * by using "/dev/null", or all core files can be stored in "/cores/%U/%N-%P". * This is controlled by the sysctl variable kern.corefile (see above). */ static int corefile_open(const char *comm, uid_t uid, pid_t pid, struct thread *td, int compress, struct vnode **vpp, char **namep) { struct nameidata nd; struct sbuf sb; const char *format; char *hostname, *name; int indexpos, i, error, cmode, flags, oflags; hostname = NULL; format = corefilename; name = malloc(MAXPATHLEN, M_TEMP, M_WAITOK | M_ZERO); indexpos = -1; (void)sbuf_new(&sb, name, MAXPATHLEN, SBUF_FIXEDLEN); for (i = 0; format[i] != '\0'; i++) { switch (format[i]) { case '%': /* Format character */ i++; switch (format[i]) { case '%': sbuf_putc(&sb, '%'); break; case 'H': /* hostname */ if (hostname == NULL) { hostname = malloc(MAXHOSTNAMELEN, M_TEMP, M_WAITOK); } getcredhostname(td->td_ucred, hostname, MAXHOSTNAMELEN); sbuf_printf(&sb, "%s", hostname); break; case 'I': /* autoincrementing index */ sbuf_printf(&sb, "0"); indexpos = sbuf_len(&sb) - 1; break; case 'N': /* process name */ sbuf_printf(&sb, "%s", comm); break; case 'P': /* process id */ sbuf_printf(&sb, "%u", pid); break; case 'U': /* user id */ sbuf_printf(&sb, "%u", uid); break; default: log(LOG_ERR, "Unknown format character %c in " "corename `%s'\n", format[i], format); break; } break; default: sbuf_putc(&sb, format[i]); break; } } free(hostname, M_TEMP); #ifdef COMPRESS_USER_CORES if (compress) sbuf_printf(&sb, GZ_SUFFIX); #endif if (sbuf_error(&sb) != 0) { log(LOG_ERR, "pid %ld (%s), uid (%lu): corename is too " "long\n", (long)pid, comm, (u_long)uid); sbuf_delete(&sb); free(name, M_TEMP); return (ENOMEM); } sbuf_finish(&sb); sbuf_delete(&sb); cmode = S_IRUSR | S_IWUSR; oflags = VN_OPEN_NOAUDIT | (capmode_coredump ? VN_OPEN_NOCAPCHECK : 0); /* * If the core format has a %I in it, then we need to check * for existing corefiles before returning a name. * To do this we iterate over 0..num_cores to find a * non-existing core file name to use. */ if (indexpos != -1) { for (i = 0; i < num_cores; i++) { flags = O_CREAT | O_EXCL | FWRITE | O_NOFOLLOW; name[indexpos] = '0' + i; NDINIT(&nd, LOOKUP, NOFOLLOW, UIO_SYSSPACE, name, td); error = vn_open_cred(&nd, &flags, cmode, oflags, td->td_ucred, NULL); if (error) { if (error == EEXIST) continue; log(LOG_ERR, "pid %d (%s), uid (%u): Path `%s' failed " "on initial open test, error = %d\n", pid, comm, uid, name, error); } goto out; } } flags = O_CREAT | FWRITE | O_NOFOLLOW; NDINIT(&nd, LOOKUP, NOFOLLOW, UIO_SYSSPACE, name, td); error = vn_open_cred(&nd, &flags, cmode, oflags, td->td_ucred, NULL); out: if (error) { #ifdef AUDIT audit_proc_coredump(td, name, error); #endif free(name, M_TEMP); return (error); } NDFREE(&nd, NDF_ONLY_PNBUF); *vpp = nd.ni_vp; *namep = name; return (0); } /* * Dump a process' core. The main routine does some * policy checking, and creates the name of the coredump; * then it passes on a vnode and a size limit to the process-specific * coredump routine if there is one; if there _is not_ one, it returns * ENOSYS; otherwise it returns the error from the process-specific routine. */ static int coredump(struct thread *td) { struct proc *p = td->td_proc; struct ucred *cred = td->td_ucred; struct vnode *vp; struct flock lf; struct vattr vattr; int error, error1, locked; char *name; /* name of corefile */ void *rl_cookie; off_t limit; int compress; #ifdef COMPRESS_USER_CORES compress = compress_user_cores; #else compress = 0; #endif PROC_LOCK_ASSERT(p, MA_OWNED); MPASS((p->p_flag & P_HADTHREADS) == 0 || p->p_singlethread == td); _STOPEVENT(p, S_CORE, 0); if (!do_coredump || (!sugid_coredump && (p->p_flag & P_SUGID) != 0)) { PROC_UNLOCK(p); return (EFAULT); } /* * Note that the bulk of limit checking is done after * the corefile is created. The exception is if the limit * for corefiles is 0, in which case we don't bother * creating the corefile at all. This layout means that * a corefile is truncated instead of not being created, * if it is larger than the limit. */ limit = (off_t)lim_cur(p, RLIMIT_CORE); if (limit == 0 || racct_get_available(p, RACCT_CORE) == 0) { PROC_UNLOCK(p); return (EFBIG); } PROC_UNLOCK(p); error = corefile_open(p->p_comm, cred->cr_uid, p->p_pid, td, compress, &vp, &name); if (error != 0) return (error); /* * Don't dump to non-regular files or files with links. * Do not dump into system files. */ if (vp->v_type != VREG || VOP_GETATTR(vp, &vattr, cred) != 0 || vattr.va_nlink != 1 || (vp->v_vflag & VV_SYSTEM) != 0) { VOP_UNLOCK(vp, 0); error = EFAULT; goto close; } VOP_UNLOCK(vp, 0); /* Postpone other writers, including core dumps of other processes. */ rl_cookie = vn_rangelock_wlock(vp, 0, OFF_MAX); lf.l_whence = SEEK_SET; lf.l_start = 0; lf.l_len = 0; lf.l_type = F_WRLCK; locked = (VOP_ADVLOCK(vp, (caddr_t)p, F_SETLK, &lf, F_FLOCK) == 0); VATTR_NULL(&vattr); vattr.va_size = 0; if (set_core_nodump_flag) vattr.va_flags = UF_NODUMP; vn_lock(vp, LK_EXCLUSIVE | LK_RETRY); VOP_SETATTR(vp, &vattr, cred); VOP_UNLOCK(vp, 0); PROC_LOCK(p); p->p_acflag |= ACORE; PROC_UNLOCK(p); if (p->p_sysent->sv_coredump != NULL) { error = p->p_sysent->sv_coredump(td, vp, limit, compress ? IMGACT_CORE_COMPRESS : 0); } else { error = ENOSYS; } if (locked) { lf.l_type = F_UNLCK; VOP_ADVLOCK(vp, (caddr_t)p, F_UNLCK, &lf, F_FLOCK); } vn_rangelock_unlock(vp, rl_cookie); close: error1 = vn_close(vp, FWRITE, cred, td); if (error == 0) error = error1; #ifdef AUDIT audit_proc_coredump(td, name, error); #endif free(name, M_TEMP); return (error); } /* * Nonexistent system call-- signal process (may want to handle it). Flag * error in case process won't see signal immediately (blocked or ignored). */ #ifndef _SYS_SYSPROTO_H_ struct nosys_args { int dummy; }; #endif /* ARGSUSED */ int nosys(td, args) struct thread *td; struct nosys_args *args; { struct proc *p = td->td_proc; PROC_LOCK(p); tdsignal(td, SIGSYS); PROC_UNLOCK(p); return (ENOSYS); } /* * Send a SIGIO or SIGURG signal to a process or process group using stored * credentials rather than those of the current process. */ void pgsigio(sigiop, sig, checkctty) struct sigio **sigiop; int sig, checkctty; { ksiginfo_t ksi; struct sigio *sigio; ksiginfo_init(&ksi); ksi.ksi_signo = sig; ksi.ksi_code = SI_KERNEL; SIGIO_LOCK(); sigio = *sigiop; if (sigio == NULL) { SIGIO_UNLOCK(); return; } if (sigio->sio_pgid > 0) { PROC_LOCK(sigio->sio_proc); if (CANSIGIO(sigio->sio_ucred, sigio->sio_proc->p_ucred)) kern_psignal(sigio->sio_proc, sig); PROC_UNLOCK(sigio->sio_proc); } else if (sigio->sio_pgid < 0) { struct proc *p; PGRP_LOCK(sigio->sio_pgrp); LIST_FOREACH(p, &sigio->sio_pgrp->pg_members, p_pglist) { PROC_LOCK(p); if (p->p_state == PRS_NORMAL && CANSIGIO(sigio->sio_ucred, p->p_ucred) && (checkctty == 0 || (p->p_flag & P_CONTROLT))) kern_psignal(p, sig); PROC_UNLOCK(p); } PGRP_UNLOCK(sigio->sio_pgrp); } SIGIO_UNLOCK(); } static int filt_sigattach(struct knote *kn) { struct proc *p = curproc; kn->kn_ptr.p_proc = p; kn->kn_flags |= EV_CLEAR; /* automatically set */ knlist_add(&p->p_klist, kn, 0); return (0); } static void filt_sigdetach(struct knote *kn) { struct proc *p = kn->kn_ptr.p_proc; knlist_remove(&p->p_klist, kn, 0); } /* * signal knotes are shared with proc knotes, so we apply a mask to * the hint in order to differentiate them from process hints. This * could be avoided by using a signal-specific knote list, but probably * isn't worth the trouble. */ static int filt_signal(struct knote *kn, long hint) { if (hint & NOTE_SIGNAL) { hint &= ~NOTE_SIGNAL; if (kn->kn_id == hint) kn->kn_data++; } return (kn->kn_data != 0); } struct sigacts * sigacts_alloc(void) { struct sigacts *ps; ps = malloc(sizeof(struct sigacts), M_SUBPROC, M_WAITOK | M_ZERO); refcount_init(&ps->ps_refcnt, 1); mtx_init(&ps->ps_mtx, "sigacts", NULL, MTX_DEF); return (ps); } void sigacts_free(struct sigacts *ps) { if (refcount_release(&ps->ps_refcnt) == 0) return; mtx_destroy(&ps->ps_mtx); free(ps, M_SUBPROC); } struct sigacts * sigacts_hold(struct sigacts *ps) { refcount_acquire(&ps->ps_refcnt); return (ps); } void sigacts_copy(struct sigacts *dest, struct sigacts *src) { KASSERT(dest->ps_refcnt == 1, ("sigacts_copy to shared dest")); mtx_lock(&src->ps_mtx); bcopy(src, dest, offsetof(struct sigacts, ps_refcnt)); mtx_unlock(&src->ps_mtx); } int sigacts_shared(struct sigacts *ps) { return (ps->ps_refcnt > 1); } Index: projects/clang350-import/sys/kern/kern_thread.c =================================================================== --- projects/clang350-import/sys/kern/kern_thread.c (revision 275748) +++ projects/clang350-import/sys/kern/kern_thread.c (revision 275749) @@ -1,1069 +1,1112 @@ /*- * Copyright (C) 2001 Julian Elischer . * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice(s), this list of conditions and the following disclaimer as * the first lines of this file unmodified other than the possible * addition of one or more copyright notices. * 2. Redistributions in binary form must reproduce the above copyright * notice(s), this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER(S) ``AS IS'' AND ANY * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE * DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER(S) BE LIABLE FOR ANY * DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR * SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER * CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH * DAMAGE. */ #include "opt_witness.h" #include "opt_hwpmc_hooks.h" #include __FBSDID("$FreeBSD$"); #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #ifdef HWPMC_HOOKS #include #endif #include #include #include #include #include SDT_PROVIDER_DECLARE(proc); SDT_PROBE_DEFINE(proc, , , lwp__exit); /* * thread related storage. */ static uma_zone_t thread_zone; TAILQ_HEAD(, thread) zombie_threads = TAILQ_HEAD_INITIALIZER(zombie_threads); static struct mtx zombie_lock; MTX_SYSINIT(zombie_lock, &zombie_lock, "zombie lock", MTX_SPIN); static void thread_zombie(struct thread *); #define TID_BUFFER_SIZE 1024 struct mtx tid_lock; static struct unrhdr *tid_unrhdr; static lwpid_t tid_buffer[TID_BUFFER_SIZE]; static int tid_head, tid_tail; static MALLOC_DEFINE(M_TIDHASH, "tidhash", "thread hash"); struct tidhashhead *tidhashtbl; u_long tidhash; struct rwlock tidhash_lock; static lwpid_t tid_alloc(void) { lwpid_t tid; tid = alloc_unr(tid_unrhdr); if (tid != -1) return (tid); mtx_lock(&tid_lock); if (tid_head == tid_tail) { mtx_unlock(&tid_lock); return (-1); } tid = tid_buffer[tid_head]; tid_head = (tid_head + 1) % TID_BUFFER_SIZE; mtx_unlock(&tid_lock); return (tid); } static void tid_free(lwpid_t tid) { lwpid_t tmp_tid = -1; mtx_lock(&tid_lock); if ((tid_tail + 1) % TID_BUFFER_SIZE == tid_head) { tmp_tid = tid_buffer[tid_head]; tid_head = (tid_head + 1) % TID_BUFFER_SIZE; } tid_buffer[tid_tail] = tid; tid_tail = (tid_tail + 1) % TID_BUFFER_SIZE; mtx_unlock(&tid_lock); if (tmp_tid != -1) free_unr(tid_unrhdr, tmp_tid); } /* * Prepare a thread for use. */ static int thread_ctor(void *mem, int size, void *arg, int flags) { struct thread *td; td = (struct thread *)mem; td->td_state = TDS_INACTIVE; td->td_oncpu = NOCPU; td->td_tid = tid_alloc(); /* * Note that td_critnest begins life as 1 because the thread is not * running and is thereby implicitly waiting to be on the receiving * end of a context switch. */ td->td_critnest = 1; td->td_lend_user_pri = PRI_MAX; EVENTHANDLER_INVOKE(thread_ctor, td); #ifdef AUDIT audit_thread_alloc(td); #endif umtx_thread_alloc(td); return (0); } /* * Reclaim a thread after use. */ static void thread_dtor(void *mem, int size, void *arg) { struct thread *td; td = (struct thread *)mem; #ifdef INVARIANTS /* Verify that this thread is in a safe state to free. */ switch (td->td_state) { case TDS_INHIBITED: case TDS_RUNNING: case TDS_CAN_RUN: case TDS_RUNQ: /* * We must never unlink a thread that is in one of * these states, because it is currently active. */ panic("bad state for thread unlinking"); /* NOTREACHED */ case TDS_INACTIVE: break; default: panic("bad thread state"); /* NOTREACHED */ } #endif #ifdef AUDIT audit_thread_free(td); #endif /* Free all OSD associated to this thread. */ osd_thread_exit(td); EVENTHANDLER_INVOKE(thread_dtor, td); tid_free(td->td_tid); } /* * Initialize type-stable parts of a thread (when newly created). */ static int thread_init(void *mem, int size, int flags) { struct thread *td; td = (struct thread *)mem; td->td_sleepqueue = sleepq_alloc(); td->td_turnstile = turnstile_alloc(); td->td_rlqe = NULL; EVENTHANDLER_INVOKE(thread_init, td); td->td_sched = (struct td_sched *)&td[1]; umtx_thread_init(td); td->td_kstack = 0; return (0); } /* * Tear down type-stable parts of a thread (just before being discarded). */ static void thread_fini(void *mem, int size) { struct thread *td; td = (struct thread *)mem; EVENTHANDLER_INVOKE(thread_fini, td); rlqentry_free(td->td_rlqe); turnstile_free(td->td_turnstile); sleepq_free(td->td_sleepqueue); umtx_thread_fini(td); seltdfini(td); } /* * For a newly created process, * link up all the structures and its initial threads etc. * called from: * {arch}/{arch}/machdep.c {arch}_init(), init386() etc. * proc_dtor() (should go away) * proc_init() */ void proc_linkup0(struct proc *p, struct thread *td) { TAILQ_INIT(&p->p_threads); /* all threads in proc */ proc_linkup(p, td); } void proc_linkup(struct proc *p, struct thread *td) { sigqueue_init(&p->p_sigqueue, p); p->p_ksi = ksiginfo_alloc(1); if (p->p_ksi != NULL) { /* XXX p_ksi may be null if ksiginfo zone is not ready */ p->p_ksi->ksi_flags = KSI_EXT | KSI_INS; } LIST_INIT(&p->p_mqnotifier); p->p_numthreads = 0; thread_link(td, p); } /* * Initialize global thread allocation resources. */ void threadinit(void) { mtx_init(&tid_lock, "TID lock", NULL, MTX_DEF); /* * pid_max cannot be greater than PID_MAX. * leave one number for thread0. */ tid_unrhdr = new_unrhdr(PID_MAX + 2, INT_MAX, &tid_lock); thread_zone = uma_zcreate("THREAD", sched_sizeof_thread(), thread_ctor, thread_dtor, thread_init, thread_fini, 16 - 1, 0); tidhashtbl = hashinit(maxproc / 2, M_TIDHASH, &tidhash); rw_init(&tidhash_lock, "tidhash"); } /* * Place an unused thread on the zombie list. * Use the slpq as that must be unused by now. */ void thread_zombie(struct thread *td) { mtx_lock_spin(&zombie_lock); TAILQ_INSERT_HEAD(&zombie_threads, td, td_slpq); mtx_unlock_spin(&zombie_lock); } /* * Release a thread that has exited after cpu_throw(). */ void thread_stash(struct thread *td) { atomic_subtract_rel_int(&td->td_proc->p_exitthreads, 1); thread_zombie(td); } /* * Reap zombie resources. */ void thread_reap(void) { struct thread *td_first, *td_next; /* * Don't even bother to lock if none at this instant, * we really don't care about the next instant.. */ if (!TAILQ_EMPTY(&zombie_threads)) { mtx_lock_spin(&zombie_lock); td_first = TAILQ_FIRST(&zombie_threads); if (td_first) TAILQ_INIT(&zombie_threads); mtx_unlock_spin(&zombie_lock); while (td_first) { td_next = TAILQ_NEXT(td_first, td_slpq); if (td_first->td_ucred) crfree(td_first->td_ucred); thread_free(td_first); td_first = td_next; } } } /* * Allocate a thread. */ struct thread * thread_alloc(int pages) { struct thread *td; thread_reap(); /* check if any zombies to get */ td = (struct thread *)uma_zalloc(thread_zone, M_WAITOK); KASSERT(td->td_kstack == 0, ("thread_alloc got thread with kstack")); if (!vm_thread_new(td, pages)) { uma_zfree(thread_zone, td); return (NULL); } cpu_thread_alloc(td); return (td); } int thread_alloc_stack(struct thread *td, int pages) { KASSERT(td->td_kstack == 0, ("thread_alloc_stack called on a thread with kstack")); if (!vm_thread_new(td, pages)) return (0); cpu_thread_alloc(td); return (1); } /* * Deallocate a thread. */ void thread_free(struct thread *td) { lock_profile_thread_exit(td); if (td->td_cpuset) cpuset_rel(td->td_cpuset); td->td_cpuset = NULL; cpu_thread_free(td); if (td->td_kstack != 0) vm_thread_dispose(td); uma_zfree(thread_zone, td); } /* * Discard the current thread and exit from its context. * Always called with scheduler locked. * * Because we can't free a thread while we're operating under its context, * push the current thread into our CPU's deadthread holder. This means * we needn't worry about someone else grabbing our context before we * do a cpu_throw(). */ void thread_exit(void) { uint64_t runtime, new_switchtime; struct thread *td; struct thread *td2; struct proc *p; int wakeup_swapper; td = curthread; p = td->td_proc; PROC_SLOCK_ASSERT(p, MA_OWNED); mtx_assert(&Giant, MA_NOTOWNED); PROC_LOCK_ASSERT(p, MA_OWNED); KASSERT(p != NULL, ("thread exiting without a process")); CTR3(KTR_PROC, "thread_exit: thread %p (pid %ld, %s)", td, (long)p->p_pid, td->td_name); KASSERT(TAILQ_EMPTY(&td->td_sigqueue.sq_list), ("signal pending")); #ifdef AUDIT AUDIT_SYSCALL_EXIT(0, td); #endif umtx_thread_exit(td); /* * drop FPU & debug register state storage, or any other * architecture specific resources that * would not be on a new untouched process. */ cpu_thread_exit(td); /* XXXSMP */ /* * The last thread is left attached to the process * So that the whole bundle gets recycled. Skip * all this stuff if we never had threads. * EXIT clears all sign of other threads when * it goes to single threading, so the last thread always * takes the short path. */ if (p->p_flag & P_HADTHREADS) { if (p->p_numthreads > 1) { atomic_add_int(&td->td_proc->p_exitthreads, 1); thread_unlink(td); td2 = FIRST_THREAD_IN_PROC(p); sched_exit_thread(td2, td); /* * The test below is NOT true if we are the * sole exiting thread. P_STOPPED_SINGLE is unset * in exit1() after it is the only survivor. */ if (P_SHOULDSTOP(p) == P_STOPPED_SINGLE) { if (p->p_numthreads == p->p_suspcount) { thread_lock(p->p_singlethread); wakeup_swapper = thread_unsuspend_one( - p->p_singlethread); + p->p_singlethread, p); thread_unlock(p->p_singlethread); if (wakeup_swapper) kick_proc0(); } } PCPU_SET(deadthread, td); } else { /* * The last thread is exiting.. but not through exit() */ panic ("thread_exit: Last thread exiting on its own"); } } #ifdef HWPMC_HOOKS /* * If this thread is part of a process that is being tracked by hwpmc(4), * inform the module of the thread's impending exit. */ if (PMC_PROC_IS_USING_PMCS(td->td_proc)) PMC_SWITCH_CONTEXT(td, PMC_FN_CSW_OUT); #endif PROC_UNLOCK(p); PROC_STATLOCK(p); thread_lock(td); PROC_SUNLOCK(p); /* Do the same timestamp bookkeeping that mi_switch() would do. */ new_switchtime = cpu_ticks(); runtime = new_switchtime - PCPU_GET(switchtime); td->td_runtime += runtime; td->td_incruntime += runtime; PCPU_SET(switchtime, new_switchtime); PCPU_SET(switchticks, ticks); PCPU_INC(cnt.v_swtch); /* Save our resource usage in our process. */ td->td_ru.ru_nvcsw++; ruxagg(p, td); rucollect(&p->p_ru, &td->td_ru); PROC_STATUNLOCK(p); td->td_state = TDS_INACTIVE; #ifdef WITNESS witness_thread_exit(td); #endif CTR1(KTR_PROC, "thread_exit: cpu_throw() thread %p", td); sched_throw(td); panic("I'm a teapot!"); /* NOTREACHED */ } /* * Do any thread specific cleanups that may be needed in wait() * called with Giant, proc and schedlock not held. */ void thread_wait(struct proc *p) { struct thread *td; mtx_assert(&Giant, MA_NOTOWNED); KASSERT(p->p_numthreads == 1, ("multiple threads in thread_wait()")); KASSERT(p->p_exitthreads == 0, ("p_exitthreads leaking")); td = FIRST_THREAD_IN_PROC(p); /* Lock the last thread so we spin until it exits cpu_throw(). */ thread_lock(td); thread_unlock(td); lock_profile_thread_exit(td); cpuset_rel(td->td_cpuset); td->td_cpuset = NULL; cpu_thread_clean(td); crfree(td->td_ucred); thread_reap(); /* check for zombie threads etc. */ } /* * Link a thread to a process. * set up anything that needs to be initialized for it to * be used by the process. */ void thread_link(struct thread *td, struct proc *p) { /* * XXX This can't be enabled because it's called for proc0 before * its lock has been created. * PROC_LOCK_ASSERT(p, MA_OWNED); */ td->td_state = TDS_INACTIVE; td->td_proc = p; td->td_flags = TDF_INMEM; LIST_INIT(&td->td_contested); LIST_INIT(&td->td_lprof[0]); LIST_INIT(&td->td_lprof[1]); sigqueue_init(&td->td_sigqueue, p); callout_init(&td->td_slpcallout, CALLOUT_MPSAFE); TAILQ_INSERT_TAIL(&p->p_threads, td, td_plist); p->p_numthreads++; } /* * Called from: * thread_exit() */ void thread_unlink(struct thread *td) { struct proc *p = td->td_proc; PROC_LOCK_ASSERT(p, MA_OWNED); TAILQ_REMOVE(&p->p_threads, td, td_plist); p->p_numthreads--; /* could clear a few other things here */ /* Must NOT clear links to proc! */ } static int calc_remaining(struct proc *p, int mode) { int remaining; PROC_LOCK_ASSERT(p, MA_OWNED); PROC_SLOCK_ASSERT(p, MA_OWNED); if (mode == SINGLE_EXIT) remaining = p->p_numthreads; else if (mode == SINGLE_BOUNDARY) remaining = p->p_numthreads - p->p_boundary_count; - else if (mode == SINGLE_NO_EXIT) + else if (mode == SINGLE_NO_EXIT || mode == SINGLE_ALLPROC) remaining = p->p_numthreads - p->p_suspcount; else panic("calc_remaining: wrong mode %d", mode); return (remaining); } static int remain_for_mode(int mode) { - return (1); + return (mode == SINGLE_ALLPROC ? 0 : 1); } static int weed_inhib(int mode, struct thread *td2, struct proc *p) { int wakeup_swapper; PROC_LOCK_ASSERT(p, MA_OWNED); PROC_SLOCK_ASSERT(p, MA_OWNED); THREAD_LOCK_ASSERT(td2, MA_OWNED); wakeup_swapper = 0; switch (mode) { case SINGLE_EXIT: if (TD_IS_SUSPENDED(td2)) - wakeup_swapper |= thread_unsuspend_one(td2); + wakeup_swapper |= thread_unsuspend_one(td2, p); if (TD_ON_SLEEPQ(td2) && (td2->td_flags & TDF_SINTR) != 0) wakeup_swapper |= sleepq_abort(td2, EINTR); break; case SINGLE_BOUNDARY: if (TD_IS_SUSPENDED(td2) && (td2->td_flags & TDF_BOUNDARY) == 0) - wakeup_swapper |= thread_unsuspend_one(td2); + wakeup_swapper |= thread_unsuspend_one(td2, p); if (TD_ON_SLEEPQ(td2) && (td2->td_flags & TDF_SINTR) != 0) wakeup_swapper |= sleepq_abort(td2, ERESTART); break; case SINGLE_NO_EXIT: if (TD_IS_SUSPENDED(td2) && (td2->td_flags & TDF_BOUNDARY) == 0) - wakeup_swapper |= thread_unsuspend_one(td2); + wakeup_swapper |= thread_unsuspend_one(td2, p); if (TD_ON_SLEEPQ(td2) && (td2->td_flags & TDF_SINTR) != 0) wakeup_swapper |= sleepq_abort(td2, ERESTART); + case SINGLE_ALLPROC: + /* + * ALLPROC suspend tries to avoid spurious EINTR for + * threads sleeping interruptable, by suspending the + * thread directly, similarly to sig_suspend_threads(). + * Since such sleep is not performed at the user + * boundary, TDF_BOUNDARY flag is not set, and TDF_ALLPROCSUSP + * is used to avoid immediate un-suspend. + */ + if (TD_IS_SUSPENDED(td2) && (td2->td_flags & (TDF_BOUNDARY | + TDF_ALLPROCSUSP)) == 0) + wakeup_swapper |= thread_unsuspend_one(td2, p); + if (TD_ON_SLEEPQ(td2) && (td2->td_flags & TDF_SINTR) != 0) { + if ((td2->td_flags & TDF_SBDRY) == 0) { + thread_suspend_one(td2); + td2->td_flags |= TDF_ALLPROCSUSP; + } else { + wakeup_swapper |= sleepq_abort(td2, ERESTART); + } + } break; } return (wakeup_swapper); } /* * Enforce single-threading. * * Returns 1 if the caller must abort (another thread is waiting to * exit the process or similar). Process is locked! * Returns 0 when you are successfully the only thread running. * A process has successfully single threaded in the suspend mode when * There are no threads in user mode. Threads in the kernel must be * allowed to continue until they get to the user boundary. They may even * copy out their return values and data before suspending. They may however be * accelerated in reaching the user boundary as we will wake up * any sleeping threads that are interruptable. (PCATCH). */ int -thread_single(int mode) +thread_single(struct proc *p, int mode) { struct thread *td; struct thread *td2; - struct proc *p; int remaining, wakeup_swapper; td = curthread; - p = td->td_proc; + KASSERT(mode == SINGLE_EXIT || mode == SINGLE_BOUNDARY || + mode == SINGLE_ALLPROC || mode == SINGLE_NO_EXIT, + ("invalid mode %d", mode)); + /* + * If allowing non-ALLPROC singlethreading for non-curproc + * callers, calc_remaining() and remain_for_mode() should be + * adjusted to also account for td->td_proc != p. For now + * this is not implemented because it is not used. + */ + KASSERT((mode == SINGLE_ALLPROC && td->td_proc != p) || + (mode != SINGLE_ALLPROC && td->td_proc == p), + ("mode %d proc %p curproc %p", mode, p, td->td_proc)); mtx_assert(&Giant, MA_NOTOWNED); PROC_LOCK_ASSERT(p, MA_OWNED); - if ((p->p_flag & P_HADTHREADS) == 0) + if ((p->p_flag & P_HADTHREADS) == 0 && mode != SINGLE_ALLPROC) return (0); /* Is someone already single threading? */ if (p->p_singlethread != NULL && p->p_singlethread != td) return (1); if (mode == SINGLE_EXIT) { p->p_flag |= P_SINGLE_EXIT; p->p_flag &= ~P_SINGLE_BOUNDARY; } else { p->p_flag &= ~P_SINGLE_EXIT; if (mode == SINGLE_BOUNDARY) p->p_flag |= P_SINGLE_BOUNDARY; else p->p_flag &= ~P_SINGLE_BOUNDARY; } + if (mode == SINGLE_ALLPROC) + p->p_flag |= P_TOTAL_STOP; p->p_flag |= P_STOPPED_SINGLE; PROC_SLOCK(p); p->p_singlethread = td; remaining = calc_remaining(p, mode); while (remaining != remain_for_mode(mode)) { if (P_SHOULDSTOP(p) != P_STOPPED_SINGLE) goto stopme; wakeup_swapper = 0; FOREACH_THREAD_IN_PROC(p, td2) { if (td2 == td) continue; thread_lock(td2); td2->td_flags |= TDF_ASTPENDING | TDF_NEEDSUSPCHK; - if (TD_IS_INHIBITED(td2)) + if (TD_IS_INHIBITED(td2)) { wakeup_swapper |= weed_inhib(mode, td2, p); #ifdef SMP - else if (TD_IS_RUNNING(td2) && td != td2) { + } else if (TD_IS_RUNNING(td2) && td != td2) { forward_signal(td2); - } #endif + } thread_unlock(td2); } if (wakeup_swapper) kick_proc0(); remaining = calc_remaining(p, mode); /* * Maybe we suspended some threads.. was it enough? */ if (remaining == remain_for_mode(mode)) break; stopme: /* * Wake us up when everyone else has suspended. * In the mean time we suspend as well. */ - thread_suspend_switch(td); + thread_suspend_switch(td, p); remaining = calc_remaining(p, mode); } if (mode == SINGLE_EXIT) { /* * Convert the process to an unthreaded process. The * SINGLE_EXIT is called by exit1() or execve(), in * both cases other threads must be retired. */ KASSERT(p->p_numthreads == 1, ("Unthreading with >1 threads")); p->p_singlethread = NULL; p->p_flag &= ~(P_STOPPED_SINGLE | P_SINGLE_EXIT | P_HADTHREADS); /* * Wait for any remaining threads to exit cpu_throw(). */ while (p->p_exitthreads != 0) { PROC_SUNLOCK(p); PROC_UNLOCK(p); sched_relinquish(td); PROC_LOCK(p); PROC_SLOCK(p); } } PROC_SUNLOCK(p); return (0); } bool thread_suspend_check_needed(void) { struct proc *p; struct thread *td; td = curthread; p = td->td_proc; PROC_LOCK_ASSERT(p, MA_OWNED); return (P_SHOULDSTOP(p) || ((p->p_flag & P_TRACED) != 0 && (td->td_dbgflags & TDB_SUSPEND) != 0)); } /* * Called in from locations that can safely check to see * whether we have to suspend or at least throttle for a * single-thread event (e.g. fork). * * Such locations include userret(). * If the "return_instead" argument is non zero, the thread must be able to * accept 0 (caller may continue), or 1 (caller must abort) as a result. * * The 'return_instead' argument tells the function if it may do a * thread_exit() or suspend, or whether the caller must abort and back * out instead. * * If the thread that set the single_threading request has set the * P_SINGLE_EXIT bit in the process flags then this call will never return * if 'return_instead' is false, but will exit. * * P_SINGLE_EXIT | return_instead == 0| return_instead != 0 *---------------+--------------------+--------------------- * 0 | returns 0 | returns 0 or 1 * | when ST ends | immediately *---------------+--------------------+--------------------- * 1 | thread exits | returns 1 * | | immediately * 0 = thread_exit() or suspension ok, * other = return error instead of stopping the thread. * * While a full suspension is under effect, even a single threading * thread would be suspended if it made this call (but it shouldn't). * This call should only be made from places where * thread_exit() would be safe as that may be the outcome unless * return_instead is set. */ int thread_suspend_check(int return_instead) { struct thread *td; struct proc *p; int wakeup_swapper; td = curthread; p = td->td_proc; mtx_assert(&Giant, MA_NOTOWNED); PROC_LOCK_ASSERT(p, MA_OWNED); while (thread_suspend_check_needed()) { if (P_SHOULDSTOP(p) == P_STOPPED_SINGLE) { KASSERT(p->p_singlethread != NULL, ("singlethread not set")); /* * The only suspension in action is a * single-threading. Single threader need not stop. * XXX Should be safe to access unlocked * as it can only be set to be true by us. */ if (p->p_singlethread == td) return (0); /* Exempt from stopping. */ } if ((p->p_flag & P_SINGLE_EXIT) && return_instead) return (EINTR); /* Should we goto user boundary if we didn't come from there? */ if (P_SHOULDSTOP(p) == P_STOPPED_SINGLE && (p->p_flag & P_SINGLE_BOUNDARY) && return_instead) return (ERESTART); /* * Ignore suspend requests for stop signals if they * are deferred. */ - if (P_SHOULDSTOP(p) == P_STOPPED_SIG && - td->td_flags & TDF_SBDRY) { + if ((P_SHOULDSTOP(p) == P_STOPPED_SIG || + (p->p_flag & P_TOTAL_STOP) != 0) && + (td->td_flags & TDF_SBDRY) != 0) { KASSERT(return_instead, ("TDF_SBDRY set for unsafe thread_suspend_check")); return (0); } /* * If the process is waiting for us to exit, * this thread should just suicide. * Assumes that P_SINGLE_EXIT implies P_STOPPED_SINGLE. */ if ((p->p_flag & P_SINGLE_EXIT) && (p->p_singlethread != td)) { PROC_UNLOCK(p); tidhash_remove(td); PROC_LOCK(p); tdsigcleanup(td); PROC_SLOCK(p); thread_stopped(p); thread_exit(); } PROC_SLOCK(p); thread_stopped(p); if (P_SHOULDSTOP(p) == P_STOPPED_SINGLE) { if (p->p_numthreads == p->p_suspcount + 1) { thread_lock(p->p_singlethread); wakeup_swapper = - thread_unsuspend_one(p->p_singlethread); + thread_unsuspend_one(p->p_singlethread, p); thread_unlock(p->p_singlethread); if (wakeup_swapper) kick_proc0(); } } PROC_UNLOCK(p); thread_lock(td); /* * When a thread suspends, it just * gets taken off all queues. */ thread_suspend_one(td); if (return_instead == 0) { p->p_boundary_count++; td->td_flags |= TDF_BOUNDARY; } PROC_SUNLOCK(p); mi_switch(SW_INVOL | SWT_SUSPEND, NULL); if (return_instead == 0) td->td_flags &= ~TDF_BOUNDARY; thread_unlock(td); PROC_LOCK(p); if (return_instead == 0) { PROC_SLOCK(p); p->p_boundary_count--; PROC_SUNLOCK(p); } } return (0); } void -thread_suspend_switch(struct thread *td) +thread_suspend_switch(struct thread *td, struct proc *p) { - struct proc *p; - p = td->td_proc; KASSERT(!TD_IS_SUSPENDED(td), ("already suspended")); PROC_LOCK_ASSERT(p, MA_OWNED); PROC_SLOCK_ASSERT(p, MA_OWNED); /* * We implement thread_suspend_one in stages here to avoid * dropping the proc lock while the thread lock is owned. */ - thread_stopped(p); - p->p_suspcount++; + if (p == td->td_proc) { + thread_stopped(p); + p->p_suspcount++; + } PROC_UNLOCK(p); thread_lock(td); td->td_flags &= ~TDF_NEEDSUSPCHK; TD_SET_SUSPENDED(td); sched_sleep(td, 0); PROC_SUNLOCK(p); DROP_GIANT(); mi_switch(SW_VOL | SWT_SUSPEND, NULL); thread_unlock(td); PICKUP_GIANT(); PROC_LOCK(p); PROC_SLOCK(p); } void thread_suspend_one(struct thread *td) { - struct proc *p = td->td_proc; + struct proc *p; + p = td->td_proc; PROC_SLOCK_ASSERT(p, MA_OWNED); THREAD_LOCK_ASSERT(td, MA_OWNED); KASSERT(!TD_IS_SUSPENDED(td), ("already suspended")); p->p_suspcount++; td->td_flags &= ~TDF_NEEDSUSPCHK; TD_SET_SUSPENDED(td); sched_sleep(td, 0); } int -thread_unsuspend_one(struct thread *td) +thread_unsuspend_one(struct thread *td, struct proc *p) { - struct proc *p = td->td_proc; - PROC_SLOCK_ASSERT(p, MA_OWNED); THREAD_LOCK_ASSERT(td, MA_OWNED); KASSERT(TD_IS_SUSPENDED(td), ("Thread not suspended")); TD_CLR_SUSPENDED(td); - p->p_suspcount--; + td->td_flags &= ~TDF_ALLPROCSUSP; + if (td->td_proc == p) { + PROC_SLOCK_ASSERT(p, MA_OWNED); + p->p_suspcount--; + } return (setrunnable(td)); } /* * Allow all threads blocked by single threading to continue running. */ void thread_unsuspend(struct proc *p) { struct thread *td; int wakeup_swapper; PROC_LOCK_ASSERT(p, MA_OWNED); PROC_SLOCK_ASSERT(p, MA_OWNED); wakeup_swapper = 0; if (!P_SHOULDSTOP(p)) { FOREACH_THREAD_IN_PROC(p, td) { thread_lock(td); if (TD_IS_SUSPENDED(td)) { - wakeup_swapper |= thread_unsuspend_one(td); + wakeup_swapper |= thread_unsuspend_one(td, p); } thread_unlock(td); } } else if ((P_SHOULDSTOP(p) == P_STOPPED_SINGLE) && (p->p_numthreads == p->p_suspcount)) { /* * Stopping everything also did the job for the single * threading request. Now we've downgraded to single-threaded, * let it continue. */ - thread_lock(p->p_singlethread); - wakeup_swapper = thread_unsuspend_one(p->p_singlethread); - thread_unlock(p->p_singlethread); + if (p->p_singlethread->td_proc == p) { + thread_lock(p->p_singlethread); + wakeup_swapper = thread_unsuspend_one( + p->p_singlethread, p); + thread_unlock(p->p_singlethread); + } } if (wakeup_swapper) kick_proc0(); } /* * End the single threading mode.. */ void -thread_single_end(void) +thread_single_end(struct proc *p, int mode) { struct thread *td; - struct proc *p; int wakeup_swapper; - p = curproc; + KASSERT(mode == SINGLE_EXIT || mode == SINGLE_BOUNDARY || + mode == SINGLE_ALLPROC || mode == SINGLE_NO_EXIT, + ("invalid mode %d", mode)); PROC_LOCK_ASSERT(p, MA_OWNED); - p->p_flag &= ~(P_STOPPED_SINGLE | P_SINGLE_EXIT | P_SINGLE_BOUNDARY); + KASSERT((mode == SINGLE_ALLPROC && (p->p_flag & P_TOTAL_STOP) != 0) || + (mode != SINGLE_ALLPROC && (p->p_flag & P_TOTAL_STOP) == 0), + ("mode %d does not match P_TOTAL_STOP", mode)); + p->p_flag &= ~(P_STOPPED_SINGLE | P_SINGLE_EXIT | P_SINGLE_BOUNDARY | + P_TOTAL_STOP); PROC_SLOCK(p); p->p_singlethread = NULL; wakeup_swapper = 0; /* * If there are other threads they may now run, * unless of course there is a blanket 'stop order' * on the process. The single threader must be allowed * to continue however as this is a bad place to stop. */ - if (p->p_numthreads != remain_for_mode(SINGLE_EXIT) && - !P_SHOULDSTOP(p)) { + if (p->p_numthreads != remain_for_mode(mode) && !P_SHOULDSTOP(p)) { FOREACH_THREAD_IN_PROC(p, td) { thread_lock(td); if (TD_IS_SUSPENDED(td)) { - wakeup_swapper |= thread_unsuspend_one(td); + wakeup_swapper |= thread_unsuspend_one(td, p); } thread_unlock(td); } } PROC_SUNLOCK(p); if (wakeup_swapper) kick_proc0(); } struct thread * thread_find(struct proc *p, lwpid_t tid) { struct thread *td; PROC_LOCK_ASSERT(p, MA_OWNED); FOREACH_THREAD_IN_PROC(p, td) { if (td->td_tid == tid) break; } return (td); } /* Locate a thread by number; return with proc lock held. */ struct thread * tdfind(lwpid_t tid, pid_t pid) { #define RUN_THRESH 16 struct thread *td; int run = 0; rw_rlock(&tidhash_lock); LIST_FOREACH(td, TIDHASH(tid), td_hash) { if (td->td_tid == tid) { if (pid != -1 && td->td_proc->p_pid != pid) { td = NULL; break; } PROC_LOCK(td->td_proc); if (td->td_proc->p_state == PRS_NEW) { PROC_UNLOCK(td->td_proc); td = NULL; break; } if (run > RUN_THRESH) { if (rw_try_upgrade(&tidhash_lock)) { LIST_REMOVE(td, td_hash); LIST_INSERT_HEAD(TIDHASH(td->td_tid), td, td_hash); rw_wunlock(&tidhash_lock); return (td); } } break; } run++; } rw_runlock(&tidhash_lock); return (td); } void tidhash_add(struct thread *td) { rw_wlock(&tidhash_lock); LIST_INSERT_HEAD(TIDHASH(td->td_tid), td, td_hash); rw_wunlock(&tidhash_lock); } void tidhash_remove(struct thread *td) { rw_wlock(&tidhash_lock); LIST_REMOVE(td, td_hash); rw_wunlock(&tidhash_lock); } Index: projects/clang350-import/sys/kern/subr_clock.c =================================================================== --- projects/clang350-import/sys/kern/subr_clock.c (revision 275748) +++ projects/clang350-import/sys/kern/subr_clock.c (revision 275749) @@ -1,226 +1,224 @@ /*- * Copyright (c) 1988 University of Utah. * Copyright (c) 1982, 1990, 1993 * The Regents of the University of California. All rights reserved. * * This code is derived from software contributed to Berkeley by * the Systems Programming Group of the University of Utah Computer * Science Department. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 4. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * from: Utah $Hdr: clock.c 1.18 91/01/21$ * from: @(#)clock.c 8.2 (Berkeley) 1/12/94 * from: NetBSD: clock_subr.c,v 1.6 2001/07/07 17:04:02 thorpej Exp * and * from: src/sys/i386/isa/clock.c,v 1.176 2001/09/04 */ #include __FBSDID("$FreeBSD$"); #include #include #include #include #include #include #include #include int tz_minuteswest; int tz_dsttime; /* * The adjkerntz and wall_cmos_clock sysctls are in the "machdep" sysctl * namespace because they were misplaced there originally. */ static int adjkerntz; static int sysctl_machdep_adjkerntz(SYSCTL_HANDLER_ARGS) { int error; error = sysctl_handle_int(oidp, oidp->oid_arg1, oidp->oid_arg2, req); if (!error && req->newptr) resettodr(); return (error); } SYSCTL_PROC(_machdep, OID_AUTO, adjkerntz, CTLTYPE_INT|CTLFLAG_RW, &adjkerntz, 0, sysctl_machdep_adjkerntz, "I", "Local offset from UTC in seconds"); static int ct_debug; SYSCTL_INT(_debug, OID_AUTO, clocktime, CTLFLAG_RW, &ct_debug, 0, "Enable printing of clocktime debugging"); static int wall_cmos_clock; SYSCTL_INT(_machdep, OID_AUTO, wall_cmos_clock, CTLFLAG_RW, &wall_cmos_clock, 0, "Enables application of machdep.adjkerntz"); /*--------------------------------------------------------------------* * Generic routines to convert between a POSIX date * (seconds since 1/1/1970) and yr/mo/day/hr/min/sec * Derived from NetBSD arch/hp300/hp300/clock.c */ #define FEBRUARY 2 #define days_in_year(y) (leapyear(y) ? 366 : 365) #define days_in_month(y, m) \ (month_days[(m) - 1] + (m == FEBRUARY ? leapyear(y) : 0)) /* Day of week. Days are counted from 1/1/1970, which was a Thursday */ #define day_of_week(days) (((days) + 4) % 7) static const int month_days[12] = { 31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31 }; /* * This inline avoids some unnecessary modulo operations * as compared with the usual macro: * ( ((year % 4) == 0 && * (year % 100) != 0) || * ((year % 400) == 0) ) * It is otherwise equivalent. */ static int leapyear(int year) { int rv = 0; if ((year & 3) == 0) { rv = 1; if ((year % 100) == 0) { rv = 0; if ((year % 400) == 0) rv = 1; } } return (rv); } static void print_ct(struct clocktime *ct) { printf("[%04d-%02d-%02d %02d:%02d:%02d]", ct->year, ct->mon, ct->day, ct->hour, ct->min, ct->sec); } int clock_ct_to_ts(struct clocktime *ct, struct timespec *ts) { - time_t secs; int i, year, days; year = ct->year; if (ct_debug) { printf("ct_to_ts("); print_ct(ct); printf(")"); } /* Sanity checks. */ if (ct->mon < 1 || ct->mon > 12 || ct->day < 1 || ct->day > days_in_month(year, ct->mon) || ct->hour > 23 || ct->min > 59 || ct->sec > 59 || (sizeof(time_t) == 4 && year > 2037)) { /* time_t overflow */ if (ct_debug) printf(" = EINVAL\n"); return (EINVAL); } /* * Compute days since start of time * First from years, then from months. */ days = 0; for (i = POSIX_BASE_YEAR; i < year; i++) days += days_in_year(i); /* Months */ for (i = 1; i < ct->mon; i++) days += days_in_month(year, i); days += (ct->day - 1); - /* Add hours, minutes, seconds. */ - secs = ((days * 24 + ct->hour) * 60 + ct->min) * 60 + ct->sec; - - ts->tv_sec = secs; + ts->tv_sec = (((time_t)days * 24 + ct->hour) * 60 + ct->min) * 60 + + ct->sec; ts->tv_nsec = ct->nsec; + if (ct_debug) printf(" = %ld.%09ld\n", (long)ts->tv_sec, (long)ts->tv_nsec); return (0); } void clock_ts_to_ct(struct timespec *ts, struct clocktime *ct) { int i, year, days; time_t rsec; /* remainder seconds */ time_t secs; secs = ts->tv_sec; days = secs / SECDAY; rsec = secs % SECDAY; ct->dow = day_of_week(days); /* Subtract out whole years, counting them in i. */ for (year = POSIX_BASE_YEAR; days >= days_in_year(year); year++) days -= days_in_year(year); ct->year = year; /* Subtract out whole months, counting them in i. */ for (i = 1; days >= days_in_month(year, i); i++) days -= days_in_month(year, i); ct->mon = i; /* Days are what is left over (+1) from all that. */ ct->day = days + 1; /* Hours, minutes, seconds are easy */ ct->hour = rsec / 3600; rsec = rsec % 3600; ct->min = rsec / 60; rsec = rsec % 60; ct->sec = rsec; ct->nsec = ts->tv_nsec; if (ct_debug) { printf("ts_to_ct(%ld.%09ld) = ", (long)ts->tv_sec, (long)ts->tv_nsec); print_ct(ct); printf("\n"); } } int utc_offset(void) { return (tz_minuteswest * 60 + (wall_cmos_clock ? adjkerntz : 0)); } Index: projects/clang350-import/sys/kern/vfs_subr.c =================================================================== --- projects/clang350-import/sys/kern/vfs_subr.c (revision 275748) +++ projects/clang350-import/sys/kern/vfs_subr.c (revision 275749) @@ -1,4864 +1,4878 @@ /*- * Copyright (c) 1989, 1993 * The Regents of the University of California. All rights reserved. * (c) UNIX System Laboratories, Inc. * All or some portions of this file are derived from material licensed * to the University of California by American Telephone and Telegraph * Co. or Unix System Laboratories, Inc. and are reproduced herein with * the permission of UNIX System Laboratories, Inc. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 4. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * @(#)vfs_subr.c 8.31 (Berkeley) 5/26/95 */ /* * External virtual filesystem routines */ #include __FBSDID("$FreeBSD$"); #include "opt_compat.h" #include "opt_ddb.h" #include "opt_watchdog.h" #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #ifdef DDB #include #endif static void delmntque(struct vnode *vp); static int flushbuflist(struct bufv *bufv, int flags, struct bufobj *bo, int slpflag, int slptimeo); static void syncer_shutdown(void *arg, int howto); static int vtryrecycle(struct vnode *vp); static void v_incr_usecount(struct vnode *); static void v_decr_usecount(struct vnode *); static void v_decr_useonly(struct vnode *); static void v_upgrade_usecount(struct vnode *); static void vnlru_free(int); static void vgonel(struct vnode *); static void vfs_knllock(void *arg); static void vfs_knlunlock(void *arg); static void vfs_knl_assert_locked(void *arg); static void vfs_knl_assert_unlocked(void *arg); static void destroy_vpollinfo(struct vpollinfo *vi); /* * Number of vnodes in existence. Increased whenever getnewvnode() * allocates a new vnode, decreased in vdropl() for VI_DOOMED vnode. */ static unsigned long numvnodes; SYSCTL_ULONG(_vfs, OID_AUTO, numvnodes, CTLFLAG_RD, &numvnodes, 0, "Number of vnodes in existence"); /* * Conversion tables for conversion from vnode types to inode formats * and back. */ enum vtype iftovt_tab[16] = { VNON, VFIFO, VCHR, VNON, VDIR, VNON, VBLK, VNON, VREG, VNON, VLNK, VNON, VSOCK, VNON, VNON, VBAD, }; int vttoif_tab[10] = { 0, S_IFREG, S_IFDIR, S_IFBLK, S_IFCHR, S_IFLNK, S_IFSOCK, S_IFIFO, S_IFMT, S_IFMT }; /* * List of vnodes that are ready for recycling. */ static TAILQ_HEAD(freelst, vnode) vnode_free_list; /* * Free vnode target. Free vnodes may simply be files which have been stat'd * but not read. This is somewhat common, and a small cache of such files * should be kept to avoid recreation costs. */ static u_long wantfreevnodes; SYSCTL_ULONG(_vfs, OID_AUTO, wantfreevnodes, CTLFLAG_RW, &wantfreevnodes, 0, ""); /* Number of vnodes in the free list. */ static u_long freevnodes; SYSCTL_ULONG(_vfs, OID_AUTO, freevnodes, CTLFLAG_RD, &freevnodes, 0, "Number of vnodes in the free list"); static int vlru_allow_cache_src; SYSCTL_INT(_vfs, OID_AUTO, vlru_allow_cache_src, CTLFLAG_RW, &vlru_allow_cache_src, 0, "Allow vlru to reclaim source vnode"); /* * Various variables used for debugging the new implementation of * reassignbuf(). * XXX these are probably of (very) limited utility now. */ static int reassignbufcalls; SYSCTL_INT(_vfs, OID_AUTO, reassignbufcalls, CTLFLAG_RW, &reassignbufcalls, 0, "Number of calls to reassignbuf"); /* * Cache for the mount type id assigned to NFS. This is used for * special checks in nfs/nfs_nqlease.c and vm/vnode_pager.c. */ int nfs_mount_type = -1; /* To keep more than one thread at a time from running vfs_getnewfsid */ static struct mtx mntid_mtx; /* * Lock for any access to the following: * vnode_free_list * numvnodes * freevnodes */ static struct mtx vnode_free_list_mtx; /* Publicly exported FS */ struct nfs_public nfs_pub; static uma_zone_t buf_trie_zone; /* Zone for allocation of new vnodes - used exclusively by getnewvnode() */ static uma_zone_t vnode_zone; static uma_zone_t vnodepoll_zone; /* * The workitem queue. * * It is useful to delay writes of file data and filesystem metadata * for tens of seconds so that quickly created and deleted files need * not waste disk bandwidth being created and removed. To realize this, * we append vnodes to a "workitem" queue. When running with a soft * updates implementation, most pending metadata dependencies should * not wait for more than a few seconds. Thus, mounted on block devices * are delayed only about a half the time that file data is delayed. * Similarly, directory updates are more critical, so are only delayed * about a third the time that file data is delayed. Thus, there are * SYNCER_MAXDELAY queues that are processed round-robin at a rate of * one each second (driven off the filesystem syncer process). The * syncer_delayno variable indicates the next queue that is to be processed. * Items that need to be processed soon are placed in this queue: * * syncer_workitem_pending[syncer_delayno] * * A delay of fifteen seconds is done by placing the request fifteen * entries later in the queue: * * syncer_workitem_pending[(syncer_delayno + 15) & syncer_mask] * */ static int syncer_delayno; static long syncer_mask; LIST_HEAD(synclist, bufobj); static struct synclist *syncer_workitem_pending; /* * The sync_mtx protects: * bo->bo_synclist * sync_vnode_count * syncer_delayno * syncer_state * syncer_workitem_pending * syncer_worklist_len * rushjob */ static struct mtx sync_mtx; static struct cv sync_wakeup; #define SYNCER_MAXDELAY 32 static int syncer_maxdelay = SYNCER_MAXDELAY; /* maximum delay time */ static int syncdelay = 30; /* max time to delay syncing data */ static int filedelay = 30; /* time to delay syncing files */ SYSCTL_INT(_kern, OID_AUTO, filedelay, CTLFLAG_RW, &filedelay, 0, "Time to delay syncing files (in seconds)"); static int dirdelay = 29; /* time to delay syncing directories */ SYSCTL_INT(_kern, OID_AUTO, dirdelay, CTLFLAG_RW, &dirdelay, 0, "Time to delay syncing directories (in seconds)"); static int metadelay = 28; /* time to delay syncing metadata */ SYSCTL_INT(_kern, OID_AUTO, metadelay, CTLFLAG_RW, &metadelay, 0, "Time to delay syncing metadata (in seconds)"); static int rushjob; /* number of slots to run ASAP */ static int stat_rush_requests; /* number of times I/O speeded up */ SYSCTL_INT(_debug, OID_AUTO, rush_requests, CTLFLAG_RW, &stat_rush_requests, 0, "Number of times I/O speeded up (rush requests)"); /* * When shutting down the syncer, run it at four times normal speed. */ #define SYNCER_SHUTDOWN_SPEEDUP 4 static int sync_vnode_count; static int syncer_worklist_len; static enum { SYNCER_RUNNING, SYNCER_SHUTTING_DOWN, SYNCER_FINAL_DELAY } syncer_state; /* * Number of vnodes we want to exist at any one time. This is mostly used * to size hash tables in vnode-related code. It is normally not used in * getnewvnode(), as wantfreevnodes is normally nonzero.) * * XXX desiredvnodes is historical cruft and should not exist. */ int desiredvnodes; SYSCTL_INT(_kern, KERN_MAXVNODES, maxvnodes, CTLFLAG_RW, &desiredvnodes, 0, "Maximum number of vnodes"); SYSCTL_ULONG(_kern, OID_AUTO, minvnodes, CTLFLAG_RW, &wantfreevnodes, 0, "Minimum number of vnodes (legacy)"); static int vnlru_nowhere; SYSCTL_INT(_debug, OID_AUTO, vnlru_nowhere, CTLFLAG_RW, &vnlru_nowhere, 0, "Number of times the vnlru process ran without success"); /* Shift count for (uintptr_t)vp to initialize vp->v_hash. */ static int vnsz2log; /* * Support for the bufobj clean & dirty pctrie. */ static void * buf_trie_alloc(struct pctrie *ptree) { return uma_zalloc(buf_trie_zone, M_NOWAIT); } static void buf_trie_free(struct pctrie *ptree, void *node) { uma_zfree(buf_trie_zone, node); } PCTRIE_DEFINE(BUF, buf, b_lblkno, buf_trie_alloc, buf_trie_free); /* * Initialize the vnode management data structures. * * Reevaluate the following cap on the number of vnodes after the physical * memory size exceeds 512GB. In the limit, as the physical memory size * grows, the ratio of physical pages to vnodes approaches sixteen to one. */ #ifndef MAXVNODES_MAX #define MAXVNODES_MAX (512 * (1024 * 1024 * 1024 / (int)PAGE_SIZE / 16)) #endif static void vntblinit(void *dummy __unused) { u_int i; int physvnodes, virtvnodes; /* * Desiredvnodes is a function of the physical memory size and the * kernel's heap size. Generally speaking, it scales with the * physical memory size. The ratio of desiredvnodes to physical pages * is one to four until desiredvnodes exceeds 98,304. Thereafter, the * marginal ratio of desiredvnodes to physical pages is one to * sixteen. However, desiredvnodes is limited by the kernel's heap * size. The memory required by desiredvnodes vnodes and vm objects * may not exceed one seventh of the kernel's heap size. */ physvnodes = maxproc + vm_cnt.v_page_count / 16 + 3 * min(98304 * 4, vm_cnt.v_page_count) / 16; virtvnodes = vm_kmem_size / (7 * (sizeof(struct vm_object) + sizeof(struct vnode))); desiredvnodes = min(physvnodes, virtvnodes); if (desiredvnodes > MAXVNODES_MAX) { if (bootverbose) printf("Reducing kern.maxvnodes %d -> %d\n", desiredvnodes, MAXVNODES_MAX); desiredvnodes = MAXVNODES_MAX; } wantfreevnodes = desiredvnodes / 4; mtx_init(&mntid_mtx, "mntid", NULL, MTX_DEF); TAILQ_INIT(&vnode_free_list); mtx_init(&vnode_free_list_mtx, "vnode_free_list", NULL, MTX_DEF); vnode_zone = uma_zcreate("VNODE", sizeof (struct vnode), NULL, NULL, NULL, NULL, UMA_ALIGN_PTR, 0); vnodepoll_zone = uma_zcreate("VNODEPOLL", sizeof (struct vpollinfo), NULL, NULL, NULL, NULL, UMA_ALIGN_PTR, 0); /* * Preallocate enough nodes to support one-per buf so that * we can not fail an insert. reassignbuf() callers can not * tolerate the insertion failure. */ buf_trie_zone = uma_zcreate("BUF TRIE", pctrie_node_size(), NULL, NULL, pctrie_zone_init, NULL, UMA_ALIGN_PTR, UMA_ZONE_NOFREE | UMA_ZONE_VM); uma_prealloc(buf_trie_zone, nbuf); /* * Initialize the filesystem syncer. */ syncer_workitem_pending = hashinit(syncer_maxdelay, M_VNODE, &syncer_mask); syncer_maxdelay = syncer_mask + 1; mtx_init(&sync_mtx, "Syncer mtx", NULL, MTX_DEF); cv_init(&sync_wakeup, "syncer"); for (i = 1; i <= sizeof(struct vnode); i <<= 1) vnsz2log++; vnsz2log--; } SYSINIT(vfs, SI_SUB_VFS, SI_ORDER_FIRST, vntblinit, NULL); /* * Mark a mount point as busy. Used to synchronize access and to delay * unmounting. Eventually, mountlist_mtx is not released on failure. * * vfs_busy() is a custom lock, it can block the caller. * vfs_busy() only sleeps if the unmount is active on the mount point. * For a mountpoint mp, vfs_busy-enforced lock is before lock of any * vnode belonging to mp. * * Lookup uses vfs_busy() to traverse mount points. * root fs var fs * / vnode lock A / vnode lock (/var) D * /var vnode lock B /log vnode lock(/var/log) E * vfs_busy lock C vfs_busy lock F * * Within each file system, the lock order is C->A->B and F->D->E. * * When traversing across mounts, the system follows that lock order: * * C->A->B * | * +->F->D->E * * The lookup() process for namei("/var") illustrates the process: * VOP_LOOKUP() obtains B while A is held * vfs_busy() obtains a shared lock on F while A and B are held * vput() releases lock on B * vput() releases lock on A * VFS_ROOT() obtains lock on D while shared lock on F is held * vfs_unbusy() releases shared lock on F * vn_lock() obtains lock on deadfs vnode vp_crossmp instead of A. * Attempt to lock A (instead of vp_crossmp) while D is held would * violate the global order, causing deadlocks. * * dounmount() locks B while F is drained. */ int vfs_busy(struct mount *mp, int flags) { MPASS((flags & ~MBF_MASK) == 0); CTR3(KTR_VFS, "%s: mp %p with flags %d", __func__, mp, flags); MNT_ILOCK(mp); MNT_REF(mp); /* * If mount point is currenly being unmounted, sleep until the * mount point fate is decided. If thread doing the unmounting fails, * it will clear MNTK_UNMOUNT flag before waking us up, indicating * that this mount point has survived the unmount attempt and vfs_busy * should retry. Otherwise the unmounter thread will set MNTK_REFEXPIRE * flag in addition to MNTK_UNMOUNT, indicating that mount point is * about to be really destroyed. vfs_busy needs to release its * reference on the mount point in this case and return with ENOENT, * telling the caller that mount mount it tried to busy is no longer * valid. */ while (mp->mnt_kern_flag & MNTK_UNMOUNT) { if (flags & MBF_NOWAIT || mp->mnt_kern_flag & MNTK_REFEXPIRE) { MNT_REL(mp); MNT_IUNLOCK(mp); CTR1(KTR_VFS, "%s: failed busying before sleeping", __func__); return (ENOENT); } if (flags & MBF_MNTLSTLOCK) mtx_unlock(&mountlist_mtx); mp->mnt_kern_flag |= MNTK_MWAIT; msleep(mp, MNT_MTX(mp), PVFS | PDROP, "vfs_busy", 0); if (flags & MBF_MNTLSTLOCK) mtx_lock(&mountlist_mtx); MNT_ILOCK(mp); } if (flags & MBF_MNTLSTLOCK) mtx_unlock(&mountlist_mtx); mp->mnt_lockref++; MNT_IUNLOCK(mp); return (0); } /* * Free a busy filesystem. */ void vfs_unbusy(struct mount *mp) { CTR2(KTR_VFS, "%s: mp %p", __func__, mp); MNT_ILOCK(mp); MNT_REL(mp); KASSERT(mp->mnt_lockref > 0, ("negative mnt_lockref")); mp->mnt_lockref--; if (mp->mnt_lockref == 0 && (mp->mnt_kern_flag & MNTK_DRAINING) != 0) { MPASS(mp->mnt_kern_flag & MNTK_UNMOUNT); CTR1(KTR_VFS, "%s: waking up waiters", __func__); mp->mnt_kern_flag &= ~MNTK_DRAINING; wakeup(&mp->mnt_lockref); } MNT_IUNLOCK(mp); } /* * Lookup a mount point by filesystem identifier. */ struct mount * vfs_getvfs(fsid_t *fsid) { struct mount *mp; CTR2(KTR_VFS, "%s: fsid %p", __func__, fsid); mtx_lock(&mountlist_mtx); TAILQ_FOREACH(mp, &mountlist, mnt_list) { if (mp->mnt_stat.f_fsid.val[0] == fsid->val[0] && mp->mnt_stat.f_fsid.val[1] == fsid->val[1]) { vfs_ref(mp); mtx_unlock(&mountlist_mtx); return (mp); } } mtx_unlock(&mountlist_mtx); CTR2(KTR_VFS, "%s: lookup failed for %p id", __func__, fsid); return ((struct mount *) 0); } /* * Lookup a mount point by filesystem identifier, busying it before * returning. * * To avoid congestion on mountlist_mtx, implement simple direct-mapped * cache for popular filesystem identifiers. The cache is lockess, using * the fact that struct mount's are never freed. In worst case we may * get pointer to unmounted or even different filesystem, so we have to * check what we got, and go slow way if so. */ struct mount * vfs_busyfs(fsid_t *fsid) { #define FSID_CACHE_SIZE 256 typedef struct mount * volatile vmp_t; static vmp_t cache[FSID_CACHE_SIZE]; struct mount *mp; int error; uint32_t hash; CTR2(KTR_VFS, "%s: fsid %p", __func__, fsid); hash = fsid->val[0] ^ fsid->val[1]; hash = (hash >> 16 ^ hash) & (FSID_CACHE_SIZE - 1); mp = cache[hash]; if (mp == NULL || mp->mnt_stat.f_fsid.val[0] != fsid->val[0] || mp->mnt_stat.f_fsid.val[1] != fsid->val[1]) goto slow; if (vfs_busy(mp, 0) != 0) { cache[hash] = NULL; goto slow; } if (mp->mnt_stat.f_fsid.val[0] == fsid->val[0] && mp->mnt_stat.f_fsid.val[1] == fsid->val[1]) return (mp); else vfs_unbusy(mp); slow: mtx_lock(&mountlist_mtx); TAILQ_FOREACH(mp, &mountlist, mnt_list) { if (mp->mnt_stat.f_fsid.val[0] == fsid->val[0] && mp->mnt_stat.f_fsid.val[1] == fsid->val[1]) { error = vfs_busy(mp, MBF_MNTLSTLOCK); if (error) { cache[hash] = NULL; mtx_unlock(&mountlist_mtx); return (NULL); } cache[hash] = mp; return (mp); } } CTR2(KTR_VFS, "%s: lookup failed for %p id", __func__, fsid); mtx_unlock(&mountlist_mtx); return ((struct mount *) 0); } /* * Check if a user can access privileged mount options. */ int vfs_suser(struct mount *mp, struct thread *td) { int error; /* * If the thread is jailed, but this is not a jail-friendly file * system, deny immediately. */ if (!(mp->mnt_vfc->vfc_flags & VFCF_JAIL) && jailed(td->td_ucred)) return (EPERM); /* * If the file system was mounted outside the jail of the calling * thread, deny immediately. */ if (prison_check(td->td_ucred, mp->mnt_cred) != 0) return (EPERM); /* * If file system supports delegated administration, we don't check * for the PRIV_VFS_MOUNT_OWNER privilege - it will be better verified * by the file system itself. * If this is not the user that did original mount, we check for * the PRIV_VFS_MOUNT_OWNER privilege. */ if (!(mp->mnt_vfc->vfc_flags & VFCF_DELEGADMIN) && mp->mnt_cred->cr_uid != td->td_ucred->cr_uid) { if ((error = priv_check(td, PRIV_VFS_MOUNT_OWNER)) != 0) return (error); } return (0); } /* * Get a new unique fsid. Try to make its val[0] unique, since this value * will be used to create fake device numbers for stat(). Also try (but * not so hard) make its val[0] unique mod 2^16, since some emulators only * support 16-bit device numbers. We end up with unique val[0]'s for the * first 2^16 calls and unique val[0]'s mod 2^16 for the first 2^8 calls. * * Keep in mind that several mounts may be running in parallel. Starting * the search one past where the previous search terminated is both a * micro-optimization and a defense against returning the same fsid to * different mounts. */ void vfs_getnewfsid(struct mount *mp) { static uint16_t mntid_base; struct mount *nmp; fsid_t tfsid; int mtype; CTR2(KTR_VFS, "%s: mp %p", __func__, mp); mtx_lock(&mntid_mtx); mtype = mp->mnt_vfc->vfc_typenum; tfsid.val[1] = mtype; mtype = (mtype & 0xFF) << 24; for (;;) { tfsid.val[0] = makedev(255, mtype | ((mntid_base & 0xFF00) << 8) | (mntid_base & 0xFF)); mntid_base++; if ((nmp = vfs_getvfs(&tfsid)) == NULL) break; vfs_rel(nmp); } mp->mnt_stat.f_fsid.val[0] = tfsid.val[0]; mp->mnt_stat.f_fsid.val[1] = tfsid.val[1]; mtx_unlock(&mntid_mtx); } /* * Knob to control the precision of file timestamps: * * 0 = seconds only; nanoseconds zeroed. * 1 = seconds and nanoseconds, accurate within 1/HZ. * 2 = seconds and nanoseconds, truncated to microseconds. * >=3 = seconds and nanoseconds, maximum precision. */ enum { TSP_SEC, TSP_HZ, TSP_USEC, TSP_NSEC }; static int timestamp_precision = TSP_SEC; SYSCTL_INT(_vfs, OID_AUTO, timestamp_precision, CTLFLAG_RW, ×tamp_precision, 0, "File timestamp precision (0: seconds, " "1: sec + ns accurate to 1/HZ, 2: sec + ns truncated to ms, " "3+: sec + ns (max. precision))"); /* * Get a current timestamp. */ void vfs_timestamp(struct timespec *tsp) { struct timeval tv; switch (timestamp_precision) { case TSP_SEC: tsp->tv_sec = time_second; tsp->tv_nsec = 0; break; case TSP_HZ: getnanotime(tsp); break; case TSP_USEC: microtime(&tv); TIMEVAL_TO_TIMESPEC(&tv, tsp); break; case TSP_NSEC: default: nanotime(tsp); break; } } /* * Set vnode attributes to VNOVAL */ void vattr_null(struct vattr *vap) { vap->va_type = VNON; vap->va_size = VNOVAL; vap->va_bytes = VNOVAL; vap->va_mode = VNOVAL; vap->va_nlink = VNOVAL; vap->va_uid = VNOVAL; vap->va_gid = VNOVAL; vap->va_fsid = VNOVAL; vap->va_fileid = VNOVAL; vap->va_blocksize = VNOVAL; vap->va_rdev = VNOVAL; vap->va_atime.tv_sec = VNOVAL; vap->va_atime.tv_nsec = VNOVAL; vap->va_mtime.tv_sec = VNOVAL; vap->va_mtime.tv_nsec = VNOVAL; vap->va_ctime.tv_sec = VNOVAL; vap->va_ctime.tv_nsec = VNOVAL; vap->va_birthtime.tv_sec = VNOVAL; vap->va_birthtime.tv_nsec = VNOVAL; vap->va_flags = VNOVAL; vap->va_gen = VNOVAL; vap->va_vaflags = 0; } /* * This routine is called when we have too many vnodes. It attempts * to free vnodes and will potentially free vnodes that still * have VM backing store (VM backing store is typically the cause * of a vnode blowout so we want to do this). Therefore, this operation * is not considered cheap. * * A number of conditions may prevent a vnode from being reclaimed. * the buffer cache may have references on the vnode, a directory * vnode may still have references due to the namei cache representing * underlying files, or the vnode may be in active use. It is not * desireable to reuse such vnodes. These conditions may cause the * number of vnodes to reach some minimum value regardless of what * you set kern.maxvnodes to. Do not set kern.maxvnodes too low. */ static int vlrureclaim(struct mount *mp) { struct vnode *vp; int done; int trigger; int usevnodes; int count; /* * Calculate the trigger point, don't allow user * screwups to blow us up. This prevents us from * recycling vnodes with lots of resident pages. We * aren't trying to free memory, we are trying to * free vnodes. */ usevnodes = desiredvnodes; if (usevnodes <= 0) usevnodes = 1; trigger = vm_cnt.v_page_count * 2 / usevnodes; done = 0; vn_start_write(NULL, &mp, V_WAIT); MNT_ILOCK(mp); count = mp->mnt_nvnodelistsize / 10 + 1; while (count != 0) { vp = TAILQ_FIRST(&mp->mnt_nvnodelist); while (vp != NULL && vp->v_type == VMARKER) vp = TAILQ_NEXT(vp, v_nmntvnodes); if (vp == NULL) break; TAILQ_REMOVE(&mp->mnt_nvnodelist, vp, v_nmntvnodes); TAILQ_INSERT_TAIL(&mp->mnt_nvnodelist, vp, v_nmntvnodes); --count; if (!VI_TRYLOCK(vp)) goto next_iter; /* * If it's been deconstructed already, it's still * referenced, or it exceeds the trigger, skip it. */ if (vp->v_usecount || (!vlru_allow_cache_src && !LIST_EMPTY(&(vp)->v_cache_src)) || (vp->v_iflag & VI_DOOMED) != 0 || (vp->v_object != NULL && vp->v_object->resident_page_count > trigger)) { VI_UNLOCK(vp); goto next_iter; } MNT_IUNLOCK(mp); vholdl(vp); if (VOP_LOCK(vp, LK_INTERLOCK|LK_EXCLUSIVE|LK_NOWAIT)) { vdrop(vp); goto next_iter_mntunlocked; } VI_LOCK(vp); /* * v_usecount may have been bumped after VOP_LOCK() dropped * the vnode interlock and before it was locked again. * * It is not necessary to recheck VI_DOOMED because it can * only be set by another thread that holds both the vnode * lock and vnode interlock. If another thread has the * vnode lock before we get to VOP_LOCK() and obtains the * vnode interlock after VOP_LOCK() drops the vnode * interlock, the other thread will be unable to drop the * vnode lock before our VOP_LOCK() call fails. */ if (vp->v_usecount || (!vlru_allow_cache_src && !LIST_EMPTY(&(vp)->v_cache_src)) || (vp->v_object != NULL && vp->v_object->resident_page_count > trigger)) { VOP_UNLOCK(vp, LK_INTERLOCK); vdrop(vp); goto next_iter_mntunlocked; } KASSERT((vp->v_iflag & VI_DOOMED) == 0, ("VI_DOOMED unexpectedly detected in vlrureclaim()")); vgonel(vp); VOP_UNLOCK(vp, 0); vdropl(vp); done++; next_iter_mntunlocked: if (!should_yield()) goto relock_mnt; goto yield; next_iter: if (!should_yield()) continue; MNT_IUNLOCK(mp); yield: kern_yield(PRI_USER); relock_mnt: MNT_ILOCK(mp); } MNT_IUNLOCK(mp); vn_finished_write(mp); return done; } /* * Attempt to keep the free list at wantfreevnodes length. */ static void vnlru_free(int count) { struct vnode *vp; mtx_assert(&vnode_free_list_mtx, MA_OWNED); for (; count > 0; count--) { vp = TAILQ_FIRST(&vnode_free_list); /* * The list can be modified while the free_list_mtx * has been dropped and vp could be NULL here. */ if (!vp) break; VNASSERT(vp->v_op != NULL, vp, ("vnlru_free: vnode already reclaimed.")); KASSERT((vp->v_iflag & VI_FREE) != 0, ("Removing vnode not on freelist")); KASSERT((vp->v_iflag & VI_ACTIVE) == 0, ("Mangling active vnode")); TAILQ_REMOVE(&vnode_free_list, vp, v_actfreelist); /* * Don't recycle if we can't get the interlock. */ if (!VI_TRYLOCK(vp)) { TAILQ_INSERT_TAIL(&vnode_free_list, vp, v_actfreelist); continue; } VNASSERT((vp->v_iflag & VI_FREE) != 0 && vp->v_holdcnt == 0, vp, ("vp inconsistent on freelist")); /* * The clear of VI_FREE prevents activation of the * vnode. There is no sense in putting the vnode on * the mount point active list, only to remove it * later during recycling. Inline the relevant part * of vholdl(), to avoid triggering assertions or * activating. */ freevnodes--; vp->v_iflag &= ~VI_FREE; vp->v_holdcnt++; mtx_unlock(&vnode_free_list_mtx); VI_UNLOCK(vp); vtryrecycle(vp); /* * If the recycled succeeded this vdrop will actually free * the vnode. If not it will simply place it back on * the free list. */ vdrop(vp); mtx_lock(&vnode_free_list_mtx); } } /* * Attempt to recycle vnodes in a context that is always safe to block. * Calling vlrurecycle() from the bowels of filesystem code has some * interesting deadlock problems. */ static struct proc *vnlruproc; static int vnlruproc_sig; static void vnlru_proc(void) { struct mount *mp, *nmp; int done; struct proc *p = vnlruproc; EVENTHANDLER_REGISTER(shutdown_pre_sync, kproc_shutdown, p, SHUTDOWN_PRI_FIRST); for (;;) { kproc_suspend_check(p); mtx_lock(&vnode_free_list_mtx); if (freevnodes > wantfreevnodes) vnlru_free(freevnodes - wantfreevnodes); if (numvnodes <= desiredvnodes * 9 / 10) { vnlruproc_sig = 0; wakeup(&vnlruproc_sig); msleep(vnlruproc, &vnode_free_list_mtx, PVFS|PDROP, "vlruwt", hz); continue; } mtx_unlock(&vnode_free_list_mtx); done = 0; mtx_lock(&mountlist_mtx); for (mp = TAILQ_FIRST(&mountlist); mp != NULL; mp = nmp) { if (vfs_busy(mp, MBF_NOWAIT | MBF_MNTLSTLOCK)) { nmp = TAILQ_NEXT(mp, mnt_list); continue; } done += vlrureclaim(mp); mtx_lock(&mountlist_mtx); nmp = TAILQ_NEXT(mp, mnt_list); vfs_unbusy(mp); } mtx_unlock(&mountlist_mtx); if (done == 0) { #if 0 /* These messages are temporary debugging aids */ if (vnlru_nowhere < 5) printf("vnlru process getting nowhere..\n"); else if (vnlru_nowhere == 5) printf("vnlru process messages stopped.\n"); #endif vnlru_nowhere++; tsleep(vnlruproc, PPAUSE, "vlrup", hz * 3); } else kern_yield(PRI_USER); } } static struct kproc_desc vnlru_kp = { "vnlru", vnlru_proc, &vnlruproc }; SYSINIT(vnlru, SI_SUB_KTHREAD_UPDATE, SI_ORDER_FIRST, kproc_start, &vnlru_kp); /* * Routines having to do with the management of the vnode table. */ /* * Try to recycle a freed vnode. We abort if anyone picks up a reference * before we actually vgone(). This function must be called with the vnode * held to prevent the vnode from being returned to the free list midway * through vgone(). */ static int vtryrecycle(struct vnode *vp) { struct mount *vnmp; CTR2(KTR_VFS, "%s: vp %p", __func__, vp); VNASSERT(vp->v_holdcnt, vp, ("vtryrecycle: Recycling vp %p without a reference.", vp)); /* * This vnode may found and locked via some other list, if so we * can't recycle it yet. */ if (VOP_LOCK(vp, LK_EXCLUSIVE | LK_NOWAIT) != 0) { CTR2(KTR_VFS, "%s: impossible to recycle, vp %p lock is already held", __func__, vp); return (EWOULDBLOCK); } /* * Don't recycle if its filesystem is being suspended. */ if (vn_start_write(vp, &vnmp, V_NOWAIT) != 0) { VOP_UNLOCK(vp, 0); CTR2(KTR_VFS, "%s: impossible to recycle, cannot start the write for %p", __func__, vp); return (EBUSY); } /* * If we got this far, we need to acquire the interlock and see if * anyone picked up this vnode from another list. If not, we will * mark it with DOOMED via vgonel() so that anyone who does find it * will skip over it. */ VI_LOCK(vp); if (vp->v_usecount) { VOP_UNLOCK(vp, LK_INTERLOCK); vn_finished_write(vnmp); CTR2(KTR_VFS, "%s: impossible to recycle, %p is already referenced", __func__, vp); return (EBUSY); } if ((vp->v_iflag & VI_DOOMED) == 0) vgonel(vp); VOP_UNLOCK(vp, LK_INTERLOCK); vn_finished_write(vnmp); return (0); } /* * Wait for available vnodes. */ static int getnewvnode_wait(int suspended) { mtx_assert(&vnode_free_list_mtx, MA_OWNED); if (numvnodes > desiredvnodes) { if (suspended) { /* * File system is beeing suspended, we cannot risk a * deadlock here, so allocate new vnode anyway. */ if (freevnodes > wantfreevnodes) vnlru_free(freevnodes - wantfreevnodes); return (0); } if (vnlruproc_sig == 0) { vnlruproc_sig = 1; /* avoid unnecessary wakeups */ wakeup(vnlruproc); } msleep(&vnlruproc_sig, &vnode_free_list_mtx, PVFS, "vlruwk", hz); } return (numvnodes > desiredvnodes ? ENFILE : 0); } void getnewvnode_reserve(u_int count) { struct thread *td; td = curthread; /* First try to be quick and racy. */ if (atomic_fetchadd_long(&numvnodes, count) + count <= desiredvnodes) { td->td_vp_reserv += count; return; } else atomic_subtract_long(&numvnodes, count); mtx_lock(&vnode_free_list_mtx); while (count > 0) { if (getnewvnode_wait(0) == 0) { count--; td->td_vp_reserv++; atomic_add_long(&numvnodes, 1); } } mtx_unlock(&vnode_free_list_mtx); } void getnewvnode_drop_reserve(void) { struct thread *td; td = curthread; atomic_subtract_long(&numvnodes, td->td_vp_reserv); td->td_vp_reserv = 0; } /* * Return the next vnode from the free list. */ int getnewvnode(const char *tag, struct mount *mp, struct vop_vector *vops, struct vnode **vpp) { struct vnode *vp; struct bufobj *bo; struct thread *td; int error; CTR3(KTR_VFS, "%s: mp %p with tag %s", __func__, mp, tag); vp = NULL; td = curthread; if (td->td_vp_reserv > 0) { td->td_vp_reserv -= 1; goto alloc; } mtx_lock(&vnode_free_list_mtx); /* * Lend our context to reclaim vnodes if they've exceeded the max. */ if (freevnodes > wantfreevnodes) vnlru_free(1); error = getnewvnode_wait(mp != NULL && (mp->mnt_kern_flag & MNTK_SUSPEND)); #if 0 /* XXX Not all VFS_VGET/ffs_vget callers check returns. */ if (error != 0) { mtx_unlock(&vnode_free_list_mtx); return (error); } #endif atomic_add_long(&numvnodes, 1); mtx_unlock(&vnode_free_list_mtx); alloc: vp = (struct vnode *) uma_zalloc(vnode_zone, M_WAITOK|M_ZERO); /* * Setup locks. */ vp->v_vnlock = &vp->v_lock; mtx_init(&vp->v_interlock, "vnode interlock", NULL, MTX_DEF); /* * By default, don't allow shared locks unless filesystems * opt-in. */ lockinit(vp->v_vnlock, PVFS, tag, VLKTIMEOUT, LK_NOSHARE | LK_IS_VNODE); /* * Initialize bufobj. */ bo = &vp->v_bufobj; bo->__bo_vnode = vp; rw_init(BO_LOCKPTR(bo), "bufobj interlock"); bo->bo_ops = &buf_ops_bio; bo->bo_private = vp; TAILQ_INIT(&bo->bo_clean.bv_hd); TAILQ_INIT(&bo->bo_dirty.bv_hd); /* * Initialize namecache. */ LIST_INIT(&vp->v_cache_src); TAILQ_INIT(&vp->v_cache_dst); /* * Finalize various vnode identity bits. */ vp->v_type = VNON; vp->v_tag = tag; vp->v_op = vops; v_incr_usecount(vp); vp->v_data = NULL; #ifdef MAC mac_vnode_init(vp); if (mp != NULL && (mp->mnt_flag & MNT_MULTILABEL) == 0) mac_vnode_associate_singlelabel(mp, vp); else if (mp == NULL && vops != &dead_vnodeops) printf("NULL mp in getnewvnode()\n"); #endif if (mp != NULL) { bo->bo_bsize = mp->mnt_stat.f_iosize; if ((mp->mnt_kern_flag & MNTK_NOKNOTE) != 0) vp->v_vflag |= VV_NOKNOTE; } rangelock_init(&vp->v_rl); /* * For the filesystems which do not use vfs_hash_insert(), * still initialize v_hash to have vfs_hash_index() useful. * E.g., nullfs uses vfs_hash_index() on the lower vnode for * its own hashing. */ vp->v_hash = (uintptr_t)vp >> vnsz2log; *vpp = vp; return (0); } /* * Delete from old mount point vnode list, if on one. */ static void delmntque(struct vnode *vp) { struct mount *mp; int active; mp = vp->v_mount; if (mp == NULL) return; MNT_ILOCK(mp); VI_LOCK(vp); KASSERT(mp->mnt_activevnodelistsize <= mp->mnt_nvnodelistsize, ("Active vnode list size %d > Vnode list size %d", mp->mnt_activevnodelistsize, mp->mnt_nvnodelistsize)); active = vp->v_iflag & VI_ACTIVE; vp->v_iflag &= ~VI_ACTIVE; if (active) { mtx_lock(&vnode_free_list_mtx); TAILQ_REMOVE(&mp->mnt_activevnodelist, vp, v_actfreelist); mp->mnt_activevnodelistsize--; mtx_unlock(&vnode_free_list_mtx); } vp->v_mount = NULL; VI_UNLOCK(vp); VNASSERT(mp->mnt_nvnodelistsize > 0, vp, ("bad mount point vnode list size")); TAILQ_REMOVE(&mp->mnt_nvnodelist, vp, v_nmntvnodes); mp->mnt_nvnodelistsize--; MNT_REL(mp); MNT_IUNLOCK(mp); } static void insmntque_stddtr(struct vnode *vp, void *dtr_arg) { vp->v_data = NULL; vp->v_op = &dead_vnodeops; vgone(vp); vput(vp); } /* * Insert into list of vnodes for the new mount point, if available. */ int insmntque1(struct vnode *vp, struct mount *mp, void (*dtr)(struct vnode *, void *), void *dtr_arg) { KASSERT(vp->v_mount == NULL, ("insmntque: vnode already on per mount vnode list")); VNASSERT(mp != NULL, vp, ("Don't call insmntque(foo, NULL)")); ASSERT_VOP_ELOCKED(vp, "insmntque: non-locked vp"); /* * We acquire the vnode interlock early to ensure that the * vnode cannot be recycled by another process releasing a * holdcnt on it before we get it on both the vnode list * and the active vnode list. The mount mutex protects only * manipulation of the vnode list and the vnode freelist * mutex protects only manipulation of the active vnode list. * Hence the need to hold the vnode interlock throughout. */ MNT_ILOCK(mp); VI_LOCK(vp); if (((mp->mnt_kern_flag & MNTK_NOINSMNTQ) != 0 && ((mp->mnt_kern_flag & MNTK_UNMOUNTF) != 0 || mp->mnt_nvnodelistsize == 0)) && (vp->v_vflag & VV_FORCEINSMQ) == 0) { VI_UNLOCK(vp); MNT_IUNLOCK(mp); if (dtr != NULL) dtr(vp, dtr_arg); return (EBUSY); } vp->v_mount = mp; MNT_REF(mp); TAILQ_INSERT_TAIL(&mp->mnt_nvnodelist, vp, v_nmntvnodes); VNASSERT(mp->mnt_nvnodelistsize >= 0, vp, ("neg mount point vnode list size")); mp->mnt_nvnodelistsize++; KASSERT((vp->v_iflag & VI_ACTIVE) == 0, ("Activating already active vnode")); vp->v_iflag |= VI_ACTIVE; mtx_lock(&vnode_free_list_mtx); TAILQ_INSERT_HEAD(&mp->mnt_activevnodelist, vp, v_actfreelist); mp->mnt_activevnodelistsize++; mtx_unlock(&vnode_free_list_mtx); VI_UNLOCK(vp); MNT_IUNLOCK(mp); return (0); } int insmntque(struct vnode *vp, struct mount *mp) { return (insmntque1(vp, mp, insmntque_stddtr, NULL)); } /* * Flush out and invalidate all buffers associated with a bufobj * Called with the underlying object locked. */ int bufobj_invalbuf(struct bufobj *bo, int flags, int slpflag, int slptimeo) { int error; BO_LOCK(bo); if (flags & V_SAVE) { error = bufobj_wwait(bo, slpflag, slptimeo); if (error) { BO_UNLOCK(bo); return (error); } if (bo->bo_dirty.bv_cnt > 0) { BO_UNLOCK(bo); if ((error = BO_SYNC(bo, MNT_WAIT)) != 0) return (error); /* * XXX We could save a lock/unlock if this was only * enabled under INVARIANTS */ BO_LOCK(bo); if (bo->bo_numoutput > 0 || bo->bo_dirty.bv_cnt > 0) panic("vinvalbuf: dirty bufs"); } } /* * If you alter this loop please notice that interlock is dropped and * reacquired in flushbuflist. Special care is needed to ensure that * no race conditions occur from this. */ do { error = flushbuflist(&bo->bo_clean, flags, bo, slpflag, slptimeo); if (error == 0 && !(flags & V_CLEANONLY)) error = flushbuflist(&bo->bo_dirty, flags, bo, slpflag, slptimeo); if (error != 0 && error != EAGAIN) { BO_UNLOCK(bo); return (error); } } while (error != 0); /* * Wait for I/O to complete. XXX needs cleaning up. The vnode can * have write I/O in-progress but if there is a VM object then the * VM object can also have read-I/O in-progress. */ do { bufobj_wwait(bo, 0, 0); BO_UNLOCK(bo); if (bo->bo_object != NULL) { VM_OBJECT_WLOCK(bo->bo_object); vm_object_pip_wait(bo->bo_object, "bovlbx"); VM_OBJECT_WUNLOCK(bo->bo_object); } BO_LOCK(bo); } while (bo->bo_numoutput > 0); BO_UNLOCK(bo); /* * Destroy the copy in the VM cache, too. */ if (bo->bo_object != NULL && (flags & (V_ALT | V_NORMAL | V_CLEANONLY)) == 0) { VM_OBJECT_WLOCK(bo->bo_object); vm_object_page_remove(bo->bo_object, 0, 0, (flags & V_SAVE) ? OBJPR_CLEANONLY : 0); VM_OBJECT_WUNLOCK(bo->bo_object); } #ifdef INVARIANTS BO_LOCK(bo); if ((flags & (V_ALT | V_NORMAL | V_CLEANONLY)) == 0 && (bo->bo_dirty.bv_cnt > 0 || bo->bo_clean.bv_cnt > 0)) panic("vinvalbuf: flush failed"); BO_UNLOCK(bo); #endif return (0); } /* * Flush out and invalidate all buffers associated with a vnode. * Called with the underlying object locked. */ int vinvalbuf(struct vnode *vp, int flags, int slpflag, int slptimeo) { CTR3(KTR_VFS, "%s: vp %p with flags %d", __func__, vp, flags); ASSERT_VOP_LOCKED(vp, "vinvalbuf"); if (vp->v_object != NULL && vp->v_object->handle != vp) return (0); return (bufobj_invalbuf(&vp->v_bufobj, flags, slpflag, slptimeo)); } /* * Flush out buffers on the specified list. * */ static int flushbuflist(struct bufv *bufv, int flags, struct bufobj *bo, int slpflag, int slptimeo) { struct buf *bp, *nbp; int retval, error; daddr_t lblkno; b_xflags_t xflags; ASSERT_BO_WLOCKED(bo); retval = 0; TAILQ_FOREACH_SAFE(bp, &bufv->bv_hd, b_bobufs, nbp) { if (((flags & V_NORMAL) && (bp->b_xflags & BX_ALTDATA)) || ((flags & V_ALT) && (bp->b_xflags & BX_ALTDATA) == 0)) { continue; } lblkno = 0; xflags = 0; if (nbp != NULL) { lblkno = nbp->b_lblkno; xflags = nbp->b_xflags & (BX_VNDIRTY | BX_VNCLEAN); } retval = EAGAIN; error = BUF_TIMELOCK(bp, LK_EXCLUSIVE | LK_SLEEPFAIL | LK_INTERLOCK, BO_LOCKPTR(bo), "flushbuf", slpflag, slptimeo); if (error) { BO_LOCK(bo); return (error != ENOLCK ? error : EAGAIN); } KASSERT(bp->b_bufobj == bo, ("bp %p wrong b_bufobj %p should be %p", bp, bp->b_bufobj, bo)); if (bp->b_bufobj != bo) { /* XXX: necessary ? */ BUF_UNLOCK(bp); BO_LOCK(bo); return (EAGAIN); } /* * XXX Since there are no node locks for NFS, I * believe there is a slight chance that a delayed * write will occur while sleeping just above, so * check for it. */ if (((bp->b_flags & (B_DELWRI | B_INVAL)) == B_DELWRI) && (flags & V_SAVE)) { bremfree(bp); bp->b_flags |= B_ASYNC; bwrite(bp); BO_LOCK(bo); return (EAGAIN); /* XXX: why not loop ? */ } bremfree(bp); bp->b_flags |= (B_INVAL | B_RELBUF); bp->b_flags &= ~B_ASYNC; brelse(bp); BO_LOCK(bo); if (nbp != NULL && (nbp->b_bufobj != bo || nbp->b_lblkno != lblkno || (nbp->b_xflags & (BX_VNDIRTY | BX_VNCLEAN)) != xflags)) break; /* nbp invalid */ } return (retval); } /* * Truncate a file's buffer and pages to a specified length. This * is in lieu of the old vinvalbuf mechanism, which performed unneeded * sync activity. */ int vtruncbuf(struct vnode *vp, struct ucred *cred, off_t length, int blksize) { struct buf *bp, *nbp; int anyfreed; int trunclbn; struct bufobj *bo; CTR5(KTR_VFS, "%s: vp %p with cred %p and block %d:%ju", __func__, vp, cred, blksize, (uintmax_t)length); /* * Round up to the *next* lbn. */ trunclbn = (length + blksize - 1) / blksize; ASSERT_VOP_LOCKED(vp, "vtruncbuf"); restart: bo = &vp->v_bufobj; BO_LOCK(bo); anyfreed = 1; for (;anyfreed;) { anyfreed = 0; TAILQ_FOREACH_SAFE(bp, &bo->bo_clean.bv_hd, b_bobufs, nbp) { if (bp->b_lblkno < trunclbn) continue; if (BUF_LOCK(bp, LK_EXCLUSIVE | LK_SLEEPFAIL | LK_INTERLOCK, BO_LOCKPTR(bo)) == ENOLCK) goto restart; bremfree(bp); bp->b_flags |= (B_INVAL | B_RELBUF); bp->b_flags &= ~B_ASYNC; brelse(bp); anyfreed = 1; BO_LOCK(bo); if (nbp != NULL && (((nbp->b_xflags & BX_VNCLEAN) == 0) || (nbp->b_vp != vp) || (nbp->b_flags & B_DELWRI))) { BO_UNLOCK(bo); goto restart; } } TAILQ_FOREACH_SAFE(bp, &bo->bo_dirty.bv_hd, b_bobufs, nbp) { if (bp->b_lblkno < trunclbn) continue; if (BUF_LOCK(bp, LK_EXCLUSIVE | LK_SLEEPFAIL | LK_INTERLOCK, BO_LOCKPTR(bo)) == ENOLCK) goto restart; bremfree(bp); bp->b_flags |= (B_INVAL | B_RELBUF); bp->b_flags &= ~B_ASYNC; brelse(bp); anyfreed = 1; BO_LOCK(bo); if (nbp != NULL && (((nbp->b_xflags & BX_VNDIRTY) == 0) || (nbp->b_vp != vp) || (nbp->b_flags & B_DELWRI) == 0)) { BO_UNLOCK(bo); goto restart; } } } if (length > 0) { restartsync: TAILQ_FOREACH_SAFE(bp, &bo->bo_dirty.bv_hd, b_bobufs, nbp) { if (bp->b_lblkno > 0) continue; /* * Since we hold the vnode lock this should only * fail if we're racing with the buf daemon. */ if (BUF_LOCK(bp, LK_EXCLUSIVE | LK_SLEEPFAIL | LK_INTERLOCK, BO_LOCKPTR(bo)) == ENOLCK) { goto restart; } VNASSERT((bp->b_flags & B_DELWRI), vp, ("buf(%p) on dirty queue without DELWRI", bp)); bremfree(bp); bawrite(bp); BO_LOCK(bo); goto restartsync; } } bufobj_wwait(bo, 0, 0); BO_UNLOCK(bo); vnode_pager_setsize(vp, length); return (0); } static void buf_vlist_remove(struct buf *bp) { struct bufv *bv; KASSERT(bp->b_bufobj != NULL, ("No b_bufobj %p", bp)); ASSERT_BO_WLOCKED(bp->b_bufobj); KASSERT((bp->b_xflags & (BX_VNDIRTY|BX_VNCLEAN)) != (BX_VNDIRTY|BX_VNCLEAN), ("buf_vlist_remove: Buf %p is on two lists", bp)); if (bp->b_xflags & BX_VNDIRTY) bv = &bp->b_bufobj->bo_dirty; else bv = &bp->b_bufobj->bo_clean; BUF_PCTRIE_REMOVE(&bv->bv_root, bp->b_lblkno); TAILQ_REMOVE(&bv->bv_hd, bp, b_bobufs); bv->bv_cnt--; bp->b_xflags &= ~(BX_VNDIRTY | BX_VNCLEAN); } /* * Add the buffer to the sorted clean or dirty block list. * * NOTE: xflags is passed as a constant, optimizing this inline function! */ static void buf_vlist_add(struct buf *bp, struct bufobj *bo, b_xflags_t xflags) { struct bufv *bv; struct buf *n; int error; ASSERT_BO_WLOCKED(bo); + KASSERT((bo->bo_flag & BO_DEAD) == 0, ("dead bo %p", bo)); KASSERT((bp->b_xflags & (BX_VNDIRTY|BX_VNCLEAN)) == 0, ("buf_vlist_add: Buf %p has existing xflags %d", bp, bp->b_xflags)); bp->b_xflags |= xflags; if (xflags & BX_VNDIRTY) bv = &bo->bo_dirty; else bv = &bo->bo_clean; /* * Keep the list ordered. Optimize empty list insertion. Assume * we tend to grow at the tail so lookup_le should usually be cheaper * than _ge. */ if (bv->bv_cnt == 0 || bp->b_lblkno > TAILQ_LAST(&bv->bv_hd, buflists)->b_lblkno) TAILQ_INSERT_TAIL(&bv->bv_hd, bp, b_bobufs); else if ((n = BUF_PCTRIE_LOOKUP_LE(&bv->bv_root, bp->b_lblkno)) == NULL) TAILQ_INSERT_HEAD(&bv->bv_hd, bp, b_bobufs); else TAILQ_INSERT_AFTER(&bv->bv_hd, n, bp, b_bobufs); error = BUF_PCTRIE_INSERT(&bv->bv_root, bp); if (error) panic("buf_vlist_add: Preallocated nodes insufficient."); bv->bv_cnt++; } /* * Lookup a buffer using the splay tree. Note that we specifically avoid * shadow buffers used in background bitmap writes. * * This code isn't quite efficient as it could be because we are maintaining * two sorted lists and do not know which list the block resides in. * * During a "make buildworld" the desired buffer is found at one of * the roots more than 60% of the time. Thus, checking both roots * before performing either splay eliminates unnecessary splays on the * first tree splayed. */ struct buf * gbincore(struct bufobj *bo, daddr_t lblkno) { struct buf *bp; ASSERT_BO_LOCKED(bo); bp = BUF_PCTRIE_LOOKUP(&bo->bo_clean.bv_root, lblkno); if (bp != NULL) return (bp); return BUF_PCTRIE_LOOKUP(&bo->bo_dirty.bv_root, lblkno); } /* * Associate a buffer with a vnode. */ void bgetvp(struct vnode *vp, struct buf *bp) { struct bufobj *bo; bo = &vp->v_bufobj; ASSERT_BO_WLOCKED(bo); VNASSERT(bp->b_vp == NULL, bp->b_vp, ("bgetvp: not free")); CTR3(KTR_BUF, "bgetvp(%p) vp %p flags %X", bp, vp, bp->b_flags); VNASSERT((bp->b_xflags & (BX_VNDIRTY|BX_VNCLEAN)) == 0, vp, ("bgetvp: bp already attached! %p", bp)); vhold(vp); bp->b_vp = vp; bp->b_bufobj = bo; /* * Insert onto list for new vnode. */ buf_vlist_add(bp, bo, BX_VNCLEAN); } /* * Disassociate a buffer from a vnode. */ void brelvp(struct buf *bp) { struct bufobj *bo; struct vnode *vp; CTR3(KTR_BUF, "brelvp(%p) vp %p flags %X", bp, bp->b_vp, bp->b_flags); KASSERT(bp->b_vp != NULL, ("brelvp: NULL")); /* * Delete from old vnode list, if on one. */ vp = bp->b_vp; /* XXX */ bo = bp->b_bufobj; BO_LOCK(bo); if (bp->b_xflags & (BX_VNDIRTY | BX_VNCLEAN)) buf_vlist_remove(bp); else panic("brelvp: Buffer %p not on queue.", bp); if ((bo->bo_flag & BO_ONWORKLST) && bo->bo_dirty.bv_cnt == 0) { bo->bo_flag &= ~BO_ONWORKLST; mtx_lock(&sync_mtx); LIST_REMOVE(bo, bo_synclist); syncer_worklist_len--; mtx_unlock(&sync_mtx); } bp->b_vp = NULL; bp->b_bufobj = NULL; BO_UNLOCK(bo); vdrop(vp); } /* * Add an item to the syncer work queue. */ static void vn_syncer_add_to_worklist(struct bufobj *bo, int delay) { int slot; ASSERT_BO_WLOCKED(bo); mtx_lock(&sync_mtx); if (bo->bo_flag & BO_ONWORKLST) LIST_REMOVE(bo, bo_synclist); else { bo->bo_flag |= BO_ONWORKLST; syncer_worklist_len++; } if (delay > syncer_maxdelay - 2) delay = syncer_maxdelay - 2; slot = (syncer_delayno + delay) & syncer_mask; LIST_INSERT_HEAD(&syncer_workitem_pending[slot], bo, bo_synclist); mtx_unlock(&sync_mtx); } static int sysctl_vfs_worklist_len(SYSCTL_HANDLER_ARGS) { int error, len; mtx_lock(&sync_mtx); len = syncer_worklist_len - sync_vnode_count; mtx_unlock(&sync_mtx); error = SYSCTL_OUT(req, &len, sizeof(len)); return (error); } SYSCTL_PROC(_vfs, OID_AUTO, worklist_len, CTLTYPE_INT | CTLFLAG_RD, NULL, 0, sysctl_vfs_worklist_len, "I", "Syncer thread worklist length"); static struct proc *updateproc; static void sched_sync(void); static struct kproc_desc up_kp = { "syncer", sched_sync, &updateproc }; SYSINIT(syncer, SI_SUB_KTHREAD_UPDATE, SI_ORDER_FIRST, kproc_start, &up_kp); static int sync_vnode(struct synclist *slp, struct bufobj **bo, struct thread *td) { struct vnode *vp; struct mount *mp; *bo = LIST_FIRST(slp); if (*bo == NULL) return (0); vp = (*bo)->__bo_vnode; /* XXX */ if (VOP_ISLOCKED(vp) != 0 || VI_TRYLOCK(vp) == 0) return (1); /* * We use vhold in case the vnode does not * successfully sync. vhold prevents the vnode from * going away when we unlock the sync_mtx so that * we can acquire the vnode interlock. */ vholdl(vp); mtx_unlock(&sync_mtx); VI_UNLOCK(vp); if (vn_start_write(vp, &mp, V_NOWAIT) != 0) { vdrop(vp); mtx_lock(&sync_mtx); return (*bo == LIST_FIRST(slp)); } vn_lock(vp, LK_EXCLUSIVE | LK_RETRY); (void) VOP_FSYNC(vp, MNT_LAZY, td); VOP_UNLOCK(vp, 0); vn_finished_write(mp); BO_LOCK(*bo); if (((*bo)->bo_flag & BO_ONWORKLST) != 0) { /* * Put us back on the worklist. The worklist * routine will remove us from our current * position and then add us back in at a later * position. */ vn_syncer_add_to_worklist(*bo, syncdelay); } BO_UNLOCK(*bo); vdrop(vp); mtx_lock(&sync_mtx); return (0); } static int first_printf = 1; /* * System filesystem synchronizer daemon. */ static void sched_sync(void) { struct synclist *next, *slp; struct bufobj *bo; long starttime; struct thread *td = curthread; int last_work_seen; int net_worklist_len; int syncer_final_iter; int error; last_work_seen = 0; syncer_final_iter = 0; syncer_state = SYNCER_RUNNING; starttime = time_uptime; td->td_pflags |= TDP_NORUNNINGBUF; EVENTHANDLER_REGISTER(shutdown_pre_sync, syncer_shutdown, td->td_proc, SHUTDOWN_PRI_LAST); mtx_lock(&sync_mtx); for (;;) { if (syncer_state == SYNCER_FINAL_DELAY && syncer_final_iter == 0) { mtx_unlock(&sync_mtx); kproc_suspend_check(td->td_proc); mtx_lock(&sync_mtx); } net_worklist_len = syncer_worklist_len - sync_vnode_count; if (syncer_state != SYNCER_RUNNING && starttime != time_uptime) { if (first_printf) { printf("\nSyncing disks, vnodes remaining..."); first_printf = 0; } printf("%d ", net_worklist_len); } starttime = time_uptime; /* * Push files whose dirty time has expired. Be careful * of interrupt race on slp queue. * * Skip over empty worklist slots when shutting down. */ do { slp = &syncer_workitem_pending[syncer_delayno]; syncer_delayno += 1; if (syncer_delayno == syncer_maxdelay) syncer_delayno = 0; next = &syncer_workitem_pending[syncer_delayno]; /* * If the worklist has wrapped since the * it was emptied of all but syncer vnodes, * switch to the FINAL_DELAY state and run * for one more second. */ if (syncer_state == SYNCER_SHUTTING_DOWN && net_worklist_len == 0 && last_work_seen == syncer_delayno) { syncer_state = SYNCER_FINAL_DELAY; syncer_final_iter = SYNCER_SHUTDOWN_SPEEDUP; } } while (syncer_state != SYNCER_RUNNING && LIST_EMPTY(slp) && syncer_worklist_len > 0); /* * Keep track of the last time there was anything * on the worklist other than syncer vnodes. * Return to the SHUTTING_DOWN state if any * new work appears. */ if (net_worklist_len > 0 || syncer_state == SYNCER_RUNNING) last_work_seen = syncer_delayno; if (net_worklist_len > 0 && syncer_state == SYNCER_FINAL_DELAY) syncer_state = SYNCER_SHUTTING_DOWN; while (!LIST_EMPTY(slp)) { error = sync_vnode(slp, &bo, td); if (error == 1) { LIST_REMOVE(bo, bo_synclist); LIST_INSERT_HEAD(next, bo, bo_synclist); continue; } if (first_printf == 0) { /* * Drop the sync mutex, because some watchdog * drivers need to sleep while patting */ mtx_unlock(&sync_mtx); wdog_kern_pat(WD_LASTVAL); mtx_lock(&sync_mtx); } } if (syncer_state == SYNCER_FINAL_DELAY && syncer_final_iter > 0) syncer_final_iter--; /* * The variable rushjob allows the kernel to speed up the * processing of the filesystem syncer process. A rushjob * value of N tells the filesystem syncer to process the next * N seconds worth of work on its queue ASAP. Currently rushjob * is used by the soft update code to speed up the filesystem * syncer process when the incore state is getting so far * ahead of the disk that the kernel memory pool is being * threatened with exhaustion. */ if (rushjob > 0) { rushjob -= 1; continue; } /* * Just sleep for a short period of time between * iterations when shutting down to allow some I/O * to happen. * * If it has taken us less than a second to process the * current work, then wait. Otherwise start right over * again. We can still lose time if any single round * takes more than two seconds, but it does not really * matter as we are just trying to generally pace the * filesystem activity. */ if (syncer_state != SYNCER_RUNNING || time_uptime == starttime) { thread_lock(td); sched_prio(td, PPAUSE); thread_unlock(td); } if (syncer_state != SYNCER_RUNNING) cv_timedwait(&sync_wakeup, &sync_mtx, hz / SYNCER_SHUTDOWN_SPEEDUP); else if (time_uptime == starttime) cv_timedwait(&sync_wakeup, &sync_mtx, hz); } } /* * Request the syncer daemon to speed up its work. * We never push it to speed up more than half of its * normal turn time, otherwise it could take over the cpu. */ int speedup_syncer(void) { int ret = 0; mtx_lock(&sync_mtx); if (rushjob < syncdelay / 2) { rushjob += 1; stat_rush_requests += 1; ret = 1; } mtx_unlock(&sync_mtx); cv_broadcast(&sync_wakeup); return (ret); } /* * Tell the syncer to speed up its work and run though its work * list several times, then tell it to shut down. */ static void syncer_shutdown(void *arg, int howto) { if (howto & RB_NOSYNC) return; mtx_lock(&sync_mtx); syncer_state = SYNCER_SHUTTING_DOWN; rushjob = 0; mtx_unlock(&sync_mtx); cv_broadcast(&sync_wakeup); kproc_shutdown(arg, howto); } void syncer_suspend(void) { syncer_shutdown(updateproc, 0); } void syncer_resume(void) { mtx_lock(&sync_mtx); first_printf = 1; syncer_state = SYNCER_RUNNING; mtx_unlock(&sync_mtx); cv_broadcast(&sync_wakeup); kproc_resume(updateproc); } /* * Reassign a buffer from one vnode to another. * Used to assign file specific control information * (indirect blocks) to the vnode to which they belong. */ void reassignbuf(struct buf *bp) { struct vnode *vp; struct bufobj *bo; int delay; #ifdef INVARIANTS struct bufv *bv; #endif vp = bp->b_vp; bo = bp->b_bufobj; ++reassignbufcalls; CTR3(KTR_BUF, "reassignbuf(%p) vp %p flags %X", bp, bp->b_vp, bp->b_flags); /* * B_PAGING flagged buffers cannot be reassigned because their vp * is not fully linked in. */ if (bp->b_flags & B_PAGING) panic("cannot reassign paging buffer"); /* * Delete from old vnode list, if on one. */ BO_LOCK(bo); if (bp->b_xflags & (BX_VNDIRTY | BX_VNCLEAN)) buf_vlist_remove(bp); else panic("reassignbuf: Buffer %p not on queue.", bp); /* * If dirty, put on list of dirty buffers; otherwise insert onto list * of clean buffers. */ if (bp->b_flags & B_DELWRI) { if ((bo->bo_flag & BO_ONWORKLST) == 0) { switch (vp->v_type) { case VDIR: delay = dirdelay; break; case VCHR: delay = metadelay; break; default: delay = filedelay; } vn_syncer_add_to_worklist(bo, delay); } buf_vlist_add(bp, bo, BX_VNDIRTY); } else { buf_vlist_add(bp, bo, BX_VNCLEAN); if ((bo->bo_flag & BO_ONWORKLST) && bo->bo_dirty.bv_cnt == 0) { mtx_lock(&sync_mtx); LIST_REMOVE(bo, bo_synclist); syncer_worklist_len--; mtx_unlock(&sync_mtx); bo->bo_flag &= ~BO_ONWORKLST; } } #ifdef INVARIANTS bv = &bo->bo_clean; bp = TAILQ_FIRST(&bv->bv_hd); KASSERT(bp == NULL || bp->b_bufobj == bo, ("bp %p wrong b_bufobj %p should be %p", bp, bp->b_bufobj, bo)); bp = TAILQ_LAST(&bv->bv_hd, buflists); KASSERT(bp == NULL || bp->b_bufobj == bo, ("bp %p wrong b_bufobj %p should be %p", bp, bp->b_bufobj, bo)); bv = &bo->bo_dirty; bp = TAILQ_FIRST(&bv->bv_hd); KASSERT(bp == NULL || bp->b_bufobj == bo, ("bp %p wrong b_bufobj %p should be %p", bp, bp->b_bufobj, bo)); bp = TAILQ_LAST(&bv->bv_hd, buflists); KASSERT(bp == NULL || bp->b_bufobj == bo, ("bp %p wrong b_bufobj %p should be %p", bp, bp->b_bufobj, bo)); #endif BO_UNLOCK(bo); } /* * Increment the use and hold counts on the vnode, taking care to reference * the driver's usecount if this is a chardev. The vholdl() will remove * the vnode from the free list if it is presently free. Requires the * vnode interlock and returns with it held. */ static void v_incr_usecount(struct vnode *vp) { CTR2(KTR_VFS, "%s: vp %p", __func__, vp); vholdl(vp); vp->v_usecount++; if (vp->v_type == VCHR && vp->v_rdev != NULL) { dev_lock(); vp->v_rdev->si_usecount++; dev_unlock(); } } /* * Turn a holdcnt into a use+holdcnt such that only one call to * v_decr_usecount is needed. */ static void v_upgrade_usecount(struct vnode *vp) { CTR2(KTR_VFS, "%s: vp %p", __func__, vp); vp->v_usecount++; if (vp->v_type == VCHR && vp->v_rdev != NULL) { dev_lock(); vp->v_rdev->si_usecount++; dev_unlock(); } } /* * Decrement the vnode use and hold count along with the driver's usecount * if this is a chardev. The vdropl() below releases the vnode interlock * as it may free the vnode. */ static void v_decr_usecount(struct vnode *vp) { ASSERT_VI_LOCKED(vp, __FUNCTION__); VNASSERT(vp->v_usecount > 0, vp, ("v_decr_usecount: negative usecount")); CTR2(KTR_VFS, "%s: vp %p", __func__, vp); vp->v_usecount--; if (vp->v_type == VCHR && vp->v_rdev != NULL) { dev_lock(); vp->v_rdev->si_usecount--; dev_unlock(); } vdropl(vp); } /* * Decrement only the use count and driver use count. This is intended to * be paired with a follow on vdropl() to release the remaining hold count. * In this way we may vgone() a vnode with a 0 usecount without risk of * having it end up on a free list because the hold count is kept above 0. */ static void v_decr_useonly(struct vnode *vp) { ASSERT_VI_LOCKED(vp, __FUNCTION__); VNASSERT(vp->v_usecount > 0, vp, ("v_decr_useonly: negative usecount")); CTR2(KTR_VFS, "%s: vp %p", __func__, vp); vp->v_usecount--; if (vp->v_type == VCHR && vp->v_rdev != NULL) { dev_lock(); vp->v_rdev->si_usecount--; dev_unlock(); } } /* * Grab a particular vnode from the free list, increment its * reference count and lock it. VI_DOOMED is set if the vnode * is being destroyed. Only callers who specify LK_RETRY will * see doomed vnodes. If inactive processing was delayed in * vput try to do it here. */ int vget(struct vnode *vp, int flags, struct thread *td) { int error; error = 0; VNASSERT((flags & LK_TYPE_MASK) != 0, vp, ("vget: invalid lock operation")); CTR3(KTR_VFS, "%s: vp %p with flags %d", __func__, vp, flags); if ((flags & LK_INTERLOCK) == 0) VI_LOCK(vp); vholdl(vp); if ((error = vn_lock(vp, flags | LK_INTERLOCK)) != 0) { vdrop(vp); CTR2(KTR_VFS, "%s: impossible to lock vnode %p", __func__, vp); return (error); } if (vp->v_iflag & VI_DOOMED && (flags & LK_RETRY) == 0) panic("vget: vn_lock failed to return ENOENT\n"); VI_LOCK(vp); /* Upgrade our holdcnt to a usecount. */ v_upgrade_usecount(vp); /* * We don't guarantee that any particular close will * trigger inactive processing so just make a best effort * here at preventing a reference to a removed file. If * we don't succeed no harm is done. */ if (vp->v_iflag & VI_OWEINACT) { if (VOP_ISLOCKED(vp) == LK_EXCLUSIVE && (flags & LK_NOWAIT) == 0) vinactive(vp, td); vp->v_iflag &= ~VI_OWEINACT; } VI_UNLOCK(vp); return (0); } /* * Increase the reference count of a vnode. */ void vref(struct vnode *vp) { CTR2(KTR_VFS, "%s: vp %p", __func__, vp); VI_LOCK(vp); v_incr_usecount(vp); VI_UNLOCK(vp); } /* * Return reference count of a vnode. * * The results of this call are only guaranteed when some mechanism other * than the VI lock is used to stop other processes from gaining references * to the vnode. This may be the case if the caller holds the only reference. * This is also useful when stale data is acceptable as race conditions may * be accounted for by some other means. */ int vrefcnt(struct vnode *vp) { int usecnt; VI_LOCK(vp); usecnt = vp->v_usecount; VI_UNLOCK(vp); return (usecnt); } #define VPUTX_VRELE 1 #define VPUTX_VPUT 2 #define VPUTX_VUNREF 3 static void vputx(struct vnode *vp, int func) { int error; KASSERT(vp != NULL, ("vputx: null vp")); if (func == VPUTX_VUNREF) ASSERT_VOP_LOCKED(vp, "vunref"); else if (func == VPUTX_VPUT) ASSERT_VOP_LOCKED(vp, "vput"); else KASSERT(func == VPUTX_VRELE, ("vputx: wrong func")); CTR2(KTR_VFS, "%s: vp %p", __func__, vp); VI_LOCK(vp); /* Skip this v_writecount check if we're going to panic below. */ VNASSERT(vp->v_writecount < vp->v_usecount || vp->v_usecount < 1, vp, ("vputx: missed vn_close")); error = 0; if (vp->v_usecount > 1 || ((vp->v_iflag & VI_DOINGINACT) && vp->v_usecount == 1)) { if (func == VPUTX_VPUT) VOP_UNLOCK(vp, 0); v_decr_usecount(vp); return; } if (vp->v_usecount != 1) { vprint("vputx: negative ref count", vp); panic("vputx: negative ref cnt"); } CTR2(KTR_VFS, "%s: return vnode %p to the freelist", __func__, vp); /* * We want to hold the vnode until the inactive finishes to * prevent vgone() races. We drop the use count here and the * hold count below when we're done. */ v_decr_useonly(vp); /* * We must call VOP_INACTIVE with the node locked. Mark * as VI_DOINGINACT to avoid recursion. */ vp->v_iflag |= VI_OWEINACT; switch (func) { case VPUTX_VRELE: error = vn_lock(vp, LK_EXCLUSIVE | LK_INTERLOCK); VI_LOCK(vp); break; case VPUTX_VPUT: if (VOP_ISLOCKED(vp) != LK_EXCLUSIVE) { error = VOP_LOCK(vp, LK_UPGRADE | LK_INTERLOCK | LK_NOWAIT); VI_LOCK(vp); } break; case VPUTX_VUNREF: if (VOP_ISLOCKED(vp) != LK_EXCLUSIVE) { error = VOP_LOCK(vp, LK_TRYUPGRADE | LK_INTERLOCK); VI_LOCK(vp); } break; } if (vp->v_usecount > 0) vp->v_iflag &= ~VI_OWEINACT; if (error == 0) { if (vp->v_iflag & VI_OWEINACT) vinactive(vp, curthread); if (func != VPUTX_VUNREF) VOP_UNLOCK(vp, 0); } vdropl(vp); } /* * Vnode put/release. * If count drops to zero, call inactive routine and return to freelist. */ void vrele(struct vnode *vp) { vputx(vp, VPUTX_VRELE); } /* * Release an already locked vnode. This give the same effects as * unlock+vrele(), but takes less time and avoids releasing and * re-aquiring the lock (as vrele() acquires the lock internally.) */ void vput(struct vnode *vp) { vputx(vp, VPUTX_VPUT); } /* * Release an exclusively locked vnode. Do not unlock the vnode lock. */ void vunref(struct vnode *vp) { vputx(vp, VPUTX_VUNREF); } /* * Somebody doesn't want the vnode recycled. */ void vhold(struct vnode *vp) { VI_LOCK(vp); vholdl(vp); VI_UNLOCK(vp); } /* * Increase the hold count and activate if this is the first reference. */ void vholdl(struct vnode *vp) { struct mount *mp; CTR2(KTR_VFS, "%s: vp %p", __func__, vp); #ifdef INVARIANTS /* getnewvnode() calls v_incr_usecount() without holding interlock. */ if (vp->v_type != VNON || vp->v_data != NULL) { ASSERT_VI_LOCKED(vp, "vholdl"); VNASSERT(vp->v_holdcnt > 0 || (vp->v_iflag & VI_FREE) != 0, vp, ("vholdl: free vnode is held")); } #endif vp->v_holdcnt++; if ((vp->v_iflag & VI_FREE) == 0) return; VNASSERT(vp->v_holdcnt == 1, vp, ("vholdl: wrong hold count")); VNASSERT(vp->v_op != NULL, vp, ("vholdl: vnode already reclaimed.")); /* * Remove a vnode from the free list, mark it as in use, * and put it on the active list. */ mtx_lock(&vnode_free_list_mtx); TAILQ_REMOVE(&vnode_free_list, vp, v_actfreelist); freevnodes--; vp->v_iflag &= ~(VI_FREE|VI_AGE); KASSERT((vp->v_iflag & VI_ACTIVE) == 0, ("Activating already active vnode")); vp->v_iflag |= VI_ACTIVE; mp = vp->v_mount; TAILQ_INSERT_HEAD(&mp->mnt_activevnodelist, vp, v_actfreelist); mp->mnt_activevnodelistsize++; mtx_unlock(&vnode_free_list_mtx); } /* * Note that there is one less who cares about this vnode. * vdrop() is the opposite of vhold(). */ void vdrop(struct vnode *vp) { VI_LOCK(vp); vdropl(vp); } /* * Drop the hold count of the vnode. If this is the last reference to * the vnode we place it on the free list unless it has been vgone'd * (marked VI_DOOMED) in which case we will free it. */ void vdropl(struct vnode *vp) { struct bufobj *bo; struct mount *mp; int active; ASSERT_VI_LOCKED(vp, "vdropl"); CTR2(KTR_VFS, "%s: vp %p", __func__, vp); if (vp->v_holdcnt <= 0) panic("vdrop: holdcnt %d", vp->v_holdcnt); vp->v_holdcnt--; VNASSERT(vp->v_holdcnt >= vp->v_usecount, vp, ("hold count less than use count")); if (vp->v_holdcnt > 0) { VI_UNLOCK(vp); return; } if ((vp->v_iflag & VI_DOOMED) == 0) { /* * Mark a vnode as free: remove it from its active list * and put it up for recycling on the freelist. */ VNASSERT(vp->v_op != NULL, vp, ("vdropl: vnode already reclaimed.")); VNASSERT((vp->v_iflag & VI_FREE) == 0, vp, ("vnode already free")); VNASSERT(vp->v_holdcnt == 0, vp, ("vdropl: freeing when we shouldn't")); active = vp->v_iflag & VI_ACTIVE; vp->v_iflag &= ~VI_ACTIVE; mp = vp->v_mount; mtx_lock(&vnode_free_list_mtx); if (active) { TAILQ_REMOVE(&mp->mnt_activevnodelist, vp, v_actfreelist); mp->mnt_activevnodelistsize--; } if (vp->v_iflag & VI_AGE) { TAILQ_INSERT_HEAD(&vnode_free_list, vp, v_actfreelist); } else { TAILQ_INSERT_TAIL(&vnode_free_list, vp, v_actfreelist); } freevnodes++; vp->v_iflag &= ~VI_AGE; vp->v_iflag |= VI_FREE; mtx_unlock(&vnode_free_list_mtx); VI_UNLOCK(vp); return; } /* * The vnode has been marked for destruction, so free it. */ CTR2(KTR_VFS, "%s: destroying the vnode %p", __func__, vp); atomic_subtract_long(&numvnodes, 1); bo = &vp->v_bufobj; VNASSERT((vp->v_iflag & VI_FREE) == 0, vp, ("cleaned vnode still on the free list.")); VNASSERT(vp->v_data == NULL, vp, ("cleaned vnode isn't")); VNASSERT(vp->v_holdcnt == 0, vp, ("Non-zero hold count")); VNASSERT(vp->v_usecount == 0, vp, ("Non-zero use count")); VNASSERT(vp->v_writecount == 0, vp, ("Non-zero write count")); VNASSERT(bo->bo_numoutput == 0, vp, ("Clean vnode has pending I/O's")); VNASSERT(bo->bo_clean.bv_cnt == 0, vp, ("cleanbufcnt not 0")); VNASSERT(pctrie_is_empty(&bo->bo_clean.bv_root), vp, ("clean blk trie not empty")); VNASSERT(bo->bo_dirty.bv_cnt == 0, vp, ("dirtybufcnt not 0")); VNASSERT(pctrie_is_empty(&bo->bo_dirty.bv_root), vp, ("dirty blk trie not empty")); VNASSERT(TAILQ_EMPTY(&vp->v_cache_dst), vp, ("vp has namecache dst")); VNASSERT(LIST_EMPTY(&vp->v_cache_src), vp, ("vp has namecache src")); VNASSERT(vp->v_cache_dd == NULL, vp, ("vp has namecache for ..")); VI_UNLOCK(vp); #ifdef MAC mac_vnode_destroy(vp); #endif if (vp->v_pollinfo != NULL) destroy_vpollinfo(vp->v_pollinfo); #ifdef INVARIANTS /* XXX Elsewhere we detect an already freed vnode via NULL v_op. */ vp->v_op = NULL; #endif rangelock_destroy(&vp->v_rl); lockdestroy(vp->v_vnlock); mtx_destroy(&vp->v_interlock); rw_destroy(BO_LOCKPTR(bo)); uma_zfree(vnode_zone, vp); } /* * Call VOP_INACTIVE on the vnode and manage the DOINGINACT and OWEINACT * flags. DOINGINACT prevents us from recursing in calls to vinactive. * OWEINACT tracks whether a vnode missed a call to inactive due to a * failed lock upgrade. */ void vinactive(struct vnode *vp, struct thread *td) { struct vm_object *obj; ASSERT_VOP_ELOCKED(vp, "vinactive"); ASSERT_VI_LOCKED(vp, "vinactive"); VNASSERT((vp->v_iflag & VI_DOINGINACT) == 0, vp, ("vinactive: recursed on VI_DOINGINACT")); CTR2(KTR_VFS, "%s: vp %p", __func__, vp); vp->v_iflag |= VI_DOINGINACT; vp->v_iflag &= ~VI_OWEINACT; VI_UNLOCK(vp); /* * Before moving off the active list, we must be sure that any * modified pages are on the vnode's dirty list since these will * no longer be checked once the vnode is on the inactive list. * Because the vnode vm object keeps a hold reference on the vnode * if there is at least one resident non-cached page, the vnode * cannot leave the active list without the page cleanup done. */ obj = vp->v_object; if (obj != NULL && (obj->flags & OBJ_MIGHTBEDIRTY) != 0) { VM_OBJECT_WLOCK(obj); vm_object_page_clean(obj, 0, 0, OBJPC_NOSYNC); VM_OBJECT_WUNLOCK(obj); } VOP_INACTIVE(vp, td); VI_LOCK(vp); VNASSERT(vp->v_iflag & VI_DOINGINACT, vp, ("vinactive: lost VI_DOINGINACT")); vp->v_iflag &= ~VI_DOINGINACT; } /* * Remove any vnodes in the vnode table belonging to mount point mp. * * If FORCECLOSE is not specified, there should not be any active ones, * return error if any are found (nb: this is a user error, not a * system error). If FORCECLOSE is specified, detach any active vnodes * that are found. * * If WRITECLOSE is set, only flush out regular file vnodes open for * writing. * * SKIPSYSTEM causes any vnodes marked VV_SYSTEM to be skipped. * * `rootrefs' specifies the base reference count for the root vnode * of this filesystem. The root vnode is considered busy if its * v_usecount exceeds this value. On a successful return, vflush(, td) * will call vrele() on the root vnode exactly rootrefs times. * If the SKIPSYSTEM or WRITECLOSE flags are specified, rootrefs must * be zero. */ #ifdef DIAGNOSTIC static int busyprt = 0; /* print out busy vnodes */ SYSCTL_INT(_debug, OID_AUTO, busyprt, CTLFLAG_RW, &busyprt, 0, "Print out busy vnodes"); #endif int vflush(struct mount *mp, int rootrefs, int flags, struct thread *td) { struct vnode *vp, *mvp, *rootvp = NULL; struct vattr vattr; int busy = 0, error; CTR4(KTR_VFS, "%s: mp %p with rootrefs %d and flags %d", __func__, mp, rootrefs, flags); if (rootrefs > 0) { KASSERT((flags & (SKIPSYSTEM | WRITECLOSE)) == 0, ("vflush: bad args")); /* * Get the filesystem root vnode. We can vput() it * immediately, since with rootrefs > 0, it won't go away. */ if ((error = VFS_ROOT(mp, LK_EXCLUSIVE, &rootvp)) != 0) { CTR2(KTR_VFS, "%s: vfs_root lookup failed with %d", __func__, error); return (error); } vput(rootvp); } loop: MNT_VNODE_FOREACH_ALL(vp, mp, mvp) { vholdl(vp); error = vn_lock(vp, LK_INTERLOCK | LK_EXCLUSIVE); if (error) { vdrop(vp); MNT_VNODE_FOREACH_ALL_ABORT(mp, mvp); goto loop; } /* * Skip over a vnodes marked VV_SYSTEM. */ if ((flags & SKIPSYSTEM) && (vp->v_vflag & VV_SYSTEM)) { VOP_UNLOCK(vp, 0); vdrop(vp); continue; } /* * If WRITECLOSE is set, flush out unlinked but still open * files (even if open only for reading) and regular file * vnodes open for writing. */ if (flags & WRITECLOSE) { if (vp->v_object != NULL) { VM_OBJECT_WLOCK(vp->v_object); vm_object_page_clean(vp->v_object, 0, 0, 0); VM_OBJECT_WUNLOCK(vp->v_object); } error = VOP_FSYNC(vp, MNT_WAIT, td); if (error != 0) { VOP_UNLOCK(vp, 0); vdrop(vp); MNT_VNODE_FOREACH_ALL_ABORT(mp, mvp); return (error); } error = VOP_GETATTR(vp, &vattr, td->td_ucred); VI_LOCK(vp); if ((vp->v_type == VNON || (error == 0 && vattr.va_nlink > 0)) && (vp->v_writecount == 0 || vp->v_type != VREG)) { VOP_UNLOCK(vp, 0); vdropl(vp); continue; } } else VI_LOCK(vp); /* * With v_usecount == 0, all we need to do is clear out the * vnode data structures and we are done. * * If FORCECLOSE is set, forcibly close the vnode. */ if (vp->v_usecount == 0 || (flags & FORCECLOSE)) { VNASSERT(vp->v_usecount == 0 || (vp->v_type != VCHR && vp->v_type != VBLK), vp, ("device VNODE %p is FORCECLOSED", vp)); vgonel(vp); } else { busy++; #ifdef DIAGNOSTIC if (busyprt) vprint("vflush: busy vnode", vp); #endif } VOP_UNLOCK(vp, 0); vdropl(vp); } if (rootrefs > 0 && (flags & FORCECLOSE) == 0) { /* * If just the root vnode is busy, and if its refcount * is equal to `rootrefs', then go ahead and kill it. */ VI_LOCK(rootvp); KASSERT(busy > 0, ("vflush: not busy")); VNASSERT(rootvp->v_usecount >= rootrefs, rootvp, ("vflush: usecount %d < rootrefs %d", rootvp->v_usecount, rootrefs)); if (busy == 1 && rootvp->v_usecount == rootrefs) { VOP_LOCK(rootvp, LK_EXCLUSIVE|LK_INTERLOCK); vgone(rootvp); VOP_UNLOCK(rootvp, 0); busy = 0; } else VI_UNLOCK(rootvp); } if (busy) { CTR2(KTR_VFS, "%s: failing as %d vnodes are busy", __func__, busy); return (EBUSY); } for (; rootrefs > 0; rootrefs--) vrele(rootvp); return (0); } /* * Recycle an unused vnode to the front of the free list. */ int vrecycle(struct vnode *vp) { int recycled; ASSERT_VOP_ELOCKED(vp, "vrecycle"); CTR2(KTR_VFS, "%s: vp %p", __func__, vp); recycled = 0; VI_LOCK(vp); if (vp->v_usecount == 0) { recycled = 1; vgonel(vp); } VI_UNLOCK(vp); return (recycled); } /* * Eliminate all activity associated with a vnode * in preparation for reuse. */ void vgone(struct vnode *vp) { VI_LOCK(vp); vgonel(vp); VI_UNLOCK(vp); } static void notify_lowervp_vfs_dummy(struct mount *mp __unused, struct vnode *lowervp __unused) { } /* * Notify upper mounts about reclaimed or unlinked vnode. */ void vfs_notify_upper(struct vnode *vp, int event) { static struct vfsops vgonel_vfsops = { .vfs_reclaim_lowervp = notify_lowervp_vfs_dummy, .vfs_unlink_lowervp = notify_lowervp_vfs_dummy, }; struct mount *mp, *ump, *mmp; mp = vp->v_mount; if (mp == NULL) return; MNT_ILOCK(mp); if (TAILQ_EMPTY(&mp->mnt_uppers)) goto unlock; MNT_IUNLOCK(mp); mmp = malloc(sizeof(struct mount), M_TEMP, M_WAITOK | M_ZERO); mmp->mnt_op = &vgonel_vfsops; mmp->mnt_kern_flag |= MNTK_MARKER; MNT_ILOCK(mp); mp->mnt_kern_flag |= MNTK_VGONE_UPPER; for (ump = TAILQ_FIRST(&mp->mnt_uppers); ump != NULL;) { if ((ump->mnt_kern_flag & MNTK_MARKER) != 0) { ump = TAILQ_NEXT(ump, mnt_upper_link); continue; } TAILQ_INSERT_AFTER(&mp->mnt_uppers, ump, mmp, mnt_upper_link); MNT_IUNLOCK(mp); switch (event) { case VFS_NOTIFY_UPPER_RECLAIM: VFS_RECLAIM_LOWERVP(ump, vp); break; case VFS_NOTIFY_UPPER_UNLINK: VFS_UNLINK_LOWERVP(ump, vp); break; default: KASSERT(0, ("invalid event %d", event)); break; } MNT_ILOCK(mp); ump = TAILQ_NEXT(mmp, mnt_upper_link); TAILQ_REMOVE(&mp->mnt_uppers, mmp, mnt_upper_link); } free(mmp, M_TEMP); mp->mnt_kern_flag &= ~MNTK_VGONE_UPPER; if ((mp->mnt_kern_flag & MNTK_VGONE_WAITER) != 0) { mp->mnt_kern_flag &= ~MNTK_VGONE_WAITER; wakeup(&mp->mnt_uppers); } unlock: MNT_IUNLOCK(mp); } /* * vgone, with the vp interlock held. */ void vgonel(struct vnode *vp) { struct thread *td; int oweinact; int active; struct mount *mp; ASSERT_VOP_ELOCKED(vp, "vgonel"); ASSERT_VI_LOCKED(vp, "vgonel"); VNASSERT(vp->v_holdcnt, vp, ("vgonel: vp %p has no reference.", vp)); CTR2(KTR_VFS, "%s: vp %p", __func__, vp); td = curthread; /* * Don't vgonel if we're already doomed. */ if (vp->v_iflag & VI_DOOMED) return; vp->v_iflag |= VI_DOOMED; /* * Check to see if the vnode is in use. If so, we have to call * VOP_CLOSE() and VOP_INACTIVE(). */ active = vp->v_usecount; oweinact = (vp->v_iflag & VI_OWEINACT); VI_UNLOCK(vp); vfs_notify_upper(vp, VFS_NOTIFY_UPPER_RECLAIM); /* - * Clean out any buffers associated with the vnode. - * If the flush fails, just toss the buffers. - */ - mp = NULL; - if (!TAILQ_EMPTY(&vp->v_bufobj.bo_dirty.bv_hd)) - (void) vn_start_secondary_write(vp, &mp, V_WAIT); - if (vinvalbuf(vp, V_SAVE, 0, 0) != 0) - vinvalbuf(vp, 0, 0, 0); - - /* * If purging an active vnode, it must be closed and * deactivated before being reclaimed. */ if (active) VOP_CLOSE(vp, FNONBLOCK, NOCRED, td); if (oweinact || active) { VI_LOCK(vp); if ((vp->v_iflag & VI_DOINGINACT) == 0) vinactive(vp, td); VI_UNLOCK(vp); } if (vp->v_type == VSOCK) vfs_unp_reclaim(vp); + + /* + * Clean out any buffers associated with the vnode. + * If the flush fails, just toss the buffers. + */ + mp = NULL; + if (!TAILQ_EMPTY(&vp->v_bufobj.bo_dirty.bv_hd)) + (void) vn_start_secondary_write(vp, &mp, V_WAIT); + if (vinvalbuf(vp, V_SAVE, 0, 0) != 0) { + while (vinvalbuf(vp, 0, 0, 0) != 0) + ; + } +#ifdef INVARIANTS + BO_LOCK(&vp->v_bufobj); + KASSERT(TAILQ_EMPTY(&vp->v_bufobj.bo_dirty.bv_hd) && + vp->v_bufobj.bo_dirty.bv_cnt == 0 && + TAILQ_EMPTY(&vp->v_bufobj.bo_clean.bv_hd) && + vp->v_bufobj.bo_clean.bv_cnt == 0, + ("vp %p bufobj not invalidated", vp)); + vp->v_bufobj.bo_flag |= BO_DEAD; + BO_UNLOCK(&vp->v_bufobj); +#endif + /* * Reclaim the vnode. */ if (VOP_RECLAIM(vp, td)) panic("vgone: cannot reclaim"); if (mp != NULL) vn_finished_secondary_write(mp); VNASSERT(vp->v_object == NULL, vp, ("vop_reclaim left v_object vp=%p, tag=%s", vp, vp->v_tag)); /* * Clear the advisory locks and wake up waiting threads. */ (void)VOP_ADVLOCKPURGE(vp); /* * Delete from old mount point vnode list. */ delmntque(vp); cache_purge(vp); /* * Done with purge, reset to the standard lock and invalidate * the vnode. */ VI_LOCK(vp); vp->v_vnlock = &vp->v_lock; vp->v_op = &dead_vnodeops; vp->v_tag = "none"; vp->v_type = VBAD; } /* * Calculate the total number of references to a special device. */ int vcount(struct vnode *vp) { int count; dev_lock(); count = vp->v_rdev->si_usecount; dev_unlock(); return (count); } /* * Same as above, but using the struct cdev *as argument */ int count_dev(struct cdev *dev) { int count; dev_lock(); count = dev->si_usecount; dev_unlock(); return(count); } /* * Print out a description of a vnode. */ static char *typename[] = {"VNON", "VREG", "VDIR", "VBLK", "VCHR", "VLNK", "VSOCK", "VFIFO", "VBAD", "VMARKER"}; void vn_printf(struct vnode *vp, const char *fmt, ...) { va_list ap; char buf[256], buf2[16]; u_long flags; va_start(ap, fmt); vprintf(fmt, ap); va_end(ap); printf("%p: ", (void *)vp); printf("tag %s, type %s\n", vp->v_tag, typename[vp->v_type]); printf(" usecount %d, writecount %d, refcount %d mountedhere %p\n", vp->v_usecount, vp->v_writecount, vp->v_holdcnt, vp->v_mountedhere); buf[0] = '\0'; buf[1] = '\0'; if (vp->v_vflag & VV_ROOT) strlcat(buf, "|VV_ROOT", sizeof(buf)); if (vp->v_vflag & VV_ISTTY) strlcat(buf, "|VV_ISTTY", sizeof(buf)); if (vp->v_vflag & VV_NOSYNC) strlcat(buf, "|VV_NOSYNC", sizeof(buf)); if (vp->v_vflag & VV_ETERNALDEV) strlcat(buf, "|VV_ETERNALDEV", sizeof(buf)); if (vp->v_vflag & VV_CACHEDLABEL) strlcat(buf, "|VV_CACHEDLABEL", sizeof(buf)); if (vp->v_vflag & VV_TEXT) strlcat(buf, "|VV_TEXT", sizeof(buf)); if (vp->v_vflag & VV_COPYONWRITE) strlcat(buf, "|VV_COPYONWRITE", sizeof(buf)); if (vp->v_vflag & VV_SYSTEM) strlcat(buf, "|VV_SYSTEM", sizeof(buf)); if (vp->v_vflag & VV_PROCDEP) strlcat(buf, "|VV_PROCDEP", sizeof(buf)); if (vp->v_vflag & VV_NOKNOTE) strlcat(buf, "|VV_NOKNOTE", sizeof(buf)); if (vp->v_vflag & VV_DELETED) strlcat(buf, "|VV_DELETED", sizeof(buf)); if (vp->v_vflag & VV_MD) strlcat(buf, "|VV_MD", sizeof(buf)); if (vp->v_vflag & VV_FORCEINSMQ) strlcat(buf, "|VV_FORCEINSMQ", sizeof(buf)); flags = vp->v_vflag & ~(VV_ROOT | VV_ISTTY | VV_NOSYNC | VV_ETERNALDEV | VV_CACHEDLABEL | VV_TEXT | VV_COPYONWRITE | VV_SYSTEM | VV_PROCDEP | VV_NOKNOTE | VV_DELETED | VV_MD | VV_FORCEINSMQ); if (flags != 0) { snprintf(buf2, sizeof(buf2), "|VV(0x%lx)", flags); strlcat(buf, buf2, sizeof(buf)); } if (vp->v_iflag & VI_MOUNT) strlcat(buf, "|VI_MOUNT", sizeof(buf)); if (vp->v_iflag & VI_AGE) strlcat(buf, "|VI_AGE", sizeof(buf)); if (vp->v_iflag & VI_DOOMED) strlcat(buf, "|VI_DOOMED", sizeof(buf)); if (vp->v_iflag & VI_FREE) strlcat(buf, "|VI_FREE", sizeof(buf)); if (vp->v_iflag & VI_ACTIVE) strlcat(buf, "|VI_ACTIVE", sizeof(buf)); if (vp->v_iflag & VI_DOINGINACT) strlcat(buf, "|VI_DOINGINACT", sizeof(buf)); if (vp->v_iflag & VI_OWEINACT) strlcat(buf, "|VI_OWEINACT", sizeof(buf)); flags = vp->v_iflag & ~(VI_MOUNT | VI_AGE | VI_DOOMED | VI_FREE | VI_ACTIVE | VI_DOINGINACT | VI_OWEINACT); if (flags != 0) { snprintf(buf2, sizeof(buf2), "|VI(0x%lx)", flags); strlcat(buf, buf2, sizeof(buf)); } printf(" flags (%s)\n", buf + 1); if (mtx_owned(VI_MTX(vp))) printf(" VI_LOCKed"); if (vp->v_object != NULL) printf(" v_object %p ref %d pages %d " "cleanbuf %d dirtybuf %d\n", vp->v_object, vp->v_object->ref_count, vp->v_object->resident_page_count, vp->v_bufobj.bo_dirty.bv_cnt, vp->v_bufobj.bo_clean.bv_cnt); printf(" "); lockmgr_printinfo(vp->v_vnlock); if (vp->v_data != NULL) VOP_PRINT(vp); } #ifdef DDB /* * List all of the locked vnodes in the system. * Called when debugging the kernel. */ DB_SHOW_COMMAND(lockedvnods, lockedvnodes) { struct mount *mp; struct vnode *vp; /* * Note: because this is DDB, we can't obey the locking semantics * for these structures, which means we could catch an inconsistent * state and dereference a nasty pointer. Not much to be done * about that. */ db_printf("Locked vnodes\n"); TAILQ_FOREACH(mp, &mountlist, mnt_list) { TAILQ_FOREACH(vp, &mp->mnt_nvnodelist, v_nmntvnodes) { if (vp->v_type != VMARKER && VOP_ISLOCKED(vp)) vprint("", vp); } } } /* * Show details about the given vnode. */ DB_SHOW_COMMAND(vnode, db_show_vnode) { struct vnode *vp; if (!have_addr) return; vp = (struct vnode *)addr; vn_printf(vp, "vnode "); } /* * Show details about the given mount point. */ DB_SHOW_COMMAND(mount, db_show_mount) { struct mount *mp; struct vfsopt *opt; struct statfs *sp; struct vnode *vp; char buf[512]; uint64_t mflags; u_int flags; if (!have_addr) { /* No address given, print short info about all mount points. */ TAILQ_FOREACH(mp, &mountlist, mnt_list) { db_printf("%p %s on %s (%s)\n", mp, mp->mnt_stat.f_mntfromname, mp->mnt_stat.f_mntonname, mp->mnt_stat.f_fstypename); if (db_pager_quit) break; } db_printf("\nMore info: show mount \n"); return; } mp = (struct mount *)addr; db_printf("%p %s on %s (%s)\n", mp, mp->mnt_stat.f_mntfromname, mp->mnt_stat.f_mntonname, mp->mnt_stat.f_fstypename); buf[0] = '\0'; mflags = mp->mnt_flag; #define MNT_FLAG(flag) do { \ if (mflags & (flag)) { \ if (buf[0] != '\0') \ strlcat(buf, ", ", sizeof(buf)); \ strlcat(buf, (#flag) + 4, sizeof(buf)); \ mflags &= ~(flag); \ } \ } while (0) MNT_FLAG(MNT_RDONLY); MNT_FLAG(MNT_SYNCHRONOUS); MNT_FLAG(MNT_NOEXEC); MNT_FLAG(MNT_NOSUID); MNT_FLAG(MNT_NFS4ACLS); MNT_FLAG(MNT_UNION); MNT_FLAG(MNT_ASYNC); MNT_FLAG(MNT_SUIDDIR); MNT_FLAG(MNT_SOFTDEP); MNT_FLAG(MNT_NOSYMFOLLOW); MNT_FLAG(MNT_GJOURNAL); MNT_FLAG(MNT_MULTILABEL); MNT_FLAG(MNT_ACLS); MNT_FLAG(MNT_NOATIME); MNT_FLAG(MNT_NOCLUSTERR); MNT_FLAG(MNT_NOCLUSTERW); MNT_FLAG(MNT_SUJ); MNT_FLAG(MNT_EXRDONLY); MNT_FLAG(MNT_EXPORTED); MNT_FLAG(MNT_DEFEXPORTED); MNT_FLAG(MNT_EXPORTANON); MNT_FLAG(MNT_EXKERB); MNT_FLAG(MNT_EXPUBLIC); MNT_FLAG(MNT_LOCAL); MNT_FLAG(MNT_QUOTA); MNT_FLAG(MNT_ROOTFS); MNT_FLAG(MNT_USER); MNT_FLAG(MNT_IGNORE); MNT_FLAG(MNT_UPDATE); MNT_FLAG(MNT_DELEXPORT); MNT_FLAG(MNT_RELOAD); MNT_FLAG(MNT_FORCE); MNT_FLAG(MNT_SNAPSHOT); MNT_FLAG(MNT_BYFSID); #undef MNT_FLAG if (mflags != 0) { if (buf[0] != '\0') strlcat(buf, ", ", sizeof(buf)); snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), "0x%016jx", mflags); } db_printf(" mnt_flag = %s\n", buf); buf[0] = '\0'; flags = mp->mnt_kern_flag; #define MNT_KERN_FLAG(flag) do { \ if (flags & (flag)) { \ if (buf[0] != '\0') \ strlcat(buf, ", ", sizeof(buf)); \ strlcat(buf, (#flag) + 5, sizeof(buf)); \ flags &= ~(flag); \ } \ } while (0) MNT_KERN_FLAG(MNTK_UNMOUNTF); MNT_KERN_FLAG(MNTK_ASYNC); MNT_KERN_FLAG(MNTK_SOFTDEP); MNT_KERN_FLAG(MNTK_NOINSMNTQ); MNT_KERN_FLAG(MNTK_DRAINING); MNT_KERN_FLAG(MNTK_REFEXPIRE); MNT_KERN_FLAG(MNTK_EXTENDED_SHARED); MNT_KERN_FLAG(MNTK_SHARED_WRITES); MNT_KERN_FLAG(MNTK_NO_IOPF); MNT_KERN_FLAG(MNTK_VGONE_UPPER); MNT_KERN_FLAG(MNTK_VGONE_WAITER); MNT_KERN_FLAG(MNTK_LOOKUP_EXCL_DOTDOT); MNT_KERN_FLAG(MNTK_MARKER); MNT_KERN_FLAG(MNTK_NOASYNC); MNT_KERN_FLAG(MNTK_UNMOUNT); MNT_KERN_FLAG(MNTK_MWAIT); MNT_KERN_FLAG(MNTK_SUSPEND); MNT_KERN_FLAG(MNTK_SUSPEND2); MNT_KERN_FLAG(MNTK_SUSPENDED); MNT_KERN_FLAG(MNTK_LOOKUP_SHARED); MNT_KERN_FLAG(MNTK_NOKNOTE); #undef MNT_KERN_FLAG if (flags != 0) { if (buf[0] != '\0') strlcat(buf, ", ", sizeof(buf)); snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), "0x%08x", flags); } db_printf(" mnt_kern_flag = %s\n", buf); db_printf(" mnt_opt = "); opt = TAILQ_FIRST(mp->mnt_opt); if (opt != NULL) { db_printf("%s", opt->name); opt = TAILQ_NEXT(opt, link); while (opt != NULL) { db_printf(", %s", opt->name); opt = TAILQ_NEXT(opt, link); } } db_printf("\n"); sp = &mp->mnt_stat; db_printf(" mnt_stat = { version=%u type=%u flags=0x%016jx " "bsize=%ju iosize=%ju blocks=%ju bfree=%ju bavail=%jd files=%ju " "ffree=%jd syncwrites=%ju asyncwrites=%ju syncreads=%ju " "asyncreads=%ju namemax=%u owner=%u fsid=[%d, %d] }\n", (u_int)sp->f_version, (u_int)sp->f_type, (uintmax_t)sp->f_flags, (uintmax_t)sp->f_bsize, (uintmax_t)sp->f_iosize, (uintmax_t)sp->f_blocks, (uintmax_t)sp->f_bfree, (intmax_t)sp->f_bavail, (uintmax_t)sp->f_files, (intmax_t)sp->f_ffree, (uintmax_t)sp->f_syncwrites, (uintmax_t)sp->f_asyncwrites, (uintmax_t)sp->f_syncreads, (uintmax_t)sp->f_asyncreads, (u_int)sp->f_namemax, (u_int)sp->f_owner, (int)sp->f_fsid.val[0], (int)sp->f_fsid.val[1]); db_printf(" mnt_cred = { uid=%u ruid=%u", (u_int)mp->mnt_cred->cr_uid, (u_int)mp->mnt_cred->cr_ruid); if (jailed(mp->mnt_cred)) db_printf(", jail=%d", mp->mnt_cred->cr_prison->pr_id); db_printf(" }\n"); db_printf(" mnt_ref = %d\n", mp->mnt_ref); db_printf(" mnt_gen = %d\n", mp->mnt_gen); db_printf(" mnt_nvnodelistsize = %d\n", mp->mnt_nvnodelistsize); db_printf(" mnt_activevnodelistsize = %d\n", mp->mnt_activevnodelistsize); db_printf(" mnt_writeopcount = %d\n", mp->mnt_writeopcount); db_printf(" mnt_maxsymlinklen = %d\n", mp->mnt_maxsymlinklen); db_printf(" mnt_iosize_max = %d\n", mp->mnt_iosize_max); db_printf(" mnt_hashseed = %u\n", mp->mnt_hashseed); db_printf(" mnt_secondary_writes = %d\n", mp->mnt_secondary_writes); db_printf(" mnt_secondary_accwrites = %d\n", mp->mnt_secondary_accwrites); db_printf(" mnt_gjprovider = %s\n", mp->mnt_gjprovider != NULL ? mp->mnt_gjprovider : "NULL"); db_printf("\n\nList of active vnodes\n"); TAILQ_FOREACH(vp, &mp->mnt_activevnodelist, v_actfreelist) { if (vp->v_type != VMARKER) { vn_printf(vp, "vnode "); if (db_pager_quit) break; } } db_printf("\n\nList of inactive vnodes\n"); TAILQ_FOREACH(vp, &mp->mnt_nvnodelist, v_nmntvnodes) { if (vp->v_type != VMARKER && (vp->v_iflag & VI_ACTIVE) == 0) { vn_printf(vp, "vnode "); if (db_pager_quit) break; } } } #endif /* DDB */ /* * Fill in a struct xvfsconf based on a struct vfsconf. */ static int vfsconf2x(struct sysctl_req *req, struct vfsconf *vfsp) { struct xvfsconf xvfsp; bzero(&xvfsp, sizeof(xvfsp)); strcpy(xvfsp.vfc_name, vfsp->vfc_name); xvfsp.vfc_typenum = vfsp->vfc_typenum; xvfsp.vfc_refcount = vfsp->vfc_refcount; xvfsp.vfc_flags = vfsp->vfc_flags; /* * These are unused in userland, we keep them * to not break binary compatibility. */ xvfsp.vfc_vfsops = NULL; xvfsp.vfc_next = NULL; return (SYSCTL_OUT(req, &xvfsp, sizeof(xvfsp))); } #ifdef COMPAT_FREEBSD32 struct xvfsconf32 { uint32_t vfc_vfsops; char vfc_name[MFSNAMELEN]; int32_t vfc_typenum; int32_t vfc_refcount; int32_t vfc_flags; uint32_t vfc_next; }; static int vfsconf2x32(struct sysctl_req *req, struct vfsconf *vfsp) { struct xvfsconf32 xvfsp; strcpy(xvfsp.vfc_name, vfsp->vfc_name); xvfsp.vfc_typenum = vfsp->vfc_typenum; xvfsp.vfc_refcount = vfsp->vfc_refcount; xvfsp.vfc_flags = vfsp->vfc_flags; xvfsp.vfc_vfsops = 0; xvfsp.vfc_next = 0; return (SYSCTL_OUT(req, &xvfsp, sizeof(xvfsp))); } #endif /* * Top level filesystem related information gathering. */ static int sysctl_vfs_conflist(SYSCTL_HANDLER_ARGS) { struct vfsconf *vfsp; int error; error = 0; vfsconf_slock(); TAILQ_FOREACH(vfsp, &vfsconf, vfc_list) { #ifdef COMPAT_FREEBSD32 if (req->flags & SCTL_MASK32) error = vfsconf2x32(req, vfsp); else #endif error = vfsconf2x(req, vfsp); if (error) break; } vfsconf_sunlock(); return (error); } SYSCTL_PROC(_vfs, OID_AUTO, conflist, CTLTYPE_OPAQUE | CTLFLAG_RD | CTLFLAG_MPSAFE, NULL, 0, sysctl_vfs_conflist, "S,xvfsconf", "List of all configured filesystems"); #ifndef BURN_BRIDGES static int sysctl_ovfs_conf(SYSCTL_HANDLER_ARGS); static int vfs_sysctl(SYSCTL_HANDLER_ARGS) { int *name = (int *)arg1 - 1; /* XXX */ u_int namelen = arg2 + 1; /* XXX */ struct vfsconf *vfsp; log(LOG_WARNING, "userland calling deprecated sysctl, " "please rebuild world\n"); #if 1 || defined(COMPAT_PRELITE2) /* Resolve ambiguity between VFS_VFSCONF and VFS_GENERIC. */ if (namelen == 1) return (sysctl_ovfs_conf(oidp, arg1, arg2, req)); #endif switch (name[1]) { case VFS_MAXTYPENUM: if (namelen != 2) return (ENOTDIR); return (SYSCTL_OUT(req, &maxvfsconf, sizeof(int))); case VFS_CONF: if (namelen != 3) return (ENOTDIR); /* overloaded */ vfsconf_slock(); TAILQ_FOREACH(vfsp, &vfsconf, vfc_list) { if (vfsp->vfc_typenum == name[2]) break; } vfsconf_sunlock(); if (vfsp == NULL) return (EOPNOTSUPP); #ifdef COMPAT_FREEBSD32 if (req->flags & SCTL_MASK32) return (vfsconf2x32(req, vfsp)); else #endif return (vfsconf2x(req, vfsp)); } return (EOPNOTSUPP); } static SYSCTL_NODE(_vfs, VFS_GENERIC, generic, CTLFLAG_RD | CTLFLAG_SKIP | CTLFLAG_MPSAFE, vfs_sysctl, "Generic filesystem"); #if 1 || defined(COMPAT_PRELITE2) static int sysctl_ovfs_conf(SYSCTL_HANDLER_ARGS) { int error; struct vfsconf *vfsp; struct ovfsconf ovfs; vfsconf_slock(); TAILQ_FOREACH(vfsp, &vfsconf, vfc_list) { bzero(&ovfs, sizeof(ovfs)); ovfs.vfc_vfsops = vfsp->vfc_vfsops; /* XXX used as flag */ strcpy(ovfs.vfc_name, vfsp->vfc_name); ovfs.vfc_index = vfsp->vfc_typenum; ovfs.vfc_refcount = vfsp->vfc_refcount; ovfs.vfc_flags = vfsp->vfc_flags; error = SYSCTL_OUT(req, &ovfs, sizeof ovfs); if (error != 0) { vfsconf_sunlock(); return (error); } } vfsconf_sunlock(); return (0); } #endif /* 1 || COMPAT_PRELITE2 */ #endif /* !BURN_BRIDGES */ #define KINFO_VNODESLOP 10 #ifdef notyet /* * Dump vnode list (via sysctl). */ /* ARGSUSED */ static int sysctl_vnode(SYSCTL_HANDLER_ARGS) { struct xvnode *xvn; struct mount *mp; struct vnode *vp; int error, len, n; /* * Stale numvnodes access is not fatal here. */ req->lock = 0; len = (numvnodes + KINFO_VNODESLOP) * sizeof *xvn; if (!req->oldptr) /* Make an estimate */ return (SYSCTL_OUT(req, 0, len)); error = sysctl_wire_old_buffer(req, 0); if (error != 0) return (error); xvn = malloc(len, M_TEMP, M_ZERO | M_WAITOK); n = 0; mtx_lock(&mountlist_mtx); TAILQ_FOREACH(mp, &mountlist, mnt_list) { if (vfs_busy(mp, MBF_NOWAIT | MBF_MNTLSTLOCK)) continue; MNT_ILOCK(mp); TAILQ_FOREACH(vp, &mp->mnt_nvnodelist, v_nmntvnodes) { if (n == len) break; vref(vp); xvn[n].xv_size = sizeof *xvn; xvn[n].xv_vnode = vp; xvn[n].xv_id = 0; /* XXX compat */ #define XV_COPY(field) xvn[n].xv_##field = vp->v_##field XV_COPY(usecount); XV_COPY(writecount); XV_COPY(holdcnt); XV_COPY(mount); XV_COPY(numoutput); XV_COPY(type); #undef XV_COPY xvn[n].xv_flag = vp->v_vflag; switch (vp->v_type) { case VREG: case VDIR: case VLNK: break; case VBLK: case VCHR: if (vp->v_rdev == NULL) { vrele(vp); continue; } xvn[n].xv_dev = dev2udev(vp->v_rdev); break; case VSOCK: xvn[n].xv_socket = vp->v_socket; break; case VFIFO: xvn[n].xv_fifo = vp->v_fifoinfo; break; case VNON: case VBAD: default: /* shouldn't happen? */ vrele(vp); continue; } vrele(vp); ++n; } MNT_IUNLOCK(mp); mtx_lock(&mountlist_mtx); vfs_unbusy(mp); if (n == len) break; } mtx_unlock(&mountlist_mtx); error = SYSCTL_OUT(req, xvn, n * sizeof *xvn); free(xvn, M_TEMP); return (error); } SYSCTL_PROC(_kern, KERN_VNODE, vnode, CTLTYPE_OPAQUE | CTLFLAG_RD | CTLFLAG_MPSAFE, 0, 0, sysctl_vnode, "S,xvnode", ""); #endif /* * Unmount all filesystems. The list is traversed in reverse order * of mounting to avoid dependencies. */ void vfs_unmountall(void) { struct mount *mp; struct thread *td; int error; CTR1(KTR_VFS, "%s: unmounting all filesystems", __func__); td = curthread; /* * Since this only runs when rebooting, it is not interlocked. */ while(!TAILQ_EMPTY(&mountlist)) { mp = TAILQ_LAST(&mountlist, mntlist); error = dounmount(mp, MNT_FORCE, td); if (error) { TAILQ_REMOVE(&mountlist, mp, mnt_list); /* * XXX: Due to the way in which we mount the root * file system off of devfs, devfs will generate a * "busy" warning when we try to unmount it before * the root. Don't print a warning as a result in * order to avoid false positive errors that may * cause needless upset. */ if (strcmp(mp->mnt_vfc->vfc_name, "devfs") != 0) { printf("unmount of %s failed (", mp->mnt_stat.f_mntonname); if (error == EBUSY) printf("BUSY)\n"); else printf("%d)\n", error); } } else { /* The unmount has removed mp from the mountlist */ } } } /* * perform msync on all vnodes under a mount point * the mount point must be locked. */ void vfs_msync(struct mount *mp, int flags) { struct vnode *vp, *mvp; struct vm_object *obj; CTR2(KTR_VFS, "%s: mp %p", __func__, mp); MNT_VNODE_FOREACH_ACTIVE(vp, mp, mvp) { obj = vp->v_object; if (obj != NULL && (obj->flags & OBJ_MIGHTBEDIRTY) != 0 && (flags == MNT_WAIT || VOP_ISLOCKED(vp) == 0)) { if (!vget(vp, LK_EXCLUSIVE | LK_RETRY | LK_INTERLOCK, curthread)) { if (vp->v_vflag & VV_NOSYNC) { /* unlinked */ vput(vp); continue; } obj = vp->v_object; if (obj != NULL) { VM_OBJECT_WLOCK(obj); vm_object_page_clean(obj, 0, 0, flags == MNT_WAIT ? OBJPC_SYNC : OBJPC_NOSYNC); VM_OBJECT_WUNLOCK(obj); } vput(vp); } } else VI_UNLOCK(vp); } } static void destroy_vpollinfo_free(struct vpollinfo *vi) { knlist_destroy(&vi->vpi_selinfo.si_note); mtx_destroy(&vi->vpi_lock); uma_zfree(vnodepoll_zone, vi); } static void destroy_vpollinfo(struct vpollinfo *vi) { knlist_clear(&vi->vpi_selinfo.si_note, 1); seldrain(&vi->vpi_selinfo); destroy_vpollinfo_free(vi); } /* * Initalize per-vnode helper structure to hold poll-related state. */ void v_addpollinfo(struct vnode *vp) { struct vpollinfo *vi; if (vp->v_pollinfo != NULL) return; vi = uma_zalloc(vnodepoll_zone, M_WAITOK); mtx_init(&vi->vpi_lock, "vnode pollinfo", NULL, MTX_DEF); knlist_init(&vi->vpi_selinfo.si_note, vp, vfs_knllock, vfs_knlunlock, vfs_knl_assert_locked, vfs_knl_assert_unlocked); VI_LOCK(vp); if (vp->v_pollinfo != NULL) { VI_UNLOCK(vp); destroy_vpollinfo_free(vi); return; } vp->v_pollinfo = vi; VI_UNLOCK(vp); } /* * Record a process's interest in events which might happen to * a vnode. Because poll uses the historic select-style interface * internally, this routine serves as both the ``check for any * pending events'' and the ``record my interest in future events'' * functions. (These are done together, while the lock is held, * to avoid race conditions.) */ int vn_pollrecord(struct vnode *vp, struct thread *td, int events) { v_addpollinfo(vp); mtx_lock(&vp->v_pollinfo->vpi_lock); if (vp->v_pollinfo->vpi_revents & events) { /* * This leaves events we are not interested * in available for the other process which * which presumably had requested them * (otherwise they would never have been * recorded). */ events &= vp->v_pollinfo->vpi_revents; vp->v_pollinfo->vpi_revents &= ~events; mtx_unlock(&vp->v_pollinfo->vpi_lock); return (events); } vp->v_pollinfo->vpi_events |= events; selrecord(td, &vp->v_pollinfo->vpi_selinfo); mtx_unlock(&vp->v_pollinfo->vpi_lock); return (0); } /* * Routine to create and manage a filesystem syncer vnode. */ #define sync_close ((int (*)(struct vop_close_args *))nullop) static int sync_fsync(struct vop_fsync_args *); static int sync_inactive(struct vop_inactive_args *); static int sync_reclaim(struct vop_reclaim_args *); static struct vop_vector sync_vnodeops = { .vop_bypass = VOP_EOPNOTSUPP, .vop_close = sync_close, /* close */ .vop_fsync = sync_fsync, /* fsync */ .vop_inactive = sync_inactive, /* inactive */ .vop_reclaim = sync_reclaim, /* reclaim */ .vop_lock1 = vop_stdlock, /* lock */ .vop_unlock = vop_stdunlock, /* unlock */ .vop_islocked = vop_stdislocked, /* islocked */ }; /* * Create a new filesystem syncer vnode for the specified mount point. */ void vfs_allocate_syncvnode(struct mount *mp) { struct vnode *vp; struct bufobj *bo; static long start, incr, next; int error; /* Allocate a new vnode */ error = getnewvnode("syncer", mp, &sync_vnodeops, &vp); if (error != 0) panic("vfs_allocate_syncvnode: getnewvnode() failed"); vp->v_type = VNON; vn_lock(vp, LK_EXCLUSIVE | LK_RETRY); vp->v_vflag |= VV_FORCEINSMQ; error = insmntque(vp, mp); if (error != 0) panic("vfs_allocate_syncvnode: insmntque() failed"); vp->v_vflag &= ~VV_FORCEINSMQ; VOP_UNLOCK(vp, 0); /* * Place the vnode onto the syncer worklist. We attempt to * scatter them about on the list so that they will go off * at evenly distributed times even if all the filesystems * are mounted at once. */ next += incr; if (next == 0 || next > syncer_maxdelay) { start /= 2; incr /= 2; if (start == 0) { start = syncer_maxdelay / 2; incr = syncer_maxdelay; } next = start; } bo = &vp->v_bufobj; BO_LOCK(bo); vn_syncer_add_to_worklist(bo, syncdelay > 0 ? next % syncdelay : 0); /* XXX - vn_syncer_add_to_worklist() also grabs and drops sync_mtx. */ mtx_lock(&sync_mtx); sync_vnode_count++; if (mp->mnt_syncer == NULL) { mp->mnt_syncer = vp; vp = NULL; } mtx_unlock(&sync_mtx); BO_UNLOCK(bo); if (vp != NULL) { vn_lock(vp, LK_EXCLUSIVE | LK_RETRY); vgone(vp); vput(vp); } } void vfs_deallocate_syncvnode(struct mount *mp) { struct vnode *vp; mtx_lock(&sync_mtx); vp = mp->mnt_syncer; if (vp != NULL) mp->mnt_syncer = NULL; mtx_unlock(&sync_mtx); if (vp != NULL) vrele(vp); } /* * Do a lazy sync of the filesystem. */ static int sync_fsync(struct vop_fsync_args *ap) { struct vnode *syncvp = ap->a_vp; struct mount *mp = syncvp->v_mount; int error, save; struct bufobj *bo; /* * We only need to do something if this is a lazy evaluation. */ if (ap->a_waitfor != MNT_LAZY) return (0); /* * Move ourselves to the back of the sync list. */ bo = &syncvp->v_bufobj; BO_LOCK(bo); vn_syncer_add_to_worklist(bo, syncdelay); BO_UNLOCK(bo); /* * Walk the list of vnodes pushing all that are dirty and * not already on the sync list. */ if (vfs_busy(mp, MBF_NOWAIT) != 0) return (0); if (vn_start_write(NULL, &mp, V_NOWAIT) != 0) { vfs_unbusy(mp); return (0); } save = curthread_pflags_set(TDP_SYNCIO); vfs_msync(mp, MNT_NOWAIT); error = VFS_SYNC(mp, MNT_LAZY); curthread_pflags_restore(save); vn_finished_write(mp); vfs_unbusy(mp); return (error); } /* * The syncer vnode is no referenced. */ static int sync_inactive(struct vop_inactive_args *ap) { vgone(ap->a_vp); return (0); } /* * The syncer vnode is no longer needed and is being decommissioned. * * Modifications to the worklist must be protected by sync_mtx. */ static int sync_reclaim(struct vop_reclaim_args *ap) { struct vnode *vp = ap->a_vp; struct bufobj *bo; bo = &vp->v_bufobj; BO_LOCK(bo); mtx_lock(&sync_mtx); if (vp->v_mount->mnt_syncer == vp) vp->v_mount->mnt_syncer = NULL; if (bo->bo_flag & BO_ONWORKLST) { LIST_REMOVE(bo, bo_synclist); syncer_worklist_len--; sync_vnode_count--; bo->bo_flag &= ~BO_ONWORKLST; } mtx_unlock(&sync_mtx); BO_UNLOCK(bo); return (0); } /* * Check if vnode represents a disk device */ int vn_isdisk(struct vnode *vp, int *errp) { int error; if (vp->v_type != VCHR) { error = ENOTBLK; goto out; } error = 0; dev_lock(); if (vp->v_rdev == NULL) error = ENXIO; else if (vp->v_rdev->si_devsw == NULL) error = ENXIO; else if (!(vp->v_rdev->si_devsw->d_flags & D_DISK)) error = ENOTBLK; dev_unlock(); out: if (errp != NULL) *errp = error; return (error == 0); } /* * Common filesystem object access control check routine. Accepts a * vnode's type, "mode", uid and gid, requested access mode, credentials, * and optional call-by-reference privused argument allowing vaccess() * to indicate to the caller whether privilege was used to satisfy the * request (obsoleted). Returns 0 on success, or an errno on failure. */ int vaccess(enum vtype type, mode_t file_mode, uid_t file_uid, gid_t file_gid, accmode_t accmode, struct ucred *cred, int *privused) { accmode_t dac_granted; accmode_t priv_granted; KASSERT((accmode & ~(VEXEC | VWRITE | VREAD | VADMIN | VAPPEND)) == 0, ("invalid bit in accmode")); KASSERT((accmode & VAPPEND) == 0 || (accmode & VWRITE), ("VAPPEND without VWRITE")); /* * Look for a normal, non-privileged way to access the file/directory * as requested. If it exists, go with that. */ if (privused != NULL) *privused = 0; dac_granted = 0; /* Check the owner. */ if (cred->cr_uid == file_uid) { dac_granted |= VADMIN; if (file_mode & S_IXUSR) dac_granted |= VEXEC; if (file_mode & S_IRUSR) dac_granted |= VREAD; if (file_mode & S_IWUSR) dac_granted |= (VWRITE | VAPPEND); if ((accmode & dac_granted) == accmode) return (0); goto privcheck; } /* Otherwise, check the groups (first match) */ if (groupmember(file_gid, cred)) { if (file_mode & S_IXGRP) dac_granted |= VEXEC; if (file_mode & S_IRGRP) dac_granted |= VREAD; if (file_mode & S_IWGRP) dac_granted |= (VWRITE | VAPPEND); if ((accmode & dac_granted) == accmode) return (0); goto privcheck; } /* Otherwise, check everyone else. */ if (file_mode & S_IXOTH) dac_granted |= VEXEC; if (file_mode & S_IROTH) dac_granted |= VREAD; if (file_mode & S_IWOTH) dac_granted |= (VWRITE | VAPPEND); if ((accmode & dac_granted) == accmode) return (0); privcheck: /* * Build a privilege mask to determine if the set of privileges * satisfies the requirements when combined with the granted mask * from above. For each privilege, if the privilege is required, * bitwise or the request type onto the priv_granted mask. */ priv_granted = 0; if (type == VDIR) { /* * For directories, use PRIV_VFS_LOOKUP to satisfy VEXEC * requests, instead of PRIV_VFS_EXEC. */ if ((accmode & VEXEC) && ((dac_granted & VEXEC) == 0) && !priv_check_cred(cred, PRIV_VFS_LOOKUP, 0)) priv_granted |= VEXEC; } else { /* * Ensure that at least one execute bit is on. Otherwise, * a privileged user will always succeed, and we don't want * this to happen unless the file really is executable. */ if ((accmode & VEXEC) && ((dac_granted & VEXEC) == 0) && (file_mode & (S_IXUSR | S_IXGRP | S_IXOTH)) != 0 && !priv_check_cred(cred, PRIV_VFS_EXEC, 0)) priv_granted |= VEXEC; } if ((accmode & VREAD) && ((dac_granted & VREAD) == 0) && !priv_check_cred(cred, PRIV_VFS_READ, 0)) priv_granted |= VREAD; if ((accmode & VWRITE) && ((dac_granted & VWRITE) == 0) && !priv_check_cred(cred, PRIV_VFS_WRITE, 0)) priv_granted |= (VWRITE | VAPPEND); if ((accmode & VADMIN) && ((dac_granted & VADMIN) == 0) && !priv_check_cred(cred, PRIV_VFS_ADMIN, 0)) priv_granted |= VADMIN; if ((accmode & (priv_granted | dac_granted)) == accmode) { /* XXX audit: privilege used */ if (privused != NULL) *privused = 1; return (0); } return ((accmode & VADMIN) ? EPERM : EACCES); } /* * Credential check based on process requesting service, and per-attribute * permissions. */ int extattr_check_cred(struct vnode *vp, int attrnamespace, struct ucred *cred, struct thread *td, accmode_t accmode) { /* * Kernel-invoked always succeeds. */ if (cred == NOCRED) return (0); /* * Do not allow privileged processes in jail to directly manipulate * system attributes. */ switch (attrnamespace) { case EXTATTR_NAMESPACE_SYSTEM: /* Potentially should be: return (EPERM); */ return (priv_check_cred(cred, PRIV_VFS_EXTATTR_SYSTEM, 0)); case EXTATTR_NAMESPACE_USER: return (VOP_ACCESS(vp, accmode, cred, td)); default: return (EPERM); } } #ifdef DEBUG_VFS_LOCKS /* * This only exists to supress warnings from unlocked specfs accesses. It is * no longer ok to have an unlocked VFS. */ #define IGNORE_LOCK(vp) (panicstr != NULL || (vp) == NULL || \ (vp)->v_type == VCHR || (vp)->v_type == VBAD) int vfs_badlock_ddb = 1; /* Drop into debugger on violation. */ SYSCTL_INT(_debug, OID_AUTO, vfs_badlock_ddb, CTLFLAG_RW, &vfs_badlock_ddb, 0, "Drop into debugger on lock violation"); int vfs_badlock_mutex = 1; /* Check for interlock across VOPs. */ SYSCTL_INT(_debug, OID_AUTO, vfs_badlock_mutex, CTLFLAG_RW, &vfs_badlock_mutex, 0, "Check for interlock across VOPs"); int vfs_badlock_print = 1; /* Print lock violations. */ SYSCTL_INT(_debug, OID_AUTO, vfs_badlock_print, CTLFLAG_RW, &vfs_badlock_print, 0, "Print lock violations"); #ifdef KDB int vfs_badlock_backtrace = 1; /* Print backtrace at lock violations. */ SYSCTL_INT(_debug, OID_AUTO, vfs_badlock_backtrace, CTLFLAG_RW, &vfs_badlock_backtrace, 0, "Print backtrace at lock violations"); #endif static void vfs_badlock(const char *msg, const char *str, struct vnode *vp) { #ifdef KDB if (vfs_badlock_backtrace) kdb_backtrace(); #endif if (vfs_badlock_print) printf("%s: %p %s\n", str, (void *)vp, msg); if (vfs_badlock_ddb) kdb_enter(KDB_WHY_VFSLOCK, "lock violation"); } void assert_vi_locked(struct vnode *vp, const char *str) { if (vfs_badlock_mutex && !mtx_owned(VI_MTX(vp))) vfs_badlock("interlock is not locked but should be", str, vp); } void assert_vi_unlocked(struct vnode *vp, const char *str) { if (vfs_badlock_mutex && mtx_owned(VI_MTX(vp))) vfs_badlock("interlock is locked but should not be", str, vp); } void assert_vop_locked(struct vnode *vp, const char *str) { int locked; if (!IGNORE_LOCK(vp)) { locked = VOP_ISLOCKED(vp); if (locked == 0 || locked == LK_EXCLOTHER) vfs_badlock("is not locked but should be", str, vp); } } void assert_vop_unlocked(struct vnode *vp, const char *str) { if (!IGNORE_LOCK(vp) && VOP_ISLOCKED(vp) == LK_EXCLUSIVE) vfs_badlock("is locked but should not be", str, vp); } void assert_vop_elocked(struct vnode *vp, const char *str) { if (!IGNORE_LOCK(vp) && VOP_ISLOCKED(vp) != LK_EXCLUSIVE) vfs_badlock("is not exclusive locked but should be", str, vp); } #if 0 void assert_vop_elocked_other(struct vnode *vp, const char *str) { if (!IGNORE_LOCK(vp) && VOP_ISLOCKED(vp) != LK_EXCLOTHER) vfs_badlock("is not exclusive locked by another thread", str, vp); } void assert_vop_slocked(struct vnode *vp, const char *str) { if (!IGNORE_LOCK(vp) && VOP_ISLOCKED(vp) != LK_SHARED) vfs_badlock("is not locked shared but should be", str, vp); } #endif /* 0 */ #endif /* DEBUG_VFS_LOCKS */ void vop_rename_fail(struct vop_rename_args *ap) { if (ap->a_tvp != NULL) vput(ap->a_tvp); if (ap->a_tdvp == ap->a_tvp) vrele(ap->a_tdvp); else vput(ap->a_tdvp); vrele(ap->a_fdvp); vrele(ap->a_fvp); } void vop_rename_pre(void *ap) { struct vop_rename_args *a = ap; #ifdef DEBUG_VFS_LOCKS if (a->a_tvp) ASSERT_VI_UNLOCKED(a->a_tvp, "VOP_RENAME"); ASSERT_VI_UNLOCKED(a->a_tdvp, "VOP_RENAME"); ASSERT_VI_UNLOCKED(a->a_fvp, "VOP_RENAME"); ASSERT_VI_UNLOCKED(a->a_fdvp, "VOP_RENAME"); /* Check the source (from). */ if (a->a_tdvp->v_vnlock != a->a_fdvp->v_vnlock && (a->a_tvp == NULL || a->a_tvp->v_vnlock != a->a_fdvp->v_vnlock)) ASSERT_VOP_UNLOCKED(a->a_fdvp, "vop_rename: fdvp locked"); if (a->a_tvp == NULL || a->a_tvp->v_vnlock != a->a_fvp->v_vnlock) ASSERT_VOP_UNLOCKED(a->a_fvp, "vop_rename: fvp locked"); /* Check the target. */ if (a->a_tvp) ASSERT_VOP_LOCKED(a->a_tvp, "vop_rename: tvp not locked"); ASSERT_VOP_LOCKED(a->a_tdvp, "vop_rename: tdvp not locked"); #endif if (a->a_tdvp != a->a_fdvp) vhold(a->a_fdvp); if (a->a_tvp != a->a_fvp) vhold(a->a_fvp); vhold(a->a_tdvp); if (a->a_tvp) vhold(a->a_tvp); } void vop_strategy_pre(void *ap) { #ifdef DEBUG_VFS_LOCKS struct vop_strategy_args *a; struct buf *bp; a = ap; bp = a->a_bp; /* * Cluster ops lock their component buffers but not the IO container. */ if ((bp->b_flags & B_CLUSTER) != 0) return; if (panicstr == NULL && !BUF_ISLOCKED(bp)) { if (vfs_badlock_print) printf( "VOP_STRATEGY: bp is not locked but should be\n"); if (vfs_badlock_ddb) kdb_enter(KDB_WHY_VFSLOCK, "lock violation"); } #endif } void vop_lock_pre(void *ap) { #ifdef DEBUG_VFS_LOCKS struct vop_lock1_args *a = ap; if ((a->a_flags & LK_INTERLOCK) == 0) ASSERT_VI_UNLOCKED(a->a_vp, "VOP_LOCK"); else ASSERT_VI_LOCKED(a->a_vp, "VOP_LOCK"); #endif } void vop_lock_post(void *ap, int rc) { #ifdef DEBUG_VFS_LOCKS struct vop_lock1_args *a = ap; ASSERT_VI_UNLOCKED(a->a_vp, "VOP_LOCK"); if (rc == 0 && (a->a_flags & LK_EXCLOTHER) == 0) ASSERT_VOP_LOCKED(a->a_vp, "VOP_LOCK"); #endif } void vop_unlock_pre(void *ap) { #ifdef DEBUG_VFS_LOCKS struct vop_unlock_args *a = ap; if (a->a_flags & LK_INTERLOCK) ASSERT_VI_LOCKED(a->a_vp, "VOP_UNLOCK"); ASSERT_VOP_LOCKED(a->a_vp, "VOP_UNLOCK"); #endif } void vop_unlock_post(void *ap, int rc) { #ifdef DEBUG_VFS_LOCKS struct vop_unlock_args *a = ap; if (a->a_flags & LK_INTERLOCK) ASSERT_VI_UNLOCKED(a->a_vp, "VOP_UNLOCK"); #endif } void vop_create_post(void *ap, int rc) { struct vop_create_args *a = ap; if (!rc) VFS_KNOTE_LOCKED(a->a_dvp, NOTE_WRITE); } void vop_deleteextattr_post(void *ap, int rc) { struct vop_deleteextattr_args *a = ap; if (!rc) VFS_KNOTE_LOCKED(a->a_vp, NOTE_ATTRIB); } void vop_link_post(void *ap, int rc) { struct vop_link_args *a = ap; if (!rc) { VFS_KNOTE_LOCKED(a->a_vp, NOTE_LINK); VFS_KNOTE_LOCKED(a->a_tdvp, NOTE_WRITE); } } void vop_mkdir_post(void *ap, int rc) { struct vop_mkdir_args *a = ap; if (!rc) VFS_KNOTE_LOCKED(a->a_dvp, NOTE_WRITE | NOTE_LINK); } void vop_mknod_post(void *ap, int rc) { struct vop_mknod_args *a = ap; if (!rc) VFS_KNOTE_LOCKED(a->a_dvp, NOTE_WRITE); } void vop_remove_post(void *ap, int rc) { struct vop_remove_args *a = ap; if (!rc) { VFS_KNOTE_LOCKED(a->a_dvp, NOTE_WRITE); VFS_KNOTE_LOCKED(a->a_vp, NOTE_DELETE); } } void vop_rename_post(void *ap, int rc) { struct vop_rename_args *a = ap; if (!rc) { VFS_KNOTE_UNLOCKED(a->a_fdvp, NOTE_WRITE); VFS_KNOTE_UNLOCKED(a->a_tdvp, NOTE_WRITE); VFS_KNOTE_UNLOCKED(a->a_fvp, NOTE_RENAME); if (a->a_tvp) VFS_KNOTE_UNLOCKED(a->a_tvp, NOTE_DELETE); } if (a->a_tdvp != a->a_fdvp) vdrop(a->a_fdvp); if (a->a_tvp != a->a_fvp) vdrop(a->a_fvp); vdrop(a->a_tdvp); if (a->a_tvp) vdrop(a->a_tvp); } void vop_rmdir_post(void *ap, int rc) { struct vop_rmdir_args *a = ap; if (!rc) { VFS_KNOTE_LOCKED(a->a_dvp, NOTE_WRITE | NOTE_LINK); VFS_KNOTE_LOCKED(a->a_vp, NOTE_DELETE); } } void vop_setattr_post(void *ap, int rc) { struct vop_setattr_args *a = ap; if (!rc) VFS_KNOTE_LOCKED(a->a_vp, NOTE_ATTRIB); } void vop_setextattr_post(void *ap, int rc) { struct vop_setextattr_args *a = ap; if (!rc) VFS_KNOTE_LOCKED(a->a_vp, NOTE_ATTRIB); } void vop_symlink_post(void *ap, int rc) { struct vop_symlink_args *a = ap; if (!rc) VFS_KNOTE_LOCKED(a->a_dvp, NOTE_WRITE); } static struct knlist fs_knlist; static void vfs_event_init(void *arg) { knlist_init_mtx(&fs_knlist, NULL); } /* XXX - correct order? */ SYSINIT(vfs_knlist, SI_SUB_VFS, SI_ORDER_ANY, vfs_event_init, NULL); void vfs_event_signal(fsid_t *fsid, uint32_t event, intptr_t data __unused) { KNOTE_UNLOCKED(&fs_knlist, event); } static int filt_fsattach(struct knote *kn); static void filt_fsdetach(struct knote *kn); static int filt_fsevent(struct knote *kn, long hint); struct filterops fs_filtops = { .f_isfd = 0, .f_attach = filt_fsattach, .f_detach = filt_fsdetach, .f_event = filt_fsevent }; static int filt_fsattach(struct knote *kn) { kn->kn_flags |= EV_CLEAR; knlist_add(&fs_knlist, kn, 0); return (0); } static void filt_fsdetach(struct knote *kn) { knlist_remove(&fs_knlist, kn, 0); } static int filt_fsevent(struct knote *kn, long hint) { kn->kn_fflags |= hint; return (kn->kn_fflags != 0); } static int sysctl_vfs_ctl(SYSCTL_HANDLER_ARGS) { struct vfsidctl vc; int error; struct mount *mp; error = SYSCTL_IN(req, &vc, sizeof(vc)); if (error) return (error); if (vc.vc_vers != VFS_CTL_VERS1) return (EINVAL); mp = vfs_getvfs(&vc.vc_fsid); if (mp == NULL) return (ENOENT); /* ensure that a specific sysctl goes to the right filesystem. */ if (strcmp(vc.vc_fstypename, "*") != 0 && strcmp(vc.vc_fstypename, mp->mnt_vfc->vfc_name) != 0) { vfs_rel(mp); return (EINVAL); } VCTLTOREQ(&vc, req); error = VFS_SYSCTL(mp, vc.vc_op, req); vfs_rel(mp); return (error); } SYSCTL_PROC(_vfs, OID_AUTO, ctl, CTLTYPE_OPAQUE | CTLFLAG_WR, NULL, 0, sysctl_vfs_ctl, "", "Sysctl by fsid"); /* * Function to initialize a va_filerev field sensibly. * XXX: Wouldn't a random number make a lot more sense ?? */ u_quad_t init_va_filerev(void) { struct bintime bt; getbinuptime(&bt); return (((u_quad_t)bt.sec << 32LL) | (bt.frac >> 32LL)); } static int filt_vfsread(struct knote *kn, long hint); static int filt_vfswrite(struct knote *kn, long hint); static int filt_vfsvnode(struct knote *kn, long hint); static void filt_vfsdetach(struct knote *kn); static struct filterops vfsread_filtops = { .f_isfd = 1, .f_detach = filt_vfsdetach, .f_event = filt_vfsread }; static struct filterops vfswrite_filtops = { .f_isfd = 1, .f_detach = filt_vfsdetach, .f_event = filt_vfswrite }; static struct filterops vfsvnode_filtops = { .f_isfd = 1, .f_detach = filt_vfsdetach, .f_event = filt_vfsvnode }; static void vfs_knllock(void *arg) { struct vnode *vp = arg; vn_lock(vp, LK_EXCLUSIVE | LK_RETRY); } static void vfs_knlunlock(void *arg) { struct vnode *vp = arg; VOP_UNLOCK(vp, 0); } static void vfs_knl_assert_locked(void *arg) { #ifdef DEBUG_VFS_LOCKS struct vnode *vp = arg; ASSERT_VOP_LOCKED(vp, "vfs_knl_assert_locked"); #endif } static void vfs_knl_assert_unlocked(void *arg) { #ifdef DEBUG_VFS_LOCKS struct vnode *vp = arg; ASSERT_VOP_UNLOCKED(vp, "vfs_knl_assert_unlocked"); #endif } int vfs_kqfilter(struct vop_kqfilter_args *ap) { struct vnode *vp = ap->a_vp; struct knote *kn = ap->a_kn; struct knlist *knl; switch (kn->kn_filter) { case EVFILT_READ: kn->kn_fop = &vfsread_filtops; break; case EVFILT_WRITE: kn->kn_fop = &vfswrite_filtops; break; case EVFILT_VNODE: kn->kn_fop = &vfsvnode_filtops; break; default: return (EINVAL); } kn->kn_hook = (caddr_t)vp; v_addpollinfo(vp); if (vp->v_pollinfo == NULL) return (ENOMEM); knl = &vp->v_pollinfo->vpi_selinfo.si_note; vhold(vp); knlist_add(knl, kn, 0); return (0); } /* * Detach knote from vnode */ static void filt_vfsdetach(struct knote *kn) { struct vnode *vp = (struct vnode *)kn->kn_hook; KASSERT(vp->v_pollinfo != NULL, ("Missing v_pollinfo")); knlist_remove(&vp->v_pollinfo->vpi_selinfo.si_note, kn, 0); vdrop(vp); } /*ARGSUSED*/ static int filt_vfsread(struct knote *kn, long hint) { struct vnode *vp = (struct vnode *)kn->kn_hook; struct vattr va; int res; /* * filesystem is gone, so set the EOF flag and schedule * the knote for deletion. */ if (hint == NOTE_REVOKE) { VI_LOCK(vp); kn->kn_flags |= (EV_EOF | EV_ONESHOT); VI_UNLOCK(vp); return (1); } if (VOP_GETATTR(vp, &va, curthread->td_ucred)) return (0); VI_LOCK(vp); kn->kn_data = va.va_size - kn->kn_fp->f_offset; res = (kn->kn_data != 0); VI_UNLOCK(vp); return (res); } /*ARGSUSED*/ static int filt_vfswrite(struct knote *kn, long hint) { struct vnode *vp = (struct vnode *)kn->kn_hook; VI_LOCK(vp); /* * filesystem is gone, so set the EOF flag and schedule * the knote for deletion. */ if (hint == NOTE_REVOKE) kn->kn_flags |= (EV_EOF | EV_ONESHOT); kn->kn_data = 0; VI_UNLOCK(vp); return (1); } static int filt_vfsvnode(struct knote *kn, long hint) { struct vnode *vp = (struct vnode *)kn->kn_hook; int res; VI_LOCK(vp); if (kn->kn_sfflags & hint) kn->kn_fflags |= hint; if (hint == NOTE_REVOKE) { kn->kn_flags |= EV_EOF; VI_UNLOCK(vp); return (1); } res = (kn->kn_fflags != 0); VI_UNLOCK(vp); return (res); } int vfs_read_dirent(struct vop_readdir_args *ap, struct dirent *dp, off_t off) { int error; if (dp->d_reclen > ap->a_uio->uio_resid) return (ENAMETOOLONG); error = uiomove(dp, dp->d_reclen, ap->a_uio); if (error) { if (ap->a_ncookies != NULL) { if (ap->a_cookies != NULL) free(ap->a_cookies, M_TEMP); ap->a_cookies = NULL; *ap->a_ncookies = 0; } return (error); } if (ap->a_ncookies == NULL) return (0); KASSERT(ap->a_cookies, ("NULL ap->a_cookies value with non-NULL ap->a_ncookies!")); *ap->a_cookies = realloc(*ap->a_cookies, (*ap->a_ncookies + 1) * sizeof(u_long), M_TEMP, M_WAITOK | M_ZERO); (*ap->a_cookies)[*ap->a_ncookies] = off; return (0); } /* * Mark for update the access time of the file if the filesystem * supports VOP_MARKATIME. This functionality is used by execve and * mmap, so we want to avoid the I/O implied by directly setting * va_atime for the sake of efficiency. */ void vfs_mark_atime(struct vnode *vp, struct ucred *cred) { struct mount *mp; mp = vp->v_mount; ASSERT_VOP_LOCKED(vp, "vfs_mark_atime"); if (mp != NULL && (mp->mnt_flag & (MNT_NOATIME | MNT_RDONLY)) == 0) (void)VOP_MARKATIME(vp); } /* * The purpose of this routine is to remove granularity from accmode_t, * reducing it into standard unix access bits - VEXEC, VREAD, VWRITE, * VADMIN and VAPPEND. * * If it returns 0, the caller is supposed to continue with the usual * access checks using 'accmode' as modified by this routine. If it * returns nonzero value, the caller is supposed to return that value * as errno. * * Note that after this routine runs, accmode may be zero. */ int vfs_unixify_accmode(accmode_t *accmode) { /* * There is no way to specify explicit "deny" rule using * file mode or POSIX.1e ACLs. */ if (*accmode & VEXPLICIT_DENY) { *accmode = 0; return (0); } /* * None of these can be translated into usual access bits. * Also, the common case for NFSv4 ACLs is to not contain * either of these bits. Caller should check for VWRITE * on the containing directory instead. */ if (*accmode & (VDELETE_CHILD | VDELETE)) return (EPERM); if (*accmode & VADMIN_PERMS) { *accmode &= ~VADMIN_PERMS; *accmode |= VADMIN; } /* * There is no way to deny VREAD_ATTRIBUTES, VREAD_ACL * or VSYNCHRONIZE using file mode or POSIX.1e ACL. */ *accmode &= ~(VSTAT_PERMS | VSYNCHRONIZE); return (0); } /* * These are helper functions for filesystems to traverse all * their vnodes. See MNT_VNODE_FOREACH_ALL() in sys/mount.h. * * This interface replaces MNT_VNODE_FOREACH. */ MALLOC_DEFINE(M_VNODE_MARKER, "vnodemarker", "vnode marker"); struct vnode * __mnt_vnode_next_all(struct vnode **mvp, struct mount *mp) { struct vnode *vp; if (should_yield()) kern_yield(PRI_USER); MNT_ILOCK(mp); KASSERT((*mvp)->v_mount == mp, ("marker vnode mount list mismatch")); vp = TAILQ_NEXT(*mvp, v_nmntvnodes); while (vp != NULL && (vp->v_type == VMARKER || (vp->v_iflag & VI_DOOMED) != 0)) vp = TAILQ_NEXT(vp, v_nmntvnodes); /* Check if we are done */ if (vp == NULL) { __mnt_vnode_markerfree_all(mvp, mp); /* MNT_IUNLOCK(mp); -- done in above function */ mtx_assert(MNT_MTX(mp), MA_NOTOWNED); return (NULL); } TAILQ_REMOVE(&mp->mnt_nvnodelist, *mvp, v_nmntvnodes); TAILQ_INSERT_AFTER(&mp->mnt_nvnodelist, vp, *mvp, v_nmntvnodes); VI_LOCK(vp); MNT_IUNLOCK(mp); return (vp); } struct vnode * __mnt_vnode_first_all(struct vnode **mvp, struct mount *mp) { struct vnode *vp; *mvp = malloc(sizeof(struct vnode), M_VNODE_MARKER, M_WAITOK | M_ZERO); MNT_ILOCK(mp); MNT_REF(mp); (*mvp)->v_type = VMARKER; vp = TAILQ_FIRST(&mp->mnt_nvnodelist); while (vp != NULL && (vp->v_type == VMARKER || (vp->v_iflag & VI_DOOMED) != 0)) vp = TAILQ_NEXT(vp, v_nmntvnodes); /* Check if we are done */ if (vp == NULL) { MNT_REL(mp); MNT_IUNLOCK(mp); free(*mvp, M_VNODE_MARKER); *mvp = NULL; return (NULL); } (*mvp)->v_mount = mp; TAILQ_INSERT_AFTER(&mp->mnt_nvnodelist, vp, *mvp, v_nmntvnodes); VI_LOCK(vp); MNT_IUNLOCK(mp); return (vp); } void __mnt_vnode_markerfree_all(struct vnode **mvp, struct mount *mp) { if (*mvp == NULL) { MNT_IUNLOCK(mp); return; } mtx_assert(MNT_MTX(mp), MA_OWNED); KASSERT((*mvp)->v_mount == mp, ("marker vnode mount list mismatch")); TAILQ_REMOVE(&mp->mnt_nvnodelist, *mvp, v_nmntvnodes); MNT_REL(mp); MNT_IUNLOCK(mp); free(*mvp, M_VNODE_MARKER); *mvp = NULL; } /* * These are helper functions for filesystems to traverse their * active vnodes. See MNT_VNODE_FOREACH_ACTIVE() in sys/mount.h */ static void mnt_vnode_markerfree_active(struct vnode **mvp, struct mount *mp) { KASSERT((*mvp)->v_mount == mp, ("marker vnode mount list mismatch")); MNT_ILOCK(mp); MNT_REL(mp); MNT_IUNLOCK(mp); free(*mvp, M_VNODE_MARKER); *mvp = NULL; } static struct vnode * mnt_vnode_next_active(struct vnode **mvp, struct mount *mp) { struct vnode *vp, *nvp; mtx_assert(&vnode_free_list_mtx, MA_OWNED); KASSERT((*mvp)->v_mount == mp, ("marker vnode mount list mismatch")); restart: vp = TAILQ_NEXT(*mvp, v_actfreelist); TAILQ_REMOVE(&mp->mnt_activevnodelist, *mvp, v_actfreelist); while (vp != NULL) { if (vp->v_type == VMARKER) { vp = TAILQ_NEXT(vp, v_actfreelist); continue; } if (!VI_TRYLOCK(vp)) { if (mp_ncpus == 1 || should_yield()) { TAILQ_INSERT_BEFORE(vp, *mvp, v_actfreelist); mtx_unlock(&vnode_free_list_mtx); pause("vnacti", 1); mtx_lock(&vnode_free_list_mtx); goto restart; } continue; } KASSERT(vp->v_type != VMARKER, ("locked marker %p", vp)); KASSERT(vp->v_mount == mp || vp->v_mount == NULL, ("alien vnode on the active list %p %p", vp, mp)); if (vp->v_mount == mp && (vp->v_iflag & VI_DOOMED) == 0) break; nvp = TAILQ_NEXT(vp, v_actfreelist); VI_UNLOCK(vp); vp = nvp; } /* Check if we are done */ if (vp == NULL) { mtx_unlock(&vnode_free_list_mtx); mnt_vnode_markerfree_active(mvp, mp); return (NULL); } TAILQ_INSERT_AFTER(&mp->mnt_activevnodelist, vp, *mvp, v_actfreelist); mtx_unlock(&vnode_free_list_mtx); ASSERT_VI_LOCKED(vp, "active iter"); KASSERT((vp->v_iflag & VI_ACTIVE) != 0, ("Non-active vp %p", vp)); return (vp); } struct vnode * __mnt_vnode_next_active(struct vnode **mvp, struct mount *mp) { if (should_yield()) kern_yield(PRI_USER); mtx_lock(&vnode_free_list_mtx); return (mnt_vnode_next_active(mvp, mp)); } struct vnode * __mnt_vnode_first_active(struct vnode **mvp, struct mount *mp) { struct vnode *vp; *mvp = malloc(sizeof(struct vnode), M_VNODE_MARKER, M_WAITOK | M_ZERO); MNT_ILOCK(mp); MNT_REF(mp); MNT_IUNLOCK(mp); (*mvp)->v_type = VMARKER; (*mvp)->v_mount = mp; mtx_lock(&vnode_free_list_mtx); vp = TAILQ_FIRST(&mp->mnt_activevnodelist); if (vp == NULL) { mtx_unlock(&vnode_free_list_mtx); mnt_vnode_markerfree_active(mvp, mp); return (NULL); } TAILQ_INSERT_BEFORE(vp, *mvp, v_actfreelist); return (mnt_vnode_next_active(mvp, mp)); } void __mnt_vnode_markerfree_active(struct vnode **mvp, struct mount *mp) { if (*mvp == NULL) return; mtx_lock(&vnode_free_list_mtx); TAILQ_REMOVE(&mp->mnt_activevnodelist, *mvp, v_actfreelist); mtx_unlock(&vnode_free_list_mtx); mnt_vnode_markerfree_active(mvp, mp); } Index: projects/clang350-import/sys/kern/vfs_vnops.c =================================================================== --- projects/clang350-import/sys/kern/vfs_vnops.c (revision 275748) +++ projects/clang350-import/sys/kern/vfs_vnops.c (revision 275749) @@ -1,2345 +1,2348 @@ /*- * Copyright (c) 1982, 1986, 1989, 1993 * The Regents of the University of California. All rights reserved. * (c) UNIX System Laboratories, Inc. * All or some portions of this file are derived from material licensed * to the University of California by American Telephone and Telegraph * Co. or Unix System Laboratories, Inc. and are reproduced herein with * the permission of UNIX System Laboratories, Inc. * * Copyright (c) 2012 Konstantin Belousov * Copyright (c) 2013, 2014 The FreeBSD Foundation * * Portions of this software were developed by Konstantin Belousov * under sponsorship from the FreeBSD Foundation. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 4. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * @(#)vfs_vnops.c 8.2 (Berkeley) 1/21/94 */ #include __FBSDID("$FreeBSD$"); #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include static fo_rdwr_t vn_read; static fo_rdwr_t vn_write; static fo_rdwr_t vn_io_fault; static fo_truncate_t vn_truncate; static fo_ioctl_t vn_ioctl; static fo_poll_t vn_poll; static fo_kqfilter_t vn_kqfilter; static fo_stat_t vn_statfile; static fo_close_t vn_closefile; struct fileops vnops = { .fo_read = vn_io_fault, .fo_write = vn_io_fault, .fo_truncate = vn_truncate, .fo_ioctl = vn_ioctl, .fo_poll = vn_poll, .fo_kqfilter = vn_kqfilter, .fo_stat = vn_statfile, .fo_close = vn_closefile, .fo_chmod = vn_chmod, .fo_chown = vn_chown, .fo_sendfile = vn_sendfile, .fo_seek = vn_seek, .fo_fill_kinfo = vn_fill_kinfo, .fo_flags = DFLAG_PASSABLE | DFLAG_SEEKABLE }; static const int io_hold_cnt = 16; static int vn_io_fault_enable = 1; SYSCTL_INT(_debug, OID_AUTO, vn_io_fault_enable, CTLFLAG_RW, &vn_io_fault_enable, 0, "Enable vn_io_fault lock avoidance"); static u_long vn_io_faults_cnt; SYSCTL_ULONG(_debug, OID_AUTO, vn_io_faults, CTLFLAG_RD, &vn_io_faults_cnt, 0, "Count of vn_io_fault lock avoidance triggers"); /* * Returns true if vn_io_fault mode of handling the i/o request should * be used. */ static bool do_vn_io_fault(struct vnode *vp, struct uio *uio) { struct mount *mp; return (uio->uio_segflg == UIO_USERSPACE && vp->v_type == VREG && (mp = vp->v_mount) != NULL && (mp->mnt_kern_flag & MNTK_NO_IOPF) != 0 && vn_io_fault_enable); } /* * Structure used to pass arguments to vn_io_fault1(), to do either * file- or vnode-based I/O calls. */ struct vn_io_fault_args { enum { VN_IO_FAULT_FOP, VN_IO_FAULT_VOP } kind; struct ucred *cred; int flags; union { struct fop_args_tag { struct file *fp; fo_rdwr_t *doio; } fop_args; struct vop_args_tag { struct vnode *vp; } vop_args; } args; }; static int vn_io_fault1(struct vnode *vp, struct uio *uio, struct vn_io_fault_args *args, struct thread *td); int vn_open(ndp, flagp, cmode, fp) struct nameidata *ndp; int *flagp, cmode; struct file *fp; { struct thread *td = ndp->ni_cnd.cn_thread; return (vn_open_cred(ndp, flagp, cmode, 0, td->td_ucred, fp)); } /* * Common code for vnode open operations via a name lookup. * Lookup the vnode and invoke VOP_CREATE if needed. * Check permissions, and call the VOP_OPEN or VOP_CREATE routine. * * Note that this does NOT free nameidata for the successful case, * due to the NDINIT being done elsewhere. */ int vn_open_cred(struct nameidata *ndp, int *flagp, int cmode, u_int vn_open_flags, struct ucred *cred, struct file *fp) { struct vnode *vp; struct mount *mp; struct thread *td = ndp->ni_cnd.cn_thread; struct vattr vat; struct vattr *vap = &vat; int fmode, error; restart: fmode = *flagp; if (fmode & O_CREAT) { ndp->ni_cnd.cn_nameiop = CREATE; ndp->ni_cnd.cn_flags = ISOPEN | LOCKPARENT | LOCKLEAF; if ((fmode & O_EXCL) == 0 && (fmode & O_NOFOLLOW) == 0) ndp->ni_cnd.cn_flags |= FOLLOW; if (!(vn_open_flags & VN_OPEN_NOAUDIT)) ndp->ni_cnd.cn_flags |= AUDITVNODE1; if (vn_open_flags & VN_OPEN_NOCAPCHECK) ndp->ni_cnd.cn_flags |= NOCAPCHECK; bwillwrite(); if ((error = namei(ndp)) != 0) return (error); if (ndp->ni_vp == NULL) { VATTR_NULL(vap); vap->va_type = VREG; vap->va_mode = cmode; if (fmode & O_EXCL) vap->va_vaflags |= VA_EXCLUSIVE; if (vn_start_write(ndp->ni_dvp, &mp, V_NOWAIT) != 0) { NDFREE(ndp, NDF_ONLY_PNBUF); vput(ndp->ni_dvp); if ((error = vn_start_write(NULL, &mp, V_XSLEEP | PCATCH)) != 0) return (error); goto restart; } #ifdef MAC error = mac_vnode_check_create(cred, ndp->ni_dvp, &ndp->ni_cnd, vap); if (error == 0) #endif error = VOP_CREATE(ndp->ni_dvp, &ndp->ni_vp, &ndp->ni_cnd, vap); vput(ndp->ni_dvp); vn_finished_write(mp); if (error) { NDFREE(ndp, NDF_ONLY_PNBUF); return (error); } fmode &= ~O_TRUNC; vp = ndp->ni_vp; } else { if (ndp->ni_dvp == ndp->ni_vp) vrele(ndp->ni_dvp); else vput(ndp->ni_dvp); ndp->ni_dvp = NULL; vp = ndp->ni_vp; if (fmode & O_EXCL) { error = EEXIST; goto bad; } fmode &= ~O_CREAT; } } else { ndp->ni_cnd.cn_nameiop = LOOKUP; ndp->ni_cnd.cn_flags = ISOPEN | ((fmode & O_NOFOLLOW) ? NOFOLLOW : FOLLOW) | LOCKLEAF; if (!(fmode & FWRITE)) ndp->ni_cnd.cn_flags |= LOCKSHARED; if (!(vn_open_flags & VN_OPEN_NOAUDIT)) ndp->ni_cnd.cn_flags |= AUDITVNODE1; if (vn_open_flags & VN_OPEN_NOCAPCHECK) ndp->ni_cnd.cn_flags |= NOCAPCHECK; if ((error = namei(ndp)) != 0) return (error); vp = ndp->ni_vp; } error = vn_open_vnode(vp, fmode, cred, td, fp); if (error) goto bad; *flagp = fmode; return (0); bad: NDFREE(ndp, NDF_ONLY_PNBUF); vput(vp); *flagp = fmode; ndp->ni_vp = NULL; return (error); } /* * Common code for vnode open operations once a vnode is located. * Check permissions, and call the VOP_OPEN routine. */ int vn_open_vnode(struct vnode *vp, int fmode, struct ucred *cred, struct thread *td, struct file *fp) { struct mount *mp; accmode_t accmode; struct flock lf; int error, have_flock, lock_flags, type; if (vp->v_type == VLNK) return (EMLINK); if (vp->v_type == VSOCK) return (EOPNOTSUPP); if (vp->v_type != VDIR && fmode & O_DIRECTORY) return (ENOTDIR); accmode = 0; if (fmode & (FWRITE | O_TRUNC)) { if (vp->v_type == VDIR) return (EISDIR); accmode |= VWRITE; } if (fmode & FREAD) accmode |= VREAD; if (fmode & FEXEC) accmode |= VEXEC; if ((fmode & O_APPEND) && (fmode & FWRITE)) accmode |= VAPPEND; #ifdef MAC error = mac_vnode_check_open(cred, vp, accmode); if (error) return (error); #endif if ((fmode & O_CREAT) == 0) { if (accmode & VWRITE) { error = vn_writechk(vp); if (error) return (error); } if (accmode) { error = VOP_ACCESS(vp, accmode, cred, td); if (error) return (error); } } if (vp->v_type == VFIFO && VOP_ISLOCKED(vp) != LK_EXCLUSIVE) vn_lock(vp, LK_UPGRADE | LK_RETRY); if ((error = VOP_OPEN(vp, fmode, cred, td, fp)) != 0) return (error); if (fmode & (O_EXLOCK | O_SHLOCK)) { KASSERT(fp != NULL, ("open with flock requires fp")); lock_flags = VOP_ISLOCKED(vp); VOP_UNLOCK(vp, 0); lf.l_whence = SEEK_SET; lf.l_start = 0; lf.l_len = 0; if (fmode & O_EXLOCK) lf.l_type = F_WRLCK; else lf.l_type = F_RDLCK; type = F_FLOCK; if ((fmode & FNONBLOCK) == 0) type |= F_WAIT; error = VOP_ADVLOCK(vp, (caddr_t)fp, F_SETLK, &lf, type); have_flock = (error == 0); vn_lock(vp, lock_flags | LK_RETRY); if (error == 0 && vp->v_iflag & VI_DOOMED) error = ENOENT; /* * Another thread might have used this vnode as an * executable while the vnode lock was dropped. * Ensure the vnode is still able to be opened for * writing after the lock has been obtained. */ if (error == 0 && accmode & VWRITE) error = vn_writechk(vp); if (error) { VOP_UNLOCK(vp, 0); if (have_flock) { lf.l_whence = SEEK_SET; lf.l_start = 0; lf.l_len = 0; lf.l_type = F_UNLCK; (void) VOP_ADVLOCK(vp, fp, F_UNLCK, &lf, F_FLOCK); } vn_start_write(vp, &mp, V_WAIT); vn_lock(vp, lock_flags | LK_RETRY); (void)VOP_CLOSE(vp, fmode, cred, td); vn_finished_write(mp); /* Prevent second close from fdrop()->vn_close(). */ if (fp != NULL) fp->f_ops= &badfileops; return (error); } fp->f_flag |= FHASLOCK; } if (fmode & FWRITE) { VOP_ADD_WRITECOUNT(vp, 1); CTR3(KTR_VFS, "%s: vp %p v_writecount increased to %d", __func__, vp, vp->v_writecount); } ASSERT_VOP_LOCKED(vp, "vn_open_vnode"); return (0); } /* * Check for write permissions on the specified vnode. * Prototype text segments cannot be written. */ int vn_writechk(vp) register struct vnode *vp; { ASSERT_VOP_LOCKED(vp, "vn_writechk"); /* * If there's shared text associated with * the vnode, try to free it up once. If * we fail, we can't allow writing. */ if (VOP_IS_TEXT(vp)) return (ETXTBSY); return (0); } /* * Vnode close call */ int vn_close(vp, flags, file_cred, td) register struct vnode *vp; int flags; struct ucred *file_cred; struct thread *td; { struct mount *mp; int error, lock_flags; if (vp->v_type != VFIFO && (flags & FWRITE) == 0 && MNT_EXTENDED_SHARED(vp->v_mount)) lock_flags = LK_SHARED; else lock_flags = LK_EXCLUSIVE; vn_start_write(vp, &mp, V_WAIT); vn_lock(vp, lock_flags | LK_RETRY); if (flags & FWRITE) { VNASSERT(vp->v_writecount > 0, vp, ("vn_close: negative writecount")); VOP_ADD_WRITECOUNT(vp, -1); CTR3(KTR_VFS, "%s: vp %p v_writecount decreased to %d", __func__, vp, vp->v_writecount); } error = VOP_CLOSE(vp, flags, file_cred, td); vput(vp); vn_finished_write(mp); return (error); } /* * Heuristic to detect sequential operation. */ static int sequential_heuristic(struct uio *uio, struct file *fp) { ASSERT_VOP_LOCKED(fp->f_vnode, __func__); if (fp->f_flag & FRDAHEAD) return (fp->f_seqcount << IO_SEQSHIFT); /* * Offset 0 is handled specially. open() sets f_seqcount to 1 so * that the first I/O is normally considered to be slightly * sequential. Seeking to offset 0 doesn't change sequentiality * unless previous seeks have reduced f_seqcount to 0, in which * case offset 0 is not special. */ if ((uio->uio_offset == 0 && fp->f_seqcount > 0) || uio->uio_offset == fp->f_nextoff) { /* * f_seqcount is in units of fixed-size blocks so that it * depends mainly on the amount of sequential I/O and not * much on the number of sequential I/O's. The fixed size * of 16384 is hard-coded here since it is (not quite) just * a magic size that works well here. This size is more * closely related to the best I/O size for real disks than * to any block size used by software. */ fp->f_seqcount += howmany(uio->uio_resid, 16384); if (fp->f_seqcount > IO_SEQMAX) fp->f_seqcount = IO_SEQMAX; return (fp->f_seqcount << IO_SEQSHIFT); } /* Not sequential. Quickly draw-down sequentiality. */ if (fp->f_seqcount > 1) fp->f_seqcount = 1; else fp->f_seqcount = 0; return (0); } /* * Package up an I/O request on a vnode into a uio and do it. */ int vn_rdwr(enum uio_rw rw, struct vnode *vp, void *base, int len, off_t offset, enum uio_seg segflg, int ioflg, struct ucred *active_cred, struct ucred *file_cred, ssize_t *aresid, struct thread *td) { struct uio auio; struct iovec aiov; struct mount *mp; struct ucred *cred; void *rl_cookie; struct vn_io_fault_args args; int error, lock_flags; auio.uio_iov = &aiov; auio.uio_iovcnt = 1; aiov.iov_base = base; aiov.iov_len = len; auio.uio_resid = len; auio.uio_offset = offset; auio.uio_segflg = segflg; auio.uio_rw = rw; auio.uio_td = td; error = 0; if ((ioflg & IO_NODELOCKED) == 0) { if ((ioflg & IO_RANGELOCKED) == 0) { if (rw == UIO_READ) { rl_cookie = vn_rangelock_rlock(vp, offset, offset + len); } else { rl_cookie = vn_rangelock_wlock(vp, offset, offset + len); } } else rl_cookie = NULL; mp = NULL; if (rw == UIO_WRITE) { if (vp->v_type != VCHR && (error = vn_start_write(vp, &mp, V_WAIT | PCATCH)) != 0) goto out; if (MNT_SHARED_WRITES(mp) || ((mp == NULL) && MNT_SHARED_WRITES(vp->v_mount))) lock_flags = LK_SHARED; else lock_flags = LK_EXCLUSIVE; } else lock_flags = LK_SHARED; vn_lock(vp, lock_flags | LK_RETRY); } else rl_cookie = NULL; ASSERT_VOP_LOCKED(vp, "IO_NODELOCKED with no vp lock held"); #ifdef MAC if ((ioflg & IO_NOMACCHECK) == 0) { if (rw == UIO_READ) error = mac_vnode_check_read(active_cred, file_cred, vp); else error = mac_vnode_check_write(active_cred, file_cred, vp); } #endif if (error == 0) { if (file_cred != NULL) cred = file_cred; else cred = active_cred; if (do_vn_io_fault(vp, &auio)) { args.kind = VN_IO_FAULT_VOP; args.cred = cred; args.flags = ioflg; args.args.vop_args.vp = vp; error = vn_io_fault1(vp, &auio, &args, td); } else if (rw == UIO_READ) { error = VOP_READ(vp, &auio, ioflg, cred); } else /* if (rw == UIO_WRITE) */ { error = VOP_WRITE(vp, &auio, ioflg, cred); } } if (aresid) *aresid = auio.uio_resid; else if (auio.uio_resid && error == 0) error = EIO; if ((ioflg & IO_NODELOCKED) == 0) { VOP_UNLOCK(vp, 0); if (mp != NULL) vn_finished_write(mp); } out: if (rl_cookie != NULL) vn_rangelock_unlock(vp, rl_cookie); return (error); } /* * Package up an I/O request on a vnode into a uio and do it. The I/O * request is split up into smaller chunks and we try to avoid saturating * the buffer cache while potentially holding a vnode locked, so we * check bwillwrite() before calling vn_rdwr(). We also call kern_yield() * to give other processes a chance to lock the vnode (either other processes * core'ing the same binary, or unrelated processes scanning the directory). */ int vn_rdwr_inchunks(rw, vp, base, len, offset, segflg, ioflg, active_cred, file_cred, aresid, td) enum uio_rw rw; struct vnode *vp; void *base; size_t len; off_t offset; enum uio_seg segflg; int ioflg; struct ucred *active_cred; struct ucred *file_cred; size_t *aresid; struct thread *td; { int error = 0; ssize_t iaresid; do { int chunk; /* * Force `offset' to a multiple of MAXBSIZE except possibly * for the first chunk, so that filesystems only need to * write full blocks except possibly for the first and last * chunks. */ chunk = MAXBSIZE - (uoff_t)offset % MAXBSIZE; if (chunk > len) chunk = len; if (rw != UIO_READ && vp->v_type == VREG) bwillwrite(); iaresid = 0; error = vn_rdwr(rw, vp, base, chunk, offset, segflg, ioflg, active_cred, file_cred, &iaresid, td); len -= chunk; /* aresid calc already includes length */ if (error) break; offset += chunk; base = (char *)base + chunk; kern_yield(PRI_USER); } while (len); if (aresid) *aresid = len + iaresid; return (error); } off_t foffset_lock(struct file *fp, int flags) { struct mtx *mtxp; off_t res; KASSERT((flags & FOF_OFFSET) == 0, ("FOF_OFFSET passed")); #if OFF_MAX <= LONG_MAX /* * Caller only wants the current f_offset value. Assume that * the long and shorter integer types reads are atomic. */ if ((flags & FOF_NOLOCK) != 0) return (fp->f_offset); #endif /* * According to McKusick the vn lock was protecting f_offset here. * It is now protected by the FOFFSET_LOCKED flag. */ mtxp = mtx_pool_find(mtxpool_sleep, fp); mtx_lock(mtxp); if ((flags & FOF_NOLOCK) == 0) { while (fp->f_vnread_flags & FOFFSET_LOCKED) { fp->f_vnread_flags |= FOFFSET_LOCK_WAITING; msleep(&fp->f_vnread_flags, mtxp, PUSER -1, "vofflock", 0); } fp->f_vnread_flags |= FOFFSET_LOCKED; } res = fp->f_offset; mtx_unlock(mtxp); return (res); } void foffset_unlock(struct file *fp, off_t val, int flags) { struct mtx *mtxp; KASSERT((flags & FOF_OFFSET) == 0, ("FOF_OFFSET passed")); #if OFF_MAX <= LONG_MAX if ((flags & FOF_NOLOCK) != 0) { if ((flags & FOF_NOUPDATE) == 0) fp->f_offset = val; if ((flags & FOF_NEXTOFF) != 0) fp->f_nextoff = val; return; } #endif mtxp = mtx_pool_find(mtxpool_sleep, fp); mtx_lock(mtxp); if ((flags & FOF_NOUPDATE) == 0) fp->f_offset = val; if ((flags & FOF_NEXTOFF) != 0) fp->f_nextoff = val; if ((flags & FOF_NOLOCK) == 0) { KASSERT((fp->f_vnread_flags & FOFFSET_LOCKED) != 0, ("Lost FOFFSET_LOCKED")); if (fp->f_vnread_flags & FOFFSET_LOCK_WAITING) wakeup(&fp->f_vnread_flags); fp->f_vnread_flags = 0; } mtx_unlock(mtxp); } void foffset_lock_uio(struct file *fp, struct uio *uio, int flags) { if ((flags & FOF_OFFSET) == 0) uio->uio_offset = foffset_lock(fp, flags); } void foffset_unlock_uio(struct file *fp, struct uio *uio, int flags) { if ((flags & FOF_OFFSET) == 0) foffset_unlock(fp, uio->uio_offset, flags); } static int get_advice(struct file *fp, struct uio *uio) { struct mtx *mtxp; int ret; ret = POSIX_FADV_NORMAL; if (fp->f_advice == NULL) return (ret); mtxp = mtx_pool_find(mtxpool_sleep, fp); mtx_lock(mtxp); if (uio->uio_offset >= fp->f_advice->fa_start && uio->uio_offset + uio->uio_resid <= fp->f_advice->fa_end) ret = fp->f_advice->fa_advice; mtx_unlock(mtxp); return (ret); } /* * File table vnode read routine. */ static int vn_read(fp, uio, active_cred, flags, td) struct file *fp; struct uio *uio; struct ucred *active_cred; int flags; struct thread *td; { struct vnode *vp; struct mtx *mtxp; int error, ioflag; int advice; off_t offset, start, end; KASSERT(uio->uio_td == td, ("uio_td %p is not td %p", uio->uio_td, td)); KASSERT(flags & FOF_OFFSET, ("No FOF_OFFSET")); vp = fp->f_vnode; ioflag = 0; if (fp->f_flag & FNONBLOCK) ioflag |= IO_NDELAY; if (fp->f_flag & O_DIRECT) ioflag |= IO_DIRECT; advice = get_advice(fp, uio); vn_lock(vp, LK_SHARED | LK_RETRY); switch (advice) { case POSIX_FADV_NORMAL: case POSIX_FADV_SEQUENTIAL: case POSIX_FADV_NOREUSE: ioflag |= sequential_heuristic(uio, fp); break; case POSIX_FADV_RANDOM: /* Disable read-ahead for random I/O. */ break; } offset = uio->uio_offset; #ifdef MAC error = mac_vnode_check_read(active_cred, fp->f_cred, vp); if (error == 0) #endif error = VOP_READ(vp, uio, ioflag, fp->f_cred); fp->f_nextoff = uio->uio_offset; VOP_UNLOCK(vp, 0); if (error == 0 && advice == POSIX_FADV_NOREUSE && offset != uio->uio_offset) { /* * Use POSIX_FADV_DONTNEED to flush clean pages and * buffers for the backing file after a * POSIX_FADV_NOREUSE read(2). To optimize the common * case of using POSIX_FADV_NOREUSE with sequential * access, track the previous implicit DONTNEED * request and grow this request to include the * current read(2) in addition to the previous * DONTNEED. With purely sequential access this will * cause the DONTNEED requests to continously grow to * cover all of the previously read regions of the * file. This allows filesystem blocks that are * accessed by multiple calls to read(2) to be flushed * once the last read(2) finishes. */ start = offset; end = uio->uio_offset - 1; mtxp = mtx_pool_find(mtxpool_sleep, fp); mtx_lock(mtxp); if (fp->f_advice != NULL && fp->f_advice->fa_advice == POSIX_FADV_NOREUSE) { if (start != 0 && fp->f_advice->fa_prevend + 1 == start) start = fp->f_advice->fa_prevstart; else if (fp->f_advice->fa_prevstart != 0 && fp->f_advice->fa_prevstart == end + 1) end = fp->f_advice->fa_prevend; fp->f_advice->fa_prevstart = start; fp->f_advice->fa_prevend = end; } mtx_unlock(mtxp); error = VOP_ADVISE(vp, start, end, POSIX_FADV_DONTNEED); } return (error); } /* * File table vnode write routine. */ static int vn_write(fp, uio, active_cred, flags, td) struct file *fp; struct uio *uio; struct ucred *active_cred; int flags; struct thread *td; { struct vnode *vp; struct mount *mp; struct mtx *mtxp; int error, ioflag, lock_flags; int advice; off_t offset, start, end; KASSERT(uio->uio_td == td, ("uio_td %p is not td %p", uio->uio_td, td)); KASSERT(flags & FOF_OFFSET, ("No FOF_OFFSET")); vp = fp->f_vnode; if (vp->v_type == VREG) bwillwrite(); ioflag = IO_UNIT; if (vp->v_type == VREG && (fp->f_flag & O_APPEND)) ioflag |= IO_APPEND; if (fp->f_flag & FNONBLOCK) ioflag |= IO_NDELAY; if (fp->f_flag & O_DIRECT) ioflag |= IO_DIRECT; if ((fp->f_flag & O_FSYNC) || (vp->v_mount && (vp->v_mount->mnt_flag & MNT_SYNCHRONOUS))) ioflag |= IO_SYNC; mp = NULL; if (vp->v_type != VCHR && (error = vn_start_write(vp, &mp, V_WAIT | PCATCH)) != 0) goto unlock; advice = get_advice(fp, uio); if (MNT_SHARED_WRITES(mp) || (mp == NULL && MNT_SHARED_WRITES(vp->v_mount))) { lock_flags = LK_SHARED; } else { lock_flags = LK_EXCLUSIVE; } vn_lock(vp, lock_flags | LK_RETRY); switch (advice) { case POSIX_FADV_NORMAL: case POSIX_FADV_SEQUENTIAL: case POSIX_FADV_NOREUSE: ioflag |= sequential_heuristic(uio, fp); break; case POSIX_FADV_RANDOM: /* XXX: Is this correct? */ break; } offset = uio->uio_offset; #ifdef MAC error = mac_vnode_check_write(active_cred, fp->f_cred, vp); if (error == 0) #endif error = VOP_WRITE(vp, uio, ioflag, fp->f_cred); fp->f_nextoff = uio->uio_offset; VOP_UNLOCK(vp, 0); if (vp->v_type != VCHR) vn_finished_write(mp); if (error == 0 && advice == POSIX_FADV_NOREUSE && offset != uio->uio_offset) { /* * Use POSIX_FADV_DONTNEED to flush clean pages and * buffers for the backing file after a * POSIX_FADV_NOREUSE write(2). To optimize the * common case of using POSIX_FADV_NOREUSE with * sequential access, track the previous implicit * DONTNEED request and grow this request to include * the current write(2) in addition to the previous * DONTNEED. With purely sequential access this will * cause the DONTNEED requests to continously grow to * cover all of the previously written regions of the * file. * * Note that the blocks just written are almost * certainly still dirty, so this only works when * VOP_ADVISE() calls from subsequent writes push out * the data written by this write(2) once the backing * buffers are clean. However, as compared to forcing * IO_DIRECT, this gives much saner behavior. Write * clustering is still allowed, and clean pages are * merely moved to the cache page queue rather than * outright thrown away. This means a subsequent * read(2) can still avoid hitting the disk if the * pages have not been reclaimed. * * This does make POSIX_FADV_NOREUSE largely useless * with non-sequential access. However, sequential * access is the more common use case and the flag is * merely advisory. */ start = offset; end = uio->uio_offset - 1; mtxp = mtx_pool_find(mtxpool_sleep, fp); mtx_lock(mtxp); if (fp->f_advice != NULL && fp->f_advice->fa_advice == POSIX_FADV_NOREUSE) { if (start != 0 && fp->f_advice->fa_prevend + 1 == start) start = fp->f_advice->fa_prevstart; else if (fp->f_advice->fa_prevstart != 0 && fp->f_advice->fa_prevstart == end + 1) end = fp->f_advice->fa_prevend; fp->f_advice->fa_prevstart = start; fp->f_advice->fa_prevend = end; } mtx_unlock(mtxp); error = VOP_ADVISE(vp, start, end, POSIX_FADV_DONTNEED); } unlock: return (error); } /* * The vn_io_fault() is a wrapper around vn_read() and vn_write() to * prevent the following deadlock: * * Assume that the thread A reads from the vnode vp1 into userspace * buffer buf1 backed by the pages of vnode vp2. If a page in buf1 is * currently not resident, then system ends up with the call chain * vn_read() -> VOP_READ(vp1) -> uiomove() -> [Page Fault] -> * vm_fault(buf1) -> vnode_pager_getpages(vp2) -> VOP_GETPAGES(vp2) * which establishes lock order vp1->vn_lock, then vp2->vn_lock. * If, at the same time, thread B reads from vnode vp2 into buffer buf2 * backed by the pages of vnode vp1, and some page in buf2 is not * resident, we get a reversed order vp2->vn_lock, then vp1->vn_lock. * * To prevent the lock order reversal and deadlock, vn_io_fault() does * not allow page faults to happen during VOP_READ() or VOP_WRITE(). * Instead, it first tries to do the whole range i/o with pagefaults * disabled. If all pages in the i/o buffer are resident and mapped, * VOP will succeed (ignoring the genuine filesystem errors). * Otherwise, we get back EFAULT, and vn_io_fault() falls back to do * i/o in chunks, with all pages in the chunk prefaulted and held * using vm_fault_quick_hold_pages(). * * Filesystems using this deadlock avoidance scheme should use the * array of the held pages from uio, saved in the curthread->td_ma, * instead of doing uiomove(). A helper function * vn_io_fault_uiomove() converts uiomove request into * uiomove_fromphys() over td_ma array. * * Since vnode locks do not cover the whole i/o anymore, rangelocks * make the current i/o request atomic with respect to other i/os and * truncations. */ /* * Decode vn_io_fault_args and perform the corresponding i/o. */ static int vn_io_fault_doio(struct vn_io_fault_args *args, struct uio *uio, struct thread *td) { switch (args->kind) { case VN_IO_FAULT_FOP: return ((args->args.fop_args.doio)(args->args.fop_args.fp, uio, args->cred, args->flags, td)); case VN_IO_FAULT_VOP: if (uio->uio_rw == UIO_READ) { return (VOP_READ(args->args.vop_args.vp, uio, args->flags, args->cred)); } else if (uio->uio_rw == UIO_WRITE) { return (VOP_WRITE(args->args.vop_args.vp, uio, args->flags, args->cred)); } break; } panic("vn_io_fault_doio: unknown kind of io %d %d", args->kind, uio->uio_rw); } /* * Common code for vn_io_fault(), agnostic to the kind of i/o request. * Uses vn_io_fault_doio() to make the call to an actual i/o function. * Used from vn_rdwr() and vn_io_fault(), which encode the i/o request * into args and call vn_io_fault1() to handle faults during the user * mode buffer accesses. */ static int vn_io_fault1(struct vnode *vp, struct uio *uio, struct vn_io_fault_args *args, struct thread *td) { vm_page_t ma[io_hold_cnt + 2]; struct uio *uio_clone, short_uio; struct iovec short_iovec[1]; vm_page_t *prev_td_ma; vm_prot_t prot; vm_offset_t addr, end; size_t len, resid; ssize_t adv; int error, cnt, save, saveheld, prev_td_ma_cnt; prot = uio->uio_rw == UIO_READ ? VM_PROT_WRITE : VM_PROT_READ; /* * The UFS follows IO_UNIT directive and replays back both * uio_offset and uio_resid if an error is encountered during the * operation. But, since the iovec may be already advanced, * uio is still in an inconsistent state. * * Cache a copy of the original uio, which is advanced to the redo * point using UIO_NOCOPY below. */ uio_clone = cloneuio(uio); resid = uio->uio_resid; short_uio.uio_segflg = UIO_USERSPACE; short_uio.uio_rw = uio->uio_rw; short_uio.uio_td = uio->uio_td; save = vm_fault_disable_pagefaults(); error = vn_io_fault_doio(args, uio, td); if (error != EFAULT) goto out; atomic_add_long(&vn_io_faults_cnt, 1); uio_clone->uio_segflg = UIO_NOCOPY; uiomove(NULL, resid - uio->uio_resid, uio_clone); uio_clone->uio_segflg = uio->uio_segflg; saveheld = curthread_pflags_set(TDP_UIOHELD); prev_td_ma = td->td_ma; prev_td_ma_cnt = td->td_ma_cnt; while (uio_clone->uio_resid != 0) { len = uio_clone->uio_iov->iov_len; if (len == 0) { KASSERT(uio_clone->uio_iovcnt >= 1, ("iovcnt underflow")); uio_clone->uio_iov++; uio_clone->uio_iovcnt--; continue; } if (len > io_hold_cnt * PAGE_SIZE) len = io_hold_cnt * PAGE_SIZE; addr = (uintptr_t)uio_clone->uio_iov->iov_base; end = round_page(addr + len); if (end < addr) { error = EFAULT; break; } cnt = atop(end - trunc_page(addr)); /* * A perfectly misaligned address and length could cause * both the start and the end of the chunk to use partial * page. +2 accounts for such a situation. */ cnt = vm_fault_quick_hold_pages(&td->td_proc->p_vmspace->vm_map, addr, len, prot, ma, io_hold_cnt + 2); if (cnt == -1) { error = EFAULT; break; } short_uio.uio_iov = &short_iovec[0]; short_iovec[0].iov_base = (void *)addr; short_uio.uio_iovcnt = 1; short_uio.uio_resid = short_iovec[0].iov_len = len; short_uio.uio_offset = uio_clone->uio_offset; td->td_ma = ma; td->td_ma_cnt = cnt; error = vn_io_fault_doio(args, &short_uio, td); vm_page_unhold_pages(ma, cnt); adv = len - short_uio.uio_resid; uio_clone->uio_iov->iov_base = (char *)uio_clone->uio_iov->iov_base + adv; uio_clone->uio_iov->iov_len -= adv; uio_clone->uio_resid -= adv; uio_clone->uio_offset += adv; uio->uio_resid -= adv; uio->uio_offset += adv; if (error != 0 || adv == 0) break; } td->td_ma = prev_td_ma; td->td_ma_cnt = prev_td_ma_cnt; curthread_pflags_restore(saveheld); out: vm_fault_enable_pagefaults(save); free(uio_clone, M_IOV); return (error); } static int vn_io_fault(struct file *fp, struct uio *uio, struct ucred *active_cred, int flags, struct thread *td) { fo_rdwr_t *doio; struct vnode *vp; void *rl_cookie; struct vn_io_fault_args args; int error; doio = uio->uio_rw == UIO_READ ? vn_read : vn_write; vp = fp->f_vnode; foffset_lock_uio(fp, uio, flags); if (do_vn_io_fault(vp, uio)) { args.kind = VN_IO_FAULT_FOP; args.args.fop_args.fp = fp; args.args.fop_args.doio = doio; args.cred = active_cred; args.flags = flags | FOF_OFFSET; if (uio->uio_rw == UIO_READ) { rl_cookie = vn_rangelock_rlock(vp, uio->uio_offset, uio->uio_offset + uio->uio_resid); } else if ((fp->f_flag & O_APPEND) != 0 || (flags & FOF_OFFSET) == 0) { /* For appenders, punt and lock the whole range. */ rl_cookie = vn_rangelock_wlock(vp, 0, OFF_MAX); } else { rl_cookie = vn_rangelock_wlock(vp, uio->uio_offset, uio->uio_offset + uio->uio_resid); } error = vn_io_fault1(vp, uio, &args, td); vn_rangelock_unlock(vp, rl_cookie); } else { error = doio(fp, uio, active_cred, flags | FOF_OFFSET, td); } foffset_unlock_uio(fp, uio, flags); return (error); } /* * Helper function to perform the requested uiomove operation using * the held pages for io->uio_iov[0].iov_base buffer instead of * copyin/copyout. Access to the pages with uiomove_fromphys() * instead of iov_base prevents page faults that could occur due to * pmap_collect() invalidating the mapping created by * vm_fault_quick_hold_pages(), or pageout daemon, page laundry or * object cleanup revoking the write access from page mappings. * * Filesystems specified MNTK_NO_IOPF shall use vn_io_fault_uiomove() * instead of plain uiomove(). */ int vn_io_fault_uiomove(char *data, int xfersize, struct uio *uio) { struct uio transp_uio; struct iovec transp_iov[1]; struct thread *td; size_t adv; int error, pgadv; td = curthread; if ((td->td_pflags & TDP_UIOHELD) == 0 || uio->uio_segflg != UIO_USERSPACE) return (uiomove(data, xfersize, uio)); KASSERT(uio->uio_iovcnt == 1, ("uio_iovcnt %d", uio->uio_iovcnt)); transp_iov[0].iov_base = data; transp_uio.uio_iov = &transp_iov[0]; transp_uio.uio_iovcnt = 1; if (xfersize > uio->uio_resid) xfersize = uio->uio_resid; transp_uio.uio_resid = transp_iov[0].iov_len = xfersize; transp_uio.uio_offset = 0; transp_uio.uio_segflg = UIO_SYSSPACE; /* * Since transp_iov points to data, and td_ma page array * corresponds to original uio->uio_iov, we need to invert the * direction of the i/o operation as passed to * uiomove_fromphys(). */ switch (uio->uio_rw) { case UIO_WRITE: transp_uio.uio_rw = UIO_READ; break; case UIO_READ: transp_uio.uio_rw = UIO_WRITE; break; } transp_uio.uio_td = uio->uio_td; error = uiomove_fromphys(td->td_ma, ((vm_offset_t)uio->uio_iov->iov_base) & PAGE_MASK, xfersize, &transp_uio); adv = xfersize - transp_uio.uio_resid; pgadv = (((vm_offset_t)uio->uio_iov->iov_base + adv) >> PAGE_SHIFT) - (((vm_offset_t)uio->uio_iov->iov_base) >> PAGE_SHIFT); td->td_ma += pgadv; KASSERT(td->td_ma_cnt >= pgadv, ("consumed pages %d %d", td->td_ma_cnt, pgadv)); td->td_ma_cnt -= pgadv; uio->uio_iov->iov_base = (char *)uio->uio_iov->iov_base + adv; uio->uio_iov->iov_len -= adv; uio->uio_resid -= adv; uio->uio_offset += adv; return (error); } int vn_io_fault_pgmove(vm_page_t ma[], vm_offset_t offset, int xfersize, struct uio *uio) { struct thread *td; vm_offset_t iov_base; int cnt, pgadv; td = curthread; if ((td->td_pflags & TDP_UIOHELD) == 0 || uio->uio_segflg != UIO_USERSPACE) return (uiomove_fromphys(ma, offset, xfersize, uio)); KASSERT(uio->uio_iovcnt == 1, ("uio_iovcnt %d", uio->uio_iovcnt)); cnt = xfersize > uio->uio_resid ? uio->uio_resid : xfersize; iov_base = (vm_offset_t)uio->uio_iov->iov_base; switch (uio->uio_rw) { case UIO_WRITE: pmap_copy_pages(td->td_ma, iov_base & PAGE_MASK, ma, offset, cnt); break; case UIO_READ: pmap_copy_pages(ma, offset, td->td_ma, iov_base & PAGE_MASK, cnt); break; } pgadv = ((iov_base + cnt) >> PAGE_SHIFT) - (iov_base >> PAGE_SHIFT); td->td_ma += pgadv; KASSERT(td->td_ma_cnt >= pgadv, ("consumed pages %d %d", td->td_ma_cnt, pgadv)); td->td_ma_cnt -= pgadv; uio->uio_iov->iov_base = (char *)(iov_base + cnt); uio->uio_iov->iov_len -= cnt; uio->uio_resid -= cnt; uio->uio_offset += cnt; return (0); } /* * File table truncate routine. */ static int vn_truncate(struct file *fp, off_t length, struct ucred *active_cred, struct thread *td) { struct vattr vattr; struct mount *mp; struct vnode *vp; void *rl_cookie; int error; vp = fp->f_vnode; /* * Lock the whole range for truncation. Otherwise split i/o * might happen partly before and partly after the truncation. */ rl_cookie = vn_rangelock_wlock(vp, 0, OFF_MAX); error = vn_start_write(vp, &mp, V_WAIT | PCATCH); if (error) goto out1; vn_lock(vp, LK_EXCLUSIVE | LK_RETRY); if (vp->v_type == VDIR) { error = EISDIR; goto out; } #ifdef MAC error = mac_vnode_check_write(active_cred, fp->f_cred, vp); if (error) goto out; #endif error = vn_writechk(vp); if (error == 0) { VATTR_NULL(&vattr); vattr.va_size = length; error = VOP_SETATTR(vp, &vattr, fp->f_cred); } out: VOP_UNLOCK(vp, 0); vn_finished_write(mp); out1: vn_rangelock_unlock(vp, rl_cookie); return (error); } /* * File table vnode stat routine. */ static int vn_statfile(fp, sb, active_cred, td) struct file *fp; struct stat *sb; struct ucred *active_cred; struct thread *td; { struct vnode *vp = fp->f_vnode; int error; vn_lock(vp, LK_SHARED | LK_RETRY); error = vn_stat(vp, sb, active_cred, fp->f_cred, td); VOP_UNLOCK(vp, 0); return (error); } /* * Stat a vnode; implementation for the stat syscall */ int vn_stat(vp, sb, active_cred, file_cred, td) struct vnode *vp; register struct stat *sb; struct ucred *active_cred; struct ucred *file_cred; struct thread *td; { struct vattr vattr; register struct vattr *vap; int error; u_short mode; #ifdef MAC error = mac_vnode_check_stat(active_cred, file_cred, vp); if (error) return (error); #endif vap = &vattr; /* * Initialize defaults for new and unusual fields, so that file * systems which don't support these fields don't need to know * about them. */ vap->va_birthtime.tv_sec = -1; vap->va_birthtime.tv_nsec = 0; vap->va_fsid = VNOVAL; vap->va_rdev = NODEV; error = VOP_GETATTR(vp, vap, active_cred); if (error) return (error); /* * Zero the spare stat fields */ bzero(sb, sizeof *sb); /* * Copy from vattr table */ if (vap->va_fsid != VNOVAL) sb->st_dev = vap->va_fsid; else sb->st_dev = vp->v_mount->mnt_stat.f_fsid.val[0]; sb->st_ino = vap->va_fileid; mode = vap->va_mode; switch (vap->va_type) { case VREG: mode |= S_IFREG; break; case VDIR: mode |= S_IFDIR; break; case VBLK: mode |= S_IFBLK; break; case VCHR: mode |= S_IFCHR; break; case VLNK: mode |= S_IFLNK; break; case VSOCK: mode |= S_IFSOCK; break; case VFIFO: mode |= S_IFIFO; break; default: return (EBADF); }; sb->st_mode = mode; sb->st_nlink = vap->va_nlink; sb->st_uid = vap->va_uid; sb->st_gid = vap->va_gid; sb->st_rdev = vap->va_rdev; if (vap->va_size > OFF_MAX) return (EOVERFLOW); sb->st_size = vap->va_size; sb->st_atim = vap->va_atime; sb->st_mtim = vap->va_mtime; sb->st_ctim = vap->va_ctime; sb->st_birthtim = vap->va_birthtime; /* * According to www.opengroup.org, the meaning of st_blksize is * "a filesystem-specific preferred I/O block size for this * object. In some filesystem types, this may vary from file * to file" * Use miminum/default of PAGE_SIZE (e.g. for VCHR). */ sb->st_blksize = max(PAGE_SIZE, vap->va_blocksize); sb->st_flags = vap->va_flags; if (priv_check(td, PRIV_VFS_GENERATION)) sb->st_gen = 0; else sb->st_gen = vap->va_gen; sb->st_blocks = vap->va_bytes / S_BLKSIZE; return (0); } /* * File table vnode ioctl routine. */ static int vn_ioctl(fp, com, data, active_cred, td) struct file *fp; u_long com; void *data; struct ucred *active_cred; struct thread *td; { struct vattr vattr; struct vnode *vp; int error; vp = fp->f_vnode; switch (vp->v_type) { case VDIR: case VREG: switch (com) { case FIONREAD: vn_lock(vp, LK_SHARED | LK_RETRY); error = VOP_GETATTR(vp, &vattr, active_cred); VOP_UNLOCK(vp, 0); if (error == 0) *(int *)data = vattr.va_size - fp->f_offset; return (error); case FIONBIO: case FIOASYNC: return (0); default: return (VOP_IOCTL(vp, com, data, fp->f_flag, active_cred, td)); } default: return (ENOTTY); } } /* * File table vnode poll routine. */ static int vn_poll(fp, events, active_cred, td) struct file *fp; int events; struct ucred *active_cred; struct thread *td; { struct vnode *vp; int error; vp = fp->f_vnode; #ifdef MAC vn_lock(vp, LK_EXCLUSIVE | LK_RETRY); error = mac_vnode_check_poll(active_cred, fp->f_cred, vp); VOP_UNLOCK(vp, 0); if (!error) #endif error = VOP_POLL(vp, events, fp->f_cred, td); return (error); } /* * Acquire the requested lock and then check for validity. LK_RETRY * permits vn_lock to return doomed vnodes. */ int _vn_lock(struct vnode *vp, int flags, char *file, int line) { int error; VNASSERT((flags & LK_TYPE_MASK) != 0, vp, ("vn_lock called with no locktype.")); do { #ifdef DEBUG_VFS_LOCKS KASSERT(vp->v_holdcnt != 0, ("vn_lock %p: zero hold count", vp)); #endif error = VOP_LOCK1(vp, flags, file, line); flags &= ~LK_INTERLOCK; /* Interlock is always dropped. */ KASSERT((flags & LK_RETRY) == 0 || error == 0, ("LK_RETRY set with incompatible flags (0x%x) or an error occured (%d)", flags, error)); /* * Callers specify LK_RETRY if they wish to get dead vnodes. * If RETRY is not set, we return ENOENT instead. */ if (error == 0 && vp->v_iflag & VI_DOOMED && (flags & LK_RETRY) == 0) { VOP_UNLOCK(vp, 0); error = ENOENT; break; } } while (flags & LK_RETRY && error != 0); return (error); } /* * File table vnode close routine. */ static int vn_closefile(fp, td) struct file *fp; struct thread *td; { struct vnode *vp; struct flock lf; int error; vp = fp->f_vnode; fp->f_ops = &badfileops; if (fp->f_type == DTYPE_VNODE && fp->f_flag & FHASLOCK) vref(vp); error = vn_close(vp, fp->f_flag, fp->f_cred, td); if (fp->f_type == DTYPE_VNODE && fp->f_flag & FHASLOCK) { lf.l_whence = SEEK_SET; lf.l_start = 0; lf.l_len = 0; lf.l_type = F_UNLCK; (void) VOP_ADVLOCK(vp, fp, F_UNLCK, &lf, F_FLOCK); vrele(vp); } return (error); } static bool vn_suspendable_mp(struct mount *mp) { return ((mp->mnt_kern_flag & MNTK_SUSPENDABLE) != 0); } static bool vn_suspendable(struct vnode *vp, struct mount **mpp) { if (vp != NULL) *mpp = vp->v_mount; if (*mpp == NULL) return (false); return (vn_suspendable_mp(*mpp)); } /* * Preparing to start a filesystem write operation. If the operation is * permitted, then we bump the count of operations in progress and * proceed. If a suspend request is in progress, we wait until the * suspension is over, and then proceed. */ static int vn_start_write_locked(struct mount *mp, int flags) { - int error; + int error, mflags; mtx_assert(MNT_MTX(mp), MA_OWNED); error = 0; /* * Check on status of suspension. */ if ((curthread->td_pflags & TDP_IGNSUSP) == 0 || mp->mnt_susp_owner != curthread) { + mflags = ((mp->mnt_vfc->vfc_flags & VFCF_SBDRY) != 0 ? + (flags & PCATCH) : 0) | (PUSER - 1); while ((mp->mnt_kern_flag & MNTK_SUSPEND) != 0) { if (flags & V_NOWAIT) { error = EWOULDBLOCK; goto unlock; } - error = msleep(&mp->mnt_flag, MNT_MTX(mp), - (PUSER - 1) | (flags & PCATCH), "suspfs", 0); + error = msleep(&mp->mnt_flag, MNT_MTX(mp), mflags, + "suspfs", 0); if (error) goto unlock; } } if (flags & V_XSLEEP) goto unlock; mp->mnt_writeopcount++; unlock: if (error != 0 || (flags & V_XSLEEP) != 0) MNT_REL(mp); MNT_IUNLOCK(mp); return (error); } int vn_start_write(vp, mpp, flags) struct vnode *vp; struct mount **mpp; int flags; { struct mount *mp; int error; if (!vn_suspendable(vp, mpp)) return (0); error = 0; /* * If a vnode is provided, get and return the mount point that * to which it will write. */ if (vp != NULL) { if ((error = VOP_GETWRITEMOUNT(vp, mpp)) != 0) { *mpp = NULL; if (error != EOPNOTSUPP) return (error); return (0); } } if ((mp = *mpp) == NULL) return (0); /* * VOP_GETWRITEMOUNT() returns with the mp refcount held through * a vfs_ref(). * As long as a vnode is not provided we need to acquire a * refcount for the provided mountpoint too, in order to * emulate a vfs_ref(). */ MNT_ILOCK(mp); if (vp == NULL) MNT_REF(mp); return (vn_start_write_locked(mp, flags)); } /* * Secondary suspension. Used by operations such as vop_inactive * routines that are needed by the higher level functions. These * are allowed to proceed until all the higher level functions have * completed (indicated by mnt_writeopcount dropping to zero). At that * time, these operations are halted until the suspension is over. */ int vn_start_secondary_write(vp, mpp, flags) struct vnode *vp; struct mount **mpp; int flags; { struct mount *mp; int error; if (!vn_suspendable(vp, mpp)) return (0); retry: if (vp != NULL) { if ((error = VOP_GETWRITEMOUNT(vp, mpp)) != 0) { *mpp = NULL; if (error != EOPNOTSUPP) return (error); return (0); } } /* * If we are not suspended or have not yet reached suspended * mode, then let the operation proceed. */ if ((mp = *mpp) == NULL) return (0); /* * VOP_GETWRITEMOUNT() returns with the mp refcount held through * a vfs_ref(). * As long as a vnode is not provided we need to acquire a * refcount for the provided mountpoint too, in order to * emulate a vfs_ref(). */ MNT_ILOCK(mp); if (vp == NULL) MNT_REF(mp); if ((mp->mnt_kern_flag & (MNTK_SUSPENDED | MNTK_SUSPEND2)) == 0) { mp->mnt_secondary_writes++; mp->mnt_secondary_accwrites++; MNT_IUNLOCK(mp); return (0); } if (flags & V_NOWAIT) { MNT_REL(mp); MNT_IUNLOCK(mp); return (EWOULDBLOCK); } /* * Wait for the suspension to finish. */ - error = msleep(&mp->mnt_flag, MNT_MTX(mp), - (PUSER - 1) | (flags & PCATCH) | PDROP, "suspfs", 0); + error = msleep(&mp->mnt_flag, MNT_MTX(mp), (PUSER - 1) | PDROP | + ((mp->mnt_vfc->vfc_flags & VFCF_SBDRY) != 0 ? (flags & PCATCH) : 0), + "suspfs", 0); vfs_rel(mp); if (error == 0) goto retry; return (error); } /* * Filesystem write operation has completed. If we are suspending and this * operation is the last one, notify the suspender that the suspension is * now in effect. */ void vn_finished_write(mp) struct mount *mp; { if (mp == NULL || !vn_suspendable_mp(mp)) return; MNT_ILOCK(mp); MNT_REL(mp); mp->mnt_writeopcount--; if (mp->mnt_writeopcount < 0) panic("vn_finished_write: neg cnt"); if ((mp->mnt_kern_flag & MNTK_SUSPEND) != 0 && mp->mnt_writeopcount <= 0) wakeup(&mp->mnt_writeopcount); MNT_IUNLOCK(mp); } /* * Filesystem secondary write operation has completed. If we are * suspending and this operation is the last one, notify the suspender * that the suspension is now in effect. */ void vn_finished_secondary_write(mp) struct mount *mp; { if (mp == NULL || !vn_suspendable_mp(mp)) return; MNT_ILOCK(mp); MNT_REL(mp); mp->mnt_secondary_writes--; if (mp->mnt_secondary_writes < 0) panic("vn_finished_secondary_write: neg cnt"); if ((mp->mnt_kern_flag & MNTK_SUSPEND) != 0 && mp->mnt_secondary_writes <= 0) wakeup(&mp->mnt_secondary_writes); MNT_IUNLOCK(mp); } /* * Request a filesystem to suspend write operations. */ int vfs_write_suspend(struct mount *mp, int flags) { int error; MPASS(vn_suspendable_mp(mp)); MNT_ILOCK(mp); if (mp->mnt_susp_owner == curthread) { MNT_IUNLOCK(mp); return (EALREADY); } while (mp->mnt_kern_flag & MNTK_SUSPEND) msleep(&mp->mnt_flag, MNT_MTX(mp), PUSER - 1, "wsuspfs", 0); /* * Unmount holds a write reference on the mount point. If we * own busy reference and drain for writers, we deadlock with * the reference draining in the unmount path. Callers of * vfs_write_suspend() must specify VS_SKIP_UNMOUNT if * vfs_busy() reference is owned and caller is not in the * unmount context. */ if ((flags & VS_SKIP_UNMOUNT) != 0 && (mp->mnt_kern_flag & MNTK_UNMOUNT) != 0) { MNT_IUNLOCK(mp); return (EBUSY); } mp->mnt_kern_flag |= MNTK_SUSPEND; mp->mnt_susp_owner = curthread; if (mp->mnt_writeopcount > 0) (void) msleep(&mp->mnt_writeopcount, MNT_MTX(mp), (PUSER - 1)|PDROP, "suspwt", 0); else MNT_IUNLOCK(mp); if ((error = VFS_SYNC(mp, MNT_SUSPEND)) != 0) vfs_write_resume(mp, 0); return (error); } /* * Request a filesystem to resume write operations. */ void vfs_write_resume(struct mount *mp, int flags) { MPASS(vn_suspendable_mp(mp)); MNT_ILOCK(mp); if ((mp->mnt_kern_flag & MNTK_SUSPEND) != 0) { KASSERT(mp->mnt_susp_owner == curthread, ("mnt_susp_owner")); mp->mnt_kern_flag &= ~(MNTK_SUSPEND | MNTK_SUSPEND2 | MNTK_SUSPENDED); mp->mnt_susp_owner = NULL; wakeup(&mp->mnt_writeopcount); wakeup(&mp->mnt_flag); curthread->td_pflags &= ~TDP_IGNSUSP; if ((flags & VR_START_WRITE) != 0) { MNT_REF(mp); mp->mnt_writeopcount++; } MNT_IUNLOCK(mp); if ((flags & VR_NO_SUSPCLR) == 0) VFS_SUSP_CLEAN(mp); } else if ((flags & VR_START_WRITE) != 0) { MNT_REF(mp); vn_start_write_locked(mp, 0); } else { MNT_IUNLOCK(mp); } } /* * Helper loop around vfs_write_suspend() for filesystem unmount VFS * methods. */ int vfs_write_suspend_umnt(struct mount *mp) { int error; MPASS(vn_suspendable_mp(mp)); KASSERT((curthread->td_pflags & TDP_IGNSUSP) == 0, ("vfs_write_suspend_umnt: recursed")); /* dounmount() already called vn_start_write(). */ for (;;) { vn_finished_write(mp); error = vfs_write_suspend(mp, 0); if (error != 0) { vn_start_write(NULL, &mp, V_WAIT); return (error); } MNT_ILOCK(mp); if ((mp->mnt_kern_flag & MNTK_SUSPENDED) != 0) break; MNT_IUNLOCK(mp); vn_start_write(NULL, &mp, V_WAIT); } mp->mnt_kern_flag &= ~(MNTK_SUSPENDED | MNTK_SUSPEND2); wakeup(&mp->mnt_flag); MNT_IUNLOCK(mp); curthread->td_pflags |= TDP_IGNSUSP; return (0); } /* * Implement kqueues for files by translating it to vnode operation. */ static int vn_kqfilter(struct file *fp, struct knote *kn) { return (VOP_KQFILTER(fp->f_vnode, kn)); } /* * Simplified in-kernel wrapper calls for extended attribute access. * Both calls pass in a NULL credential, authorizing as "kernel" access. * Set IO_NODELOCKED in ioflg if the vnode is already locked. */ int vn_extattr_get(struct vnode *vp, int ioflg, int attrnamespace, const char *attrname, int *buflen, char *buf, struct thread *td) { struct uio auio; struct iovec iov; int error; iov.iov_len = *buflen; iov.iov_base = buf; auio.uio_iov = &iov; auio.uio_iovcnt = 1; auio.uio_rw = UIO_READ; auio.uio_segflg = UIO_SYSSPACE; auio.uio_td = td; auio.uio_offset = 0; auio.uio_resid = *buflen; if ((ioflg & IO_NODELOCKED) == 0) vn_lock(vp, LK_SHARED | LK_RETRY); ASSERT_VOP_LOCKED(vp, "IO_NODELOCKED with no vp lock held"); /* authorize attribute retrieval as kernel */ error = VOP_GETEXTATTR(vp, attrnamespace, attrname, &auio, NULL, NULL, td); if ((ioflg & IO_NODELOCKED) == 0) VOP_UNLOCK(vp, 0); if (error == 0) { *buflen = *buflen - auio.uio_resid; } return (error); } /* * XXX failure mode if partially written? */ int vn_extattr_set(struct vnode *vp, int ioflg, int attrnamespace, const char *attrname, int buflen, char *buf, struct thread *td) { struct uio auio; struct iovec iov; struct mount *mp; int error; iov.iov_len = buflen; iov.iov_base = buf; auio.uio_iov = &iov; auio.uio_iovcnt = 1; auio.uio_rw = UIO_WRITE; auio.uio_segflg = UIO_SYSSPACE; auio.uio_td = td; auio.uio_offset = 0; auio.uio_resid = buflen; if ((ioflg & IO_NODELOCKED) == 0) { if ((error = vn_start_write(vp, &mp, V_WAIT)) != 0) return (error); vn_lock(vp, LK_EXCLUSIVE | LK_RETRY); } ASSERT_VOP_LOCKED(vp, "IO_NODELOCKED with no vp lock held"); /* authorize attribute setting as kernel */ error = VOP_SETEXTATTR(vp, attrnamespace, attrname, &auio, NULL, td); if ((ioflg & IO_NODELOCKED) == 0) { vn_finished_write(mp); VOP_UNLOCK(vp, 0); } return (error); } int vn_extattr_rm(struct vnode *vp, int ioflg, int attrnamespace, const char *attrname, struct thread *td) { struct mount *mp; int error; if ((ioflg & IO_NODELOCKED) == 0) { if ((error = vn_start_write(vp, &mp, V_WAIT)) != 0) return (error); vn_lock(vp, LK_EXCLUSIVE | LK_RETRY); } ASSERT_VOP_LOCKED(vp, "IO_NODELOCKED with no vp lock held"); /* authorize attribute removal as kernel */ error = VOP_DELETEEXTATTR(vp, attrnamespace, attrname, NULL, td); if (error == EOPNOTSUPP) error = VOP_SETEXTATTR(vp, attrnamespace, attrname, NULL, NULL, td); if ((ioflg & IO_NODELOCKED) == 0) { vn_finished_write(mp); VOP_UNLOCK(vp, 0); } return (error); } static int vn_get_ino_alloc_vget(struct mount *mp, void *arg, int lkflags, struct vnode **rvp) { return (VFS_VGET(mp, *(ino_t *)arg, lkflags, rvp)); } int vn_vget_ino(struct vnode *vp, ino_t ino, int lkflags, struct vnode **rvp) { return (vn_vget_ino_gen(vp, vn_get_ino_alloc_vget, &ino, lkflags, rvp)); } int vn_vget_ino_gen(struct vnode *vp, vn_get_ino_t alloc, void *alloc_arg, int lkflags, struct vnode **rvp) { struct mount *mp; int ltype, error; ASSERT_VOP_LOCKED(vp, "vn_vget_ino_get"); mp = vp->v_mount; ltype = VOP_ISLOCKED(vp); KASSERT(ltype == LK_EXCLUSIVE || ltype == LK_SHARED, ("vn_vget_ino: vp not locked")); error = vfs_busy(mp, MBF_NOWAIT); if (error != 0) { vfs_ref(mp); VOP_UNLOCK(vp, 0); error = vfs_busy(mp, 0); vn_lock(vp, ltype | LK_RETRY); vfs_rel(mp); if (error != 0) return (ENOENT); if (vp->v_iflag & VI_DOOMED) { vfs_unbusy(mp); return (ENOENT); } } VOP_UNLOCK(vp, 0); error = alloc(mp, alloc_arg, lkflags, rvp); vfs_unbusy(mp); if (*rvp != vp) vn_lock(vp, ltype | LK_RETRY); if (vp->v_iflag & VI_DOOMED) { if (error == 0) { if (*rvp == vp) vunref(vp); else vput(*rvp); } error = ENOENT; } return (error); } int vn_rlimit_fsize(const struct vnode *vp, const struct uio *uio, const struct thread *td) { if (vp->v_type != VREG || td == NULL) return (0); PROC_LOCK(td->td_proc); if ((uoff_t)uio->uio_offset + uio->uio_resid > lim_cur(td->td_proc, RLIMIT_FSIZE)) { kern_psignal(td->td_proc, SIGXFSZ); PROC_UNLOCK(td->td_proc); return (EFBIG); } PROC_UNLOCK(td->td_proc); return (0); } int vn_chmod(struct file *fp, mode_t mode, struct ucred *active_cred, struct thread *td) { struct vnode *vp; vp = fp->f_vnode; #ifdef AUDIT vn_lock(vp, LK_SHARED | LK_RETRY); AUDIT_ARG_VNODE1(vp); VOP_UNLOCK(vp, 0); #endif return (setfmode(td, active_cred, vp, mode)); } int vn_chown(struct file *fp, uid_t uid, gid_t gid, struct ucred *active_cred, struct thread *td) { struct vnode *vp; vp = fp->f_vnode; #ifdef AUDIT vn_lock(vp, LK_SHARED | LK_RETRY); AUDIT_ARG_VNODE1(vp); VOP_UNLOCK(vp, 0); #endif return (setfown(td, active_cred, vp, uid, gid)); } void vn_pages_remove(struct vnode *vp, vm_pindex_t start, vm_pindex_t end) { vm_object_t object; if ((object = vp->v_object) == NULL) return; VM_OBJECT_WLOCK(object); vm_object_page_remove(object, start, end, 0); VM_OBJECT_WUNLOCK(object); } int vn_bmap_seekhole(struct vnode *vp, u_long cmd, off_t *off, struct ucred *cred) { struct vattr va; daddr_t bn, bnp; uint64_t bsize; off_t noff; int error; KASSERT(cmd == FIOSEEKHOLE || cmd == FIOSEEKDATA, ("Wrong command %lu", cmd)); if (vn_lock(vp, LK_SHARED) != 0) return (EBADF); if (vp->v_type != VREG) { error = ENOTTY; goto unlock; } error = VOP_GETATTR(vp, &va, cred); if (error != 0) goto unlock; noff = *off; if (noff >= va.va_size) { error = ENXIO; goto unlock; } bsize = vp->v_mount->mnt_stat.f_iosize; for (bn = noff / bsize; noff < va.va_size; bn++, noff += bsize) { error = VOP_BMAP(vp, bn, NULL, &bnp, NULL, NULL); if (error == EOPNOTSUPP) { error = ENOTTY; goto unlock; } if ((bnp == -1 && cmd == FIOSEEKHOLE) || (bnp != -1 && cmd == FIOSEEKDATA)) { noff = bn * bsize; if (noff < *off) noff = *off; goto unlock; } } if (noff > va.va_size) noff = va.va_size; /* noff == va.va_size. There is an implicit hole at the end of file. */ if (cmd == FIOSEEKDATA) error = ENXIO; unlock: VOP_UNLOCK(vp, 0); if (error == 0) *off = noff; return (error); } int vn_seek(struct file *fp, off_t offset, int whence, struct thread *td) { struct ucred *cred; struct vnode *vp; struct vattr vattr; off_t foffset, size; int error, noneg; cred = td->td_ucred; vp = fp->f_vnode; foffset = foffset_lock(fp, 0); noneg = (vp->v_type != VCHR); error = 0; switch (whence) { case L_INCR: if (noneg && (foffset < 0 || (offset > 0 && foffset > OFF_MAX - offset))) { error = EOVERFLOW; break; } offset += foffset; break; case L_XTND: vn_lock(vp, LK_SHARED | LK_RETRY); error = VOP_GETATTR(vp, &vattr, cred); VOP_UNLOCK(vp, 0); if (error) break; /* * If the file references a disk device, then fetch * the media size and use that to determine the ending * offset. */ if (vattr.va_size == 0 && vp->v_type == VCHR && fo_ioctl(fp, DIOCGMEDIASIZE, &size, cred, td) == 0) vattr.va_size = size; if (noneg && (vattr.va_size > OFF_MAX || (offset > 0 && vattr.va_size > OFF_MAX - offset))) { error = EOVERFLOW; break; } offset += vattr.va_size; break; case L_SET: break; case SEEK_DATA: error = fo_ioctl(fp, FIOSEEKDATA, &offset, cred, td); break; case SEEK_HOLE: error = fo_ioctl(fp, FIOSEEKHOLE, &offset, cred, td); break; default: error = EINVAL; } if (error == 0 && noneg && offset < 0) error = EINVAL; if (error != 0) goto drop; VFS_KNOTE_UNLOCKED(vp, 0); td->td_uretoff.tdu_off = offset; drop: foffset_unlock(fp, offset, error != 0 ? FOF_NOUPDATE : 0); return (error); } int vn_utimes_perm(struct vnode *vp, struct vattr *vap, struct ucred *cred, struct thread *td) { int error; /* * Grant permission if the caller is the owner of the file, or * the super-user, or has ACL_WRITE_ATTRIBUTES permission on * on the file. If the time pointer is null, then write * permission on the file is also sufficient. * * From NFSv4.1, draft 21, 6.2.1.3.1, Discussion of Mask Attributes: * A user having ACL_WRITE_DATA or ACL_WRITE_ATTRIBUTES * will be allowed to set the times [..] to the current * server time. */ error = VOP_ACCESSX(vp, VWRITE_ATTRIBUTES, cred, td); if (error != 0 && (vap->va_vaflags & VA_UTIMES_NULL) != 0) error = VOP_ACCESS(vp, VWRITE, cred, td); return (error); } int vn_fill_kinfo(struct file *fp, struct kinfo_file *kif, struct filedesc *fdp) { struct vnode *vp; int error; if (fp->f_type == DTYPE_FIFO) kif->kf_type = KF_TYPE_FIFO; else kif->kf_type = KF_TYPE_VNODE; vp = fp->f_vnode; vref(vp); FILEDESC_SUNLOCK(fdp); error = vn_fill_kinfo_vnode(vp, kif); vrele(vp); FILEDESC_SLOCK(fdp); return (error); } int vn_fill_kinfo_vnode(struct vnode *vp, struct kinfo_file *kif) { struct vattr va; char *fullpath, *freepath; int error; kif->kf_vnode_type = vntype_to_kinfo(vp->v_type); freepath = NULL; fullpath = "-"; error = vn_fullpath(curthread, vp, &fullpath, &freepath); if (error == 0) { strlcpy(kif->kf_path, fullpath, sizeof(kif->kf_path)); } if (freepath != NULL) free(freepath, M_TEMP); /* * Retrieve vnode attributes. */ va.va_fsid = VNOVAL; va.va_rdev = NODEV; vn_lock(vp, LK_SHARED | LK_RETRY); error = VOP_GETATTR(vp, &va, curthread->td_ucred); VOP_UNLOCK(vp, 0); if (error != 0) return (error); if (va.va_fsid != VNOVAL) kif->kf_un.kf_file.kf_file_fsid = va.va_fsid; else kif->kf_un.kf_file.kf_file_fsid = vp->v_mount->mnt_stat.f_fsid.val[0]; kif->kf_un.kf_file.kf_file_fileid = va.va_fileid; kif->kf_un.kf_file.kf_file_mode = MAKEIMODE(va.va_type, va.va_mode); kif->kf_un.kf_file.kf_file_size = va.va_size; kif->kf_un.kf_file.kf_file_rdev = va.va_rdev; return (0); } Index: projects/clang350-import/sys/libkern/timingsafe_bcmp.c =================================================================== --- projects/clang350-import/sys/libkern/timingsafe_bcmp.c (nonexistent) +++ projects/clang350-import/sys/libkern/timingsafe_bcmp.c (revision 275749) @@ -0,0 +1,32 @@ +/* $OpenBSD: timingsafe_bcmp.c,v 1.2 2014/06/10 04:16:57 deraadt Exp $ */ +/* + * Copyright (c) 2010 Damien Miller. All rights reserved. + * + * Permission to use, copy, modify, and distribute this software for any + * purpose with or without fee is hereby granted, provided that the above + * copyright notice and this permission notice appear in all copies. + * + * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES + * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF + * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR + * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES + * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN + * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF + * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. + * + * $FreeBSD$ + * + */ + +#include + +int +timingsafe_bcmp(const void *b1, const void *b2, size_t n) +{ + const unsigned char *p1 = b1, *p2 = b2; + int ret = 0; + + for (; n > 0; n--) + ret |= *p1++ ^ *p2++; + return (ret != 0); +} Property changes on: projects/clang350-import/sys/libkern/timingsafe_bcmp.c ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: projects/clang350-import/sys/mips/rmi/dev/sec/rmisec.c =================================================================== --- projects/clang350-import/sys/mips/rmi/dev/sec/rmisec.c (revision 275748) +++ projects/clang350-import/sys/mips/rmi/dev/sec/rmisec.c (revision 275749) @@ -1,576 +1,571 @@ /*- * Copyright (c) 2003-2009 RMI Corporation * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of RMI Corporation, nor the names of its contributors, * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * RMI_BSD */ #include __FBSDID("$FreeBSD$"); #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include "cryptodev_if.h" #include #include #include /* #define RMI_SEC_DEBUG */ void xlr_sec_print_data(struct cryptop *crp); static int xlr_sec_newsession(device_t dev, uint32_t * sidp, struct cryptoini *cri); static int xlr_sec_freesession(device_t dev, uint64_t tid); static int xlr_sec_process(device_t dev, struct cryptop *crp, int hint); static int xlr_sec_probe(device_t); static int xlr_sec_attach(device_t); static int xlr_sec_detach(device_t); static device_method_t xlr_sec_methods[] = { /* device interface */ DEVMETHOD(device_probe, xlr_sec_probe), DEVMETHOD(device_attach, xlr_sec_attach), DEVMETHOD(device_detach, xlr_sec_detach), /* crypto device methods */ DEVMETHOD(cryptodev_newsession, xlr_sec_newsession), DEVMETHOD(cryptodev_freesession,xlr_sec_freesession), DEVMETHOD(cryptodev_process, xlr_sec_process), DEVMETHOD_END }; static driver_t xlr_sec_driver = { "rmisec", xlr_sec_methods, sizeof(struct xlr_sec_softc) }; static devclass_t xlr_sec_devclass; DRIVER_MODULE(rmisec, iodi, xlr_sec_driver, xlr_sec_devclass, 0, 0); MODULE_DEPEND(rmisec, crypto, 1, 1, 1); static int xlr_sec_probe(device_t dev) { device_set_desc(dev, "XLR Security Accelerator"); return (BUS_PROBE_DEFAULT); } /* * Attach an interface that successfully probed. */ static int xlr_sec_attach(device_t dev) { struct xlr_sec_softc *sc = device_get_softc(dev); sc->sc_dev = dev; mtx_init(&sc->sc_mtx, device_get_nameunit(dev), "rmi crypto driver", MTX_DEF); sc->sc_cid = crypto_get_driverid(dev, CRYPTOCAP_F_HARDWARE); if (sc->sc_cid < 0) { printf("xlr_sec - error : could not get the driver id\n"); goto error_exit; } if (crypto_register(sc->sc_cid, CRYPTO_DES_CBC, 0, 0) != 0) printf("register failed for CRYPTO_DES_CBC\n"); if (crypto_register(sc->sc_cid, CRYPTO_3DES_CBC, 0, 0) != 0) printf("register failed for CRYPTO_3DES_CBC\n"); if (crypto_register(sc->sc_cid, CRYPTO_AES_CBC, 0, 0) != 0) printf("register failed for CRYPTO_AES_CBC\n"); if (crypto_register(sc->sc_cid, CRYPTO_ARC4, 0, 0) != 0) printf("register failed for CRYPTO_ARC4\n"); if (crypto_register(sc->sc_cid, CRYPTO_MD5, 0, 0) != 0) printf("register failed for CRYPTO_MD5\n"); if (crypto_register(sc->sc_cid, CRYPTO_SHA1, 0, 0) != 0) printf("register failed for CRYPTO_SHA1\n"); if (crypto_register(sc->sc_cid, CRYPTO_MD5_HMAC, 0, 0) != 0) printf("register failed for CRYPTO_MD5_HMAC\n"); if (crypto_register(sc->sc_cid, CRYPTO_SHA1_HMAC, 0, 0) != 0) printf("register failed for CRYPTO_SHA1_HMAC\n"); xlr_sec_init(sc); device_printf(dev, "Initialization complete!\n"); return (0); error_exit: return (ENXIO); } /* * Detach an interface that successfully probed. */ static int xlr_sec_detach(device_t dev) { int sesn; struct xlr_sec_softc *sc = device_get_softc(dev); struct xlr_sec_session *ses = NULL; symkey_desc_pt desc; for (sesn = 0; sesn < sc->sc_nsessions; sesn++) { ses = &sc->sc_sessions[sesn]; desc = (symkey_desc_pt) ses->desc_ptr; free(desc->user.kern_src, M_DEVBUF); free(desc->user.kern_dest, M_DEVBUF); free(desc->next_src_buf, M_DEVBUF); free(desc->next_dest_buf, M_DEVBUF); free(ses->desc_ptr, M_DEVBUF); } return (0); } /* * Allocate a new 'session' and return an encoded session id. 'sidp' * contains our registration id, and should contain an encoded session * id on successful allocation. */ static int xlr_sec_newsession(device_t dev, u_int32_t *sidp, struct cryptoini *cri) { struct cryptoini *c; struct xlr_sec_softc *sc = device_get_softc(dev); int mac = 0, cry = 0, sesn; struct xlr_sec_session *ses = NULL; if (sidp == NULL || cri == NULL || sc == NULL) return (EINVAL); if (sc->sc_sessions == NULL) { ses = sc->sc_sessions = (struct xlr_sec_session *)malloc( sizeof(struct xlr_sec_session), M_DEVBUF, M_NOWAIT); if (ses == NULL) return (ENOMEM); sesn = 0; sc->sc_nsessions = 1; } else { for (sesn = 0; sesn < sc->sc_nsessions; sesn++) { if (!sc->sc_sessions[sesn].hs_used) { ses = &sc->sc_sessions[sesn]; break; } } if (ses == NULL) { sesn = sc->sc_nsessions; ses = (struct xlr_sec_session *)malloc((sesn + 1) * sizeof(struct xlr_sec_session), M_DEVBUF, M_NOWAIT); if (ses == NULL) return (ENOMEM); bcopy(sc->sc_sessions, ses, sesn * sizeof(*ses)); bzero(sc->sc_sessions, sesn * sizeof(*ses)); free(sc->sc_sessions, M_DEVBUF); sc->sc_sessions = ses; ses = &sc->sc_sessions[sesn]; sc->sc_nsessions++; } } bzero(ses, sizeof(*ses)); ses->sessionid = sesn; ses->desc_ptr = xlr_sec_allocate_desc(ses); if (ses->desc_ptr == NULL) return (ENOMEM); ses->hs_used = 1; for (c = cri; c != NULL; c = c->cri_next) { switch (c->cri_alg) { case CRYPTO_MD5: case CRYPTO_SHA1: case CRYPTO_MD5_HMAC: case CRYPTO_SHA1_HMAC: if (mac) return (EINVAL); mac = 1; ses->hs_mlen = c->cri_mlen; if (ses->hs_mlen == 0) { switch (c->cri_alg) { case CRYPTO_MD5: case CRYPTO_MD5_HMAC: ses->hs_mlen = 16; break; case CRYPTO_SHA1: case CRYPTO_SHA1_HMAC: ses->hs_mlen = 20; break; } } break; case CRYPTO_DES_CBC: case CRYPTO_3DES_CBC: case CRYPTO_AES_CBC: /* XXX this may read fewer, does it matter? */ /* * read_random(ses->hs_iv, c->cri_alg == * CRYPTO_AES_CBC ? XLR_SEC_AES_IV_LENGTH : * XLR_SEC_IV_LENGTH); */ /* FALLTHROUGH */ case CRYPTO_ARC4: if (cry) return (EINVAL); cry = 1; break; default: return (EINVAL); } } if (mac == 0 && cry == 0) return (EINVAL); *sidp = XLR_SEC_SID(device_get_unit(sc->sc_dev), sesn); return (0); } /* * Deallocate a session. * XXX this routine should run a zero'd mac/encrypt key into context ram. * XXX to blow away any keys already stored there. */ static int xlr_sec_freesession(device_t dev, u_int64_t tid) { struct xlr_sec_softc *sc = device_get_softc(dev); int session; u_int32_t sid = CRYPTO_SESID2LID(tid); if (sc == NULL) return (EINVAL); session = XLR_SEC_SESSION(sid); if (session >= sc->sc_nsessions) return (EINVAL); sc->sc_sessions[session].hs_used = 0; return (0); } #ifdef RMI_SEC_DEBUG void xlr_sec_print_data(struct cryptop *crp) { int i, key_len; struct cryptodesc *crp_desc; printf("session id = 0x%llx, crp_ilen = %d, crp_olen=%d \n", crp->crp_sid, crp->crp_ilen, crp->crp_olen); printf("crp_flags = 0x%x\n", crp->crp_flags); printf("crp buf:\n"); for (i = 0; i < crp->crp_ilen; i++) { printf("%c ", crp->crp_buf[i]); if (i % 10 == 0) printf("\n"); } printf("\n"); printf("****************** desc ****************\n"); crp_desc = crp->crp_desc; printf("crd_skip=%d, crd_len=%d, crd_flags=0x%x, crd_alg=%d\n", crp_desc->crd_skip, crp_desc->crd_len, crp_desc->crd_flags, crp_desc->crd_alg); key_len = crp_desc->crd_klen / 8; printf("key(%d) :\n", key_len); for (i = 0; i < key_len; i++) printf("%d", crp_desc->crd_key[i]); printf("\n"); printf(" IV : \n"); for (i = 0; i < EALG_MAX_BLOCK_LEN; i++) printf("%d", crp_desc->crd_iv[i]); printf("\n"); printf("crd_next=%p\n", crp_desc->crd_next); return; } #endif static int xlr_sec_process(device_t dev, struct cryptop *crp, int hint) { struct xlr_sec_softc *sc = device_get_softc(dev); struct xlr_sec_command *cmd = NULL; int session, err; struct cryptodesc *crd1, *crd2, *maccrd, *enccrd; struct xlr_sec_session *ses; if (crp == NULL || crp->crp_callback == NULL) { return (EINVAL); } session = XLR_SEC_SESSION(crp->crp_sid); if (sc == NULL || session >= sc->sc_nsessions) { err = EINVAL; goto errout; } ses = &sc->sc_sessions[session]; cmd = &ses->cmd; if (cmd == NULL) { err = ENOMEM; goto errout; } crd1 = crp->crp_desc; if (crd1 == NULL) { err = EINVAL; goto errout; } crd2 = crd1->crd_next; if (crd2 == NULL) { if (crd1->crd_alg == CRYPTO_MD5_HMAC || crd1->crd_alg == CRYPTO_SHA1_HMAC || crd1->crd_alg == CRYPTO_SHA1 || crd1->crd_alg == CRYPTO_MD5) { maccrd = crd1; enccrd = NULL; } else if (crd1->crd_alg == CRYPTO_DES_CBC || crd1->crd_alg == CRYPTO_3DES_CBC || crd1->crd_alg == CRYPTO_AES_CBC || crd1->crd_alg == CRYPTO_ARC4) { maccrd = NULL; enccrd = crd1; } else { err = EINVAL; goto errout; } } else { if ((crd1->crd_alg == CRYPTO_MD5_HMAC || crd1->crd_alg == CRYPTO_SHA1_HMAC || crd1->crd_alg == CRYPTO_MD5 || crd1->crd_alg == CRYPTO_SHA1) && (crd2->crd_alg == CRYPTO_DES_CBC || crd2->crd_alg == CRYPTO_3DES_CBC || crd2->crd_alg == CRYPTO_AES_CBC || crd2->crd_alg == CRYPTO_ARC4)) { maccrd = crd1; enccrd = crd2; } else if ((crd1->crd_alg == CRYPTO_DES_CBC || crd1->crd_alg == CRYPTO_ARC4 || crd1->crd_alg == CRYPTO_3DES_CBC || crd1->crd_alg == CRYPTO_AES_CBC) && (crd2->crd_alg == CRYPTO_MD5_HMAC || crd2->crd_alg == CRYPTO_SHA1_HMAC || crd2->crd_alg == CRYPTO_MD5 || crd2->crd_alg == CRYPTO_SHA1) && (crd1->crd_flags & CRD_F_ENCRYPT)) { enccrd = crd1; maccrd = crd2; } else { err = EINVAL; goto errout; } } bzero(&cmd->op, sizeof(xlr_sec_io_t)); cmd->op.source_buf = (uint64_t) (unsigned long)crp->crp_buf; cmd->op.source_buf_size = crp->crp_ilen; - if (crp->crp_flags & CRYPTO_F_REL) { - cmd->op.dest_buf = (uint64_t) (unsigned long)crp->crp_buf; - cmd->op.dest_buf_size = crp->crp_ilen; - } else { - cmd->op.dest_buf = (uint64_t) (unsigned long)crp->crp_buf; - cmd->op.dest_buf_size = crp->crp_ilen; - } + cmd->op.dest_buf = (uint64_t) (unsigned long)crp->crp_buf; + cmd->op.dest_buf_size = crp->crp_ilen; cmd->op.num_packets = 1; cmd->op.num_fragments = 1; if (cmd->op.source_buf_size > SEC_MAX_FRAG_LEN) { ses->multi_frag_flag = 1; } else { ses->multi_frag_flag = 0; } if (maccrd) { cmd->maccrd = maccrd; cmd->op.cipher_op = XLR_SEC_CIPHER_MODE_PASS; cmd->op.cipher_mode = XLR_SEC_CIPHER_MODE_NONE; cmd->op.cipher_type = XLR_SEC_CIPHER_TYPE_NONE; cmd->op.cipher_init = 0; cmd->op.cipher_offset = 0; switch (maccrd->crd_alg) { case CRYPTO_MD5: cmd->op.digest_type = XLR_SEC_DIGEST_TYPE_MD5; cmd->op.digest_init = XLR_SEC_DIGEST_INIT_NEWKEY; cmd->op.digest_src = XLR_SEC_DIGEST_SRC_DMA; cmd->op.digest_offset = 0; cmd->op.cksum_type = XLR_SEC_CKSUM_TYPE_NOP; cmd->op.cksum_src = XLR_SEC_CKSUM_SRC_CIPHER; cmd->op.cksum_offset = 0; cmd->op.pkt_hmac = XLR_SEC_LOADHMACKEY_MODE_OLD; cmd->op.pkt_hash = XLR_SEC_PADHASH_PAD; cmd->op.pkt_hashbytes = XLR_SEC_HASHBYTES_ALL8; cmd->op.pkt_next = XLR_SEC_NEXT_FINISH; cmd->op.pkt_iv = XLR_SEC_PKT_IV_OLD; cmd->op.pkt_lastword = XLR_SEC_LASTWORD_128; default: printf("currently not handled\n"); } } if (enccrd) { cmd->enccrd = enccrd; #ifdef RMI_SEC_DEBUG xlr_sec_print_data(crp); #endif if (enccrd->crd_flags & CRD_F_ENCRYPT) { cmd->op.cipher_op = XLR_SEC_CIPHER_OP_ENCRYPT; } else cmd->op.cipher_op = XLR_SEC_CIPHER_OP_DECRYPT; switch (enccrd->crd_alg) { case CRYPTO_DES_CBC: case CRYPTO_3DES_CBC: if (enccrd->crd_alg == CRYPTO_DES_CBC) { cmd->op.cipher_type = XLR_SEC_CIPHER_TYPE_DES; memcpy(&cmd->op.crypt_key[0], enccrd->crd_key, XLR_SEC_DES_KEY_LENGTH); } else { cmd->op.cipher_type = XLR_SEC_CIPHER_TYPE_3DES; //if (enccrd->crd_flags & CRD_F_KEY_EXPLICIT) { memcpy(&cmd->op.crypt_key[0], enccrd->crd_key, XLR_SEC_3DES_KEY_LENGTH); } } cmd->op.cipher_mode = XLR_SEC_CIPHER_MODE_CBC; cmd->op.cipher_init = XLR_SEC_CIPHER_INIT_NK; cmd->op.cipher_offset = XLR_SEC_DES_IV_LENGTH; cmd->op.digest_type = XLR_SEC_DIGEST_TYPE_NONE; cmd->op.digest_init = XLR_SEC_DIGEST_INIT_OLDKEY; cmd->op.digest_src = XLR_SEC_DIGEST_SRC_DMA; cmd->op.digest_offset = 0; cmd->op.cksum_type = XLR_SEC_CKSUM_TYPE_NOP; cmd->op.cksum_src = XLR_SEC_CKSUM_SRC_CIPHER; cmd->op.cksum_offset = 0; cmd->op.pkt_hmac = XLR_SEC_LOADHMACKEY_MODE_OLD; cmd->op.pkt_hash = XLR_SEC_PADHASH_PAD; cmd->op.pkt_hashbytes = XLR_SEC_HASHBYTES_ALL8; cmd->op.pkt_next = XLR_SEC_NEXT_FINISH; cmd->op.pkt_iv = XLR_SEC_PKT_IV_NEW; cmd->op.pkt_lastword = XLR_SEC_LASTWORD_128; //if ((!(enccrd->crd_flags & CRD_F_IV_PRESENT)) && if ((enccrd->crd_flags & CRD_F_IV_EXPLICIT)) { memcpy(&cmd->op.initial_vector[0], enccrd->crd_iv, XLR_SEC_DES_IV_LENGTH); } break; case CRYPTO_AES_CBC: if (enccrd->crd_alg == CRYPTO_AES_CBC) { cmd->op.cipher_type = XLR_SEC_CIPHER_TYPE_AES128; //if (enccrd->crd_flags & CRD_F_KEY_EXPLICIT) { memcpy(&cmd->op.crypt_key[0], enccrd->crd_key, XLR_SEC_AES128_KEY_LENGTH); } } cmd->op.cipher_mode = XLR_SEC_CIPHER_MODE_CBC; cmd->op.cipher_init = XLR_SEC_CIPHER_INIT_NK; cmd->op.cipher_offset = XLR_SEC_AES_BLOCK_SIZE; cmd->op.digest_type = XLR_SEC_DIGEST_TYPE_NONE; cmd->op.digest_init = XLR_SEC_DIGEST_INIT_OLDKEY; cmd->op.digest_src = XLR_SEC_DIGEST_SRC_DMA; cmd->op.digest_offset = 0; cmd->op.cksum_type = XLR_SEC_CKSUM_TYPE_NOP; cmd->op.cksum_src = XLR_SEC_CKSUM_SRC_CIPHER; cmd->op.cksum_offset = 0; cmd->op.pkt_hmac = XLR_SEC_LOADHMACKEY_MODE_OLD; cmd->op.pkt_hash = XLR_SEC_PADHASH_PAD; cmd->op.pkt_hashbytes = XLR_SEC_HASHBYTES_ALL8; cmd->op.pkt_next = XLR_SEC_NEXT_FINISH; cmd->op.pkt_iv = XLR_SEC_PKT_IV_NEW; cmd->op.pkt_lastword = XLR_SEC_LASTWORD_128; //if (!(enccrd->crd_flags & CRD_F_IV_PRESENT)) { if ((enccrd->crd_flags & CRD_F_IV_EXPLICIT)) { memcpy(&cmd->op.initial_vector[0], enccrd->crd_iv, XLR_SEC_AES_BLOCK_SIZE); } //} break; } } cmd->crp = crp; cmd->session_num = session; xlr_sec_setup(ses, cmd, (symkey_desc_pt) ses->desc_ptr); return (0); errout: if (cmd != NULL) free(cmd, M_DEVBUF); crp->crp_etype = err; crypto_done(crp); return (err); } Index: projects/clang350-import/sys/modules/aesni/Makefile =================================================================== --- projects/clang350-import/sys/modules/aesni/Makefile (revision 275748) +++ projects/clang350-import/sys/modules/aesni/Makefile (revision 275749) @@ -1,19 +1,24 @@ # $FreeBSD$ .PATH: ${.CURDIR}/../../crypto/aesni KMOD= aesni SRCS= aesni.c SRCS+= aeskeys_${MACHINE_CPUARCH}.S SRCS+= device_if.h bus_if.h opt_bus.h cryptodev_if.h -OBJS+= aesni_wrap.o +OBJS+= aesni_ghash.o aesni_wrap.o # Remove -nostdinc so we can get the intrinsics. +aesni_ghash.o: aesni_ghash.c + # XXX - gcc won't understand -mpclmul + ${CC} -c ${CFLAGS:C/^-O2$/-O3/:N-nostdinc} ${WERROR} ${PROF} \ + -mmmx -msse -msse4 -maes -mpclmul ${.IMPSRC} + ${CTFCONVERT_CMD} + aesni_wrap.o: aesni_wrap.c ${CC} -c ${CFLAGS:C/^-O2$/-O3/:N-nostdinc} ${WERROR} ${PROF} \ - -mmmx -msse -maes ${.IMPSRC} + -mmmx -msse -msse4 -maes ${.IMPSRC} ${CTFCONVERT_CMD} .include - Index: projects/clang350-import/sys/modules/crypto/Makefile =================================================================== --- projects/clang350-import/sys/modules/crypto/Makefile (revision 275748) +++ projects/clang350-import/sys/modules/crypto/Makefile (revision 275749) @@ -1,24 +1,25 @@ # $FreeBSD$ .PATH: ${.CURDIR}/../../opencrypto .PATH: ${.CURDIR}/../../crypto .PATH: ${.CURDIR}/../../crypto/blowfish .PATH: ${.CURDIR}/../../crypto/camellia .PATH: ${.CURDIR}/../../crypto/des .PATH: ${.CURDIR}/../../crypto/rijndael .PATH: ${.CURDIR}/../../crypto/sha2 .PATH: ${.CURDIR}/../../crypto/siphash KMOD = crypto SRCS = crypto.c cryptodev_if.c SRCS += criov.c cryptosoft.c xform.c SRCS += cast.c cryptodeflate.c rmd160.c rijndael-alg-fst.c rijndael-api.c SRCS += skipjack.c bf_enc.c bf_ecb.c bf_skey.c SRCS += camellia.c camellia-api.c SRCS += des_ecb.c des_enc.c des_setkey.c SRCS += sha1.c sha2.c sha256c.c SRCS += siphash.c +SRCS += gmac.c gfmult.c SRCS += opt_param.h cryptodev_if.h bus_if.h device_if.h SRCS += opt_ddb.h .include Index: projects/clang350-import/sys/net/if_dead.c =================================================================== --- projects/clang350-import/sys/net/if_dead.c (revision 275748) +++ projects/clang350-import/sys/net/if_dead.c (revision 275749) @@ -1,107 +1,115 @@ /*- * Copyright (c) 2009 Robert N. M. Watson * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ /* * When an interface has been detached but not yet freed, we set the various * ifnet function pointers to "ifdead" versions. This prevents unexpected * calls from the network stack into the device driver after if_detach() has * returned. */ #include __FBSDID("$FreeBSD$"); #include #include #include #include #include static int ifdead_output(struct ifnet *ifp, struct mbuf *m, const struct sockaddr *sa, struct route *ro) { m_freem(m); return (ENXIO); } static void ifdead_input(struct ifnet *ifp, struct mbuf *m) { m_freem(m); } static void ifdead_start(struct ifnet *ifp) { } static int ifdead_ioctl(struct ifnet *ifp, u_long cmd, caddr_t data) { return (ENXIO); } static int ifdead_resolvemulti(struct ifnet *ifp, struct sockaddr **llsa, struct sockaddr *sa) { *llsa = NULL; return (ENXIO); } static void ifdead_qflush(struct ifnet *ifp) { } static int ifdead_transmit(struct ifnet *ifp, struct mbuf *m) { m_freem(m); return (ENXIO); } +static uint64_t +ifdead_get_counter(struct ifnet *ifp, ift_counter cnt) +{ + + return (0); +} + void if_dead(struct ifnet *ifp) { ifp->if_output = ifdead_output; ifp->if_input = ifdead_input; ifp->if_start = ifdead_start; ifp->if_ioctl = ifdead_ioctl; ifp->if_resolvemulti = ifdead_resolvemulti; ifp->if_qflush = ifdead_qflush; ifp->if_transmit = ifdead_transmit; + ifp->if_get_counter = ifdead_get_counter; } Index: projects/clang350-import/sys/netinet/sctp_input.c =================================================================== --- projects/clang350-import/sys/netinet/sctp_input.c (revision 275748) +++ projects/clang350-import/sys/netinet/sctp_input.c (revision 275749) @@ -1,6155 +1,6153 @@ /*- * Copyright (c) 2001-2008, by Cisco Systems, Inc. All rights reserved. * Copyright (c) 2008-2012, by Randall Stewart. All rights reserved. * Copyright (c) 2008-2012, by Michael Tuexen. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are met: * * a) Redistributions of source code must retain the above copyright notice, * this list of conditions and the following disclaimer. * * b) Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in * the documentation and/or other materials provided with the distribution. * * c) Neither the name of Cisco Systems, Inc. nor the names of its * contributors may be used to endorse or promote products derived * from this software without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF * THE POSSIBILITY OF SUCH DAMAGE. */ #include __FBSDID("$FreeBSD$"); #include #include #include #include #include #include #include #include #include #include #include #include #include #include #if defined(INET) || defined(INET6) #include #endif #include static void sctp_stop_all_cookie_timers(struct sctp_tcb *stcb) { struct sctp_nets *net; /* * This now not only stops all cookie timers it also stops any INIT * timers as well. This will make sure that the timers are stopped * in all collision cases. */ SCTP_TCB_LOCK_ASSERT(stcb); TAILQ_FOREACH(net, &stcb->asoc.nets, sctp_next) { if (net->rxt_timer.type == SCTP_TIMER_TYPE_COOKIE) { sctp_timer_stop(SCTP_TIMER_TYPE_COOKIE, stcb->sctp_ep, stcb, net, SCTP_FROM_SCTP_INPUT + SCTP_LOC_1); } else if (net->rxt_timer.type == SCTP_TIMER_TYPE_INIT) { sctp_timer_stop(SCTP_TIMER_TYPE_INIT, stcb->sctp_ep, stcb, net, SCTP_FROM_SCTP_INPUT + SCTP_LOC_2); } } } /* INIT handler */ static void sctp_handle_init(struct mbuf *m, int iphlen, int offset, struct sockaddr *src, struct sockaddr *dst, struct sctphdr *sh, struct sctp_init_chunk *cp, struct sctp_inpcb *inp, struct sctp_tcb *stcb, int *abort_no_unlock, uint8_t mflowtype, uint32_t mflowid, uint32_t vrf_id, uint16_t port) { struct sctp_init *init; struct mbuf *op_err; SCTPDBG(SCTP_DEBUG_INPUT2, "sctp_handle_init: handling INIT tcb:%p\n", (void *)stcb); if (stcb == NULL) { SCTP_INP_RLOCK(inp); } /* validate length */ if (ntohs(cp->ch.chunk_length) < sizeof(struct sctp_init_chunk)) { op_err = sctp_generate_cause(SCTP_CAUSE_INVALID_PARAM, ""); sctp_abort_association(inp, stcb, m, iphlen, src, dst, sh, op_err, mflowtype, mflowid, vrf_id, port); if (stcb) *abort_no_unlock = 1; goto outnow; } /* validate parameters */ init = &cp->init; if (init->initiate_tag == 0) { /* protocol error... send abort */ op_err = sctp_generate_cause(SCTP_CAUSE_INVALID_PARAM, ""); sctp_abort_association(inp, stcb, m, iphlen, src, dst, sh, op_err, mflowtype, mflowid, vrf_id, port); if (stcb) *abort_no_unlock = 1; goto outnow; } if (ntohl(init->a_rwnd) < SCTP_MIN_RWND) { /* invalid parameter... send abort */ op_err = sctp_generate_cause(SCTP_CAUSE_INVALID_PARAM, ""); sctp_abort_association(inp, stcb, m, iphlen, src, dst, sh, op_err, mflowtype, mflowid, vrf_id, port); if (stcb) *abort_no_unlock = 1; goto outnow; } if (init->num_inbound_streams == 0) { /* protocol error... send abort */ op_err = sctp_generate_cause(SCTP_CAUSE_INVALID_PARAM, ""); sctp_abort_association(inp, stcb, m, iphlen, src, dst, sh, op_err, mflowtype, mflowid, vrf_id, port); if (stcb) *abort_no_unlock = 1; goto outnow; } if (init->num_outbound_streams == 0) { /* protocol error... send abort */ op_err = sctp_generate_cause(SCTP_CAUSE_INVALID_PARAM, ""); sctp_abort_association(inp, stcb, m, iphlen, src, dst, sh, op_err, mflowtype, mflowid, vrf_id, port); if (stcb) *abort_no_unlock = 1; goto outnow; } if (sctp_validate_init_auth_params(m, offset + sizeof(*cp), offset + ntohs(cp->ch.chunk_length))) { /* auth parameter(s) error... send abort */ op_err = sctp_generate_cause(SCTP_BASE_SYSCTL(sctp_diag_info_code), "Problem with AUTH parameters"); sctp_abort_association(inp, stcb, m, iphlen, src, dst, sh, op_err, mflowtype, mflowid, vrf_id, port); if (stcb) *abort_no_unlock = 1; goto outnow; } /* * We are only accepting if we have a socket with positive * so_qlimit. */ if ((stcb == NULL) && ((inp->sctp_flags & SCTP_PCB_FLAGS_SOCKET_GONE) || (inp->sctp_flags & SCTP_PCB_FLAGS_SOCKET_ALLGONE) || (inp->sctp_socket == NULL) || (inp->sctp_socket->so_qlimit == 0))) { /* * FIX ME ?? What about TCP model and we have a * match/restart case? Actually no fix is needed. the lookup * will always find the existing assoc so stcb would not be * NULL. It may be questionable to do this since we COULD * just send back the INIT-ACK and hope that the app did * accept()'s by the time the COOKIE was sent. But there is * a price to pay for COOKIE generation and I don't want to * pay it on the chance that the app will actually do some * accepts(). The App just looses and should NOT be in this * state :-) */ if (SCTP_BASE_SYSCTL(sctp_blackhole) == 0) { op_err = sctp_generate_cause(SCTP_BASE_SYSCTL(sctp_diag_info_code), "No listener"); sctp_send_abort(m, iphlen, src, dst, sh, 0, op_err, mflowtype, mflowid, vrf_id, port); } goto outnow; } if ((stcb != NULL) && (SCTP_GET_STATE(&stcb->asoc) == SCTP_STATE_SHUTDOWN_ACK_SENT)) { SCTPDBG(SCTP_DEBUG_INPUT3, "sctp_handle_init: sending SHUTDOWN-ACK\n"); sctp_send_shutdown_ack(stcb, NULL); sctp_chunk_output(inp, stcb, SCTP_OUTPUT_FROM_CONTROL_PROC, SCTP_SO_NOT_LOCKED); } else { SCTPDBG(SCTP_DEBUG_INPUT3, "sctp_handle_init: sending INIT-ACK\n"); sctp_send_initiate_ack(inp, stcb, m, iphlen, offset, src, dst, sh, cp, mflowtype, mflowid, vrf_id, port, ((stcb == NULL) ? SCTP_HOLDS_LOCK : SCTP_NOT_LOCKED)); } outnow: if (stcb == NULL) { SCTP_INP_RUNLOCK(inp); } } /* * process peer "INIT/INIT-ACK" chunk returns value < 0 on error */ int sctp_is_there_unsent_data(struct sctp_tcb *stcb, int so_locked #if !defined(__APPLE__) && !defined(SCTP_SO_LOCK_TESTING) SCTP_UNUSED #endif ) { int unsent_data = 0; unsigned int i; struct sctp_stream_queue_pending *sp; struct sctp_association *asoc; /* * This function returns the number of streams that have true unsent * data on them. Note that as it looks through it will clean up any * places that have old data that has been sent but left at top of * stream queue. */ asoc = &stcb->asoc; SCTP_TCB_SEND_LOCK(stcb); if (!stcb->asoc.ss_functions.sctp_ss_is_empty(stcb, asoc)) { /* Check to see if some data queued */ for (i = 0; i < stcb->asoc.streamoutcnt; i++) { /* sa_ignore FREED_MEMORY */ sp = TAILQ_FIRST(&stcb->asoc.strmout[i].outqueue); if (sp == NULL) { continue; } if ((sp->msg_is_complete) && (sp->length == 0) && (sp->sender_all_done)) { /* * We are doing differed cleanup. Last time * through when we took all the data the * sender_all_done was not set. */ if (sp->put_last_out == 0) { SCTP_PRINTF("Gak, put out entire msg with NO end!-1\n"); SCTP_PRINTF("sender_done:%d len:%d msg_comp:%d put_last_out:%d\n", sp->sender_all_done, sp->length, sp->msg_is_complete, sp->put_last_out); } atomic_subtract_int(&stcb->asoc.stream_queue_cnt, 1); TAILQ_REMOVE(&stcb->asoc.strmout[i].outqueue, sp, next); if (sp->net) { sctp_free_remote_addr(sp->net); sp->net = NULL; } if (sp->data) { sctp_m_freem(sp->data); sp->data = NULL; } sctp_free_a_strmoq(stcb, sp, so_locked); } else { unsent_data++; break; } } } SCTP_TCB_SEND_UNLOCK(stcb); return (unsent_data); } static int sctp_process_init(struct sctp_init_chunk *cp, struct sctp_tcb *stcb) { struct sctp_init *init; struct sctp_association *asoc; struct sctp_nets *lnet; unsigned int i; init = &cp->init; asoc = &stcb->asoc; /* save off parameters */ asoc->peer_vtag = ntohl(init->initiate_tag); asoc->peers_rwnd = ntohl(init->a_rwnd); /* init tsn's */ asoc->highest_tsn_inside_map = asoc->asconf_seq_in = ntohl(init->initial_tsn) - 1; if (!TAILQ_EMPTY(&asoc->nets)) { /* update any ssthresh's that may have a default */ TAILQ_FOREACH(lnet, &asoc->nets, sctp_next) { lnet->ssthresh = asoc->peers_rwnd; if (SCTP_BASE_SYSCTL(sctp_logging_level) & (SCTP_CWND_MONITOR_ENABLE | SCTP_CWND_LOGGING_ENABLE)) { sctp_log_cwnd(stcb, lnet, 0, SCTP_CWND_INITIALIZATION); } } } SCTP_TCB_SEND_LOCK(stcb); if (asoc->pre_open_streams > ntohs(init->num_inbound_streams)) { unsigned int newcnt; struct sctp_stream_out *outs; struct sctp_stream_queue_pending *sp, *nsp; struct sctp_tmit_chunk *chk, *nchk; /* abandon the upper streams */ newcnt = ntohs(init->num_inbound_streams); TAILQ_FOREACH_SAFE(chk, &asoc->send_queue, sctp_next, nchk) { if (chk->rec.data.stream_number >= newcnt) { TAILQ_REMOVE(&asoc->send_queue, chk, sctp_next); asoc->send_queue_cnt--; if (asoc->strmout[chk->rec.data.stream_number].chunks_on_queues > 0) { asoc->strmout[chk->rec.data.stream_number].chunks_on_queues--; #ifdef INVARIANTS } else { panic("No chunks on the queues for sid %u.", chk->rec.data.stream_number); #endif } if (chk->data != NULL) { sctp_free_bufspace(stcb, asoc, chk, 1); sctp_ulp_notify(SCTP_NOTIFY_UNSENT_DG_FAIL, stcb, 0, chk, SCTP_SO_NOT_LOCKED); if (chk->data) { sctp_m_freem(chk->data); chk->data = NULL; } } sctp_free_a_chunk(stcb, chk, SCTP_SO_NOT_LOCKED); /* sa_ignore FREED_MEMORY */ } } if (asoc->strmout) { for (i = newcnt; i < asoc->pre_open_streams; i++) { outs = &asoc->strmout[i]; TAILQ_FOREACH_SAFE(sp, &outs->outqueue, next, nsp) { TAILQ_REMOVE(&outs->outqueue, sp, next); asoc->stream_queue_cnt--; sctp_ulp_notify(SCTP_NOTIFY_SPECIAL_SP_FAIL, stcb, 0, sp, SCTP_SO_NOT_LOCKED); if (sp->data) { sctp_m_freem(sp->data); sp->data = NULL; } if (sp->net) { sctp_free_remote_addr(sp->net); sp->net = NULL; } /* Free the chunk */ sctp_free_a_strmoq(stcb, sp, SCTP_SO_NOT_LOCKED); /* sa_ignore FREED_MEMORY */ } } } /* cut back the count */ asoc->pre_open_streams = newcnt; } SCTP_TCB_SEND_UNLOCK(stcb); asoc->strm_realoutsize = asoc->streamoutcnt = asoc->pre_open_streams; /* EY - nr_sack: initialize highest tsn in nr_mapping_array */ asoc->highest_tsn_inside_nr_map = asoc->highest_tsn_inside_map; if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_MAP_LOGGING_ENABLE) { sctp_log_map(0, 5, asoc->highest_tsn_inside_map, SCTP_MAP_SLIDE_RESULT); } /* This is the next one we expect */ asoc->str_reset_seq_in = asoc->asconf_seq_in + 1; asoc->mapping_array_base_tsn = ntohl(init->initial_tsn); asoc->tsn_last_delivered = asoc->cumulative_tsn = asoc->asconf_seq_in; asoc->advanced_peer_ack_point = asoc->last_acked_seq; /* open the requested streams */ if (asoc->strmin != NULL) { /* Free the old ones */ struct sctp_queued_to_read *ctl, *nctl; for (i = 0; i < asoc->streamincnt; i++) { TAILQ_FOREACH_SAFE(ctl, &asoc->strmin[i].inqueue, next, nctl) { TAILQ_REMOVE(&asoc->strmin[i].inqueue, ctl, next); sctp_free_remote_addr(ctl->whoFrom); ctl->whoFrom = NULL; sctp_m_freem(ctl->data); ctl->data = NULL; sctp_free_a_readq(stcb, ctl); } } SCTP_FREE(asoc->strmin, SCTP_M_STRMI); } if (asoc->max_inbound_streams > ntohs(init->num_outbound_streams)) { asoc->streamincnt = ntohs(init->num_outbound_streams); } else { asoc->streamincnt = asoc->max_inbound_streams; } SCTP_MALLOC(asoc->strmin, struct sctp_stream_in *, asoc->streamincnt * sizeof(struct sctp_stream_in), SCTP_M_STRMI); if (asoc->strmin == NULL) { /* we didn't get memory for the streams! */ SCTPDBG(SCTP_DEBUG_INPUT2, "process_init: couldn't get memory for the streams!\n"); return (-1); } for (i = 0; i < asoc->streamincnt; i++) { asoc->strmin[i].stream_no = i; asoc->strmin[i].last_sequence_delivered = 0xffff; TAILQ_INIT(&asoc->strmin[i].inqueue); asoc->strmin[i].delivery_started = 0; } /* * load_address_from_init will put the addresses into the * association when the COOKIE is processed or the INIT-ACK is * processed. Both types of COOKIE's existing and new call this * routine. It will remove addresses that are no longer in the * association (for the restarting case where addresses are * removed). Up front when the INIT arrives we will discard it if it * is a restart and new addresses have been added. */ /* sa_ignore MEMLEAK */ return (0); } /* * INIT-ACK message processing/consumption returns value < 0 on error */ static int sctp_process_init_ack(struct mbuf *m, int iphlen, int offset, struct sockaddr *src, struct sockaddr *dst, struct sctphdr *sh, struct sctp_init_ack_chunk *cp, struct sctp_tcb *stcb, struct sctp_nets *net, int *abort_no_unlock, uint8_t mflowtype, uint32_t mflowid, uint32_t vrf_id) { struct sctp_association *asoc; struct mbuf *op_err; int retval, abort_flag; uint32_t initack_limit; int nat_friendly = 0; /* First verify that we have no illegal param's */ abort_flag = 0; op_err = sctp_arethere_unrecognized_parameters(m, (offset + sizeof(struct sctp_init_chunk)), &abort_flag, (struct sctp_chunkhdr *)cp, &nat_friendly); if (abort_flag) { /* Send an abort and notify peer */ sctp_abort_an_association(stcb->sctp_ep, stcb, op_err, SCTP_SO_NOT_LOCKED); *abort_no_unlock = 1; return (-1); } asoc = &stcb->asoc; asoc->peer_supports_nat = (uint8_t) nat_friendly; /* process the peer's parameters in the INIT-ACK */ retval = sctp_process_init((struct sctp_init_chunk *)cp, stcb); if (retval < 0) { return (retval); } initack_limit = offset + ntohs(cp->ch.chunk_length); /* load all addresses */ if ((retval = sctp_load_addresses_from_init(stcb, m, (offset + sizeof(struct sctp_init_chunk)), initack_limit, src, dst, NULL))) { op_err = sctp_generate_cause(SCTP_BASE_SYSCTL(sctp_diag_info_code), "Problem with address parameters"); SCTPDBG(SCTP_DEBUG_INPUT1, "Load addresses from INIT causes an abort %d\n", retval); sctp_abort_association(stcb->sctp_ep, stcb, m, iphlen, src, dst, sh, op_err, mflowtype, mflowid, vrf_id, net->port); *abort_no_unlock = 1; return (-1); } /* if the peer doesn't support asconf, flush the asconf queue */ if (asoc->asconf_supported == 0) { struct sctp_asconf_addr *param, *nparam; TAILQ_FOREACH_SAFE(param, &asoc->asconf_queue, next, nparam) { TAILQ_REMOVE(&asoc->asconf_queue, param, next); SCTP_FREE(param, SCTP_M_ASC_ADDR); } } stcb->asoc.peer_hmac_id = sctp_negotiate_hmacid(stcb->asoc.peer_hmacs, stcb->asoc.local_hmacs); if (op_err) { sctp_queue_op_err(stcb, op_err); /* queuing will steal away the mbuf chain to the out queue */ op_err = NULL; } /* extract the cookie and queue it to "echo" it back... */ if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_THRESHOLD_LOGGING) { sctp_misc_ints(SCTP_THRESHOLD_CLEAR, stcb->asoc.overall_error_count, 0, SCTP_FROM_SCTP_INPUT, __LINE__); } stcb->asoc.overall_error_count = 0; net->error_count = 0; /* * Cancel the INIT timer, We do this first before queueing the * cookie. We always cancel at the primary to assue that we are * canceling the timer started by the INIT which always goes to the * primary. */ sctp_timer_stop(SCTP_TIMER_TYPE_INIT, stcb->sctp_ep, stcb, asoc->primary_destination, SCTP_FROM_SCTP_INPUT + SCTP_LOC_4); /* calculate the RTO */ net->RTO = sctp_calculate_rto(stcb, asoc, net, &asoc->time_entered, sctp_align_safe_nocopy, SCTP_RTT_FROM_NON_DATA); retval = sctp_send_cookie_echo(m, offset, stcb, net); if (retval < 0) { /* * No cookie, we probably should send a op error. But in any * case if there is no cookie in the INIT-ACK, we can * abandon the peer, its broke. */ if (retval == -3) { /* We abort with an error of missing mandatory param */ op_err = sctp_generate_cause(SCTP_CAUSE_MISSING_PARAM, ""); if (op_err) { /* * Expand beyond to include the mandatory * param cookie */ struct sctp_inv_mandatory_param *mp; SCTP_BUF_LEN(op_err) = sizeof(struct sctp_inv_mandatory_param); mp = mtod(op_err, struct sctp_inv_mandatory_param *); /* Subtract the reserved param */ mp->length = htons(sizeof(struct sctp_inv_mandatory_param) - 2); mp->num_param = htonl(1); mp->param = htons(SCTP_STATE_COOKIE); mp->resv = 0; } sctp_abort_association(stcb->sctp_ep, stcb, m, iphlen, src, dst, sh, op_err, mflowtype, mflowid, vrf_id, net->port); *abort_no_unlock = 1; } return (retval); } return (0); } static void sctp_handle_heartbeat_ack(struct sctp_heartbeat_chunk *cp, struct sctp_tcb *stcb, struct sctp_nets *net) { union sctp_sockstore store; struct sctp_nets *r_net, *f_net; struct timeval tv; int req_prim = 0; uint16_t old_error_counter; if (ntohs(cp->ch.chunk_length) != sizeof(struct sctp_heartbeat_chunk)) { /* Invalid length */ return; } memset(&store, 0, sizeof(store)); switch (cp->heartbeat.hb_info.addr_family) { #ifdef INET case AF_INET: if (cp->heartbeat.hb_info.addr_len == sizeof(struct sockaddr_in)) { store.sin.sin_family = cp->heartbeat.hb_info.addr_family; store.sin.sin_len = cp->heartbeat.hb_info.addr_len; store.sin.sin_port = stcb->rport; memcpy(&store.sin.sin_addr, cp->heartbeat.hb_info.address, sizeof(store.sin.sin_addr)); } else { return; } break; #endif #ifdef INET6 case AF_INET6: if (cp->heartbeat.hb_info.addr_len == sizeof(struct sockaddr_in6)) { store.sin6.sin6_family = cp->heartbeat.hb_info.addr_family; store.sin6.sin6_len = cp->heartbeat.hb_info.addr_len; store.sin6.sin6_port = stcb->rport; memcpy(&store.sin6.sin6_addr, cp->heartbeat.hb_info.address, sizeof(struct in6_addr)); } else { return; } break; #endif default: return; } r_net = sctp_findnet(stcb, &store.sa); if (r_net == NULL) { SCTPDBG(SCTP_DEBUG_INPUT1, "Huh? I can't find the address I sent it to, discard\n"); return; } if ((r_net && (r_net->dest_state & SCTP_ADDR_UNCONFIRMED)) && (r_net->heartbeat_random1 == cp->heartbeat.hb_info.random_value1) && (r_net->heartbeat_random2 == cp->heartbeat.hb_info.random_value2)) { /* * If the its a HB and it's random value is correct when can * confirm the destination. */ r_net->dest_state &= ~SCTP_ADDR_UNCONFIRMED; if (r_net->dest_state & SCTP_ADDR_REQ_PRIMARY) { stcb->asoc.primary_destination = r_net; r_net->dest_state &= ~SCTP_ADDR_REQ_PRIMARY; f_net = TAILQ_FIRST(&stcb->asoc.nets); if (f_net != r_net) { /* * first one on the list is NOT the primary * sctp_cmpaddr() is much more efficent if * the primary is the first on the list, * make it so. */ TAILQ_REMOVE(&stcb->asoc.nets, r_net, sctp_next); TAILQ_INSERT_HEAD(&stcb->asoc.nets, r_net, sctp_next); } req_prim = 1; } sctp_ulp_notify(SCTP_NOTIFY_INTERFACE_CONFIRMED, stcb, 0, (void *)r_net, SCTP_SO_NOT_LOCKED); sctp_timer_stop(SCTP_TIMER_TYPE_HEARTBEAT, stcb->sctp_ep, stcb, r_net, SCTP_FROM_SCTP_INPUT + SCTP_LOC_3); sctp_timer_start(SCTP_TIMER_TYPE_HEARTBEAT, stcb->sctp_ep, stcb, r_net); } old_error_counter = r_net->error_count; r_net->error_count = 0; r_net->hb_responded = 1; tv.tv_sec = cp->heartbeat.hb_info.time_value_1; tv.tv_usec = cp->heartbeat.hb_info.time_value_2; /* Now lets do a RTO with this */ r_net->RTO = sctp_calculate_rto(stcb, &stcb->asoc, r_net, &tv, sctp_align_safe_nocopy, SCTP_RTT_FROM_NON_DATA); if (!(r_net->dest_state & SCTP_ADDR_REACHABLE)) { r_net->dest_state |= SCTP_ADDR_REACHABLE; sctp_ulp_notify(SCTP_NOTIFY_INTERFACE_UP, stcb, 0, (void *)r_net, SCTP_SO_NOT_LOCKED); } if (r_net->dest_state & SCTP_ADDR_PF) { r_net->dest_state &= ~SCTP_ADDR_PF; stcb->asoc.cc_functions.sctp_cwnd_update_exit_pf(stcb, net); } if (old_error_counter > 0) { sctp_timer_stop(SCTP_TIMER_TYPE_HEARTBEAT, stcb->sctp_ep, stcb, r_net, SCTP_FROM_SCTP_INPUT + SCTP_LOC_3); sctp_timer_start(SCTP_TIMER_TYPE_HEARTBEAT, stcb->sctp_ep, stcb, r_net); } if (r_net == stcb->asoc.primary_destination) { if (stcb->asoc.alternate) { /* release the alternate, primary is good */ sctp_free_remote_addr(stcb->asoc.alternate); stcb->asoc.alternate = NULL; } } /* Mobility adaptation */ if (req_prim) { if ((sctp_is_mobility_feature_on(stcb->sctp_ep, SCTP_MOBILITY_BASE) || sctp_is_mobility_feature_on(stcb->sctp_ep, SCTP_MOBILITY_FASTHANDOFF)) && sctp_is_mobility_feature_on(stcb->sctp_ep, SCTP_MOBILITY_PRIM_DELETED)) { sctp_timer_stop(SCTP_TIMER_TYPE_PRIM_DELETED, stcb->sctp_ep, stcb, NULL, SCTP_FROM_SCTP_TIMER + SCTP_LOC_7); if (sctp_is_mobility_feature_on(stcb->sctp_ep, SCTP_MOBILITY_FASTHANDOFF)) { sctp_assoc_immediate_retrans(stcb, stcb->asoc.primary_destination); } if (sctp_is_mobility_feature_on(stcb->sctp_ep, SCTP_MOBILITY_BASE)) { sctp_move_chunks_from_net(stcb, stcb->asoc.deleted_primary); } sctp_delete_prim_timer(stcb->sctp_ep, stcb, stcb->asoc.deleted_primary); } } } static int sctp_handle_nat_colliding_state(struct sctp_tcb *stcb) { /* * return 0 means we want you to proceed with the abort non-zero * means no abort processing */ struct sctpasochead *head; if (SCTP_GET_STATE(&stcb->asoc) == SCTP_STATE_COOKIE_WAIT) { /* generate a new vtag and send init */ LIST_REMOVE(stcb, sctp_asocs); stcb->asoc.my_vtag = sctp_select_a_tag(stcb->sctp_ep, stcb->sctp_ep->sctp_lport, stcb->rport, 1); head = &SCTP_BASE_INFO(sctp_asochash)[SCTP_PCBHASH_ASOC(stcb->asoc.my_vtag, SCTP_BASE_INFO(hashasocmark))]; /* * put it in the bucket in the vtag hash of assoc's for the * system */ LIST_INSERT_HEAD(head, stcb, sctp_asocs); sctp_send_initiate(stcb->sctp_ep, stcb, SCTP_SO_NOT_LOCKED); return (1); } if (SCTP_GET_STATE(&stcb->asoc) == SCTP_STATE_COOKIE_ECHOED) { /* * treat like a case where the cookie expired i.e.: - dump * current cookie. - generate a new vtag. - resend init. */ /* generate a new vtag and send init */ LIST_REMOVE(stcb, sctp_asocs); stcb->asoc.state &= ~SCTP_STATE_COOKIE_ECHOED; stcb->asoc.state |= SCTP_STATE_COOKIE_WAIT; sctp_stop_all_cookie_timers(stcb); sctp_toss_old_cookies(stcb, &stcb->asoc); stcb->asoc.my_vtag = sctp_select_a_tag(stcb->sctp_ep, stcb->sctp_ep->sctp_lport, stcb->rport, 1); head = &SCTP_BASE_INFO(sctp_asochash)[SCTP_PCBHASH_ASOC(stcb->asoc.my_vtag, SCTP_BASE_INFO(hashasocmark))]; /* * put it in the bucket in the vtag hash of assoc's for the * system */ LIST_INSERT_HEAD(head, stcb, sctp_asocs); sctp_send_initiate(stcb->sctp_ep, stcb, SCTP_SO_NOT_LOCKED); return (1); } return (0); } static int sctp_handle_nat_missing_state(struct sctp_tcb *stcb, struct sctp_nets *net) { /* * return 0 means we want you to proceed with the abort non-zero * means no abort processing */ if (stcb->asoc.auth_supported == 0) { SCTPDBG(SCTP_DEBUG_INPUT2, "sctp_handle_nat_missing_state: Peer does not support AUTH, cannot send an asconf\n"); return (0); } sctp_asconf_send_nat_state_update(stcb, net); return (1); } static void sctp_handle_abort(struct sctp_abort_chunk *abort, struct sctp_tcb *stcb, struct sctp_nets *net) { #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) struct socket *so; #endif uint16_t len; uint16_t error; SCTPDBG(SCTP_DEBUG_INPUT2, "sctp_handle_abort: handling ABORT\n"); if (stcb == NULL) return; len = ntohs(abort->ch.chunk_length); if (len > sizeof(struct sctp_chunkhdr)) { /* * Need to check the cause codes for our two magic nat * aborts which don't kill the assoc necessarily. */ struct sctp_missing_nat_state *natc; natc = (struct sctp_missing_nat_state *)(abort + 1); error = ntohs(natc->cause); if (error == SCTP_CAUSE_NAT_COLLIDING_STATE) { SCTPDBG(SCTP_DEBUG_INPUT2, "Received Colliding state abort flags:%x\n", abort->ch.chunk_flags); if (sctp_handle_nat_colliding_state(stcb)) { return; } } else if (error == SCTP_CAUSE_NAT_MISSING_STATE) { SCTPDBG(SCTP_DEBUG_INPUT2, "Received missing state abort flags:%x\n", abort->ch.chunk_flags); if (sctp_handle_nat_missing_state(stcb, net)) { return; } } } else { error = 0; } /* stop any receive timers */ sctp_timer_stop(SCTP_TIMER_TYPE_RECV, stcb->sctp_ep, stcb, net, SCTP_FROM_SCTP_INPUT + SCTP_LOC_6); /* notify user of the abort and clean up... */ sctp_abort_notification(stcb, 1, error, abort, SCTP_SO_NOT_LOCKED); /* free the tcb */ SCTP_STAT_INCR_COUNTER32(sctps_aborted); if ((SCTP_GET_STATE(&stcb->asoc) == SCTP_STATE_OPEN) || (SCTP_GET_STATE(&stcb->asoc) == SCTP_STATE_SHUTDOWN_RECEIVED)) { SCTP_STAT_DECR_GAUGE32(sctps_currestab); } #ifdef SCTP_ASOCLOG_OF_TSNS sctp_print_out_track_log(stcb); #endif #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) so = SCTP_INP_SO(stcb->sctp_ep); atomic_add_int(&stcb->asoc.refcnt, 1); SCTP_TCB_UNLOCK(stcb); SCTP_SOCKET_LOCK(so, 1); SCTP_TCB_LOCK(stcb); atomic_subtract_int(&stcb->asoc.refcnt, 1); #endif stcb->asoc.state |= SCTP_STATE_WAS_ABORTED; (void)sctp_free_assoc(stcb->sctp_ep, stcb, SCTP_NORMAL_PROC, SCTP_FROM_SCTP_INPUT + SCTP_LOC_6); #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) SCTP_SOCKET_UNLOCK(so, 1); #endif SCTPDBG(SCTP_DEBUG_INPUT2, "sctp_handle_abort: finished\n"); } static void sctp_start_net_timers(struct sctp_tcb *stcb) { uint32_t cnt_hb_sent; struct sctp_nets *net; cnt_hb_sent = 0; TAILQ_FOREACH(net, &stcb->asoc.nets, sctp_next) { /* * For each network start: 1) A pmtu timer. 2) A HB timer 3) * If the dest in unconfirmed send a hb as well if under * max_hb_burst have been sent. */ sctp_timer_start(SCTP_TIMER_TYPE_PATHMTURAISE, stcb->sctp_ep, stcb, net); sctp_timer_start(SCTP_TIMER_TYPE_HEARTBEAT, stcb->sctp_ep, stcb, net); if ((net->dest_state & SCTP_ADDR_UNCONFIRMED) && (cnt_hb_sent < SCTP_BASE_SYSCTL(sctp_hb_maxburst))) { sctp_send_hb(stcb, net, SCTP_SO_NOT_LOCKED); cnt_hb_sent++; } } if (cnt_hb_sent) { sctp_chunk_output(stcb->sctp_ep, stcb, SCTP_OUTPUT_FROM_COOKIE_ACK, SCTP_SO_NOT_LOCKED); } } static void sctp_handle_shutdown(struct sctp_shutdown_chunk *cp, struct sctp_tcb *stcb, struct sctp_nets *net, int *abort_flag) { struct sctp_association *asoc; int some_on_streamwheel; #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) struct socket *so; #endif SCTPDBG(SCTP_DEBUG_INPUT2, "sctp_handle_shutdown: handling SHUTDOWN\n"); if (stcb == NULL) return; asoc = &stcb->asoc; if ((SCTP_GET_STATE(asoc) == SCTP_STATE_COOKIE_WAIT) || (SCTP_GET_STATE(asoc) == SCTP_STATE_COOKIE_ECHOED)) { return; } if (ntohs(cp->ch.chunk_length) != sizeof(struct sctp_shutdown_chunk)) { /* Shutdown NOT the expected size */ return; } else { sctp_update_acked(stcb, cp, abort_flag); if (*abort_flag) { return; } } if (asoc->control_pdapi) { /* * With a normal shutdown we assume the end of last record. */ SCTP_INP_READ_LOCK(stcb->sctp_ep); asoc->control_pdapi->end_added = 1; asoc->control_pdapi->pdapi_aborted = 1; asoc->control_pdapi = NULL; SCTP_INP_READ_UNLOCK(stcb->sctp_ep); #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) so = SCTP_INP_SO(stcb->sctp_ep); atomic_add_int(&stcb->asoc.refcnt, 1); SCTP_TCB_UNLOCK(stcb); SCTP_SOCKET_LOCK(so, 1); SCTP_TCB_LOCK(stcb); atomic_subtract_int(&stcb->asoc.refcnt, 1); if (stcb->asoc.state & SCTP_STATE_CLOSED_SOCKET) { /* assoc was freed while we were unlocked */ SCTP_SOCKET_UNLOCK(so, 1); return; } #endif sctp_sorwakeup(stcb->sctp_ep, stcb->sctp_socket); #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) SCTP_SOCKET_UNLOCK(so, 1); #endif } /* goto SHUTDOWN_RECEIVED state to block new requests */ if (stcb->sctp_socket) { if ((SCTP_GET_STATE(asoc) != SCTP_STATE_SHUTDOWN_RECEIVED) && (SCTP_GET_STATE(asoc) != SCTP_STATE_SHUTDOWN_ACK_SENT) && (SCTP_GET_STATE(asoc) != SCTP_STATE_SHUTDOWN_SENT)) { SCTP_SET_STATE(asoc, SCTP_STATE_SHUTDOWN_RECEIVED); SCTP_CLEAR_SUBSTATE(asoc, SCTP_STATE_SHUTDOWN_PENDING); /* * notify upper layer that peer has initiated a * shutdown */ sctp_ulp_notify(SCTP_NOTIFY_PEER_SHUTDOWN, stcb, 0, NULL, SCTP_SO_NOT_LOCKED); /* reset time */ (void)SCTP_GETTIME_TIMEVAL(&asoc->time_entered); } } if (SCTP_GET_STATE(asoc) == SCTP_STATE_SHUTDOWN_SENT) { /* * stop the shutdown timer, since we WILL move to * SHUTDOWN-ACK-SENT. */ sctp_timer_stop(SCTP_TIMER_TYPE_SHUTDOWN, stcb->sctp_ep, stcb, net, SCTP_FROM_SCTP_INPUT + SCTP_LOC_8); } /* Now is there unsent data on a stream somewhere? */ some_on_streamwheel = sctp_is_there_unsent_data(stcb, SCTP_SO_NOT_LOCKED); if (!TAILQ_EMPTY(&asoc->send_queue) || !TAILQ_EMPTY(&asoc->sent_queue) || some_on_streamwheel) { /* By returning we will push more data out */ return; } else { /* no outstanding data to send, so move on... */ /* send SHUTDOWN-ACK */ /* move to SHUTDOWN-ACK-SENT state */ if ((SCTP_GET_STATE(asoc) == SCTP_STATE_OPEN) || (SCTP_GET_STATE(asoc) == SCTP_STATE_SHUTDOWN_RECEIVED)) { SCTP_STAT_DECR_GAUGE32(sctps_currestab); } SCTP_SET_STATE(asoc, SCTP_STATE_SHUTDOWN_ACK_SENT); SCTP_CLEAR_SUBSTATE(asoc, SCTP_STATE_SHUTDOWN_PENDING); sctp_stop_timers_for_shutdown(stcb); sctp_send_shutdown_ack(stcb, net); sctp_timer_start(SCTP_TIMER_TYPE_SHUTDOWNACK, stcb->sctp_ep, stcb, net); } } static void sctp_handle_shutdown_ack(struct sctp_shutdown_ack_chunk *cp SCTP_UNUSED, struct sctp_tcb *stcb, struct sctp_nets *net) { struct sctp_association *asoc; #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) struct socket *so; so = SCTP_INP_SO(stcb->sctp_ep); #endif SCTPDBG(SCTP_DEBUG_INPUT2, "sctp_handle_shutdown_ack: handling SHUTDOWN ACK\n"); if (stcb == NULL) return; asoc = &stcb->asoc; /* process according to association state */ if ((SCTP_GET_STATE(asoc) == SCTP_STATE_COOKIE_WAIT) || (SCTP_GET_STATE(asoc) == SCTP_STATE_COOKIE_ECHOED)) { /* unexpected SHUTDOWN-ACK... do OOTB handling... */ sctp_send_shutdown_complete(stcb, net, 1); SCTP_TCB_UNLOCK(stcb); return; } if ((SCTP_GET_STATE(asoc) != SCTP_STATE_SHUTDOWN_SENT) && (SCTP_GET_STATE(asoc) != SCTP_STATE_SHUTDOWN_ACK_SENT)) { /* unexpected SHUTDOWN-ACK... so ignore... */ SCTP_TCB_UNLOCK(stcb); return; } if (asoc->control_pdapi) { /* * With a normal shutdown we assume the end of last record. */ SCTP_INP_READ_LOCK(stcb->sctp_ep); asoc->control_pdapi->end_added = 1; asoc->control_pdapi->pdapi_aborted = 1; asoc->control_pdapi = NULL; SCTP_INP_READ_UNLOCK(stcb->sctp_ep); #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) atomic_add_int(&stcb->asoc.refcnt, 1); SCTP_TCB_UNLOCK(stcb); SCTP_SOCKET_LOCK(so, 1); SCTP_TCB_LOCK(stcb); atomic_subtract_int(&stcb->asoc.refcnt, 1); if (stcb->asoc.state & SCTP_STATE_CLOSED_SOCKET) { /* assoc was freed while we were unlocked */ SCTP_SOCKET_UNLOCK(so, 1); return; } #endif sctp_sorwakeup(stcb->sctp_ep, stcb->sctp_socket); #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) SCTP_SOCKET_UNLOCK(so, 1); #endif } #ifdef INVARIANTS if (!TAILQ_EMPTY(&asoc->send_queue) || !TAILQ_EMPTY(&asoc->sent_queue) || !stcb->asoc.ss_functions.sctp_ss_is_empty(stcb, asoc)) { panic("Queues are not empty when handling SHUTDOWN-ACK"); } #endif /* stop the timer */ sctp_timer_stop(SCTP_TIMER_TYPE_SHUTDOWN, stcb->sctp_ep, stcb, net, SCTP_FROM_SCTP_INPUT + SCTP_LOC_9); /* send SHUTDOWN-COMPLETE */ sctp_send_shutdown_complete(stcb, net, 0); /* notify upper layer protocol */ if (stcb->sctp_socket) { if ((stcb->sctp_ep->sctp_flags & SCTP_PCB_FLAGS_TCPTYPE) || (stcb->sctp_ep->sctp_flags & SCTP_PCB_FLAGS_IN_TCPPOOL)) { stcb->sctp_socket->so_snd.sb_cc = 0; } sctp_ulp_notify(SCTP_NOTIFY_ASSOC_DOWN, stcb, 0, NULL, SCTP_SO_NOT_LOCKED); } SCTP_STAT_INCR_COUNTER32(sctps_shutdown); /* free the TCB but first save off the ep */ #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) atomic_add_int(&stcb->asoc.refcnt, 1); SCTP_TCB_UNLOCK(stcb); SCTP_SOCKET_LOCK(so, 1); SCTP_TCB_LOCK(stcb); atomic_subtract_int(&stcb->asoc.refcnt, 1); #endif (void)sctp_free_assoc(stcb->sctp_ep, stcb, SCTP_NORMAL_PROC, SCTP_FROM_SCTP_INPUT + SCTP_LOC_10); #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) SCTP_SOCKET_UNLOCK(so, 1); #endif } /* * Skip past the param header and then we will find the chunk that caused the * problem. There are two possiblities ASCONF or FWD-TSN other than that and * our peer must be broken. */ static void sctp_process_unrecog_chunk(struct sctp_tcb *stcb, struct sctp_paramhdr *phdr, struct sctp_nets *net) { struct sctp_chunkhdr *chk; chk = (struct sctp_chunkhdr *)((caddr_t)phdr + sizeof(*phdr)); switch (chk->chunk_type) { case SCTP_ASCONF_ACK: case SCTP_ASCONF: sctp_asconf_cleanup(stcb, net); break; case SCTP_FORWARD_CUM_TSN: stcb->asoc.prsctp_supported = 0; break; default: SCTPDBG(SCTP_DEBUG_INPUT2, "Peer does not support chunk type %d(%x)??\n", chk->chunk_type, (uint32_t) chk->chunk_type); break; } } /* * Skip past the param header and then we will find the param that caused the * problem. There are a number of param's in a ASCONF OR the prsctp param * these will turn of specific features. * XXX: Is this the right thing to do? */ static void sctp_process_unrecog_param(struct sctp_tcb *stcb, struct sctp_paramhdr *phdr) { struct sctp_paramhdr *pbad; pbad = phdr + 1; switch (ntohs(pbad->param_type)) { /* pr-sctp draft */ case SCTP_PRSCTP_SUPPORTED: stcb->asoc.prsctp_supported = 0; break; case SCTP_SUPPORTED_CHUNK_EXT: break; /* draft-ietf-tsvwg-addip-sctp */ case SCTP_HAS_NAT_SUPPORT: stcb->asoc.peer_supports_nat = 0; break; case SCTP_ADD_IP_ADDRESS: case SCTP_DEL_IP_ADDRESS: case SCTP_SET_PRIM_ADDR: stcb->asoc.asconf_supported = 0; break; case SCTP_SUCCESS_REPORT: case SCTP_ERROR_CAUSE_IND: SCTPDBG(SCTP_DEBUG_INPUT2, "Huh, the peer does not support success? or error cause?\n"); SCTPDBG(SCTP_DEBUG_INPUT2, "Turning off ASCONF to this strange peer\n"); stcb->asoc.asconf_supported = 0; break; default: SCTPDBG(SCTP_DEBUG_INPUT2, "Peer does not support param type %d(%x)??\n", pbad->param_type, (uint32_t) pbad->param_type); break; } } static int sctp_handle_error(struct sctp_chunkhdr *ch, struct sctp_tcb *stcb, struct sctp_nets *net) { int chklen; struct sctp_paramhdr *phdr; uint16_t error, error_type; uint16_t error_len; struct sctp_association *asoc; int adjust; #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) struct socket *so; #endif /* parse through all of the errors and process */ asoc = &stcb->asoc; phdr = (struct sctp_paramhdr *)((caddr_t)ch + sizeof(struct sctp_chunkhdr)); chklen = ntohs(ch->chunk_length) - sizeof(struct sctp_chunkhdr); error = 0; while ((size_t)chklen >= sizeof(struct sctp_paramhdr)) { /* Process an Error Cause */ error_type = ntohs(phdr->param_type); error_len = ntohs(phdr->param_length); if ((error_len > chklen) || (error_len == 0)) { /* invalid param length for this param */ SCTPDBG(SCTP_DEBUG_INPUT1, "Bogus length in error param- chunk left:%d errorlen:%d\n", chklen, error_len); return (0); } if (error == 0) { /* report the first error cause */ error = error_type; } switch (error_type) { case SCTP_CAUSE_INVALID_STREAM: case SCTP_CAUSE_MISSING_PARAM: case SCTP_CAUSE_INVALID_PARAM: case SCTP_CAUSE_NO_USER_DATA: SCTPDBG(SCTP_DEBUG_INPUT1, "Software error we got a %d back? We have a bug :/ (or do they?)\n", error_type); break; case SCTP_CAUSE_NAT_COLLIDING_STATE: SCTPDBG(SCTP_DEBUG_INPUT2, "Received Colliding state abort flags:%x\n", ch->chunk_flags); if (sctp_handle_nat_colliding_state(stcb)) { return (0); } break; case SCTP_CAUSE_NAT_MISSING_STATE: SCTPDBG(SCTP_DEBUG_INPUT2, "Received missing state abort flags:%x\n", ch->chunk_flags); if (sctp_handle_nat_missing_state(stcb, net)) { return (0); } break; case SCTP_CAUSE_STALE_COOKIE: /* * We only act if we have echoed a cookie and are * waiting. */ if (SCTP_GET_STATE(asoc) == SCTP_STATE_COOKIE_ECHOED) { int *p; p = (int *)((caddr_t)phdr + sizeof(*phdr)); /* Save the time doubled */ asoc->cookie_preserve_req = ntohl(*p) << 1; asoc->stale_cookie_count++; if (asoc->stale_cookie_count > asoc->max_init_times) { sctp_abort_notification(stcb, 0, 0, NULL, SCTP_SO_NOT_LOCKED); /* now free the asoc */ #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) so = SCTP_INP_SO(stcb->sctp_ep); atomic_add_int(&stcb->asoc.refcnt, 1); SCTP_TCB_UNLOCK(stcb); SCTP_SOCKET_LOCK(so, 1); SCTP_TCB_LOCK(stcb); atomic_subtract_int(&stcb->asoc.refcnt, 1); #endif (void)sctp_free_assoc(stcb->sctp_ep, stcb, SCTP_NORMAL_PROC, SCTP_FROM_SCTP_INPUT + SCTP_LOC_11); #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) SCTP_SOCKET_UNLOCK(so, 1); #endif return (-1); } /* blast back to INIT state */ sctp_toss_old_cookies(stcb, &stcb->asoc); asoc->state &= ~SCTP_STATE_COOKIE_ECHOED; asoc->state |= SCTP_STATE_COOKIE_WAIT; sctp_stop_all_cookie_timers(stcb); sctp_send_initiate(stcb->sctp_ep, stcb, SCTP_SO_NOT_LOCKED); } break; case SCTP_CAUSE_UNRESOLVABLE_ADDR: /* * Nothing we can do here, we don't do hostname * addresses so if the peer does not like my IPv6 * (or IPv4 for that matter) it does not matter. If * they don't support that type of address, they can * NOT possibly get that packet type... i.e. with no * IPv6 you can't recieve a IPv6 packet. so we can * safely ignore this one. If we ever added support * for HOSTNAME Addresses, then we would need to do * something here. */ break; case SCTP_CAUSE_UNRECOG_CHUNK: sctp_process_unrecog_chunk(stcb, phdr, net); break; case SCTP_CAUSE_UNRECOG_PARAM: sctp_process_unrecog_param(stcb, phdr); break; case SCTP_CAUSE_COOKIE_IN_SHUTDOWN: /* * We ignore this since the timer will drive out a * new cookie anyway and there timer will drive us * to send a SHUTDOWN_COMPLETE. We can't send one * here since we don't have their tag. */ break; case SCTP_CAUSE_DELETING_LAST_ADDR: case SCTP_CAUSE_RESOURCE_SHORTAGE: case SCTP_CAUSE_DELETING_SRC_ADDR: /* * We should NOT get these here, but in a * ASCONF-ACK. */ SCTPDBG(SCTP_DEBUG_INPUT2, "Peer sends ASCONF errors in a Operational Error?<%d>?\n", error_type); break; case SCTP_CAUSE_OUT_OF_RESC: /* * And what, pray tell do we do with the fact that * the peer is out of resources? Not really sure we * could do anything but abort. I suspect this * should have came WITH an abort instead of in a * OP-ERROR. */ break; default: SCTPDBG(SCTP_DEBUG_INPUT1, "sctp_handle_error: unknown error type = 0x%xh\n", error_type); break; } adjust = SCTP_SIZE32(error_len); chklen -= adjust; phdr = (struct sctp_paramhdr *)((caddr_t)phdr + adjust); } sctp_ulp_notify(SCTP_NOTIFY_REMOTE_ERROR, stcb, error, ch, SCTP_SO_NOT_LOCKED); return (0); } static int sctp_handle_init_ack(struct mbuf *m, int iphlen, int offset, struct sockaddr *src, struct sockaddr *dst, struct sctphdr *sh, struct sctp_init_ack_chunk *cp, struct sctp_tcb *stcb, struct sctp_nets *net, int *abort_no_unlock, uint8_t mflowtype, uint32_t mflowid, uint32_t vrf_id) { struct sctp_init_ack *init_ack; struct mbuf *op_err; SCTPDBG(SCTP_DEBUG_INPUT2, "sctp_handle_init_ack: handling INIT-ACK\n"); if (stcb == NULL) { SCTPDBG(SCTP_DEBUG_INPUT2, "sctp_handle_init_ack: TCB is null\n"); return (-1); } if (ntohs(cp->ch.chunk_length) < sizeof(struct sctp_init_ack_chunk)) { /* Invalid length */ op_err = sctp_generate_cause(SCTP_CAUSE_INVALID_PARAM, ""); sctp_abort_association(stcb->sctp_ep, stcb, m, iphlen, src, dst, sh, op_err, mflowtype, mflowid, vrf_id, net->port); *abort_no_unlock = 1; return (-1); } init_ack = &cp->init; /* validate parameters */ if (init_ack->initiate_tag == 0) { /* protocol error... send an abort */ op_err = sctp_generate_cause(SCTP_CAUSE_INVALID_PARAM, ""); sctp_abort_association(stcb->sctp_ep, stcb, m, iphlen, src, dst, sh, op_err, mflowtype, mflowid, vrf_id, net->port); *abort_no_unlock = 1; return (-1); } if (ntohl(init_ack->a_rwnd) < SCTP_MIN_RWND) { /* protocol error... send an abort */ op_err = sctp_generate_cause(SCTP_CAUSE_INVALID_PARAM, ""); sctp_abort_association(stcb->sctp_ep, stcb, m, iphlen, src, dst, sh, op_err, mflowtype, mflowid, vrf_id, net->port); *abort_no_unlock = 1; return (-1); } if (init_ack->num_inbound_streams == 0) { /* protocol error... send an abort */ op_err = sctp_generate_cause(SCTP_CAUSE_INVALID_PARAM, ""); sctp_abort_association(stcb->sctp_ep, stcb, m, iphlen, src, dst, sh, op_err, mflowtype, mflowid, vrf_id, net->port); *abort_no_unlock = 1; return (-1); } if (init_ack->num_outbound_streams == 0) { /* protocol error... send an abort */ op_err = sctp_generate_cause(SCTP_CAUSE_INVALID_PARAM, ""); sctp_abort_association(stcb->sctp_ep, stcb, m, iphlen, src, dst, sh, op_err, mflowtype, mflowid, vrf_id, net->port); *abort_no_unlock = 1; return (-1); } /* process according to association state... */ switch (stcb->asoc.state & SCTP_STATE_MASK) { case SCTP_STATE_COOKIE_WAIT: /* this is the expected state for this chunk */ /* process the INIT-ACK parameters */ if (stcb->asoc.primary_destination->dest_state & SCTP_ADDR_UNCONFIRMED) { /* * The primary is where we sent the INIT, we can * always consider it confirmed when the INIT-ACK is * returned. Do this before we load addresses * though. */ stcb->asoc.primary_destination->dest_state &= ~SCTP_ADDR_UNCONFIRMED; sctp_ulp_notify(SCTP_NOTIFY_INTERFACE_CONFIRMED, stcb, 0, (void *)stcb->asoc.primary_destination, SCTP_SO_NOT_LOCKED); } if (sctp_process_init_ack(m, iphlen, offset, src, dst, sh, cp, stcb, net, abort_no_unlock, mflowtype, mflowid, vrf_id) < 0) { /* error in parsing parameters */ return (-1); } /* update our state */ SCTPDBG(SCTP_DEBUG_INPUT2, "moving to COOKIE-ECHOED state\n"); SCTP_SET_STATE(&stcb->asoc, SCTP_STATE_COOKIE_ECHOED); /* reset the RTO calc */ if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_THRESHOLD_LOGGING) { sctp_misc_ints(SCTP_THRESHOLD_CLEAR, stcb->asoc.overall_error_count, 0, SCTP_FROM_SCTP_INPUT, __LINE__); } stcb->asoc.overall_error_count = 0; (void)SCTP_GETTIME_TIMEVAL(&stcb->asoc.time_entered); /* * collapse the init timer back in case of a exponential * backoff */ sctp_timer_start(SCTP_TIMER_TYPE_COOKIE, stcb->sctp_ep, stcb, net); /* * the send at the end of the inbound data processing will * cause the cookie to be sent */ break; case SCTP_STATE_SHUTDOWN_SENT: /* incorrect state... discard */ break; case SCTP_STATE_COOKIE_ECHOED: /* incorrect state... discard */ break; case SCTP_STATE_OPEN: /* incorrect state... discard */ break; case SCTP_STATE_EMPTY: case SCTP_STATE_INUSE: default: /* incorrect state... discard */ return (-1); break; } SCTPDBG(SCTP_DEBUG_INPUT1, "Leaving handle-init-ack end\n"); return (0); } static struct sctp_tcb * sctp_process_cookie_new(struct mbuf *m, int iphlen, int offset, struct sockaddr *src, struct sockaddr *dst, struct sctphdr *sh, struct sctp_state_cookie *cookie, int cookie_len, struct sctp_inpcb *inp, struct sctp_nets **netp, struct sockaddr *init_src, int *notification, int auth_skipped, uint32_t auth_offset, uint32_t auth_len, uint8_t mflowtype, uint32_t mflowid, uint32_t vrf_id, uint16_t port); /* * handle a state cookie for an existing association m: input packet mbuf * chain-- assumes a pullup on IP/SCTP/COOKIE-ECHO chunk note: this is a * "split" mbuf and the cookie signature does not exist offset: offset into * mbuf to the cookie-echo chunk */ static struct sctp_tcb * sctp_process_cookie_existing(struct mbuf *m, int iphlen, int offset, struct sockaddr *src, struct sockaddr *dst, struct sctphdr *sh, struct sctp_state_cookie *cookie, int cookie_len, struct sctp_inpcb *inp, struct sctp_tcb *stcb, struct sctp_nets **netp, struct sockaddr *init_src, int *notification, int auth_skipped, uint32_t auth_offset, uint32_t auth_len, uint8_t mflowtype, uint32_t mflowid, uint32_t vrf_id, uint16_t port) { struct sctp_association *asoc; struct sctp_init_chunk *init_cp, init_buf; struct sctp_init_ack_chunk *initack_cp, initack_buf; struct sctp_nets *net; struct mbuf *op_err; int init_offset, initack_offset, i; int retval; int spec_flag = 0; uint32_t how_indx; #if defined(SCTP_DETAILED_STR_STATS) int j; #endif net = *netp; /* I know that the TCB is non-NULL from the caller */ asoc = &stcb->asoc; for (how_indx = 0; how_indx < sizeof(asoc->cookie_how); how_indx++) { if (asoc->cookie_how[how_indx] == 0) break; } if (how_indx < sizeof(asoc->cookie_how)) { asoc->cookie_how[how_indx] = 1; } if (SCTP_GET_STATE(asoc) == SCTP_STATE_SHUTDOWN_ACK_SENT) { /* SHUTDOWN came in after sending INIT-ACK */ sctp_send_shutdown_ack(stcb, stcb->asoc.primary_destination); op_err = sctp_generate_cause(SCTP_CAUSE_COOKIE_IN_SHUTDOWN, ""); sctp_send_operr_to(src, dst, sh, cookie->peers_vtag, op_err, mflowtype, mflowid, vrf_id, net->port); if (how_indx < sizeof(asoc->cookie_how)) asoc->cookie_how[how_indx] = 2; return (NULL); } /* * find and validate the INIT chunk in the cookie (peer's info) the * INIT should start after the cookie-echo header struct (chunk * header, state cookie header struct) */ init_offset = offset += sizeof(struct sctp_cookie_echo_chunk); init_cp = (struct sctp_init_chunk *) sctp_m_getptr(m, init_offset, sizeof(struct sctp_init_chunk), (uint8_t *) & init_buf); if (init_cp == NULL) { /* could not pull a INIT chunk in cookie */ return (NULL); } if (init_cp->ch.chunk_type != SCTP_INITIATION) { return (NULL); } /* * find and validate the INIT-ACK chunk in the cookie (my info) the * INIT-ACK follows the INIT chunk */ initack_offset = init_offset + SCTP_SIZE32(ntohs(init_cp->ch.chunk_length)); initack_cp = (struct sctp_init_ack_chunk *) sctp_m_getptr(m, initack_offset, sizeof(struct sctp_init_ack_chunk), (uint8_t *) & initack_buf); if (initack_cp == NULL) { /* could not pull INIT-ACK chunk in cookie */ return (NULL); } if (initack_cp->ch.chunk_type != SCTP_INITIATION_ACK) { return (NULL); } if ((ntohl(initack_cp->init.initiate_tag) == asoc->my_vtag) && (ntohl(init_cp->init.initiate_tag) == asoc->peer_vtag)) { /* * case D in Section 5.2.4 Table 2: MMAA process accordingly * to get into the OPEN state */ if (ntohl(initack_cp->init.initial_tsn) != asoc->init_seq_number) { /*- * Opps, this means that we somehow generated two vtag's * the same. I.e. we did: * Us Peer * <---INIT(tag=a)------ * ----INIT-ACK(tag=t)--> * ----INIT(tag=t)------> *1 * <---INIT-ACK(tag=a)--- * <----CE(tag=t)------------- *2 * * At point *1 we should be generating a different * tag t'. Which means we would throw away the CE and send * ours instead. Basically this is case C (throw away side). */ if (how_indx < sizeof(asoc->cookie_how)) asoc->cookie_how[how_indx] = 17; return (NULL); } switch (SCTP_GET_STATE(asoc)) { case SCTP_STATE_COOKIE_WAIT: case SCTP_STATE_COOKIE_ECHOED: /* * INIT was sent but got a COOKIE_ECHO with the * correct tags... just accept it...but we must * process the init so that we can make sure we have * the right seq no's. */ /* First we must process the INIT !! */ retval = sctp_process_init(init_cp, stcb); if (retval < 0) { if (how_indx < sizeof(asoc->cookie_how)) asoc->cookie_how[how_indx] = 3; return (NULL); } /* we have already processed the INIT so no problem */ sctp_timer_stop(SCTP_TIMER_TYPE_HEARTBEAT, inp, stcb, net, SCTP_FROM_SCTP_INPUT + SCTP_LOC_12); sctp_timer_stop(SCTP_TIMER_TYPE_INIT, inp, stcb, net, SCTP_FROM_SCTP_INPUT + SCTP_LOC_13); /* update current state */ if (SCTP_GET_STATE(asoc) == SCTP_STATE_COOKIE_ECHOED) SCTP_STAT_INCR_COUNTER32(sctps_activeestab); else SCTP_STAT_INCR_COUNTER32(sctps_collisionestab); SCTP_SET_STATE(asoc, SCTP_STATE_OPEN); if (asoc->state & SCTP_STATE_SHUTDOWN_PENDING) { sctp_timer_start(SCTP_TIMER_TYPE_SHUTDOWNGUARD, stcb->sctp_ep, stcb, asoc->primary_destination); } SCTP_STAT_INCR_GAUGE32(sctps_currestab); sctp_stop_all_cookie_timers(stcb); if (((stcb->sctp_ep->sctp_flags & SCTP_PCB_FLAGS_TCPTYPE) || (stcb->sctp_ep->sctp_flags & SCTP_PCB_FLAGS_IN_TCPPOOL)) && (inp->sctp_socket->so_qlimit == 0) ) { #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) struct socket *so; #endif /* * Here is where collision would go if we * did a connect() and instead got a * init/init-ack/cookie done before the * init-ack came back.. */ stcb->sctp_ep->sctp_flags |= SCTP_PCB_FLAGS_CONNECTED; #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) so = SCTP_INP_SO(stcb->sctp_ep); atomic_add_int(&stcb->asoc.refcnt, 1); SCTP_TCB_UNLOCK(stcb); SCTP_SOCKET_LOCK(so, 1); SCTP_TCB_LOCK(stcb); atomic_add_int(&stcb->asoc.refcnt, -1); if (stcb->asoc.state & SCTP_STATE_CLOSED_SOCKET) { SCTP_SOCKET_UNLOCK(so, 1); return (NULL); } #endif soisconnected(stcb->sctp_socket); #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) SCTP_SOCKET_UNLOCK(so, 1); #endif } /* notify upper layer */ *notification = SCTP_NOTIFY_ASSOC_UP; /* * since we did not send a HB make sure we don't * double things */ net->hb_responded = 1; net->RTO = sctp_calculate_rto(stcb, asoc, net, &cookie->time_entered, sctp_align_unsafe_makecopy, SCTP_RTT_FROM_NON_DATA); if (stcb->asoc.sctp_autoclose_ticks && (sctp_is_feature_on(inp, SCTP_PCB_FLAGS_AUTOCLOSE))) { sctp_timer_start(SCTP_TIMER_TYPE_AUTOCLOSE, inp, stcb, NULL); } break; default: /* * we're in the OPEN state (or beyond), so peer must * have simply lost the COOKIE-ACK */ break; } /* end switch */ sctp_stop_all_cookie_timers(stcb); /* * We ignore the return code here.. not sure if we should * somehow abort.. but we do have an existing asoc. This * really should not fail. */ if (sctp_load_addresses_from_init(stcb, m, init_offset + sizeof(struct sctp_init_chunk), initack_offset, src, dst, init_src)) { if (how_indx < sizeof(asoc->cookie_how)) asoc->cookie_how[how_indx] = 4; return (NULL); } /* respond with a COOKIE-ACK */ sctp_toss_old_cookies(stcb, asoc); sctp_send_cookie_ack(stcb); if (how_indx < sizeof(asoc->cookie_how)) asoc->cookie_how[how_indx] = 5; return (stcb); } if (ntohl(initack_cp->init.initiate_tag) != asoc->my_vtag && ntohl(init_cp->init.initiate_tag) == asoc->peer_vtag && cookie->tie_tag_my_vtag == 0 && cookie->tie_tag_peer_vtag == 0) { /* * case C in Section 5.2.4 Table 2: XMOO silently discard */ if (how_indx < sizeof(asoc->cookie_how)) asoc->cookie_how[how_indx] = 6; return (NULL); } /* * If nat support, and the below and stcb is established, send back * a ABORT(colliding state) if we are established. */ if ((SCTP_GET_STATE(asoc) == SCTP_STATE_OPEN) && (asoc->peer_supports_nat) && ((ntohl(initack_cp->init.initiate_tag) == asoc->my_vtag) && ((ntohl(init_cp->init.initiate_tag) != asoc->peer_vtag) || (asoc->peer_vtag == 0)))) { /* * Special case - Peer's support nat. We may have two init's * that we gave out the same tag on since one was not * established.. i.e. we get INIT from host-1 behind the nat * and we respond tag-a, we get a INIT from host-2 behind * the nat and we get tag-a again. Then we bring up host-1 * (or 2's) assoc, Then comes the cookie from hsot-2 (or 1). * Now we have colliding state. We must send an abort here * with colliding state indication. */ op_err = sctp_generate_cause(SCTP_CAUSE_NAT_COLLIDING_STATE, ""); sctp_send_abort(m, iphlen, src, dst, sh, 0, op_err, mflowtype, mflowid, vrf_id, port); return (NULL); } if ((ntohl(initack_cp->init.initiate_tag) == asoc->my_vtag) && ((ntohl(init_cp->init.initiate_tag) != asoc->peer_vtag) || (asoc->peer_vtag == 0))) { /* * case B in Section 5.2.4 Table 2: MXAA or MOAA my info * should be ok, re-accept peer info */ if (ntohl(initack_cp->init.initial_tsn) != asoc->init_seq_number) { /* * Extension of case C. If we hit this, then the * random number generator returned the same vtag * when we first sent our INIT-ACK and when we later * sent our INIT. The side with the seq numbers that * are different will be the one that normnally * would have hit case C. This in effect "extends" * our vtags in this collision case to be 64 bits. * The same collision could occur aka you get both * vtag and seq number the same twice in a row.. but * is much less likely. If it did happen then we * would proceed through and bring up the assoc.. we * may end up with the wrong stream setup however.. * which would be bad.. but there is no way to * tell.. until we send on a stream that does not * exist :-) */ if (how_indx < sizeof(asoc->cookie_how)) asoc->cookie_how[how_indx] = 7; return (NULL); } if (how_indx < sizeof(asoc->cookie_how)) asoc->cookie_how[how_indx] = 8; sctp_timer_stop(SCTP_TIMER_TYPE_HEARTBEAT, inp, stcb, net, SCTP_FROM_SCTP_INPUT + SCTP_LOC_14); sctp_stop_all_cookie_timers(stcb); /* * since we did not send a HB make sure we don't double * things */ net->hb_responded = 1; if (stcb->asoc.sctp_autoclose_ticks && sctp_is_feature_on(inp, SCTP_PCB_FLAGS_AUTOCLOSE)) { sctp_timer_start(SCTP_TIMER_TYPE_AUTOCLOSE, inp, stcb, NULL); } asoc->my_rwnd = ntohl(initack_cp->init.a_rwnd); asoc->pre_open_streams = ntohs(initack_cp->init.num_outbound_streams); if (ntohl(init_cp->init.initiate_tag) != asoc->peer_vtag) { /* * Ok the peer probably discarded our data (if we * echoed a cookie+data). So anything on the * sent_queue should be marked for retransmit, we * may not get something to kick us so it COULD * still take a timeout to move these.. but it can't * hurt to mark them. */ struct sctp_tmit_chunk *chk; TAILQ_FOREACH(chk, &stcb->asoc.sent_queue, sctp_next) { if (chk->sent < SCTP_DATAGRAM_RESEND) { chk->sent = SCTP_DATAGRAM_RESEND; sctp_flight_size_decrease(chk); sctp_total_flight_decrease(stcb, chk); sctp_ucount_incr(stcb->asoc.sent_queue_retran_cnt); spec_flag++; } } } /* process the INIT info (peer's info) */ retval = sctp_process_init(init_cp, stcb); if (retval < 0) { if (how_indx < sizeof(asoc->cookie_how)) asoc->cookie_how[how_indx] = 9; return (NULL); } if (sctp_load_addresses_from_init(stcb, m, init_offset + sizeof(struct sctp_init_chunk), initack_offset, src, dst, init_src)) { if (how_indx < sizeof(asoc->cookie_how)) asoc->cookie_how[how_indx] = 10; return (NULL); } if ((asoc->state & SCTP_STATE_COOKIE_WAIT) || (asoc->state & SCTP_STATE_COOKIE_ECHOED)) { *notification = SCTP_NOTIFY_ASSOC_UP; if (((stcb->sctp_ep->sctp_flags & SCTP_PCB_FLAGS_TCPTYPE) || (stcb->sctp_ep->sctp_flags & SCTP_PCB_FLAGS_IN_TCPPOOL)) && (inp->sctp_socket->so_qlimit == 0)) { #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) struct socket *so; #endif stcb->sctp_ep->sctp_flags |= SCTP_PCB_FLAGS_CONNECTED; #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) so = SCTP_INP_SO(stcb->sctp_ep); atomic_add_int(&stcb->asoc.refcnt, 1); SCTP_TCB_UNLOCK(stcb); SCTP_SOCKET_LOCK(so, 1); SCTP_TCB_LOCK(stcb); atomic_add_int(&stcb->asoc.refcnt, -1); if (stcb->asoc.state & SCTP_STATE_CLOSED_SOCKET) { SCTP_SOCKET_UNLOCK(so, 1); return (NULL); } #endif soisconnected(stcb->sctp_socket); #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) SCTP_SOCKET_UNLOCK(so, 1); #endif } if (SCTP_GET_STATE(asoc) == SCTP_STATE_COOKIE_ECHOED) SCTP_STAT_INCR_COUNTER32(sctps_activeestab); else SCTP_STAT_INCR_COUNTER32(sctps_collisionestab); SCTP_STAT_INCR_GAUGE32(sctps_currestab); } else if (SCTP_GET_STATE(asoc) == SCTP_STATE_OPEN) { SCTP_STAT_INCR_COUNTER32(sctps_restartestab); } else { SCTP_STAT_INCR_COUNTER32(sctps_collisionestab); } SCTP_SET_STATE(asoc, SCTP_STATE_OPEN); if (asoc->state & SCTP_STATE_SHUTDOWN_PENDING) { sctp_timer_start(SCTP_TIMER_TYPE_SHUTDOWNGUARD, stcb->sctp_ep, stcb, asoc->primary_destination); } sctp_stop_all_cookie_timers(stcb); sctp_toss_old_cookies(stcb, asoc); sctp_send_cookie_ack(stcb); if (spec_flag) { /* * only if we have retrans set do we do this. What * this call does is get only the COOKIE-ACK out and * then when we return the normal call to * sctp_chunk_output will get the retrans out behind * this. */ sctp_chunk_output(inp, stcb, SCTP_OUTPUT_FROM_COOKIE_ACK, SCTP_SO_NOT_LOCKED); } if (how_indx < sizeof(asoc->cookie_how)) asoc->cookie_how[how_indx] = 11; return (stcb); } if ((ntohl(initack_cp->init.initiate_tag) != asoc->my_vtag && ntohl(init_cp->init.initiate_tag) != asoc->peer_vtag) && cookie->tie_tag_my_vtag == asoc->my_vtag_nonce && cookie->tie_tag_peer_vtag == asoc->peer_vtag_nonce && cookie->tie_tag_peer_vtag != 0) { struct sctpasochead *head; #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) struct socket *so; #endif if (asoc->peer_supports_nat) { /* * This is a gross gross hack. Just call the * cookie_new code since we are allowing a duplicate * association. I hope this works... */ return (sctp_process_cookie_new(m, iphlen, offset, src, dst, sh, cookie, cookie_len, inp, netp, init_src, notification, auth_skipped, auth_offset, auth_len, mflowtype, mflowid, vrf_id, port)); } /* * case A in Section 5.2.4 Table 2: XXMM (peer restarted) */ /* temp code */ if (how_indx < sizeof(asoc->cookie_how)) asoc->cookie_how[how_indx] = 12; sctp_timer_stop(SCTP_TIMER_TYPE_INIT, inp, stcb, net, SCTP_FROM_SCTP_INPUT + SCTP_LOC_15); sctp_timer_stop(SCTP_TIMER_TYPE_HEARTBEAT, inp, stcb, net, SCTP_FROM_SCTP_INPUT + SCTP_LOC_16); /* notify upper layer */ *notification = SCTP_NOTIFY_ASSOC_RESTART; atomic_add_int(&stcb->asoc.refcnt, 1); if ((SCTP_GET_STATE(asoc) != SCTP_STATE_OPEN) && (SCTP_GET_STATE(asoc) != SCTP_STATE_SHUTDOWN_RECEIVED) && (SCTP_GET_STATE(asoc) != SCTP_STATE_SHUTDOWN_SENT)) { SCTP_STAT_INCR_GAUGE32(sctps_currestab); } if (SCTP_GET_STATE(asoc) == SCTP_STATE_OPEN) { SCTP_STAT_INCR_GAUGE32(sctps_restartestab); } else if (SCTP_GET_STATE(asoc) != SCTP_STATE_SHUTDOWN_SENT) { SCTP_STAT_INCR_GAUGE32(sctps_collisionestab); } if (asoc->state & SCTP_STATE_SHUTDOWN_PENDING) { SCTP_SET_STATE(asoc, SCTP_STATE_OPEN); sctp_timer_start(SCTP_TIMER_TYPE_SHUTDOWNGUARD, stcb->sctp_ep, stcb, asoc->primary_destination); } else if (!(asoc->state & SCTP_STATE_SHUTDOWN_SENT)) { /* move to OPEN state, if not in SHUTDOWN_SENT */ SCTP_SET_STATE(asoc, SCTP_STATE_OPEN); } asoc->pre_open_streams = ntohs(initack_cp->init.num_outbound_streams); asoc->init_seq_number = ntohl(initack_cp->init.initial_tsn); asoc->sending_seq = asoc->asconf_seq_out = asoc->str_reset_seq_out = asoc->init_seq_number; asoc->asconf_seq_out_acked = asoc->asconf_seq_out - 1; asoc->asconf_seq_in = asoc->last_acked_seq = asoc->init_seq_number - 1; asoc->str_reset_seq_in = asoc->init_seq_number; asoc->advanced_peer_ack_point = asoc->last_acked_seq; if (asoc->mapping_array) { memset(asoc->mapping_array, 0, asoc->mapping_array_size); } if (asoc->nr_mapping_array) { memset(asoc->nr_mapping_array, 0, asoc->mapping_array_size); } SCTP_TCB_UNLOCK(stcb); #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) so = SCTP_INP_SO(stcb->sctp_ep); SCTP_SOCKET_LOCK(so, 1); #endif SCTP_INP_INFO_WLOCK(); SCTP_INP_WLOCK(stcb->sctp_ep); SCTP_TCB_LOCK(stcb); atomic_add_int(&stcb->asoc.refcnt, -1); /* send up all the data */ SCTP_TCB_SEND_LOCK(stcb); sctp_report_all_outbound(stcb, 0, 1, SCTP_SO_LOCKED); for (i = 0; i < stcb->asoc.streamoutcnt; i++) { stcb->asoc.strmout[i].chunks_on_queues = 0; #if defined(SCTP_DETAILED_STR_STATS) for (j = 0; j < SCTP_PR_SCTP_MAX + 1; j++) { asoc->strmout[i].abandoned_sent[j] = 0; asoc->strmout[i].abandoned_unsent[j] = 0; } #else asoc->strmout[i].abandoned_sent[0] = 0; asoc->strmout[i].abandoned_unsent[0] = 0; #endif stcb->asoc.strmout[i].stream_no = i; stcb->asoc.strmout[i].next_sequence_send = 0; stcb->asoc.strmout[i].last_msg_incomplete = 0; } /* process the INIT-ACK info (my info) */ asoc->my_vtag = ntohl(initack_cp->init.initiate_tag); asoc->my_rwnd = ntohl(initack_cp->init.a_rwnd); /* pull from vtag hash */ LIST_REMOVE(stcb, sctp_asocs); /* re-insert to new vtag position */ head = &SCTP_BASE_INFO(sctp_asochash)[SCTP_PCBHASH_ASOC(stcb->asoc.my_vtag, SCTP_BASE_INFO(hashasocmark))]; /* * put it in the bucket in the vtag hash of assoc's for the * system */ LIST_INSERT_HEAD(head, stcb, sctp_asocs); SCTP_TCB_SEND_UNLOCK(stcb); SCTP_INP_WUNLOCK(stcb->sctp_ep); SCTP_INP_INFO_WUNLOCK(); #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) SCTP_SOCKET_UNLOCK(so, 1); #endif asoc->total_flight = 0; asoc->total_flight_count = 0; /* process the INIT info (peer's info) */ retval = sctp_process_init(init_cp, stcb); if (retval < 0) { if (how_indx < sizeof(asoc->cookie_how)) asoc->cookie_how[how_indx] = 13; return (NULL); } /* * since we did not send a HB make sure we don't double * things */ net->hb_responded = 1; if (sctp_load_addresses_from_init(stcb, m, init_offset + sizeof(struct sctp_init_chunk), initack_offset, src, dst, init_src)) { if (how_indx < sizeof(asoc->cookie_how)) asoc->cookie_how[how_indx] = 14; return (NULL); } /* respond with a COOKIE-ACK */ sctp_stop_all_cookie_timers(stcb); sctp_toss_old_cookies(stcb, asoc); sctp_send_cookie_ack(stcb); if (how_indx < sizeof(asoc->cookie_how)) asoc->cookie_how[how_indx] = 15; return (stcb); } if (how_indx < sizeof(asoc->cookie_how)) asoc->cookie_how[how_indx] = 16; /* all other cases... */ return (NULL); } /* * handle a state cookie for a new association m: input packet mbuf chain-- * assumes a pullup on IP/SCTP/COOKIE-ECHO chunk note: this is a "split" mbuf * and the cookie signature does not exist offset: offset into mbuf to the * cookie-echo chunk length: length of the cookie chunk to: where the init * was from returns a new TCB */ static struct sctp_tcb * sctp_process_cookie_new(struct mbuf *m, int iphlen, int offset, struct sockaddr *src, struct sockaddr *dst, struct sctphdr *sh, struct sctp_state_cookie *cookie, int cookie_len, struct sctp_inpcb *inp, struct sctp_nets **netp, struct sockaddr *init_src, int *notification, int auth_skipped, uint32_t auth_offset, uint32_t auth_len, uint8_t mflowtype, uint32_t mflowid, uint32_t vrf_id, uint16_t port) { struct sctp_tcb *stcb; struct sctp_init_chunk *init_cp, init_buf; struct sctp_init_ack_chunk *initack_cp, initack_buf; union sctp_sockstore store; struct sctp_association *asoc; int init_offset, initack_offset, initack_limit; int retval; int error = 0; uint8_t auth_chunk_buf[SCTP_PARAM_BUFFER_SIZE]; #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) struct socket *so; so = SCTP_INP_SO(inp); #endif /* * find and validate the INIT chunk in the cookie (peer's info) the * INIT should start after the cookie-echo header struct (chunk * header, state cookie header struct) */ init_offset = offset + sizeof(struct sctp_cookie_echo_chunk); init_cp = (struct sctp_init_chunk *) sctp_m_getptr(m, init_offset, sizeof(struct sctp_init_chunk), (uint8_t *) & init_buf); if (init_cp == NULL) { /* could not pull a INIT chunk in cookie */ SCTPDBG(SCTP_DEBUG_INPUT1, "process_cookie_new: could not pull INIT chunk hdr\n"); return (NULL); } if (init_cp->ch.chunk_type != SCTP_INITIATION) { SCTPDBG(SCTP_DEBUG_INPUT1, "HUH? process_cookie_new: could not find INIT chunk!\n"); return (NULL); } initack_offset = init_offset + SCTP_SIZE32(ntohs(init_cp->ch.chunk_length)); /* * find and validate the INIT-ACK chunk in the cookie (my info) the * INIT-ACK follows the INIT chunk */ initack_cp = (struct sctp_init_ack_chunk *) sctp_m_getptr(m, initack_offset, sizeof(struct sctp_init_ack_chunk), (uint8_t *) & initack_buf); if (initack_cp == NULL) { /* could not pull INIT-ACK chunk in cookie */ SCTPDBG(SCTP_DEBUG_INPUT1, "process_cookie_new: could not pull INIT-ACK chunk hdr\n"); return (NULL); } if (initack_cp->ch.chunk_type != SCTP_INITIATION_ACK) { return (NULL); } /* * NOTE: We can't use the INIT_ACK's chk_length to determine the * "initack_limit" value. This is because the chk_length field * includes the length of the cookie, but the cookie is omitted when * the INIT and INIT_ACK are tacked onto the cookie... */ initack_limit = offset + cookie_len; /* * now that we know the INIT/INIT-ACK are in place, create a new TCB * and popluate */ /* * Here we do a trick, we set in NULL for the proc/thread argument. * We do this since in effect we only use the p argument when the * socket is unbound and we must do an implicit bind. Since we are * getting a cookie, we cannot be unbound. */ stcb = sctp_aloc_assoc(inp, init_src, &error, ntohl(initack_cp->init.initiate_tag), vrf_id, (struct thread *)NULL ); if (stcb == NULL) { struct mbuf *op_err; /* memory problem? */ SCTPDBG(SCTP_DEBUG_INPUT1, "process_cookie_new: no room for another TCB!\n"); op_err = sctp_generate_cause(SCTP_CAUSE_OUT_OF_RESC, ""); sctp_abort_association(inp, (struct sctp_tcb *)NULL, m, iphlen, src, dst, sh, op_err, mflowtype, mflowid, vrf_id, port); return (NULL); } /* get the correct sctp_nets */ if (netp) *netp = sctp_findnet(stcb, init_src); asoc = &stcb->asoc; /* get scope variables out of cookie */ asoc->scope.ipv4_local_scope = cookie->ipv4_scope; asoc->scope.site_scope = cookie->site_scope; asoc->scope.local_scope = cookie->local_scope; asoc->scope.loopback_scope = cookie->loopback_scope; if ((asoc->scope.ipv4_addr_legal != cookie->ipv4_addr_legal) || (asoc->scope.ipv6_addr_legal != cookie->ipv6_addr_legal)) { struct mbuf *op_err; /* * Houston we have a problem. The EP changed while the * cookie was in flight. Only recourse is to abort the * association. */ atomic_add_int(&stcb->asoc.refcnt, 1); op_err = sctp_generate_cause(SCTP_CAUSE_OUT_OF_RESC, ""); sctp_abort_association(inp, (struct sctp_tcb *)NULL, m, iphlen, src, dst, sh, op_err, mflowtype, mflowid, vrf_id, port); #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) SCTP_TCB_UNLOCK(stcb); SCTP_SOCKET_LOCK(so, 1); SCTP_TCB_LOCK(stcb); #endif (void)sctp_free_assoc(inp, stcb, SCTP_NORMAL_PROC, SCTP_FROM_SCTP_INPUT + SCTP_LOC_16); #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) SCTP_SOCKET_UNLOCK(so, 1); #endif atomic_subtract_int(&stcb->asoc.refcnt, 1); return (NULL); } /* process the INIT-ACK info (my info) */ asoc->my_vtag = ntohl(initack_cp->init.initiate_tag); asoc->my_rwnd = ntohl(initack_cp->init.a_rwnd); asoc->pre_open_streams = ntohs(initack_cp->init.num_outbound_streams); asoc->init_seq_number = ntohl(initack_cp->init.initial_tsn); asoc->sending_seq = asoc->asconf_seq_out = asoc->str_reset_seq_out = asoc->init_seq_number; asoc->asconf_seq_out_acked = asoc->asconf_seq_out - 1; asoc->asconf_seq_in = asoc->last_acked_seq = asoc->init_seq_number - 1; asoc->str_reset_seq_in = asoc->init_seq_number; asoc->advanced_peer_ack_point = asoc->last_acked_seq; /* process the INIT info (peer's info) */ if (netp) retval = sctp_process_init(init_cp, stcb); else retval = 0; if (retval < 0) { atomic_add_int(&stcb->asoc.refcnt, 1); #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) SCTP_TCB_UNLOCK(stcb); SCTP_SOCKET_LOCK(so, 1); SCTP_TCB_LOCK(stcb); #endif (void)sctp_free_assoc(inp, stcb, SCTP_NORMAL_PROC, SCTP_FROM_SCTP_INPUT + SCTP_LOC_16); #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) SCTP_SOCKET_UNLOCK(so, 1); #endif atomic_subtract_int(&stcb->asoc.refcnt, 1); return (NULL); } /* load all addresses */ if (sctp_load_addresses_from_init(stcb, m, init_offset + sizeof(struct sctp_init_chunk), initack_offset, src, dst, init_src)) { atomic_add_int(&stcb->asoc.refcnt, 1); #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) SCTP_TCB_UNLOCK(stcb); SCTP_SOCKET_LOCK(so, 1); SCTP_TCB_LOCK(stcb); #endif (void)sctp_free_assoc(inp, stcb, SCTP_NORMAL_PROC, SCTP_FROM_SCTP_INPUT + SCTP_LOC_17); #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) SCTP_SOCKET_UNLOCK(so, 1); #endif atomic_subtract_int(&stcb->asoc.refcnt, 1); return (NULL); } /* * verify any preceding AUTH chunk that was skipped */ /* pull the local authentication parameters from the cookie/init-ack */ sctp_auth_get_cookie_params(stcb, m, initack_offset + sizeof(struct sctp_init_ack_chunk), initack_limit - (initack_offset + sizeof(struct sctp_init_ack_chunk))); if (auth_skipped) { struct sctp_auth_chunk *auth; auth = (struct sctp_auth_chunk *) sctp_m_getptr(m, auth_offset, auth_len, auth_chunk_buf); if ((auth == NULL) || sctp_handle_auth(stcb, auth, m, auth_offset)) { /* auth HMAC failed, dump the assoc and packet */ SCTPDBG(SCTP_DEBUG_AUTH1, "COOKIE-ECHO: AUTH failed\n"); atomic_add_int(&stcb->asoc.refcnt, 1); #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) SCTP_TCB_UNLOCK(stcb); SCTP_SOCKET_LOCK(so, 1); SCTP_TCB_LOCK(stcb); #endif (void)sctp_free_assoc(inp, stcb, SCTP_NORMAL_PROC, SCTP_FROM_SCTP_INPUT + SCTP_LOC_18); #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) SCTP_SOCKET_UNLOCK(so, 1); #endif atomic_subtract_int(&stcb->asoc.refcnt, 1); return (NULL); } else { /* remaining chunks checked... good to go */ stcb->asoc.authenticated = 1; } } /* update current state */ SCTPDBG(SCTP_DEBUG_INPUT2, "moving to OPEN state\n"); SCTP_SET_STATE(asoc, SCTP_STATE_OPEN); if (asoc->state & SCTP_STATE_SHUTDOWN_PENDING) { sctp_timer_start(SCTP_TIMER_TYPE_SHUTDOWNGUARD, stcb->sctp_ep, stcb, asoc->primary_destination); } sctp_stop_all_cookie_timers(stcb); SCTP_STAT_INCR_COUNTER32(sctps_passiveestab); SCTP_STAT_INCR_GAUGE32(sctps_currestab); /* * if we're doing ASCONFs, check to see if we have any new local * addresses that need to get added to the peer (eg. addresses * changed while cookie echo in flight). This needs to be done * after we go to the OPEN state to do the correct asconf * processing. else, make sure we have the correct addresses in our * lists */ /* warning, we re-use sin, sin6, sa_store here! */ /* pull in local_address (our "from" address) */ switch (cookie->laddr_type) { #ifdef INET case SCTP_IPV4_ADDRESS: /* source addr is IPv4 */ memset(&store.sin, 0, sizeof(struct sockaddr_in)); store.sin.sin_family = AF_INET; store.sin.sin_len = sizeof(struct sockaddr_in); store.sin.sin_addr.s_addr = cookie->laddress[0]; break; #endif #ifdef INET6 case SCTP_IPV6_ADDRESS: /* source addr is IPv6 */ memset(&store.sin6, 0, sizeof(struct sockaddr_in6)); store.sin6.sin6_family = AF_INET6; store.sin6.sin6_len = sizeof(struct sockaddr_in6); store.sin6.sin6_scope_id = cookie->scope_id; memcpy(&store.sin6.sin6_addr, cookie->laddress, sizeof(struct in6_addr)); break; #endif default: atomic_add_int(&stcb->asoc.refcnt, 1); #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) SCTP_TCB_UNLOCK(stcb); SCTP_SOCKET_LOCK(so, 1); SCTP_TCB_LOCK(stcb); #endif (void)sctp_free_assoc(inp, stcb, SCTP_NORMAL_PROC, SCTP_FROM_SCTP_INPUT + SCTP_LOC_19); #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) SCTP_SOCKET_UNLOCK(so, 1); #endif atomic_subtract_int(&stcb->asoc.refcnt, 1); return (NULL); } /* set up to notify upper layer */ *notification = SCTP_NOTIFY_ASSOC_UP; if (((stcb->sctp_ep->sctp_flags & SCTP_PCB_FLAGS_TCPTYPE) || (stcb->sctp_ep->sctp_flags & SCTP_PCB_FLAGS_IN_TCPPOOL)) && (inp->sctp_socket->so_qlimit == 0)) { /* * This is an endpoint that called connect() how it got a * cookie that is NEW is a bit of a mystery. It must be that * the INIT was sent, but before it got there.. a complete * INIT/INIT-ACK/COOKIE arrived. But of course then it * should have went to the other code.. not here.. oh well.. * a bit of protection is worth having.. */ stcb->sctp_ep->sctp_flags |= SCTP_PCB_FLAGS_CONNECTED; #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) atomic_add_int(&stcb->asoc.refcnt, 1); SCTP_TCB_UNLOCK(stcb); SCTP_SOCKET_LOCK(so, 1); SCTP_TCB_LOCK(stcb); atomic_subtract_int(&stcb->asoc.refcnt, 1); if (stcb->asoc.state & SCTP_STATE_CLOSED_SOCKET) { SCTP_SOCKET_UNLOCK(so, 1); return (NULL); } #endif soisconnected(stcb->sctp_socket); #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) SCTP_SOCKET_UNLOCK(so, 1); #endif } else if ((stcb->sctp_ep->sctp_flags & SCTP_PCB_FLAGS_TCPTYPE) && (inp->sctp_socket->so_qlimit)) { /* * We don't want to do anything with this one. Since it is * the listening guy. The timer will get started for * accepted connections in the caller. */ ; } /* since we did not send a HB make sure we don't double things */ if ((netp) && (*netp)) (*netp)->hb_responded = 1; if (stcb->asoc.sctp_autoclose_ticks && sctp_is_feature_on(inp, SCTP_PCB_FLAGS_AUTOCLOSE)) { sctp_timer_start(SCTP_TIMER_TYPE_AUTOCLOSE, inp, stcb, NULL); } /* calculate the RTT */ (void)SCTP_GETTIME_TIMEVAL(&stcb->asoc.time_entered); if ((netp) && (*netp)) { (*netp)->RTO = sctp_calculate_rto(stcb, asoc, *netp, &cookie->time_entered, sctp_align_unsafe_makecopy, SCTP_RTT_FROM_NON_DATA); } /* respond with a COOKIE-ACK */ sctp_send_cookie_ack(stcb); /* * check the address lists for any ASCONFs that need to be sent * AFTER the cookie-ack is sent */ sctp_check_address_list(stcb, m, initack_offset + sizeof(struct sctp_init_ack_chunk), initack_limit - (initack_offset + sizeof(struct sctp_init_ack_chunk)), &store.sa, cookie->local_scope, cookie->site_scope, cookie->ipv4_scope, cookie->loopback_scope); return (stcb); } /* * CODE LIKE THIS NEEDS TO RUN IF the peer supports the NAT extension, i.e * we NEED to make sure we are not already using the vtag. If so we * need to send back an ABORT-TRY-AGAIN-WITH-NEW-TAG No middle box bit! head = &SCTP_BASE_INFO(sctp_asochash)[SCTP_PCBHASH_ASOC(tag, SCTP_BASE_INFO(hashasocmark))]; LIST_FOREACH(stcb, head, sctp_asocs) { if ((stcb->asoc.my_vtag == tag) && (stcb->rport == rport) && (inp == stcb->sctp_ep)) { -- SEND ABORT - TRY AGAIN -- } } */ /* * handles a COOKIE-ECHO message stcb: modified to either a new or left as * existing (non-NULL) TCB */ static struct mbuf * sctp_handle_cookie_echo(struct mbuf *m, int iphlen, int offset, struct sockaddr *src, struct sockaddr *dst, struct sctphdr *sh, struct sctp_cookie_echo_chunk *cp, struct sctp_inpcb **inp_p, struct sctp_tcb **stcb, struct sctp_nets **netp, int auth_skipped, uint32_t auth_offset, uint32_t auth_len, struct sctp_tcb **locked_tcb, uint8_t mflowtype, uint32_t mflowid, uint32_t vrf_id, uint16_t port) { struct sctp_state_cookie *cookie; struct sctp_tcb *l_stcb = *stcb; struct sctp_inpcb *l_inp; struct sockaddr *to; struct sctp_pcb *ep; struct mbuf *m_sig; uint8_t calc_sig[SCTP_SIGNATURE_SIZE], tmp_sig[SCTP_SIGNATURE_SIZE]; uint8_t *sig; uint8_t cookie_ok = 0; unsigned int sig_offset, cookie_offset; unsigned int cookie_len; struct timeval now; struct timeval time_expires; int notification = 0; struct sctp_nets *netl; int had_a_existing_tcb = 0; int send_int_conf = 0; #ifdef INET struct sockaddr_in sin; #endif #ifdef INET6 struct sockaddr_in6 sin6; #endif SCTPDBG(SCTP_DEBUG_INPUT2, "sctp_handle_cookie: handling COOKIE-ECHO\n"); if (inp_p == NULL) { return (NULL); } cookie = &cp->cookie; cookie_offset = offset + sizeof(struct sctp_chunkhdr); cookie_len = ntohs(cp->ch.chunk_length); if ((cookie->peerport != sh->src_port) && (cookie->myport != sh->dest_port) && (cookie->my_vtag != sh->v_tag)) { /* * invalid ports or bad tag. Note that we always leave the * v_tag in the header in network order and when we stored * it in the my_vtag slot we also left it in network order. * This maintains the match even though it may be in the * opposite byte order of the machine :-> */ return (NULL); } if (cookie_len < sizeof(struct sctp_cookie_echo_chunk) + sizeof(struct sctp_init_chunk) + sizeof(struct sctp_init_ack_chunk) + SCTP_SIGNATURE_SIZE) { /* cookie too small */ return (NULL); } /* * split off the signature into its own mbuf (since it should not be * calculated in the sctp_hmac_m() call). */ sig_offset = offset + cookie_len - SCTP_SIGNATURE_SIZE; m_sig = m_split(m, sig_offset, M_NOWAIT); if (m_sig == NULL) { /* out of memory or ?? */ return (NULL); } #ifdef SCTP_MBUF_LOGGING if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_MBUF_LOGGING_ENABLE) { struct mbuf *mat; for (mat = m_sig; mat; mat = SCTP_BUF_NEXT(mat)) { if (SCTP_BUF_IS_EXTENDED(mat)) { sctp_log_mb(mat, SCTP_MBUF_SPLIT); } } } #endif /* * compute the signature/digest for the cookie */ ep = &(*inp_p)->sctp_ep; l_inp = *inp_p; if (l_stcb) { SCTP_TCB_UNLOCK(l_stcb); } SCTP_INP_RLOCK(l_inp); if (l_stcb) { SCTP_TCB_LOCK(l_stcb); } /* which cookie is it? */ if ((cookie->time_entered.tv_sec < (long)ep->time_of_secret_change) && (ep->current_secret_number != ep->last_secret_number)) { /* it's the old cookie */ (void)sctp_hmac_m(SCTP_HMAC, (uint8_t *) ep->secret_key[(int)ep->last_secret_number], SCTP_SECRET_SIZE, m, cookie_offset, calc_sig, 0); } else { /* it's the current cookie */ (void)sctp_hmac_m(SCTP_HMAC, (uint8_t *) ep->secret_key[(int)ep->current_secret_number], SCTP_SECRET_SIZE, m, cookie_offset, calc_sig, 0); } /* get the signature */ SCTP_INP_RUNLOCK(l_inp); sig = (uint8_t *) sctp_m_getptr(m_sig, 0, SCTP_SIGNATURE_SIZE, (uint8_t *) & tmp_sig); if (sig == NULL) { /* couldn't find signature */ sctp_m_freem(m_sig); return (NULL); } /* compare the received digest with the computed digest */ if (memcmp(calc_sig, sig, SCTP_SIGNATURE_SIZE) != 0) { /* try the old cookie? */ if ((cookie->time_entered.tv_sec == (long)ep->time_of_secret_change) && (ep->current_secret_number != ep->last_secret_number)) { /* compute digest with old */ (void)sctp_hmac_m(SCTP_HMAC, (uint8_t *) ep->secret_key[(int)ep->last_secret_number], SCTP_SECRET_SIZE, m, cookie_offset, calc_sig, 0); /* compare */ if (memcmp(calc_sig, sig, SCTP_SIGNATURE_SIZE) == 0) cookie_ok = 1; } } else { cookie_ok = 1; } /* * Now before we continue we must reconstruct our mbuf so that * normal processing of any other chunks will work. */ { struct mbuf *m_at; m_at = m; while (SCTP_BUF_NEXT(m_at) != NULL) { m_at = SCTP_BUF_NEXT(m_at); } SCTP_BUF_NEXT(m_at) = m_sig; } if (cookie_ok == 0) { SCTPDBG(SCTP_DEBUG_INPUT2, "handle_cookie_echo: cookie signature validation failed!\n"); SCTPDBG(SCTP_DEBUG_INPUT2, "offset = %u, cookie_offset = %u, sig_offset = %u\n", (uint32_t) offset, cookie_offset, sig_offset); return (NULL); } /* * check the cookie timestamps to be sure it's not stale */ (void)SCTP_GETTIME_TIMEVAL(&now); /* Expire time is in Ticks, so we convert to seconds */ time_expires.tv_sec = cookie->time_entered.tv_sec + TICKS_TO_SEC(cookie->cookie_life); time_expires.tv_usec = cookie->time_entered.tv_usec; /* * TODO sctp_constants.h needs alternative time macros when _KERNEL * is undefined. */ if (timevalcmp(&now, &time_expires, >)) { /* cookie is stale! */ struct mbuf *op_err; struct sctp_stale_cookie_msg *scm; uint32_t tim; op_err = sctp_get_mbuf_for_msg(sizeof(struct sctp_stale_cookie_msg), 0, M_NOWAIT, 1, MT_DATA); if (op_err == NULL) { /* FOOBAR */ return (NULL); } /* Set the len */ SCTP_BUF_LEN(op_err) = sizeof(struct sctp_stale_cookie_msg); scm = mtod(op_err, struct sctp_stale_cookie_msg *); scm->ph.param_type = htons(SCTP_CAUSE_STALE_COOKIE); scm->ph.param_length = htons((sizeof(struct sctp_paramhdr) + (sizeof(uint32_t)))); /* seconds to usec */ tim = (now.tv_sec - time_expires.tv_sec) * 1000000; /* add in usec */ if (tim == 0) tim = now.tv_usec - cookie->time_entered.tv_usec; scm->time_usec = htonl(tim); sctp_send_operr_to(src, dst, sh, cookie->peers_vtag, op_err, mflowtype, mflowid, vrf_id, port); return (NULL); } /* * Now we must see with the lookup address if we have an existing * asoc. This will only happen if we were in the COOKIE-WAIT state * and a INIT collided with us and somewhere the peer sent the * cookie on another address besides the single address our assoc * had for him. In this case we will have one of the tie-tags set at * least AND the address field in the cookie can be used to look it * up. */ to = NULL; switch (cookie->addr_type) { #ifdef INET6 case SCTP_IPV6_ADDRESS: memset(&sin6, 0, sizeof(sin6)); sin6.sin6_family = AF_INET6; sin6.sin6_len = sizeof(sin6); sin6.sin6_port = sh->src_port; sin6.sin6_scope_id = cookie->scope_id; memcpy(&sin6.sin6_addr.s6_addr, cookie->address, sizeof(sin6.sin6_addr.s6_addr)); to = (struct sockaddr *)&sin6; break; #endif #ifdef INET case SCTP_IPV4_ADDRESS: memset(&sin, 0, sizeof(sin)); sin.sin_family = AF_INET; sin.sin_len = sizeof(sin); sin.sin_port = sh->src_port; sin.sin_addr.s_addr = cookie->address[0]; to = (struct sockaddr *)&sin; break; #endif default: /* This should not happen */ return (NULL); } if ((*stcb == NULL) && to) { /* Yep, lets check */ *stcb = sctp_findassociation_ep_addr(inp_p, to, netp, dst, NULL); if (*stcb == NULL) { /* * We should have only got back the same inp. If we * got back a different ep we have a problem. The * original findep got back l_inp and now */ if (l_inp != *inp_p) { SCTP_PRINTF("Bad problem find_ep got a diff inp then special_locate?\n"); } } else { if (*locked_tcb == NULL) { /* * In this case we found the assoc only * after we locked the create lock. This * means we are in a colliding case and we * must make sure that we unlock the tcb if * its one of the cases where we throw away * the incoming packets. */ *locked_tcb = *stcb; /* * We must also increment the inp ref count * since the ref_count flags was set when we * did not find the TCB, now we found it * which reduces the refcount.. we must * raise it back out to balance it all :-) */ SCTP_INP_INCR_REF((*stcb)->sctp_ep); if ((*stcb)->sctp_ep != l_inp) { SCTP_PRINTF("Huh? ep:%p diff then l_inp:%p?\n", (void *)(*stcb)->sctp_ep, (void *)l_inp); } } } } if (to == NULL) { return (NULL); } cookie_len -= SCTP_SIGNATURE_SIZE; if (*stcb == NULL) { /* this is the "normal" case... get a new TCB */ *stcb = sctp_process_cookie_new(m, iphlen, offset, src, dst, sh, cookie, cookie_len, *inp_p, netp, to, ¬ification, auth_skipped, auth_offset, auth_len, mflowtype, mflowid, vrf_id, port); } else { /* this is abnormal... cookie-echo on existing TCB */ had_a_existing_tcb = 1; *stcb = sctp_process_cookie_existing(m, iphlen, offset, src, dst, sh, cookie, cookie_len, *inp_p, *stcb, netp, to, ¬ification, auth_skipped, auth_offset, auth_len, mflowtype, mflowid, vrf_id, port); } if (*stcb == NULL) { /* still no TCB... must be bad cookie-echo */ return (NULL); } if ((*netp != NULL) && (mflowtype != M_HASHTYPE_NONE)) { (*netp)->flowtype = mflowtype; } /* * Ok, we built an association so confirm the address we sent the * INIT-ACK to. */ netl = sctp_findnet(*stcb, to); /* * This code should in theory NOT run but */ if (netl == NULL) { /* TSNH! Huh, why do I need to add this address here? */ if (sctp_add_remote_addr(*stcb, to, NULL, SCTP_DONOT_SETSCOPE, SCTP_IN_COOKIE_PROC)) { return (NULL); } netl = sctp_findnet(*stcb, to); } if (netl) { if (netl->dest_state & SCTP_ADDR_UNCONFIRMED) { netl->dest_state &= ~SCTP_ADDR_UNCONFIRMED; (void)sctp_set_primary_addr((*stcb), (struct sockaddr *)NULL, netl); send_int_conf = 1; } } sctp_start_net_timers(*stcb); if ((*inp_p)->sctp_flags & SCTP_PCB_FLAGS_TCPTYPE) { if (!had_a_existing_tcb || (((*inp_p)->sctp_flags & SCTP_PCB_FLAGS_CONNECTED) == 0)) { /* * If we have a NEW cookie or the connect never * reached the connected state during collision we * must do the TCP accept thing. */ struct socket *so, *oso; struct sctp_inpcb *inp; if (notification == SCTP_NOTIFY_ASSOC_RESTART) { /* * For a restart we will keep the same * socket, no need to do anything. I THINK!! */ sctp_ulp_notify(notification, *stcb, 0, NULL, SCTP_SO_NOT_LOCKED); if (send_int_conf) { sctp_ulp_notify(SCTP_NOTIFY_INTERFACE_CONFIRMED, (*stcb), 0, (void *)netl, SCTP_SO_NOT_LOCKED); } return (m); } oso = (*inp_p)->sctp_socket; atomic_add_int(&(*stcb)->asoc.refcnt, 1); SCTP_TCB_UNLOCK((*stcb)); CURVNET_SET(oso->so_vnet); so = sonewconn(oso, 0 ); CURVNET_RESTORE(); SCTP_TCB_LOCK((*stcb)); atomic_subtract_int(&(*stcb)->asoc.refcnt, 1); if (so == NULL) { struct mbuf *op_err; #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) struct socket *pcb_so; #endif /* Too many sockets */ SCTPDBG(SCTP_DEBUG_INPUT1, "process_cookie_new: no room for another socket!\n"); op_err = sctp_generate_cause(SCTP_CAUSE_OUT_OF_RESC, ""); sctp_abort_association(*inp_p, NULL, m, iphlen, src, dst, sh, op_err, mflowtype, mflowid, vrf_id, port); #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) pcb_so = SCTP_INP_SO(*inp_p); atomic_add_int(&(*stcb)->asoc.refcnt, 1); SCTP_TCB_UNLOCK((*stcb)); SCTP_SOCKET_LOCK(pcb_so, 1); SCTP_TCB_LOCK((*stcb)); atomic_subtract_int(&(*stcb)->asoc.refcnt, 1); #endif (void)sctp_free_assoc(*inp_p, *stcb, SCTP_NORMAL_PROC, SCTP_FROM_SCTP_INPUT + SCTP_LOC_20); #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) SCTP_SOCKET_UNLOCK(pcb_so, 1); #endif return (NULL); } inp = (struct sctp_inpcb *)so->so_pcb; SCTP_INP_INCR_REF(inp); /* * We add the unbound flag here so that if we get an * soabort() before we get the move_pcb done, we * will properly cleanup. */ inp->sctp_flags = (SCTP_PCB_FLAGS_TCPTYPE | SCTP_PCB_FLAGS_CONNECTED | SCTP_PCB_FLAGS_IN_TCPPOOL | SCTP_PCB_FLAGS_UNBOUND | (SCTP_PCB_COPY_FLAGS & (*inp_p)->sctp_flags) | SCTP_PCB_FLAGS_DONT_WAKE); inp->sctp_features = (*inp_p)->sctp_features; inp->sctp_mobility_features = (*inp_p)->sctp_mobility_features; inp->sctp_socket = so; inp->sctp_frag_point = (*inp_p)->sctp_frag_point; inp->sctp_cmt_on_off = (*inp_p)->sctp_cmt_on_off; inp->ecn_supported = (*inp_p)->ecn_supported; inp->prsctp_supported = (*inp_p)->prsctp_supported; inp->auth_supported = (*inp_p)->auth_supported; inp->asconf_supported = (*inp_p)->asconf_supported; inp->reconfig_supported = (*inp_p)->reconfig_supported; inp->nrsack_supported = (*inp_p)->nrsack_supported; inp->pktdrop_supported = (*inp_p)->pktdrop_supported; inp->partial_delivery_point = (*inp_p)->partial_delivery_point; inp->sctp_context = (*inp_p)->sctp_context; inp->local_strreset_support = (*inp_p)->local_strreset_support; inp->inp_starting_point_for_iterator = NULL; /* * copy in the authentication parameters from the * original endpoint */ if (inp->sctp_ep.local_hmacs) sctp_free_hmaclist(inp->sctp_ep.local_hmacs); inp->sctp_ep.local_hmacs = sctp_copy_hmaclist((*inp_p)->sctp_ep.local_hmacs); if (inp->sctp_ep.local_auth_chunks) sctp_free_chunklist(inp->sctp_ep.local_auth_chunks); inp->sctp_ep.local_auth_chunks = sctp_copy_chunklist((*inp_p)->sctp_ep.local_auth_chunks); /* * Now we must move it from one hash table to * another and get the tcb in the right place. */ /* * This is where the one-2-one socket is put into * the accept state waiting for the accept! */ if (*stcb) { (*stcb)->asoc.state |= SCTP_STATE_IN_ACCEPT_QUEUE; } sctp_move_pcb_and_assoc(*inp_p, inp, *stcb); atomic_add_int(&(*stcb)->asoc.refcnt, 1); SCTP_TCB_UNLOCK((*stcb)); sctp_pull_off_control_to_new_inp((*inp_p), inp, *stcb, 0); SCTP_TCB_LOCK((*stcb)); atomic_subtract_int(&(*stcb)->asoc.refcnt, 1); /* * now we must check to see if we were aborted while * the move was going on and the lock/unlock * happened. */ if (inp->sctp_flags & SCTP_PCB_FLAGS_SOCKET_GONE) { /* * yep it was, we leave the assoc attached * to the socket since the sctp_inpcb_free() * call will send an abort for us. */ SCTP_INP_DECR_REF(inp); return (NULL); } SCTP_INP_DECR_REF(inp); /* Switch over to the new guy */ *inp_p = inp; sctp_ulp_notify(notification, *stcb, 0, NULL, SCTP_SO_NOT_LOCKED); if (send_int_conf) { sctp_ulp_notify(SCTP_NOTIFY_INTERFACE_CONFIRMED, (*stcb), 0, (void *)netl, SCTP_SO_NOT_LOCKED); } /* * Pull it from the incomplete queue and wake the * guy */ #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) atomic_add_int(&(*stcb)->asoc.refcnt, 1); SCTP_TCB_UNLOCK((*stcb)); SCTP_SOCKET_LOCK(so, 1); #endif soisconnected(so); #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) SCTP_TCB_LOCK((*stcb)); atomic_subtract_int(&(*stcb)->asoc.refcnt, 1); SCTP_SOCKET_UNLOCK(so, 1); #endif return (m); } } if (notification) { sctp_ulp_notify(notification, *stcb, 0, NULL, SCTP_SO_NOT_LOCKED); } if (send_int_conf) { sctp_ulp_notify(SCTP_NOTIFY_INTERFACE_CONFIRMED, (*stcb), 0, (void *)netl, SCTP_SO_NOT_LOCKED); } return (m); } static void sctp_handle_cookie_ack(struct sctp_cookie_ack_chunk *cp SCTP_UNUSED, struct sctp_tcb *stcb, struct sctp_nets *net) { /* cp must not be used, others call this without a c-ack :-) */ struct sctp_association *asoc; SCTPDBG(SCTP_DEBUG_INPUT2, "sctp_handle_cookie_ack: handling COOKIE-ACK\n"); if ((stcb == NULL) || (net == NULL)) { return; } asoc = &stcb->asoc; sctp_stop_all_cookie_timers(stcb); /* process according to association state */ if (SCTP_GET_STATE(asoc) == SCTP_STATE_COOKIE_ECHOED) { /* state change only needed when I am in right state */ SCTPDBG(SCTP_DEBUG_INPUT2, "moving to OPEN state\n"); SCTP_SET_STATE(asoc, SCTP_STATE_OPEN); sctp_start_net_timers(stcb); if (asoc->state & SCTP_STATE_SHUTDOWN_PENDING) { sctp_timer_start(SCTP_TIMER_TYPE_SHUTDOWNGUARD, stcb->sctp_ep, stcb, asoc->primary_destination); } /* update RTO */ SCTP_STAT_INCR_COUNTER32(sctps_activeestab); SCTP_STAT_INCR_GAUGE32(sctps_currestab); if (asoc->overall_error_count == 0) { net->RTO = sctp_calculate_rto(stcb, asoc, net, &asoc->time_entered, sctp_align_safe_nocopy, SCTP_RTT_FROM_NON_DATA); } (void)SCTP_GETTIME_TIMEVAL(&asoc->time_entered); sctp_ulp_notify(SCTP_NOTIFY_ASSOC_UP, stcb, 0, NULL, SCTP_SO_NOT_LOCKED); if ((stcb->sctp_ep->sctp_flags & SCTP_PCB_FLAGS_TCPTYPE) || (stcb->sctp_ep->sctp_flags & SCTP_PCB_FLAGS_IN_TCPPOOL)) { #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) struct socket *so; #endif stcb->sctp_ep->sctp_flags |= SCTP_PCB_FLAGS_CONNECTED; #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) so = SCTP_INP_SO(stcb->sctp_ep); atomic_add_int(&stcb->asoc.refcnt, 1); SCTP_TCB_UNLOCK(stcb); SCTP_SOCKET_LOCK(so, 1); SCTP_TCB_LOCK(stcb); atomic_subtract_int(&stcb->asoc.refcnt, 1); #endif if ((stcb->asoc.state & SCTP_STATE_CLOSED_SOCKET) == 0) { soisconnected(stcb->sctp_socket); } #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) SCTP_SOCKET_UNLOCK(so, 1); #endif } /* * since we did not send a HB make sure we don't double * things */ net->hb_responded = 1; if (stcb->asoc.state & SCTP_STATE_CLOSED_SOCKET) { /* * We don't need to do the asconf thing, nor hb or * autoclose if the socket is closed. */ goto closed_socket; } sctp_timer_start(SCTP_TIMER_TYPE_HEARTBEAT, stcb->sctp_ep, stcb, net); if (stcb->asoc.sctp_autoclose_ticks && sctp_is_feature_on(stcb->sctp_ep, SCTP_PCB_FLAGS_AUTOCLOSE)) { sctp_timer_start(SCTP_TIMER_TYPE_AUTOCLOSE, stcb->sctp_ep, stcb, NULL); } /* * send ASCONF if parameters are pending and ASCONFs are * allowed (eg. addresses changed when init/cookie echo were * in flight) */ if ((sctp_is_feature_on(stcb->sctp_ep, SCTP_PCB_FLAGS_DO_ASCONF)) && (stcb->asoc.asconf_supported == 1) && (!TAILQ_EMPTY(&stcb->asoc.asconf_queue))) { #ifdef SCTP_TIMER_BASED_ASCONF sctp_timer_start(SCTP_TIMER_TYPE_ASCONF, stcb->sctp_ep, stcb, stcb->asoc.primary_destination); #else sctp_send_asconf(stcb, stcb->asoc.primary_destination, SCTP_ADDR_NOT_LOCKED); #endif } } closed_socket: /* Toss the cookie if I can */ sctp_toss_old_cookies(stcb, asoc); if (!TAILQ_EMPTY(&asoc->sent_queue)) { /* Restart the timer if we have pending data */ struct sctp_tmit_chunk *chk; chk = TAILQ_FIRST(&asoc->sent_queue); sctp_timer_start(SCTP_TIMER_TYPE_SEND, stcb->sctp_ep, stcb, chk->whoTo); } } static void sctp_handle_ecn_echo(struct sctp_ecne_chunk *cp, struct sctp_tcb *stcb) { struct sctp_nets *net; struct sctp_tmit_chunk *lchk; struct sctp_ecne_chunk bkup; uint8_t override_bit; uint32_t tsn, window_data_tsn; int len; unsigned int pkt_cnt; len = ntohs(cp->ch.chunk_length); if ((len != sizeof(struct sctp_ecne_chunk)) && (len != sizeof(struct old_sctp_ecne_chunk))) { return; } if (len == sizeof(struct old_sctp_ecne_chunk)) { /* Its the old format */ memcpy(&bkup, cp, sizeof(struct old_sctp_ecne_chunk)); bkup.num_pkts_since_cwr = htonl(1); cp = &bkup; } SCTP_STAT_INCR(sctps_recvecne); tsn = ntohl(cp->tsn); pkt_cnt = ntohl(cp->num_pkts_since_cwr); lchk = TAILQ_LAST(&stcb->asoc.send_queue, sctpchunk_listhead); if (lchk == NULL) { window_data_tsn = stcb->asoc.sending_seq - 1; } else { window_data_tsn = lchk->rec.data.TSN_seq; } /* Find where it was sent to if possible. */ net = NULL; TAILQ_FOREACH(lchk, &stcb->asoc.sent_queue, sctp_next) { if (lchk->rec.data.TSN_seq == tsn) { net = lchk->whoTo; net->ecn_prev_cwnd = lchk->rec.data.cwnd_at_send; break; } if (SCTP_TSN_GT(lchk->rec.data.TSN_seq, tsn)) { break; } } if (net == NULL) { /* * What to do. A previous send of a CWR was possibly lost. * See how old it is, we may have it marked on the actual * net. */ TAILQ_FOREACH(net, &stcb->asoc.nets, sctp_next) { if (tsn == net->last_cwr_tsn) { /* Found him, send it off */ break; } } if (net == NULL) { /* * If we reach here, we need to send a special CWR * that says hey, we did this a long time ago and * you lost the response. */ net = TAILQ_FIRST(&stcb->asoc.nets); if (net == NULL) { /* TSNH */ return; } override_bit = SCTP_CWR_REDUCE_OVERRIDE; } else { override_bit = 0; } } else { override_bit = 0; } if (SCTP_TSN_GT(tsn, net->cwr_window_tsn) && ((override_bit & SCTP_CWR_REDUCE_OVERRIDE) == 0)) { /* * JRS - Use the congestion control given in the pluggable * CC module */ stcb->asoc.cc_functions.sctp_cwnd_update_after_ecn_echo(stcb, net, 0, pkt_cnt); /* * We reduce once every RTT. So we will only lower cwnd at * the next sending seq i.e. the window_data_tsn */ net->cwr_window_tsn = window_data_tsn; net->ecn_ce_pkt_cnt += pkt_cnt; net->lost_cnt = pkt_cnt; net->last_cwr_tsn = tsn; } else { override_bit |= SCTP_CWR_IN_SAME_WINDOW; if (SCTP_TSN_GT(tsn, net->last_cwr_tsn) && ((override_bit & SCTP_CWR_REDUCE_OVERRIDE) == 0)) { /* * Another loss in the same window update how many * marks/packets lost we have had. */ int cnt = 1; if (pkt_cnt > net->lost_cnt) { /* Should be the case */ cnt = (pkt_cnt - net->lost_cnt); net->ecn_ce_pkt_cnt += cnt; } net->lost_cnt = pkt_cnt; net->last_cwr_tsn = tsn; /* * Most CC functions will ignore this call, since we * are in-window yet of the initial CE the peer saw. */ stcb->asoc.cc_functions.sctp_cwnd_update_after_ecn_echo(stcb, net, 1, cnt); } } /* * We always send a CWR this way if our previous one was lost our * peer will get an update, or if it is not time again to reduce we * still get the cwr to the peer. Note we set the override when we * could not find the TSN on the chunk or the destination network. */ sctp_send_cwr(stcb, net, net->last_cwr_tsn, override_bit); } static void sctp_handle_ecn_cwr(struct sctp_cwr_chunk *cp, struct sctp_tcb *stcb, struct sctp_nets *net) { /* * Here we get a CWR from the peer. We must look in the outqueue and * make sure that we have a covered ECNE in the control chunk part. * If so remove it. */ struct sctp_tmit_chunk *chk; struct sctp_ecne_chunk *ecne; int override; uint32_t cwr_tsn; cwr_tsn = ntohl(cp->tsn); override = cp->ch.chunk_flags & SCTP_CWR_REDUCE_OVERRIDE; TAILQ_FOREACH(chk, &stcb->asoc.control_send_queue, sctp_next) { if (chk->rec.chunk_id.id != SCTP_ECN_ECHO) { continue; } if ((override == 0) && (chk->whoTo != net)) { /* Must be from the right src unless override is set */ continue; } ecne = mtod(chk->data, struct sctp_ecne_chunk *); if (SCTP_TSN_GE(cwr_tsn, ntohl(ecne->tsn))) { /* this covers this ECNE, we can remove it */ stcb->asoc.ecn_echo_cnt_onq--; TAILQ_REMOVE(&stcb->asoc.control_send_queue, chk, sctp_next); if (chk->data) { sctp_m_freem(chk->data); chk->data = NULL; } stcb->asoc.ctrl_queue_cnt--; sctp_free_a_chunk(stcb, chk, SCTP_SO_NOT_LOCKED); if (override == 0) { break; } } } } static void sctp_handle_shutdown_complete(struct sctp_shutdown_complete_chunk *cp SCTP_UNUSED, struct sctp_tcb *stcb, struct sctp_nets *net) { struct sctp_association *asoc; #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) struct socket *so; #endif SCTPDBG(SCTP_DEBUG_INPUT2, "sctp_handle_shutdown_complete: handling SHUTDOWN-COMPLETE\n"); if (stcb == NULL) return; asoc = &stcb->asoc; /* process according to association state */ if (SCTP_GET_STATE(asoc) != SCTP_STATE_SHUTDOWN_ACK_SENT) { /* unexpected SHUTDOWN-COMPLETE... so ignore... */ SCTPDBG(SCTP_DEBUG_INPUT2, "sctp_handle_shutdown_complete: not in SCTP_STATE_SHUTDOWN_ACK_SENT --- ignore\n"); SCTP_TCB_UNLOCK(stcb); return; } /* notify upper layer protocol */ if (stcb->sctp_socket) { sctp_ulp_notify(SCTP_NOTIFY_ASSOC_DOWN, stcb, 0, NULL, SCTP_SO_NOT_LOCKED); } #ifdef INVARIANTS if (!TAILQ_EMPTY(&asoc->send_queue) || !TAILQ_EMPTY(&asoc->sent_queue) || !stcb->asoc.ss_functions.sctp_ss_is_empty(stcb, asoc)) { panic("Queues are not empty when handling SHUTDOWN-COMPLETE"); } #endif /* stop the timer */ sctp_timer_stop(SCTP_TIMER_TYPE_SHUTDOWNACK, stcb->sctp_ep, stcb, net, SCTP_FROM_SCTP_INPUT + SCTP_LOC_22); SCTP_STAT_INCR_COUNTER32(sctps_shutdown); /* free the TCB */ SCTPDBG(SCTP_DEBUG_INPUT2, "sctp_handle_shutdown_complete: calls free-asoc\n"); #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) so = SCTP_INP_SO(stcb->sctp_ep); atomic_add_int(&stcb->asoc.refcnt, 1); SCTP_TCB_UNLOCK(stcb); SCTP_SOCKET_LOCK(so, 1); SCTP_TCB_LOCK(stcb); atomic_subtract_int(&stcb->asoc.refcnt, 1); #endif (void)sctp_free_assoc(stcb->sctp_ep, stcb, SCTP_NORMAL_PROC, SCTP_FROM_SCTP_INPUT + SCTP_LOC_23); #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) SCTP_SOCKET_UNLOCK(so, 1); #endif return; } static int process_chunk_drop(struct sctp_tcb *stcb, struct sctp_chunk_desc *desc, struct sctp_nets *net, uint8_t flg) { switch (desc->chunk_type) { case SCTP_DATA: /* find the tsn to resend (possibly */ { uint32_t tsn; struct sctp_tmit_chunk *tp1; tsn = ntohl(desc->tsn_ifany); TAILQ_FOREACH(tp1, &stcb->asoc.sent_queue, sctp_next) { if (tp1->rec.data.TSN_seq == tsn) { /* found it */ break; } if (SCTP_TSN_GT(tp1->rec.data.TSN_seq, tsn)) { /* not found */ tp1 = NULL; break; } } if (tp1 == NULL) { /* * Do it the other way , aka without paying * attention to queue seq order. */ SCTP_STAT_INCR(sctps_pdrpdnfnd); TAILQ_FOREACH(tp1, &stcb->asoc.sent_queue, sctp_next) { if (tp1->rec.data.TSN_seq == tsn) { /* found it */ break; } } } if (tp1 == NULL) { SCTP_STAT_INCR(sctps_pdrptsnnf); } if ((tp1) && (tp1->sent < SCTP_DATAGRAM_ACKED)) { uint8_t *ddp; if (((flg & SCTP_BADCRC) == 0) && ((flg & SCTP_FROM_MIDDLE_BOX) == 0)) { return (0); } if ((stcb->asoc.peers_rwnd == 0) && ((flg & SCTP_FROM_MIDDLE_BOX) == 0)) { SCTP_STAT_INCR(sctps_pdrpdiwnp); return (0); } if (stcb->asoc.peers_rwnd == 0 && (flg & SCTP_FROM_MIDDLE_BOX)) { SCTP_STAT_INCR(sctps_pdrpdizrw); return (0); } ddp = (uint8_t *) (mtod(tp1->data, caddr_t)+ sizeof(struct sctp_data_chunk)); { unsigned int iii; for (iii = 0; iii < sizeof(desc->data_bytes); iii++) { if (ddp[iii] != desc->data_bytes[iii]) { SCTP_STAT_INCR(sctps_pdrpbadd); return (-1); } } } if (tp1->do_rtt) { /* * this guy had a RTO calculation * pending on it, cancel it */ if (tp1->whoTo->rto_needed == 0) { tp1->whoTo->rto_needed = 1; } tp1->do_rtt = 0; } SCTP_STAT_INCR(sctps_pdrpmark); if (tp1->sent != SCTP_DATAGRAM_RESEND) sctp_ucount_incr(stcb->asoc.sent_queue_retran_cnt); /* * mark it as if we were doing a FR, since * we will be getting gap ack reports behind * the info from the router. */ tp1->rec.data.doing_fast_retransmit = 1; /* * mark the tsn with what sequences can * cause a new FR. */ if (TAILQ_EMPTY(&stcb->asoc.send_queue)) { tp1->rec.data.fast_retran_tsn = stcb->asoc.sending_seq; } else { tp1->rec.data.fast_retran_tsn = (TAILQ_FIRST(&stcb->asoc.send_queue))->rec.data.TSN_seq; } /* restart the timer */ sctp_timer_stop(SCTP_TIMER_TYPE_SEND, stcb->sctp_ep, stcb, tp1->whoTo, SCTP_FROM_SCTP_INPUT + SCTP_LOC_24); sctp_timer_start(SCTP_TIMER_TYPE_SEND, stcb->sctp_ep, stcb, tp1->whoTo); /* fix counts and things */ if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_FLIGHT_LOGGING_ENABLE) { sctp_misc_ints(SCTP_FLIGHT_LOG_DOWN_PDRP, tp1->whoTo->flight_size, tp1->book_size, (uintptr_t) stcb, tp1->rec.data.TSN_seq); } if (tp1->sent < SCTP_DATAGRAM_RESEND) { sctp_flight_size_decrease(tp1); sctp_total_flight_decrease(stcb, tp1); } tp1->sent = SCTP_DATAGRAM_RESEND; } { /* audit code */ unsigned int audit; audit = 0; TAILQ_FOREACH(tp1, &stcb->asoc.sent_queue, sctp_next) { if (tp1->sent == SCTP_DATAGRAM_RESEND) audit++; } TAILQ_FOREACH(tp1, &stcb->asoc.control_send_queue, sctp_next) { if (tp1->sent == SCTP_DATAGRAM_RESEND) audit++; } if (audit != stcb->asoc.sent_queue_retran_cnt) { SCTP_PRINTF("**Local Audit finds cnt:%d asoc cnt:%d\n", audit, stcb->asoc.sent_queue_retran_cnt); #ifndef SCTP_AUDITING_ENABLED stcb->asoc.sent_queue_retran_cnt = audit; #endif } } } break; case SCTP_ASCONF: { struct sctp_tmit_chunk *asconf; TAILQ_FOREACH(asconf, &stcb->asoc.control_send_queue, sctp_next) { if (asconf->rec.chunk_id.id == SCTP_ASCONF) { break; } } if (asconf) { if (asconf->sent != SCTP_DATAGRAM_RESEND) sctp_ucount_incr(stcb->asoc.sent_queue_retran_cnt); asconf->sent = SCTP_DATAGRAM_RESEND; asconf->snd_count--; } } break; case SCTP_INITIATION: /* resend the INIT */ stcb->asoc.dropped_special_cnt++; if (stcb->asoc.dropped_special_cnt < SCTP_RETRY_DROPPED_THRESH) { /* * If we can get it in, in a few attempts we do * this, otherwise we let the timer fire. */ sctp_timer_stop(SCTP_TIMER_TYPE_INIT, stcb->sctp_ep, stcb, net, SCTP_FROM_SCTP_INPUT + SCTP_LOC_25); sctp_send_initiate(stcb->sctp_ep, stcb, SCTP_SO_NOT_LOCKED); } break; case SCTP_SELECTIVE_ACK: case SCTP_NR_SELECTIVE_ACK: /* resend the sack */ sctp_send_sack(stcb, SCTP_SO_NOT_LOCKED); break; case SCTP_HEARTBEAT_REQUEST: /* resend a demand HB */ if ((stcb->asoc.overall_error_count + 3) < stcb->asoc.max_send_times) { /* * Only retransmit if we KNOW we wont destroy the * tcb */ sctp_send_hb(stcb, net, SCTP_SO_NOT_LOCKED); } break; case SCTP_SHUTDOWN: sctp_send_shutdown(stcb, net); break; case SCTP_SHUTDOWN_ACK: sctp_send_shutdown_ack(stcb, net); break; case SCTP_COOKIE_ECHO: { struct sctp_tmit_chunk *cookie; cookie = NULL; TAILQ_FOREACH(cookie, &stcb->asoc.control_send_queue, sctp_next) { if (cookie->rec.chunk_id.id == SCTP_COOKIE_ECHO) { break; } } if (cookie) { if (cookie->sent != SCTP_DATAGRAM_RESEND) sctp_ucount_incr(stcb->asoc.sent_queue_retran_cnt); cookie->sent = SCTP_DATAGRAM_RESEND; sctp_stop_all_cookie_timers(stcb); } } break; case SCTP_COOKIE_ACK: sctp_send_cookie_ack(stcb); break; case SCTP_ASCONF_ACK: /* resend last asconf ack */ sctp_send_asconf_ack(stcb); break; case SCTP_FORWARD_CUM_TSN: send_forward_tsn(stcb, &stcb->asoc); break; /* can't do anything with these */ case SCTP_PACKET_DROPPED: case SCTP_INITIATION_ACK: /* this should not happen */ case SCTP_HEARTBEAT_ACK: case SCTP_ABORT_ASSOCIATION: case SCTP_OPERATION_ERROR: case SCTP_SHUTDOWN_COMPLETE: case SCTP_ECN_ECHO: case SCTP_ECN_CWR: default: break; } return (0); } void sctp_reset_in_stream(struct sctp_tcb *stcb, uint32_t number_entries, uint16_t * list) { uint32_t i; uint16_t temp; /* * We set things to 0xffff since this is the last delivered sequence * and we will be sending in 0 after the reset. */ if (number_entries) { for (i = 0; i < number_entries; i++) { temp = ntohs(list[i]); if (temp >= stcb->asoc.streamincnt) { continue; } stcb->asoc.strmin[temp].last_sequence_delivered = 0xffff; } } else { list = NULL; for (i = 0; i < stcb->asoc.streamincnt; i++) { stcb->asoc.strmin[i].last_sequence_delivered = 0xffff; } } sctp_ulp_notify(SCTP_NOTIFY_STR_RESET_RECV, stcb, number_entries, (void *)list, SCTP_SO_NOT_LOCKED); } static void sctp_reset_out_streams(struct sctp_tcb *stcb, uint32_t number_entries, uint16_t * list) { uint32_t i; uint16_t temp; if (number_entries > 0) { for (i = 0; i < number_entries; i++) { temp = ntohs(list[i]); if (temp >= stcb->asoc.streamoutcnt) { /* no such stream */ continue; } stcb->asoc.strmout[temp].next_sequence_send = 0; } } else { for (i = 0; i < stcb->asoc.streamoutcnt; i++) { stcb->asoc.strmout[i].next_sequence_send = 0; } } sctp_ulp_notify(SCTP_NOTIFY_STR_RESET_SEND, stcb, number_entries, (void *)list, SCTP_SO_NOT_LOCKED); } struct sctp_stream_reset_request * sctp_find_stream_reset(struct sctp_tcb *stcb, uint32_t seq, struct sctp_tmit_chunk **bchk) { struct sctp_association *asoc; struct sctp_chunkhdr *ch; struct sctp_stream_reset_request *r; struct sctp_tmit_chunk *chk; int len, clen; asoc = &stcb->asoc; if (TAILQ_EMPTY(&stcb->asoc.control_send_queue)) { asoc->stream_reset_outstanding = 0; return (NULL); } if (stcb->asoc.str_reset == NULL) { asoc->stream_reset_outstanding = 0; return (NULL); } chk = stcb->asoc.str_reset; if (chk->data == NULL) { return (NULL); } if (bchk) { /* he wants a copy of the chk pointer */ *bchk = chk; } clen = chk->send_size; ch = mtod(chk->data, struct sctp_chunkhdr *); r = (struct sctp_stream_reset_request *)(ch + 1); if (ntohl(r->request_seq) == seq) { /* found it */ return (r); } len = SCTP_SIZE32(ntohs(r->ph.param_length)); if (clen > (len + (int)sizeof(struct sctp_chunkhdr))) { /* move to the next one, there can only be a max of two */ r = (struct sctp_stream_reset_request *)((caddr_t)r + len); if (ntohl(r->request_seq) == seq) { return (r); } } /* that seq is not here */ return (NULL); } static void sctp_clean_up_stream_reset(struct sctp_tcb *stcb) { struct sctp_association *asoc; struct sctp_tmit_chunk *chk = stcb->asoc.str_reset; if (stcb->asoc.str_reset == NULL) { return; } asoc = &stcb->asoc; sctp_timer_stop(SCTP_TIMER_TYPE_STRRESET, stcb->sctp_ep, stcb, chk->whoTo, SCTP_FROM_SCTP_INPUT + SCTP_LOC_26); TAILQ_REMOVE(&asoc->control_send_queue, chk, sctp_next); if (chk->data) { sctp_m_freem(chk->data); chk->data = NULL; } asoc->ctrl_queue_cnt--; sctp_free_a_chunk(stcb, chk, SCTP_SO_NOT_LOCKED); /* sa_ignore NO_NULL_CHK */ stcb->asoc.str_reset = NULL; } static int sctp_handle_stream_reset_response(struct sctp_tcb *stcb, uint32_t seq, uint32_t action, struct sctp_stream_reset_response *respin) { uint16_t type; int lparm_len; struct sctp_association *asoc = &stcb->asoc; struct sctp_tmit_chunk *chk; struct sctp_stream_reset_request *req_param; struct sctp_stream_reset_out_request *req_out_param; struct sctp_stream_reset_in_request *req_in_param; uint32_t number_entries; if (asoc->stream_reset_outstanding == 0) { /* duplicate */ return (0); } if (seq == stcb->asoc.str_reset_seq_out) { req_param = sctp_find_stream_reset(stcb, seq, &chk); if (req_param != NULL) { stcb->asoc.str_reset_seq_out++; type = ntohs(req_param->ph.param_type); lparm_len = ntohs(req_param->ph.param_length); if (type == SCTP_STR_RESET_OUT_REQUEST) { req_out_param = (struct sctp_stream_reset_out_request *)req_param; number_entries = (lparm_len - sizeof(struct sctp_stream_reset_out_request)) / sizeof(uint16_t); asoc->stream_reset_out_is_outstanding = 0; if (asoc->stream_reset_outstanding) asoc->stream_reset_outstanding--; if (action == SCTP_STREAM_RESET_RESULT_PERFORMED) { /* do it */ sctp_reset_out_streams(stcb, number_entries, req_out_param->list_of_streams); } else if (action == SCTP_STREAM_RESET_RESULT_DENIED) { sctp_ulp_notify(SCTP_NOTIFY_STR_RESET_DENIED_OUT, stcb, number_entries, req_out_param->list_of_streams, SCTP_SO_NOT_LOCKED); } else { sctp_ulp_notify(SCTP_NOTIFY_STR_RESET_FAILED_OUT, stcb, number_entries, req_out_param->list_of_streams, SCTP_SO_NOT_LOCKED); } } else if (type == SCTP_STR_RESET_IN_REQUEST) { req_in_param = (struct sctp_stream_reset_in_request *)req_param; number_entries = (lparm_len - sizeof(struct sctp_stream_reset_in_request)) / sizeof(uint16_t); if (asoc->stream_reset_outstanding) asoc->stream_reset_outstanding--; if (action == SCTP_STREAM_RESET_RESULT_DENIED) { sctp_ulp_notify(SCTP_NOTIFY_STR_RESET_DENIED_IN, stcb, number_entries, req_in_param->list_of_streams, SCTP_SO_NOT_LOCKED); } else if (action != SCTP_STREAM_RESET_RESULT_PERFORMED) { sctp_ulp_notify(SCTP_NOTIFY_STR_RESET_FAILED_IN, stcb, number_entries, req_in_param->list_of_streams, SCTP_SO_NOT_LOCKED); } } else if (type == SCTP_STR_RESET_ADD_OUT_STREAMS) { /* Ok we now may have more streams */ int num_stream; num_stream = stcb->asoc.strm_pending_add_size; if (num_stream > (stcb->asoc.strm_realoutsize - stcb->asoc.streamoutcnt)) { /* TSNH */ num_stream = stcb->asoc.strm_realoutsize - stcb->asoc.streamoutcnt; } stcb->asoc.strm_pending_add_size = 0; if (asoc->stream_reset_outstanding) asoc->stream_reset_outstanding--; if (action == SCTP_STREAM_RESET_RESULT_PERFORMED) { /* Put the new streams into effect */ stcb->asoc.streamoutcnt += num_stream; sctp_notify_stream_reset_add(stcb, stcb->asoc.streamincnt, stcb->asoc.streamoutcnt, 0); } else if (action == SCTP_STREAM_RESET_RESULT_DENIED) { sctp_notify_stream_reset_add(stcb, stcb->asoc.streamincnt, stcb->asoc.streamoutcnt, SCTP_STREAM_CHANGE_DENIED); } else { sctp_notify_stream_reset_add(stcb, stcb->asoc.streamincnt, stcb->asoc.streamoutcnt, SCTP_STREAM_CHANGE_FAILED); } } else if (type == SCTP_STR_RESET_ADD_IN_STREAMS) { if (asoc->stream_reset_outstanding) asoc->stream_reset_outstanding--; if (action == SCTP_STREAM_RESET_RESULT_DENIED) { sctp_notify_stream_reset_add(stcb, stcb->asoc.streamincnt, stcb->asoc.streamoutcnt, SCTP_STREAM_CHANGE_DENIED); } else if (action != SCTP_STREAM_RESET_RESULT_PERFORMED) { sctp_notify_stream_reset_add(stcb, stcb->asoc.streamincnt, stcb->asoc.streamoutcnt, SCTP_STREAM_CHANGE_FAILED); } } else if (type == SCTP_STR_RESET_TSN_REQUEST) { /** * a) Adopt the new in tsn. * b) reset the map * c) Adopt the new out-tsn */ struct sctp_stream_reset_response_tsn *resp; struct sctp_forward_tsn_chunk fwdtsn; int abort_flag = 0; if (respin == NULL) { /* huh ? */ return (0); } if (action == SCTP_STREAM_RESET_RESULT_PERFORMED) { resp = (struct sctp_stream_reset_response_tsn *)respin; asoc->stream_reset_outstanding--; fwdtsn.ch.chunk_length = htons(sizeof(struct sctp_forward_tsn_chunk)); fwdtsn.ch.chunk_type = SCTP_FORWARD_CUM_TSN; fwdtsn.new_cumulative_tsn = htonl(ntohl(resp->senders_next_tsn) - 1); sctp_handle_forward_tsn(stcb, &fwdtsn, &abort_flag, NULL, 0); if (abort_flag) { return (1); } stcb->asoc.highest_tsn_inside_map = (ntohl(resp->senders_next_tsn) - 1); if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_MAP_LOGGING_ENABLE) { sctp_log_map(0, 7, asoc->highest_tsn_inside_map, SCTP_MAP_SLIDE_RESULT); } stcb->asoc.tsn_last_delivered = stcb->asoc.cumulative_tsn = stcb->asoc.highest_tsn_inside_map; stcb->asoc.mapping_array_base_tsn = ntohl(resp->senders_next_tsn); memset(stcb->asoc.mapping_array, 0, stcb->asoc.mapping_array_size); stcb->asoc.highest_tsn_inside_nr_map = stcb->asoc.highest_tsn_inside_map; memset(stcb->asoc.nr_mapping_array, 0, stcb->asoc.mapping_array_size); stcb->asoc.sending_seq = ntohl(resp->receivers_next_tsn); stcb->asoc.last_acked_seq = stcb->asoc.cumulative_tsn; sctp_reset_out_streams(stcb, 0, (uint16_t *) NULL); sctp_reset_in_stream(stcb, 0, (uint16_t *) NULL); sctp_notify_stream_reset_tsn(stcb, stcb->asoc.sending_seq, (stcb->asoc.mapping_array_base_tsn + 1), 0); } else if (action == SCTP_STREAM_RESET_RESULT_DENIED) { sctp_notify_stream_reset_tsn(stcb, stcb->asoc.sending_seq, (stcb->asoc.mapping_array_base_tsn + 1), SCTP_ASSOC_RESET_DENIED); } else { sctp_notify_stream_reset_tsn(stcb, stcb->asoc.sending_seq, (stcb->asoc.mapping_array_base_tsn + 1), SCTP_ASSOC_RESET_FAILED); } } /* get rid of the request and get the request flags */ if (asoc->stream_reset_outstanding == 0) { sctp_clean_up_stream_reset(stcb); } } } return (0); } static void sctp_handle_str_reset_request_in(struct sctp_tcb *stcb, struct sctp_tmit_chunk *chk, struct sctp_stream_reset_in_request *req, int trunc) { uint32_t seq; int len, i; int number_entries; uint16_t temp; /* * peer wants me to send a str-reset to him for my outgoing seq's if * seq_in is right. */ struct sctp_association *asoc = &stcb->asoc; seq = ntohl(req->request_seq); if (asoc->str_reset_seq_in == seq) { asoc->last_reset_action[1] = asoc->last_reset_action[0]; if (!(asoc->local_strreset_support & SCTP_ENABLE_RESET_STREAM_REQ)) { asoc->last_reset_action[0] = SCTP_STREAM_RESET_RESULT_DENIED; } else if (trunc) { /* Can't do it, since they exceeded our buffer size */ asoc->last_reset_action[0] = SCTP_STREAM_RESET_RESULT_DENIED; } else if (stcb->asoc.stream_reset_out_is_outstanding == 0) { len = ntohs(req->ph.param_length); number_entries = ((len - sizeof(struct sctp_stream_reset_in_request)) / sizeof(uint16_t)); for (i = 0; i < number_entries; i++) { temp = ntohs(req->list_of_streams[i]); req->list_of_streams[i] = temp; } asoc->last_reset_action[0] = SCTP_STREAM_RESET_RESULT_PERFORMED; sctp_add_stream_reset_out(chk, number_entries, req->list_of_streams, asoc->str_reset_seq_out, seq, (asoc->sending_seq - 1)); asoc->stream_reset_out_is_outstanding = 1; asoc->str_reset = chk; sctp_timer_start(SCTP_TIMER_TYPE_STRRESET, stcb->sctp_ep, stcb, chk->whoTo); stcb->asoc.stream_reset_outstanding++; } else { /* Can't do it, since we have sent one out */ asoc->last_reset_action[0] = SCTP_STREAM_RESET_RESULT_ERR_IN_PROGRESS; } sctp_add_stream_reset_result(chk, seq, asoc->last_reset_action[0]); asoc->str_reset_seq_in++; } else if (asoc->str_reset_seq_in - 1 == seq) { sctp_add_stream_reset_result(chk, seq, asoc->last_reset_action[0]); } else if (asoc->str_reset_seq_in - 2 == seq) { sctp_add_stream_reset_result(chk, seq, asoc->last_reset_action[1]); } else { sctp_add_stream_reset_result(chk, seq, SCTP_STREAM_RESET_RESULT_ERR_BAD_SEQNO); } } static int sctp_handle_str_reset_request_tsn(struct sctp_tcb *stcb, struct sctp_tmit_chunk *chk, struct sctp_stream_reset_tsn_request *req) { /* reset all in and out and update the tsn */ /* * A) reset my str-seq's on in and out. B) Select a receive next, * and set cum-ack to it. Also process this selected number as a * fwd-tsn as well. C) set in the response my next sending seq. */ struct sctp_forward_tsn_chunk fwdtsn; struct sctp_association *asoc = &stcb->asoc; int abort_flag = 0; uint32_t seq; seq = ntohl(req->request_seq); if (asoc->str_reset_seq_in == seq) { asoc->last_reset_action[1] = stcb->asoc.last_reset_action[0]; if (!(asoc->local_strreset_support & SCTP_ENABLE_CHANGE_ASSOC_REQ)) { asoc->last_reset_action[0] = SCTP_STREAM_RESET_RESULT_DENIED; } else { fwdtsn.ch.chunk_length = htons(sizeof(struct sctp_forward_tsn_chunk)); fwdtsn.ch.chunk_type = SCTP_FORWARD_CUM_TSN; fwdtsn.ch.chunk_flags = 0; fwdtsn.new_cumulative_tsn = htonl(stcb->asoc.highest_tsn_inside_map + 1); sctp_handle_forward_tsn(stcb, &fwdtsn, &abort_flag, NULL, 0); if (abort_flag) { return (1); } asoc->highest_tsn_inside_map += SCTP_STREAM_RESET_TSN_DELTA; if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_MAP_LOGGING_ENABLE) { sctp_log_map(0, 10, asoc->highest_tsn_inside_map, SCTP_MAP_SLIDE_RESULT); } asoc->tsn_last_delivered = asoc->cumulative_tsn = asoc->highest_tsn_inside_map; asoc->mapping_array_base_tsn = asoc->highest_tsn_inside_map + 1; memset(asoc->mapping_array, 0, asoc->mapping_array_size); asoc->highest_tsn_inside_nr_map = asoc->highest_tsn_inside_map; memset(asoc->nr_mapping_array, 0, asoc->mapping_array_size); atomic_add_int(&asoc->sending_seq, 1); /* save off historical data for retrans */ asoc->last_sending_seq[1] = asoc->last_sending_seq[0]; asoc->last_sending_seq[0] = asoc->sending_seq; asoc->last_base_tsnsent[1] = asoc->last_base_tsnsent[0]; asoc->last_base_tsnsent[0] = asoc->mapping_array_base_tsn; sctp_reset_out_streams(stcb, 0, (uint16_t *) NULL); sctp_reset_in_stream(stcb, 0, (uint16_t *) NULL); asoc->last_reset_action[0] = SCTP_STREAM_RESET_RESULT_PERFORMED; sctp_notify_stream_reset_tsn(stcb, asoc->sending_seq, (asoc->mapping_array_base_tsn + 1), 0); } sctp_add_stream_reset_result_tsn(chk, seq, asoc->last_reset_action[0], asoc->last_sending_seq[0], asoc->last_base_tsnsent[0]); asoc->str_reset_seq_in++; } else if (asoc->str_reset_seq_in - 1 == seq) { sctp_add_stream_reset_result_tsn(chk, seq, asoc->last_reset_action[0], asoc->last_sending_seq[0], asoc->last_base_tsnsent[0]); } else if (asoc->str_reset_seq_in - 2 == seq) { sctp_add_stream_reset_result_tsn(chk, seq, asoc->last_reset_action[1], asoc->last_sending_seq[1], asoc->last_base_tsnsent[1]); } else { sctp_add_stream_reset_result(chk, seq, SCTP_STREAM_RESET_RESULT_ERR_BAD_SEQNO); } return (0); } static void sctp_handle_str_reset_request_out(struct sctp_tcb *stcb, struct sctp_tmit_chunk *chk, struct sctp_stream_reset_out_request *req, int trunc) { uint32_t seq, tsn; int number_entries, len; struct sctp_association *asoc = &stcb->asoc; seq = ntohl(req->request_seq); /* now if its not a duplicate we process it */ if (asoc->str_reset_seq_in == seq) { len = ntohs(req->ph.param_length); number_entries = ((len - sizeof(struct sctp_stream_reset_out_request)) / sizeof(uint16_t)); /* * the sender is resetting, handle the list issue.. we must * a) verify if we can do the reset, if so no problem b) If * we can't do the reset we must copy the request. c) queue * it, and setup the data in processor to trigger it off * when needed and dequeue all the queued data. */ tsn = ntohl(req->send_reset_at_tsn); /* move the reset action back one */ asoc->last_reset_action[1] = asoc->last_reset_action[0]; if (!(asoc->local_strreset_support & SCTP_ENABLE_RESET_STREAM_REQ)) { asoc->last_reset_action[0] = SCTP_STREAM_RESET_RESULT_DENIED; } else if (trunc) { asoc->last_reset_action[0] = SCTP_STREAM_RESET_RESULT_DENIED; } else if (SCTP_TSN_GE(asoc->cumulative_tsn, tsn)) { /* we can do it now */ sctp_reset_in_stream(stcb, number_entries, req->list_of_streams); asoc->last_reset_action[0] = SCTP_STREAM_RESET_RESULT_PERFORMED; } else { /* * we must queue it up and thus wait for the TSN's * to arrive that are at or before tsn */ struct sctp_stream_reset_list *liste; int siz; siz = sizeof(struct sctp_stream_reset_list) + (number_entries * sizeof(uint16_t)); SCTP_MALLOC(liste, struct sctp_stream_reset_list *, siz, SCTP_M_STRESET); if (liste == NULL) { /* gak out of memory */ asoc->last_reset_action[0] = SCTP_STREAM_RESET_RESULT_DENIED; sctp_add_stream_reset_result(chk, seq, asoc->last_reset_action[0]); return; } liste->tsn = tsn; liste->number_entries = number_entries; memcpy(&liste->list_of_streams, req->list_of_streams, number_entries * sizeof(uint16_t)); TAILQ_INSERT_TAIL(&asoc->resetHead, liste, next_resp); asoc->last_reset_action[0] = SCTP_STREAM_RESET_RESULT_PERFORMED; } sctp_add_stream_reset_result(chk, seq, asoc->last_reset_action[0]); asoc->str_reset_seq_in++; } else if ((asoc->str_reset_seq_in - 1) == seq) { /* * one seq back, just echo back last action since my * response was lost. */ sctp_add_stream_reset_result(chk, seq, asoc->last_reset_action[0]); } else if ((asoc->str_reset_seq_in - 2) == seq) { /* * two seq back, just echo back last action since my * response was lost. */ sctp_add_stream_reset_result(chk, seq, asoc->last_reset_action[1]); } else { sctp_add_stream_reset_result(chk, seq, SCTP_STREAM_RESET_RESULT_ERR_BAD_SEQNO); } } static void sctp_handle_str_reset_add_strm(struct sctp_tcb *stcb, struct sctp_tmit_chunk *chk, struct sctp_stream_reset_add_strm *str_add) { /* * Peer is requesting to add more streams. If its within our * max-streams we will allow it. */ uint32_t num_stream, i; uint32_t seq; struct sctp_association *asoc = &stcb->asoc; struct sctp_queued_to_read *ctl, *nctl; /* Get the number. */ seq = ntohl(str_add->request_seq); num_stream = ntohs(str_add->number_of_streams); /* Now what would be the new total? */ if (asoc->str_reset_seq_in == seq) { num_stream += stcb->asoc.streamincnt; stcb->asoc.last_reset_action[1] = stcb->asoc.last_reset_action[0]; if (!(asoc->local_strreset_support & SCTP_ENABLE_CHANGE_ASSOC_REQ)) { asoc->last_reset_action[0] = SCTP_STREAM_RESET_RESULT_DENIED; } else if ((num_stream > stcb->asoc.max_inbound_streams) || (num_stream > 0xffff)) { /* We must reject it they ask for to many */ denied: stcb->asoc.last_reset_action[0] = SCTP_STREAM_RESET_RESULT_DENIED; } else { /* Ok, we can do that :-) */ struct sctp_stream_in *oldstrm; /* save off the old */ oldstrm = stcb->asoc.strmin; SCTP_MALLOC(stcb->asoc.strmin, struct sctp_stream_in *, (num_stream * sizeof(struct sctp_stream_in)), SCTP_M_STRMI); if (stcb->asoc.strmin == NULL) { stcb->asoc.strmin = oldstrm; goto denied; } /* copy off the old data */ for (i = 0; i < stcb->asoc.streamincnt; i++) { TAILQ_INIT(&stcb->asoc.strmin[i].inqueue); stcb->asoc.strmin[i].stream_no = i; stcb->asoc.strmin[i].last_sequence_delivered = oldstrm[i].last_sequence_delivered; stcb->asoc.strmin[i].delivery_started = oldstrm[i].delivery_started; /* now anything on those queues? */ TAILQ_FOREACH_SAFE(ctl, &oldstrm[i].inqueue, next, nctl) { TAILQ_REMOVE(&oldstrm[i].inqueue, ctl, next); TAILQ_INSERT_TAIL(&stcb->asoc.strmin[i].inqueue, ctl, next); } } /* Init the new streams */ for (i = stcb->asoc.streamincnt; i < num_stream; i++) { TAILQ_INIT(&stcb->asoc.strmin[i].inqueue); stcb->asoc.strmin[i].stream_no = i; stcb->asoc.strmin[i].last_sequence_delivered = 0xffff; stcb->asoc.strmin[i].delivery_started = 0; } SCTP_FREE(oldstrm, SCTP_M_STRMI); /* update the size */ stcb->asoc.streamincnt = num_stream; stcb->asoc.last_reset_action[0] = SCTP_STREAM_RESET_RESULT_PERFORMED; sctp_notify_stream_reset_add(stcb, stcb->asoc.streamincnt, stcb->asoc.streamoutcnt, 0); } sctp_add_stream_reset_result(chk, seq, asoc->last_reset_action[0]); asoc->str_reset_seq_in++; } else if ((asoc->str_reset_seq_in - 1) == seq) { /* * one seq back, just echo back last action since my * response was lost. */ sctp_add_stream_reset_result(chk, seq, asoc->last_reset_action[0]); } else if ((asoc->str_reset_seq_in - 2) == seq) { /* * two seq back, just echo back last action since my * response was lost. */ sctp_add_stream_reset_result(chk, seq, asoc->last_reset_action[1]); } else { sctp_add_stream_reset_result(chk, seq, SCTP_STREAM_RESET_RESULT_ERR_BAD_SEQNO); } } static void sctp_handle_str_reset_add_out_strm(struct sctp_tcb *stcb, struct sctp_tmit_chunk *chk, struct sctp_stream_reset_add_strm *str_add) { /* * Peer is requesting to add more streams. If its within our * max-streams we will allow it. */ uint16_t num_stream; uint32_t seq; struct sctp_association *asoc = &stcb->asoc; /* Get the number. */ seq = ntohl(str_add->request_seq); num_stream = ntohs(str_add->number_of_streams); /* Now what would be the new total? */ if (asoc->str_reset_seq_in == seq) { stcb->asoc.last_reset_action[1] = stcb->asoc.last_reset_action[0]; if (!(asoc->local_strreset_support & SCTP_ENABLE_CHANGE_ASSOC_REQ)) { asoc->last_reset_action[0] = SCTP_STREAM_RESET_RESULT_DENIED; } else if (stcb->asoc.stream_reset_outstanding) { /* We must reject it we have something pending */ stcb->asoc.last_reset_action[0] = SCTP_STREAM_RESET_RESULT_ERR_IN_PROGRESS; } else { /* Ok, we can do that :-) */ int mychk; mychk = stcb->asoc.streamoutcnt; mychk += num_stream; if (mychk < 0x10000) { stcb->asoc.last_reset_action[0] = SCTP_STREAM_RESET_RESULT_PERFORMED; if (sctp_send_str_reset_req(stcb, 0, NULL, 0, 0, 0, 1, num_stream, 0, 1)) { stcb->asoc.last_reset_action[0] = SCTP_STREAM_RESET_RESULT_DENIED; } } else { stcb->asoc.last_reset_action[0] = SCTP_STREAM_RESET_RESULT_DENIED; } } sctp_add_stream_reset_result(chk, seq, stcb->asoc.last_reset_action[0]); asoc->str_reset_seq_in++; } else if ((asoc->str_reset_seq_in - 1) == seq) { /* * one seq back, just echo back last action since my * response was lost. */ sctp_add_stream_reset_result(chk, seq, asoc->last_reset_action[0]); } else if ((asoc->str_reset_seq_in - 2) == seq) { /* * two seq back, just echo back last action since my * response was lost. */ sctp_add_stream_reset_result(chk, seq, asoc->last_reset_action[1]); } else { sctp_add_stream_reset_result(chk, seq, SCTP_STREAM_RESET_RESULT_ERR_BAD_SEQNO); } } #ifdef __GNUC__ __attribute__((noinline)) #endif static int sctp_handle_stream_reset(struct sctp_tcb *stcb, struct mbuf *m, int offset, struct sctp_chunkhdr *ch_req) { int chk_length, param_len, ptype; struct sctp_paramhdr pstore; uint8_t cstore[SCTP_CHUNK_BUFFER_SIZE]; uint32_t seq = 0; int num_req = 0; int trunc = 0; struct sctp_tmit_chunk *chk; struct sctp_chunkhdr *ch; struct sctp_paramhdr *ph; int ret_code = 0; int num_param = 0; /* now it may be a reset or a reset-response */ chk_length = ntohs(ch_req->chunk_length); /* setup for adding the response */ sctp_alloc_a_chunk(stcb, chk); if (chk == NULL) { return (ret_code); } chk->copy_by_ref = 0; chk->rec.chunk_id.id = SCTP_STREAM_RESET; chk->rec.chunk_id.can_take_data = 0; chk->flags = 0; chk->asoc = &stcb->asoc; chk->no_fr_allowed = 0; chk->book_size = chk->send_size = sizeof(struct sctp_chunkhdr); chk->book_size_scale = 0; chk->data = sctp_get_mbuf_for_msg(MCLBYTES, 0, M_NOWAIT, 1, MT_DATA); if (chk->data == NULL) { strres_nochunk: if (chk->data) { sctp_m_freem(chk->data); chk->data = NULL; } sctp_free_a_chunk(stcb, chk, SCTP_SO_NOT_LOCKED); return (ret_code); } SCTP_BUF_RESV_UF(chk->data, SCTP_MIN_OVERHEAD); /* setup chunk parameters */ chk->sent = SCTP_DATAGRAM_UNSENT; chk->snd_count = 0; chk->whoTo = NULL; ch = mtod(chk->data, struct sctp_chunkhdr *); ch->chunk_type = SCTP_STREAM_RESET; ch->chunk_flags = 0; ch->chunk_length = htons(chk->send_size); SCTP_BUF_LEN(chk->data) = SCTP_SIZE32(chk->send_size); offset += sizeof(struct sctp_chunkhdr); while ((size_t)chk_length >= sizeof(struct sctp_stream_reset_tsn_request)) { ph = (struct sctp_paramhdr *)sctp_m_getptr(m, offset, sizeof(pstore), (uint8_t *) & pstore); if (ph == NULL) break; param_len = ntohs(ph->param_length); if (param_len < (int)sizeof(struct sctp_stream_reset_tsn_request)) { /* bad param */ break; } ph = (struct sctp_paramhdr *)sctp_m_getptr(m, offset, min(param_len, (int)sizeof(cstore)), (uint8_t *) & cstore); ptype = ntohs(ph->param_type); num_param++; if (param_len > (int)sizeof(cstore)) { trunc = 1; } else { trunc = 0; } if (num_param > SCTP_MAX_RESET_PARAMS) { /* hit the max of parameters already sorry.. */ break; } if (ptype == SCTP_STR_RESET_OUT_REQUEST) { struct sctp_stream_reset_out_request *req_out; req_out = (struct sctp_stream_reset_out_request *)ph; num_req++; if (stcb->asoc.stream_reset_outstanding) { seq = ntohl(req_out->response_seq); if (seq == stcb->asoc.str_reset_seq_out) { /* implicit ack */ (void)sctp_handle_stream_reset_response(stcb, seq, SCTP_STREAM_RESET_RESULT_PERFORMED, NULL); } } sctp_handle_str_reset_request_out(stcb, chk, req_out, trunc); } else if (ptype == SCTP_STR_RESET_ADD_OUT_STREAMS) { struct sctp_stream_reset_add_strm *str_add; str_add = (struct sctp_stream_reset_add_strm *)ph; num_req++; sctp_handle_str_reset_add_strm(stcb, chk, str_add); } else if (ptype == SCTP_STR_RESET_ADD_IN_STREAMS) { struct sctp_stream_reset_add_strm *str_add; str_add = (struct sctp_stream_reset_add_strm *)ph; num_req++; sctp_handle_str_reset_add_out_strm(stcb, chk, str_add); } else if (ptype == SCTP_STR_RESET_IN_REQUEST) { struct sctp_stream_reset_in_request *req_in; num_req++; req_in = (struct sctp_stream_reset_in_request *)ph; sctp_handle_str_reset_request_in(stcb, chk, req_in, trunc); } else if (ptype == SCTP_STR_RESET_TSN_REQUEST) { struct sctp_stream_reset_tsn_request *req_tsn; num_req++; req_tsn = (struct sctp_stream_reset_tsn_request *)ph; if (sctp_handle_str_reset_request_tsn(stcb, chk, req_tsn)) { ret_code = 1; goto strres_nochunk; } /* no more */ break; } else if (ptype == SCTP_STR_RESET_RESPONSE) { struct sctp_stream_reset_response *resp; uint32_t result; resp = (struct sctp_stream_reset_response *)ph; seq = ntohl(resp->response_seq); result = ntohl(resp->result); if (sctp_handle_stream_reset_response(stcb, seq, result, resp)) { ret_code = 1; goto strres_nochunk; } } else { break; } offset += SCTP_SIZE32(param_len); chk_length -= SCTP_SIZE32(param_len); } if (num_req == 0) { /* we have no response free the stuff */ goto strres_nochunk; } /* ok we have a chunk to link in */ TAILQ_INSERT_TAIL(&stcb->asoc.control_send_queue, chk, sctp_next); stcb->asoc.ctrl_queue_cnt++; return (ret_code); } /* * Handle a router or endpoints report of a packet loss, there are two ways * to handle this, either we get the whole packet and must disect it * ourselves (possibly with truncation and or corruption) or it is a summary * from a middle box that did the disectting for us. */ static void sctp_handle_packet_dropped(struct sctp_pktdrop_chunk *cp, struct sctp_tcb *stcb, struct sctp_nets *net, uint32_t limit) { uint32_t bottle_bw, on_queue; uint16_t trunc_len; unsigned int chlen; unsigned int at; struct sctp_chunk_desc desc; struct sctp_chunkhdr *ch; chlen = ntohs(cp->ch.chunk_length); chlen -= sizeof(struct sctp_pktdrop_chunk); /* XXX possible chlen underflow */ if (chlen == 0) { ch = NULL; if (cp->ch.chunk_flags & SCTP_FROM_MIDDLE_BOX) SCTP_STAT_INCR(sctps_pdrpbwrpt); } else { ch = (struct sctp_chunkhdr *)(cp->data + sizeof(struct sctphdr)); chlen -= sizeof(struct sctphdr); /* XXX possible chlen underflow */ memset(&desc, 0, sizeof(desc)); } trunc_len = (uint16_t) ntohs(cp->trunc_len); if (trunc_len > limit) { trunc_len = limit; } /* now the chunks themselves */ while ((ch != NULL) && (chlen >= sizeof(struct sctp_chunkhdr))) { desc.chunk_type = ch->chunk_type; /* get amount we need to move */ at = ntohs(ch->chunk_length); if (at < sizeof(struct sctp_chunkhdr)) { /* corrupt chunk, maybe at the end? */ SCTP_STAT_INCR(sctps_pdrpcrupt); break; } if (trunc_len == 0) { /* we are supposed to have all of it */ if (at > chlen) { /* corrupt skip it */ SCTP_STAT_INCR(sctps_pdrpcrupt); break; } } else { /* is there enough of it left ? */ if (desc.chunk_type == SCTP_DATA) { if (chlen < (sizeof(struct sctp_data_chunk) + sizeof(desc.data_bytes))) { break; } } else { if (chlen < sizeof(struct sctp_chunkhdr)) { break; } } } if (desc.chunk_type == SCTP_DATA) { /* can we get out the tsn? */ if ((cp->ch.chunk_flags & SCTP_FROM_MIDDLE_BOX)) SCTP_STAT_INCR(sctps_pdrpmbda); if (chlen >= (sizeof(struct sctp_data_chunk) + sizeof(uint32_t))) { /* yep */ struct sctp_data_chunk *dcp; uint8_t *ddp; unsigned int iii; dcp = (struct sctp_data_chunk *)ch; ddp = (uint8_t *) (dcp + 1); for (iii = 0; iii < sizeof(desc.data_bytes); iii++) { desc.data_bytes[iii] = ddp[iii]; } desc.tsn_ifany = dcp->dp.tsn; } else { /* nope we are done. */ SCTP_STAT_INCR(sctps_pdrpnedat); break; } } else { if ((cp->ch.chunk_flags & SCTP_FROM_MIDDLE_BOX)) SCTP_STAT_INCR(sctps_pdrpmbct); } if (process_chunk_drop(stcb, &desc, net, cp->ch.chunk_flags)) { SCTP_STAT_INCR(sctps_pdrppdbrk); break; } if (SCTP_SIZE32(at) > chlen) { break; } chlen -= SCTP_SIZE32(at); if (chlen < sizeof(struct sctp_chunkhdr)) { /* done, none left */ break; } ch = (struct sctp_chunkhdr *)((caddr_t)ch + SCTP_SIZE32(at)); } /* Now update any rwnd --- possibly */ if ((cp->ch.chunk_flags & SCTP_FROM_MIDDLE_BOX) == 0) { /* From a peer, we get a rwnd report */ uint32_t a_rwnd; SCTP_STAT_INCR(sctps_pdrpfehos); bottle_bw = ntohl(cp->bottle_bw); on_queue = ntohl(cp->current_onq); if (bottle_bw && on_queue) { /* a rwnd report is in here */ if (bottle_bw > on_queue) a_rwnd = bottle_bw - on_queue; else a_rwnd = 0; if (a_rwnd == 0) stcb->asoc.peers_rwnd = 0; else { if (a_rwnd > stcb->asoc.total_flight) { stcb->asoc.peers_rwnd = a_rwnd - stcb->asoc.total_flight; } else { stcb->asoc.peers_rwnd = 0; } if (stcb->asoc.peers_rwnd < stcb->sctp_ep->sctp_ep.sctp_sws_sender) { /* SWS sender side engages */ stcb->asoc.peers_rwnd = 0; } } } } else { SCTP_STAT_INCR(sctps_pdrpfmbox); } /* now middle boxes in sat networks get a cwnd bump */ if ((cp->ch.chunk_flags & SCTP_FROM_MIDDLE_BOX) && (stcb->asoc.sat_t3_loss_recovery == 0) && (stcb->asoc.sat_network)) { /* * This is debateable but for sat networks it makes sense * Note if a T3 timer has went off, we will prohibit any * changes to cwnd until we exit the t3 loss recovery. */ stcb->asoc.cc_functions.sctp_cwnd_update_after_packet_dropped(stcb, net, cp, &bottle_bw, &on_queue); } } /* * handles all control chunks in a packet inputs: - m: mbuf chain, assumed to * still contain IP/SCTP header - stcb: is the tcb found for this packet - * offset: offset into the mbuf chain to first chunkhdr - length: is the * length of the complete packet outputs: - length: modified to remaining * length after control processing - netp: modified to new sctp_nets after * cookie-echo processing - return NULL to discard the packet (ie. no asoc, * bad packet,...) otherwise return the tcb for this packet */ #ifdef __GNUC__ __attribute__((noinline)) #endif static struct sctp_tcb * sctp_process_control(struct mbuf *m, int iphlen, int *offset, int length, struct sockaddr *src, struct sockaddr *dst, struct sctphdr *sh, struct sctp_chunkhdr *ch, struct sctp_inpcb *inp, struct sctp_tcb *stcb, struct sctp_nets **netp, int *fwd_tsn_seen, uint8_t mflowtype, uint32_t mflowid, uint32_t vrf_id, uint16_t port) { struct sctp_association *asoc; struct mbuf *op_err; char msg[SCTP_DIAG_INFO_LEN]; uint32_t vtag_in; int num_chunks = 0; /* number of control chunks processed */ uint32_t chk_length; int ret; int abort_no_unlock = 0; int ecne_seen = 0; /* * How big should this be, and should it be alloc'd? Lets try the * d-mtu-ceiling for now (2k) and that should hopefully work ... * until we get into jumbo grams and such.. */ uint8_t chunk_buf[SCTP_CHUNK_BUFFER_SIZE]; struct sctp_tcb *locked_tcb = stcb; int got_auth = 0; uint32_t auth_offset = 0, auth_len = 0; int auth_skipped = 0; int asconf_cnt = 0; #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) struct socket *so; #endif SCTPDBG(SCTP_DEBUG_INPUT1, "sctp_process_control: iphlen=%u, offset=%u, length=%u stcb:%p\n", iphlen, *offset, length, (void *)stcb); /* validate chunk header length... */ if (ntohs(ch->chunk_length) < sizeof(*ch)) { SCTPDBG(SCTP_DEBUG_INPUT1, "Invalid header length %d\n", ntohs(ch->chunk_length)); if (locked_tcb) { SCTP_TCB_UNLOCK(locked_tcb); } return (NULL); } /* * validate the verification tag */ vtag_in = ntohl(sh->v_tag); if (locked_tcb) { SCTP_TCB_LOCK_ASSERT(locked_tcb); } if (ch->chunk_type == SCTP_INITIATION) { SCTPDBG(SCTP_DEBUG_INPUT1, "Its an INIT of len:%d vtag:%x\n", ntohs(ch->chunk_length), vtag_in); if (vtag_in != 0) { /* protocol error- silently discard... */ SCTP_STAT_INCR(sctps_badvtag); if (locked_tcb) { SCTP_TCB_UNLOCK(locked_tcb); } return (NULL); } } else if (ch->chunk_type != SCTP_COOKIE_ECHO) { /* * If there is no stcb, skip the AUTH chunk and process * later after a stcb is found (to validate the lookup was * valid. */ if ((ch->chunk_type == SCTP_AUTHENTICATION) && (stcb == NULL) && (inp->auth_supported == 1)) { /* save this chunk for later processing */ auth_skipped = 1; auth_offset = *offset; auth_len = ntohs(ch->chunk_length); /* (temporarily) move past this chunk */ *offset += SCTP_SIZE32(auth_len); if (*offset >= length) { /* no more data left in the mbuf chain */ *offset = length; if (locked_tcb) { SCTP_TCB_UNLOCK(locked_tcb); } return (NULL); } ch = (struct sctp_chunkhdr *)sctp_m_getptr(m, *offset, sizeof(struct sctp_chunkhdr), chunk_buf); } if (ch == NULL) { /* Help */ *offset = length; if (locked_tcb) { SCTP_TCB_UNLOCK(locked_tcb); } return (NULL); } if (ch->chunk_type == SCTP_COOKIE_ECHO) { goto process_control_chunks; } /* * first check if it's an ASCONF with an unknown src addr we * need to look inside to find the association */ if (ch->chunk_type == SCTP_ASCONF && stcb == NULL) { struct sctp_chunkhdr *asconf_ch = ch; uint32_t asconf_offset = 0, asconf_len = 0; /* inp's refcount may be reduced */ SCTP_INP_INCR_REF(inp); asconf_offset = *offset; do { asconf_len = ntohs(asconf_ch->chunk_length); if (asconf_len < sizeof(struct sctp_asconf_paramhdr)) break; stcb = sctp_findassociation_ep_asconf(m, *offset, dst, sh, &inp, netp, vrf_id); if (stcb != NULL) break; asconf_offset += SCTP_SIZE32(asconf_len); asconf_ch = (struct sctp_chunkhdr *)sctp_m_getptr(m, asconf_offset, sizeof(struct sctp_chunkhdr), chunk_buf); } while (asconf_ch != NULL && asconf_ch->chunk_type == SCTP_ASCONF); if (stcb == NULL) { /* * reduce inp's refcount if not reduced in * sctp_findassociation_ep_asconf(). */ SCTP_INP_DECR_REF(inp); } else { locked_tcb = stcb; } /* now go back and verify any auth chunk to be sure */ if (auth_skipped && (stcb != NULL)) { struct sctp_auth_chunk *auth; auth = (struct sctp_auth_chunk *) sctp_m_getptr(m, auth_offset, auth_len, chunk_buf); got_auth = 1; auth_skipped = 0; if ((auth == NULL) || sctp_handle_auth(stcb, auth, m, auth_offset)) { /* auth HMAC failed so dump it */ *offset = length; if (locked_tcb) { SCTP_TCB_UNLOCK(locked_tcb); } return (NULL); } else { /* remaining chunks are HMAC checked */ stcb->asoc.authenticated = 1; } } } if (stcb == NULL) { snprintf(msg, sizeof(msg), "OOTB, %s:%d at %s\n", __FILE__, __LINE__, __FUNCTION__); op_err = sctp_generate_cause(SCTP_BASE_SYSCTL(sctp_diag_info_code), msg); /* no association, so it's out of the blue... */ sctp_handle_ootb(m, iphlen, *offset, src, dst, sh, inp, op_err, mflowtype, mflowid, vrf_id, port); *offset = length; if (locked_tcb) { SCTP_TCB_UNLOCK(locked_tcb); } return (NULL); } asoc = &stcb->asoc; /* ABORT and SHUTDOWN can use either v_tag... */ if ((ch->chunk_type == SCTP_ABORT_ASSOCIATION) || (ch->chunk_type == SCTP_SHUTDOWN_COMPLETE) || (ch->chunk_type == SCTP_PACKET_DROPPED)) { /* Take the T-bit always into account. */ if ((((ch->chunk_flags & SCTP_HAD_NO_TCB) == 0) && (vtag_in == asoc->my_vtag)) || (((ch->chunk_flags & SCTP_HAD_NO_TCB) == SCTP_HAD_NO_TCB) && (vtag_in == asoc->peer_vtag))) { /* this is valid */ } else { /* drop this packet... */ SCTP_STAT_INCR(sctps_badvtag); if (locked_tcb) { SCTP_TCB_UNLOCK(locked_tcb); } return (NULL); } } else if (ch->chunk_type == SCTP_SHUTDOWN_ACK) { if (vtag_in != asoc->my_vtag) { /* * this could be a stale SHUTDOWN-ACK or the * peer never got the SHUTDOWN-COMPLETE and * is still hung; we have started a new asoc * but it won't complete until the shutdown * is completed */ if (locked_tcb) { SCTP_TCB_UNLOCK(locked_tcb); } snprintf(msg, sizeof(msg), "OOTB, %s:%d at %s\n", __FILE__, __LINE__, __FUNCTION__); op_err = sctp_generate_cause(SCTP_BASE_SYSCTL(sctp_diag_info_code), msg); sctp_handle_ootb(m, iphlen, *offset, src, dst, sh, inp, op_err, mflowtype, mflowid, vrf_id, port); return (NULL); } } else { /* for all other chunks, vtag must match */ if (vtag_in != asoc->my_vtag) { /* invalid vtag... */ SCTPDBG(SCTP_DEBUG_INPUT3, "invalid vtag: %xh, expect %xh\n", vtag_in, asoc->my_vtag); SCTP_STAT_INCR(sctps_badvtag); if (locked_tcb) { SCTP_TCB_UNLOCK(locked_tcb); } *offset = length; return (NULL); } } } /* end if !SCTP_COOKIE_ECHO */ /* * process all control chunks... */ if (((ch->chunk_type == SCTP_SELECTIVE_ACK) || (ch->chunk_type == SCTP_NR_SELECTIVE_ACK) || (ch->chunk_type == SCTP_HEARTBEAT_REQUEST)) && (SCTP_GET_STATE(&stcb->asoc) == SCTP_STATE_COOKIE_ECHOED)) { /* implied cookie-ack.. we must have lost the ack */ if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_THRESHOLD_LOGGING) { sctp_misc_ints(SCTP_THRESHOLD_CLEAR, stcb->asoc.overall_error_count, 0, SCTP_FROM_SCTP_INPUT, __LINE__); } stcb->asoc.overall_error_count = 0; sctp_handle_cookie_ack((struct sctp_cookie_ack_chunk *)ch, stcb, *netp); } process_control_chunks: while (IS_SCTP_CONTROL(ch)) { /* validate chunk length */ chk_length = ntohs(ch->chunk_length); SCTPDBG(SCTP_DEBUG_INPUT2, "sctp_process_control: processing a chunk type=%u, len=%u\n", ch->chunk_type, chk_length); SCTP_LTRACE_CHK(inp, stcb, ch->chunk_type, chk_length); if (chk_length < sizeof(*ch) || (*offset + (int)chk_length) > length) { *offset = length; if (locked_tcb) { SCTP_TCB_UNLOCK(locked_tcb); } return (NULL); } SCTP_STAT_INCR_COUNTER64(sctps_incontrolchunks); /* * INIT-ACK only gets the init ack "header" portion only * because we don't have to process the peer's COOKIE. All * others get a complete chunk. */ if ((ch->chunk_type == SCTP_INITIATION_ACK) || (ch->chunk_type == SCTP_INITIATION)) { /* get an init-ack chunk */ ch = (struct sctp_chunkhdr *)sctp_m_getptr(m, *offset, sizeof(struct sctp_init_ack_chunk), chunk_buf); if (ch == NULL) { *offset = length; if (locked_tcb) { SCTP_TCB_UNLOCK(locked_tcb); } return (NULL); } } else { /* For cookies and all other chunks. */ if (chk_length > sizeof(chunk_buf)) { /* * use just the size of the chunk buffer so * the front part of our chunks fit in * contiguous space up to the chunk buffer * size (508 bytes). For chunks that need to * get more than that they must use the * sctp_m_getptr() function or other means * (e.g. know how to parse mbuf chains). * Cookies do this already. */ ch = (struct sctp_chunkhdr *)sctp_m_getptr(m, *offset, (sizeof(chunk_buf) - 4), chunk_buf); if (ch == NULL) { *offset = length; if (locked_tcb) { SCTP_TCB_UNLOCK(locked_tcb); } return (NULL); } } else { /* We can fit it all */ ch = (struct sctp_chunkhdr *)sctp_m_getptr(m, *offset, chk_length, chunk_buf); if (ch == NULL) { SCTP_PRINTF("sctp_process_control: Can't get the all data....\n"); *offset = length; if (locked_tcb) { SCTP_TCB_UNLOCK(locked_tcb); } return (NULL); } } } num_chunks++; /* Save off the last place we got a control from */ if (stcb != NULL) { if (((netp != NULL) && (*netp != NULL)) || (ch->chunk_type == SCTP_ASCONF)) { /* * allow last_control to be NULL if * ASCONF... ASCONF processing will find the * right net later */ if ((netp != NULL) && (*netp != NULL)) stcb->asoc.last_control_chunk_from = *netp; } } #ifdef SCTP_AUDITING_ENABLED sctp_audit_log(0xB0, ch->chunk_type); #endif /* check to see if this chunk required auth, but isn't */ if ((stcb != NULL) && (stcb->asoc.auth_supported == 1) && sctp_auth_is_required_chunk(ch->chunk_type, stcb->asoc.local_auth_chunks) && !stcb->asoc.authenticated) { /* "silently" ignore */ SCTP_STAT_INCR(sctps_recvauthmissing); goto next_chunk; } switch (ch->chunk_type) { case SCTP_INITIATION: SCTPDBG(SCTP_DEBUG_INPUT3, "SCTP_INIT\n"); /* The INIT chunk must be the only chunk. */ if ((num_chunks > 1) || (length - *offset > (int)SCTP_SIZE32(chk_length))) { op_err = sctp_generate_cause(SCTP_BASE_SYSCTL(sctp_diag_info_code), "INIT not the only chunk"); sctp_abort_association(inp, stcb, m, iphlen, src, dst, sh, op_err, mflowtype, mflowid, vrf_id, port); *offset = length; return (NULL); } /* Honor our resource limit. */ if (chk_length > SCTP_LARGEST_INIT_ACCEPTED) { op_err = sctp_generate_cause(SCTP_CAUSE_OUT_OF_RESC, ""); sctp_abort_association(inp, stcb, m, iphlen, src, dst, sh, op_err, mflowtype, mflowid, vrf_id, port); *offset = length; return (NULL); } sctp_handle_init(m, iphlen, *offset, src, dst, sh, (struct sctp_init_chunk *)ch, inp, stcb, &abort_no_unlock, mflowtype, mflowid, vrf_id, port); *offset = length; if ((!abort_no_unlock) && (locked_tcb)) { SCTP_TCB_UNLOCK(locked_tcb); } return (NULL); break; case SCTP_PAD_CHUNK: break; case SCTP_INITIATION_ACK: SCTPDBG(SCTP_DEBUG_INPUT3, "SCTP_INIT-ACK\n"); if (inp->sctp_flags & SCTP_PCB_FLAGS_SOCKET_GONE) { /* We are not interested anymore */ if ((stcb) && (stcb->asoc.total_output_queue_size)) { ; } else { if (locked_tcb != stcb) { /* Very unlikely */ SCTP_TCB_UNLOCK(locked_tcb); } *offset = length; if (stcb) { #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) so = SCTP_INP_SO(inp); atomic_add_int(&stcb->asoc.refcnt, 1); SCTP_TCB_UNLOCK(stcb); SCTP_SOCKET_LOCK(so, 1); SCTP_TCB_LOCK(stcb); atomic_subtract_int(&stcb->asoc.refcnt, 1); #endif (void)sctp_free_assoc(inp, stcb, SCTP_NORMAL_PROC, SCTP_FROM_SCTP_INPUT + SCTP_LOC_27); #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) SCTP_SOCKET_UNLOCK(so, 1); #endif } return (NULL); } } /* The INIT-ACK chunk must be the only chunk. */ if ((num_chunks > 1) || (length - *offset > (int)SCTP_SIZE32(chk_length))) { *offset = length; if (locked_tcb) { SCTP_TCB_UNLOCK(locked_tcb); } return (NULL); } if ((netp) && (*netp)) { ret = sctp_handle_init_ack(m, iphlen, *offset, src, dst, sh, (struct sctp_init_ack_chunk *)ch, stcb, *netp, &abort_no_unlock, mflowtype, mflowid, vrf_id); } else { ret = -1; } *offset = length; if (abort_no_unlock) { return (NULL); } /* * Special case, I must call the output routine to * get the cookie echoed */ if ((stcb != NULL) && (ret == 0)) { sctp_chunk_output(stcb->sctp_ep, stcb, SCTP_OUTPUT_FROM_CONTROL_PROC, SCTP_SO_NOT_LOCKED); } if (locked_tcb) { SCTP_TCB_UNLOCK(locked_tcb); } return (NULL); break; case SCTP_SELECTIVE_ACK: { struct sctp_sack_chunk *sack; int abort_now = 0; uint32_t a_rwnd, cum_ack; uint16_t num_seg, num_dup; uint8_t flags; int offset_seg, offset_dup; SCTPDBG(SCTP_DEBUG_INPUT3, "SCTP_SACK\n"); SCTP_STAT_INCR(sctps_recvsacks); if (stcb == NULL) { SCTPDBG(SCTP_DEBUG_INDATA1, "No stcb when processing SACK chunk\n"); break; } if (chk_length < sizeof(struct sctp_sack_chunk)) { SCTPDBG(SCTP_DEBUG_INDATA1, "Bad size on SACK chunk, too small\n"); break; } if (SCTP_GET_STATE(&stcb->asoc) == SCTP_STATE_SHUTDOWN_ACK_SENT) { /*- * If we have sent a shutdown-ack, we will pay no * attention to a sack sent in to us since * we don't care anymore. */ break; } sack = (struct sctp_sack_chunk *)ch; flags = ch->chunk_flags; cum_ack = ntohl(sack->sack.cum_tsn_ack); num_seg = ntohs(sack->sack.num_gap_ack_blks); num_dup = ntohs(sack->sack.num_dup_tsns); a_rwnd = (uint32_t) ntohl(sack->sack.a_rwnd); if (sizeof(struct sctp_sack_chunk) + num_seg * sizeof(struct sctp_gap_ack_block) + num_dup * sizeof(uint32_t) != chk_length) { SCTPDBG(SCTP_DEBUG_INDATA1, "Bad size of SACK chunk\n"); break; } offset_seg = *offset + sizeof(struct sctp_sack_chunk); offset_dup = offset_seg + num_seg * sizeof(struct sctp_gap_ack_block); SCTPDBG(SCTP_DEBUG_INPUT3, "SCTP_SACK process cum_ack:%x num_seg:%d a_rwnd:%d\n", cum_ack, num_seg, a_rwnd); stcb->asoc.seen_a_sack_this_pkt = 1; if ((stcb->asoc.pr_sctp_cnt == 0) && (num_seg == 0) && SCTP_TSN_GE(cum_ack, stcb->asoc.last_acked_seq) && (stcb->asoc.saw_sack_with_frags == 0) && (stcb->asoc.saw_sack_with_nr_frags == 0) && (!TAILQ_EMPTY(&stcb->asoc.sent_queue)) ) { /* * We have a SIMPLE sack having no * prior segments and data on sent * queue to be acked.. Use the * faster path sack processing. We * also allow window update sacks * with no missing segments to go * this way too. */ sctp_express_handle_sack(stcb, cum_ack, a_rwnd, &abort_now, ecne_seen); } else { if (netp && *netp) sctp_handle_sack(m, offset_seg, offset_dup, stcb, num_seg, 0, num_dup, &abort_now, flags, cum_ack, a_rwnd, ecne_seen); } if (abort_now) { /* ABORT signal from sack processing */ *offset = length; return (NULL); } if (TAILQ_EMPTY(&stcb->asoc.send_queue) && TAILQ_EMPTY(&stcb->asoc.sent_queue) && (stcb->asoc.stream_queue_cnt == 0)) { sctp_ulp_notify(SCTP_NOTIFY_SENDER_DRY, stcb, 0, NULL, SCTP_SO_NOT_LOCKED); } } break; /* * EY - nr_sack: If the received chunk is an * nr_sack chunk */ case SCTP_NR_SELECTIVE_ACK: { struct sctp_nr_sack_chunk *nr_sack; int abort_now = 0; uint32_t a_rwnd, cum_ack; uint16_t num_seg, num_nr_seg, num_dup; uint8_t flags; int offset_seg, offset_dup; SCTPDBG(SCTP_DEBUG_INPUT3, "SCTP_NR_SACK\n"); SCTP_STAT_INCR(sctps_recvsacks); if (stcb == NULL) { SCTPDBG(SCTP_DEBUG_INDATA1, "No stcb when processing NR-SACK chunk\n"); break; } if (stcb->asoc.nrsack_supported == 0) { goto unknown_chunk; } if (chk_length < sizeof(struct sctp_nr_sack_chunk)) { SCTPDBG(SCTP_DEBUG_INDATA1, "Bad size on NR-SACK chunk, too small\n"); break; } if (SCTP_GET_STATE(&stcb->asoc) == SCTP_STATE_SHUTDOWN_ACK_SENT) { /*- * If we have sent a shutdown-ack, we will pay no * attention to a sack sent in to us since * we don't care anymore. */ break; } nr_sack = (struct sctp_nr_sack_chunk *)ch; flags = ch->chunk_flags; cum_ack = ntohl(nr_sack->nr_sack.cum_tsn_ack); num_seg = ntohs(nr_sack->nr_sack.num_gap_ack_blks); num_nr_seg = ntohs(nr_sack->nr_sack.num_nr_gap_ack_blks); num_dup = ntohs(nr_sack->nr_sack.num_dup_tsns); a_rwnd = (uint32_t) ntohl(nr_sack->nr_sack.a_rwnd); if (sizeof(struct sctp_nr_sack_chunk) + (num_seg + num_nr_seg) * sizeof(struct sctp_gap_ack_block) + num_dup * sizeof(uint32_t) != chk_length) { SCTPDBG(SCTP_DEBUG_INDATA1, "Bad size of NR_SACK chunk\n"); break; } offset_seg = *offset + sizeof(struct sctp_nr_sack_chunk); offset_dup = offset_seg + num_seg * sizeof(struct sctp_gap_ack_block); SCTPDBG(SCTP_DEBUG_INPUT3, "SCTP_NR_SACK process cum_ack:%x num_seg:%d a_rwnd:%d\n", cum_ack, num_seg, a_rwnd); stcb->asoc.seen_a_sack_this_pkt = 1; if ((stcb->asoc.pr_sctp_cnt == 0) && (num_seg == 0) && (num_nr_seg == 0) && SCTP_TSN_GE(cum_ack, stcb->asoc.last_acked_seq) && (stcb->asoc.saw_sack_with_frags == 0) && (stcb->asoc.saw_sack_with_nr_frags == 0) && (!TAILQ_EMPTY(&stcb->asoc.sent_queue))) { /* * We have a SIMPLE sack having no * prior segments and data on sent * queue to be acked. Use the faster * path sack processing. We also * allow window update sacks with no * missing segments to go this way * too. */ sctp_express_handle_sack(stcb, cum_ack, a_rwnd, &abort_now, ecne_seen); } else { if (netp && *netp) sctp_handle_sack(m, offset_seg, offset_dup, stcb, num_seg, num_nr_seg, num_dup, &abort_now, flags, cum_ack, a_rwnd, ecne_seen); } if (abort_now) { /* ABORT signal from sack processing */ *offset = length; return (NULL); } if (TAILQ_EMPTY(&stcb->asoc.send_queue) && TAILQ_EMPTY(&stcb->asoc.sent_queue) && (stcb->asoc.stream_queue_cnt == 0)) { sctp_ulp_notify(SCTP_NOTIFY_SENDER_DRY, stcb, 0, NULL, SCTP_SO_NOT_LOCKED); } } break; case SCTP_HEARTBEAT_REQUEST: SCTPDBG(SCTP_DEBUG_INPUT3, "SCTP_HEARTBEAT\n"); if ((stcb) && netp && *netp) { SCTP_STAT_INCR(sctps_recvheartbeat); sctp_send_heartbeat_ack(stcb, m, *offset, chk_length, *netp); /* He's alive so give him credit */ if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_THRESHOLD_LOGGING) { sctp_misc_ints(SCTP_THRESHOLD_CLEAR, stcb->asoc.overall_error_count, 0, SCTP_FROM_SCTP_INPUT, __LINE__); } stcb->asoc.overall_error_count = 0; } break; case SCTP_HEARTBEAT_ACK: SCTPDBG(SCTP_DEBUG_INPUT3, "SCTP_HEARTBEAT-ACK\n"); if ((stcb == NULL) || (chk_length != sizeof(struct sctp_heartbeat_chunk))) { /* Its not ours */ *offset = length; if (locked_tcb) { SCTP_TCB_UNLOCK(locked_tcb); } return (NULL); } /* He's alive so give him credit */ if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_THRESHOLD_LOGGING) { sctp_misc_ints(SCTP_THRESHOLD_CLEAR, stcb->asoc.overall_error_count, 0, SCTP_FROM_SCTP_INPUT, __LINE__); } stcb->asoc.overall_error_count = 0; SCTP_STAT_INCR(sctps_recvheartbeatack); if (netp && *netp) sctp_handle_heartbeat_ack((struct sctp_heartbeat_chunk *)ch, stcb, *netp); break; case SCTP_ABORT_ASSOCIATION: SCTPDBG(SCTP_DEBUG_INPUT3, "SCTP_ABORT, stcb %p\n", (void *)stcb); if ((stcb) && netp && *netp) sctp_handle_abort((struct sctp_abort_chunk *)ch, stcb, *netp); *offset = length; return (NULL); break; case SCTP_SHUTDOWN: SCTPDBG(SCTP_DEBUG_INPUT3, "SCTP_SHUTDOWN, stcb %p\n", (void *)stcb); if ((stcb == NULL) || (chk_length != sizeof(struct sctp_shutdown_chunk))) { *offset = length; if (locked_tcb) { SCTP_TCB_UNLOCK(locked_tcb); } return (NULL); } if (netp && *netp) { int abort_flag = 0; sctp_handle_shutdown((struct sctp_shutdown_chunk *)ch, stcb, *netp, &abort_flag); if (abort_flag) { *offset = length; return (NULL); } } break; case SCTP_SHUTDOWN_ACK: SCTPDBG(SCTP_DEBUG_INPUT3, "SCTP_SHUTDOWN-ACK, stcb %p\n", (void *)stcb); if ((stcb) && (netp) && (*netp)) sctp_handle_shutdown_ack((struct sctp_shutdown_ack_chunk *)ch, stcb, *netp); *offset = length; return (NULL); break; case SCTP_OPERATION_ERROR: SCTPDBG(SCTP_DEBUG_INPUT3, "SCTP_OP-ERR\n"); if ((stcb) && netp && *netp && sctp_handle_error(ch, stcb, *netp) < 0) { *offset = length; return (NULL); } break; case SCTP_COOKIE_ECHO: SCTPDBG(SCTP_DEBUG_INPUT3, "SCTP_COOKIE-ECHO, stcb %p\n", (void *)stcb); if ((stcb) && (stcb->asoc.total_output_queue_size)) { ; } else { if (inp->sctp_flags & SCTP_PCB_FLAGS_SOCKET_GONE) { /* We are not interested anymore */ abend: if (stcb) { SCTP_TCB_UNLOCK(stcb); } *offset = length; return (NULL); } } /* * First are we accepting? We do this again here * since it is possible that a previous endpoint WAS * listening responded to a INIT-ACK and then * closed. We opened and bound.. and are now no * longer listening. */ if ((stcb == NULL) && (inp->sctp_socket->so_qlen >= inp->sctp_socket->so_qlimit)) { if ((inp->sctp_flags & SCTP_PCB_FLAGS_TCPTYPE) && (SCTP_BASE_SYSCTL(sctp_abort_if_one_2_one_hits_limit))) { op_err = sctp_generate_cause(SCTP_CAUSE_OUT_OF_RESC, ""); sctp_abort_association(inp, stcb, m, iphlen, src, dst, sh, op_err, mflowtype, mflowid, vrf_id, port); } *offset = length; return (NULL); } else { struct mbuf *ret_buf; struct sctp_inpcb *linp; if (stcb) { linp = NULL; } else { linp = inp; } if (linp) { SCTP_ASOC_CREATE_LOCK(linp); if ((inp->sctp_flags & SCTP_PCB_FLAGS_SOCKET_GONE) || (inp->sctp_flags & SCTP_PCB_FLAGS_SOCKET_ALLGONE)) { SCTP_ASOC_CREATE_UNLOCK(linp); goto abend; } } if (netp) { ret_buf = sctp_handle_cookie_echo(m, iphlen, *offset, src, dst, sh, (struct sctp_cookie_echo_chunk *)ch, &inp, &stcb, netp, auth_skipped, auth_offset, auth_len, &locked_tcb, mflowtype, mflowid, vrf_id, port); } else { ret_buf = NULL; } if (linp) { SCTP_ASOC_CREATE_UNLOCK(linp); } if (ret_buf == NULL) { if (locked_tcb) { SCTP_TCB_UNLOCK(locked_tcb); } SCTPDBG(SCTP_DEBUG_INPUT3, "GAK, null buffer\n"); *offset = length; return (NULL); } /* if AUTH skipped, see if it verified... */ if (auth_skipped) { got_auth = 1; auth_skipped = 0; } if (!TAILQ_EMPTY(&stcb->asoc.sent_queue)) { /* * Restart the timer if we have * pending data */ struct sctp_tmit_chunk *chk; chk = TAILQ_FIRST(&stcb->asoc.sent_queue); sctp_timer_start(SCTP_TIMER_TYPE_SEND, stcb->sctp_ep, stcb, chk->whoTo); } } break; case SCTP_COOKIE_ACK: SCTPDBG(SCTP_DEBUG_INPUT3, "SCTP_COOKIE-ACK, stcb %p\n", (void *)stcb); if ((stcb == NULL) || chk_length != sizeof(struct sctp_cookie_ack_chunk)) { if (locked_tcb) { SCTP_TCB_UNLOCK(locked_tcb); } return (NULL); } if (inp->sctp_flags & SCTP_PCB_FLAGS_SOCKET_GONE) { /* We are not interested anymore */ if ((stcb) && (stcb->asoc.total_output_queue_size)) { ; } else if (stcb) { #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) so = SCTP_INP_SO(inp); atomic_add_int(&stcb->asoc.refcnt, 1); SCTP_TCB_UNLOCK(stcb); SCTP_SOCKET_LOCK(so, 1); SCTP_TCB_LOCK(stcb); atomic_subtract_int(&stcb->asoc.refcnt, 1); #endif (void)sctp_free_assoc(inp, stcb, SCTP_NORMAL_PROC, SCTP_FROM_SCTP_INPUT + SCTP_LOC_27); #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) SCTP_SOCKET_UNLOCK(so, 1); #endif *offset = length; return (NULL); } } /* He's alive so give him credit */ if ((stcb) && netp && *netp) { if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_THRESHOLD_LOGGING) { sctp_misc_ints(SCTP_THRESHOLD_CLEAR, stcb->asoc.overall_error_count, 0, SCTP_FROM_SCTP_INPUT, __LINE__); } stcb->asoc.overall_error_count = 0; sctp_handle_cookie_ack((struct sctp_cookie_ack_chunk *)ch, stcb, *netp); } break; case SCTP_ECN_ECHO: SCTPDBG(SCTP_DEBUG_INPUT3, "SCTP_ECN-ECHO\n"); /* He's alive so give him credit */ if ((stcb == NULL) || (chk_length != sizeof(struct sctp_ecne_chunk))) { /* Its not ours */ if (locked_tcb) { SCTP_TCB_UNLOCK(locked_tcb); } *offset = length; return (NULL); } if (stcb) { if (stcb->asoc.ecn_supported == 0) { goto unknown_chunk; } if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_THRESHOLD_LOGGING) { sctp_misc_ints(SCTP_THRESHOLD_CLEAR, stcb->asoc.overall_error_count, 0, SCTP_FROM_SCTP_INPUT, __LINE__); } stcb->asoc.overall_error_count = 0; sctp_handle_ecn_echo((struct sctp_ecne_chunk *)ch, stcb); ecne_seen = 1; } break; case SCTP_ECN_CWR: SCTPDBG(SCTP_DEBUG_INPUT3, "SCTP_ECN-CWR\n"); /* He's alive so give him credit */ if ((stcb == NULL) || (chk_length != sizeof(struct sctp_cwr_chunk))) { /* Its not ours */ if (locked_tcb) { SCTP_TCB_UNLOCK(locked_tcb); } *offset = length; return (NULL); } if (stcb) { if (stcb->asoc.ecn_supported == 0) { goto unknown_chunk; } if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_THRESHOLD_LOGGING) { sctp_misc_ints(SCTP_THRESHOLD_CLEAR, stcb->asoc.overall_error_count, 0, SCTP_FROM_SCTP_INPUT, __LINE__); } stcb->asoc.overall_error_count = 0; sctp_handle_ecn_cwr((struct sctp_cwr_chunk *)ch, stcb, *netp); } break; case SCTP_SHUTDOWN_COMPLETE: SCTPDBG(SCTP_DEBUG_INPUT3, "SCTP_SHUTDOWN-COMPLETE, stcb %p\n", (void *)stcb); /* must be first and only chunk */ if ((num_chunks > 1) || (length - *offset > (int)SCTP_SIZE32(chk_length))) { *offset = length; if (locked_tcb) { SCTP_TCB_UNLOCK(locked_tcb); } return (NULL); } if ((stcb) && netp && *netp) { sctp_handle_shutdown_complete((struct sctp_shutdown_complete_chunk *)ch, stcb, *netp); } *offset = length; return (NULL); break; case SCTP_ASCONF: SCTPDBG(SCTP_DEBUG_INPUT3, "SCTP_ASCONF\n"); /* He's alive so give him credit */ if (stcb) { if (stcb->asoc.asconf_supported == 0) { goto unknown_chunk; } if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_THRESHOLD_LOGGING) { sctp_misc_ints(SCTP_THRESHOLD_CLEAR, stcb->asoc.overall_error_count, 0, SCTP_FROM_SCTP_INPUT, __LINE__); } stcb->asoc.overall_error_count = 0; sctp_handle_asconf(m, *offset, src, (struct sctp_asconf_chunk *)ch, stcb, asconf_cnt == 0); asconf_cnt++; } break; case SCTP_ASCONF_ACK: SCTPDBG(SCTP_DEBUG_INPUT3, "SCTP_ASCONF-ACK\n"); if (chk_length < sizeof(struct sctp_asconf_ack_chunk)) { /* Its not ours */ if (locked_tcb) { SCTP_TCB_UNLOCK(locked_tcb); } *offset = length; return (NULL); } if ((stcb) && netp && *netp) { if (stcb->asoc.asconf_supported == 0) { goto unknown_chunk; } /* He's alive so give him credit */ if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_THRESHOLD_LOGGING) { sctp_misc_ints(SCTP_THRESHOLD_CLEAR, stcb->asoc.overall_error_count, 0, SCTP_FROM_SCTP_INPUT, __LINE__); } stcb->asoc.overall_error_count = 0; sctp_handle_asconf_ack(m, *offset, (struct sctp_asconf_ack_chunk *)ch, stcb, *netp, &abort_no_unlock); if (abort_no_unlock) return (NULL); } break; case SCTP_FORWARD_CUM_TSN: SCTPDBG(SCTP_DEBUG_INPUT3, "SCTP_FWD-TSN\n"); if (chk_length < sizeof(struct sctp_forward_tsn_chunk)) { /* Its not ours */ if (locked_tcb) { SCTP_TCB_UNLOCK(locked_tcb); } *offset = length; return (NULL); } /* He's alive so give him credit */ if (stcb) { int abort_flag = 0; if (stcb->asoc.prsctp_supported == 0) { goto unknown_chunk; } stcb->asoc.overall_error_count = 0; if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_THRESHOLD_LOGGING) { sctp_misc_ints(SCTP_THRESHOLD_CLEAR, stcb->asoc.overall_error_count, 0, SCTP_FROM_SCTP_INPUT, __LINE__); } *fwd_tsn_seen = 1; if (inp->sctp_flags & SCTP_PCB_FLAGS_SOCKET_GONE) { /* We are not interested anymore */ #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) so = SCTP_INP_SO(inp); atomic_add_int(&stcb->asoc.refcnt, 1); SCTP_TCB_UNLOCK(stcb); SCTP_SOCKET_LOCK(so, 1); SCTP_TCB_LOCK(stcb); atomic_subtract_int(&stcb->asoc.refcnt, 1); #endif (void)sctp_free_assoc(inp, stcb, SCTP_NORMAL_PROC, SCTP_FROM_SCTP_INPUT + SCTP_LOC_29); #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) SCTP_SOCKET_UNLOCK(so, 1); #endif *offset = length; return (NULL); } sctp_handle_forward_tsn(stcb, (struct sctp_forward_tsn_chunk *)ch, &abort_flag, m, *offset); if (abort_flag) { *offset = length; return (NULL); } else { if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_THRESHOLD_LOGGING) { sctp_misc_ints(SCTP_THRESHOLD_CLEAR, stcb->asoc.overall_error_count, 0, SCTP_FROM_SCTP_INPUT, __LINE__); } stcb->asoc.overall_error_count = 0; } } break; case SCTP_STREAM_RESET: SCTPDBG(SCTP_DEBUG_INPUT3, "SCTP_STREAM_RESET\n"); if (((stcb == NULL) || (ch == NULL) || (chk_length < sizeof(struct sctp_stream_reset_tsn_req)))) { /* Its not ours */ if (locked_tcb) { SCTP_TCB_UNLOCK(locked_tcb); } *offset = length; return (NULL); } if (stcb->asoc.reconfig_supported == 0) { goto unknown_chunk; } if (sctp_handle_stream_reset(stcb, m, *offset, ch)) { /* stop processing */ *offset = length; return (NULL); } break; case SCTP_PACKET_DROPPED: SCTPDBG(SCTP_DEBUG_INPUT3, "SCTP_PACKET_DROPPED\n"); /* re-get it all please */ if (chk_length < sizeof(struct sctp_pktdrop_chunk)) { /* Its not ours */ if (locked_tcb) { SCTP_TCB_UNLOCK(locked_tcb); } *offset = length; return (NULL); } if (ch && (stcb) && netp && (*netp)) { if (stcb->asoc.pktdrop_supported == 0) { goto unknown_chunk; } sctp_handle_packet_dropped((struct sctp_pktdrop_chunk *)ch, stcb, *netp, min(chk_length, (sizeof(chunk_buf) - 4))); } break; case SCTP_AUTHENTICATION: SCTPDBG(SCTP_DEBUG_INPUT3, "SCTP_AUTHENTICATION\n"); if (stcb == NULL) { /* save the first AUTH for later processing */ if (auth_skipped == 0) { auth_offset = *offset; auth_len = chk_length; auth_skipped = 1; } /* skip this chunk (temporarily) */ goto next_chunk; } if (stcb->asoc.auth_supported == 0) { goto unknown_chunk; } if ((chk_length < (sizeof(struct sctp_auth_chunk))) || (chk_length > (sizeof(struct sctp_auth_chunk) + SCTP_AUTH_DIGEST_LEN_MAX))) { /* Its not ours */ if (locked_tcb) { SCTP_TCB_UNLOCK(locked_tcb); } *offset = length; return (NULL); } if (got_auth == 1) { /* skip this chunk... it's already auth'd */ goto next_chunk; } got_auth = 1; if ((ch == NULL) || sctp_handle_auth(stcb, (struct sctp_auth_chunk *)ch, m, *offset)) { /* auth HMAC failed so dump the packet */ *offset = length; return (stcb); } else { /* remaining chunks are HMAC checked */ stcb->asoc.authenticated = 1; } break; default: unknown_chunk: /* it's an unknown chunk! */ if ((ch->chunk_type & 0x40) && (stcb != NULL)) { struct mbuf *mm; struct sctp_paramhdr *phd; int len; mm = sctp_get_mbuf_for_msg(sizeof(struct sctp_paramhdr), 0, M_NOWAIT, 1, MT_DATA); if (mm) { len = min(SCTP_SIZE32(chk_length), (uint32_t) (length - *offset)); phd = mtod(mm, struct sctp_paramhdr *); /* * We cheat and use param type since * we did not bother to define a * error cause struct. They are the * same basic format with different * names. */ phd->param_type = htons(SCTP_CAUSE_UNRECOG_CHUNK); phd->param_length = htons(len + sizeof(*phd)); SCTP_BUF_LEN(mm) = sizeof(*phd); SCTP_BUF_NEXT(mm) = SCTP_M_COPYM(m, *offset, len, M_NOWAIT); if (SCTP_BUF_NEXT(mm)) { if (sctp_pad_lastmbuf(SCTP_BUF_NEXT(mm), SCTP_SIZE32(len) - len, NULL) == NULL) { sctp_m_freem(mm); } else { #ifdef SCTP_MBUF_LOGGING if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_MBUF_LOGGING_ENABLE) { struct mbuf *mat; for (mat = SCTP_BUF_NEXT(mm); mat; mat = SCTP_BUF_NEXT(mat)) { if (SCTP_BUF_IS_EXTENDED(mat)) { sctp_log_mb(mat, SCTP_MBUF_ICOPY); } } } #endif sctp_queue_op_err(stcb, mm); } } else { sctp_m_freem(mm); } } } if ((ch->chunk_type & 0x80) == 0) { /* discard this packet */ *offset = length; return (stcb); } /* else skip this bad chunk and continue... */ break; } /* switch (ch->chunk_type) */ next_chunk: /* get the next chunk */ *offset += SCTP_SIZE32(chk_length); if (*offset >= length) { /* no more data left in the mbuf chain */ break; } ch = (struct sctp_chunkhdr *)sctp_m_getptr(m, *offset, sizeof(struct sctp_chunkhdr), chunk_buf); if (ch == NULL) { if (locked_tcb) { SCTP_TCB_UNLOCK(locked_tcb); } *offset = length; return (NULL); } } /* while */ if (asconf_cnt > 0 && stcb != NULL) { sctp_send_asconf_ack(stcb); } return (stcb); } #ifdef INVARIANTS #ifdef __GNUC__ __attribute__((noinline)) #endif void sctp_validate_no_locks(struct sctp_inpcb *inp) { struct sctp_tcb *lstcb; LIST_FOREACH(lstcb, &inp->sctp_asoc_list, sctp_tcblist) { if (mtx_owned(&lstcb->tcb_mtx)) { panic("Own lock on stcb at return from input"); } } if (mtx_owned(&inp->inp_create_mtx)) { panic("Own create lock on inp"); } if (mtx_owned(&inp->inp_mtx)) { panic("Own inp lock on inp"); } } #endif /* * common input chunk processing (v4 and v6) */ void sctp_common_input_processing(struct mbuf **mm, int iphlen, int offset, int length, struct sockaddr *src, struct sockaddr *dst, struct sctphdr *sh, struct sctp_chunkhdr *ch, #if !defined(SCTP_WITH_NO_CSUM) uint8_t compute_crc, #endif uint8_t ecn_bits, uint8_t mflowtype, uint32_t mflowid, uint32_t vrf_id, uint16_t port) { uint32_t high_tsn; int fwd_tsn_seen = 0, data_processed = 0; struct mbuf *m = *mm, *op_err; char msg[SCTP_DIAG_INFO_LEN]; int un_sent; int cnt_ctrl_ready = 0; struct sctp_inpcb *inp = NULL, *inp_decr = NULL; struct sctp_tcb *stcb = NULL; struct sctp_nets *net = NULL; SCTP_STAT_INCR(sctps_recvdatagrams); #ifdef SCTP_AUDITING_ENABLED sctp_audit_log(0xE0, 1); sctp_auditing(0, inp, stcb, net); #endif #if !defined(SCTP_WITH_NO_CSUM) if (compute_crc != 0) { uint32_t check, calc_check; check = sh->checksum; sh->checksum = 0; calc_check = sctp_calculate_cksum(m, iphlen); sh->checksum = check; if (calc_check != check) { SCTPDBG(SCTP_DEBUG_INPUT1, "Bad CSUM on SCTP packet calc_check:%x check:%x m:%p mlen:%d iphlen:%d\n", calc_check, check, (void *)m, length, iphlen); stcb = sctp_findassociation_addr(m, offset, src, dst, sh, ch, &inp, &net, vrf_id); #if defined(INET) || defined(INET6) if ((net != NULL) && (port != 0)) { if (net->port == 0) { sctp_pathmtu_adjustment(stcb, net->mtu - sizeof(struct udphdr)); } net->port = port; } #endif if ((net != NULL) && (mflowtype != M_HASHTYPE_NONE)) { net->flowtype = mflowtype; } if ((inp != NULL) && (stcb != NULL)) { sctp_send_packet_dropped(stcb, net, m, length, iphlen, 1); sctp_chunk_output(inp, stcb, SCTP_OUTPUT_FROM_INPUT_ERROR, SCTP_SO_NOT_LOCKED); } else if ((inp != NULL) && (stcb == NULL)) { inp_decr = inp; } SCTP_STAT_INCR(sctps_badsum); SCTP_STAT_INCR_COUNTER32(sctps_checksumerrors); goto out; } } #endif /* Destination port of 0 is illegal, based on RFC4960. */ if (sh->dest_port == 0) { SCTP_STAT_INCR(sctps_hdrops); goto out; } stcb = sctp_findassociation_addr(m, offset, src, dst, sh, ch, &inp, &net, vrf_id); #if defined(INET) || defined(INET6) if ((net != NULL) && (port != 0)) { if (net->port == 0) { sctp_pathmtu_adjustment(stcb, net->mtu - sizeof(struct udphdr)); } net->port = port; } #endif if ((net != NULL) && (mflowtype != M_HASHTYPE_NONE)) { net->flowtype = mflowtype; } if (inp == NULL) { SCTP_STAT_INCR(sctps_noport); if (badport_bandlim(BANDLIM_SCTP_OOTB) < 0) { goto out; } if (ch->chunk_type == SCTP_SHUTDOWN_ACK) { sctp_send_shutdown_complete2(src, dst, sh, mflowtype, mflowid, vrf_id, port); goto out; } if (ch->chunk_type == SCTP_SHUTDOWN_COMPLETE) { goto out; } if (ch->chunk_type != SCTP_ABORT_ASSOCIATION) { if ((SCTP_BASE_SYSCTL(sctp_blackhole) == 0) || ((SCTP_BASE_SYSCTL(sctp_blackhole) == 1) && (ch->chunk_type != SCTP_INIT))) { op_err = sctp_generate_cause(SCTP_BASE_SYSCTL(sctp_diag_info_code), "Out of the blue"); sctp_send_abort(m, iphlen, src, dst, sh, 0, op_err, mflowtype, mflowid, vrf_id, port); } } goto out; } else if (stcb == NULL) { inp_decr = inp; } #ifdef IPSEC /*- * I very much doubt any of the IPSEC stuff will work but I have no * idea, so I will leave it in place. */ if (inp != NULL) { switch (dst->sa_family) { #ifdef INET case AF_INET: if (ipsec4_in_reject(m, &inp->ip_inp.inp)) { - IPSECSTAT_INC(ips_in_polvio); SCTP_STAT_INCR(sctps_hdrops); goto out; } break; #endif #ifdef INET6 case AF_INET6: if (ipsec6_in_reject(m, &inp->ip_inp.inp)) { - IPSEC6STAT_INC(ips_in_polvio); SCTP_STAT_INCR(sctps_hdrops); goto out; } break; #endif default: break; } } #endif SCTPDBG(SCTP_DEBUG_INPUT1, "Ok, Common input processing called, m:%p iphlen:%d offset:%d length:%d stcb:%p\n", (void *)m, iphlen, offset, length, (void *)stcb); if (stcb) { /* always clear this before beginning a packet */ stcb->asoc.authenticated = 0; stcb->asoc.seen_a_sack_this_pkt = 0; SCTPDBG(SCTP_DEBUG_INPUT1, "stcb:%p state:%x\n", (void *)stcb, stcb->asoc.state); if ((stcb->asoc.state & SCTP_STATE_WAS_ABORTED) || (stcb->asoc.state & SCTP_STATE_ABOUT_TO_BE_FREED)) { /*- * If we hit here, we had a ref count * up when the assoc was aborted and the * timer is clearing out the assoc, we should * NOT respond to any packet.. its OOTB. */ SCTP_TCB_UNLOCK(stcb); stcb = NULL; snprintf(msg, sizeof(msg), "OOTB, %s:%d at %s\n", __FILE__, __LINE__, __FUNCTION__); op_err = sctp_generate_cause(SCTP_BASE_SYSCTL(sctp_diag_info_code), msg); sctp_handle_ootb(m, iphlen, offset, src, dst, sh, inp, op_err, mflowtype, mflowid, vrf_id, port); goto out; } } if (IS_SCTP_CONTROL(ch)) { /* process the control portion of the SCTP packet */ /* sa_ignore NO_NULL_CHK */ stcb = sctp_process_control(m, iphlen, &offset, length, src, dst, sh, ch, inp, stcb, &net, &fwd_tsn_seen, mflowtype, mflowid, vrf_id, port); if (stcb) { /* * This covers us if the cookie-echo was there and * it changes our INP. */ inp = stcb->sctp_ep; #if defined(INET) || defined(INET6) if ((net) && (port)) { if (net->port == 0) { sctp_pathmtu_adjustment(stcb, net->mtu - sizeof(struct udphdr)); } net->port = port; } #endif } } else { /* * no control chunks, so pre-process DATA chunks (these * checks are taken care of by control processing) */ /* * if DATA only packet, and auth is required, then punt... * can't have authenticated without any AUTH (control) * chunks */ if ((stcb != NULL) && (stcb->asoc.auth_supported == 1) && sctp_auth_is_required_chunk(SCTP_DATA, stcb->asoc.local_auth_chunks)) { /* "silently" ignore */ SCTP_STAT_INCR(sctps_recvauthmissing); goto out; } if (stcb == NULL) { /* out of the blue DATA chunk */ snprintf(msg, sizeof(msg), "OOTB, %s:%d at %s\n", __FILE__, __LINE__, __FUNCTION__); op_err = sctp_generate_cause(SCTP_BASE_SYSCTL(sctp_diag_info_code), msg); sctp_handle_ootb(m, iphlen, offset, src, dst, sh, inp, op_err, mflowtype, mflowid, vrf_id, port); goto out; } if (stcb->asoc.my_vtag != ntohl(sh->v_tag)) { /* v_tag mismatch! */ SCTP_STAT_INCR(sctps_badvtag); goto out; } } if (stcb == NULL) { /* * no valid TCB for this packet, or we found it's a bad * packet while processing control, or we're done with this * packet (done or skip rest of data), so we drop it... */ goto out; } /* * DATA chunk processing */ /* plow through the data chunks while length > offset */ /* * Rest should be DATA only. Check authentication state if AUTH for * DATA is required. */ if ((length > offset) && (stcb != NULL) && (stcb->asoc.auth_supported == 1) && sctp_auth_is_required_chunk(SCTP_DATA, stcb->asoc.local_auth_chunks) && !stcb->asoc.authenticated) { /* "silently" ignore */ SCTP_STAT_INCR(sctps_recvauthmissing); SCTPDBG(SCTP_DEBUG_AUTH1, "Data chunk requires AUTH, skipped\n"); goto trigger_send; } if (length > offset) { int retval; /* * First check to make sure our state is correct. We would * not get here unless we really did have a tag, so we don't * abort if this happens, just dump the chunk silently. */ switch (SCTP_GET_STATE(&stcb->asoc)) { case SCTP_STATE_COOKIE_ECHOED: /* * we consider data with valid tags in this state * shows us the cookie-ack was lost. Imply it was * there. */ if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_THRESHOLD_LOGGING) { sctp_misc_ints(SCTP_THRESHOLD_CLEAR, stcb->asoc.overall_error_count, 0, SCTP_FROM_SCTP_INPUT, __LINE__); } stcb->asoc.overall_error_count = 0; sctp_handle_cookie_ack((struct sctp_cookie_ack_chunk *)ch, stcb, net); break; case SCTP_STATE_COOKIE_WAIT: /* * We consider OOTB any data sent during asoc setup. */ snprintf(msg, sizeof(msg), "OOTB, %s:%d at %s\n", __FILE__, __LINE__, __FUNCTION__); op_err = sctp_generate_cause(SCTP_BASE_SYSCTL(sctp_diag_info_code), msg); sctp_handle_ootb(m, iphlen, offset, src, dst, sh, inp, op_err, mflowtype, mflowid, vrf_id, port); goto out; /* sa_ignore NOTREACHED */ break; case SCTP_STATE_EMPTY: /* should not happen */ case SCTP_STATE_INUSE: /* should not happen */ case SCTP_STATE_SHUTDOWN_RECEIVED: /* This is a peer error */ case SCTP_STATE_SHUTDOWN_ACK_SENT: default: goto out; /* sa_ignore NOTREACHED */ break; case SCTP_STATE_OPEN: case SCTP_STATE_SHUTDOWN_SENT: break; } /* plow through the data chunks while length > offset */ retval = sctp_process_data(mm, iphlen, &offset, length, src, dst, sh, inp, stcb, net, &high_tsn, mflowtype, mflowid, vrf_id, port); if (retval == 2) { /* * The association aborted, NO UNLOCK needed since * the association is destroyed. */ stcb = NULL; goto out; } data_processed = 1; /* * Anything important needs to have been m_copy'ed in * process_data */ } /* take care of ecn */ if ((data_processed == 1) && (stcb->asoc.ecn_supported == 1) && ((ecn_bits & SCTP_CE_BITS) == SCTP_CE_BITS)) { /* Yep, we need to add a ECNE */ sctp_send_ecn_echo(stcb, net, high_tsn); } if ((data_processed == 0) && (fwd_tsn_seen)) { int was_a_gap; uint32_t highest_tsn; if (SCTP_TSN_GT(stcb->asoc.highest_tsn_inside_nr_map, stcb->asoc.highest_tsn_inside_map)) { highest_tsn = stcb->asoc.highest_tsn_inside_nr_map; } else { highest_tsn = stcb->asoc.highest_tsn_inside_map; } was_a_gap = SCTP_TSN_GT(highest_tsn, stcb->asoc.cumulative_tsn); stcb->asoc.send_sack = 1; sctp_sack_check(stcb, was_a_gap); } else if (fwd_tsn_seen) { stcb->asoc.send_sack = 1; } /* trigger send of any chunks in queue... */ trigger_send: #ifdef SCTP_AUDITING_ENABLED sctp_audit_log(0xE0, 2); sctp_auditing(1, inp, stcb, net); #endif SCTPDBG(SCTP_DEBUG_INPUT1, "Check for chunk output prw:%d tqe:%d tf=%d\n", stcb->asoc.peers_rwnd, TAILQ_EMPTY(&stcb->asoc.control_send_queue), stcb->asoc.total_flight); un_sent = (stcb->asoc.total_output_queue_size - stcb->asoc.total_flight); if (!TAILQ_EMPTY(&stcb->asoc.control_send_queue)) { cnt_ctrl_ready = stcb->asoc.ctrl_queue_cnt - stcb->asoc.ecn_echo_cnt_onq; } if (cnt_ctrl_ready || ((un_sent) && (stcb->asoc.peers_rwnd > 0 || (stcb->asoc.peers_rwnd <= 0 && stcb->asoc.total_flight == 0)))) { SCTPDBG(SCTP_DEBUG_INPUT3, "Calling chunk OUTPUT\n"); sctp_chunk_output(inp, stcb, SCTP_OUTPUT_FROM_CONTROL_PROC, SCTP_SO_NOT_LOCKED); SCTPDBG(SCTP_DEBUG_INPUT3, "chunk OUTPUT returns\n"); } #ifdef SCTP_AUDITING_ENABLED sctp_audit_log(0xE0, 3); sctp_auditing(2, inp, stcb, net); #endif out: if (stcb != NULL) { SCTP_TCB_UNLOCK(stcb); } if (inp_decr != NULL) { /* reduce ref-count */ SCTP_INP_WLOCK(inp_decr); SCTP_INP_DECR_REF(inp_decr); SCTP_INP_WUNLOCK(inp_decr); } #ifdef INVARIANTS if (inp != NULL) { sctp_validate_no_locks(inp); } #endif return; } #if 0 static void sctp_print_mbuf_chain(struct mbuf *m) { for (; m; m = SCTP_BUF_NEXT(m)) { SCTP_PRINTF("%p: m_len = %ld\n", (void *)m, SCTP_BUF_LEN(m)); if (SCTP_BUF_IS_EXTENDED(m)) SCTP_PRINTF("%p: extend_size = %d\n", (void *)m, SCTP_BUF_EXTEND_SIZE(m)); } } #endif #ifdef INET void sctp_input_with_port(struct mbuf *i_pak, int off, uint16_t port) { struct mbuf *m; int iphlen; uint32_t vrf_id = 0; uint8_t ecn_bits; struct sockaddr_in src, dst; struct ip *ip; struct sctphdr *sh; struct sctp_chunkhdr *ch; int length, offset; #if !defined(SCTP_WITH_NO_CSUM) uint8_t compute_crc; #endif uint32_t mflowid; uint8_t mflowtype; iphlen = off; if (SCTP_GET_PKT_VRFID(i_pak, vrf_id)) { SCTP_RELEASE_PKT(i_pak); return; } m = SCTP_HEADER_TO_CHAIN(i_pak); #ifdef SCTP_MBUF_LOGGING /* Log in any input mbufs */ if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_MBUF_LOGGING_ENABLE) { struct mbuf *mat; for (mat = m; mat; mat = SCTP_BUF_NEXT(mat)) { if (SCTP_BUF_IS_EXTENDED(mat)) { sctp_log_mb(mat, SCTP_MBUF_INPUT); } } } #endif #ifdef SCTP_PACKET_LOGGING if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_LAST_PACKET_TRACING) { sctp_packet_log(m); } #endif SCTPDBG(SCTP_DEBUG_CRCOFFLOAD, "sctp_input(): Packet of length %d received on %s with csum_flags 0x%b.\n", m->m_pkthdr.len, if_name(m->m_pkthdr.rcvif), (int)m->m_pkthdr.csum_flags, CSUM_BITS); mflowid = m->m_pkthdr.flowid; mflowtype = M_HASHTYPE_GET(m); SCTP_STAT_INCR(sctps_recvpackets); SCTP_STAT_INCR_COUNTER64(sctps_inpackets); /* Get IP, SCTP, and first chunk header together in the first mbuf. */ offset = iphlen + sizeof(struct sctphdr) + sizeof(struct sctp_chunkhdr); if (SCTP_BUF_LEN(m) < offset) { if ((m = m_pullup(m, offset)) == NULL) { SCTP_STAT_INCR(sctps_hdrops); return; } } ip = mtod(m, struct ip *); sh = (struct sctphdr *)((caddr_t)ip + iphlen); ch = (struct sctp_chunkhdr *)((caddr_t)sh + sizeof(struct sctphdr)); offset -= sizeof(struct sctp_chunkhdr); memset(&src, 0, sizeof(struct sockaddr_in)); src.sin_family = AF_INET; src.sin_len = sizeof(struct sockaddr_in); src.sin_port = sh->src_port; src.sin_addr = ip->ip_src; memset(&dst, 0, sizeof(struct sockaddr_in)); dst.sin_family = AF_INET; dst.sin_len = sizeof(struct sockaddr_in); dst.sin_port = sh->dest_port; dst.sin_addr = ip->ip_dst; length = ntohs(ip->ip_len); /* Validate mbuf chain length with IP payload length. */ if (SCTP_HEADER_LEN(m) != length) { SCTPDBG(SCTP_DEBUG_INPUT1, "sctp_input() length:%d reported length:%d\n", length, SCTP_HEADER_LEN(m)); SCTP_STAT_INCR(sctps_hdrops); goto out; } /* SCTP does not allow broadcasts or multicasts */ if (IN_MULTICAST(ntohl(dst.sin_addr.s_addr))) { goto out; } if (SCTP_IS_IT_BROADCAST(dst.sin_addr, m)) { goto out; } ecn_bits = ip->ip_tos; #if defined(SCTP_WITH_NO_CSUM) SCTP_STAT_INCR(sctps_recvnocrc); #else if (m->m_pkthdr.csum_flags & CSUM_SCTP_VALID) { SCTP_STAT_INCR(sctps_recvhwcrc); compute_crc = 0; } else { SCTP_STAT_INCR(sctps_recvswcrc); compute_crc = 1; } #endif sctp_common_input_processing(&m, iphlen, offset, length, (struct sockaddr *)&src, (struct sockaddr *)&dst, sh, ch, #if !defined(SCTP_WITH_NO_CSUM) compute_crc, #endif ecn_bits, mflowtype, mflowid, vrf_id, port); out: if (m) { sctp_m_freem(m); } return; } #if defined(__FreeBSD__) && defined(SCTP_MCORE_INPUT) && defined(SMP) extern int *sctp_cpuarry; #endif int sctp_input(struct mbuf **mp, int *offp, int proto SCTP_UNUSED) { struct mbuf *m; int off; m = *mp; off = *offp; #if defined(__FreeBSD__) && defined(SCTP_MCORE_INPUT) && defined(SMP) if (mp_ncpus > 1) { struct ip *ip; struct sctphdr *sh; int offset; int cpu_to_use; uint32_t flowid, tag; if (M_HASHTYPE_GET(m) != M_HASHTYPE_NONE) { flowid = m->m_pkthdr.flowid; } else { /* * No flow id built by lower layers fix it so we * create one. */ offset = off + sizeof(struct sctphdr); if (SCTP_BUF_LEN(m) < offset) { if ((m = m_pullup(m, offset)) == NULL) { SCTP_STAT_INCR(sctps_hdrops); return (IPPROTO_DONE); } } ip = mtod(m, struct ip *); sh = (struct sctphdr *)((caddr_t)ip + off); tag = htonl(sh->v_tag); flowid = tag ^ ntohs(sh->dest_port) ^ ntohs(sh->src_port); m->m_pkthdr.flowid = flowid; M_HASHTYPE_SET(m, M_HASHTYPE_OPAQUE); } cpu_to_use = sctp_cpuarry[flowid % mp_ncpus]; sctp_queue_to_mcore(m, off, cpu_to_use); return (IPPROTO_DONE); } #endif sctp_input_with_port(m, off, 0); return (IPPROTO_DONE); } #endif Index: projects/clang350-import/sys/netinet/tcp_input.c =================================================================== --- projects/clang350-import/sys/netinet/tcp_input.c (revision 275748) +++ projects/clang350-import/sys/netinet/tcp_input.c (revision 275749) @@ -1,3678 +1,3676 @@ /*- * Copyright (c) 1982, 1986, 1988, 1990, 1993, 1994, 1995 * The Regents of the University of California. All rights reserved. * Copyright (c) 2007-2008,2010 * Swinburne University of Technology, Melbourne, Australia. * Copyright (c) 2009-2010 Lawrence Stewart * Copyright (c) 2010 The FreeBSD Foundation * Copyright (c) 2010-2011 Juniper Networks, Inc. * All rights reserved. * * Portions of this software were developed at the Centre for Advanced Internet * Architectures, Swinburne University of Technology, by Lawrence Stewart, * James Healy and David Hayes, made possible in part by a grant from the Cisco * University Research Program Fund at Community Foundation Silicon Valley. * * Portions of this software were developed at the Centre for Advanced * Internet Architectures, Swinburne University of Technology, Melbourne, * Australia by David Hayes under sponsorship from the FreeBSD Foundation. * * Portions of this software were developed by Robert N. M. Watson under * contract to Juniper Networks, Inc. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 4. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * @(#)tcp_input.c 8.12 (Berkeley) 5/24/95 */ #include __FBSDID("$FreeBSD$"); #include "opt_ipfw.h" /* for ipfw_fwd */ #include "opt_inet.h" #include "opt_inet6.h" #include "opt_ipsec.h" #include "opt_tcpdebug.h" #include #include #include #include #include #include /* for proc0 declaration */ #include #include #include #include #include #include #include #include #include /* before tcp_seq.h, for tcp_random18() */ #include #include #include #include #include #define TCPSTATES /* for logging */ #include #include #include #include #include #include #include #include /* required for icmp_var.h */ #include /* for ICMP_BANDLIM */ #include #include #include #include #include #include #include #include #include #include #include #include #include #include #ifdef TCPDEBUG #include #endif /* TCPDEBUG */ #ifdef TCP_OFFLOAD #include #endif #ifdef IPSEC #include #include #endif /*IPSEC*/ #include #include const int tcprexmtthresh = 3; int tcp_log_in_vain = 0; SYSCTL_INT(_net_inet_tcp, OID_AUTO, log_in_vain, CTLFLAG_RW, &tcp_log_in_vain, 0, "Log all incoming TCP segments to closed ports"); VNET_DEFINE(int, blackhole) = 0; #define V_blackhole VNET(blackhole) SYSCTL_INT(_net_inet_tcp, OID_AUTO, blackhole, CTLFLAG_VNET | CTLFLAG_RW, &VNET_NAME(blackhole), 0, "Do not send RST on segments to closed ports"); VNET_DEFINE(int, tcp_delack_enabled) = 1; SYSCTL_INT(_net_inet_tcp, OID_AUTO, delayed_ack, CTLFLAG_VNET | CTLFLAG_RW, &VNET_NAME(tcp_delack_enabled), 0, "Delay ACK to try and piggyback it onto a data packet"); VNET_DEFINE(int, drop_synfin) = 0; #define V_drop_synfin VNET(drop_synfin) SYSCTL_INT(_net_inet_tcp, OID_AUTO, drop_synfin, CTLFLAG_VNET | CTLFLAG_RW, &VNET_NAME(drop_synfin), 0, "Drop TCP packets with SYN+FIN set"); VNET_DEFINE(int, tcp_do_rfc3042) = 1; #define V_tcp_do_rfc3042 VNET(tcp_do_rfc3042) SYSCTL_INT(_net_inet_tcp, OID_AUTO, rfc3042, CTLFLAG_VNET | CTLFLAG_RW, &VNET_NAME(tcp_do_rfc3042), 0, "Enable RFC 3042 (Limited Transmit)"); VNET_DEFINE(int, tcp_do_rfc3390) = 1; SYSCTL_INT(_net_inet_tcp, OID_AUTO, rfc3390, CTLFLAG_VNET | CTLFLAG_RW, &VNET_NAME(tcp_do_rfc3390), 0, "Enable RFC 3390 (Increasing TCP's Initial Congestion Window)"); SYSCTL_NODE(_net_inet_tcp, OID_AUTO, experimental, CTLFLAG_RW, 0, "Experimental TCP extensions"); VNET_DEFINE(int, tcp_do_initcwnd10) = 1; SYSCTL_INT(_net_inet_tcp_experimental, OID_AUTO, initcwnd10, CTLFLAG_VNET | CTLFLAG_RW, &VNET_NAME(tcp_do_initcwnd10), 0, "Enable RFC 6928 (Increasing initial CWND to 10)"); VNET_DEFINE(int, tcp_do_rfc3465) = 1; SYSCTL_INT(_net_inet_tcp, OID_AUTO, rfc3465, CTLFLAG_VNET | CTLFLAG_RW, &VNET_NAME(tcp_do_rfc3465), 0, "Enable RFC 3465 (Appropriate Byte Counting)"); VNET_DEFINE(int, tcp_abc_l_var) = 2; SYSCTL_INT(_net_inet_tcp, OID_AUTO, abc_l_var, CTLFLAG_VNET | CTLFLAG_RW, &VNET_NAME(tcp_abc_l_var), 2, "Cap the max cwnd increment during slow-start to this number of segments"); static SYSCTL_NODE(_net_inet_tcp, OID_AUTO, ecn, CTLFLAG_RW, 0, "TCP ECN"); VNET_DEFINE(int, tcp_do_ecn) = 0; SYSCTL_INT(_net_inet_tcp_ecn, OID_AUTO, enable, CTLFLAG_VNET | CTLFLAG_RW, &VNET_NAME(tcp_do_ecn), 0, "TCP ECN support"); VNET_DEFINE(int, tcp_ecn_maxretries) = 1; SYSCTL_INT(_net_inet_tcp_ecn, OID_AUTO, maxretries, CTLFLAG_VNET | CTLFLAG_RW, &VNET_NAME(tcp_ecn_maxretries), 0, "Max retries before giving up on ECN"); VNET_DEFINE(int, tcp_insecure_syn) = 0; #define V_tcp_insecure_syn VNET(tcp_insecure_syn) SYSCTL_INT(_net_inet_tcp, OID_AUTO, insecure_syn, CTLFLAG_VNET | CTLFLAG_RW, &VNET_NAME(tcp_insecure_syn), 0, "Follow RFC793 instead of RFC5961 criteria for accepting SYN packets"); VNET_DEFINE(int, tcp_insecure_rst) = 0; #define V_tcp_insecure_rst VNET(tcp_insecure_rst) SYSCTL_INT(_net_inet_tcp, OID_AUTO, insecure_rst, CTLFLAG_VNET | CTLFLAG_RW, &VNET_NAME(tcp_insecure_rst), 0, "Follow RFC793 instead of RFC5961 criteria for accepting RST packets"); VNET_DEFINE(int, tcp_recvspace) = 1024*64; #define V_tcp_recvspace VNET(tcp_recvspace) SYSCTL_INT(_net_inet_tcp, TCPCTL_RECVSPACE, recvspace, CTLFLAG_VNET | CTLFLAG_RW, &VNET_NAME(tcp_recvspace), 0, "Initial receive socket buffer size"); VNET_DEFINE(int, tcp_do_autorcvbuf) = 1; #define V_tcp_do_autorcvbuf VNET(tcp_do_autorcvbuf) SYSCTL_INT(_net_inet_tcp, OID_AUTO, recvbuf_auto, CTLFLAG_VNET | CTLFLAG_RW, &VNET_NAME(tcp_do_autorcvbuf), 0, "Enable automatic receive buffer sizing"); VNET_DEFINE(int, tcp_autorcvbuf_inc) = 16*1024; #define V_tcp_autorcvbuf_inc VNET(tcp_autorcvbuf_inc) SYSCTL_INT(_net_inet_tcp, OID_AUTO, recvbuf_inc, CTLFLAG_VNET | CTLFLAG_RW, &VNET_NAME(tcp_autorcvbuf_inc), 0, "Incrementor step size of automatic receive buffer"); VNET_DEFINE(int, tcp_autorcvbuf_max) = 2*1024*1024; #define V_tcp_autorcvbuf_max VNET(tcp_autorcvbuf_max) SYSCTL_INT(_net_inet_tcp, OID_AUTO, recvbuf_max, CTLFLAG_VNET | CTLFLAG_RW, &VNET_NAME(tcp_autorcvbuf_max), 0, "Max size of automatic receive buffer"); VNET_DEFINE(struct inpcbhead, tcb); #define tcb6 tcb /* for KAME src sync over BSD*'s */ VNET_DEFINE(struct inpcbinfo, tcbinfo); static void tcp_dooptions(struct tcpopt *, u_char *, int, int); static void tcp_do_segment(struct mbuf *, struct tcphdr *, struct socket *, struct tcpcb *, int, int, uint8_t, int); static void tcp_dropwithreset(struct mbuf *, struct tcphdr *, struct tcpcb *, int, int); static void tcp_pulloutofband(struct socket *, struct tcphdr *, struct mbuf *, int); static void tcp_xmit_timer(struct tcpcb *, int); static void tcp_newreno_partial_ack(struct tcpcb *, struct tcphdr *); static void inline cc_ack_received(struct tcpcb *tp, struct tcphdr *th, uint16_t type); static void inline cc_conn_init(struct tcpcb *tp); static void inline cc_post_recovery(struct tcpcb *tp, struct tcphdr *th); static void inline hhook_run_tcp_est_in(struct tcpcb *tp, struct tcphdr *th, struct tcpopt *to); /* * TCP statistics are stored in an "array" of counter(9)s. */ VNET_PCPUSTAT_DEFINE(struct tcpstat, tcpstat); VNET_PCPUSTAT_SYSINIT(tcpstat); SYSCTL_VNET_PCPUSTAT(_net_inet_tcp, TCPCTL_STATS, stats, struct tcpstat, tcpstat, "TCP statistics (struct tcpstat, netinet/tcp_var.h)"); #ifdef VIMAGE VNET_PCPUSTAT_SYSUNINIT(tcpstat); #endif /* VIMAGE */ /* * Kernel module interface for updating tcpstat. The argument is an index * into tcpstat treated as an array. */ void kmod_tcpstat_inc(int statnum) { counter_u64_add(VNET(tcpstat)[statnum], 1); } /* * Wrapper for the TCP established input helper hook. */ static void inline hhook_run_tcp_est_in(struct tcpcb *tp, struct tcphdr *th, struct tcpopt *to) { struct tcp_hhook_data hhook_data; if (V_tcp_hhh[HHOOK_TCP_EST_IN]->hhh_nhooks > 0) { hhook_data.tp = tp; hhook_data.th = th; hhook_data.to = to; hhook_run_hooks(V_tcp_hhh[HHOOK_TCP_EST_IN], &hhook_data, tp->osd); } } /* * CC wrapper hook functions */ static void inline cc_ack_received(struct tcpcb *tp, struct tcphdr *th, uint16_t type) { INP_WLOCK_ASSERT(tp->t_inpcb); tp->ccv->bytes_this_ack = BYTES_THIS_ACK(tp, th); if (tp->snd_cwnd <= tp->snd_wnd) tp->ccv->flags |= CCF_CWND_LIMITED; else tp->ccv->flags &= ~CCF_CWND_LIMITED; if (type == CC_ACK) { if (tp->snd_cwnd > tp->snd_ssthresh) { tp->t_bytes_acked += min(tp->ccv->bytes_this_ack, V_tcp_abc_l_var * tp->t_maxseg); if (tp->t_bytes_acked >= tp->snd_cwnd) { tp->t_bytes_acked -= tp->snd_cwnd; tp->ccv->flags |= CCF_ABC_SENTAWND; } } else { tp->ccv->flags &= ~CCF_ABC_SENTAWND; tp->t_bytes_acked = 0; } } if (CC_ALGO(tp)->ack_received != NULL) { /* XXXLAS: Find a way to live without this */ tp->ccv->curack = th->th_ack; CC_ALGO(tp)->ack_received(tp->ccv, type); } } static void inline cc_conn_init(struct tcpcb *tp) { struct hc_metrics_lite metrics; struct inpcb *inp = tp->t_inpcb; int rtt; INP_WLOCK_ASSERT(tp->t_inpcb); tcp_hc_get(&inp->inp_inc, &metrics); if (tp->t_srtt == 0 && (rtt = metrics.rmx_rtt)) { tp->t_srtt = rtt; tp->t_rttbest = tp->t_srtt + TCP_RTT_SCALE; TCPSTAT_INC(tcps_usedrtt); if (metrics.rmx_rttvar) { tp->t_rttvar = metrics.rmx_rttvar; TCPSTAT_INC(tcps_usedrttvar); } else { /* default variation is +- 1 rtt */ tp->t_rttvar = tp->t_srtt * TCP_RTTVAR_SCALE / TCP_RTT_SCALE; } TCPT_RANGESET(tp->t_rxtcur, ((tp->t_srtt >> 2) + tp->t_rttvar) >> 1, tp->t_rttmin, TCPTV_REXMTMAX); } if (metrics.rmx_ssthresh) { /* * There's some sort of gateway or interface * buffer limit on the path. Use this to set * the slow start threshhold, but set the * threshold to no less than 2*mss. */ tp->snd_ssthresh = max(2 * tp->t_maxseg, metrics.rmx_ssthresh); TCPSTAT_INC(tcps_usedssthresh); } /* * Set the initial slow-start flight size. * * RFC5681 Section 3.1 specifies the default conservative values. * RFC3390 specifies slightly more aggressive values. * RFC6928 increases it to ten segments. * * If a SYN or SYN/ACK was lost and retransmitted, we have to * reduce the initial CWND to one segment as congestion is likely * requiring us to be cautious. */ if (tp->snd_cwnd == 1) tp->snd_cwnd = tp->t_maxseg; /* SYN(-ACK) lost */ else if (V_tcp_do_initcwnd10) tp->snd_cwnd = min(10 * tp->t_maxseg, max(2 * tp->t_maxseg, 14600)); else if (V_tcp_do_rfc3390) tp->snd_cwnd = min(4 * tp->t_maxseg, max(2 * tp->t_maxseg, 4380)); else { /* Per RFC5681 Section 3.1 */ if (tp->t_maxseg > 2190) tp->snd_cwnd = 2 * tp->t_maxseg; else if (tp->t_maxseg > 1095) tp->snd_cwnd = 3 * tp->t_maxseg; else tp->snd_cwnd = 4 * tp->t_maxseg; } if (CC_ALGO(tp)->conn_init != NULL) CC_ALGO(tp)->conn_init(tp->ccv); } void inline cc_cong_signal(struct tcpcb *tp, struct tcphdr *th, uint32_t type) { INP_WLOCK_ASSERT(tp->t_inpcb); switch(type) { case CC_NDUPACK: if (!IN_FASTRECOVERY(tp->t_flags)) { tp->snd_recover = tp->snd_max; if (tp->t_flags & TF_ECN_PERMIT) tp->t_flags |= TF_ECN_SND_CWR; } break; case CC_ECN: if (!IN_CONGRECOVERY(tp->t_flags)) { TCPSTAT_INC(tcps_ecn_rcwnd); tp->snd_recover = tp->snd_max; if (tp->t_flags & TF_ECN_PERMIT) tp->t_flags |= TF_ECN_SND_CWR; } break; case CC_RTO: tp->t_dupacks = 0; tp->t_bytes_acked = 0; EXIT_RECOVERY(tp->t_flags); tp->snd_ssthresh = max(2, min(tp->snd_wnd, tp->snd_cwnd) / 2 / tp->t_maxseg) * tp->t_maxseg; tp->snd_cwnd = tp->t_maxseg; break; case CC_RTO_ERR: TCPSTAT_INC(tcps_sndrexmitbad); /* RTO was unnecessary, so reset everything. */ tp->snd_cwnd = tp->snd_cwnd_prev; tp->snd_ssthresh = tp->snd_ssthresh_prev; tp->snd_recover = tp->snd_recover_prev; if (tp->t_flags & TF_WASFRECOVERY) ENTER_FASTRECOVERY(tp->t_flags); if (tp->t_flags & TF_WASCRECOVERY) ENTER_CONGRECOVERY(tp->t_flags); tp->snd_nxt = tp->snd_max; tp->t_flags &= ~TF_PREVVALID; tp->t_badrxtwin = 0; break; } if (CC_ALGO(tp)->cong_signal != NULL) { if (th != NULL) tp->ccv->curack = th->th_ack; CC_ALGO(tp)->cong_signal(tp->ccv, type); } } static void inline cc_post_recovery(struct tcpcb *tp, struct tcphdr *th) { INP_WLOCK_ASSERT(tp->t_inpcb); /* XXXLAS: KASSERT that we're in recovery? */ if (CC_ALGO(tp)->post_recovery != NULL) { tp->ccv->curack = th->th_ack; CC_ALGO(tp)->post_recovery(tp->ccv); } /* XXXLAS: EXIT_RECOVERY ? */ tp->t_bytes_acked = 0; } #ifdef TCP_SIGNATURE static inline int tcp_signature_verify_input(struct mbuf *m, int off0, int tlen, int optlen, struct tcpopt *to, struct tcphdr *th, u_int tcpbflag) { int ret; tcp_fields_to_net(th); ret = tcp_signature_verify(m, off0, tlen, optlen, to, th, tcpbflag); tcp_fields_to_host(th); return (ret); } #endif /* Neighbor Discovery, Neighbor Unreachability Detection Upper layer hint. */ #ifdef INET6 #define ND6_HINT(tp) \ do { \ if ((tp) && (tp)->t_inpcb && \ ((tp)->t_inpcb->inp_vflag & INP_IPV6) != 0) \ nd6_nud_hint(NULL, NULL, 0); \ } while (0) #else #define ND6_HINT(tp) #endif /* * Indicate whether this ack should be delayed. We can delay the ack if * following conditions are met: * - There is no delayed ack timer in progress. * - Our last ack wasn't a 0-sized window. We never want to delay * the ack that opens up a 0-sized window. * - LRO wasn't used for this segment. We make sure by checking that the * segment size is not larger than the MSS. * - Delayed acks are enabled or this is a half-synchronized T/TCP * connection. */ #define DELAY_ACK(tp, tlen) \ ((!tcp_timer_active(tp, TT_DELACK) && \ (tp->t_flags & TF_RXWIN0SENT) == 0) && \ (tlen <= tp->t_maxopd) && \ (V_tcp_delack_enabled || (tp->t_flags & TF_NEEDSYN))) /* * TCP input handling is split into multiple parts: * tcp6_input is a thin wrapper around tcp_input for the extended * ip6_protox[] call format in ip6_input * tcp_input handles primary segment validation, inpcb lookup and * SYN processing on listen sockets * tcp_do_segment processes the ACK and text of the segment for * establishing, established and closing connections */ #ifdef INET6 int tcp6_input(struct mbuf **mp, int *offp, int proto) { struct mbuf *m = *mp; struct in6_ifaddr *ia6; struct ip6_hdr *ip6; IP6_EXTHDR_CHECK(m, *offp, sizeof(struct tcphdr), IPPROTO_DONE); /* * draft-itojun-ipv6-tcp-to-anycast * better place to put this in? */ ip6 = mtod(m, struct ip6_hdr *); ia6 = in6ifa_ifwithaddr(&ip6->ip6_dst, 0 /* XXX */); if (ia6 && (ia6->ia6_flags & IN6_IFF_ANYCAST)) { struct ip6_hdr *ip6; ifa_free(&ia6->ia_ifa); ip6 = mtod(m, struct ip6_hdr *); icmp6_error(m, ICMP6_DST_UNREACH, ICMP6_DST_UNREACH_ADDR, (caddr_t)&ip6->ip6_dst - (caddr_t)ip6); return (IPPROTO_DONE); } if (ia6) ifa_free(&ia6->ia_ifa); return (tcp_input(mp, offp, proto)); } #endif /* INET6 */ int tcp_input(struct mbuf **mp, int *offp, int proto) { struct mbuf *m = *mp; struct tcphdr *th = NULL; struct ip *ip = NULL; struct inpcb *inp = NULL; struct tcpcb *tp = NULL; struct socket *so = NULL; u_char *optp = NULL; int off0; int optlen = 0; #ifdef INET int len; #endif int tlen = 0, off; int drop_hdrlen; int thflags; int rstreason = 0; /* For badport_bandlim accounting purposes */ #ifdef TCP_SIGNATURE uint8_t sig_checked = 0; #endif uint8_t iptos = 0; struct m_tag *fwd_tag = NULL; #ifdef INET6 struct ip6_hdr *ip6 = NULL; int isipv6; #else const void *ip6 = NULL; #endif /* INET6 */ struct tcpopt to; /* options in this segment */ char *s = NULL; /* address and port logging */ int ti_locked; #define TI_UNLOCKED 1 #define TI_WLOCKED 2 #ifdef TCPDEBUG /* * The size of tcp_saveipgen must be the size of the max ip header, * now IPv6. */ u_char tcp_saveipgen[IP6_HDR_LEN]; struct tcphdr tcp_savetcp; short ostate = 0; #endif #ifdef INET6 isipv6 = (mtod(m, struct ip *)->ip_v == 6) ? 1 : 0; #endif off0 = *offp; m = *mp; *mp = NULL; to.to_flags = 0; TCPSTAT_INC(tcps_rcvtotal); #ifdef INET6 if (isipv6) { /* IP6_EXTHDR_CHECK() is already done at tcp6_input(). */ if (m->m_len < (sizeof(*ip6) + sizeof(*th))) { m = m_pullup(m, sizeof(*ip6) + sizeof(*th)); if (m == NULL) { TCPSTAT_INC(tcps_rcvshort); return (IPPROTO_DONE); } } ip6 = mtod(m, struct ip6_hdr *); th = (struct tcphdr *)((caddr_t)ip6 + off0); tlen = sizeof(*ip6) + ntohs(ip6->ip6_plen) - off0; if (m->m_pkthdr.csum_flags & CSUM_DATA_VALID_IPV6) { if (m->m_pkthdr.csum_flags & CSUM_PSEUDO_HDR) th->th_sum = m->m_pkthdr.csum_data; else th->th_sum = in6_cksum_pseudo(ip6, tlen, IPPROTO_TCP, m->m_pkthdr.csum_data); th->th_sum ^= 0xffff; } else th->th_sum = in6_cksum(m, IPPROTO_TCP, off0, tlen); if (th->th_sum) { TCPSTAT_INC(tcps_rcvbadsum); goto drop; } /* * Be proactive about unspecified IPv6 address in source. * As we use all-zero to indicate unbounded/unconnected pcb, * unspecified IPv6 address can be used to confuse us. * * Note that packets with unspecified IPv6 destination is * already dropped in ip6_input. */ if (IN6_IS_ADDR_UNSPECIFIED(&ip6->ip6_src)) { /* XXX stat */ goto drop; } } #endif #if defined(INET) && defined(INET6) else #endif #ifdef INET { /* * Get IP and TCP header together in first mbuf. * Note: IP leaves IP header in first mbuf. */ if (off0 > sizeof (struct ip)) { ip_stripoptions(m); off0 = sizeof(struct ip); } if (m->m_len < sizeof (struct tcpiphdr)) { if ((m = m_pullup(m, sizeof (struct tcpiphdr))) == NULL) { TCPSTAT_INC(tcps_rcvshort); return (IPPROTO_DONE); } } ip = mtod(m, struct ip *); th = (struct tcphdr *)((caddr_t)ip + off0); tlen = ntohs(ip->ip_len) - off0; if (m->m_pkthdr.csum_flags & CSUM_DATA_VALID) { if (m->m_pkthdr.csum_flags & CSUM_PSEUDO_HDR) th->th_sum = m->m_pkthdr.csum_data; else th->th_sum = in_pseudo(ip->ip_src.s_addr, ip->ip_dst.s_addr, htonl(m->m_pkthdr.csum_data + tlen + IPPROTO_TCP)); th->th_sum ^= 0xffff; } else { struct ipovly *ipov = (struct ipovly *)ip; /* * Checksum extended TCP header and data. */ len = off0 + tlen; bzero(ipov->ih_x1, sizeof(ipov->ih_x1)); ipov->ih_len = htons(tlen); th->th_sum = in_cksum(m, len); /* Reset length for SDT probes. */ ip->ip_len = htons(tlen + off0); } if (th->th_sum) { TCPSTAT_INC(tcps_rcvbadsum); goto drop; } /* Re-initialization for later version check */ ip->ip_v = IPVERSION; } #endif /* INET */ #ifdef INET6 if (isipv6) iptos = (ntohl(ip6->ip6_flow) >> 20) & 0xff; #endif #if defined(INET) && defined(INET6) else #endif #ifdef INET iptos = ip->ip_tos; #endif /* * Check that TCP offset makes sense, * pull out TCP options and adjust length. XXX */ off = th->th_off << 2; if (off < sizeof (struct tcphdr) || off > tlen) { TCPSTAT_INC(tcps_rcvbadoff); goto drop; } tlen -= off; /* tlen is used instead of ti->ti_len */ if (off > sizeof (struct tcphdr)) { #ifdef INET6 if (isipv6) { IP6_EXTHDR_CHECK(m, off0, off, IPPROTO_DONE); ip6 = mtod(m, struct ip6_hdr *); th = (struct tcphdr *)((caddr_t)ip6 + off0); } #endif #if defined(INET) && defined(INET6) else #endif #ifdef INET { if (m->m_len < sizeof(struct ip) + off) { if ((m = m_pullup(m, sizeof (struct ip) + off)) == NULL) { TCPSTAT_INC(tcps_rcvshort); return (IPPROTO_DONE); } ip = mtod(m, struct ip *); th = (struct tcphdr *)((caddr_t)ip + off0); } } #endif optlen = off - sizeof (struct tcphdr); optp = (u_char *)(th + 1); } thflags = th->th_flags; /* * Convert TCP protocol specific fields to host format. */ tcp_fields_to_host(th); /* * Delay dropping TCP, IP headers, IPv6 ext headers, and TCP options. */ drop_hdrlen = off0 + off; /* * Locate pcb for segment; if we're likely to add or remove a * connection then first acquire pcbinfo lock. There are three cases * where we might discover later we need a write lock despite the * flags: ACKs moving a connection out of the syncache, ACKs for a * connection in TIMEWAIT and SYNs not targeting a listening socket. */ if ((thflags & (TH_FIN | TH_RST)) != 0) { INP_INFO_WLOCK(&V_tcbinfo); ti_locked = TI_WLOCKED; } else ti_locked = TI_UNLOCKED; /* * Grab info from PACKET_TAG_IPFORWARD tag prepended to the chain. */ if ( #ifdef INET6 (isipv6 && (m->m_flags & M_IP6_NEXTHOP)) #ifdef INET || (!isipv6 && (m->m_flags & M_IP_NEXTHOP)) #endif #endif #if defined(INET) && !defined(INET6) (m->m_flags & M_IP_NEXTHOP) #endif ) fwd_tag = m_tag_find(m, PACKET_TAG_IPFORWARD, NULL); findpcb: #ifdef INVARIANTS if (ti_locked == TI_WLOCKED) { INP_INFO_WLOCK_ASSERT(&V_tcbinfo); } else { INP_INFO_UNLOCK_ASSERT(&V_tcbinfo); } #endif #ifdef INET6 if (isipv6 && fwd_tag != NULL) { struct sockaddr_in6 *next_hop6; next_hop6 = (struct sockaddr_in6 *)(fwd_tag + 1); /* * Transparently forwarded. Pretend to be the destination. * Already got one like this? */ inp = in6_pcblookup_mbuf(&V_tcbinfo, &ip6->ip6_src, th->th_sport, &ip6->ip6_dst, th->th_dport, INPLOOKUP_WLOCKPCB, m->m_pkthdr.rcvif, m); if (!inp) { /* * It's new. Try to find the ambushing socket. * Because we've rewritten the destination address, * any hardware-generated hash is ignored. */ inp = in6_pcblookup(&V_tcbinfo, &ip6->ip6_src, th->th_sport, &next_hop6->sin6_addr, next_hop6->sin6_port ? ntohs(next_hop6->sin6_port) : th->th_dport, INPLOOKUP_WILDCARD | INPLOOKUP_WLOCKPCB, m->m_pkthdr.rcvif); } } else if (isipv6) { inp = in6_pcblookup_mbuf(&V_tcbinfo, &ip6->ip6_src, th->th_sport, &ip6->ip6_dst, th->th_dport, INPLOOKUP_WILDCARD | INPLOOKUP_WLOCKPCB, m->m_pkthdr.rcvif, m); } #endif /* INET6 */ #if defined(INET6) && defined(INET) else #endif #ifdef INET if (fwd_tag != NULL) { struct sockaddr_in *next_hop; next_hop = (struct sockaddr_in *)(fwd_tag+1); /* * Transparently forwarded. Pretend to be the destination. * already got one like this? */ inp = in_pcblookup_mbuf(&V_tcbinfo, ip->ip_src, th->th_sport, ip->ip_dst, th->th_dport, INPLOOKUP_WLOCKPCB, m->m_pkthdr.rcvif, m); if (!inp) { /* * It's new. Try to find the ambushing socket. * Because we've rewritten the destination address, * any hardware-generated hash is ignored. */ inp = in_pcblookup(&V_tcbinfo, ip->ip_src, th->th_sport, next_hop->sin_addr, next_hop->sin_port ? ntohs(next_hop->sin_port) : th->th_dport, INPLOOKUP_WILDCARD | INPLOOKUP_WLOCKPCB, m->m_pkthdr.rcvif); } } else inp = in_pcblookup_mbuf(&V_tcbinfo, ip->ip_src, th->th_sport, ip->ip_dst, th->th_dport, INPLOOKUP_WILDCARD | INPLOOKUP_WLOCKPCB, m->m_pkthdr.rcvif, m); #endif /* INET */ /* * If the INPCB does not exist then all data in the incoming * segment is discarded and an appropriate RST is sent back. * XXX MRT Send RST using which routing table? */ if (inp == NULL) { /* * Log communication attempts to ports that are not * in use. */ if ((tcp_log_in_vain == 1 && (thflags & TH_SYN)) || tcp_log_in_vain == 2) { if ((s = tcp_log_vain(NULL, th, (void *)ip, ip6))) log(LOG_INFO, "%s; %s: Connection attempt " "to closed port\n", s, __func__); } /* * When blackholing do not respond with a RST but * completely ignore the segment and drop it. */ if ((V_blackhole == 1 && (thflags & TH_SYN)) || V_blackhole == 2) goto dropunlock; rstreason = BANDLIM_RST_CLOSEDPORT; goto dropwithreset; } INP_WLOCK_ASSERT(inp); if ((inp->inp_flowtype == M_HASHTYPE_NONE) && (M_HASHTYPE_GET(m) != M_HASHTYPE_NONE) && ((inp->inp_socket == NULL) || (inp->inp_socket->so_options & SO_ACCEPTCONN) == 0)) { inp->inp_flowid = m->m_pkthdr.flowid; inp->inp_flowtype = M_HASHTYPE_GET(m); } #ifdef IPSEC #ifdef INET6 if (isipv6 && ipsec6_in_reject(m, inp)) { - IPSEC6STAT_INC(ips_in_polvio); goto dropunlock; } else #endif /* INET6 */ if (ipsec4_in_reject(m, inp) != 0) { - IPSECSTAT_INC(ips_in_polvio); goto dropunlock; } #endif /* IPSEC */ /* * Check the minimum TTL for socket. */ if (inp->inp_ip_minttl != 0) { #ifdef INET6 if (isipv6 && inp->inp_ip_minttl > ip6->ip6_hlim) goto dropunlock; else #endif if (inp->inp_ip_minttl > ip->ip_ttl) goto dropunlock; } /* * A previous connection in TIMEWAIT state is supposed to catch stray * or duplicate segments arriving late. If this segment was a * legitimate new connection attempt, the old INPCB gets removed and * we can try again to find a listening socket. * * At this point, due to earlier optimism, we may hold only an inpcb * lock, and not the inpcbinfo write lock. If so, we need to try to * acquire it, or if that fails, acquire a reference on the inpcb, * drop all locks, acquire a global write lock, and then re-acquire * the inpcb lock. We may at that point discover that another thread * has tried to free the inpcb, in which case we need to loop back * and try to find a new inpcb to deliver to. * * XXXRW: It may be time to rethink timewait locking. */ relocked: if (inp->inp_flags & INP_TIMEWAIT) { if (ti_locked == TI_UNLOCKED) { if (INP_INFO_TRY_WLOCK(&V_tcbinfo) == 0) { in_pcbref(inp); INP_WUNLOCK(inp); INP_INFO_WLOCK(&V_tcbinfo); ti_locked = TI_WLOCKED; INP_WLOCK(inp); if (in_pcbrele_wlocked(inp)) { inp = NULL; goto findpcb; } } else ti_locked = TI_WLOCKED; } INP_INFO_WLOCK_ASSERT(&V_tcbinfo); if (thflags & TH_SYN) tcp_dooptions(&to, optp, optlen, TO_SYN); /* * NB: tcp_twcheck unlocks the INP and frees the mbuf. */ if (tcp_twcheck(inp, &to, th, m, tlen)) goto findpcb; INP_INFO_WUNLOCK(&V_tcbinfo); return (IPPROTO_DONE); } /* * The TCPCB may no longer exist if the connection is winding * down or it is in the CLOSED state. Either way we drop the * segment and send an appropriate response. */ tp = intotcpcb(inp); if (tp == NULL || tp->t_state == TCPS_CLOSED) { rstreason = BANDLIM_RST_CLOSEDPORT; goto dropwithreset; } #ifdef TCP_OFFLOAD if (tp->t_flags & TF_TOE) { tcp_offload_input(tp, m); m = NULL; /* consumed by the TOE driver */ goto dropunlock; } #endif /* * We've identified a valid inpcb, but it could be that we need an * inpcbinfo write lock but don't hold it. In this case, attempt to * acquire using the same strategy as the TIMEWAIT case above. If we * relock, we have to jump back to 'relocked' as the connection might * now be in TIMEWAIT. */ #ifdef INVARIANTS if ((thflags & (TH_FIN | TH_RST)) != 0) INP_INFO_WLOCK_ASSERT(&V_tcbinfo); #endif if (!((tp->t_state == TCPS_ESTABLISHED && (thflags & TH_SYN) == 0) || (tp->t_state == TCPS_LISTEN && (thflags & TH_SYN)))) { if (ti_locked == TI_UNLOCKED) { if (INP_INFO_TRY_WLOCK(&V_tcbinfo) == 0) { in_pcbref(inp); INP_WUNLOCK(inp); INP_INFO_WLOCK(&V_tcbinfo); ti_locked = TI_WLOCKED; INP_WLOCK(inp); if (in_pcbrele_wlocked(inp)) { inp = NULL; goto findpcb; } goto relocked; } else ti_locked = TI_WLOCKED; } INP_INFO_WLOCK_ASSERT(&V_tcbinfo); } #ifdef MAC INP_WLOCK_ASSERT(inp); if (mac_inpcb_check_deliver(inp, m)) goto dropunlock; #endif so = inp->inp_socket; KASSERT(so != NULL, ("%s: so == NULL", __func__)); #ifdef TCPDEBUG if (so->so_options & SO_DEBUG) { ostate = tp->t_state; #ifdef INET6 if (isipv6) { bcopy((char *)ip6, (char *)tcp_saveipgen, sizeof(*ip6)); } else #endif bcopy((char *)ip, (char *)tcp_saveipgen, sizeof(*ip)); tcp_savetcp = *th; } #endif /* TCPDEBUG */ /* * When the socket is accepting connections (the INPCB is in LISTEN * state) we look into the SYN cache if this is a new connection * attempt or the completion of a previous one. */ if (so->so_options & SO_ACCEPTCONN) { struct in_conninfo inc; KASSERT(tp->t_state == TCPS_LISTEN, ("%s: so accepting but " "tp not listening", __func__)); bzero(&inc, sizeof(inc)); #ifdef INET6 if (isipv6) { inc.inc_flags |= INC_ISIPV6; inc.inc6_faddr = ip6->ip6_src; inc.inc6_laddr = ip6->ip6_dst; } else #endif { inc.inc_faddr = ip->ip_src; inc.inc_laddr = ip->ip_dst; } inc.inc_fport = th->th_sport; inc.inc_lport = th->th_dport; inc.inc_fibnum = so->so_fibnum; /* * Check for an existing connection attempt in syncache if * the flag is only ACK. A successful lookup creates a new * socket appended to the listen queue in SYN_RECEIVED state. */ if ((thflags & (TH_RST|TH_ACK|TH_SYN)) == TH_ACK) { INP_INFO_WLOCK_ASSERT(&V_tcbinfo); /* * Parse the TCP options here because * syncookies need access to the reflected * timestamp. */ tcp_dooptions(&to, optp, optlen, 0); /* * NB: syncache_expand() doesn't unlock * inp and tcpinfo locks. */ if (!syncache_expand(&inc, &to, th, &so, m)) { /* * No syncache entry or ACK was not * for our SYN/ACK. Send a RST. * NB: syncache did its own logging * of the failure cause. */ rstreason = BANDLIM_RST_OPENPORT; goto dropwithreset; } if (so == NULL) { /* * We completed the 3-way handshake * but could not allocate a socket * either due to memory shortage, * listen queue length limits or * global socket limits. Send RST * or wait and have the remote end * retransmit the ACK for another * try. */ if ((s = tcp_log_addrs(&inc, th, NULL, NULL))) log(LOG_DEBUG, "%s; %s: Listen socket: " "Socket allocation failed due to " "limits or memory shortage, %s\n", s, __func__, V_tcp_sc_rst_sock_fail ? "sending RST" : "try again"); if (V_tcp_sc_rst_sock_fail) { rstreason = BANDLIM_UNLIMITED; goto dropwithreset; } else goto dropunlock; } /* * Socket is created in state SYN_RECEIVED. * Unlock the listen socket, lock the newly * created socket and update the tp variable. */ INP_WUNLOCK(inp); /* listen socket */ inp = sotoinpcb(so); INP_WLOCK(inp); /* new connection */ tp = intotcpcb(inp); KASSERT(tp->t_state == TCPS_SYN_RECEIVED, ("%s: ", __func__)); #ifdef TCP_SIGNATURE if (sig_checked == 0) { tcp_dooptions(&to, optp, optlen, (thflags & TH_SYN) ? TO_SYN : 0); if (!tcp_signature_verify_input(m, off0, tlen, optlen, &to, th, tp->t_flags)) { /* * In SYN_SENT state if it receives an * RST, it is allowed for further * processing. */ if ((thflags & TH_RST) == 0 || (tp->t_state == TCPS_SYN_SENT) == 0) goto dropunlock; } sig_checked = 1; } #endif /* * Process the segment and the data it * contains. tcp_do_segment() consumes * the mbuf chain and unlocks the inpcb. */ tcp_do_segment(m, th, so, tp, drop_hdrlen, tlen, iptos, ti_locked); INP_INFO_UNLOCK_ASSERT(&V_tcbinfo); return (IPPROTO_DONE); } /* * Segment flag validation for new connection attempts: * * Our (SYN|ACK) response was rejected. * Check with syncache and remove entry to prevent * retransmits. * * NB: syncache_chkrst does its own logging of failure * causes. */ if (thflags & TH_RST) { syncache_chkrst(&inc, th); goto dropunlock; } /* * We can't do anything without SYN. */ if ((thflags & TH_SYN) == 0) { if ((s = tcp_log_addrs(&inc, th, NULL, NULL))) log(LOG_DEBUG, "%s; %s: Listen socket: " "SYN is missing, segment ignored\n", s, __func__); TCPSTAT_INC(tcps_badsyn); goto dropunlock; } /* * (SYN|ACK) is bogus on a listen socket. */ if (thflags & TH_ACK) { if ((s = tcp_log_addrs(&inc, th, NULL, NULL))) log(LOG_DEBUG, "%s; %s: Listen socket: " "SYN|ACK invalid, segment rejected\n", s, __func__); syncache_badack(&inc); /* XXX: Not needed! */ TCPSTAT_INC(tcps_badsyn); rstreason = BANDLIM_RST_OPENPORT; goto dropwithreset; } /* * If the drop_synfin option is enabled, drop all * segments with both the SYN and FIN bits set. * This prevents e.g. nmap from identifying the * TCP/IP stack. * XXX: Poor reasoning. nmap has other methods * and is constantly refining its stack detection * strategies. * XXX: This is a violation of the TCP specification * and was used by RFC1644. */ if ((thflags & TH_FIN) && V_drop_synfin) { if ((s = tcp_log_addrs(&inc, th, NULL, NULL))) log(LOG_DEBUG, "%s; %s: Listen socket: " "SYN|FIN segment ignored (based on " "sysctl setting)\n", s, __func__); TCPSTAT_INC(tcps_badsyn); goto dropunlock; } /* * Segment's flags are (SYN) or (SYN|FIN). * * TH_PUSH, TH_URG, TH_ECE, TH_CWR are ignored * as they do not affect the state of the TCP FSM. * The data pointed to by TH_URG and th_urp is ignored. */ KASSERT((thflags & (TH_RST|TH_ACK)) == 0, ("%s: Listen socket: TH_RST or TH_ACK set", __func__)); KASSERT(thflags & (TH_SYN), ("%s: Listen socket: TH_SYN not set", __func__)); #ifdef INET6 /* * If deprecated address is forbidden, * we do not accept SYN to deprecated interface * address to prevent any new inbound connection from * getting established. * When we do not accept SYN, we send a TCP RST, * with deprecated source address (instead of dropping * it). We compromise it as it is much better for peer * to send a RST, and RST will be the final packet * for the exchange. * * If we do not forbid deprecated addresses, we accept * the SYN packet. RFC2462 does not suggest dropping * SYN in this case. * If we decipher RFC2462 5.5.4, it says like this: * 1. use of deprecated addr with existing * communication is okay - "SHOULD continue to be * used" * 2. use of it with new communication: * (2a) "SHOULD NOT be used if alternate address * with sufficient scope is available" * (2b) nothing mentioned otherwise. * Here we fall into (2b) case as we have no choice in * our source address selection - we must obey the peer. * * The wording in RFC2462 is confusing, and there are * multiple description text for deprecated address * handling - worse, they are not exactly the same. * I believe 5.5.4 is the best one, so we follow 5.5.4. */ if (isipv6 && !V_ip6_use_deprecated) { struct in6_ifaddr *ia6; ia6 = in6ifa_ifwithaddr(&ip6->ip6_dst, 0 /* XXX */); if (ia6 != NULL && (ia6->ia6_flags & IN6_IFF_DEPRECATED)) { ifa_free(&ia6->ia_ifa); if ((s = tcp_log_addrs(&inc, th, NULL, NULL))) log(LOG_DEBUG, "%s; %s: Listen socket: " "Connection attempt to deprecated " "IPv6 address rejected\n", s, __func__); rstreason = BANDLIM_RST_OPENPORT; goto dropwithreset; } if (ia6) ifa_free(&ia6->ia_ifa); } #endif /* INET6 */ /* * Basic sanity checks on incoming SYN requests: * Don't respond if the destination is a link layer * broadcast according to RFC1122 4.2.3.10, p. 104. * If it is from this socket it must be forged. * Don't respond if the source or destination is a * global or subnet broad- or multicast address. * Note that it is quite possible to receive unicast * link-layer packets with a broadcast IP address. Use * in_broadcast() to find them. */ if (m->m_flags & (M_BCAST|M_MCAST)) { if ((s = tcp_log_addrs(&inc, th, NULL, NULL))) log(LOG_DEBUG, "%s; %s: Listen socket: " "Connection attempt from broad- or multicast " "link layer address ignored\n", s, __func__); goto dropunlock; } #ifdef INET6 if (isipv6) { if (th->th_dport == th->th_sport && IN6_ARE_ADDR_EQUAL(&ip6->ip6_dst, &ip6->ip6_src)) { if ((s = tcp_log_addrs(&inc, th, NULL, NULL))) log(LOG_DEBUG, "%s; %s: Listen socket: " "Connection attempt to/from self " "ignored\n", s, __func__); goto dropunlock; } if (IN6_IS_ADDR_MULTICAST(&ip6->ip6_dst) || IN6_IS_ADDR_MULTICAST(&ip6->ip6_src)) { if ((s = tcp_log_addrs(&inc, th, NULL, NULL))) log(LOG_DEBUG, "%s; %s: Listen socket: " "Connection attempt from/to multicast " "address ignored\n", s, __func__); goto dropunlock; } } #endif #if defined(INET) && defined(INET6) else #endif #ifdef INET { if (th->th_dport == th->th_sport && ip->ip_dst.s_addr == ip->ip_src.s_addr) { if ((s = tcp_log_addrs(&inc, th, NULL, NULL))) log(LOG_DEBUG, "%s; %s: Listen socket: " "Connection attempt from/to self " "ignored\n", s, __func__); goto dropunlock; } if (IN_MULTICAST(ntohl(ip->ip_dst.s_addr)) || IN_MULTICAST(ntohl(ip->ip_src.s_addr)) || ip->ip_src.s_addr == htonl(INADDR_BROADCAST) || in_broadcast(ip->ip_dst, m->m_pkthdr.rcvif)) { if ((s = tcp_log_addrs(&inc, th, NULL, NULL))) log(LOG_DEBUG, "%s; %s: Listen socket: " "Connection attempt from/to broad- " "or multicast address ignored\n", s, __func__); goto dropunlock; } } #endif /* * SYN appears to be valid. Create compressed TCP state * for syncache. */ #ifdef TCPDEBUG if (so->so_options & SO_DEBUG) tcp_trace(TA_INPUT, ostate, tp, (void *)tcp_saveipgen, &tcp_savetcp, 0); #endif tcp_dooptions(&to, optp, optlen, TO_SYN); syncache_add(&inc, &to, th, inp, &so, m, NULL, NULL); /* * Entry added to syncache and mbuf consumed. * Only the listen socket is unlocked by syncache_add(). */ if (ti_locked == TI_WLOCKED) { INP_INFO_WUNLOCK(&V_tcbinfo); ti_locked = TI_UNLOCKED; } INP_INFO_UNLOCK_ASSERT(&V_tcbinfo); return (IPPROTO_DONE); } else if (tp->t_state == TCPS_LISTEN) { /* * When a listen socket is torn down the SO_ACCEPTCONN * flag is removed first while connections are drained * from the accept queue in a unlock/lock cycle of the * ACCEPT_LOCK, opening a race condition allowing a SYN * attempt go through unhandled. */ goto dropunlock; } #ifdef TCP_SIGNATURE if (sig_checked == 0) { tcp_dooptions(&to, optp, optlen, (thflags & TH_SYN) ? TO_SYN : 0); if (!tcp_signature_verify_input(m, off0, tlen, optlen, &to, th, tp->t_flags)) { /* * In SYN_SENT state if it receives an RST, it is * allowed for further processing. */ if ((thflags & TH_RST) == 0 || (tp->t_state == TCPS_SYN_SENT) == 0) goto dropunlock; } sig_checked = 1; } #endif TCP_PROBE5(receive, NULL, tp, mtod(m, const char *), tp, th); /* * Segment belongs to a connection in SYN_SENT, ESTABLISHED or later * state. tcp_do_segment() always consumes the mbuf chain, unlocks * the inpcb, and unlocks pcbinfo. */ tcp_do_segment(m, th, so, tp, drop_hdrlen, tlen, iptos, ti_locked); INP_INFO_UNLOCK_ASSERT(&V_tcbinfo); return (IPPROTO_DONE); dropwithreset: TCP_PROBE5(receive, NULL, tp, mtod(m, const char *), tp, th); if (ti_locked == TI_WLOCKED) { INP_INFO_WUNLOCK(&V_tcbinfo); ti_locked = TI_UNLOCKED; } #ifdef INVARIANTS else { KASSERT(ti_locked == TI_UNLOCKED, ("%s: dropwithreset " "ti_locked: %d", __func__, ti_locked)); INP_INFO_UNLOCK_ASSERT(&V_tcbinfo); } #endif if (inp != NULL) { tcp_dropwithreset(m, th, tp, tlen, rstreason); INP_WUNLOCK(inp); } else tcp_dropwithreset(m, th, NULL, tlen, rstreason); m = NULL; /* mbuf chain got consumed. */ goto drop; dropunlock: if (m != NULL) TCP_PROBE5(receive, NULL, tp, mtod(m, const char *), tp, th); if (ti_locked == TI_WLOCKED) { INP_INFO_WUNLOCK(&V_tcbinfo); ti_locked = TI_UNLOCKED; } #ifdef INVARIANTS else { KASSERT(ti_locked == TI_UNLOCKED, ("%s: dropunlock " "ti_locked: %d", __func__, ti_locked)); INP_INFO_UNLOCK_ASSERT(&V_tcbinfo); } #endif if (inp != NULL) INP_WUNLOCK(inp); drop: INP_INFO_UNLOCK_ASSERT(&V_tcbinfo); if (s != NULL) free(s, M_TCPLOG); if (m != NULL) m_freem(m); return (IPPROTO_DONE); } static void tcp_do_segment(struct mbuf *m, struct tcphdr *th, struct socket *so, struct tcpcb *tp, int drop_hdrlen, int tlen, uint8_t iptos, int ti_locked) { int thflags, acked, ourfinisacked, needoutput = 0; int rstreason, todrop, win; u_long tiwin; char *s; struct in_conninfo *inc; struct mbuf *mfree; struct tcpopt to; #ifdef TCPDEBUG /* * The size of tcp_saveipgen must be the size of the max ip header, * now IPv6. */ u_char tcp_saveipgen[IP6_HDR_LEN]; struct tcphdr tcp_savetcp; short ostate = 0; #endif thflags = th->th_flags; inc = &tp->t_inpcb->inp_inc; tp->sackhint.last_sack_ack = 0; /* * If this is either a state-changing packet or current state isn't * established, we require a write lock on tcbinfo. Otherwise, we * allow the tcbinfo to be in either alocked or unlocked, as the * caller may have unnecessarily acquired a write lock due to a race. */ if ((thflags & (TH_SYN | TH_FIN | TH_RST)) != 0 || tp->t_state != TCPS_ESTABLISHED) { KASSERT(ti_locked == TI_WLOCKED, ("%s ti_locked %d for " "SYN/FIN/RST/!EST", __func__, ti_locked)); INP_INFO_WLOCK_ASSERT(&V_tcbinfo); } else { #ifdef INVARIANTS if (ti_locked == TI_WLOCKED) INP_INFO_WLOCK_ASSERT(&V_tcbinfo); else { KASSERT(ti_locked == TI_UNLOCKED, ("%s: EST " "ti_locked: %d", __func__, ti_locked)); INP_INFO_UNLOCK_ASSERT(&V_tcbinfo); } #endif } INP_WLOCK_ASSERT(tp->t_inpcb); KASSERT(tp->t_state > TCPS_LISTEN, ("%s: TCPS_LISTEN", __func__)); KASSERT(tp->t_state != TCPS_TIME_WAIT, ("%s: TCPS_TIME_WAIT", __func__)); /* * Segment received on connection. * Reset idle time and keep-alive timer. * XXX: This should be done after segment * validation to ignore broken/spoofed segs. */ tp->t_rcvtime = ticks; if (TCPS_HAVEESTABLISHED(tp->t_state)) tcp_timer_activate(tp, TT_KEEP, TP_KEEPIDLE(tp)); /* * Unscale the window into a 32-bit value. * For the SYN_SENT state the scale is zero. */ tiwin = th->th_win << tp->snd_scale; /* * TCP ECN processing. */ if (tp->t_flags & TF_ECN_PERMIT) { if (thflags & TH_CWR) tp->t_flags &= ~TF_ECN_SND_ECE; switch (iptos & IPTOS_ECN_MASK) { case IPTOS_ECN_CE: tp->t_flags |= TF_ECN_SND_ECE; TCPSTAT_INC(tcps_ecn_ce); break; case IPTOS_ECN_ECT0: TCPSTAT_INC(tcps_ecn_ect0); break; case IPTOS_ECN_ECT1: TCPSTAT_INC(tcps_ecn_ect1); break; } /* Congestion experienced. */ if (thflags & TH_ECE) { cc_cong_signal(tp, th, CC_ECN); } } /* * Parse options on any incoming segment. */ tcp_dooptions(&to, (u_char *)(th + 1), (th->th_off << 2) - sizeof(struct tcphdr), (thflags & TH_SYN) ? TO_SYN : 0); /* * If echoed timestamp is later than the current time, * fall back to non RFC1323 RTT calculation. Normalize * timestamp if syncookies were used when this connection * was established. */ if ((to.to_flags & TOF_TS) && (to.to_tsecr != 0)) { to.to_tsecr -= tp->ts_offset; if (TSTMP_GT(to.to_tsecr, tcp_ts_getticks())) to.to_tsecr = 0; } /* * If timestamps were negotiated during SYN/ACK they should * appear on every segment during this session and vice versa. */ if ((tp->t_flags & TF_RCVD_TSTMP) && !(to.to_flags & TOF_TS)) { if ((s = tcp_log_addrs(inc, th, NULL, NULL))) { log(LOG_DEBUG, "%s; %s: Timestamp missing, " "no action\n", s, __func__); free(s, M_TCPLOG); } } if (!(tp->t_flags & TF_RCVD_TSTMP) && (to.to_flags & TOF_TS)) { if ((s = tcp_log_addrs(inc, th, NULL, NULL))) { log(LOG_DEBUG, "%s; %s: Timestamp not expected, " "no action\n", s, __func__); free(s, M_TCPLOG); } } /* * Process options only when we get SYN/ACK back. The SYN case * for incoming connections is handled in tcp_syncache. * According to RFC1323 the window field in a SYN (i.e., a * or ) segment itself is never scaled. * XXX this is traditional behavior, may need to be cleaned up. */ if (tp->t_state == TCPS_SYN_SENT && (thflags & TH_SYN)) { if ((to.to_flags & TOF_SCALE) && (tp->t_flags & TF_REQ_SCALE)) { tp->t_flags |= TF_RCVD_SCALE; tp->snd_scale = to.to_wscale; } /* * Initial send window. It will be updated with * the next incoming segment to the scaled value. */ tp->snd_wnd = th->th_win; if (to.to_flags & TOF_TS) { tp->t_flags |= TF_RCVD_TSTMP; tp->ts_recent = to.to_tsval; tp->ts_recent_age = tcp_ts_getticks(); } if (to.to_flags & TOF_MSS) tcp_mss(tp, to.to_mss); if ((tp->t_flags & TF_SACK_PERMIT) && (to.to_flags & TOF_SACKPERM) == 0) tp->t_flags &= ~TF_SACK_PERMIT; } /* * Header prediction: check for the two common cases * of a uni-directional data xfer. If the packet has * no control flags, is in-sequence, the window didn't * change and we're not retransmitting, it's a * candidate. If the length is zero and the ack moved * forward, we're the sender side of the xfer. Just * free the data acked & wake any higher level process * that was blocked waiting for space. If the length * is non-zero and the ack didn't move, we're the * receiver side. If we're getting packets in-order * (the reassembly queue is empty), add the data to * the socket buffer and note that we need a delayed ack. * Make sure that the hidden state-flags are also off. * Since we check for TCPS_ESTABLISHED first, it can only * be TH_NEEDSYN. */ if (tp->t_state == TCPS_ESTABLISHED && th->th_seq == tp->rcv_nxt && (thflags & (TH_SYN|TH_FIN|TH_RST|TH_URG|TH_ACK)) == TH_ACK && tp->snd_nxt == tp->snd_max && tiwin && tiwin == tp->snd_wnd && ((tp->t_flags & (TF_NEEDSYN|TF_NEEDFIN)) == 0) && tp->t_segq == NULL && ((to.to_flags & TOF_TS) == 0 || TSTMP_GEQ(to.to_tsval, tp->ts_recent)) ) { /* * If last ACK falls within this segment's sequence numbers, * record the timestamp. * NOTE that the test is modified according to the latest * proposal of the tcplw@cray.com list (Braden 1993/04/26). */ if ((to.to_flags & TOF_TS) != 0 && SEQ_LEQ(th->th_seq, tp->last_ack_sent)) { tp->ts_recent_age = tcp_ts_getticks(); tp->ts_recent = to.to_tsval; } if (tlen == 0) { if (SEQ_GT(th->th_ack, tp->snd_una) && SEQ_LEQ(th->th_ack, tp->snd_max) && !IN_RECOVERY(tp->t_flags) && (to.to_flags & TOF_SACK) == 0 && TAILQ_EMPTY(&tp->snd_holes)) { /* * This is a pure ack for outstanding data. */ if (ti_locked == TI_WLOCKED) INP_INFO_WUNLOCK(&V_tcbinfo); ti_locked = TI_UNLOCKED; TCPSTAT_INC(tcps_predack); /* * "bad retransmit" recovery. */ if (tp->t_rxtshift == 1 && tp->t_flags & TF_PREVVALID && (int)(ticks - tp->t_badrxtwin) < 0) { cc_cong_signal(tp, th, CC_RTO_ERR); } /* * Recalculate the transmit timer / rtt. * * Some boxes send broken timestamp replies * during the SYN+ACK phase, ignore * timestamps of 0 or we could calculate a * huge RTT and blow up the retransmit timer. */ if ((to.to_flags & TOF_TS) != 0 && to.to_tsecr) { u_int t; t = tcp_ts_getticks() - to.to_tsecr; if (!tp->t_rttlow || tp->t_rttlow > t) tp->t_rttlow = t; tcp_xmit_timer(tp, TCP_TS_TO_TICKS(t) + 1); } else if (tp->t_rtttime && SEQ_GT(th->th_ack, tp->t_rtseq)) { if (!tp->t_rttlow || tp->t_rttlow > ticks - tp->t_rtttime) tp->t_rttlow = ticks - tp->t_rtttime; tcp_xmit_timer(tp, ticks - tp->t_rtttime); } acked = BYTES_THIS_ACK(tp, th); /* Run HHOOK_TCP_ESTABLISHED_IN helper hooks. */ hhook_run_tcp_est_in(tp, th, &to); TCPSTAT_INC(tcps_rcvackpack); TCPSTAT_ADD(tcps_rcvackbyte, acked); sbdrop(&so->so_snd, acked); if (SEQ_GT(tp->snd_una, tp->snd_recover) && SEQ_LEQ(th->th_ack, tp->snd_recover)) tp->snd_recover = th->th_ack - 1; /* * Let the congestion control algorithm update * congestion control related information. This * typically means increasing the congestion * window. */ cc_ack_received(tp, th, CC_ACK); tp->snd_una = th->th_ack; /* * Pull snd_wl2 up to prevent seq wrap relative * to th_ack. */ tp->snd_wl2 = th->th_ack; tp->t_dupacks = 0; m_freem(m); ND6_HINT(tp); /* Some progress has been made. */ /* * If all outstanding data are acked, stop * retransmit timer, otherwise restart timer * using current (possibly backed-off) value. * If process is waiting for space, * wakeup/selwakeup/signal. If data * are ready to send, let tcp_output * decide between more output or persist. */ #ifdef TCPDEBUG if (so->so_options & SO_DEBUG) tcp_trace(TA_INPUT, ostate, tp, (void *)tcp_saveipgen, &tcp_savetcp, 0); #endif if (tp->snd_una == tp->snd_max) tcp_timer_activate(tp, TT_REXMT, 0); else if (!tcp_timer_active(tp, TT_PERSIST)) tcp_timer_activate(tp, TT_REXMT, tp->t_rxtcur); sowwakeup(so); if (sbavail(&so->so_snd)) (void) tcp_output(tp); goto check_delack; } } else if (th->th_ack == tp->snd_una && tlen <= sbspace(&so->so_rcv)) { int newsize = 0; /* automatic sockbuf scaling */ /* * This is a pure, in-sequence data packet with * nothing on the reassembly queue and we have enough * buffer space to take it. */ if (ti_locked == TI_WLOCKED) INP_INFO_WUNLOCK(&V_tcbinfo); ti_locked = TI_UNLOCKED; /* Clean receiver SACK report if present */ if ((tp->t_flags & TF_SACK_PERMIT) && tp->rcv_numsacks) tcp_clean_sackreport(tp); TCPSTAT_INC(tcps_preddat); tp->rcv_nxt += tlen; /* * Pull snd_wl1 up to prevent seq wrap relative to * th_seq. */ tp->snd_wl1 = th->th_seq; /* * Pull rcv_up up to prevent seq wrap relative to * rcv_nxt. */ tp->rcv_up = tp->rcv_nxt; TCPSTAT_INC(tcps_rcvpack); TCPSTAT_ADD(tcps_rcvbyte, tlen); ND6_HINT(tp); /* Some progress has been made */ #ifdef TCPDEBUG if (so->so_options & SO_DEBUG) tcp_trace(TA_INPUT, ostate, tp, (void *)tcp_saveipgen, &tcp_savetcp, 0); #endif /* * Automatic sizing of receive socket buffer. Often the send * buffer size is not optimally adjusted to the actual network * conditions at hand (delay bandwidth product). Setting the * buffer size too small limits throughput on links with high * bandwidth and high delay (eg. trans-continental/oceanic links). * * On the receive side the socket buffer memory is only rarely * used to any significant extent. This allows us to be much * more aggressive in scaling the receive socket buffer. For * the case that the buffer space is actually used to a large * extent and we run out of kernel memory we can simply drop * the new segments; TCP on the sender will just retransmit it * later. Setting the buffer size too big may only consume too * much kernel memory if the application doesn't read() from * the socket or packet loss or reordering makes use of the * reassembly queue. * * The criteria to step up the receive buffer one notch are: * 1. Application has not set receive buffer size with * SO_RCVBUF. Setting SO_RCVBUF clears SB_AUTOSIZE. * 2. the number of bytes received during the time it takes * one timestamp to be reflected back to us (the RTT); * 3. received bytes per RTT is within seven eighth of the * current socket buffer size; * 4. receive buffer size has not hit maximal automatic size; * * This algorithm does one step per RTT at most and only if * we receive a bulk stream w/o packet losses or reorderings. * Shrinking the buffer during idle times is not necessary as * it doesn't consume any memory when idle. * * TODO: Only step up if the application is actually serving * the buffer to better manage the socket buffer resources. */ if (V_tcp_do_autorcvbuf && to.to_tsecr && (so->so_rcv.sb_flags & SB_AUTOSIZE)) { if (TSTMP_GT(to.to_tsecr, tp->rfbuf_ts) && to.to_tsecr - tp->rfbuf_ts < hz) { if (tp->rfbuf_cnt > (so->so_rcv.sb_hiwat / 8 * 7) && so->so_rcv.sb_hiwat < V_tcp_autorcvbuf_max) { newsize = min(so->so_rcv.sb_hiwat + V_tcp_autorcvbuf_inc, V_tcp_autorcvbuf_max); } /* Start over with next RTT. */ tp->rfbuf_ts = 0; tp->rfbuf_cnt = 0; } else tp->rfbuf_cnt += tlen; /* add up */ } /* Add data to socket buffer. */ SOCKBUF_LOCK(&so->so_rcv); if (so->so_rcv.sb_state & SBS_CANTRCVMORE) { m_freem(m); } else { /* * Set new socket buffer size. * Give up when limit is reached. */ if (newsize) if (!sbreserve_locked(&so->so_rcv, newsize, so, NULL)) so->so_rcv.sb_flags &= ~SB_AUTOSIZE; m_adj(m, drop_hdrlen); /* delayed header drop */ sbappendstream_locked(&so->so_rcv, m, 0); } /* NB: sorwakeup_locked() does an implicit unlock. */ sorwakeup_locked(so); if (DELAY_ACK(tp, tlen)) { tp->t_flags |= TF_DELACK; } else { tp->t_flags |= TF_ACKNOW; tcp_output(tp); } goto check_delack; } } /* * Calculate amount of space in receive window, * and then do TCP input processing. * Receive window is amount of space in rcv queue, * but not less than advertised window. */ win = sbspace(&so->so_rcv); if (win < 0) win = 0; tp->rcv_wnd = imax(win, (int)(tp->rcv_adv - tp->rcv_nxt)); /* Reset receive buffer auto scaling when not in bulk receive mode. */ tp->rfbuf_ts = 0; tp->rfbuf_cnt = 0; switch (tp->t_state) { /* * If the state is SYN_RECEIVED: * if seg contains an ACK, but not for our SYN/ACK, send a RST. */ case TCPS_SYN_RECEIVED: if ((thflags & TH_ACK) && (SEQ_LEQ(th->th_ack, tp->snd_una) || SEQ_GT(th->th_ack, tp->snd_max))) { rstreason = BANDLIM_RST_OPENPORT; goto dropwithreset; } break; /* * If the state is SYN_SENT: * if seg contains an ACK, but not for our SYN, drop the input. * if seg contains a RST, then drop the connection. * if seg does not contain SYN, then drop it. * Otherwise this is an acceptable SYN segment * initialize tp->rcv_nxt and tp->irs * if seg contains ack then advance tp->snd_una * if seg contains an ECE and ECN support is enabled, the stream * is ECN capable. * if SYN has been acked change to ESTABLISHED else SYN_RCVD state * arrange for segment to be acked (eventually) * continue processing rest of data/controls, beginning with URG */ case TCPS_SYN_SENT: if ((thflags & TH_ACK) && (SEQ_LEQ(th->th_ack, tp->iss) || SEQ_GT(th->th_ack, tp->snd_max))) { rstreason = BANDLIM_UNLIMITED; goto dropwithreset; } if ((thflags & (TH_ACK|TH_RST)) == (TH_ACK|TH_RST)) { TCP_PROBE5(connect__refused, NULL, tp, mtod(m, const char *), tp, th); tp = tcp_drop(tp, ECONNREFUSED); } if (thflags & TH_RST) goto drop; if (!(thflags & TH_SYN)) goto drop; tp->irs = th->th_seq; tcp_rcvseqinit(tp); if (thflags & TH_ACK) { TCPSTAT_INC(tcps_connects); soisconnected(so); #ifdef MAC mac_socketpeer_set_from_mbuf(m, so); #endif /* Do window scaling on this connection? */ if ((tp->t_flags & (TF_RCVD_SCALE|TF_REQ_SCALE)) == (TF_RCVD_SCALE|TF_REQ_SCALE)) { tp->rcv_scale = tp->request_r_scale; } tp->rcv_adv += imin(tp->rcv_wnd, TCP_MAXWIN << tp->rcv_scale); tp->snd_una++; /* SYN is acked */ /* * If there's data, delay ACK; if there's also a FIN * ACKNOW will be turned on later. */ if (DELAY_ACK(tp, tlen) && tlen != 0) tcp_timer_activate(tp, TT_DELACK, tcp_delacktime); else tp->t_flags |= TF_ACKNOW; if ((thflags & TH_ECE) && V_tcp_do_ecn) { tp->t_flags |= TF_ECN_PERMIT; TCPSTAT_INC(tcps_ecn_shs); } /* * Received in SYN_SENT[*] state. * Transitions: * SYN_SENT --> ESTABLISHED * SYN_SENT* --> FIN_WAIT_1 */ tp->t_starttime = ticks; if (tp->t_flags & TF_NEEDFIN) { tcp_state_change(tp, TCPS_FIN_WAIT_1); tp->t_flags &= ~TF_NEEDFIN; thflags &= ~TH_SYN; } else { tcp_state_change(tp, TCPS_ESTABLISHED); TCP_PROBE5(connect__established, NULL, tp, mtod(m, const char *), tp, th); cc_conn_init(tp); tcp_timer_activate(tp, TT_KEEP, TP_KEEPIDLE(tp)); } } else { /* * Received initial SYN in SYN-SENT[*] state => * simultaneous open. * If it succeeds, connection is * half-synchronized. * Otherwise, do 3-way handshake: * SYN-SENT -> SYN-RECEIVED * SYN-SENT* -> SYN-RECEIVED* */ tp->t_flags |= (TF_ACKNOW | TF_NEEDSYN); tcp_timer_activate(tp, TT_REXMT, 0); tcp_state_change(tp, TCPS_SYN_RECEIVED); } KASSERT(ti_locked == TI_WLOCKED, ("%s: trimthenstep6: " "ti_locked %d", __func__, ti_locked)); INP_INFO_WLOCK_ASSERT(&V_tcbinfo); INP_WLOCK_ASSERT(tp->t_inpcb); /* * Advance th->th_seq to correspond to first data byte. * If data, trim to stay within window, * dropping FIN if necessary. */ th->th_seq++; if (tlen > tp->rcv_wnd) { todrop = tlen - tp->rcv_wnd; m_adj(m, -todrop); tlen = tp->rcv_wnd; thflags &= ~TH_FIN; TCPSTAT_INC(tcps_rcvpackafterwin); TCPSTAT_ADD(tcps_rcvbyteafterwin, todrop); } tp->snd_wl1 = th->th_seq - 1; tp->rcv_up = th->th_seq; /* * Client side of transaction: already sent SYN and data. * If the remote host used T/TCP to validate the SYN, * our data will be ACK'd; if so, enter normal data segment * processing in the middle of step 5, ack processing. * Otherwise, goto step 6. */ if (thflags & TH_ACK) goto process_ACK; goto step6; /* * If the state is LAST_ACK or CLOSING or TIME_WAIT: * do normal processing. * * NB: Leftover from RFC1644 T/TCP. Cases to be reused later. */ case TCPS_LAST_ACK: case TCPS_CLOSING: break; /* continue normal processing */ } /* * States other than LISTEN or SYN_SENT. * First check the RST flag and sequence number since reset segments * are exempt from the timestamp and connection count tests. This * fixes a bug introduced by the Stevens, vol. 2, p. 960 bugfix * below which allowed reset segments in half the sequence space * to fall though and be processed (which gives forged reset * segments with a random sequence number a 50 percent chance of * killing a connection). * Then check timestamp, if present. * Then check the connection count, if present. * Then check that at least some bytes of segment are within * receive window. If segment begins before rcv_nxt, * drop leading data (and SYN); if nothing left, just ack. */ if (thflags & TH_RST) { /* * RFC5961 Section 3.2 * * - RST drops connection only if SEG.SEQ == RCV.NXT. * - If RST is in window, we send challenge ACK. * * Note: to take into account delayed ACKs, we should * test against last_ack_sent instead of rcv_nxt. * Note 2: we handle special case of closed window, not * covered by the RFC. */ if ((SEQ_GEQ(th->th_seq, tp->last_ack_sent) && SEQ_LT(th->th_seq, tp->last_ack_sent + tp->rcv_wnd)) || (tp->rcv_wnd == 0 && tp->last_ack_sent == th->th_seq)) { INP_INFO_WLOCK_ASSERT(&V_tcbinfo); KASSERT(ti_locked == TI_WLOCKED, ("%s: TH_RST ti_locked %d, th %p tp %p", __func__, ti_locked, th, tp)); KASSERT(tp->t_state != TCPS_SYN_SENT, ("%s: TH_RST for TCPS_SYN_SENT th %p tp %p", __func__, th, tp)); if (V_tcp_insecure_rst || tp->last_ack_sent == th->th_seq) { TCPSTAT_INC(tcps_drops); /* Drop the connection. */ switch (tp->t_state) { case TCPS_SYN_RECEIVED: so->so_error = ECONNREFUSED; goto close; case TCPS_ESTABLISHED: case TCPS_FIN_WAIT_1: case TCPS_FIN_WAIT_2: case TCPS_CLOSE_WAIT: so->so_error = ECONNRESET; close: tcp_state_change(tp, TCPS_CLOSED); /* FALLTHROUGH */ default: tp = tcp_close(tp); } } else { TCPSTAT_INC(tcps_badrst); /* Send challenge ACK. */ tcp_respond(tp, mtod(m, void *), th, m, tp->rcv_nxt, tp->snd_nxt, TH_ACK); tp->last_ack_sent = tp->rcv_nxt; m = NULL; } } goto drop; } /* * RFC5961 Section 4.2 * Send challenge ACK for any SYN in synchronized state. */ if ((thflags & TH_SYN) && tp->t_state != TCPS_SYN_SENT) { KASSERT(ti_locked == TI_WLOCKED, ("tcp_do_segment: TH_SYN ti_locked %d", ti_locked)); INP_INFO_WLOCK_ASSERT(&V_tcbinfo); TCPSTAT_INC(tcps_badsyn); if (V_tcp_insecure_syn && SEQ_GEQ(th->th_seq, tp->last_ack_sent) && SEQ_LT(th->th_seq, tp->last_ack_sent + tp->rcv_wnd)) { tp = tcp_drop(tp, ECONNRESET); rstreason = BANDLIM_UNLIMITED; } else { /* Send challenge ACK. */ tcp_respond(tp, mtod(m, void *), th, m, tp->rcv_nxt, tp->snd_nxt, TH_ACK); tp->last_ack_sent = tp->rcv_nxt; m = NULL; } goto drop; } /* * RFC 1323 PAWS: If we have a timestamp reply on this segment * and it's less than ts_recent, drop it. */ if ((to.to_flags & TOF_TS) != 0 && tp->ts_recent && TSTMP_LT(to.to_tsval, tp->ts_recent)) { /* Check to see if ts_recent is over 24 days old. */ if (tcp_ts_getticks() - tp->ts_recent_age > TCP_PAWS_IDLE) { /* * Invalidate ts_recent. If this segment updates * ts_recent, the age will be reset later and ts_recent * will get a valid value. If it does not, setting * ts_recent to zero will at least satisfy the * requirement that zero be placed in the timestamp * echo reply when ts_recent isn't valid. The * age isn't reset until we get a valid ts_recent * because we don't want out-of-order segments to be * dropped when ts_recent is old. */ tp->ts_recent = 0; } else { TCPSTAT_INC(tcps_rcvduppack); TCPSTAT_ADD(tcps_rcvdupbyte, tlen); TCPSTAT_INC(tcps_pawsdrop); if (tlen) goto dropafterack; goto drop; } } /* * In the SYN-RECEIVED state, validate that the packet belongs to * this connection before trimming the data to fit the receive * window. Check the sequence number versus IRS since we know * the sequence numbers haven't wrapped. This is a partial fix * for the "LAND" DoS attack. */ if (tp->t_state == TCPS_SYN_RECEIVED && SEQ_LT(th->th_seq, tp->irs)) { rstreason = BANDLIM_RST_OPENPORT; goto dropwithreset; } todrop = tp->rcv_nxt - th->th_seq; if (todrop > 0) { if (thflags & TH_SYN) { thflags &= ~TH_SYN; th->th_seq++; if (th->th_urp > 1) th->th_urp--; else thflags &= ~TH_URG; todrop--; } /* * Following if statement from Stevens, vol. 2, p. 960. */ if (todrop > tlen || (todrop == tlen && (thflags & TH_FIN) == 0)) { /* * Any valid FIN must be to the left of the window. * At this point the FIN must be a duplicate or out * of sequence; drop it. */ thflags &= ~TH_FIN; /* * Send an ACK to resynchronize and drop any data. * But keep on processing for RST or ACK. */ tp->t_flags |= TF_ACKNOW; todrop = tlen; TCPSTAT_INC(tcps_rcvduppack); TCPSTAT_ADD(tcps_rcvdupbyte, todrop); } else { TCPSTAT_INC(tcps_rcvpartduppack); TCPSTAT_ADD(tcps_rcvpartdupbyte, todrop); } drop_hdrlen += todrop; /* drop from the top afterwards */ th->th_seq += todrop; tlen -= todrop; if (th->th_urp > todrop) th->th_urp -= todrop; else { thflags &= ~TH_URG; th->th_urp = 0; } } /* * If new data are received on a connection after the * user processes are gone, then RST the other end. */ if ((so->so_state & SS_NOFDREF) && tp->t_state > TCPS_CLOSE_WAIT && tlen) { KASSERT(ti_locked == TI_WLOCKED, ("%s: SS_NOFDEREF && " "CLOSE_WAIT && tlen ti_locked %d", __func__, ti_locked)); INP_INFO_WLOCK_ASSERT(&V_tcbinfo); if ((s = tcp_log_addrs(inc, th, NULL, NULL))) { log(LOG_DEBUG, "%s; %s: %s: Received %d bytes of data " "after socket was closed, " "sending RST and removing tcpcb\n", s, __func__, tcpstates[tp->t_state], tlen); free(s, M_TCPLOG); } tp = tcp_close(tp); TCPSTAT_INC(tcps_rcvafterclose); rstreason = BANDLIM_UNLIMITED; goto dropwithreset; } /* * If segment ends after window, drop trailing data * (and PUSH and FIN); if nothing left, just ACK. */ todrop = (th->th_seq + tlen) - (tp->rcv_nxt + tp->rcv_wnd); if (todrop > 0) { TCPSTAT_INC(tcps_rcvpackafterwin); if (todrop >= tlen) { TCPSTAT_ADD(tcps_rcvbyteafterwin, tlen); /* * If window is closed can only take segments at * window edge, and have to drop data and PUSH from * incoming segments. Continue processing, but * remember to ack. Otherwise, drop segment * and ack. */ if (tp->rcv_wnd == 0 && th->th_seq == tp->rcv_nxt) { tp->t_flags |= TF_ACKNOW; TCPSTAT_INC(tcps_rcvwinprobe); } else goto dropafterack; } else TCPSTAT_ADD(tcps_rcvbyteafterwin, todrop); m_adj(m, -todrop); tlen -= todrop; thflags &= ~(TH_PUSH|TH_FIN); } /* * If last ACK falls within this segment's sequence numbers, * record its timestamp. * NOTE: * 1) That the test incorporates suggestions from the latest * proposal of the tcplw@cray.com list (Braden 1993/04/26). * 2) That updating only on newer timestamps interferes with * our earlier PAWS tests, so this check should be solely * predicated on the sequence space of this segment. * 3) That we modify the segment boundary check to be * Last.ACK.Sent <= SEG.SEQ + SEG.Len * instead of RFC1323's * Last.ACK.Sent < SEG.SEQ + SEG.Len, * This modified check allows us to overcome RFC1323's * limitations as described in Stevens TCP/IP Illustrated * Vol. 2 p.869. In such cases, we can still calculate the * RTT correctly when RCV.NXT == Last.ACK.Sent. */ if ((to.to_flags & TOF_TS) != 0 && SEQ_LEQ(th->th_seq, tp->last_ack_sent) && SEQ_LEQ(tp->last_ack_sent, th->th_seq + tlen + ((thflags & (TH_SYN|TH_FIN)) != 0))) { tp->ts_recent_age = tcp_ts_getticks(); tp->ts_recent = to.to_tsval; } /* * If the ACK bit is off: if in SYN-RECEIVED state or SENDSYN * flag is on (half-synchronized state), then queue data for * later processing; else drop segment and return. */ if ((thflags & TH_ACK) == 0) { if (tp->t_state == TCPS_SYN_RECEIVED || (tp->t_flags & TF_NEEDSYN)) goto step6; else if (tp->t_flags & TF_ACKNOW) goto dropafterack; else goto drop; } /* * Ack processing. */ switch (tp->t_state) { /* * In SYN_RECEIVED state, the ack ACKs our SYN, so enter * ESTABLISHED state and continue processing. * The ACK was checked above. */ case TCPS_SYN_RECEIVED: TCPSTAT_INC(tcps_connects); soisconnected(so); /* Do window scaling? */ if ((tp->t_flags & (TF_RCVD_SCALE|TF_REQ_SCALE)) == (TF_RCVD_SCALE|TF_REQ_SCALE)) { tp->rcv_scale = tp->request_r_scale; tp->snd_wnd = tiwin; } /* * Make transitions: * SYN-RECEIVED -> ESTABLISHED * SYN-RECEIVED* -> FIN-WAIT-1 */ tp->t_starttime = ticks; if (tp->t_flags & TF_NEEDFIN) { tcp_state_change(tp, TCPS_FIN_WAIT_1); tp->t_flags &= ~TF_NEEDFIN; } else { tcp_state_change(tp, TCPS_ESTABLISHED); TCP_PROBE5(accept__established, NULL, tp, mtod(m, const char *), tp, th); cc_conn_init(tp); tcp_timer_activate(tp, TT_KEEP, TP_KEEPIDLE(tp)); } /* * If segment contains data or ACK, will call tcp_reass() * later; if not, do so now to pass queued data to user. */ if (tlen == 0 && (thflags & TH_FIN) == 0) (void) tcp_reass(tp, (struct tcphdr *)0, 0, (struct mbuf *)0); tp->snd_wl1 = th->th_seq - 1; /* FALLTHROUGH */ /* * In ESTABLISHED state: drop duplicate ACKs; ACK out of range * ACKs. If the ack is in the range * tp->snd_una < th->th_ack <= tp->snd_max * then advance tp->snd_una to th->th_ack and drop * data from the retransmission queue. If this ACK reflects * more up to date window information we update our window information. */ case TCPS_ESTABLISHED: case TCPS_FIN_WAIT_1: case TCPS_FIN_WAIT_2: case TCPS_CLOSE_WAIT: case TCPS_CLOSING: case TCPS_LAST_ACK: if (SEQ_GT(th->th_ack, tp->snd_max)) { TCPSTAT_INC(tcps_rcvacktoomuch); goto dropafterack; } if ((tp->t_flags & TF_SACK_PERMIT) && ((to.to_flags & TOF_SACK) || !TAILQ_EMPTY(&tp->snd_holes))) tcp_sack_doack(tp, &to, th->th_ack); /* Run HHOOK_TCP_ESTABLISHED_IN helper hooks. */ hhook_run_tcp_est_in(tp, th, &to); if (SEQ_LEQ(th->th_ack, tp->snd_una)) { if (tlen == 0 && tiwin == tp->snd_wnd) { /* * If this is the first time we've seen a * FIN from the remote, this is not a * duplicate and it needs to be processed * normally. This happens during a * simultaneous close. */ if ((thflags & TH_FIN) && (TCPS_HAVERCVDFIN(tp->t_state) == 0)) { tp->t_dupacks = 0; break; } TCPSTAT_INC(tcps_rcvdupack); /* * If we have outstanding data (other than * a window probe), this is a completely * duplicate ack (ie, window info didn't * change and FIN isn't set), * the ack is the biggest we've * seen and we've seen exactly our rexmt * threshhold of them, assume a packet * has been dropped and retransmit it. * Kludge snd_nxt & the congestion * window so we send only this one * packet. * * We know we're losing at the current * window size so do congestion avoidance * (set ssthresh to half the current window * and pull our congestion window back to * the new ssthresh). * * Dup acks mean that packets have left the * network (they're now cached at the receiver) * so bump cwnd by the amount in the receiver * to keep a constant cwnd packets in the * network. * * When using TCP ECN, notify the peer that * we reduced the cwnd. */ if (!tcp_timer_active(tp, TT_REXMT) || th->th_ack != tp->snd_una) tp->t_dupacks = 0; else if (++tp->t_dupacks > tcprexmtthresh || IN_FASTRECOVERY(tp->t_flags)) { cc_ack_received(tp, th, CC_DUPACK); if ((tp->t_flags & TF_SACK_PERMIT) && IN_FASTRECOVERY(tp->t_flags)) { int awnd; /* * Compute the amount of data in flight first. * We can inject new data into the pipe iff * we have less than 1/2 the original window's * worth of data in flight. */ awnd = (tp->snd_nxt - tp->snd_fack) + tp->sackhint.sack_bytes_rexmit; if (awnd < tp->snd_ssthresh) { tp->snd_cwnd += tp->t_maxseg; if (tp->snd_cwnd > tp->snd_ssthresh) tp->snd_cwnd = tp->snd_ssthresh; } } else tp->snd_cwnd += tp->t_maxseg; (void) tcp_output(tp); goto drop; } else if (tp->t_dupacks == tcprexmtthresh) { tcp_seq onxt = tp->snd_nxt; /* * If we're doing sack, check to * see if we're already in sack * recovery. If we're not doing sack, * check to see if we're in newreno * recovery. */ if (tp->t_flags & TF_SACK_PERMIT) { if (IN_FASTRECOVERY(tp->t_flags)) { tp->t_dupacks = 0; break; } } else { if (SEQ_LEQ(th->th_ack, tp->snd_recover)) { tp->t_dupacks = 0; break; } } /* Congestion signal before ack. */ cc_cong_signal(tp, th, CC_NDUPACK); cc_ack_received(tp, th, CC_DUPACK); tcp_timer_activate(tp, TT_REXMT, 0); tp->t_rtttime = 0; if (tp->t_flags & TF_SACK_PERMIT) { TCPSTAT_INC( tcps_sack_recovery_episode); tp->sack_newdata = tp->snd_nxt; tp->snd_cwnd = tp->t_maxseg; (void) tcp_output(tp); goto drop; } tp->snd_nxt = th->th_ack; tp->snd_cwnd = tp->t_maxseg; (void) tcp_output(tp); KASSERT(tp->snd_limited <= 2, ("%s: tp->snd_limited too big", __func__)); tp->snd_cwnd = tp->snd_ssthresh + tp->t_maxseg * (tp->t_dupacks - tp->snd_limited); if (SEQ_GT(onxt, tp->snd_nxt)) tp->snd_nxt = onxt; goto drop; } else if (V_tcp_do_rfc3042) { cc_ack_received(tp, th, CC_DUPACK); u_long oldcwnd = tp->snd_cwnd; tcp_seq oldsndmax = tp->snd_max; u_int sent; int avail; KASSERT(tp->t_dupacks == 1 || tp->t_dupacks == 2, ("%s: dupacks not 1 or 2", __func__)); if (tp->t_dupacks == 1) tp->snd_limited = 0; tp->snd_cwnd = (tp->snd_nxt - tp->snd_una) + (tp->t_dupacks - tp->snd_limited) * tp->t_maxseg; /* * Only call tcp_output when there * is new data available to be sent. * Otherwise we would send pure ACKs. */ SOCKBUF_LOCK(&so->so_snd); avail = sbavail(&so->so_snd) - (tp->snd_nxt - tp->snd_una); SOCKBUF_UNLOCK(&so->so_snd); if (avail > 0) (void) tcp_output(tp); sent = tp->snd_max - oldsndmax; if (sent > tp->t_maxseg) { KASSERT((tp->t_dupacks == 2 && tp->snd_limited == 0) || (sent == tp->t_maxseg + 1 && tp->t_flags & TF_SENTFIN), ("%s: sent too much", __func__)); tp->snd_limited = 2; } else if (sent > 0) ++tp->snd_limited; tp->snd_cwnd = oldcwnd; goto drop; } } else tp->t_dupacks = 0; break; } KASSERT(SEQ_GT(th->th_ack, tp->snd_una), ("%s: th_ack <= snd_una", __func__)); /* * If the congestion window was inflated to account * for the other side's cached packets, retract it. */ if (IN_FASTRECOVERY(tp->t_flags)) { if (SEQ_LT(th->th_ack, tp->snd_recover)) { if (tp->t_flags & TF_SACK_PERMIT) tcp_sack_partialack(tp, th); else tcp_newreno_partial_ack(tp, th); } else cc_post_recovery(tp, th); } tp->t_dupacks = 0; /* * If we reach this point, ACK is not a duplicate, * i.e., it ACKs something we sent. */ if (tp->t_flags & TF_NEEDSYN) { /* * T/TCP: Connection was half-synchronized, and our * SYN has been ACK'd (so connection is now fully * synchronized). Go to non-starred state, * increment snd_una for ACK of SYN, and check if * we can do window scaling. */ tp->t_flags &= ~TF_NEEDSYN; tp->snd_una++; /* Do window scaling? */ if ((tp->t_flags & (TF_RCVD_SCALE|TF_REQ_SCALE)) == (TF_RCVD_SCALE|TF_REQ_SCALE)) { tp->rcv_scale = tp->request_r_scale; /* Send window already scaled. */ } } process_ACK: INP_WLOCK_ASSERT(tp->t_inpcb); acked = BYTES_THIS_ACK(tp, th); TCPSTAT_INC(tcps_rcvackpack); TCPSTAT_ADD(tcps_rcvackbyte, acked); /* * If we just performed our first retransmit, and the ACK * arrives within our recovery window, then it was a mistake * to do the retransmit in the first place. Recover our * original cwnd and ssthresh, and proceed to transmit where * we left off. */ if (tp->t_rxtshift == 1 && tp->t_flags & TF_PREVVALID && (int)(ticks - tp->t_badrxtwin) < 0) cc_cong_signal(tp, th, CC_RTO_ERR); /* * If we have a timestamp reply, update smoothed * round trip time. If no timestamp is present but * transmit timer is running and timed sequence * number was acked, update smoothed round trip time. * Since we now have an rtt measurement, cancel the * timer backoff (cf., Phil Karn's retransmit alg.). * Recompute the initial retransmit timer. * * Some boxes send broken timestamp replies * during the SYN+ACK phase, ignore * timestamps of 0 or we could calculate a * huge RTT and blow up the retransmit timer. */ if ((to.to_flags & TOF_TS) != 0 && to.to_tsecr) { u_int t; t = tcp_ts_getticks() - to.to_tsecr; if (!tp->t_rttlow || tp->t_rttlow > t) tp->t_rttlow = t; tcp_xmit_timer(tp, TCP_TS_TO_TICKS(t) + 1); } else if (tp->t_rtttime && SEQ_GT(th->th_ack, tp->t_rtseq)) { if (!tp->t_rttlow || tp->t_rttlow > ticks - tp->t_rtttime) tp->t_rttlow = ticks - tp->t_rtttime; tcp_xmit_timer(tp, ticks - tp->t_rtttime); } /* * If all outstanding data is acked, stop retransmit * timer and remember to restart (more output or persist). * If there is more data to be acked, restart retransmit * timer, using current (possibly backed-off) value. */ if (th->th_ack == tp->snd_max) { tcp_timer_activate(tp, TT_REXMT, 0); needoutput = 1; } else if (!tcp_timer_active(tp, TT_PERSIST)) tcp_timer_activate(tp, TT_REXMT, tp->t_rxtcur); /* * If no data (only SYN) was ACK'd, * skip rest of ACK processing. */ if (acked == 0) goto step6; /* * Let the congestion control algorithm update congestion * control related information. This typically means increasing * the congestion window. */ cc_ack_received(tp, th, CC_ACK); SOCKBUF_LOCK(&so->so_snd); if (acked > sbavail(&so->so_snd)) { tp->snd_wnd -= sbavail(&so->so_snd); mfree = sbcut_locked(&so->so_snd, (int)sbavail(&so->so_snd)); ourfinisacked = 1; } else { mfree = sbcut_locked(&so->so_snd, acked); tp->snd_wnd -= acked; ourfinisacked = 0; } /* NB: sowwakeup_locked() does an implicit unlock. */ sowwakeup_locked(so); m_freem(mfree); /* Detect una wraparound. */ if (!IN_RECOVERY(tp->t_flags) && SEQ_GT(tp->snd_una, tp->snd_recover) && SEQ_LEQ(th->th_ack, tp->snd_recover)) tp->snd_recover = th->th_ack - 1; /* XXXLAS: Can this be moved up into cc_post_recovery? */ if (IN_RECOVERY(tp->t_flags) && SEQ_GEQ(th->th_ack, tp->snd_recover)) { EXIT_RECOVERY(tp->t_flags); } tp->snd_una = th->th_ack; if (tp->t_flags & TF_SACK_PERMIT) { if (SEQ_GT(tp->snd_una, tp->snd_recover)) tp->snd_recover = tp->snd_una; } if (SEQ_LT(tp->snd_nxt, tp->snd_una)) tp->snd_nxt = tp->snd_una; switch (tp->t_state) { /* * In FIN_WAIT_1 STATE in addition to the processing * for the ESTABLISHED state if our FIN is now acknowledged * then enter FIN_WAIT_2. */ case TCPS_FIN_WAIT_1: if (ourfinisacked) { /* * If we can't receive any more * data, then closing user can proceed. * Starting the timer is contrary to the * specification, but if we don't get a FIN * we'll hang forever. * * XXXjl: * we should release the tp also, and use a * compressed state. */ if (so->so_rcv.sb_state & SBS_CANTRCVMORE) { soisdisconnected(so); tcp_timer_activate(tp, TT_2MSL, (tcp_fast_finwait2_recycle ? tcp_finwait2_timeout : TP_MAXIDLE(tp))); } tcp_state_change(tp, TCPS_FIN_WAIT_2); } break; /* * In CLOSING STATE in addition to the processing for * the ESTABLISHED state if the ACK acknowledges our FIN * then enter the TIME-WAIT state, otherwise ignore * the segment. */ case TCPS_CLOSING: if (ourfinisacked) { INP_INFO_WLOCK_ASSERT(&V_tcbinfo); tcp_twstart(tp); INP_INFO_WUNLOCK(&V_tcbinfo); m_freem(m); return; } break; /* * In LAST_ACK, we may still be waiting for data to drain * and/or to be acked, as well as for the ack of our FIN. * If our FIN is now acknowledged, delete the TCB, * enter the closed state and return. */ case TCPS_LAST_ACK: if (ourfinisacked) { INP_INFO_WLOCK_ASSERT(&V_tcbinfo); tp = tcp_close(tp); goto drop; } break; } } step6: INP_WLOCK_ASSERT(tp->t_inpcb); /* * Update window information. * Don't look at window if no ACK: TAC's send garbage on first SYN. */ if ((thflags & TH_ACK) && (SEQ_LT(tp->snd_wl1, th->th_seq) || (tp->snd_wl1 == th->th_seq && (SEQ_LT(tp->snd_wl2, th->th_ack) || (tp->snd_wl2 == th->th_ack && tiwin > tp->snd_wnd))))) { /* keep track of pure window updates */ if (tlen == 0 && tp->snd_wl2 == th->th_ack && tiwin > tp->snd_wnd) TCPSTAT_INC(tcps_rcvwinupd); tp->snd_wnd = tiwin; tp->snd_wl1 = th->th_seq; tp->snd_wl2 = th->th_ack; if (tp->snd_wnd > tp->max_sndwnd) tp->max_sndwnd = tp->snd_wnd; needoutput = 1; } /* * Process segments with URG. */ if ((thflags & TH_URG) && th->th_urp && TCPS_HAVERCVDFIN(tp->t_state) == 0) { /* * This is a kludge, but if we receive and accept * random urgent pointers, we'll crash in * soreceive. It's hard to imagine someone * actually wanting to send this much urgent data. */ SOCKBUF_LOCK(&so->so_rcv); if (th->th_urp + sbavail(&so->so_rcv) > sb_max) { th->th_urp = 0; /* XXX */ thflags &= ~TH_URG; /* XXX */ SOCKBUF_UNLOCK(&so->so_rcv); /* XXX */ goto dodata; /* XXX */ } /* * If this segment advances the known urgent pointer, * then mark the data stream. This should not happen * in CLOSE_WAIT, CLOSING, LAST_ACK or TIME_WAIT STATES since * a FIN has been received from the remote side. * In these states we ignore the URG. * * According to RFC961 (Assigned Protocols), * the urgent pointer points to the last octet * of urgent data. We continue, however, * to consider it to indicate the first octet * of data past the urgent section as the original * spec states (in one of two places). */ if (SEQ_GT(th->th_seq+th->th_urp, tp->rcv_up)) { tp->rcv_up = th->th_seq + th->th_urp; so->so_oobmark = sbavail(&so->so_rcv) + (tp->rcv_up - tp->rcv_nxt) - 1; if (so->so_oobmark == 0) so->so_rcv.sb_state |= SBS_RCVATMARK; sohasoutofband(so); tp->t_oobflags &= ~(TCPOOB_HAVEDATA | TCPOOB_HADDATA); } SOCKBUF_UNLOCK(&so->so_rcv); /* * Remove out of band data so doesn't get presented to user. * This can happen independent of advancing the URG pointer, * but if two URG's are pending at once, some out-of-band * data may creep in... ick. */ if (th->th_urp <= (u_long)tlen && !(so->so_options & SO_OOBINLINE)) { /* hdr drop is delayed */ tcp_pulloutofband(so, th, m, drop_hdrlen); } } else { /* * If no out of band data is expected, * pull receive urgent pointer along * with the receive window. */ if (SEQ_GT(tp->rcv_nxt, tp->rcv_up)) tp->rcv_up = tp->rcv_nxt; } dodata: /* XXX */ INP_WLOCK_ASSERT(tp->t_inpcb); /* * Process the segment text, merging it into the TCP sequencing queue, * and arranging for acknowledgment of receipt if necessary. * This process logically involves adjusting tp->rcv_wnd as data * is presented to the user (this happens in tcp_usrreq.c, * case PRU_RCVD). If a FIN has already been received on this * connection then we just ignore the text. */ if ((tlen || (thflags & TH_FIN)) && TCPS_HAVERCVDFIN(tp->t_state) == 0) { tcp_seq save_start = th->th_seq; m_adj(m, drop_hdrlen); /* delayed header drop */ /* * Insert segment which includes th into TCP reassembly queue * with control block tp. Set thflags to whether reassembly now * includes a segment with FIN. This handles the common case * inline (segment is the next to be received on an established * connection, and the queue is empty), avoiding linkage into * and removal from the queue and repetition of various * conversions. * Set DELACK for segments received in order, but ack * immediately when segments are out of order (so * fast retransmit can work). */ if (th->th_seq == tp->rcv_nxt && tp->t_segq == NULL && TCPS_HAVEESTABLISHED(tp->t_state)) { if (DELAY_ACK(tp, tlen)) tp->t_flags |= TF_DELACK; else tp->t_flags |= TF_ACKNOW; tp->rcv_nxt += tlen; thflags = th->th_flags & TH_FIN; TCPSTAT_INC(tcps_rcvpack); TCPSTAT_ADD(tcps_rcvbyte, tlen); ND6_HINT(tp); SOCKBUF_LOCK(&so->so_rcv); if (so->so_rcv.sb_state & SBS_CANTRCVMORE) m_freem(m); else sbappendstream_locked(&so->so_rcv, m, 0); /* NB: sorwakeup_locked() does an implicit unlock. */ sorwakeup_locked(so); } else { /* * XXX: Due to the header drop above "th" is * theoretically invalid by now. Fortunately * m_adj() doesn't actually frees any mbufs * when trimming from the head. */ thflags = tcp_reass(tp, th, &tlen, m); tp->t_flags |= TF_ACKNOW; } if (tlen > 0 && (tp->t_flags & TF_SACK_PERMIT)) tcp_update_sack_list(tp, save_start, save_start + tlen); #if 0 /* * Note the amount of data that peer has sent into * our window, in order to estimate the sender's * buffer size. * XXX: Unused. */ if (SEQ_GT(tp->rcv_adv, tp->rcv_nxt)) len = so->so_rcv.sb_hiwat - (tp->rcv_adv - tp->rcv_nxt); else len = so->so_rcv.sb_hiwat; #endif } else { m_freem(m); thflags &= ~TH_FIN; } /* * If FIN is received ACK the FIN and let the user know * that the connection is closing. */ if (thflags & TH_FIN) { if (TCPS_HAVERCVDFIN(tp->t_state) == 0) { socantrcvmore(so); /* * If connection is half-synchronized * (ie NEEDSYN flag on) then delay ACK, * so it may be piggybacked when SYN is sent. * Otherwise, since we received a FIN then no * more input can be expected, send ACK now. */ if (tp->t_flags & TF_NEEDSYN) tp->t_flags |= TF_DELACK; else tp->t_flags |= TF_ACKNOW; tp->rcv_nxt++; } switch (tp->t_state) { /* * In SYN_RECEIVED and ESTABLISHED STATES * enter the CLOSE_WAIT state. */ case TCPS_SYN_RECEIVED: tp->t_starttime = ticks; /* FALLTHROUGH */ case TCPS_ESTABLISHED: tcp_state_change(tp, TCPS_CLOSE_WAIT); break; /* * If still in FIN_WAIT_1 STATE FIN has not been acked so * enter the CLOSING state. */ case TCPS_FIN_WAIT_1: tcp_state_change(tp, TCPS_CLOSING); break; /* * In FIN_WAIT_2 state enter the TIME_WAIT state, * starting the time-wait timer, turning off the other * standard timers. */ case TCPS_FIN_WAIT_2: INP_INFO_WLOCK_ASSERT(&V_tcbinfo); KASSERT(ti_locked == TI_WLOCKED, ("%s: dodata " "TCP_FIN_WAIT_2 ti_locked: %d", __func__, ti_locked)); tcp_twstart(tp); INP_INFO_WUNLOCK(&V_tcbinfo); return; } } if (ti_locked == TI_WLOCKED) INP_INFO_WUNLOCK(&V_tcbinfo); ti_locked = TI_UNLOCKED; #ifdef TCPDEBUG if (so->so_options & SO_DEBUG) tcp_trace(TA_INPUT, ostate, tp, (void *)tcp_saveipgen, &tcp_savetcp, 0); #endif /* * Return any desired output. */ if (needoutput || (tp->t_flags & TF_ACKNOW)) (void) tcp_output(tp); check_delack: KASSERT(ti_locked == TI_UNLOCKED, ("%s: check_delack ti_locked %d", __func__, ti_locked)); INP_INFO_UNLOCK_ASSERT(&V_tcbinfo); INP_WLOCK_ASSERT(tp->t_inpcb); if (tp->t_flags & TF_DELACK) { tp->t_flags &= ~TF_DELACK; tcp_timer_activate(tp, TT_DELACK, tcp_delacktime); } INP_WUNLOCK(tp->t_inpcb); return; dropafterack: /* * Generate an ACK dropping incoming segment if it occupies * sequence space, where the ACK reflects our state. * * We can now skip the test for the RST flag since all * paths to this code happen after packets containing * RST have been dropped. * * In the SYN-RECEIVED state, don't send an ACK unless the * segment we received passes the SYN-RECEIVED ACK test. * If it fails send a RST. This breaks the loop in the * "LAND" DoS attack, and also prevents an ACK storm * between two listening ports that have been sent forged * SYN segments, each with the source address of the other. */ if (tp->t_state == TCPS_SYN_RECEIVED && (thflags & TH_ACK) && (SEQ_GT(tp->snd_una, th->th_ack) || SEQ_GT(th->th_ack, tp->snd_max)) ) { rstreason = BANDLIM_RST_OPENPORT; goto dropwithreset; } #ifdef TCPDEBUG if (so->so_options & SO_DEBUG) tcp_trace(TA_DROP, ostate, tp, (void *)tcp_saveipgen, &tcp_savetcp, 0); #endif if (ti_locked == TI_WLOCKED) INP_INFO_WUNLOCK(&V_tcbinfo); ti_locked = TI_UNLOCKED; tp->t_flags |= TF_ACKNOW; (void) tcp_output(tp); INP_WUNLOCK(tp->t_inpcb); m_freem(m); return; dropwithreset: if (ti_locked == TI_WLOCKED) INP_INFO_WUNLOCK(&V_tcbinfo); ti_locked = TI_UNLOCKED; if (tp != NULL) { tcp_dropwithreset(m, th, tp, tlen, rstreason); INP_WUNLOCK(tp->t_inpcb); } else tcp_dropwithreset(m, th, NULL, tlen, rstreason); return; drop: if (ti_locked == TI_WLOCKED) { INP_INFO_WUNLOCK(&V_tcbinfo); ti_locked = TI_UNLOCKED; } #ifdef INVARIANTS else INP_INFO_UNLOCK_ASSERT(&V_tcbinfo); #endif /* * Drop space held by incoming segment and return. */ #ifdef TCPDEBUG if (tp == NULL || (tp->t_inpcb->inp_socket->so_options & SO_DEBUG)) tcp_trace(TA_DROP, ostate, tp, (void *)tcp_saveipgen, &tcp_savetcp, 0); #endif if (tp != NULL) INP_WUNLOCK(tp->t_inpcb); m_freem(m); } /* * Issue RST and make ACK acceptable to originator of segment. * The mbuf must still include the original packet header. * tp may be NULL. */ static void tcp_dropwithreset(struct mbuf *m, struct tcphdr *th, struct tcpcb *tp, int tlen, int rstreason) { #ifdef INET struct ip *ip; #endif #ifdef INET6 struct ip6_hdr *ip6; #endif if (tp != NULL) { INP_WLOCK_ASSERT(tp->t_inpcb); } /* Don't bother if destination was broadcast/multicast. */ if ((th->th_flags & TH_RST) || m->m_flags & (M_BCAST|M_MCAST)) goto drop; #ifdef INET6 if (mtod(m, struct ip *)->ip_v == 6) { ip6 = mtod(m, struct ip6_hdr *); if (IN6_IS_ADDR_MULTICAST(&ip6->ip6_dst) || IN6_IS_ADDR_MULTICAST(&ip6->ip6_src)) goto drop; /* IPv6 anycast check is done at tcp6_input() */ } #endif #if defined(INET) && defined(INET6) else #endif #ifdef INET { ip = mtod(m, struct ip *); if (IN_MULTICAST(ntohl(ip->ip_dst.s_addr)) || IN_MULTICAST(ntohl(ip->ip_src.s_addr)) || ip->ip_src.s_addr == htonl(INADDR_BROADCAST) || in_broadcast(ip->ip_dst, m->m_pkthdr.rcvif)) goto drop; } #endif /* Perform bandwidth limiting. */ if (badport_bandlim(rstreason) < 0) goto drop; /* tcp_respond consumes the mbuf chain. */ if (th->th_flags & TH_ACK) { tcp_respond(tp, mtod(m, void *), th, m, (tcp_seq)0, th->th_ack, TH_RST); } else { if (th->th_flags & TH_SYN) tlen++; tcp_respond(tp, mtod(m, void *), th, m, th->th_seq+tlen, (tcp_seq)0, TH_RST|TH_ACK); } return; drop: m_freem(m); } /* * Parse TCP options and place in tcpopt. */ static void tcp_dooptions(struct tcpopt *to, u_char *cp, int cnt, int flags) { int opt, optlen; to->to_flags = 0; for (; cnt > 0; cnt -= optlen, cp += optlen) { opt = cp[0]; if (opt == TCPOPT_EOL) break; if (opt == TCPOPT_NOP) optlen = 1; else { if (cnt < 2) break; optlen = cp[1]; if (optlen < 2 || optlen > cnt) break; } switch (opt) { case TCPOPT_MAXSEG: if (optlen != TCPOLEN_MAXSEG) continue; if (!(flags & TO_SYN)) continue; to->to_flags |= TOF_MSS; bcopy((char *)cp + 2, (char *)&to->to_mss, sizeof(to->to_mss)); to->to_mss = ntohs(to->to_mss); break; case TCPOPT_WINDOW: if (optlen != TCPOLEN_WINDOW) continue; if (!(flags & TO_SYN)) continue; to->to_flags |= TOF_SCALE; to->to_wscale = min(cp[2], TCP_MAX_WINSHIFT); break; case TCPOPT_TIMESTAMP: if (optlen != TCPOLEN_TIMESTAMP) continue; to->to_flags |= TOF_TS; bcopy((char *)cp + 2, (char *)&to->to_tsval, sizeof(to->to_tsval)); to->to_tsval = ntohl(to->to_tsval); bcopy((char *)cp + 6, (char *)&to->to_tsecr, sizeof(to->to_tsecr)); to->to_tsecr = ntohl(to->to_tsecr); break; #ifdef TCP_SIGNATURE /* * XXX In order to reply to a host which has set the * TCP_SIGNATURE option in its initial SYN, we have to * record the fact that the option was observed here * for the syncache code to perform the correct response. */ case TCPOPT_SIGNATURE: if (optlen != TCPOLEN_SIGNATURE) continue; to->to_flags |= TOF_SIGNATURE; to->to_signature = cp + 2; break; #endif case TCPOPT_SACK_PERMITTED: if (optlen != TCPOLEN_SACK_PERMITTED) continue; if (!(flags & TO_SYN)) continue; if (!V_tcp_do_sack) continue; to->to_flags |= TOF_SACKPERM; break; case TCPOPT_SACK: if (optlen <= 2 || (optlen - 2) % TCPOLEN_SACK != 0) continue; if (flags & TO_SYN) continue; to->to_flags |= TOF_SACK; to->to_nsacks = (optlen - 2) / TCPOLEN_SACK; to->to_sacks = cp + 2; TCPSTAT_INC(tcps_sack_rcv_blocks); break; default: continue; } } } /* * Pull out of band byte out of a segment so * it doesn't appear in the user's data queue. * It is still reflected in the segment length for * sequencing purposes. */ static void tcp_pulloutofband(struct socket *so, struct tcphdr *th, struct mbuf *m, int off) { int cnt = off + th->th_urp - 1; while (cnt >= 0) { if (m->m_len > cnt) { char *cp = mtod(m, caddr_t) + cnt; struct tcpcb *tp = sototcpcb(so); INP_WLOCK_ASSERT(tp->t_inpcb); tp->t_iobc = *cp; tp->t_oobflags |= TCPOOB_HAVEDATA; bcopy(cp+1, cp, (unsigned)(m->m_len - cnt - 1)); m->m_len--; if (m->m_flags & M_PKTHDR) m->m_pkthdr.len--; return; } cnt -= m->m_len; m = m->m_next; if (m == NULL) break; } panic("tcp_pulloutofband"); } /* * Collect new round-trip time estimate * and update averages and current timeout. */ static void tcp_xmit_timer(struct tcpcb *tp, int rtt) { int delta; INP_WLOCK_ASSERT(tp->t_inpcb); TCPSTAT_INC(tcps_rttupdated); tp->t_rttupdated++; if (tp->t_srtt != 0) { /* * srtt is stored as fixed point with 5 bits after the * binary point (i.e., scaled by 8). The following magic * is equivalent to the smoothing algorithm in rfc793 with * an alpha of .875 (srtt = rtt/8 + srtt*7/8 in fixed * point). Adjust rtt to origin 0. */ delta = ((rtt - 1) << TCP_DELTA_SHIFT) - (tp->t_srtt >> (TCP_RTT_SHIFT - TCP_DELTA_SHIFT)); if ((tp->t_srtt += delta) <= 0) tp->t_srtt = 1; /* * We accumulate a smoothed rtt variance (actually, a * smoothed mean difference), then set the retransmit * timer to smoothed rtt + 4 times the smoothed variance. * rttvar is stored as fixed point with 4 bits after the * binary point (scaled by 16). The following is * equivalent to rfc793 smoothing with an alpha of .75 * (rttvar = rttvar*3/4 + |delta| / 4). This replaces * rfc793's wired-in beta. */ if (delta < 0) delta = -delta; delta -= tp->t_rttvar >> (TCP_RTTVAR_SHIFT - TCP_DELTA_SHIFT); if ((tp->t_rttvar += delta) <= 0) tp->t_rttvar = 1; if (tp->t_rttbest > tp->t_srtt + tp->t_rttvar) tp->t_rttbest = tp->t_srtt + tp->t_rttvar; } else { /* * No rtt measurement yet - use the unsmoothed rtt. * Set the variance to half the rtt (so our first * retransmit happens at 3*rtt). */ tp->t_srtt = rtt << TCP_RTT_SHIFT; tp->t_rttvar = rtt << (TCP_RTTVAR_SHIFT - 1); tp->t_rttbest = tp->t_srtt + tp->t_rttvar; } tp->t_rtttime = 0; tp->t_rxtshift = 0; /* * the retransmit should happen at rtt + 4 * rttvar. * Because of the way we do the smoothing, srtt and rttvar * will each average +1/2 tick of bias. When we compute * the retransmit timer, we want 1/2 tick of rounding and * 1 extra tick because of +-1/2 tick uncertainty in the * firing of the timer. The bias will give us exactly the * 1.5 tick we need. But, because the bias is * statistical, we have to test that we don't drop below * the minimum feasible timer (which is 2 ticks). */ TCPT_RANGESET(tp->t_rxtcur, TCP_REXMTVAL(tp), max(tp->t_rttmin, rtt + 2), TCPTV_REXMTMAX); /* * We received an ack for a packet that wasn't retransmitted; * it is probably safe to discard any error indications we've * received recently. This isn't quite right, but close enough * for now (a route might have failed after we sent a segment, * and the return path might not be symmetrical). */ tp->t_softerror = 0; } /* * Determine a reasonable value for maxseg size. * If the route is known, check route for mtu. * If none, use an mss that can be handled on the outgoing interface * without forcing IP to fragment. If no route is found, route has no mtu, * or the destination isn't local, use a default, hopefully conservative * size (usually 512 or the default IP max size, but no more than the mtu * of the interface), as we can't discover anything about intervening * gateways or networks. We also initialize the congestion/slow start * window to be a single segment if the destination isn't local. * While looking at the routing entry, we also initialize other path-dependent * parameters from pre-set or cached values in the routing entry. * * Also take into account the space needed for options that we * send regularly. Make maxseg shorter by that amount to assure * that we can send maxseg amount of data even when the options * are present. Store the upper limit of the length of options plus * data in maxopd. * * NOTE that this routine is only called when we process an incoming * segment, or an ICMP need fragmentation datagram. Outgoing SYN/ACK MSS * settings are handled in tcp_mssopt(). */ void tcp_mss_update(struct tcpcb *tp, int offer, int mtuoffer, struct hc_metrics_lite *metricptr, struct tcp_ifcap *cap) { int mss = 0; u_long maxmtu = 0; struct inpcb *inp = tp->t_inpcb; struct hc_metrics_lite metrics; int origoffer; #ifdef INET6 int isipv6 = ((inp->inp_vflag & INP_IPV6) != 0) ? 1 : 0; size_t min_protoh = isipv6 ? sizeof (struct ip6_hdr) + sizeof (struct tcphdr) : sizeof (struct tcpiphdr); #else const size_t min_protoh = sizeof(struct tcpiphdr); #endif INP_WLOCK_ASSERT(tp->t_inpcb); if (mtuoffer != -1) { KASSERT(offer == -1, ("%s: conflict", __func__)); offer = mtuoffer - min_protoh; } origoffer = offer; /* Initialize. */ #ifdef INET6 if (isipv6) { maxmtu = tcp_maxmtu6(&inp->inp_inc, cap); tp->t_maxopd = tp->t_maxseg = V_tcp_v6mssdflt; } #endif #if defined(INET) && defined(INET6) else #endif #ifdef INET { maxmtu = tcp_maxmtu(&inp->inp_inc, cap); tp->t_maxopd = tp->t_maxseg = V_tcp_mssdflt; } #endif /* * No route to sender, stay with default mss and return. */ if (maxmtu == 0) { /* * In case we return early we need to initialize metrics * to a defined state as tcp_hc_get() would do for us * if there was no cache hit. */ if (metricptr != NULL) bzero(metricptr, sizeof(struct hc_metrics_lite)); return; } /* What have we got? */ switch (offer) { case 0: /* * Offer == 0 means that there was no MSS on the SYN * segment, in this case we use tcp_mssdflt as * already assigned to t_maxopd above. */ offer = tp->t_maxopd; break; case -1: /* * Offer == -1 means that we didn't receive SYN yet. */ /* FALLTHROUGH */ default: /* * Prevent DoS attack with too small MSS. Round up * to at least minmss. */ offer = max(offer, V_tcp_minmss); } /* * rmx information is now retrieved from tcp_hostcache. */ tcp_hc_get(&inp->inp_inc, &metrics); if (metricptr != NULL) bcopy(&metrics, metricptr, sizeof(struct hc_metrics_lite)); /* * If there's a discovered mtu in tcp hostcache, use it. * Else, use the link mtu. */ if (metrics.rmx_mtu) mss = min(metrics.rmx_mtu, maxmtu) - min_protoh; else { #ifdef INET6 if (isipv6) { mss = maxmtu - min_protoh; if (!V_path_mtu_discovery && !in6_localaddr(&inp->in6p_faddr)) mss = min(mss, V_tcp_v6mssdflt); } #endif #if defined(INET) && defined(INET6) else #endif #ifdef INET { mss = maxmtu - min_protoh; if (!V_path_mtu_discovery && !in_localaddr(inp->inp_faddr)) mss = min(mss, V_tcp_mssdflt); } #endif /* * XXX - The above conditional (mss = maxmtu - min_protoh) * probably violates the TCP spec. * The problem is that, since we don't know the * other end's MSS, we are supposed to use a conservative * default. But, if we do that, then MTU discovery will * never actually take place, because the conservative * default is much less than the MTUs typically seen * on the Internet today. For the moment, we'll sweep * this under the carpet. * * The conservative default might not actually be a problem * if the only case this occurs is when sending an initial * SYN with options and data to a host we've never talked * to before. Then, they will reply with an MSS value which * will get recorded and the new parameters should get * recomputed. For Further Study. */ } mss = min(mss, offer); /* * Sanity check: make sure that maxopd will be large * enough to allow some data on segments even if the * all the option space is used (40bytes). Otherwise * funny things may happen in tcp_output. */ mss = max(mss, 64); /* * maxopd stores the maximum length of data AND options * in a segment; maxseg is the amount of data in a normal * segment. We need to store this value (maxopd) apart * from maxseg, because now every segment carries options * and thus we normally have somewhat less data in segments. */ tp->t_maxopd = mss; /* * origoffer==-1 indicates that no segments were received yet. * In this case we just guess. */ if ((tp->t_flags & (TF_REQ_TSTMP|TF_NOOPT)) == TF_REQ_TSTMP && (origoffer == -1 || (tp->t_flags & TF_RCVD_TSTMP) == TF_RCVD_TSTMP)) mss -= TCPOLEN_TSTAMP_APPA; tp->t_maxseg = mss; } void tcp_mss(struct tcpcb *tp, int offer) { int mss; u_long bufsize; struct inpcb *inp; struct socket *so; struct hc_metrics_lite metrics; struct tcp_ifcap cap; KASSERT(tp != NULL, ("%s: tp == NULL", __func__)); bzero(&cap, sizeof(cap)); tcp_mss_update(tp, offer, -1, &metrics, &cap); mss = tp->t_maxseg; inp = tp->t_inpcb; /* * If there's a pipesize, change the socket buffer to that size, * don't change if sb_hiwat is different than default (then it * has been changed on purpose with setsockopt). * Make the socket buffers an integral number of mss units; * if the mss is larger than the socket buffer, decrease the mss. */ so = inp->inp_socket; SOCKBUF_LOCK(&so->so_snd); if ((so->so_snd.sb_hiwat == V_tcp_sendspace) && metrics.rmx_sendpipe) bufsize = metrics.rmx_sendpipe; else bufsize = so->so_snd.sb_hiwat; if (bufsize < mss) mss = bufsize; else { bufsize = roundup(bufsize, mss); if (bufsize > sb_max) bufsize = sb_max; if (bufsize > so->so_snd.sb_hiwat) (void)sbreserve_locked(&so->so_snd, bufsize, so, NULL); } SOCKBUF_UNLOCK(&so->so_snd); tp->t_maxseg = mss; SOCKBUF_LOCK(&so->so_rcv); if ((so->so_rcv.sb_hiwat == V_tcp_recvspace) && metrics.rmx_recvpipe) bufsize = metrics.rmx_recvpipe; else bufsize = so->so_rcv.sb_hiwat; if (bufsize > mss) { bufsize = roundup(bufsize, mss); if (bufsize > sb_max) bufsize = sb_max; if (bufsize > so->so_rcv.sb_hiwat) (void)sbreserve_locked(&so->so_rcv, bufsize, so, NULL); } SOCKBUF_UNLOCK(&so->so_rcv); /* Check the interface for TSO capabilities. */ if (cap.ifcap & CSUM_TSO) { tp->t_flags |= TF_TSO; tp->t_tsomax = cap.tsomax; tp->t_tsomaxsegcount = cap.tsomaxsegcount; tp->t_tsomaxsegsize = cap.tsomaxsegsize; } } /* * Determine the MSS option to send on an outgoing SYN. */ int tcp_mssopt(struct in_conninfo *inc) { int mss = 0; u_long maxmtu = 0; u_long thcmtu = 0; size_t min_protoh; KASSERT(inc != NULL, ("tcp_mssopt with NULL in_conninfo pointer")); #ifdef INET6 if (inc->inc_flags & INC_ISIPV6) { mss = V_tcp_v6mssdflt; maxmtu = tcp_maxmtu6(inc, NULL); min_protoh = sizeof(struct ip6_hdr) + sizeof(struct tcphdr); } #endif #if defined(INET) && defined(INET6) else #endif #ifdef INET { mss = V_tcp_mssdflt; maxmtu = tcp_maxmtu(inc, NULL); min_protoh = sizeof(struct tcpiphdr); } #endif #if defined(INET6) || defined(INET) thcmtu = tcp_hc_getmtu(inc); /* IPv4 and IPv6 */ #endif if (maxmtu && thcmtu) mss = min(maxmtu, thcmtu) - min_protoh; else if (maxmtu || thcmtu) mss = max(maxmtu, thcmtu) - min_protoh; return (mss); } /* * On a partial ack arrives, force the retransmission of the * next unacknowledged segment. Do not clear tp->t_dupacks. * By setting snd_nxt to ti_ack, this forces retransmission timer to * be started again. */ static void tcp_newreno_partial_ack(struct tcpcb *tp, struct tcphdr *th) { tcp_seq onxt = tp->snd_nxt; u_long ocwnd = tp->snd_cwnd; INP_WLOCK_ASSERT(tp->t_inpcb); tcp_timer_activate(tp, TT_REXMT, 0); tp->t_rtttime = 0; tp->snd_nxt = th->th_ack; /* * Set snd_cwnd to one segment beyond acknowledged offset. * (tp->snd_una has not yet been updated when this function is called.) */ tp->snd_cwnd = tp->t_maxseg + BYTES_THIS_ACK(tp, th); tp->t_flags |= TF_ACKNOW; (void) tcp_output(tp); tp->snd_cwnd = ocwnd; if (SEQ_GT(onxt, tp->snd_nxt)) tp->snd_nxt = onxt; /* * Partial window deflation. Relies on fact that tp->snd_una * not updated yet. */ if (tp->snd_cwnd > BYTES_THIS_ACK(tp, th)) tp->snd_cwnd -= BYTES_THIS_ACK(tp, th); else tp->snd_cwnd = 0; tp->snd_cwnd += tp->t_maxseg; } Index: projects/clang350-import/sys/netinet/udp_usrreq.c =================================================================== --- projects/clang350-import/sys/netinet/udp_usrreq.c (revision 275748) +++ projects/clang350-import/sys/netinet/udp_usrreq.c (revision 275749) @@ -1,1899 +1,1898 @@ /*- * Copyright (c) 1982, 1986, 1988, 1990, 1993, 1995 * The Regents of the University of California. * Copyright (c) 2008 Robert N. M. Watson * Copyright (c) 2010-2011 Juniper Networks, Inc. * Copyright (c) 2014 Kevin Lo * All rights reserved. * * Portions of this software were developed by Robert N. M. Watson under * contract to Juniper Networks, Inc. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 4. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * @(#)udp_usrreq.c 8.6 (Berkeley) 5/23/95 */ #include __FBSDID("$FreeBSD$"); #include "opt_ipfw.h" #include "opt_inet.h" #include "opt_inet6.h" #include "opt_ipsec.h" #include "opt_rss.h" #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #ifdef INET6 #include #endif #include #include #include #include #ifdef INET6 #include #endif #include #include #include #include #ifdef IPSEC #include #include #endif #include #include /* * UDP and UDP-Lite protocols implementation. * Per RFC 768, August, 1980. * Per RFC 3828, July, 2004. */ /* * BSD 4.2 defaulted the udp checksum to be off. Turning off udp checksums * removes the only data integrity mechanism for packets and malformed * packets that would otherwise be discarded due to bad checksums, and may * cause problems (especially for NFS data blocks). */ VNET_DEFINE(int, udp_cksum) = 1; SYSCTL_INT(_net_inet_udp, UDPCTL_CHECKSUM, checksum, CTLFLAG_VNET | CTLFLAG_RW, &VNET_NAME(udp_cksum), 0, "compute udp checksum"); int udp_log_in_vain = 0; SYSCTL_INT(_net_inet_udp, OID_AUTO, log_in_vain, CTLFLAG_RW, &udp_log_in_vain, 0, "Log all incoming UDP packets"); VNET_DEFINE(int, udp_blackhole) = 0; SYSCTL_INT(_net_inet_udp, OID_AUTO, blackhole, CTLFLAG_VNET | CTLFLAG_RW, &VNET_NAME(udp_blackhole), 0, "Do not send port unreachables for refused connects"); u_long udp_sendspace = 9216; /* really max datagram size */ /* 40 1K datagrams */ SYSCTL_ULONG(_net_inet_udp, UDPCTL_MAXDGRAM, maxdgram, CTLFLAG_RW, &udp_sendspace, 0, "Maximum outgoing UDP datagram size"); u_long udp_recvspace = 40 * (1024 + #ifdef INET6 sizeof(struct sockaddr_in6) #else sizeof(struct sockaddr_in) #endif ); SYSCTL_ULONG(_net_inet_udp, UDPCTL_RECVSPACE, recvspace, CTLFLAG_RW, &udp_recvspace, 0, "Maximum space for incoming UDP datagrams"); VNET_DEFINE(struct inpcbhead, udb); /* from udp_var.h */ VNET_DEFINE(struct inpcbinfo, udbinfo); VNET_DEFINE(struct inpcbhead, ulitecb); VNET_DEFINE(struct inpcbinfo, ulitecbinfo); static VNET_DEFINE(uma_zone_t, udpcb_zone); #define V_udpcb_zone VNET(udpcb_zone) #ifndef UDBHASHSIZE #define UDBHASHSIZE 128 #endif VNET_PCPUSTAT_DEFINE(struct udpstat, udpstat); /* from udp_var.h */ VNET_PCPUSTAT_SYSINIT(udpstat); SYSCTL_VNET_PCPUSTAT(_net_inet_udp, UDPCTL_STATS, stats, struct udpstat, udpstat, "UDP statistics (struct udpstat, netinet/udp_var.h)"); #ifdef VIMAGE VNET_PCPUSTAT_SYSUNINIT(udpstat); #endif /* VIMAGE */ #ifdef INET static void udp_detach(struct socket *so); static int udp_output(struct inpcb *, struct mbuf *, struct sockaddr *, struct mbuf *, struct thread *); #endif #ifdef IPSEC #ifdef IPSEC_NAT_T #define UF_ESPINUDP_ALL (UF_ESPINUDP_NON_IKE|UF_ESPINUDP) #ifdef INET static struct mbuf *udp4_espdecap(struct inpcb *, struct mbuf *, int); #endif #endif /* IPSEC_NAT_T */ #endif /* IPSEC */ static void udp_zone_change(void *tag) { uma_zone_set_max(V_udbinfo.ipi_zone, maxsockets); uma_zone_set_max(V_udpcb_zone, maxsockets); } static int udp_inpcb_init(void *mem, int size, int flags) { struct inpcb *inp; inp = mem; INP_LOCK_INIT(inp, "inp", "udpinp"); return (0); } static int udplite_inpcb_init(void *mem, int size, int flags) { struct inpcb *inp; inp = mem; INP_LOCK_INIT(inp, "inp", "udpliteinp"); return (0); } void udp_init(void) { /* * For now default to 2-tuple UDP hashing - until the fragment * reassembly code can also update the flowid. * * Once we can calculate the flowid that way and re-establish * a 4-tuple, flip this to 4-tuple. */ in_pcbinfo_init(&V_udbinfo, "udp", &V_udb, UDBHASHSIZE, UDBHASHSIZE, "udp_inpcb", udp_inpcb_init, NULL, 0, IPI_HASHFIELDS_2TUPLE); V_udpcb_zone = uma_zcreate("udpcb", sizeof(struct udpcb), NULL, NULL, NULL, NULL, UMA_ALIGN_PTR, 0); uma_zone_set_max(V_udpcb_zone, maxsockets); uma_zone_set_warning(V_udpcb_zone, "kern.ipc.maxsockets limit reached"); EVENTHANDLER_REGISTER(maxsockets_change, udp_zone_change, NULL, EVENTHANDLER_PRI_ANY); } void udplite_init(void) { in_pcbinfo_init(&V_ulitecbinfo, "udplite", &V_ulitecb, UDBHASHSIZE, UDBHASHSIZE, "udplite_inpcb", udplite_inpcb_init, NULL, 0, IPI_HASHFIELDS_2TUPLE); } /* * Kernel module interface for updating udpstat. The argument is an index * into udpstat treated as an array of u_long. While this encodes the * general layout of udpstat into the caller, it doesn't encode its location, * so that future changes to add, for example, per-CPU stats support won't * cause binary compatibility problems for kernel modules. */ void kmod_udpstat_inc(int statnum) { counter_u64_add(VNET(udpstat)[statnum], 1); } int udp_newudpcb(struct inpcb *inp) { struct udpcb *up; up = uma_zalloc(V_udpcb_zone, M_NOWAIT | M_ZERO); if (up == NULL) return (ENOBUFS); inp->inp_ppcb = up; return (0); } void udp_discardcb(struct udpcb *up) { uma_zfree(V_udpcb_zone, up); } #ifdef VIMAGE void udp_destroy(void) { in_pcbinfo_destroy(&V_udbinfo); uma_zdestroy(V_udpcb_zone); } void udplite_destroy(void) { in_pcbinfo_destroy(&V_ulitecbinfo); } #endif #ifdef INET /* * Subroutine of udp_input(), which appends the provided mbuf chain to the * passed pcb/socket. The caller must provide a sockaddr_in via udp_in that * contains the source address. If the socket ends up being an IPv6 socket, * udp_append() will convert to a sockaddr_in6 before passing the address * into the socket code. */ static void udp_append(struct inpcb *inp, struct ip *ip, struct mbuf *n, int off, struct sockaddr_in *udp_in) { struct sockaddr *append_sa; struct socket *so; struct mbuf *opts = 0; #ifdef INET6 struct sockaddr_in6 udp_in6; #endif struct udpcb *up; INP_LOCK_ASSERT(inp); /* * Engage the tunneling protocol. */ up = intoudpcb(inp); if (up->u_tun_func != NULL) { (*up->u_tun_func)(n, off, inp, (struct sockaddr *)udp_in, up->u_tun_ctx); return; } off += sizeof(struct udphdr); #ifdef IPSEC /* Check AH/ESP integrity. */ if (ipsec4_in_reject(n, inp)) { m_freem(n); - IPSECSTAT_INC(ips_in_polvio); return; } #ifdef IPSEC_NAT_T up = intoudpcb(inp); KASSERT(up != NULL, ("%s: udpcb NULL", __func__)); if (up->u_flags & UF_ESPINUDP_ALL) { /* IPSec UDP encaps. */ n = udp4_espdecap(inp, n, off); if (n == NULL) /* Consumed. */ return; } #endif /* IPSEC_NAT_T */ #endif /* IPSEC */ #ifdef MAC if (mac_inpcb_check_deliver(inp, n) != 0) { m_freem(n); return; } #endif /* MAC */ if (inp->inp_flags & INP_CONTROLOPTS || inp->inp_socket->so_options & (SO_TIMESTAMP | SO_BINTIME)) { #ifdef INET6 if (inp->inp_vflag & INP_IPV6) (void)ip6_savecontrol_v4(inp, n, &opts, NULL); else #endif /* INET6 */ ip_savecontrol(inp, &opts, ip, n); } #ifdef INET6 if (inp->inp_vflag & INP_IPV6) { bzero(&udp_in6, sizeof(udp_in6)); udp_in6.sin6_len = sizeof(udp_in6); udp_in6.sin6_family = AF_INET6; in6_sin_2_v4mapsin6(udp_in, &udp_in6); append_sa = (struct sockaddr *)&udp_in6; } else #endif /* INET6 */ append_sa = (struct sockaddr *)udp_in; m_adj(n, off); so = inp->inp_socket; SOCKBUF_LOCK(&so->so_rcv); if (sbappendaddr_locked(&so->so_rcv, append_sa, n, opts) == 0) { SOCKBUF_UNLOCK(&so->so_rcv); m_freem(n); if (opts) m_freem(opts); UDPSTAT_INC(udps_fullsock); } else sorwakeup_locked(so); } int udp_input(struct mbuf **mp, int *offp, int proto) { struct ip *ip; struct udphdr *uh; struct ifnet *ifp; struct inpcb *inp; uint16_t len, ip_len; struct inpcbinfo *pcbinfo; struct ip save_ip; struct sockaddr_in udp_in; struct mbuf *m; struct m_tag *fwd_tag; int cscov_partial, iphlen; m = *mp; iphlen = *offp; ifp = m->m_pkthdr.rcvif; *mp = NULL; UDPSTAT_INC(udps_ipackets); /* * Strip IP options, if any; should skip this, make available to * user, and use on returned packets, but we don't yet have a way to * check the checksum with options still present. */ if (iphlen > sizeof (struct ip)) { ip_stripoptions(m); iphlen = sizeof(struct ip); } /* * Get IP and UDP header together in first mbuf. */ ip = mtod(m, struct ip *); if (m->m_len < iphlen + sizeof(struct udphdr)) { if ((m = m_pullup(m, iphlen + sizeof(struct udphdr))) == NULL) { UDPSTAT_INC(udps_hdrops); return (IPPROTO_DONE); } ip = mtod(m, struct ip *); } uh = (struct udphdr *)((caddr_t)ip + iphlen); cscov_partial = (proto == IPPROTO_UDPLITE) ? 1 : 0; /* * Destination port of 0 is illegal, based on RFC768. */ if (uh->uh_dport == 0) goto badunlocked; /* * Construct sockaddr format source address. Stuff source address * and datagram in user buffer. */ bzero(&udp_in, sizeof(udp_in)); udp_in.sin_len = sizeof(udp_in); udp_in.sin_family = AF_INET; udp_in.sin_port = uh->uh_sport; udp_in.sin_addr = ip->ip_src; /* * Make mbuf data length reflect UDP length. If not enough data to * reflect UDP length, drop. */ len = ntohs((u_short)uh->uh_ulen); ip_len = ntohs(ip->ip_len) - iphlen; if (proto == IPPROTO_UDPLITE && (len == 0 || len == ip_len)) { /* Zero means checksum over the complete packet. */ if (len == 0) len = ip_len; cscov_partial = 0; } if (ip_len != len) { if (len > ip_len || len < sizeof(struct udphdr)) { UDPSTAT_INC(udps_badlen); goto badunlocked; } if (proto == IPPROTO_UDP) m_adj(m, len - ip_len); } /* * Save a copy of the IP header in case we want restore it for * sending an ICMP error message in response. */ if (!V_udp_blackhole) save_ip = *ip; else memset(&save_ip, 0, sizeof(save_ip)); /* * Checksum extended UDP header and data. */ if (uh->uh_sum) { u_short uh_sum; if ((m->m_pkthdr.csum_flags & CSUM_DATA_VALID) && !cscov_partial) { if (m->m_pkthdr.csum_flags & CSUM_PSEUDO_HDR) uh_sum = m->m_pkthdr.csum_data; else uh_sum = in_pseudo(ip->ip_src.s_addr, ip->ip_dst.s_addr, htonl((u_short)len + m->m_pkthdr.csum_data + proto)); uh_sum ^= 0xffff; } else { char b[9]; bcopy(((struct ipovly *)ip)->ih_x1, b, 9); bzero(((struct ipovly *)ip)->ih_x1, 9); ((struct ipovly *)ip)->ih_len = (proto == IPPROTO_UDP) ? uh->uh_ulen : htons(ip_len); uh_sum = in_cksum(m, len + sizeof (struct ip)); bcopy(b, ((struct ipovly *)ip)->ih_x1, 9); } if (uh_sum) { UDPSTAT_INC(udps_badsum); m_freem(m); return (IPPROTO_DONE); } } else { if (proto == IPPROTO_UDP) { UDPSTAT_INC(udps_nosum); } else { /* UDPLite requires a checksum */ /* XXX: What is the right UDPLite MIB counter here? */ m_freem(m); return (IPPROTO_DONE); } } pcbinfo = get_inpcbinfo(proto); if (IN_MULTICAST(ntohl(ip->ip_dst.s_addr)) || in_broadcast(ip->ip_dst, ifp)) { struct inpcb *last; struct inpcbhead *pcblist; struct ip_moptions *imo; INP_INFO_RLOCK(pcbinfo); pcblist = get_pcblist(proto); last = NULL; LIST_FOREACH(inp, pcblist, inp_list) { if (inp->inp_lport != uh->uh_dport) continue; #ifdef INET6 if ((inp->inp_vflag & INP_IPV4) == 0) continue; #endif if (inp->inp_laddr.s_addr != INADDR_ANY && inp->inp_laddr.s_addr != ip->ip_dst.s_addr) continue; if (inp->inp_faddr.s_addr != INADDR_ANY && inp->inp_faddr.s_addr != ip->ip_src.s_addr) continue; if (inp->inp_fport != 0 && inp->inp_fport != uh->uh_sport) continue; INP_RLOCK(inp); /* * XXXRW: Because we weren't holding either the inpcb * or the hash lock when we checked for a match * before, we should probably recheck now that the * inpcb lock is held. */ /* * Handle socket delivery policy for any-source * and source-specific multicast. [RFC3678] */ imo = inp->inp_moptions; if (IN_MULTICAST(ntohl(ip->ip_dst.s_addr))) { struct sockaddr_in group; int blocked; if (imo == NULL) { INP_RUNLOCK(inp); continue; } bzero(&group, sizeof(struct sockaddr_in)); group.sin_len = sizeof(struct sockaddr_in); group.sin_family = AF_INET; group.sin_addr = ip->ip_dst; blocked = imo_multi_filter(imo, ifp, (struct sockaddr *)&group, (struct sockaddr *)&udp_in); if (blocked != MCAST_PASS) { if (blocked == MCAST_NOTGMEMBER) IPSTAT_INC(ips_notmember); if (blocked == MCAST_NOTSMEMBER || blocked == MCAST_MUTED) UDPSTAT_INC(udps_filtermcast); INP_RUNLOCK(inp); continue; } } if (last != NULL) { struct mbuf *n; if ((n = m_copy(m, 0, M_COPYALL)) != NULL) { UDP_PROBE(receive, NULL, last, ip, last, uh); udp_append(last, ip, n, iphlen, &udp_in); } INP_RUNLOCK(last); } last = inp; /* * Don't look for additional matches if this one does * not have either the SO_REUSEPORT or SO_REUSEADDR * socket options set. This heuristic avoids * searching through all pcbs in the common case of a * non-shared port. It assumes that an application * will never clear these options after setting them. */ if ((last->inp_socket->so_options & (SO_REUSEPORT|SO_REUSEADDR)) == 0) break; } if (last == NULL) { /* * No matching pcb found; discard datagram. (No need * to send an ICMP Port Unreachable for a broadcast * or multicast datgram.) */ UDPSTAT_INC(udps_noportbcast); if (inp) INP_RUNLOCK(inp); INP_INFO_RUNLOCK(pcbinfo); goto badunlocked; } UDP_PROBE(receive, NULL, last, ip, last, uh); udp_append(last, ip, m, iphlen, &udp_in); INP_RUNLOCK(last); INP_INFO_RUNLOCK(pcbinfo); return (IPPROTO_DONE); } /* * Locate pcb for datagram. */ /* * Grab info from PACKET_TAG_IPFORWARD tag prepended to the chain. */ if ((m->m_flags & M_IP_NEXTHOP) && (fwd_tag = m_tag_find(m, PACKET_TAG_IPFORWARD, NULL)) != NULL) { struct sockaddr_in *next_hop; next_hop = (struct sockaddr_in *)(fwd_tag + 1); /* * Transparently forwarded. Pretend to be the destination. * Already got one like this? */ inp = in_pcblookup_mbuf(pcbinfo, ip->ip_src, uh->uh_sport, ip->ip_dst, uh->uh_dport, INPLOOKUP_RLOCKPCB, ifp, m); if (!inp) { /* * It's new. Try to find the ambushing socket. * Because we've rewritten the destination address, * any hardware-generated hash is ignored. */ inp = in_pcblookup(pcbinfo, ip->ip_src, uh->uh_sport, next_hop->sin_addr, next_hop->sin_port ? htons(next_hop->sin_port) : uh->uh_dport, INPLOOKUP_WILDCARD | INPLOOKUP_RLOCKPCB, ifp); } /* Remove the tag from the packet. We don't need it anymore. */ m_tag_delete(m, fwd_tag); m->m_flags &= ~M_IP_NEXTHOP; } else inp = in_pcblookup_mbuf(pcbinfo, ip->ip_src, uh->uh_sport, ip->ip_dst, uh->uh_dport, INPLOOKUP_WILDCARD | INPLOOKUP_RLOCKPCB, ifp, m); if (inp == NULL) { if (udp_log_in_vain) { char buf[4*sizeof "123"]; strcpy(buf, inet_ntoa(ip->ip_dst)); log(LOG_INFO, "Connection attempt to UDP %s:%d from %s:%d\n", buf, ntohs(uh->uh_dport), inet_ntoa(ip->ip_src), ntohs(uh->uh_sport)); } UDPSTAT_INC(udps_noport); if (m->m_flags & (M_BCAST | M_MCAST)) { UDPSTAT_INC(udps_noportbcast); goto badunlocked; } if (V_udp_blackhole) goto badunlocked; if (badport_bandlim(BANDLIM_ICMP_UNREACH) < 0) goto badunlocked; *ip = save_ip; icmp_error(m, ICMP_UNREACH, ICMP_UNREACH_PORT, 0, 0); return (IPPROTO_DONE); } /* * Check the minimum TTL for socket. */ INP_RLOCK_ASSERT(inp); if (inp->inp_ip_minttl && inp->inp_ip_minttl > ip->ip_ttl) { INP_RUNLOCK(inp); m_freem(m); return (IPPROTO_DONE); } if (cscov_partial) { struct udpcb *up; up = intoudpcb(inp); if (up->u_rxcslen == 0 || up->u_rxcslen > len) { INP_RUNLOCK(inp); m_freem(m); return (IPPROTO_DONE); } } UDP_PROBE(receive, NULL, inp, ip, inp, uh); udp_append(inp, ip, m, iphlen, &udp_in); INP_RUNLOCK(inp); return (IPPROTO_DONE); badunlocked: m_freem(m); return (IPPROTO_DONE); } #endif /* INET */ /* * Notify a udp user of an asynchronous error; just wake up so that they can * collect error status. */ struct inpcb * udp_notify(struct inpcb *inp, int errno) { /* * While udp_ctlinput() always calls udp_notify() with a read lock * when invoking it directly, in_pcbnotifyall() currently uses write * locks due to sharing code with TCP. For now, accept either a read * or a write lock, but a read lock is sufficient. */ INP_LOCK_ASSERT(inp); inp->inp_socket->so_error = errno; sorwakeup(inp->inp_socket); sowwakeup(inp->inp_socket); return (inp); } #ifdef INET static void udp_common_ctlinput(int cmd, struct sockaddr *sa, void *vip, struct inpcbinfo *pcbinfo) { struct ip *ip = vip; struct udphdr *uh; struct in_addr faddr; struct inpcb *inp; faddr = ((struct sockaddr_in *)sa)->sin_addr; if (sa->sa_family != AF_INET || faddr.s_addr == INADDR_ANY) return; /* * Redirects don't need to be handled up here. */ if (PRC_IS_REDIRECT(cmd)) return; /* * Hostdead is ugly because it goes linearly through all PCBs. * * XXX: We never get this from ICMP, otherwise it makes an excellent * DoS attack on machines with many connections. */ if (cmd == PRC_HOSTDEAD) ip = NULL; else if ((unsigned)cmd >= PRC_NCMDS || inetctlerrmap[cmd] == 0) return; if (ip != NULL) { uh = (struct udphdr *)((caddr_t)ip + (ip->ip_hl << 2)); inp = in_pcblookup(pcbinfo, faddr, uh->uh_dport, ip->ip_src, uh->uh_sport, INPLOOKUP_RLOCKPCB, NULL); if (inp != NULL) { INP_RLOCK_ASSERT(inp); if (inp->inp_socket != NULL) { udp_notify(inp, inetctlerrmap[cmd]); } INP_RUNLOCK(inp); } } else in_pcbnotifyall(pcbinfo, faddr, inetctlerrmap[cmd], udp_notify); } void udp_ctlinput(int cmd, struct sockaddr *sa, void *vip) { return (udp_common_ctlinput(cmd, sa, vip, &V_udbinfo)); } void udplite_ctlinput(int cmd, struct sockaddr *sa, void *vip) { return (udp_common_ctlinput(cmd, sa, vip, &V_ulitecbinfo)); } #endif /* INET */ static int udp_pcblist(SYSCTL_HANDLER_ARGS) { int error, i, n; struct inpcb *inp, **inp_list; inp_gen_t gencnt; struct xinpgen xig; /* * The process of preparing the PCB list is too time-consuming and * resource-intensive to repeat twice on every request. */ if (req->oldptr == 0) { n = V_udbinfo.ipi_count; n += imax(n / 8, 10); req->oldidx = 2 * (sizeof xig) + n * sizeof(struct xinpcb); return (0); } if (req->newptr != 0) return (EPERM); /* * OK, now we're committed to doing something. */ INP_INFO_RLOCK(&V_udbinfo); gencnt = V_udbinfo.ipi_gencnt; n = V_udbinfo.ipi_count; INP_INFO_RUNLOCK(&V_udbinfo); error = sysctl_wire_old_buffer(req, 2 * (sizeof xig) + n * sizeof(struct xinpcb)); if (error != 0) return (error); xig.xig_len = sizeof xig; xig.xig_count = n; xig.xig_gen = gencnt; xig.xig_sogen = so_gencnt; error = SYSCTL_OUT(req, &xig, sizeof xig); if (error) return (error); inp_list = malloc(n * sizeof *inp_list, M_TEMP, M_WAITOK); if (inp_list == 0) return (ENOMEM); INP_INFO_RLOCK(&V_udbinfo); for (inp = LIST_FIRST(V_udbinfo.ipi_listhead), i = 0; inp && i < n; inp = LIST_NEXT(inp, inp_list)) { INP_WLOCK(inp); if (inp->inp_gencnt <= gencnt && cr_canseeinpcb(req->td->td_ucred, inp) == 0) { in_pcbref(inp); inp_list[i++] = inp; } INP_WUNLOCK(inp); } INP_INFO_RUNLOCK(&V_udbinfo); n = i; error = 0; for (i = 0; i < n; i++) { inp = inp_list[i]; INP_RLOCK(inp); if (inp->inp_gencnt <= gencnt) { struct xinpcb xi; bzero(&xi, sizeof(xi)); xi.xi_len = sizeof xi; /* XXX should avoid extra copy */ bcopy(inp, &xi.xi_inp, sizeof *inp); if (inp->inp_socket) sotoxsocket(inp->inp_socket, &xi.xi_socket); xi.xi_inp.inp_gencnt = inp->inp_gencnt; INP_RUNLOCK(inp); error = SYSCTL_OUT(req, &xi, sizeof xi); } else INP_RUNLOCK(inp); } INP_INFO_WLOCK(&V_udbinfo); for (i = 0; i < n; i++) { inp = inp_list[i]; INP_RLOCK(inp); if (!in_pcbrele_rlocked(inp)) INP_RUNLOCK(inp); } INP_INFO_WUNLOCK(&V_udbinfo); if (!error) { /* * Give the user an updated idea of our state. If the * generation differs from what we told her before, she knows * that something happened while we were processing this * request, and it might be necessary to retry. */ INP_INFO_RLOCK(&V_udbinfo); xig.xig_gen = V_udbinfo.ipi_gencnt; xig.xig_sogen = so_gencnt; xig.xig_count = V_udbinfo.ipi_count; INP_INFO_RUNLOCK(&V_udbinfo); error = SYSCTL_OUT(req, &xig, sizeof xig); } free(inp_list, M_TEMP); return (error); } SYSCTL_PROC(_net_inet_udp, UDPCTL_PCBLIST, pcblist, CTLTYPE_OPAQUE | CTLFLAG_RD, NULL, 0, udp_pcblist, "S,xinpcb", "List of active UDP sockets"); #ifdef INET static int udp_getcred(SYSCTL_HANDLER_ARGS) { struct xucred xuc; struct sockaddr_in addrs[2]; struct inpcb *inp; int error; error = priv_check(req->td, PRIV_NETINET_GETCRED); if (error) return (error); error = SYSCTL_IN(req, addrs, sizeof(addrs)); if (error) return (error); inp = in_pcblookup(&V_udbinfo, addrs[1].sin_addr, addrs[1].sin_port, addrs[0].sin_addr, addrs[0].sin_port, INPLOOKUP_WILDCARD | INPLOOKUP_RLOCKPCB, NULL); if (inp != NULL) { INP_RLOCK_ASSERT(inp); if (inp->inp_socket == NULL) error = ENOENT; if (error == 0) error = cr_canseeinpcb(req->td->td_ucred, inp); if (error == 0) cru2x(inp->inp_cred, &xuc); INP_RUNLOCK(inp); } else error = ENOENT; if (error == 0) error = SYSCTL_OUT(req, &xuc, sizeof(struct xucred)); return (error); } SYSCTL_PROC(_net_inet_udp, OID_AUTO, getcred, CTLTYPE_OPAQUE|CTLFLAG_RW|CTLFLAG_PRISON, 0, 0, udp_getcred, "S,xucred", "Get the xucred of a UDP connection"); #endif /* INET */ int udp_ctloutput(struct socket *so, struct sockopt *sopt) { struct inpcb *inp; struct udpcb *up; int isudplite, error, optval; error = 0; isudplite = (so->so_proto->pr_protocol == IPPROTO_UDPLITE) ? 1 : 0; inp = sotoinpcb(so); KASSERT(inp != NULL, ("%s: inp == NULL", __func__)); INP_WLOCK(inp); if (sopt->sopt_level != so->so_proto->pr_protocol) { #ifdef INET6 if (INP_CHECK_SOCKAF(so, AF_INET6)) { INP_WUNLOCK(inp); error = ip6_ctloutput(so, sopt); } #endif #if defined(INET) && defined(INET6) else #endif #ifdef INET { INP_WUNLOCK(inp); error = ip_ctloutput(so, sopt); } #endif return (error); } switch (sopt->sopt_dir) { case SOPT_SET: switch (sopt->sopt_name) { case UDP_ENCAP: INP_WUNLOCK(inp); error = sooptcopyin(sopt, &optval, sizeof optval, sizeof optval); if (error) break; inp = sotoinpcb(so); KASSERT(inp != NULL, ("%s: inp == NULL", __func__)); INP_WLOCK(inp); #ifdef IPSEC_NAT_T up = intoudpcb(inp); KASSERT(up != NULL, ("%s: up == NULL", __func__)); #endif switch (optval) { case 0: /* Clear all UDP encap. */ #ifdef IPSEC_NAT_T up->u_flags &= ~UF_ESPINUDP_ALL; #endif break; #ifdef IPSEC_NAT_T case UDP_ENCAP_ESPINUDP: case UDP_ENCAP_ESPINUDP_NON_IKE: up->u_flags &= ~UF_ESPINUDP_ALL; if (optval == UDP_ENCAP_ESPINUDP) up->u_flags |= UF_ESPINUDP; else if (optval == UDP_ENCAP_ESPINUDP_NON_IKE) up->u_flags |= UF_ESPINUDP_NON_IKE; break; #endif default: error = EINVAL; break; } INP_WUNLOCK(inp); break; case UDPLITE_SEND_CSCOV: case UDPLITE_RECV_CSCOV: if (!isudplite) { INP_WUNLOCK(inp); error = ENOPROTOOPT; break; } INP_WUNLOCK(inp); error = sooptcopyin(sopt, &optval, sizeof(optval), sizeof(optval)); if (error != 0) break; inp = sotoinpcb(so); KASSERT(inp != NULL, ("%s: inp == NULL", __func__)); INP_WLOCK(inp); up = intoudpcb(inp); KASSERT(up != NULL, ("%s: up == NULL", __func__)); if ((optval != 0 && optval < 8) || (optval > 65535)) { INP_WUNLOCK(inp); error = EINVAL; break; } if (sopt->sopt_name == UDPLITE_SEND_CSCOV) up->u_txcslen = optval; else up->u_rxcslen = optval; INP_WUNLOCK(inp); break; default: INP_WUNLOCK(inp); error = ENOPROTOOPT; break; } break; case SOPT_GET: switch (sopt->sopt_name) { #ifdef IPSEC_NAT_T case UDP_ENCAP: up = intoudpcb(inp); KASSERT(up != NULL, ("%s: up == NULL", __func__)); optval = up->u_flags & UF_ESPINUDP_ALL; INP_WUNLOCK(inp); error = sooptcopyout(sopt, &optval, sizeof optval); break; #endif case UDPLITE_SEND_CSCOV: case UDPLITE_RECV_CSCOV: if (!isudplite) { INP_WUNLOCK(inp); error = ENOPROTOOPT; break; } up = intoudpcb(inp); KASSERT(up != NULL, ("%s: up == NULL", __func__)); if (sopt->sopt_name == UDPLITE_SEND_CSCOV) optval = up->u_txcslen; else optval = up->u_rxcslen; INP_WUNLOCK(inp); error = sooptcopyout(sopt, &optval, sizeof(optval)); break; default: INP_WUNLOCK(inp); error = ENOPROTOOPT; break; } break; } return (error); } #ifdef INET #define UH_WLOCKED 2 #define UH_RLOCKED 1 #define UH_UNLOCKED 0 static int udp_output(struct inpcb *inp, struct mbuf *m, struct sockaddr *addr, struct mbuf *control, struct thread *td) { struct udpiphdr *ui; int len = m->m_pkthdr.len; struct in_addr faddr, laddr; struct cmsghdr *cm; struct inpcbinfo *pcbinfo; struct sockaddr_in *sin, src; int cscov_partial = 0; int error = 0; int ipflags; u_short fport, lport; int unlock_udbinfo; u_char tos; uint8_t pr; uint16_t cscov = 0; uint32_t flowid = 0; uint8_t flowtype = M_HASHTYPE_NONE; /* * udp_output() may need to temporarily bind or connect the current * inpcb. As such, we don't know up front whether we will need the * pcbinfo lock or not. Do any work to decide what is needed up * front before acquiring any locks. */ if (len + sizeof(struct udpiphdr) > IP_MAXPACKET) { if (control) m_freem(control); m_freem(m); return (EMSGSIZE); } src.sin_family = 0; INP_RLOCK(inp); tos = inp->inp_ip_tos; if (control != NULL) { /* * XXX: Currently, we assume all the optional information is * stored in a single mbuf. */ if (control->m_next) { INP_RUNLOCK(inp); m_freem(control); m_freem(m); return (EINVAL); } for (; control->m_len > 0; control->m_data += CMSG_ALIGN(cm->cmsg_len), control->m_len -= CMSG_ALIGN(cm->cmsg_len)) { cm = mtod(control, struct cmsghdr *); if (control->m_len < sizeof(*cm) || cm->cmsg_len == 0 || cm->cmsg_len > control->m_len) { error = EINVAL; break; } if (cm->cmsg_level != IPPROTO_IP) continue; switch (cm->cmsg_type) { case IP_SENDSRCADDR: if (cm->cmsg_len != CMSG_LEN(sizeof(struct in_addr))) { error = EINVAL; break; } bzero(&src, sizeof(src)); src.sin_family = AF_INET; src.sin_len = sizeof(src); src.sin_port = inp->inp_lport; src.sin_addr = *(struct in_addr *)CMSG_DATA(cm); break; case IP_TOS: if (cm->cmsg_len != CMSG_LEN(sizeof(u_char))) { error = EINVAL; break; } tos = *(u_char *)CMSG_DATA(cm); break; case IP_FLOWID: if (cm->cmsg_len != CMSG_LEN(sizeof(uint32_t))) { error = EINVAL; break; } flowid = *(uint32_t *) CMSG_DATA(cm); break; case IP_FLOWTYPE: if (cm->cmsg_len != CMSG_LEN(sizeof(uint32_t))) { error = EINVAL; break; } flowtype = *(uint32_t *) CMSG_DATA(cm); break; #ifdef RSS case IP_RSSBUCKETID: if (cm->cmsg_len != CMSG_LEN(sizeof(uint32_t))) { error = EINVAL; break; } /* This is just a placeholder for now */ break; #endif /* RSS */ default: error = ENOPROTOOPT; break; } if (error) break; } m_freem(control); } if (error) { INP_RUNLOCK(inp); m_freem(m); return (error); } /* * Depending on whether or not the application has bound or connected * the socket, we may have to do varying levels of work. The optimal * case is for a connected UDP socket, as a global lock isn't * required at all. * * In order to decide which we need, we require stability of the * inpcb binding, which we ensure by acquiring a read lock on the * inpcb. This doesn't strictly follow the lock order, so we play * the trylock and retry game; note that we may end up with more * conservative locks than required the second time around, so later * assertions have to accept that. Further analysis of the number of * misses under contention is required. * * XXXRW: Check that hash locking update here is correct. */ pr = inp->inp_socket->so_proto->pr_protocol; pcbinfo = get_inpcbinfo(pr); sin = (struct sockaddr_in *)addr; if (sin != NULL && (inp->inp_laddr.s_addr == INADDR_ANY && inp->inp_lport == 0)) { INP_RUNLOCK(inp); INP_WLOCK(inp); INP_HASH_WLOCK(pcbinfo); unlock_udbinfo = UH_WLOCKED; } else if ((sin != NULL && ( (sin->sin_addr.s_addr == INADDR_ANY) || (sin->sin_addr.s_addr == INADDR_BROADCAST) || (inp->inp_laddr.s_addr == INADDR_ANY) || (inp->inp_lport == 0))) || (src.sin_family == AF_INET)) { INP_HASH_RLOCK(pcbinfo); unlock_udbinfo = UH_RLOCKED; } else unlock_udbinfo = UH_UNLOCKED; /* * If the IP_SENDSRCADDR control message was specified, override the * source address for this datagram. Its use is invalidated if the * address thus specified is incomplete or clobbers other inpcbs. */ laddr = inp->inp_laddr; lport = inp->inp_lport; if (src.sin_family == AF_INET) { INP_HASH_LOCK_ASSERT(pcbinfo); if ((lport == 0) || (laddr.s_addr == INADDR_ANY && src.sin_addr.s_addr == INADDR_ANY)) { error = EINVAL; goto release; } error = in_pcbbind_setup(inp, (struct sockaddr *)&src, &laddr.s_addr, &lport, td->td_ucred); if (error) goto release; } /* * If a UDP socket has been connected, then a local address/port will * have been selected and bound. * * If a UDP socket has not been connected to, then an explicit * destination address must be used, in which case a local * address/port may not have been selected and bound. */ if (sin != NULL) { INP_LOCK_ASSERT(inp); if (inp->inp_faddr.s_addr != INADDR_ANY) { error = EISCONN; goto release; } /* * Jail may rewrite the destination address, so let it do * that before we use it. */ error = prison_remote_ip4(td->td_ucred, &sin->sin_addr); if (error) goto release; /* * If a local address or port hasn't yet been selected, or if * the destination address needs to be rewritten due to using * a special INADDR_ constant, invoke in_pcbconnect_setup() * to do the heavy lifting. Once a port is selected, we * commit the binding back to the socket; we also commit the * binding of the address if in jail. * * If we already have a valid binding and we're not * requesting a destination address rewrite, use a fast path. */ if (inp->inp_laddr.s_addr == INADDR_ANY || inp->inp_lport == 0 || sin->sin_addr.s_addr == INADDR_ANY || sin->sin_addr.s_addr == INADDR_BROADCAST) { INP_HASH_LOCK_ASSERT(pcbinfo); error = in_pcbconnect_setup(inp, addr, &laddr.s_addr, &lport, &faddr.s_addr, &fport, NULL, td->td_ucred); if (error) goto release; /* * XXXRW: Why not commit the port if the address is * !INADDR_ANY? */ /* Commit the local port if newly assigned. */ if (inp->inp_laddr.s_addr == INADDR_ANY && inp->inp_lport == 0) { INP_WLOCK_ASSERT(inp); INP_HASH_WLOCK_ASSERT(pcbinfo); /* * Remember addr if jailed, to prevent * rebinding. */ if (prison_flag(td->td_ucred, PR_IP4)) inp->inp_laddr = laddr; inp->inp_lport = lport; if (in_pcbinshash(inp) != 0) { inp->inp_lport = 0; error = EAGAIN; goto release; } inp->inp_flags |= INP_ANONPORT; } } else { faddr = sin->sin_addr; fport = sin->sin_port; } } else { INP_LOCK_ASSERT(inp); faddr = inp->inp_faddr; fport = inp->inp_fport; if (faddr.s_addr == INADDR_ANY) { error = ENOTCONN; goto release; } } /* * Calculate data length and get a mbuf for UDP, IP, and possible * link-layer headers. Immediate slide the data pointer back forward * since we won't use that space at this layer. */ M_PREPEND(m, sizeof(struct udpiphdr) + max_linkhdr, M_NOWAIT); if (m == NULL) { error = ENOBUFS; goto release; } m->m_data += max_linkhdr; m->m_len -= max_linkhdr; m->m_pkthdr.len -= max_linkhdr; /* * Fill in mbuf with extended UDP header and addresses and length put * into network format. */ ui = mtod(m, struct udpiphdr *); bzero(ui->ui_x1, sizeof(ui->ui_x1)); /* XXX still needed? */ ui->ui_pr = pr; ui->ui_src = laddr; ui->ui_dst = faddr; ui->ui_sport = lport; ui->ui_dport = fport; ui->ui_ulen = htons((u_short)len + sizeof(struct udphdr)); if (pr == IPPROTO_UDPLITE) { struct udpcb *up; uint16_t plen; up = intoudpcb(inp); cscov = up->u_txcslen; plen = (u_short)len + sizeof(struct udphdr); if (cscov >= plen) cscov = 0; ui->ui_len = htons(plen); ui->ui_ulen = htons(cscov); /* * For UDP-Lite, checksum coverage length of zero means * the entire UDPLite packet is covered by the checksum. */ cscov_partial = (cscov == 0) ? 0 : 1; } else ui->ui_v = IPVERSION << 4; /* * Set the Don't Fragment bit in the IP header. */ if (inp->inp_flags & INP_DONTFRAG) { struct ip *ip; ip = (struct ip *)&ui->ui_i; ip->ip_off |= htons(IP_DF); } ipflags = 0; if (inp->inp_socket->so_options & SO_DONTROUTE) ipflags |= IP_ROUTETOIF; if (inp->inp_socket->so_options & SO_BROADCAST) ipflags |= IP_ALLOWBROADCAST; if (inp->inp_flags & INP_ONESBCAST) ipflags |= IP_SENDONES; #ifdef MAC mac_inpcb_create_mbuf(inp, m); #endif /* * Set up checksum and output datagram. */ ui->ui_sum = 0; if (pr == IPPROTO_UDPLITE) { if (inp->inp_flags & INP_ONESBCAST) faddr.s_addr = INADDR_BROADCAST; if (cscov_partial) { if ((ui->ui_sum = in_cksum(m, sizeof(struct ip) + cscov)) == 0) ui->ui_sum = 0xffff; } else { if ((ui->ui_sum = in_cksum(m, sizeof(struct udpiphdr) + len)) == 0) ui->ui_sum = 0xffff; } } else if (V_udp_cksum) { if (inp->inp_flags & INP_ONESBCAST) faddr.s_addr = INADDR_BROADCAST; ui->ui_sum = in_pseudo(ui->ui_src.s_addr, faddr.s_addr, htons((u_short)len + sizeof(struct udphdr) + pr)); m->m_pkthdr.csum_flags = CSUM_UDP; m->m_pkthdr.csum_data = offsetof(struct udphdr, uh_sum); } ((struct ip *)ui)->ip_len = htons(sizeof(struct udpiphdr) + len); ((struct ip *)ui)->ip_ttl = inp->inp_ip_ttl; /* XXX */ ((struct ip *)ui)->ip_tos = tos; /* XXX */ UDPSTAT_INC(udps_opackets); /* * Setup flowid / RSS information for outbound socket. * * Once the UDP code decides to set a flowid some other way, * this allows the flowid to be overridden by userland. */ if (flowtype != M_HASHTYPE_NONE) { m->m_pkthdr.flowid = flowid; M_HASHTYPE_SET(m, flowtype); #ifdef RSS } else { uint32_t hash_val, hash_type; /* * Calculate an appropriate RSS hash for UDP and * UDP Lite. * * The called function will take care of figuring out * whether a 2-tuple or 4-tuple hash is required based * on the currently configured scheme. * * Later later on connected socket values should be * cached in the inpcb and reused, rather than constantly * re-calculating it. * * UDP Lite is a different protocol number and will * likely end up being hashed as a 2-tuple until * RSS / NICs grow UDP Lite protocol awareness. */ if (rss_proto_software_hash_v4(faddr, laddr, fport, lport, pr, &hash_val, &hash_type) == 0) { m->m_pkthdr.flowid = hash_val; M_HASHTYPE_SET(m, hash_type); } #endif } #ifdef RSS /* * Don't override with the inp cached flowid value. * * Depending upon the kind of send being done, the inp * flowid/flowtype values may actually not be appropriate * for this particular socket send. * * We should either leave the flowid at zero (which is what is * currently done) or set it to some software generated * hash value based on the packet contents. */ ipflags |= IP_NODEFAULTFLOWID; #endif /* RSS */ if (unlock_udbinfo == UH_WLOCKED) INP_HASH_WUNLOCK(pcbinfo); else if (unlock_udbinfo == UH_RLOCKED) INP_HASH_RUNLOCK(pcbinfo); UDP_PROBE(send, NULL, inp, &ui->ui_i, inp, &ui->ui_u); error = ip_output(m, inp->inp_options, NULL, ipflags, inp->inp_moptions, inp); if (unlock_udbinfo == UH_WLOCKED) INP_WUNLOCK(inp); else INP_RUNLOCK(inp); return (error); release: if (unlock_udbinfo == UH_WLOCKED) { INP_HASH_WUNLOCK(pcbinfo); INP_WUNLOCK(inp); } else if (unlock_udbinfo == UH_RLOCKED) { INP_HASH_RUNLOCK(pcbinfo); INP_RUNLOCK(inp); } else INP_RUNLOCK(inp); m_freem(m); return (error); } #if defined(IPSEC) && defined(IPSEC_NAT_T) /* * Potentially decap ESP in UDP frame. Check for an ESP header * and optional marker; if present, strip the UDP header and * push the result through IPSec. * * Returns mbuf to be processed (potentially re-allocated) or * NULL if consumed and/or processed. */ static struct mbuf * udp4_espdecap(struct inpcb *inp, struct mbuf *m, int off) { size_t minlen, payload, skip, iphlen; caddr_t data; struct udpcb *up; struct m_tag *tag; struct udphdr *udphdr; struct ip *ip; INP_RLOCK_ASSERT(inp); /* * Pull up data so the longest case is contiguous: * IP/UDP hdr + non ESP marker + ESP hdr. */ minlen = off + sizeof(uint64_t) + sizeof(struct esp); if (minlen > m->m_pkthdr.len) minlen = m->m_pkthdr.len; if ((m = m_pullup(m, minlen)) == NULL) { IPSECSTAT_INC(ips_in_inval); return (NULL); /* Bypass caller processing. */ } data = mtod(m, caddr_t); /* Points to ip header. */ payload = m->m_len - off; /* Size of payload. */ if (payload == 1 && data[off] == '\xff') return (m); /* NB: keepalive packet, no decap. */ up = intoudpcb(inp); KASSERT(up != NULL, ("%s: udpcb NULL", __func__)); KASSERT((up->u_flags & UF_ESPINUDP_ALL) != 0, ("u_flags 0x%x", up->u_flags)); /* * Check that the payload is large enough to hold an * ESP header and compute the amount of data to remove. * * NB: the caller has already done a pullup for us. * XXX can we assume alignment and eliminate bcopys? */ if (up->u_flags & UF_ESPINUDP_NON_IKE) { /* * draft-ietf-ipsec-nat-t-ike-0[01].txt and * draft-ietf-ipsec-udp-encaps-(00/)01.txt, ignoring * possible AH mode non-IKE marker+non-ESP marker * from draft-ietf-ipsec-udp-encaps-00.txt. */ uint64_t marker; if (payload <= sizeof(uint64_t) + sizeof(struct esp)) return (m); /* NB: no decap. */ bcopy(data + off, &marker, sizeof(uint64_t)); if (marker != 0) /* Non-IKE marker. */ return (m); /* NB: no decap. */ skip = sizeof(uint64_t) + sizeof(struct udphdr); } else { uint32_t spi; if (payload <= sizeof(struct esp)) { IPSECSTAT_INC(ips_in_inval); m_freem(m); return (NULL); /* Discard. */ } bcopy(data + off, &spi, sizeof(uint32_t)); if (spi == 0) /* Non-ESP marker. */ return (m); /* NB: no decap. */ skip = sizeof(struct udphdr); } /* * Setup a PACKET_TAG_IPSEC_NAT_T_PORT tag to remember * the UDP ports. This is required if we want to select * the right SPD for multiple hosts behind same NAT. * * NB: ports are maintained in network byte order everywhere * in the NAT-T code. */ tag = m_tag_get(PACKET_TAG_IPSEC_NAT_T_PORTS, 2 * sizeof(uint16_t), M_NOWAIT); if (tag == NULL) { IPSECSTAT_INC(ips_in_nomem); m_freem(m); return (NULL); /* Discard. */ } iphlen = off - sizeof(struct udphdr); udphdr = (struct udphdr *)(data + iphlen); ((uint16_t *)(tag + 1))[0] = udphdr->uh_sport; ((uint16_t *)(tag + 1))[1] = udphdr->uh_dport; m_tag_prepend(m, tag); /* * Remove the UDP header (and possibly the non ESP marker) * IP header length is iphlen * Before: * <--- off ---> * +----+------+-----+ * | IP | UDP | ESP | * +----+------+-----+ * <-skip-> * After: * +----+-----+ * | IP | ESP | * +----+-----+ * <-skip-> */ ovbcopy(data, data + skip, iphlen); m_adj(m, skip); ip = mtod(m, struct ip *); ip->ip_len = htons(ntohs(ip->ip_len) - skip); ip->ip_p = IPPROTO_ESP; /* * We cannot yet update the cksums so clear any * h/w cksum flags as they are no longer valid. */ if (m->m_pkthdr.csum_flags & CSUM_DATA_VALID) m->m_pkthdr.csum_flags &= ~(CSUM_DATA_VALID|CSUM_PSEUDO_HDR); (void) ipsec4_common_input(m, iphlen, ip->ip_p); return (NULL); /* NB: consumed, bypass processing. */ } #endif /* defined(IPSEC) && defined(IPSEC_NAT_T) */ static void udp_abort(struct socket *so) { struct inpcb *inp; struct inpcbinfo *pcbinfo; pcbinfo = get_inpcbinfo(so->so_proto->pr_protocol); inp = sotoinpcb(so); KASSERT(inp != NULL, ("udp_abort: inp == NULL")); INP_WLOCK(inp); if (inp->inp_faddr.s_addr != INADDR_ANY) { INP_HASH_WLOCK(pcbinfo); in_pcbdisconnect(inp); inp->inp_laddr.s_addr = INADDR_ANY; INP_HASH_WUNLOCK(pcbinfo); soisdisconnected(so); } INP_WUNLOCK(inp); } static int udp_attach(struct socket *so, int proto, struct thread *td) { struct inpcb *inp; struct inpcbinfo *pcbinfo; int error; pcbinfo = get_inpcbinfo(so->so_proto->pr_protocol); inp = sotoinpcb(so); KASSERT(inp == NULL, ("udp_attach: inp != NULL")); error = soreserve(so, udp_sendspace, udp_recvspace); if (error) return (error); INP_INFO_WLOCK(pcbinfo); error = in_pcballoc(so, pcbinfo); if (error) { INP_INFO_WUNLOCK(pcbinfo); return (error); } inp = sotoinpcb(so); inp->inp_vflag |= INP_IPV4; inp->inp_ip_ttl = V_ip_defttl; error = udp_newudpcb(inp); if (error) { in_pcbdetach(inp); in_pcbfree(inp); INP_INFO_WUNLOCK(pcbinfo); return (error); } INP_WUNLOCK(inp); INP_INFO_WUNLOCK(pcbinfo); return (0); } #endif /* INET */ int udp_set_kernel_tunneling(struct socket *so, udp_tun_func_t f, void *ctx) { struct inpcb *inp; struct udpcb *up; KASSERT(so->so_type == SOCK_DGRAM, ("udp_set_kernel_tunneling: !dgram")); inp = sotoinpcb(so); KASSERT(inp != NULL, ("udp_set_kernel_tunneling: inp == NULL")); INP_WLOCK(inp); up = intoudpcb(inp); if (up->u_tun_func != NULL) { INP_WUNLOCK(inp); return (EBUSY); } up->u_tun_func = f; up->u_tun_ctx = ctx; INP_WUNLOCK(inp); return (0); } #ifdef INET static int udp_bind(struct socket *so, struct sockaddr *nam, struct thread *td) { struct inpcb *inp; struct inpcbinfo *pcbinfo; int error; pcbinfo = get_inpcbinfo(so->so_proto->pr_protocol); inp = sotoinpcb(so); KASSERT(inp != NULL, ("udp_bind: inp == NULL")); INP_WLOCK(inp); INP_HASH_WLOCK(pcbinfo); error = in_pcbbind(inp, nam, td->td_ucred); INP_HASH_WUNLOCK(pcbinfo); INP_WUNLOCK(inp); return (error); } static void udp_close(struct socket *so) { struct inpcb *inp; struct inpcbinfo *pcbinfo; pcbinfo = get_inpcbinfo(so->so_proto->pr_protocol); inp = sotoinpcb(so); KASSERT(inp != NULL, ("udp_close: inp == NULL")); INP_WLOCK(inp); if (inp->inp_faddr.s_addr != INADDR_ANY) { INP_HASH_WLOCK(pcbinfo); in_pcbdisconnect(inp); inp->inp_laddr.s_addr = INADDR_ANY; INP_HASH_WUNLOCK(pcbinfo); soisdisconnected(so); } INP_WUNLOCK(inp); } static int udp_connect(struct socket *so, struct sockaddr *nam, struct thread *td) { struct inpcb *inp; struct inpcbinfo *pcbinfo; struct sockaddr_in *sin; int error; pcbinfo = get_inpcbinfo(so->so_proto->pr_protocol); inp = sotoinpcb(so); KASSERT(inp != NULL, ("udp_connect: inp == NULL")); INP_WLOCK(inp); if (inp->inp_faddr.s_addr != INADDR_ANY) { INP_WUNLOCK(inp); return (EISCONN); } sin = (struct sockaddr_in *)nam; error = prison_remote_ip4(td->td_ucred, &sin->sin_addr); if (error != 0) { INP_WUNLOCK(inp); return (error); } INP_HASH_WLOCK(pcbinfo); error = in_pcbconnect(inp, nam, td->td_ucred); INP_HASH_WUNLOCK(pcbinfo); if (error == 0) soisconnected(so); INP_WUNLOCK(inp); return (error); } static void udp_detach(struct socket *so) { struct inpcb *inp; struct inpcbinfo *pcbinfo; struct udpcb *up; pcbinfo = get_inpcbinfo(so->so_proto->pr_protocol); inp = sotoinpcb(so); KASSERT(inp != NULL, ("udp_detach: inp == NULL")); KASSERT(inp->inp_faddr.s_addr == INADDR_ANY, ("udp_detach: not disconnected")); INP_INFO_WLOCK(pcbinfo); INP_WLOCK(inp); up = intoudpcb(inp); KASSERT(up != NULL, ("%s: up == NULL", __func__)); inp->inp_ppcb = NULL; in_pcbdetach(inp); in_pcbfree(inp); INP_INFO_WUNLOCK(pcbinfo); udp_discardcb(up); } static int udp_disconnect(struct socket *so) { struct inpcb *inp; struct inpcbinfo *pcbinfo; pcbinfo = get_inpcbinfo(so->so_proto->pr_protocol); inp = sotoinpcb(so); KASSERT(inp != NULL, ("udp_disconnect: inp == NULL")); INP_WLOCK(inp); if (inp->inp_faddr.s_addr == INADDR_ANY) { INP_WUNLOCK(inp); return (ENOTCONN); } INP_HASH_WLOCK(pcbinfo); in_pcbdisconnect(inp); inp->inp_laddr.s_addr = INADDR_ANY; INP_HASH_WUNLOCK(pcbinfo); SOCK_LOCK(so); so->so_state &= ~SS_ISCONNECTED; /* XXX */ SOCK_UNLOCK(so); INP_WUNLOCK(inp); return (0); } static int udp_send(struct socket *so, int flags, struct mbuf *m, struct sockaddr *addr, struct mbuf *control, struct thread *td) { struct inpcb *inp; inp = sotoinpcb(so); KASSERT(inp != NULL, ("udp_send: inp == NULL")); return (udp_output(inp, m, addr, control, td)); } #endif /* INET */ int udp_shutdown(struct socket *so) { struct inpcb *inp; inp = sotoinpcb(so); KASSERT(inp != NULL, ("udp_shutdown: inp == NULL")); INP_WLOCK(inp); socantsendmore(so); INP_WUNLOCK(inp); return (0); } #ifdef INET struct pr_usrreqs udp_usrreqs = { .pru_abort = udp_abort, .pru_attach = udp_attach, .pru_bind = udp_bind, .pru_connect = udp_connect, .pru_control = in_control, .pru_detach = udp_detach, .pru_disconnect = udp_disconnect, .pru_peeraddr = in_getpeeraddr, .pru_send = udp_send, .pru_soreceive = soreceive_dgram, .pru_sosend = sosend_dgram, .pru_shutdown = udp_shutdown, .pru_sockaddr = in_getsockaddr, .pru_sosetlabel = in_pcbsosetlabel, .pru_close = udp_close, }; #endif /* INET */ Index: projects/clang350-import/sys/netinet6/ip6_forward.c =================================================================== --- projects/clang350-import/sys/netinet6/ip6_forward.c (revision 275748) +++ projects/clang350-import/sys/netinet6/ip6_forward.c (revision 275749) @@ -1,643 +1,640 @@ /*- * Copyright (C) 1995, 1996, 1997, and 1998 WIDE Project. * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of the project nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE PROJECT AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE PROJECT OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * $KAME: ip6_forward.c,v 1.69 2001/05/17 03:48:30 itojun Exp $ */ #include __FBSDID("$FreeBSD$"); #include "opt_inet.h" #include "opt_inet6.h" #include "opt_ipfw.h" #include "opt_ipsec.h" #include "opt_ipstealth.h" #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #ifdef IPSEC +#include #include #include #include #endif /* IPSEC */ /* * Forward a packet. If some error occurs return the sender * an icmp packet. Note we can't always generate a meaningful * icmp message because icmp doesn't have a large enough repertoire * of codes and types. * * If not forwarding, just drop the packet. This could be confusing * if ipforwarding was zero but some routing protocol was advancing * us as a gateway to somewhere. However, we must let the routing * protocol deal with that. * */ void ip6_forward(struct mbuf *m, int srcrt) { struct ip6_hdr *ip6 = mtod(m, struct ip6_hdr *); struct sockaddr_in6 *dst = NULL; struct rtentry *rt = NULL; struct route_in6 rin6; int error, type = 0, code = 0; struct mbuf *mcopy = NULL; struct ifnet *origifp; /* maybe unnecessary */ u_int32_t inzone, outzone; struct in6_addr src_in6, dst_in6, odst; #ifdef IPSEC struct secpolicy *sp = NULL; #endif #ifdef SCTP int sw_csum; #endif struct m_tag *fwd_tag; char ip6bufs[INET6_ADDRSTRLEN], ip6bufd[INET6_ADDRSTRLEN]; -#ifdef IPSEC /* - * Check AH/ESP integrity. - */ - /* - * Don't increment ip6s_cantforward because this is the check - * before forwarding packet actually. - */ - if (ipsec6_in_reject(m, NULL)) { - IPSEC6STAT_INC(ips_in_polvio); - m_freem(m); - return; - } -#endif /* IPSEC */ - - /* * Do not forward packets to multicast destination (should be handled * by ip6_mforward(). * Do not forward packets with unspecified source. It was discussed * in July 2000, on the ipngwg mailing list. */ if ((m->m_flags & (M_BCAST|M_MCAST)) != 0 || IN6_IS_ADDR_MULTICAST(&ip6->ip6_dst) || IN6_IS_ADDR_UNSPECIFIED(&ip6->ip6_src)) { IP6STAT_INC(ip6s_cantforward); /* XXX in6_ifstat_inc(rt->rt_ifp, ifs6_in_discard) */ if (V_ip6_log_time + V_ip6_log_interval < time_uptime) { V_ip6_log_time = time_uptime; log(LOG_DEBUG, "cannot forward " "from %s to %s nxt %d received on %s\n", ip6_sprintf(ip6bufs, &ip6->ip6_src), ip6_sprintf(ip6bufd, &ip6->ip6_dst), ip6->ip6_nxt, if_name(m->m_pkthdr.rcvif)); } m_freem(m); return; } +#ifdef IPSEC + /* + * Check if this packet has an active SA and needs to be dropped + * instead of forwarded. + */ + if (ip6_ipsec_fwd(m) != 0) { + IP6STAT_INC(ip6s_cantforward); + m_freem(m); + return; + } +#endif /* IPSEC */ #ifdef IPSTEALTH if (!V_ip6stealth) { #endif if (ip6->ip6_hlim <= IPV6_HLIMDEC) { /* XXX in6_ifstat_inc(rt->rt_ifp, ifs6_in_discard) */ icmp6_error(m, ICMP6_TIME_EXCEEDED, ICMP6_TIME_EXCEED_TRANSIT, 0); return; } ip6->ip6_hlim -= IPV6_HLIMDEC; #ifdef IPSTEALTH } #endif /* * Save at most ICMPV6_PLD_MAXLEN (= the min IPv6 MTU - * size of IPv6 + ICMPv6 headers) bytes of the packet in case * we need to generate an ICMP6 message to the src. * Thanks to M_EXT, in most cases copy will not occur. * * It is important to save it before IPsec processing as IPsec * processing may modify the mbuf. */ mcopy = m_copy(m, 0, imin(m->m_pkthdr.len, ICMPV6_PLD_MAXLEN)); #ifdef IPSEC /* get a security policy for this packet */ sp = ipsec_getpolicybyaddr(m, IPSEC_DIR_OUTBOUND, &error); if (sp == NULL) { IPSEC6STAT_INC(ips_out_inval); IP6STAT_INC(ip6s_cantforward); if (mcopy) { #if 0 /* XXX: what icmp ? */ #else m_freem(mcopy); #endif } m_freem(m); return; } error = 0; /* check policy */ switch (sp->policy) { case IPSEC_POLICY_DISCARD: /* * This packet is just discarded. */ IPSEC6STAT_INC(ips_out_polvio); IP6STAT_INC(ip6s_cantforward); KEY_FREESP(&sp); if (mcopy) { #if 0 /* XXX: what icmp ? */ #else m_freem(mcopy); #endif } m_freem(m); return; case IPSEC_POLICY_BYPASS: case IPSEC_POLICY_NONE: /* no need to do IPsec. */ KEY_FREESP(&sp); goto skip_ipsec; case IPSEC_POLICY_IPSEC: if (sp->req == NULL) { /* XXX should be panic ? */ printf("ip6_forward: No IPsec request specified.\n"); IP6STAT_INC(ip6s_cantforward); KEY_FREESP(&sp); if (mcopy) { #if 0 /* XXX: what icmp ? */ #else m_freem(mcopy); #endif } m_freem(m); return; } /* do IPsec */ break; case IPSEC_POLICY_ENTRUST: default: /* should be panic ?? */ printf("ip6_forward: Invalid policy found. %d\n", sp->policy); KEY_FREESP(&sp); goto skip_ipsec; } { struct ipsecrequest *isr = NULL; /* * when the kernel forwards a packet, it is not proper to apply * IPsec transport mode to the packet is not proper. this check * avoid from this. * at present, if there is even a transport mode SA request in the * security policy, the kernel does not apply IPsec to the packet. * this check is not enough because the following case is valid. * ipsec esp/tunnel/xxx-xxx/require esp/transport//require; */ for (isr = sp->req; isr; isr = isr->next) { if (isr->saidx.mode == IPSEC_MODE_ANY) goto doipsectunnel; if (isr->saidx.mode == IPSEC_MODE_TUNNEL) goto doipsectunnel; } /* * if there's no need for tunnel mode IPsec, skip. */ if (!isr) goto skip_ipsec; doipsectunnel: /* * All the extension headers will become inaccessible * (since they can be encrypted). * Don't panic, we need no more updates to extension headers * on inner IPv6 packet (since they are now encapsulated). * * IPv6 [ESP|AH] IPv6 [extension headers] payload */ /* * If we need to encapsulate the packet, do it here * ipsec6_proces_packet will send the packet using ip6_output */ error = ipsec6_process_packet(m, sp->req); KEY_FREESP(&sp); if (error == EJUSTRETURN) { /* * We had a SP with a level of 'use' and no SA. We * will just continue to process the packet without * IPsec processing. */ error = 0; goto skip_ipsec; } if (error) { /* mbuf is already reclaimed in ipsec6_process_packet. */ switch (error) { case EHOSTUNREACH: case ENETUNREACH: case EMSGSIZE: case ENOBUFS: case ENOMEM: break; default: printf("ip6_output (ipsec): error code %d\n", error); /* FALLTHROUGH */ case ENOENT: /* don't show these error codes to the user */ break; } IP6STAT_INC(ip6s_cantforward); if (mcopy) { #if 0 /* XXX: what icmp ? */ #else m_freem(mcopy); #endif } return; } else { /* * In the FAST IPSec case we have already * re-injected the packet and it has been freed * by the ipsec_done() function. So, just clean * up after ourselves. */ m = NULL; goto freecopy; } } skip_ipsec: #endif again: bzero(&rin6, sizeof(struct route_in6)); dst = (struct sockaddr_in6 *)&rin6.ro_dst; dst->sin6_len = sizeof(struct sockaddr_in6); dst->sin6_family = AF_INET6; dst->sin6_addr = ip6->ip6_dst; again2: rin6.ro_rt = in6_rtalloc1((struct sockaddr *)dst, 0, 0, M_GETFIB(m)); if (rin6.ro_rt != NULL) RT_UNLOCK(rin6.ro_rt); else { IP6STAT_INC(ip6s_noroute); in6_ifstat_inc(m->m_pkthdr.rcvif, ifs6_in_noroute); if (mcopy) { icmp6_error(mcopy, ICMP6_DST_UNREACH, ICMP6_DST_UNREACH_NOROUTE, 0); } goto bad; } rt = rin6.ro_rt; /* * Source scope check: if a packet can't be delivered to its * destination for the reason that the destination is beyond the scope * of the source address, discard the packet and return an icmp6 * destination unreachable error with Code 2 (beyond scope of source * address). We use a local copy of ip6_src, since in6_setscope() * will possibly modify its first argument. * [draft-ietf-ipngwg-icmp-v3-04.txt, Section 3.1] */ src_in6 = ip6->ip6_src; if (in6_setscope(&src_in6, rt->rt_ifp, &outzone)) { /* XXX: this should not happen */ IP6STAT_INC(ip6s_cantforward); IP6STAT_INC(ip6s_badscope); goto bad; } if (in6_setscope(&src_in6, m->m_pkthdr.rcvif, &inzone)) { IP6STAT_INC(ip6s_cantforward); IP6STAT_INC(ip6s_badscope); goto bad; } if (inzone != outzone) { IP6STAT_INC(ip6s_cantforward); IP6STAT_INC(ip6s_badscope); in6_ifstat_inc(rt->rt_ifp, ifs6_in_discard); if (V_ip6_log_time + V_ip6_log_interval < time_uptime) { V_ip6_log_time = time_uptime; log(LOG_DEBUG, "cannot forward " "src %s, dst %s, nxt %d, rcvif %s, outif %s\n", ip6_sprintf(ip6bufs, &ip6->ip6_src), ip6_sprintf(ip6bufd, &ip6->ip6_dst), ip6->ip6_nxt, if_name(m->m_pkthdr.rcvif), if_name(rt->rt_ifp)); } if (mcopy) icmp6_error(mcopy, ICMP6_DST_UNREACH, ICMP6_DST_UNREACH_BEYONDSCOPE, 0); goto bad; } /* * Destination scope check: if a packet is going to break the scope * zone of packet's destination address, discard it. This case should * usually be prevented by appropriately-configured routing table, but * we need an explicit check because we may mistakenly forward the * packet to a different zone by (e.g.) a default route. */ dst_in6 = ip6->ip6_dst; if (in6_setscope(&dst_in6, m->m_pkthdr.rcvif, &inzone) != 0 || in6_setscope(&dst_in6, rt->rt_ifp, &outzone) != 0 || inzone != outzone) { IP6STAT_INC(ip6s_cantforward); IP6STAT_INC(ip6s_badscope); goto bad; } if (m->m_pkthdr.len > IN6_LINKMTU(rt->rt_ifp)) { in6_ifstat_inc(rt->rt_ifp, ifs6_in_toobig); if (mcopy) { u_long mtu; #ifdef IPSEC size_t ipsechdrsiz; #endif /* IPSEC */ mtu = IN6_LINKMTU(rt->rt_ifp); #ifdef IPSEC /* * When we do IPsec tunnel ingress, we need to play * with the link value (decrement IPsec header size * from mtu value). The code is much simpler than v4 * case, as we have the outgoing interface for * encapsulated packet as "rt->rt_ifp". */ ipsechdrsiz = ipsec_hdrsiz(mcopy, IPSEC_DIR_OUTBOUND, NULL); if (ipsechdrsiz < mtu) mtu -= ipsechdrsiz; /* * if mtu becomes less than minimum MTU, * tell minimum MTU (and I'll need to fragment it). */ if (mtu < IPV6_MMTU) mtu = IPV6_MMTU; #endif /* IPSEC */ icmp6_error(mcopy, ICMP6_PACKET_TOO_BIG, 0, mtu); } goto bad; } if (rt->rt_flags & RTF_GATEWAY) dst = (struct sockaddr_in6 *)rt->rt_gateway; /* * If we are to forward the packet using the same interface * as one we got the packet from, perhaps we should send a redirect * to sender to shortcut a hop. * Only send redirect if source is sending directly to us, * and if packet was not source routed (or has any options). * Also, don't send redirect if forwarding using a route * modified by a redirect. */ if (V_ip6_sendredirects && rt->rt_ifp == m->m_pkthdr.rcvif && !srcrt && (rt->rt_flags & (RTF_DYNAMIC|RTF_MODIFIED)) == 0) { if ((rt->rt_ifp->if_flags & IFF_POINTOPOINT) != 0) { /* * If the incoming interface is equal to the outgoing * one, and the link attached to the interface is * point-to-point, then it will be highly probable * that a routing loop occurs. Thus, we immediately * drop the packet and send an ICMPv6 error message. * * type/code is based on suggestion by Rich Draves. * not sure if it is the best pick. */ icmp6_error(mcopy, ICMP6_DST_UNREACH, ICMP6_DST_UNREACH_ADDR, 0); goto bad; } type = ND_REDIRECT; } /* * Fake scoped addresses. Note that even link-local source or * destinaion can appear, if the originating node just sends the * packet to us (without address resolution for the destination). * Since both icmp6_error and icmp6_redirect_output fill the embedded * link identifiers, we can do this stuff after making a copy for * returning an error. */ if ((rt->rt_ifp->if_flags & IFF_LOOPBACK) != 0) { /* * See corresponding comments in ip6_output. * XXX: but is it possible that ip6_forward() sends a packet * to a loopback interface? I don't think so, and thus * I bark here. (jinmei@kame.net) * XXX: it is common to route invalid packets to loopback. * also, the codepath will be visited on use of ::1 in * rthdr. (itojun) */ #if 1 if (0) #else if ((rt->rt_flags & (RTF_BLACKHOLE|RTF_REJECT)) == 0) #endif { printf("ip6_forward: outgoing interface is loopback. " "src %s, dst %s, nxt %d, rcvif %s, outif %s\n", ip6_sprintf(ip6bufs, &ip6->ip6_src), ip6_sprintf(ip6bufd, &ip6->ip6_dst), ip6->ip6_nxt, if_name(m->m_pkthdr.rcvif), if_name(rt->rt_ifp)); } /* we can just use rcvif in forwarding. */ origifp = m->m_pkthdr.rcvif; } else origifp = rt->rt_ifp; /* * clear embedded scope identifiers if necessary. * in6_clearscope will touch the addresses only when necessary. */ in6_clearscope(&ip6->ip6_src); in6_clearscope(&ip6->ip6_dst); /* Jump over all PFIL processing if hooks are not active. */ if (!PFIL_HOOKED(&V_inet6_pfil_hook)) goto pass; odst = ip6->ip6_dst; /* Run through list of hooks for output packets. */ error = pfil_run_hooks(&V_inet6_pfil_hook, &m, rt->rt_ifp, PFIL_OUT, NULL); if (error != 0 || m == NULL) goto freecopy; /* consumed by filter */ ip6 = mtod(m, struct ip6_hdr *); /* See if destination IP address was changed by packet filter. */ if (!IN6_ARE_ADDR_EQUAL(&odst, &ip6->ip6_dst)) { m->m_flags |= M_SKIP_FIREWALL; /* If destination is now ourself drop to ip6_input(). */ if (in6_localip(&ip6->ip6_dst)) { m->m_flags |= M_FASTFWD_OURS; if (m->m_pkthdr.rcvif == NULL) m->m_pkthdr.rcvif = V_loif; if (m->m_pkthdr.csum_flags & CSUM_DELAY_DATA_IPV6) { m->m_pkthdr.csum_flags |= CSUM_DATA_VALID_IPV6 | CSUM_PSEUDO_HDR; m->m_pkthdr.csum_data = 0xffff; } #ifdef SCTP if (m->m_pkthdr.csum_flags & CSUM_SCTP_IPV6) m->m_pkthdr.csum_flags |= CSUM_SCTP_VALID; #endif error = netisr_queue(NETISR_IPV6, m); goto out; } else goto again; /* Redo the routing table lookup. */ } /* See if local, if yes, send it to netisr. */ if (m->m_flags & M_FASTFWD_OURS) { if (m->m_pkthdr.rcvif == NULL) m->m_pkthdr.rcvif = V_loif; if (m->m_pkthdr.csum_flags & CSUM_DELAY_DATA_IPV6) { m->m_pkthdr.csum_flags |= CSUM_DATA_VALID_IPV6 | CSUM_PSEUDO_HDR; m->m_pkthdr.csum_data = 0xffff; } #ifdef SCTP if (m->m_pkthdr.csum_flags & CSUM_SCTP_IPV6) m->m_pkthdr.csum_flags |= CSUM_SCTP_VALID; #endif error = netisr_queue(NETISR_IPV6, m); goto out; } /* Or forward to some other address? */ if ((m->m_flags & M_IP6_NEXTHOP) && (fwd_tag = m_tag_find(m, PACKET_TAG_IPFORWARD, NULL)) != NULL) { dst = (struct sockaddr_in6 *)&rin6.ro_dst; bcopy((fwd_tag+1), dst, sizeof(struct sockaddr_in6)); m->m_flags |= M_SKIP_FIREWALL; m->m_flags &= ~M_IP6_NEXTHOP; m_tag_delete(m, fwd_tag); goto again2; } pass: error = nd6_output(rt->rt_ifp, origifp, m, dst, rt); if (error) { in6_ifstat_inc(rt->rt_ifp, ifs6_out_discard); IP6STAT_INC(ip6s_cantforward); } else { IP6STAT_INC(ip6s_forward); in6_ifstat_inc(rt->rt_ifp, ifs6_out_forward); if (type) IP6STAT_INC(ip6s_redirectsent); else { if (mcopy) goto freecopy; } } if (mcopy == NULL) goto out; switch (error) { case 0: if (type == ND_REDIRECT) { icmp6_redirect_output(mcopy, rt); goto out; } goto freecopy; case EMSGSIZE: /* xxx MTU is constant in PPP? */ goto freecopy; case ENOBUFS: /* Tell source to slow down like source quench in IP? */ goto freecopy; case ENETUNREACH: /* shouldn't happen, checked above */ case EHOSTUNREACH: case ENETDOWN: case EHOSTDOWN: default: type = ICMP6_DST_UNREACH; code = ICMP6_DST_UNREACH_ADDR; break; } icmp6_error(mcopy, type, code, 0); goto out; freecopy: m_freem(mcopy); goto out; bad: m_freem(m); out: if (rt != NULL) RTFREE(rt); } Index: projects/clang350-import/sys/netinet6/ip6_ipsec.c =================================================================== --- projects/clang350-import/sys/netinet6/ip6_ipsec.c (revision 275748) +++ projects/clang350-import/sys/netinet6/ip6_ipsec.c (revision 275749) @@ -1,342 +1,316 @@ /*- * Copyright (c) 1982, 1986, 1988, 1993 * The Regents of the University of California. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 4. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #include __FBSDID("$FreeBSD$"); #include "opt_inet.h" #include "opt_inet6.h" #include "opt_ipsec.h" #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #ifdef IPSEC #include #include #include #include #ifdef IPSEC_DEBUG #include #else #define KEYDEBUG(lev,arg) #endif #endif /*IPSEC*/ #include #include extern struct protosw inet6sw[]; #ifdef INET6 #ifdef IPSEC #ifdef IPSEC_FILTERTUNNEL static VNET_DEFINE(int, ip6_ipsec6_filtertunnel) = 1; #else static VNET_DEFINE(int, ip6_ipsec6_filtertunnel) = 0; #endif #define V_ip6_ipsec6_filtertunnel VNET(ip6_ipsec6_filtertunnel) SYSCTL_DECL(_net_inet6_ipsec6); SYSCTL_INT(_net_inet6_ipsec6, OID_AUTO, filtertunnel, CTLFLAG_VNET | CTLFLAG_RW, &VNET_NAME(ip6_ipsec6_filtertunnel), 0, "If set filter packets from an IPsec tunnel."); #endif /* IPSEC */ #endif /* INET6 */ /* * Check if we have to jump over firewall processing for this packet. * Called from ip6_input(). * 1 = jump over firewall, 0 = packet goes through firewall. */ int ip6_ipsec_filtertunnel(struct mbuf *m) { #ifdef IPSEC /* * Bypass packet filtering for packets previously handled by IPsec. */ if (!V_ip6_ipsec6_filtertunnel && m_tag_find(m, PACKET_TAG_IPSEC_IN_DONE, NULL) != NULL) return 1; #endif return 0; } /* * Check if this packet has an active SA and needs to be dropped instead * of forwarded. - * Called from ip6_input(). + * Called from ip6_forward(). * 1 = drop packet, 0 = forward packet. */ int ip6_ipsec_fwd(struct mbuf *m) { -#ifdef IPSEC - struct secpolicy *sp; - int error; - sp = ipsec_getpolicybyaddr(m, IPSEC_DIR_INBOUND, &error); - if (sp != NULL) { - /* - * Check security policy against packet attributes. - */ - error = ipsec_in_reject(sp, m); - KEY_FREESP(&sp); - } - if (error != 0) - return (1); -#endif /* IPSEC */ +#ifdef IPSEC + return (ipsec6_in_reject(m, NULL)); +#else return (0); +#endif /* !IPSEC */ } /* * Check if protocol type doesn't have a further header and do IPSEC * decryption or reject right now. Protocols with further headers get * their IPSEC treatment within the protocol specific processing. * Called from ip6_input(). * 1 = drop packet, 0 = continue processing packet. */ int ip6_ipsec_input(struct mbuf *m, int nxt) { + #ifdef IPSEC - struct secpolicy *sp; - int error; /* * enforce IPsec policy checking if we are seeing last header. * note that we do not visit this with protocols with pcb layer * code - like udp/tcp/raw ip. */ - if ((inet6sw[ip6_protox[nxt]].pr_flags & PR_LASTHDR) != 0 && - ipsec6_in_reject(m, NULL)) { - sp = ipsec_getpolicybyaddr(m, IPSEC_DIR_INBOUND, &error); - if (sp != NULL) { - /* - * Check security policy against packet attributes. - */ - error = ipsec_in_reject(sp, m); - KEY_FREESP(&sp); - } else { - /* XXX error stat??? */ - error = EINVAL; - DPRINTF(("%s: no SP, packet discarded\n", __func__));/*XXX*/ - } - if (error != 0) - return (1); - } + if ((inet6sw[ip6_protox[nxt]].pr_flags & PR_LASTHDR) != 0) + return (ipsec6_in_reject(m, NULL)); #endif /* IPSEC */ return (0); } /* * Called from ip6_output(). * 1 = drop packet, 0 = continue processing packet, * -1 = packet was reinjected and stop processing packet */ int ip6_ipsec_output(struct mbuf **m, struct inpcb *inp, int *error) { #ifdef IPSEC struct secpolicy *sp; /* * Check the security policy (SP) for the packet and, if * required, do IPsec-related processing. There are two * cases here; the first time a packet is sent through * it will be untagged and handled by ipsec4_checkpolicy. * If the packet is resubmitted to ip6_output (e.g. after * AH, ESP, etc. processing), there will be a tag to bypass * the lookup and related policy checking. */ if (m_tag_find(*m, PACKET_TAG_IPSEC_OUT_DONE, NULL) != NULL) { *error = 0; return (0); } sp = ipsec4_checkpolicy(*m, IPSEC_DIR_OUTBOUND, error, inp); /* * There are four return cases: * sp != NULL apply IPsec policy * sp == NULL, error == 0 no IPsec handling needed * sp == NULL, error == -EINVAL discard packet w/o error * sp == NULL, error != 0 discard packet, report error */ if (sp != NULL) { /* * Do delayed checksums now because we send before * this is done in the normal processing path. */ #ifdef INET if ((*m)->m_pkthdr.csum_flags & CSUM_DELAY_DATA) { in_delayed_cksum(*m); (*m)->m_pkthdr.csum_flags &= ~CSUM_DELAY_DATA; } #endif if ((*m)->m_pkthdr.csum_flags & CSUM_DELAY_DATA_IPV6) { in6_delayed_cksum(*m, (*m)->m_pkthdr.len - sizeof(struct ip6_hdr), sizeof(struct ip6_hdr)); (*m)->m_pkthdr.csum_flags &= ~CSUM_DELAY_DATA_IPV6; } #ifdef SCTP if ((*m)->m_pkthdr.csum_flags & CSUM_SCTP_IPV6) { sctp_delayed_cksum(*m, sizeof(struct ip6_hdr)); (*m)->m_pkthdr.csum_flags &= ~CSUM_SCTP_IPV6; } #endif /* NB: callee frees mbuf */ *error = ipsec6_process_packet(*m, sp->req); if (*error == EJUSTRETURN) { /* * We had a SP with a level of 'use' and no SA. We * will just continue to process the packet without * IPsec processing. */ *error = 0; goto done; } /* * Preserve KAME behaviour: ENOENT can be returned * when an SA acquire is in progress. Don't propagate * this to user-level; it confuses applications. * * XXX this will go away when the SADB is redone. */ if (*error == ENOENT) *error = 0; goto reinjected; } else { /* sp == NULL */ if (*error != 0) { /* * Hack: -EINVAL is used to signal that a packet * should be silently discarded. This is typically * because we asked key management for an SA and * it was delayed (e.g. kicked up to IKE). */ if (*error == -EINVAL) *error = 0; goto bad; } /* No IPsec processing for this packet. */ } done: if (sp != NULL) KEY_FREESP(&sp); return 0; reinjected: if (sp != NULL) KEY_FREESP(&sp); return -1; bad: if (sp != NULL) KEY_FREESP(&sp); return 1; #endif /* IPSEC */ return 0; } #if 0 /* * Compute the MTU for a forwarded packet that gets IPSEC encapsulated. * Called from ip_forward(). * Returns MTU suggestion for ICMP needfrag reply. */ int ip6_ipsec_mtu(struct mbuf *m) { int mtu = 0; /* * If the packet is routed over IPsec tunnel, tell the * originator the tunnel MTU. * tunnel MTU = if MTU - sizeof(IP) - ESP/AH hdrsiz * XXX quickhack!!! */ #ifdef IPSEC struct secpolicy *sp = NULL; int ipsecerror; int ipsechdr; struct route *ro; sp = ipsec_getpolicybyaddr(m, IPSEC_DIR_OUTBOUND, IP_FORWARDING, &ipsecerror); if (sp != NULL) { /* count IPsec header size */ ipsechdr = ipsec_hdrsiz(m, IPSEC_DIR_OUTBOUND, NULL); /* * find the correct route for outer IPv4 * header, compute tunnel MTU. */ if (sp->req != NULL && sp->req->sav != NULL && sp->req->sav->sah != NULL) { ro = &sp->req->sav->sah->route_cache.sa_route; if (ro->ro_rt && ro->ro_rt->rt_ifp) { mtu = ro->ro_rt->rt_mtu ? ro->ro_rt->rt_mtu : ro->ro_rt->rt_ifp->if_mtu; mtu -= ipsechdr; } } KEY_FREESP(&sp); } #endif /* IPSEC */ /* XXX else case missing. */ return mtu; } #endif Index: projects/clang350-import/sys/netinet6/raw_ip6.c =================================================================== --- projects/clang350-import/sys/netinet6/raw_ip6.c (revision 275748) +++ projects/clang350-import/sys/netinet6/raw_ip6.c (revision 275749) @@ -1,926 +1,924 @@ /*- * Copyright (C) 1995, 1996, 1997, and 1998 WIDE Project. * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of the project nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE PROJECT AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE PROJECT OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ /*- * Copyright (c) 1982, 1986, 1988, 1993 * The Regents of the University of California. * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 4. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * @(#)raw_ip.c 8.2 (Berkeley) 1/4/94 */ #include __FBSDID("$FreeBSD$"); #include "opt_ipsec.h" #include "opt_inet6.h" #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #ifdef IPSEC #include #include #endif /* IPSEC */ #include #define satosin6(sa) ((struct sockaddr_in6 *)(sa)) #define ifatoia6(ifa) ((struct in6_ifaddr *)(ifa)) /* * Raw interface to IP6 protocol. */ VNET_DECLARE(struct inpcbhead, ripcb); VNET_DECLARE(struct inpcbinfo, ripcbinfo); #define V_ripcb VNET(ripcb) #define V_ripcbinfo VNET(ripcbinfo) extern u_long rip_sendspace; extern u_long rip_recvspace; VNET_PCPUSTAT_DEFINE(struct rip6stat, rip6stat); VNET_PCPUSTAT_SYSINIT(rip6stat); #ifdef VIMAGE VNET_PCPUSTAT_SYSUNINIT(rip6stat); #endif /* VIMAGE */ /* * Hooks for multicast routing. They all default to NULL, so leave them not * initialized and rely on BSS being set to 0. */ /* * The socket used to communicate with the multicast routing daemon. */ VNET_DEFINE(struct socket *, ip6_mrouter); /* * The various mrouter functions. */ int (*ip6_mrouter_set)(struct socket *, struct sockopt *); int (*ip6_mrouter_get)(struct socket *, struct sockopt *); int (*ip6_mrouter_done)(void); int (*ip6_mforward)(struct ip6_hdr *, struct ifnet *, struct mbuf *); int (*mrt6_ioctl)(u_long, caddr_t); /* * Setup generic address and protocol structures for raw_input routine, then * pass them along with mbuf chain. */ int rip6_input(struct mbuf **mp, int *offp, int proto) { struct ifnet *ifp; struct mbuf *m = *mp; register struct ip6_hdr *ip6 = mtod(m, struct ip6_hdr *); register struct inpcb *in6p; struct inpcb *last = 0; struct mbuf *opts = NULL; struct sockaddr_in6 fromsa; RIP6STAT_INC(rip6s_ipackets); init_sin6(&fromsa, m); /* general init */ ifp = m->m_pkthdr.rcvif; INP_INFO_RLOCK(&V_ripcbinfo); LIST_FOREACH(in6p, &V_ripcb, inp_list) { /* XXX inp locking */ if ((in6p->inp_vflag & INP_IPV6) == 0) continue; if (in6p->inp_ip_p && in6p->inp_ip_p != proto) continue; if (!IN6_IS_ADDR_UNSPECIFIED(&in6p->in6p_laddr) && !IN6_ARE_ADDR_EQUAL(&in6p->in6p_laddr, &ip6->ip6_dst)) continue; if (!IN6_IS_ADDR_UNSPECIFIED(&in6p->in6p_faddr) && !IN6_ARE_ADDR_EQUAL(&in6p->in6p_faddr, &ip6->ip6_src)) continue; if (jailed_without_vnet(in6p->inp_cred)) { /* * Allow raw socket in jail to receive multicast; * assume process had PRIV_NETINET_RAW at attach, * and fall through into normal filter path if so. */ if (!IN6_IS_ADDR_MULTICAST(&ip6->ip6_dst) && prison_check_ip6(in6p->inp_cred, &ip6->ip6_dst) != 0) continue; } INP_RLOCK(in6p); if (in6p->in6p_cksum != -1) { RIP6STAT_INC(rip6s_isum); if (in6_cksum(m, proto, *offp, m->m_pkthdr.len - *offp)) { INP_RUNLOCK(in6p); RIP6STAT_INC(rip6s_badsum); continue; } } /* * If this raw socket has multicast state, and we * have received a multicast, check if this socket * should receive it, as multicast filtering is now * the responsibility of the transport layer. */ if (in6p->in6p_moptions && IN6_IS_ADDR_MULTICAST(&ip6->ip6_dst)) { /* * If the incoming datagram is for MLD, allow it * through unconditionally to the raw socket. * * Use the M_RTALERT_MLD flag to check for MLD * traffic without having to inspect the mbuf chain * more deeply, as all MLDv1/v2 host messages MUST * contain the Router Alert option. * * In the case of MLDv1, we may not have explicitly * joined the group, and may have set IFF_ALLMULTI * on the interface. im6o_mc_filter() may discard * control traffic we actually need to see. * * Userland multicast routing daemons should continue * filter the control traffic appropriately. */ int blocked; blocked = MCAST_PASS; if ((m->m_flags & M_RTALERT_MLD) == 0) { struct sockaddr_in6 mcaddr; bzero(&mcaddr, sizeof(struct sockaddr_in6)); mcaddr.sin6_len = sizeof(struct sockaddr_in6); mcaddr.sin6_family = AF_INET6; mcaddr.sin6_addr = ip6->ip6_dst; blocked = im6o_mc_filter(in6p->in6p_moptions, ifp, (struct sockaddr *)&mcaddr, (struct sockaddr *)&fromsa); } if (blocked != MCAST_PASS) { IP6STAT_INC(ip6s_notmember); INP_RUNLOCK(in6p); continue; } } if (last != NULL) { struct mbuf *n = m_copy(m, 0, (int)M_COPYALL); #ifdef IPSEC /* * Check AH/ESP integrity. */ if (n && ipsec6_in_reject(n, last)) { m_freem(n); - IPSEC6STAT_INC(ips_in_polvio); /* Do not inject data into pcb. */ } else #endif /* IPSEC */ if (n) { if (last->inp_flags & INP_CONTROLOPTS || last->inp_socket->so_options & SO_TIMESTAMP) ip6_savecontrol(last, n, &opts); /* strip intermediate headers */ m_adj(n, *offp); if (sbappendaddr(&last->inp_socket->so_rcv, (struct sockaddr *)&fromsa, n, opts) == 0) { m_freem(n); if (opts) m_freem(opts); RIP6STAT_INC(rip6s_fullsock); } else sorwakeup(last->inp_socket); opts = NULL; } INP_RUNLOCK(last); } last = in6p; } INP_INFO_RUNLOCK(&V_ripcbinfo); #ifdef IPSEC /* * Check AH/ESP integrity. */ if ((last != NULL) && ipsec6_in_reject(m, last)) { m_freem(m); - IPSEC6STAT_INC(ips_in_polvio); IP6STAT_DEC(ip6s_delivered); /* Do not inject data into pcb. */ INP_RUNLOCK(last); } else #endif /* IPSEC */ if (last != NULL) { if (last->inp_flags & INP_CONTROLOPTS || last->inp_socket->so_options & SO_TIMESTAMP) ip6_savecontrol(last, m, &opts); /* Strip intermediate headers. */ m_adj(m, *offp); if (sbappendaddr(&last->inp_socket->so_rcv, (struct sockaddr *)&fromsa, m, opts) == 0) { m_freem(m); if (opts) m_freem(opts); RIP6STAT_INC(rip6s_fullsock); } else sorwakeup(last->inp_socket); INP_RUNLOCK(last); } else { RIP6STAT_INC(rip6s_nosock); if (m->m_flags & M_MCAST) RIP6STAT_INC(rip6s_nosockmcast); if (proto == IPPROTO_NONE) m_freem(m); else { char *prvnxtp = ip6_get_prevhdr(m, *offp); /* XXX */ icmp6_error(m, ICMP6_PARAM_PROB, ICMP6_PARAMPROB_NEXTHEADER, prvnxtp - mtod(m, char *)); } IP6STAT_DEC(ip6s_delivered); } return (IPPROTO_DONE); } void rip6_ctlinput(int cmd, struct sockaddr *sa, void *d) { struct ip6_hdr *ip6; struct mbuf *m; int off = 0; struct ip6ctlparam *ip6cp = NULL; const struct sockaddr_in6 *sa6_src = NULL; void *cmdarg; struct inpcb *(*notify)(struct inpcb *, int) = in6_rtchange; if (sa->sa_family != AF_INET6 || sa->sa_len != sizeof(struct sockaddr_in6)) return; if ((unsigned)cmd >= PRC_NCMDS) return; if (PRC_IS_REDIRECT(cmd)) notify = in6_rtchange, d = NULL; else if (cmd == PRC_HOSTDEAD) d = NULL; else if (inet6ctlerrmap[cmd] == 0) return; /* * If the parameter is from icmp6, decode it. */ if (d != NULL) { ip6cp = (struct ip6ctlparam *)d; m = ip6cp->ip6c_m; ip6 = ip6cp->ip6c_ip6; off = ip6cp->ip6c_off; cmdarg = ip6cp->ip6c_cmdarg; sa6_src = ip6cp->ip6c_src; } else { m = NULL; ip6 = NULL; cmdarg = NULL; sa6_src = &sa6_any; } (void) in6_pcbnotify(&V_ripcbinfo, sa, 0, (const struct sockaddr *)sa6_src, 0, cmd, cmdarg, notify); } /* * Generate IPv6 header and pass packet to ip6_output. Tack on options user * may have setup with control call. */ int rip6_output(struct mbuf *m, struct socket *so, ...) { struct mbuf *control; struct m_tag *mtag; struct sockaddr_in6 *dstsock; struct in6_addr *dst; struct ip6_hdr *ip6; struct inpcb *in6p; u_int plen = m->m_pkthdr.len; int error = 0; struct ip6_pktopts opt, *optp; struct ifnet *oifp = NULL; int type = 0, code = 0; /* for ICMPv6 output statistics only */ int scope_ambiguous = 0; int use_defzone = 0; struct in6_addr in6a; va_list ap; va_start(ap, so); dstsock = va_arg(ap, struct sockaddr_in6 *); control = va_arg(ap, struct mbuf *); va_end(ap); in6p = sotoinpcb(so); INP_WLOCK(in6p); dst = &dstsock->sin6_addr; if (control != NULL) { if ((error = ip6_setpktopts(control, &opt, in6p->in6p_outputopts, so->so_cred, so->so_proto->pr_protocol)) != 0) { goto bad; } optp = &opt; } else optp = in6p->in6p_outputopts; /* * Check and convert scope zone ID into internal form. * * XXX: we may still need to determine the zone later. */ if (!(so->so_state & SS_ISCONNECTED)) { if (!optp || !optp->ip6po_pktinfo || !optp->ip6po_pktinfo->ipi6_ifindex) use_defzone = V_ip6_use_defzone; if (dstsock->sin6_scope_id == 0 && !use_defzone) scope_ambiguous = 1; if ((error = sa6_embedscope(dstsock, use_defzone)) != 0) goto bad; } /* * For an ICMPv6 packet, we should know its type and code to update * statistics. */ if (so->so_proto->pr_protocol == IPPROTO_ICMPV6) { struct icmp6_hdr *icmp6; if (m->m_len < sizeof(struct icmp6_hdr) && (m = m_pullup(m, sizeof(struct icmp6_hdr))) == NULL) { error = ENOBUFS; goto bad; } icmp6 = mtod(m, struct icmp6_hdr *); type = icmp6->icmp6_type; code = icmp6->icmp6_code; } M_PREPEND(m, sizeof(*ip6), M_NOWAIT); if (m == NULL) { error = ENOBUFS; goto bad; } ip6 = mtod(m, struct ip6_hdr *); /* * Source address selection. */ error = in6_selectsrc(dstsock, optp, in6p, NULL, so->so_cred, &oifp, &in6a); if (error) goto bad; error = prison_check_ip6(in6p->inp_cred, &in6a); if (error != 0) goto bad; ip6->ip6_src = in6a; if (oifp && scope_ambiguous) { /* * Application should provide a proper zone ID or the use of * default zone IDs should be enabled. Unfortunately, some * applications do not behave as it should, so we need a * workaround. Even if an appropriate ID is not determined * (when it's required), if we can determine the outgoing * interface. determine the zone ID based on the interface. */ error = in6_setscope(&dstsock->sin6_addr, oifp, NULL); if (error != 0) goto bad; } ip6->ip6_dst = dstsock->sin6_addr; /* * Fill in the rest of the IPv6 header fields. */ ip6->ip6_flow = (ip6->ip6_flow & ~IPV6_FLOWINFO_MASK) | (in6p->inp_flow & IPV6_FLOWINFO_MASK); ip6->ip6_vfc = (ip6->ip6_vfc & ~IPV6_VERSION_MASK) | (IPV6_VERSION & IPV6_VERSION_MASK); /* * ip6_plen will be filled in ip6_output, so not fill it here. */ ip6->ip6_nxt = in6p->inp_ip_p; ip6->ip6_hlim = in6_selecthlim(in6p, oifp); if (so->so_proto->pr_protocol == IPPROTO_ICMPV6 || in6p->in6p_cksum != -1) { struct mbuf *n; int off; u_int16_t *p; /* Compute checksum. */ if (so->so_proto->pr_protocol == IPPROTO_ICMPV6) off = offsetof(struct icmp6_hdr, icmp6_cksum); else off = in6p->in6p_cksum; if (plen < off + 1) { error = EINVAL; goto bad; } off += sizeof(struct ip6_hdr); n = m; while (n && n->m_len <= off) { off -= n->m_len; n = n->m_next; } if (!n) goto bad; p = (u_int16_t *)(mtod(n, caddr_t) + off); *p = 0; *p = in6_cksum(m, ip6->ip6_nxt, sizeof(*ip6), plen); } /* * Send RA/RS messages to user land for protection, before sending * them to rtadvd/rtsol. */ if ((send_sendso_input_hook != NULL) && so->so_proto->pr_protocol == IPPROTO_ICMPV6) { switch (type) { case ND_ROUTER_ADVERT: case ND_ROUTER_SOLICIT: mtag = m_tag_get(PACKET_TAG_ND_OUTGOING, sizeof(unsigned short), M_NOWAIT); if (mtag == NULL) goto bad; m_tag_prepend(m, mtag); } } error = ip6_output(m, optp, NULL, 0, in6p->in6p_moptions, &oifp, in6p); if (so->so_proto->pr_protocol == IPPROTO_ICMPV6) { if (oifp) icmp6_ifoutstat_inc(oifp, type, code); ICMP6STAT_INC(icp6s_outhist[type]); } else RIP6STAT_INC(rip6s_opackets); goto freectl; bad: if (m) m_freem(m); freectl: if (control != NULL) { ip6_clearpktopts(&opt, -1); m_freem(control); } INP_WUNLOCK(in6p); return (error); } /* * Raw IPv6 socket option processing. */ int rip6_ctloutput(struct socket *so, struct sockopt *sopt) { struct inpcb *inp; int error; if (sopt->sopt_level == IPPROTO_ICMPV6) /* * XXX: is it better to call icmp6_ctloutput() directly * from protosw? */ return (icmp6_ctloutput(so, sopt)); else if (sopt->sopt_level != IPPROTO_IPV6) { if (sopt->sopt_level == SOL_SOCKET && sopt->sopt_name == SO_SETFIB) { inp = sotoinpcb(so); INP_WLOCK(inp); inp->inp_inc.inc_fibnum = so->so_fibnum; INP_WUNLOCK(inp); return (0); } return (EINVAL); } error = 0; switch (sopt->sopt_dir) { case SOPT_GET: switch (sopt->sopt_name) { case MRT6_INIT: case MRT6_DONE: case MRT6_ADD_MIF: case MRT6_DEL_MIF: case MRT6_ADD_MFC: case MRT6_DEL_MFC: case MRT6_PIM: error = ip6_mrouter_get ? ip6_mrouter_get(so, sopt) : EOPNOTSUPP; break; case IPV6_CHECKSUM: error = ip6_raw_ctloutput(so, sopt); break; default: error = ip6_ctloutput(so, sopt); break; } break; case SOPT_SET: switch (sopt->sopt_name) { case MRT6_INIT: case MRT6_DONE: case MRT6_ADD_MIF: case MRT6_DEL_MIF: case MRT6_ADD_MFC: case MRT6_DEL_MFC: case MRT6_PIM: error = ip6_mrouter_set ? ip6_mrouter_set(so, sopt) : EOPNOTSUPP; break; case IPV6_CHECKSUM: error = ip6_raw_ctloutput(so, sopt); break; default: error = ip6_ctloutput(so, sopt); break; } break; } return (error); } static int rip6_attach(struct socket *so, int proto, struct thread *td) { struct inpcb *inp; struct icmp6_filter *filter; int error; inp = sotoinpcb(so); KASSERT(inp == NULL, ("rip6_attach: inp != NULL")); error = priv_check(td, PRIV_NETINET_RAW); if (error) return (error); error = soreserve(so, rip_sendspace, rip_recvspace); if (error) return (error); filter = malloc(sizeof(struct icmp6_filter), M_PCB, M_NOWAIT); if (filter == NULL) return (ENOMEM); INP_INFO_WLOCK(&V_ripcbinfo); error = in_pcballoc(so, &V_ripcbinfo); if (error) { INP_INFO_WUNLOCK(&V_ripcbinfo); free(filter, M_PCB); return (error); } inp = (struct inpcb *)so->so_pcb; INP_INFO_WUNLOCK(&V_ripcbinfo); inp->inp_vflag |= INP_IPV6; inp->inp_ip_p = (long)proto; inp->in6p_hops = -1; /* use kernel default */ inp->in6p_cksum = -1; inp->in6p_icmp6filt = filter; ICMP6_FILTER_SETPASSALL(inp->in6p_icmp6filt); INP_WUNLOCK(inp); return (0); } static void rip6_detach(struct socket *so) { struct inpcb *inp; inp = sotoinpcb(so); KASSERT(inp != NULL, ("rip6_detach: inp == NULL")); if (so == V_ip6_mrouter && ip6_mrouter_done) ip6_mrouter_done(); /* xxx: RSVP */ INP_INFO_WLOCK(&V_ripcbinfo); INP_WLOCK(inp); free(inp->in6p_icmp6filt, M_PCB); in_pcbdetach(inp); in_pcbfree(inp); INP_INFO_WUNLOCK(&V_ripcbinfo); } /* XXXRW: This can't ever be called. */ static void rip6_abort(struct socket *so) { struct inpcb *inp; inp = sotoinpcb(so); KASSERT(inp != NULL, ("rip6_abort: inp == NULL")); soisdisconnected(so); } static void rip6_close(struct socket *so) { struct inpcb *inp; inp = sotoinpcb(so); KASSERT(inp != NULL, ("rip6_close: inp == NULL")); soisdisconnected(so); } static int rip6_disconnect(struct socket *so) { struct inpcb *inp; inp = sotoinpcb(so); KASSERT(inp != NULL, ("rip6_disconnect: inp == NULL")); if ((so->so_state & SS_ISCONNECTED) == 0) return (ENOTCONN); inp->in6p_faddr = in6addr_any; rip6_abort(so); return (0); } static int rip6_bind(struct socket *so, struct sockaddr *nam, struct thread *td) { struct inpcb *inp; struct sockaddr_in6 *addr = (struct sockaddr_in6 *)nam; struct ifaddr *ifa = NULL; int error = 0; inp = sotoinpcb(so); KASSERT(inp != NULL, ("rip6_bind: inp == NULL")); if (nam->sa_len != sizeof(*addr)) return (EINVAL); if ((error = prison_check_ip6(td->td_ucred, &addr->sin6_addr)) != 0) return (error); if (TAILQ_EMPTY(&V_ifnet) || addr->sin6_family != AF_INET6) return (EADDRNOTAVAIL); if ((error = sa6_embedscope(addr, V_ip6_use_defzone)) != 0) return (error); if (!IN6_IS_ADDR_UNSPECIFIED(&addr->sin6_addr) && (ifa = ifa_ifwithaddr((struct sockaddr *)addr)) == NULL) return (EADDRNOTAVAIL); if (ifa != NULL && ((struct in6_ifaddr *)ifa)->ia6_flags & (IN6_IFF_ANYCAST|IN6_IFF_NOTREADY| IN6_IFF_DETACHED|IN6_IFF_DEPRECATED)) { ifa_free(ifa); return (EADDRNOTAVAIL); } if (ifa != NULL) ifa_free(ifa); INP_INFO_WLOCK(&V_ripcbinfo); INP_WLOCK(inp); inp->in6p_laddr = addr->sin6_addr; INP_WUNLOCK(inp); INP_INFO_WUNLOCK(&V_ripcbinfo); return (0); } static int rip6_connect(struct socket *so, struct sockaddr *nam, struct thread *td) { struct inpcb *inp; struct sockaddr_in6 *addr = (struct sockaddr_in6 *)nam; struct in6_addr in6a; struct ifnet *ifp = NULL; int error = 0, scope_ambiguous = 0; inp = sotoinpcb(so); KASSERT(inp != NULL, ("rip6_connect: inp == NULL")); if (nam->sa_len != sizeof(*addr)) return (EINVAL); if (TAILQ_EMPTY(&V_ifnet)) return (EADDRNOTAVAIL); if (addr->sin6_family != AF_INET6) return (EAFNOSUPPORT); /* * Application should provide a proper zone ID or the use of default * zone IDs should be enabled. Unfortunately, some applications do * not behave as it should, so we need a workaround. Even if an * appropriate ID is not determined, we'll see if we can determine * the outgoing interface. If we can, determine the zone ID based on * the interface below. */ if (addr->sin6_scope_id == 0 && !V_ip6_use_defzone) scope_ambiguous = 1; if ((error = sa6_embedscope(addr, V_ip6_use_defzone)) != 0) return (error); INP_INFO_WLOCK(&V_ripcbinfo); INP_WLOCK(inp); /* Source address selection. XXX: need pcblookup? */ error = in6_selectsrc(addr, inp->in6p_outputopts, inp, NULL, so->so_cred, &ifp, &in6a); if (error) { INP_WUNLOCK(inp); INP_INFO_WUNLOCK(&V_ripcbinfo); return (error); } /* XXX: see above */ if (ifp && scope_ambiguous && (error = in6_setscope(&addr->sin6_addr, ifp, NULL)) != 0) { INP_WUNLOCK(inp); INP_INFO_WUNLOCK(&V_ripcbinfo); return (error); } inp->in6p_faddr = addr->sin6_addr; inp->in6p_laddr = in6a; soisconnected(so); INP_WUNLOCK(inp); INP_INFO_WUNLOCK(&V_ripcbinfo); return (0); } static int rip6_shutdown(struct socket *so) { struct inpcb *inp; inp = sotoinpcb(so); KASSERT(inp != NULL, ("rip6_shutdown: inp == NULL")); INP_WLOCK(inp); socantsendmore(so); INP_WUNLOCK(inp); return (0); } static int rip6_send(struct socket *so, int flags, struct mbuf *m, struct sockaddr *nam, struct mbuf *control, struct thread *td) { struct inpcb *inp; struct sockaddr_in6 tmp; struct sockaddr_in6 *dst; int ret; inp = sotoinpcb(so); KASSERT(inp != NULL, ("rip6_send: inp == NULL")); /* Always copy sockaddr to avoid overwrites. */ /* Unlocked read. */ if (so->so_state & SS_ISCONNECTED) { if (nam) { m_freem(m); return (EISCONN); } /* XXX */ bzero(&tmp, sizeof(tmp)); tmp.sin6_family = AF_INET6; tmp.sin6_len = sizeof(struct sockaddr_in6); INP_RLOCK(inp); bcopy(&inp->in6p_faddr, &tmp.sin6_addr, sizeof(struct in6_addr)); INP_RUNLOCK(inp); dst = &tmp; } else { if (nam == NULL) { m_freem(m); return (ENOTCONN); } if (nam->sa_len != sizeof(struct sockaddr_in6)) { m_freem(m); return (EINVAL); } tmp = *(struct sockaddr_in6 *)nam; dst = &tmp; if (dst->sin6_family == AF_UNSPEC) { /* * XXX: we allow this case for backward * compatibility to buggy applications that * rely on old (and wrong) kernel behavior. */ log(LOG_INFO, "rip6 SEND: address family is " "unspec. Assume AF_INET6\n"); dst->sin6_family = AF_INET6; } else if (dst->sin6_family != AF_INET6) { m_freem(m); return(EAFNOSUPPORT); } } ret = rip6_output(m, so, dst, control); return (ret); } struct pr_usrreqs rip6_usrreqs = { .pru_abort = rip6_abort, .pru_attach = rip6_attach, .pru_bind = rip6_bind, .pru_connect = rip6_connect, .pru_control = in6_control, .pru_detach = rip6_detach, .pru_disconnect = rip6_disconnect, .pru_peeraddr = in6_getpeeraddr, .pru_send = rip6_send, .pru_shutdown = rip6_shutdown, .pru_sockaddr = in6_getsockaddr, .pru_close = rip6_close, }; Index: projects/clang350-import/sys/netinet6/udp6_usrreq.c =================================================================== --- projects/clang350-import/sys/netinet6/udp6_usrreq.c (revision 275748) +++ projects/clang350-import/sys/netinet6/udp6_usrreq.c (revision 275749) @@ -1,1250 +1,1249 @@ /*- * Copyright (C) 1995, 1996, 1997, and 1998 WIDE Project. * Copyright (c) 2010-2011 Juniper Networks, Inc. * Copyright (c) 2014 Kevin Lo * All rights reserved. * * Portions of this software were developed by Robert N. M. Watson under * contract to Juniper Networks, Inc. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of the project nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE PROJECT AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE PROJECT OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * $KAME: udp6_usrreq.c,v 1.27 2001/05/21 05:45:10 jinmei Exp $ * $KAME: udp6_output.c,v 1.31 2001/05/21 16:39:15 jinmei Exp $ */ /*- * Copyright (c) 1982, 1986, 1988, 1990, 1993, 1995 * The Regents of the University of California. * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 4. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * @(#)udp_usrreq.c 8.6 (Berkeley) 5/23/95 */ #include __FBSDID("$FreeBSD$"); #include "opt_inet.h" #include "opt_inet6.h" #include "opt_ipfw.h" #include "opt_ipsec.h" #include "opt_rss.h" #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #ifdef IPSEC #include #include #endif /* IPSEC */ #include /* * UDP protocol implementation. * Per RFC 768, August, 1980. */ extern struct protosw inetsw[]; static void udp6_detach(struct socket *so); static void udp6_append(struct inpcb *inp, struct mbuf *n, int off, struct sockaddr_in6 *fromsa) { struct socket *so; struct mbuf *opts; struct udpcb *up; INP_LOCK_ASSERT(inp); /* * Engage the tunneling protocol. */ up = intoudpcb(inp); if (up->u_tun_func != NULL) { (*up->u_tun_func)(n, off, inp, (struct sockaddr *)fromsa, up->u_tun_ctx); return; } #ifdef IPSEC /* Check AH/ESP integrity. */ if (ipsec6_in_reject(n, inp)) { m_freem(n); - IPSEC6STAT_INC(ips_in_polvio); return; } #endif /* IPSEC */ #ifdef MAC if (mac_inpcb_check_deliver(inp, n) != 0) { m_freem(n); return; } #endif opts = NULL; if (inp->inp_flags & INP_CONTROLOPTS || inp->inp_socket->so_options & SO_TIMESTAMP) ip6_savecontrol(inp, n, &opts); m_adj(n, off + sizeof(struct udphdr)); so = inp->inp_socket; SOCKBUF_LOCK(&so->so_rcv); if (sbappendaddr_locked(&so->so_rcv, (struct sockaddr *)fromsa, n, opts) == 0) { SOCKBUF_UNLOCK(&so->so_rcv); m_freem(n); if (opts) m_freem(opts); UDPSTAT_INC(udps_fullsock); } else sorwakeup_locked(so); } int udp6_input(struct mbuf **mp, int *offp, int proto) { struct mbuf *m = *mp; struct ifnet *ifp; struct ip6_hdr *ip6; struct udphdr *uh; struct inpcb *inp; struct inpcbinfo *pcbinfo; struct udpcb *up; int off = *offp; int cscov_partial; int plen, ulen; struct sockaddr_in6 fromsa; struct m_tag *fwd_tag; uint16_t uh_sum; uint8_t nxt; ifp = m->m_pkthdr.rcvif; ip6 = mtod(m, struct ip6_hdr *); #ifndef PULLDOWN_TEST IP6_EXTHDR_CHECK(m, off, sizeof(struct udphdr), IPPROTO_DONE); ip6 = mtod(m, struct ip6_hdr *); uh = (struct udphdr *)((caddr_t)ip6 + off); #else IP6_EXTHDR_GET(uh, struct udphdr *, m, off, sizeof(*uh)); if (!uh) return (IPPROTO_DONE); #endif UDPSTAT_INC(udps_ipackets); /* * Destination port of 0 is illegal, based on RFC768. */ if (uh->uh_dport == 0) goto badunlocked; plen = ntohs(ip6->ip6_plen) - off + sizeof(*ip6); ulen = ntohs((u_short)uh->uh_ulen); nxt = ip6->ip6_nxt; cscov_partial = (nxt == IPPROTO_UDPLITE) ? 1 : 0; if (nxt == IPPROTO_UDPLITE) { /* Zero means checksum over the complete packet. */ if (ulen == 0) ulen = plen; if (ulen == plen) cscov_partial = 0; if ((ulen < sizeof(struct udphdr)) || (ulen > plen)) { /* XXX: What is the right UDPLite MIB counter? */ goto badunlocked; } if (uh->uh_sum == 0) { /* XXX: What is the right UDPLite MIB counter? */ goto badunlocked; } } else { if ((ulen < sizeof(struct udphdr)) || (plen != ulen)) { UDPSTAT_INC(udps_badlen); goto badunlocked; } if (uh->uh_sum == 0) { UDPSTAT_INC(udps_nosum); goto badunlocked; } } if ((m->m_pkthdr.csum_flags & CSUM_DATA_VALID_IPV6) && !cscov_partial) { if (m->m_pkthdr.csum_flags & CSUM_PSEUDO_HDR) uh_sum = m->m_pkthdr.csum_data; else uh_sum = in6_cksum_pseudo(ip6, ulen, nxt, m->m_pkthdr.csum_data); uh_sum ^= 0xffff; } else uh_sum = in6_cksum_partial(m, nxt, off, plen, ulen); if (uh_sum != 0) { UDPSTAT_INC(udps_badsum); goto badunlocked; } /* * Construct sockaddr format source address. */ init_sin6(&fromsa, m); fromsa.sin6_port = uh->uh_sport; pcbinfo = get_inpcbinfo(nxt); if (IN6_IS_ADDR_MULTICAST(&ip6->ip6_dst)) { struct inpcb *last; struct inpcbhead *pcblist; struct ip6_moptions *imo; INP_INFO_RLOCK(pcbinfo); /* * In the event that laddr should be set to the link-local * address (this happens in RIPng), the multicast address * specified in the received packet will not match laddr. To * handle this situation, matching is relaxed if the * receiving interface is the same as one specified in the * socket and if the destination multicast address matches * one of the multicast groups specified in the socket. */ /* * KAME note: traditionally we dropped udpiphdr from mbuf * here. We need udphdr for IPsec processing so we do that * later. */ pcblist = get_pcblist(nxt); last = NULL; LIST_FOREACH(inp, pcblist, inp_list) { if ((inp->inp_vflag & INP_IPV6) == 0) continue; if (inp->inp_lport != uh->uh_dport) continue; if (inp->inp_fport != 0 && inp->inp_fport != uh->uh_sport) continue; if (!IN6_IS_ADDR_UNSPECIFIED(&inp->in6p_laddr)) { if (!IN6_ARE_ADDR_EQUAL(&inp->in6p_laddr, &ip6->ip6_dst)) continue; } if (!IN6_IS_ADDR_UNSPECIFIED(&inp->in6p_faddr)) { if (!IN6_ARE_ADDR_EQUAL(&inp->in6p_faddr, &ip6->ip6_src) || inp->inp_fport != uh->uh_sport) continue; } /* * XXXRW: Because we weren't holding either the inpcb * or the hash lock when we checked for a match * before, we should probably recheck now that the * inpcb lock is (supposed to be) held. */ /* * Handle socket delivery policy for any-source * and source-specific multicast. [RFC3678] */ imo = inp->in6p_moptions; if (imo && IN6_IS_ADDR_MULTICAST(&ip6->ip6_dst)) { struct sockaddr_in6 mcaddr; int blocked; INP_RLOCK(inp); bzero(&mcaddr, sizeof(struct sockaddr_in6)); mcaddr.sin6_len = sizeof(struct sockaddr_in6); mcaddr.sin6_family = AF_INET6; mcaddr.sin6_addr = ip6->ip6_dst; blocked = im6o_mc_filter(imo, ifp, (struct sockaddr *)&mcaddr, (struct sockaddr *)&fromsa); if (blocked != MCAST_PASS) { if (blocked == MCAST_NOTGMEMBER) IP6STAT_INC(ip6s_notmember); if (blocked == MCAST_NOTSMEMBER || blocked == MCAST_MUTED) UDPSTAT_INC(udps_filtermcast); INP_RUNLOCK(inp); /* XXX */ continue; } INP_RUNLOCK(inp); } if (last != NULL) { struct mbuf *n; if ((n = m_copy(m, 0, M_COPYALL)) != NULL) { INP_RLOCK(last); UDP_PROBE(receive, NULL, last, ip6, last, uh); udp6_append(last, n, off, &fromsa); INP_RUNLOCK(last); } } last = inp; /* * Don't look for additional matches if this one does * not have either the SO_REUSEPORT or SO_REUSEADDR * socket options set. This heuristic avoids * searching through all pcbs in the common case of a * non-shared port. It assumes that an application * will never clear these options after setting them. */ if ((last->inp_socket->so_options & (SO_REUSEPORT|SO_REUSEADDR)) == 0) break; } if (last == NULL) { /* * No matching pcb found; discard datagram. (No need * to send an ICMP Port Unreachable for a broadcast * or multicast datgram.) */ UDPSTAT_INC(udps_noport); UDPSTAT_INC(udps_noportmcast); goto badheadlocked; } INP_RLOCK(last); INP_INFO_RUNLOCK(pcbinfo); UDP_PROBE(receive, NULL, last, ip6, last, uh); udp6_append(last, m, off, &fromsa); INP_RUNLOCK(last); return (IPPROTO_DONE); } /* * Locate pcb for datagram. */ /* * Grab info from PACKET_TAG_IPFORWARD tag prepended to the chain. */ if ((m->m_flags & M_IP6_NEXTHOP) && (fwd_tag = m_tag_find(m, PACKET_TAG_IPFORWARD, NULL)) != NULL) { struct sockaddr_in6 *next_hop6; next_hop6 = (struct sockaddr_in6 *)(fwd_tag + 1); /* * Transparently forwarded. Pretend to be the destination. * Already got one like this? */ inp = in6_pcblookup_mbuf(pcbinfo, &ip6->ip6_src, uh->uh_sport, &ip6->ip6_dst, uh->uh_dport, INPLOOKUP_RLOCKPCB, m->m_pkthdr.rcvif, m); if (!inp) { /* * It's new. Try to find the ambushing socket. * Because we've rewritten the destination address, * any hardware-generated hash is ignored. */ inp = in6_pcblookup(pcbinfo, &ip6->ip6_src, uh->uh_sport, &next_hop6->sin6_addr, next_hop6->sin6_port ? htons(next_hop6->sin6_port) : uh->uh_dport, INPLOOKUP_WILDCARD | INPLOOKUP_RLOCKPCB, m->m_pkthdr.rcvif); } /* Remove the tag from the packet. We don't need it anymore. */ m_tag_delete(m, fwd_tag); m->m_flags &= ~M_IP6_NEXTHOP; } else inp = in6_pcblookup_mbuf(pcbinfo, &ip6->ip6_src, uh->uh_sport, &ip6->ip6_dst, uh->uh_dport, INPLOOKUP_WILDCARD | INPLOOKUP_RLOCKPCB, m->m_pkthdr.rcvif, m); if (inp == NULL) { if (udp_log_in_vain) { char ip6bufs[INET6_ADDRSTRLEN]; char ip6bufd[INET6_ADDRSTRLEN]; log(LOG_INFO, "Connection attempt to UDP [%s]:%d from [%s]:%d\n", ip6_sprintf(ip6bufd, &ip6->ip6_dst), ntohs(uh->uh_dport), ip6_sprintf(ip6bufs, &ip6->ip6_src), ntohs(uh->uh_sport)); } UDPSTAT_INC(udps_noport); if (m->m_flags & M_MCAST) { printf("UDP6: M_MCAST is set in a unicast packet.\n"); UDPSTAT_INC(udps_noportmcast); goto badunlocked; } if (V_udp_blackhole) goto badunlocked; if (badport_bandlim(BANDLIM_ICMP6_UNREACH) < 0) goto badunlocked; icmp6_error(m, ICMP6_DST_UNREACH, ICMP6_DST_UNREACH_NOPORT, 0); return (IPPROTO_DONE); } INP_RLOCK_ASSERT(inp); up = intoudpcb(inp); if (cscov_partial) { if (up->u_rxcslen == 0 || up->u_rxcslen > ulen) { INP_RUNLOCK(inp); m_freem(m); return (IPPROTO_DONE); } } UDP_PROBE(receive, NULL, inp, ip6, inp, uh); udp6_append(inp, m, off, &fromsa); INP_RUNLOCK(inp); return (IPPROTO_DONE); badheadlocked: INP_INFO_RUNLOCK(pcbinfo); badunlocked: if (m) m_freem(m); return (IPPROTO_DONE); } static void udp6_common_ctlinput(int cmd, struct sockaddr *sa, void *d, struct inpcbinfo *pcbinfo) { struct udphdr uh; struct ip6_hdr *ip6; struct mbuf *m; int off = 0; struct ip6ctlparam *ip6cp = NULL; const struct sockaddr_in6 *sa6_src = NULL; void *cmdarg; struct inpcb *(*notify)(struct inpcb *, int) = udp_notify; struct udp_portonly { u_int16_t uh_sport; u_int16_t uh_dport; } *uhp; if (sa->sa_family != AF_INET6 || sa->sa_len != sizeof(struct sockaddr_in6)) return; if ((unsigned)cmd >= PRC_NCMDS) return; if (PRC_IS_REDIRECT(cmd)) notify = in6_rtchange, d = NULL; else if (cmd == PRC_HOSTDEAD) d = NULL; else if (inet6ctlerrmap[cmd] == 0) return; /* if the parameter is from icmp6, decode it. */ if (d != NULL) { ip6cp = (struct ip6ctlparam *)d; m = ip6cp->ip6c_m; ip6 = ip6cp->ip6c_ip6; off = ip6cp->ip6c_off; cmdarg = ip6cp->ip6c_cmdarg; sa6_src = ip6cp->ip6c_src; } else { m = NULL; ip6 = NULL; cmdarg = NULL; sa6_src = &sa6_any; } if (ip6) { /* * XXX: We assume that when IPV6 is non NULL, * M and OFF are valid. */ /* Check if we can safely examine src and dst ports. */ if (m->m_pkthdr.len < off + sizeof(*uhp)) return; bzero(&uh, sizeof(uh)); m_copydata(m, off, sizeof(*uhp), (caddr_t)&uh); (void)in6_pcbnotify(pcbinfo, sa, uh.uh_dport, (struct sockaddr *)ip6cp->ip6c_src, uh.uh_sport, cmd, cmdarg, notify); } else (void)in6_pcbnotify(pcbinfo, sa, 0, (const struct sockaddr *)sa6_src, 0, cmd, cmdarg, notify); } void udp6_ctlinput(int cmd, struct sockaddr *sa, void *d) { return (udp6_common_ctlinput(cmd, sa, d, &V_udbinfo)); } void udplite6_ctlinput(int cmd, struct sockaddr *sa, void *d) { return (udp6_common_ctlinput(cmd, sa, d, &V_ulitecbinfo)); } static int udp6_getcred(SYSCTL_HANDLER_ARGS) { struct xucred xuc; struct sockaddr_in6 addrs[2]; struct inpcb *inp; int error; error = priv_check(req->td, PRIV_NETINET_GETCRED); if (error) return (error); if (req->newlen != sizeof(addrs)) return (EINVAL); if (req->oldlen != sizeof(struct xucred)) return (EINVAL); error = SYSCTL_IN(req, addrs, sizeof(addrs)); if (error) return (error); if ((error = sa6_embedscope(&addrs[0], V_ip6_use_defzone)) != 0 || (error = sa6_embedscope(&addrs[1], V_ip6_use_defzone)) != 0) { return (error); } inp = in6_pcblookup(&V_udbinfo, &addrs[1].sin6_addr, addrs[1].sin6_port, &addrs[0].sin6_addr, addrs[0].sin6_port, INPLOOKUP_WILDCARD | INPLOOKUP_RLOCKPCB, NULL); if (inp != NULL) { INP_RLOCK_ASSERT(inp); if (inp->inp_socket == NULL) error = ENOENT; if (error == 0) error = cr_canseesocket(req->td->td_ucred, inp->inp_socket); if (error == 0) cru2x(inp->inp_cred, &xuc); INP_RUNLOCK(inp); } else error = ENOENT; if (error == 0) error = SYSCTL_OUT(req, &xuc, sizeof(struct xucred)); return (error); } SYSCTL_PROC(_net_inet6_udp6, OID_AUTO, getcred, CTLTYPE_OPAQUE|CTLFLAG_RW, 0, 0, udp6_getcred, "S,xucred", "Get the xucred of a UDP6 connection"); static int udp6_output(struct inpcb *inp, struct mbuf *m, struct sockaddr *addr6, struct mbuf *control, struct thread *td) { u_int32_t ulen = m->m_pkthdr.len; u_int32_t plen = sizeof(struct udphdr) + ulen; struct ip6_hdr *ip6; struct udphdr *udp6; struct in6_addr *laddr, *faddr, in6a; struct sockaddr_in6 *sin6 = NULL; struct ifnet *oifp = NULL; int cscov_partial = 0; int scope_ambiguous = 0; u_short fport; int error = 0; uint8_t nxt; uint16_t cscov = 0; struct ip6_pktopts *optp, opt; int af = AF_INET6, hlen = sizeof(struct ip6_hdr); int flags; struct sockaddr_in6 tmp; INP_WLOCK_ASSERT(inp); INP_HASH_WLOCK_ASSERT(inp->inp_pcbinfo); if (addr6) { /* addr6 has been validated in udp6_send(). */ sin6 = (struct sockaddr_in6 *)addr6; /* protect *sin6 from overwrites */ tmp = *sin6; sin6 = &tmp; /* * Application should provide a proper zone ID or the use of * default zone IDs should be enabled. Unfortunately, some * applications do not behave as it should, so we need a * workaround. Even if an appropriate ID is not determined, * we'll see if we can determine the outgoing interface. If we * can, determine the zone ID based on the interface below. */ if (sin6->sin6_scope_id == 0 && !V_ip6_use_defzone) scope_ambiguous = 1; if ((error = sa6_embedscope(sin6, V_ip6_use_defzone)) != 0) return (error); } if (control) { if ((error = ip6_setpktopts(control, &opt, inp->in6p_outputopts, td->td_ucred, IPPROTO_UDP)) != 0) goto release; optp = &opt; } else optp = inp->in6p_outputopts; if (sin6) { faddr = &sin6->sin6_addr; /* * Since we saw no essential reason for calling in_pcbconnect, * we get rid of such kind of logic, and call in6_selectsrc * and in6_pcbsetport in order to fill in the local address * and the local port. */ if (sin6->sin6_port == 0) { error = EADDRNOTAVAIL; goto release; } if (!IN6_IS_ADDR_UNSPECIFIED(&inp->in6p_faddr)) { /* how about ::ffff:0.0.0.0 case? */ error = EISCONN; goto release; } fport = sin6->sin6_port; /* allow 0 port */ if (IN6_IS_ADDR_V4MAPPED(faddr)) { if ((inp->inp_flags & IN6P_IPV6_V6ONLY)) { /* * I believe we should explicitly discard the * packet when mapped addresses are disabled, * rather than send the packet as an IPv6 one. * If we chose the latter approach, the packet * might be sent out on the wire based on the * default route, the situation which we'd * probably want to avoid. * (20010421 jinmei@kame.net) */ error = EINVAL; goto release; } if (!IN6_IS_ADDR_UNSPECIFIED(&inp->in6p_laddr) && !IN6_IS_ADDR_V4MAPPED(&inp->in6p_laddr)) { /* * when remote addr is an IPv4-mapped address, * local addr should not be an IPv6 address, * since you cannot determine how to map IPv6 * source address to IPv4. */ error = EINVAL; goto release; } af = AF_INET; } if (!IN6_IS_ADDR_V4MAPPED(faddr)) { error = in6_selectsrc(sin6, optp, inp, NULL, td->td_ucred, &oifp, &in6a); if (error) goto release; if (oifp && scope_ambiguous && (error = in6_setscope(&sin6->sin6_addr, oifp, NULL))) { goto release; } laddr = &in6a; } else laddr = &inp->in6p_laddr; /* XXX */ if (laddr == NULL) { if (error == 0) error = EADDRNOTAVAIL; goto release; } if (inp->inp_lport == 0 && (error = in6_pcbsetport(laddr, inp, td->td_ucred)) != 0) { /* Undo an address bind that may have occurred. */ inp->in6p_laddr = in6addr_any; goto release; } } else { if (IN6_IS_ADDR_UNSPECIFIED(&inp->in6p_faddr)) { error = ENOTCONN; goto release; } if (IN6_IS_ADDR_V4MAPPED(&inp->in6p_faddr)) { if ((inp->inp_flags & IN6P_IPV6_V6ONLY)) { /* * XXX: this case would happen when the * application sets the V6ONLY flag after * connecting the foreign address. * Such applications should be fixed, * so we bark here. */ log(LOG_INFO, "udp6_output: IPV6_V6ONLY " "option was set for a connected socket\n"); error = EINVAL; goto release; } else af = AF_INET; } laddr = &inp->in6p_laddr; faddr = &inp->in6p_faddr; fport = inp->inp_fport; } if (af == AF_INET) hlen = sizeof(struct ip); /* * Calculate data length and get a mbuf * for UDP and IP6 headers. */ M_PREPEND(m, hlen + sizeof(struct udphdr), M_NOWAIT); if (m == 0) { error = ENOBUFS; goto release; } /* * Stuff checksum and output datagram. */ nxt = (inp->inp_socket->so_proto->pr_protocol == IPPROTO_UDP) ? IPPROTO_UDP : IPPROTO_UDPLITE; udp6 = (struct udphdr *)(mtod(m, caddr_t) + hlen); udp6->uh_sport = inp->inp_lport; /* lport is always set in the PCB */ udp6->uh_dport = fport; if (nxt == IPPROTO_UDPLITE) { struct udpcb *up; up = intoudpcb(inp); cscov = up->u_txcslen; if (cscov >= plen) cscov = 0; udp6->uh_ulen = htons(cscov); /* * For UDP-Lite, checksum coverage length of zero means * the entire UDPLite packet is covered by the checksum. */ cscov_partial = (cscov == 0) ? 0 : 1; } else if (plen <= 0xffff) udp6->uh_ulen = htons((u_short)plen); else udp6->uh_ulen = 0; udp6->uh_sum = 0; switch (af) { case AF_INET6: ip6 = mtod(m, struct ip6_hdr *); ip6->ip6_flow = inp->inp_flow & IPV6_FLOWINFO_MASK; ip6->ip6_vfc &= ~IPV6_VERSION_MASK; ip6->ip6_vfc |= IPV6_VERSION; ip6->ip6_plen = htons((u_short)plen); ip6->ip6_nxt = nxt; ip6->ip6_hlim = in6_selecthlim(inp, NULL); ip6->ip6_src = *laddr; ip6->ip6_dst = *faddr; if (cscov_partial) { if ((udp6->uh_sum = in6_cksum_partial(m, nxt, sizeof(struct ip6_hdr), plen, cscov)) == 0) udp6->uh_sum = 0xffff; } else { udp6->uh_sum = in6_cksum_pseudo(ip6, plen, nxt, 0); m->m_pkthdr.csum_flags = CSUM_UDP_IPV6; m->m_pkthdr.csum_data = offsetof(struct udphdr, uh_sum); } /* * XXX for now assume UDP is 2-tuple. * Later on this may become configurable as 4-tuple; * we should support that. * * XXX .. and we should likely cache this in the inpcb. */ #ifdef RSS m->m_pkthdr.flowid = rss_hash_ip6_2tuple(*faddr, *laddr); M_HASHTYPE_SET(m, M_HASHTYPE_RSS_IPV6); #endif flags = 0; #ifdef RSS /* * Don't override with the inp cached flowid. * * Until the whole UDP path is vetted, it may actually * be incorrect. */ flags |= IP_NODEFAULTFLOWID; #endif UDP_PROBE(send, NULL, inp, ip6, inp, udp6); UDPSTAT_INC(udps_opackets); error = ip6_output(m, optp, NULL, flags, inp->in6p_moptions, NULL, inp); break; case AF_INET: error = EAFNOSUPPORT; goto release; } goto releaseopt; release: m_freem(m); releaseopt: if (control) { ip6_clearpktopts(&opt, -1); m_freem(control); } return (error); } static void udp6_abort(struct socket *so) { struct inpcb *inp; struct inpcbinfo *pcbinfo; pcbinfo = get_inpcbinfo(so->so_proto->pr_protocol); inp = sotoinpcb(so); KASSERT(inp != NULL, ("udp6_abort: inp == NULL")); #ifdef INET if (inp->inp_vflag & INP_IPV4) { struct pr_usrreqs *pru; pru = inetsw[ip_protox[IPPROTO_UDP]].pr_usrreqs; (*pru->pru_abort)(so); return; } #endif INP_WLOCK(inp); if (!IN6_IS_ADDR_UNSPECIFIED(&inp->in6p_faddr)) { INP_HASH_WLOCK(pcbinfo); in6_pcbdisconnect(inp); inp->in6p_laddr = in6addr_any; INP_HASH_WUNLOCK(pcbinfo); soisdisconnected(so); } INP_WUNLOCK(inp); } static int udp6_attach(struct socket *so, int proto, struct thread *td) { struct inpcb *inp; struct inpcbinfo *pcbinfo; int error; pcbinfo = get_inpcbinfo(so->so_proto->pr_protocol); inp = sotoinpcb(so); KASSERT(inp == NULL, ("udp6_attach: inp != NULL")); if (so->so_snd.sb_hiwat == 0 || so->so_rcv.sb_hiwat == 0) { error = soreserve(so, udp_sendspace, udp_recvspace); if (error) return (error); } INP_INFO_WLOCK(pcbinfo); error = in_pcballoc(so, pcbinfo); if (error) { INP_INFO_WUNLOCK(pcbinfo); return (error); } inp = (struct inpcb *)so->so_pcb; inp->inp_vflag |= INP_IPV6; if ((inp->inp_flags & IN6P_IPV6_V6ONLY) == 0) inp->inp_vflag |= INP_IPV4; inp->in6p_hops = -1; /* use kernel default */ inp->in6p_cksum = -1; /* just to be sure */ /* * XXX: ugly!! * IPv4 TTL initialization is necessary for an IPv6 socket as well, * because the socket may be bound to an IPv6 wildcard address, * which may match an IPv4-mapped IPv6 address. */ inp->inp_ip_ttl = V_ip_defttl; error = udp_newudpcb(inp); if (error) { in_pcbdetach(inp); in_pcbfree(inp); INP_INFO_WUNLOCK(pcbinfo); return (error); } INP_WUNLOCK(inp); INP_INFO_WUNLOCK(pcbinfo); return (0); } static int udp6_bind(struct socket *so, struct sockaddr *nam, struct thread *td) { struct inpcb *inp; struct inpcbinfo *pcbinfo; int error; pcbinfo = get_inpcbinfo(so->so_proto->pr_protocol); inp = sotoinpcb(so); KASSERT(inp != NULL, ("udp6_bind: inp == NULL")); INP_WLOCK(inp); INP_HASH_WLOCK(pcbinfo); inp->inp_vflag &= ~INP_IPV4; inp->inp_vflag |= INP_IPV6; if ((inp->inp_flags & IN6P_IPV6_V6ONLY) == 0) { struct sockaddr_in6 *sin6_p; sin6_p = (struct sockaddr_in6 *)nam; if (IN6_IS_ADDR_UNSPECIFIED(&sin6_p->sin6_addr)) inp->inp_vflag |= INP_IPV4; #ifdef INET else if (IN6_IS_ADDR_V4MAPPED(&sin6_p->sin6_addr)) { struct sockaddr_in sin; in6_sin6_2_sin(&sin, sin6_p); inp->inp_vflag |= INP_IPV4; inp->inp_vflag &= ~INP_IPV6; error = in_pcbbind(inp, (struct sockaddr *)&sin, td->td_ucred); goto out; } #endif } error = in6_pcbbind(inp, nam, td->td_ucred); #ifdef INET out: #endif INP_HASH_WUNLOCK(pcbinfo); INP_WUNLOCK(inp); return (error); } static void udp6_close(struct socket *so) { struct inpcb *inp; struct inpcbinfo *pcbinfo; pcbinfo = get_inpcbinfo(so->so_proto->pr_protocol); inp = sotoinpcb(so); KASSERT(inp != NULL, ("udp6_close: inp == NULL")); #ifdef INET if (inp->inp_vflag & INP_IPV4) { struct pr_usrreqs *pru; pru = inetsw[ip_protox[IPPROTO_UDP]].pr_usrreqs; (*pru->pru_disconnect)(so); return; } #endif INP_WLOCK(inp); if (!IN6_IS_ADDR_UNSPECIFIED(&inp->in6p_faddr)) { INP_HASH_WLOCK(pcbinfo); in6_pcbdisconnect(inp); inp->in6p_laddr = in6addr_any; INP_HASH_WUNLOCK(pcbinfo); soisdisconnected(so); } INP_WUNLOCK(inp); } static int udp6_connect(struct socket *so, struct sockaddr *nam, struct thread *td) { struct inpcb *inp; struct inpcbinfo *pcbinfo; struct sockaddr_in6 *sin6; int error; pcbinfo = get_inpcbinfo(so->so_proto->pr_protocol); inp = sotoinpcb(so); sin6 = (struct sockaddr_in6 *)nam; KASSERT(inp != NULL, ("udp6_connect: inp == NULL")); /* * XXXRW: Need to clarify locking of v4/v6 flags. */ INP_WLOCK(inp); #ifdef INET if (IN6_IS_ADDR_V4MAPPED(&sin6->sin6_addr)) { struct sockaddr_in sin; if ((inp->inp_flags & IN6P_IPV6_V6ONLY) != 0) { error = EINVAL; goto out; } if (inp->inp_faddr.s_addr != INADDR_ANY) { error = EISCONN; goto out; } in6_sin6_2_sin(&sin, sin6); inp->inp_vflag |= INP_IPV4; inp->inp_vflag &= ~INP_IPV6; error = prison_remote_ip4(td->td_ucred, &sin.sin_addr); if (error != 0) goto out; INP_HASH_WLOCK(pcbinfo); error = in_pcbconnect(inp, (struct sockaddr *)&sin, td->td_ucred); INP_HASH_WUNLOCK(pcbinfo); if (error == 0) soisconnected(so); goto out; } #endif if (!IN6_IS_ADDR_UNSPECIFIED(&inp->in6p_faddr)) { error = EISCONN; goto out; } inp->inp_vflag &= ~INP_IPV4; inp->inp_vflag |= INP_IPV6; error = prison_remote_ip6(td->td_ucred, &sin6->sin6_addr); if (error != 0) goto out; INP_HASH_WLOCK(pcbinfo); error = in6_pcbconnect(inp, nam, td->td_ucred); INP_HASH_WUNLOCK(pcbinfo); if (error == 0) soisconnected(so); out: INP_WUNLOCK(inp); return (error); } static void udp6_detach(struct socket *so) { struct inpcb *inp; struct inpcbinfo *pcbinfo; struct udpcb *up; pcbinfo = get_inpcbinfo(so->so_proto->pr_protocol); inp = sotoinpcb(so); KASSERT(inp != NULL, ("udp6_detach: inp == NULL")); INP_INFO_WLOCK(pcbinfo); INP_WLOCK(inp); up = intoudpcb(inp); KASSERT(up != NULL, ("%s: up == NULL", __func__)); in_pcbdetach(inp); in_pcbfree(inp); INP_INFO_WUNLOCK(pcbinfo); udp_discardcb(up); } static int udp6_disconnect(struct socket *so) { struct inpcb *inp; struct inpcbinfo *pcbinfo; int error; pcbinfo = get_inpcbinfo(so->so_proto->pr_protocol); inp = sotoinpcb(so); KASSERT(inp != NULL, ("udp6_disconnect: inp == NULL")); #ifdef INET if (inp->inp_vflag & INP_IPV4) { struct pr_usrreqs *pru; pru = inetsw[ip_protox[IPPROTO_UDP]].pr_usrreqs; (void)(*pru->pru_disconnect)(so); return (0); } #endif INP_WLOCK(inp); if (IN6_IS_ADDR_UNSPECIFIED(&inp->in6p_faddr)) { error = ENOTCONN; goto out; } INP_HASH_WLOCK(pcbinfo); in6_pcbdisconnect(inp); inp->in6p_laddr = in6addr_any; INP_HASH_WUNLOCK(pcbinfo); SOCK_LOCK(so); so->so_state &= ~SS_ISCONNECTED; /* XXX */ SOCK_UNLOCK(so); out: INP_WUNLOCK(inp); return (0); } static int udp6_send(struct socket *so, int flags, struct mbuf *m, struct sockaddr *addr, struct mbuf *control, struct thread *td) { struct inpcb *inp; struct inpcbinfo *pcbinfo; int error = 0; pcbinfo = get_inpcbinfo(so->so_proto->pr_protocol); inp = sotoinpcb(so); KASSERT(inp != NULL, ("udp6_send: inp == NULL")); INP_WLOCK(inp); if (addr) { if (addr->sa_len != sizeof(struct sockaddr_in6)) { error = EINVAL; goto bad; } if (addr->sa_family != AF_INET6) { error = EAFNOSUPPORT; goto bad; } } #ifdef INET if ((inp->inp_flags & IN6P_IPV6_V6ONLY) == 0) { int hasv4addr; struct sockaddr_in6 *sin6 = 0; if (addr == 0) hasv4addr = (inp->inp_vflag & INP_IPV4); else { sin6 = (struct sockaddr_in6 *)addr; hasv4addr = IN6_IS_ADDR_V4MAPPED(&sin6->sin6_addr) ? 1 : 0; } if (hasv4addr) { struct pr_usrreqs *pru; /* * XXXRW: We release UDP-layer locks before calling * udp_send() in order to avoid recursion. However, * this does mean there is a short window where inp's * fields are unstable. Could this lead to a * potential race in which the factors causing us to * select the UDPv4 output routine are invalidated? */ INP_WUNLOCK(inp); if (sin6) in6_sin6_2_sin_in_sock(addr); pru = inetsw[ip_protox[IPPROTO_UDP]].pr_usrreqs; /* addr will just be freed in sendit(). */ return ((*pru->pru_send)(so, flags, m, addr, control, td)); } } #endif #ifdef MAC mac_inpcb_create_mbuf(inp, m); #endif INP_HASH_WLOCK(pcbinfo); error = udp6_output(inp, m, addr, control, td); INP_HASH_WUNLOCK(pcbinfo); #ifdef INET #endif INP_WUNLOCK(inp); return (error); bad: INP_WUNLOCK(inp); m_freem(m); return (error); } struct pr_usrreqs udp6_usrreqs = { .pru_abort = udp6_abort, .pru_attach = udp6_attach, .pru_bind = udp6_bind, .pru_connect = udp6_connect, .pru_control = in6_control, .pru_detach = udp6_detach, .pru_disconnect = udp6_disconnect, .pru_peeraddr = in6_mapped_peeraddr, .pru_send = udp6_send, .pru_shutdown = udp_shutdown, .pru_sockaddr = in6_mapped_sockaddr, .pru_soreceive = soreceive_dgram, .pru_sosend = sosend_dgram, .pru_sosetlabel = in_pcbsosetlabel, .pru_close = udp6_close }; Index: projects/clang350-import/sys/opencrypto/criov.c =================================================================== --- projects/clang350-import/sys/opencrypto/criov.c (revision 275748) +++ projects/clang350-import/sys/opencrypto/criov.c (revision 275749) @@ -1,198 +1,238 @@ /* $OpenBSD: criov.c,v 1.9 2002/01/29 15:48:29 jason Exp $ */ /*- * Copyright (c) 1999 Theo de Raadt * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. The name of the author may not be used to endorse or promote products * derived from this software without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. * IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. */ #include __FBSDID("$FreeBSD$"); #include #include #include #include #include #include #include #include #include /* * This macro is only for avoiding code duplication, as we need to skip * given number of bytes in the same way in three functions below. */ #define CUIO_SKIP() do { \ KASSERT(off >= 0, ("%s: off %d < 0", __func__, off)); \ KASSERT(len >= 0, ("%s: len %d < 0", __func__, len)); \ while (off > 0) { \ KASSERT(iol >= 0, ("%s: empty in skip", __func__)); \ if (off < iov->iov_len) \ break; \ off -= iov->iov_len; \ iol--; \ iov++; \ } \ } while (0) void cuio_copydata(struct uio* uio, int off, int len, caddr_t cp) { struct iovec *iov = uio->uio_iov; int iol = uio->uio_iovcnt; unsigned count; CUIO_SKIP(); while (len > 0) { KASSERT(iol >= 0, ("%s: empty", __func__)); count = min(iov->iov_len - off, len); bcopy(((caddr_t)iov->iov_base) + off, cp, count); len -= count; cp += count; off = 0; iol--; iov++; } } void cuio_copyback(struct uio* uio, int off, int len, caddr_t cp) { struct iovec *iov = uio->uio_iov; int iol = uio->uio_iovcnt; unsigned count; CUIO_SKIP(); while (len > 0) { KASSERT(iol >= 0, ("%s: empty", __func__)); count = min(iov->iov_len - off, len); bcopy(cp, ((caddr_t)iov->iov_base) + off, count); len -= count; cp += count; off = 0; iol--; iov++; } } /* - * Return a pointer to iov/offset of location in iovec list. + * Return the index and offset of location in iovec list. */ -struct iovec * +int cuio_getptr(struct uio *uio, int loc, int *off) { - struct iovec *iov = uio->uio_iov; - int iol = uio->uio_iovcnt; + int ind, len; - while (loc >= 0) { - /* Normal end of search */ - if (loc < iov->iov_len) { + ind = 0; + while (loc >= 0 && ind < uio->uio_iovcnt) { + len = uio->uio_iov[ind].iov_len; + if (len > loc) { *off = loc; - return (iov); + return (ind); } + loc -= len; + ind++; + } - loc -= iov->iov_len; - if (iol == 0) { - if (loc == 0) { - /* Point at the end of valid data */ - *off = iov->iov_len; - return (iov); - } else - return (NULL); - } else { - iov++, iol--; - } - } + if (ind > 0 && loc == 0) { + ind--; + *off = uio->uio_iov[ind].iov_len; + return (ind); + } - return (NULL); + return (-1); } /* * Apply function f to the data in an iovec list starting "off" bytes from * the beginning, continuing for "len" bytes. */ int cuio_apply(struct uio *uio, int off, int len, int (*f)(void *, void *, u_int), void *arg) { struct iovec *iov = uio->uio_iov; int iol = uio->uio_iovcnt; unsigned count; int rval; CUIO_SKIP(); while (len > 0) { KASSERT(iol >= 0, ("%s: empty", __func__)); count = min(iov->iov_len - off, len); rval = (*f)(arg, ((caddr_t)iov->iov_base) + off, count); if (rval) return (rval); len -= count; off = 0; iol--; iov++; } return (0); } void crypto_copyback(int flags, caddr_t buf, int off, int size, caddr_t in) { if ((flags & CRYPTO_F_IMBUF) != 0) m_copyback((struct mbuf *)buf, off, size, in); else if ((flags & CRYPTO_F_IOV) != 0) cuio_copyback((struct uio *)buf, off, size, in); else bcopy(in, buf + off, size); } void crypto_copydata(int flags, caddr_t buf, int off, int size, caddr_t out) { if ((flags & CRYPTO_F_IMBUF) != 0) m_copydata((struct mbuf *)buf, off, size, out); else if ((flags & CRYPTO_F_IOV) != 0) cuio_copydata((struct uio *)buf, off, size, out); else bcopy(buf + off, out, size); } int crypto_apply(int flags, caddr_t buf, int off, int len, int (*f)(void *, void *, u_int), void *arg) { int error; if ((flags & CRYPTO_F_IMBUF) != 0) error = m_apply((struct mbuf *)buf, off, len, f, arg); else if ((flags & CRYPTO_F_IOV) != 0) error = cuio_apply((struct uio *)buf, off, len, f, arg); else error = (*f)(arg, buf + off, len); return (error); +} + +void +crypto_mbuftoiov(struct mbuf *mbuf, struct iovec **iovptr, int *cnt, + int *allocated) +{ + struct iovec *iov; + struct mbuf *m, *mtmp; + int i, j; + + *allocated = 0; + iov = *iovptr; + if (iov == NULL) + *cnt = 0; + + m = mbuf; + i = 0; + while (m != NULL) { + if (i == *cnt) { + /* we need to allocate a larger array */ + j = 1; + mtmp = m; + while ((mtmp = mtmp->m_next) != NULL) + j++; + iov = malloc(sizeof *iov * (i + j), M_CRYPTO_DATA, + M_WAITOK); + *allocated = 1; + *cnt = i + j; + memcpy(iov, *iovptr, sizeof *iov * i); + } + + iov[i].iov_base = m->m_data; + iov[i].iov_len = m->m_len; + + i++; + m = m->m_next; + } + + if (*allocated) + KASSERT(*cnt == i, ("did not allocate correct amount: %d != %d", + *cnt, i)); + + *iovptr = iov; + *cnt = i; } Index: projects/clang350-import/sys/opencrypto/crypto.c =================================================================== --- projects/clang350-import/sys/opencrypto/crypto.c (revision 275748) +++ projects/clang350-import/sys/opencrypto/crypto.c (revision 275749) @@ -1,1575 +1,1577 @@ /*- * Copyright (c) 2002-2006 Sam Leffler. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. * IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. */ #include __FBSDID("$FreeBSD$"); /* * Cryptographic Subsystem. * * This code is derived from the Openbsd Cryptographic Framework (OCF) * that has the copyright shown below. Very little of the original * code remains. */ /*- * The author of this code is Angelos D. Keromytis (angelos@cis.upenn.edu) * * This code was written by Angelos D. Keromytis in Athens, Greece, in * February 2000. Network Security Technologies Inc. (NSTI) kindly * supported the development of this code. * * Copyright (c) 2000, 2001 Angelos D. Keromytis * * Permission to use, copy, and modify this software with or without fee * is hereby granted, provided that this entire notice is included in * all source code copies of any software which is or includes a copy or * modification of this software. * * THIS SOFTWARE IS BEING PROVIDED "AS IS", WITHOUT ANY EXPRESS OR * IMPLIED WARRANTY. IN PARTICULAR, NONE OF THE AUTHORS MAKES ANY * REPRESENTATION OR WARRANTY OF ANY KIND CONCERNING THE * MERCHANTABILITY OF THIS SOFTWARE OR ITS FITNESS FOR ANY PARTICULAR * PURPOSE. */ #define CRYPTO_TIMING /* enable timing support */ #include "opt_ddb.h" #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include /* XXX for M_XDATA */ #include #include #include "cryptodev_if.h" #if defined(__i386__) || defined(__amd64__) #include #endif SDT_PROVIDER_DEFINE(opencrypto); /* * Crypto drivers register themselves by allocating a slot in the * crypto_drivers table with crypto_get_driverid() and then registering * each algorithm they support with crypto_register() and crypto_kregister(). */ static struct mtx crypto_drivers_mtx; /* lock on driver table */ #define CRYPTO_DRIVER_LOCK() mtx_lock(&crypto_drivers_mtx) #define CRYPTO_DRIVER_UNLOCK() mtx_unlock(&crypto_drivers_mtx) #define CRYPTO_DRIVER_ASSERT() mtx_assert(&crypto_drivers_mtx, MA_OWNED) /* * Crypto device/driver capabilities structure. * * Synchronization: * (d) - protected by CRYPTO_DRIVER_LOCK() * (q) - protected by CRYPTO_Q_LOCK() * Not tagged fields are read-only. */ struct cryptocap { device_t cc_dev; /* (d) device/driver */ u_int32_t cc_sessions; /* (d) # of sessions */ u_int32_t cc_koperations; /* (d) # os asym operations */ /* * Largest possible operator length (in bits) for each type of * encryption algorithm. XXX not used */ u_int16_t cc_max_op_len[CRYPTO_ALGORITHM_MAX + 1]; u_int8_t cc_alg[CRYPTO_ALGORITHM_MAX + 1]; u_int8_t cc_kalg[CRK_ALGORITHM_MAX + 1]; int cc_flags; /* (d) flags */ #define CRYPTOCAP_F_CLEANUP 0x80000000 /* needs resource cleanup */ int cc_qblocked; /* (q) symmetric q blocked */ int cc_kqblocked; /* (q) asymmetric q blocked */ }; static struct cryptocap *crypto_drivers = NULL; static int crypto_drivers_num = 0; /* * There are two queues for crypto requests; one for symmetric (e.g. * cipher) operations and one for asymmetric (e.g. MOD)operations. * A single mutex is used to lock access to both queues. We could * have one per-queue but having one simplifies handling of block/unblock * operations. */ static int crp_sleep = 0; static TAILQ_HEAD(,cryptop) crp_q; /* request queues */ static TAILQ_HEAD(,cryptkop) crp_kq; static struct mtx crypto_q_mtx; #define CRYPTO_Q_LOCK() mtx_lock(&crypto_q_mtx) #define CRYPTO_Q_UNLOCK() mtx_unlock(&crypto_q_mtx) /* * There are two queues for processing completed crypto requests; one * for the symmetric and one for the asymmetric ops. We only need one * but have two to avoid type futzing (cryptop vs. cryptkop). A single * mutex is used to lock access to both queues. Note that this lock * must be separate from the lock on request queues to insure driver * callbacks don't generate lock order reversals. */ static TAILQ_HEAD(,cryptop) crp_ret_q; /* callback queues */ static TAILQ_HEAD(,cryptkop) crp_ret_kq; static struct mtx crypto_ret_q_mtx; #define CRYPTO_RETQ_LOCK() mtx_lock(&crypto_ret_q_mtx) #define CRYPTO_RETQ_UNLOCK() mtx_unlock(&crypto_ret_q_mtx) #define CRYPTO_RETQ_EMPTY() (TAILQ_EMPTY(&crp_ret_q) && TAILQ_EMPTY(&crp_ret_kq)) static uma_zone_t cryptop_zone; static uma_zone_t cryptodesc_zone; int crypto_userasymcrypto = 1; /* userland may do asym crypto reqs */ SYSCTL_INT(_kern, OID_AUTO, userasymcrypto, CTLFLAG_RW, &crypto_userasymcrypto, 0, "Enable/disable user-mode access to asymmetric crypto support"); int crypto_devallowsoft = 0; /* only use hardware crypto */ SYSCTL_INT(_kern, OID_AUTO, cryptodevallowsoft, CTLFLAG_RW, &crypto_devallowsoft, 0, "Enable/disable use of software crypto by /dev/crypto"); MALLOC_DEFINE(M_CRYPTO_DATA, "crypto", "crypto session records"); static void crypto_proc(void); static struct proc *cryptoproc; static void crypto_ret_proc(void); static struct proc *cryptoretproc; static void crypto_destroy(void); static int crypto_invoke(struct cryptocap *cap, struct cryptop *crp, int hint); static int crypto_kinvoke(struct cryptkop *krp, int flags); static struct cryptostats cryptostats; SYSCTL_STRUCT(_kern, OID_AUTO, crypto_stats, CTLFLAG_RW, &cryptostats, cryptostats, "Crypto system statistics"); #ifdef CRYPTO_TIMING static int crypto_timing = 0; SYSCTL_INT(_debug, OID_AUTO, crypto_timing, CTLFLAG_RW, &crypto_timing, 0, "Enable/disable crypto timing support"); #endif static int crypto_init(void) { int error; mtx_init(&crypto_drivers_mtx, "crypto", "crypto driver table", MTX_DEF|MTX_QUIET); TAILQ_INIT(&crp_q); TAILQ_INIT(&crp_kq); mtx_init(&crypto_q_mtx, "crypto", "crypto op queues", MTX_DEF); TAILQ_INIT(&crp_ret_q); TAILQ_INIT(&crp_ret_kq); mtx_init(&crypto_ret_q_mtx, "crypto", "crypto return queues", MTX_DEF); cryptop_zone = uma_zcreate("cryptop", sizeof (struct cryptop), 0, 0, 0, 0, UMA_ALIGN_PTR, UMA_ZONE_ZINIT); cryptodesc_zone = uma_zcreate("cryptodesc", sizeof (struct cryptodesc), 0, 0, 0, 0, UMA_ALIGN_PTR, UMA_ZONE_ZINIT); if (cryptodesc_zone == NULL || cryptop_zone == NULL) { printf("crypto_init: cannot setup crypto zones\n"); error = ENOMEM; goto bad; } crypto_drivers_num = CRYPTO_DRIVERS_INITIAL; crypto_drivers = malloc(crypto_drivers_num * sizeof(struct cryptocap), M_CRYPTO_DATA, M_NOWAIT | M_ZERO); if (crypto_drivers == NULL) { printf("crypto_init: cannot setup crypto drivers\n"); error = ENOMEM; goto bad; } error = kproc_create((void (*)(void *)) crypto_proc, NULL, &cryptoproc, 0, 0, "crypto"); if (error) { printf("crypto_init: cannot start crypto thread; error %d", error); goto bad; } error = kproc_create((void (*)(void *)) crypto_ret_proc, NULL, &cryptoretproc, 0, 0, "crypto returns"); if (error) { printf("crypto_init: cannot start cryptoret thread; error %d", error); goto bad; } return 0; bad: crypto_destroy(); return error; } /* * Signal a crypto thread to terminate. We use the driver * table lock to synchronize the sleep/wakeups so that we * are sure the threads have terminated before we release * the data structures they use. See crypto_finis below * for the other half of this song-and-dance. */ static void crypto_terminate(struct proc **pp, void *q) { struct proc *p; mtx_assert(&crypto_drivers_mtx, MA_OWNED); p = *pp; *pp = NULL; if (p) { wakeup_one(q); PROC_LOCK(p); /* NB: insure we don't miss wakeup */ CRYPTO_DRIVER_UNLOCK(); /* let crypto_finis progress */ msleep(p, &p->p_mtx, PWAIT, "crypto_destroy", 0); PROC_UNLOCK(p); CRYPTO_DRIVER_LOCK(); } } static void crypto_destroy(void) { /* * Terminate any crypto threads. */ CRYPTO_DRIVER_LOCK(); crypto_terminate(&cryptoproc, &crp_q); crypto_terminate(&cryptoretproc, &crp_ret_q); CRYPTO_DRIVER_UNLOCK(); /* XXX flush queues??? */ /* * Reclaim dynamically allocated resources. */ if (crypto_drivers != NULL) free(crypto_drivers, M_CRYPTO_DATA); if (cryptodesc_zone != NULL) uma_zdestroy(cryptodesc_zone); if (cryptop_zone != NULL) uma_zdestroy(cryptop_zone); mtx_destroy(&crypto_q_mtx); mtx_destroy(&crypto_ret_q_mtx); mtx_destroy(&crypto_drivers_mtx); } static struct cryptocap * crypto_checkdriver(u_int32_t hid) { if (crypto_drivers == NULL) return NULL; return (hid >= crypto_drivers_num ? NULL : &crypto_drivers[hid]); } /* * Compare a driver's list of supported algorithms against another * list; return non-zero if all algorithms are supported. */ static int driver_suitable(const struct cryptocap *cap, const struct cryptoini *cri) { const struct cryptoini *cr; /* See if all the algorithms are supported. */ for (cr = cri; cr; cr = cr->cri_next) if (cap->cc_alg[cr->cri_alg] == 0) return 0; return 1; } /* * Select a driver for a new session that supports the specified * algorithms and, optionally, is constrained according to the flags. * The algorithm we use here is pretty stupid; just use the * first driver that supports all the algorithms we need. If there * are multiple drivers we choose the driver with the fewest active * sessions. We prefer hardware-backed drivers to software ones. * * XXX We need more smarts here (in real life too, but that's * XXX another story altogether). */ static struct cryptocap * crypto_select_driver(const struct cryptoini *cri, int flags) { struct cryptocap *cap, *best; int match, hid; CRYPTO_DRIVER_ASSERT(); /* * Look first for hardware crypto devices if permitted. */ if (flags & CRYPTOCAP_F_HARDWARE) match = CRYPTOCAP_F_HARDWARE; else match = CRYPTOCAP_F_SOFTWARE; best = NULL; again: for (hid = 0; hid < crypto_drivers_num; hid++) { cap = &crypto_drivers[hid]; /* * If it's not initialized, is in the process of * going away, or is not appropriate (hardware * or software based on match), then skip. */ if (cap->cc_dev == NULL || (cap->cc_flags & CRYPTOCAP_F_CLEANUP) || (cap->cc_flags & match) == 0) continue; /* verify all the algorithms are supported. */ if (driver_suitable(cap, cri)) { if (best == NULL || cap->cc_sessions < best->cc_sessions) best = cap; } } - if (best != NULL) - return best; - if (match == CRYPTOCAP_F_HARDWARE && (flags & CRYPTOCAP_F_SOFTWARE)) { + if (best == NULL && match == CRYPTOCAP_F_HARDWARE && + (flags & CRYPTOCAP_F_SOFTWARE)) { /* sort of an Algol 68-style for loop */ match = CRYPTOCAP_F_SOFTWARE; goto again; } return best; } /* * Create a new session. The crid argument specifies a crypto * driver to use or constraints on a driver to select (hardware * only, software only, either). Whatever driver is selected * must be capable of the requested crypto algorithms. */ int crypto_newsession(u_int64_t *sid, struct cryptoini *cri, int crid) { struct cryptocap *cap; u_int32_t hid, lid; int err; CRYPTO_DRIVER_LOCK(); if ((crid & (CRYPTOCAP_F_HARDWARE | CRYPTOCAP_F_SOFTWARE)) == 0) { /* * Use specified driver; verify it is capable. */ cap = crypto_checkdriver(crid); if (cap != NULL && !driver_suitable(cap, cri)) cap = NULL; } else { /* * No requested driver; select based on crid flags. */ cap = crypto_select_driver(cri, crid); /* * if NULL then can't do everything in one session. * XXX Fix this. We need to inject a "virtual" session * XXX layer right about here. */ } if (cap != NULL) { /* Call the driver initialization routine. */ hid = cap - crypto_drivers; lid = hid; /* Pass the driver ID. */ err = CRYPTODEV_NEWSESSION(cap->cc_dev, &lid, cri); if (err == 0) { (*sid) = (cap->cc_flags & 0xff000000) | (hid & 0x00ffffff); (*sid) <<= 32; (*sid) |= (lid & 0xffffffff); cap->cc_sessions++; - } - } else + } else + CRYPTDEB("dev newsession failed"); + } else { + CRYPTDEB("no driver"); err = EINVAL; + } CRYPTO_DRIVER_UNLOCK(); return err; } static void crypto_remove(struct cryptocap *cap) { mtx_assert(&crypto_drivers_mtx, MA_OWNED); if (cap->cc_sessions == 0 && cap->cc_koperations == 0) bzero(cap, sizeof(*cap)); } /* * Delete an existing session (or a reserved session on an unregistered * driver). */ int crypto_freesession(u_int64_t sid) { struct cryptocap *cap; u_int32_t hid; int err; CRYPTO_DRIVER_LOCK(); if (crypto_drivers == NULL) { err = EINVAL; goto done; } /* Determine two IDs. */ hid = CRYPTO_SESID2HID(sid); if (hid >= crypto_drivers_num) { err = ENOENT; goto done; } cap = &crypto_drivers[hid]; if (cap->cc_sessions) cap->cc_sessions--; /* Call the driver cleanup routine, if available. */ err = CRYPTODEV_FREESESSION(cap->cc_dev, sid); if (cap->cc_flags & CRYPTOCAP_F_CLEANUP) crypto_remove(cap); done: CRYPTO_DRIVER_UNLOCK(); return err; } /* * Return an unused driver id. Used by drivers prior to registering * support for the algorithms they handle. */ int32_t crypto_get_driverid(device_t dev, int flags) { struct cryptocap *newdrv; int i; if ((flags & (CRYPTOCAP_F_HARDWARE | CRYPTOCAP_F_SOFTWARE)) == 0) { printf("%s: no flags specified when registering driver\n", device_get_nameunit(dev)); return -1; } CRYPTO_DRIVER_LOCK(); for (i = 0; i < crypto_drivers_num; i++) { if (crypto_drivers[i].cc_dev == NULL && (crypto_drivers[i].cc_flags & CRYPTOCAP_F_CLEANUP) == 0) { break; } } /* Out of entries, allocate some more. */ if (i == crypto_drivers_num) { /* Be careful about wrap-around. */ if (2 * crypto_drivers_num <= crypto_drivers_num) { CRYPTO_DRIVER_UNLOCK(); printf("crypto: driver count wraparound!\n"); return -1; } newdrv = malloc(2 * crypto_drivers_num * sizeof(struct cryptocap), M_CRYPTO_DATA, M_NOWAIT|M_ZERO); if (newdrv == NULL) { CRYPTO_DRIVER_UNLOCK(); printf("crypto: no space to expand driver table!\n"); return -1; } bcopy(crypto_drivers, newdrv, crypto_drivers_num * sizeof(struct cryptocap)); crypto_drivers_num *= 2; free(crypto_drivers, M_CRYPTO_DATA); crypto_drivers = newdrv; } /* NB: state is zero'd on free */ crypto_drivers[i].cc_sessions = 1; /* Mark */ crypto_drivers[i].cc_dev = dev; crypto_drivers[i].cc_flags = flags; if (bootverbose) printf("crypto: assign %s driver id %u, flags %u\n", device_get_nameunit(dev), i, flags); CRYPTO_DRIVER_UNLOCK(); return i; } /* * Lookup a driver by name. We match against the full device * name and unit, and against just the name. The latter gives * us a simple widlcarding by device name. On success return the * driver/hardware identifier; otherwise return -1. */ int crypto_find_driver(const char *match) { int i, len = strlen(match); CRYPTO_DRIVER_LOCK(); for (i = 0; i < crypto_drivers_num; i++) { device_t dev = crypto_drivers[i].cc_dev; if (dev == NULL || (crypto_drivers[i].cc_flags & CRYPTOCAP_F_CLEANUP)) continue; if (strncmp(match, device_get_nameunit(dev), len) == 0 || strncmp(match, device_get_name(dev), len) == 0) break; } CRYPTO_DRIVER_UNLOCK(); return i < crypto_drivers_num ? i : -1; } /* * Return the device_t for the specified driver or NULL * if the driver identifier is invalid. */ device_t crypto_find_device_byhid(int hid) { struct cryptocap *cap = crypto_checkdriver(hid); return cap != NULL ? cap->cc_dev : NULL; } /* * Return the device/driver capabilities. */ int crypto_getcaps(int hid) { struct cryptocap *cap = crypto_checkdriver(hid); return cap != NULL ? cap->cc_flags : 0; } /* * Register support for a key-related algorithm. This routine * is called once for each algorithm supported a driver. */ int crypto_kregister(u_int32_t driverid, int kalg, u_int32_t flags) { struct cryptocap *cap; int err; CRYPTO_DRIVER_LOCK(); cap = crypto_checkdriver(driverid); if (cap != NULL && (CRK_ALGORITM_MIN <= kalg && kalg <= CRK_ALGORITHM_MAX)) { /* * XXX Do some performance testing to determine placing. * XXX We probably need an auxiliary data structure that * XXX describes relative performances. */ cap->cc_kalg[kalg] = flags | CRYPTO_ALG_FLAG_SUPPORTED; if (bootverbose) printf("crypto: %s registers key alg %u flags %u\n" , device_get_nameunit(cap->cc_dev) , kalg , flags ); err = 0; } else err = EINVAL; CRYPTO_DRIVER_UNLOCK(); return err; } /* * Register support for a non-key-related algorithm. This routine * is called once for each such algorithm supported by a driver. */ int crypto_register(u_int32_t driverid, int alg, u_int16_t maxoplen, u_int32_t flags) { struct cryptocap *cap; int err; CRYPTO_DRIVER_LOCK(); cap = crypto_checkdriver(driverid); /* NB: algorithms are in the range [1..max] */ if (cap != NULL && (CRYPTO_ALGORITHM_MIN <= alg && alg <= CRYPTO_ALGORITHM_MAX)) { /* * XXX Do some performance testing to determine placing. * XXX We probably need an auxiliary data structure that * XXX describes relative performances. */ cap->cc_alg[alg] = flags | CRYPTO_ALG_FLAG_SUPPORTED; cap->cc_max_op_len[alg] = maxoplen; if (bootverbose) printf("crypto: %s registers alg %u flags %u maxoplen %u\n" , device_get_nameunit(cap->cc_dev) , alg , flags , maxoplen ); cap->cc_sessions = 0; /* Unmark */ err = 0; } else err = EINVAL; CRYPTO_DRIVER_UNLOCK(); return err; } static void driver_finis(struct cryptocap *cap) { u_int32_t ses, kops; CRYPTO_DRIVER_ASSERT(); ses = cap->cc_sessions; kops = cap->cc_koperations; bzero(cap, sizeof(*cap)); if (ses != 0 || kops != 0) { /* * If there are pending sessions, * just mark as invalid. */ cap->cc_flags |= CRYPTOCAP_F_CLEANUP; cap->cc_sessions = ses; cap->cc_koperations = kops; } } /* * Unregister a crypto driver. If there are pending sessions using it, * leave enough information around so that subsequent calls using those * sessions will correctly detect the driver has been unregistered and * reroute requests. */ int crypto_unregister(u_int32_t driverid, int alg) { struct cryptocap *cap; int i, err; CRYPTO_DRIVER_LOCK(); cap = crypto_checkdriver(driverid); if (cap != NULL && (CRYPTO_ALGORITHM_MIN <= alg && alg <= CRYPTO_ALGORITHM_MAX) && cap->cc_alg[alg] != 0) { cap->cc_alg[alg] = 0; cap->cc_max_op_len[alg] = 0; /* Was this the last algorithm ? */ for (i = 1; i <= CRYPTO_ALGORITHM_MAX; i++) if (cap->cc_alg[i] != 0) break; if (i == CRYPTO_ALGORITHM_MAX + 1) driver_finis(cap); err = 0; } else err = EINVAL; CRYPTO_DRIVER_UNLOCK(); return err; } /* * Unregister all algorithms associated with a crypto driver. * If there are pending sessions using it, leave enough information * around so that subsequent calls using those sessions will * correctly detect the driver has been unregistered and reroute * requests. */ int crypto_unregister_all(u_int32_t driverid) { struct cryptocap *cap; int err; CRYPTO_DRIVER_LOCK(); cap = crypto_checkdriver(driverid); if (cap != NULL) { driver_finis(cap); err = 0; } else err = EINVAL; CRYPTO_DRIVER_UNLOCK(); return err; } /* * Clear blockage on a driver. The what parameter indicates whether * the driver is now ready for cryptop's and/or cryptokop's. */ int crypto_unblock(u_int32_t driverid, int what) { struct cryptocap *cap; int err; CRYPTO_Q_LOCK(); cap = crypto_checkdriver(driverid); if (cap != NULL) { if (what & CRYPTO_SYMQ) cap->cc_qblocked = 0; if (what & CRYPTO_ASYMQ) cap->cc_kqblocked = 0; if (crp_sleep) wakeup_one(&crp_q); err = 0; } else err = EINVAL; CRYPTO_Q_UNLOCK(); return err; } /* * Add a crypto request to a queue, to be processed by the kernel thread. */ int crypto_dispatch(struct cryptop *crp) { struct cryptocap *cap; u_int32_t hid; int result; cryptostats.cs_ops++; #ifdef CRYPTO_TIMING if (crypto_timing) binuptime(&crp->crp_tstamp); #endif hid = CRYPTO_SESID2HID(crp->crp_sid); if ((crp->crp_flags & CRYPTO_F_BATCH) == 0) { /* * Caller marked the request to be processed * immediately; dispatch it directly to the * driver unless the driver is currently blocked. */ cap = crypto_checkdriver(hid); /* Driver cannot disappeared when there is an active session. */ KASSERT(cap != NULL, ("%s: Driver disappeared.", __func__)); if (!cap->cc_qblocked) { result = crypto_invoke(cap, crp, 0); if (result != ERESTART) return (result); /* * The driver ran out of resources, put the request on * the queue. */ } } CRYPTO_Q_LOCK(); TAILQ_INSERT_TAIL(&crp_q, crp, crp_next); if (crp_sleep) wakeup_one(&crp_q); CRYPTO_Q_UNLOCK(); return 0; } /* * Add an asymetric crypto request to a queue, * to be processed by the kernel thread. */ int crypto_kdispatch(struct cryptkop *krp) { int error; cryptostats.cs_kops++; error = crypto_kinvoke(krp, krp->krp_crid); if (error == ERESTART) { CRYPTO_Q_LOCK(); TAILQ_INSERT_TAIL(&crp_kq, krp, krp_next); if (crp_sleep) wakeup_one(&crp_q); CRYPTO_Q_UNLOCK(); error = 0; } return error; } /* * Verify a driver is suitable for the specified operation. */ static __inline int kdriver_suitable(const struct cryptocap *cap, const struct cryptkop *krp) { return (cap->cc_kalg[krp->krp_op] & CRYPTO_ALG_FLAG_SUPPORTED) != 0; } /* * Select a driver for an asym operation. The driver must * support the necessary algorithm. The caller can constrain * which device is selected with the flags parameter. The * algorithm we use here is pretty stupid; just use the first * driver that supports the algorithms we need. If there are * multiple suitable drivers we choose the driver with the * fewest active operations. We prefer hardware-backed * drivers to software ones when either may be used. */ static struct cryptocap * crypto_select_kdriver(const struct cryptkop *krp, int flags) { struct cryptocap *cap, *best, *blocked; int match, hid; CRYPTO_DRIVER_ASSERT(); /* * Look first for hardware crypto devices if permitted. */ if (flags & CRYPTOCAP_F_HARDWARE) match = CRYPTOCAP_F_HARDWARE; else match = CRYPTOCAP_F_SOFTWARE; best = NULL; blocked = NULL; again: for (hid = 0; hid < crypto_drivers_num; hid++) { cap = &crypto_drivers[hid]; /* * If it's not initialized, is in the process of * going away, or is not appropriate (hardware * or software based on match), then skip. */ if (cap->cc_dev == NULL || (cap->cc_flags & CRYPTOCAP_F_CLEANUP) || (cap->cc_flags & match) == 0) continue; /* verify all the algorithms are supported. */ if (kdriver_suitable(cap, krp)) { if (best == NULL || cap->cc_koperations < best->cc_koperations) best = cap; } } if (best != NULL) return best; if (match == CRYPTOCAP_F_HARDWARE && (flags & CRYPTOCAP_F_SOFTWARE)) { /* sort of an Algol 68-style for loop */ match = CRYPTOCAP_F_SOFTWARE; goto again; } return best; } /* * Dispatch an assymetric crypto request. */ static int crypto_kinvoke(struct cryptkop *krp, int crid) { struct cryptocap *cap = NULL; int error; KASSERT(krp != NULL, ("%s: krp == NULL", __func__)); KASSERT(krp->krp_callback != NULL, ("%s: krp->crp_callback == NULL", __func__)); CRYPTO_DRIVER_LOCK(); if ((crid & (CRYPTOCAP_F_HARDWARE | CRYPTOCAP_F_SOFTWARE)) == 0) { cap = crypto_checkdriver(crid); if (cap != NULL) { /* * Driver present, it must support the necessary * algorithm and, if s/w drivers are excluded, * it must be registered as hardware-backed. */ if (!kdriver_suitable(cap, krp) || (!crypto_devallowsoft && (cap->cc_flags & CRYPTOCAP_F_HARDWARE) == 0)) cap = NULL; } } else { /* * No requested driver; select based on crid flags. */ if (!crypto_devallowsoft) /* NB: disallow s/w drivers */ crid &= ~CRYPTOCAP_F_SOFTWARE; cap = crypto_select_kdriver(krp, crid); } if (cap != NULL && !cap->cc_kqblocked) { krp->krp_hid = cap - crypto_drivers; cap->cc_koperations++; CRYPTO_DRIVER_UNLOCK(); error = CRYPTODEV_KPROCESS(cap->cc_dev, krp, 0); CRYPTO_DRIVER_LOCK(); if (error == ERESTART) { cap->cc_koperations--; CRYPTO_DRIVER_UNLOCK(); return (error); } } else { /* * NB: cap is !NULL if device is blocked; in * that case return ERESTART so the operation * is resubmitted if possible. */ error = (cap == NULL) ? ENODEV : ERESTART; } CRYPTO_DRIVER_UNLOCK(); if (error) { krp->krp_status = error; crypto_kdone(krp); } return 0; } #ifdef CRYPTO_TIMING static void crypto_tstat(struct cryptotstat *ts, struct bintime *bt) { struct bintime now, delta; struct timespec t; uint64_t u; binuptime(&now); u = now.frac; delta.frac = now.frac - bt->frac; delta.sec = now.sec - bt->sec; if (u < delta.frac) delta.sec--; bintime2timespec(&delta, &t); timespecadd(&ts->acc, &t); if (timespeccmp(&t, &ts->min, <)) ts->min = t; if (timespeccmp(&t, &ts->max, >)) ts->max = t; ts->count++; *bt = now; } #endif /* * Dispatch a crypto request to the appropriate crypto devices. */ static int crypto_invoke(struct cryptocap *cap, struct cryptop *crp, int hint) { KASSERT(crp != NULL, ("%s: crp == NULL", __func__)); KASSERT(crp->crp_callback != NULL, ("%s: crp->crp_callback == NULL", __func__)); KASSERT(crp->crp_desc != NULL, ("%s: crp->crp_desc == NULL", __func__)); #ifdef CRYPTO_TIMING if (crypto_timing) crypto_tstat(&cryptostats.cs_invoke, &crp->crp_tstamp); #endif if (cap->cc_flags & CRYPTOCAP_F_CLEANUP) { struct cryptodesc *crd; u_int64_t nid; /* * Driver has unregistered; migrate the session and return * an error to the caller so they'll resubmit the op. * * XXX: What if there are more already queued requests for this * session? */ crypto_freesession(crp->crp_sid); for (crd = crp->crp_desc; crd->crd_next; crd = crd->crd_next) crd->CRD_INI.cri_next = &(crd->crd_next->CRD_INI); /* XXX propagate flags from initial session? */ if (crypto_newsession(&nid, &(crp->crp_desc->CRD_INI), CRYPTOCAP_F_HARDWARE | CRYPTOCAP_F_SOFTWARE) == 0) crp->crp_sid = nid; crp->crp_etype = EAGAIN; crypto_done(crp); return 0; } else { /* * Invoke the driver to process the request. */ return CRYPTODEV_PROCESS(cap->cc_dev, crp, hint); } } /* * Release a set of crypto descriptors. */ void crypto_freereq(struct cryptop *crp) { struct cryptodesc *crd; if (crp == NULL) return; #ifdef DIAGNOSTIC { struct cryptop *crp2; CRYPTO_Q_LOCK(); TAILQ_FOREACH(crp2, &crp_q, crp_next) { KASSERT(crp2 != crp, ("Freeing cryptop from the crypto queue (%p).", crp)); } CRYPTO_Q_UNLOCK(); CRYPTO_RETQ_LOCK(); TAILQ_FOREACH(crp2, &crp_ret_q, crp_next) { KASSERT(crp2 != crp, ("Freeing cryptop from the return queue (%p).", crp)); } CRYPTO_RETQ_UNLOCK(); } #endif while ((crd = crp->crp_desc) != NULL) { crp->crp_desc = crd->crd_next; uma_zfree(cryptodesc_zone, crd); } uma_zfree(cryptop_zone, crp); } /* * Acquire a set of crypto descriptors. */ struct cryptop * crypto_getreq(int num) { struct cryptodesc *crd; struct cryptop *crp; crp = uma_zalloc(cryptop_zone, M_NOWAIT|M_ZERO); if (crp != NULL) { while (num--) { crd = uma_zalloc(cryptodesc_zone, M_NOWAIT|M_ZERO); if (crd == NULL) { crypto_freereq(crp); return NULL; } crd->crd_next = crp->crp_desc; crp->crp_desc = crd; } } return crp; } /* * Invoke the callback on behalf of the driver. */ void crypto_done(struct cryptop *crp) { KASSERT((crp->crp_flags & CRYPTO_F_DONE) == 0, ("crypto_done: op already done, flags 0x%x", crp->crp_flags)); crp->crp_flags |= CRYPTO_F_DONE; if (crp->crp_etype != 0) cryptostats.cs_errs++; #ifdef CRYPTO_TIMING if (crypto_timing) crypto_tstat(&cryptostats.cs_done, &crp->crp_tstamp); #endif /* * CBIMM means unconditionally do the callback immediately; * CBIFSYNC means do the callback immediately only if the * operation was done synchronously. Both are used to avoid * doing extraneous context switches; the latter is mostly * used with the software crypto driver. */ if ((crp->crp_flags & CRYPTO_F_CBIMM) || ((crp->crp_flags & CRYPTO_F_CBIFSYNC) && (CRYPTO_SESID2CAPS(crp->crp_sid) & CRYPTOCAP_F_SYNC))) { /* * Do the callback directly. This is ok when the * callback routine does very little (e.g. the * /dev/crypto callback method just does a wakeup). */ #ifdef CRYPTO_TIMING if (crypto_timing) { /* * NB: We must copy the timestamp before * doing the callback as the cryptop is * likely to be reclaimed. */ struct bintime t = crp->crp_tstamp; crypto_tstat(&cryptostats.cs_cb, &t); crp->crp_callback(crp); crypto_tstat(&cryptostats.cs_finis, &t); } else #endif crp->crp_callback(crp); } else { /* * Normal case; queue the callback for the thread. */ CRYPTO_RETQ_LOCK(); if (CRYPTO_RETQ_EMPTY()) wakeup_one(&crp_ret_q); /* shared wait channel */ TAILQ_INSERT_TAIL(&crp_ret_q, crp, crp_next); CRYPTO_RETQ_UNLOCK(); } } /* * Invoke the callback on behalf of the driver. */ void crypto_kdone(struct cryptkop *krp) { struct cryptocap *cap; if (krp->krp_status != 0) cryptostats.cs_kerrs++; CRYPTO_DRIVER_LOCK(); /* XXX: What if driver is loaded in the meantime? */ if (krp->krp_hid < crypto_drivers_num) { cap = &crypto_drivers[krp->krp_hid]; cap->cc_koperations--; KASSERT(cap->cc_koperations >= 0, ("cc_koperations < 0")); if (cap->cc_flags & CRYPTOCAP_F_CLEANUP) crypto_remove(cap); } CRYPTO_DRIVER_UNLOCK(); CRYPTO_RETQ_LOCK(); if (CRYPTO_RETQ_EMPTY()) wakeup_one(&crp_ret_q); /* shared wait channel */ TAILQ_INSERT_TAIL(&crp_ret_kq, krp, krp_next); CRYPTO_RETQ_UNLOCK(); } int crypto_getfeat(int *featp) { int hid, kalg, feat = 0; CRYPTO_DRIVER_LOCK(); for (hid = 0; hid < crypto_drivers_num; hid++) { const struct cryptocap *cap = &crypto_drivers[hid]; if ((cap->cc_flags & CRYPTOCAP_F_SOFTWARE) && !crypto_devallowsoft) { continue; } for (kalg = 0; kalg < CRK_ALGORITHM_MAX; kalg++) if (cap->cc_kalg[kalg] & CRYPTO_ALG_FLAG_SUPPORTED) feat |= 1 << kalg; } CRYPTO_DRIVER_UNLOCK(); *featp = feat; return (0); } /* * Terminate a thread at module unload. The process that * initiated this is waiting for us to signal that we're gone; * wake it up and exit. We use the driver table lock to insure * we don't do the wakeup before they're waiting. There is no * race here because the waiter sleeps on the proc lock for the * thread so it gets notified at the right time because of an * extra wakeup that's done in exit1(). */ static void crypto_finis(void *chan) { CRYPTO_DRIVER_LOCK(); wakeup_one(chan); CRYPTO_DRIVER_UNLOCK(); kproc_exit(0); } /* * Crypto thread, dispatches crypto requests. */ static void crypto_proc(void) { struct cryptop *crp, *submit; struct cryptkop *krp; struct cryptocap *cap; u_int32_t hid; int result, hint; #if defined(__i386__) || defined(__amd64__) fpu_kern_thread(FPU_KERN_NORMAL); #endif CRYPTO_Q_LOCK(); for (;;) { /* * Find the first element in the queue that can be * processed and look-ahead to see if multiple ops * are ready for the same driver. */ submit = NULL; hint = 0; TAILQ_FOREACH(crp, &crp_q, crp_next) { hid = CRYPTO_SESID2HID(crp->crp_sid); cap = crypto_checkdriver(hid); /* * Driver cannot disappeared when there is an active * session. */ KASSERT(cap != NULL, ("%s:%u Driver disappeared.", __func__, __LINE__)); if (cap == NULL || cap->cc_dev == NULL) { /* Op needs to be migrated, process it. */ if (submit == NULL) submit = crp; break; } if (!cap->cc_qblocked) { if (submit != NULL) { /* * We stop on finding another op, * regardless whether its for the same * driver or not. We could keep * searching the queue but it might be * better to just use a per-driver * queue instead. */ if (CRYPTO_SESID2HID(submit->crp_sid) == hid) hint = CRYPTO_HINT_MORE; break; } else { submit = crp; if ((submit->crp_flags & CRYPTO_F_BATCH) == 0) break; /* keep scanning for more are q'd */ } } } if (submit != NULL) { TAILQ_REMOVE(&crp_q, submit, crp_next); hid = CRYPTO_SESID2HID(submit->crp_sid); cap = crypto_checkdriver(hid); KASSERT(cap != NULL, ("%s:%u Driver disappeared.", __func__, __LINE__)); result = crypto_invoke(cap, submit, hint); if (result == ERESTART) { /* * The driver ran out of resources, mark the * driver ``blocked'' for cryptop's and put * the request back in the queue. It would * best to put the request back where we got * it but that's hard so for now we put it * at the front. This should be ok; putting * it at the end does not work. */ /* XXX validate sid again? */ crypto_drivers[CRYPTO_SESID2HID(submit->crp_sid)].cc_qblocked = 1; TAILQ_INSERT_HEAD(&crp_q, submit, crp_next); cryptostats.cs_blocks++; } } /* As above, but for key ops */ TAILQ_FOREACH(krp, &crp_kq, krp_next) { cap = crypto_checkdriver(krp->krp_hid); if (cap == NULL || cap->cc_dev == NULL) { /* * Operation needs to be migrated, invalidate * the assigned device so it will reselect a * new one below. Propagate the original * crid selection flags if supplied. */ krp->krp_hid = krp->krp_crid & (CRYPTOCAP_F_SOFTWARE|CRYPTOCAP_F_HARDWARE); if (krp->krp_hid == 0) krp->krp_hid = CRYPTOCAP_F_SOFTWARE|CRYPTOCAP_F_HARDWARE; break; } if (!cap->cc_kqblocked) break; } if (krp != NULL) { TAILQ_REMOVE(&crp_kq, krp, krp_next); result = crypto_kinvoke(krp, krp->krp_hid); if (result == ERESTART) { /* * The driver ran out of resources, mark the * driver ``blocked'' for cryptkop's and put * the request back in the queue. It would * best to put the request back where we got * it but that's hard so for now we put it * at the front. This should be ok; putting * it at the end does not work. */ /* XXX validate sid again? */ crypto_drivers[krp->krp_hid].cc_kqblocked = 1; TAILQ_INSERT_HEAD(&crp_kq, krp, krp_next); cryptostats.cs_kblocks++; } } if (submit == NULL && krp == NULL) { /* * Nothing more to be processed. Sleep until we're * woken because there are more ops to process. * This happens either by submission or by a driver * becoming unblocked and notifying us through * crypto_unblock. Note that when we wakeup we * start processing each queue again from the * front. It's not clear that it's important to * preserve this ordering since ops may finish * out of order if dispatched to different devices * and some become blocked while others do not. */ crp_sleep = 1; msleep(&crp_q, &crypto_q_mtx, PWAIT, "crypto_wait", 0); crp_sleep = 0; if (cryptoproc == NULL) break; cryptostats.cs_intrs++; } } CRYPTO_Q_UNLOCK(); crypto_finis(&crp_q); } /* * Crypto returns thread, does callbacks for processed crypto requests. * Callbacks are done here, rather than in the crypto drivers, because * callbacks typically are expensive and would slow interrupt handling. */ static void crypto_ret_proc(void) { struct cryptop *crpt; struct cryptkop *krpt; CRYPTO_RETQ_LOCK(); for (;;) { /* Harvest return q's for completed ops */ crpt = TAILQ_FIRST(&crp_ret_q); if (crpt != NULL) TAILQ_REMOVE(&crp_ret_q, crpt, crp_next); krpt = TAILQ_FIRST(&crp_ret_kq); if (krpt != NULL) TAILQ_REMOVE(&crp_ret_kq, krpt, krp_next); if (crpt != NULL || krpt != NULL) { CRYPTO_RETQ_UNLOCK(); /* * Run callbacks unlocked. */ if (crpt != NULL) { #ifdef CRYPTO_TIMING if (crypto_timing) { /* * NB: We must copy the timestamp before * doing the callback as the cryptop is * likely to be reclaimed. */ struct bintime t = crpt->crp_tstamp; crypto_tstat(&cryptostats.cs_cb, &t); crpt->crp_callback(crpt); crypto_tstat(&cryptostats.cs_finis, &t); } else #endif crpt->crp_callback(crpt); } if (krpt != NULL) krpt->krp_callback(krpt); CRYPTO_RETQ_LOCK(); } else { /* * Nothing more to be processed. Sleep until we're * woken because there are more returns to process. */ msleep(&crp_ret_q, &crypto_ret_q_mtx, PWAIT, "crypto_ret_wait", 0); if (cryptoretproc == NULL) break; cryptostats.cs_rets++; } } CRYPTO_RETQ_UNLOCK(); crypto_finis(&crp_ret_q); } #ifdef DDB static void db_show_drivers(void) { int hid; db_printf("%12s %4s %4s %8s %2s %2s\n" , "Device" , "Ses" , "Kops" , "Flags" , "QB" , "KB" ); for (hid = 0; hid < crypto_drivers_num; hid++) { const struct cryptocap *cap = &crypto_drivers[hid]; if (cap->cc_dev == NULL) continue; db_printf("%-12s %4u %4u %08x %2u %2u\n" , device_get_nameunit(cap->cc_dev) , cap->cc_sessions , cap->cc_koperations , cap->cc_flags , cap->cc_qblocked , cap->cc_kqblocked ); } } DB_SHOW_COMMAND(crypto, db_show_crypto) { struct cryptop *crp; db_show_drivers(); db_printf("\n"); db_printf("%4s %8s %4s %4s %4s %4s %8s %8s\n", "HID", "Caps", "Ilen", "Olen", "Etype", "Flags", "Desc", "Callback"); TAILQ_FOREACH(crp, &crp_q, crp_next) { db_printf("%4u %08x %4u %4u %4u %04x %8p %8p\n" , (int) CRYPTO_SESID2HID(crp->crp_sid) , (int) CRYPTO_SESID2CAPS(crp->crp_sid) , crp->crp_ilen, crp->crp_olen , crp->crp_etype , crp->crp_flags , crp->crp_desc , crp->crp_callback ); } if (!TAILQ_EMPTY(&crp_ret_q)) { db_printf("\n%4s %4s %4s %8s\n", "HID", "Etype", "Flags", "Callback"); TAILQ_FOREACH(crp, &crp_ret_q, crp_next) { db_printf("%4u %4u %04x %8p\n" , (int) CRYPTO_SESID2HID(crp->crp_sid) , crp->crp_etype , crp->crp_flags , crp->crp_callback ); } } } DB_SHOW_COMMAND(kcrypto, db_show_kcrypto) { struct cryptkop *krp; db_show_drivers(); db_printf("\n"); db_printf("%4s %5s %4s %4s %8s %4s %8s\n", "Op", "Status", "#IP", "#OP", "CRID", "HID", "Callback"); TAILQ_FOREACH(krp, &crp_kq, krp_next) { db_printf("%4u %5u %4u %4u %08x %4u %8p\n" , krp->krp_op , krp->krp_status , krp->krp_iparams, krp->krp_oparams , krp->krp_crid, krp->krp_hid , krp->krp_callback ); } if (!TAILQ_EMPTY(&crp_ret_q)) { db_printf("%4s %5s %8s %4s %8s\n", "Op", "Status", "CRID", "HID", "Callback"); TAILQ_FOREACH(krp, &crp_ret_kq, krp_next) { db_printf("%4u %5u %08x %4u %8p\n" , krp->krp_op , krp->krp_status , krp->krp_crid, krp->krp_hid , krp->krp_callback ); } } } #endif int crypto_modevent(module_t mod, int type, void *unused); /* * Initialization code, both for static and dynamic loading. * Note this is not invoked with the usual MODULE_DECLARE * mechanism but instead is listed as a dependency by the * cryptosoft driver. This guarantees proper ordering of * calls on module load/unload. */ int crypto_modevent(module_t mod, int type, void *unused) { int error = EINVAL; switch (type) { case MOD_LOAD: error = crypto_init(); if (error == 0 && bootverbose) printf("crypto: \n"); break; case MOD_UNLOAD: /*XXX disallow if active sessions */ error = 0; crypto_destroy(); return 0; } return error; } MODULE_VERSION(crypto, 1); MODULE_DEPEND(crypto, zlib, 1, 1, 1); Index: projects/clang350-import/sys/opencrypto/cryptodev.c =================================================================== --- projects/clang350-import/sys/opencrypto/cryptodev.c (revision 275748) +++ projects/clang350-import/sys/opencrypto/cryptodev.c (revision 275749) @@ -1,1141 +1,1374 @@ /* $OpenBSD: cryptodev.c,v 1.52 2002/06/19 07:22:46 deraadt Exp $ */ /*- * Copyright (c) 2001 Theo de Raadt * Copyright (c) 2002-2006 Sam Leffler, Errno Consulting + * Copyright (c) 2014 The FreeBSD Foundation + * All rights reserved. * + * Portions of this software were developed by John-Mark Gurney + * under sponsorship of the FreeBSD Foundation and + * Rubicon Communications, LLC (Netgate). + * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. The name of the author may not be used to endorse or promote products * derived from this software without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. * IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. * * Effort sponsored in part by the Defense Advanced Research Projects * Agency (DARPA) and Air Force Research Laboratory, Air Force * Materiel Command, USAF, under agreement number F30602-01-2-0537. */ #include __FBSDID("$FreeBSD$"); #include "opt_compat.h" +#include "opt_kdtrace.h" #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include +#include #include #include +SDT_PROVIDER_DECLARE(opencrypto); + +SDT_PROBE_DEFINE1(opencrypto, dev, ioctl, error, "int"/*line number*/); + #ifdef COMPAT_FREEBSD32 #include #include struct session_op32 { u_int32_t cipher; u_int32_t mac; u_int32_t keylen; u_int32_t key; int mackeylen; u_int32_t mackey; u_int32_t ses; }; struct session2_op32 { u_int32_t cipher; u_int32_t mac; u_int32_t keylen; u_int32_t key; int mackeylen; u_int32_t mackey; u_int32_t ses; int crid; int pad[4]; }; struct crypt_op32 { u_int32_t ses; u_int16_t op; u_int16_t flags; u_int len; u_int32_t src, dst; u_int32_t mac; u_int32_t iv; }; struct crparam32 { u_int32_t crp_p; u_int crp_nbits; }; struct crypt_kop32 { u_int crk_op; u_int crk_status; u_short crk_iparams; u_short crk_oparams; u_int crk_crid; struct crparam32 crk_param[CRK_MAXPARAM]; }; struct cryptotstat32 { struct timespec32 acc; struct timespec32 min; struct timespec32 max; u_int32_t count; }; struct cryptostats32 { u_int32_t cs_ops; u_int32_t cs_errs; u_int32_t cs_kops; u_int32_t cs_kerrs; u_int32_t cs_intrs; u_int32_t cs_rets; u_int32_t cs_blocks; u_int32_t cs_kblocks; struct cryptotstat32 cs_invoke; struct cryptotstat32 cs_done; struct cryptotstat32 cs_cb; struct cryptotstat32 cs_finis; }; #define CIOCGSESSION32 _IOWR('c', 101, struct session_op32) #define CIOCCRYPT32 _IOWR('c', 103, struct crypt_op32) #define CIOCKEY32 _IOWR('c', 104, struct crypt_kop32) #define CIOCGSESSION232 _IOWR('c', 106, struct session2_op32) #define CIOCKEY232 _IOWR('c', 107, struct crypt_kop32) static void session_op_from_32(const struct session_op32 *from, struct session_op *to) { CP(*from, *to, cipher); CP(*from, *to, mac); CP(*from, *to, keylen); PTRIN_CP(*from, *to, key); CP(*from, *to, mackeylen); PTRIN_CP(*from, *to, mackey); CP(*from, *to, ses); } static void session2_op_from_32(const struct session2_op32 *from, struct session2_op *to) { session_op_from_32((const struct session_op32 *)from, (struct session_op *)to); CP(*from, *to, crid); } static void session_op_to_32(const struct session_op *from, struct session_op32 *to) { CP(*from, *to, cipher); CP(*from, *to, mac); CP(*from, *to, keylen); PTROUT_CP(*from, *to, key); CP(*from, *to, mackeylen); PTROUT_CP(*from, *to, mackey); CP(*from, *to, ses); } static void session2_op_to_32(const struct session2_op *from, struct session2_op32 *to) { session_op_to_32((const struct session_op *)from, (struct session_op32 *)to); CP(*from, *to, crid); } static void crypt_op_from_32(const struct crypt_op32 *from, struct crypt_op *to) { CP(*from, *to, ses); CP(*from, *to, op); CP(*from, *to, flags); CP(*from, *to, len); PTRIN_CP(*from, *to, src); PTRIN_CP(*from, *to, dst); PTRIN_CP(*from, *to, mac); PTRIN_CP(*from, *to, iv); } static void crypt_op_to_32(const struct crypt_op *from, struct crypt_op32 *to) { CP(*from, *to, ses); CP(*from, *to, op); CP(*from, *to, flags); CP(*from, *to, len); PTROUT_CP(*from, *to, src); PTROUT_CP(*from, *to, dst); PTROUT_CP(*from, *to, mac); PTROUT_CP(*from, *to, iv); } static void crparam_from_32(const struct crparam32 *from, struct crparam *to) { PTRIN_CP(*from, *to, crp_p); CP(*from, *to, crp_nbits); } static void crparam_to_32(const struct crparam *from, struct crparam32 *to) { PTROUT_CP(*from, *to, crp_p); CP(*from, *to, crp_nbits); } static void crypt_kop_from_32(const struct crypt_kop32 *from, struct crypt_kop *to) { int i; CP(*from, *to, crk_op); CP(*from, *to, crk_status); CP(*from, *to, crk_iparams); CP(*from, *to, crk_oparams); CP(*from, *to, crk_crid); for (i = 0; i < CRK_MAXPARAM; i++) crparam_from_32(&from->crk_param[i], &to->crk_param[i]); } static void crypt_kop_to_32(const struct crypt_kop *from, struct crypt_kop32 *to) { int i; CP(*from, *to, crk_op); CP(*from, *to, crk_status); CP(*from, *to, crk_iparams); CP(*from, *to, crk_oparams); CP(*from, *to, crk_crid); for (i = 0; i < CRK_MAXPARAM; i++) crparam_to_32(&from->crk_param[i], &to->crk_param[i]); } #endif struct csession { TAILQ_ENTRY(csession) next; u_int64_t sid; u_int32_t ses; struct mtx lock; /* for op submission */ u_int32_t cipher; struct enc_xform *txform; u_int32_t mac; struct auth_hash *thash; caddr_t key; int keylen; u_char tmp_iv[EALG_MAX_BLOCK_LEN]; caddr_t mackey; int mackeylen; struct iovec iovec; struct uio uio; int error; }; struct fcrypt { TAILQ_HEAD(csessionlist, csession) csessions; int sesn; }; static int cryptof_ioctl(struct file *, u_long, void *, struct ucred *, struct thread *); static int cryptof_stat(struct file *, struct stat *, struct ucred *, struct thread *); static int cryptof_close(struct file *, struct thread *); static int cryptof_fill_kinfo(struct file *, struct kinfo_file *, struct filedesc *); static struct fileops cryptofops = { .fo_read = invfo_rdwr, .fo_write = invfo_rdwr, .fo_truncate = invfo_truncate, .fo_ioctl = cryptof_ioctl, .fo_poll = invfo_poll, .fo_kqfilter = invfo_kqfilter, .fo_stat = cryptof_stat, .fo_close = cryptof_close, .fo_chmod = invfo_chmod, .fo_chown = invfo_chown, .fo_sendfile = invfo_sendfile, .fo_fill_kinfo = cryptof_fill_kinfo, }; static struct csession *csefind(struct fcrypt *, u_int); static int csedelete(struct fcrypt *, struct csession *); static struct csession *cseadd(struct fcrypt *, struct csession *); static struct csession *csecreate(struct fcrypt *, u_int64_t, caddr_t, u_int64_t, caddr_t, u_int64_t, u_int32_t, u_int32_t, struct enc_xform *, struct auth_hash *); static int csefree(struct csession *); static int cryptodev_op(struct csession *, struct crypt_op *, struct ucred *, struct thread *td); +static int cryptodev_aead(struct csession *, struct crypt_aead *, + struct ucred *, struct thread *); static int cryptodev_key(struct crypt_kop *); static int cryptodev_find(struct crypt_find_op *); /* * Check a crypto identifier to see if it requested * a software device/driver. This can be done either * by device name/class or through search constraints. */ static int -checkforsoftware(int crid) +checkforsoftware(int *cridp) { + int crid; + crid = *cridp; + if (!crypto_devallowsoft) { - if (crid & CRYPTOCAP_F_SOFTWARE) - return EINVAL; /* XXX */ + if (crid & CRYPTOCAP_F_SOFTWARE) { + if (crid & CRYPTOCAP_F_HARDWARE) { + *cridp = CRYPTOCAP_F_HARDWARE; + return 0; + } + return EINVAL; + } if ((crid & CRYPTOCAP_F_HARDWARE) == 0 && (crypto_getcaps(crid) & CRYPTOCAP_F_HARDWARE) == 0) - return EINVAL; /* XXX */ + return EINVAL; } return 0; } /* ARGSUSED */ static int cryptof_ioctl( struct file *fp, u_long cmd, void *data, struct ucred *active_cred, struct thread *td) { #define SES2(p) ((struct session2_op *)p) struct cryptoini cria, crie; struct fcrypt *fcr = fp->f_data; struct csession *cse; struct session_op *sop; struct crypt_op *cop; + struct crypt_aead *caead; struct enc_xform *txform = NULL; struct auth_hash *thash = NULL; struct crypt_kop *kop; u_int64_t sid; u_int32_t ses; int error = 0, crid; #ifdef COMPAT_FREEBSD32 struct session2_op sopc; struct crypt_op copc; struct crypt_kop kopc; #endif switch (cmd) { case CIOCGSESSION: case CIOCGSESSION2: #ifdef COMPAT_FREEBSD32 case CIOCGSESSION32: case CIOCGSESSION232: if (cmd == CIOCGSESSION32) { session_op_from_32(data, (struct session_op *)&sopc); sop = (struct session_op *)&sopc; } else if (cmd == CIOCGSESSION232) { session2_op_from_32(data, &sopc); sop = (struct session_op *)&sopc; } else #endif sop = (struct session_op *)data; switch (sop->cipher) { case 0: break; case CRYPTO_DES_CBC: txform = &enc_xform_des; break; case CRYPTO_3DES_CBC: txform = &enc_xform_3des; break; case CRYPTO_BLF_CBC: txform = &enc_xform_blf; break; case CRYPTO_CAST_CBC: txform = &enc_xform_cast5; break; case CRYPTO_SKIPJACK_CBC: txform = &enc_xform_skipjack; break; case CRYPTO_AES_CBC: txform = &enc_xform_rijndael128; break; case CRYPTO_AES_XTS: txform = &enc_xform_aes_xts; break; case CRYPTO_NULL_CBC: txform = &enc_xform_null; break; case CRYPTO_ARC4: txform = &enc_xform_arc4; break; case CRYPTO_CAMELLIA_CBC: txform = &enc_xform_camellia; break; + case CRYPTO_AES_ICM: + txform = &enc_xform_aes_icm; + break; + case CRYPTO_AES_NIST_GCM_16: + txform = &enc_xform_aes_nist_gcm; + break; + default: + CRYPTDEB("invalid cipher"); return (EINVAL); } switch (sop->mac) { case 0: break; case CRYPTO_MD5_HMAC: thash = &auth_hash_hmac_md5; break; case CRYPTO_SHA1_HMAC: thash = &auth_hash_hmac_sha1; break; case CRYPTO_SHA2_256_HMAC: thash = &auth_hash_hmac_sha2_256; break; case CRYPTO_SHA2_384_HMAC: thash = &auth_hash_hmac_sha2_384; break; case CRYPTO_SHA2_512_HMAC: thash = &auth_hash_hmac_sha2_512; break; case CRYPTO_RIPEMD160_HMAC: thash = &auth_hash_hmac_ripemd_160; break; + case CRYPTO_AES_128_NIST_GMAC: + thash = &auth_hash_nist_gmac_aes_128; + break; + case CRYPTO_AES_192_NIST_GMAC: + thash = &auth_hash_nist_gmac_aes_192; + break; + case CRYPTO_AES_256_NIST_GMAC: + thash = &auth_hash_nist_gmac_aes_256; + break; + #ifdef notdef case CRYPTO_MD5: thash = &auth_hash_md5; break; case CRYPTO_SHA1: thash = &auth_hash_sha1; break; #endif case CRYPTO_NULL_HMAC: thash = &auth_hash_null; break; default: + CRYPTDEB("invalid mac"); return (EINVAL); } bzero(&crie, sizeof(crie)); bzero(&cria, sizeof(cria)); if (txform) { crie.cri_alg = txform->type; crie.cri_klen = sop->keylen * 8; if (sop->keylen > txform->maxkey || sop->keylen < txform->minkey) { + CRYPTDEB("invalid cipher parameters"); error = EINVAL; goto bail; } crie.cri_key = malloc(crie.cri_klen / 8, M_XDATA, M_WAITOK); if ((error = copyin(sop->key, crie.cri_key, - crie.cri_klen / 8))) + crie.cri_klen / 8))) { + CRYPTDEB("invalid key"); goto bail; + } if (thash) crie.cri_next = &cria; } if (thash) { cria.cri_alg = thash->type; cria.cri_klen = sop->mackeylen * 8; if (sop->mackeylen != thash->keysize) { + CRYPTDEB("invalid mac key length"); error = EINVAL; goto bail; } if (cria.cri_klen) { cria.cri_key = malloc(cria.cri_klen / 8, M_XDATA, M_WAITOK); if ((error = copyin(sop->mackey, cria.cri_key, - cria.cri_klen / 8))) + cria.cri_klen / 8))) { + CRYPTDEB("invalid mac key"); goto bail; + } } } /* NB: CIOCGSESSION2 has the crid */ if (cmd == CIOCGSESSION2 #ifdef COMPAT_FREEBSD32 || cmd == CIOCGSESSION232 #endif ) { crid = SES2(sop)->crid; - error = checkforsoftware(crid); - if (error) + error = checkforsoftware(&crid); + if (error) { + CRYPTDEB("checkforsoftware"); goto bail; + } } else crid = CRYPTOCAP_F_HARDWARE; error = crypto_newsession(&sid, (txform ? &crie : &cria), crid); - if (error) + if (error) { + CRYPTDEB("crypto_newsession"); goto bail; + } cse = csecreate(fcr, sid, crie.cri_key, crie.cri_klen, cria.cri_key, cria.cri_klen, sop->cipher, sop->mac, txform, thash); if (cse == NULL) { crypto_freesession(sid); error = EINVAL; + CRYPTDEB("csecreate"); goto bail; } sop->ses = cse->ses; if (cmd == CIOCGSESSION2 #ifdef COMPAT_FREEBSD32 || cmd == CIOCGSESSION232 #endif ) { /* return hardware/driver id */ SES2(sop)->crid = CRYPTO_SESID2HID(cse->sid); } bail: if (error) { if (crie.cri_key) free(crie.cri_key, M_XDATA); if (cria.cri_key) free(cria.cri_key, M_XDATA); } #ifdef COMPAT_FREEBSD32 else { if (cmd == CIOCGSESSION32) session_op_to_32(sop, data); else if (cmd == CIOCGSESSION232) session2_op_to_32((struct session2_op *)sop, data); } #endif break; case CIOCFSESSION: ses = *(u_int32_t *)data; cse = csefind(fcr, ses); if (cse == NULL) return (EINVAL); csedelete(fcr, cse); error = csefree(cse); break; case CIOCCRYPT: #ifdef COMPAT_FREEBSD32 case CIOCCRYPT32: if (cmd == CIOCCRYPT32) { cop = &copc; crypt_op_from_32(data, cop); } else #endif cop = (struct crypt_op *)data; cse = csefind(fcr, cop->ses); - if (cse == NULL) + if (cse == NULL) { + SDT_PROBE1(opencrypto, dev, ioctl, error, __LINE__); return (EINVAL); + } error = cryptodev_op(cse, cop, active_cred, td); #ifdef COMPAT_FREEBSD32 if (error == 0 && cmd == CIOCCRYPT32) crypt_op_to_32(cop, data); #endif break; case CIOCKEY: case CIOCKEY2: #ifdef COMPAT_FREEBSD32 case CIOCKEY32: case CIOCKEY232: #endif if (!crypto_userasymcrypto) return (EPERM); /* XXX compat? */ #ifdef COMPAT_FREEBSD32 if (cmd == CIOCKEY32 || cmd == CIOCKEY232) { kop = &kopc; crypt_kop_from_32(data, kop); } else #endif kop = (struct crypt_kop *)data; if (cmd == CIOCKEY #ifdef COMPAT_FREEBSD32 || cmd == CIOCKEY32 #endif ) { /* NB: crypto core enforces s/w driver use */ kop->crk_crid = CRYPTOCAP_F_HARDWARE | CRYPTOCAP_F_SOFTWARE; } mtx_lock(&Giant); error = cryptodev_key(kop); mtx_unlock(&Giant); #ifdef COMPAT_FREEBSD32 if (cmd == CIOCKEY32 || cmd == CIOCKEY232) crypt_kop_to_32(kop, data); #endif break; case CIOCASYMFEAT: if (!crypto_userasymcrypto) { /* * NB: if user asym crypto operations are * not permitted return "no algorithms" * so well-behaved applications will just * fallback to doing them in software. */ *(int *)data = 0; } else error = crypto_getfeat((int *)data); break; case CIOCFINDDEV: error = cryptodev_find((struct crypt_find_op *)data); break; + case CIOCCRYPTAEAD: + caead = (struct crypt_aead *)data; + cse = csefind(fcr, caead->ses); + if (cse == NULL) + return (EINVAL); + error = cryptodev_aead(cse, caead, active_cred, td); + break; default: error = EINVAL; break; } return (error); #undef SES2 } static int cryptodev_cb(void *); static int cryptodev_op( struct csession *cse, struct crypt_op *cop, struct ucred *active_cred, struct thread *td) { struct cryptop *crp = NULL; struct cryptodesc *crde = NULL, *crda = NULL; int error; - if (cop->len > 256*1024-4) + if (cop->len > 256*1024-4) { + SDT_PROBE1(opencrypto, dev, ioctl, error, __LINE__); return (E2BIG); + } if (cse->txform) { - if (cop->len == 0 || (cop->len % cse->txform->blocksize) != 0) + if (cop->len == 0 || (cop->len % cse->txform->blocksize) != 0) { + SDT_PROBE1(opencrypto, dev, ioctl, error, __LINE__); return (EINVAL); + } } cse->uio.uio_iov = &cse->iovec; cse->uio.uio_iovcnt = 1; cse->uio.uio_offset = 0; cse->uio.uio_resid = cop->len; cse->uio.uio_segflg = UIO_SYSSPACE; cse->uio.uio_rw = UIO_WRITE; cse->uio.uio_td = td; cse->uio.uio_iov[0].iov_len = cop->len; if (cse->thash) { cse->uio.uio_iov[0].iov_len += cse->thash->hashsize; cse->uio.uio_resid += cse->thash->hashsize; } cse->uio.uio_iov[0].iov_base = malloc(cse->uio.uio_iov[0].iov_len, M_XDATA, M_WAITOK); crp = crypto_getreq((cse->txform != NULL) + (cse->thash != NULL)); if (crp == NULL) { + SDT_PROBE1(opencrypto, dev, ioctl, error, __LINE__); error = ENOMEM; goto bail; } if (cse->thash) { crda = crp->crp_desc; if (cse->txform) crde = crda->crd_next; } else { if (cse->txform) crde = crp->crp_desc; else { + SDT_PROBE1(opencrypto, dev, ioctl, error, __LINE__); error = EINVAL; goto bail; } } - if ((error = copyin(cop->src, cse->uio.uio_iov[0].iov_base, cop->len))) + if ((error = copyin(cop->src, cse->uio.uio_iov[0].iov_base, + cop->len))) { + SDT_PROBE1(opencrypto, dev, ioctl, error, __LINE__); goto bail; + } if (crda) { crda->crd_skip = 0; crda->crd_len = cop->len; crda->crd_inject = cop->len; crda->crd_alg = cse->mac; crda->crd_key = cse->mackey; crda->crd_klen = cse->mackeylen * 8; } if (crde) { if (cop->op == COP_ENCRYPT) crde->crd_flags |= CRD_F_ENCRYPT; else crde->crd_flags &= ~CRD_F_ENCRYPT; crde->crd_len = cop->len; crde->crd_inject = 0; crde->crd_alg = cse->cipher; crde->crd_key = cse->key; crde->crd_klen = cse->keylen * 8; } crp->crp_ilen = cop->len; crp->crp_flags = CRYPTO_F_IOV | CRYPTO_F_CBIMM | (cop->flags & COP_F_BATCH); crp->crp_buf = (caddr_t)&cse->uio; crp->crp_callback = (int (*) (struct cryptop *)) cryptodev_cb; crp->crp_sid = cse->sid; crp->crp_opaque = (void *)cse; if (cop->iv) { if (crde == NULL) { + SDT_PROBE1(opencrypto, dev, ioctl, error, __LINE__); error = EINVAL; goto bail; } if (cse->cipher == CRYPTO_ARC4) { /* XXX use flag? */ + SDT_PROBE1(opencrypto, dev, ioctl, error, __LINE__); error = EINVAL; goto bail; } - if ((error = copyin(cop->iv, cse->tmp_iv, cse->txform->blocksize))) + if ((error = copyin(cop->iv, cse->tmp_iv, + cse->txform->blocksize))) { + SDT_PROBE1(opencrypto, dev, ioctl, error, __LINE__); goto bail; + } bcopy(cse->tmp_iv, crde->crd_iv, cse->txform->blocksize); crde->crd_flags |= CRD_F_IV_EXPLICIT | CRD_F_IV_PRESENT; crde->crd_skip = 0; } else if (cse->cipher == CRYPTO_ARC4) { /* XXX use flag? */ crde->crd_skip = 0; } else if (crde) { crde->crd_flags |= CRD_F_IV_PRESENT; crde->crd_skip = cse->txform->blocksize; crde->crd_len -= cse->txform->blocksize; } if (cop->mac && crda == NULL) { + SDT_PROBE1(opencrypto, dev, ioctl, error, __LINE__); error = EINVAL; goto bail; } again: /* * Let the dispatch run unlocked, then, interlock against the * callback before checking if the operation completed and going * to sleep. This insures drivers don't inherit our lock which * results in a lock order reversal between crypto_dispatch forced * entry and the crypto_done callback into us. */ error = crypto_dispatch(crp); mtx_lock(&cse->lock); if (error == 0 && (crp->crp_flags & CRYPTO_F_DONE) == 0) error = msleep(crp, &cse->lock, PWAIT, "crydev", 0); mtx_unlock(&cse->lock); - if (error != 0) + if (error != 0) { + SDT_PROBE1(opencrypto, dev, ioctl, error, __LINE__); goto bail; + } if (crp->crp_etype == EAGAIN) { crp->crp_etype = 0; crp->crp_flags &= ~CRYPTO_F_DONE; goto again; } if (crp->crp_etype != 0) { + SDT_PROBE1(opencrypto, dev, ioctl, error, __LINE__); error = crp->crp_etype; goto bail; } if (cse->error) { + SDT_PROBE1(opencrypto, dev, ioctl, error, __LINE__); error = cse->error; goto bail; } if (cop->dst && - (error = copyout(cse->uio.uio_iov[0].iov_base, cop->dst, cop->len))) + (error = copyout(cse->uio.uio_iov[0].iov_base, cop->dst, + cop->len))) { + SDT_PROBE1(opencrypto, dev, ioctl, error, __LINE__); goto bail; + } if (cop->mac && (error = copyout((caddr_t)cse->uio.uio_iov[0].iov_base + cop->len, - cop->mac, cse->thash->hashsize))) + cop->mac, cse->thash->hashsize))) { + SDT_PROBE1(opencrypto, dev, ioctl, error, __LINE__); goto bail; + } bail: if (crp) crypto_freereq(crp); if (cse->uio.uio_iov[0].iov_base) free(cse->uio.uio_iov[0].iov_base, M_XDATA); return (error); } static int +cryptodev_aead( + struct csession *cse, + struct crypt_aead *caead, + struct ucred *active_cred, + struct thread *td) +{ + struct uio *uio; + struct cryptop *crp = NULL; + struct cryptodesc *crde = NULL, *crda = NULL; + int error; + + if (caead->len > 256*1024-4 || caead->aadlen > 256*1024-4) + return (E2BIG); + + if (cse->txform == NULL || cse->thash == NULL || caead->tag == NULL || + (caead->len % cse->txform->blocksize) != 0) + return (EINVAL); + + uio = &cse->uio; + uio->uio_iov = &cse->iovec; + uio->uio_iovcnt = 1; + uio->uio_offset = 0; + uio->uio_resid = caead->len + caead->aadlen + cse->thash->hashsize; + uio->uio_segflg = UIO_SYSSPACE; + uio->uio_rw = UIO_WRITE; + uio->uio_td = td; + uio->uio_iov[0].iov_len = uio->uio_resid; + + uio->uio_iov[0].iov_base = malloc(uio->uio_iov[0].iov_len, + M_XDATA, M_WAITOK); + + crp = crypto_getreq(2); + if (crp == NULL) { + error = ENOMEM; + goto bail; + } + + crda = crp->crp_desc; + crde = crda->crd_next; + + if ((error = copyin(caead->src, cse->uio.uio_iov[0].iov_base, + caead->len))) + goto bail; + + if ((error = copyin(caead->aad, (char *)cse->uio.uio_iov[0].iov_base + + caead->len, caead->aadlen))) + goto bail; + + crda->crd_skip = caead->len; + crda->crd_len = caead->aadlen; + crda->crd_inject = caead->len + caead->aadlen; + + crda->crd_alg = cse->mac; + crda->crd_key = cse->mackey; + crda->crd_klen = cse->mackeylen * 8; + + if (caead->op == COP_ENCRYPT) + crde->crd_flags |= CRD_F_ENCRYPT; + else + crde->crd_flags &= ~CRD_F_ENCRYPT; + /* crde->crd_skip set below */ + crde->crd_len = caead->len; + crde->crd_inject = 0; + + crde->crd_alg = cse->cipher; + crde->crd_key = cse->key; + crde->crd_klen = cse->keylen * 8; + + crp->crp_ilen = caead->len + caead->aadlen; + crp->crp_flags = CRYPTO_F_IOV | CRYPTO_F_CBIMM + | (caead->flags & COP_F_BATCH); + crp->crp_buf = (caddr_t)&cse->uio.uio_iov; + crp->crp_callback = (int (*) (struct cryptop *)) cryptodev_cb; + crp->crp_sid = cse->sid; + crp->crp_opaque = (void *)cse; + + if (caead->iv) { + if (caead->ivlen > sizeof cse->tmp_iv) { + error = EINVAL; + goto bail; + } + + if ((error = copyin(caead->iv, cse->tmp_iv, caead->ivlen))) + goto bail; + bcopy(cse->tmp_iv, crde->crd_iv, caead->ivlen); + crde->crd_flags |= CRD_F_IV_EXPLICIT | CRD_F_IV_PRESENT; + crde->crd_skip = 0; + } else { + crde->crd_flags |= CRD_F_IV_PRESENT; + crde->crd_skip = cse->txform->blocksize; + crde->crd_len -= cse->txform->blocksize; + } + + if ((error = copyin(caead->tag, (caddr_t)cse->uio.uio_iov[0].iov_base + + caead->len + caead->aadlen, cse->thash->hashsize))) + goto bail; +again: + /* + * Let the dispatch run unlocked, then, interlock against the + * callback before checking if the operation completed and going + * to sleep. This insures drivers don't inherit our lock which + * results in a lock order reversal between crypto_dispatch forced + * entry and the crypto_done callback into us. + */ + error = crypto_dispatch(crp); + mtx_lock(&cse->lock); + if (error == 0 && (crp->crp_flags & CRYPTO_F_DONE) == 0) + error = msleep(crp, &cse->lock, PWAIT, "crydev", 0); + mtx_unlock(&cse->lock); + + if (error != 0) + goto bail; + + if (crp->crp_etype == EAGAIN) { + crp->crp_etype = 0; + crp->crp_flags &= ~CRYPTO_F_DONE; + goto again; + } + + if (crp->crp_etype != 0) { + error = crp->crp_etype; + goto bail; + } + + if (cse->error) { + error = cse->error; + goto bail; + } + + if (caead->dst && (error = copyout(cse->uio.uio_iov[0].iov_base, + caead->dst, caead->len))) + goto bail; + + if ((error = copyout((caddr_t)cse->uio.uio_iov[0].iov_base + + caead->len + caead->aadlen, caead->tag, cse->thash->hashsize))) + goto bail; + +bail: + crypto_freereq(crp); + free(cse->uio.uio_iov[0].iov_base, M_XDATA); + + return (error); +} + +static int cryptodev_cb(void *op) { struct cryptop *crp = (struct cryptop *) op; struct csession *cse = (struct csession *)crp->crp_opaque; mtx_lock(&cse->lock); cse->error = crp->crp_etype; wakeup_one(crp); mtx_unlock(&cse->lock); return (0); } static int cryptodevkey_cb(void *op) { struct cryptkop *krp = (struct cryptkop *) op; wakeup_one(krp); return (0); } static int cryptodev_key(struct crypt_kop *kop) { struct cryptkop *krp = NULL; int error = EINVAL; int in, out, size, i; if (kop->crk_iparams + kop->crk_oparams > CRK_MAXPARAM) { return (EFBIG); } in = kop->crk_iparams; out = kop->crk_oparams; switch (kop->crk_op) { case CRK_MOD_EXP: if (in == 3 && out == 1) break; return (EINVAL); case CRK_MOD_EXP_CRT: if (in == 6 && out == 1) break; return (EINVAL); case CRK_DSA_SIGN: if (in == 5 && out == 2) break; return (EINVAL); case CRK_DSA_VERIFY: if (in == 7 && out == 0) break; return (EINVAL); case CRK_DH_COMPUTE_KEY: if (in == 3 && out == 1) break; return (EINVAL); default: return (EINVAL); } krp = (struct cryptkop *)malloc(sizeof *krp, M_XDATA, M_WAITOK|M_ZERO); if (!krp) return (ENOMEM); krp->krp_op = kop->crk_op; krp->krp_status = kop->crk_status; krp->krp_iparams = kop->crk_iparams; krp->krp_oparams = kop->crk_oparams; krp->krp_crid = kop->crk_crid; krp->krp_status = 0; krp->krp_callback = (int (*) (struct cryptkop *)) cryptodevkey_cb; for (i = 0; i < CRK_MAXPARAM; i++) { if (kop->crk_param[i].crp_nbits > 65536) /* Limit is the same as in OpenBSD */ goto fail; krp->krp_param[i].crp_nbits = kop->crk_param[i].crp_nbits; } for (i = 0; i < krp->krp_iparams + krp->krp_oparams; i++) { size = (krp->krp_param[i].crp_nbits + 7) / 8; if (size == 0) continue; krp->krp_param[i].crp_p = malloc(size, M_XDATA, M_WAITOK); if (i >= krp->krp_iparams) continue; error = copyin(kop->crk_param[i].crp_p, krp->krp_param[i].crp_p, size); if (error) goto fail; } error = crypto_kdispatch(krp); if (error) goto fail; error = tsleep(krp, PSOCK, "crydev", 0); if (error) { /* XXX can this happen? if so, how do we recover? */ goto fail; } kop->crk_crid = krp->krp_crid; /* device that did the work */ if (krp->krp_status != 0) { error = krp->krp_status; goto fail; } for (i = krp->krp_iparams; i < krp->krp_iparams + krp->krp_oparams; i++) { size = (krp->krp_param[i].crp_nbits + 7) / 8; if (size == 0) continue; error = copyout(krp->krp_param[i].crp_p, kop->crk_param[i].crp_p, size); if (error) goto fail; } fail: if (krp) { kop->crk_status = krp->krp_status; for (i = 0; i < CRK_MAXPARAM; i++) { if (krp->krp_param[i].crp_p) free(krp->krp_param[i].crp_p, M_XDATA); } free(krp, M_XDATA); } return (error); } static int cryptodev_find(struct crypt_find_op *find) { device_t dev; + size_t fnlen = sizeof find->name; if (find->crid != -1) { dev = crypto_find_device_byhid(find->crid); if (dev == NULL) return (ENOENT); - strlcpy(find->name, device_get_nameunit(dev), - sizeof(find->name)); + strncpy(find->name, device_get_nameunit(dev), fnlen); + find->name[fnlen - 1] = '\x0'; } else { + find->name[fnlen - 1] = '\x0'; find->crid = crypto_find_driver(find->name); if (find->crid == -1) return (ENOENT); } return (0); } /* ARGSUSED */ static int cryptof_stat( struct file *fp, struct stat *sb, struct ucred *active_cred, struct thread *td) { return (EOPNOTSUPP); } /* ARGSUSED */ static int cryptof_close(struct file *fp, struct thread *td) { struct fcrypt *fcr = fp->f_data; struct csession *cse; while ((cse = TAILQ_FIRST(&fcr->csessions))) { TAILQ_REMOVE(&fcr->csessions, cse, next); (void)csefree(cse); } free(fcr, M_XDATA); fp->f_data = NULL; return 0; } static int cryptof_fill_kinfo(struct file *fp, struct kinfo_file *kif, struct filedesc *fdp) { kif->kf_type = KF_TYPE_CRYPTO; return (0); } static struct csession * csefind(struct fcrypt *fcr, u_int ses) { struct csession *cse; TAILQ_FOREACH(cse, &fcr->csessions, next) if (cse->ses == ses) return (cse); return (NULL); } static int csedelete(struct fcrypt *fcr, struct csession *cse_del) { struct csession *cse; TAILQ_FOREACH(cse, &fcr->csessions, next) { if (cse == cse_del) { TAILQ_REMOVE(&fcr->csessions, cse, next); return (1); } } return (0); } static struct csession * cseadd(struct fcrypt *fcr, struct csession *cse) { TAILQ_INSERT_TAIL(&fcr->csessions, cse, next); cse->ses = fcr->sesn++; return (cse); } struct csession * csecreate(struct fcrypt *fcr, u_int64_t sid, caddr_t key, u_int64_t keylen, caddr_t mackey, u_int64_t mackeylen, u_int32_t cipher, u_int32_t mac, struct enc_xform *txform, struct auth_hash *thash) { struct csession *cse; cse = malloc(sizeof(struct csession), M_XDATA, M_NOWAIT | M_ZERO); if (cse == NULL) return NULL; mtx_init(&cse->lock, "cryptodev", "crypto session lock", MTX_DEF); cse->key = key; cse->keylen = keylen/8; cse->mackey = mackey; cse->mackeylen = mackeylen/8; cse->sid = sid; cse->cipher = cipher; cse->mac = mac; cse->txform = txform; cse->thash = thash; cseadd(fcr, cse); return (cse); } static int csefree(struct csession *cse) { int error; error = crypto_freesession(cse->sid); mtx_destroy(&cse->lock); if (cse->key) free(cse->key, M_XDATA); if (cse->mackey) free(cse->mackey, M_XDATA); free(cse, M_XDATA); return (error); } static int cryptoopen(struct cdev *dev, int oflags, int devtype, struct thread *td) { return (0); } static int cryptoread(struct cdev *dev, struct uio *uio, int ioflag) { return (EIO); } static int cryptowrite(struct cdev *dev, struct uio *uio, int ioflag) { return (EIO); } static int cryptoioctl(struct cdev *dev, u_long cmd, caddr_t data, int flag, struct thread *td) { struct file *f; struct fcrypt *fcr; int fd, error; switch (cmd) { case CRIOGET: fcr = malloc(sizeof(struct fcrypt), M_XDATA, M_WAITOK); TAILQ_INIT(&fcr->csessions); fcr->sesn = 0; error = falloc(td, &f, &fd, 0); if (error) { free(fcr, M_XDATA); return (error); } /* falloc automatically provides an extra reference to 'f'. */ finit(f, FREAD | FWRITE, DTYPE_CRYPTO, fcr, &cryptofops); *(u_int32_t *)data = fd; fdrop(f, td); break; case CRIOFINDDEV: error = cryptodev_find((struct crypt_find_op *)data); break; case CRIOASYMFEAT: error = crypto_getfeat((int *)data); break; default: error = EINVAL; break; } return (error); } static struct cdevsw crypto_cdevsw = { .d_version = D_VERSION, .d_flags = D_NEEDGIANT, .d_open = cryptoopen, .d_read = cryptoread, .d_write = cryptowrite, .d_ioctl = cryptoioctl, .d_name = "crypto", }; static struct cdev *crypto_dev; /* * Initialization code, both for static and dynamic loading. */ static int cryptodev_modevent(module_t mod, int type, void *unused) { switch (type) { case MOD_LOAD: if (bootverbose) printf("crypto: \n"); crypto_dev = make_dev(&crypto_cdevsw, 0, UID_ROOT, GID_WHEEL, 0666, "crypto"); return 0; case MOD_UNLOAD: /*XXX disallow if active sessions */ destroy_dev(crypto_dev); return 0; } return EINVAL; } static moduledata_t cryptodev_mod = { "cryptodev", cryptodev_modevent, 0 }; MODULE_VERSION(cryptodev, 1); DECLARE_MODULE(cryptodev, cryptodev_mod, SI_SUB_PSEUDO, SI_ORDER_ANY); MODULE_DEPEND(cryptodev, crypto, 1, 1, 1); MODULE_DEPEND(cryptodev, zlib, 1, 1, 1); Index: projects/clang350-import/sys/opencrypto/cryptodev.h =================================================================== --- projects/clang350-import/sys/opencrypto/cryptodev.h (revision 275748) +++ projects/clang350-import/sys/opencrypto/cryptodev.h (revision 275749) @@ -1,432 +1,478 @@ /* $FreeBSD$ */ /* $OpenBSD: cryptodev.h,v 1.31 2002/06/11 11:14:29 beck Exp $ */ /*- * The author of this code is Angelos D. Keromytis (angelos@cis.upenn.edu) * Copyright (c) 2002-2006 Sam Leffler, Errno Consulting * * This code was written by Angelos D. Keromytis in Athens, Greece, in * February 2000. Network Security Technologies Inc. (NSTI) kindly * supported the development of this code. * * Copyright (c) 2000 Angelos D. Keromytis * * Permission to use, copy, and modify this software with or without fee * is hereby granted, provided that this entire notice is included in * all source code copies of any software which is or includes a copy or * modification of this software. * * THIS SOFTWARE IS BEING PROVIDED "AS IS", WITHOUT ANY EXPRESS OR * IMPLIED WARRANTY. IN PARTICULAR, NONE OF THE AUTHORS MAKES ANY * REPRESENTATION OR WARRANTY OF ANY KIND CONCERNING THE * MERCHANTABILITY OF THIS SOFTWARE OR ITS FITNESS FOR ANY PARTICULAR * PURPOSE. * * Copyright (c) 2001 Theo de Raadt + * Copyright (c) 2014 The FreeBSD Foundation + * All rights reserved. * + * Portions of this software were developed by John-Mark Gurney + * under sponsorship of the FreeBSD Foundation and + * Rubicon Communications, LLC (Netgate). + * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. The name of the author may not be used to endorse or promote products * derived from this software without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. * IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. * * Effort sponsored in part by the Defense Advanced Research Projects * Agency (DARPA) and Air Force Research Laboratory, Air Force * Materiel Command, USAF, under agreement number F30602-01-2-0537. * */ #ifndef _CRYPTO_CRYPTO_H_ #define _CRYPTO_CRYPTO_H_ #include /* Some initial values */ #define CRYPTO_DRIVERS_INITIAL 4 #define CRYPTO_SW_SESSIONS 32 /* Hash values */ #define NULL_HASH_LEN 16 #define MD5_HASH_LEN 16 #define SHA1_HASH_LEN 20 #define RIPEMD160_HASH_LEN 20 #define SHA2_256_HASH_LEN 32 #define SHA2_384_HASH_LEN 48 #define SHA2_512_HASH_LEN 64 #define MD5_KPDK_HASH_LEN 16 #define SHA1_KPDK_HASH_LEN 20 /* Maximum hash algorithm result length */ #define HASH_MAX_LEN SHA2_512_HASH_LEN /* Keep this updated */ /* HMAC values */ #define NULL_HMAC_BLOCK_LEN 64 #define MD5_HMAC_BLOCK_LEN 64 #define SHA1_HMAC_BLOCK_LEN 64 #define RIPEMD160_HMAC_BLOCK_LEN 64 #define SHA2_256_HMAC_BLOCK_LEN 64 #define SHA2_384_HMAC_BLOCK_LEN 128 #define SHA2_512_HMAC_BLOCK_LEN 128 /* Maximum HMAC block length */ #define HMAC_MAX_BLOCK_LEN SHA2_512_HMAC_BLOCK_LEN /* Keep this updated */ #define HMAC_IPAD_VAL 0x36 #define HMAC_OPAD_VAL 0x5C /* Encryption algorithm block sizes */ #define NULL_BLOCK_LEN 4 #define DES_BLOCK_LEN 8 #define DES3_BLOCK_LEN 8 #define BLOWFISH_BLOCK_LEN 8 #define SKIPJACK_BLOCK_LEN 8 #define CAST128_BLOCK_LEN 8 #define RIJNDAEL128_BLOCK_LEN 16 #define AES_BLOCK_LEN RIJNDAEL128_BLOCK_LEN #define CAMELLIA_BLOCK_LEN 16 #define EALG_MAX_BLOCK_LEN AES_BLOCK_LEN /* Keep this updated */ +/* Maximum hash algorithm result length */ +#define AALG_MAX_RESULT_LEN 64 /* Keep this updated */ + #define CRYPTO_ALGORITHM_MIN 1 -#define CRYPTO_DES_CBC 1 -#define CRYPTO_3DES_CBC 2 -#define CRYPTO_BLF_CBC 3 -#define CRYPTO_CAST_CBC 4 -#define CRYPTO_SKIPJACK_CBC 5 -#define CRYPTO_MD5_HMAC 6 -#define CRYPTO_SHA1_HMAC 7 -#define CRYPTO_RIPEMD160_HMAC 8 -#define CRYPTO_MD5_KPDK 9 -#define CRYPTO_SHA1_KPDK 10 -#define CRYPTO_RIJNDAEL128_CBC 11 /* 128 bit blocksize */ -#define CRYPTO_AES_CBC 11 /* 128 bit blocksize -- the same as above */ -#define CRYPTO_ARC4 12 +#define CRYPTO_DES_CBC 1 +#define CRYPTO_3DES_CBC 2 +#define CRYPTO_BLF_CBC 3 +#define CRYPTO_CAST_CBC 4 +#define CRYPTO_SKIPJACK_CBC 5 +#define CRYPTO_MD5_HMAC 6 +#define CRYPTO_SHA1_HMAC 7 +#define CRYPTO_RIPEMD160_HMAC 8 +#define CRYPTO_MD5_KPDK 9 +#define CRYPTO_SHA1_KPDK 10 +#define CRYPTO_RIJNDAEL128_CBC 11 /* 128 bit blocksize */ +#define CRYPTO_AES_CBC 11 /* 128 bit blocksize -- the same as above */ +#define CRYPTO_ARC4 12 #define CRYPTO_MD5 13 #define CRYPTO_SHA1 14 #define CRYPTO_NULL_HMAC 15 #define CRYPTO_NULL_CBC 16 #define CRYPTO_DEFLATE_COMP 17 /* Deflate compression algorithm */ #define CRYPTO_SHA2_256_HMAC 18 #define CRYPTO_SHA2_384_HMAC 19 #define CRYPTO_SHA2_512_HMAC 20 -#define CRYPTO_CAMELLIA_CBC 21 +#define CRYPTO_CAMELLIA_CBC 21 #define CRYPTO_AES_XTS 22 -#define CRYPTO_ALGORITHM_MAX 22 /* Keep updated - see below */ +#define CRYPTO_AES_ICM 23 /* commonly known as CTR mode */ +#define CRYPTO_AES_NIST_GMAC 24 /* cipher side */ +#define CRYPTO_AES_NIST_GCM_16 25 /* 16 byte ICV */ +#define CRYPTO_AES_128_NIST_GMAC 26 /* auth side */ +#define CRYPTO_AES_192_NIST_GMAC 27 /* auth side */ +#define CRYPTO_AES_256_NIST_GMAC 28 /* auth side */ +#define CRYPTO_ALGORITHM_MAX 28 /* Keep updated - see below */ +#define CRYPTO_ALGO_VALID(x) ((x) >= CRYPTO_ALGORITHM_MIN && \ + (x) <= CRYPTO_ALGORITHM_MAX) + /* Algorithm flags */ #define CRYPTO_ALG_FLAG_SUPPORTED 0x01 /* Algorithm is supported */ #define CRYPTO_ALG_FLAG_RNG_ENABLE 0x02 /* Has HW RNG for DH/DSA */ #define CRYPTO_ALG_FLAG_DSA_SHA 0x04 /* Can do SHA on msg */ /* * Crypto driver/device flags. They can set in the crid * parameter when creating a session or submitting a key * op to affect the device/driver assigned. If neither * of these are specified then the crid is assumed to hold * the driver id of an existing (and suitable) device that * must be used to satisfy the request. */ #define CRYPTO_FLAG_HARDWARE 0x01000000 /* hardware accelerated */ #define CRYPTO_FLAG_SOFTWARE 0x02000000 /* software implementation */ /* NB: deprecated */ struct session_op { u_int32_t cipher; /* ie. CRYPTO_DES_CBC */ u_int32_t mac; /* ie. CRYPTO_MD5_HMAC */ u_int32_t keylen; /* cipher key */ caddr_t key; int mackeylen; /* mac key */ caddr_t mackey; u_int32_t ses; /* returns: session # */ }; struct session2_op { u_int32_t cipher; /* ie. CRYPTO_DES_CBC */ u_int32_t mac; /* ie. CRYPTO_MD5_HMAC */ u_int32_t keylen; /* cipher key */ caddr_t key; int mackeylen; /* mac key */ caddr_t mackey; u_int32_t ses; /* returns: session # */ int crid; /* driver id + flags (rw) */ int pad[4]; /* for future expansion */ }; struct crypt_op { u_int32_t ses; u_int16_t op; /* i.e. COP_ENCRYPT */ #define COP_ENCRYPT 1 #define COP_DECRYPT 2 u_int16_t flags; #define COP_F_BATCH 0x0008 /* Batch op if possible */ u_int len; caddr_t src, dst; /* become iov[] inside kernel */ caddr_t mac; /* must be big enough for chosen MAC */ caddr_t iv; }; +/* op and flags the same as crypt_op */ +struct crypt_aead { + u_int32_t ses; + u_int16_t op; /* i.e. COP_ENCRYPT */ + u_int16_t flags; + u_int len; + u_int aadlen; + u_int ivlen; + caddr_t src, dst; /* become iov[] inside kernel */ + caddr_t aad; /* additional authenticated data */ + caddr_t tag; /* must fit for chosen TAG length */ + caddr_t iv; +}; + /* * Parameters for looking up a crypto driver/device by * device name or by id. The latter are returned for * created sessions (crid) and completed key operations. */ struct crypt_find_op { int crid; /* driver id + flags */ char name[32]; /* device/driver name */ }; /* bignum parameter, in packed bytes, ... */ struct crparam { caddr_t crp_p; u_int crp_nbits; }; #define CRK_MAXPARAM 8 struct crypt_kop { u_int crk_op; /* ie. CRK_MOD_EXP or other */ u_int crk_status; /* return status */ u_short crk_iparams; /* # of input parameters */ u_short crk_oparams; /* # of output parameters */ u_int crk_crid; /* NB: only used by CIOCKEY2 (rw) */ struct crparam crk_param[CRK_MAXPARAM]; }; #define CRK_ALGORITM_MIN 0 #define CRK_MOD_EXP 0 #define CRK_MOD_EXP_CRT 1 #define CRK_DSA_SIGN 2 #define CRK_DSA_VERIFY 3 #define CRK_DH_COMPUTE_KEY 4 #define CRK_ALGORITHM_MAX 4 /* Keep updated - see below */ #define CRF_MOD_EXP (1 << CRK_MOD_EXP) #define CRF_MOD_EXP_CRT (1 << CRK_MOD_EXP_CRT) #define CRF_DSA_SIGN (1 << CRK_DSA_SIGN) #define CRF_DSA_VERIFY (1 << CRK_DSA_VERIFY) #define CRF_DH_COMPUTE_KEY (1 << CRK_DH_COMPUTE_KEY) /* * done against open of /dev/crypto, to get a cloned descriptor. * Please use F_SETFD against the cloned descriptor. */ #define CRIOGET _IOWR('c', 100, u_int32_t) #define CRIOASYMFEAT CIOCASYMFEAT #define CRIOFINDDEV CIOCFINDDEV /* the following are done against the cloned descriptor */ #define CIOCGSESSION _IOWR('c', 101, struct session_op) #define CIOCFSESSION _IOW('c', 102, u_int32_t) #define CIOCCRYPT _IOWR('c', 103, struct crypt_op) #define CIOCKEY _IOWR('c', 104, struct crypt_kop) #define CIOCASYMFEAT _IOR('c', 105, u_int32_t) #define CIOCGSESSION2 _IOWR('c', 106, struct session2_op) #define CIOCKEY2 _IOWR('c', 107, struct crypt_kop) #define CIOCFINDDEV _IOWR('c', 108, struct crypt_find_op) +#define CIOCCRYPTAEAD _IOWR('c', 109, struct crypt_aead) struct cryptotstat { struct timespec acc; /* total accumulated time */ struct timespec min; /* min time */ struct timespec max; /* max time */ u_int32_t count; /* number of observations */ }; struct cryptostats { u_int32_t cs_ops; /* symmetric crypto ops submitted */ u_int32_t cs_errs; /* symmetric crypto ops that failed */ u_int32_t cs_kops; /* asymetric/key ops submitted */ u_int32_t cs_kerrs; /* asymetric/key ops that failed */ u_int32_t cs_intrs; /* crypto swi thread activations */ u_int32_t cs_rets; /* crypto return thread activations */ u_int32_t cs_blocks; /* symmetric op driver block */ u_int32_t cs_kblocks; /* symmetric op driver block */ /* * When CRYPTO_TIMING is defined at compile time and the * sysctl debug.crypto is set to 1, the crypto system will * accumulate statistics about how long it takes to process * crypto requests at various points during processing. */ struct cryptotstat cs_invoke; /* crypto_dipsatch -> crypto_invoke */ struct cryptotstat cs_done; /* crypto_invoke -> crypto_done */ struct cryptotstat cs_cb; /* crypto_done -> callback */ struct cryptotstat cs_finis; /* callback -> callback return */ }; #ifdef _KERNEL + +#if 0 +#define CRYPTDEB(s) do { printf("%s:%d: %s\n", __FILE__, __LINE__, s); \ + } while (0) +#else +#define CRYPTDEB(s) do { } while (0) +#endif + /* Standard initialization structure beginning */ struct cryptoini { int cri_alg; /* Algorithm to use */ int cri_klen; /* Key length, in bits */ int cri_mlen; /* Number of bytes we want from the entire hash. 0 means all. */ caddr_t cri_key; /* key to use */ u_int8_t cri_iv[EALG_MAX_BLOCK_LEN]; /* IV to use */ struct cryptoini *cri_next; }; /* Describe boundaries of a single crypto operation */ struct cryptodesc { int crd_skip; /* How many bytes to ignore from start */ int crd_len; /* How many bytes to process */ int crd_inject; /* Where to inject results, if applicable */ int crd_flags; #define CRD_F_ENCRYPT 0x01 /* Set when doing encryption */ #define CRD_F_IV_PRESENT 0x02 /* When encrypting, IV is already in place, so don't copy. */ #define CRD_F_IV_EXPLICIT 0x04 /* IV explicitly provided */ #define CRD_F_DSA_SHA_NEEDED 0x08 /* Compute SHA-1 of buffer for DSA */ +#define CRD_F_COMP 0x0f /* Set when doing compression */ #define CRD_F_KEY_EXPLICIT 0x10 /* Key explicitly provided */ -#define CRD_F_COMP 0x0f /* Set when doing compression */ struct cryptoini CRD_INI; /* Initialization/context data */ -#define crd_iv CRD_INI.cri_iv -#define crd_key CRD_INI.cri_key -#define crd_alg CRD_INI.cri_alg -#define crd_klen CRD_INI.cri_klen +#define crd_esn CRD_INI.cri_esn +#define crd_iv CRD_INI.cri_iv +#define crd_key CRD_INI.cri_key +#define crd_alg CRD_INI.cri_alg +#define crd_klen CRD_INI.cri_klen struct cryptodesc *crd_next; }; /* Structure describing complete operation */ struct cryptop { TAILQ_ENTRY(cryptop) crp_next; u_int64_t crp_sid; /* Session ID */ int crp_ilen; /* Input data total length */ int crp_olen; /* Result total length */ int crp_etype; /* * Error type (zero means no error). * All error codes except EAGAIN * indicate possible data corruption (as in, * the data have been touched). On all * errors, the crp_sid may have changed * (reset to a new one), so the caller * should always check and use the new * value on future requests. */ int crp_flags; -#define CRYPTO_F_IMBUF 0x0001 /* Input/output are mbuf chains */ -#define CRYPTO_F_IOV 0x0002 /* Input/output are uio */ -#define CRYPTO_F_REL 0x0004 /* Must return data in same place */ +#define CRYPTO_F_IMBUF 0x0001 /* Input/output are mbuf chains */ +#define CRYPTO_F_IOV 0x0002 /* Input/output are uio */ #define CRYPTO_F_BATCH 0x0008 /* Batch op if possible */ #define CRYPTO_F_CBIMM 0x0010 /* Do callback immediately */ #define CRYPTO_F_DONE 0x0020 /* Operation completed */ #define CRYPTO_F_CBIFSYNC 0x0040 /* Do CBIMM if op is synchronous */ caddr_t crp_buf; /* Data to be processed */ caddr_t crp_opaque; /* Opaque pointer, passed along */ struct cryptodesc *crp_desc; /* Linked list of processing descriptors */ int (*crp_callback)(struct cryptop *); /* Callback function */ struct bintime crp_tstamp; /* performance time stamp */ }; -#define CRYPTO_BUF_CONTIG 0x0 -#define CRYPTO_BUF_IOV 0x1 -#define CRYPTO_BUF_MBUF 0x2 +#define CRYPTO_BUF_CONTIG 0x0 +#define CRYPTO_BUF_IOV 0x1 +#define CRYPTO_BUF_MBUF 0x2 -#define CRYPTO_OP_DECRYPT 0x0 -#define CRYPTO_OP_ENCRYPT 0x1 +#define CRYPTO_OP_DECRYPT 0x0 +#define CRYPTO_OP_ENCRYPT 0x1 /* * Hints passed to process methods. */ #define CRYPTO_HINT_MORE 0x1 /* more ops coming shortly */ struct cryptkop { TAILQ_ENTRY(cryptkop) krp_next; u_int krp_op; /* ie. CRK_MOD_EXP or other */ u_int krp_status; /* return status */ u_short krp_iparams; /* # of input parameters */ u_short krp_oparams; /* # of output parameters */ u_int krp_crid; /* desired device, etc. */ u_int32_t krp_hid; struct crparam krp_param[CRK_MAXPARAM]; /* kvm */ int (*krp_callback)(struct cryptkop *); }; /* * Session ids are 64 bits. The lower 32 bits contain a "local id" which * is a driver-private session identifier. The upper 32 bits contain a * "hardware id" used by the core crypto code to identify the driver and * a copy of the driver's capabilities that can be used by client code to * optimize operation. */ #define CRYPTO_SESID2HID(_sid) (((_sid) >> 32) & 0x00ffffff) #define CRYPTO_SESID2CAPS(_sid) (((_sid) >> 32) & 0xff000000) #define CRYPTO_SESID2LID(_sid) (((u_int32_t) (_sid)) & 0xffffffff) MALLOC_DECLARE(M_CRYPTO_DATA); extern int crypto_newsession(u_int64_t *sid, struct cryptoini *cri, int hard); extern int crypto_freesession(u_int64_t sid); -#define CRYPTOCAP_F_HARDWARE CRYPTO_FLAG_HARDWARE -#define CRYPTOCAP_F_SOFTWARE CRYPTO_FLAG_SOFTWARE -#define CRYPTOCAP_F_SYNC 0x04000000 /* operates synchronously */ +#define CRYPTOCAP_F_HARDWARE CRYPTO_FLAG_HARDWARE +#define CRYPTOCAP_F_SOFTWARE CRYPTO_FLAG_SOFTWARE +#define CRYPTOCAP_F_SYNC 0x04000000 /* operates synchronously */ extern int32_t crypto_get_driverid(device_t dev, int flags); extern int crypto_find_driver(const char *); extern device_t crypto_find_device_byhid(int hid); extern int crypto_getcaps(int hid); extern int crypto_register(u_int32_t driverid, int alg, u_int16_t maxoplen, u_int32_t flags); extern int crypto_kregister(u_int32_t, int, u_int32_t); extern int crypto_unregister(u_int32_t driverid, int alg); extern int crypto_unregister_all(u_int32_t driverid); extern int crypto_dispatch(struct cryptop *crp); extern int crypto_kdispatch(struct cryptkop *); #define CRYPTO_SYMQ 0x1 #define CRYPTO_ASYMQ 0x2 extern int crypto_unblock(u_int32_t, int); extern void crypto_done(struct cryptop *crp); extern void crypto_kdone(struct cryptkop *); extern int crypto_getfeat(int *); extern void crypto_freereq(struct cryptop *crp); extern struct cryptop *crypto_getreq(int num); extern int crypto_usercrypto; /* userland may do crypto requests */ extern int crypto_userasymcrypto; /* userland may do asym crypto reqs */ extern int crypto_devallowsoft; /* only use hardware crypto */ /* * Crypto-related utility routines used mainly by drivers. * * XXX these don't really belong here; but for now they're * kept apart from the rest of the system. */ struct uio; extern void cuio_copydata(struct uio* uio, int off, int len, caddr_t cp); extern void cuio_copyback(struct uio* uio, int off, int len, caddr_t cp); -extern struct iovec *cuio_getptr(struct uio *uio, int loc, int *off); +extern int cuio_getptr(struct uio *uio, int loc, int *off); extern int cuio_apply(struct uio *uio, int off, int len, int (*f)(void *, void *, u_int), void *arg); + +struct mbuf; +struct iovec; +extern void crypto_mbuftoiov(struct mbuf *mbuf, struct iovec **iovptr, + int *cnt, int *allocated); extern void crypto_copyback(int flags, caddr_t buf, int off, int size, caddr_t in); extern void crypto_copydata(int flags, caddr_t buf, int off, int size, caddr_t out); extern int crypto_apply(int flags, caddr_t buf, int off, int len, int (*f)(void *, void *, u_int), void *arg); #endif /* _KERNEL */ #endif /* _CRYPTO_CRYPTO_H_ */ Index: projects/clang350-import/sys/opencrypto/cryptosoft.c =================================================================== --- projects/clang350-import/sys/opencrypto/cryptosoft.c (revision 275748) +++ projects/clang350-import/sys/opencrypto/cryptosoft.c (revision 275749) @@ -1,1197 +1,1289 @@ /* $OpenBSD: cryptosoft.c,v 1.35 2002/04/26 08:43:50 deraadt Exp $ */ /*- * The author of this code is Angelos D. Keromytis (angelos@cis.upenn.edu) * Copyright (c) 2002-2006 Sam Leffler, Errno Consulting * * This code was written by Angelos D. Keromytis in Athens, Greece, in * February 2000. Network Security Technologies Inc. (NSTI) kindly * supported the development of this code. * * Copyright (c) 2000, 2001 Angelos D. Keromytis + * Copyright (c) 2014 The FreeBSD Foundation + * All rights reserved. * + * Portions of this software were developed by John-Mark Gurney + * under sponsorship of the FreeBSD Foundation and + * Rubicon Communications, LLC (Netgate). + * * Permission to use, copy, and modify this software with or without fee * is hereby granted, provided that this entire notice is included in * all source code copies of any software which is or includes a copy or * modification of this software. * * THIS SOFTWARE IS BEING PROVIDED "AS IS", WITHOUT ANY EXPRESS OR * IMPLIED WARRANTY. IN PARTICULAR, NONE OF THE AUTHORS MAKES ANY * REPRESENTATION OR WARRANTY OF ANY KIND CONCERNING THE * MERCHANTABILITY OF THIS SOFTWARE OR ITS FITNESS FOR ANY PARTICULAR * PURPOSE. */ #include __FBSDID("$FreeBSD$"); #include #include #include #include #include #include #include #include #include #include #include #include +#include +#include #include #include #include #include #include #include #include #include #include #include #include #include "cryptodev_if.h" static int32_t swcr_id; static struct swcr_data **swcr_sessions = NULL; static u_int32_t swcr_sesnum; /* Protects swcr_sessions pointer, not data. */ static struct rwlock swcr_sessions_lock; u_int8_t hmac_ipad_buffer[HMAC_MAX_BLOCK_LEN]; u_int8_t hmac_opad_buffer[HMAC_MAX_BLOCK_LEN]; static int swcr_encdec(struct cryptodesc *, struct swcr_data *, caddr_t, int); static int swcr_authcompute(struct cryptodesc *, struct swcr_data *, caddr_t, int); +static int swcr_authenc(struct cryptop *crp); static int swcr_compdec(struct cryptodesc *, struct swcr_data *, caddr_t, int); static int swcr_freesession(device_t dev, u_int64_t tid); static int swcr_freesession_locked(device_t dev, u_int64_t tid); /* * Apply a symmetric encryption/decryption algorithm. */ static int swcr_encdec(struct cryptodesc *crd, struct swcr_data *sw, caddr_t buf, int flags) { unsigned char iv[EALG_MAX_BLOCK_LEN], blk[EALG_MAX_BLOCK_LEN], *idat; - unsigned char *ivp, piv[EALG_MAX_BLOCK_LEN]; + unsigned char *ivp, *nivp, iv2[EALG_MAX_BLOCK_LEN]; struct enc_xform *exf; - int i, k, j, blks; + int i, j, k, blks, ind, count, ivlen; + struct uio *uio, uiolcl; + struct iovec iovlcl[4]; + struct iovec *iov; + int iovcnt, iovalloc; + int error; + error = 0; + exf = sw->sw_exf; blks = exf->blocksize; + ivlen = exf->ivsize; /* Check for non-padded data */ if (crd->crd_len % blks) return EINVAL; + if (crd->crd_alg == CRYPTO_AES_ICM && + (crd->crd_flags & CRD_F_IV_EXPLICIT) == 0) + return (EINVAL); + /* Initialize the IV */ if (crd->crd_flags & CRD_F_ENCRYPT) { /* IV explicitly provided ? */ if (crd->crd_flags & CRD_F_IV_EXPLICIT) - bcopy(crd->crd_iv, iv, blks); + bcopy(crd->crd_iv, iv, ivlen); else - arc4rand(iv, blks, 0); + arc4rand(iv, ivlen, 0); /* Do we need to write the IV */ if (!(crd->crd_flags & CRD_F_IV_PRESENT)) - crypto_copyback(flags, buf, crd->crd_inject, blks, iv); + crypto_copyback(flags, buf, crd->crd_inject, ivlen, iv); } else { /* Decryption */ - /* IV explicitly provided ? */ + /* IV explicitly provided ? */ if (crd->crd_flags & CRD_F_IV_EXPLICIT) - bcopy(crd->crd_iv, iv, blks); + bcopy(crd->crd_iv, iv, ivlen); else { /* Get IV off buf */ - crypto_copydata(flags, buf, crd->crd_inject, blks, iv); + crypto_copydata(flags, buf, crd->crd_inject, ivlen, iv); } } if (crd->crd_flags & CRD_F_KEY_EXPLICIT) { int error; if (sw->sw_kschedule) exf->zerokey(&(sw->sw_kschedule)); + error = exf->setkey(&sw->sw_kschedule, crd->crd_key, crd->crd_klen / 8); if (error) return (error); } + iov = iovlcl; + iovcnt = nitems(iovlcl); + iovalloc = 0; + uio = &uiolcl; + if ((flags & CRYPTO_F_IMBUF) != 0) { + crypto_mbuftoiov((struct mbuf *)buf, &iov, &iovcnt, + &iovalloc); + uio->uio_iov = iov; + uio->uio_iovcnt = iovcnt; + } else if ((flags & CRYPTO_F_IOV) != 0) + uio = (struct uio *)buf; + else { + iov[0].iov_base = buf; + iov[0].iov_len = crd->crd_skip + crd->crd_len; + uio->uio_iov = iov; + uio->uio_iovcnt = 1; + } + ivp = iv; - /* - * xforms that provide a reinit method perform all IV - * handling themselves. - */ - if (exf->reinit) + if (exf->reinit) { + /* + * xforms that provide a reinit method perform all IV + * handling themselves. + */ exf->reinit(sw->sw_kschedule, iv); + } - if (flags & CRYPTO_F_IMBUF) { - struct mbuf *m = (struct mbuf *) buf; + count = crd->crd_skip; + ind = cuio_getptr(uio, count, &k); + if (ind == -1) { + error = EINVAL; + goto out; + } - /* Find beginning of data */ - m = m_getptr(m, crd->crd_skip, &k); - if (m == NULL) - return EINVAL; + i = crd->crd_len; - i = crd->crd_len; + while (i > 0) { + /* + * If there's insufficient data at the end of + * an iovec, we have to do some copying. + */ + if (uio->uio_iov[ind].iov_len < k + blks && + uio->uio_iov[ind].iov_len != k) { + cuio_copydata(uio, count, blks, blk); - while (i > 0) { - /* - * If there's insufficient data at the end of - * an mbuf, we have to do some copying. - */ - if (m->m_len < k + blks && m->m_len != k) { - m_copydata(m, k, blks, blk); - - /* Actual encryption/decryption */ - if (exf->reinit) { - if (crd->crd_flags & CRD_F_ENCRYPT) { - exf->encrypt(sw->sw_kschedule, - blk); - } else { - exf->decrypt(sw->sw_kschedule, - blk); - } - } else if (crd->crd_flags & CRD_F_ENCRYPT) { - /* XOR with previous block */ - for (j = 0; j < blks; j++) - blk[j] ^= ivp[j]; - - exf->encrypt(sw->sw_kschedule, blk); - - /* - * Keep encrypted block for XOR'ing - * with next block - */ - bcopy(blk, iv, blks); - ivp = iv; - } else { /* decrypt */ - /* - * Keep encrypted block for XOR'ing - * with next block - */ - if (ivp == iv) - bcopy(blk, piv, blks); - else - bcopy(blk, iv, blks); - - exf->decrypt(sw->sw_kschedule, blk); - - /* XOR with previous block */ - for (j = 0; j < blks; j++) - blk[j] ^= ivp[j]; - - if (ivp == iv) - bcopy(piv, iv, blks); - else - ivp = iv; + /* Actual encryption/decryption */ + if (exf->reinit) { + if (crd->crd_flags & CRD_F_ENCRYPT) { + exf->encrypt(sw->sw_kschedule, + blk); + } else { + exf->decrypt(sw->sw_kschedule, + blk); } + } else if (crd->crd_flags & CRD_F_ENCRYPT) { + /* XOR with previous block */ + for (j = 0; j < blks; j++) + blk[j] ^= ivp[j]; - /* Copy back decrypted block */ - m_copyback(m, k, blks, blk); + exf->encrypt(sw->sw_kschedule, blk); - /* Advance pointer */ - m = m_getptr(m, k + blks, &k); - if (m == NULL) - return EINVAL; + /* + * Keep encrypted block for XOR'ing + * with next block + */ + bcopy(blk, iv, blks); + ivp = iv; + } else { /* decrypt */ + /* + * Keep encrypted block for XOR'ing + * with next block + */ + nivp = (ivp == iv) ? iv2 : iv; + bcopy(blk, nivp, blks); - i -= blks; + exf->decrypt(sw->sw_kschedule, blk); - /* Could be done... */ - if (i == 0) - break; - } + /* XOR with previous block */ + for (j = 0; j < blks; j++) + blk[j] ^= ivp[j]; - /* Skip possibly empty mbufs */ - if (k == m->m_len) { - for (m = m->m_next; m && m->m_len == 0; - m = m->m_next) - ; - k = 0; + ivp = nivp; } - /* Sanity check */ - if (m == NULL) - return EINVAL; + /* Copy back decrypted block */ + cuio_copyback(uio, count, blks, blk); - /* - * Warning: idat may point to garbage here, but - * we only use it in the while() loop, only if - * there are indeed enough data. - */ - idat = mtod(m, unsigned char *) + k; + count += blks; - while (m->m_len >= k + blks && i > 0) { - if (exf->reinit) { - if (crd->crd_flags & CRD_F_ENCRYPT) { - exf->encrypt(sw->sw_kschedule, - idat); - } else { - exf->decrypt(sw->sw_kschedule, - idat); - } - } else if (crd->crd_flags & CRD_F_ENCRYPT) { - /* XOR with previous block/IV */ - for (j = 0; j < blks; j++) - idat[j] ^= ivp[j]; + /* Advance pointer */ + ind = cuio_getptr(uio, count, &k); + if (ind == -1) { + error = EINVAL; + goto out; + } - exf->encrypt(sw->sw_kschedule, idat); - ivp = idat; - } else { /* decrypt */ - /* - * Keep encrypted block to be used - * in next block's processing. - */ - if (ivp == iv) - bcopy(idat, piv, blks); - else - bcopy(idat, iv, blks); + i -= blks; - exf->decrypt(sw->sw_kschedule, idat); - - /* XOR with previous block/IV */ - for (j = 0; j < blks; j++) - idat[j] ^= ivp[j]; - - if (ivp == iv) - bcopy(piv, iv, blks); - else - ivp = iv; - } - - idat += blks; - k += blks; - i -= blks; - } + /* Could be done... */ + if (i == 0) + break; } - return 0; /* Done with mbuf encryption/decryption */ - } else if (flags & CRYPTO_F_IOV) { - struct uio *uio = (struct uio *) buf; - struct iovec *iov; + /* + * Warning: idat may point to garbage here, but + * we only use it in the while() loop, only if + * there are indeed enough data. + */ + idat = (char *)uio->uio_iov[ind].iov_base + k; - /* Find beginning of data */ - iov = cuio_getptr(uio, crd->crd_skip, &k); - if (iov == NULL) - return EINVAL; - - i = crd->crd_len; - - while (i > 0) { - /* - * If there's insufficient data at the end of - * an iovec, we have to do some copying. - */ - if (iov->iov_len < k + blks && iov->iov_len != k) { - cuio_copydata(uio, k, blks, blk); - - /* Actual encryption/decryption */ - if (exf->reinit) { - if (crd->crd_flags & CRD_F_ENCRYPT) { - exf->encrypt(sw->sw_kschedule, - blk); - } else { - exf->decrypt(sw->sw_kschedule, - blk); - } - } else if (crd->crd_flags & CRD_F_ENCRYPT) { - /* XOR with previous block */ - for (j = 0; j < blks; j++) - blk[j] ^= ivp[j]; - - exf->encrypt(sw->sw_kschedule, blk); - - /* - * Keep encrypted block for XOR'ing - * with next block - */ - bcopy(blk, iv, blks); - ivp = iv; - } else { /* decrypt */ - /* - * Keep encrypted block for XOR'ing - * with next block - */ - if (ivp == iv) - bcopy(blk, piv, blks); - else - bcopy(blk, iv, blks); - - exf->decrypt(sw->sw_kschedule, blk); - - /* XOR with previous block */ - for (j = 0; j < blks; j++) - blk[j] ^= ivp[j]; - - if (ivp == iv) - bcopy(piv, iv, blks); - else - ivp = iv; + while (uio->uio_iov[ind].iov_len >= k + blks && i > 0) { + if (exf->reinit) { + if (crd->crd_flags & CRD_F_ENCRYPT) { + exf->encrypt(sw->sw_kschedule, + idat); + } else { + exf->decrypt(sw->sw_kschedule, + idat); } + } else if (crd->crd_flags & CRD_F_ENCRYPT) { + /* XOR with previous block/IV */ + for (j = 0; j < blks; j++) + idat[j] ^= ivp[j]; - /* Copy back decrypted block */ - cuio_copyback(uio, k, blks, blk); + exf->encrypt(sw->sw_kschedule, idat); + ivp = idat; + } else { /* decrypt */ + /* + * Keep encrypted block to be used + * in next block's processing. + */ + nivp = (ivp == iv) ? iv2 : iv; + bcopy(idat, nivp, blks); - /* Advance pointer */ - iov = cuio_getptr(uio, k + blks, &k); - if (iov == NULL) - return EINVAL; + exf->decrypt(sw->sw_kschedule, idat); - i -= blks; + /* XOR with previous block/IV */ + for (j = 0; j < blks; j++) + idat[j] ^= ivp[j]; - /* Could be done... */ - if (i == 0) - break; + ivp = nivp; } - /* - * Warning: idat may point to garbage here, but - * we only use it in the while() loop, only if - * there are indeed enough data. - */ - idat = (char *)iov->iov_base + k; - - while (iov->iov_len >= k + blks && i > 0) { - if (exf->reinit) { - if (crd->crd_flags & CRD_F_ENCRYPT) { - exf->encrypt(sw->sw_kschedule, - idat); - } else { - exf->decrypt(sw->sw_kschedule, - idat); - } - } else if (crd->crd_flags & CRD_F_ENCRYPT) { - /* XOR with previous block/IV */ - for (j = 0; j < blks; j++) - idat[j] ^= ivp[j]; - - exf->encrypt(sw->sw_kschedule, idat); - ivp = idat; - } else { /* decrypt */ - /* - * Keep encrypted block to be used - * in next block's processing. - */ - if (ivp == iv) - bcopy(idat, piv, blks); - else - bcopy(idat, iv, blks); - - exf->decrypt(sw->sw_kschedule, idat); - - /* XOR with previous block/IV */ - for (j = 0; j < blks; j++) - idat[j] ^= ivp[j]; - - if (ivp == iv) - bcopy(piv, iv, blks); - else - ivp = iv; - } - - idat += blks; - k += blks; - i -= blks; - } - if (k == iov->iov_len) { - iov++; - k = 0; - } + idat += blks; + count += blks; + k += blks; + i -= blks; } - return 0; /* Done with iovec encryption/decryption */ - } else { /* contiguous buffer */ - if (exf->reinit) { - for (i = crd->crd_skip; - i < crd->crd_skip + crd->crd_len; i += blks) { - if (crd->crd_flags & CRD_F_ENCRYPT) - exf->encrypt(sw->sw_kschedule, buf + i); - else - exf->decrypt(sw->sw_kschedule, buf + i); + /* + * Advance to the next iov if the end of the current iov + * is aligned with the end of a cipher block. + * Note that the code is equivalent to calling: + * ind = cuio_getptr(uio, count, &k); + */ + if (i > 0 && k == uio->uio_iov[ind].iov_len) { + k = 0; + ind++; + if (ind >= uio->uio_iovcnt) { + error = EINVAL; + goto out; } - } else if (crd->crd_flags & CRD_F_ENCRYPT) { - for (i = crd->crd_skip; - i < crd->crd_skip + crd->crd_len; i += blks) { - /* XOR with the IV/previous block, as appropriate. */ - if (i == crd->crd_skip) - for (k = 0; k < blks; k++) - buf[i + k] ^= ivp[k]; - else - for (k = 0; k < blks; k++) - buf[i + k] ^= buf[i + k - blks]; - exf->encrypt(sw->sw_kschedule, buf + i); - } - } else { /* Decrypt */ - /* - * Start at the end, so we don't need to keep the encrypted - * block as the IV for the next block. - */ - for (i = crd->crd_skip + crd->crd_len - blks; - i >= crd->crd_skip; i -= blks) { - exf->decrypt(sw->sw_kschedule, buf + i); - - /* XOR with the IV/previous block, as appropriate */ - if (i == crd->crd_skip) - for (k = 0; k < blks; k++) - buf[i + k] ^= ivp[k]; - else - for (k = 0; k < blks; k++) - buf[i + k] ^= buf[i + k - blks]; - } } - - return 0; /* Done with contiguous buffer encryption/decryption */ } - /* Unreachable */ - return EINVAL; +out: + if (iovalloc) + free(iov, M_CRYPTO_DATA); + + return (error); } static void swcr_authprepare(struct auth_hash *axf, struct swcr_data *sw, u_char *key, int klen) { int k; klen /= 8; switch (axf->type) { case CRYPTO_MD5_HMAC: case CRYPTO_SHA1_HMAC: case CRYPTO_SHA2_256_HMAC: case CRYPTO_SHA2_384_HMAC: case CRYPTO_SHA2_512_HMAC: case CRYPTO_NULL_HMAC: case CRYPTO_RIPEMD160_HMAC: for (k = 0; k < klen; k++) key[k] ^= HMAC_IPAD_VAL; axf->Init(sw->sw_ictx); axf->Update(sw->sw_ictx, key, klen); axf->Update(sw->sw_ictx, hmac_ipad_buffer, axf->blocksize - klen); for (k = 0; k < klen; k++) key[k] ^= (HMAC_IPAD_VAL ^ HMAC_OPAD_VAL); axf->Init(sw->sw_octx); axf->Update(sw->sw_octx, key, klen); axf->Update(sw->sw_octx, hmac_opad_buffer, axf->blocksize - klen); for (k = 0; k < klen; k++) key[k] ^= HMAC_OPAD_VAL; break; case CRYPTO_MD5_KPDK: case CRYPTO_SHA1_KPDK: { /* * We need a buffer that can hold an md5 and a sha1 result * just to throw it away. * What we do here is the initial part of: * ALGO( key, keyfill, .. ) * adding the key to sw_ictx and abusing Final() to get the * "keyfill" padding. * In addition we abuse the sw_octx to save the key to have * it to be able to append it at the end in swcr_authcompute(). */ u_char buf[SHA1_RESULTLEN]; sw->sw_klen = klen; bcopy(key, sw->sw_octx, klen); axf->Init(sw->sw_ictx); axf->Update(sw->sw_ictx, key, klen); axf->Final(buf, sw->sw_ictx); break; } default: printf("%s: CRD_F_KEY_EXPLICIT flag given, but algorithm %d " "doesn't use keys.\n", __func__, axf->type); } } /* * Compute keyed-hash authenticator. */ static int swcr_authcompute(struct cryptodesc *crd, struct swcr_data *sw, caddr_t buf, int flags) { unsigned char aalg[HASH_MAX_LEN]; struct auth_hash *axf; union authctx ctx; int err; if (sw->sw_ictx == 0) return EINVAL; axf = sw->sw_axf; if (crd->crd_flags & CRD_F_KEY_EXPLICIT) swcr_authprepare(axf, sw, crd->crd_key, crd->crd_klen); bcopy(sw->sw_ictx, &ctx, axf->ctxsize); err = crypto_apply(flags, buf, crd->crd_skip, crd->crd_len, (int (*)(void *, void *, unsigned int))axf->Update, (caddr_t)&ctx); if (err) return err; switch (sw->sw_alg) { case CRYPTO_MD5_HMAC: case CRYPTO_SHA1_HMAC: case CRYPTO_SHA2_256_HMAC: case CRYPTO_SHA2_384_HMAC: case CRYPTO_SHA2_512_HMAC: case CRYPTO_RIPEMD160_HMAC: if (sw->sw_octx == NULL) return EINVAL; axf->Final(aalg, &ctx); bcopy(sw->sw_octx, &ctx, axf->ctxsize); axf->Update(&ctx, aalg, axf->hashsize); axf->Final(aalg, &ctx); break; case CRYPTO_MD5_KPDK: case CRYPTO_SHA1_KPDK: /* If we have no key saved, return error. */ if (sw->sw_octx == NULL) return EINVAL; /* * Add the trailing copy of the key (see comment in * swcr_authprepare()) after the data: * ALGO( .., key, algofill ) * and let Final() do the proper, natural "algofill" * padding. */ axf->Update(&ctx, sw->sw_octx, sw->sw_klen); axf->Final(aalg, &ctx); break; case CRYPTO_NULL_HMAC: axf->Final(aalg, &ctx); break; } /* Inject the authentication data */ crypto_copyback(flags, buf, crd->crd_inject, sw->sw_mlen == 0 ? axf->hashsize : sw->sw_mlen, aalg); return 0; } +CTASSERT(INT_MAX <= (1ll<<39) - 256); /* GCM: plain text < 2^39-256 */ +CTASSERT(INT_MAX <= (uint64_t)-1); /* GCM: associated data <= 2^64-1 */ + /* + * Apply a combined encryption-authentication transformation + */ +static int +swcr_authenc(struct cryptop *crp) +{ + uint32_t blkbuf[howmany(EALG_MAX_BLOCK_LEN, sizeof(uint32_t))]; + u_char *blk = (u_char *)blkbuf; + u_char aalg[AALG_MAX_RESULT_LEN]; + u_char uaalg[AALG_MAX_RESULT_LEN]; + u_char iv[EALG_MAX_BLOCK_LEN]; + union authctx ctx; + struct cryptodesc *crd, *crda = NULL, *crde = NULL; + struct swcr_data *sw, *swa, *swe = NULL; + struct auth_hash *axf = NULL; + struct enc_xform *exf = NULL; + caddr_t buf = (caddr_t)crp->crp_buf; + uint32_t *blkp; + int aadlen, blksz, i, ivlen, len, iskip, oskip, r; + + ivlen = blksz = iskip = oskip = 0; + + for (crd = crp->crp_desc; crd; crd = crd->crd_next) { + for (sw = swcr_sessions[crp->crp_sid & 0xffffffff]; + sw && sw->sw_alg != crd->crd_alg; + sw = sw->sw_next) + ; + if (sw == NULL) + return (EINVAL); + + switch (sw->sw_alg) { + case CRYPTO_AES_NIST_GCM_16: + case CRYPTO_AES_NIST_GMAC: + swe = sw; + crde = crd; + exf = swe->sw_exf; + ivlen = 12; + break; + case CRYPTO_AES_128_NIST_GMAC: + case CRYPTO_AES_192_NIST_GMAC: + case CRYPTO_AES_256_NIST_GMAC: + swa = sw; + crda = crd; + axf = swa->sw_axf; + if (swa->sw_ictx == 0) + return (EINVAL); + bcopy(swa->sw_ictx, &ctx, axf->ctxsize); + blksz = axf->blocksize; + break; + default: + return (EINVAL); + } + } + if (crde == NULL || crda == NULL) + return (EINVAL); + + if (crde->crd_alg == CRYPTO_AES_NIST_GCM_16 && + (crde->crd_flags & CRD_F_IV_EXPLICIT) == 0) + return (EINVAL); + + if (crde->crd_klen != crda->crd_klen) + return (EINVAL); + + /* Initialize the IV */ + if (crde->crd_flags & CRD_F_ENCRYPT) { + /* IV explicitly provided ? */ + if (crde->crd_flags & CRD_F_IV_EXPLICIT) + bcopy(crde->crd_iv, iv, ivlen); + else + arc4rand(iv, ivlen, 0); + + /* Do we need to write the IV */ + if (!(crde->crd_flags & CRD_F_IV_PRESENT)) + crypto_copyback(crp->crp_flags, buf, crde->crd_inject, + ivlen, iv); + + } else { /* Decryption */ + /* IV explicitly provided ? */ + if (crde->crd_flags & CRD_F_IV_EXPLICIT) + bcopy(crde->crd_iv, iv, ivlen); + else { + /* Get IV off buf */ + crypto_copydata(crp->crp_flags, buf, crde->crd_inject, + ivlen, iv); + } + } + + /* Supply MAC with IV */ + if (axf->Reinit) + axf->Reinit(&ctx, iv, ivlen); + + /* Supply MAC with AAD */ + aadlen = crda->crd_len; + + for (i = iskip; i < crda->crd_len; i += blksz) { + len = MIN(crda->crd_len - i, blksz - oskip); + crypto_copydata(crp->crp_flags, buf, crda->crd_skip + i, len, + blk + oskip); + bzero(blk + len + oskip, blksz - len - oskip); + axf->Update(&ctx, blk, blksz); + oskip = 0; /* reset initial output offset */ + } + + if (exf->reinit) + exf->reinit(swe->sw_kschedule, iv); + + /* Do encryption/decryption with MAC */ + for (i = 0; i < crde->crd_len; i += blksz) { + len = MIN(crde->crd_len - i, blksz); + if (len < blksz) + bzero(blk, blksz); + crypto_copydata(crp->crp_flags, buf, crde->crd_skip + i, len, + blk); + if (crde->crd_flags & CRD_F_ENCRYPT) { + exf->encrypt(swe->sw_kschedule, blk); + axf->Update(&ctx, blk, len); + crypto_copyback(crp->crp_flags, buf, + crde->crd_skip + i, len, blk); + } else { + axf->Update(&ctx, blk, len); + } + } + + /* Do any required special finalization */ + switch (crda->crd_alg) { + case CRYPTO_AES_128_NIST_GMAC: + case CRYPTO_AES_192_NIST_GMAC: + case CRYPTO_AES_256_NIST_GMAC: + /* length block */ + bzero(blk, blksz); + blkp = (uint32_t *)blk + 1; + *blkp = htobe32(aadlen * 8); + blkp = (uint32_t *)blk + 3; + *blkp = htobe32(crde->crd_len * 8); + axf->Update(&ctx, blk, blksz); + break; + } + + /* Finalize MAC */ + axf->Final(aalg, &ctx); + + /* Validate tag */ + if (!(crde->crd_flags & CRD_F_ENCRYPT)) { + crypto_copydata(crp->crp_flags, buf, crda->crd_inject, + axf->hashsize, uaalg); + + r = timingsafe_bcmp(aalg, uaalg, axf->hashsize); + if (r == 0) { + /* tag matches, decrypt data */ + for (i = 0; i < crde->crd_len; i += blksz) { + len = MIN(crde->crd_len - i, blksz); + if (len < blksz) + bzero(blk, blksz); + crypto_copydata(crp->crp_flags, buf, + crde->crd_skip + i, len, blk); + if (!(crde->crd_flags & CRD_F_ENCRYPT)) { + exf->decrypt(swe->sw_kschedule, blk); + } + crypto_copyback(crp->crp_flags, buf, + crde->crd_skip + i, len, blk); + } + } else + return (EBADMSG); + } else { + /* Inject the authentication data */ + crypto_copyback(crp->crp_flags, buf, crda->crd_inject, + axf->hashsize, aalg); + } + + return (0); +} + +/* * Apply a compression/decompression algorithm */ static int swcr_compdec(struct cryptodesc *crd, struct swcr_data *sw, caddr_t buf, int flags) { u_int8_t *data, *out; struct comp_algo *cxf; int adj; u_int32_t result; cxf = sw->sw_cxf; /* We must handle the whole buffer of data in one time * then if there is not all the data in the mbuf, we must * copy in a buffer. */ data = malloc(crd->crd_len, M_CRYPTO_DATA, M_NOWAIT); if (data == NULL) return (EINVAL); crypto_copydata(flags, buf, crd->crd_skip, crd->crd_len, data); if (crd->crd_flags & CRD_F_COMP) result = cxf->compress(data, crd->crd_len, &out); else result = cxf->decompress(data, crd->crd_len, &out); free(data, M_CRYPTO_DATA); if (result == 0) return EINVAL; /* Copy back the (de)compressed data. m_copyback is * extending the mbuf as necessary. */ sw->sw_size = result; /* Check the compressed size when doing compression */ if (crd->crd_flags & CRD_F_COMP) { if (result >= crd->crd_len) { /* Compression was useless, we lost time */ free(out, M_CRYPTO_DATA); return 0; } } crypto_copyback(flags, buf, crd->crd_skip, result, out); if (result < crd->crd_len) { adj = result - crd->crd_len; if (flags & CRYPTO_F_IMBUF) { adj = result - crd->crd_len; m_adj((struct mbuf *)buf, adj); } else if (flags & CRYPTO_F_IOV) { struct uio *uio = (struct uio *)buf; int ind; adj = crd->crd_len - result; ind = uio->uio_iovcnt - 1; while (adj > 0 && ind >= 0) { if (adj < uio->uio_iov[ind].iov_len) { uio->uio_iov[ind].iov_len -= adj; break; } adj -= uio->uio_iov[ind].iov_len; uio->uio_iov[ind].iov_len = 0; ind--; uio->uio_iovcnt--; } } } free(out, M_CRYPTO_DATA); return 0; } /* * Generate a new software session. */ static int swcr_newsession(device_t dev, u_int32_t *sid, struct cryptoini *cri) { struct swcr_data **swd; struct auth_hash *axf; struct enc_xform *txf; struct comp_algo *cxf; u_int32_t i; int error; if (sid == NULL || cri == NULL) return EINVAL; rw_wlock(&swcr_sessions_lock); if (swcr_sessions) { for (i = 1; i < swcr_sesnum; i++) if (swcr_sessions[i] == NULL) break; } else i = 1; /* NB: to silence compiler warning */ if (swcr_sessions == NULL || i == swcr_sesnum) { if (swcr_sessions == NULL) { i = 1; /* We leave swcr_sessions[0] empty */ swcr_sesnum = CRYPTO_SW_SESSIONS; } else swcr_sesnum *= 2; swd = malloc(swcr_sesnum * sizeof(struct swcr_data *), M_CRYPTO_DATA, M_NOWAIT|M_ZERO); if (swd == NULL) { /* Reset session number */ if (swcr_sesnum == CRYPTO_SW_SESSIONS) swcr_sesnum = 0; else swcr_sesnum /= 2; rw_wunlock(&swcr_sessions_lock); return ENOBUFS; } /* Copy existing sessions */ if (swcr_sessions != NULL) { bcopy(swcr_sessions, swd, (swcr_sesnum / 2) * sizeof(struct swcr_data *)); free(swcr_sessions, M_CRYPTO_DATA); } swcr_sessions = swd; } rw_downgrade(&swcr_sessions_lock); swd = &swcr_sessions[i]; *sid = i; while (cri) { *swd = malloc(sizeof(struct swcr_data), M_CRYPTO_DATA, M_NOWAIT|M_ZERO); if (*swd == NULL) { swcr_freesession_locked(dev, i); rw_runlock(&swcr_sessions_lock); return ENOBUFS; } switch (cri->cri_alg) { case CRYPTO_DES_CBC: txf = &enc_xform_des; goto enccommon; case CRYPTO_3DES_CBC: txf = &enc_xform_3des; goto enccommon; case CRYPTO_BLF_CBC: txf = &enc_xform_blf; goto enccommon; case CRYPTO_CAST_CBC: txf = &enc_xform_cast5; goto enccommon; case CRYPTO_SKIPJACK_CBC: txf = &enc_xform_skipjack; goto enccommon; case CRYPTO_RIJNDAEL128_CBC: txf = &enc_xform_rijndael128; goto enccommon; case CRYPTO_AES_XTS: txf = &enc_xform_aes_xts; goto enccommon; + case CRYPTO_AES_ICM: + txf = &enc_xform_aes_icm; + goto enccommon; + case CRYPTO_AES_NIST_GCM_16: + txf = &enc_xform_aes_nist_gcm; + goto enccommon; + case CRYPTO_AES_NIST_GMAC: + txf = &enc_xform_aes_nist_gmac; + (*swd)->sw_exf = txf; + break; case CRYPTO_CAMELLIA_CBC: txf = &enc_xform_camellia; goto enccommon; case CRYPTO_NULL_CBC: txf = &enc_xform_null; goto enccommon; enccommon: if (cri->cri_key != NULL) { error = txf->setkey(&((*swd)->sw_kschedule), cri->cri_key, cri->cri_klen / 8); if (error) { swcr_freesession_locked(dev, i); rw_runlock(&swcr_sessions_lock); return error; } } (*swd)->sw_exf = txf; break; case CRYPTO_MD5_HMAC: axf = &auth_hash_hmac_md5; goto authcommon; case CRYPTO_SHA1_HMAC: axf = &auth_hash_hmac_sha1; goto authcommon; case CRYPTO_SHA2_256_HMAC: axf = &auth_hash_hmac_sha2_256; goto authcommon; case CRYPTO_SHA2_384_HMAC: axf = &auth_hash_hmac_sha2_384; goto authcommon; case CRYPTO_SHA2_512_HMAC: axf = &auth_hash_hmac_sha2_512; goto authcommon; case CRYPTO_NULL_HMAC: axf = &auth_hash_null; goto authcommon; case CRYPTO_RIPEMD160_HMAC: axf = &auth_hash_hmac_ripemd_160; authcommon: (*swd)->sw_ictx = malloc(axf->ctxsize, M_CRYPTO_DATA, M_NOWAIT); if ((*swd)->sw_ictx == NULL) { swcr_freesession_locked(dev, i); rw_runlock(&swcr_sessions_lock); return ENOBUFS; } (*swd)->sw_octx = malloc(axf->ctxsize, M_CRYPTO_DATA, M_NOWAIT); if ((*swd)->sw_octx == NULL) { swcr_freesession_locked(dev, i); rw_runlock(&swcr_sessions_lock); return ENOBUFS; } if (cri->cri_key != NULL) { swcr_authprepare(axf, *swd, cri->cri_key, cri->cri_klen); } (*swd)->sw_mlen = cri->cri_mlen; (*swd)->sw_axf = axf; break; case CRYPTO_MD5_KPDK: axf = &auth_hash_key_md5; goto auth2common; case CRYPTO_SHA1_KPDK: axf = &auth_hash_key_sha1; auth2common: (*swd)->sw_ictx = malloc(axf->ctxsize, M_CRYPTO_DATA, M_NOWAIT); if ((*swd)->sw_ictx == NULL) { swcr_freesession_locked(dev, i); rw_runlock(&swcr_sessions_lock); return ENOBUFS; } (*swd)->sw_octx = malloc(cri->cri_klen / 8, M_CRYPTO_DATA, M_NOWAIT); if ((*swd)->sw_octx == NULL) { swcr_freesession_locked(dev, i); rw_runlock(&swcr_sessions_lock); return ENOBUFS; } /* Store the key so we can "append" it to the payload */ if (cri->cri_key != NULL) { swcr_authprepare(axf, *swd, cri->cri_key, cri->cri_klen); } (*swd)->sw_mlen = cri->cri_mlen; (*swd)->sw_axf = axf; break; #ifdef notdef case CRYPTO_MD5: axf = &auth_hash_md5; goto auth3common; case CRYPTO_SHA1: axf = &auth_hash_sha1; auth3common: (*swd)->sw_ictx = malloc(axf->ctxsize, M_CRYPTO_DATA, M_NOWAIT); if ((*swd)->sw_ictx == NULL) { swcr_freesession_locked(dev, i); rw_runlock(&swcr_sessions_lock); return ENOBUFS; } axf->Init((*swd)->sw_ictx); (*swd)->sw_mlen = cri->cri_mlen; (*swd)->sw_axf = axf; break; #endif + + case CRYPTO_AES_128_NIST_GMAC: + axf = &auth_hash_nist_gmac_aes_128; + goto auth4common; + + case CRYPTO_AES_192_NIST_GMAC: + axf = &auth_hash_nist_gmac_aes_192; + goto auth4common; + + case CRYPTO_AES_256_NIST_GMAC: + axf = &auth_hash_nist_gmac_aes_256; + auth4common: + (*swd)->sw_ictx = malloc(axf->ctxsize, M_CRYPTO_DATA, + M_NOWAIT); + if ((*swd)->sw_ictx == NULL) { + swcr_freesession_locked(dev, i); + rw_runlock(&swcr_sessions_lock); + return ENOBUFS; + } + axf->Init((*swd)->sw_ictx); + axf->Setkey((*swd)->sw_ictx, cri->cri_key, + cri->cri_klen / 8); + (*swd)->sw_axf = axf; + break; + case CRYPTO_DEFLATE_COMP: cxf = &comp_algo_deflate; (*swd)->sw_cxf = cxf; break; default: swcr_freesession_locked(dev, i); rw_runlock(&swcr_sessions_lock); return EINVAL; } (*swd)->sw_alg = cri->cri_alg; cri = cri->cri_next; swd = &((*swd)->sw_next); } rw_runlock(&swcr_sessions_lock); return 0; } static int swcr_freesession(device_t dev, u_int64_t tid) { int error; rw_rlock(&swcr_sessions_lock); error = swcr_freesession_locked(dev, tid); rw_runlock(&swcr_sessions_lock); return error; } /* * Free a session. */ static int swcr_freesession_locked(device_t dev, u_int64_t tid) { struct swcr_data *swd; struct enc_xform *txf; struct auth_hash *axf; struct comp_algo *cxf; u_int32_t sid = CRYPTO_SESID2LID(tid); if (sid > swcr_sesnum || swcr_sessions == NULL || swcr_sessions[sid] == NULL) return EINVAL; /* Silently accept and return */ if (sid == 0) return 0; while ((swd = swcr_sessions[sid]) != NULL) { swcr_sessions[sid] = swd->sw_next; switch (swd->sw_alg) { case CRYPTO_DES_CBC: case CRYPTO_3DES_CBC: case CRYPTO_BLF_CBC: case CRYPTO_CAST_CBC: case CRYPTO_SKIPJACK_CBC: case CRYPTO_RIJNDAEL128_CBC: case CRYPTO_AES_XTS: + case CRYPTO_AES_ICM: + case CRYPTO_AES_NIST_GCM_16: + case CRYPTO_AES_NIST_GMAC: case CRYPTO_CAMELLIA_CBC: case CRYPTO_NULL_CBC: txf = swd->sw_exf; if (swd->sw_kschedule) txf->zerokey(&(swd->sw_kschedule)); break; case CRYPTO_MD5_HMAC: case CRYPTO_SHA1_HMAC: case CRYPTO_SHA2_256_HMAC: case CRYPTO_SHA2_384_HMAC: case CRYPTO_SHA2_512_HMAC: case CRYPTO_RIPEMD160_HMAC: case CRYPTO_NULL_HMAC: axf = swd->sw_axf; if (swd->sw_ictx) { bzero(swd->sw_ictx, axf->ctxsize); free(swd->sw_ictx, M_CRYPTO_DATA); } if (swd->sw_octx) { bzero(swd->sw_octx, axf->ctxsize); free(swd->sw_octx, M_CRYPTO_DATA); } break; case CRYPTO_MD5_KPDK: case CRYPTO_SHA1_KPDK: axf = swd->sw_axf; if (swd->sw_ictx) { bzero(swd->sw_ictx, axf->ctxsize); free(swd->sw_ictx, M_CRYPTO_DATA); } if (swd->sw_octx) { bzero(swd->sw_octx, swd->sw_klen); free(swd->sw_octx, M_CRYPTO_DATA); } break; case CRYPTO_MD5: case CRYPTO_SHA1: axf = swd->sw_axf; if (swd->sw_ictx) free(swd->sw_ictx, M_CRYPTO_DATA); break; case CRYPTO_DEFLATE_COMP: cxf = swd->sw_cxf; break; } free(swd, M_CRYPTO_DATA); } return 0; } /* * Process a software request. */ static int swcr_process(device_t dev, struct cryptop *crp, int hint) { struct cryptodesc *crd; struct swcr_data *sw; u_int32_t lid; /* Sanity check */ if (crp == NULL) return EINVAL; if (crp->crp_desc == NULL || crp->crp_buf == NULL) { crp->crp_etype = EINVAL; goto done; } lid = CRYPTO_SESID2LID(crp->crp_sid); rw_rlock(&swcr_sessions_lock); if (swcr_sessions == NULL || lid >= swcr_sesnum || lid == 0 || swcr_sessions[lid] == NULL) { rw_runlock(&swcr_sessions_lock); crp->crp_etype = ENOENT; goto done; } rw_runlock(&swcr_sessions_lock); /* Go through crypto descriptors, processing as we go */ for (crd = crp->crp_desc; crd; crd = crd->crd_next) { /* * Find the crypto context. * * XXX Note that the logic here prevents us from having * XXX the same algorithm multiple times in a session * XXX (or rather, we can but it won't give us the right * XXX results). To do that, we'd need some way of differentiating * XXX between the various instances of an algorithm (so we can * XXX locate the correct crypto context). */ rw_rlock(&swcr_sessions_lock); if (swcr_sessions == NULL) { rw_runlock(&swcr_sessions_lock); crp->crp_etype = ENOENT; goto done; } for (sw = swcr_sessions[lid]; sw && sw->sw_alg != crd->crd_alg; sw = sw->sw_next) ; rw_runlock(&swcr_sessions_lock); /* No such context ? */ if (sw == NULL) { crp->crp_etype = EINVAL; goto done; } switch (sw->sw_alg) { case CRYPTO_DES_CBC: case CRYPTO_3DES_CBC: case CRYPTO_BLF_CBC: case CRYPTO_CAST_CBC: case CRYPTO_SKIPJACK_CBC: case CRYPTO_RIJNDAEL128_CBC: case CRYPTO_AES_XTS: + case CRYPTO_AES_ICM: case CRYPTO_CAMELLIA_CBC: if ((crp->crp_etype = swcr_encdec(crd, sw, crp->crp_buf, crp->crp_flags)) != 0) goto done; break; case CRYPTO_NULL_CBC: crp->crp_etype = 0; break; case CRYPTO_MD5_HMAC: case CRYPTO_SHA1_HMAC: case CRYPTO_SHA2_256_HMAC: case CRYPTO_SHA2_384_HMAC: case CRYPTO_SHA2_512_HMAC: case CRYPTO_RIPEMD160_HMAC: case CRYPTO_NULL_HMAC: case CRYPTO_MD5_KPDK: case CRYPTO_SHA1_KPDK: case CRYPTO_MD5: case CRYPTO_SHA1: if ((crp->crp_etype = swcr_authcompute(crd, sw, crp->crp_buf, crp->crp_flags)) != 0) goto done; break; + case CRYPTO_AES_NIST_GCM_16: + case CRYPTO_AES_NIST_GMAC: + case CRYPTO_AES_128_NIST_GMAC: + case CRYPTO_AES_192_NIST_GMAC: + case CRYPTO_AES_256_NIST_GMAC: + crp->crp_etype = swcr_authenc(crp); + goto done; + case CRYPTO_DEFLATE_COMP: if ((crp->crp_etype = swcr_compdec(crd, sw, crp->crp_buf, crp->crp_flags)) != 0) goto done; else crp->crp_olen = (int)sw->sw_size; break; default: /* Unknown/unsupported algorithm */ crp->crp_etype = EINVAL; goto done; } } done: crypto_done(crp); return 0; } static void swcr_identify(driver_t *drv, device_t parent) { /* NB: order 10 is so we get attached after h/w devices */ if (device_find_child(parent, "cryptosoft", -1) == NULL && BUS_ADD_CHILD(parent, 10, "cryptosoft", 0) == 0) panic("cryptosoft: could not attach"); } static int swcr_probe(device_t dev) { device_set_desc(dev, "software crypto"); return (BUS_PROBE_NOWILDCARD); } static int swcr_attach(device_t dev) { rw_init(&swcr_sessions_lock, "swcr_sessions_lock"); memset(hmac_ipad_buffer, HMAC_IPAD_VAL, HMAC_MAX_BLOCK_LEN); memset(hmac_opad_buffer, HMAC_OPAD_VAL, HMAC_MAX_BLOCK_LEN); swcr_id = crypto_get_driverid(dev, CRYPTOCAP_F_SOFTWARE | CRYPTOCAP_F_SYNC); if (swcr_id < 0) { device_printf(dev, "cannot initialize!"); return ENOMEM; } #define REGISTER(alg) \ crypto_register(swcr_id, alg, 0,0) REGISTER(CRYPTO_DES_CBC); REGISTER(CRYPTO_3DES_CBC); REGISTER(CRYPTO_BLF_CBC); REGISTER(CRYPTO_CAST_CBC); REGISTER(CRYPTO_SKIPJACK_CBC); REGISTER(CRYPTO_NULL_CBC); REGISTER(CRYPTO_MD5_HMAC); REGISTER(CRYPTO_SHA1_HMAC); REGISTER(CRYPTO_SHA2_256_HMAC); REGISTER(CRYPTO_SHA2_384_HMAC); REGISTER(CRYPTO_SHA2_512_HMAC); REGISTER(CRYPTO_RIPEMD160_HMAC); REGISTER(CRYPTO_NULL_HMAC); REGISTER(CRYPTO_MD5_KPDK); REGISTER(CRYPTO_SHA1_KPDK); REGISTER(CRYPTO_MD5); REGISTER(CRYPTO_SHA1); REGISTER(CRYPTO_RIJNDAEL128_CBC); REGISTER(CRYPTO_AES_XTS); + REGISTER(CRYPTO_AES_ICM); + REGISTER(CRYPTO_AES_NIST_GCM_16); + REGISTER(CRYPTO_AES_NIST_GMAC); + REGISTER(CRYPTO_AES_128_NIST_GMAC); + REGISTER(CRYPTO_AES_192_NIST_GMAC); + REGISTER(CRYPTO_AES_256_NIST_GMAC); REGISTER(CRYPTO_CAMELLIA_CBC); REGISTER(CRYPTO_DEFLATE_COMP); #undef REGISTER return 0; } static int swcr_detach(device_t dev) { crypto_unregister_all(swcr_id); rw_wlock(&swcr_sessions_lock); free(swcr_sessions, M_CRYPTO_DATA); swcr_sessions = NULL; rw_wunlock(&swcr_sessions_lock); rw_destroy(&swcr_sessions_lock); return 0; } static device_method_t swcr_methods[] = { DEVMETHOD(device_identify, swcr_identify), DEVMETHOD(device_probe, swcr_probe), DEVMETHOD(device_attach, swcr_attach), DEVMETHOD(device_detach, swcr_detach), DEVMETHOD(cryptodev_newsession, swcr_newsession), DEVMETHOD(cryptodev_freesession,swcr_freesession), DEVMETHOD(cryptodev_process, swcr_process), {0, 0}, }; static driver_t swcr_driver = { "cryptosoft", swcr_methods, 0, /* NB: no softc */ }; static devclass_t swcr_devclass; /* * NB: We explicitly reference the crypto module so we * get the necessary ordering when built as a loadable * module. This is required because we bundle the crypto * module code together with the cryptosoft driver (otherwise * normal module dependencies would handle things). */ extern int crypto_modevent(struct module *, int, void *); /* XXX where to attach */ DRIVER_MODULE(cryptosoft, nexus, swcr_driver, swcr_devclass, crypto_modevent,0); MODULE_VERSION(cryptosoft, 1); MODULE_DEPEND(cryptosoft, crypto, 1, 1, 1); Index: projects/clang350-import/sys/opencrypto/gfmult.c =================================================================== --- projects/clang350-import/sys/opencrypto/gfmult.c (nonexistent) +++ projects/clang350-import/sys/opencrypto/gfmult.c (revision 275749) @@ -0,0 +1,275 @@ +/*- + * Copyright (c) 2014 The FreeBSD Foundation + * All rights reserved. + * + * This software was developed by John-Mark Gurney under + * the sponsorship of the FreeBSD Foundation and + * Rubicon Communications, LLC (Netgate). + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * $FreeBSD$ + * + */ + +#include "gfmult.h" + +#define REV_POLY_REDUCT 0xe1 /* 0x87 bit reversed */ + +/* reverse the bits of a nibble */ +static const uint8_t nib_rev[] = { + 0x0, 0x8, 0x4, 0xc, 0x2, 0xa, 0x6, 0xe, + 0x1, 0x9, 0x5, 0xd, 0x3, 0xb, 0x7, 0xf, +}; + +/* calulate v * 2 */ +static inline struct gf128 +gf128_mulalpha(struct gf128 v) +{ + uint64_t mask; + + mask = !!(v.v[1] & 1); + mask = ~(mask - 1); + v.v[1] = (v.v[1] >> 1) | ((v.v[0] & 1) << 63); + v.v[0] = (v.v[0] >> 1) ^ ((mask & REV_POLY_REDUCT) << 56); + + return v; +} + +/* + * Generate a table for 0-16 * h. Store the results in the table w/ indexes + * bit reversed, and the words striped across the values. + */ +void +gf128_genmultable(struct gf128 h, struct gf128table *t) +{ + struct gf128 tbl[16]; + int i; + + tbl[0] = MAKE_GF128(0, 0); + tbl[1] = h; + + for (i = 2; i < 16; i += 2) { + tbl[i] = gf128_mulalpha(tbl[i / 2]); + tbl[i + 1] = gf128_add(tbl[i], h); + } + + for (i = 0; i < 16; i++) { + t->a[nib_rev[i]] = tbl[i].v[0] >> 32; + t->b[nib_rev[i]] = tbl[i].v[0]; + t->c[nib_rev[i]] = tbl[i].v[1] >> 32; + t->d[nib_rev[i]] = tbl[i].v[1]; + } +} + +/* + * Generate tables containing h, h^2, h^3 and h^4, starting at 0. + */ +void +gf128_genmultable4(struct gf128 h, struct gf128table4 *t) +{ + struct gf128 h2, h3, h4; + + gf128_genmultable(h, &t->tbls[0]); + + h2 = gf128_mul(h, &t->tbls[0]); + + gf128_genmultable(h2, &t->tbls[1]); + + h3 = gf128_mul(h, &t->tbls[1]); + gf128_genmultable(h3, &t->tbls[2]); + + h4 = gf128_mul(h2, &t->tbls[1]); + gf128_genmultable(h4, &t->tbls[3]); +} + +/* + * Read a row from the table. + */ +static inline struct gf128 +readrow(struct gf128table *tbl, unsigned bits) +{ + struct gf128 r; + + bits = bits % 16; + + r.v[0] = ((uint64_t)tbl->a[bits] << 32) | tbl->b[bits]; + r.v[1] = ((uint64_t)tbl->c[bits] << 32) | tbl->d[bits]; + + return r; +} + +/* + * These are the reduction values. Since we are dealing with bit reversed + * version, the values need to be bit reversed, AND the indexes are also + * bit reversed to make lookups quicker. + */ +static uint16_t reduction[] = { + 0x0000, 0x1c20, 0x3840, 0x2460, 0x7080, 0x6ca0, 0x48c0, 0x54e0, + 0xe100, 0xfd20, 0xd940, 0xc560, 0x9180, 0x8da0, 0xa9c0, 0xb5e0, +}; + +/* + * Calculate: + * (x*2^4 + word[3,0]*h) * + * 2^4 + word[7,4]*h) * + * ... + * 2^4 + word[63,60]*h + */ +static struct gf128 +gfmultword(uint64_t word, struct gf128 x, struct gf128table *tbl) +{ + struct gf128 row; + unsigned bits; + unsigned redbits; + int i; + + for (i = 0; i < 64; i += 4) { + bits = word % 16; + + /* fetch row */ + row = readrow(tbl, bits); + + /* x * 2^4 */ + redbits = x.v[1] % 16; + x.v[1] = (x.v[1] >> 4) | (x.v[0] % 16) << 60; + x.v[0] >>= 4; + x.v[0] ^= (uint64_t)reduction[redbits] << (64 - 16); + + word >>= 4; + + x = gf128_add(x, row); + } + + return x; +} + +/* + * Calculate + * (x*2^4 + worda[3,0]*h^4+wordb[3,0]*h^3+...+wordd[3,0]*h) * + * ... + * 2^4 + worda[63,60]*h^4+ ... + wordd[63,60]*h + * + * Passing/returning struct is .5% faster than passing in via pointer on + * amd64. + */ +static struct gf128 +gfmultword4(uint64_t worda, uint64_t wordb, uint64_t wordc, uint64_t wordd, + struct gf128 x, struct gf128table4 *tbl) +{ + struct gf128 rowa, rowb, rowc, rowd; + unsigned bitsa, bitsb, bitsc, bitsd; + unsigned redbits; + int i; + + /* + * XXX - nibble reverse words to save a shift? probably not as + * nibble reverse would take 20 ops (5 * 4) verse 16 + */ + + for (i = 0; i < 64; i += 4) { + bitsa = worda % 16; + bitsb = wordb % 16; + bitsc = wordc % 16; + bitsd = wordd % 16; + + /* fetch row */ + rowa = readrow(&tbl->tbls[3], bitsa); + rowb = readrow(&tbl->tbls[2], bitsb); + rowc = readrow(&tbl->tbls[1], bitsc); + rowd = readrow(&tbl->tbls[0], bitsd); + + /* x * 2^4 */ + redbits = x.v[1] % 16; + x.v[1] = (x.v[1] >> 4) | (x.v[0] % 16) << 60; + x.v[0] >>= 4; + x.v[0] ^= (uint64_t)reduction[redbits] << (64 - 16); + + worda >>= 4; + wordb >>= 4; + wordc >>= 4; + wordd >>= 4; + + x = gf128_add(x, gf128_add(rowa, gf128_add(rowb, + gf128_add(rowc, rowd)))); + } + + return x; +} + +struct gf128 +gf128_mul(struct gf128 v, struct gf128table *tbl) +{ + struct gf128 ret; + + ret = MAKE_GF128(0, 0); + + ret = gfmultword(v.v[1], ret, tbl); + ret = gfmultword(v.v[0], ret, tbl); + + return ret; +} + +/* + * Calculate a*h^4 + b*h^3 + c*h^2 + d*h, or: + * (((a*h+b)*h+c)*h+d)*h + */ +struct gf128 +gf128_mul4(struct gf128 a, struct gf128 b, struct gf128 c, struct gf128 d, + struct gf128table4 *tbl) +{ + struct gf128 tmp; + + tmp = MAKE_GF128(0, 0); + + tmp = gfmultword4(a.v[1], b.v[1], c.v[1], d.v[1], tmp, tbl); + tmp = gfmultword4(a.v[0], b.v[0], c.v[0], d.v[0], tmp, tbl); + + return tmp; +} + +/* + * a = data[0..15] + r + * b = data[16..31] + * c = data[32..47] + * d = data[48..63] + * + * Calculate a*h^4 + b*h^3 + c*h^2 + d*h, or: + * (((a*h+b)*h+c)*h+d)*h + */ +struct gf128 +gf128_mul4b(struct gf128 r, const uint8_t *v, struct gf128table4 *tbl) +{ + struct gf128 a, b, c, d; + struct gf128 tmp; + + tmp = MAKE_GF128(0, 0); + + a = gf128_add(r, gf128_read(&v[0*16])); + b = gf128_read(&v[1*16]); + c = gf128_read(&v[2*16]); + d = gf128_read(&v[3*16]); + + tmp = gfmultword4(a.v[1], b.v[1], c.v[1], d.v[1], tmp, tbl); + tmp = gfmultword4(a.v[0], b.v[0], c.v[0], d.v[0], tmp, tbl); + + return tmp; +} Property changes on: projects/clang350-import/sys/opencrypto/gfmult.c ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: projects/clang350-import/sys/opencrypto/gfmult.h =================================================================== --- projects/clang350-import/sys/opencrypto/gfmult.h (nonexistent) +++ projects/clang350-import/sys/opencrypto/gfmult.h (revision 275749) @@ -0,0 +1,128 @@ +/*- + * Copyright (c) 2014 The FreeBSD Foundation + * All rights reserved. + * + * This software was developed by John-Mark Gurney under + * the sponsorship of the FreeBSD Foundation and + * Rubicon Communications, LLC (Netgate). + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * $FreeBSD$ + * + */ + +#ifndef _GFMULT_H_ +#define _GFMULT_H_ + +#ifdef __APPLE__ +#define __aligned(x) __attribute__((__aligned__(x))) +#define be64dec(buf) __builtin_bswap64(*(uint64_t *)buf) +#define be64enc(buf, x) (*(uint64_t *)buf = __builtin_bswap64(x)) +#else +#include +#endif + +#ifdef _KERNEL +#include +#else +#include +#include +#endif + +#define REQ_ALIGN (16 * 4) +/* + * The rows are striped across cache lines. Note that the indexes + * are bit reversed to make accesses quicker. + */ +struct gf128table { + uint32_t a[16] __aligned(REQ_ALIGN); /* bits 0 - 31 */ + uint32_t b[16] __aligned(REQ_ALIGN); /* bits 63 - 32 */ + uint32_t c[16] __aligned(REQ_ALIGN); /* bits 95 - 64 */ + uint32_t d[16] __aligned(REQ_ALIGN); /* bits 127 - 96 */ +} __aligned(REQ_ALIGN); + +/* + * A set of tables that contain h, h^2, h^3, h^4. To be used w/ gf128_mul4. + */ +struct gf128table4 { + struct gf128table tbls[4]; +}; + +/* + * GCM per spec is bit reversed in memory. So byte 0 is really bit reversed + * and contains bits 0-7. We can deal w/ this by using right shifts and + * related math instead of having to bit reverse everything. This means that + * the low bits are in v[0] (bits 0-63) and reverse order, while the high + * bits are in v[1] (bits 64-127) and reverse order. The high bit of v[0] is + * bit 0, and the low bit of v[1] is bit 127. + */ +struct gf128 { + uint64_t v[2]; +}; + +/* Note that we don't bit reverse in MAKE_GF128. */ +#define MAKE_GF128(a, b) ((struct gf128){.v = { (a), (b) } }) +#define GF128_EQ(a, b) ((((a).v[0] ^ (b).v[0]) | \ + ((a).v[1] ^ (b).v[1])) == 0) + +static inline struct gf128 +gf128_read(const uint8_t *buf) +{ + struct gf128 r; + + r.v[0] = be64dec(buf); + buf += sizeof(uint64_t); + + r.v[1] = be64dec(buf); + + return r; +} + +static inline void +gf128_write(struct gf128 v, uint8_t *buf) +{ + uint64_t tmp; + + be64enc(buf, v.v[0]); + buf += sizeof tmp; + + be64enc(buf, v.v[1]); +} + +static inline struct gf128 __pure /* XXX - __pure2 instead */ +gf128_add(struct gf128 a, struct gf128 b) +{ + a.v[0] ^= b.v[0]; + a.v[1] ^= b.v[1]; + + return a; +} + +void gf128_genmultable(struct gf128 h, struct gf128table *t); +void gf128_genmultable4(struct gf128 h, struct gf128table4 *t); +struct gf128 gf128_mul(struct gf128 v, struct gf128table *tbl); +struct gf128 gf128_mul4(struct gf128 a, struct gf128 b, struct gf128 c, + struct gf128 d, struct gf128table4 *tbl); +struct gf128 gf128_mul4b(struct gf128 r, const uint8_t *v, + struct gf128table4 *tbl); + +#endif /* _GFMULT_H_ */ Property changes on: projects/clang350-import/sys/opencrypto/gfmult.h ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: projects/clang350-import/sys/opencrypto/gmac.c =================================================================== --- projects/clang350-import/sys/opencrypto/gmac.c (nonexistent) +++ projects/clang350-import/sys/opencrypto/gmac.c (revision 275749) @@ -0,0 +1,119 @@ +/*- + * Copyright (c) 2014 The FreeBSD Foundation + * All rights reserved. + * + * This software was developed by John-Mark Gurney under + * the sponsorship of the FreeBSD Foundation and + * Rubicon Communications, LLC (Netgate). + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * $FreeBSD$ + * + */ + +#include +#include +#include +#include + +void +AES_GMAC_Init(struct aes_gmac_ctx *agc) +{ + + bzero(agc, sizeof *agc); +} + +void +AES_GMAC_Setkey(struct aes_gmac_ctx *agc, const uint8_t *key, uint16_t klen) +{ + const uint8_t zeros[GMAC_BLOCK_LEN] = {}; + struct gf128 h; + uint8_t hbuf[GMAC_BLOCK_LEN]; + + agc->rounds = rijndaelKeySetupEnc(agc->keysched, key, klen * 8); + + rijndaelEncrypt(agc->keysched, agc->rounds, zeros, hbuf); + + h = gf128_read(hbuf); + gf128_genmultable4(h, &agc->ghashtbl); + + explicit_bzero(&h, sizeof h); + explicit_bzero(hbuf, sizeof hbuf); +} + +void +AES_GMAC_Reinit(struct aes_gmac_ctx *agc, const uint8_t *iv, uint16_t ivlen) +{ + + KASSERT(ivlen <= sizeof agc->counter, ("passed ivlen too large!")); + bcopy(iv, agc->counter, ivlen); +} + +int +AES_GMAC_Update(struct aes_gmac_ctx *agc, const uint8_t *data, uint16_t len) +{ + struct gf128 v; + uint8_t buf[GMAC_BLOCK_LEN] = {}; + int i; + + v = agc->hash; + + while (len > 0) { + if (len >= 4*GMAC_BLOCK_LEN) { + i = 4*GMAC_BLOCK_LEN; + v = gf128_mul4b(v, data, &agc->ghashtbl); + } else if (len >= GMAC_BLOCK_LEN) { + i = GMAC_BLOCK_LEN; + v = gf128_add(v, gf128_read(data)); + v = gf128_mul(v, &agc->ghashtbl.tbls[0]); + } else { + i = len; + bcopy(data, buf, i); + v = gf128_add(v, gf128_read(&buf[0])); + v = gf128_mul(v, &agc->ghashtbl.tbls[0]); + explicit_bzero(buf, sizeof buf); + } + len -= i; + data += i; + } + + agc->hash = v; + explicit_bzero(&v, sizeof v); + + return (0); +} + +void +AES_GMAC_Final(uint8_t digest[GMAC_DIGEST_LEN], struct aes_gmac_ctx *agc) +{ + uint8_t enccntr[GMAC_BLOCK_LEN]; + struct gf128 a; + + /* XXX - zero additional bytes? */ + agc->counter[GMAC_BLOCK_LEN - 1] = 1; + + rijndaelEncrypt(agc->keysched, agc->rounds, agc->counter, enccntr); + a = gf128_add(agc->hash, gf128_read(enccntr)); + gf128_write(a, digest); + + explicit_bzero(enccntr, sizeof enccntr); +} Property changes on: projects/clang350-import/sys/opencrypto/gmac.c ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: projects/clang350-import/sys/opencrypto/gmac.h =================================================================== --- projects/clang350-import/sys/opencrypto/gmac.h (nonexistent) +++ projects/clang350-import/sys/opencrypto/gmac.h (revision 275749) @@ -0,0 +1,55 @@ +/*- + * Copyright (c) 2014 The FreeBSD Foundation + * All rights reserved. + * + * This software was developed by John-Mark Gurney under + * the sponsorship of the FreeBSD Foundation and + * Rubicon Communications, LLC (Netgate). + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * $FreeBSD$ + * + */ + +#ifndef _GMAC_H_ + +#include "gfmult.h" +#include + +#define GMAC_BLOCK_LEN 16 +#define GMAC_DIGEST_LEN 16 + +struct aes_gmac_ctx { + struct gf128table4 ghashtbl; + struct gf128 hash; + uint32_t keysched[4*(RIJNDAEL_MAXNR + 1)]; + uint8_t counter[GMAC_BLOCK_LEN]; + int rounds; +}; + +void AES_GMAC_Init(struct aes_gmac_ctx *); +void AES_GMAC_Setkey(struct aes_gmac_ctx *, const uint8_t *, uint16_t); +void AES_GMAC_Reinit(struct aes_gmac_ctx *, const uint8_t *, uint16_t); +int AES_GMAC_Update(struct aes_gmac_ctx *, const uint8_t *, uint16_t); +void AES_GMAC_Final(uint8_t [GMAC_DIGEST_LEN], struct aes_gmac_ctx *); + +#endif /* _GMAC_H_ */ Property changes on: projects/clang350-import/sys/opencrypto/gmac.h ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: projects/clang350-import/sys/opencrypto/xform.c =================================================================== --- projects/clang350-import/sys/opencrypto/xform.c (revision 275748) +++ projects/clang350-import/sys/opencrypto/xform.c (revision 275749) @@ -1,818 +1,972 @@ /* $OpenBSD: xform.c,v 1.16 2001/08/28 12:20:43 ben Exp $ */ /*- * The authors of this code are John Ioannidis (ji@tla.org), * Angelos D. Keromytis (kermit@csd.uch.gr), * Niels Provos (provos@physnet.uni-hamburg.de) and * Damien Miller (djm@mindrot.org). * * This code was written by John Ioannidis for BSD/OS in Athens, Greece, * in November 1995. * * Ported to OpenBSD and NetBSD, with additional transforms, in December 1996, * by Angelos D. Keromytis. * * Additional transforms and features in 1997 and 1998 by Angelos D. Keromytis * and Niels Provos. * * Additional features in 1999 by Angelos D. Keromytis. * * AES XTS implementation in 2008 by Damien Miller * * Copyright (C) 1995, 1996, 1997, 1998, 1999 by John Ioannidis, * Angelos D. Keromytis and Niels Provos. * * Copyright (C) 2001, Angelos D. Keromytis. * * Copyright (C) 2008, Damien Miller + * Copyright (c) 2014 The FreeBSD Foundation + * All rights reserved. * + * Portions of this software were developed by John-Mark Gurney + * under sponsorship of the FreeBSD Foundation and + * Rubicon Communications, LLC (Netgate). + * * Permission to use, copy, and modify this software with or without fee * is hereby granted, provided that this entire notice is included in * all copies of any software which is or includes a copy or * modification of this software. * You may use this code under the GNU public license if you so wish. Please * contribute changes back to the authors under this freer than GPL license * so that we may further the use of strong encryption without limitations to * all. * * THIS SOFTWARE IS BEING PROVIDED "AS IS", WITHOUT ANY EXPRESS OR * IMPLIED WARRANTY. IN PARTICULAR, NONE OF THE AUTHORS MAKES ANY * REPRESENTATION OR WARRANTY OF ANY KIND CONCERNING THE * MERCHANTABILITY OF THIS SOFTWARE OR ITS FITNESS FOR ANY PARTICULAR * PURPOSE. */ #include __FBSDID("$FreeBSD$"); #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include static int null_setkey(u_int8_t **, u_int8_t *, int); static int des1_setkey(u_int8_t **, u_int8_t *, int); static int des3_setkey(u_int8_t **, u_int8_t *, int); static int blf_setkey(u_int8_t **, u_int8_t *, int); static int cast5_setkey(u_int8_t **, u_int8_t *, int); static int skipjack_setkey(u_int8_t **, u_int8_t *, int); static int rijndael128_setkey(u_int8_t **, u_int8_t *, int); +static int aes_icm_setkey(u_int8_t **, u_int8_t *, int); static int aes_xts_setkey(u_int8_t **, u_int8_t *, int); static int cml_setkey(u_int8_t **, u_int8_t *, int); static void null_encrypt(caddr_t, u_int8_t *); static void des1_encrypt(caddr_t, u_int8_t *); static void des3_encrypt(caddr_t, u_int8_t *); static void blf_encrypt(caddr_t, u_int8_t *); static void cast5_encrypt(caddr_t, u_int8_t *); static void skipjack_encrypt(caddr_t, u_int8_t *); static void rijndael128_encrypt(caddr_t, u_int8_t *); static void aes_xts_encrypt(caddr_t, u_int8_t *); static void cml_encrypt(caddr_t, u_int8_t *); static void null_decrypt(caddr_t, u_int8_t *); static void des1_decrypt(caddr_t, u_int8_t *); static void des3_decrypt(caddr_t, u_int8_t *); static void blf_decrypt(caddr_t, u_int8_t *); static void cast5_decrypt(caddr_t, u_int8_t *); static void skipjack_decrypt(caddr_t, u_int8_t *); static void rijndael128_decrypt(caddr_t, u_int8_t *); static void aes_xts_decrypt(caddr_t, u_int8_t *); static void cml_decrypt(caddr_t, u_int8_t *); +static void aes_icm_crypt(caddr_t, u_int8_t *); + static void null_zerokey(u_int8_t **); static void des1_zerokey(u_int8_t **); static void des3_zerokey(u_int8_t **); static void blf_zerokey(u_int8_t **); static void cast5_zerokey(u_int8_t **); static void skipjack_zerokey(u_int8_t **); static void rijndael128_zerokey(u_int8_t **); +static void aes_icm_zerokey(u_int8_t **); static void aes_xts_zerokey(u_int8_t **); static void cml_zerokey(u_int8_t **); +static void aes_icm_reinit(caddr_t, u_int8_t *); static void aes_xts_reinit(caddr_t, u_int8_t *); +static void aes_gcm_reinit(caddr_t, u_int8_t *); static void null_init(void *); -static int null_update(void *, u_int8_t *, u_int16_t); +static void null_reinit(void *ctx, const u_int8_t *buf, u_int16_t len); +static int null_update(void *, const u_int8_t *, u_int16_t); static void null_final(u_int8_t *, void *); -static int MD5Update_int(void *, u_int8_t *, u_int16_t); +static int MD5Update_int(void *, const u_int8_t *, u_int16_t); static void SHA1Init_int(void *); -static int SHA1Update_int(void *, u_int8_t *, u_int16_t); +static int SHA1Update_int(void *, const u_int8_t *, u_int16_t); static void SHA1Final_int(u_int8_t *, void *); -static int RMD160Update_int(void *, u_int8_t *, u_int16_t); -static int SHA256Update_int(void *, u_int8_t *, u_int16_t); -static int SHA384Update_int(void *, u_int8_t *, u_int16_t); -static int SHA512Update_int(void *, u_int8_t *, u_int16_t); +static int RMD160Update_int(void *, const u_int8_t *, u_int16_t); +static int SHA256Update_int(void *, const u_int8_t *, u_int16_t); +static int SHA384Update_int(void *, const u_int8_t *, u_int16_t); +static int SHA512Update_int(void *, const u_int8_t *, u_int16_t); static u_int32_t deflate_compress(u_int8_t *, u_int32_t, u_int8_t **); static u_int32_t deflate_decompress(u_int8_t *, u_int32_t, u_int8_t **); +#define AESICM_BLOCKSIZE 16 + +struct aes_icm_ctx { + u_int32_t ac_ek[4*(RIJNDAEL_MAXNR + 1)]; + /* ac_block is initalized to IV */ + u_int8_t ac_block[AESICM_BLOCKSIZE]; + int ac_nr; +}; + MALLOC_DEFINE(M_XDATA, "xform", "xform data buffers"); /* Encryption instances */ struct enc_xform enc_xform_null = { CRYPTO_NULL_CBC, "NULL", /* NB: blocksize of 4 is to generate a properly aligned ESP header */ - NULL_BLOCK_LEN, 0, 256, /* 2048 bits, max key */ + NULL_BLOCK_LEN, NULL_BLOCK_LEN, 0, 256, /* 2048 bits, max key */ null_encrypt, null_decrypt, null_setkey, null_zerokey, - NULL + NULL, }; struct enc_xform enc_xform_des = { CRYPTO_DES_CBC, "DES", - DES_BLOCK_LEN, 8, 8, + DES_BLOCK_LEN, DES_BLOCK_LEN, 8, 8, des1_encrypt, des1_decrypt, des1_setkey, des1_zerokey, - NULL + NULL, }; struct enc_xform enc_xform_3des = { CRYPTO_3DES_CBC, "3DES", - DES3_BLOCK_LEN, 24, 24, + DES3_BLOCK_LEN, DES3_BLOCK_LEN, 24, 24, des3_encrypt, des3_decrypt, des3_setkey, des3_zerokey, - NULL + NULL, }; struct enc_xform enc_xform_blf = { CRYPTO_BLF_CBC, "Blowfish", - BLOWFISH_BLOCK_LEN, 5, 56 /* 448 bits, max key */, + BLOWFISH_BLOCK_LEN, BLOWFISH_BLOCK_LEN, 5, 56 /* 448 bits, max key */, blf_encrypt, blf_decrypt, blf_setkey, blf_zerokey, - NULL + NULL, }; struct enc_xform enc_xform_cast5 = { CRYPTO_CAST_CBC, "CAST-128", - CAST128_BLOCK_LEN, 5, 16, + CAST128_BLOCK_LEN, CAST128_BLOCK_LEN, 5, 16, cast5_encrypt, cast5_decrypt, cast5_setkey, cast5_zerokey, - NULL + NULL, }; struct enc_xform enc_xform_skipjack = { CRYPTO_SKIPJACK_CBC, "Skipjack", - SKIPJACK_BLOCK_LEN, 10, 10, + SKIPJACK_BLOCK_LEN, SKIPJACK_BLOCK_LEN, 10, 10, skipjack_encrypt, - skipjack_decrypt, - skipjack_setkey, + skipjack_decrypt, skipjack_setkey, skipjack_zerokey, - NULL + NULL, }; struct enc_xform enc_xform_rijndael128 = { CRYPTO_RIJNDAEL128_CBC, "Rijndael-128/AES", - RIJNDAEL128_BLOCK_LEN, 8, 32, + RIJNDAEL128_BLOCK_LEN, RIJNDAEL128_BLOCK_LEN, 16, 32, rijndael128_encrypt, rijndael128_decrypt, rijndael128_setkey, rijndael128_zerokey, - NULL + NULL, }; +struct enc_xform enc_xform_aes_icm = { + CRYPTO_AES_ICM, "AES-ICM", + RIJNDAEL128_BLOCK_LEN, RIJNDAEL128_BLOCK_LEN, 16, 32, + aes_icm_crypt, + aes_icm_crypt, + aes_icm_setkey, + rijndael128_zerokey, + aes_icm_reinit, +}; + +struct enc_xform enc_xform_aes_nist_gcm = { + CRYPTO_AES_NIST_GCM_16, "AES-GCM", + 1, 12, 16, 32, + aes_icm_crypt, + aes_icm_crypt, + aes_icm_setkey, + aes_icm_zerokey, + aes_gcm_reinit, +}; + +struct enc_xform enc_xform_aes_nist_gmac = { + CRYPTO_AES_NIST_GMAC, "AES-GMAC", + 1, 12, 16, 32, + NULL, + NULL, + NULL, + NULL, + NULL, +}; + struct enc_xform enc_xform_aes_xts = { CRYPTO_AES_XTS, "AES-XTS", - RIJNDAEL128_BLOCK_LEN, 32, 64, + RIJNDAEL128_BLOCK_LEN, 8, 32, 64, aes_xts_encrypt, aes_xts_decrypt, aes_xts_setkey, aes_xts_zerokey, aes_xts_reinit }; struct enc_xform enc_xform_arc4 = { CRYPTO_ARC4, "ARC4", - 1, 1, 32, + 1, 1, 1, 32, NULL, NULL, NULL, NULL, - NULL + NULL, }; struct enc_xform enc_xform_camellia = { CRYPTO_CAMELLIA_CBC, "Camellia", - CAMELLIA_BLOCK_LEN, 8, 32, + CAMELLIA_BLOCK_LEN, CAMELLIA_BLOCK_LEN, 8, 32, cml_encrypt, cml_decrypt, cml_setkey, cml_zerokey, - NULL + NULL, }; /* Authentication instances */ -struct auth_hash auth_hash_null = { +struct auth_hash auth_hash_null = { /* NB: context isn't used */ CRYPTO_NULL_HMAC, "NULL-HMAC", - 0, NULL_HASH_LEN, NULL_HMAC_BLOCK_LEN, sizeof(int), /* NB: context isn't used */ - null_init, null_update, null_final + 0, NULL_HASH_LEN, sizeof(int), NULL_HMAC_BLOCK_LEN, + null_init, null_reinit, null_reinit, null_update, null_final }; struct auth_hash auth_hash_hmac_md5 = { CRYPTO_MD5_HMAC, "HMAC-MD5", - 16, MD5_HASH_LEN, MD5_HMAC_BLOCK_LEN, sizeof(MD5_CTX), - (void (*) (void *)) MD5Init, MD5Update_int, + 16, MD5_HASH_LEN, sizeof(MD5_CTX), MD5_HMAC_BLOCK_LEN, + (void (*) (void *)) MD5Init, NULL, NULL, MD5Update_int, (void (*) (u_int8_t *, void *)) MD5Final }; struct auth_hash auth_hash_hmac_sha1 = { CRYPTO_SHA1_HMAC, "HMAC-SHA1", - 20, SHA1_HASH_LEN, SHA1_HMAC_BLOCK_LEN, sizeof(SHA1_CTX), - SHA1Init_int, SHA1Update_int, SHA1Final_int + 20, SHA1_HASH_LEN, sizeof(SHA1_CTX), SHA1_HMAC_BLOCK_LEN, + SHA1Init_int, NULL, NULL, SHA1Update_int, SHA1Final_int }; struct auth_hash auth_hash_hmac_ripemd_160 = { CRYPTO_RIPEMD160_HMAC, "HMAC-RIPEMD-160", - 20, RIPEMD160_HASH_LEN, RIPEMD160_HMAC_BLOCK_LEN, sizeof(RMD160_CTX), - (void (*)(void *)) RMD160Init, RMD160Update_int, + 20, RIPEMD160_HASH_LEN, sizeof(RMD160_CTX), RIPEMD160_HMAC_BLOCK_LEN, + (void (*)(void *)) RMD160Init, NULL, NULL, RMD160Update_int, (void (*)(u_int8_t *, void *)) RMD160Final }; struct auth_hash auth_hash_key_md5 = { CRYPTO_MD5_KPDK, "Keyed MD5", - 0, MD5_KPDK_HASH_LEN, 0, sizeof(MD5_CTX), - (void (*)(void *)) MD5Init, MD5Update_int, + 0, MD5_KPDK_HASH_LEN, sizeof(MD5_CTX), 0, + (void (*)(void *)) MD5Init, NULL, NULL, MD5Update_int, (void (*)(u_int8_t *, void *)) MD5Final }; struct auth_hash auth_hash_key_sha1 = { CRYPTO_SHA1_KPDK, "Keyed SHA1", - 0, SHA1_KPDK_HASH_LEN, 0, sizeof(SHA1_CTX), - SHA1Init_int, SHA1Update_int, SHA1Final_int + 0, SHA1_KPDK_HASH_LEN, sizeof(SHA1_CTX), 0, + SHA1Init_int, NULL, NULL, SHA1Update_int, SHA1Final_int }; struct auth_hash auth_hash_hmac_sha2_256 = { CRYPTO_SHA2_256_HMAC, "HMAC-SHA2-256", - 32, SHA2_256_HASH_LEN, SHA2_256_HMAC_BLOCK_LEN, sizeof(SHA256_CTX), - (void (*)(void *)) SHA256_Init, SHA256Update_int, + 32, SHA2_256_HASH_LEN, sizeof(SHA256_CTX), SHA2_256_HMAC_BLOCK_LEN, + (void (*)(void *)) SHA256_Init, NULL, NULL, SHA256Update_int, (void (*)(u_int8_t *, void *)) SHA256_Final }; struct auth_hash auth_hash_hmac_sha2_384 = { CRYPTO_SHA2_384_HMAC, "HMAC-SHA2-384", - 48, SHA2_384_HASH_LEN, SHA2_384_HMAC_BLOCK_LEN, sizeof(SHA384_CTX), - (void (*)(void *)) SHA384_Init, SHA384Update_int, + 48, SHA2_384_HASH_LEN, sizeof(SHA384_CTX), SHA2_384_HMAC_BLOCK_LEN, + (void (*)(void *)) SHA384_Init, NULL, NULL, SHA384Update_int, (void (*)(u_int8_t *, void *)) SHA384_Final }; struct auth_hash auth_hash_hmac_sha2_512 = { CRYPTO_SHA2_512_HMAC, "HMAC-SHA2-512", - 64, SHA2_512_HASH_LEN, SHA2_512_HMAC_BLOCK_LEN, sizeof(SHA512_CTX), - (void (*)(void *)) SHA512_Init, SHA512Update_int, + 64, SHA2_512_HASH_LEN, sizeof(SHA512_CTX), SHA2_512_HMAC_BLOCK_LEN, + (void (*)(void *)) SHA512_Init, NULL, NULL, SHA512Update_int, (void (*)(u_int8_t *, void *)) SHA512_Final }; +struct auth_hash auth_hash_nist_gmac_aes_128 = { + CRYPTO_AES_128_NIST_GMAC, "GMAC-AES-128", + 16, 16, sizeof(struct aes_gmac_ctx), GMAC_BLOCK_LEN, + (void (*)(void *)) AES_GMAC_Init, + (void (*)(void *, const u_int8_t *, u_int16_t)) AES_GMAC_Setkey, + (void (*)(void *, const u_int8_t *, u_int16_t)) AES_GMAC_Reinit, + (int (*)(void *, const u_int8_t *, u_int16_t)) AES_GMAC_Update, + (void (*)(u_int8_t *, void *)) AES_GMAC_Final +}; + +struct auth_hash auth_hash_nist_gmac_aes_192 = { + CRYPTO_AES_192_NIST_GMAC, "GMAC-AES-192", + 24, 16, sizeof(struct aes_gmac_ctx), GMAC_BLOCK_LEN, + (void (*)(void *)) AES_GMAC_Init, + (void (*)(void *, const u_int8_t *, u_int16_t)) AES_GMAC_Setkey, + (void (*)(void *, const u_int8_t *, u_int16_t)) AES_GMAC_Reinit, + (int (*)(void *, const u_int8_t *, u_int16_t)) AES_GMAC_Update, + (void (*)(u_int8_t *, void *)) AES_GMAC_Final +}; + +struct auth_hash auth_hash_nist_gmac_aes_256 = { + CRYPTO_AES_256_NIST_GMAC, "GMAC-AES-256", + 32, 16, sizeof(struct aes_gmac_ctx), GMAC_BLOCK_LEN, + (void (*)(void *)) AES_GMAC_Init, + (void (*)(void *, const u_int8_t *, u_int16_t)) AES_GMAC_Setkey, + (void (*)(void *, const u_int8_t *, u_int16_t)) AES_GMAC_Reinit, + (int (*)(void *, const u_int8_t *, u_int16_t)) AES_GMAC_Update, + (void (*)(u_int8_t *, void *)) AES_GMAC_Final +}; + /* Compression instance */ struct comp_algo comp_algo_deflate = { CRYPTO_DEFLATE_COMP, "Deflate", 90, deflate_compress, deflate_decompress }; /* * Encryption wrapper routines. */ static void null_encrypt(caddr_t key, u_int8_t *blk) { } static void null_decrypt(caddr_t key, u_int8_t *blk) { } static int null_setkey(u_int8_t **sched, u_int8_t *key, int len) { *sched = NULL; return 0; } static void null_zerokey(u_int8_t **sched) { *sched = NULL; } static void des1_encrypt(caddr_t key, u_int8_t *blk) { des_cblock *cb = (des_cblock *) blk; des_key_schedule *p = (des_key_schedule *) key; des_ecb_encrypt(cb, cb, p[0], DES_ENCRYPT); } static void des1_decrypt(caddr_t key, u_int8_t *blk) { des_cblock *cb = (des_cblock *) blk; des_key_schedule *p = (des_key_schedule *) key; des_ecb_encrypt(cb, cb, p[0], DES_DECRYPT); } static int des1_setkey(u_int8_t **sched, u_int8_t *key, int len) { des_key_schedule *p; int err; p = malloc(sizeof (des_key_schedule), M_CRYPTO_DATA, M_NOWAIT|M_ZERO); if (p != NULL) { des_set_key((des_cblock *) key, p[0]); err = 0; } else err = ENOMEM; *sched = (u_int8_t *) p; return err; } static void des1_zerokey(u_int8_t **sched) { bzero(*sched, sizeof (des_key_schedule)); free(*sched, M_CRYPTO_DATA); *sched = NULL; } static void des3_encrypt(caddr_t key, u_int8_t *blk) { des_cblock *cb = (des_cblock *) blk; des_key_schedule *p = (des_key_schedule *) key; des_ecb3_encrypt(cb, cb, p[0], p[1], p[2], DES_ENCRYPT); } static void des3_decrypt(caddr_t key, u_int8_t *blk) { des_cblock *cb = (des_cblock *) blk; des_key_schedule *p = (des_key_schedule *) key; des_ecb3_encrypt(cb, cb, p[0], p[1], p[2], DES_DECRYPT); } static int des3_setkey(u_int8_t **sched, u_int8_t *key, int len) { des_key_schedule *p; int err; p = malloc(3*sizeof (des_key_schedule), M_CRYPTO_DATA, M_NOWAIT|M_ZERO); if (p != NULL) { des_set_key((des_cblock *)(key + 0), p[0]); des_set_key((des_cblock *)(key + 8), p[1]); des_set_key((des_cblock *)(key + 16), p[2]); err = 0; } else err = ENOMEM; *sched = (u_int8_t *) p; return err; } static void des3_zerokey(u_int8_t **sched) { bzero(*sched, 3*sizeof (des_key_schedule)); free(*sched, M_CRYPTO_DATA); *sched = NULL; } static void blf_encrypt(caddr_t key, u_int8_t *blk) { BF_LONG t[2]; memcpy(t, blk, sizeof (t)); t[0] = ntohl(t[0]); t[1] = ntohl(t[1]); /* NB: BF_encrypt expects the block in host order! */ BF_encrypt(t, (BF_KEY *) key); t[0] = htonl(t[0]); t[1] = htonl(t[1]); memcpy(blk, t, sizeof (t)); } static void blf_decrypt(caddr_t key, u_int8_t *blk) { BF_LONG t[2]; memcpy(t, blk, sizeof (t)); t[0] = ntohl(t[0]); t[1] = ntohl(t[1]); /* NB: BF_decrypt expects the block in host order! */ BF_decrypt(t, (BF_KEY *) key); t[0] = htonl(t[0]); t[1] = htonl(t[1]); memcpy(blk, t, sizeof (t)); } static int blf_setkey(u_int8_t **sched, u_int8_t *key, int len) { int err; *sched = malloc(sizeof(BF_KEY), M_CRYPTO_DATA, M_NOWAIT|M_ZERO); if (*sched != NULL) { BF_set_key((BF_KEY *) *sched, len, key); err = 0; } else err = ENOMEM; return err; } static void blf_zerokey(u_int8_t **sched) { bzero(*sched, sizeof(BF_KEY)); free(*sched, M_CRYPTO_DATA); *sched = NULL; } static void cast5_encrypt(caddr_t key, u_int8_t *blk) { cast_encrypt((cast_key *) key, blk, blk); } static void cast5_decrypt(caddr_t key, u_int8_t *blk) { cast_decrypt((cast_key *) key, blk, blk); } static int cast5_setkey(u_int8_t **sched, u_int8_t *key, int len) { int err; *sched = malloc(sizeof(cast_key), M_CRYPTO_DATA, M_NOWAIT|M_ZERO); if (*sched != NULL) { cast_setkey((cast_key *)*sched, key, len); err = 0; } else err = ENOMEM; return err; } static void cast5_zerokey(u_int8_t **sched) { bzero(*sched, sizeof(cast_key)); free(*sched, M_CRYPTO_DATA); *sched = NULL; } static void skipjack_encrypt(caddr_t key, u_int8_t *blk) { skipjack_forwards(blk, blk, (u_int8_t **) key); } static void skipjack_decrypt(caddr_t key, u_int8_t *blk) { skipjack_backwards(blk, blk, (u_int8_t **) key); } static int skipjack_setkey(u_int8_t **sched, u_int8_t *key, int len) { int err; /* NB: allocate all the memory that's needed at once */ *sched = malloc(10 * (sizeof(u_int8_t *) + 0x100), M_CRYPTO_DATA, M_NOWAIT|M_ZERO); if (*sched != NULL) { u_int8_t** key_tables = (u_int8_t**) *sched; u_int8_t* table = (u_int8_t*) &key_tables[10]; int k; for (k = 0; k < 10; k++) { key_tables[k] = table; table += 0x100; } subkey_table_gen(key, (u_int8_t **) *sched); err = 0; } else err = ENOMEM; return err; } static void skipjack_zerokey(u_int8_t **sched) { bzero(*sched, 10 * (sizeof(u_int8_t *) + 0x100)); free(*sched, M_CRYPTO_DATA); *sched = NULL; } static void rijndael128_encrypt(caddr_t key, u_int8_t *blk) { rijndael_encrypt((rijndael_ctx *) key, (u_char *) blk, (u_char *) blk); } static void rijndael128_decrypt(caddr_t key, u_int8_t *blk) { rijndael_decrypt(((rijndael_ctx *) key), (u_char *) blk, (u_char *) blk); } static int rijndael128_setkey(u_int8_t **sched, u_int8_t *key, int len) { int err; if (len != 16 && len != 24 && len != 32) return (EINVAL); *sched = malloc(sizeof(rijndael_ctx), M_CRYPTO_DATA, M_NOWAIT|M_ZERO); if (*sched != NULL) { rijndael_set_key((rijndael_ctx *) *sched, (u_char *) key, len * 8); err = 0; } else err = ENOMEM; return err; } static void rijndael128_zerokey(u_int8_t **sched) { bzero(*sched, sizeof(rijndael_ctx)); free(*sched, M_CRYPTO_DATA); *sched = NULL; } +void +aes_icm_reinit(caddr_t key, u_int8_t *iv) +{ + struct aes_icm_ctx *ctx; + + ctx = (struct aes_icm_ctx *)key; + bcopy(iv, ctx->ac_block, AESICM_BLOCKSIZE); +} + +void +aes_gcm_reinit(caddr_t key, u_int8_t *iv) +{ + struct aes_icm_ctx *ctx; + + aes_icm_reinit(key, iv); + + ctx = (struct aes_icm_ctx *)key; + /* GCM starts with 2 as counter 1 is used for final xor of tag. */ + bzero(&ctx->ac_block[AESICM_BLOCKSIZE - 4], 4); + ctx->ac_block[AESICM_BLOCKSIZE - 1] = 2; +} + +void +aes_icm_crypt(caddr_t key, u_int8_t *data) +{ + struct aes_icm_ctx *ctx; + u_int8_t keystream[AESICM_BLOCKSIZE]; + int i; + + ctx = (struct aes_icm_ctx *)key; + rijndaelEncrypt(ctx->ac_ek, ctx->ac_nr, ctx->ac_block, keystream); + for (i = 0; i < AESICM_BLOCKSIZE; i++) + data[i] ^= keystream[i]; + explicit_bzero(keystream, sizeof(keystream)); + + /* increment counter */ + for (i = AESICM_BLOCKSIZE - 1; + i >= 0; i--) + if (++ctx->ac_block[i]) /* continue on overflow */ + break; +} + +int +aes_icm_setkey(u_int8_t **sched, u_int8_t *key, int len) +{ + struct aes_icm_ctx *ctx; + + *sched = malloc(sizeof(struct aes_icm_ctx), M_CRYPTO_DATA, + M_NOWAIT | M_ZERO); + if (*sched == NULL) + return ENOMEM; + + ctx = (struct aes_icm_ctx *)*sched; + ctx->ac_nr = rijndaelKeySetupEnc(ctx->ac_ek, (u_char *)key, len * 8); + if (ctx->ac_nr == 0) + return EINVAL; + return 0; +} + +void +aes_icm_zerokey(u_int8_t **sched) +{ + + bzero(*sched, sizeof(struct aes_icm_ctx)); + free(*sched, M_CRYPTO_DATA); + *sched = NULL; +} + #define AES_XTS_BLOCKSIZE 16 #define AES_XTS_IVSIZE 8 #define AES_XTS_ALPHA 0x87 /* GF(2^128) generator polynomial */ struct aes_xts_ctx { rijndael_ctx key1; rijndael_ctx key2; u_int8_t tweak[AES_XTS_BLOCKSIZE]; }; void aes_xts_reinit(caddr_t key, u_int8_t *iv) { struct aes_xts_ctx *ctx = (struct aes_xts_ctx *)key; u_int64_t blocknum; u_int i; /* * Prepare tweak as E_k2(IV). IV is specified as LE representation * of a 64-bit block number which we allow to be passed in directly. */ bcopy(iv, &blocknum, AES_XTS_IVSIZE); for (i = 0; i < AES_XTS_IVSIZE; i++) { ctx->tweak[i] = blocknum & 0xff; blocknum >>= 8; } /* Last 64 bits of IV are always zero */ bzero(ctx->tweak + AES_XTS_IVSIZE, AES_XTS_IVSIZE); rijndael_encrypt(&ctx->key2, ctx->tweak, ctx->tweak); } static void aes_xts_crypt(struct aes_xts_ctx *ctx, u_int8_t *data, u_int do_encrypt) { u_int8_t block[AES_XTS_BLOCKSIZE]; u_int i, carry_in, carry_out; for (i = 0; i < AES_XTS_BLOCKSIZE; i++) block[i] = data[i] ^ ctx->tweak[i]; if (do_encrypt) rijndael_encrypt(&ctx->key1, block, data); else rijndael_decrypt(&ctx->key1, block, data); for (i = 0; i < AES_XTS_BLOCKSIZE; i++) data[i] ^= ctx->tweak[i]; /* Exponentiate tweak */ carry_in = 0; for (i = 0; i < AES_XTS_BLOCKSIZE; i++) { carry_out = ctx->tweak[i] & 0x80; ctx->tweak[i] = (ctx->tweak[i] << 1) | (carry_in ? 1 : 0); carry_in = carry_out; } if (carry_in) ctx->tweak[0] ^= AES_XTS_ALPHA; bzero(block, sizeof(block)); } void aes_xts_encrypt(caddr_t key, u_int8_t *data) { aes_xts_crypt((struct aes_xts_ctx *)key, data, 1); } void aes_xts_decrypt(caddr_t key, u_int8_t *data) { aes_xts_crypt((struct aes_xts_ctx *)key, data, 0); } int aes_xts_setkey(u_int8_t **sched, u_int8_t *key, int len) { struct aes_xts_ctx *ctx; if (len != 32 && len != 64) return EINVAL; *sched = malloc(sizeof(struct aes_xts_ctx), M_CRYPTO_DATA, M_NOWAIT | M_ZERO); if (*sched == NULL) return ENOMEM; ctx = (struct aes_xts_ctx *)*sched; rijndael_set_key(&ctx->key1, key, len * 4); rijndael_set_key(&ctx->key2, key + (len / 2), len * 4); return 0; } void aes_xts_zerokey(u_int8_t **sched) { bzero(*sched, sizeof(struct aes_xts_ctx)); free(*sched, M_CRYPTO_DATA); *sched = NULL; } static void cml_encrypt(caddr_t key, u_int8_t *blk) { camellia_encrypt((camellia_ctx *) key, (u_char *) blk, (u_char *) blk); } static void cml_decrypt(caddr_t key, u_int8_t *blk) { camellia_decrypt(((camellia_ctx *) key), (u_char *) blk, (u_char *) blk); } static int cml_setkey(u_int8_t **sched, u_int8_t *key, int len) { int err; if (len != 16 && len != 24 && len != 32) return (EINVAL); *sched = malloc(sizeof(camellia_ctx), M_CRYPTO_DATA, M_NOWAIT|M_ZERO); if (*sched != NULL) { camellia_set_key((camellia_ctx *) *sched, (u_char *) key, len * 8); err = 0; } else err = ENOMEM; return err; } static void cml_zerokey(u_int8_t **sched) { bzero(*sched, sizeof(camellia_ctx)); free(*sched, M_CRYPTO_DATA); *sched = NULL; } /* * And now for auth. */ static void null_init(void *ctx) { } +static void +null_reinit(void *ctx, const u_int8_t *buf, u_int16_t len) +{ +} + static int -null_update(void *ctx, u_int8_t *buf, u_int16_t len) +null_update(void *ctx, const u_int8_t *buf, u_int16_t len) { return 0; } static void null_final(u_int8_t *buf, void *ctx) { if (buf != (u_int8_t *) 0) bzero(buf, 12); } static int -RMD160Update_int(void *ctx, u_int8_t *buf, u_int16_t len) +RMD160Update_int(void *ctx, const u_int8_t *buf, u_int16_t len) { RMD160Update(ctx, buf, len); return 0; } static int -MD5Update_int(void *ctx, u_int8_t *buf, u_int16_t len) +MD5Update_int(void *ctx, const u_int8_t *buf, u_int16_t len) { MD5Update(ctx, buf, len); return 0; } static void SHA1Init_int(void *ctx) { SHA1Init(ctx); } static int -SHA1Update_int(void *ctx, u_int8_t *buf, u_int16_t len) +SHA1Update_int(void *ctx, const u_int8_t *buf, u_int16_t len) { SHA1Update(ctx, buf, len); return 0; } static void SHA1Final_int(u_int8_t *blk, void *ctx) { SHA1Final(blk, ctx); } static int -SHA256Update_int(void *ctx, u_int8_t *buf, u_int16_t len) +SHA256Update_int(void *ctx, const u_int8_t *buf, u_int16_t len) { SHA256_Update(ctx, buf, len); return 0; } static int -SHA384Update_int(void *ctx, u_int8_t *buf, u_int16_t len) +SHA384Update_int(void *ctx, const u_int8_t *buf, u_int16_t len) { SHA384_Update(ctx, buf, len); return 0; } static int -SHA512Update_int(void *ctx, u_int8_t *buf, u_int16_t len) +SHA512Update_int(void *ctx, const u_int8_t *buf, u_int16_t len) { SHA512_Update(ctx, buf, len); return 0; } /* * And compression */ static u_int32_t deflate_compress(data, size, out) u_int8_t *data; u_int32_t size; u_int8_t **out; { return deflate_global(data, size, 0, out); } static u_int32_t deflate_decompress(data, size, out) u_int8_t *data; u_int32_t size; u_int8_t **out; { return deflate_global(data, size, 1, out); } Index: projects/clang350-import/sys/opencrypto/xform.h =================================================================== --- projects/clang350-import/sys/opencrypto/xform.h (revision 275748) +++ projects/clang350-import/sys/opencrypto/xform.h (revision 275749) @@ -1,105 +1,122 @@ /* $FreeBSD$ */ /* $OpenBSD: xform.h,v 1.8 2001/08/28 12:20:43 ben Exp $ */ /*- * The author of this code is Angelos D. Keromytis (angelos@cis.upenn.edu) * * This code was written by Angelos D. Keromytis in Athens, Greece, in * February 2000. Network Security Technologies Inc. (NSTI) kindly * supported the development of this code. * * Copyright (c) 2000 Angelos D. Keromytis + * Copyright (c) 2014 The FreeBSD Foundation + * All rights reserved. * + * Portions of this software were developed by John-Mark Gurney + * under sponsorship of the FreeBSD Foundation and + * Rubicon Communications, LLC (Netgate). + * * Permission to use, copy, and modify this software without fee * is hereby granted, provided that this entire notice is included in * all source code copies of any software which is or includes a copy or * modification of this software. * * THIS SOFTWARE IS BEING PROVIDED "AS IS", WITHOUT ANY EXPRESS OR * IMPLIED WARRANTY. IN PARTICULAR, NONE OF THE AUTHORS MAKES ANY * REPRESENTATION OR WARRANTY OF ANY KIND CONCERNING THE * MERCHANTABILITY OF THIS SOFTWARE OR ITS FITNESS FOR ANY PARTICULAR * PURPOSE. */ #ifndef _CRYPTO_XFORM_H_ #define _CRYPTO_XFORM_H_ #include #include #include #include +#include /* Declarations */ struct auth_hash { int type; char *name; u_int16_t keysize; u_int16_t hashsize; - u_int16_t blocksize; u_int16_t ctxsize; + u_int16_t blocksize; void (*Init) (void *); - int (*Update) (void *, u_int8_t *, u_int16_t); + void (*Setkey) (void *, const u_int8_t *, u_int16_t); + void (*Reinit) (void *, const u_int8_t *, u_int16_t); + int (*Update) (void *, const u_int8_t *, u_int16_t); void (*Final) (u_int8_t *, void *); }; /* XXX use a define common with other hash stuff ! */ #define AH_ALEN_MAX 64 /* max authenticator hash length */ struct enc_xform { int type; char *name; u_int16_t blocksize; + u_int16_t ivsize; u_int16_t minkey, maxkey; void (*encrypt) (caddr_t, u_int8_t *); void (*decrypt) (caddr_t, u_int8_t *); int (*setkey) (u_int8_t **, u_int8_t *, int len); void (*zerokey) (u_int8_t **); void (*reinit) (caddr_t, u_int8_t *); }; struct comp_algo { int type; char *name; size_t minlen; u_int32_t (*compress) (u_int8_t *, u_int32_t, u_int8_t **); u_int32_t (*decompress) (u_int8_t *, u_int32_t, u_int8_t **); }; union authctx { MD5_CTX md5ctx; SHA1_CTX sha1ctx; RMD160_CTX rmd160ctx; SHA256_CTX sha256ctx; SHA384_CTX sha384ctx; SHA512_CTX sha512ctx; + struct aes_gmac_ctx aes_gmac_ctx; }; extern struct enc_xform enc_xform_null; extern struct enc_xform enc_xform_des; extern struct enc_xform enc_xform_3des; extern struct enc_xform enc_xform_blf; extern struct enc_xform enc_xform_cast5; extern struct enc_xform enc_xform_skipjack; extern struct enc_xform enc_xform_rijndael128; +extern struct enc_xform enc_xform_aes_icm; +extern struct enc_xform enc_xform_aes_nist_gcm; +extern struct enc_xform enc_xform_aes_nist_gmac; extern struct enc_xform enc_xform_aes_xts; extern struct enc_xform enc_xform_arc4; extern struct enc_xform enc_xform_camellia; extern struct auth_hash auth_hash_null; extern struct auth_hash auth_hash_key_md5; extern struct auth_hash auth_hash_key_sha1; extern struct auth_hash auth_hash_hmac_md5; extern struct auth_hash auth_hash_hmac_sha1; extern struct auth_hash auth_hash_hmac_ripemd_160; extern struct auth_hash auth_hash_hmac_sha2_256; extern struct auth_hash auth_hash_hmac_sha2_384; extern struct auth_hash auth_hash_hmac_sha2_512; +extern struct auth_hash auth_hash_nist_gmac_aes_128; +extern struct auth_hash auth_hash_nist_gmac_aes_192; +extern struct auth_hash auth_hash_nist_gmac_aes_256; extern struct comp_algo comp_algo_deflate; #ifdef _KERNEL #include MALLOC_DECLARE(M_XDATA); #endif #endif /* _CRYPTO_XFORM_H_ */ Index: projects/clang350-import/sys/rpc/svc.c =================================================================== --- projects/clang350-import/sys/rpc/svc.c (revision 275748) +++ projects/clang350-import/sys/rpc/svc.c (revision 275749) @@ -1,1434 +1,1435 @@ /* $NetBSD: svc.c,v 1.21 2000/07/06 03:10:35 christos Exp $ */ /*- * Copyright (c) 2009, Sun Microsystems, Inc. * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are met: * - Redistributions of source code must retain the above copyright notice, * this list of conditions and the following disclaimer. * - Redistributions in binary form must reproduce the above copyright notice, * this list of conditions and the following disclaimer in the documentation * and/or other materials provided with the distribution. * - Neither the name of Sun Microsystems, Inc. nor the names of its * contributors may be used to endorse or promote products derived * from this software without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ #if defined(LIBC_SCCS) && !defined(lint) static char *sccsid2 = "@(#)svc.c 1.44 88/02/08 Copyr 1984 Sun Micro"; static char *sccsid = "@(#)svc.c 2.4 88/08/11 4.0 RPCSRC"; #endif #include __FBSDID("$FreeBSD$"); /* * svc.c, Server-side remote procedure call interface. * * There are two sets of procedures here. The xprt routines are * for handling transport handles. The svc routines handle the * list of service routines. * * Copyright (C) 1984, Sun Microsystems, Inc. */ #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #define SVC_VERSQUIET 0x0001 /* keep quiet about vers mismatch */ #define version_keepquiet(xp) (SVC_EXT(xp)->xp_flags & SVC_VERSQUIET) static struct svc_callout *svc_find(SVCPOOL *pool, rpcprog_t, rpcvers_t, char *); static void svc_new_thread(SVCGROUP *grp); static void xprt_unregister_locked(SVCXPRT *xprt); static void svc_change_space_used(SVCPOOL *pool, int delta); static bool_t svc_request_space_available(SVCPOOL *pool); /* *************** SVCXPRT related stuff **************** */ static int svcpool_minthread_sysctl(SYSCTL_HANDLER_ARGS); static int svcpool_maxthread_sysctl(SYSCTL_HANDLER_ARGS); static int svcpool_threads_sysctl(SYSCTL_HANDLER_ARGS); SVCPOOL* svcpool_create(const char *name, struct sysctl_oid_list *sysctl_base) { SVCPOOL *pool; SVCGROUP *grp; int g; pool = malloc(sizeof(SVCPOOL), M_RPC, M_WAITOK|M_ZERO); mtx_init(&pool->sp_lock, "sp_lock", NULL, MTX_DEF); pool->sp_name = name; pool->sp_state = SVCPOOL_INIT; pool->sp_proc = NULL; TAILQ_INIT(&pool->sp_callouts); TAILQ_INIT(&pool->sp_lcallouts); pool->sp_minthreads = 1; pool->sp_maxthreads = 1; pool->sp_groupcount = 1; for (g = 0; g < SVC_MAXGROUPS; g++) { grp = &pool->sp_groups[g]; mtx_init(&grp->sg_lock, "sg_lock", NULL, MTX_DEF); grp->sg_pool = pool; grp->sg_state = SVCPOOL_ACTIVE; TAILQ_INIT(&grp->sg_xlist); TAILQ_INIT(&grp->sg_active); LIST_INIT(&grp->sg_idlethreads); grp->sg_minthreads = 1; grp->sg_maxthreads = 1; } /* * Don't use more than a quarter of mbuf clusters or more than * 45Mb buffering requests. */ pool->sp_space_high = nmbclusters * MCLBYTES / 4; if (pool->sp_space_high > 45 << 20) pool->sp_space_high = 45 << 20; pool->sp_space_low = 2 * pool->sp_space_high / 3; sysctl_ctx_init(&pool->sp_sysctl); if (sysctl_base) { SYSCTL_ADD_PROC(&pool->sp_sysctl, sysctl_base, OID_AUTO, "minthreads", CTLTYPE_INT | CTLFLAG_RW, pool, 0, svcpool_minthread_sysctl, "I", "Minimal number of threads"); SYSCTL_ADD_PROC(&pool->sp_sysctl, sysctl_base, OID_AUTO, "maxthreads", CTLTYPE_INT | CTLFLAG_RW, pool, 0, svcpool_maxthread_sysctl, "I", "Maximal number of threads"); SYSCTL_ADD_PROC(&pool->sp_sysctl, sysctl_base, OID_AUTO, "threads", CTLTYPE_INT | CTLFLAG_RD, pool, 0, svcpool_threads_sysctl, "I", "Current number of threads"); SYSCTL_ADD_INT(&pool->sp_sysctl, sysctl_base, OID_AUTO, "groups", CTLFLAG_RD, &pool->sp_groupcount, 0, "Number of thread groups"); SYSCTL_ADD_UINT(&pool->sp_sysctl, sysctl_base, OID_AUTO, "request_space_used", CTLFLAG_RD, &pool->sp_space_used, 0, "Space in parsed but not handled requests."); SYSCTL_ADD_UINT(&pool->sp_sysctl, sysctl_base, OID_AUTO, "request_space_used_highest", CTLFLAG_RD, &pool->sp_space_used_highest, 0, "Highest space used since reboot."); SYSCTL_ADD_UINT(&pool->sp_sysctl, sysctl_base, OID_AUTO, "request_space_high", CTLFLAG_RW, &pool->sp_space_high, 0, "Maximum space in parsed but not handled requests."); SYSCTL_ADD_UINT(&pool->sp_sysctl, sysctl_base, OID_AUTO, "request_space_low", CTLFLAG_RW, &pool->sp_space_low, 0, "Low water mark for request space."); SYSCTL_ADD_INT(&pool->sp_sysctl, sysctl_base, OID_AUTO, "request_space_throttled", CTLFLAG_RD, &pool->sp_space_throttled, 0, "Whether nfs requests are currently throttled"); SYSCTL_ADD_INT(&pool->sp_sysctl, sysctl_base, OID_AUTO, "request_space_throttle_count", CTLFLAG_RD, &pool->sp_space_throttle_count, 0, "Count of times throttling based on request space has occurred"); } return pool; } void svcpool_destroy(SVCPOOL *pool) { SVCGROUP *grp; SVCXPRT *xprt, *nxprt; struct svc_callout *s; struct svc_loss_callout *sl; struct svcxprt_list cleanup; int g; TAILQ_INIT(&cleanup); for (g = 0; g < SVC_MAXGROUPS; g++) { grp = &pool->sp_groups[g]; mtx_lock(&grp->sg_lock); while ((xprt = TAILQ_FIRST(&grp->sg_xlist)) != NULL) { xprt_unregister_locked(xprt); TAILQ_INSERT_TAIL(&cleanup, xprt, xp_link); } mtx_unlock(&grp->sg_lock); } TAILQ_FOREACH_SAFE(xprt, &cleanup, xp_link, nxprt) { SVC_RELEASE(xprt); } mtx_lock(&pool->sp_lock); while ((s = TAILQ_FIRST(&pool->sp_callouts)) != NULL) { mtx_unlock(&pool->sp_lock); svc_unreg(pool, s->sc_prog, s->sc_vers); mtx_lock(&pool->sp_lock); } while ((sl = TAILQ_FIRST(&pool->sp_lcallouts)) != NULL) { mtx_unlock(&pool->sp_lock); svc_loss_unreg(pool, sl->slc_dispatch); mtx_lock(&pool->sp_lock); } mtx_unlock(&pool->sp_lock); for (g = 0; g < SVC_MAXGROUPS; g++) { grp = &pool->sp_groups[g]; mtx_destroy(&grp->sg_lock); } mtx_destroy(&pool->sp_lock); if (pool->sp_rcache) replay_freecache(pool->sp_rcache); sysctl_ctx_free(&pool->sp_sysctl); free(pool, M_RPC); } /* * Sysctl handler to get the present thread count on a pool */ static int svcpool_threads_sysctl(SYSCTL_HANDLER_ARGS) { SVCPOOL *pool; int threads, error, g; pool = oidp->oid_arg1; threads = 0; mtx_lock(&pool->sp_lock); for (g = 0; g < pool->sp_groupcount; g++) threads += pool->sp_groups[g].sg_threadcount; mtx_unlock(&pool->sp_lock); error = sysctl_handle_int(oidp, &threads, 0, req); return (error); } /* * Sysctl handler to set the minimum thread count on a pool */ static int svcpool_minthread_sysctl(SYSCTL_HANDLER_ARGS) { SVCPOOL *pool; int newminthreads, error, g; pool = oidp->oid_arg1; newminthreads = pool->sp_minthreads; error = sysctl_handle_int(oidp, &newminthreads, 0, req); if (error == 0 && newminthreads != pool->sp_minthreads) { if (newminthreads > pool->sp_maxthreads) return (EINVAL); mtx_lock(&pool->sp_lock); pool->sp_minthreads = newminthreads; for (g = 0; g < pool->sp_groupcount; g++) { pool->sp_groups[g].sg_minthreads = max(1, pool->sp_minthreads / pool->sp_groupcount); } mtx_unlock(&pool->sp_lock); } return (error); } /* * Sysctl handler to set the maximum thread count on a pool */ static int svcpool_maxthread_sysctl(SYSCTL_HANDLER_ARGS) { SVCPOOL *pool; int newmaxthreads, error, g; pool = oidp->oid_arg1; newmaxthreads = pool->sp_maxthreads; error = sysctl_handle_int(oidp, &newmaxthreads, 0, req); if (error == 0 && newmaxthreads != pool->sp_maxthreads) { if (newmaxthreads < pool->sp_minthreads) return (EINVAL); mtx_lock(&pool->sp_lock); pool->sp_maxthreads = newmaxthreads; for (g = 0; g < pool->sp_groupcount; g++) { pool->sp_groups[g].sg_maxthreads = max(1, pool->sp_maxthreads / pool->sp_groupcount); } mtx_unlock(&pool->sp_lock); } return (error); } /* * Activate a transport handle. */ void xprt_register(SVCXPRT *xprt) { SVCPOOL *pool = xprt->xp_pool; SVCGROUP *grp; int g; SVC_ACQUIRE(xprt); g = atomic_fetchadd_int(&pool->sp_nextgroup, 1) % pool->sp_groupcount; xprt->xp_group = grp = &pool->sp_groups[g]; mtx_lock(&grp->sg_lock); xprt->xp_registered = TRUE; xprt->xp_active = FALSE; TAILQ_INSERT_TAIL(&grp->sg_xlist, xprt, xp_link); mtx_unlock(&grp->sg_lock); } /* * De-activate a transport handle. Note: the locked version doesn't * release the transport - caller must do that after dropping the pool * lock. */ static void xprt_unregister_locked(SVCXPRT *xprt) { SVCGROUP *grp = xprt->xp_group; mtx_assert(&grp->sg_lock, MA_OWNED); KASSERT(xprt->xp_registered == TRUE, ("xprt_unregister_locked: not registered")); xprt_inactive_locked(xprt); TAILQ_REMOVE(&grp->sg_xlist, xprt, xp_link); xprt->xp_registered = FALSE; } void xprt_unregister(SVCXPRT *xprt) { SVCGROUP *grp = xprt->xp_group; mtx_lock(&grp->sg_lock); if (xprt->xp_registered == FALSE) { /* Already unregistered by another thread */ mtx_unlock(&grp->sg_lock); return; } xprt_unregister_locked(xprt); mtx_unlock(&grp->sg_lock); SVC_RELEASE(xprt); } /* * Attempt to assign a service thread to this transport. */ static int xprt_assignthread(SVCXPRT *xprt) { SVCGROUP *grp = xprt->xp_group; SVCTHREAD *st; mtx_assert(&grp->sg_lock, MA_OWNED); st = LIST_FIRST(&grp->sg_idlethreads); if (st) { LIST_REMOVE(st, st_ilink); SVC_ACQUIRE(xprt); xprt->xp_thread = st; st->st_xprt = xprt; cv_signal(&st->st_cond); return (TRUE); } else { /* * See if we can create a new thread. The * actual thread creation happens in * svc_run_internal because our locking state * is poorly defined (we are typically called * from a socket upcall). Don't create more * than one thread per second. */ if (grp->sg_state == SVCPOOL_ACTIVE && grp->sg_lastcreatetime < time_uptime && grp->sg_threadcount < grp->sg_maxthreads) { grp->sg_state = SVCPOOL_THREADWANTED; } } return (FALSE); } void xprt_active(SVCXPRT *xprt) { SVCGROUP *grp = xprt->xp_group; mtx_lock(&grp->sg_lock); if (!xprt->xp_registered) { /* * Race with xprt_unregister - we lose. */ mtx_unlock(&grp->sg_lock); return; } if (!xprt->xp_active) { xprt->xp_active = TRUE; if (xprt->xp_thread == NULL) { if (!svc_request_space_available(xprt->xp_pool) || !xprt_assignthread(xprt)) TAILQ_INSERT_TAIL(&grp->sg_active, xprt, xp_alink); } } mtx_unlock(&grp->sg_lock); } void xprt_inactive_locked(SVCXPRT *xprt) { SVCGROUP *grp = xprt->xp_group; mtx_assert(&grp->sg_lock, MA_OWNED); if (xprt->xp_active) { if (xprt->xp_thread == NULL) TAILQ_REMOVE(&grp->sg_active, xprt, xp_alink); xprt->xp_active = FALSE; } } void xprt_inactive(SVCXPRT *xprt) { SVCGROUP *grp = xprt->xp_group; mtx_lock(&grp->sg_lock); xprt_inactive_locked(xprt); mtx_unlock(&grp->sg_lock); } /* * Variant of xprt_inactive() for use only when sure that port is * assigned to thread. For example, withing receive handlers. */ void xprt_inactive_self(SVCXPRT *xprt) { KASSERT(xprt->xp_thread != NULL, ("xprt_inactive_self(%p) with NULL xp_thread", xprt)); xprt->xp_active = FALSE; } /* * Add a service program to the callout list. * The dispatch routine will be called when a rpc request for this * program number comes in. */ bool_t svc_reg(SVCXPRT *xprt, const rpcprog_t prog, const rpcvers_t vers, void (*dispatch)(struct svc_req *, SVCXPRT *), const struct netconfig *nconf) { SVCPOOL *pool = xprt->xp_pool; struct svc_callout *s; char *netid = NULL; int flag = 0; /* VARIABLES PROTECTED BY svc_lock: s, svc_head */ if (xprt->xp_netid) { netid = strdup(xprt->xp_netid, M_RPC); flag = 1; } else if (nconf && nconf->nc_netid) { netid = strdup(nconf->nc_netid, M_RPC); flag = 1; } /* must have been created with svc_raw_create */ if ((netid == NULL) && (flag == 1)) { return (FALSE); } mtx_lock(&pool->sp_lock); if ((s = svc_find(pool, prog, vers, netid)) != NULL) { if (netid) free(netid, M_RPC); if (s->sc_dispatch == dispatch) goto rpcb_it; /* he is registering another xptr */ mtx_unlock(&pool->sp_lock); return (FALSE); } s = malloc(sizeof (struct svc_callout), M_RPC, M_NOWAIT); if (s == NULL) { if (netid) free(netid, M_RPC); mtx_unlock(&pool->sp_lock); return (FALSE); } s->sc_prog = prog; s->sc_vers = vers; s->sc_dispatch = dispatch; s->sc_netid = netid; TAILQ_INSERT_TAIL(&pool->sp_callouts, s, sc_link); if ((xprt->xp_netid == NULL) && (flag == 1) && netid) ((SVCXPRT *) xprt)->xp_netid = strdup(netid, M_RPC); rpcb_it: mtx_unlock(&pool->sp_lock); /* now register the information with the local binder service */ if (nconf) { bool_t dummy; struct netconfig tnc; struct netbuf nb; tnc = *nconf; nb.buf = &xprt->xp_ltaddr; nb.len = xprt->xp_ltaddr.ss_len; dummy = rpcb_set(prog, vers, &tnc, &nb); return (dummy); } return (TRUE); } /* * Remove a service program from the callout list. */ void svc_unreg(SVCPOOL *pool, const rpcprog_t prog, const rpcvers_t vers) { struct svc_callout *s; /* unregister the information anyway */ (void) rpcb_unset(prog, vers, NULL); mtx_lock(&pool->sp_lock); while ((s = svc_find(pool, prog, vers, NULL)) != NULL) { TAILQ_REMOVE(&pool->sp_callouts, s, sc_link); if (s->sc_netid) mem_free(s->sc_netid, sizeof (s->sc_netid) + 1); mem_free(s, sizeof (struct svc_callout)); } mtx_unlock(&pool->sp_lock); } /* * Add a service connection loss program to the callout list. * The dispatch routine will be called when some port in ths pool die. */ bool_t svc_loss_reg(SVCXPRT *xprt, void (*dispatch)(SVCXPRT *)) { SVCPOOL *pool = xprt->xp_pool; struct svc_loss_callout *s; mtx_lock(&pool->sp_lock); TAILQ_FOREACH(s, &pool->sp_lcallouts, slc_link) { if (s->slc_dispatch == dispatch) break; } if (s != NULL) { mtx_unlock(&pool->sp_lock); return (TRUE); } s = malloc(sizeof (struct svc_callout), M_RPC, M_NOWAIT); if (s == NULL) { mtx_unlock(&pool->sp_lock); return (FALSE); } s->slc_dispatch = dispatch; TAILQ_INSERT_TAIL(&pool->sp_lcallouts, s, slc_link); mtx_unlock(&pool->sp_lock); return (TRUE); } /* * Remove a service connection loss program from the callout list. */ void svc_loss_unreg(SVCPOOL *pool, void (*dispatch)(SVCXPRT *)) { struct svc_loss_callout *s; mtx_lock(&pool->sp_lock); TAILQ_FOREACH(s, &pool->sp_lcallouts, slc_link) { if (s->slc_dispatch == dispatch) { TAILQ_REMOVE(&pool->sp_lcallouts, s, slc_link); free(s, M_RPC); break; } } mtx_unlock(&pool->sp_lock); } /* ********************** CALLOUT list related stuff ************* */ /* * Search the callout list for a program number, return the callout * struct. */ static struct svc_callout * svc_find(SVCPOOL *pool, rpcprog_t prog, rpcvers_t vers, char *netid) { struct svc_callout *s; mtx_assert(&pool->sp_lock, MA_OWNED); TAILQ_FOREACH(s, &pool->sp_callouts, sc_link) { if (s->sc_prog == prog && s->sc_vers == vers && (netid == NULL || s->sc_netid == NULL || strcmp(netid, s->sc_netid) == 0)) break; } return (s); } /* ******************* REPLY GENERATION ROUTINES ************ */ static bool_t svc_sendreply_common(struct svc_req *rqstp, struct rpc_msg *rply, struct mbuf *body) { SVCXPRT *xprt = rqstp->rq_xprt; bool_t ok; if (rqstp->rq_args) { m_freem(rqstp->rq_args); rqstp->rq_args = NULL; } if (xprt->xp_pool->sp_rcache) replay_setreply(xprt->xp_pool->sp_rcache, rply, svc_getrpccaller(rqstp), body); if (!SVCAUTH_WRAP(&rqstp->rq_auth, &body)) return (FALSE); ok = SVC_REPLY(xprt, rply, rqstp->rq_addr, body, &rqstp->rq_reply_seq); if (rqstp->rq_addr) { free(rqstp->rq_addr, M_SONAME); rqstp->rq_addr = NULL; } return (ok); } /* * Send a reply to an rpc request */ bool_t svc_sendreply(struct svc_req *rqstp, xdrproc_t xdr_results, void * xdr_location) { struct rpc_msg rply; struct mbuf *m; XDR xdrs; bool_t ok; rply.rm_xid = rqstp->rq_xid; rply.rm_direction = REPLY; rply.rm_reply.rp_stat = MSG_ACCEPTED; rply.acpted_rply.ar_verf = rqstp->rq_verf; rply.acpted_rply.ar_stat = SUCCESS; rply.acpted_rply.ar_results.where = NULL; rply.acpted_rply.ar_results.proc = (xdrproc_t) xdr_void; m = m_getcl(M_WAITOK, MT_DATA, 0); xdrmbuf_create(&xdrs, m, XDR_ENCODE); ok = xdr_results(&xdrs, xdr_location); XDR_DESTROY(&xdrs); if (ok) { return (svc_sendreply_common(rqstp, &rply, m)); } else { m_freem(m); return (FALSE); } } bool_t svc_sendreply_mbuf(struct svc_req *rqstp, struct mbuf *m) { struct rpc_msg rply; rply.rm_xid = rqstp->rq_xid; rply.rm_direction = REPLY; rply.rm_reply.rp_stat = MSG_ACCEPTED; rply.acpted_rply.ar_verf = rqstp->rq_verf; rply.acpted_rply.ar_stat = SUCCESS; rply.acpted_rply.ar_results.where = NULL; rply.acpted_rply.ar_results.proc = (xdrproc_t) xdr_void; return (svc_sendreply_common(rqstp, &rply, m)); } /* * No procedure error reply */ void svcerr_noproc(struct svc_req *rqstp) { SVCXPRT *xprt = rqstp->rq_xprt; struct rpc_msg rply; rply.rm_xid = rqstp->rq_xid; rply.rm_direction = REPLY; rply.rm_reply.rp_stat = MSG_ACCEPTED; rply.acpted_rply.ar_verf = rqstp->rq_verf; rply.acpted_rply.ar_stat = PROC_UNAVAIL; if (xprt->xp_pool->sp_rcache) replay_setreply(xprt->xp_pool->sp_rcache, &rply, svc_getrpccaller(rqstp), NULL); svc_sendreply_common(rqstp, &rply, NULL); } /* * Can't decode args error reply */ void svcerr_decode(struct svc_req *rqstp) { SVCXPRT *xprt = rqstp->rq_xprt; struct rpc_msg rply; rply.rm_xid = rqstp->rq_xid; rply.rm_direction = REPLY; rply.rm_reply.rp_stat = MSG_ACCEPTED; rply.acpted_rply.ar_verf = rqstp->rq_verf; rply.acpted_rply.ar_stat = GARBAGE_ARGS; if (xprt->xp_pool->sp_rcache) replay_setreply(xprt->xp_pool->sp_rcache, &rply, (struct sockaddr *) &xprt->xp_rtaddr, NULL); svc_sendreply_common(rqstp, &rply, NULL); } /* * Some system error */ void svcerr_systemerr(struct svc_req *rqstp) { SVCXPRT *xprt = rqstp->rq_xprt; struct rpc_msg rply; rply.rm_xid = rqstp->rq_xid; rply.rm_direction = REPLY; rply.rm_reply.rp_stat = MSG_ACCEPTED; rply.acpted_rply.ar_verf = rqstp->rq_verf; rply.acpted_rply.ar_stat = SYSTEM_ERR; if (xprt->xp_pool->sp_rcache) replay_setreply(xprt->xp_pool->sp_rcache, &rply, svc_getrpccaller(rqstp), NULL); svc_sendreply_common(rqstp, &rply, NULL); } /* * Authentication error reply */ void svcerr_auth(struct svc_req *rqstp, enum auth_stat why) { SVCXPRT *xprt = rqstp->rq_xprt; struct rpc_msg rply; rply.rm_xid = rqstp->rq_xid; rply.rm_direction = REPLY; rply.rm_reply.rp_stat = MSG_DENIED; rply.rjcted_rply.rj_stat = AUTH_ERROR; rply.rjcted_rply.rj_why = why; if (xprt->xp_pool->sp_rcache) replay_setreply(xprt->xp_pool->sp_rcache, &rply, svc_getrpccaller(rqstp), NULL); svc_sendreply_common(rqstp, &rply, NULL); } /* * Auth too weak error reply */ void svcerr_weakauth(struct svc_req *rqstp) { svcerr_auth(rqstp, AUTH_TOOWEAK); } /* * Program unavailable error reply */ void svcerr_noprog(struct svc_req *rqstp) { SVCXPRT *xprt = rqstp->rq_xprt; struct rpc_msg rply; rply.rm_xid = rqstp->rq_xid; rply.rm_direction = REPLY; rply.rm_reply.rp_stat = MSG_ACCEPTED; rply.acpted_rply.ar_verf = rqstp->rq_verf; rply.acpted_rply.ar_stat = PROG_UNAVAIL; if (xprt->xp_pool->sp_rcache) replay_setreply(xprt->xp_pool->sp_rcache, &rply, svc_getrpccaller(rqstp), NULL); svc_sendreply_common(rqstp, &rply, NULL); } /* * Program version mismatch error reply */ void svcerr_progvers(struct svc_req *rqstp, rpcvers_t low_vers, rpcvers_t high_vers) { SVCXPRT *xprt = rqstp->rq_xprt; struct rpc_msg rply; rply.rm_xid = rqstp->rq_xid; rply.rm_direction = REPLY; rply.rm_reply.rp_stat = MSG_ACCEPTED; rply.acpted_rply.ar_verf = rqstp->rq_verf; rply.acpted_rply.ar_stat = PROG_MISMATCH; rply.acpted_rply.ar_vers.low = (uint32_t)low_vers; rply.acpted_rply.ar_vers.high = (uint32_t)high_vers; if (xprt->xp_pool->sp_rcache) replay_setreply(xprt->xp_pool->sp_rcache, &rply, svc_getrpccaller(rqstp), NULL); svc_sendreply_common(rqstp, &rply, NULL); } /* * Allocate a new server transport structure. All fields are * initialized to zero and xp_p3 is initialized to point at an * extension structure to hold various flags and authentication * parameters. */ SVCXPRT * svc_xprt_alloc() { SVCXPRT *xprt; SVCXPRT_EXT *ext; xprt = mem_alloc(sizeof(SVCXPRT)); memset(xprt, 0, sizeof(SVCXPRT)); ext = mem_alloc(sizeof(SVCXPRT_EXT)); memset(ext, 0, sizeof(SVCXPRT_EXT)); xprt->xp_p3 = ext; refcount_init(&xprt->xp_refs, 1); return (xprt); } /* * Free a server transport structure. */ void svc_xprt_free(xprt) SVCXPRT *xprt; { mem_free(xprt->xp_p3, sizeof(SVCXPRT_EXT)); mem_free(xprt, sizeof(SVCXPRT)); } /* ******************* SERVER INPUT STUFF ******************* */ /* * Read RPC requests from a transport and queue them to be * executed. We handle authentication and replay cache replies here. * Actually dispatching the RPC is deferred till svc_executereq. */ static enum xprt_stat svc_getreq(SVCXPRT *xprt, struct svc_req **rqstp_ret) { SVCPOOL *pool = xprt->xp_pool; struct svc_req *r; struct rpc_msg msg; struct mbuf *args; struct svc_loss_callout *s; enum xprt_stat stat; /* now receive msgs from xprtprt (support batch calls) */ r = malloc(sizeof(*r), M_RPC, M_WAITOK|M_ZERO); msg.rm_call.cb_cred.oa_base = r->rq_credarea; msg.rm_call.cb_verf.oa_base = &r->rq_credarea[MAX_AUTH_BYTES]; r->rq_clntcred = &r->rq_credarea[2*MAX_AUTH_BYTES]; if (SVC_RECV(xprt, &msg, &r->rq_addr, &args)) { enum auth_stat why; /* * Handle replays and authenticate before queuing the * request to be executed. */ SVC_ACQUIRE(xprt); r->rq_xprt = xprt; if (pool->sp_rcache) { struct rpc_msg repmsg; struct mbuf *repbody; enum replay_state rs; rs = replay_find(pool->sp_rcache, &msg, svc_getrpccaller(r), &repmsg, &repbody); switch (rs) { case RS_NEW: break; case RS_DONE: SVC_REPLY(xprt, &repmsg, r->rq_addr, repbody, &r->rq_reply_seq); if (r->rq_addr) { free(r->rq_addr, M_SONAME); r->rq_addr = NULL; } m_freem(args); goto call_done; default: m_freem(args); goto call_done; } } r->rq_xid = msg.rm_xid; r->rq_prog = msg.rm_call.cb_prog; r->rq_vers = msg.rm_call.cb_vers; r->rq_proc = msg.rm_call.cb_proc; r->rq_size = sizeof(*r) + m_length(args, NULL); r->rq_args = args; if ((why = _authenticate(r, &msg)) != AUTH_OK) { /* * RPCSEC_GSS uses this return code * for requests that form part of its * context establishment protocol and * should not be dispatched to the * application. */ if (why != RPCSEC_GSS_NODISPATCH) svcerr_auth(r, why); goto call_done; } if (!SVCAUTH_UNWRAP(&r->rq_auth, &r->rq_args)) { svcerr_decode(r); goto call_done; } /* * Everything checks out, return request to caller. */ *rqstp_ret = r; r = NULL; } call_done: if (r) { svc_freereq(r); r = NULL; } if ((stat = SVC_STAT(xprt)) == XPRT_DIED) { TAILQ_FOREACH(s, &pool->sp_lcallouts, slc_link) (*s->slc_dispatch)(xprt); xprt_unregister(xprt); } return (stat); } static void svc_executereq(struct svc_req *rqstp) { SVCXPRT *xprt = rqstp->rq_xprt; SVCPOOL *pool = xprt->xp_pool; int prog_found; rpcvers_t low_vers; rpcvers_t high_vers; struct svc_callout *s; /* now match message with a registered service*/ prog_found = FALSE; low_vers = (rpcvers_t) -1L; high_vers = (rpcvers_t) 0L; TAILQ_FOREACH(s, &pool->sp_callouts, sc_link) { if (s->sc_prog == rqstp->rq_prog) { if (s->sc_vers == rqstp->rq_vers) { /* * We hand ownership of r to the * dispatch method - they must call * svc_freereq. */ (*s->sc_dispatch)(rqstp, xprt); return; } /* found correct version */ prog_found = TRUE; if (s->sc_vers < low_vers) low_vers = s->sc_vers; if (s->sc_vers > high_vers) high_vers = s->sc_vers; } /* found correct program */ } /* * if we got here, the program or version * is not served ... */ if (prog_found) svcerr_progvers(rqstp, low_vers, high_vers); else svcerr_noprog(rqstp); svc_freereq(rqstp); } static void svc_checkidle(SVCGROUP *grp) { SVCXPRT *xprt, *nxprt; time_t timo; struct svcxprt_list cleanup; TAILQ_INIT(&cleanup); TAILQ_FOREACH_SAFE(xprt, &grp->sg_xlist, xp_link, nxprt) { /* * Only some transports have idle timers. Don't time * something out which is just waking up. */ if (!xprt->xp_idletimeout || xprt->xp_thread) continue; timo = xprt->xp_lastactive + xprt->xp_idletimeout; if (time_uptime > timo) { xprt_unregister_locked(xprt); TAILQ_INSERT_TAIL(&cleanup, xprt, xp_link); } } mtx_unlock(&grp->sg_lock); TAILQ_FOREACH_SAFE(xprt, &cleanup, xp_link, nxprt) { SVC_RELEASE(xprt); } mtx_lock(&grp->sg_lock); } static void svc_assign_waiting_sockets(SVCPOOL *pool) { SVCGROUP *grp; SVCXPRT *xprt; int g; for (g = 0; g < pool->sp_groupcount; g++) { grp = &pool->sp_groups[g]; mtx_lock(&grp->sg_lock); while ((xprt = TAILQ_FIRST(&grp->sg_active)) != NULL) { if (xprt_assignthread(xprt)) TAILQ_REMOVE(&grp->sg_active, xprt, xp_alink); else break; } mtx_unlock(&grp->sg_lock); } } static void svc_change_space_used(SVCPOOL *pool, int delta) { unsigned int value; value = atomic_fetchadd_int(&pool->sp_space_used, delta) + delta; if (delta > 0) { if (value >= pool->sp_space_high && !pool->sp_space_throttled) { pool->sp_space_throttled = TRUE; pool->sp_space_throttle_count++; } if (value > pool->sp_space_used_highest) pool->sp_space_used_highest = value; } else { if (value < pool->sp_space_low && pool->sp_space_throttled) { pool->sp_space_throttled = FALSE; svc_assign_waiting_sockets(pool); } } } static bool_t svc_request_space_available(SVCPOOL *pool) { if (pool->sp_space_throttled) return (FALSE); return (TRUE); } static void svc_run_internal(SVCGROUP *grp, bool_t ismaster) { SVCPOOL *pool = grp->sg_pool; SVCTHREAD *st, *stpref; SVCXPRT *xprt; enum xprt_stat stat; struct svc_req *rqstp; struct proc *p; size_t sz; int error; st = mem_alloc(sizeof(*st)); mtx_init(&st->st_lock, "st_lock", NULL, MTX_DEF); st->st_pool = pool; st->st_xprt = NULL; STAILQ_INIT(&st->st_reqs); cv_init(&st->st_cond, "rpcsvc"); mtx_lock(&grp->sg_lock); /* * If we are a new thread which was spawned to cope with * increased load, set the state back to SVCPOOL_ACTIVE. */ if (grp->sg_state == SVCPOOL_THREADSTARTING) grp->sg_state = SVCPOOL_ACTIVE; while (grp->sg_state != SVCPOOL_CLOSING) { /* * Create new thread if requested. */ if (grp->sg_state == SVCPOOL_THREADWANTED) { grp->sg_state = SVCPOOL_THREADSTARTING; grp->sg_lastcreatetime = time_uptime; mtx_unlock(&grp->sg_lock); svc_new_thread(grp); mtx_lock(&grp->sg_lock); continue; } /* * Check for idle transports once per second. */ if (time_uptime > grp->sg_lastidlecheck) { grp->sg_lastidlecheck = time_uptime; svc_checkidle(grp); } xprt = st->st_xprt; if (!xprt) { /* * Enforce maxthreads count. */ if (grp->sg_threadcount > grp->sg_maxthreads) break; /* * Before sleeping, see if we can find an * active transport which isn't being serviced * by a thread. */ if (svc_request_space_available(pool) && (xprt = TAILQ_FIRST(&grp->sg_active)) != NULL) { TAILQ_REMOVE(&grp->sg_active, xprt, xp_alink); SVC_ACQUIRE(xprt); xprt->xp_thread = st; st->st_xprt = xprt; continue; } LIST_INSERT_HEAD(&grp->sg_idlethreads, st, st_ilink); if (ismaster || (!ismaster && grp->sg_threadcount > grp->sg_minthreads)) error = cv_timedwait_sig(&st->st_cond, &grp->sg_lock, 5 * hz); else error = cv_wait_sig(&st->st_cond, &grp->sg_lock); if (st->st_xprt == NULL) LIST_REMOVE(st, st_ilink); /* * Reduce worker thread count when idle. */ if (error == EWOULDBLOCK) { if (!ismaster && (grp->sg_threadcount > grp->sg_minthreads) && !st->st_xprt) break; } else if (error != 0) { KASSERT(error == EINTR || error == ERESTART, ("non-signal error %d", error)); mtx_unlock(&grp->sg_lock); p = curproc; PROC_LOCK(p); - if (P_SHOULDSTOP(p)) { + if (P_SHOULDSTOP(p) || + (p->p_flag & P_TOTAL_STOP) != 0) { thread_suspend_check(0); PROC_UNLOCK(p); mtx_lock(&grp->sg_lock); } else { PROC_UNLOCK(p); svc_exit(pool); mtx_lock(&grp->sg_lock); break; } } continue; } mtx_unlock(&grp->sg_lock); /* * Drain the transport socket and queue up any RPCs. */ xprt->xp_lastactive = time_uptime; do { if (!svc_request_space_available(pool)) break; rqstp = NULL; stat = svc_getreq(xprt, &rqstp); if (rqstp) { svc_change_space_used(pool, rqstp->rq_size); /* * See if the application has a preference * for some other thread. */ if (pool->sp_assign) { stpref = pool->sp_assign(st, rqstp); rqstp->rq_thread = stpref; STAILQ_INSERT_TAIL(&stpref->st_reqs, rqstp, rq_link); mtx_unlock(&stpref->st_lock); if (stpref != st) rqstp = NULL; } else { rqstp->rq_thread = st; STAILQ_INSERT_TAIL(&st->st_reqs, rqstp, rq_link); } } } while (rqstp == NULL && stat == XPRT_MOREREQS && grp->sg_state != SVCPOOL_CLOSING); /* * Move this transport to the end of the active list to * ensure fairness when multiple transports are active. * If this was the last queued request, svc_getreq will end * up calling xprt_inactive to remove from the active list. */ mtx_lock(&grp->sg_lock); xprt->xp_thread = NULL; st->st_xprt = NULL; if (xprt->xp_active) { if (!svc_request_space_available(pool) || !xprt_assignthread(xprt)) TAILQ_INSERT_TAIL(&grp->sg_active, xprt, xp_alink); } mtx_unlock(&grp->sg_lock); SVC_RELEASE(xprt); /* * Execute what we have queued. */ sz = 0; mtx_lock(&st->st_lock); while ((rqstp = STAILQ_FIRST(&st->st_reqs)) != NULL) { STAILQ_REMOVE_HEAD(&st->st_reqs, rq_link); mtx_unlock(&st->st_lock); sz += rqstp->rq_size; svc_executereq(rqstp); mtx_lock(&st->st_lock); } mtx_unlock(&st->st_lock); svc_change_space_used(pool, -sz); mtx_lock(&grp->sg_lock); } if (st->st_xprt) { xprt = st->st_xprt; st->st_xprt = NULL; SVC_RELEASE(xprt); } KASSERT(STAILQ_EMPTY(&st->st_reqs), ("stray reqs on exit")); mtx_destroy(&st->st_lock); cv_destroy(&st->st_cond); mem_free(st, sizeof(*st)); grp->sg_threadcount--; if (!ismaster) wakeup(grp); mtx_unlock(&grp->sg_lock); } static void svc_thread_start(void *arg) { svc_run_internal((SVCGROUP *) arg, FALSE); kthread_exit(); } static void svc_new_thread(SVCGROUP *grp) { SVCPOOL *pool = grp->sg_pool; struct thread *td; grp->sg_threadcount++; kthread_add(svc_thread_start, grp, pool->sp_proc, &td, 0, 0, "%s: service", pool->sp_name); } void svc_run(SVCPOOL *pool) { int g, i; struct proc *p; struct thread *td; SVCGROUP *grp; p = curproc; td = curthread; snprintf(td->td_name, sizeof(td->td_name), "%s: master", pool->sp_name); pool->sp_state = SVCPOOL_ACTIVE; pool->sp_proc = p; /* Choose group count based on number of threads and CPUs. */ pool->sp_groupcount = max(1, min(SVC_MAXGROUPS, min(pool->sp_maxthreads / 2, mp_ncpus) / 6)); for (g = 0; g < pool->sp_groupcount; g++) { grp = &pool->sp_groups[g]; grp->sg_minthreads = max(1, pool->sp_minthreads / pool->sp_groupcount); grp->sg_maxthreads = max(1, pool->sp_maxthreads / pool->sp_groupcount); grp->sg_lastcreatetime = time_uptime; } /* Starting threads */ for (g = 0; g < pool->sp_groupcount; g++) { grp = &pool->sp_groups[g]; for (i = ((g == 0) ? 1 : 0); i < grp->sg_minthreads; i++) svc_new_thread(grp); } pool->sp_groups[0].sg_threadcount++; svc_run_internal(&pool->sp_groups[0], TRUE); /* Waiting for threads to stop. */ for (g = 0; g < pool->sp_groupcount; g++) { grp = &pool->sp_groups[g]; mtx_lock(&grp->sg_lock); while (grp->sg_threadcount > 0) msleep(grp, &grp->sg_lock, 0, "svcexit", 0); mtx_unlock(&grp->sg_lock); } } void svc_exit(SVCPOOL *pool) { SVCGROUP *grp; SVCTHREAD *st; int g; pool->sp_state = SVCPOOL_CLOSING; for (g = 0; g < pool->sp_groupcount; g++) { grp = &pool->sp_groups[g]; mtx_lock(&grp->sg_lock); if (grp->sg_state != SVCPOOL_CLOSING) { grp->sg_state = SVCPOOL_CLOSING; LIST_FOREACH(st, &grp->sg_idlethreads, st_ilink) cv_signal(&st->st_cond); } mtx_unlock(&grp->sg_lock); } } bool_t svc_getargs(struct svc_req *rqstp, xdrproc_t xargs, void *args) { struct mbuf *m; XDR xdrs; bool_t stat; m = rqstp->rq_args; rqstp->rq_args = NULL; xdrmbuf_create(&xdrs, m, XDR_DECODE); stat = xargs(&xdrs, args); XDR_DESTROY(&xdrs); return (stat); } bool_t svc_freeargs(struct svc_req *rqstp, xdrproc_t xargs, void *args) { XDR xdrs; if (rqstp->rq_addr) { free(rqstp->rq_addr, M_SONAME); rqstp->rq_addr = NULL; } xdrs.x_op = XDR_FREE; return (xargs(&xdrs, args)); } void svc_freereq(struct svc_req *rqstp) { SVCTHREAD *st; SVCPOOL *pool; st = rqstp->rq_thread; if (st) { pool = st->st_pool; if (pool->sp_done) pool->sp_done(st, rqstp); } if (rqstp->rq_auth.svc_ah_ops) SVCAUTH_RELEASE(&rqstp->rq_auth); if (rqstp->rq_xprt) { SVC_RELEASE(rqstp->rq_xprt); } if (rqstp->rq_addr) free(rqstp->rq_addr, M_SONAME); if (rqstp->rq_args) m_freem(rqstp->rq_args); free(rqstp, M_RPC); } Index: projects/clang350-import/sys/sys/bufobj.h =================================================================== --- projects/clang350-import/sys/sys/bufobj.h (revision 275748) +++ projects/clang350-import/sys/sys/bufobj.h (revision 275749) @@ -1,134 +1,135 @@ /*- * Copyright (c) 2004 Poul-Henning Kamp * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * $FreeBSD$ */ /* * Architectural notes: * * bufobj is a new object which is what buffers hang from in the buffer * cache. * * This used to be vnodes, but we need non-vnode code to be able * to use the buffer cache as well, specifically geom classes like gbde, * raid3 and raid5. * * All vnodes will contain a bufobj initially, but down the road we may * want to only allocate bufobjs when they are needed. There could be a * large number of vnodes in the system which wouldn't need a bufobj during * their lifetime. * * The exact relationship to the vmobject is not determined at this point, * it may in fact be that we find them to be two sides of the same object * once things starts to crystalize. */ #ifndef _SYS_BUFOBJ_H_ #define _SYS_BUFOBJ_H_ #if defined(_KERNEL) || defined(_KVM_VNODE) #include #include #include #include struct bufobj; struct buf_ops; extern struct buf_ops buf_ops_bio; TAILQ_HEAD(buflists, buf); /* A Buffer list & trie */ struct bufv { struct buflists bv_hd; /* Sorted blocklist */ struct pctrie bv_root; /* Buf trie */ int bv_cnt; /* Number of buffers */ }; typedef void b_strategy_t(struct bufobj *, struct buf *); typedef int b_write_t(struct buf *); typedef int b_sync_t(struct bufobj *, int waitfor); typedef void b_bdflush_t(struct bufobj *, struct buf *); struct buf_ops { char *bop_name; b_write_t *bop_write; b_strategy_t *bop_strategy; b_sync_t *bop_sync; b_bdflush_t *bop_bdflush; }; #define BO_STRATEGY(bo, bp) ((bo)->bo_ops->bop_strategy((bo), (bp))) #define BO_SYNC(bo, w) ((bo)->bo_ops->bop_sync((bo), (w))) #define BO_WRITE(bo, bp) ((bo)->bo_ops->bop_write((bp))) #define BO_BDFLUSH(bo, bp) ((bo)->bo_ops->bop_bdflush((bo), (bp))) struct bufobj { struct rwlock bo_lock; /* Lock which protects "i" things */ struct buf_ops *bo_ops; /* - Buffer operations */ struct vm_object *bo_object; /* v Place to store VM object */ LIST_ENTRY(bufobj) bo_synclist; /* S dirty vnode list */ void *bo_private; /* private pointer */ struct vnode *__bo_vnode; /* * XXX: This vnode pointer is here * XXX: only to keep the syncer working * XXX: for now. */ struct bufv bo_clean; /* i Clean buffers */ struct bufv bo_dirty; /* i Dirty buffers */ long bo_numoutput; /* i Writes in progress */ u_int bo_flag; /* i Flags */ int bo_bsize; /* - Block size for i/o */ }; /* * XXX BO_ONWORKLST could be replaced with a check for NULL list elements * in v_synclist. */ #define BO_ONWORKLST (1 << 0) /* On syncer work-list */ #define BO_WWAIT (1 << 1) /* Wait for output to complete */ +#define BO_DEAD (1 << 2) /* Dead; only with INVARIANTS */ #define BO_LOCKPTR(bo) (&(bo)->bo_lock) #define BO_LOCK(bo) rw_wlock(BO_LOCKPTR((bo))) #define BO_UNLOCK(bo) rw_wunlock(BO_LOCKPTR((bo))) #define BO_RLOCK(bo) rw_rlock(BO_LOCKPTR((bo))) #define BO_RUNLOCK(bo) rw_runlock(BO_LOCKPTR((bo))) #define ASSERT_BO_WLOCKED(bo) rw_assert(BO_LOCKPTR((bo)), RA_WLOCKED) #define ASSERT_BO_LOCKED(bo) rw_assert(BO_LOCKPTR((bo)), RA_LOCKED) #define ASSERT_BO_UNLOCKED(bo) rw_assert(BO_LOCKPTR((bo)), RA_UNLOCKED) void bufobj_wdrop(struct bufobj *bo); void bufobj_wref(struct bufobj *bo); void bufobj_wrefl(struct bufobj *bo); int bufobj_invalbuf(struct bufobj *bo, int flags, int slpflag, int slptimeo); int bufobj_wwait(struct bufobj *bo, int slpflag, int timeo); int bufsync(struct bufobj *bo, int waitfor); void bufbdflush(struct bufobj *bo, struct buf *bp); #endif /* defined(_KERNEL) || defined(_KVM_VNODE) */ #endif /* _SYS_BUFOBJ_H_ */ Index: projects/clang350-import/sys/sys/libkern.h =================================================================== --- projects/clang350-import/sys/sys/libkern.h (revision 275748) +++ projects/clang350-import/sys/sys/libkern.h (revision 275749) @@ -1,203 +1,204 @@ /*- * Copyright (c) 1992, 1993 * The Regents of the University of California. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 4. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * @(#)libkern.h 8.1 (Berkeley) 6/10/93 * $FreeBSD$ */ #ifndef _SYS_LIBKERN_H_ #define _SYS_LIBKERN_H_ #include #include #ifdef _KERNEL #include #endif #ifndef LIBKERN_INLINE #define LIBKERN_INLINE static __inline #define LIBKERN_BODY #endif /* BCD conversions. */ extern u_char const bcd2bin_data[]; extern u_char const bin2bcd_data[]; extern char const hex2ascii_data[]; #define bcd2bin(bcd) (bcd2bin_data[bcd]) #define bin2bcd(bin) (bin2bcd_data[bin]) #define hex2ascii(hex) (hex2ascii_data[hex]) static __inline int imax(int a, int b) { return (a > b ? a : b); } static __inline int imin(int a, int b) { return (a < b ? a : b); } static __inline long lmax(long a, long b) { return (a > b ? a : b); } static __inline long lmin(long a, long b) { return (a < b ? a : b); } static __inline u_int max(u_int a, u_int b) { return (a > b ? a : b); } static __inline u_int min(u_int a, u_int b) { return (a < b ? a : b); } static __inline quad_t qmax(quad_t a, quad_t b) { return (a > b ? a : b); } static __inline quad_t qmin(quad_t a, quad_t b) { return (a < b ? a : b); } static __inline u_long ulmax(u_long a, u_long b) { return (a > b ? a : b); } static __inline u_long ulmin(u_long a, u_long b) { return (a < b ? a : b); } static __inline off_t omax(off_t a, off_t b) { return (a > b ? a : b); } static __inline off_t omin(off_t a, off_t b) { return (a < b ? a : b); } static __inline int abs(int a) { return (a < 0 ? -a : a); } static __inline long labs(long a) { return (a < 0 ? -a : a); } static __inline quad_t qabs(quad_t a) { return (a < 0 ? -a : a); } #define ARC4_ENTR_NONE 0 /* Don't have entropy yet. */ #define ARC4_ENTR_HAVE 1 /* Have entropy. */ #define ARC4_ENTR_SEED 2 /* Reseeding. */ extern int arc4rand_iniseed_state; /* Prototypes for non-quad routines. */ struct malloc_type; uint32_t arc4random(void); void arc4rand(void *ptr, u_int len, int reseed); int bcmp(const void *, const void *, size_t); +int timingsafe_bcmp(const void *, const void *, size_t); void *bsearch(const void *, const void *, size_t, size_t, int (*)(const void *, const void *)); #ifndef HAVE_INLINE_FFS int ffs(int); #endif #ifndef HAVE_INLINE_FFSL int ffsl(long); #endif #ifndef HAVE_INLINE_FLS int fls(int); #endif #ifndef HAVE_INLINE_FLSL int flsl(long); #endif #ifndef HAVE_INLINE_FLSLL int flsll(long long); #endif int fnmatch(const char *, const char *, int); int locc(int, char *, u_int); void *memchr(const void *s, int c, size_t n); void *memcchr(const void *s, int c, size_t n); int memcmp(const void *b1, const void *b2, size_t len); void *memmem(const void *l, size_t l_len, const void *s, size_t s_len); void qsort(void *base, size_t nmemb, size_t size, int (*compar)(const void *, const void *)); void qsort_r(void *base, size_t nmemb, size_t size, void *thunk, int (*compar)(void *, const void *, const void *)); u_long random(void); int scanc(u_int, const u_char *, const u_char *, int); void srandom(u_long); int strcasecmp(const char *, const char *); char *strcat(char * __restrict, const char * __restrict); char *strchr(const char *, int); int strcmp(const char *, const char *); char *strcpy(char * __restrict, const char * __restrict); size_t strcspn(const char * __restrict, const char * __restrict) __pure; char *strdup(const char *__restrict, struct malloc_type *); char *strndup(const char *__restrict, size_t, struct malloc_type *); size_t strlcat(char *, const char *, size_t); size_t strlcpy(char *, const char *, size_t); size_t strlen(const char *); int strncasecmp(const char *, const char *, size_t); int strncmp(const char *, const char *, size_t); char *strncpy(char * __restrict, const char * __restrict, size_t); size_t strnlen(const char *, size_t); char *strrchr(const char *, int); char *strsep(char **, const char *delim); size_t strspn(const char *, const char *); char *strstr(const char *, const char *); int strvalid(const char *, size_t); extern const uint32_t crc32_tab[]; static __inline uint32_t crc32_raw(const void *buf, size_t size, uint32_t crc) { const uint8_t *p = (const uint8_t *)buf; while (size--) crc = crc32_tab[(crc ^ *p++) & 0xFF] ^ (crc >> 8); return (crc); } static __inline uint32_t crc32(const void *buf, size_t size) { uint32_t crc; crc = crc32_raw(buf, size, ~0U); return (crc ^ ~0U); } uint32_t calculate_crc32c(uint32_t crc32c, const unsigned char *buffer, unsigned int length); LIBKERN_INLINE void *memset(void *, int, size_t); #ifdef LIBKERN_BODY LIBKERN_INLINE void * memset(void *b, int c, size_t len) { char *bb; if (c == 0) bzero(b, len); else for (bb = (char *)b; len--; ) *bb++ = c; return (b); } #endif static __inline char * index(const char *p, int ch) { return (strchr(p, ch)); } static __inline char * rindex(const char *p, int ch) { return (strrchr(p, ch)); } /* fnmatch() return values. */ #define FNM_NOMATCH 1 /* Match failed. */ /* fnmatch() flags. */ #define FNM_NOESCAPE 0x01 /* Disable backslash escaping. */ #define FNM_PATHNAME 0x02 /* Slash must be matched by slash. */ #define FNM_PERIOD 0x04 /* Period must be matched by period. */ #define FNM_LEADING_DIR 0x08 /* Ignore / after Imatch. */ #define FNM_CASEFOLD 0x10 /* Case insensitive search. */ #define FNM_IGNORECASE FNM_CASEFOLD #define FNM_FILE_NAME FNM_PATHNAME #endif /* !_SYS_LIBKERN_H_ */ Index: projects/clang350-import/sys/sys/proc.h =================================================================== --- projects/clang350-import/sys/sys/proc.h (revision 275748) +++ projects/clang350-import/sys/sys/proc.h (revision 275749) @@ -1,1003 +1,1008 @@ /*- * Copyright (c) 1986, 1989, 1991, 1993 * The Regents of the University of California. All rights reserved. * (c) UNIX System Laboratories, Inc. * All or some portions of this file are derived from material licensed * to the University of California by American Telephone and Telegraph * Co. or Unix System Laboratories, Inc. and are reproduced herein with * the permission of UNIX System Laboratories, Inc. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 4. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * @(#)proc.h 8.15 (Berkeley) 5/19/95 * $FreeBSD$ */ #ifndef _SYS_PROC_H_ #define _SYS_PROC_H_ #include /* For struct callout. */ #include /* For struct klist. */ #include #ifndef _KERNEL #include #endif #include #include #include #include #include #include #include /* XXX. */ #include #include #include #include #include #ifndef _KERNEL #include /* For structs itimerval, timeval. */ #else #include #endif #include #include #include /* Machine-dependent proc substruct. */ /* * One structure allocated per session. * * List of locks * (m) locked by s_mtx mtx * (e) locked by proctree_lock sx * (c) const until freeing */ struct session { u_int s_count; /* Ref cnt; pgrps in session - atomic. */ struct proc *s_leader; /* (m + e) Session leader. */ struct vnode *s_ttyvp; /* (m) Vnode of controlling tty. */ struct cdev_priv *s_ttydp; /* (m) Device of controlling tty. */ struct tty *s_ttyp; /* (e) Controlling tty. */ pid_t s_sid; /* (c) Session ID. */ /* (m) Setlogin() name: */ char s_login[roundup(MAXLOGNAME, sizeof(long))]; struct mtx s_mtx; /* Mutex to protect members. */ }; /* * One structure allocated per process group. * * List of locks * (m) locked by pg_mtx mtx * (e) locked by proctree_lock sx * (c) const until freeing */ struct pgrp { LIST_ENTRY(pgrp) pg_hash; /* (e) Hash chain. */ LIST_HEAD(, proc) pg_members; /* (m + e) Pointer to pgrp members. */ struct session *pg_session; /* (c) Pointer to session. */ struct sigiolst pg_sigiolst; /* (m) List of sigio sources. */ pid_t pg_id; /* (c) Process group id. */ int pg_jobc; /* (m) Job control process count. */ struct mtx pg_mtx; /* Mutex to protect members */ }; /* * pargs, used to hold a copy of the command line, if it had a sane length. */ struct pargs { u_int ar_ref; /* Reference count. */ u_int ar_length; /* Length. */ u_char ar_args[1]; /* Arguments. */ }; /*- * Description of a process. * * This structure contains the information needed to manage a thread of * control, known in UN*X as a process; it has references to substructures * containing descriptions of things that the process uses, but may share * with related processes. The process structure and the substructures * are always addressable except for those marked "(CPU)" below, * which might be addressable only on a processor on which the process * is running. * * Below is a key of locks used to protect each member of struct proc. The * lock is indicated by a reference to a specific character in parens in the * associated comment. * * - not yet protected * a - only touched by curproc or parent during fork/wait * b - created at fork, never changes * (exception aiods switch vmspaces, but they are also * marked 'P_SYSTEM' so hopefully it will be left alone) * c - locked by proc mtx * d - locked by allproc_lock lock * e - locked by proctree_lock lock * f - session mtx * g - process group mtx * h - callout_lock mtx * i - by curproc or the master session mtx * j - locked by proc slock * k - only accessed by curthread * k*- only accessed by curthread and from an interrupt * l - the attaching proc or attaching proc parent * m - Giant * n - not locked, lazy * o - ktrace lock * q - td_contested lock * r - p_peers lock * t - thread lock * u - process stat lock * w - process timer lock * x - created at fork, only changes during single threading in exec * y - created at first aio, doesn't change until exit or exec at which * point we are single-threaded and only curthread changes it * z - zombie threads lock * * If the locking key specifies two identifiers (for example, p_pptr) then * either lock is sufficient for read access, but both locks must be held * for write access. */ struct cpuset; struct kaioinfo; struct kaudit_record; struct kdtrace_proc; struct kdtrace_thread; struct mqueue_notifier; struct nlminfo; struct p_sched; struct proc; struct procdesc; struct racct; struct sbuf; struct sleepqueue; struct td_sched; struct thread; struct trapframe; struct turnstile; /* * XXX: Does this belong in resource.h or resourcevar.h instead? * Resource usage extension. The times in rusage structs in the kernel are * never up to date. The actual times are kept as runtimes and tick counts * (with control info in the "previous" times), and are converted when * userland asks for rusage info. Backwards compatibility prevents putting * this directly in the user-visible rusage struct. * * Locking for p_rux: (cu) means (u) for p_rux and (c) for p_crux. * Locking for td_rux: (t) for all fields. */ struct rusage_ext { uint64_t rux_runtime; /* (cu) Real time. */ uint64_t rux_uticks; /* (cu) Statclock hits in user mode. */ uint64_t rux_sticks; /* (cu) Statclock hits in sys mode. */ uint64_t rux_iticks; /* (cu) Statclock hits in intr mode. */ uint64_t rux_uu; /* (c) Previous user time in usec. */ uint64_t rux_su; /* (c) Previous sys time in usec. */ uint64_t rux_tu; /* (c) Previous total time in usec. */ }; /* * Kernel runnable context (thread). * This is what is put to sleep and reactivated. * Thread context. Processes may have multiple threads. */ struct thread { struct mtx *volatile td_lock; /* replaces sched lock */ struct proc *td_proc; /* (*) Associated process. */ TAILQ_ENTRY(thread) td_plist; /* (*) All threads in this proc. */ TAILQ_ENTRY(thread) td_runq; /* (t) Run queue. */ TAILQ_ENTRY(thread) td_slpq; /* (t) Sleep queue. */ TAILQ_ENTRY(thread) td_lockq; /* (t) Lock queue. */ LIST_ENTRY(thread) td_hash; /* (d) Hash chain. */ struct cpuset *td_cpuset; /* (t) CPU affinity mask. */ struct seltd *td_sel; /* Select queue/channel. */ struct sleepqueue *td_sleepqueue; /* (k) Associated sleep queue. */ struct turnstile *td_turnstile; /* (k) Associated turnstile. */ struct rl_q_entry *td_rlqe; /* (k) Associated range lock entry. */ struct umtx_q *td_umtxq; /* (c?) Link for when we're blocked. */ lwpid_t td_tid; /* (b) Thread ID. */ sigqueue_t td_sigqueue; /* (c) Sigs arrived, not delivered. */ #define td_siglist td_sigqueue.sq_signals u_char td_lend_user_pri; /* (t) Lend user pri. */ /* Cleared during fork1() */ #define td_startzero td_flags int td_flags; /* (t) TDF_* flags. */ int td_inhibitors; /* (t) Why can not run. */ int td_pflags; /* (k) Private thread (TDP_*) flags. */ int td_dupfd; /* (k) Ret value from fdopen. XXX */ int td_sqqueue; /* (t) Sleepqueue queue blocked on. */ void *td_wchan; /* (t) Sleep address. */ const char *td_wmesg; /* (t) Reason for sleep. */ int td_lastcpu; /* (t) Last cpu we were on. */ int td_oncpu; /* (t) Which cpu we are on. */ volatile u_char td_owepreempt; /* (k*) Preempt on last critical_exit */ u_char td_tsqueue; /* (t) Turnstile queue blocked on. */ short td_locks; /* (k) Count of non-spin locks. */ short td_rw_rlocks; /* (k) Count of rwlock read locks. */ short td_lk_slocks; /* (k) Count of lockmgr shared locks. */ short td_stopsched; /* (k) Scheduler stopped. */ struct turnstile *td_blocked; /* (t) Lock thread is blocked on. */ const char *td_lockname; /* (t) Name of lock blocked on. */ LIST_HEAD(, turnstile) td_contested; /* (q) Contested locks. */ struct lock_list_entry *td_sleeplocks; /* (k) Held sleep locks. */ int td_intr_nesting_level; /* (k) Interrupt recursion. */ int td_pinned; /* (k) Temporary cpu pin count. */ struct ucred *td_ucred; /* (k) Reference to credentials. */ u_int td_estcpu; /* (t) estimated cpu utilization */ int td_slptick; /* (t) Time at sleep. */ int td_blktick; /* (t) Time spent blocked. */ int td_swvoltick; /* (t) Time at last SW_VOL switch. */ u_int td_cow; /* (*) Number of copy-on-write faults */ struct rusage td_ru; /* (t) rusage information. */ struct rusage_ext td_rux; /* (t) Internal rusage information. */ uint64_t td_incruntime; /* (t) Cpu ticks to transfer to proc. */ uint64_t td_runtime; /* (t) How many cpu ticks we've run. */ u_int td_pticks; /* (t) Statclock hits for profiling */ u_int td_sticks; /* (t) Statclock hits in system mode. */ u_int td_iticks; /* (t) Statclock hits in intr mode. */ u_int td_uticks; /* (t) Statclock hits in user mode. */ int td_intrval; /* (t) Return value for sleepq. */ sigset_t td_oldsigmask; /* (k) Saved mask from pre sigpause. */ volatile u_int td_generation; /* (k) For detection of preemption */ stack_t td_sigstk; /* (k) Stack ptr and on-stack flag. */ int td_xsig; /* (c) Signal for ptrace */ u_long td_profil_addr; /* (k) Temporary addr until AST. */ u_int td_profil_ticks; /* (k) Temporary ticks until AST. */ char td_name[MAXCOMLEN + 1]; /* (*) Thread name. */ struct file *td_fpop; /* (k) file referencing cdev under op */ int td_dbgflags; /* (c) Userland debugger flags */ struct ksiginfo td_dbgksi; /* (c) ksi reflected to debugger. */ int td_ng_outbound; /* (k) Thread entered ng from above. */ struct osd td_osd; /* (k) Object specific data. */ struct vm_map_entry *td_map_def_user; /* (k) Deferred entries. */ pid_t td_dbg_forked; /* (c) Child pid for debugger. */ u_int td_vp_reserv; /* (k) Count of reserved vnodes. */ int td_no_sleeping; /* (k) Sleeping disabled count. */ int td_dom_rr_idx; /* (k) RR Numa domain selection. */ #define td_endzero td_sigmask /* Copied during fork1() or create_thread(). */ #define td_startcopy td_endzero sigset_t td_sigmask; /* (c) Current signal mask. */ u_char td_rqindex; /* (t) Run queue index. */ u_char td_base_pri; /* (t) Thread base kernel priority. */ u_char td_priority; /* (t) Thread active priority. */ u_char td_pri_class; /* (t) Scheduling class. */ u_char td_user_pri; /* (t) User pri from estcpu and nice. */ u_char td_base_user_pri; /* (t) Base user pri */ #define td_endcopy td_pcb /* * Fields that must be manually set in fork1() or create_thread() * or already have been set in the allocator, constructor, etc. */ struct pcb *td_pcb; /* (k) Kernel VA of pcb and kstack. */ enum { TDS_INACTIVE = 0x0, TDS_INHIBITED, TDS_CAN_RUN, TDS_RUNQ, TDS_RUNNING } td_state; /* (t) thread state */ union { register_t tdu_retval[2]; off_t tdu_off; } td_uretoff; /* (k) Syscall aux returns. */ #define td_retval td_uretoff.tdu_retval struct callout td_slpcallout; /* (h) Callout for sleep. */ struct trapframe *td_frame; /* (k) */ struct vm_object *td_kstack_obj;/* (a) Kstack object. */ vm_offset_t td_kstack; /* (a) Kernel VA of kstack. */ int td_kstack_pages; /* (a) Size of the kstack. */ volatile u_int td_critnest; /* (k*) Critical section nest level. */ struct mdthread td_md; /* (k) Any machine-dependent fields. */ struct td_sched *td_sched; /* (*) Scheduler-specific data. */ struct kaudit_record *td_ar; /* (k) Active audit record, if any. */ struct lpohead td_lprof[2]; /* (a) lock profiling objects. */ struct kdtrace_thread *td_dtrace; /* (*) DTrace-specific data. */ int td_errno; /* Error returned by last syscall. */ struct vnet *td_vnet; /* (k) Effective vnet. */ const char *td_vnet_lpush; /* (k) Debugging vnet push / pop. */ struct trapframe *td_intr_frame;/* (k) Frame of the current irq */ struct proc *td_rfppwait_p; /* (k) The vforked child */ struct vm_page **td_ma; /* (k) uio pages held */ int td_ma_cnt; /* (k) size of *td_ma */ }; struct mtx *thread_lock_block(struct thread *); void thread_lock_unblock(struct thread *, struct mtx *); void thread_lock_set(struct thread *, struct mtx *); #define THREAD_LOCK_ASSERT(td, type) \ do { \ struct mtx *__m = (td)->td_lock; \ if (__m != &blocked_lock) \ mtx_assert(__m, (type)); \ } while (0) #ifdef INVARIANTS #define THREAD_LOCKPTR_ASSERT(td, lock) \ do { \ struct mtx *__m = (td)->td_lock; \ KASSERT((__m == &blocked_lock || __m == (lock)), \ ("Thread %p lock %p does not match %p", td, __m, (lock))); \ } while (0) #else #define THREAD_LOCKPTR_ASSERT(td, lock) #endif /* * Flags kept in td_flags: * To change these you MUST have the scheduler lock. */ #define TDF_BORROWING 0x00000001 /* Thread is borrowing pri from another. */ #define TDF_INPANIC 0x00000002 /* Caused a panic, let it drive crashdump. */ #define TDF_INMEM 0x00000004 /* Thread's stack is in memory. */ #define TDF_SINTR 0x00000008 /* Sleep is interruptible. */ #define TDF_TIMEOUT 0x00000010 /* Timing out during sleep. */ #define TDF_IDLETD 0x00000020 /* This is a per-CPU idle thread. */ #define TDF_CANSWAP 0x00000040 /* Thread can be swapped. */ #define TDF_SLEEPABORT 0x00000080 /* sleepq_abort was called. */ #define TDF_KTH_SUSP 0x00000100 /* kthread is suspended */ -#define TDF_UNUSED09 0x00000200 /* --available-- */ +#define TDF_ALLPROCSUSP 0x00000200 /* suspended by SINGLE_ALLPROC */ #define TDF_BOUNDARY 0x00000400 /* Thread suspended at user boundary */ #define TDF_ASTPENDING 0x00000800 /* Thread has some asynchronous events. */ #define TDF_TIMOFAIL 0x00001000 /* Timeout from sleep after we were awake. */ #define TDF_SBDRY 0x00002000 /* Stop only on usermode boundary. */ #define TDF_UPIBLOCKED 0x00004000 /* Thread blocked on user PI mutex. */ #define TDF_NEEDSUSPCHK 0x00008000 /* Thread may need to suspend. */ #define TDF_NEEDRESCHED 0x00010000 /* Thread needs to yield. */ #define TDF_NEEDSIGCHK 0x00020000 /* Thread may need signal delivery. */ #define TDF_NOLOAD 0x00040000 /* Ignore during load avg calculations. */ #define TDF_UNUSED19 0x00080000 /* --available-- */ #define TDF_THRWAKEUP 0x00100000 /* Libthr thread must not suspend itself. */ #define TDF_UNUSED21 0x00200000 /* --available-- */ #define TDF_SWAPINREQ 0x00400000 /* Swapin request due to wakeup. */ #define TDF_UNUSED23 0x00800000 /* --available-- */ #define TDF_SCHED0 0x01000000 /* Reserved for scheduler private use */ #define TDF_SCHED1 0x02000000 /* Reserved for scheduler private use */ #define TDF_SCHED2 0x04000000 /* Reserved for scheduler private use */ #define TDF_SCHED3 0x08000000 /* Reserved for scheduler private use */ #define TDF_ALRMPEND 0x10000000 /* Pending SIGVTALRM needs to be posted. */ #define TDF_PROFPEND 0x20000000 /* Pending SIGPROF needs to be posted. */ #define TDF_MACPEND 0x40000000 /* AST-based MAC event pending. */ /* Userland debug flags */ #define TDB_SUSPEND 0x00000001 /* Thread is suspended by debugger */ #define TDB_XSIG 0x00000002 /* Thread is exchanging signal under trace */ #define TDB_USERWR 0x00000004 /* Debugger modified memory or registers */ #define TDB_SCE 0x00000008 /* Thread performs syscall enter */ #define TDB_SCX 0x00000010 /* Thread performs syscall exit */ #define TDB_EXEC 0x00000020 /* TDB_SCX from exec(2) family */ #define TDB_FORK 0x00000040 /* TDB_SCX from fork(2) that created new process */ #define TDB_STOPATFORK 0x00000080 /* Stop at the return from fork (child only) */ #define TDB_CHILD 0x00000100 /* New child indicator for ptrace() */ /* * "Private" flags kept in td_pflags: * These are only written by curthread and thus need no locking. */ #define TDP_OLDMASK 0x00000001 /* Need to restore mask after suspend. */ #define TDP_INKTR 0x00000002 /* Thread is currently in KTR code. */ #define TDP_INKTRACE 0x00000004 /* Thread is currently in KTRACE code. */ #define TDP_BUFNEED 0x00000008 /* Do not recurse into the buf flush */ #define TDP_COWINPROGRESS 0x00000010 /* Snapshot copy-on-write in progress. */ #define TDP_ALTSTACK 0x00000020 /* Have alternate signal stack. */ #define TDP_DEADLKTREAT 0x00000040 /* Lock aquisition - deadlock treatment. */ #define TDP_NOFAULTING 0x00000080 /* Do not handle page faults. */ #define TDP_UNUSED9 0x00000100 /* --available-- */ #define TDP_OWEUPC 0x00000200 /* Call addupc() at next AST. */ #define TDP_ITHREAD 0x00000400 /* Thread is an interrupt thread. */ #define TDP_SYNCIO 0x00000800 /* Local override, disable async i/o. */ #define TDP_SCHED1 0x00001000 /* Reserved for scheduler private use */ #define TDP_SCHED2 0x00002000 /* Reserved for scheduler private use */ #define TDP_SCHED3 0x00004000 /* Reserved for scheduler private use */ #define TDP_SCHED4 0x00008000 /* Reserved for scheduler private use */ #define TDP_GEOM 0x00010000 /* Settle GEOM before finishing syscall */ #define TDP_SOFTDEP 0x00020000 /* Stuck processing softdep worklist */ #define TDP_NORUNNINGBUF 0x00040000 /* Ignore runningbufspace check */ #define TDP_WAKEUP 0x00080000 /* Don't sleep in umtx cond_wait */ #define TDP_INBDFLUSH 0x00100000 /* Already in BO_BDFLUSH, do not recurse */ #define TDP_KTHREAD 0x00200000 /* This is an official kernel thread */ #define TDP_CALLCHAIN 0x00400000 /* Capture thread's callchain */ #define TDP_IGNSUSP 0x00800000 /* Permission to ignore the MNTK_SUSPEND* */ #define TDP_AUDITREC 0x01000000 /* Audit record pending on thread */ #define TDP_RFPPWAIT 0x02000000 /* Handle RFPPWAIT on syscall exit */ #define TDP_RESETSPUR 0x04000000 /* Reset spurious page fault history. */ #define TDP_NERRNO 0x08000000 /* Last errno is already in td_errno */ #define TDP_UIOHELD 0x10000000 /* Current uio has pages held in td_ma */ #define TDP_DEVMEMIO 0x20000000 /* Accessing memory for /dev/mem */ #define TDP_EXECVMSPC 0x40000000 /* Execve destroyed old vmspace */ /* * Reasons that the current thread can not be run yet. * More than one may apply. */ #define TDI_SUSPENDED 0x0001 /* On suspension queue. */ #define TDI_SLEEPING 0x0002 /* Actually asleep! (tricky). */ #define TDI_SWAPPED 0x0004 /* Stack not in mem. Bad juju if run. */ #define TDI_LOCK 0x0008 /* Stopped on a lock. */ #define TDI_IWAIT 0x0010 /* Awaiting interrupt. */ #define TD_IS_SLEEPING(td) ((td)->td_inhibitors & TDI_SLEEPING) #define TD_ON_SLEEPQ(td) ((td)->td_wchan != NULL) #define TD_IS_SUSPENDED(td) ((td)->td_inhibitors & TDI_SUSPENDED) #define TD_IS_SWAPPED(td) ((td)->td_inhibitors & TDI_SWAPPED) #define TD_ON_LOCK(td) ((td)->td_inhibitors & TDI_LOCK) #define TD_AWAITING_INTR(td) ((td)->td_inhibitors & TDI_IWAIT) #define TD_IS_RUNNING(td) ((td)->td_state == TDS_RUNNING) #define TD_ON_RUNQ(td) ((td)->td_state == TDS_RUNQ) #define TD_CAN_RUN(td) ((td)->td_state == TDS_CAN_RUN) #define TD_IS_INHIBITED(td) ((td)->td_state == TDS_INHIBITED) #define TD_ON_UPILOCK(td) ((td)->td_flags & TDF_UPIBLOCKED) #define TD_IS_IDLETHREAD(td) ((td)->td_flags & TDF_IDLETD) #define TD_SET_INHIB(td, inhib) do { \ (td)->td_state = TDS_INHIBITED; \ (td)->td_inhibitors |= (inhib); \ } while (0) #define TD_CLR_INHIB(td, inhib) do { \ if (((td)->td_inhibitors & (inhib)) && \ (((td)->td_inhibitors &= ~(inhib)) == 0)) \ (td)->td_state = TDS_CAN_RUN; \ } while (0) #define TD_SET_SLEEPING(td) TD_SET_INHIB((td), TDI_SLEEPING) #define TD_SET_SWAPPED(td) TD_SET_INHIB((td), TDI_SWAPPED) #define TD_SET_LOCK(td) TD_SET_INHIB((td), TDI_LOCK) #define TD_SET_SUSPENDED(td) TD_SET_INHIB((td), TDI_SUSPENDED) #define TD_SET_IWAIT(td) TD_SET_INHIB((td), TDI_IWAIT) #define TD_SET_EXITING(td) TD_SET_INHIB((td), TDI_EXITING) #define TD_CLR_SLEEPING(td) TD_CLR_INHIB((td), TDI_SLEEPING) #define TD_CLR_SWAPPED(td) TD_CLR_INHIB((td), TDI_SWAPPED) #define TD_CLR_LOCK(td) TD_CLR_INHIB((td), TDI_LOCK) #define TD_CLR_SUSPENDED(td) TD_CLR_INHIB((td), TDI_SUSPENDED) #define TD_CLR_IWAIT(td) TD_CLR_INHIB((td), TDI_IWAIT) #define TD_SET_RUNNING(td) (td)->td_state = TDS_RUNNING #define TD_SET_RUNQ(td) (td)->td_state = TDS_RUNQ #define TD_SET_CAN_RUN(td) (td)->td_state = TDS_CAN_RUN /* * Process structure. */ struct proc { LIST_ENTRY(proc) p_list; /* (d) List of all processes. */ TAILQ_HEAD(, thread) p_threads; /* (c) all threads. */ struct mtx p_slock; /* process spin lock */ struct ucred *p_ucred; /* (c) Process owner's identity. */ struct filedesc *p_fd; /* (b) Open files. */ struct filedesc_to_leader *p_fdtol; /* (b) Tracking node */ struct pstats *p_stats; /* (b) Accounting/statistics (CPU). */ struct plimit *p_limit; /* (c) Process limits. */ struct callout p_limco; /* (c) Limit callout handle */ struct sigacts *p_sigacts; /* (x) Signal actions, state (CPU). */ int p_flag; /* (c) P_* flags. */ int p_flag2; /* (c) P2_* flags. */ enum { PRS_NEW = 0, /* In creation */ PRS_NORMAL, /* threads can be run. */ PRS_ZOMBIE } p_state; /* (j/c) Process status. */ pid_t p_pid; /* (b) Process identifier. */ LIST_ENTRY(proc) p_hash; /* (d) Hash chain. */ LIST_ENTRY(proc) p_pglist; /* (g + e) List of processes in pgrp. */ struct proc *p_pptr; /* (c + e) Pointer to parent process. */ LIST_ENTRY(proc) p_sibling; /* (e) List of sibling processes. */ LIST_HEAD(, proc) p_children; /* (e) Pointer to list of children. */ struct mtx p_mtx; /* (n) Lock for this struct. */ struct mtx p_statmtx; /* Lock for the stats */ struct mtx p_itimmtx; /* Lock for the virt/prof timers */ struct mtx p_profmtx; /* Lock for the profiling */ struct ksiginfo *p_ksi; /* Locked by parent proc lock */ sigqueue_t p_sigqueue; /* (c) Sigs not delivered to a td. */ #define p_siglist p_sigqueue.sq_signals /* The following fields are all zeroed upon creation in fork. */ #define p_startzero p_oppid pid_t p_oppid; /* (c + e) Save ppid in ptrace. XXX */ struct vmspace *p_vmspace; /* (b) Address space. */ u_int p_swtick; /* (c) Tick when swapped in or out. */ struct itimerval p_realtimer; /* (c) Alarm timer. */ struct rusage p_ru; /* (a) Exit information. */ struct rusage_ext p_rux; /* (cu) Internal resource usage. */ struct rusage_ext p_crux; /* (c) Internal child resource usage. */ int p_profthreads; /* (c) Num threads in addupc_task. */ volatile int p_exitthreads; /* (j) Number of threads exiting */ int p_traceflag; /* (o) Kernel trace points. */ struct vnode *p_tracevp; /* (c + o) Trace to vnode. */ struct ucred *p_tracecred; /* (o) Credentials to trace with. */ struct vnode *p_textvp; /* (b) Vnode of executable. */ u_int p_lock; /* (c) Proclock (prevent swap) count. */ struct sigiolst p_sigiolst; /* (c) List of sigio sources. */ int p_sigparent; /* (c) Signal to parent on exit. */ int p_sig; /* (n) For core dump/debugger XXX. */ u_long p_code; /* (n) For core dump/debugger XXX. */ u_int p_stops; /* (c) Stop event bitmask. */ u_int p_stype; /* (c) Stop event type. */ char p_step; /* (c) Process is stopped. */ u_char p_pfsflags; /* (c) Procfs flags. */ struct nlminfo *p_nlminfo; /* (?) Only used by/for lockd. */ struct kaioinfo *p_aioinfo; /* (y) ASYNC I/O info. */ struct thread *p_singlethread;/* (c + j) If single threading this is it */ int p_suspcount; /* (j) Num threads in suspended mode. */ struct thread *p_xthread; /* (c) Trap thread */ int p_boundary_count;/* (j) Num threads at user boundary */ int p_pendingcnt; /* how many signals are pending */ struct itimers *p_itimers; /* (c) POSIX interval timers. */ struct procdesc *p_procdesc; /* (e) Process descriptor, if any. */ u_int p_treeflag; /* (e) P_TREE flags */ /* End area that is zeroed on creation. */ #define p_endzero p_magic /* The following fields are all copied upon creation in fork. */ #define p_startcopy p_endzero u_int p_magic; /* (b) Magic number. */ int p_osrel; /* (x) osreldate for the binary (from ELF note, if any) */ char p_comm[MAXCOMLEN + 1]; /* (b) Process name. */ struct pgrp *p_pgrp; /* (c + e) Pointer to process group. */ struct sysentvec *p_sysent; /* (b) Syscall dispatch info. */ struct pargs *p_args; /* (c) Process arguments. */ rlim_t p_cpulimit; /* (c) Current CPU limit in seconds. */ signed char p_nice; /* (c) Process "nice" value. */ int p_fibnum; /* in this routing domain XXX MRT */ /* End area that is copied on creation. */ #define p_endcopy p_xstat u_short p_xstat; /* (c) Exit status; also stop sig. */ struct knlist p_klist; /* (c) Knotes attached to this proc. */ int p_numthreads; /* (c) Number of threads. */ struct mdproc p_md; /* Any machine-dependent fields. */ struct callout p_itcallout; /* (h + c) Interval timer callout. */ u_short p_acflag; /* (c) Accounting flags. */ struct proc *p_peers; /* (r) */ struct proc *p_leader; /* (b) */ void *p_emuldata; /* (c) Emulator state data. */ struct label *p_label; /* (*) Proc (not subject) MAC label. */ struct p_sched *p_sched; /* (*) Scheduler-specific data. */ STAILQ_HEAD(, ktr_request) p_ktr; /* (o) KTR event queue. */ LIST_HEAD(, mqueue_notifier) p_mqnotifier; /* (c) mqueue notifiers.*/ struct kdtrace_proc *p_dtrace; /* (*) DTrace-specific data. */ struct cv p_pwait; /* (*) wait cv for exit/exec. */ struct cv p_dbgwait; /* (*) wait cv for debugger attach after fork. */ uint64_t p_prev_runtime; /* (c) Resource usage accounting. */ struct racct *p_racct; /* (b) Resource accounting. */ u_char p_throttled; /* (c) Flag for racct pcpu throttling */ /* * An orphan is the child that has beed re-parented to the * debugger as a result of attaching to it. Need to keep * track of them for parent to be able to collect the exit * status of what used to be children. */ LIST_ENTRY(proc) p_orphan; /* (e) List of orphan processes. */ LIST_HEAD(, proc) p_orphans; /* (e) Pointer to list of orphans. */ }; #define p_session p_pgrp->pg_session #define p_pgid p_pgrp->pg_id #define NOCPU (-1) /* For when we aren't on a CPU. */ #define NOCPU_OLD (255) #define MAXCPU_OLD (254) #define PROC_SLOCK(p) mtx_lock_spin(&(p)->p_slock) #define PROC_SUNLOCK(p) mtx_unlock_spin(&(p)->p_slock) #define PROC_SLOCK_ASSERT(p, type) mtx_assert(&(p)->p_slock, (type)) #define PROC_STATLOCK(p) mtx_lock_spin(&(p)->p_statmtx) #define PROC_STATUNLOCK(p) mtx_unlock_spin(&(p)->p_statmtx) #define PROC_STATLOCK_ASSERT(p, type) mtx_assert(&(p)->p_statmtx, (type)) #define PROC_ITIMLOCK(p) mtx_lock_spin(&(p)->p_itimmtx) #define PROC_ITIMUNLOCK(p) mtx_unlock_spin(&(p)->p_itimmtx) #define PROC_ITIMLOCK_ASSERT(p, type) mtx_assert(&(p)->p_itimmtx, (type)) #define PROC_PROFLOCK(p) mtx_lock_spin(&(p)->p_profmtx) #define PROC_PROFUNLOCK(p) mtx_unlock_spin(&(p)->p_profmtx) #define PROC_PROFLOCK_ASSERT(p, type) mtx_assert(&(p)->p_profmtx, (type)) /* These flags are kept in p_flag. */ #define P_ADVLOCK 0x00001 /* Process may hold a POSIX advisory lock. */ #define P_CONTROLT 0x00002 /* Has a controlling terminal. */ #define P_KTHREAD 0x00004 /* Kernel thread (*). */ #define P_FOLLOWFORK 0x00008 /* Attach parent debugger to children. */ #define P_PPWAIT 0x00010 /* Parent is waiting for child to exec/exit. */ #define P_PROFIL 0x00020 /* Has started profiling. */ #define P_STOPPROF 0x00040 /* Has thread requesting to stop profiling. */ #define P_HADTHREADS 0x00080 /* Has had threads (no cleanup shortcuts) */ #define P_SUGID 0x00100 /* Had set id privileges since last exec. */ #define P_SYSTEM 0x00200 /* System proc: no sigs, stats or swapping. */ #define P_SINGLE_EXIT 0x00400 /* Threads suspending should exit, not wait. */ #define P_TRACED 0x00800 /* Debugged process being traced. */ #define P_WAITED 0x01000 /* Someone is waiting for us. */ #define P_WEXIT 0x02000 /* Working on exiting. */ #define P_EXEC 0x04000 /* Process called exec. */ #define P_WKILLED 0x08000 /* Killed, go to kernel/user boundary ASAP. */ #define P_CONTINUED 0x10000 /* Proc has continued from a stopped state. */ #define P_STOPPED_SIG 0x20000 /* Stopped due to SIGSTOP/SIGTSTP. */ #define P_STOPPED_TRACE 0x40000 /* Stopped because of tracing. */ #define P_STOPPED_SINGLE 0x80000 /* Only 1 thread can continue (not to user). */ #define P_PROTECTED 0x100000 /* Do not kill on memory overcommit. */ #define P_SIGEVENT 0x200000 /* Process pending signals changed. */ #define P_SINGLE_BOUNDARY 0x400000 /* Threads should suspend at user boundary. */ #define P_HWPMC 0x800000 /* Process is using HWPMCs */ #define P_JAILED 0x1000000 /* Process is in jail. */ -#define P_UNUSED1 0x2000000 +#define P_TOTAL_STOP 0x2000000 /* Stopped in proc_stop_total. */ #define P_INEXEC 0x4000000 /* Process is in execve(). */ #define P_STATCHILD 0x8000000 /* Child process stopped or exited. */ #define P_INMEM 0x10000000 /* Loaded into memory. */ #define P_SWAPPINGOUT 0x20000000 /* Process is being swapped out. */ #define P_SWAPPINGIN 0x40000000 /* Process is being swapped in. */ #define P_PPTRACE 0x80000000 /* PT_TRACEME by vforked child. */ #define P_STOPPED (P_STOPPED_SIG|P_STOPPED_SINGLE|P_STOPPED_TRACE) #define P_SHOULDSTOP(p) ((p)->p_flag & P_STOPPED) #define P_KILLED(p) ((p)->p_flag & P_WKILLED) /* These flags are kept in p_flag2. */ #define P2_INHERIT_PROTECTED 0x00000001 /* New children get P_PROTECTED. */ /* Flags protected by proctree_lock, kept in p_treeflags. */ #define P_TREE_ORPHANED 0x00000001 /* Reparented, on orphan list */ #define P_TREE_FIRST_ORPHAN 0x00000002 /* First element of orphan list */ /* * These were process status values (p_stat), now they are only used in * legacy conversion code. */ #define SIDL 1 /* Process being created by fork. */ #define SRUN 2 /* Currently runnable. */ #define SSLEEP 3 /* Sleeping on an address. */ #define SSTOP 4 /* Process debugging or suspension. */ #define SZOMB 5 /* Awaiting collection by parent. */ #define SWAIT 6 /* Waiting for interrupt. */ #define SLOCK 7 /* Blocked on a lock. */ #define P_MAGIC 0xbeefface #ifdef _KERNEL /* Types and flags for mi_switch(). */ #define SW_TYPE_MASK 0xff /* First 8 bits are switch type */ #define SWT_NONE 0 /* Unspecified switch. */ #define SWT_PREEMPT 1 /* Switching due to preemption. */ #define SWT_OWEPREEMPT 2 /* Switching due to opepreempt. */ #define SWT_TURNSTILE 3 /* Turnstile contention. */ #define SWT_SLEEPQ 4 /* Sleepq wait. */ #define SWT_SLEEPQTIMO 5 /* Sleepq timeout wait. */ #define SWT_RELINQUISH 6 /* yield call. */ #define SWT_NEEDRESCHED 7 /* NEEDRESCHED was set. */ #define SWT_IDLE 8 /* Switching from the idle thread. */ #define SWT_IWAIT 9 /* Waiting for interrupts. */ #define SWT_SUSPEND 10 /* Thread suspended. */ #define SWT_REMOTEPREEMPT 11 /* Remote processor preempted. */ #define SWT_REMOTEWAKEIDLE 12 /* Remote processor preempted idle. */ #define SWT_COUNT 13 /* Number of switch types. */ /* Flags */ #define SW_VOL 0x0100 /* Voluntary switch. */ #define SW_INVOL 0x0200 /* Involuntary switch. */ #define SW_PREEMPT 0x0400 /* The invol switch is a preemption */ /* How values for thread_single(). */ #define SINGLE_NO_EXIT 0 #define SINGLE_EXIT 1 #define SINGLE_BOUNDARY 2 +#define SINGLE_ALLPROC 3 #ifdef MALLOC_DECLARE MALLOC_DECLARE(M_PARGS); MALLOC_DECLARE(M_PGRP); MALLOC_DECLARE(M_SESSION); MALLOC_DECLARE(M_SUBPROC); #endif #define FOREACH_PROC_IN_SYSTEM(p) \ LIST_FOREACH((p), &allproc, p_list) #define FOREACH_THREAD_IN_PROC(p, td) \ TAILQ_FOREACH((td), &(p)->p_threads, td_plist) #define FIRST_THREAD_IN_PROC(p) TAILQ_FIRST(&(p)->p_threads) /* * We use process IDs <= pid_max <= PID_MAX; PID_MAX + 1 must also fit * in a pid_t, as it is used to represent "no process group". */ #define PID_MAX 99999 #define NO_PID 100000 extern pid_t pid_max; #define SESS_LEADER(p) ((p)->p_session->s_leader == (p)) #define STOPEVENT(p, e, v) do { \ if ((p)->p_stops & (e)) { \ PROC_LOCK(p); \ stopevent((p), (e), (v)); \ PROC_UNLOCK(p); \ } \ } while (0) #define _STOPEVENT(p, e, v) do { \ PROC_LOCK_ASSERT(p, MA_OWNED); \ WITNESS_WARN(WARN_GIANTOK | WARN_SLEEPOK, &p->p_mtx.lock_object, \ "checking stopevent %d", (e)); \ if ((p)->p_stops & (e)) \ stopevent((p), (e), (v)); \ } while (0) /* Lock and unlock a process. */ #define PROC_LOCK(p) mtx_lock(&(p)->p_mtx) #define PROC_TRYLOCK(p) mtx_trylock(&(p)->p_mtx) #define PROC_UNLOCK(p) mtx_unlock(&(p)->p_mtx) #define PROC_LOCKED(p) mtx_owned(&(p)->p_mtx) #define PROC_LOCK_ASSERT(p, type) mtx_assert(&(p)->p_mtx, (type)) /* Lock and unlock a process group. */ #define PGRP_LOCK(pg) mtx_lock(&(pg)->pg_mtx) #define PGRP_UNLOCK(pg) mtx_unlock(&(pg)->pg_mtx) #define PGRP_LOCKED(pg) mtx_owned(&(pg)->pg_mtx) #define PGRP_LOCK_ASSERT(pg, type) mtx_assert(&(pg)->pg_mtx, (type)) #define PGRP_LOCK_PGSIGNAL(pg) do { \ if ((pg) != NULL) \ PGRP_LOCK(pg); \ } while (0) #define PGRP_UNLOCK_PGSIGNAL(pg) do { \ if ((pg) != NULL) \ PGRP_UNLOCK(pg); \ } while (0) /* Lock and unlock a session. */ #define SESS_LOCK(s) mtx_lock(&(s)->s_mtx) #define SESS_UNLOCK(s) mtx_unlock(&(s)->s_mtx) #define SESS_LOCKED(s) mtx_owned(&(s)->s_mtx) #define SESS_LOCK_ASSERT(s, type) mtx_assert(&(s)->s_mtx, (type)) /* Hold process U-area in memory, normally for ptrace/procfs work. */ #define PHOLD(p) do { \ PROC_LOCK(p); \ _PHOLD(p); \ PROC_UNLOCK(p); \ } while (0) #define _PHOLD(p) do { \ PROC_LOCK_ASSERT((p), MA_OWNED); \ KASSERT(!((p)->p_flag & P_WEXIT) || (p) == curproc, \ ("PHOLD of exiting process")); \ (p)->p_lock++; \ if (((p)->p_flag & P_INMEM) == 0) \ faultin((p)); \ } while (0) #define PROC_ASSERT_HELD(p) do { \ KASSERT((p)->p_lock > 0, ("process not held")); \ } while (0) #define PRELE(p) do { \ PROC_LOCK((p)); \ _PRELE((p)); \ PROC_UNLOCK((p)); \ } while (0) #define _PRELE(p) do { \ PROC_LOCK_ASSERT((p), MA_OWNED); \ PROC_ASSERT_HELD(p); \ (--(p)->p_lock); \ if (((p)->p_flag & P_WEXIT) && (p)->p_lock == 0) \ wakeup(&(p)->p_lock); \ } while (0) #define PROC_ASSERT_NOT_HELD(p) do { \ KASSERT((p)->p_lock == 0, ("process held")); \ } while (0) /* Check whether a thread is safe to be swapped out. */ #define thread_safetoswapout(td) ((td)->td_flags & TDF_CANSWAP) /* Control whether or not it is safe for curthread to sleep. */ #define THREAD_NO_SLEEPING() ((curthread)->td_no_sleeping++) #define THREAD_SLEEPING_OK() ((curthread)->td_no_sleeping--) #define THREAD_CAN_SLEEP() ((curthread)->td_no_sleeping == 0) #define PIDHASH(pid) (&pidhashtbl[(pid) & pidhash]) extern LIST_HEAD(pidhashhead, proc) *pidhashtbl; extern u_long pidhash; #define TIDHASH(tid) (&tidhashtbl[(tid) & tidhash]) extern LIST_HEAD(tidhashhead, thread) *tidhashtbl; extern u_long tidhash; extern struct rwlock tidhash_lock; #define PGRPHASH(pgid) (&pgrphashtbl[(pgid) & pgrphash]) extern LIST_HEAD(pgrphashhead, pgrp) *pgrphashtbl; extern u_long pgrphash; extern struct sx allproc_lock; +extern int allproc_gen; extern struct sx proctree_lock; extern struct mtx ppeers_lock; extern struct proc proc0; /* Process slot for swapper. */ extern struct thread thread0; /* Primary thread in proc0. */ extern struct vmspace vmspace0; /* VM space for proc0. */ extern int hogticks; /* Limit on kernel cpu hogs. */ extern int lastpid; extern int nprocs, maxproc; /* Current and max number of procs. */ extern int maxprocperuid; /* Max procs per uid. */ extern u_long ps_arg_cache_limit; LIST_HEAD(proclist, proc); TAILQ_HEAD(procqueue, proc); TAILQ_HEAD(threadqueue, thread); extern struct proclist allproc; /* List of all processes. */ extern struct proclist zombproc; /* List of zombie processes. */ extern struct proc *initproc, *pageproc; /* Process slots for init, pager. */ extern struct uma_zone *proc_zone; struct proc *pfind(pid_t); /* Find process by id. */ struct proc *pfind_locked(pid_t pid); struct pgrp *pgfind(pid_t); /* Find process group by id. */ struct proc *zpfind(pid_t); /* Find zombie process by id. */ /* * pget() flags. */ #define PGET_HOLD 0x00001 /* Hold the process. */ #define PGET_CANSEE 0x00002 /* Check against p_cansee(). */ #define PGET_CANDEBUG 0x00004 /* Check against p_candebug(). */ #define PGET_ISCURRENT 0x00008 /* Check that the found process is current. */ #define PGET_NOTWEXIT 0x00010 /* Check that the process is not in P_WEXIT. */ #define PGET_NOTINEXEC 0x00020 /* Check that the process is not in P_INEXEC. */ #define PGET_NOTID 0x00040 /* Do not assume tid if pid > PID_MAX. */ #define PGET_WANTREAD (PGET_HOLD | PGET_CANDEBUG | PGET_NOTWEXIT) int pget(pid_t pid, int flags, struct proc **pp); void ast(struct trapframe *framep); struct thread *choosethread(void); int cr_cansignal(struct ucred *cred, struct proc *proc, int signum); int enterpgrp(struct proc *p, pid_t pgid, struct pgrp *pgrp, struct session *sess); int enterthispgrp(struct proc *p, struct pgrp *pgrp); void faultin(struct proc *p); void fixjobc(struct proc *p, struct pgrp *pgrp, int entering); int fork1(struct thread *, int, int, struct proc **, int *, int); void fork_exit(void (*)(void *, struct trapframe *), void *, struct trapframe *); void fork_return(struct thread *, struct trapframe *); int inferior(struct proc *p); void kern_yield(int); void kick_proc0(void); int leavepgrp(struct proc *p); int maybe_preempt(struct thread *td); void maybe_yield(void); void mi_switch(int flags, struct thread *newtd); int p_candebug(struct thread *td, struct proc *p); int p_cansee(struct thread *td, struct proc *p); int p_cansched(struct thread *td, struct proc *p); int p_cansignal(struct thread *td, struct proc *p, int signum); int p_canwait(struct thread *td, struct proc *p); struct pargs *pargs_alloc(int len); void pargs_drop(struct pargs *pa); void pargs_hold(struct pargs *pa); int proc_getargv(struct thread *td, struct proc *p, struct sbuf *sb); int proc_getauxv(struct thread *td, struct proc *p, struct sbuf *sb); int proc_getenvv(struct thread *td, struct proc *p, struct sbuf *sb); void procinit(void); void proc_linkup0(struct proc *p, struct thread *td); void proc_linkup(struct proc *p, struct thread *td); struct proc *proc_realparent(struct proc *child); void proc_reap(struct thread *td, struct proc *p, int *status, int options); void proc_reparent(struct proc *child, struct proc *newparent); struct pstats *pstats_alloc(void); void pstats_fork(struct pstats *src, struct pstats *dst); void pstats_free(struct pstats *ps); int securelevel_ge(struct ucred *cr, int level); int securelevel_gt(struct ucred *cr, int level); void sess_hold(struct session *); void sess_release(struct session *); int setrunnable(struct thread *); void setsugid(struct proc *p); int should_yield(void); int sigonstack(size_t sp); void stopevent(struct proc *, u_int, u_int); struct thread *tdfind(lwpid_t, pid_t); void threadinit(void); void tidhash_add(struct thread *); void tidhash_remove(struct thread *); void cpu_idle(int); int cpu_idle_wakeup(int); extern void (*cpu_idle_hook)(sbintime_t); /* Hook to machdep CPU idler. */ void cpu_switch(struct thread *, struct thread *, struct mtx *); void cpu_throw(struct thread *, struct thread *) __dead2; void unsleep(struct thread *); void userret(struct thread *, struct trapframe *); void cpu_exit(struct thread *); void exit1(struct thread *, int) __dead2; struct syscall_args; int cpu_fetch_syscall_args(struct thread *td, struct syscall_args *sa); void cpu_fork(struct thread *, struct proc *, struct thread *, int); void cpu_set_fork_handler(struct thread *, void (*)(void *), void *); void cpu_set_syscall_retval(struct thread *, int); void cpu_set_upcall(struct thread *td, struct thread *td0); void cpu_set_upcall_kse(struct thread *, void (*)(void *), void *, stack_t *); int cpu_set_user_tls(struct thread *, void *tls_base); void cpu_thread_alloc(struct thread *); void cpu_thread_clean(struct thread *); void cpu_thread_exit(struct thread *); void cpu_thread_free(struct thread *); void cpu_thread_swapin(struct thread *); void cpu_thread_swapout(struct thread *); struct thread *thread_alloc(int pages); int thread_alloc_stack(struct thread *, int pages); void thread_exit(void) __dead2; void thread_free(struct thread *td); void thread_link(struct thread *td, struct proc *p); void thread_reap(void); -int thread_single(int how); -void thread_single_end(void); +int thread_single(struct proc *p, int how); +void thread_single_end(struct proc *p, int how); void thread_stash(struct thread *td); void thread_stopped(struct proc *p); void childproc_stopped(struct proc *child, int reason); void childproc_continued(struct proc *child); void childproc_exited(struct proc *child); int thread_suspend_check(int how); bool thread_suspend_check_needed(void); -void thread_suspend_switch(struct thread *); +void thread_suspend_switch(struct thread *, struct proc *p); void thread_suspend_one(struct thread *td); void thread_unlink(struct thread *td); void thread_unsuspend(struct proc *p); -int thread_unsuspend_one(struct thread *td); +int thread_unsuspend_one(struct thread *td, struct proc *p); void thread_wait(struct proc *p); struct thread *thread_find(struct proc *p, lwpid_t tid); + +void stop_all_proc(void); +void resume_all_proc(void); static __inline int curthread_pflags_set(int flags) { struct thread *td; int save; td = curthread; save = ~flags | (td->td_pflags & flags); td->td_pflags |= flags; return (save); } static __inline void curthread_pflags_restore(int save) { curthread->td_pflags &= save; } #endif /* _KERNEL */ #endif /* !_SYS_PROC_H_ */ Index: projects/clang350-import/sys =================================================================== --- projects/clang350-import/sys (revision 275748) +++ projects/clang350-import/sys (revision 275749) Property changes on: projects/clang350-import/sys ___________________________________________________________________ Modified: svn:mergeinfo ## -0,0 +0,1 ## Merged /head/sys:r275715-275748 Index: projects/clang350-import/tests/sys/Makefile =================================================================== --- projects/clang350-import/tests/sys/Makefile (revision 275748) +++ projects/clang350-import/tests/sys/Makefile (revision 275749) @@ -1,13 +1,14 @@ # $FreeBSD$ .include TESTSDIR= ${TESTSBASE}/sys TESTS_SUBDIRS+= kern TESTS_SUBDIRS+= netinet +TESTS_SUBDIRS+= opencrypto # Items not integrated into kyua runs by default SUBDIR+= pjdfstest .include Index: projects/clang350-import/tests/sys/opencrypto/Makefile =================================================================== --- projects/clang350-import/tests/sys/opencrypto/Makefile (nonexistent) +++ projects/clang350-import/tests/sys/opencrypto/Makefile (revision 275749) @@ -0,0 +1,14 @@ +# $FreeBSD$ + +TESTSDIR= ${TESTSBASE}/sys/opencrypto +BINDIR= ${TESTSDIR} + +PLAIN_TESTS_SH= runtests + +TEST_METADATA.foo+=required_programs="python" +PYMODULES= cryptodev.py cryptodevh.py cryptotest.py dpkt.py + +FILESDIR= ${TESTSDIR} +FILES= ${PYMODULES} + +.include Property changes on: projects/clang350-import/tests/sys/opencrypto/Makefile ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: projects/clang350-import/tests/sys/opencrypto/cryptodev.py =================================================================== --- projects/clang350-import/tests/sys/opencrypto/cryptodev.py (nonexistent) +++ projects/clang350-import/tests/sys/opencrypto/cryptodev.py (revision 275749) @@ -0,0 +1,561 @@ +#!/usr/bin/env python +# +# Copyright (c) 2014 The FreeBSD Foundation +# Copyright 2014 John-Mark Gurney +# All rights reserved. +# +# This software was developed by John-Mark Gurney under +# the sponsorship from the FreeBSD Foundation. +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# 1. Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# 2. Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# +# THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND +# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +# ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE +# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS +# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) +# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT +# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY +# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF +# SUCH DAMAGE. +# +# $FreeBSD$ +# + +import array +import dpkt +from fcntl import ioctl +import os +import signal +from struct import pack as _pack + +from cryptodevh import * + +__all__ = [ 'Crypto', 'MismatchError', ] + +class FindOp(dpkt.Packet): + __byte_order__ = '@' + __hdr__ = ( ('crid', 'i', 0), + ('name', '32s', 0), + ) + +class SessionOp(dpkt.Packet): + __byte_order__ = '@' + __hdr__ = ( ('cipher', 'I', 0), + ('mac', 'I', 0), + ('keylen', 'I', 0), + ('key', 'P', 0), + ('mackeylen', 'i', 0), + ('mackey', 'P', 0), + ('ses', 'I', 0), + ) + +class SessionOp2(dpkt.Packet): + __byte_order__ = '@' + __hdr__ = ( ('cipher', 'I', 0), + ('mac', 'I', 0), + ('keylen', 'I', 0), + ('key', 'P', 0), + ('mackeylen', 'i', 0), + ('mackey', 'P', 0), + ('ses', 'I', 0), + ('crid', 'i', 0), + ('pad0', 'i', 0), + ('pad1', 'i', 0), + ('pad2', 'i', 0), + ('pad3', 'i', 0), + ) + +class CryptOp(dpkt.Packet): + __byte_order__ = '@' + __hdr__ = ( ('ses', 'I', 0), + ('op', 'H', 0), + ('flags', 'H', 0), + ('len', 'I', 0), + ('src', 'P', 0), + ('dst', 'P', 0), + ('mac', 'P', 0), + ('iv', 'P', 0), + ) + +class CryptAEAD(dpkt.Packet): + __byte_order__ = '@' + __hdr__ = ( + ('ses', 'I', 0), + ('op', 'H', 0), + ('flags', 'H', 0), + ('len', 'I', 0), + ('aadlen', 'I', 0), + ('ivlen', 'I', 0), + ('src', 'P', 0), + ('dst', 'P', 0), + ('aad', 'P', 0), + ('tag', 'P', 0), + ('iv', 'P', 0), + ) + +# h2py.py can't handle multiarg macros +CRIOGET = 3221513060 +CIOCGSESSION = 3224396645 +CIOCGSESSION2 = 3225445226 +CIOCFSESSION = 2147771238 +CIOCCRYPT = 3224396647 +CIOCKEY = 3230688104 +CIOCASYMFEAT = 1074029417 +CIOCKEY2 = 3230688107 +CIOCFINDDEV = 3223610220 +CIOCCRYPTAEAD = 3225445229 + +def _getdev(): + fd = os.open('/dev/crypto', os.O_RDWR) + buf = array.array('I', [0]) + ioctl(fd, CRIOGET, buf, 1) + os.close(fd) + + return buf[0] + +_cryptodev = _getdev() + +def _findop(crid, name): + fop = FindOp() + fop.crid = crid + fop.name = name + s = array.array('B', fop.pack_hdr()) + ioctl(_cryptodev, CIOCFINDDEV, s, 1) + fop.unpack(s) + + try: + idx = fop.name.index('\x00') + name = fop.name[:idx] + except ValueError: + name = fop.name + + return fop.crid, name + +class Crypto: + @staticmethod + def findcrid(name): + return _findop(-1, name)[0] + + @staticmethod + def getcridname(crid): + return _findop(crid, '')[1] + + def __init__(self, cipher=0, key=None, mac=0, mackey=None, + crid=CRYPTOCAP_F_SOFTWARE | CRYPTOCAP_F_HARDWARE): + self._ses = None + ses = SessionOp2() + ses.cipher = cipher + ses.mac = mac + + if key is not None: + ses.keylen = len(key) + k = array.array('B', key) + ses.key = k.buffer_info()[0] + else: + self.key = None + + if mackey is not None: + ses.mackeylen = len(mackey) + mk = array.array('B', mackey) + ses.mackey = mk.buffer_info()[0] + self._maclen = 16 # parameterize? + else: + self._maclen = None + + if not cipher and not mac: + raise ValueError('one of cipher or mac MUST be specified.') + ses.crid = CRYPTOCAP_F_SOFTWARE | CRYPTOCAP_F_HARDWARE + #ses.crid = CRYPTOCAP_F_HARDWARE + #ses.crid = CRYPTOCAP_F_SOFTWARE + #ses.crid = 0 + #print `ses` + s = array.array('B', ses.pack_hdr()) + #print `s` + ioctl(_cryptodev, CIOCGSESSION2, s, 1) + ses.unpack(s) + + self._ses = ses.ses + + def __del__(self): + if self._ses is None: + return + + try: + ioctl(_cryptodev, CIOCFSESSION, _pack('I', self._ses)) + except TypeError: + pass + self._ses = None + + def _doop(self, op, src, iv): + cop = CryptOp() + cop.ses = self._ses + cop.op = op + cop.flags = 0 + cop.len = len(src) + s = array.array('B', src) + cop.src = cop.dst = s.buffer_info()[0] + if self._maclen is not None: + m = array.array('B', [0] * self._maclen) + cop.mac = m.buffer_info()[0] + ivbuf = array.array('B', iv) + cop.iv = ivbuf.buffer_info()[0] + + #print 'cop:', `cop` + ioctl(_cryptodev, CIOCCRYPT, str(cop)) + + s = s.tostring() + if self._maclen is not None: + return s, m.tostring() + + return s + + def _doaead(self, op, src, aad, iv, tag=None): + caead = CryptAEAD() + caead.ses = self._ses + caead.op = op + caead.flags = CRD_F_IV_EXPLICIT + caead.flags = 0 + caead.len = len(src) + s = array.array('B', src) + caead.src = caead.dst = s.buffer_info()[0] + caead.aadlen = len(aad) + saad = array.array('B', aad) + caead.aad = saad.buffer_info()[0] + + if self._maclen is None: + raise ValueError('must have a tag length') + + if tag is None: + tag = array.array('B', [0] * self._maclen) + else: + assert len(tag) == self._maclen, `len(tag), self._maclen` + tag = array.array('B', tag) + + caead.tag = tag.buffer_info()[0] + + ivbuf = array.array('B', iv) + caead.ivlen = len(iv) + caead.iv = ivbuf.buffer_info()[0] + + ioctl(_cryptodev, CIOCCRYPTAEAD, str(caead)) + + s = s.tostring() + + return s, tag.tostring() + + def perftest(self, op, size, timeo=3): + import random + import time + + inp = array.array('B', (random.randint(0, 255) for x in xrange(size))) + out = array.array('B', inp) + + # prep ioctl + cop = CryptOp() + cop.ses = self._ses + cop.op = op + cop.flags = 0 + cop.len = len(inp) + s = array.array('B', inp) + cop.src = s.buffer_info()[0] + cop.dst = out.buffer_info()[0] + if self._maclen is not None: + m = array.array('B', [0] * self._maclen) + cop.mac = m.buffer_info()[0] + ivbuf = array.array('B', (random.randint(0, 255) for x in xrange(16))) + cop.iv = ivbuf.buffer_info()[0] + + exit = [ False ] + def alarmhandle(a, b, exit=exit): + exit[0] = True + + oldalarm = signal.signal(signal.SIGALRM, alarmhandle) + signal.alarm(timeo) + + start = time.time() + reps = 0 + while not exit[0]: + ioctl(_cryptodev, CIOCCRYPT, str(cop)) + reps += 1 + + end = time.time() + + signal.signal(signal.SIGALRM, oldalarm) + + print 'time:', end - start + print 'perf MB/sec:', (reps * size) / (end - start) / 1024 / 1024 + + def encrypt(self, data, iv, aad=None): + if aad is None: + return self._doop(COP_ENCRYPT, data, iv) + else: + return self._doaead(COP_ENCRYPT, data, aad, + iv) + + def decrypt(self, data, iv, aad=None, tag=None): + if aad is None: + return self._doop(COP_DECRYPT, data, iv) + else: + return self._doaead(COP_DECRYPT, data, aad, + iv, tag=tag) + +class MismatchError(Exception): + pass + +class KATParser: + def __init__(self, fname, fields): + self.fp = open(fname) + self.fields = set(fields) + self._pending = None + + def __iter__(self): + while True: + didread = False + if self._pending is not None: + i = self._pending + self._pending = None + else: + i = self.fp.readline() + didread = True + + if didread and not i: + return + + if (i and i[0] == '#') or not i.strip(): + continue + if i[0] == '[': + yield i[1:].split(']', 1)[0], self.fielditer() + else: + raise ValueError('unknown line: %s' % `i`) + + def eatblanks(self): + while True: + line = self.fp.readline() + if line == '': + break + + line = line.strip() + if line: + break + + return line + + def fielditer(self): + while True: + values = {} + + line = self.eatblanks() + if not line or line[0] == '[': + self._pending = line + return + + while True: + try: + f, v = line.split(' =') + except: + if line == 'FAIL': + f, v = 'FAIL', '' + else: + print 'line:', `line` + raise + v = v.strip() + + if f in values: + raise ValueError('already present: %s' % `f`) + values[f] = v + line = self.fp.readline().strip() + if not line: + break + + # we should have everything + remain = self.fields.copy() - set(values.keys()) + # XXX - special case GCM decrypt + if remain and not ('FAIL' in values and 'PT' in remain): + raise ValueError('not all fields found: %s' % `remain`) + + yield values + +def _spdechex(s): + return ''.join(s.split()).decode('hex') + +if __name__ == '__main__': + if True: + try: + crid = Crypto.findcrid('aesni0') + print 'aesni:', crid + except IOError: + print 'aesni0 not found' + + for i in xrange(10): + try: + name = Crypto.getcridname(i) + print '%2d: %s' % (i, `name`) + except IOError: + pass + elif False: + kp = KATParser('/usr/home/jmg/aesni.testing/format tweak value input - data unit seq no/XTSGenAES128.rsp', [ 'COUNT', 'DataUnitLen', 'Key', 'DataUnitSeqNumber', 'PT', 'CT' ]) + for mode, ni in kp: + print `i`, `ni` + for j in ni: + print `j` + elif False: + key = _spdechex('c939cc13397c1d37de6ae0e1cb7c423c') + iv = _spdechex('00000000000000000000000000000001') + pt = _spdechex('ab3cabed693a32946055524052afe3c9cb49664f09fc8b7da824d924006b7496353b8c1657c5dec564d8f38d7432e1de35aae9d95590e66278d4acce883e51abaf94977fcd3679660109a92bf7b2973ccd547f065ec6cee4cb4a72a5e9f45e615d920d76cb34cba482467b3e21422a7242e7d931330c0fbf465c3a3a46fae943029fd899626dda542750a1eee253df323c6ef1573f1c8c156613e2ea0a6cdbf2ae9701020be2d6a83ecb7f3f9d8e') + #pt = _spdechex('00000000000000000000000000000000') + ct = _spdechex('f42c33853ecc5ce2949865fdb83de3bff1089e9360c94f830baebfaff72836ab5236f77212f1e7396c8c54ac73d81986375a6e9e299cfeca5ba051ed25e8d1affa5beaf6c1d2b45e90802408f2ced21663497e906de5f29341e5e52ddfea5363d628b3eb7806835e17bae051b3a6da3f8e2941fe44384eac17a9d298d2c331ca8320c775b5d53263a5e905059d891b21dede2d8110fd427c7bd5a9a274ddb47b1945ee79522203b6e297d0e399ef') + + c = Crypto(CRYPTO_AES_ICM, key) + enc = c.encrypt(pt, iv) + + print 'enc:', enc.encode('hex') + print ' ct:', ct.encode('hex') + + assert ct == enc + + dec = c.decrypt(ct, iv) + + print 'dec:', dec.encode('hex') + print ' pt:', pt.encode('hex') + + assert pt == dec + elif False: + key = _spdechex('c939cc13397c1d37de6ae0e1cb7c423c') + iv = _spdechex('00000000000000000000000000000001') + pt = _spdechex('ab3cabed693a32946055524052afe3c9cb49664f09fc8b7da824d924006b7496353b8c1657c5dec564d8f38d7432e1de35aae9d95590e66278d4acce883e51abaf94977fcd3679660109a92bf7b2973ccd547f065ec6cee4cb4a72a5e9f45e615d920d76cb34cba482467b3e21422a7242e7d931330c0fbf465c3a3a46fae943029fd899626dda542750a1eee253df323c6ef1573f1c8c156613e2ea0a6cdbf2ae9701020be2d6a83ecb7f3f9d8e0a3f') + #pt = _spdechex('00000000000000000000000000000000') + ct = _spdechex('f42c33853ecc5ce2949865fdb83de3bff1089e9360c94f830baebfaff72836ab5236f77212f1e7396c8c54ac73d81986375a6e9e299cfeca5ba051ed25e8d1affa5beaf6c1d2b45e90802408f2ced21663497e906de5f29341e5e52ddfea5363d628b3eb7806835e17bae051b3a6da3f8e2941fe44384eac17a9d298d2c331ca8320c775b5d53263a5e905059d891b21dede2d8110fd427c7bd5a9a274ddb47b1945ee79522203b6e297d0e399ef3768') + + c = Crypto(CRYPTO_AES_ICM, key) + enc = c.encrypt(pt, iv) + + print 'enc:', enc.encode('hex') + print ' ct:', ct.encode('hex') + + assert ct == enc + + dec = c.decrypt(ct, iv) + + print 'dec:', dec.encode('hex') + print ' pt:', pt.encode('hex') + + assert pt == dec + elif False: + key = _spdechex('c939cc13397c1d37de6ae0e1cb7c423c') + iv = _spdechex('6eba2716ec0bd6fa5cdef5e6d3a795bc') + pt = _spdechex('ab3cabed693a32946055524052afe3c9cb49664f09fc8b7da824d924006b7496353b8c1657c5dec564d8f38d7432e1de35aae9d95590e66278d4acce883e51abaf94977fcd3679660109a92bf7b2973ccd547f065ec6cee4cb4a72a5e9f45e615d920d76cb34cba482467b3e21422a7242e7d931330c0fbf465c3a3a46fae943029fd899626dda542750a1eee253df323c6ef1573f1c8c156613e2ea0a6cdbf2ae9701020be2d6a83ecb7f3f9d8e0a3f') + ct = _spdechex('f1f81f12e72e992dbdc304032705dc75dc3e4180eff8ee4819906af6aee876d5b00b7c36d282a445ce3620327be481e8e53a8e5a8e5ca9abfeb2281be88d12ffa8f46d958d8224738c1f7eea48bda03edbf9adeb900985f4fa25648b406d13a886c25e70cfdecdde0ad0f2991420eb48a61c64fd797237cf2798c2675b9bb744360b0a3f329ac53bbceb4e3e7456e6514f1a9d2f06c236c31d0f080b79c15dce1096357416602520daa098b17d1af427') + c = Crypto(CRYPTO_AES_CBC, key) + + enc = c.encrypt(pt, iv) + + print 'enc:', enc.encode('hex') + print ' ct:', ct.encode('hex') + + assert ct == enc + + dec = c.decrypt(ct, iv) + + print 'dec:', dec.encode('hex') + print ' pt:', pt.encode('hex') + + assert pt == dec + elif False: + key = _spdechex('c939cc13397c1d37de6ae0e1cb7c423c') + iv = _spdechex('b3d8cc017cbb89b39e0f67e2') + pt = _spdechex('c3b3c41f113a31b73d9a5cd4321030') + aad = _spdechex('24825602bd12a984e0092d3e448eda5f') + ct = _spdechex('93fe7d9e9bfd10348a5606e5cafa7354') + ct = _spdechex('93fe7d9e9bfd10348a5606e5cafa73') + tag = _spdechex('0032a1dc85f1c9786925a2e71d8272dd') + tag = _spdechex('8d11a0929cb3fbe1fef01a4a38d5f8ea') + + c = Crypto(CRYPTO_AES_NIST_GCM_16, key, + mac=CRYPTO_AES_128_NIST_GMAC, mackey=key) + + enc, enctag = c.encrypt(pt, iv, aad=aad) + + print 'enc:', enc.encode('hex') + print ' ct:', ct.encode('hex') + + assert enc == ct + + print 'etg:', enctag.encode('hex') + print 'tag:', tag.encode('hex') + assert enctag == tag + + # Make sure we get EBADMSG + #enctag = enctag[:-1] + 'a' + dec, dectag = c.decrypt(ct, iv, aad=aad, tag=enctag) + + print 'dec:', dec.encode('hex') + print ' pt:', pt.encode('hex') + + assert dec == pt + + print 'dtg:', dectag.encode('hex') + print 'tag:', tag.encode('hex') + + assert dectag == tag + elif False: + key = _spdechex('c939cc13397c1d37de6ae0e1cb7c423c') + iv = _spdechex('b3d8cc017cbb89b39e0f67e2') + key = key + iv[:4] + iv = iv[4:] + pt = _spdechex('c3b3c41f113a31b73d9a5cd432103069') + aad = _spdechex('24825602bd12a984e0092d3e448eda5f') + ct = _spdechex('93fe7d9e9bfd10348a5606e5cafa7354') + tag = _spdechex('0032a1dc85f1c9786925a2e71d8272dd') + + c = Crypto(CRYPTO_AES_GCM_16, key, mac=CRYPTO_AES_128_GMAC, mackey=key) + + enc, enctag = c.encrypt(pt, iv, aad=aad) + + print 'enc:', enc.encode('hex') + print ' ct:', ct.encode('hex') + + assert enc == ct + + print 'etg:', enctag.encode('hex') + print 'tag:', tag.encode('hex') + assert enctag == tag + elif False: + for i in xrange(100000): + c = Crypto(CRYPTO_AES_XTS, '1bbfeadf539daedcae33ced497343f3ca1f2474ad932b903997d44707db41382'.decode('hex')) + data = '52a42bca4e9425a25bbc8c8bf6129dec'.decode('hex') + ct = '517e602becd066b65fa4f4f56ddfe240'.decode('hex') + iv = _pack('QQ', 71, 0) + + enc = c.encrypt(data, iv) + assert enc == ct + elif True: + c = Crypto(CRYPTO_AES_XTS, '1bbfeadf539daedcae33ced497343f3ca1f2474ad932b903997d44707db41382'.decode('hex')) + data = '52a42bca4e9425a25bbc8c8bf6129dec'.decode('hex') + ct = '517e602becd066b65fa4f4f56ddfe240'.decode('hex') + iv = _pack('QQ', 71, 0) + + enc = c.encrypt(data, iv) + assert enc == ct + + dec = c.decrypt(enc, iv) + assert dec == data + + #c.perftest(COP_ENCRYPT, 192*1024, reps=30000) + + else: + key = '1bbfeadf539daedcae33ced497343f3ca1f2474ad932b903997d44707db41382'.decode('hex') + print 'XTS %d testing:' % (len(key) * 8) + c = Crypto(CRYPTO_AES_XTS, key) + for i in [ 8192, 192*1024]: + print 'block size: %d' % i + c.perftest(COP_ENCRYPT, i) + c.perftest(COP_DECRYPT, i) Property changes on: projects/clang350-import/tests/sys/opencrypto/cryptodev.py ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: projects/clang350-import/tests/sys/opencrypto/cryptodevh.py =================================================================== --- projects/clang350-import/tests/sys/opencrypto/cryptodevh.py (nonexistent) +++ projects/clang350-import/tests/sys/opencrypto/cryptodevh.py (revision 275749) @@ -0,0 +1,250 @@ +# $FreeBSD$ +# Generated by h2py from stdin + +# Included from sys/ioccom.h +IOCPARM_SHIFT = 13 +IOCPARM_MASK = ((1 << IOCPARM_SHIFT) - 1) +def IOCPARM_LEN(x): return (((x) >> 16) & IOCPARM_MASK) + +def IOCBASECMD(x): return ((x) & ~(IOCPARM_MASK << 16)) + +def IOCGROUP(x): return (((x) >> 8) & 0xff) + +IOCPARM_MAX = (1 << IOCPARM_SHIFT) +IOC_VOID = 0x20000000 +IOC_OUT = 0x40000000 +IOC_IN = 0x80000000 +IOC_INOUT = (IOC_IN|IOC_OUT) +IOC_DIRMASK = (IOC_VOID|IOC_OUT|IOC_IN) + +# Included from sys/cdefs.h +def __has_feature(x): return 0 + +def __has_include(x): return 0 + +def __has_builtin(x): return 0 + +__GNUCLIKE_ASM = 3 +__GNUCLIKE_ASM = 2 +__GNUCLIKE___TYPEOF = 1 +__GNUCLIKE___OFFSETOF = 1 +__GNUCLIKE___SECTION = 1 +__GNUCLIKE_CTOR_SECTION_HANDLING = 1 +__GNUCLIKE_BUILTIN_CONSTANT_P = 1 +__GNUCLIKE_BUILTIN_VARARGS = 1 +__GNUCLIKE_BUILTIN_STDARG = 1 +__GNUCLIKE_BUILTIN_VAALIST = 1 +__GNUC_VA_LIST_COMPATIBILITY = 1 +__GNUCLIKE_BUILTIN_NEXT_ARG = 1 +__GNUCLIKE_BUILTIN_MEMCPY = 1 +__CC_SUPPORTS_INLINE = 1 +__CC_SUPPORTS___INLINE = 1 +__CC_SUPPORTS___INLINE__ = 1 +__CC_SUPPORTS___FUNC__ = 1 +__CC_SUPPORTS_WARNING = 1 +__CC_SUPPORTS_VARADIC_XXX = 1 +__CC_SUPPORTS_DYNAMIC_ARRAY_INIT = 1 +def __P(protos): return protos + +def __STRING(x): return #x + +def __XSTRING(x): return __STRING(x) + +def __P(protos): return () + +def __STRING(x): return "x" + +def __aligned(x): return __attribute__((__aligned__(x))) + +def __section(x): return __attribute__((__section__(x))) + +def __aligned(x): return __attribute__((__aligned__(x))) + +def __section(x): return __attribute__((__section__(x))) + +def _Alignas(x): return alignas(x) + +def _Alignas(x): return __aligned(x) + +def _Alignof(x): return alignof(x) + +def _Alignof(x): return __alignof(x) + +def __nonnull(x): return __attribute__((__nonnull__(x))) + +def __predict_true(exp): return __builtin_expect((exp), 1) + +def __predict_false(exp): return __builtin_expect((exp), 0) + +def __predict_true(exp): return (exp) + +def __predict_false(exp): return (exp) + +def __format_arg(fmtarg): return __attribute__((__format_arg__ (fmtarg))) + +def __GLOBL(sym): return __GLOBL1(sym) + +def __FBSDID(s): return __IDSTRING(__CONCAT(__rcsid_,__LINE__),s) + +def __RCSID(s): return __IDSTRING(__CONCAT(__rcsid_,__LINE__),s) + +def __RCSID_SOURCE(s): return __IDSTRING(__CONCAT(__rcsid_source_,__LINE__),s) + +def __SCCSID(s): return __IDSTRING(__CONCAT(__sccsid_,__LINE__),s) + +def __COPYRIGHT(s): return __IDSTRING(__CONCAT(__copyright_,__LINE__),s) + +_POSIX_C_SOURCE = 199009 +_POSIX_C_SOURCE = 199209 +__XSI_VISIBLE = 700 +_POSIX_C_SOURCE = 200809 +__XSI_VISIBLE = 600 +_POSIX_C_SOURCE = 200112 +__XSI_VISIBLE = 500 +_POSIX_C_SOURCE = 199506 +_POSIX_C_SOURCE = 198808 +__POSIX_VISIBLE = 200809 +__ISO_C_VISIBLE = 1999 +__POSIX_VISIBLE = 200112 +__ISO_C_VISIBLE = 1999 +__POSIX_VISIBLE = 199506 +__ISO_C_VISIBLE = 1990 +__POSIX_VISIBLE = 199309 +__ISO_C_VISIBLE = 1990 +__POSIX_VISIBLE = 199209 +__ISO_C_VISIBLE = 1990 +__POSIX_VISIBLE = 199009 +__ISO_C_VISIBLE = 1990 +__POSIX_VISIBLE = 198808 +__ISO_C_VISIBLE = 0 +__POSIX_VISIBLE = 0 +__XSI_VISIBLE = 0 +__BSD_VISIBLE = 0 +__ISO_C_VISIBLE = 1990 +__POSIX_VISIBLE = 0 +__XSI_VISIBLE = 0 +__BSD_VISIBLE = 0 +__ISO_C_VISIBLE = 1999 +__POSIX_VISIBLE = 0 +__XSI_VISIBLE = 0 +__BSD_VISIBLE = 0 +__ISO_C_VISIBLE = 2011 +__POSIX_VISIBLE = 200809 +__XSI_VISIBLE = 700 +__BSD_VISIBLE = 1 +__ISO_C_VISIBLE = 2011 +__NO_TLS = 1 +CRYPTO_DRIVERS_INITIAL = 4 +CRYPTO_SW_SESSIONS = 32 +NULL_HASH_LEN = 16 +MD5_HASH_LEN = 16 +SHA1_HASH_LEN = 20 +RIPEMD160_HASH_LEN = 20 +SHA2_256_HASH_LEN = 32 +SHA2_384_HASH_LEN = 48 +SHA2_512_HASH_LEN = 64 +MD5_KPDK_HASH_LEN = 16 +SHA1_KPDK_HASH_LEN = 20 +HASH_MAX_LEN = SHA2_512_HASH_LEN +NULL_HMAC_BLOCK_LEN = 64 +MD5_HMAC_BLOCK_LEN = 64 +SHA1_HMAC_BLOCK_LEN = 64 +RIPEMD160_HMAC_BLOCK_LEN = 64 +SHA2_256_HMAC_BLOCK_LEN = 64 +SHA2_384_HMAC_BLOCK_LEN = 128 +SHA2_512_HMAC_BLOCK_LEN = 128 +HMAC_MAX_BLOCK_LEN = SHA2_512_HMAC_BLOCK_LEN +HMAC_IPAD_VAL = 0x36 +HMAC_OPAD_VAL = 0x5C +NULL_BLOCK_LEN = 4 +DES_BLOCK_LEN = 8 +DES3_BLOCK_LEN = 8 +BLOWFISH_BLOCK_LEN = 8 +SKIPJACK_BLOCK_LEN = 8 +CAST128_BLOCK_LEN = 8 +RIJNDAEL128_BLOCK_LEN = 16 +AES_BLOCK_LEN = RIJNDAEL128_BLOCK_LEN +CAMELLIA_BLOCK_LEN = 16 +EALG_MAX_BLOCK_LEN = AES_BLOCK_LEN +AALG_MAX_RESULT_LEN = 64 +CRYPTO_ALGORITHM_MIN = 1 +CRYPTO_DES_CBC = 1 +CRYPTO_3DES_CBC = 2 +CRYPTO_BLF_CBC = 3 +CRYPTO_CAST_CBC = 4 +CRYPTO_SKIPJACK_CBC = 5 +CRYPTO_MD5_HMAC = 6 +CRYPTO_SHA1_HMAC = 7 +CRYPTO_RIPEMD160_HMAC = 8 +CRYPTO_MD5_KPDK = 9 +CRYPTO_SHA1_KPDK = 10 +CRYPTO_RIJNDAEL128_CBC = 11 +CRYPTO_AES_CBC = 11 +CRYPTO_ARC4 = 12 +CRYPTO_MD5 = 13 +CRYPTO_SHA1 = 14 +CRYPTO_NULL_HMAC = 15 +CRYPTO_NULL_CBC = 16 +CRYPTO_DEFLATE_COMP = 17 +CRYPTO_SHA2_256_HMAC = 18 +CRYPTO_SHA2_384_HMAC = 19 +CRYPTO_SHA2_512_HMAC = 20 +CRYPTO_CAMELLIA_CBC = 21 +CRYPTO_AES_XTS = 22 +CRYPTO_AES_ICM = 23 +CRYPTO_AES_NIST_GMAC = 24 +CRYPTO_AES_NIST_GCM_16 = 25 +CRYPTO_AES_128_NIST_GMAC = 26 +CRYPTO_AES_192_NIST_GMAC = 27 +CRYPTO_AES_256_NIST_GMAC = 28 +CRYPTO_ALGORITHM_MAX = 28 +CRYPTO_ALG_FLAG_SUPPORTED = 0x01 +CRYPTO_ALG_FLAG_RNG_ENABLE = 0x02 +CRYPTO_ALG_FLAG_DSA_SHA = 0x04 +CRYPTO_FLAG_HARDWARE = 0x01000000 +CRYPTO_FLAG_SOFTWARE = 0x02000000 +COP_ENCRYPT = 1 +COP_DECRYPT = 2 +COP_F_BATCH = 0x0008 +CRK_MAXPARAM = 8 +CRK_ALGORITM_MIN = 0 +CRK_MOD_EXP = 0 +CRK_MOD_EXP_CRT = 1 +CRK_DSA_SIGN = 2 +CRK_DSA_VERIFY = 3 +CRK_DH_COMPUTE_KEY = 4 +CRK_ALGORITHM_MAX = 4 +CRF_MOD_EXP = (1 << CRK_MOD_EXP) +CRF_MOD_EXP_CRT = (1 << CRK_MOD_EXP_CRT) +CRF_DSA_SIGN = (1 << CRK_DSA_SIGN) +CRF_DSA_VERIFY = (1 << CRK_DSA_VERIFY) +CRF_DH_COMPUTE_KEY = (1 << CRK_DH_COMPUTE_KEY) +CRD_F_ENCRYPT = 0x01 +CRD_F_IV_PRESENT = 0x02 +CRD_F_IV_EXPLICIT = 0x04 +CRD_F_DSA_SHA_NEEDED = 0x08 +CRD_F_COMP = 0x0f +CRD_F_KEY_EXPLICIT = 0x10 +CRYPTO_F_IMBUF = 0x0001 +CRYPTO_F_IOV = 0x0002 +CRYPTO_F_BATCH = 0x0008 +CRYPTO_F_CBIMM = 0x0010 +CRYPTO_F_DONE = 0x0020 +CRYPTO_F_CBIFSYNC = 0x0040 +CRYPTO_BUF_CONTIG = 0x0 +CRYPTO_BUF_IOV = 0x1 +CRYPTO_BUF_MBUF = 0x2 +CRYPTO_OP_DECRYPT = 0x0 +CRYPTO_OP_ENCRYPT = 0x1 +CRYPTO_HINT_MORE = 0x1 +def CRYPTO_SESID2HID(_sid): return (((_sid) >> 32) & 0x00ffffff) + +def CRYPTO_SESID2CAPS(_sid): return (((_sid) >> 32) & 0xff000000) + +def CRYPTO_SESID2LID(_sid): return (((u_int32_t) (_sid)) & 0xffffffff) + +CRYPTOCAP_F_HARDWARE = CRYPTO_FLAG_HARDWARE +CRYPTOCAP_F_SOFTWARE = CRYPTO_FLAG_SOFTWARE +CRYPTOCAP_F_SYNC = 0x04000000 +CRYPTO_SYMQ = 0x1 +CRYPTO_ASYMQ = 0x2 Property changes on: projects/clang350-import/tests/sys/opencrypto/cryptodevh.py ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: projects/clang350-import/tests/sys/opencrypto/cryptotest.py =================================================================== --- projects/clang350-import/tests/sys/opencrypto/cryptotest.py (nonexistent) +++ projects/clang350-import/tests/sys/opencrypto/cryptotest.py (revision 275749) @@ -0,0 +1,265 @@ +#!/usr/bin/env python +# +# Copyright (c) 2014 The FreeBSD Foundation +# All rights reserved. +# +# This software was developed by John-Mark Gurney under +# the sponsorship from the FreeBSD Foundation. +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# 1. Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# 2. Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# +# THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND +# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +# ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE +# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS +# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) +# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT +# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY +# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF +# SUCH DAMAGE. +# +# $FreeBSD$ +# + +import cryptodev +import itertools +import os +import struct +import unittest +from cryptodev import * +from glob import iglob + +katdir = '/usr/local/share/nist-kat' + +def katg(base, glob): + return iglob(os.path.join(katdir, base, glob)) + +aesmodules = [ 'cryptosoft0', 'aesni0', ] +desmodules = [ 'cryptosoft0', ] +shamodules = [ 'cryptosoft0', ] + +def GenTestCase(cname): + try: + crid = cryptodev.Crypto.findcrid(cname) + except IOError: + return None + + class GendCryptoTestCase(unittest.TestCase): + ############### + ##### AES ##### + ############### + @unittest.skipIf(cname not in aesmodules, 'skipping AES on %s' % `cname`) + def test_xts(self): + for i in katg('XTSTestVectors/format tweak value input - data unit seq no', '*.rsp'): + self.runXTS(i, cryptodev.CRYPTO_AES_XTS) + + def test_cbc(self): + for i in katg('KAT_AES', 'CBC[GKV]*.rsp'): + self.runCBC(i) + + def test_gcm(self): + for i in katg('gcmtestvectors', 'gcmEncrypt*'): + self.runGCM(i, 'ENCRYPT') + + for i in katg('gcmtestvectors', 'gcmDecrypt*'): + self.runGCM(i, 'DECRYPT') + + _gmacsizes = { 32: cryptodev.CRYPTO_AES_256_NIST_GMAC, + 24: cryptodev.CRYPTO_AES_192_NIST_GMAC, + 16: cryptodev.CRYPTO_AES_128_NIST_GMAC, + } + def runGCM(self, fname, mode): + curfun = None + if mode == 'ENCRYPT': + swapptct = False + curfun = Crypto.encrypt + elif mode == 'DECRYPT': + swapptct = True + curfun = Crypto.decrypt + else: + raise RuntimeError('unknown mode: %s' % `mode`) + + for bogusmode, lines in cryptodev.KATParser(fname, + [ 'Count', 'Key', 'IV', 'CT', 'AAD', 'Tag', 'PT', ]): + for data in lines: + curcnt = int(data['Count']) + cipherkey = data['Key'].decode('hex') + iv = data['IV'].decode('hex') + aad = data['AAD'].decode('hex') + tag = data['Tag'].decode('hex') + if 'FAIL' not in data: + pt = data['PT'].decode('hex') + ct = data['CT'].decode('hex') + + if len(iv) != 12: + # XXX - isn't supported + continue + + c = Crypto(cryptodev.CRYPTO_AES_NIST_GCM_16, + cipherkey, + mac=self._gmacsizes[len(cipherkey)], + mackey=cipherkey, crid=crid) + + if mode == 'ENCRYPT': + rct, rtag = c.encrypt(pt, iv, aad) + rtag = rtag[:len(tag)] + data['rct'] = rct.encode('hex') + data['rtag'] = rtag.encode('hex') + self.assertEqual(rct, ct, `data`) + self.assertEqual(rtag, tag, `data`) + else: + if len(tag) != 16: + continue + args = (ct, iv, aad, tag) + if 'FAIL' in data: + self.assertRaises(IOError, + c.decrypt, *args) + else: + rpt, rtag = c.decrypt(*args) + data['rpt'] = rpt.encode('hex') + data['rtag'] = rtag.encode('hex') + self.assertEqual(rpt, pt, + `data`) + + def runCBC(self, fname): + curfun = None + for mode, lines in cryptodev.KATParser(fname, + [ 'COUNT', 'KEY', 'IV', 'PLAINTEXT', 'CIPHERTEXT', ]): + if mode == 'ENCRYPT': + swapptct = False + curfun = Crypto.encrypt + elif mode == 'DECRYPT': + swapptct = True + curfun = Crypto.decrypt + else: + raise RuntimeError('unknown mode: %s' % `mode`) + + for data in lines: + curcnt = int(data['COUNT']) + cipherkey = data['KEY'].decode('hex') + iv = data['IV'].decode('hex') + pt = data['PLAINTEXT'].decode('hex') + ct = data['CIPHERTEXT'].decode('hex') + + if swapptct: + pt, ct = ct, pt + # run the fun + c = Crypto(cryptodev.CRYPTO_AES_CBC, cipherkey, crid=crid) + r = curfun(c, pt, iv) + self.assertEqual(r, ct) + + def runXTS(self, fname, meth): + curfun = None + for mode, lines in cryptodev.KATParser(fname, + [ 'COUNT', 'DataUnitLen', 'Key', 'DataUnitSeqNumber', 'PT', + 'CT' ]): + if mode == 'ENCRYPT': + swapptct = False + curfun = Crypto.encrypt + elif mode == 'DECRYPT': + swapptct = True + curfun = Crypto.decrypt + else: + raise RuntimeError('unknown mode: %s' % `mode`) + + for data in lines: + curcnt = int(data['COUNT']) + nbits = int(data['DataUnitLen']) + cipherkey = data['Key'].decode('hex') + iv = struct.pack('QQ', int(data['DataUnitSeqNumber']), 0) + pt = data['PT'].decode('hex') + ct = data['CT'].decode('hex') + + if nbits % 128 != 0: + # XXX - mark as skipped + continue + if swapptct: + pt, ct = ct, pt + # run the fun + c = Crypto(meth, cipherkey, crid=crid) + r = curfun(c, pt, iv) + self.assertEqual(r, ct) + + ############### + ##### DES ##### + ############### + @unittest.skipIf(cname not in desmodules, 'skipping DES on %s' % `cname`) + def test_tdes(self): + for i in katg('KAT_TDES', 'TCBC[a-z]*.rsp'): + self.runTDES(i) + + def runTDES(self, fname): + curfun = None + for mode, lines in cryptodev.KATParser(fname, + [ 'COUNT', 'KEYs', 'IV', 'PLAINTEXT', 'CIPHERTEXT', ]): + if mode == 'ENCRYPT': + swapptct = False + curfun = Crypto.encrypt + elif mode == 'DECRYPT': + swapptct = True + curfun = Crypto.decrypt + else: + raise RuntimeError('unknown mode: %s' % `mode`) + + for data in lines: + curcnt = int(data['COUNT']) + key = data['KEYs'] * 3 + cipherkey = key.decode('hex') + iv = data['IV'].decode('hex') + pt = data['PLAINTEXT'].decode('hex') + ct = data['CIPHERTEXT'].decode('hex') + + if swapptct: + pt, ct = ct, pt + # run the fun + c = Crypto(cryptodev.CRYPTO_3DES_CBC, cipherkey, crid=crid) + r = curfun(c, pt, iv) + self.assertEqual(r, ct) + + ############### + ##### SHA ##### + ############### + @unittest.skipIf(cname not in shamodules, 'skipping SHA on %s' % `cname`) + def test_sha(self): + # SHA not available in software + pass + #for i in iglob('SHA1*'): + # self.runSHA(i) + + def test_sha1hmac(self): + for i in katg('hmactestvectors', 'HMAC.rsp'): + self.runSHA1HMAC(i) + + def runSHA1HMAC(self, fname): + for bogusmode, lines in cryptodev.KATParser(fname, + [ 'Count', 'Klen', 'Tlen', 'Key', 'Msg', 'Mac' ]): + for data in lines: + key = data['Key'].decode('hex') + msg = data['Msg'].decode('hex') + mac = data['Mac'].decode('hex') + + if len(key) != 20: + # XXX - implementation bug + continue + + c = Crypto(mac=cryptodev.CRYPTO_SHA1_HMAC, + mackey=key, crid=crid) + + r = c.encrypt(msg) + self.assertEqual(r, mac, `data`) + + return GendCryptoTestCase + +cryptosoft = GenTestCase('cryptosoft0') +aesni = GenTestCase('aesni0') + +if __name__ == '__main__': + unittest.main() Property changes on: projects/clang350-import/tests/sys/opencrypto/cryptotest.py ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: projects/clang350-import/tests/sys/opencrypto/dpkt.py =================================================================== --- projects/clang350-import/tests/sys/opencrypto/dpkt.py (nonexistent) +++ projects/clang350-import/tests/sys/opencrypto/dpkt.py (revision 275749) @@ -0,0 +1,160 @@ +# $FreeBSD$ +# $Id: dpkt.py 114 2005-09-11 15:15:12Z dugsong $ + +"""fast, simple packet creation / parsing, with definitions for the +basic TCP/IP protocols. +""" + +__author__ = 'Dug Song ' +__copyright__ = 'Copyright (c) 2004 Dug Song' +__license__ = 'BSD' +__url__ = 'http://monkey.org/~dugsong/dpkt/' +__version__ = '1.2' + +try: + from itertools import izip as _it_izip +except ImportError: + _it_izip = zip + +from struct import calcsize as _st_calcsize, \ + pack as _st_pack, unpack as _st_unpack, error as _st_error +from re import compile as _re_compile + +intchr = _re_compile(r"(?P[0-9]+)(?P.)") + +class MetaPacket(type): + def __new__(cls, clsname, clsbases, clsdict): + if '__hdr__' in clsdict: + st = clsdict['__hdr__'] + clsdict['__hdr_fields__'] = [ x[0] for x in st ] + clsdict['__hdr_fmt__'] = clsdict.get('__byte_order__', '>') + \ + ''.join([ x[1] for x in st ]) + clsdict['__hdr_len__'] = _st_calcsize(clsdict['__hdr_fmt__']) + clsdict['__hdr_defaults__'] = \ + dict(zip(clsdict['__hdr_fields__'], [ x[2] for x in st ])) + clsdict['__slots__'] = clsdict['__hdr_fields__'] + return type.__new__(cls, clsname, clsbases, clsdict) + +class Packet(object): + """Packet class + + __hdr__ should be defined as a list of (name, structfmt, default) tuples + __byte_order__ can be set to override the default ('>') + """ + __metaclass__ = MetaPacket + data = '' + + def __init__(self, *args, **kwargs): + """Packet constructor with ([buf], [field=val,...]) prototype. + + Arguments: + + buf -- packet buffer to unpack + + Optional keyword arguments correspond to packet field names. + """ + if args: + self.unpack(args[0]) + else: + for k in self.__hdr_fields__: + setattr(self, k, self.__hdr_defaults__[k]) + for k, v in kwargs.iteritems(): + setattr(self, k, v) + + def __len__(self): + return self.__hdr_len__ + len(self.data) + + def __repr__(self): + l = [ '%s=%r' % (k, getattr(self, k)) + for k in self.__hdr_defaults__ + if getattr(self, k) != self.__hdr_defaults__[k] ] + if self.data: + l.append('data=%r' % self.data) + return '%s(%s)' % (self.__class__.__name__, ', '.join(l)) + + def __str__(self): + return self.pack_hdr() + str(self.data) + + def pack_hdr(self): + """Return packed header string.""" + try: + return _st_pack(self.__hdr_fmt__, + *[ getattr(self, k) for k in self.__hdr_fields__ ]) + except _st_error: + vals = [] + for k in self.__hdr_fields__: + v = getattr(self, k) + if isinstance(v, tuple): + vals.extend(v) + else: + vals.append(v) + return _st_pack(self.__hdr_fmt__, *vals) + + def unpack(self, buf): + """Unpack packet header fields from buf, and set self.data.""" + + res = list(_st_unpack(self.__hdr_fmt__, buf[:self.__hdr_len__])) + for e, k in enumerate(self.__slots__): + sfmt = self.__hdr__[e][1] + mat = intchr.match(sfmt) + if mat and mat.group('chr') != 's': + cnt = int(mat.group('int')) + setattr(self, k, list(res[:cnt])) + del res[:cnt] + else: + if sfmt[-1] == 's': + i = res[0].find('\x00') + if i != -1: + res[0] = res[0][:i] + setattr(self, k, res[0]) + del res[0] + assert len(res) == 0 + self.data = buf[self.__hdr_len__:] + +# XXX - ''.join([(len(`chr(x)`)==3) and chr(x) or '.' for x in range(256)]) +__vis_filter = """................................ !"#$%&\'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[.]^_`abcdefghijklmnopqrstuvwxyz{|}~.................................................................................................................................""" + +def hexdump(buf, length=16): + """Return a hexdump output string of the given buffer.""" + n = 0 + res = [] + while buf: + line, buf = buf[:length], buf[length:] + hexa = ' '.join(['%02x' % ord(x) for x in line]) + line = line.translate(__vis_filter) + res.append(' %04d: %-*s %s' % (n, length * 3, hexa, line)) + n += length + return '\n'.join(res) + +def in_cksum_add(s, buf): + """in_cksum_add(cksum, buf) -> cksum + + Return accumulated Internet checksum. + """ + nleft = len(buf) + i = 0 + while nleft > 1: + s += ord(buf[i]) * 256 + ord(buf[i+1]) + i += 2 + nleft -= 2 + if nleft: + s += ord(buf[i]) * 256 + return s + +def in_cksum_done(s): + """Fold and return Internet checksum.""" + while (s >> 16): + s = (s >> 16) + (s & 0xffff) + return (~s & 0xffff) + +def in_cksum(buf): + """Return computed Internet checksum.""" + return in_cksum_done(in_cksum_add(0, buf)) + +try: + import psyco + psyco.bind(in_cksum) + psyco.bind(Packet) +except ImportError: + pass + Property changes on: projects/clang350-import/tests/sys/opencrypto/dpkt.py ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: projects/clang350-import/tests/sys/opencrypto/runtests.sh =================================================================== --- projects/clang350-import/tests/sys/opencrypto/runtests.sh (nonexistent) +++ projects/clang350-import/tests/sys/opencrypto/runtests.sh (revision 275749) @@ -0,0 +1,60 @@ +#!/bin/sh - +# +# Copyright (c) 2014 The FreeBSD Foundation +# All rights reserved. +# +# This software was developed by John-Mark Gurney under +# the sponsorship from the FreeBSD Foundation. +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# 1. Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# 2. Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# +# THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND +# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +# ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE +# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS +# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) +# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT +# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY +# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF +# SUCH DAMAGE. +# +# $FreeBSD$ +# + +set -e + +if [ ! -d /usr/local/share/nist-kat ]; then + echo 'Skipping, nist-kat package not installed for test vectors.' + exit 0 +fi + +if kldload aesni 2>/dev/null; then + unloadaesni=1 +fi + +if kldload cryptodev 2>/dev/null; then + unloadcdev=1 +fi + +# Run software crypto test +oldcdas=$(sysctl -e kern.cryptodevallowsoft) +sysctl kern.cryptodevallowsoft=1 + +python $(dirname $0)/cryptotest.py + +sysctl "$oldcdas" + +if [ x"$unloadcdev" = x"1" ]; then + kldunload cryptodev +fi +if [ x"$unloadaesni" = x"1" ]; then + kldunload aesni +fi Property changes on: projects/clang350-import/tests/sys/opencrypto/runtests.sh ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:executable ## -0,0 +1 ## +* \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: projects/clang350-import/usr.bin/mkimg/Makefile =================================================================== --- projects/clang350-import/usr.bin/mkimg/Makefile (revision 275748) +++ projects/clang350-import/usr.bin/mkimg/Makefile (revision 275749) @@ -1,42 +1,42 @@ # $FreeBSD$ .include PROG= mkimg SRCS= format.c image.c mkimg.c scheme.c MAN= mkimg.1 -MKIMG_VERSION=20141003 +MKIMG_VERSION=20141211 mkimg.o: Makefile CFLAGS+=-DMKIMG_VERSION=${MKIMG_VERSION} CFLAGS+=-DSPARSE_WRITE # List of formats to support SRCS+= \ qcow.c \ raw.c \ vhd.c \ vmdk.c # List of schemes to support SRCS+= \ apm.c \ bsd.c \ ebr.c \ gpt.c \ mbr.c \ pc98.c \ vtoc8.c BINDIR?=/usr/bin LIBADD= util WARNS?= 6 .if ${MK_TESTS} != "no" SUBDIR+= tests .endif .include Index: projects/clang350-import/usr.bin/mkimg/qcow.c =================================================================== --- projects/clang350-import/usr.bin/mkimg/qcow.c (revision 275748) +++ projects/clang350-import/usr.bin/mkimg/qcow.c (revision 275749) @@ -1,369 +1,370 @@ /*- * Copyright (c) 2014 Marcel Moolenaar * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #include __FBSDID("$FreeBSD$"); #include #include #include #include #include #include #include #include #include "image.h" #include "format.h" #include "mkimg.h" /* Default cluster sizes. */ #define QCOW1_CLSTR_LOG2SZ 12 /* 4KB */ #define QCOW2_CLSTR_LOG2SZ 16 /* 64KB */ /* Flag bits in cluster offsets */ #define QCOW_CLSTR_COMPRESSED (1ULL << 62) #define QCOW_CLSTR_COPIED (1ULL << 63) struct qcow_header { uint32_t magic; #define QCOW_MAGIC 0x514649fb uint32_t version; #define QCOW_VERSION_1 1 #define QCOW_VERSION_2 2 uint64_t path_offset; uint32_t path_length; uint32_t clstr_log2sz; /* v2 only */ uint64_t disk_size; union { struct { uint8_t clstr_log2sz; uint8_t l2_log2sz; uint16_t _pad; uint32_t encryption; uint64_t l1_offset; } v1; struct { uint32_t encryption; uint32_t l1_entries; uint64_t l1_offset; uint64_t refcnt_offset; - uint32_t refcnt_entries; + uint32_t refcnt_clstrs; uint32_t snapshot_count; uint64_t snapshot_offset; } v2; } u; }; static u_int clstr_log2sz; static uint64_t round_clstr(uint64_t ofs) { uint64_t clstrsz; clstrsz = 1UL << clstr_log2sz; return ((ofs + clstrsz - 1) & ~(clstrsz - 1)); } static int qcow_resize(lba_t imgsz, u_int version) { uint64_t imagesz; switch (version) { case QCOW_VERSION_1: clstr_log2sz = QCOW1_CLSTR_LOG2SZ; break; case QCOW_VERSION_2: clstr_log2sz = QCOW2_CLSTR_LOG2SZ; break; default: return (EDOOFUS); } imagesz = round_clstr(imgsz * secsz); if (verbose) fprintf(stderr, "QCOW: image size = %ju, cluster size = %u\n", (uintmax_t)imagesz, (u_int)(1U << clstr_log2sz)); return (image_set_size(imagesz / secsz)); } static int qcow1_resize(lba_t imgsz) { return (qcow_resize(imgsz, QCOW_VERSION_1)); } static int qcow2_resize(lba_t imgsz) { return (qcow_resize(imgsz, QCOW_VERSION_2)); } static int qcow_write(int fd, u_int version) { struct qcow_header *hdr; uint64_t *l1tbl, *l2tbl, *rctbl; uint16_t *rcblk; uint64_t clstr_imgsz, clstr_l2tbls, clstr_l1tblsz; uint64_t clstr_rcblks, clstr_rctblsz; uint64_t n, imagesz, nclstrs, ofs, ofsflags; lba_t blk, blkofs, blk_imgsz; u_int l1clno, l2clno, rcclno; - u_int blk_clstrsz; + u_int blk_clstrsz, refcnt_clstrs; u_int clstrsz, l1idx, l2idx; int error; if (clstr_log2sz == 0) return (EDOOFUS); clstrsz = 1U << clstr_log2sz; blk_clstrsz = clstrsz / secsz; blk_imgsz = image_get_size(); imagesz = blk_imgsz * secsz; clstr_imgsz = imagesz >> clstr_log2sz; clstr_l2tbls = round_clstr(clstr_imgsz * 8) >> clstr_log2sz; clstr_l1tblsz = round_clstr(clstr_l2tbls * 8) >> clstr_log2sz; nclstrs = clstr_imgsz + clstr_l2tbls + clstr_l1tblsz + 1; clstr_rcblks = clstr_rctblsz = 0; do { n = clstr_rcblks + clstr_rctblsz; clstr_rcblks = round_clstr((nclstrs + n) * 2) >> clstr_log2sz; clstr_rctblsz = round_clstr(clstr_rcblks * 8) >> clstr_log2sz; } while (n < (clstr_rcblks + clstr_rctblsz)); /* * We got all the sizes in clusters. Start the layout. * 0 - header * 1 - L1 table * 2 - RC table (v2 only) * 3 - L2 tables * 4 - RC block (v2 only) * 5 - data */ l1clno = 1; rcclno = 0; rctbl = l2tbl = l1tbl = NULL; rcblk = NULL; hdr = calloc(1, clstrsz); if (hdr == NULL) return (errno); be32enc(&hdr->magic, QCOW_MAGIC); be32enc(&hdr->version, version); be64enc(&hdr->disk_size, imagesz); switch (version) { case QCOW_VERSION_1: ofsflags = 0; l2clno = l1clno + clstr_l1tblsz; hdr->u.v1.clstr_log2sz = clstr_log2sz; hdr->u.v1.l2_log2sz = clstr_log2sz - 3; be64enc(&hdr->u.v1.l1_offset, clstrsz * l1clno); break; case QCOW_VERSION_2: ofsflags = QCOW_CLSTR_COPIED; rcclno = l1clno + clstr_l1tblsz; l2clno = rcclno + clstr_rctblsz; be32enc(&hdr->clstr_log2sz, clstr_log2sz); be32enc(&hdr->u.v2.l1_entries, clstr_l2tbls); be64enc(&hdr->u.v2.l1_offset, clstrsz * l1clno); be64enc(&hdr->u.v2.refcnt_offset, clstrsz * rcclno); - be32enc(&hdr->u.v2.refcnt_entries, clstr_rcblks); + refcnt_clstrs = round_clstr(clstr_rcblks * 8) >> clstr_log2sz; + be32enc(&hdr->u.v2.refcnt_clstrs, refcnt_clstrs); break; default: return (EDOOFUS); } if (sparse_write(fd, hdr, clstrsz) < 0) { - error = errno; + error = errno; goto out; } free(hdr); hdr = NULL; ofs = clstrsz * l2clno; nclstrs = 1 + clstr_l1tblsz + clstr_rctblsz; l1tbl = calloc(1, clstrsz * clstr_l1tblsz); if (l1tbl == NULL) { error = ENOMEM; goto out; } for (n = 0; n < clstr_imgsz; n++) { blk = n * blk_clstrsz; if (image_data(blk, blk_clstrsz)) { nclstrs++; l1idx = n >> (clstr_log2sz - 3); if (l1tbl[l1idx] == 0) { be64enc(l1tbl + l1idx, ofs + ofsflags); ofs += clstrsz; nclstrs++; } } } if (sparse_write(fd, l1tbl, clstrsz * clstr_l1tblsz) < 0) { error = errno; goto out; } clstr_rcblks = 0; do { n = clstr_rcblks; clstr_rcblks = round_clstr((nclstrs + n) * 2) >> clstr_log2sz; } while (n < clstr_rcblks); if (rcclno > 0) { rctbl = calloc(1, clstrsz * clstr_rctblsz); if (rctbl == NULL) { error = ENOMEM; goto out; } for (n = 0; n < clstr_rcblks; n++) { be64enc(rctbl + n, ofs); ofs += clstrsz; nclstrs++; } if (sparse_write(fd, rctbl, clstrsz * clstr_rctblsz) < 0) { error = errno; goto out; } free(rctbl); rctbl = NULL; } l2tbl = malloc(clstrsz); if (l2tbl == NULL) { error = ENOMEM; goto out; } for (l1idx = 0; l1idx < clstr_l2tbls; l1idx++) { if (l1tbl[l1idx] == 0) continue; memset(l2tbl, 0, clstrsz); blkofs = (lba_t)l1idx * blk_clstrsz * (clstrsz >> 3); for (l2idx = 0; l2idx < (clstrsz >> 3); l2idx++) { blk = blkofs + (lba_t)l2idx * blk_clstrsz; if (blk >= blk_imgsz) break; if (image_data(blk, blk_clstrsz)) { be64enc(l2tbl + l2idx, ofs + ofsflags); ofs += clstrsz; } } if (sparse_write(fd, l2tbl, clstrsz) < 0) { error = errno; goto out; } } free(l2tbl); l2tbl = NULL; free(l1tbl); l1tbl = NULL; if (rcclno > 0) { rcblk = calloc(1, clstrsz * clstr_rcblks); if (rcblk == NULL) { error = ENOMEM; goto out; } for (n = 0; n < nclstrs; n++) be16enc(rcblk + n, 1); if (sparse_write(fd, rcblk, clstrsz * clstr_rcblks) < 0) { error = errno; goto out; } free(rcblk); rcblk = NULL; } error = 0; for (n = 0; n < clstr_imgsz; n++) { blk = n * blk_clstrsz; if (image_data(blk, blk_clstrsz)) { error = image_copyout_region(fd, blk, blk_clstrsz); if (error) break; } } if (!error) error = image_copyout_done(fd); out: if (rcblk != NULL) free(rcblk); if (l2tbl != NULL) free(l2tbl); if (rctbl != NULL) free(rctbl); if (l1tbl != NULL) free(l1tbl); if (hdr != NULL) free(hdr); return (error); } static int qcow1_write(int fd) { return (qcow_write(fd, QCOW_VERSION_1)); } static int qcow2_write(int fd) { return (qcow_write(fd, QCOW_VERSION_2)); } static struct mkimg_format qcow1_format = { .name = "qcow", .description = "QEMU Copy-On-Write, version 1", .resize = qcow1_resize, .write = qcow1_write, }; FORMAT_DEFINE(qcow1_format); static struct mkimg_format qcow2_format = { .name = "qcow2", .description = "QEMU Copy-On-Write, version 2", .resize = qcow2_resize, .write = qcow2_write, }; FORMAT_DEFINE(qcow2_format); Index: projects/clang350-import/usr.bin/mkimg =================================================================== --- projects/clang350-import/usr.bin/mkimg (revision 275748) +++ projects/clang350-import/usr.bin/mkimg (revision 275749) Property changes on: projects/clang350-import/usr.bin/mkimg ___________________________________________________________________ Modified: svn:mergeinfo ## -0,0 +0,1 ## Merged /head/usr.bin/mkimg:r274961,275076-275748 Index: projects/clang350-import/usr.sbin/autofs/autounmountd.8 =================================================================== --- projects/clang350-import/usr.sbin/autofs/autounmountd.8 (revision 275748) +++ projects/clang350-import/usr.sbin/autofs/autounmountd.8 (revision 275749) @@ -1,88 +1,88 @@ .\" Copyright (c) 2014 The FreeBSD Foundation .\" All rights reserved. .\" .\" This software was developed by Edward Tomasz Napierala under sponsorship .\" from the FreeBSD Foundation. .\" .\" Redistribution and use in source and binary forms, with or without .\" modification, are permitted provided that the following conditions .\" are met: .\" 1. Redistributions of source code must retain the above copyright .\" notice, this list of conditions and the following disclaimer. .\" 2. Redistributions in binary form must reproduce the above copyright .\" notice, this list of conditions and the following disclaimer in the .\" documentation and/or other materials provided with the distribution. .\" .\" THIS SOFTWARE IS PROVIDED BY THE AUTHORS AND CONTRIBUTORS ``AS IS'' AND .\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE .\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE .\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE LIABLE .\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL .\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS .\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) .\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT .\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY .\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF .\" SUCH DAMAGE. .\" .\" $FreeBSD$ .\" -.Dd April 20, 2014 +.Dd December 13, 2014 .Dt AUTOUNMOUNTD 8 .Os .Sh NAME .Nm autounmountd .Nd daemon unmounting automounted filesystems .Sh SYNOPSIS .Nm .Op Fl d -.Op Fl r time -.Op Fl t time +.Op Fl r Ar time +.Op Fl t Ar time .Op Fl v .Sh DESCRIPTION The .Nm daemon is responsible for unmounting filesystems mounted by .Xr automountd 8 . On startup, .Nm retrieves a list of filesystems that have the .Li automounted mount option set. The list is updated every time a filesystem is mounted or unmounted. After a specified time passes, .Nm attempts to unmount a filesystem, retrying after some time if necessary. .Pp These options are available: .Bl -tag -width ".Fl v" .It Fl d Debug mode: increase verbosity and do not daemonize. .It Fl r Number of seconds to wait before trying to unmount an expired filesystem after a previous attempt failed, possibly due to filesystem being busy. The default value is 600, or ten minutes. .It Fl t Number of seconds to wait before trying to unmount a filesystem. The default value is 600, or ten minutes. .It Fl v Increase verbosity. .El .Sh EXIT STATUS .Ex -std .Sh SEE ALSO .Xr auto_master 5 , .Xr autofs 5 , .Xr automount 8 , .Xr automountd 8 .Sh HISTORY The .Nm daemon appeared in .Fx 10.1 . .Sh AUTHORS The .Nm was developed by .An Edward Tomasz Napierala Aq Mt trasz@FreeBSD.org under sponsorship from the FreeBSD Foundation. Index: projects/clang350-import/usr.sbin/syslogd/syslogd.c =================================================================== --- projects/clang350-import/usr.sbin/syslogd/syslogd.c (revision 275748) +++ projects/clang350-import/usr.sbin/syslogd/syslogd.c (revision 275749) @@ -1,2759 +1,2759 @@ /* * Copyright (c) 1983, 1988, 1993, 1994 * The Regents of the University of California. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 4. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #ifndef lint static const char copyright[] = "@(#) Copyright (c) 1983, 1988, 1993, 1994\n\ The Regents of the University of California. All rights reserved.\n"; #endif /* not lint */ #ifndef lint #if 0 static char sccsid[] = "@(#)syslogd.c 8.3 (Berkeley) 4/4/94"; #endif #endif /* not lint */ #include __FBSDID("$FreeBSD$"); /* * syslogd -- log system messages * * This program implements a system log. It takes a series of lines. * Each line may have a priority, signified as "" as * the first characters of the line. If this is * not present, a default priority is used. * * To kill syslogd, send a signal 15 (terminate). A signal 1 (hup) will * cause it to reread its configuration file. * * Defined Constants: * * MAXLINE -- the maximum line length that can be handled. * DEFUPRI -- the default priority for user messages * DEFSPRI -- the default priority for kernel messages * * Author: Eric Allman * extensive changes by Ralph Campbell * more extensive changes by Eric Allman (again) * Extension to log by program name as well as facility and priority * by Peter da Silva. * -u and -v by Harlan Stenn. * Priority comparison code by Harlan Stenn. */ #define MAXLINE 1024 /* maximum line length */ #define MAXSVLINE 120 /* maximum saved line length */ #define DEFUPRI (LOG_USER|LOG_NOTICE) #define DEFSPRI (LOG_KERN|LOG_CRIT) #define TIMERINTVL 30 /* interval for checking flush, mark */ #define TTYMSGTIME 1 /* timeout passed to ttymsg */ #define RCVBUF_MINSIZE (80 * 1024) /* minimum size of dgram rcv buffer */ #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include "pathnames.h" #include "ttymsg.h" #define SYSLOG_NAMES #include const char *ConfFile = _PATH_LOGCONF; const char *PidFile = _PATH_LOGPID; const char ctty[] = _PATH_CONSOLE; #define dprintf if (Debug) printf #define MAXUNAMES 20 /* maximum number of user names */ /* * Unix sockets. * We have two default sockets, one with 666 permissions, * and one for privileged programs. */ struct funix { int s; const char *name; mode_t mode; STAILQ_ENTRY(funix) next; }; struct funix funix_secure = { -1, _PATH_LOG_PRIV, S_IRUSR | S_IWUSR, { NULL } }; struct funix funix_default = { -1, _PATH_LOG, DEFFILEMODE, { &funix_secure } }; STAILQ_HEAD(, funix) funixes = { &funix_default, &(funix_secure.next.stqe_next) }; /* * Flags to logmsg(). */ #define IGN_CONS 0x001 /* don't print on console */ #define SYNC_FILE 0x002 /* do fsync on file after printing */ #define ADDDATE 0x004 /* add a date to the message */ #define MARK 0x008 /* this message is a mark */ #define ISKERNEL 0x010 /* kernel generated message */ /* * This structure represents the files that will have log * copies printed. * We require f_file to be valid if f_type is F_FILE, F_CONSOLE, F_TTY * or if f_type if F_PIPE and f_pid > 0. */ struct filed { struct filed *f_next; /* next in linked list */ short f_type; /* entry type, see below */ short f_file; /* file descriptor */ time_t f_time; /* time this was last written */ char *f_host; /* host from which to recd. */ u_char f_pmask[LOG_NFACILITIES+1]; /* priority mask */ u_char f_pcmp[LOG_NFACILITIES+1]; /* compare priority */ #define PRI_LT 0x1 #define PRI_EQ 0x2 #define PRI_GT 0x4 char *f_program; /* program this applies to */ union { char f_uname[MAXUNAMES][MAXLOGNAME]; struct { char f_hname[MAXHOSTNAMELEN]; struct addrinfo *f_addr; } f_forw; /* forwarding address */ char f_fname[MAXPATHLEN]; struct { char f_pname[MAXPATHLEN]; pid_t f_pid; } f_pipe; } f_un; char f_prevline[MAXSVLINE]; /* last message logged */ char f_lasttime[16]; /* time of last occurrence */ char f_prevhost[MAXHOSTNAMELEN]; /* host from which recd. */ int f_prevpri; /* pri of f_prevline */ int f_prevlen; /* length of f_prevline */ int f_prevcount; /* repetition cnt of prevline */ u_int f_repeatcount; /* number of "repeated" msgs */ int f_flags; /* file-specific flags */ #define FFLAG_SYNC 0x01 #define FFLAG_NEEDSYNC 0x02 }; /* * Queue of about-to-be dead processes we should watch out for. */ TAILQ_HEAD(stailhead, deadq_entry) deadq_head; struct stailhead *deadq_headp; struct deadq_entry { pid_t dq_pid; int dq_timeout; TAILQ_ENTRY(deadq_entry) dq_entries; }; /* * The timeout to apply to processes waiting on the dead queue. Unit * of measure is `mark intervals', i.e. 20 minutes by default. * Processes on the dead queue will be terminated after that time. */ #define DQ_TIMO_INIT 2 typedef struct deadq_entry *dq_t; /* * Struct to hold records of network addresses that are allowed to log * to us. */ struct allowedpeer { int isnumeric; u_short port; union { struct { struct sockaddr_storage addr; struct sockaddr_storage mask; } numeric; char *name; } u; #define a_addr u.numeric.addr #define a_mask u.numeric.mask #define a_name u.name }; /* * Intervals at which we flush out "message repeated" messages, * in seconds after previous message is logged. After each flush, * we move to the next interval until we reach the largest. */ int repeatinterval[] = { 30, 120, 600 }; /* # of secs before flush */ #define MAXREPEAT ((sizeof(repeatinterval) / sizeof(repeatinterval[0])) - 1) #define REPEATTIME(f) ((f)->f_time + repeatinterval[(f)->f_repeatcount]) #define BACKOFF(f) { if (++(f)->f_repeatcount > MAXREPEAT) \ (f)->f_repeatcount = MAXREPEAT; \ } /* values for f_type */ #define F_UNUSED 0 /* unused entry */ #define F_FILE 1 /* regular file */ #define F_TTY 2 /* terminal */ #define F_CONSOLE 3 /* console terminal */ #define F_FORW 4 /* remote machine */ #define F_USERS 5 /* list of users */ #define F_WALL 6 /* everyone logged on */ #define F_PIPE 7 /* pipe to program */ const char *TypeNames[8] = { "UNUSED", "FILE", "TTY", "CONSOLE", "FORW", "USERS", "WALL", "PIPE" }; static struct filed *Files; /* Log files that we write to */ static struct filed consfile; /* Console */ static int Debug; /* debug flag */ static int resolve = 1; /* resolve hostname */ static char LocalHostName[MAXHOSTNAMELEN]; /* our hostname */ static const char *LocalDomain; /* our local domain name */ static int *finet; /* Internet datagram socket */ static int fklog = -1; /* /dev/klog */ static int Initialized; /* set when we have initialized ourselves */ static int MarkInterval = 20 * 60; /* interval between marks in seconds */ static int MarkSeq; /* mark sequence number */ static int NoBind; /* don't bind() as suggested by RFC 3164 */ static int SecureMode; /* when true, receive only unix domain socks */ #ifdef INET6 static int family = PF_UNSPEC; /* protocol family (IPv4, IPv6 or both) */ #else static int family = PF_INET; /* protocol family (IPv4 only) */ #endif static int mask_C1 = 1; /* mask characters from 0x80 - 0x9F */ static int send_to_all; /* send message to all IPv4/IPv6 addresses */ static int use_bootfile; /* log entire bootfile for every kern msg */ static int no_compress; /* don't compress messages (1=pipes, 2=all) */ static int logflags = O_WRONLY|O_APPEND; /* flags used to open log files */ static char bootfile[MAXLINE+1]; /* booted kernel file */ struct allowedpeer *AllowedPeers; /* List of allowed peers */ static int NumAllowed; /* Number of entries in AllowedPeers */ static int RemoteAddDate; /* Always set the date on remote messages */ static int UniquePriority; /* Only log specified priority? */ static int LogFacPri; /* Put facility and priority in log message: */ /* 0=no, 1=numeric, 2=names */ static int KeepKernFac; /* Keep remotely logged kernel facility */ static int needdofsync = 0; /* Are any file(s) waiting to be fsynced? */ static struct pidfh *pfh; volatile sig_atomic_t MarkSet, WantDie; static int allowaddr(char *); static void cfline(const char *, struct filed *, const char *, const char *); static const char *cvthname(struct sockaddr *); static void deadq_enter(pid_t, const char *); static int deadq_remove(pid_t); static int decode(const char *, const CODE *); static void die(int); static void dodie(int); static void dofsync(void); static void domark(int); static void fprintlog(struct filed *, int, const char *); static int *socksetup(int, char *); static void init(int); static void logerror(const char *); static void logmsg(int, const char *, const char *, int); static void log_deadchild(pid_t, int, const char *); static void markit(void); static int skip_message(const char *, const char *, int); static void printline(const char *, char *, int); static void printsys(char *); static int p_open(const char *, pid_t *); static void readklog(void); static void reapchild(int); static void usage(void); static int validate(struct sockaddr *, const char *); static void unmapped(struct sockaddr *); static void wallmsg(struct filed *, struct iovec *, const int iovlen); static int waitdaemon(int, int, int); static void timedout(int); static void increase_rcvbuf(int); int main(int argc, char *argv[]) { int ch, i, fdsrmax = 0, l; struct sockaddr_un sunx, fromunix; struct sockaddr_storage frominet; fd_set *fdsr = NULL; char line[MAXLINE + 1]; char *bindhostname; const char *hname; struct timeval tv, *tvp; struct sigaction sact; struct funix *fx, *fx1; sigset_t mask; pid_t ppid = 1, spid; socklen_t len; if (madvise(NULL, 0, MADV_PROTECT) != 0) dprintf("madvise() failed: %s\n", strerror(errno)); bindhostname = NULL; while ((ch = getopt(argc, argv, "468Aa:b:cCdf:kl:m:nNop:P:sS:Tuv")) != -1) switch (ch) { case '4': family = PF_INET; break; #ifdef INET6 case '6': family = PF_INET6; break; #endif case '8': mask_C1 = 0; break; case 'A': send_to_all++; break; case 'a': /* allow specific network addresses only */ if (allowaddr(optarg) == -1) usage(); break; case 'b': bindhostname = optarg; break; case 'c': no_compress++; break; case 'C': logflags |= O_CREAT; break; case 'd': /* debug */ Debug++; break; case 'f': /* configuration file */ ConfFile = optarg; break; case 'k': /* keep remote kern fac */ KeepKernFac = 1; break; case 'l': { long perml; mode_t mode; char *name, *ep; if (optarg[0] == '/') { mode = DEFFILEMODE; name = optarg; } else if ((name = strchr(optarg, ':')) != NULL) { *name++ = '\0'; if (name[0] != '/') errx(1, "socket name must be absolute " "path"); if (isdigit(*optarg)) { perml = strtol(optarg, &ep, 8); if (*ep || perml < 0 || perml & ~(S_IRWXU|S_IRWXG|S_IRWXO)) errx(1, "invalid mode %s, exiting", optarg); mode = (mode_t )perml; } else errx(1, "invalid mode %s, exiting", optarg); } else /* doesn't begin with '/', and no ':' */ errx(1, "can't parse path %s", optarg); if (strlen(name) >= sizeof(sunx.sun_path)) errx(1, "%s path too long, exiting", name); if ((fx = malloc(sizeof(struct funix))) == NULL) errx(1, "malloc failed"); fx->s = -1; fx->name = name; fx->mode = mode; STAILQ_INSERT_TAIL(&funixes, fx, next); break; } case 'm': /* mark interval */ MarkInterval = atoi(optarg) * 60; break; case 'N': NoBind = 1; SecureMode = 1; break; case 'n': resolve = 0; break; case 'o': use_bootfile = 1; break; case 'p': /* path */ if (strlen(optarg) >= sizeof(sunx.sun_path)) errx(1, "%s path too long, exiting", optarg); funix_default.name = optarg; break; case 'P': /* path for alt. PID */ PidFile = optarg; break; case 's': /* no network mode */ SecureMode++; break; case 'S': /* path for privileged originator */ if (strlen(optarg) >= sizeof(sunx.sun_path)) errx(1, "%s path too long, exiting", optarg); funix_secure.name = optarg; break; case 'T': RemoteAddDate = 1; break; case 'u': /* only log specified priority */ UniquePriority++; break; case 'v': /* log facility and priority */ LogFacPri++; break; default: usage(); } if ((argc -= optind) != 0) usage(); pfh = pidfile_open(PidFile, 0600, &spid); if (pfh == NULL) { if (errno == EEXIST) errx(1, "syslogd already running, pid: %d", spid); warn("cannot open pid file"); } if (!Debug) { ppid = waitdaemon(0, 0, 30); if (ppid < 0) { warn("could not become daemon"); pidfile_remove(pfh); exit(1); } } else { setlinebuf(stdout); } if (NumAllowed) endservent(); consfile.f_type = F_CONSOLE; (void)strlcpy(consfile.f_un.f_fname, ctty + sizeof _PATH_DEV - 1, sizeof(consfile.f_un.f_fname)); (void)strlcpy(bootfile, getbootfile(), sizeof(bootfile)); (void)signal(SIGTERM, dodie); (void)signal(SIGINT, Debug ? dodie : SIG_IGN); (void)signal(SIGQUIT, Debug ? dodie : SIG_IGN); /* * We don't want the SIGCHLD and SIGHUP handlers to interfere * with each other; they are likely candidates for being called * simultaneously (SIGHUP closes pipe descriptor, process dies, * SIGCHLD happens). */ sigemptyset(&mask); sigaddset(&mask, SIGHUP); sact.sa_handler = reapchild; sact.sa_mask = mask; sact.sa_flags = SA_RESTART; (void)sigaction(SIGCHLD, &sact, NULL); (void)signal(SIGALRM, domark); (void)signal(SIGPIPE, SIG_IGN); /* We'll catch EPIPE instead. */ (void)alarm(TIMERINTVL); TAILQ_INIT(&deadq_head); #ifndef SUN_LEN #define SUN_LEN(unp) (strlen((unp)->sun_path) + 2) #endif STAILQ_FOREACH_SAFE(fx, &funixes, next, fx1) { (void)unlink(fx->name); memset(&sunx, 0, sizeof(sunx)); sunx.sun_family = AF_LOCAL; (void)strlcpy(sunx.sun_path, fx->name, sizeof(sunx.sun_path)); fx->s = socket(PF_LOCAL, SOCK_DGRAM, 0); if (fx->s < 0 || bind(fx->s, (struct sockaddr *)&sunx, SUN_LEN(&sunx)) < 0 || chmod(fx->name, fx->mode) < 0) { (void)snprintf(line, sizeof line, "cannot create %s", fx->name); logerror(line); dprintf("cannot create %s (%d)\n", fx->name, errno); if (fx == &funix_default || fx == &funix_secure) die(0); else { STAILQ_REMOVE(&funixes, fx, funix, next); continue; } } increase_rcvbuf(fx->s); } if (SecureMode <= 1) finet = socksetup(family, bindhostname); if (finet) { if (SecureMode) { for (i = 0; i < *finet; i++) { if (shutdown(finet[i+1], SHUT_RD) < 0) { logerror("shutdown"); if (!Debug) die(0); } } } else { dprintf("listening on inet and/or inet6 socket\n"); } dprintf("sending on inet and/or inet6 socket\n"); } if ((fklog = open(_PATH_KLOG, O_RDONLY, 0)) >= 0) if (fcntl(fklog, F_SETFL, O_NONBLOCK) < 0) fklog = -1; if (fklog < 0) dprintf("can't open %s (%d)\n", _PATH_KLOG, errno); /* tuck my process id away */ pidfile_write(pfh); dprintf("off & running....\n"); init(0); /* prevent SIGHUP and SIGCHLD handlers from running in parallel */ sigemptyset(&mask); sigaddset(&mask, SIGCHLD); sact.sa_handler = init; sact.sa_mask = mask; sact.sa_flags = SA_RESTART; (void)sigaction(SIGHUP, &sact, NULL); tvp = &tv; tv.tv_sec = tv.tv_usec = 0; if (fklog != -1 && fklog > fdsrmax) fdsrmax = fklog; if (finet && !SecureMode) { for (i = 0; i < *finet; i++) { if (finet[i+1] != -1 && finet[i+1] > fdsrmax) fdsrmax = finet[i+1]; } } STAILQ_FOREACH(fx, &funixes, next) if (fx->s > fdsrmax) fdsrmax = fx->s; fdsr = (fd_set *)calloc(howmany(fdsrmax+1, NFDBITS), sizeof(fd_mask)); if (fdsr == NULL) errx(1, "calloc fd_set"); for (;;) { if (MarkSet) markit(); if (WantDie) die(WantDie); bzero(fdsr, howmany(fdsrmax+1, NFDBITS) * sizeof(fd_mask)); if (fklog != -1) FD_SET(fklog, fdsr); if (finet && !SecureMode) { for (i = 0; i < *finet; i++) { if (finet[i+1] != -1) FD_SET(finet[i+1], fdsr); } } STAILQ_FOREACH(fx, &funixes, next) FD_SET(fx->s, fdsr); i = select(fdsrmax+1, fdsr, NULL, NULL, needdofsync ? &tv : tvp); switch (i) { case 0: dofsync(); needdofsync = 0; if (tvp) { tvp = NULL; if (ppid != 1) kill(ppid, SIGALRM); } continue; case -1: if (errno != EINTR) logerror("select"); continue; } if (fklog != -1 && FD_ISSET(fklog, fdsr)) readklog(); if (finet && !SecureMode) { for (i = 0; i < *finet; i++) { if (FD_ISSET(finet[i+1], fdsr)) { len = sizeof(frominet); l = recvfrom(finet[i+1], line, MAXLINE, 0, (struct sockaddr *)&frominet, &len); if (l > 0) { line[l] = '\0'; hname = cvthname((struct sockaddr *)&frominet); unmapped((struct sockaddr *)&frominet); if (validate((struct sockaddr *)&frominet, hname)) printline(hname, line, RemoteAddDate ? ADDDATE : 0); } else if (l < 0 && errno != EINTR) logerror("recvfrom inet"); } } } STAILQ_FOREACH(fx, &funixes, next) { if (FD_ISSET(fx->s, fdsr)) { len = sizeof(fromunix); l = recvfrom(fx->s, line, MAXLINE, 0, (struct sockaddr *)&fromunix, &len); if (l > 0) { line[l] = '\0'; printline(LocalHostName, line, 0); } else if (l < 0 && errno != EINTR) logerror("recvfrom unix"); } } } if (fdsr) free(fdsr); } static void unmapped(struct sockaddr *sa) { struct sockaddr_in6 *sin6; struct sockaddr_in sin4; if (sa->sa_family != AF_INET6) return; if (sa->sa_len != sizeof(struct sockaddr_in6) || sizeof(sin4) > sa->sa_len) return; sin6 = (struct sockaddr_in6 *)sa; if (!IN6_IS_ADDR_V4MAPPED(&sin6->sin6_addr)) return; memset(&sin4, 0, sizeof(sin4)); sin4.sin_family = AF_INET; sin4.sin_len = sizeof(struct sockaddr_in); memcpy(&sin4.sin_addr, &sin6->sin6_addr.s6_addr[12], sizeof(sin4.sin_addr)); sin4.sin_port = sin6->sin6_port; memcpy(sa, &sin4, sin4.sin_len); } static void usage(void) { fprintf(stderr, "%s\n%s\n%s\n%s\n", "usage: syslogd [-468ACcdknosTuv] [-a allowed_peer]", " [-b bind_address] [-f config_file]", " [-l [mode:]path] [-m mark_interval]", " [-P pid_file] [-p log_socket]"); exit(1); } /* * Take a raw input line, decode the message, and print the message * on the appropriate log files. */ static void printline(const char *hname, char *msg, int flags) { char *p, *q; long n; int c, pri; char line[MAXLINE + 1]; /* test for special codes */ p = msg; pri = DEFUPRI; if (*p == '<') { errno = 0; n = strtol(p + 1, &q, 10); if (*q == '>' && n >= 0 && n < INT_MAX && errno == 0) { p = q + 1; pri = n; } } if (pri &~ (LOG_FACMASK|LOG_PRIMASK)) pri = DEFUPRI; /* * Don't allow users to log kernel messages. * NOTE: since LOG_KERN == 0 this will also match * messages with no facility specified. */ if ((pri & LOG_FACMASK) == LOG_KERN && !KeepKernFac) pri = LOG_MAKEPRI(LOG_USER, LOG_PRI(pri)); q = line; while ((c = (unsigned char)*p++) != '\0' && q < &line[sizeof(line) - 4]) { if (mask_C1 && (c & 0x80) && c < 0xA0) { c &= 0x7F; *q++ = 'M'; *q++ = '-'; } if (isascii(c) && iscntrl(c)) { if (c == '\n') { *q++ = ' '; } else if (c == '\t') { *q++ = '\t'; } else { *q++ = '^'; *q++ = c ^ 0100; } } else { *q++ = c; } } *q = '\0'; logmsg(pri, line, hname, flags); } /* * Read /dev/klog while data are available, split into lines. */ static void readklog(void) { char *p, *q, line[MAXLINE + 1]; int len, i; len = 0; for (;;) { i = read(fklog, line + len, MAXLINE - 1 - len); if (i > 0) { line[i + len] = '\0'; } else { if (i < 0 && errno != EINTR && errno != EAGAIN) { logerror("klog"); fklog = -1; } break; } for (p = line; (q = strchr(p, '\n')) != NULL; p = q + 1) { *q = '\0'; printsys(p); } len = strlen(p); if (len >= MAXLINE - 1) { printsys(p); len = 0; } if (len > 0) memmove(line, p, len + 1); } if (len > 0) printsys(line); } /* * Take a raw input line from /dev/klog, format similar to syslog(). */ static void printsys(char *msg) { char *p, *q; long n; int flags, isprintf, pri; flags = ISKERNEL | SYNC_FILE | ADDDATE; /* fsync after write */ p = msg; pri = DEFSPRI; isprintf = 1; if (*p == '<') { errno = 0; n = strtol(p + 1, &q, 10); if (*q == '>' && n >= 0 && n < INT_MAX && errno == 0) { p = q + 1; pri = n; isprintf = 0; } } /* * Kernel printf's and LOG_CONSOLE messages have been displayed * on the console already. */ if (isprintf || (pri & LOG_FACMASK) == LOG_CONSOLE) flags |= IGN_CONS; if (pri &~ (LOG_FACMASK|LOG_PRIMASK)) pri = DEFSPRI; logmsg(pri, p, LocalHostName, flags); } static time_t now; /* * Match a program or host name against a specification. * Return a non-0 value if the message must be ignored * based on the specification. */ static int skip_message(const char *name, const char *spec, int checkcase) { const char *s; char prev, next; int exclude = 0; /* Behaviour on explicit match */ if (spec == NULL) return 0; switch (*spec) { case '-': exclude = 1; /*FALLTHROUGH*/ case '+': spec++; break; default: break; } if (checkcase) s = strstr (spec, name); else s = strcasestr (spec, name); if (s != NULL) { prev = (s == spec ? ',' : *(s - 1)); next = *(s + strlen (name)); if (prev == ',' && (next == '\0' || next == ',')) /* Explicit match: skip iff the spec is an exclusive one. */ return exclude; } /* No explicit match for this name: skip the message iff the spec is an inclusive one. */ return !exclude; } /* * Log a message to the appropriate log files, users, etc. based on * the priority. */ static void logmsg(int pri, const char *msg, const char *from, int flags) { struct filed *f; int i, fac, msglen, omask, prilev; const char *timestamp; char prog[NAME_MAX+1]; char buf[MAXLINE+1]; dprintf("logmsg: pri %o, flags %x, from %s, msg %s\n", pri, flags, from, msg); omask = sigblock(sigmask(SIGHUP)|sigmask(SIGALRM)); /* * Check to see if msg looks non-standard. */ msglen = strlen(msg); if (msglen < 16 || msg[3] != ' ' || msg[6] != ' ' || msg[9] != ':' || msg[12] != ':' || msg[15] != ' ') flags |= ADDDATE; (void)time(&now); if (flags & ADDDATE) { timestamp = ctime(&now) + 4; } else { timestamp = msg; msg += 16; msglen -= 16; } /* skip leading blanks */ while (isspace(*msg)) { msg++; msglen--; } /* extract facility and priority level */ if (flags & MARK) fac = LOG_NFACILITIES; else fac = LOG_FAC(pri); /* Check maximum facility number. */ if (fac > LOG_NFACILITIES) { (void)sigsetmask(omask); return; } prilev = LOG_PRI(pri); /* extract program name */ for (i = 0; i < NAME_MAX; i++) { if (!isprint(msg[i]) || msg[i] == ':' || msg[i] == '[' || msg[i] == '/' || isspace(msg[i])) break; prog[i] = msg[i]; } prog[i] = 0; /* add kernel prefix for kernel messages */ if (flags & ISKERNEL) { snprintf(buf, sizeof(buf), "%s: %s", use_bootfile ? bootfile : "kernel", msg); msg = buf; msglen = strlen(buf); } /* log the message to the particular outputs */ if (!Initialized) { f = &consfile; /* * Open in non-blocking mode to avoid hangs during open * and close(waiting for the port to drain). */ f->f_file = open(ctty, O_WRONLY | O_NONBLOCK, 0); if (f->f_file >= 0) { (void)strlcpy(f->f_lasttime, timestamp, sizeof(f->f_lasttime)); fprintlog(f, flags, msg); (void)close(f->f_file); } (void)sigsetmask(omask); return; } for (f = Files; f; f = f->f_next) { /* skip messages that are incorrect priority */ if (!(((f->f_pcmp[fac] & PRI_EQ) && (f->f_pmask[fac] == prilev)) ||((f->f_pcmp[fac] & PRI_LT) && (f->f_pmask[fac] < prilev)) ||((f->f_pcmp[fac] & PRI_GT) && (f->f_pmask[fac] > prilev)) ) || f->f_pmask[fac] == INTERNAL_NOPRI) continue; /* skip messages with the incorrect hostname */ if (skip_message(from, f->f_host, 0)) continue; /* skip messages with the incorrect program name */ if (skip_message(prog, f->f_program, 1)) continue; /* skip message to console if it has already been printed */ if (f->f_type == F_CONSOLE && (flags & IGN_CONS)) continue; /* don't output marks to recently written files */ if ((flags & MARK) && (now - f->f_time) < MarkInterval / 2) continue; /* * suppress duplicate lines to this file */ if (no_compress - (f->f_type != F_PIPE) < 1 && (flags & MARK) == 0 && msglen == f->f_prevlen && f->f_prevline && !strcmp(msg, f->f_prevline) && !strcasecmp(from, f->f_prevhost)) { (void)strlcpy(f->f_lasttime, timestamp, sizeof(f->f_lasttime)); f->f_prevcount++; dprintf("msg repeated %d times, %ld sec of %d\n", f->f_prevcount, (long)(now - f->f_time), repeatinterval[f->f_repeatcount]); /* * If domark would have logged this by now, * flush it now (so we don't hold isolated messages), * but back off so we'll flush less often * in the future. */ if (now > REPEATTIME(f)) { fprintlog(f, flags, (char *)NULL); BACKOFF(f); } } else { /* new line, save it */ if (f->f_prevcount) fprintlog(f, 0, (char *)NULL); f->f_repeatcount = 0; f->f_prevpri = pri; (void)strlcpy(f->f_lasttime, timestamp, sizeof(f->f_lasttime)); (void)strlcpy(f->f_prevhost, from, sizeof(f->f_prevhost)); if (msglen < MAXSVLINE) { f->f_prevlen = msglen; (void)strlcpy(f->f_prevline, msg, sizeof(f->f_prevline)); fprintlog(f, flags, (char *)NULL); } else { f->f_prevline[0] = 0; f->f_prevlen = 0; fprintlog(f, flags, msg); } } } (void)sigsetmask(omask); } static void dofsync(void) { struct filed *f; for (f = Files; f; f = f->f_next) { if ((f->f_type == F_FILE) && (f->f_flags & FFLAG_NEEDSYNC)) { f->f_flags &= ~FFLAG_NEEDSYNC; (void)fsync(f->f_file); } } } #define IOV_SIZE 7 static void fprintlog(struct filed *f, int flags, const char *msg) { struct iovec iov[IOV_SIZE]; struct iovec *v; struct addrinfo *r; int i, l, lsent = 0; char line[MAXLINE + 1], repbuf[80], greetings[200], *wmsg = NULL; char nul[] = "", space[] = " ", lf[] = "\n", crlf[] = "\r\n"; const char *msgret; v = iov; if (f->f_type == F_WALL) { v->iov_base = greetings; /* The time displayed is not synchornized with the other log * destinations (like messages). Following fragment was using * ctime(&now), which was updating the time every 30 sec. * With f_lasttime, time is synchronized correctly. */ v->iov_len = snprintf(greetings, sizeof greetings, "\r\n\7Message from syslogd@%s at %.24s ...\r\n", f->f_prevhost, f->f_lasttime); if (v->iov_len >= sizeof greetings) v->iov_len = sizeof greetings - 1; v++; v->iov_base = nul; v->iov_len = 0; v++; } else { v->iov_base = f->f_lasttime; v->iov_len = strlen(f->f_lasttime); v++; v->iov_base = space; v->iov_len = 1; v++; } if (LogFacPri) { static char fp_buf[30]; /* Hollow laugh */ int fac = f->f_prevpri & LOG_FACMASK; int pri = LOG_PRI(f->f_prevpri); const char *f_s = NULL; char f_n[5]; /* Hollow laugh */ const char *p_s = NULL; char p_n[5]; /* Hollow laugh */ if (LogFacPri > 1) { const CODE *c; for (c = facilitynames; c->c_name; c++) { if (c->c_val == fac) { f_s = c->c_name; break; } } for (c = prioritynames; c->c_name; c++) { if (c->c_val == pri) { p_s = c->c_name; break; } } } if (!f_s) { snprintf(f_n, sizeof f_n, "%d", LOG_FAC(fac)); f_s = f_n; } if (!p_s) { snprintf(p_n, sizeof p_n, "%d", pri); p_s = p_n; } snprintf(fp_buf, sizeof fp_buf, "<%s.%s> ", f_s, p_s); v->iov_base = fp_buf; v->iov_len = strlen(fp_buf); } else { v->iov_base = nul; v->iov_len = 0; } v++; v->iov_base = f->f_prevhost; v->iov_len = strlen(v->iov_base); v++; v->iov_base = space; v->iov_len = 1; v++; if (msg) { wmsg = strdup(msg); /* XXX iov_base needs a `const' sibling. */ if (wmsg == NULL) { logerror("strdup"); exit(1); } v->iov_base = wmsg; v->iov_len = strlen(msg); } else if (f->f_prevcount > 1) { v->iov_base = repbuf; v->iov_len = snprintf(repbuf, sizeof repbuf, "last message repeated %d times", f->f_prevcount); } else if (f->f_prevline) { v->iov_base = f->f_prevline; v->iov_len = f->f_prevlen; } else { return; } v++; dprintf("Logging to %s", TypeNames[f->f_type]); f->f_time = now; switch (f->f_type) { int port; case F_UNUSED: dprintf("\n"); break; case F_FORW: port = (int)ntohs(((struct sockaddr_in *) (f->f_un.f_forw.f_addr->ai_addr))->sin_port); if (port != 514) { dprintf(" %s:%d\n", f->f_un.f_forw.f_hname, port); } else { dprintf(" %s\n", f->f_un.f_forw.f_hname); } /* check for local vs remote messages */ if (strcasecmp(f->f_prevhost, LocalHostName)) l = snprintf(line, sizeof line - 1, "<%d>%.15s Forwarded from %s: %s", f->f_prevpri, (char *)iov[0].iov_base, f->f_prevhost, (char *)iov[5].iov_base); else l = snprintf(line, sizeof line - 1, "<%d>%.15s %s", f->f_prevpri, (char *)iov[0].iov_base, (char *)iov[5].iov_base); if (l < 0) l = 0; else if (l > MAXLINE) l = MAXLINE; if (finet) { for (r = f->f_un.f_forw.f_addr; r; r = r->ai_next) { for (i = 0; i < *finet; i++) { #if 0 /* * should we check AF first, or just * trial and error? FWD */ if (r->ai_family == address_family_of(finet[i+1])) #endif lsent = sendto(finet[i+1], line, l, 0, r->ai_addr, r->ai_addrlen); if (lsent == l) break; } if (lsent == l && !send_to_all) break; } dprintf("lsent/l: %d/%d\n", lsent, l); if (lsent != l) { int e = errno; logerror("sendto"); errno = e; switch (errno) { case ENOBUFS: case ENETDOWN: case ENETUNREACH: case EHOSTUNREACH: case EHOSTDOWN: case EADDRNOTAVAIL: break; /* case EBADF: */ /* case EACCES: */ /* case ENOTSOCK: */ /* case EFAULT: */ /* case EMSGSIZE: */ /* case EAGAIN: */ /* case ENOBUFS: */ /* case ECONNREFUSED: */ default: dprintf("removing entry: errno=%d\n", e); f->f_type = F_UNUSED; break; } } } break; case F_FILE: dprintf(" %s\n", f->f_un.f_fname); v->iov_base = lf; v->iov_len = 1; if (writev(f->f_file, iov, IOV_SIZE) < 0) { /* * If writev(2) fails for potentially transient errors * like the filesystem being full, ignore it. * Otherwise remove this logfile from the list. */ if (errno != ENOSPC) { int e = errno; (void)close(f->f_file); f->f_type = F_UNUSED; errno = e; logerror(f->f_un.f_fname); } } else if ((flags & SYNC_FILE) && (f->f_flags & FFLAG_SYNC)) { f->f_flags |= FFLAG_NEEDSYNC; needdofsync = 1; } break; case F_PIPE: dprintf(" %s\n", f->f_un.f_pipe.f_pname); v->iov_base = lf; v->iov_len = 1; if (f->f_un.f_pipe.f_pid == 0) { if ((f->f_file = p_open(f->f_un.f_pipe.f_pname, &f->f_un.f_pipe.f_pid)) < 0) { f->f_type = F_UNUSED; logerror(f->f_un.f_pipe.f_pname); break; } } if (writev(f->f_file, iov, IOV_SIZE) < 0) { int e = errno; (void)close(f->f_file); if (f->f_un.f_pipe.f_pid > 0) deadq_enter(f->f_un.f_pipe.f_pid, f->f_un.f_pipe.f_pname); f->f_un.f_pipe.f_pid = 0; errno = e; logerror(f->f_un.f_pipe.f_pname); } break; case F_CONSOLE: if (flags & IGN_CONS) { dprintf(" (ignored)\n"); break; } /* FALLTHROUGH */ case F_TTY: dprintf(" %s%s\n", _PATH_DEV, f->f_un.f_fname); v->iov_base = crlf; v->iov_len = 2; errno = 0; /* ttymsg() only sometimes returns an errno */ if ((msgret = ttymsg(iov, IOV_SIZE, f->f_un.f_fname, 10))) { f->f_type = F_UNUSED; logerror(msgret); } break; case F_USERS: case F_WALL: dprintf("\n"); v->iov_base = crlf; v->iov_len = 2; wallmsg(f, iov, IOV_SIZE); break; } f->f_prevcount = 0; free(wmsg); } /* * WALLMSG -- Write a message to the world at large * * Write the specified message to either the entire * world, or a list of approved users. */ static void wallmsg(struct filed *f, struct iovec *iov, const int iovlen) { static int reenter; /* avoid calling ourselves */ struct utmpx *ut; int i; const char *p; if (reenter++) return; setutxent(); /* NOSTRICT */ while ((ut = getutxent()) != NULL) { if (ut->ut_type != USER_PROCESS) continue; if (f->f_type == F_WALL) { if ((p = ttymsg(iov, iovlen, ut->ut_line, TTYMSGTIME)) != NULL) { errno = 0; /* already in msg */ logerror(p); } continue; } /* should we send the message to this user? */ for (i = 0; i < MAXUNAMES; i++) { if (!f->f_un.f_uname[i][0]) break; if (!strcmp(f->f_un.f_uname[i], ut->ut_user)) { if ((p = ttymsg(iov, iovlen, ut->ut_line, TTYMSGTIME)) != NULL) { errno = 0; /* already in msg */ logerror(p); } break; } } } endutxent(); reenter = 0; } static void reapchild(int signo __unused) { int status; pid_t pid; struct filed *f; while ((pid = wait3(&status, WNOHANG, (struct rusage *)NULL)) > 0) { if (!Initialized) /* Don't tell while we are initting. */ continue; /* First, look if it's a process from the dead queue. */ if (deadq_remove(pid)) goto oncemore; /* Now, look in list of active processes. */ for (f = Files; f; f = f->f_next) if (f->f_type == F_PIPE && f->f_un.f_pipe.f_pid == pid) { (void)close(f->f_file); f->f_un.f_pipe.f_pid = 0; log_deadchild(pid, status, f->f_un.f_pipe.f_pname); break; } oncemore: continue; } } /* * Return a printable representation of a host address. */ static const char * cvthname(struct sockaddr *f) { int error, hl; sigset_t omask, nmask; static char hname[NI_MAXHOST], ip[NI_MAXHOST]; error = getnameinfo((struct sockaddr *)f, ((struct sockaddr *)f)->sa_len, ip, sizeof ip, NULL, 0, NI_NUMERICHOST); dprintf("cvthname(%s)\n", ip); if (error) { dprintf("Malformed from address %s\n", gai_strerror(error)); return ("???"); } if (!resolve) return (ip); sigemptyset(&nmask); sigaddset(&nmask, SIGHUP); sigprocmask(SIG_BLOCK, &nmask, &omask); error = getnameinfo((struct sockaddr *)f, ((struct sockaddr *)f)->sa_len, hname, sizeof hname, NULL, 0, NI_NAMEREQD); sigprocmask(SIG_SETMASK, &omask, NULL); if (error) { dprintf("Host name for your address (%s) unknown\n", ip); return (ip); } hl = strlen(hname); if (hl > 0 && hname[hl-1] == '.') hname[--hl] = '\0'; trimdomain(hname, hl); return (hname); } static void dodie(int signo) { WantDie = signo; } static void domark(int signo __unused) { MarkSet = 1; } /* * Print syslogd errors some place. */ static void logerror(const char *type) { char buf[512]; static int recursed = 0; /* If there's an error while trying to log an error, give up. */ if (recursed) return; recursed++; if (errno) (void)snprintf(buf, sizeof buf, "syslogd: %s: %s", type, strerror(errno)); else (void)snprintf(buf, sizeof buf, "syslogd: %s", type); errno = 0; dprintf("%s\n", buf); logmsg(LOG_SYSLOG|LOG_ERR, buf, LocalHostName, ADDDATE); recursed--; } static void die(int signo) { struct filed *f; struct funix *fx; int was_initialized; char buf[100]; was_initialized = Initialized; Initialized = 0; /* Don't log SIGCHLDs. */ for (f = Files; f != NULL; f = f->f_next) { /* flush any pending output */ if (f->f_prevcount) fprintlog(f, 0, (char *)NULL); if (f->f_type == F_PIPE && f->f_un.f_pipe.f_pid > 0) { (void)close(f->f_file); f->f_un.f_pipe.f_pid = 0; } } Initialized = was_initialized; if (signo) { dprintf("syslogd: exiting on signal %d\n", signo); (void)snprintf(buf, sizeof(buf), "exiting on signal %d", signo); errno = 0; logerror(buf); } STAILQ_FOREACH(fx, &funixes, next) (void)unlink(fx->name); pidfile_remove(pfh); exit(1); } /* * INIT -- Initialize syslogd from configuration table */ static void init(int signo) { int i; FILE *cf; struct filed *f, *next, **nextp; char *p; char cline[LINE_MAX]; - char prog[NAME_MAX+1]; + char prog[LINE_MAX]; char host[MAXHOSTNAMELEN]; char oldLocalHostName[MAXHOSTNAMELEN]; char hostMsg[2*MAXHOSTNAMELEN+40]; char bootfileMsg[LINE_MAX]; dprintf("init\n"); /* * Load hostname (may have changed). */ if (signo != 0) (void)strlcpy(oldLocalHostName, LocalHostName, sizeof(oldLocalHostName)); if (gethostname(LocalHostName, sizeof(LocalHostName))) err(EX_OSERR, "gethostname() failed"); if ((p = strchr(LocalHostName, '.')) != NULL) { *p++ = '\0'; LocalDomain = p; } else { LocalDomain = ""; } /* * Close all open log files. */ Initialized = 0; for (f = Files; f != NULL; f = next) { /* flush any pending output */ if (f->f_prevcount) fprintlog(f, 0, (char *)NULL); switch (f->f_type) { case F_FILE: case F_FORW: case F_CONSOLE: case F_TTY: (void)close(f->f_file); break; case F_PIPE: if (f->f_un.f_pipe.f_pid > 0) { (void)close(f->f_file); deadq_enter(f->f_un.f_pipe.f_pid, f->f_un.f_pipe.f_pname); } f->f_un.f_pipe.f_pid = 0; break; } next = f->f_next; if (f->f_program) free(f->f_program); if (f->f_host) free(f->f_host); free((char *)f); } Files = NULL; nextp = &Files; /* open the configuration file */ if ((cf = fopen(ConfFile, "r")) == NULL) { dprintf("cannot open %s\n", ConfFile); *nextp = (struct filed *)calloc(1, sizeof(*f)); if (*nextp == NULL) { logerror("calloc"); exit(1); } cfline("*.ERR\t/dev/console", *nextp, "*", "*"); (*nextp)->f_next = (struct filed *)calloc(1, sizeof(*f)); if ((*nextp)->f_next == NULL) { logerror("calloc"); exit(1); } cfline("*.PANIC\t*", (*nextp)->f_next, "*", "*"); Initialized = 1; return; } /* * Foreach line in the conf table, open that file. */ f = NULL; (void)strlcpy(host, "*", sizeof(host)); (void)strlcpy(prog, "*", sizeof(prog)); while (fgets(cline, sizeof(cline), cf) != NULL) { /* * check for end-of-section, comments, strip off trailing * spaces and newline character. #!prog is treated specially: * following lines apply only to that program. */ for (p = cline; isspace(*p); ++p) continue; if (*p == 0) continue; if (*p == '#') { p++; if (*p != '!' && *p != '+' && *p != '-') continue; } if (*p == '+' || *p == '-') { host[0] = *p++; while (isspace(*p)) p++; if ((!*p) || (*p == '*')) { (void)strlcpy(host, "*", sizeof(host)); continue; } if (*p == '@') p = LocalHostName; for (i = 1; i < MAXHOSTNAMELEN - 1; i++) { if (!isalnum(*p) && *p != '.' && *p != '-' && *p != ',' && *p != ':' && *p != '%') break; host[i] = *p++; } host[i] = '\0'; continue; } if (*p == '!') { p++; while (isspace(*p)) p++; if ((!*p) || (*p == '*')) { (void)strlcpy(prog, "*", sizeof(prog)); continue; } - for (i = 0; i < NAME_MAX; i++) { + for (i = 0; i < LINE_MAX - 1; i++) { if (!isprint(p[i]) || isspace(p[i])) break; prog[i] = p[i]; } prog[i] = 0; continue; } for (p = cline + 1; *p != '\0'; p++) { if (*p != '#') continue; if (*(p - 1) == '\\') { strcpy(p - 1, p); p--; continue; } *p = '\0'; break; } for (i = strlen(cline) - 1; i >= 0 && isspace(cline[i]); i--) cline[i] = '\0'; f = (struct filed *)calloc(1, sizeof(*f)); if (f == NULL) { logerror("calloc"); exit(1); } *nextp = f; nextp = &f->f_next; cfline(cline, f, prog, host); } /* close the configuration file */ (void)fclose(cf); Initialized = 1; if (Debug) { int port; for (f = Files; f; f = f->f_next) { for (i = 0; i <= LOG_NFACILITIES; i++) if (f->f_pmask[i] == INTERNAL_NOPRI) printf("X "); else printf("%d ", f->f_pmask[i]); printf("%s: ", TypeNames[f->f_type]); switch (f->f_type) { case F_FILE: printf("%s", f->f_un.f_fname); break; case F_CONSOLE: case F_TTY: printf("%s%s", _PATH_DEV, f->f_un.f_fname); break; case F_FORW: port = (int)ntohs(((struct sockaddr_in *) (f->f_un.f_forw.f_addr->ai_addr))->sin_port); if (port != 514) { printf("%s:%d", f->f_un.f_forw.f_hname, port); } else { printf("%s", f->f_un.f_forw.f_hname); } break; case F_PIPE: printf("%s", f->f_un.f_pipe.f_pname); break; case F_USERS: for (i = 0; i < MAXUNAMES && *f->f_un.f_uname[i]; i++) printf("%s, ", f->f_un.f_uname[i]); break; } if (f->f_program) printf(" (%s)", f->f_program); printf("\n"); } } logmsg(LOG_SYSLOG|LOG_INFO, "syslogd: restart", LocalHostName, ADDDATE); dprintf("syslogd: restarted\n"); /* * Log a change in hostname, but only on a restart. */ if (signo != 0 && strcmp(oldLocalHostName, LocalHostName) != 0) { (void)snprintf(hostMsg, sizeof(hostMsg), "syslogd: hostname changed, \"%s\" to \"%s\"", oldLocalHostName, LocalHostName); logmsg(LOG_SYSLOG|LOG_INFO, hostMsg, LocalHostName, ADDDATE); dprintf("%s\n", hostMsg); } /* * Log the kernel boot file if we aren't going to use it as * the prefix, and if this is *not* a restart. */ if (signo == 0 && !use_bootfile) { (void)snprintf(bootfileMsg, sizeof(bootfileMsg), "syslogd: kernel boot file is %s", bootfile); logmsg(LOG_KERN|LOG_INFO, bootfileMsg, LocalHostName, ADDDATE); dprintf("%s\n", bootfileMsg); } } /* * Crack a configuration file line */ static void cfline(const char *line, struct filed *f, const char *prog, const char *host) { struct addrinfo hints, *res; int error, i, pri, syncfile; const char *p, *q; char *bp; char buf[MAXLINE], ebuf[100]; dprintf("cfline(\"%s\", f, \"%s\", \"%s\")\n", line, prog, host); errno = 0; /* keep strerror() stuff out of logerror messages */ /* clear out file entry */ memset(f, 0, sizeof(*f)); for (i = 0; i <= LOG_NFACILITIES; i++) f->f_pmask[i] = INTERNAL_NOPRI; /* save hostname if any */ if (host && *host == '*') host = NULL; if (host) { int hl; f->f_host = strdup(host); if (f->f_host == NULL) { logerror("strdup"); exit(1); } hl = strlen(f->f_host); if (hl > 0 && f->f_host[hl-1] == '.') f->f_host[--hl] = '\0'; trimdomain(f->f_host, hl); } /* save program name if any */ if (prog && *prog == '*') prog = NULL; if (prog) { f->f_program = strdup(prog); if (f->f_program == NULL) { logerror("strdup"); exit(1); } } /* scan through the list of selectors */ for (p = line; *p && *p != '\t' && *p != ' ';) { int pri_done; int pri_cmp; int pri_invert; /* find the end of this facility name list */ for (q = p; *q && *q != '\t' && *q != ' ' && *q++ != '.'; ) continue; /* get the priority comparison */ pri_cmp = 0; pri_done = 0; pri_invert = 0; if (*q == '!') { pri_invert = 1; q++; } while (!pri_done) { switch (*q) { case '<': pri_cmp |= PRI_LT; q++; break; case '=': pri_cmp |= PRI_EQ; q++; break; case '>': pri_cmp |= PRI_GT; q++; break; default: pri_done++; break; } } /* collect priority name */ for (bp = buf; *q && !strchr("\t,; ", *q); ) *bp++ = *q++; *bp = '\0'; /* skip cruft */ while (strchr(",;", *q)) q++; /* decode priority name */ if (*buf == '*') { pri = LOG_PRIMASK; pri_cmp = PRI_LT | PRI_EQ | PRI_GT; } else { /* Ignore trailing spaces. */ for (i = strlen(buf) - 1; i >= 0 && buf[i] == ' '; i--) buf[i] = '\0'; pri = decode(buf, prioritynames); if (pri < 0) { errno = 0; (void)snprintf(ebuf, sizeof ebuf, "unknown priority name \"%s\"", buf); logerror(ebuf); return; } } if (!pri_cmp) pri_cmp = (UniquePriority) ? (PRI_EQ) : (PRI_EQ | PRI_GT) ; if (pri_invert) pri_cmp ^= PRI_LT | PRI_EQ | PRI_GT; /* scan facilities */ while (*p && !strchr("\t.; ", *p)) { for (bp = buf; *p && !strchr("\t,;. ", *p); ) *bp++ = *p++; *bp = '\0'; if (*buf == '*') { for (i = 0; i < LOG_NFACILITIES; i++) { f->f_pmask[i] = pri; f->f_pcmp[i] = pri_cmp; } } else { i = decode(buf, facilitynames); if (i < 0) { errno = 0; (void)snprintf(ebuf, sizeof ebuf, "unknown facility name \"%s\"", buf); logerror(ebuf); return; } f->f_pmask[i >> 3] = pri; f->f_pcmp[i >> 3] = pri_cmp; } while (*p == ',' || *p == ' ') p++; } p = q; } /* skip to action part */ while (*p == '\t' || *p == ' ') p++; if (*p == '-') { syncfile = 0; p++; } else syncfile = 1; switch (*p) { case '@': { char *tp; char endkey = ':'; /* * scan forward to see if there is a port defined. * so we can't use strlcpy.. */ i = sizeof(f->f_un.f_forw.f_hname); tp = f->f_un.f_forw.f_hname; p++; /* * an ipv6 address should start with a '[' in that case * we should scan for a ']' */ if (*p == '[') { p++; endkey = ']'; } while (*p && (*p != endkey) && (i-- > 0)) { *tp++ = *p++; } if (endkey == ']' && *p == endkey) p++; *tp = '\0'; } /* See if we copied a domain and have a port */ if (*p == ':') p++; else p = NULL; memset(&hints, 0, sizeof(hints)); hints.ai_family = family; hints.ai_socktype = SOCK_DGRAM; error = getaddrinfo(f->f_un.f_forw.f_hname, p ? p : "syslog", &hints, &res); if (error) { logerror(gai_strerror(error)); break; } f->f_un.f_forw.f_addr = res; f->f_type = F_FORW; break; case '/': if ((f->f_file = open(p, logflags, 0600)) < 0) { f->f_type = F_UNUSED; logerror(p); break; } if (syncfile) f->f_flags |= FFLAG_SYNC; if (isatty(f->f_file)) { if (strcmp(p, ctty) == 0) f->f_type = F_CONSOLE; else f->f_type = F_TTY; (void)strlcpy(f->f_un.f_fname, p + sizeof(_PATH_DEV) - 1, sizeof(f->f_un.f_fname)); } else { (void)strlcpy(f->f_un.f_fname, p, sizeof(f->f_un.f_fname)); f->f_type = F_FILE; } break; case '|': f->f_un.f_pipe.f_pid = 0; (void)strlcpy(f->f_un.f_pipe.f_pname, p + 1, sizeof(f->f_un.f_pipe.f_pname)); f->f_type = F_PIPE; break; case '*': f->f_type = F_WALL; break; default: for (i = 0; i < MAXUNAMES && *p; i++) { for (q = p; *q && *q != ','; ) q++; (void)strncpy(f->f_un.f_uname[i], p, MAXLOGNAME - 1); if ((q - p) >= MAXLOGNAME) f->f_un.f_uname[i][MAXLOGNAME - 1] = '\0'; else f->f_un.f_uname[i][q - p] = '\0'; while (*q == ',' || *q == ' ') q++; p = q; } f->f_type = F_USERS; break; } } /* * Decode a symbolic name to a numeric value */ static int decode(const char *name, const CODE *codetab) { const CODE *c; char *p, buf[40]; if (isdigit(*name)) return (atoi(name)); for (p = buf; *name && p < &buf[sizeof(buf) - 1]; p++, name++) { if (isupper(*name)) *p = tolower(*name); else *p = *name; } *p = '\0'; for (c = codetab; c->c_name; c++) if (!strcmp(buf, c->c_name)) return (c->c_val); return (-1); } static void markit(void) { struct filed *f; dq_t q, next; now = time((time_t *)NULL); MarkSeq += TIMERINTVL; if (MarkSeq >= MarkInterval) { logmsg(LOG_INFO, "-- MARK --", LocalHostName, ADDDATE|MARK); MarkSeq = 0; } for (f = Files; f; f = f->f_next) { if (f->f_prevcount && now >= REPEATTIME(f)) { dprintf("flush %s: repeated %d times, %d sec.\n", TypeNames[f->f_type], f->f_prevcount, repeatinterval[f->f_repeatcount]); fprintlog(f, 0, (char *)NULL); BACKOFF(f); } } /* Walk the dead queue, and see if we should signal somebody. */ for (q = TAILQ_FIRST(&deadq_head); q != NULL; q = next) { next = TAILQ_NEXT(q, dq_entries); switch (q->dq_timeout) { case 0: /* Already signalled once, try harder now. */ if (kill(q->dq_pid, SIGKILL) != 0) (void)deadq_remove(q->dq_pid); break; case 1: /* * Timed out on dead queue, send terminate * signal. Note that we leave the removal * from the dead queue to reapchild(), which * will also log the event (unless the process * didn't even really exist, in case we simply * drop it from the dead queue). */ if (kill(q->dq_pid, SIGTERM) != 0) (void)deadq_remove(q->dq_pid); /* FALLTHROUGH */ default: q->dq_timeout--; } } MarkSet = 0; (void)alarm(TIMERINTVL); } /* * fork off and become a daemon, but wait for the child to come online * before returing to the parent, or we get disk thrashing at boot etc. * Set a timer so we don't hang forever if it wedges. */ static int waitdaemon(int nochdir, int noclose, int maxwait) { int fd; int status; pid_t pid, childpid; switch (childpid = fork()) { case -1: return (-1); case 0: break; default: signal(SIGALRM, timedout); alarm(maxwait); while ((pid = wait3(&status, 0, NULL)) != -1) { if (WIFEXITED(status)) errx(1, "child pid %d exited with return code %d", pid, WEXITSTATUS(status)); if (WIFSIGNALED(status)) errx(1, "child pid %d exited on signal %d%s", pid, WTERMSIG(status), WCOREDUMP(status) ? " (core dumped)" : ""); if (pid == childpid) /* it's gone... */ break; } exit(0); } if (setsid() == -1) return (-1); if (!nochdir) (void)chdir("/"); if (!noclose && (fd = open(_PATH_DEVNULL, O_RDWR, 0)) != -1) { (void)dup2(fd, STDIN_FILENO); (void)dup2(fd, STDOUT_FILENO); (void)dup2(fd, STDERR_FILENO); if (fd > 2) (void)close (fd); } return (getppid()); } /* * We get a SIGALRM from the child when it's running and finished doing it's * fsync()'s or O_SYNC writes for all the boot messages. * * We also get a signal from the kernel if the timer expires, so check to * see what happened. */ static void timedout(int sig __unused) { int left; left = alarm(0); signal(SIGALRM, SIG_DFL); if (left == 0) errx(1, "timed out waiting for child"); else _exit(0); } /* * Add `s' to the list of allowable peer addresses to accept messages * from. * * `s' is a string in the form: * * [*]domainname[:{servicename|portnumber|*}] * * or * * netaddr/maskbits[:{servicename|portnumber|*}] * * Returns -1 on error, 0 if the argument was valid. */ static int allowaddr(char *s) { char *cp1, *cp2; struct allowedpeer ap; struct servent *se; int masklen = -1; struct addrinfo hints, *res; struct in_addr *addrp, *maskp; #ifdef INET6 int i; u_int32_t *addr6p, *mask6p; #endif char ip[NI_MAXHOST]; #ifdef INET6 if (*s != '[' || (cp1 = strchr(s + 1, ']')) == NULL) #endif cp1 = s; if ((cp1 = strrchr(cp1, ':'))) { /* service/port provided */ *cp1++ = '\0'; if (strlen(cp1) == 1 && *cp1 == '*') /* any port allowed */ ap.port = 0; else if ((se = getservbyname(cp1, "udp"))) { ap.port = ntohs(se->s_port); } else { ap.port = strtol(cp1, &cp2, 0); if (*cp2 != '\0') return (-1); /* port not numeric */ } } else { if ((se = getservbyname("syslog", "udp"))) ap.port = ntohs(se->s_port); else /* sanity, should not happen */ ap.port = 514; } if ((cp1 = strchr(s, '/')) != NULL && strspn(cp1 + 1, "0123456789") == strlen(cp1 + 1)) { *cp1 = '\0'; if ((masklen = atoi(cp1 + 1)) < 0) return (-1); } #ifdef INET6 if (*s == '[') { cp2 = s + strlen(s) - 1; if (*cp2 == ']') { ++s; *cp2 = '\0'; } else { cp2 = NULL; } } else { cp2 = NULL; } #endif memset(&hints, 0, sizeof(hints)); hints.ai_family = PF_UNSPEC; hints.ai_socktype = SOCK_DGRAM; hints.ai_flags = AI_PASSIVE | AI_NUMERICHOST; if (getaddrinfo(s, NULL, &hints, &res) == 0) { ap.isnumeric = 1; memcpy(&ap.a_addr, res->ai_addr, res->ai_addrlen); memset(&ap.a_mask, 0, sizeof(ap.a_mask)); ap.a_mask.ss_family = res->ai_family; if (res->ai_family == AF_INET) { ap.a_mask.ss_len = sizeof(struct sockaddr_in); maskp = &((struct sockaddr_in *)&ap.a_mask)->sin_addr; addrp = &((struct sockaddr_in *)&ap.a_addr)->sin_addr; if (masklen < 0) { /* use default netmask */ if (IN_CLASSA(ntohl(addrp->s_addr))) maskp->s_addr = htonl(IN_CLASSA_NET); else if (IN_CLASSB(ntohl(addrp->s_addr))) maskp->s_addr = htonl(IN_CLASSB_NET); else maskp->s_addr = htonl(IN_CLASSC_NET); } else if (masklen <= 32) { /* convert masklen to netmask */ if (masklen == 0) maskp->s_addr = 0; else maskp->s_addr = htonl(~((1 << (32 - masklen)) - 1)); } else { freeaddrinfo(res); return (-1); } /* Lose any host bits in the network number. */ addrp->s_addr &= maskp->s_addr; } #ifdef INET6 else if (res->ai_family == AF_INET6 && masklen <= 128) { ap.a_mask.ss_len = sizeof(struct sockaddr_in6); if (masklen < 0) masklen = 128; mask6p = (u_int32_t *)&((struct sockaddr_in6 *)&ap.a_mask)->sin6_addr; /* convert masklen to netmask */ while (masklen > 0) { if (masklen < 32) { *mask6p = htonl(~(0xffffffff >> masklen)); break; } *mask6p++ = 0xffffffff; masklen -= 32; } /* Lose any host bits in the network number. */ mask6p = (u_int32_t *)&((struct sockaddr_in6 *)&ap.a_mask)->sin6_addr; addr6p = (u_int32_t *)&((struct sockaddr_in6 *)&ap.a_addr)->sin6_addr; for (i = 0; i < 4; i++) addr6p[i] &= mask6p[i]; } #endif else { freeaddrinfo(res); return (-1); } freeaddrinfo(res); } else { /* arg `s' is domain name */ ap.isnumeric = 0; ap.a_name = s; if (cp1) *cp1 = '/'; #ifdef INET6 if (cp2) { *cp2 = ']'; --s; } #endif } if (Debug) { printf("allowaddr: rule %d: ", NumAllowed); if (ap.isnumeric) { printf("numeric, "); getnameinfo((struct sockaddr *)&ap.a_addr, ((struct sockaddr *)&ap.a_addr)->sa_len, ip, sizeof ip, NULL, 0, NI_NUMERICHOST); printf("addr = %s, ", ip); getnameinfo((struct sockaddr *)&ap.a_mask, ((struct sockaddr *)&ap.a_mask)->sa_len, ip, sizeof ip, NULL, 0, NI_NUMERICHOST); printf("mask = %s; ", ip); } else { printf("domainname = %s; ", ap.a_name); } printf("port = %d\n", ap.port); } if ((AllowedPeers = realloc(AllowedPeers, ++NumAllowed * sizeof(struct allowedpeer))) == NULL) { logerror("realloc"); exit(1); } memcpy(&AllowedPeers[NumAllowed - 1], &ap, sizeof(struct allowedpeer)); return (0); } /* * Validate that the remote peer has permission to log to us. */ static int validate(struct sockaddr *sa, const char *hname) { int i; size_t l1, l2; char *cp, name[NI_MAXHOST], ip[NI_MAXHOST], port[NI_MAXSERV]; struct allowedpeer *ap; struct sockaddr_in *sin4, *a4p = NULL, *m4p = NULL; #ifdef INET6 int j, reject; struct sockaddr_in6 *sin6, *a6p = NULL, *m6p = NULL; #endif struct addrinfo hints, *res; u_short sport; if (NumAllowed == 0) /* traditional behaviour, allow everything */ return (1); (void)strlcpy(name, hname, sizeof(name)); memset(&hints, 0, sizeof(hints)); hints.ai_family = PF_UNSPEC; hints.ai_socktype = SOCK_DGRAM; hints.ai_flags = AI_PASSIVE | AI_NUMERICHOST; if (getaddrinfo(name, NULL, &hints, &res) == 0) freeaddrinfo(res); else if (strchr(name, '.') == NULL) { strlcat(name, ".", sizeof name); strlcat(name, LocalDomain, sizeof name); } if (getnameinfo(sa, sa->sa_len, ip, sizeof ip, port, sizeof port, NI_NUMERICHOST | NI_NUMERICSERV) != 0) return (0); /* for safety, should not occur */ dprintf("validate: dgram from IP %s, port %s, name %s;\n", ip, port, name); sport = atoi(port); /* now, walk down the list */ for (i = 0, ap = AllowedPeers; i < NumAllowed; i++, ap++) { if (ap->port != 0 && ap->port != sport) { dprintf("rejected in rule %d due to port mismatch.\n", i); continue; } if (ap->isnumeric) { if (ap->a_addr.ss_family != sa->sa_family) { dprintf("rejected in rule %d due to address family mismatch.\n", i); continue; } if (ap->a_addr.ss_family == AF_INET) { sin4 = (struct sockaddr_in *)sa; a4p = (struct sockaddr_in *)&ap->a_addr; m4p = (struct sockaddr_in *)&ap->a_mask; if ((sin4->sin_addr.s_addr & m4p->sin_addr.s_addr) != a4p->sin_addr.s_addr) { dprintf("rejected in rule %d due to IP mismatch.\n", i); continue; } } #ifdef INET6 else if (ap->a_addr.ss_family == AF_INET6) { sin6 = (struct sockaddr_in6 *)sa; a6p = (struct sockaddr_in6 *)&ap->a_addr; m6p = (struct sockaddr_in6 *)&ap->a_mask; if (a6p->sin6_scope_id != 0 && sin6->sin6_scope_id != a6p->sin6_scope_id) { dprintf("rejected in rule %d due to scope mismatch.\n", i); continue; } reject = 0; for (j = 0; j < 16; j += 4) { if ((*(u_int32_t *)&sin6->sin6_addr.s6_addr[j] & *(u_int32_t *)&m6p->sin6_addr.s6_addr[j]) != *(u_int32_t *)&a6p->sin6_addr.s6_addr[j]) { ++reject; break; } } if (reject) { dprintf("rejected in rule %d due to IP mismatch.\n", i); continue; } } #endif else continue; } else { cp = ap->a_name; l1 = strlen(name); if (*cp == '*') { /* allow wildmatch */ cp++; l2 = strlen(cp); if (l2 > l1 || memcmp(cp, &name[l1 - l2], l2) != 0) { dprintf("rejected in rule %d due to name mismatch.\n", i); continue; } } else { /* exact match */ l2 = strlen(cp); if (l2 != l1 || memcmp(cp, name, l1) != 0) { dprintf("rejected in rule %d due to name mismatch.\n", i); continue; } } } dprintf("accepted in rule %d.\n", i); return (1); /* hooray! */ } return (0); } /* * Fairly similar to popen(3), but returns an open descriptor, as * opposed to a FILE *. */ static int p_open(const char *prog, pid_t *rpid) { int pfd[2], nulldesc; pid_t pid; sigset_t omask, mask; char *argv[4]; /* sh -c cmd NULL */ char errmsg[200]; if (pipe(pfd) == -1) return (-1); if ((nulldesc = open(_PATH_DEVNULL, O_RDWR)) == -1) /* we are royally screwed anyway */ return (-1); sigemptyset(&mask); sigaddset(&mask, SIGALRM); sigaddset(&mask, SIGHUP); sigprocmask(SIG_BLOCK, &mask, &omask); switch ((pid = fork())) { case -1: sigprocmask(SIG_SETMASK, &omask, 0); close(nulldesc); return (-1); case 0: argv[0] = strdup("sh"); argv[1] = strdup("-c"); argv[2] = strdup(prog); argv[3] = NULL; if (argv[0] == NULL || argv[1] == NULL || argv[2] == NULL) { logerror("strdup"); exit(1); } alarm(0); (void)setsid(); /* Avoid catching SIGHUPs. */ /* * Throw away pending signals, and reset signal * behaviour to standard values. */ signal(SIGALRM, SIG_IGN); signal(SIGHUP, SIG_IGN); sigprocmask(SIG_SETMASK, &omask, 0); signal(SIGPIPE, SIG_DFL); signal(SIGQUIT, SIG_DFL); signal(SIGALRM, SIG_DFL); signal(SIGHUP, SIG_DFL); dup2(pfd[0], STDIN_FILENO); dup2(nulldesc, STDOUT_FILENO); dup2(nulldesc, STDERR_FILENO); closefrom(3); (void)execvp(_PATH_BSHELL, argv); _exit(255); } sigprocmask(SIG_SETMASK, &omask, 0); close(nulldesc); close(pfd[0]); /* * Avoid blocking on a hung pipe. With O_NONBLOCK, we are * supposed to get an EWOULDBLOCK on writev(2), which is * caught by the logic above anyway, which will in turn close * the pipe, and fork a new logging subprocess if necessary. * The stale subprocess will be killed some time later unless * it terminated itself due to closing its input pipe (so we * get rid of really dead puppies). */ if (fcntl(pfd[1], F_SETFL, O_NONBLOCK) == -1) { /* This is bad. */ (void)snprintf(errmsg, sizeof errmsg, "Warning: cannot change pipe to PID %d to " "non-blocking behaviour.", (int)pid); logerror(errmsg); } *rpid = pid; return (pfd[1]); } static void deadq_enter(pid_t pid, const char *name) { dq_t p; int status; /* * Be paranoid, if we can't signal the process, don't enter it * into the dead queue (perhaps it's already dead). If possible, * we try to fetch and log the child's status. */ if (kill(pid, 0) != 0) { if (waitpid(pid, &status, WNOHANG) > 0) log_deadchild(pid, status, name); return; } p = malloc(sizeof(struct deadq_entry)); if (p == NULL) { logerror("malloc"); exit(1); } p->dq_pid = pid; p->dq_timeout = DQ_TIMO_INIT; TAILQ_INSERT_TAIL(&deadq_head, p, dq_entries); } static int deadq_remove(pid_t pid) { dq_t q; TAILQ_FOREACH(q, &deadq_head, dq_entries) { if (q->dq_pid == pid) { TAILQ_REMOVE(&deadq_head, q, dq_entries); free(q); return (1); } } return (0); } static void log_deadchild(pid_t pid, int status, const char *name) { int code; char buf[256]; const char *reason; errno = 0; /* Keep strerror() stuff out of logerror messages. */ if (WIFSIGNALED(status)) { reason = "due to signal"; code = WTERMSIG(status); } else { reason = "with status"; code = WEXITSTATUS(status); if (code == 0) return; } (void)snprintf(buf, sizeof buf, "Logging subprocess %d (%s) exited %s %d.", pid, name, reason, code); logerror(buf); } static int * socksetup(int af, char *bindhostname) { struct addrinfo hints, *res, *r; const char *bindservice; char *cp; int error, maxs, *s, *socks; /* * We have to handle this case for backwards compatibility: * If there are two (or more) colons but no '[' and ']', * assume this is an inet6 address without a service. */ bindservice = "syslog"; if (bindhostname != NULL) { #ifdef INET6 if (*bindhostname == '[' && (cp = strchr(bindhostname + 1, ']')) != NULL) { ++bindhostname; *cp = '\0'; if (cp[1] == ':' && cp[2] != '\0') bindservice = cp + 2; } else { #endif cp = strchr(bindhostname, ':'); if (cp != NULL && strchr(cp + 1, ':') == NULL) { *cp = '\0'; if (cp[1] != '\0') bindservice = cp + 1; if (cp == bindhostname) bindhostname = NULL; } #ifdef INET6 } #endif } memset(&hints, 0, sizeof(hints)); hints.ai_flags = AI_PASSIVE; hints.ai_family = af; hints.ai_socktype = SOCK_DGRAM; error = getaddrinfo(bindhostname, bindservice, &hints, &res); if (error) { logerror(gai_strerror(error)); errno = 0; die(0); } /* Count max number of sockets we may open */ for (maxs = 0, r = res; r; r = r->ai_next, maxs++); socks = malloc((maxs+1) * sizeof(int)); if (socks == NULL) { logerror("couldn't allocate memory for sockets"); die(0); } *socks = 0; /* num of sockets counter at start of array */ s = socks + 1; for (r = res; r; r = r->ai_next) { int on = 1; *s = socket(r->ai_family, r->ai_socktype, r->ai_protocol); if (*s < 0) { logerror("socket"); continue; } #ifdef INET6 if (r->ai_family == AF_INET6) { if (setsockopt(*s, IPPROTO_IPV6, IPV6_V6ONLY, (char *)&on, sizeof (on)) < 0) { logerror("setsockopt"); close(*s); continue; } } #endif if (setsockopt(*s, SOL_SOCKET, SO_REUSEADDR, (char *)&on, sizeof (on)) < 0) { logerror("setsockopt"); close(*s); continue; } /* * RFC 3164 recommends that client side message * should come from the privileged syslogd port. * * If the system administrator choose not to obey * this, we can skip the bind() step so that the * system will choose a port for us. */ if (!NoBind) { if (bind(*s, r->ai_addr, r->ai_addrlen) < 0) { logerror("bind"); close(*s); continue; } if (!SecureMode) increase_rcvbuf(*s); } (*socks)++; s++; } if (*socks == 0) { free(socks); if (Debug) return (NULL); else die(0); } if (res) freeaddrinfo(res); return (socks); } static void increase_rcvbuf(int fd) { socklen_t len, slen; slen = sizeof(len); if (getsockopt(fd, SOL_SOCKET, SO_RCVBUF, &len, &slen) == 0) { if (len < RCVBUF_MINSIZE) { len = RCVBUF_MINSIZE; setsockopt(fd, SOL_SOCKET, SO_RCVBUF, &len, sizeof(len)); } } } Index: projects/clang350-import =================================================================== --- projects/clang350-import (revision 275748) +++ projects/clang350-import (revision 275749) Property changes on: projects/clang350-import ___________________________________________________________________ Modified: svn:mergeinfo ## -0,0 +0,1 ## Merged /head:r275715-275748