Index: head/share/man/man4/bpf.4 =================================================================== --- head/share/man/man4/bpf.4 (revision 307706) +++ head/share/man/man4/bpf.4 (revision 307707) @@ -1,1111 +1,1115 @@ .\" Copyright (c) 2007 Seccuris Inc. .\" All rights reserved. .\" .\" This software was developed by Robert N. M. Watson under contract to .\" Seccuris Inc. .\" .\" Redistribution and use in source and binary forms, with or without .\" modification, are permitted provided that the following conditions .\" are met: .\" 1. Redistributions of source code must retain the above copyright .\" notice, this list of conditions and the following disclaimer. .\" 2. Redistributions in binary form must reproduce the above copyright .\" notice, this list of conditions and the following disclaimer in the .\" documentation and/or other materials provided with the distribution. .\" .\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND .\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE .\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE .\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE .\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL .\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS .\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) .\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT .\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY .\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF .\" SUCH DAMAGE. .\" .\" Copyright (c) 1990 The Regents of the University of California. .\" All rights reserved. .\" .\" Redistribution and use in source and binary forms, with or without .\" modification, are permitted provided that: (1) source code distributions .\" retain the above copyright notice and this paragraph in its entirety, (2) .\" distributions including binary code include the above copyright notice and .\" this paragraph in its entirety in the documentation or other materials .\" provided with the distribution, and (3) all advertising materials mentioning .\" features or use of this software display the following acknowledgement: .\" ``This product includes software developed by the University of California, .\" Lawrence Berkeley Laboratory and its contributors.'' Neither the name of .\" the University nor the names of its contributors may be used to endorse .\" or promote products derived from this software without specific prior .\" written permission. .\" THIS SOFTWARE IS PROVIDED ``AS IS'' AND WITHOUT ANY EXPRESS OR IMPLIED .\" WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF .\" MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. .\" .\" This document is derived in part from the enet man page (enet.4) .\" distributed with 4.3BSD Unix. .\" .\" $FreeBSD$ .\" -.Dd June 15, 2010 +.Dd October 21, 2016 .Dt BPF 4 .Os .Sh NAME .Nm bpf .Nd Berkeley Packet Filter .Sh SYNOPSIS .Cd device bpf .Sh DESCRIPTION The Berkeley Packet Filter provides a raw interface to data link layers in a protocol independent fashion. All packets on the network, even those destined for other hosts, are accessible through this mechanism. .Pp The packet filter appears as a character special device, .Pa /dev/bpf . After opening the device, the file descriptor must be bound to a specific network interface with the .Dv BIOCSETIF ioctl. A given interface can be shared by multiple listeners, and the filter underlying each descriptor will see an identical packet stream. .Pp A separate device file is required for each minor device. If a file is in use, the open will fail and .Va errno will be set to .Er EBUSY . .Pp Associated with each open instance of a .Nm file is a user-settable packet filter. Whenever a packet is received by an interface, all file descriptors listening on that interface apply their filter. Each descriptor that accepts the packet receives its own copy. .Pp The packet filter will support any link level protocol that has fixed length headers. Currently, only Ethernet, .Tn SLIP , and .Tn PPP drivers have been modified to interact with .Nm . .Pp Since packet data is in network byte order, applications should use the .Xr byteorder 3 macros to extract multi-byte values. .Pp A packet can be sent out on the network by writing to a .Nm file descriptor. The writes are unbuffered, meaning only one packet can be processed per write. Currently, only writes to Ethernets and .Tn SLIP links are supported. .Sh BUFFER MODES .Nm devices deliver packet data to the application via memory buffers provided by the application. The buffer mode is set using the .Dv BIOCSETBUFMODE ioctl, and read using the .Dv BIOCGETBUFMODE ioctl. .Ss Buffered read mode By default, .Nm devices operate in the .Dv BPF_BUFMODE_BUFFER mode, in which packet data is copied explicitly from kernel to user memory using the .Xr read 2 system call. The user process will declare a fixed buffer size that will be used both for sizing internal buffers and for all .Xr read 2 operations on the file. This size is queried using the .Dv BIOCGBLEN ioctl, and is set using the .Dv BIOCSBLEN ioctl. Note that an individual packet larger than the buffer size is necessarily truncated. .Ss Zero-copy buffer mode .Nm devices may also operate in the .Dv BPF_BUFMODE_ZEROCOPY mode, in which packet data is written directly into two user memory buffers by the kernel, avoiding both system call and copying overhead. Buffers are of fixed (and equal) size, page-aligned, and an even multiple of the page size. The maximum zero-copy buffer size is returned by the .Dv BIOCGETZMAX ioctl. Note that an individual packet larger than the buffer size is necessarily truncated. .Pp The user process registers two memory buffers using the .Dv BIOCSETZBUF ioctl, which accepts a .Vt struct bpf_zbuf pointer as an argument: .Bd -literal struct bpf_zbuf { void *bz_bufa; void *bz_bufb; size_t bz_buflen; }; .Ed .Pp .Vt bz_bufa is a pointer to the userspace address of the first buffer that will be filled, and .Vt bz_bufb is a pointer to the second buffer. .Nm will then cycle between the two buffers as they fill and are acknowledged. .Pp Each buffer begins with a fixed-length header to hold synchronization and data length information for the buffer: .Bd -literal struct bpf_zbuf_header { volatile u_int bzh_kernel_gen; /* Kernel generation number. */ volatile u_int bzh_kernel_len; /* Length of data in the buffer. */ volatile u_int bzh_user_gen; /* User generation number. */ /* ...padding for future use... */ }; .Ed .Pp The header structure of each buffer, including all padding, should be zeroed before it is configured using .Dv BIOCSETZBUF . Remaining space in the buffer will be used by the kernel to store packet data, laid out in the same format as with buffered read mode. .Pp The kernel and the user process follow a simple acknowledgement protocol via the buffer header to synchronize access to the buffer: when the header generation numbers, .Vt bzh_kernel_gen and .Vt bzh_user_gen , hold the same value, the kernel owns the buffer, and when they differ, userspace owns the buffer. .Pp While the kernel owns the buffer, the contents are unstable and may change asynchronously; while the user process owns the buffer, its contents are stable and will not be changed until the buffer has been acknowledged. .Pp Initializing the buffer headers to all 0's before registering the buffer has the effect of assigning initial ownership of both buffers to the kernel. The kernel signals that a buffer has been assigned to userspace by modifying .Vt bzh_kernel_gen , and userspace acknowledges the buffer and returns it to the kernel by setting the value of .Vt bzh_user_gen to the value of .Vt bzh_kernel_gen . .Pp In order to avoid caching and memory re-ordering effects, the user process must use atomic operations and memory barriers when checking for and acknowledging buffers: .Bd -literal #include /* * Return ownership of a buffer to the kernel for reuse. */ static void buffer_acknowledge(struct bpf_zbuf_header *bzh) { atomic_store_rel_int(&bzh->bzh_user_gen, bzh->bzh_kernel_gen); } /* * Check whether a buffer has been assigned to userspace by the kernel. * Return true if userspace owns the buffer, and false otherwise. */ static int buffer_check(struct bpf_zbuf_header *bzh) { return (bzh->bzh_user_gen != atomic_load_acq_int(&bzh->bzh_kernel_gen)); } .Ed .Pp The user process may force the assignment of the next buffer, if any data is pending, to userspace using the .Dv BIOCROTZBUF ioctl. This allows the user process to retrieve data in a partially filled buffer before the buffer is full, such as following a timeout; the process must recheck for buffer ownership using the header generation numbers, as the buffer will not be assigned to userspace if no data was present. .Pp As in the buffered read mode, .Xr kqueue 2 , .Xr poll 2 , and .Xr select 2 may be used to sleep awaiting the availability of a completed buffer. They will return a readable file descriptor when ownership of the next buffer is assigned to user space. .Pp In the current implementation, the kernel may assign zero, one, or both buffers to the user process; however, an earlier implementation maintained the invariant that at most one buffer could be assigned to the user process at a time. In order to both ensure progress and high performance, user processes should acknowledge a completely processed buffer as quickly as possible, returning it for reuse, and not block waiting on a second buffer while holding another buffer. .Sh IOCTLS The .Xr ioctl 2 command codes below are defined in .In net/bpf.h . All commands require these includes: .Bd -literal #include #include #include #include .Ed .Pp Additionally, .Dv BIOCGETIF and .Dv BIOCSETIF require .In sys/socket.h and .In net/if.h . .Pp In addition to .Dv FIONREAD the following commands may be applied to any open .Nm file. The (third) argument to .Xr ioctl 2 should be a pointer to the type indicated. .Bl -tag -width BIOCGETBUFMODE .It Dv BIOCGBLEN .Pq Li u_int Returns the required buffer length for reads on .Nm files. .It Dv BIOCSBLEN .Pq Li u_int Sets the buffer length for reads on .Nm files. The buffer must be set before the file is attached to an interface with .Dv BIOCSETIF . If the requested buffer size cannot be accommodated, the closest allowable size will be set and returned in the argument. A read call will result in .Er EIO if it is passed a buffer that is not this size. .It Dv BIOCGDLT .Pq Li u_int Returns the type of the data link layer underlying the attached interface. .Er EINVAL is returned if no interface has been specified. The device types, prefixed with .Dq Li DLT_ , are defined in .In net/bpf.h . .It Dv BIOCPROMISC Forces the interface into promiscuous mode. All packets, not just those destined for the local host, are processed. Since more than one file can be listening on a given interface, a listener that opened its interface non-promiscuously may receive packets promiscuously. This problem can be remedied with an appropriate filter. .It Dv BIOCFLUSH Flushes the buffer of incoming packets, and resets the statistics that are returned by BIOCGSTATS. .It Dv BIOCGETIF .Pq Li "struct ifreq" Returns the name of the hardware interface that the file is listening on. The name is returned in the ifr_name field of the .Li ifreq structure. All other fields are undefined. .It Dv BIOCSETIF .Pq Li "struct ifreq" Sets the hardware interface associate with the file. This command must be performed before any packets can be read. The device is indicated by name using the .Li ifr_name field of the .Li ifreq structure. Additionally, performs the actions of .Dv BIOCFLUSH . .It Dv BIOCSRTIMEOUT .It Dv BIOCGRTIMEOUT .Pq Li "struct timeval" Set or get the read timeout parameter. The argument specifies the length of time to wait before timing out on a read request. This parameter is initialized to zero by .Xr open 2 , indicating no timeout. .It Dv BIOCGSTATS .Pq Li "struct bpf_stat" Returns the following structure of packet statistics: .Bd -literal struct bpf_stat { u_int bs_recv; /* number of packets received */ u_int bs_drop; /* number of packets dropped */ }; .Ed .Pp The fields are: .Bl -hang -offset indent .It Li bs_recv the number of packets received by the descriptor since opened or reset (including any buffered since the last read call); and .It Li bs_drop the number of packets which were accepted by the filter but dropped by the kernel because of buffer overflows (i.e., the application's reads are not keeping up with the packet traffic). .El .It Dv BIOCIMMEDIATE .Pq Li u_int Enable or disable .Dq immediate mode , based on the truth value of the argument. When immediate mode is enabled, reads return immediately upon packet reception. Otherwise, a read will block until either the kernel buffer becomes full or a timeout occurs. This is useful for programs like .Xr rarpd 8 which must respond to messages in real time. The default for a new file is off. .It Dv BIOCSETF .It Dv BIOCSETFNR .Pq Li "struct bpf_program" Sets the read filter program used by the kernel to discard uninteresting packets. An array of instructions and its length is passed in using the following structure: .Bd -literal struct bpf_program { int bf_len; struct bpf_insn *bf_insns; }; .Ed .Pp The filter program is pointed to by the .Li bf_insns field while its length in units of .Sq Li struct bpf_insn is given by the .Li bf_len field. See section .Sx "FILTER MACHINE" for an explanation of the filter language. The only difference between .Dv BIOCSETF and .Dv BIOCSETFNR is .Dv BIOCSETF performs the actions of .Dv BIOCFLUSH while .Dv BIOCSETFNR does not. .It Dv BIOCSETWF .Pq Li "struct bpf_program" Sets the write filter program used by the kernel to control what type of packets can be written to the interface. See the .Dv BIOCSETF command for more information on the .Nm filter program. .It Dv BIOCVERSION .Pq Li "struct bpf_version" Returns the major and minor version numbers of the filter language currently recognized by the kernel. Before installing a filter, applications must check that the current version is compatible with the running kernel. Version numbers are compatible if the major numbers match and the application minor is less than or equal to the kernel minor. The kernel version number is returned in the following structure: .Bd -literal struct bpf_version { u_short bv_major; u_short bv_minor; }; .Ed .Pp The current version numbers are given by .Dv BPF_MAJOR_VERSION and .Dv BPF_MINOR_VERSION from .In net/bpf.h . An incompatible filter may result in undefined behavior (most likely, an error returned by .Fn ioctl or haphazard packet matching). .It Dv BIOCSHDRCMPLT .It Dv BIOCGHDRCMPLT .Pq Li u_int Set or get the status of the .Dq header complete flag. Set to zero if the link level source address should be filled in automatically by the interface output routine. Set to one if the link level source address will be written, as provided, to the wire. This flag is initialized to zero by default. .It Dv BIOCSSEESENT .It Dv BIOCGSEESENT .Pq Li u_int These commands are obsolete but left for compatibility. Use .Dv BIOCSDIRECTION and .Dv BIOCGDIRECTION instead. Set or get the flag determining whether locally generated packets on the interface should be returned by BPF. Set to zero to see only incoming packets on the interface. Set to one to see packets originating locally and remotely on the interface. This flag is initialized to one by default. .It Dv BIOCSDIRECTION .It Dv BIOCGDIRECTION .Pq Li u_int Set or get the setting determining whether incoming, outgoing, or all packets on the interface should be returned by BPF. Set to .Dv BPF_D_IN to see only incoming packets on the interface. Set to .Dv BPF_D_INOUT to see packets originating locally and remotely on the interface. Set to .Dv BPF_D_OUT to see only outgoing packets on the interface. This setting is initialized to .Dv BPF_D_INOUT by default. .It Dv BIOCSTSTAMP .It Dv BIOCGTSTAMP .Pq Li u_int Set or get format and resolution of the time stamps returned by BPF. Set to .Dv BPF_T_MICROTIME , .Dv BPF_T_MICROTIME_FAST , .Dv BPF_T_MICROTIME_MONOTONIC , or .Dv BPF_T_MICROTIME_MONOTONIC_FAST to get time stamps in 64-bit .Vt struct timeval format. Set to .Dv BPF_T_NANOTIME , .Dv BPF_T_NANOTIME_FAST , .Dv BPF_T_NANOTIME_MONOTONIC , or .Dv BPF_T_NANOTIME_MONOTONIC_FAST to get time stamps in 64-bit .Vt struct timespec format. Set to .Dv BPF_T_BINTIME , .Dv BPF_T_BINTIME_FAST , .Dv BPF_T_NANOTIME_MONOTONIC , or .Dv BPF_T_BINTIME_MONOTONIC_FAST to get time stamps in 64-bit .Vt struct bintime format. Set to .Dv BPF_T_NONE to ignore time stamp. All 64-bit time stamp formats are wrapped in .Vt struct bpf_ts . The .Dv BPF_T_MICROTIME_FAST , .Dv BPF_T_NANOTIME_FAST , .Dv BPF_T_BINTIME_FAST , .Dv BPF_T_MICROTIME_MONOTONIC_FAST , .Dv BPF_T_NANOTIME_MONOTONIC_FAST , and .Dv BPF_T_BINTIME_MONOTONIC_FAST are analogs of corresponding formats without _FAST suffix but do not perform a full time counter query, so their accuracy is one timer tick. The .Dv BPF_T_MICROTIME_MONOTONIC , .Dv BPF_T_NANOTIME_MONOTONIC , .Dv BPF_T_BINTIME_MONOTONIC , .Dv BPF_T_MICROTIME_MONOTONIC_FAST , .Dv BPF_T_NANOTIME_MONOTONIC_FAST , and .Dv BPF_T_BINTIME_MONOTONIC_FAST store the time elapsed since kernel boot. This setting is initialized to .Dv BPF_T_MICROTIME by default. .It Dv BIOCFEEDBACK .Pq Li u_int Set packet feedback mode. This allows injected packets to be fed back as input to the interface when output via the interface is successful. When .Dv BPF_D_INOUT direction is set, injected outgoing packet is not returned by BPF to avoid duplication. This flag is initialized to zero by default. .It Dv BIOCLOCK Set the locked flag on the .Nm descriptor. This prevents the execution of ioctl commands which could change the underlying operating parameters of the device. .It Dv BIOCGETBUFMODE .It Dv BIOCSETBUFMODE .Pq Li u_int Get or set the current .Nm buffering mode; possible values are .Dv BPF_BUFMODE_BUFFER , buffered read mode, and .Dv BPF_BUFMODE_ZBUF , zero-copy buffer mode. .It Dv BIOCSETZBUF .Pq Li struct bpf_zbuf Set the current zero-copy buffer locations; buffer locations may be set only once zero-copy buffer mode has been selected, and prior to attaching to an interface. Buffers must be of identical size, page-aligned, and an integer multiple of pages in size. The three fields .Vt bz_bufa , .Vt bz_bufb , and .Vt bz_buflen must be filled out. If buffers have already been set for this device, the ioctl will fail. .It Dv BIOCGETZMAX .Pq Li size_t Get the largest individual zero-copy buffer size allowed. As two buffers are used in zero-copy buffer mode, the limit (in practice) is twice the returned size. As zero-copy buffers consume kernel address space, conservative selection of buffer size is suggested, especially when there are multiple .Nm descriptors in use on 32-bit systems. .It Dv BIOCROTZBUF Force ownership of the next buffer to be assigned to userspace, if any data present in the buffer. If no data is present, the buffer will remain owned by the kernel. This allows consumers of zero-copy buffering to implement timeouts and retrieve partially filled buffers. In order to handle the case where no data is present in the buffer and therefore ownership is not assigned, the user process must check .Vt bzh_kernel_gen against .Vt bzh_user_gen . .El .Sh BPF HEADER One of the following structures is prepended to each packet returned by .Xr read 2 or via a zero-copy buffer: .Bd -literal struct bpf_xhdr { struct bpf_ts bh_tstamp; /* time stamp */ uint32_t bh_caplen; /* length of captured portion */ uint32_t bh_datalen; /* original length of packet */ u_short bh_hdrlen; /* length of bpf header (this struct plus alignment padding) */ }; struct bpf_hdr { struct timeval bh_tstamp; /* time stamp */ uint32_t bh_caplen; /* length of captured portion */ uint32_t bh_datalen; /* original length of packet */ u_short bh_hdrlen; /* length of bpf header (this struct plus alignment padding) */ }; .Ed .Pp The fields, whose values are stored in host order, and are: .Pp .Bl -tag -compact -width bh_datalen .It Li bh_tstamp The time at which the packet was processed by the packet filter. .It Li bh_caplen The length of the captured portion of the packet. This is the minimum of the truncation amount specified by the filter and the length of the packet. .It Li bh_datalen The length of the packet off the wire. This value is independent of the truncation amount specified by the filter. .It Li bh_hdrlen The length of the .Nm header, which may not be equal to .\" XXX - not really a function call .Fn sizeof "struct bpf_xhdr" or .Fn sizeof "struct bpf_hdr" . .El .Pp The .Li bh_hdrlen field exists to account for padding between the header and the link level protocol. The purpose here is to guarantee proper alignment of the packet data structures, which is required on alignment sensitive architectures and improves performance on many other architectures. The packet filter ensures that the .Vt bpf_xhdr , .Vt bpf_hdr and the network layer header will be word aligned. Currently, .Vt bpf_hdr is used when the time stamp is set to .Dv BPF_T_MICROTIME , .Dv BPF_T_MICROTIME_FAST , .Dv BPF_T_MICROTIME_MONOTONIC , .Dv BPF_T_MICROTIME_MONOTONIC_FAST , or .Dv BPF_T_NONE for backward compatibility reasons. Otherwise, .Vt bpf_xhdr is used. However, .Vt bpf_hdr may be deprecated in the near future. Suitable precautions must be taken when accessing the link layer protocol fields on alignment restricted machines. (This is not a problem on an Ethernet, since the type field is a short falling on an even offset, and the addresses are probably accessed in a bytewise fashion). .Pp Additionally, individual packets are padded so that each starts on a word boundary. This requires that an application has some knowledge of how to get from packet to packet. The macro .Dv BPF_WORDALIGN is defined in .In net/bpf.h to facilitate this process. It rounds up its argument to the nearest word aligned value (where a word is .Dv BPF_ALIGNMENT bytes wide). .Pp For example, if .Sq Li p points to the start of a packet, this expression will advance it to the next packet: .Dl p = (char *)p + BPF_WORDALIGN(p->bh_hdrlen + p->bh_caplen) .Pp For the alignment mechanisms to work properly, the buffer passed to .Xr read 2 must itself be word aligned. The .Xr malloc 3 function will always return an aligned buffer. .Sh FILTER MACHINE A filter program is an array of instructions, with all branches forwardly directed, terminated by a .Em return instruction. Each instruction performs some action on the pseudo-machine state, which consists of an accumulator, index register, scratch memory store, and implicit program counter. .Pp The following structure defines the instruction format: .Bd -literal struct bpf_insn { u_short code; u_char jt; u_char jf; u_long k; }; .Ed .Pp The .Li k field is used in different ways by different instructions, and the .Li jt and .Li jf fields are used as offsets by the branch instructions. The opcodes are encoded in a semi-hierarchical fashion. There are eight classes of instructions: .Dv BPF_LD , .Dv BPF_LDX , .Dv BPF_ST , .Dv BPF_STX , .Dv BPF_ALU , .Dv BPF_JMP , .Dv BPF_RET , and .Dv BPF_MISC . Various other mode and operator bits are or'd into the class to give the actual instructions. The classes and modes are defined in .In net/bpf.h . .Pp Below are the semantics for each defined .Nm instruction. We use the convention that A is the accumulator, X is the index register, P[] packet data, and M[] scratch memory store. P[i:n] gives the data at byte offset .Dq i in the packet, interpreted as a word (n=4), unsigned halfword (n=2), or unsigned byte (n=1). M[i] gives the i'th word in the scratch memory store, which is only addressed in word units. The memory store is indexed from 0 to .Dv BPF_MEMWORDS - 1. .Li k , .Li jt , and .Li jf are the corresponding fields in the instruction definition. .Dq len refers to the length of the packet. .Bl -tag -width BPF_STXx .It Dv BPF_LD These instructions copy a value into the accumulator. The type of the source operand is specified by an .Dq addressing mode and can be a constant .Pq Dv BPF_IMM , packet data at a fixed offset .Pq Dv BPF_ABS , packet data at a variable offset .Pq Dv BPF_IND , the packet length .Pq Dv BPF_LEN , or a word in the scratch memory store .Pq Dv BPF_MEM . For .Dv BPF_IND and .Dv BPF_ABS , the data size must be specified as a word .Pq Dv BPF_W , halfword .Pq Dv BPF_H , or byte .Pq Dv BPF_B . The semantics of all the recognized .Dv BPF_LD instructions follow. .Bd -literal BPF_LD+BPF_W+BPF_ABS A <- P[k:4] BPF_LD+BPF_H+BPF_ABS A <- P[k:2] BPF_LD+BPF_B+BPF_ABS A <- P[k:1] BPF_LD+BPF_W+BPF_IND A <- P[X+k:4] BPF_LD+BPF_H+BPF_IND A <- P[X+k:2] BPF_LD+BPF_B+BPF_IND A <- P[X+k:1] BPF_LD+BPF_W+BPF_LEN A <- len BPF_LD+BPF_IMM A <- k BPF_LD+BPF_MEM A <- M[k] .Ed .It Dv BPF_LDX These instructions load a value into the index register. Note that the addressing modes are more restrictive than those of the accumulator loads, but they include .Dv BPF_MSH , a hack for efficiently loading the IP header length. .Bd -literal BPF_LDX+BPF_W+BPF_IMM X <- k BPF_LDX+BPF_W+BPF_MEM X <- M[k] BPF_LDX+BPF_W+BPF_LEN X <- len BPF_LDX+BPF_B+BPF_MSH X <- 4*(P[k:1]&0xf) .Ed .It Dv BPF_ST This instruction stores the accumulator into the scratch memory. We do not need an addressing mode since there is only one possibility for the destination. .Bd -literal BPF_ST M[k] <- A .Ed .It Dv BPF_STX This instruction stores the index register in the scratch memory store. .Bd -literal BPF_STX M[k] <- X .Ed .It Dv BPF_ALU The alu instructions perform operations between the accumulator and index register or constant, and store the result back in the accumulator. For binary operations, a source mode is required .Dv ( BPF_K or .Dv BPF_X ) . .Bd -literal BPF_ALU+BPF_ADD+BPF_K A <- A + k BPF_ALU+BPF_SUB+BPF_K A <- A - k BPF_ALU+BPF_MUL+BPF_K A <- A * k BPF_ALU+BPF_DIV+BPF_K A <- A / k +BPF_ALU+BPF_MOD+BPF_K A <- A % k BPF_ALU+BPF_AND+BPF_K A <- A & k BPF_ALU+BPF_OR+BPF_K A <- A | k +BPF_ALU+BPF_XOR+BPF_K A <- A ^ k BPF_ALU+BPF_LSH+BPF_K A <- A << k BPF_ALU+BPF_RSH+BPF_K A <- A >> k BPF_ALU+BPF_ADD+BPF_X A <- A + X BPF_ALU+BPF_SUB+BPF_X A <- A - X BPF_ALU+BPF_MUL+BPF_X A <- A * X BPF_ALU+BPF_DIV+BPF_X A <- A / X +BPF_ALU+BPF_MOD+BPF_X A <- A % X BPF_ALU+BPF_AND+BPF_X A <- A & X BPF_ALU+BPF_OR+BPF_X A <- A | X +BPF_ALU+BPF_XOR+BPF_X A <- A ^ X BPF_ALU+BPF_LSH+BPF_X A <- A << X BPF_ALU+BPF_RSH+BPF_X A <- A >> X BPF_ALU+BPF_NEG A <- -A .Ed .It Dv BPF_JMP The jump instructions alter flow of control. Conditional jumps compare the accumulator against a constant .Pq Dv BPF_K or the index register .Pq Dv BPF_X . If the result is true (or non-zero), the true branch is taken, otherwise the false branch is taken. Jump offsets are encoded in 8 bits so the longest jump is 256 instructions. However, the jump always .Pq Dv BPF_JA opcode uses the 32 bit .Li k field as the offset, allowing arbitrarily distant destinations. All conditionals use unsigned comparison conventions. .Bd -literal BPF_JMP+BPF_JA pc += k BPF_JMP+BPF_JGT+BPF_K pc += (A > k) ? jt : jf BPF_JMP+BPF_JGE+BPF_K pc += (A >= k) ? jt : jf BPF_JMP+BPF_JEQ+BPF_K pc += (A == k) ? jt : jf BPF_JMP+BPF_JSET+BPF_K pc += (A & k) ? jt : jf BPF_JMP+BPF_JGT+BPF_X pc += (A > X) ? jt : jf BPF_JMP+BPF_JGE+BPF_X pc += (A >= X) ? jt : jf BPF_JMP+BPF_JEQ+BPF_X pc += (A == X) ? jt : jf BPF_JMP+BPF_JSET+BPF_X pc += (A & X) ? jt : jf .Ed .It Dv BPF_RET The return instructions terminate the filter program and specify the amount of packet to accept (i.e., they return the truncation amount). A return value of zero indicates that the packet should be ignored. The return value is either a constant .Pq Dv BPF_K or the accumulator .Pq Dv BPF_A . .Bd -literal BPF_RET+BPF_A accept A bytes BPF_RET+BPF_K accept k bytes .Ed .It Dv BPF_MISC The miscellaneous category was created for anything that does not fit into the above classes, and for any new instructions that might need to be added. Currently, these are the register transfer instructions that copy the index register to the accumulator or vice versa. .Bd -literal BPF_MISC+BPF_TAX X <- A BPF_MISC+BPF_TXA A <- X .Ed .El .Pp The .Nm interface provides the following macros to facilitate array initializers: .Fn BPF_STMT opcode operand and .Fn BPF_JUMP opcode operand true_offset false_offset . .Sh SYSCTL VARIABLES A set of .Xr sysctl 8 variables controls the behaviour of the .Nm subsystem .Bl -tag -width indent .It Va net.bpf.optimize_writers: No 0 Various programs use BPF to send (but not receive) raw packets (cdpd, lldpd, dhcpd, dhcp relays, etc. are good examples of such programs). They do not need incoming packets to be send to them. Turning this option on makes new BPF users to be attached to write-only interface list until program explicitly specifies read filter via .Fn pcap_set_filter . This removes any performance degradation for high-speed interfaces. .It Va net.bpf.stats: Binary interface for retrieving general statistics. .It Va net.bpf.zerocopy_enable: No 0 Permits zero-copy to be used with net BPF readers. Use with caution. .It Va net.bpf.maxinsns: No 512 Maximum number of instructions that BPF program can contain. Use .Xr tcpdump 1 .Fl d option to determine approximate number of instruction for any filter. .It Va net.bpf.maxbufsize: No 524288 Maximum buffer size to allocate for packets buffer. .It Va net.bpf.bufsize: No 4096 Default buffer size to allocate for packets buffer. .El .Sh EXAMPLES The following filter is taken from the Reverse ARP Daemon. It accepts only Reverse ARP requests. .Bd -literal struct bpf_insn insns[] = { BPF_STMT(BPF_LD+BPF_H+BPF_ABS, 12), BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, ETHERTYPE_REVARP, 0, 3), BPF_STMT(BPF_LD+BPF_H+BPF_ABS, 20), BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, REVARP_REQUEST, 0, 1), BPF_STMT(BPF_RET+BPF_K, sizeof(struct ether_arp) + sizeof(struct ether_header)), BPF_STMT(BPF_RET+BPF_K, 0), }; .Ed .Pp This filter accepts only IP packets between host 128.3.112.15 and 128.3.112.35. .Bd -literal struct bpf_insn insns[] = { BPF_STMT(BPF_LD+BPF_H+BPF_ABS, 12), BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, ETHERTYPE_IP, 0, 8), BPF_STMT(BPF_LD+BPF_W+BPF_ABS, 26), BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, 0x8003700f, 0, 2), BPF_STMT(BPF_LD+BPF_W+BPF_ABS, 30), BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, 0x80037023, 3, 4), BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, 0x80037023, 0, 3), BPF_STMT(BPF_LD+BPF_W+BPF_ABS, 30), BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, 0x8003700f, 0, 1), BPF_STMT(BPF_RET+BPF_K, (u_int)-1), BPF_STMT(BPF_RET+BPF_K, 0), }; .Ed .Pp Finally, this filter returns only TCP finger packets. We must parse the IP header to reach the TCP header. The .Dv BPF_JSET instruction checks that the IP fragment offset is 0 so we are sure that we have a TCP header. .Bd -literal struct bpf_insn insns[] = { BPF_STMT(BPF_LD+BPF_H+BPF_ABS, 12), BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, ETHERTYPE_IP, 0, 10), BPF_STMT(BPF_LD+BPF_B+BPF_ABS, 23), BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, IPPROTO_TCP, 0, 8), BPF_STMT(BPF_LD+BPF_H+BPF_ABS, 20), BPF_JUMP(BPF_JMP+BPF_JSET+BPF_K, 0x1fff, 6, 0), BPF_STMT(BPF_LDX+BPF_B+BPF_MSH, 14), BPF_STMT(BPF_LD+BPF_H+BPF_IND, 14), BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, 79, 2, 0), BPF_STMT(BPF_LD+BPF_H+BPF_IND, 16), BPF_JUMP(BPF_JMP+BPF_JEQ+BPF_K, 79, 0, 1), BPF_STMT(BPF_RET+BPF_K, (u_int)-1), BPF_STMT(BPF_RET+BPF_K, 0), }; .Ed .Sh SEE ALSO .Xr tcpdump 1 , .Xr ioctl 2 , .Xr kqueue 2 , .Xr poll 2 , .Xr select 2 , .Xr byteorder 3 , .Xr ng_bpf 4 , .Xr bpf 9 .Rs .%A McCanne, S. .%A Jacobson V. .%T "An efficient, extensible, and portable network monitor" .Re .Sh HISTORY The Enet packet filter was created in 1980 by Mike Accetta and Rick Rashid at Carnegie-Mellon University. Jeffrey Mogul, at Stanford, ported the code to .Bx and continued its development from 1983 on. Since then, it has evolved into the Ultrix Packet Filter at .Tn DEC , a .Tn STREAMS .Tn NIT module under .Tn SunOS 4.1 , and .Tn BPF . .Sh AUTHORS .An -nosplit .An Steven McCanne , of Lawrence Berkeley Laboratory, implemented BPF in Summer 1990. Much of the design is due to .An Van Jacobson . .Pp Support for zero-copy buffers was added by .An Robert N. M. Watson under contract to Seccuris Inc. .Sh BUGS The read buffer must be of a fixed size (returned by the .Dv BIOCGBLEN ioctl). .Pp A file that does not request promiscuous mode may receive promiscuously received packets as a side effect of another file requesting this mode on the same hardware interface. This could be fixed in the kernel with additional processing overhead. However, we favor the model where all files must assume that the interface is promiscuous, and if so desired, must utilize a filter to reject foreign packets. .Pp Data link protocols with variable length headers are not currently supported. .Pp The .Dv SEESENT , .Dv DIRECTION , and .Dv FEEDBACK settings have been observed to work incorrectly on some interface types, including those with hardware loopback rather than software loopback, and point-to-point interfaces. They appear to function correctly on a broad range of Ethernet-style interfaces. Index: head/sys/amd64/amd64/bpf_jit_machdep.c =================================================================== --- head/sys/amd64/amd64/bpf_jit_machdep.c (revision 307706) +++ head/sys/amd64/amd64/bpf_jit_machdep.c (revision 307707) @@ -1,638 +1,652 @@ /*- * Copyright (C) 2002-2003 NetGroup, Politecnico di Torino (Italy) - * Copyright (C) 2005-2009 Jung-uk Kim + * Copyright (C) 2005-2016 Jung-uk Kim * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of the Politecnico di Torino nor the names of its * contributors may be used to endorse or promote products derived from * this software without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. */ #include __FBSDID("$FreeBSD$"); #ifdef _KERNEL #include "opt_bpf.h" #include #include #include #include #include #include #include #else #include #include #include #include #endif #include #include #include #include bpf_filter_func bpf_jit_compile(struct bpf_insn *, u_int, size_t *); /* * Emit routine to update the jump table. */ static void emit_length(bpf_bin_stream *stream, __unused u_int value, u_int len) { if (stream->refs != NULL) (stream->refs)[stream->bpf_pc] += len; stream->cur_ip += len; } /* * Emit routine to output the actual binary code. */ static void emit_code(bpf_bin_stream *stream, u_int value, u_int len) { switch (len) { case 1: stream->ibuf[stream->cur_ip] = (u_char)value; stream->cur_ip++; break; case 2: *((u_short *)(void *)(stream->ibuf + stream->cur_ip)) = (u_short)value; stream->cur_ip += 2; break; case 4: *((u_int *)(void *)(stream->ibuf + stream->cur_ip)) = value; stream->cur_ip += 4; break; } return; } /* * Scan the filter program and find possible optimization. */ static int bpf_jit_optimize(struct bpf_insn *prog, u_int nins) { int flags; u_int i; /* Do we return immediately? */ if (BPF_CLASS(prog[0].code) == BPF_RET) return (BPF_JIT_FRET); for (flags = 0, i = 0; i < nins; i++) { switch (prog[i].code) { case BPF_LD|BPF_W|BPF_ABS: case BPF_LD|BPF_H|BPF_ABS: case BPF_LD|BPF_B|BPF_ABS: case BPF_LD|BPF_W|BPF_IND: case BPF_LD|BPF_H|BPF_IND: case BPF_LD|BPF_B|BPF_IND: case BPF_LDX|BPF_MSH|BPF_B: flags |= BPF_JIT_FPKT; break; case BPF_LD|BPF_MEM: case BPF_LDX|BPF_MEM: case BPF_ST: case BPF_STX: flags |= BPF_JIT_FMEM; break; case BPF_LD|BPF_W|BPF_LEN: case BPF_LDX|BPF_W|BPF_LEN: flags |= BPF_JIT_FLEN; break; case BPF_JMP|BPF_JA: case BPF_JMP|BPF_JGT|BPF_K: case BPF_JMP|BPF_JGE|BPF_K: case BPF_JMP|BPF_JEQ|BPF_K: case BPF_JMP|BPF_JSET|BPF_K: case BPF_JMP|BPF_JGT|BPF_X: case BPF_JMP|BPF_JGE|BPF_X: case BPF_JMP|BPF_JEQ|BPF_X: case BPF_JMP|BPF_JSET|BPF_X: flags |= BPF_JIT_FJMP; break; } if (flags == BPF_JIT_FLAG_ALL) break; } return (flags); } /* * Function that does the real stuff. */ bpf_filter_func bpf_jit_compile(struct bpf_insn *prog, u_int nins, size_t *size) { bpf_bin_stream stream; struct bpf_insn *ins; int flags, fret, fpkt, fmem, fjmp, flen; u_int i, pass; /* * NOTE: Do not modify the name of this variable, as it's used by * the macros to emit code. */ emit_func emitm; flags = bpf_jit_optimize(prog, nins); fret = (flags & BPF_JIT_FRET) != 0; fpkt = (flags & BPF_JIT_FPKT) != 0; fmem = (flags & BPF_JIT_FMEM) != 0; fjmp = (flags & BPF_JIT_FJMP) != 0; flen = (flags & BPF_JIT_FLEN) != 0; if (fret) nins = 1; memset(&stream, 0, sizeof(stream)); /* Allocate the reference table for the jumps. */ if (fjmp) { #ifdef _KERNEL stream.refs = malloc((nins + 1) * sizeof(u_int), M_BPFJIT, M_NOWAIT | M_ZERO); #else stream.refs = calloc(nins + 1, sizeof(u_int)); #endif if (stream.refs == NULL) return (NULL); } /* * The first pass will emit the lengths of the instructions * to create the reference table. */ emitm = emit_length; for (pass = 0; pass < 2; pass++) { ins = prog; /* Create the procedure header. */ if (fmem) { PUSH(RBP); MOVrq(RSP, RBP); SUBib(BPF_MEMWORDS * sizeof(uint32_t), RSP); } if (flen) MOVrd2(ESI, R9D); if (fpkt) { MOVrq2(RDI, R8); MOVrd(EDX, EDI); } for (i = 0; i < nins; i++) { stream.bpf_pc++; switch (ins->code) { default: #ifdef _KERNEL return (NULL); #else abort(); #endif case BPF_RET|BPF_K: MOVid(ins->k, EAX); if (fmem) LEAVE(); RET(); break; case BPF_RET|BPF_A: if (fmem) LEAVE(); RET(); break; case BPF_LD|BPF_W|BPF_ABS: MOVid(ins->k, ESI); CMPrd(EDI, ESI); JAb(12); MOVrd(EDI, ECX); SUBrd(ESI, ECX); CMPid(sizeof(int32_t), ECX); if (fmem) { JAEb(4); ZEROrd(EAX); LEAVE(); } else { JAEb(3); ZEROrd(EAX); } RET(); MOVrq3(R8, RCX); MOVobd(RCX, RSI, EAX); BSWAP(EAX); break; case BPF_LD|BPF_H|BPF_ABS: ZEROrd(EAX); MOVid(ins->k, ESI); CMPrd(EDI, ESI); JAb(12); MOVrd(EDI, ECX); SUBrd(ESI, ECX); CMPid(sizeof(int16_t), ECX); if (fmem) { JAEb(2); LEAVE(); } else JAEb(1); RET(); MOVrq3(R8, RCX); MOVobw(RCX, RSI, AX); SWAP_AX(); break; case BPF_LD|BPF_B|BPF_ABS: ZEROrd(EAX); MOVid(ins->k, ESI); CMPrd(EDI, ESI); if (fmem) { JBb(2); LEAVE(); } else JBb(1); RET(); MOVrq3(R8, RCX); MOVobb(RCX, RSI, AL); break; case BPF_LD|BPF_W|BPF_LEN: MOVrd3(R9D, EAX); break; case BPF_LDX|BPF_W|BPF_LEN: MOVrd3(R9D, EDX); break; case BPF_LD|BPF_W|BPF_IND: CMPrd(EDI, EDX); JAb(27); MOVid(ins->k, ESI); MOVrd(EDI, ECX); SUBrd(EDX, ECX); CMPrd(ESI, ECX); JBb(14); ADDrd(EDX, ESI); MOVrd(EDI, ECX); SUBrd(ESI, ECX); CMPid(sizeof(int32_t), ECX); if (fmem) { JAEb(4); ZEROrd(EAX); LEAVE(); } else { JAEb(3); ZEROrd(EAX); } RET(); MOVrq3(R8, RCX); MOVobd(RCX, RSI, EAX); BSWAP(EAX); break; case BPF_LD|BPF_H|BPF_IND: ZEROrd(EAX); CMPrd(EDI, EDX); JAb(27); MOVid(ins->k, ESI); MOVrd(EDI, ECX); SUBrd(EDX, ECX); CMPrd(ESI, ECX); JBb(14); ADDrd(EDX, ESI); MOVrd(EDI, ECX); SUBrd(ESI, ECX); CMPid(sizeof(int16_t), ECX); if (fmem) { JAEb(2); LEAVE(); } else JAEb(1); RET(); MOVrq3(R8, RCX); MOVobw(RCX, RSI, AX); SWAP_AX(); break; case BPF_LD|BPF_B|BPF_IND: ZEROrd(EAX); CMPrd(EDI, EDX); JAEb(13); MOVid(ins->k, ESI); MOVrd(EDI, ECX); SUBrd(EDX, ECX); CMPrd(ESI, ECX); if (fmem) { JAb(2); LEAVE(); } else JAb(1); RET(); MOVrq3(R8, RCX); ADDrd(EDX, ESI); MOVobb(RCX, RSI, AL); break; case BPF_LDX|BPF_MSH|BPF_B: MOVid(ins->k, ESI); CMPrd(EDI, ESI); if (fmem) { JBb(4); ZEROrd(EAX); LEAVE(); } else { JBb(3); ZEROrd(EAX); } RET(); ZEROrd(EDX); MOVrq3(R8, RCX); MOVobb(RCX, RSI, DL); ANDib(0x0f, DL); SHLib(2, EDX); break; case BPF_LD|BPF_IMM: MOVid(ins->k, EAX); break; case BPF_LDX|BPF_IMM: MOVid(ins->k, EDX); break; case BPF_LD|BPF_MEM: MOVid(ins->k * sizeof(uint32_t), ESI); MOVobd(RSP, RSI, EAX); break; case BPF_LDX|BPF_MEM: MOVid(ins->k * sizeof(uint32_t), ESI); MOVobd(RSP, RSI, EDX); break; case BPF_ST: /* * XXX this command and the following could * be optimized if the previous instruction * was already of this type */ MOVid(ins->k * sizeof(uint32_t), ESI); MOVomd(EAX, RSP, RSI); break; case BPF_STX: MOVid(ins->k * sizeof(uint32_t), ESI); MOVomd(EDX, RSP, RSI); break; case BPF_JMP|BPF_JA: JUMP(ins->k); break; case BPF_JMP|BPF_JGT|BPF_K: case BPF_JMP|BPF_JGE|BPF_K: case BPF_JMP|BPF_JEQ|BPF_K: case BPF_JMP|BPF_JSET|BPF_K: case BPF_JMP|BPF_JGT|BPF_X: case BPF_JMP|BPF_JGE|BPF_X: case BPF_JMP|BPF_JEQ|BPF_X: case BPF_JMP|BPF_JSET|BPF_X: if (ins->jt == ins->jf) { JUMP(ins->jt); break; } switch (ins->code) { case BPF_JMP|BPF_JGT|BPF_K: CMPid(ins->k, EAX); JCC(JA, JBE); break; case BPF_JMP|BPF_JGE|BPF_K: CMPid(ins->k, EAX); JCC(JAE, JB); break; case BPF_JMP|BPF_JEQ|BPF_K: CMPid(ins->k, EAX); JCC(JE, JNE); break; case BPF_JMP|BPF_JSET|BPF_K: TESTid(ins->k, EAX); JCC(JNE, JE); break; case BPF_JMP|BPF_JGT|BPF_X: CMPrd(EDX, EAX); JCC(JA, JBE); break; case BPF_JMP|BPF_JGE|BPF_X: CMPrd(EDX, EAX); JCC(JAE, JB); break; case BPF_JMP|BPF_JEQ|BPF_X: CMPrd(EDX, EAX); JCC(JE, JNE); break; case BPF_JMP|BPF_JSET|BPF_X: TESTrd(EDX, EAX); JCC(JNE, JE); break; } break; case BPF_ALU|BPF_ADD|BPF_X: ADDrd(EDX, EAX); break; case BPF_ALU|BPF_SUB|BPF_X: SUBrd(EDX, EAX); break; case BPF_ALU|BPF_MUL|BPF_X: MOVrd(EDX, ECX); MULrd(EDX); MOVrd(ECX, EDX); break; case BPF_ALU|BPF_DIV|BPF_X: + case BPF_ALU|BPF_MOD|BPF_X: TESTrd(EDX, EDX); if (fmem) { JNEb(4); ZEROrd(EAX); LEAVE(); } else { JNEb(3); ZEROrd(EAX); } RET(); MOVrd(EDX, ECX); ZEROrd(EDX); DIVrd(ECX); + if (BPF_OP(ins->code) == BPF_MOD) + MOVrd(EDX, EAX); MOVrd(ECX, EDX); break; case BPF_ALU|BPF_AND|BPF_X: ANDrd(EDX, EAX); break; case BPF_ALU|BPF_OR|BPF_X: ORrd(EDX, EAX); break; + case BPF_ALU|BPF_XOR|BPF_X: + XORrd(EDX, EAX); + break; + case BPF_ALU|BPF_LSH|BPF_X: MOVrd(EDX, ECX); SHL_CLrb(EAX); break; case BPF_ALU|BPF_RSH|BPF_X: MOVrd(EDX, ECX); SHR_CLrb(EAX); break; case BPF_ALU|BPF_ADD|BPF_K: ADD_EAXi(ins->k); break; case BPF_ALU|BPF_SUB|BPF_K: SUB_EAXi(ins->k); break; case BPF_ALU|BPF_MUL|BPF_K: MOVrd(EDX, ECX); MOVid(ins->k, EDX); MULrd(EDX); MOVrd(ECX, EDX); break; case BPF_ALU|BPF_DIV|BPF_K: + case BPF_ALU|BPF_MOD|BPF_K: MOVrd(EDX, ECX); ZEROrd(EDX); MOVid(ins->k, ESI); DIVrd(ESI); + if (BPF_OP(ins->code) == BPF_MOD) + MOVrd(EDX, EAX); MOVrd(ECX, EDX); break; case BPF_ALU|BPF_AND|BPF_K: ANDid(ins->k, EAX); break; case BPF_ALU|BPF_OR|BPF_K: ORid(ins->k, EAX); + break; + + case BPF_ALU|BPF_XOR|BPF_K: + XORid(ins->k, EAX); break; case BPF_ALU|BPF_LSH|BPF_K: SHLib((ins->k) & 0xff, EAX); break; case BPF_ALU|BPF_RSH|BPF_K: SHRib((ins->k) & 0xff, EAX); break; case BPF_ALU|BPF_NEG: NEGd(EAX); break; case BPF_MISC|BPF_TAX: MOVrd(EAX, EDX); break; case BPF_MISC|BPF_TXA: MOVrd(EDX, EAX); break; } ins++; } if (pass > 0) continue; *size = stream.cur_ip; #ifdef _KERNEL stream.ibuf = malloc(*size, M_BPFJIT, M_NOWAIT); if (stream.ibuf == NULL) break; #else stream.ibuf = mmap(NULL, *size, PROT_READ | PROT_WRITE, MAP_ANON, -1, 0); if (stream.ibuf == MAP_FAILED) { stream.ibuf = NULL; break; } #endif /* * Modify the reference table to contain the offsets and * not the lengths of the instructions. */ if (fjmp) for (i = 1; i < nins + 1; i++) stream.refs[i] += stream.refs[i - 1]; /* Reset the counters. */ stream.cur_ip = 0; stream.bpf_pc = 0; /* The second pass creates the actual code. */ emitm = emit_code; } /* * The reference table is needed only during compilation, * now we can free it. */ if (fjmp) #ifdef _KERNEL free(stream.refs, M_BPFJIT); #else free(stream.refs); #endif #ifndef _KERNEL if (stream.ibuf != NULL && mprotect(stream.ibuf, *size, PROT_READ | PROT_EXEC) != 0) { munmap(stream.ibuf, *size); stream.ibuf = NULL; } #endif return ((bpf_filter_func)(void *)stream.ibuf); } Index: head/sys/amd64/amd64/bpf_jit_machdep.h =================================================================== --- head/sys/amd64/amd64/bpf_jit_machdep.h (revision 307706) +++ head/sys/amd64/amd64/bpf_jit_machdep.h (revision 307707) @@ -1,482 +1,500 @@ /*- * Copyright (C) 2002-2003 NetGroup, Politecnico di Torino (Italy) - * Copyright (C) 2005-2009 Jung-uk Kim + * Copyright (C) 2005-2016 Jung-uk Kim * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of the Politecnico di Torino nor the names of its * contributors may be used to endorse or promote products derived from * this software without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. * * $FreeBSD$ */ #ifndef _BPF_JIT_MACHDEP_H_ #define _BPF_JIT_MACHDEP_H_ /* * Registers */ #define RAX 0 #define RCX 1 #define RDX 2 #define RBX 3 #define RSP 4 #define RBP 5 #define RSI 6 #define RDI 7 #define R8 0 #define R9 1 #define R10 2 #define R11 3 #define R12 4 #define R13 5 #define R14 6 #define R15 7 #define EAX 0 #define ECX 1 #define EDX 2 #define EBX 3 #define ESP 4 #define EBP 5 #define ESI 6 #define EDI 7 #define R8D 0 #define R9D 1 #define R10D 2 #define R11D 3 #define R12D 4 #define R13D 5 #define R14D 6 #define R15D 7 #define AX 0 #define CX 1 #define DX 2 #define BX 3 #define SP 4 #define BP 5 #define SI 6 #define DI 7 #define AL 0 #define CL 1 #define DL 2 #define BL 3 /* Optimization flags */ #define BPF_JIT_FRET 0x01 #define BPF_JIT_FPKT 0x02 #define BPF_JIT_FMEM 0x04 #define BPF_JIT_FJMP 0x08 #define BPF_JIT_FLEN 0x10 #define BPF_JIT_FLAG_ALL \ (BPF_JIT_FPKT | BPF_JIT_FMEM | BPF_JIT_FJMP | BPF_JIT_FLEN) /* A stream of native binary code */ typedef struct bpf_bin_stream { /* Current native instruction pointer. */ int cur_ip; /* * Current BPF instruction pointer, i.e. position in * the BPF program reached by the jitter. */ int bpf_pc; /* Instruction buffer, contains the generated native code. */ char *ibuf; /* Jumps reference table. */ u_int *refs; } bpf_bin_stream; /* * Prototype of the emit functions. * * Different emit functions are used to create the reference table and * to generate the actual filtering code. This allows to have simpler * instruction macros. * The first parameter is the stream that will receive the data. * The second one is a variable containing the data. * The third one is the length, that can be 1, 2, or 4 since it is possible * to emit a byte, a short, or a word at a time. */ typedef void (*emit_func)(bpf_bin_stream *stream, u_int value, u_int n); /* * Native instruction macros */ /* movl i32,r32 */ #define MOVid(i32, r32) do { \ emitm(&stream, (11 << 4) | (1 << 3) | (r32 & 0x7), 1); \ emitm(&stream, i32, 4); \ } while (0) /* movq i64,r64 */ #define MOViq(i64, r64) do { \ emitm(&stream, 0x48, 1); \ emitm(&stream, (11 << 4) | (1 << 3) | (r64 & 0x7), 1); \ emitm(&stream, i64, 4); \ emitm(&stream, (i64 >> 32), 4); \ } while (0) /* movl sr32,dr32 */ #define MOVrd(sr32, dr32) do { \ emitm(&stream, 0x89, 1); \ emitm(&stream, \ (3 << 6) | ((sr32 & 0x7) << 3) | (dr32 & 0x7), 1); \ } while (0) /* movl sr32,dr32 (dr32 = %r8-15d) */ #define MOVrd2(sr32, dr32) do { \ emitm(&stream, 0x8941, 2); \ emitm(&stream, \ (3 << 6) | ((sr32 & 0x7) << 3) | (dr32 & 0x7), 1); \ } while (0) /* movl sr32,dr32 (sr32 = %r8-15d) */ #define MOVrd3(sr32, dr32) do { \ emitm(&stream, 0x8944, 2); \ emitm(&stream, \ (3 << 6) | ((sr32 & 0x7) << 3) | (dr32 & 0x7), 1); \ } while (0) /* movq sr64,dr64 */ #define MOVrq(sr64, dr64) do { \ emitm(&stream, 0x8948, 2); \ emitm(&stream, \ (3 << 6) | ((sr64 & 0x7) << 3) | (dr64 & 0x7), 1); \ } while (0) /* movq sr64,dr64 (dr64 = %r8-15) */ #define MOVrq2(sr64, dr64) do { \ emitm(&stream, 0x8949, 2); \ emitm(&stream, \ (3 << 6) | ((sr64 & 0x7) << 3) | (dr64 & 0x7), 1); \ } while (0) /* movq sr64,dr64 (sr64 = %r8-15) */ #define MOVrq3(sr64, dr64) do { \ emitm(&stream, 0x894c, 2); \ emitm(&stream, \ (3 << 6) | ((sr64 & 0x7) << 3) | (dr64 & 0x7), 1); \ } while (0) /* movl (sr64,or64,1),dr32 */ #define MOVobd(sr64, or64, dr32) do { \ emitm(&stream, 0x8b, 1); \ emitm(&stream, ((dr32 & 0x7) << 3) | 4, 1); \ emitm(&stream, ((or64 & 0x7) << 3) | (sr64 & 0x7), 1); \ } while (0) /* movw (sr64,or64,1),dr16 */ #define MOVobw(sr64, or64, dr16) do { \ emitm(&stream, 0x8b66, 2); \ emitm(&stream, ((dr16 & 0x7) << 3) | 4, 1); \ emitm(&stream, ((or64 & 0x7) << 3) | (sr64 & 0x7), 1); \ } while (0) /* movb (sr64,or64,1),dr8 */ #define MOVobb(sr64, or64, dr8) do { \ emitm(&stream, 0x8a, 1); \ emitm(&stream, ((dr8 & 0x7) << 3) | 4, 1); \ emitm(&stream, ((or64 & 0x7) << 3) | (sr64 & 0x7), 1); \ } while (0) /* movl sr32,(dr64,or64,1) */ #define MOVomd(sr32, dr64, or64) do { \ emitm(&stream, 0x89, 1); \ emitm(&stream, ((sr32 & 0x7) << 3) | 4, 1); \ emitm(&stream, ((or64 & 0x7) << 3) | (dr64 & 0x7), 1); \ } while (0) /* bswapl dr32 */ #define BSWAP(dr32) do { \ emitm(&stream, 0xf, 1); \ emitm(&stream, (0x19 << 3) | dr32, 1); \ } while (0) /* xchgb %al,%ah */ #define SWAP_AX() do { \ emitm(&stream, 0xc486, 2); \ } while (0) /* pushq r64 */ #define PUSH(r64) do { \ emitm(&stream, (5 << 4) | (0 << 3) | (r64 & 0x7), 1); \ } while (0) /* leaveq */ #define LEAVE() do { \ emitm(&stream, 0xc9, 1); \ } while (0) /* retq */ #define RET() do { \ emitm(&stream, 0xc3, 1); \ } while (0) /* addl sr32,dr32 */ #define ADDrd(sr32, dr32) do { \ emitm(&stream, 0x01, 1); \ emitm(&stream, \ (3 << 6) | ((sr32 & 0x7) << 3) | (dr32 & 0x7), 1); \ } while (0) /* addl i32,%eax */ #define ADD_EAXi(i32) do { \ emitm(&stream, 0x05, 1); \ emitm(&stream, i32, 4); \ } while (0) /* addl i8,r32 */ #define ADDib(i8, r32) do { \ emitm(&stream, 0x83, 1); \ emitm(&stream, (24 << 3) | r32, 1); \ emitm(&stream, i8, 1); \ } while (0) /* subl sr32,dr32 */ #define SUBrd(sr32, dr32) do { \ emitm(&stream, 0x29, 1); \ emitm(&stream, \ (3 << 6) | ((sr32 & 0x7) << 3) | (dr32 & 0x7), 1); \ } while (0) /* subl i32,%eax */ #define SUB_EAXi(i32) do { \ emitm(&stream, 0x2d, 1); \ emitm(&stream, i32, 4); \ } while (0) /* subq i8,r64 */ #define SUBib(i8, r64) do { \ emitm(&stream, 0x8348, 2); \ emitm(&stream, (29 << 3) | (r64 & 0x7), 1); \ emitm(&stream, i8, 1); \ } while (0) /* mull r32 */ #define MULrd(r32) do { \ emitm(&stream, 0xf7, 1); \ emitm(&stream, (7 << 5) | (r32 & 0x7), 1); \ } while (0) /* divl r32 */ #define DIVrd(r32) do { \ emitm(&stream, 0xf7, 1); \ emitm(&stream, (15 << 4) | (r32 & 0x7), 1); \ } while (0) /* andb i8,r8 */ #define ANDib(i8, r8) do { \ if (r8 == AL) { \ emitm(&stream, 0x24, 1); \ } else { \ emitm(&stream, 0x80, 1); \ emitm(&stream, (7 << 5) | r8, 1); \ } \ emitm(&stream, i8, 1); \ } while (0) /* andl i32,r32 */ #define ANDid(i32, r32) do { \ if (r32 == EAX) { \ emitm(&stream, 0x25, 1); \ } else { \ emitm(&stream, 0x81, 1); \ emitm(&stream, (7 << 5) | r32, 1); \ } \ emitm(&stream, i32, 4); \ } while (0) /* andl sr32,dr32 */ #define ANDrd(sr32, dr32) do { \ emitm(&stream, 0x21, 1); \ emitm(&stream, \ (3 << 6) | ((sr32 & 0x7) << 3) | (dr32 & 0x7), 1); \ } while (0) /* testl i32,r32 */ #define TESTid(i32, r32) do { \ if (r32 == EAX) { \ emitm(&stream, 0xa9, 1); \ } else { \ emitm(&stream, 0xf7, 1); \ emitm(&stream, (3 << 6) | r32, 1); \ } \ emitm(&stream, i32, 4); \ } while (0) /* testl sr32,dr32 */ #define TESTrd(sr32, dr32) do { \ emitm(&stream, 0x85, 1); \ emitm(&stream, \ (3 << 6) | ((sr32 & 0x7) << 3) | (dr32 & 0x7), 1); \ } while (0) /* orl sr32,dr32 */ #define ORrd(sr32, dr32) do { \ emitm(&stream, 0x09, 1); \ emitm(&stream, \ (3 << 6) | ((sr32 & 0x7) << 3) | (dr32 & 0x7), 1); \ } while (0) /* orl i32,r32 */ #define ORid(i32, r32) do { \ if (r32 == EAX) { \ emitm(&stream, 0x0d, 1); \ + } else { \ + emitm(&stream, 0x81, 1); \ + emitm(&stream, (25 << 3) | r32, 1); \ + } \ + emitm(&stream, i32, 4); \ +} while (0) + +/* xorl sr32,dr32 */ +#define XORrd(sr32, dr32) do { \ + emitm(&stream, 0x31, 1); \ + emitm(&stream, \ + (3 << 6) | ((sr32 & 0x7) << 3) | (dr32 & 0x7), 1); \ +} while (0) + +/* xorl i32,r32 */ +#define XORid(i32, r32) do { \ + if (r32 == EAX) { \ + emitm(&stream, 0x35, 1); \ } else { \ emitm(&stream, 0x81, 1); \ emitm(&stream, (25 << 3) | r32, 1); \ } \ emitm(&stream, i32, 4); \ } while (0) /* shll i8,r32 */ #define SHLib(i8, r32) do { \ emitm(&stream, 0xc1, 1); \ emitm(&stream, (7 << 5) | (r32 & 0x7), 1); \ emitm(&stream, i8, 1); \ } while (0) /* shll %cl,dr32 */ #define SHL_CLrb(dr32) do { \ emitm(&stream, 0xd3, 1); \ emitm(&stream, (7 << 5) | (dr32 & 0x7), 1); \ } while (0) /* shrl i8,r32 */ #define SHRib(i8, r32) do { \ emitm(&stream, 0xc1, 1); \ emitm(&stream, (29 << 3) | (r32 & 0x7), 1); \ emitm(&stream, i8, 1); \ } while (0) /* shrl %cl,dr32 */ #define SHR_CLrb(dr32) do { \ emitm(&stream, 0xd3, 1); \ emitm(&stream, (29 << 3) | (dr32 & 0x7), 1); \ } while (0) /* negl r32 */ #define NEGd(r32) do { \ emitm(&stream, 0xf7, 1); \ emitm(&stream, (27 << 3) | (r32 & 0x7), 1); \ } while (0) /* cmpl sr32,dr32 */ #define CMPrd(sr32, dr32) do { \ emitm(&stream, 0x39, 1); \ emitm(&stream, \ (3 << 6) | ((sr32 & 0x7) << 3) | (dr32 & 0x7), 1); \ } while (0) /* cmpl i32,dr32 */ #define CMPid(i32, dr32) do { \ if (dr32 == EAX){ \ emitm(&stream, 0x3d, 1); \ emitm(&stream, i32, 4); \ } else { \ emitm(&stream, 0x81, 1); \ emitm(&stream, (0x1f << 3) | (dr32 & 0x7), 1); \ emitm(&stream, i32, 4); \ } \ } while (0) /* jb off8 */ #define JBb(off8) do { \ emitm(&stream, 0x72, 1); \ emitm(&stream, off8, 1); \ } while (0) /* jae off8 */ #define JAEb(off8) do { \ emitm(&stream, 0x73, 1); \ emitm(&stream, off8, 1); \ } while (0) /* jne off8 */ #define JNEb(off8) do { \ emitm(&stream, 0x75, 1); \ emitm(&stream, off8, 1); \ } while (0) /* ja off8 */ #define JAb(off8) do { \ emitm(&stream, 0x77, 1); \ emitm(&stream, off8, 1); \ } while (0) /* jmp off32 */ #define JMP(off32) do { \ emitm(&stream, 0xe9, 1); \ emitm(&stream, off32, 4); \ } while (0) /* xorl r32,r32 */ #define ZEROrd(r32) do { \ emitm(&stream, 0x31, 1); \ emitm(&stream, (3 << 6) | ((r32 & 0x7) << 3) | (r32 & 0x7), 1); \ } while (0) /* * Conditional long jumps */ #define JB 0x82 #define JAE 0x83 #define JE 0x84 #define JNE 0x85 #define JBE 0x86 #define JA 0x87 #define JCC(t, f) do { \ if (ins->jt != 0 && ins->jf != 0) { \ /* 5 is the size of the following jmp */ \ emitm(&stream, ((t) << 8) | 0x0f, 2); \ emitm(&stream, stream.refs[stream.bpf_pc + ins->jt] - \ stream.refs[stream.bpf_pc] + 5, 4); \ JMP(stream.refs[stream.bpf_pc + ins->jf] - \ stream.refs[stream.bpf_pc]); \ } else if (ins->jt != 0) { \ emitm(&stream, ((t) << 8) | 0x0f, 2); \ emitm(&stream, stream.refs[stream.bpf_pc + ins->jt] - \ stream.refs[stream.bpf_pc], 4); \ } else { \ emitm(&stream, ((f) << 8) | 0x0f, 2); \ emitm(&stream, stream.refs[stream.bpf_pc + ins->jf] - \ stream.refs[stream.bpf_pc], 4); \ } \ } while (0) #define JUMP(off) do { \ if ((off) != 0) \ JMP(stream.refs[stream.bpf_pc + (off)] - \ stream.refs[stream.bpf_pc]); \ } while (0) #endif /* _BPF_JIT_MACHDEP_H_ */ Index: head/sys/i386/i386/bpf_jit_machdep.c =================================================================== --- head/sys/i386/i386/bpf_jit_machdep.c (revision 307706) +++ head/sys/i386/i386/bpf_jit_machdep.c (revision 307707) @@ -1,667 +1,682 @@ /*- * Copyright (C) 2002-2003 NetGroup, Politecnico di Torino (Italy) - * Copyright (C) 2005-2009 Jung-uk Kim + * Copyright (C) 2005-2016 Jung-uk Kim * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of the Politecnico di Torino nor the names of its * contributors may be used to endorse or promote products derived from * this software without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. */ #include __FBSDID("$FreeBSD$"); #ifdef _KERNEL #include "opt_bpf.h" #include #include #include #include #include #include #include #else #include #include #include #include #endif #include #include #include #include bpf_filter_func bpf_jit_compile(struct bpf_insn *, u_int, size_t *); /* * Emit routine to update the jump table. */ static void emit_length(bpf_bin_stream *stream, __unused u_int value, u_int len) { if (stream->refs != NULL) (stream->refs)[stream->bpf_pc] += len; stream->cur_ip += len; } /* * Emit routine to output the actual binary code. */ static void emit_code(bpf_bin_stream *stream, u_int value, u_int len) { switch (len) { case 1: stream->ibuf[stream->cur_ip] = (u_char)value; stream->cur_ip++; break; case 2: *((u_short *)(void *)(stream->ibuf + stream->cur_ip)) = (u_short)value; stream->cur_ip += 2; break; case 4: *((u_int *)(void *)(stream->ibuf + stream->cur_ip)) = value; stream->cur_ip += 4; break; } return; } /* * Scan the filter program and find possible optimization. */ static int bpf_jit_optimize(struct bpf_insn *prog, u_int nins) { int flags; u_int i; /* Do we return immediately? */ if (BPF_CLASS(prog[0].code) == BPF_RET) return (BPF_JIT_FRET); for (flags = 0, i = 0; i < nins; i++) { switch (prog[i].code) { case BPF_LD|BPF_W|BPF_ABS: case BPF_LD|BPF_H|BPF_ABS: case BPF_LD|BPF_B|BPF_ABS: case BPF_LD|BPF_W|BPF_IND: case BPF_LD|BPF_H|BPF_IND: case BPF_LD|BPF_B|BPF_IND: case BPF_LDX|BPF_MSH|BPF_B: flags |= BPF_JIT_FPKT; break; case BPF_LD|BPF_MEM: case BPF_LDX|BPF_MEM: case BPF_ST: case BPF_STX: flags |= BPF_JIT_FMEM; break; case BPF_JMP|BPF_JA: case BPF_JMP|BPF_JGT|BPF_K: case BPF_JMP|BPF_JGE|BPF_K: case BPF_JMP|BPF_JEQ|BPF_K: case BPF_JMP|BPF_JSET|BPF_K: case BPF_JMP|BPF_JGT|BPF_X: case BPF_JMP|BPF_JGE|BPF_X: case BPF_JMP|BPF_JEQ|BPF_X: case BPF_JMP|BPF_JSET|BPF_X: flags |= BPF_JIT_FJMP; break; case BPF_ALU|BPF_DIV|BPF_K: + case BPF_ALU|BPF_MOD|BPF_K: flags |= BPF_JIT_FADK; break; } if (flags == BPF_JIT_FLAG_ALL) break; } return (flags); } /* * Function that does the real stuff. */ bpf_filter_func bpf_jit_compile(struct bpf_insn *prog, u_int nins, size_t *size) { bpf_bin_stream stream; struct bpf_insn *ins; int flags, fret, fpkt, fmem, fjmp, fadk; int save_esp; u_int i, pass; /* * NOTE: Do not modify the name of this variable, as it's used by * the macros to emit code. */ emit_func emitm; flags = bpf_jit_optimize(prog, nins); fret = (flags & BPF_JIT_FRET) != 0; fpkt = (flags & BPF_JIT_FPKT) != 0; fmem = (flags & BPF_JIT_FMEM) != 0; fjmp = (flags & BPF_JIT_FJMP) != 0; fadk = (flags & BPF_JIT_FADK) != 0; save_esp = (fpkt || fmem || fadk); /* Stack is used. */ if (fret) nins = 1; memset(&stream, 0, sizeof(stream)); /* Allocate the reference table for the jumps. */ if (fjmp) { #ifdef _KERNEL stream.refs = malloc((nins + 1) * sizeof(u_int), M_BPFJIT, M_NOWAIT | M_ZERO); #else stream.refs = calloc(nins + 1, sizeof(u_int)); #endif if (stream.refs == NULL) return (NULL); } /* * The first pass will emit the lengths of the instructions * to create the reference table. */ emitm = emit_length; for (pass = 0; pass < 2; pass++) { ins = prog; /* Create the procedure header. */ if (save_esp) { PUSH(EBP); MOVrd(ESP, EBP); } if (fmem) SUBib(BPF_MEMWORDS * sizeof(uint32_t), ESP); if (save_esp) PUSH(ESI); if (fpkt) { PUSH(EDI); PUSH(EBX); MOVodd(8, EBP, EBX); MOVodd(16, EBP, EDI); } for (i = 0; i < nins; i++) { stream.bpf_pc++; switch (ins->code) { default: #ifdef _KERNEL return (NULL); #else abort(); #endif case BPF_RET|BPF_K: MOVid(ins->k, EAX); if (save_esp) { if (fpkt) { POP(EBX); POP(EDI); } POP(ESI); LEAVE(); } RET(); break; case BPF_RET|BPF_A: if (save_esp) { if (fpkt) { POP(EBX); POP(EDI); } POP(ESI); LEAVE(); } RET(); break; case BPF_LD|BPF_W|BPF_ABS: MOVid(ins->k, ESI); CMPrd(EDI, ESI); JAb(12); MOVrd(EDI, ECX); SUBrd(ESI, ECX); CMPid(sizeof(int32_t), ECX); JAEb(7); ZEROrd(EAX); POP(EBX); POP(EDI); POP(ESI); LEAVE(); RET(); MOVobd(EBX, ESI, EAX); BSWAP(EAX); break; case BPF_LD|BPF_H|BPF_ABS: ZEROrd(EAX); MOVid(ins->k, ESI); CMPrd(EDI, ESI); JAb(12); MOVrd(EDI, ECX); SUBrd(ESI, ECX); CMPid(sizeof(int16_t), ECX); JAEb(5); POP(EBX); POP(EDI); POP(ESI); LEAVE(); RET(); MOVobw(EBX, ESI, AX); SWAP_AX(); break; case BPF_LD|BPF_B|BPF_ABS: ZEROrd(EAX); MOVid(ins->k, ESI); CMPrd(EDI, ESI); JBb(5); POP(EBX); POP(EDI); POP(ESI); LEAVE(); RET(); MOVobb(EBX, ESI, AL); break; case BPF_LD|BPF_W|BPF_LEN: if (save_esp) MOVodd(12, EBP, EAX); else { MOVrd(ESP, ECX); MOVodd(12, ECX, EAX); } break; case BPF_LDX|BPF_W|BPF_LEN: if (save_esp) MOVodd(12, EBP, EDX); else { MOVrd(ESP, ECX); MOVodd(12, ECX, EDX); } break; case BPF_LD|BPF_W|BPF_IND: CMPrd(EDI, EDX); JAb(27); MOVid(ins->k, ESI); MOVrd(EDI, ECX); SUBrd(EDX, ECX); CMPrd(ESI, ECX); JBb(14); ADDrd(EDX, ESI); MOVrd(EDI, ECX); SUBrd(ESI, ECX); CMPid(sizeof(int32_t), ECX); JAEb(7); ZEROrd(EAX); POP(EBX); POP(EDI); POP(ESI); LEAVE(); RET(); MOVobd(EBX, ESI, EAX); BSWAP(EAX); break; case BPF_LD|BPF_H|BPF_IND: ZEROrd(EAX); CMPrd(EDI, EDX); JAb(27); MOVid(ins->k, ESI); MOVrd(EDI, ECX); SUBrd(EDX, ECX); CMPrd(ESI, ECX); JBb(14); ADDrd(EDX, ESI); MOVrd(EDI, ECX); SUBrd(ESI, ECX); CMPid(sizeof(int16_t), ECX); JAEb(5); POP(EBX); POP(EDI); POP(ESI); LEAVE(); RET(); MOVobw(EBX, ESI, AX); SWAP_AX(); break; case BPF_LD|BPF_B|BPF_IND: ZEROrd(EAX); CMPrd(EDI, EDX); JAEb(13); MOVid(ins->k, ESI); MOVrd(EDI, ECX); SUBrd(EDX, ECX); CMPrd(ESI, ECX); JAb(5); POP(EBX); POP(EDI); POP(ESI); LEAVE(); RET(); ADDrd(EDX, ESI); MOVobb(EBX, ESI, AL); break; case BPF_LDX|BPF_MSH|BPF_B: MOVid(ins->k, ESI); CMPrd(EDI, ESI); JBb(7); ZEROrd(EAX); POP(EBX); POP(EDI); POP(ESI); LEAVE(); RET(); ZEROrd(EDX); MOVobb(EBX, ESI, DL); ANDib(0x0f, DL); SHLib(2, EDX); break; case BPF_LD|BPF_IMM: MOVid(ins->k, EAX); break; case BPF_LDX|BPF_IMM: MOVid(ins->k, EDX); break; case BPF_LD|BPF_MEM: MOVrd(EBP, ECX); MOVid(((int)ins->k - BPF_MEMWORDS) * sizeof(uint32_t), ESI); MOVobd(ECX, ESI, EAX); break; case BPF_LDX|BPF_MEM: MOVrd(EBP, ECX); MOVid(((int)ins->k - BPF_MEMWORDS) * sizeof(uint32_t), ESI); MOVobd(ECX, ESI, EDX); break; case BPF_ST: /* * XXX this command and the following could * be optimized if the previous instruction * was already of this type */ MOVrd(EBP, ECX); MOVid(((int)ins->k - BPF_MEMWORDS) * sizeof(uint32_t), ESI); MOVomd(EAX, ECX, ESI); break; case BPF_STX: MOVrd(EBP, ECX); MOVid(((int)ins->k - BPF_MEMWORDS) * sizeof(uint32_t), ESI); MOVomd(EDX, ECX, ESI); break; case BPF_JMP|BPF_JA: JUMP(ins->k); break; case BPF_JMP|BPF_JGT|BPF_K: case BPF_JMP|BPF_JGE|BPF_K: case BPF_JMP|BPF_JEQ|BPF_K: case BPF_JMP|BPF_JSET|BPF_K: case BPF_JMP|BPF_JGT|BPF_X: case BPF_JMP|BPF_JGE|BPF_X: case BPF_JMP|BPF_JEQ|BPF_X: case BPF_JMP|BPF_JSET|BPF_X: if (ins->jt == ins->jf) { JUMP(ins->jt); break; } switch (ins->code) { case BPF_JMP|BPF_JGT|BPF_K: CMPid(ins->k, EAX); JCC(JA, JBE); break; case BPF_JMP|BPF_JGE|BPF_K: CMPid(ins->k, EAX); JCC(JAE, JB); break; case BPF_JMP|BPF_JEQ|BPF_K: CMPid(ins->k, EAX); JCC(JE, JNE); break; case BPF_JMP|BPF_JSET|BPF_K: TESTid(ins->k, EAX); JCC(JNE, JE); break; case BPF_JMP|BPF_JGT|BPF_X: CMPrd(EDX, EAX); JCC(JA, JBE); break; case BPF_JMP|BPF_JGE|BPF_X: CMPrd(EDX, EAX); JCC(JAE, JB); break; case BPF_JMP|BPF_JEQ|BPF_X: CMPrd(EDX, EAX); JCC(JE, JNE); break; case BPF_JMP|BPF_JSET|BPF_X: TESTrd(EDX, EAX); JCC(JNE, JE); break; } break; case BPF_ALU|BPF_ADD|BPF_X: ADDrd(EDX, EAX); break; case BPF_ALU|BPF_SUB|BPF_X: SUBrd(EDX, EAX); break; case BPF_ALU|BPF_MUL|BPF_X: MOVrd(EDX, ECX); MULrd(EDX); MOVrd(ECX, EDX); break; case BPF_ALU|BPF_DIV|BPF_X: + case BPF_ALU|BPF_MOD|BPF_X: TESTrd(EDX, EDX); if (save_esp) { if (fpkt) { JNEb(7); ZEROrd(EAX); POP(EBX); POP(EDI); } else { JNEb(5); ZEROrd(EAX); } POP(ESI); LEAVE(); } else { JNEb(3); ZEROrd(EAX); } RET(); MOVrd(EDX, ECX); ZEROrd(EDX); DIVrd(ECX); + if (BPF_OP(ins->code) == BPF_MOD) + MOVrd(EDX, EAX); MOVrd(ECX, EDX); break; case BPF_ALU|BPF_AND|BPF_X: ANDrd(EDX, EAX); break; case BPF_ALU|BPF_OR|BPF_X: ORrd(EDX, EAX); break; + case BPF_ALU|BPF_XOR|BPF_X: + XORrd(EDX, EAX); + break; + case BPF_ALU|BPF_LSH|BPF_X: MOVrd(EDX, ECX); SHL_CLrb(EAX); break; case BPF_ALU|BPF_RSH|BPF_X: MOVrd(EDX, ECX); SHR_CLrb(EAX); break; case BPF_ALU|BPF_ADD|BPF_K: ADD_EAXi(ins->k); break; case BPF_ALU|BPF_SUB|BPF_K: SUB_EAXi(ins->k); break; case BPF_ALU|BPF_MUL|BPF_K: MOVrd(EDX, ECX); MOVid(ins->k, EDX); MULrd(EDX); MOVrd(ECX, EDX); break; case BPF_ALU|BPF_DIV|BPF_K: + case BPF_ALU|BPF_MOD|BPF_K: MOVrd(EDX, ECX); ZEROrd(EDX); MOVid(ins->k, ESI); DIVrd(ESI); + if (BPF_OP(ins->code) == BPF_MOD) + MOVrd(EDX, EAX); MOVrd(ECX, EDX); break; case BPF_ALU|BPF_AND|BPF_K: ANDid(ins->k, EAX); break; case BPF_ALU|BPF_OR|BPF_K: ORid(ins->k, EAX); + break; + + case BPF_ALU|BPF_XOR|BPF_K: + XORid(ins->k, EAX); break; case BPF_ALU|BPF_LSH|BPF_K: SHLib((ins->k) & 0xff, EAX); break; case BPF_ALU|BPF_RSH|BPF_K: SHRib((ins->k) & 0xff, EAX); break; case BPF_ALU|BPF_NEG: NEGd(EAX); break; case BPF_MISC|BPF_TAX: MOVrd(EAX, EDX); break; case BPF_MISC|BPF_TXA: MOVrd(EDX, EAX); break; } ins++; } if (pass > 0) continue; *size = stream.cur_ip; #ifdef _KERNEL stream.ibuf = malloc(*size, M_BPFJIT, M_NOWAIT); if (stream.ibuf == NULL) break; #else stream.ibuf = mmap(NULL, *size, PROT_READ | PROT_WRITE, MAP_ANON, -1, 0); if (stream.ibuf == MAP_FAILED) { stream.ibuf = NULL; break; } #endif /* * Modify the reference table to contain the offsets and * not the lengths of the instructions. */ if (fjmp) for (i = 1; i < nins + 1; i++) stream.refs[i] += stream.refs[i - 1]; /* Reset the counters. */ stream.cur_ip = 0; stream.bpf_pc = 0; /* The second pass creates the actual code. */ emitm = emit_code; } /* * The reference table is needed only during compilation, * now we can free it. */ if (fjmp) #ifdef _KERNEL free(stream.refs, M_BPFJIT); #else free(stream.refs); #endif #ifndef _KERNEL if (stream.ibuf != NULL && mprotect(stream.ibuf, *size, PROT_READ | PROT_EXEC) != 0) { munmap(stream.ibuf, *size); stream.ibuf = NULL; } #endif return ((bpf_filter_func)(void *)stream.ibuf); } Index: head/sys/i386/i386/bpf_jit_machdep.h =================================================================== --- head/sys/i386/i386/bpf_jit_machdep.h (revision 307706) +++ head/sys/i386/i386/bpf_jit_machdep.h (revision 307707) @@ -1,427 +1,445 @@ /*- * Copyright (C) 2002-2003 NetGroup, Politecnico di Torino (Italy) - * Copyright (C) 2005-2009 Jung-uk Kim + * Copyright (C) 2005-2016 Jung-uk Kim * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of the Politecnico di Torino nor the names of its * contributors may be used to endorse or promote products derived from * this software without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. * * $FreeBSD$ */ #ifndef _BPF_JIT_MACHDEP_H_ #define _BPF_JIT_MACHDEP_H_ /* * Registers */ #define EAX 0 #define ECX 1 #define EDX 2 #define EBX 3 #define ESP 4 #define EBP 5 #define ESI 6 #define EDI 7 #define AX 0 #define CX 1 #define DX 2 #define BX 3 #define SP 4 #define BP 5 #define SI 6 #define DI 7 #define AL 0 #define CL 1 #define DL 2 #define BL 3 /* Optimization flags */ #define BPF_JIT_FRET 0x01 #define BPF_JIT_FPKT 0x02 #define BPF_JIT_FMEM 0x04 #define BPF_JIT_FJMP 0x08 #define BPF_JIT_FADK 0x10 #define BPF_JIT_FLAG_ALL \ (BPF_JIT_FPKT | BPF_JIT_FMEM | BPF_JIT_FJMP | BPF_JIT_FADK) /* A stream of native binary code */ typedef struct bpf_bin_stream { /* Current native instruction pointer. */ int cur_ip; /* * Current BPF instruction pointer, i.e. position in * the BPF program reached by the jitter. */ int bpf_pc; /* Instruction buffer, contains the generated native code. */ char *ibuf; /* Jumps reference table. */ u_int *refs; } bpf_bin_stream; /* * Prototype of the emit functions. * * Different emit functions are used to create the reference table and * to generate the actual filtering code. This allows to have simpler * instruction macros. * The first parameter is the stream that will receive the data. * The second one is a variable containing the data. * The third one is the length, that can be 1, 2, or 4 since it is possible * to emit a byte, a short, or a word at a time. */ typedef void (*emit_func)(bpf_bin_stream *stream, u_int value, u_int n); /* * Native instruction macros */ /* movl i32,r32 */ #define MOVid(i32, r32) do { \ emitm(&stream, (11 << 4) | (1 << 3) | (r32 & 0x7), 1); \ emitm(&stream, i32, 4); \ } while (0) /* movl sr32,dr32 */ #define MOVrd(sr32, dr32) do { \ emitm(&stream, 0x89, 1); \ emitm(&stream, \ (3 << 6) | ((sr32 & 0x7) << 3) | (dr32 & 0x7), 1); \ } while (0) /* movl off(sr32),dr32 */ #define MOVodd(off, sr32, dr32) do { \ emitm(&stream, 0x8b, 1); \ emitm(&stream, \ (1 << 6) | ((dr32 & 0x7) << 3) | (sr32 & 0x7), 1); \ emitm(&stream, off, 1); \ } while (0) /* movl (sr32,or32,1),dr32 */ #define MOVobd(sr32, or32, dr32) do { \ emitm(&stream, 0x8b, 1); \ emitm(&stream, ((dr32 & 0x7) << 3) | 4, 1); \ emitm(&stream, ((or32 & 0x7) << 3) | (sr32 & 0x7), 1); \ } while (0) /* movw (sr32,or32,1),dr16 */ #define MOVobw(sr32, or32, dr16) do { \ emitm(&stream, 0x8b66, 2); \ emitm(&stream, ((dr16 & 0x7) << 3) | 4, 1); \ emitm(&stream, ((or32 & 0x7) << 3) | (sr32 & 0x7), 1); \ } while (0) /* movb (sr32,or32,1),dr8 */ #define MOVobb(sr32, or32, dr8) do { \ emitm(&stream, 0x8a, 1); \ emitm(&stream, ((dr8 & 0x7) << 3) | 4, 1); \ emitm(&stream, ((or32 & 0x7) << 3) | (sr32 & 0x7), 1); \ } while (0) /* movl sr32,(dr32,or32,1) */ #define MOVomd(sr32, dr32, or32) do { \ emitm(&stream, 0x89, 1); \ emitm(&stream, ((sr32 & 0x7) << 3) | 4, 1); \ emitm(&stream, ((or32 & 0x7) << 3) | (dr32 & 0x7), 1); \ } while (0) /* bswapl dr32 */ #define BSWAP(dr32) do { \ emitm(&stream, 0xf, 1); \ emitm(&stream, (0x19 << 3) | dr32, 1); \ } while (0) /* xchgb %al,%ah */ #define SWAP_AX() do { \ emitm(&stream, 0xc486, 2); \ } while (0) /* pushl r32 */ #define PUSH(r32) do { \ emitm(&stream, (5 << 4) | (0 << 3) | (r32 & 0x7), 1); \ } while (0) /* popl r32 */ #define POP(r32) do { \ emitm(&stream, (5 << 4) | (1 << 3) | (r32 & 0x7), 1); \ } while (0) /* leave */ #define LEAVE() do { \ emitm(&stream, 0xc9, 1); \ } while (0) /* ret */ #define RET() do { \ emitm(&stream, 0xc3, 1); \ } while (0) /* addl sr32,dr32 */ #define ADDrd(sr32, dr32) do { \ emitm(&stream, 0x01, 1); \ emitm(&stream, \ (3 << 6) | ((sr32 & 0x7) << 3) | (dr32 & 0x7), 1); \ } while (0) /* addl i32,%eax */ #define ADD_EAXi(i32) do { \ emitm(&stream, 0x05, 1); \ emitm(&stream, i32, 4); \ } while (0) /* addl i8,r32 */ #define ADDib(i8, r32) do { \ emitm(&stream, 0x83, 1); \ emitm(&stream, (24 << 3) | r32, 1); \ emitm(&stream, i8, 1); \ } while (0) /* subl sr32,dr32 */ #define SUBrd(sr32, dr32) do { \ emitm(&stream, 0x29, 1); \ emitm(&stream, \ (3 << 6) | ((sr32 & 0x7) << 3) | (dr32 & 0x7), 1); \ } while (0) /* subl i32,%eax */ #define SUB_EAXi(i32) do { \ emitm(&stream, 0x2d, 1); \ emitm(&stream, i32, 4); \ } while (0) /* subl i8,r32 */ #define SUBib(i8, r32) do { \ emitm(&stream, 0x83, 1); \ emitm(&stream, (29 << 3) | (r32 & 0x7), 1); \ emitm(&stream, i8, 1); \ } while (0) /* mull r32 */ #define MULrd(r32) do { \ emitm(&stream, 0xf7, 1); \ emitm(&stream, (7 << 5) | (r32 & 0x7), 1); \ } while (0) /* divl r32 */ #define DIVrd(r32) do { \ emitm(&stream, 0xf7, 1); \ emitm(&stream, (15 << 4) | (r32 & 0x7), 1); \ } while (0) /* andb i8,r8 */ #define ANDib(i8, r8) do { \ if (r8 == AL) { \ emitm(&stream, 0x24, 1); \ } else { \ emitm(&stream, 0x80, 1); \ emitm(&stream, (7 << 5) | r8, 1); \ } \ emitm(&stream, i8, 1); \ } while (0) /* andl i32,r32 */ #define ANDid(i32, r32) do { \ if (r32 == EAX) { \ emitm(&stream, 0x25, 1); \ } else { \ emitm(&stream, 0x81, 1); \ emitm(&stream, (7 << 5) | r32, 1); \ } \ emitm(&stream, i32, 4); \ } while (0) /* andl sr32,dr32 */ #define ANDrd(sr32, dr32) do { \ emitm(&stream, 0x21, 1); \ emitm(&stream, \ (3 << 6) | ((sr32 & 0x7) << 3) | (dr32 & 0x7), 1); \ } while (0) /* testl i32,r32 */ #define TESTid(i32, r32) do { \ if (r32 == EAX) { \ emitm(&stream, 0xa9, 1); \ } else { \ emitm(&stream, 0xf7, 1); \ emitm(&stream, (3 << 6) | r32, 1); \ } \ emitm(&stream, i32, 4); \ } while (0) /* testl sr32,dr32 */ #define TESTrd(sr32, dr32) do { \ emitm(&stream, 0x85, 1); \ emitm(&stream, \ (3 << 6) | ((sr32 & 0x7) << 3) | (dr32 & 0x7), 1); \ } while (0) /* orl sr32,dr32 */ #define ORrd(sr32, dr32) do { \ emitm(&stream, 0x09, 1); \ emitm(&stream, \ (3 << 6) | ((sr32 & 0x7) << 3) | (dr32 & 0x7), 1); \ } while (0) /* orl i32,r32 */ #define ORid(i32, r32) do { \ if (r32 == EAX) { \ emitm(&stream, 0x0d, 1); \ + } else { \ + emitm(&stream, 0x81, 1); \ + emitm(&stream, (25 << 3) | r32, 1); \ + } \ + emitm(&stream, i32, 4); \ +} while (0) + +/* xorl sr32,dr32 */ +#define XORrd(sr32, dr32) do { \ + emitm(&stream, 0x31, 1); \ + emitm(&stream, \ + (3 << 6) | ((sr32 & 0x7) << 3) | (dr32 & 0x7), 1); \ +} while (0) + +/* xorl i32,r32 */ +#define XORid(i32, r32) do { \ + if (r32 == EAX) { \ + emitm(&stream, 0x35, 1); \ } else { \ emitm(&stream, 0x81, 1); \ emitm(&stream, (25 << 3) | r32, 1); \ } \ emitm(&stream, i32, 4); \ } while (0) /* shll i8,r32 */ #define SHLib(i8, r32) do { \ emitm(&stream, 0xc1, 1); \ emitm(&stream, (7 << 5) | (r32 & 0x7), 1); \ emitm(&stream, i8, 1); \ } while (0) /* shll %cl,dr32 */ #define SHL_CLrb(dr32) do { \ emitm(&stream, 0xd3, 1); \ emitm(&stream, (7 << 5) | (dr32 & 0x7), 1); \ } while (0) /* shrl i8,r32 */ #define SHRib(i8, r32) do { \ emitm(&stream, 0xc1, 1); \ emitm(&stream, (29 << 3) | (r32 & 0x7), 1); \ emitm(&stream, i8, 1); \ } while (0) /* shrl %cl,dr32 */ #define SHR_CLrb(dr32) do { \ emitm(&stream, 0xd3, 1); \ emitm(&stream, (29 << 3) | (dr32 & 0x7), 1); \ } while (0) /* negl r32 */ #define NEGd(r32) do { \ emitm(&stream, 0xf7, 1); \ emitm(&stream, (27 << 3) | (r32 & 0x7), 1); \ } while (0) /* cmpl sr32,dr32 */ #define CMPrd(sr32, dr32) do { \ emitm(&stream, 0x39, 1); \ emitm(&stream, \ (3 << 6) | ((sr32 & 0x7) << 3) | (dr32 & 0x7), 1); \ } while (0) /* cmpl i32,dr32 */ #define CMPid(i32, dr32) do { \ if (dr32 == EAX){ \ emitm(&stream, 0x3d, 1); \ emitm(&stream, i32, 4); \ } else { \ emitm(&stream, 0x81, 1); \ emitm(&stream, (0x1f << 3) | (dr32 & 0x7), 1); \ emitm(&stream, i32, 4); \ } \ } while (0) /* jb off8 */ #define JBb(off8) do { \ emitm(&stream, 0x72, 1); \ emitm(&stream, off8, 1); \ } while (0) /* jae off8 */ #define JAEb(off8) do { \ emitm(&stream, 0x73, 1); \ emitm(&stream, off8, 1); \ } while (0) /* jne off8 */ #define JNEb(off8) do { \ emitm(&stream, 0x75, 1); \ emitm(&stream, off8, 1); \ } while (0) /* ja off8 */ #define JAb(off8) do { \ emitm(&stream, 0x77, 1); \ emitm(&stream, off8, 1); \ } while (0) /* jmp off32 */ #define JMP(off32) do { \ emitm(&stream, 0xe9, 1); \ emitm(&stream, off32, 4); \ } while (0) /* xorl r32,r32 */ #define ZEROrd(r32) do { \ emitm(&stream, 0x31, 1); \ emitm(&stream, (3 << 6) | ((r32 & 0x7) << 3) | (r32 & 0x7), 1); \ } while (0) /* * Conditional long jumps */ #define JB 0x82 #define JAE 0x83 #define JE 0x84 #define JNE 0x85 #define JBE 0x86 #define JA 0x87 #define JCC(t, f) do { \ if (ins->jt != 0 && ins->jf != 0) { \ /* 5 is the size of the following jmp */ \ emitm(&stream, ((t) << 8) | 0x0f, 2); \ emitm(&stream, stream.refs[stream.bpf_pc + ins->jt] - \ stream.refs[stream.bpf_pc] + 5, 4); \ JMP(stream.refs[stream.bpf_pc + ins->jf] - \ stream.refs[stream.bpf_pc]); \ } else if (ins->jt != 0) { \ emitm(&stream, ((t) << 8) | 0x0f, 2); \ emitm(&stream, stream.refs[stream.bpf_pc + ins->jt] - \ stream.refs[stream.bpf_pc], 4); \ } else { \ emitm(&stream, ((f) << 8) | 0x0f, 2); \ emitm(&stream, stream.refs[stream.bpf_pc + ins->jf] - \ stream.refs[stream.bpf_pc], 4); \ } \ } while (0) #define JUMP(off) do { \ if ((off) != 0) \ JMP(stream.refs[stream.bpf_pc + (off)] - \ stream.refs[stream.bpf_pc]); \ } while (0) #endif /* _BPF_JIT_MACHDEP_H_ */ Index: head/sys/net/bpf_filter.c =================================================================== --- head/sys/net/bpf_filter.c (revision 307706) +++ head/sys/net/bpf_filter.c (revision 307707) @@ -1,585 +1,604 @@ /*- * Copyright (c) 1990, 1991, 1993 * The Regents of the University of California. All rights reserved. * * This code is derived from the Stanford/CMU enet packet filter, * (net/enet.c) distributed as part of 4.3BSD, and code contributed * to Berkeley by Steven McCanne and Van Jacobson both of Lawrence * Berkeley Laboratory. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 4. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * @(#)bpf_filter.c 8.1 (Berkeley) 6/10/93 */ #include __FBSDID("$FreeBSD$"); #include #if !defined(_KERNEL) #include #endif #if !defined(_KERNEL) || defined(sun) #include #endif #ifndef __i386__ #define BPF_ALIGN #endif #ifndef BPF_ALIGN #define EXTRACT_SHORT(p) ((u_int16_t)ntohs(*(u_int16_t *)p)) #define EXTRACT_LONG(p) (ntohl(*(u_int32_t *)p)) #else #define EXTRACT_SHORT(p)\ ((u_int16_t)\ ((u_int16_t)*((u_char *)p+0)<<8|\ (u_int16_t)*((u_char *)p+1)<<0)) #define EXTRACT_LONG(p)\ ((u_int32_t)*((u_char *)p+0)<<24|\ (u_int32_t)*((u_char *)p+1)<<16|\ (u_int32_t)*((u_char *)p+2)<<8|\ (u_int32_t)*((u_char *)p+3)<<0) #endif #ifdef _KERNEL #include #else #include #endif #include #ifdef _KERNEL #define MINDEX(m, k) \ { \ register int len = m->m_len; \ \ while (k >= len) { \ k -= len; \ m = m->m_next; \ if (m == 0) \ return (0); \ len = m->m_len; \ } \ } static u_int16_t m_xhalf(struct mbuf *m, bpf_u_int32 k, int *err); static u_int32_t m_xword(struct mbuf *m, bpf_u_int32 k, int *err); static u_int32_t m_xword(struct mbuf *m, bpf_u_int32 k, int *err) { size_t len; u_char *cp, *np; struct mbuf *m0; len = m->m_len; while (k >= len) { k -= len; m = m->m_next; if (m == NULL) goto bad; len = m->m_len; } cp = mtod(m, u_char *) + k; if (len - k >= 4) { *err = 0; return (EXTRACT_LONG(cp)); } m0 = m->m_next; if (m0 == NULL || m0->m_len + len - k < 4) goto bad; *err = 0; np = mtod(m0, u_char *); switch (len - k) { case 1: return (((u_int32_t)cp[0] << 24) | ((u_int32_t)np[0] << 16) | ((u_int32_t)np[1] << 8) | (u_int32_t)np[2]); case 2: return (((u_int32_t)cp[0] << 24) | ((u_int32_t)cp[1] << 16) | ((u_int32_t)np[0] << 8) | (u_int32_t)np[1]); default: return (((u_int32_t)cp[0] << 24) | ((u_int32_t)cp[1] << 16) | ((u_int32_t)cp[2] << 8) | (u_int32_t)np[0]); } bad: *err = 1; return (0); } static u_int16_t m_xhalf(struct mbuf *m, bpf_u_int32 k, int *err) { size_t len; u_char *cp; struct mbuf *m0; len = m->m_len; while (k >= len) { k -= len; m = m->m_next; if (m == NULL) goto bad; len = m->m_len; } cp = mtod(m, u_char *) + k; if (len - k >= 2) { *err = 0; return (EXTRACT_SHORT(cp)); } m0 = m->m_next; if (m0 == NULL) goto bad; *err = 0; return ((cp[0] << 8) | mtod(m0, u_char *)[0]); bad: *err = 1; return (0); } #endif /* * Execute the filter program starting at pc on the packet p * wirelen is the length of the original packet * buflen is the amount of data present */ u_int bpf_filter(const struct bpf_insn *pc, u_char *p, u_int wirelen, u_int buflen) { u_int32_t A = 0, X = 0; bpf_u_int32 k; u_int32_t mem[BPF_MEMWORDS]; bzero(mem, sizeof(mem)); if (pc == NULL) /* * No filter means accept all. */ return ((u_int)-1); --pc; while (1) { ++pc; switch (pc->code) { default: #ifdef _KERNEL return (0); #else abort(); #endif case BPF_RET|BPF_K: return ((u_int)pc->k); case BPF_RET|BPF_A: return ((u_int)A); case BPF_LD|BPF_W|BPF_ABS: k = pc->k; if (k > buflen || sizeof(int32_t) > buflen - k) { #ifdef _KERNEL int merr; if (buflen != 0) return (0); A = m_xword((struct mbuf *)p, k, &merr); if (merr != 0) return (0); continue; #else return (0); #endif } #ifdef BPF_ALIGN if (((intptr_t)(p + k) & 3) != 0) A = EXTRACT_LONG(&p[k]); else #endif A = ntohl(*(int32_t *)(p + k)); continue; case BPF_LD|BPF_H|BPF_ABS: k = pc->k; if (k > buflen || sizeof(int16_t) > buflen - k) { #ifdef _KERNEL int merr; if (buflen != 0) return (0); A = m_xhalf((struct mbuf *)p, k, &merr); continue; #else return (0); #endif } A = EXTRACT_SHORT(&p[k]); continue; case BPF_LD|BPF_B|BPF_ABS: k = pc->k; if (k >= buflen) { #ifdef _KERNEL struct mbuf *m; if (buflen != 0) return (0); m = (struct mbuf *)p; MINDEX(m, k); A = mtod(m, u_char *)[k]; continue; #else return (0); #endif } A = p[k]; continue; case BPF_LD|BPF_W|BPF_LEN: A = wirelen; continue; case BPF_LDX|BPF_W|BPF_LEN: X = wirelen; continue; case BPF_LD|BPF_W|BPF_IND: k = X + pc->k; if (pc->k > buflen || X > buflen - pc->k || sizeof(int32_t) > buflen - k) { #ifdef _KERNEL int merr; if (buflen != 0) return (0); A = m_xword((struct mbuf *)p, k, &merr); if (merr != 0) return (0); continue; #else return (0); #endif } #ifdef BPF_ALIGN if (((intptr_t)(p + k) & 3) != 0) A = EXTRACT_LONG(&p[k]); else #endif A = ntohl(*(int32_t *)(p + k)); continue; case BPF_LD|BPF_H|BPF_IND: k = X + pc->k; if (X > buflen || pc->k > buflen - X || sizeof(int16_t) > buflen - k) { #ifdef _KERNEL int merr; if (buflen != 0) return (0); A = m_xhalf((struct mbuf *)p, k, &merr); if (merr != 0) return (0); continue; #else return (0); #endif } A = EXTRACT_SHORT(&p[k]); continue; case BPF_LD|BPF_B|BPF_IND: k = X + pc->k; if (pc->k >= buflen || X >= buflen - pc->k) { #ifdef _KERNEL struct mbuf *m; if (buflen != 0) return (0); m = (struct mbuf *)p; MINDEX(m, k); A = mtod(m, u_char *)[k]; continue; #else return (0); #endif } A = p[k]; continue; case BPF_LDX|BPF_MSH|BPF_B: k = pc->k; if (k >= buflen) { #ifdef _KERNEL register struct mbuf *m; if (buflen != 0) return (0); m = (struct mbuf *)p; MINDEX(m, k); X = (mtod(m, u_char *)[k] & 0xf) << 2; continue; #else return (0); #endif } X = (p[pc->k] & 0xf) << 2; continue; case BPF_LD|BPF_IMM: A = pc->k; continue; case BPF_LDX|BPF_IMM: X = pc->k; continue; case BPF_LD|BPF_MEM: A = mem[pc->k]; continue; case BPF_LDX|BPF_MEM: X = mem[pc->k]; continue; case BPF_ST: mem[pc->k] = A; continue; case BPF_STX: mem[pc->k] = X; continue; case BPF_JMP|BPF_JA: pc += pc->k; continue; case BPF_JMP|BPF_JGT|BPF_K: pc += (A > pc->k) ? pc->jt : pc->jf; continue; case BPF_JMP|BPF_JGE|BPF_K: pc += (A >= pc->k) ? pc->jt : pc->jf; continue; case BPF_JMP|BPF_JEQ|BPF_K: pc += (A == pc->k) ? pc->jt : pc->jf; continue; case BPF_JMP|BPF_JSET|BPF_K: pc += (A & pc->k) ? pc->jt : pc->jf; continue; case BPF_JMP|BPF_JGT|BPF_X: pc += (A > X) ? pc->jt : pc->jf; continue; case BPF_JMP|BPF_JGE|BPF_X: pc += (A >= X) ? pc->jt : pc->jf; continue; case BPF_JMP|BPF_JEQ|BPF_X: pc += (A == X) ? pc->jt : pc->jf; continue; case BPF_JMP|BPF_JSET|BPF_X: pc += (A & X) ? pc->jt : pc->jf; continue; case BPF_ALU|BPF_ADD|BPF_X: A += X; continue; case BPF_ALU|BPF_SUB|BPF_X: A -= X; continue; case BPF_ALU|BPF_MUL|BPF_X: A *= X; continue; case BPF_ALU|BPF_DIV|BPF_X: if (X == 0) return (0); A /= X; continue; + case BPF_ALU|BPF_MOD|BPF_X: + if (X == 0) + return (0); + A %= X; + continue; + case BPF_ALU|BPF_AND|BPF_X: A &= X; continue; case BPF_ALU|BPF_OR|BPF_X: A |= X; continue; + case BPF_ALU|BPF_XOR|BPF_X: + A ^= X; + continue; + case BPF_ALU|BPF_LSH|BPF_X: A <<= X; continue; case BPF_ALU|BPF_RSH|BPF_X: A >>= X; continue; case BPF_ALU|BPF_ADD|BPF_K: A += pc->k; continue; case BPF_ALU|BPF_SUB|BPF_K: A -= pc->k; continue; case BPF_ALU|BPF_MUL|BPF_K: A *= pc->k; continue; case BPF_ALU|BPF_DIV|BPF_K: A /= pc->k; continue; + case BPF_ALU|BPF_MOD|BPF_K: + A %= pc->k; + continue; + case BPF_ALU|BPF_AND|BPF_K: A &= pc->k; continue; case BPF_ALU|BPF_OR|BPF_K: A |= pc->k; continue; + case BPF_ALU|BPF_XOR|BPF_K: + A ^= pc->k; + continue; + case BPF_ALU|BPF_LSH|BPF_K: A <<= pc->k; continue; case BPF_ALU|BPF_RSH|BPF_K: A >>= pc->k; continue; case BPF_ALU|BPF_NEG: A = -A; continue; case BPF_MISC|BPF_TAX: X = A; continue; case BPF_MISC|BPF_TXA: A = X; continue; } } } #ifdef _KERNEL static const u_short bpf_code_map[] = { 0x10ff, /* 0x00-0x0f: 1111111100001000 */ 0x3070, /* 0x10-0x1f: 0000111000001100 */ 0x3131, /* 0x20-0x2f: 1000110010001100 */ 0x3031, /* 0x30-0x3f: 1000110000001100 */ 0x3131, /* 0x40-0x4f: 1000110010001100 */ 0x1011, /* 0x50-0x5f: 1000100000001000 */ 0x1013, /* 0x60-0x6f: 1100100000001000 */ 0x1010, /* 0x70-0x7f: 0000100000001000 */ 0x0093, /* 0x80-0x8f: 1100100100000000 */ - 0x0000, /* 0x90-0x9f: 0000000000000000 */ - 0x0000, /* 0xa0-0xaf: 0000000000000000 */ + 0x1010, /* 0x90-0x9f: 0000100000001000 */ + 0x1010, /* 0xa0-0xaf: 0000100000001000 */ 0x0002, /* 0xb0-0xbf: 0100000000000000 */ 0x0000, /* 0xc0-0xcf: 0000000000000000 */ 0x0000, /* 0xd0-0xdf: 0000000000000000 */ 0x0000, /* 0xe0-0xef: 0000000000000000 */ 0x0000 /* 0xf0-0xff: 0000000000000000 */ }; #define BPF_VALIDATE_CODE(c) \ ((c) <= 0xff && (bpf_code_map[(c) >> 4] & (1 << ((c) & 0xf))) != 0) /* * Return true if the 'fcode' is a valid filter program. * The constraints are that each jump be forward and to a valid * code. The code must terminate with either an accept or reject. * * The kernel needs to be able to verify an application's filter code. * Otherwise, a bogus program could easily crash the system. */ int bpf_validate(const struct bpf_insn *f, int len) { register int i; register const struct bpf_insn *p; /* Do not accept negative length filter. */ if (len < 0) return (0); /* An empty filter means accept all. */ if (len == 0) return (1); for (i = 0; i < len; ++i) { p = &f[i]; /* * Check that the code is valid. */ if (!BPF_VALIDATE_CODE(p->code)) return (0); /* * Check that that jumps are forward, and within * the code block. */ if (BPF_CLASS(p->code) == BPF_JMP) { register u_int offset; if (p->code == (BPF_JMP|BPF_JA)) offset = p->k; else offset = p->jt > p->jf ? p->jt : p->jf; if (offset >= (u_int)(len - i) - 1) return (0); continue; } /* * Check that memory operations use valid addresses. */ if (p->code == BPF_ST || p->code == BPF_STX || p->code == (BPF_LD|BPF_MEM) || p->code == (BPF_LDX|BPF_MEM)) { if (p->k >= BPF_MEMWORDS) return (0); continue; } /* * Check for constant division by 0. */ - if (p->code == (BPF_ALU|BPF_DIV|BPF_K) && p->k == 0) + if ((p->code == (BPF_ALU|BPF_DIV|BPF_K) || + p->code == (BPF_ALU|BPF_MOD|BPF_K)) && p->k == 0) return (0); } return (BPF_CLASS(f[len - 1].code) == BPF_RET); } #endif