Index: user/imp/tbemd/contrib/netcat/nc.1 =================================================================== --- user/imp/tbemd/contrib/netcat/nc.1 (revision 206684) +++ user/imp/tbemd/contrib/netcat/nc.1 (revision 206685) @@ -1,474 +1,465 @@ .\" $OpenBSD: nc.1,v 1.53 2010/02/23 23:00:52 schwarze Exp $ .\" .\" Copyright (c) 1996 David Sacerdote .\" All rights reserved. .\" .\" Redistribution and use in source and binary forms, with or without .\" modification, are permitted provided that the following conditions .\" are met: .\" 1. Redistributions of source code must retain the above copyright .\" notice, this list of conditions and the following disclaimer. .\" 2. Redistributions in binary form must reproduce the above copyright .\" notice, this list of conditions and the following disclaimer in the .\" documentation and/or other materials provided with the distribution. .\" 3. The name of the author may not be used to endorse or promote products .\" derived from this software without specific prior written permission .\" .\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR .\" IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES .\" OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. .\" IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, .\" INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT .\" NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, .\" DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY .\" THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT .\" (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF .\" THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. .\" .\" $FreeBSD$ .\" -.Dd February 23, 2010 +.Dd April 15, 2010 .Dt NC 1 .Os .Sh NAME .Nm nc .Nd arbitrary TCP and UDP connections and listens .Sh SYNOPSIS .Nm nc .Bk -words -.Op Fl 46DdEhklnorStUuvz +.Op Fl 46DdEhklnrStUuvz .Op Fl e Ar IPsec_policy .Op Fl I Ar length .Op Fl i Ar interval .Op Fl -no-tcpopt .Op Fl O Ar length .Op Fl P Ar proxy_username .Op Fl p Ar source_port .Op Fl s Ar source_ip_address .Op Fl T Ar ToS .Op Fl V Ar fib .Op Fl w Ar timeout .Op Fl X Ar proxy_protocol .Oo Xo .Fl x Ar proxy_address Ns Oo : Ns .Ar port Oc .Xc Oc .Op Ar hostname .Op Ar port .Ek .Sh DESCRIPTION The .Nm (or .Nm netcat ) utility is used for just about anything under the sun involving TCP or UDP. It can open TCP connections, send UDP packets, listen on arbitrary TCP and UDP ports, do port scanning, and deal with both IPv4 and IPv6. Unlike .Xr telnet 1 , .Nm scripts nicely, and separates error messages onto standard error instead of sending them to standard output, as .Xr telnet 1 does with some. .Pp Common uses include: .Pp .Bl -bullet -offset indent -compact .It simple TCP proxies .It shell-script based HTTP clients and servers .It network daemon testing .It a SOCKS or HTTP ProxyCommand for .Xr ssh 1 .It and much, much more .El .Pp The options are as follows: .Bl -tag -width Ds .It Fl 4 Forces .Nm to use IPv4 addresses only. .It Fl 6 Forces .Nm to use IPv6 addresses only. .It Fl D Enable debugging on the socket. .It Fl d Do not attempt to read from stdin. .It Fl E Shortcut for .Qo .Li "-e 'in ipsec esp/transport//require'" .Li "-e 'out ipsec esp/transport//require'" .Qc , which enables IPsec ESP transport mode in both directions. .It Fl e If IPsec support is available, then one can specify the IPsec policies to be used using the syntax described in .Xr ipsec_set_policy 3 . This flag can be specified up to two times, as typically one policy for each direction is needed. .It Fl h Prints out .Nm help. .It Fl I Ar length Specifies the size of the TCP receive buffer. .It Fl i Ar interval Specifies a delay time interval between lines of text sent and received. Also causes a delay time between connections to multiple ports. .It Fl k Forces .Nm to stay listening for another connection after its current connection is completed. It is an error to use this option without the .Fl l option. .It Fl l Used to specify that .Nm should listen for an incoming connection rather than initiate a connection to a remote host. It is an error to use this option in conjunction with the .Fl p , .Fl s , or .Fl z options. Additionally, any timeouts specified with the .Fl w option are ignored. .It Fl n Do not do any DNS or service lookups on any specified addresses, hostnames or ports. .It Fl -no-tcpopt Disables the use of TCP options on the socket, by setting the boolean TCP_NOOPT socket option. .It Fl O Ar length Specifies the size of the TCP send buffer. When -.It Fl o -.Dq Once-only mode . -By default, -.Nm -does not terminate on EOF condition on input, -but continues until the network side has been closed down. -Specifying -.Fl o -will make it terminate on EOF as well. .It Fl P Ar proxy_username Specifies a username to present to a proxy server that requires authentication. If no username is specified then authentication will not be attempted. Proxy authentication is only supported for HTTP CONNECT proxies at present. .It Fl p Ar source_port Specifies the source port .Nm should use, subject to privilege restrictions and availability. It is an error to use this option in conjunction with the .Fl l option. .It Fl r Specifies that source and/or destination ports should be chosen randomly instead of sequentially within a range or in the order that the system assigns them. .It Fl S Enables the RFC 2385 TCP MD5 signature option. .It Fl s Ar source_ip_address Specifies the IP of the interface which is used to send the packets. It is an error to use this option in conjunction with the .Fl l option. .It Fl T Ar ToS Specifies IP Type of Service (ToS) for the connection. Valid values are the tokens .Dq lowdelay , .Dq throughput , .Dq reliability , or an 8-bit hexadecimal value preceded by .Dq 0x . .It Fl t Causes .Nm to send RFC 854 DON'T and WON'T responses to RFC 854 DO and WILL requests. This makes it possible to use .Nm to script telnet sessions. .It Fl U Specifies to use .Ux Ns -domain sockets. .It Fl u Use UDP instead of the default option of TCP. .It Fl V Ar fib Set the routing table (FIB). The default is 0. .It Fl v Have .Nm give more verbose output. .It Fl w Ar timeout If a connection and stdin are idle for more than .Ar timeout seconds, then the connection is silently closed. The .Fl w flag has no effect on the .Fl l option, i.e.\& .Nm will listen forever for a connection, with or without the .Fl w flag. The default is no timeout. .It Fl X Ar proxy_protocol Requests that .Nm should use the specified protocol when talking to the proxy server. Supported protocols are .Dq 4 (SOCKS v.4), .Dq 5 (SOCKS v.5) and .Dq connect (HTTPS proxy). If the protocol is not specified, SOCKS version 5 is used. .It Xo .Fl x Ar proxy_address Ns Oo : Ns .Ar port Oc .Xc Requests that .Nm should connect to .Ar hostname using a proxy at .Ar proxy_address and .Ar port . If .Ar port is not specified, the well-known port for the proxy protocol is used (1080 for SOCKS, 3128 for HTTPS). .It Fl z Specifies that .Nm should just scan for listening daemons, without sending any data to them. It is an error to use this option in conjunction with the .Fl l option. .El .Pp .Ar hostname can be a numerical IP address or a symbolic hostname (unless the .Fl n option is given). In general, a hostname must be specified, unless the .Fl l option is given (in which case the local host is used). .Pp .Ar port can be a single integer or a range of ports. Ranges are in the form nn-mm. In general, a destination port must be specified, unless the .Fl U option is given (in which case a socket must be specified). .Sh CLIENT/SERVER MODEL It is quite simple to build a very basic client/server model using .Nm . On one console, start .Nm listening on a specific port for a connection. For example: .Pp .Dl $ nc -l 1234 .Pp .Nm is now listening on port 1234 for a connection. On a second console .Pq or a second machine , connect to the machine and port being listened on: .Pp .Dl $ nc 127.0.0.1 1234 .Pp There should now be a connection between the ports. Anything typed at the second console will be concatenated to the first, and vice-versa. After the connection has been set up, .Nm does not really care which side is being used as a .Sq server and which side is being used as a .Sq client . The connection may be terminated using an .Dv EOF .Pq Sq ^D . .Sh DATA TRANSFER The example in the previous section can be expanded to build a basic data transfer model. Any information input into one end of the connection will be output to the other end, and input and output can be easily captured in order to emulate file transfer. .Pp Start by using .Nm to listen on a specific port, with output captured into a file: .Pp .Dl $ nc -l 1234 \*(Gt filename.out .Pp Using a second machine, connect to the listening .Nm process, feeding it the file which is to be transferred: .Pp .Dl $ nc host.example.com 1234 \*(Lt filename.in .Pp After the file has been transferred, the connection will close automatically. .Sh TALKING TO SERVERS It is sometimes useful to talk to servers .Dq by hand rather than through a user interface. It can aid in troubleshooting, when it might be necessary to verify what data a server is sending in response to commands issued by the client. For example, to retrieve the home page of a web site: .Bd -literal -offset indent $ echo -n "GET / HTTP/1.0\er\en\er\en" | nc host.example.com 80 .Ed .Pp Note that this also displays the headers sent by the web server. They can be filtered, using a tool such as .Xr sed 1 , if necessary. .Pp More complicated examples can be built up when the user knows the format of requests required by the server. As another example, an email may be submitted to an SMTP server using: .Bd -literal -offset indent $ nc localhost 25 \*(Lt\*(Lt EOF HELO host.example.com MAIL FROM:\*(Ltuser@host.example.com\*(Gt RCPT TO:\*(Ltuser2@host.example.com\*(Gt DATA Body of email. \&. QUIT EOF .Ed .Sh PORT SCANNING It may be useful to know which ports are open and running services on a target machine. The .Fl z flag can be used to tell .Nm to report open ports, rather than initiate a connection. For example: .Bd -literal -offset indent $ nc -z host.example.com 20-30 Connection to host.example.com 22 port [tcp/ssh] succeeded! Connection to host.example.com 25 port [tcp/smtp] succeeded! .Ed .Pp The port range was specified to limit the search to ports 20 \- 30. .Pp Alternatively, it might be useful to know which server software is running, and which versions. This information is often contained within the greeting banners. In order to retrieve these, it is necessary to first make a connection, and then break the connection when the banner has been retrieved. This can be accomplished by specifying a small timeout with the .Fl w flag, or perhaps by issuing a .Qq Dv QUIT command to the server: .Bd -literal -offset indent $ echo "QUIT" | nc host.example.com 20-30 SSH-1.99-OpenSSH_3.6.1p2 Protocol mismatch. 220 host.example.com IMS SMTP Receiver Version 0.84 Ready .Ed .Sh EXAMPLES Open a TCP connection to port 42 of host.example.com, using port 31337 as the source port, with a timeout of 5 seconds: .Pp .Dl $ nc -p 31337 -w 5 host.example.com 42 .Pp Open a UDP connection to port 53 of host.example.com: .Pp .Dl $ nc -u host.example.com 53 .Pp Open a TCP connection to port 42 of host.example.com using 10.1.2.3 as the IP for the local end of the connection: .Pp .Dl $ nc -s 10.1.2.3 host.example.com 42 .Pp Open a TCP connection to port 42 of host.example.com using IPsec ESP for incoming and outgoing traffic. .Pp .Dl $ nc -E host.example.com 42 .Pp Open a TCP connection to port 42 of host.example.com using IPsec ESP for outgoing traffic only. .Pp .Dl $ nc -e 'out ipsec esp/transport//require' host.example.com 42 .Pp Create and listen on a .Ux Ns -domain socket: .Pp .Dl $ nc -lU /var/tmp/dsocket .Pp Connect to port 42 of host.example.com via an HTTP proxy at 10.2.3.4, port 8080. This example could also be used by .Xr ssh 1 ; see the .Cm ProxyCommand directive in .Xr ssh_config 5 for more information. .Pp .Dl $ nc -x10.2.3.4:8080 -Xconnect host.example.com 42 .Pp The same example again, this time enabling proxy authentication with username .Dq ruser if the proxy requires it: .Pp .Dl $ nc -x10.2.3.4:8080 -Xconnect -Pruser host.example.com 42 .Sh EXIT STATUS .Ex -std .Sh SEE ALSO .Xr cat 1 , .Xr setfib 1 , .Xr ssh 1 , .Xr tcp 4 .Sh AUTHORS Original implementation by *Hobbit* .Aq hobbit@avian.org . .br Rewritten with IPv6 support by .An Eric Jackson Aq ericj@monkey.org . .Sh CAVEATS UDP port scans will always succeed (i.e. report the port as open), rendering the .Fl uz combination of flags relatively useless. Index: user/imp/tbemd/contrib/netcat/netcat.c =================================================================== --- user/imp/tbemd/contrib/netcat/netcat.c (revision 206684) +++ user/imp/tbemd/contrib/netcat/netcat.c (revision 206685) @@ -1,1007 +1,1001 @@ /* $OpenBSD: netcat.c,v 1.95 2010/02/27 00:58:56 nicm Exp $ */ /* * Copyright (c) 2001 Eric Jackson * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. The name of the author may not be used to endorse or promote products * derived from this software without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. * IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. * * $FreeBSD$ */ /* * Re-written nc(1) for OpenBSD. Original implementation by * *Hobbit* . */ #include #include #include #include #include #include #include #include #ifdef IPSEC #include #endif #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include "atomicio.h" #ifndef SUN_LEN #define SUN_LEN(su) \ (sizeof(*(su)) - sizeof((su)->sun_path) + strlen((su)->sun_path)) #endif #define PORT_MAX 65535 #define PORT_MAX_LEN 6 /* Command Line Options */ -int Eflag; /* Use IPsec ESP */ int dflag; /* detached, no stdin */ unsigned int iflag; /* Interval Flag */ int jflag; /* use jumbo frames if we can */ int kflag; /* More than one connect */ int lflag; /* Bind to local port */ int nflag; /* Don't do name look up */ -int oflag; /* Once only: stop on EOF */ int FreeBSD_Oflag; /* Do not use TCP options */ char *Pflag; /* Proxy username */ char *pflag; /* Localport flag */ int rflag; /* Random ports flag */ char *sflag; /* Source Address */ int tflag; /* Telnet Emulation */ int uflag; /* UDP - Default to TCP */ int vflag; /* Verbosity */ int xflag; /* Socks proxy */ int zflag; /* Port Scan Flag */ int Dflag; /* sodebug */ int Iflag; /* TCP receive buffer size */ int Oflag; /* TCP send buffer size */ int Sflag; /* TCP MD5 signature option */ int Tflag = -1; /* IP Type of Service */ u_int rdomain; int timeout = -1; int family = AF_UNSPEC; char *portlist[PORT_MAX+1]; void atelnet(int, unsigned char *, unsigned int); void build_ports(char *); void help(void); int local_listen(char *, char *, struct addrinfo); void readwrite(int); int remote_connect(const char *, const char *, struct addrinfo); int socks_connect(const char *, const char *, struct addrinfo, const char *, const char *, struct addrinfo, int, const char *); int udptest(int); int unix_connect(char *); int unix_listen(char *); void set_common_sockopts(int); int parse_iptos(char *); void usage(int); #ifdef IPSEC void add_ipsec_policy(int, char *); char *ipsec_policy[2]; #endif int main(int argc, char *argv[]) { int ch, s, ret, socksv, ipsec_count; int numfibs; size_t intsize = sizeof(int); char *host, *uport; struct addrinfo hints; struct servent *sv; socklen_t len; struct sockaddr_storage cliaddr; char *proxy; const char *errstr, *proxyhost = "", *proxyport = NULL; struct addrinfo proxyhints; struct option longopts[] = { { "no-tcpopt", no_argument, &FreeBSD_Oflag, 1 }, { NULL, 0, NULL, 0 } }; rdomain = 0; ret = 1; ipsec_count = 0; s = 0; socksv = 5; host = NULL; uport = NULL; sv = NULL; while ((ch = getopt_long(argc, argv, - "46DdEe:hI:i:jklnO:oP:p:rSs:tT:UuV:vw:X:x:z", + "46DdEe:hI:i:jklnoO:P:p:rSs:tT:UuV:vw:X:x:z", longopts, NULL)) != -1) { switch (ch) { case '4': family = AF_INET; break; case '6': family = AF_INET6; break; case 'U': family = AF_UNIX; break; case 'X': if (strcasecmp(optarg, "connect") == 0) socksv = -1; /* HTTP proxy CONNECT */ else if (strcmp(optarg, "4") == 0) socksv = 4; /* SOCKS v.4 */ else if (strcmp(optarg, "5") == 0) socksv = 5; /* SOCKS v.5 */ else errx(1, "unsupported proxy protocol"); break; case 'd': dflag = 1; break; case 'e': #ifdef IPSEC ipsec_policy[ipsec_count++ % 2] = optarg; #else errx(1, "IPsec support unavailable."); #endif break; case 'E': #ifdef IPSEC ipsec_policy[0] = "in ipsec esp/transport//require"; ipsec_policy[1] = "out ipsec esp/transport//require"; #else errx(1, "IPsec support unavailable."); #endif break; case 'h': help(); break; case 'i': iflag = strtonum(optarg, 0, UINT_MAX, &errstr); if (errstr) errx(1, "interval %s: %s", errstr, optarg); break; #ifdef SO_JUMBO case 'j': jflag = 1; break; #endif case 'k': kflag = 1; break; case 'l': lflag = 1; break; case 'n': nflag = 1; break; case 'o': - oflag = 1; + fprintf(stderr, "option -o is deprecated.\n"); break; case 'P': Pflag = optarg; break; case 'p': pflag = optarg; break; case 'r': rflag = 1; break; case 's': sflag = optarg; break; case 't': tflag = 1; break; case 'u': uflag = 1; break; case 'V': if (sysctlbyname("net.fibs", &numfibs, &intsize, NULL, 0) == -1) errx(1, "Multiple FIBS not supported"); rdomain = (unsigned int)strtonum(optarg, 0, numfibs - 1, &errstr); if (errstr) errx(1, "FIB %s: %s", errstr, optarg); break; case 'v': vflag = 1; break; case 'w': timeout = strtonum(optarg, 0, INT_MAX / 1000, &errstr); if (errstr) errx(1, "timeout %s: %s", errstr, optarg); timeout *= 1000; break; case 'x': xflag = 1; if ((proxy = strdup(optarg)) == NULL) err(1, NULL); break; case 'z': zflag = 1; break; case 'D': Dflag = 1; break; case 'I': Iflag = strtonum(optarg, 1, 65536 << 14, &errstr); if (errstr != NULL) errx(1, "TCP receive window %s: %s", errstr, optarg); break; case 'O': Oflag = strtonum(optarg, 1, 65536 << 14, &errstr); if (errstr != NULL) { if (strcmp(errstr, "invalid") != 0) errx(1, "TCP send window %s: %s", errstr, optarg); } break; case 'S': Sflag = 1; break; case 'T': Tflag = parse_iptos(optarg); break; - case 0: - break; default: usage(1); } } argc -= optind; argv += optind; /* Cruft to make sure options are clean, and used properly. */ if (argv[0] && !argv[1] && family == AF_UNIX) { if (uflag) errx(1, "cannot use -u and -U"); host = argv[0]; uport = NULL; } else if (argv[0] && !argv[1]) { if (!lflag) usage(1); uport = argv[0]; host = NULL; } else if (argv[0] && argv[1]) { host = argv[0]; uport = argv[1]; } else usage(1); if (lflag && sflag) errx(1, "cannot use -s and -l"); if (lflag && pflag) errx(1, "cannot use -p and -l"); if (lflag && zflag) errx(1, "cannot use -z and -l"); if (!lflag && kflag) errx(1, "must use -l with -k"); /* Initialize addrinfo structure. */ if (family != AF_UNIX) { memset(&hints, 0, sizeof(struct addrinfo)); hints.ai_family = family; hints.ai_socktype = uflag ? SOCK_DGRAM : SOCK_STREAM; hints.ai_protocol = uflag ? IPPROTO_UDP : IPPROTO_TCP; if (nflag) hints.ai_flags |= AI_NUMERICHOST; } if (xflag) { if (uflag) errx(1, "no proxy support for UDP mode"); if (lflag) errx(1, "no proxy support for listen"); if (family == AF_UNIX) errx(1, "no proxy support for unix sockets"); /* XXX IPv6 transport to proxy would probably work */ if (family == AF_INET6) errx(1, "no proxy support for IPv6"); if (sflag) errx(1, "no proxy support for local source address"); proxyhost = strsep(&proxy, ":"); proxyport = proxy; memset(&proxyhints, 0, sizeof(struct addrinfo)); proxyhints.ai_family = family; proxyhints.ai_socktype = SOCK_STREAM; proxyhints.ai_protocol = IPPROTO_TCP; if (nflag) proxyhints.ai_flags |= AI_NUMERICHOST; } if (lflag) { int connfd; ret = 0; if (family == AF_UNIX) s = unix_listen(host); /* Allow only one connection at a time, but stay alive. */ for (;;) { if (family != AF_UNIX) s = local_listen(host, uport, hints); if (s < 0) err(1, NULL); /* * For UDP, we will use recvfrom() initially * to wait for a caller, then use the regular * functions to talk to the caller. */ if (uflag) { int rv, plen; char buf[8192]; struct sockaddr_storage z; len = sizeof(z); plen = jflag ? 8192 : 1024; rv = recvfrom(s, buf, plen, MSG_PEEK, (struct sockaddr *)&z, &len); if (rv < 0) err(1, "recvfrom"); rv = connect(s, (struct sockaddr *)&z, len); if (rv < 0) err(1, "connect"); connfd = s; } else { len = sizeof(cliaddr); connfd = accept(s, (struct sockaddr *)&cliaddr, &len); } readwrite(connfd); close(connfd); if (family != AF_UNIX) close(s); if (!kflag) break; } } else if (family == AF_UNIX) { ret = 0; if ((s = unix_connect(host)) > 0 && !zflag) { readwrite(s); close(s); } else ret = 1; exit(ret); } else { int i = 0; /* Construct the portlist[] array. */ build_ports(uport); /* Cycle through portlist, connecting to each port. */ for (i = 0; portlist[i] != NULL; i++) { if (s) close(s); if (xflag) s = socks_connect(host, portlist[i], hints, proxyhost, proxyport, proxyhints, socksv, Pflag); else s = remote_connect(host, portlist[i], hints); if (s < 0) continue; ret = 0; if (vflag || zflag) { /* For UDP, make sure we are connected. */ if (uflag) { if (udptest(s) == -1) { ret = 1; continue; } } /* Don't look up port if -n. */ if (nflag) sv = NULL; else { sv = getservbyport( ntohs(atoi(portlist[i])), uflag ? "udp" : "tcp"); } fprintf(stderr, "Connection to %s %s port [%s/%s] " "succeeded!\n", host, portlist[i], uflag ? "udp" : "tcp", sv ? sv->s_name : "*"); } if (!zflag) readwrite(s); } } if (s) close(s); exit(ret); } /* * unix_connect() * Returns a socket connected to a local unix socket. Returns -1 on failure. */ int unix_connect(char *path) { struct sockaddr_un sun; int s; if ((s = socket(AF_UNIX, SOCK_STREAM, 0)) < 0) return (-1); (void)fcntl(s, F_SETFD, 1); memset(&sun, 0, sizeof(struct sockaddr_un)); sun.sun_family = AF_UNIX; if (strlcpy(sun.sun_path, path, sizeof(sun.sun_path)) >= sizeof(sun.sun_path)) { close(s); errno = ENAMETOOLONG; return (-1); } if (connect(s, (struct sockaddr *)&sun, SUN_LEN(&sun)) < 0) { close(s); return (-1); } return (s); } /* * unix_listen() * Create a unix domain socket, and listen on it. */ int unix_listen(char *path) { struct sockaddr_un sun; int s; /* Create unix domain socket. */ if ((s = socket(AF_UNIX, SOCK_STREAM, 0)) < 0) return (-1); memset(&sun, 0, sizeof(struct sockaddr_un)); sun.sun_family = AF_UNIX; if (strlcpy(sun.sun_path, path, sizeof(sun.sun_path)) >= sizeof(sun.sun_path)) { close(s); errno = ENAMETOOLONG; return (-1); } if (bind(s, (struct sockaddr *)&sun, SUN_LEN(&sun)) < 0) { close(s); return (-1); } if (listen(s, 5) < 0) { close(s); return (-1); } return (s); } /* * remote_connect() * Returns a socket connected to a remote host. Properly binds to a local * port or source address if needed. Returns -1 on failure. */ int remote_connect(const char *host, const char *port, struct addrinfo hints) { struct addrinfo *res, *res0; int s, error, on = 1; if ((error = getaddrinfo(host, port, &hints, &res))) errx(1, "getaddrinfo: %s", gai_strerror(error)); res0 = res; do { if ((s = socket(res0->ai_family, res0->ai_socktype, res0->ai_protocol)) < 0) continue; #ifdef IPSEC if (ipsec_policy[0] != NULL) add_ipsec_policy(s, ipsec_policy[0]); if (ipsec_policy[1] != NULL) add_ipsec_policy(s, ipsec_policy[1]); #endif if (rdomain) { if (setfib(rdomain) == -1) err(1, "setfib"); } /* Bind to a local port or source address if specified. */ if (sflag || pflag) { struct addrinfo ahints, *ares; #ifdef SO_BINDANY /* try SO_BINDANY, but don't insist */ setsockopt(s, SOL_SOCKET, SO_BINDANY, &on, sizeof(on)); #endif memset(&ahints, 0, sizeof(struct addrinfo)); ahints.ai_family = res0->ai_family; ahints.ai_socktype = uflag ? SOCK_DGRAM : SOCK_STREAM; ahints.ai_protocol = uflag ? IPPROTO_UDP : IPPROTO_TCP; ahints.ai_flags = AI_PASSIVE; if ((error = getaddrinfo(sflag, pflag, &ahints, &ares))) errx(1, "getaddrinfo: %s", gai_strerror(error)); if (bind(s, (struct sockaddr *)ares->ai_addr, ares->ai_addrlen) < 0) errx(1, "bind failed: %s", strerror(errno)); freeaddrinfo(ares); } set_common_sockopts(s); if (connect(s, res0->ai_addr, res0->ai_addrlen) == 0) break; else if (vflag) warn("connect to %s port %s (%s) failed", host, port, uflag ? "udp" : "tcp"); close(s); s = -1; } while ((res0 = res0->ai_next) != NULL); freeaddrinfo(res); return (s); } /* * local_listen() * Returns a socket listening on a local port, binds to specified source * address. Returns -1 on failure. */ int local_listen(char *host, char *port, struct addrinfo hints) { struct addrinfo *res, *res0; int s, ret, x = 1; int error; /* Allow nodename to be null. */ hints.ai_flags |= AI_PASSIVE; /* * In the case of binding to a wildcard address * default to binding to an ipv4 address. */ if (host == NULL && hints.ai_family == AF_UNSPEC) hints.ai_family = AF_INET; if ((error = getaddrinfo(host, port, &hints, &res))) errx(1, "getaddrinfo: %s", gai_strerror(error)); res0 = res; do { if ((s = socket(res0->ai_family, res0->ai_socktype, res0->ai_protocol)) < 0) continue; if (rdomain) { if (setfib(rdomain) == -1) err(1, "setfib"); } ret = setsockopt(s, SOL_SOCKET, SO_REUSEPORT, &x, sizeof(x)); if (ret == -1) err(1, NULL); #ifdef IPSEC if (ipsec_policy[0] != NULL) add_ipsec_policy(s, ipsec_policy[0]); if (ipsec_policy[1] != NULL) add_ipsec_policy(s, ipsec_policy[1]); #endif if (FreeBSD_Oflag) { if (setsockopt(s, IPPROTO_TCP, TCP_NOOPT, &FreeBSD_Oflag, sizeof(FreeBSD_Oflag)) == -1) err(1, "disable TCP options"); } if (bind(s, (struct sockaddr *)res0->ai_addr, res0->ai_addrlen) == 0) break; close(s); s = -1; } while ((res0 = res0->ai_next) != NULL); if (!uflag && s != -1) { if (listen(s, 1) < 0) err(1, "listen"); } freeaddrinfo(res); return (s); } /* * readwrite() * Loop that polls on the network file descriptor and stdin. */ void readwrite(int nfd) { struct pollfd pfd[2]; unsigned char buf[8192]; int n, wfd = fileno(stdin); int lfd = fileno(stdout); int plen; plen = jflag ? 8192 : 1024; /* Setup Network FD */ pfd[0].fd = nfd; pfd[0].events = POLLIN; /* Set up STDIN FD. */ pfd[1].fd = wfd; pfd[1].events = POLLIN; while (pfd[0].fd != -1) { if (iflag) sleep(iflag); if ((n = poll(pfd, 2 - dflag, timeout)) < 0) { close(nfd); err(1, "Polling Error"); } if (n == 0) return; if (pfd[0].revents & POLLIN) { if ((n = read(nfd, buf, plen)) < 0) return; else if (n == 0) { shutdown(nfd, SHUT_RD); pfd[0].fd = -1; pfd[0].events = 0; } else { if (tflag) atelnet(nfd, buf, n); if (atomicio(vwrite, lfd, buf, n) != n) return; } } if (!dflag && pfd[1].revents & POLLIN) { - if ((n = read(wfd, buf, plen)) < 0 || - (oflag && n == 0)) { + if ((n = read(wfd, buf, plen)) < 0) return; - } else if (n == 0) { + else if (n == 0) { shutdown(nfd, SHUT_WR); pfd[1].fd = -1; pfd[1].events = 0; } else { if (atomicio(vwrite, nfd, buf, n) != n) return; } } } } /* Deal with RFC 854 WILL/WONT DO/DONT negotiation. */ void atelnet(int nfd, unsigned char *buf, unsigned int size) { unsigned char *p, *end; unsigned char obuf[4]; if (size < 3) return; end = buf + size - 2; for (p = buf; p < end; p++) { if (*p != IAC) continue; obuf[0] = IAC; p++; if ((*p == WILL) || (*p == WONT)) obuf[1] = DONT; else if ((*p == DO) || (*p == DONT)) obuf[1] = WONT; else continue; p++; obuf[2] = *p; if (atomicio(vwrite, nfd, obuf, 3) != 3) warn("Write Error!"); } } /* * build_ports() * Build an array or ports in portlist[], listing each port * that we should try to connect to. */ void build_ports(char *p) { const char *errstr; char *n; int hi, lo, cp; int x = 0; if ((n = strchr(p, '-')) != NULL) { if (lflag) errx(1, "Cannot use -l with multiple ports!"); *n = '\0'; n++; /* Make sure the ports are in order: lowest->highest. */ hi = strtonum(n, 1, PORT_MAX, &errstr); if (errstr) errx(1, "port number %s: %s", errstr, n); lo = strtonum(p, 1, PORT_MAX, &errstr); if (errstr) errx(1, "port number %s: %s", errstr, p); if (lo > hi) { cp = hi; hi = lo; lo = cp; } /* Load ports sequentially. */ for (cp = lo; cp <= hi; cp++) { portlist[x] = calloc(1, PORT_MAX_LEN); if (portlist[x] == NULL) err(1, NULL); snprintf(portlist[x], PORT_MAX_LEN, "%d", cp); x++; } /* Randomly swap ports. */ if (rflag) { int y; char *c; for (x = 0; x <= (hi - lo); x++) { y = (arc4random() & 0xFFFF) % (hi - lo); c = portlist[x]; portlist[x] = portlist[y]; portlist[y] = c; } } } else { hi = strtonum(p, 1, PORT_MAX, &errstr); if (errstr) errx(1, "port number %s: %s", errstr, p); portlist[0] = calloc(1, PORT_MAX_LEN); if (portlist[0] == NULL) err(1, NULL); portlist[0] = p; } } /* * udptest() * Do a few writes to see if the UDP port is there. * XXX - Better way of doing this? Doesn't work for IPv6. * Also fails after around 100 ports checked. */ int udptest(int s) { int i, ret; for (i = 0; i <= 3; i++) { if (write(s, "X", 1) == 1) ret = 1; else ret = -1; } return (ret); } void set_common_sockopts(int s) { int x = 1; if (Sflag) { if (setsockopt(s, IPPROTO_TCP, TCP_MD5SIG, &x, sizeof(x)) == -1) err(1, NULL); } if (Dflag) { if (setsockopt(s, SOL_SOCKET, SO_DEBUG, &x, sizeof(x)) == -1) err(1, NULL); } #ifdef SO_JUMBO if (jflag) { if (setsockopt(s, SOL_SOCKET, SO_JUMBO, &x, sizeof(x)) == -1) err(1, NULL); } #endif if (Tflag != -1) { if (setsockopt(s, IPPROTO_IP, IP_TOS, &Tflag, sizeof(Tflag)) == -1) err(1, "set IP ToS"); } if (Iflag) { if (setsockopt(s, SOL_SOCKET, SO_RCVBUF, &Iflag, sizeof(Iflag)) == -1) err(1, "set TCP receive buffer size"); } if (Oflag) { if (setsockopt(s, SOL_SOCKET, SO_SNDBUF, &Oflag, sizeof(Oflag)) == -1) err(1, "set TCP send buffer size"); } if (FreeBSD_Oflag) { if (setsockopt(s, IPPROTO_TCP, TCP_NOOPT, &FreeBSD_Oflag, sizeof(FreeBSD_Oflag)) == -1) err(1, "disable TCP options"); } } int parse_iptos(char *s) { int tos = -1; if (strcmp(s, "lowdelay") == 0) return (IPTOS_LOWDELAY); if (strcmp(s, "throughput") == 0) return (IPTOS_THROUGHPUT); if (strcmp(s, "reliability") == 0) return (IPTOS_RELIABILITY); if (sscanf(s, "0x%x", &tos) != 1 || tos < 0 || tos > 0xff) errx(1, "invalid IP Type of Service"); return (tos); } void help(void) { usage(0); fprintf(stderr, "\tCommand Summary:\n\ \t-4 Use IPv4\n\ \t-6 Use IPv6\n\ \t-D Enable the debug socket option\n\ \t-d Detach from stdin\n"); #ifdef IPSEC fprintf(stderr, "\ \t-E Use IPsec ESP\n\ \t-e policy Use specified IPsec policy\n"); #endif fprintf(stderr, "\ \t-h This help text\n\ \t-I length TCP receive buffer length\n\ \t-i secs\t Delay interval for lines sent, ports scanned\n\ \t-k Keep inbound sockets open for multiple connects\n\ \t-l Listen mode, for inbound connects\n\ \t-n Suppress name/port resolutions\n\ \t--no-tcpopt Disable TCP options\n\ \t-O length TCP send buffer length\n\ - \t-o Terminate on EOF on input\n\ \t-P proxyuser\tUsername for proxy authentication\n\ \t-p port\t Specify local port for remote connects\n\ \t-r Randomize remote ports\n\ \t-S Enable the TCP MD5 signature option\n\ \t-s addr\t Local source address\n\ \t-T ToS\t Set IP Type of Service\n\ \t-t Answer TELNET negotiation\n\ \t-U Use UNIX domain socket\n\ \t-u UDP mode\n\ \t-V fib Specify alternate routing table (FIB)\n\ \t-v Verbose\n\ \t-w secs\t Timeout for connects and final net reads\n\ \t-X proto Proxy protocol: \"4\", \"5\" (SOCKS) or \"connect\"\n\ \t-x addr[:port]\tSpecify proxy address and port\n\ \t-z Zero-I/O mode [used for scanning]\n\ Port numbers can be individual or ranges: lo-hi [inclusive]\n"); #ifdef IPSEC fprintf(stderr, "See ipsec_set_policy(3) for -e argument format\n"); #endif exit(1); } #ifdef IPSEC void add_ipsec_policy(int s, char *policy) { char *raw; int e; raw = ipsec_set_policy(policy, strlen(policy)); if (raw == NULL) errx(1, "ipsec_set_policy `%s': %s", policy, ipsec_strerror()); e = setsockopt(s, IPPROTO_IP, IP_IPSEC_POLICY, raw, ipsec_get_policylen(raw)); if (e < 0) err(1, "ipsec policy cannot be configured"); free(raw); if (vflag) fprintf(stderr, "ipsec policy configured: `%s'\n", policy); return; } #endif /* IPSEC */ void usage(int ret) { fprintf(stderr, #ifdef IPSEC - "usage: nc [-46DdEhklnorStUuvz] [-e policy] [-I length] [-i interval] [-O length]\n" + "usage: nc [-46DdEhklnrStUuvz] [-e policy] [-I length] [-i interval] [-O length]\n" #else - "usage: nc [-46DdhklnorStUuvz] [-I length] [-i interval] [-O length]\n" + "usage: nc [-46DdhklnrStUuvz] [-I length] [-i interval] [-O length]\n" #endif "\t [-P proxy_username] [-p source_port] [-s source_ip_address] [-T ToS]\n" "\t [-V fib] [-w timeout] [-X proxy_protocol]\n" "\t [-x proxy_address[:port]] [hostname] [port]\n"); if (ret) exit(1); } Index: user/imp/tbemd/contrib/tzcode/stdtime =================================================================== --- user/imp/tbemd/contrib/tzcode/stdtime (revision 206684) +++ user/imp/tbemd/contrib/tzcode/stdtime (revision 206685) Property changes on: user/imp/tbemd/contrib/tzcode/stdtime ___________________________________________________________________ Modified: svn:mergeinfo ## -0,0 +0,1 ## Merged /head/contrib/tzcode/stdtime:r206628-206684 Index: user/imp/tbemd/contrib/tzcode/zic =================================================================== --- user/imp/tbemd/contrib/tzcode/zic (revision 206684) +++ user/imp/tbemd/contrib/tzcode/zic (revision 206685) Property changes on: user/imp/tbemd/contrib/tzcode/zic ___________________________________________________________________ Modified: svn:mergeinfo ## -0,0 +0,1 ## Merged /head/contrib/tzcode/zic:r206628-206684 Index: user/imp/tbemd/contrib/tzdata =================================================================== --- user/imp/tbemd/contrib/tzdata (revision 206684) +++ user/imp/tbemd/contrib/tzdata (revision 206685) Property changes on: user/imp/tbemd/contrib/tzdata ___________________________________________________________________ Modified: svn:mergeinfo ## -0,0 +0,1 ## Merged /head/contrib/tzdata:r206628-206684 Index: user/imp/tbemd/etc/mtree/BSD.include.dist =================================================================== --- user/imp/tbemd/etc/mtree/BSD.include.dist (revision 206684) +++ user/imp/tbemd/etc/mtree/BSD.include.dist (revision 206685) @@ -1,298 +1,298 @@ # $FreeBSD$ # # Please see the file src/etc/mtree/README before making changes to this file. # /set type=dir uname=root gname=wheel mode=0755 . altq .. arpa .. bsm .. bsnmp .. c++ 4.2 backward .. bits .. debug .. ext pb_ds detail basic_tree_policy .. bin_search_tree_ .. binary_heap_ .. binomial_heap_ .. binomial_heap_base_ .. cc_hash_table_map_ .. eq_fn .. gp_hash_table_map_ .. hash_fn .. left_child_next_sibling_heap_ .. list_update_map_ .. list_update_policy .. ov_tree_map_ .. pairing_heap_ .. pat_trie_ .. rb_tree_map_ .. rc_binomial_heap_ .. resize_policy .. splay_tree_ .. thin_heap_ .. tree_policy .. trie_policy .. unordered_iterator .. .. .. .. tr1 .. .. .. cam ata .. scsi .. .. crypto .. dev acpica .. an .. bktr .. firewire .. hwpmc .. ic .. ieee488 .. iicbus .. lmc .. - mfi - .. + mfi + .. mpt mpilib .. .. ofw .. pbio .. powermac_nvram .. ppbus .. smbus .. speaker .. usb .. utopia .. vkbd .. wi .. .. fs devfs .. fdescfs .. fifofs .. msdosfs .. nfs .. ntfs .. nullfs .. nwfs .. portalfs .. procfs .. smbfs .. udf .. unionfs .. .. geom cache .. concat .. eli .. gate .. journal .. label .. mirror .. mountver .. multipath .. nop .. raid3 .. shsec .. stripe .. virstor .. .. gnu posix .. .. gpib .. gssapi .. isofs cd9660 .. .. kadm5 .. libmilter .. lwres .. machine pc .. .. net .. net80211 .. netatalk .. netgraph atm .. bluetooth include .. .. netflow .. .. netinet .. netinet6 .. netipsec .. netipx .. netnatm api .. msg .. saal .. sig .. .. netncp .. netsmb .. nfs .. nfsclient .. nfsserver .. objc .. openssl .. pcap .. protocols .. readline .. rpc .. rpcsvc .. security audit .. mac_biba .. mac_bsdextended .. mac_lomac .. mac_mls .. mac_partition .. .. ssp .. sys .. ufs ffs .. ufs .. .. vm .. .. Index: user/imp/tbemd/lib/libc/sys/sigaction.2 =================================================================== --- user/imp/tbemd/lib/libc/sys/sigaction.2 (revision 206684) +++ user/imp/tbemd/lib/libc/sys/sigaction.2 (revision 206685) @@ -1,665 +1,675 @@ .\" Copyright (c) 1980, 1990, 1993 .\" The Regents of the University of California. All rights reserved. .\" .\" Redistribution and use in source and binary forms, with or without .\" modification, are permitted provided that the following conditions .\" are met: .\" 1. Redistributions of source code must retain the above copyright .\" notice, this list of conditions and the following disclaimer. .\" 2. Redistributions in binary form must reproduce the above copyright .\" notice, this list of conditions and the following disclaimer in the .\" documentation and/or other materials provided with the distribution. .\" 4. Neither the name of the University nor the names of its contributors .\" may be used to endorse or promote products derived from this software .\" without specific prior written permission. .\" .\" THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND .\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE .\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE .\" ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE .\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL .\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS .\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) .\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT .\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY .\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF .\" SUCH DAMAGE. .\" .\" From: @(#)sigaction.2 8.2 (Berkeley) 4/3/94 .\" $FreeBSD$ .\" .Dd April 13, 2010 .Dt SIGACTION 2 .Os .Sh NAME .Nm sigaction .Nd software signal facilities .Sh LIBRARY .Lb libc .Sh SYNOPSIS .In signal.h .Bd -literal struct sigaction { union { void (*__sa_handler)(int); void (*__sa_sigaction)(int, siginfo_t *, void *); } __sigaction_u; /* signal handler */ int sa_flags; /* see signal options below */ sigset_t sa_mask; /* signal mask to apply */ }; #define sa_handler __sigaction_u.__sa_handler #define sa_sigaction __sigaction_u.__sa_sigaction .Ed .Ft int .Fo sigaction .Fa "int sig" .Fa "const struct sigaction * restrict act" .Fa "struct sigaction * restrict oact" .Fc .Sh DESCRIPTION The system defines a set of signals that may be delivered to a process. Signal delivery resembles the occurrence of a hardware interrupt: the signal is normally blocked from further occurrence, the current process context is saved, and a new one is built. A process may specify a .Em handler to which a signal is delivered, or specify that a signal is to be .Em ignored . A process may also specify that a default action is to be taken by the system when a signal occurs. A signal may also be .Em blocked , in which case its delivery is postponed until it is .Em unblocked . The action to be taken on delivery is determined at the time of delivery. Normally, signal handlers execute on the current stack of the process. This may be changed, on a per-handler basis, so that signals are taken on a special .Em "signal stack" . .Pp Signal routines normally execute with the signal that caused their invocation .Em blocked , but other signals may yet occur. A global .Em "signal mask" defines the set of signals currently blocked from delivery to a process. The signal mask for a process is initialized from that of its parent (normally empty). It may be changed with a .Xr sigprocmask 2 call, or when a signal is delivered to the process. .Pp When a signal condition arises for a process, the signal is added to a set of signals pending for the process. If the signal is not currently .Em blocked by the process then it is delivered to the process. Signals may be delivered any time a process enters the operating system (e.g., during a system call, page fault or trap, or clock interrupt). If multiple signals are ready to be delivered at the same time, any signals that could be caused by traps are delivered first. Additional signals may be processed at the same time, with each appearing to interrupt the handlers for the previous signals before their first instructions. The set of pending signals is returned by the .Xr sigpending 2 system call. When a caught signal is delivered, the current state of the process is saved, a new signal mask is calculated (as described below), and the signal handler is invoked. The call to the handler is arranged so that if the signal handling routine returns normally the process will resume execution in the context from before the signal's delivery. If the process wishes to resume in a different context, then it must arrange to restore the previous context itself. .Pp When a signal is delivered to a process a new signal mask is installed for the duration of the process' signal handler (or until a .Xr sigprocmask 2 system call is made). This mask is formed by taking the union of the current signal mask set, the signal to be delivered, and the signal mask associated with the handler to be invoked. .Pp The .Fn sigaction system call assigns an action for a signal specified by .Fa sig . If .Fa act is non-zero, it specifies an action .Dv ( SIG_DFL , .Dv SIG_IGN , or a handler routine) and mask to be used when delivering the specified signal. If .Fa oact is non-zero, the previous handling information for the signal is returned to the user. .Pp Once a signal handler is installed, it normally remains installed until another .Fn sigaction system call is made, or an .Xr execve 2 is performed. A signal-specific default action may be reset by setting .Va sa_handler to .Dv SIG_DFL . The defaults are process termination, possibly with core dump; no action; stopping the process; or continuing the process. See the signal list below for each signal's default action. If .Va sa_handler is .Dv SIG_DFL , the default action for the signal is to discard the signal, and if a signal is pending, the pending signal is discarded even if the signal is masked. If .Va sa_handler is set to .Dv SIG_IGN current and pending instances of the signal are ignored and discarded. .Pp Options may be specified by setting .Va sa_flags . The meaning of the various bits is as follows: .Bl -tag -offset indent -width SA_RESETHANDXX .It Dv SA_NOCLDSTOP If this bit is set when installing a catching function for the .Dv SIGCHLD signal, the .Dv SIGCHLD signal will be generated only when a child process exits, not when a child process stops. .It Dv SA_NOCLDWAIT If this bit is set when calling .Fn sigaction for the .Dv SIGCHLD signal, the system will not create zombie processes when children of the calling process exit. If the calling process subsequently issues a .Xr wait 2 (or equivalent), it blocks until all of the calling process's child processes terminate, and then returns a value of \-1 with .Va errno set to .Er ECHILD . The same effect of avoiding zombie creation can also be achieved by setting .Va sa_handler for .Dv SIGCHLD to .Dv SIG_IGN . .It Dv SA_ONSTACK If this bit is set, the system will deliver the signal to the process on a .Em "signal stack" , specified with .Xr sigaltstack 2 . .It Dv SA_NODEFER If this bit is set, further occurrences of the delivered signal are not masked during the execution of the handler. .It Dv SA_RESETHAND If this bit is set, the handler is reset back to .Dv SIG_DFL at the moment the signal is delivered. .It Dv SA_RESTART See paragraph below. .It Dv SA_SIGINFO If this bit is set, the handler function is assumed to be pointed to by the .Va sa_sigaction member of .Vt "struct sigaction" and should match the prototype shown above or as below in .Sx EXAMPLES . This bit should not be set when assigning .Dv SIG_DFL or .Dv SIG_IGN . .El .Pp If a signal is caught during the system calls listed below, the call may be forced to terminate with the error .Er EINTR , the call may return with a data transfer shorter than requested, or the call may be restarted. Restart of pending calls is requested by setting the .Dv SA_RESTART bit in .Va sa_flags . The affected system calls include .Xr open 2 , .Xr read 2 , .Xr write 2 , .Xr sendto 2 , .Xr recvfrom 2 , .Xr sendmsg 2 and .Xr recvmsg 2 on a communications channel or a slow device (such as a terminal, but not a regular file) and during a .Xr wait 2 or .Xr ioctl 2 . However, calls that have already committed are not restarted, but instead return a partial success (for example, a short read count). .Pp After a .Xr fork 2 or .Xr vfork 2 all signals, the signal mask, the signal stack, and the restart/interrupt flags are inherited by the child. .Pp The .Xr execve 2 system call reinstates the default action for all signals which were caught and resets all signals to be caught on the user stack. Ignored signals remain ignored; the signal mask remains the same; signals that restart pending system calls continue to do so. .Pp The following is a list of all signals with names as in the include file .In signal.h : .Bl -column SIGVTALARMXX "create core imagexxx" .It Sy "NAME Default Action Description" .It Dv SIGHUP No " terminate process" " terminal line hangup" .It Dv SIGINT No " terminate process" " interrupt program" .It Dv SIGQUIT No " create core image" " quit program" .It Dv SIGILL No " create core image" " illegal instruction" .It Dv SIGTRAP No " create core image" " trace trap" .It Dv SIGABRT No " create core image" Ta Xr abort 3 call (formerly .Dv SIGIOT ) .It Dv SIGEMT No " create core image" " emulate instruction executed" .It Dv SIGFPE No " create core image" " floating-point exception" .It Dv SIGKILL No " terminate process" " kill program" .It Dv SIGBUS No " create core image" " bus error" .It Dv SIGSEGV No " create core image" " segmentation violation" .It Dv SIGSYS No " create core image" " non-existent system call invoked" .It Dv SIGPIPE No " terminate process" " write on a pipe with no reader" .It Dv SIGALRM No " terminate process" " real-time timer expired" .It Dv SIGTERM No " terminate process" " software termination signal" .It Dv SIGURG No " discard signal" " urgent condition present on socket" .It Dv SIGSTOP No " stop process" " stop (cannot be caught or ignored)" .It Dv SIGTSTP No " stop process" " stop signal generated from keyboard" .It Dv SIGCONT No " discard signal" " continue after stop" .It Dv SIGCHLD No " discard signal" " child status has changed" .It Dv SIGTTIN No " stop process" " background read attempted from control terminal" .It Dv SIGTTOU No " stop process" " background write attempted to control terminal" .It Dv SIGIO No " discard signal" Tn " I/O" is possible on a descriptor (see .Xr fcntl 2 ) .It Dv SIGXCPU No " terminate process" " cpu time limit exceeded (see" .Xr setrlimit 2 ) .It Dv SIGXFSZ No " terminate process" " file size limit exceeded (see" .Xr setrlimit 2 ) .It Dv SIGVTALRM No " terminate process" " virtual time alarm (see" .Xr setitimer 2 ) .It Dv SIGPROF No " terminate process" " profiling timer alarm (see" .Xr setitimer 2 ) .It Dv SIGWINCH No " discard signal" " Window size change" .It Dv SIGINFO No " discard signal" " status request from keyboard" .It Dv SIGUSR1 No " terminate process" " User defined signal 1" .It Dv SIGUSR2 No " terminate process" " User defined signal 2" .El .Sh NOTE The .Va sa_mask field specified in .Fa act is not allowed to block .Dv SIGKILL or .Dv SIGSTOP . Any attempt to do so will be silently ignored. .Pp The following functions are either reentrant or not interruptible by signals and are async-signal safe. Therefore applications may invoke them, without restriction, from signal-catching functions: .Pp Base Interfaces: .Pp .Fn _exit , .Fn access , .Fn alarm , .Fn cfgetispeed , .Fn cfgetospeed , .Fn cfsetispeed , .Fn cfsetospeed , .Fn chdir , .Fn chmod , .Fn chown , .Fn close , .Fn creat , .Fn dup , .Fn dup2 , .Fn execle , .Fn execve , .Fn fcntl , .Fn fork , .Fn fpathconf , .Fn fstat , .Fn fsync , .Fn getegid , .Fn geteuid , .Fn getgid , .Fn getgroups , .Fn getpgrp , .Fn getpid , .Fn getppid , .Fn getuid , .Fn kill , .Fn link , .Fn lseek , .Fn mkdir , .Fn mkfifo , .Fn open , .Fn pathconf , .Fn pause , .Fn pipe , .Fn raise , .Fn read , .Fn rename , .Fn rmdir , .Fn setgid , .Fn setpgid , .Fn setsid , .Fn setuid , .Fn sigaction , .Fn sigaddset , .Fn sigdelset , .Fn sigemptyset , .Fn sigfillset , .Fn sigismember , .Fn signal , .Fn sigpending , .Fn sigprocmask , .Fn sigsuspend , .Fn sleep , .Fn stat , .Fn sysconf , .Fn tcdrain , .Fn tcflow , .Fn tcflush , .Fn tcgetattr , .Fn tcgetpgrp , .Fn tcsendbreak , .Fn tcsetattr , .Fn tcsetpgrp , .Fn time , .Fn times , .Fn umask , .Fn uname , .Fn unlink , .Fn utime , .Fn wait , .Fn waitpid , .Fn write . .Pp Realtime Interfaces: .Pp .Fn aio_error , .Fn clock_gettime , .Fn sigpause , .Fn timer_getoverrun , .Fn aio_return , .Fn fdatasync , .Fn sigqueue , .Fn timer_gettime , .Fn aio_suspend , .Fn sem_post , .Fn sigset , .Fn timer_settime . .Pp .Tn ANSI C Interfaces: .Pp .Fn strcpy , .Fn strcat , .Fn strncpy , .Fn strncat , and perhaps some others. .Pp Extension Interfaces: .Pp .Fn strlcpy , .Fn strlcat . .Pp All functions not in the above lists are considered to be unsafe with respect to signals. That is to say, the behaviour of such functions when called from a signal handler is undefined. In general though, signal handlers should do little more than set a flag; most other actions are not safe. .Pp Also, it is good practice to make a copy of the global variable .Va errno and restore it before returning from the signal handler. This protects against the side effect of .Va errno being set by functions called from inside the signal handler. .Sh RETURN VALUES .Rv -std sigaction .Sh EXAMPLES There are three possible prototypes the handler may match: .Bl -tag -offset indent -width short .It Tn ANSI C : .Ft void .Fn handler int ; .It Traditional BSD style: .Ft void .Fn handler int "int code" "struct sigcontext *scp" ; .It Tn POSIX Dv SA_SIGINFO : .Ft void .Fn handler int "siginfo_t *info" "ucontext_t *uap" ; .El .Pp The handler function should match the .Dv SA_SIGINFO prototype if the .Dv SA_SIGINFO bit is set in .Va sa_flags . It then should be pointed to by the .Va sa_sigaction member of .Vt "struct sigaction" . Note that you should not assign .Dv SIG_DFL or .Dv SIG_IGN this way. .Pp +If preprocessing symbol +.Va _POSIX_C_SOURCE +with the value >= 199309 is not defined, the following declaration for +the handler shall be used: +.Bl -tag -offset indent -width short +.It Tn POSIX Dv SA_SIGINFO : +.Ft void +.Fn handler int "struct __sigaction *" "void *" ; +.El +.Pp If the .Dv SA_SIGINFO flag is not set, the handler function should match either the .Tn ANSI C or traditional .Bx prototype and be pointed to by the .Va sa_handler member of .Vt "struct sigaction" . In practice, .Fx always sends the three arguments of the latter and since the .Tn ANSI C prototype is a subset, both will work. The .Va sa_handler member declaration in .Fx include files is that of .Tn ANSI C (as required by .Tn POSIX ) , so a function pointer of a .Bx Ns -style function needs to be casted to compile without warning. The traditional .Bx style is not portable and since its capabilities are a full subset of a .Dv SA_SIGINFO handler, its use is deprecated. .Pp The .Fa sig argument is the signal number, one of the .Dv SIG... values from .In signal.h . .Pp The .Fa code argument of the .Bx Ns -style handler and the .Va si_code member of the .Fa info argument to a .Dv SA_SIGINFO handler contain a numeric code explaining the cause of the signal, usually one of the .Dv SI_... values from .In sys/signal.h or codes specific to a signal, i.e., one of the .Dv FPE_... values for .Dv SIGFPE . .Pp The .Fa scp argument to a .Bx Ns -style handler points to an instance of .Vt "struct sigcontext" . .Pp The .Fa uap argument to a .Tn POSIX .Dv SA_SIGINFO handler points to an instance of ucontext_t. .Sh ERRORS The .Fn sigaction system call will fail and no new signal handler will be installed if one of the following occurs: .Bl -tag -width Er .It Bq Er EFAULT Either .Fa act or .Fa oact points to memory that is not a valid part of the process address space. .It Bq Er EINVAL The .Fa sig argument is not a valid signal number. .It Bq Er EINVAL An attempt is made to ignore or supply a handler for .Dv SIGKILL or .Dv SIGSTOP . .El .Sh SEE ALSO .Xr kill 1 , .Xr kill 2 , .Xr ptrace 2 , .Xr sigaltstack 2 , .Xr sigblock 2 , .Xr sigpause 2 , .Xr sigpending 2 , .Xr sigprocmask 2 , .Xr sigsetmask 2 , .Xr sigsuspend 2 , .Xr sigvec 2 , .Xr wait 2 , .Xr fpsetmask 3 , .Xr setjmp 3 , .Xr siginfo 3 , .Xr siginterrupt 3 , .Xr sigsetops 3 , .Xr ucontext 3 , .Xr tty 4 .Sh STANDARDS The .Fn sigaction system call is expected to conform to .St -p1003.1-90 . The .Dv SA_ONSTACK and .Dv SA_RESTART flags are Berkeley extensions, as are the signals, .Dv SIGTRAP , .Dv SIGEMT , .Dv SIGBUS , .Dv SIGSYS , .Dv SIGURG , .Dv SIGIO , .Dv SIGXCPU , .Dv SIGXFSZ , .Dv SIGVTALRM , .Dv SIGPROF , .Dv SIGWINCH , and .Dv SIGINFO . Those signals are available on most .Bx Ns \-derived systems. The .Dv SA_NODEFER and .Dv SA_RESETHAND flags are intended for backwards compatibility with other operating systems. The .Dv SA_NOCLDSTOP , and .Dv SA_NOCLDWAIT .\" and .\" SA_SIGINFO flags are featuring options commonly found in other operating systems. The flags are approved by .St -susv2 , along with the option to avoid zombie creation by ignoring .Dv SIGCHLD . Index: user/imp/tbemd/lib/libz =================================================================== --- user/imp/tbemd/lib/libz (revision 206684) +++ user/imp/tbemd/lib/libz (revision 206685) Property changes on: user/imp/tbemd/lib/libz ___________________________________________________________________ Modified: svn:mergeinfo ## -0,0 +0,1 ## Merged /head/lib/libz:r206628-206684 Index: user/imp/tbemd/sbin/geom/misc/subr.c =================================================================== --- user/imp/tbemd/sbin/geom/misc/subr.c (revision 206684) +++ user/imp/tbemd/sbin/geom/misc/subr.c (revision 206685) @@ -1,460 +1,462 @@ /*- * Copyright (c) 2004 Pawel Jakub Dawidek * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHORS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #include __FBSDID("$FreeBSD$"); #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include "misc/subr.h" struct std_metadata { char md_magic[16]; uint32_t md_version; }; static void std_metadata_decode(const u_char *data, struct std_metadata *md) { bcopy(data, md->md_magic, sizeof(md->md_magic)); md->md_version = le32dec(data + 16); } static void pathgen(const char *name, char *path, size_t size) { if (strncmp(name, _PATH_DEV, strlen(_PATH_DEV)) != 0) snprintf(path, size, "%s%s", _PATH_DEV, name); else strlcpy(path, name, size); } /* * Greatest Common Divisor. */ static unsigned gcd(unsigned a, unsigned b) { u_int c; while (b != 0) { c = a; a = b; b = (c % b); } return (a); } /* * Least Common Multiple. */ unsigned g_lcm(unsigned a, unsigned b) { return ((a * b) / gcd(a, b)); } uint32_t bitcount32(uint32_t x) { x = (x & 0x55555555) + ((x & 0xaaaaaaaa) >> 1); x = (x & 0x33333333) + ((x & 0xcccccccc) >> 2); x = (x & 0x0f0f0f0f) + ((x & 0xf0f0f0f0) >> 4); x = (x & 0x00ff00ff) + ((x & 0xff00ff00) >> 8); x = (x & 0x0000ffff) + ((x & 0xffff0000) >> 16); return (x); } off_t g_get_mediasize(const char *name) { char path[MAXPATHLEN]; off_t mediasize; int fd; pathgen(name, path, sizeof(path)); fd = open(path, O_RDONLY); if (fd == -1) return (0); if (ioctl(fd, DIOCGMEDIASIZE, &mediasize) < 0) { close(fd); return (0); } close(fd); return (mediasize); } unsigned g_get_sectorsize(const char *name) { char path[MAXPATHLEN]; unsigned sectorsize; int fd; pathgen(name, path, sizeof(path)); fd = open(path, O_RDONLY); if (fd == -1) return (0); if (ioctl(fd, DIOCGSECTORSIZE, §orsize) < 0) { close(fd); return (0); } close(fd); return (sectorsize); } int g_metadata_read(const char *name, u_char *md, size_t size, const char *magic) { struct std_metadata stdmd; char path[MAXPATHLEN]; unsigned sectorsize; off_t mediasize; u_char *sector; int error, fd; pathgen(name, path, sizeof(path)); sector = NULL; error = 0; fd = open(path, O_RDONLY); if (fd == -1) return (errno); mediasize = g_get_mediasize(name); if (mediasize == 0) { error = errno; goto out; } sectorsize = g_get_sectorsize(name); if (sectorsize == 0) { error = errno; goto out; } assert(sectorsize >= size); sector = malloc(sectorsize); if (sector == NULL) { error = ENOMEM; goto out; } if (pread(fd, sector, sectorsize, mediasize - sectorsize) != (ssize_t)sectorsize) { error = errno; goto out; } if (magic != NULL) { std_metadata_decode(sector, &stdmd); if (strcmp(stdmd.md_magic, magic) != 0) { error = EINVAL; goto out; } } bcopy(sector, md, size); out: if (sector != NULL) free(sector); close(fd); return (error); } int g_metadata_store(const char *name, u_char *md, size_t size) { char path[MAXPATHLEN]; unsigned sectorsize; off_t mediasize; u_char *sector; int error, fd; pathgen(name, path, sizeof(path)); sector = NULL; error = 0; fd = open(path, O_WRONLY); if (fd == -1) return (errno); mediasize = g_get_mediasize(name); if (mediasize == 0) { error = errno; goto out; } sectorsize = g_get_sectorsize(name); if (sectorsize == 0) { error = errno; goto out; } assert(sectorsize >= size); sector = malloc(sectorsize); if (sector == NULL) { error = ENOMEM; goto out; } bcopy(md, sector, size); if (pwrite(fd, sector, sectorsize, mediasize - sectorsize) != (ssize_t)sectorsize) { error = errno; goto out; } + (void)ioctl(fd, DIOCGFLUSH, NULL); out: if (sector != NULL) free(sector); close(fd); return (error); } int g_metadata_clear(const char *name, const char *magic) { struct std_metadata md; char path[MAXPATHLEN]; unsigned sectorsize; off_t mediasize; u_char *sector; int error, fd; pathgen(name, path, sizeof(path)); sector = NULL; error = 0; fd = open(path, O_RDWR); if (fd == -1) return (errno); mediasize = g_get_mediasize(name); if (mediasize == 0) { error = errno; goto out; } sectorsize = g_get_sectorsize(name); if (sectorsize == 0) { error = errno; goto out; } sector = malloc(sectorsize); if (sector == NULL) { error = ENOMEM; goto out; } if (magic != NULL) { if (pread(fd, sector, sectorsize, mediasize - sectorsize) != (ssize_t)sectorsize) { error = errno; goto out; } std_metadata_decode(sector, &md); if (strcmp(md.md_magic, magic) != 0) { error = EINVAL; goto out; } } bzero(sector, sectorsize); if (pwrite(fd, sector, sectorsize, mediasize - sectorsize) != (ssize_t)sectorsize) { error = errno; goto out; } + (void)ioctl(fd, DIOCGFLUSH, NULL); out: if (sector != NULL) free(sector); close(fd); return (error); } /* * Set an error message, if one does not already exist. */ void gctl_error(struct gctl_req *req, const char *error, ...) { va_list ap; if (req->error != NULL) return; va_start(ap, error); vasprintf(&req->error, error, ap); va_end(ap); } static void * gctl_get_param(struct gctl_req *req, size_t len, const char *pfmt, va_list ap) { struct gctl_req_arg *argp; char param[256]; void *p; unsigned i; vsnprintf(param, sizeof(param), pfmt, ap); for (i = 0; i < req->narg; i++) { argp = &req->arg[i]; if (strcmp(param, argp->name)) continue; if (!(argp->flag & GCTL_PARAM_RD)) continue; p = argp->value; if (len == 0) { /* We are looking for a string. */ if (argp->len < 1) { fprintf(stderr, "No length argument (%s).\n", param); abort(); } if (((char *)p)[argp->len - 1] != '\0') { fprintf(stderr, "Unterminated argument (%s).\n", param); abort(); } } else if ((int)len != argp->len) { fprintf(stderr, "Wrong length %s argument.\n", param); abort(); } return (p); } fprintf(stderr, "No such argument (%s).\n", param); abort(); } int gctl_get_int(struct gctl_req *req, const char *pfmt, ...) { int *p; va_list ap; va_start(ap, pfmt); p = gctl_get_param(req, sizeof(int), pfmt, ap); va_end(ap); return (*p); } intmax_t gctl_get_intmax(struct gctl_req *req, const char *pfmt, ...) { intmax_t *p; va_list ap; va_start(ap, pfmt); p = gctl_get_param(req, sizeof(intmax_t), pfmt, ap); va_end(ap); return (*p); } const char * gctl_get_ascii(struct gctl_req *req, const char *pfmt, ...) { const char *p; va_list ap; va_start(ap, pfmt); p = gctl_get_param(req, 0, pfmt, ap); va_end(ap); return (p); } int gctl_change_param(struct gctl_req *req, const char *name, int len, const void *value) { struct gctl_req_arg *ap; unsigned i; if (req == NULL || req->error != NULL) return (EDOOFUS); for (i = 0; i < req->narg; i++) { ap = &req->arg[i]; if (strcmp(ap->name, name) != 0) continue; ap->value = __DECONST(void *, value); if (len >= 0) { ap->flag &= ~GCTL_PARAM_ASCII; ap->len = len; } else if (len < 0) { ap->flag |= GCTL_PARAM_ASCII; ap->len = strlen(value) + 1; } return (0); } return (ENOENT); } int gctl_delete_param(struct gctl_req *req, const char *name) { struct gctl_req_arg *ap; unsigned int i; if (req == NULL || req->error != NULL) return (EDOOFUS); i = 0; while (i < req->narg) { ap = &req->arg[i]; if (strcmp(ap->name, name) == 0) break; i++; } if (i == req->narg) return (ENOENT); req->narg--; while (i < req->narg) { req->arg[i] = req->arg[i + 1]; i++; } return (0); } int gctl_has_param(struct gctl_req *req, const char *name) { struct gctl_req_arg *ap; unsigned int i; if (req == NULL || req->error != NULL) return (0); for (i = 0; i < req->narg; i++) { ap = &req->arg[i]; if (strcmp(ap->name, name) == 0) return (1); } return (0); } Index: user/imp/tbemd/sbin/hastd/primary.c =================================================================== --- user/imp/tbemd/sbin/hastd/primary.c (revision 206684) +++ user/imp/tbemd/sbin/hastd/primary.c (revision 206685) @@ -1,1793 +1,1793 @@ /*- * Copyright (c) 2009 The FreeBSD Foundation * All rights reserved. * * This software was developed by Pawel Jakub Dawidek under sponsorship from * the FreeBSD Foundation. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHORS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #include __FBSDID("$FreeBSD$"); #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include "control.h" #include "hast.h" #include "hast_proto.h" #include "hastd.h" #include "metadata.h" #include "proto.h" #include "pjdlog.h" #include "subr.h" #include "synch.h" struct hio { /* * Number of components we are still waiting for. * When this field goes to 0, we can send the request back to the * kernel. Each component has to decrease this counter by one * even on failure. */ unsigned int hio_countdown; /* * Each component has a place to store its own error. * Once the request is handled by all components we can decide if the * request overall is successful or not. */ int *hio_errors; /* * Structure used to comunicate with GEOM Gate class. */ struct g_gate_ctl_io hio_ggio; TAILQ_ENTRY(hio) *hio_next; }; #define hio_free_next hio_next[0] #define hio_done_next hio_next[0] /* * Free list holds unused structures. When free list is empty, we have to wait * until some in-progress requests are freed. */ static TAILQ_HEAD(, hio) hio_free_list; static pthread_mutex_t hio_free_list_lock; static pthread_cond_t hio_free_list_cond; /* * There is one send list for every component. One requests is placed on all * send lists - each component gets the same request, but each component is * responsible for managing his own send list. */ static TAILQ_HEAD(, hio) *hio_send_list; static pthread_mutex_t *hio_send_list_lock; static pthread_cond_t *hio_send_list_cond; /* * There is one recv list for every component, although local components don't * use recv lists as local requests are done synchronously. */ static TAILQ_HEAD(, hio) *hio_recv_list; static pthread_mutex_t *hio_recv_list_lock; static pthread_cond_t *hio_recv_list_cond; /* * Request is placed on done list by the slowest component (the one that * decreased hio_countdown from 1 to 0). */ static TAILQ_HEAD(, hio) hio_done_list; static pthread_mutex_t hio_done_list_lock; static pthread_cond_t hio_done_list_cond; /* * Structure below are for interaction with sync thread. */ static bool sync_inprogress; static pthread_mutex_t sync_lock; static pthread_cond_t sync_cond; /* * The lock below allows to synchornize access to remote connections. */ static pthread_rwlock_t *hio_remote_lock; static pthread_mutex_t hio_guard_lock; static pthread_cond_t hio_guard_cond; /* * Lock to synchronize metadata updates. Also synchronize access to * hr_primary_localcnt and hr_primary_remotecnt fields. */ static pthread_mutex_t metadata_lock; /* * Maximum number of outstanding I/O requests. */ #define HAST_HIO_MAX 256 /* * Number of components. At this point there are only two components: local * and remote, but in the future it might be possible to use multiple local * and remote components. */ #define HAST_NCOMPONENTS 2 /* * Number of seconds to sleep before next reconnect try. */ #define RECONNECT_SLEEP 5 #define ISCONNECTED(res, no) \ ((res)->hr_remotein != NULL && (res)->hr_remoteout != NULL) #define QUEUE_INSERT1(hio, name, ncomp) do { \ bool _wakeup; \ \ mtx_lock(&hio_##name##_list_lock[(ncomp)]); \ _wakeup = TAILQ_EMPTY(&hio_##name##_list[(ncomp)]); \ TAILQ_INSERT_TAIL(&hio_##name##_list[(ncomp)], (hio), \ hio_next[(ncomp)]); \ mtx_unlock(&hio_##name##_list_lock[ncomp]); \ if (_wakeup) \ cv_signal(&hio_##name##_list_cond[(ncomp)]); \ } while (0) #define QUEUE_INSERT2(hio, name) do { \ bool _wakeup; \ \ mtx_lock(&hio_##name##_list_lock); \ _wakeup = TAILQ_EMPTY(&hio_##name##_list); \ TAILQ_INSERT_TAIL(&hio_##name##_list, (hio), hio_##name##_next);\ mtx_unlock(&hio_##name##_list_lock); \ if (_wakeup) \ cv_signal(&hio_##name##_list_cond); \ } while (0) #define QUEUE_TAKE1(hio, name, ncomp) do { \ mtx_lock(&hio_##name##_list_lock[(ncomp)]); \ while (((hio) = TAILQ_FIRST(&hio_##name##_list[(ncomp)])) == NULL) { \ cv_wait(&hio_##name##_list_cond[(ncomp)], \ &hio_##name##_list_lock[(ncomp)]); \ } \ TAILQ_REMOVE(&hio_##name##_list[(ncomp)], (hio), \ hio_next[(ncomp)]); \ mtx_unlock(&hio_##name##_list_lock[(ncomp)]); \ } while (0) #define QUEUE_TAKE2(hio, name) do { \ mtx_lock(&hio_##name##_list_lock); \ while (((hio) = TAILQ_FIRST(&hio_##name##_list)) == NULL) { \ cv_wait(&hio_##name##_list_cond, \ &hio_##name##_list_lock); \ } \ TAILQ_REMOVE(&hio_##name##_list, (hio), hio_##name##_next); \ mtx_unlock(&hio_##name##_list_lock); \ } while (0) #define SYNCREQ(hio) do { (hio)->hio_ggio.gctl_unit = -1; } while (0) #define ISSYNCREQ(hio) ((hio)->hio_ggio.gctl_unit == -1) #define SYNCREQDONE(hio) do { (hio)->hio_ggio.gctl_unit = -2; } while (0) #define ISSYNCREQDONE(hio) ((hio)->hio_ggio.gctl_unit == -2) static struct hast_resource *gres; static pthread_mutex_t range_lock; static struct rangelocks *range_regular; static bool range_regular_wait; static pthread_cond_t range_regular_cond; static struct rangelocks *range_sync; static bool range_sync_wait; static pthread_cond_t range_sync_cond; static void *ggate_recv_thread(void *arg); static void *local_send_thread(void *arg); static void *remote_send_thread(void *arg); static void *remote_recv_thread(void *arg); static void *ggate_send_thread(void *arg); static void *sync_thread(void *arg); static void *guard_thread(void *arg); static void sighandler(int sig); static void cleanup(struct hast_resource *res) { int rerrno; /* Remember errno. */ rerrno = errno; /* * Close descriptor to /dev/hast/ * to work-around race in the kernel. */ close(res->hr_localfd); /* Destroy ggate provider if we created one. */ if (res->hr_ggateunit >= 0) { struct g_gate_ctl_destroy ggiod; ggiod.gctl_version = G_GATE_VERSION; ggiod.gctl_unit = res->hr_ggateunit; ggiod.gctl_force = 1; if (ioctl(res->hr_ggatefd, G_GATE_CMD_DESTROY, &ggiod) < 0) { pjdlog_warning("Unable to destroy hast/%s device", res->hr_provname); } res->hr_ggateunit = -1; } /* Restore errno. */ errno = rerrno; } static void primary_exit(int exitcode, const char *fmt, ...) { va_list ap; assert(exitcode != EX_OK); va_start(ap, fmt); pjdlogv_errno(LOG_ERR, fmt, ap); va_end(ap); cleanup(gres); exit(exitcode); } static void primary_exitx(int exitcode, const char *fmt, ...) { va_list ap; va_start(ap, fmt); pjdlogv(exitcode == EX_OK ? LOG_INFO : LOG_ERR, fmt, ap); va_end(ap); cleanup(gres); exit(exitcode); } static int hast_activemap_flush(struct hast_resource *res) { const unsigned char *buf; size_t size; buf = activemap_bitmap(res->hr_amp, &size); assert(buf != NULL); assert((size % res->hr_local_sectorsize) == 0); if (pwrite(res->hr_localfd, buf, size, METADATA_SIZE) != (ssize_t)size) { KEEP_ERRNO(pjdlog_errno(LOG_ERR, "Unable to flush activemap to disk")); return (-1); } return (0); } static void init_environment(struct hast_resource *res __unused) { struct hio *hio; unsigned int ii, ncomps; /* * In the future it might be per-resource value. */ ncomps = HAST_NCOMPONENTS; /* * Allocate memory needed by lists. */ hio_send_list = malloc(sizeof(hio_send_list[0]) * ncomps); if (hio_send_list == NULL) { primary_exitx(EX_TEMPFAIL, "Unable to allocate %zu bytes of memory for send lists.", sizeof(hio_send_list[0]) * ncomps); } hio_send_list_lock = malloc(sizeof(hio_send_list_lock[0]) * ncomps); if (hio_send_list_lock == NULL) { primary_exitx(EX_TEMPFAIL, "Unable to allocate %zu bytes of memory for send list locks.", sizeof(hio_send_list_lock[0]) * ncomps); } hio_send_list_cond = malloc(sizeof(hio_send_list_cond[0]) * ncomps); if (hio_send_list_cond == NULL) { primary_exitx(EX_TEMPFAIL, "Unable to allocate %zu bytes of memory for send list condition variables.", sizeof(hio_send_list_cond[0]) * ncomps); } hio_recv_list = malloc(sizeof(hio_recv_list[0]) * ncomps); if (hio_recv_list == NULL) { primary_exitx(EX_TEMPFAIL, "Unable to allocate %zu bytes of memory for recv lists.", sizeof(hio_recv_list[0]) * ncomps); } hio_recv_list_lock = malloc(sizeof(hio_recv_list_lock[0]) * ncomps); if (hio_recv_list_lock == NULL) { primary_exitx(EX_TEMPFAIL, "Unable to allocate %zu bytes of memory for recv list locks.", sizeof(hio_recv_list_lock[0]) * ncomps); } hio_recv_list_cond = malloc(sizeof(hio_recv_list_cond[0]) * ncomps); if (hio_recv_list_cond == NULL) { primary_exitx(EX_TEMPFAIL, "Unable to allocate %zu bytes of memory for recv list condition variables.", sizeof(hio_recv_list_cond[0]) * ncomps); } hio_remote_lock = malloc(sizeof(hio_remote_lock[0]) * ncomps); if (hio_remote_lock == NULL) { primary_exitx(EX_TEMPFAIL, "Unable to allocate %zu bytes of memory for remote connections locks.", sizeof(hio_remote_lock[0]) * ncomps); } /* * Initialize lists, their locks and theirs condition variables. */ TAILQ_INIT(&hio_free_list); mtx_init(&hio_free_list_lock); cv_init(&hio_free_list_cond); for (ii = 0; ii < HAST_NCOMPONENTS; ii++) { TAILQ_INIT(&hio_send_list[ii]); mtx_init(&hio_send_list_lock[ii]); cv_init(&hio_send_list_cond[ii]); TAILQ_INIT(&hio_recv_list[ii]); mtx_init(&hio_recv_list_lock[ii]); cv_init(&hio_recv_list_cond[ii]); rw_init(&hio_remote_lock[ii]); } TAILQ_INIT(&hio_done_list); mtx_init(&hio_done_list_lock); cv_init(&hio_done_list_cond); mtx_init(&hio_guard_lock); cv_init(&hio_guard_cond); mtx_init(&metadata_lock); /* * Allocate requests pool and initialize requests. */ for (ii = 0; ii < HAST_HIO_MAX; ii++) { hio = malloc(sizeof(*hio)); if (hio == NULL) { primary_exitx(EX_TEMPFAIL, "Unable to allocate %zu bytes of memory for hio request.", sizeof(*hio)); } hio->hio_countdown = 0; hio->hio_errors = malloc(sizeof(hio->hio_errors[0]) * ncomps); if (hio->hio_errors == NULL) { primary_exitx(EX_TEMPFAIL, "Unable allocate %zu bytes of memory for hio errors.", sizeof(hio->hio_errors[0]) * ncomps); } hio->hio_next = malloc(sizeof(hio->hio_next[0]) * ncomps); if (hio->hio_next == NULL) { primary_exitx(EX_TEMPFAIL, "Unable allocate %zu bytes of memory for hio_next field.", sizeof(hio->hio_next[0]) * ncomps); } hio->hio_ggio.gctl_version = G_GATE_VERSION; hio->hio_ggio.gctl_data = malloc(MAXPHYS); if (hio->hio_ggio.gctl_data == NULL) { primary_exitx(EX_TEMPFAIL, "Unable to allocate %zu bytes of memory for gctl_data.", MAXPHYS); } hio->hio_ggio.gctl_length = MAXPHYS; hio->hio_ggio.gctl_error = 0; TAILQ_INSERT_HEAD(&hio_free_list, hio, hio_free_next); } /* * Turn on signals handling. */ signal(SIGINT, sighandler); signal(SIGTERM, sighandler); } static void init_local(struct hast_resource *res) { unsigned char *buf; size_t mapsize; if (metadata_read(res, true) < 0) exit(EX_NOINPUT); mtx_init(&res->hr_amp_lock); if (activemap_init(&res->hr_amp, res->hr_datasize, res->hr_extentsize, res->hr_local_sectorsize, res->hr_keepdirty) < 0) { primary_exit(EX_TEMPFAIL, "Unable to create activemap"); } mtx_init(&range_lock); cv_init(&range_regular_cond); if (rangelock_init(&range_regular) < 0) primary_exit(EX_TEMPFAIL, "Unable to create regular range lock"); cv_init(&range_sync_cond); if (rangelock_init(&range_sync) < 0) primary_exit(EX_TEMPFAIL, "Unable to create sync range lock"); mapsize = activemap_ondisk_size(res->hr_amp); buf = calloc(1, mapsize); if (buf == NULL) { primary_exitx(EX_TEMPFAIL, "Unable to allocate buffer for activemap."); } if (pread(res->hr_localfd, buf, mapsize, METADATA_SIZE) != (ssize_t)mapsize) { primary_exit(EX_NOINPUT, "Unable to read activemap"); } activemap_copyin(res->hr_amp, buf, mapsize); if (res->hr_resuid != 0) return; /* * We're using provider for the first time, so we have to generate * resource unique identifier and initialize local and remote counts. */ arc4random_buf(&res->hr_resuid, sizeof(res->hr_resuid)); res->hr_primary_localcnt = 1; res->hr_primary_remotecnt = 0; if (metadata_write(res) < 0) exit(EX_NOINPUT); } static bool init_remote(struct hast_resource *res, struct proto_conn **inp, struct proto_conn **outp) { struct proto_conn *in, *out; struct nv *nvout, *nvin; const unsigned char *token; unsigned char *map; const char *errmsg; int32_t extentsize; int64_t datasize; uint32_t mapsize; size_t size; assert((inp == NULL && outp == NULL) || (inp != NULL && outp != NULL)); in = out = NULL; /* Prepare outgoing connection with remote node. */ if (proto_client(res->hr_remoteaddr, &out) < 0) { primary_exit(EX_OSERR, "Unable to create connection to %s", res->hr_remoteaddr); } /* Try to connect, but accept failure. */ if (proto_connect(out) < 0) { pjdlog_errno(LOG_WARNING, "Unable to connect to %s", res->hr_remoteaddr); goto close; } /* * First handshake step. * Setup outgoing connection with remote node. */ nvout = nv_alloc(); nv_add_string(nvout, res->hr_name, "resource"); if (nv_error(nvout) != 0) { pjdlog_common(LOG_WARNING, 0, nv_error(nvout), "Unable to allocate header for connection with %s", res->hr_remoteaddr); nv_free(nvout); goto close; } if (hast_proto_send(res, out, nvout, NULL, 0) < 0) { pjdlog_errno(LOG_WARNING, "Unable to send handshake header to %s", res->hr_remoteaddr); nv_free(nvout); goto close; } nv_free(nvout); if (hast_proto_recv_hdr(out, &nvin) < 0) { pjdlog_errno(LOG_WARNING, "Unable to receive handshake header from %s", res->hr_remoteaddr); goto close; } errmsg = nv_get_string(nvin, "errmsg"); if (errmsg != NULL) { pjdlog_warning("%s", errmsg); nv_free(nvin); goto close; } token = nv_get_uint8_array(nvin, &size, "token"); if (token == NULL) { pjdlog_warning("Handshake header from %s has no 'token' field.", res->hr_remoteaddr); nv_free(nvin); goto close; } if (size != sizeof(res->hr_token)) { pjdlog_warning("Handshake header from %s contains 'token' of wrong size (got %zu, expected %zu).", res->hr_remoteaddr, size, sizeof(res->hr_token)); nv_free(nvin); goto close; } bcopy(token, res->hr_token, sizeof(res->hr_token)); nv_free(nvin); /* * Second handshake step. * Setup incoming connection with remote node. */ if (proto_client(res->hr_remoteaddr, &in) < 0) { pjdlog_errno(LOG_WARNING, "Unable to create connection to %s", res->hr_remoteaddr); } /* Try to connect, but accept failure. */ if (proto_connect(in) < 0) { pjdlog_errno(LOG_WARNING, "Unable to connect to %s", res->hr_remoteaddr); goto close; } nvout = nv_alloc(); nv_add_string(nvout, res->hr_name, "resource"); nv_add_uint8_array(nvout, res->hr_token, sizeof(res->hr_token), "token"); nv_add_uint64(nvout, res->hr_resuid, "resuid"); nv_add_uint64(nvout, res->hr_primary_localcnt, "localcnt"); nv_add_uint64(nvout, res->hr_primary_remotecnt, "remotecnt"); if (nv_error(nvout) != 0) { pjdlog_common(LOG_WARNING, 0, nv_error(nvout), "Unable to allocate header for connection with %s", res->hr_remoteaddr); nv_free(nvout); goto close; } if (hast_proto_send(res, in, nvout, NULL, 0) < 0) { pjdlog_errno(LOG_WARNING, "Unable to send handshake header to %s", res->hr_remoteaddr); nv_free(nvout); goto close; } nv_free(nvout); if (hast_proto_recv_hdr(out, &nvin) < 0) { pjdlog_errno(LOG_WARNING, "Unable to receive handshake header from %s", res->hr_remoteaddr); goto close; } errmsg = nv_get_string(nvin, "errmsg"); if (errmsg != NULL) { pjdlog_warning("%s", errmsg); nv_free(nvin); goto close; } datasize = nv_get_int64(nvin, "datasize"); if (datasize != res->hr_datasize) { pjdlog_warning("Data size differs between nodes (local=%jd, remote=%jd).", (intmax_t)res->hr_datasize, (intmax_t)datasize); nv_free(nvin); goto close; } extentsize = nv_get_int32(nvin, "extentsize"); if (extentsize != res->hr_extentsize) { pjdlog_warning("Extent size differs between nodes (local=%zd, remote=%zd).", (ssize_t)res->hr_extentsize, (ssize_t)extentsize); nv_free(nvin); goto close; } res->hr_secondary_localcnt = nv_get_uint64(nvin, "localcnt"); res->hr_secondary_remotecnt = nv_get_uint64(nvin, "remotecnt"); res->hr_syncsrc = nv_get_uint8(nvin, "syncsrc"); map = NULL; mapsize = nv_get_uint32(nvin, "mapsize"); if (mapsize > 0) { map = malloc(mapsize); if (map == NULL) { pjdlog_error("Unable to allocate memory for remote activemap (mapsize=%ju).", (uintmax_t)mapsize); nv_free(nvin); goto close; } /* * Remote node have some dirty extents on its own, lets * download its activemap. */ if (hast_proto_recv_data(res, out, nvin, map, mapsize) < 0) { pjdlog_errno(LOG_ERR, "Unable to receive remote activemap"); nv_free(nvin); free(map); goto close; } /* * Merge local and remote bitmaps. */ activemap_merge(res->hr_amp, map, mapsize); free(map); /* * Now that we merged bitmaps from both nodes, flush it to the * disk before we start to synchronize. */ (void)hast_activemap_flush(res); } pjdlog_info("Connected to %s.", res->hr_remoteaddr); if (inp != NULL && outp != NULL) { *inp = in; *outp = out; } else { res->hr_remotein = in; res->hr_remoteout = out; } return (true); close: proto_close(out); if (in != NULL) proto_close(in); return (false); } static void sync_start(void) { mtx_lock(&sync_lock); sync_inprogress = true; mtx_unlock(&sync_lock); cv_signal(&sync_cond); } static void init_ggate(struct hast_resource *res) { struct g_gate_ctl_create ggiocreate; struct g_gate_ctl_cancel ggiocancel; /* * We communicate with ggate via /dev/ggctl. Open it. */ res->hr_ggatefd = open("/dev/" G_GATE_CTL_NAME, O_RDWR); if (res->hr_ggatefd < 0) primary_exit(EX_OSFILE, "Unable to open /dev/" G_GATE_CTL_NAME); /* * Create provider before trying to connect, as connection failure * is not critical, but may take some time. */ ggiocreate.gctl_version = G_GATE_VERSION; ggiocreate.gctl_mediasize = res->hr_datasize; ggiocreate.gctl_sectorsize = res->hr_local_sectorsize; ggiocreate.gctl_flags = 0; - ggiocreate.gctl_maxcount = 128; + ggiocreate.gctl_maxcount = G_GATE_MAX_QUEUE_SIZE; ggiocreate.gctl_timeout = 0; ggiocreate.gctl_unit = G_GATE_NAME_GIVEN; snprintf(ggiocreate.gctl_name, sizeof(ggiocreate.gctl_name), "hast/%s", res->hr_provname); bzero(ggiocreate.gctl_info, sizeof(ggiocreate.gctl_info)); if (ioctl(res->hr_ggatefd, G_GATE_CMD_CREATE, &ggiocreate) == 0) { pjdlog_info("Device hast/%s created.", res->hr_provname); res->hr_ggateunit = ggiocreate.gctl_unit; return; } if (errno != EEXIST) { primary_exit(EX_OSERR, "Unable to create hast/%s device", res->hr_provname); } pjdlog_debug(1, "Device hast/%s already exists, we will try to take it over.", res->hr_provname); /* * If we received EEXIST, we assume that the process who created the * provider died and didn't clean up. In that case we will start from * where he left of. */ ggiocancel.gctl_version = G_GATE_VERSION; ggiocancel.gctl_unit = G_GATE_NAME_GIVEN; snprintf(ggiocancel.gctl_name, sizeof(ggiocancel.gctl_name), "hast/%s", res->hr_provname); if (ioctl(res->hr_ggatefd, G_GATE_CMD_CANCEL, &ggiocancel) == 0) { pjdlog_info("Device hast/%s recovered.", res->hr_provname); res->hr_ggateunit = ggiocancel.gctl_unit; return; } primary_exit(EX_OSERR, "Unable to take over hast/%s device", res->hr_provname); } void hastd_primary(struct hast_resource *res) { pthread_t td; pid_t pid; int error; gres = res; /* * Create communication channel between parent and child. */ if (proto_client("socketpair://", &res->hr_ctrl) < 0) { KEEP_ERRNO((void)pidfile_remove(pfh)); primary_exit(EX_OSERR, "Unable to create control sockets between parent and child"); } pid = fork(); if (pid < 0) { KEEP_ERRNO((void)pidfile_remove(pfh)); primary_exit(EX_OSERR, "Unable to fork"); } if (pid > 0) { /* This is parent. */ res->hr_workerpid = pid; return; } (void)pidfile_close(pfh); setproctitle("%s (primary)", res->hr_name); init_local(res); if (init_remote(res, NULL, NULL)) sync_start(); init_ggate(res); init_environment(res); error = pthread_create(&td, NULL, ggate_recv_thread, res); assert(error == 0); error = pthread_create(&td, NULL, local_send_thread, res); assert(error == 0); error = pthread_create(&td, NULL, remote_send_thread, res); assert(error == 0); error = pthread_create(&td, NULL, remote_recv_thread, res); assert(error == 0); error = pthread_create(&td, NULL, ggate_send_thread, res); assert(error == 0); error = pthread_create(&td, NULL, sync_thread, res); assert(error == 0); error = pthread_create(&td, NULL, ctrl_thread, res); assert(error == 0); (void)guard_thread(res); } static void reqlog(int loglevel, int debuglevel, struct g_gate_ctl_io *ggio, const char *fmt, ...) { char msg[1024]; va_list ap; int len; va_start(ap, fmt); len = vsnprintf(msg, sizeof(msg), fmt, ap); va_end(ap); if ((size_t)len < sizeof(msg)) { switch (ggio->gctl_cmd) { case BIO_READ: (void)snprintf(msg + len, sizeof(msg) - len, "READ(%ju, %ju).", (uintmax_t)ggio->gctl_offset, (uintmax_t)ggio->gctl_length); break; case BIO_DELETE: (void)snprintf(msg + len, sizeof(msg) - len, "DELETE(%ju, %ju).", (uintmax_t)ggio->gctl_offset, (uintmax_t)ggio->gctl_length); break; case BIO_FLUSH: (void)snprintf(msg + len, sizeof(msg) - len, "FLUSH."); break; case BIO_WRITE: (void)snprintf(msg + len, sizeof(msg) - len, "WRITE(%ju, %ju).", (uintmax_t)ggio->gctl_offset, (uintmax_t)ggio->gctl_length); break; default: (void)snprintf(msg + len, sizeof(msg) - len, "UNKNOWN(%u).", (unsigned int)ggio->gctl_cmd); break; } } pjdlog_common(loglevel, debuglevel, -1, "%s", msg); } static void remote_close(struct hast_resource *res, int ncomp) { rw_wlock(&hio_remote_lock[ncomp]); /* * A race is possible between dropping rlock and acquiring wlock - * another thread can close connection in-between. */ if (!ISCONNECTED(res, ncomp)) { assert(res->hr_remotein == NULL); assert(res->hr_remoteout == NULL); rw_unlock(&hio_remote_lock[ncomp]); return; } assert(res->hr_remotein != NULL); assert(res->hr_remoteout != NULL); pjdlog_debug(2, "Closing old incoming connection to %s.", res->hr_remoteaddr); proto_close(res->hr_remotein); res->hr_remotein = NULL; pjdlog_debug(2, "Closing old outgoing connection to %s.", res->hr_remoteaddr); proto_close(res->hr_remoteout); res->hr_remoteout = NULL; rw_unlock(&hio_remote_lock[ncomp]); /* * Stop synchronization if in-progress. */ mtx_lock(&sync_lock); if (sync_inprogress) sync_inprogress = false; mtx_unlock(&sync_lock); /* * Wake up guard thread, so it can immediately start reconnect. */ mtx_lock(&hio_guard_lock); cv_signal(&hio_guard_cond); mtx_unlock(&hio_guard_lock); } /* * Thread receives ggate I/O requests from the kernel and passes them to * appropriate threads: * WRITE - always goes to both local_send and remote_send threads * READ (when the block is up-to-date on local component) - * only local_send thread * READ (when the block isn't up-to-date on local component) - * only remote_send thread * DELETE - always goes to both local_send and remote_send threads * FLUSH - always goes to both local_send and remote_send threads */ static void * ggate_recv_thread(void *arg) { struct hast_resource *res = arg; struct g_gate_ctl_io *ggio; struct hio *hio; unsigned int ii, ncomp, ncomps; int error; ncomps = HAST_NCOMPONENTS; for (;;) { pjdlog_debug(2, "ggate_recv: Taking free request."); QUEUE_TAKE2(hio, free); pjdlog_debug(2, "ggate_recv: (%p) Got free request.", hio); ggio = &hio->hio_ggio; ggio->gctl_unit = res->hr_ggateunit; ggio->gctl_length = MAXPHYS; ggio->gctl_error = 0; pjdlog_debug(2, "ggate_recv: (%p) Waiting for request from the kernel.", hio); if (ioctl(res->hr_ggatefd, G_GATE_CMD_START, ggio) < 0) { if (sigexit_received) pthread_exit(NULL); primary_exit(EX_OSERR, "G_GATE_CMD_START failed"); } error = ggio->gctl_error; switch (error) { case 0: break; case ECANCELED: /* Exit gracefully. */ if (!sigexit_received) { pjdlog_debug(2, "ggate_recv: (%p) Received cancel from the kernel.", hio); pjdlog_info("Received cancel from the kernel, exiting."); } pthread_exit(NULL); case ENOMEM: /* * Buffer too small? Impossible, we allocate MAXPHYS * bytes - request can't be bigger than that. */ /* FALLTHROUGH */ case ENXIO: default: primary_exitx(EX_OSERR, "G_GATE_CMD_START failed: %s.", strerror(error)); } for (ii = 0; ii < ncomps; ii++) hio->hio_errors[ii] = EINVAL; reqlog(LOG_DEBUG, 2, ggio, "ggate_recv: (%p) Request received from the kernel: ", hio); /* * Inform all components about new write request. * For read request prefer local component unless the given * range is out-of-date, then use remote component. */ switch (ggio->gctl_cmd) { case BIO_READ: pjdlog_debug(2, "ggate_recv: (%p) Moving request to the send queue.", hio); refcount_init(&hio->hio_countdown, 1); mtx_lock(&metadata_lock); if (res->hr_syncsrc == HAST_SYNCSRC_UNDEF || res->hr_syncsrc == HAST_SYNCSRC_PRIMARY) { /* * This range is up-to-date on local component, * so handle request locally. */ /* Local component is 0 for now. */ ncomp = 0; } else /* if (res->hr_syncsrc == HAST_SYNCSRC_SECONDARY) */ { assert(res->hr_syncsrc == HAST_SYNCSRC_SECONDARY); /* * This range is out-of-date on local component, * so send request to the remote node. */ /* Remote component is 1 for now. */ ncomp = 1; } mtx_unlock(&metadata_lock); QUEUE_INSERT1(hio, send, ncomp); break; case BIO_WRITE: for (;;) { mtx_lock(&range_lock); if (rangelock_islocked(range_sync, ggio->gctl_offset, ggio->gctl_length)) { pjdlog_debug(2, "regular: Range offset=%jd length=%zu locked.", (intmax_t)ggio->gctl_offset, (size_t)ggio->gctl_length); range_regular_wait = true; cv_wait(&range_regular_cond, &range_lock); range_regular_wait = false; mtx_unlock(&range_lock); continue; } if (rangelock_add(range_regular, ggio->gctl_offset, ggio->gctl_length) < 0) { mtx_unlock(&range_lock); pjdlog_debug(2, "regular: Range offset=%jd length=%zu is already locked, waiting.", (intmax_t)ggio->gctl_offset, (size_t)ggio->gctl_length); sleep(1); continue; } mtx_unlock(&range_lock); break; } mtx_lock(&res->hr_amp_lock); if (activemap_write_start(res->hr_amp, ggio->gctl_offset, ggio->gctl_length)) { (void)hast_activemap_flush(res); } mtx_unlock(&res->hr_amp_lock); /* FALLTHROUGH */ case BIO_DELETE: case BIO_FLUSH: pjdlog_debug(2, "ggate_recv: (%p) Moving request to the send queues.", hio); refcount_init(&hio->hio_countdown, ncomps); for (ii = 0; ii < ncomps; ii++) QUEUE_INSERT1(hio, send, ii); break; } } /* NOTREACHED */ return (NULL); } /* * Thread reads from or writes to local component. * If local read fails, it redirects it to remote_send thread. */ static void * local_send_thread(void *arg) { struct hast_resource *res = arg; struct g_gate_ctl_io *ggio; struct hio *hio; unsigned int ncomp, rncomp; ssize_t ret; /* Local component is 0 for now. */ ncomp = 0; /* Remote component is 1 for now. */ rncomp = 1; for (;;) { pjdlog_debug(2, "local_send: Taking request."); QUEUE_TAKE1(hio, send, ncomp); pjdlog_debug(2, "local_send: (%p) Got request.", hio); ggio = &hio->hio_ggio; switch (ggio->gctl_cmd) { case BIO_READ: ret = pread(res->hr_localfd, ggio->gctl_data, ggio->gctl_length, ggio->gctl_offset + res->hr_localoff); if (ret == ggio->gctl_length) hio->hio_errors[ncomp] = 0; else { /* * If READ failed, try to read from remote node. */ QUEUE_INSERT1(hio, send, rncomp); continue; } break; case BIO_WRITE: ret = pwrite(res->hr_localfd, ggio->gctl_data, ggio->gctl_length, ggio->gctl_offset + res->hr_localoff); if (ret < 0) hio->hio_errors[ncomp] = errno; else if (ret != ggio->gctl_length) hio->hio_errors[ncomp] = EIO; else hio->hio_errors[ncomp] = 0; break; case BIO_DELETE: ret = g_delete(res->hr_localfd, ggio->gctl_offset + res->hr_localoff, ggio->gctl_length); if (ret < 0) hio->hio_errors[ncomp] = errno; else hio->hio_errors[ncomp] = 0; break; case BIO_FLUSH: ret = g_flush(res->hr_localfd); if (ret < 0) hio->hio_errors[ncomp] = errno; else hio->hio_errors[ncomp] = 0; break; } if (refcount_release(&hio->hio_countdown)) { if (ISSYNCREQ(hio)) { mtx_lock(&sync_lock); SYNCREQDONE(hio); mtx_unlock(&sync_lock); cv_signal(&sync_cond); } else { pjdlog_debug(2, "local_send: (%p) Moving request to the done queue.", hio); QUEUE_INSERT2(hio, done); } } } /* NOTREACHED */ return (NULL); } /* * Thread sends request to secondary node. */ static void * remote_send_thread(void *arg) { struct hast_resource *res = arg; struct g_gate_ctl_io *ggio; struct hio *hio; struct nv *nv; unsigned int ncomp; bool wakeup; uint64_t offset, length; uint8_t cmd; void *data; /* Remote component is 1 for now. */ ncomp = 1; for (;;) { pjdlog_debug(2, "remote_send: Taking request."); QUEUE_TAKE1(hio, send, ncomp); pjdlog_debug(2, "remote_send: (%p) Got request.", hio); ggio = &hio->hio_ggio; switch (ggio->gctl_cmd) { case BIO_READ: cmd = HIO_READ; data = NULL; offset = ggio->gctl_offset; length = ggio->gctl_length; break; case BIO_WRITE: cmd = HIO_WRITE; data = ggio->gctl_data; offset = ggio->gctl_offset; length = ggio->gctl_length; break; case BIO_DELETE: cmd = HIO_DELETE; data = NULL; offset = ggio->gctl_offset; length = ggio->gctl_length; break; case BIO_FLUSH: cmd = HIO_FLUSH; data = NULL; offset = 0; length = 0; break; default: assert(!"invalid condition"); abort(); } nv = nv_alloc(); nv_add_uint8(nv, cmd, "cmd"); nv_add_uint64(nv, (uint64_t)ggio->gctl_seq, "seq"); nv_add_uint64(nv, offset, "offset"); nv_add_uint64(nv, length, "length"); if (nv_error(nv) != 0) { hio->hio_errors[ncomp] = nv_error(nv); pjdlog_debug(2, "remote_send: (%p) Unable to prepare header to send.", hio); reqlog(LOG_ERR, 0, ggio, "Unable to prepare header to send (%s): ", strerror(nv_error(nv))); /* Move failed request immediately to the done queue. */ goto done_queue; } pjdlog_debug(2, "remote_send: (%p) Moving request to the recv queue.", hio); /* * Protect connection from disappearing. */ rw_rlock(&hio_remote_lock[ncomp]); if (!ISCONNECTED(res, ncomp)) { rw_unlock(&hio_remote_lock[ncomp]); hio->hio_errors[ncomp] = ENOTCONN; goto done_queue; } /* * Move the request to recv queue before sending it, because * in different order we can get reply before we move request * to recv queue. */ mtx_lock(&hio_recv_list_lock[ncomp]); wakeup = TAILQ_EMPTY(&hio_recv_list[ncomp]); TAILQ_INSERT_TAIL(&hio_recv_list[ncomp], hio, hio_next[ncomp]); mtx_unlock(&hio_recv_list_lock[ncomp]); if (hast_proto_send(res, res->hr_remoteout, nv, data, data != NULL ? length : 0) < 0) { hio->hio_errors[ncomp] = errno; rw_unlock(&hio_remote_lock[ncomp]); remote_close(res, ncomp); pjdlog_debug(2, "remote_send: (%p) Unable to send request.", hio); reqlog(LOG_ERR, 0, ggio, "Unable to send request (%s): ", strerror(hio->hio_errors[ncomp])); /* * Take request back from the receive queue and move * it immediately to the done queue. */ mtx_lock(&hio_recv_list_lock[ncomp]); TAILQ_REMOVE(&hio_recv_list[ncomp], hio, hio_next[ncomp]); mtx_unlock(&hio_recv_list_lock[ncomp]); goto done_queue; } rw_unlock(&hio_remote_lock[ncomp]); nv_free(nv); if (wakeup) cv_signal(&hio_recv_list_cond[ncomp]); continue; done_queue: nv_free(nv); if (ISSYNCREQ(hio)) { if (!refcount_release(&hio->hio_countdown)) continue; mtx_lock(&sync_lock); SYNCREQDONE(hio); mtx_unlock(&sync_lock); cv_signal(&sync_cond); continue; } if (ggio->gctl_cmd == BIO_WRITE) { mtx_lock(&res->hr_amp_lock); if (activemap_need_sync(res->hr_amp, ggio->gctl_offset, ggio->gctl_length)) { (void)hast_activemap_flush(res); } mtx_unlock(&res->hr_amp_lock); } if (!refcount_release(&hio->hio_countdown)) continue; pjdlog_debug(2, "remote_send: (%p) Moving request to the done queue.", hio); QUEUE_INSERT2(hio, done); } /* NOTREACHED */ return (NULL); } /* * Thread receives answer from secondary node and passes it to ggate_send * thread. */ static void * remote_recv_thread(void *arg) { struct hast_resource *res = arg; struct g_gate_ctl_io *ggio; struct hio *hio; struct nv *nv; unsigned int ncomp; uint64_t seq; int error; /* Remote component is 1 for now. */ ncomp = 1; for (;;) { /* Wait until there is anything to receive. */ mtx_lock(&hio_recv_list_lock[ncomp]); while (TAILQ_EMPTY(&hio_recv_list[ncomp])) { pjdlog_debug(2, "remote_recv: No requests, waiting."); cv_wait(&hio_recv_list_cond[ncomp], &hio_recv_list_lock[ncomp]); } mtx_unlock(&hio_recv_list_lock[ncomp]); rw_rlock(&hio_remote_lock[ncomp]); if (!ISCONNECTED(res, ncomp)) { rw_unlock(&hio_remote_lock[ncomp]); /* * Connection is dead, so move all pending requests to * the done queue (one-by-one). */ mtx_lock(&hio_recv_list_lock[ncomp]); hio = TAILQ_FIRST(&hio_recv_list[ncomp]); assert(hio != NULL); TAILQ_REMOVE(&hio_recv_list[ncomp], hio, hio_next[ncomp]); mtx_unlock(&hio_recv_list_lock[ncomp]); goto done_queue; } if (hast_proto_recv_hdr(res->hr_remotein, &nv) < 0) { pjdlog_errno(LOG_ERR, "Unable to receive reply header"); rw_unlock(&hio_remote_lock[ncomp]); remote_close(res, ncomp); continue; } rw_unlock(&hio_remote_lock[ncomp]); seq = nv_get_uint64(nv, "seq"); if (seq == 0) { pjdlog_error("Header contains no 'seq' field."); nv_free(nv); continue; } mtx_lock(&hio_recv_list_lock[ncomp]); TAILQ_FOREACH(hio, &hio_recv_list[ncomp], hio_next[ncomp]) { if (hio->hio_ggio.gctl_seq == seq) { TAILQ_REMOVE(&hio_recv_list[ncomp], hio, hio_next[ncomp]); break; } } mtx_unlock(&hio_recv_list_lock[ncomp]); if (hio == NULL) { pjdlog_error("Found no request matching received 'seq' field (%ju).", (uintmax_t)seq); nv_free(nv); continue; } error = nv_get_int16(nv, "error"); if (error != 0) { /* Request failed on remote side. */ hio->hio_errors[ncomp] = 0; nv_free(nv); goto done_queue; } ggio = &hio->hio_ggio; switch (ggio->gctl_cmd) { case BIO_READ: rw_rlock(&hio_remote_lock[ncomp]); if (!ISCONNECTED(res, ncomp)) { rw_unlock(&hio_remote_lock[ncomp]); nv_free(nv); goto done_queue; } if (hast_proto_recv_data(res, res->hr_remotein, nv, ggio->gctl_data, ggio->gctl_length) < 0) { hio->hio_errors[ncomp] = errno; pjdlog_errno(LOG_ERR, "Unable to receive reply data"); rw_unlock(&hio_remote_lock[ncomp]); nv_free(nv); remote_close(res, ncomp); goto done_queue; } rw_unlock(&hio_remote_lock[ncomp]); break; case BIO_WRITE: case BIO_DELETE: case BIO_FLUSH: break; default: assert(!"invalid condition"); abort(); } hio->hio_errors[ncomp] = 0; nv_free(nv); done_queue: if (refcount_release(&hio->hio_countdown)) { if (ISSYNCREQ(hio)) { mtx_lock(&sync_lock); SYNCREQDONE(hio); mtx_unlock(&sync_lock); cv_signal(&sync_cond); } else { pjdlog_debug(2, "remote_recv: (%p) Moving request to the done queue.", hio); QUEUE_INSERT2(hio, done); } } } /* NOTREACHED */ return (NULL); } /* * Thread sends answer to the kernel. */ static void * ggate_send_thread(void *arg) { struct hast_resource *res = arg; struct g_gate_ctl_io *ggio; struct hio *hio; unsigned int ii, ncomp, ncomps; ncomps = HAST_NCOMPONENTS; for (;;) { pjdlog_debug(2, "ggate_send: Taking request."); QUEUE_TAKE2(hio, done); pjdlog_debug(2, "ggate_send: (%p) Got request.", hio); ggio = &hio->hio_ggio; for (ii = 0; ii < ncomps; ii++) { if (hio->hio_errors[ii] == 0) { /* * One successful request is enough to declare * success. */ ggio->gctl_error = 0; break; } } if (ii == ncomps) { /* * None of the requests were successful. * Use first error. */ ggio->gctl_error = hio->hio_errors[0]; } if (ggio->gctl_error == 0 && ggio->gctl_cmd == BIO_WRITE) { mtx_lock(&res->hr_amp_lock); activemap_write_complete(res->hr_amp, ggio->gctl_offset, ggio->gctl_length); mtx_unlock(&res->hr_amp_lock); } if (ggio->gctl_cmd == BIO_WRITE) { /* * Unlock range we locked. */ mtx_lock(&range_lock); rangelock_del(range_regular, ggio->gctl_offset, ggio->gctl_length); if (range_sync_wait) cv_signal(&range_sync_cond); mtx_unlock(&range_lock); /* * Bump local count if this is first write after * connection failure with remote node. */ ncomp = 1; rw_rlock(&hio_remote_lock[ncomp]); if (!ISCONNECTED(res, ncomp)) { mtx_lock(&metadata_lock); if (res->hr_primary_localcnt == res->hr_secondary_remotecnt) { res->hr_primary_localcnt++; pjdlog_debug(1, "Increasing localcnt to %ju.", (uintmax_t)res->hr_primary_localcnt); (void)metadata_write(res); } mtx_unlock(&metadata_lock); } rw_unlock(&hio_remote_lock[ncomp]); } if (ioctl(res->hr_ggatefd, G_GATE_CMD_DONE, ggio) < 0) primary_exit(EX_OSERR, "G_GATE_CMD_DONE failed"); pjdlog_debug(2, "ggate_send: (%p) Moving request to the free queue.", hio); QUEUE_INSERT2(hio, free); } /* NOTREACHED */ return (NULL); } /* * Thread synchronize local and remote components. */ static void * sync_thread(void *arg __unused) { struct hast_resource *res = arg; struct hio *hio; struct g_gate_ctl_io *ggio; unsigned int ii, ncomp, ncomps; off_t offset, length, synced; bool dorewind; int syncext; ncomps = HAST_NCOMPONENTS; dorewind = true; synced = 0; for (;;) { mtx_lock(&sync_lock); while (!sync_inprogress) { dorewind = true; synced = 0; cv_wait(&sync_cond, &sync_lock); } mtx_unlock(&sync_lock); /* * Obtain offset at which we should synchronize. * Rewind synchronization if needed. */ mtx_lock(&res->hr_amp_lock); if (dorewind) activemap_sync_rewind(res->hr_amp); offset = activemap_sync_offset(res->hr_amp, &length, &syncext); if (syncext != -1) { /* * We synchronized entire syncext extent, we can mark * it as clean now. */ if (activemap_extent_complete(res->hr_amp, syncext)) (void)hast_activemap_flush(res); } mtx_unlock(&res->hr_amp_lock); if (dorewind) { dorewind = false; if (offset < 0) pjdlog_info("Nodes are in sync."); else { pjdlog_info("Synchronization started. %ju bytes to go.", (uintmax_t)(res->hr_extentsize * activemap_ndirty(res->hr_amp))); } } if (offset < 0) { mtx_lock(&sync_lock); sync_inprogress = false; mtx_unlock(&sync_lock); pjdlog_debug(1, "Nothing to synchronize."); /* * Synchronization complete, make both localcnt and * remotecnt equal. */ ncomp = 1; rw_rlock(&hio_remote_lock[ncomp]); if (ISCONNECTED(res, ncomp)) { if (synced > 0) { pjdlog_info("Synchronization complete. " "%jd bytes synchronized.", (intmax_t)synced); } mtx_lock(&metadata_lock); res->hr_syncsrc = HAST_SYNCSRC_UNDEF; res->hr_primary_localcnt = res->hr_secondary_localcnt; res->hr_primary_remotecnt = res->hr_secondary_remotecnt; pjdlog_debug(1, "Setting localcnt to %ju and remotecnt to %ju.", (uintmax_t)res->hr_primary_localcnt, (uintmax_t)res->hr_secondary_localcnt); (void)metadata_write(res); mtx_unlock(&metadata_lock); } else if (synced > 0) { pjdlog_info("Synchronization interrupted. " "%jd bytes synchronized so far.", (intmax_t)synced); } rw_unlock(&hio_remote_lock[ncomp]); continue; } pjdlog_debug(2, "sync: Taking free request."); QUEUE_TAKE2(hio, free); pjdlog_debug(2, "sync: (%p) Got free request.", hio); /* * Lock the range we are going to synchronize. We don't want * race where someone writes between our read and write. */ for (;;) { mtx_lock(&range_lock); if (rangelock_islocked(range_regular, offset, length)) { pjdlog_debug(2, "sync: Range offset=%jd length=%jd locked.", (intmax_t)offset, (intmax_t)length); range_sync_wait = true; cv_wait(&range_sync_cond, &range_lock); range_sync_wait = false; mtx_unlock(&range_lock); continue; } if (rangelock_add(range_sync, offset, length) < 0) { mtx_unlock(&range_lock); pjdlog_debug(2, "sync: Range offset=%jd length=%jd is already locked, waiting.", (intmax_t)offset, (intmax_t)length); sleep(1); continue; } mtx_unlock(&range_lock); break; } /* * First read the data from synchronization source. */ SYNCREQ(hio); ggio = &hio->hio_ggio; ggio->gctl_cmd = BIO_READ; ggio->gctl_offset = offset; ggio->gctl_length = length; ggio->gctl_error = 0; for (ii = 0; ii < ncomps; ii++) hio->hio_errors[ii] = EINVAL; reqlog(LOG_DEBUG, 2, ggio, "sync: (%p) Sending sync request: ", hio); pjdlog_debug(2, "sync: (%p) Moving request to the send queue.", hio); mtx_lock(&metadata_lock); if (res->hr_syncsrc == HAST_SYNCSRC_PRIMARY) { /* * This range is up-to-date on local component, * so handle request locally. */ /* Local component is 0 for now. */ ncomp = 0; } else /* if (res->hr_syncsrc == HAST_SYNCSRC_SECONDARY) */ { assert(res->hr_syncsrc == HAST_SYNCSRC_SECONDARY); /* * This range is out-of-date on local component, * so send request to the remote node. */ /* Remote component is 1 for now. */ ncomp = 1; } mtx_unlock(&metadata_lock); refcount_init(&hio->hio_countdown, 1); QUEUE_INSERT1(hio, send, ncomp); /* * Let's wait for READ to finish. */ mtx_lock(&sync_lock); while (!ISSYNCREQDONE(hio)) cv_wait(&sync_cond, &sync_lock); mtx_unlock(&sync_lock); if (hio->hio_errors[ncomp] != 0) { pjdlog_error("Unable to read synchronization data: %s.", strerror(hio->hio_errors[ncomp])); goto free_queue; } /* * We read the data from synchronization source, now write it * to synchronization target. */ SYNCREQ(hio); ggio->gctl_cmd = BIO_WRITE; for (ii = 0; ii < ncomps; ii++) hio->hio_errors[ii] = EINVAL; reqlog(LOG_DEBUG, 2, ggio, "sync: (%p) Sending sync request: ", hio); pjdlog_debug(2, "sync: (%p) Moving request to the send queue.", hio); mtx_lock(&metadata_lock); if (res->hr_syncsrc == HAST_SYNCSRC_PRIMARY) { /* * This range is up-to-date on local component, * so we update remote component. */ /* Remote component is 1 for now. */ ncomp = 1; } else /* if (res->hr_syncsrc == HAST_SYNCSRC_SECONDARY) */ { assert(res->hr_syncsrc == HAST_SYNCSRC_SECONDARY); /* * This range is out-of-date on local component, * so we update it. */ /* Local component is 0 for now. */ ncomp = 0; } mtx_unlock(&metadata_lock); pjdlog_debug(2, "sync: (%p) Moving request to the send queues.", hio); refcount_init(&hio->hio_countdown, 1); QUEUE_INSERT1(hio, send, ncomp); /* * Let's wait for WRITE to finish. */ mtx_lock(&sync_lock); while (!ISSYNCREQDONE(hio)) cv_wait(&sync_cond, &sync_lock); mtx_unlock(&sync_lock); if (hio->hio_errors[ncomp] != 0) { pjdlog_error("Unable to write synchronization data: %s.", strerror(hio->hio_errors[ncomp])); goto free_queue; } free_queue: mtx_lock(&range_lock); rangelock_del(range_sync, offset, length); if (range_regular_wait) cv_signal(&range_regular_cond); mtx_unlock(&range_lock); synced += length; pjdlog_debug(2, "sync: (%p) Moving request to the free queue.", hio); QUEUE_INSERT2(hio, free); } /* NOTREACHED */ return (NULL); } static void sighandler(int sig) { bool unlock; switch (sig) { case SIGINT: case SIGTERM: sigexit_received = true; break; default: assert(!"invalid condition"); } /* * XXX: Racy, but if we cannot obtain hio_guard_lock here, we don't * want to risk deadlock. */ unlock = mtx_trylock(&hio_guard_lock); cv_signal(&hio_guard_cond); if (unlock) mtx_unlock(&hio_guard_lock); } /* * Thread guards remote connections and reconnects when needed, handles * signals, etc. */ static void * guard_thread(void *arg) { struct hast_resource *res = arg; struct proto_conn *in, *out; unsigned int ii, ncomps; int timeout; ncomps = HAST_NCOMPONENTS; /* The is only one remote component for now. */ #define ISREMOTE(no) ((no) == 1) for (;;) { if (sigexit_received) { primary_exitx(EX_OK, "Termination signal received, exiting."); } /* * If all the connection will be fine, we will sleep until * someone wakes us up. * If any of the connections will be broken and we won't be * able to connect, we will sleep only for RECONNECT_SLEEP * seconds so we can retry soon. */ timeout = 0; pjdlog_debug(2, "remote_guard: Checking connections."); mtx_lock(&hio_guard_lock); for (ii = 0; ii < ncomps; ii++) { if (!ISREMOTE(ii)) continue; rw_rlock(&hio_remote_lock[ii]); if (ISCONNECTED(res, ii)) { assert(res->hr_remotein != NULL); assert(res->hr_remoteout != NULL); rw_unlock(&hio_remote_lock[ii]); pjdlog_debug(2, "remote_guard: Connection to %s is ok.", res->hr_remoteaddr); } else { assert(res->hr_remotein == NULL); assert(res->hr_remoteout == NULL); /* * Upgrade the lock. It doesn't have to be * atomic as no other thread can change * connection status from disconnected to * connected. */ rw_unlock(&hio_remote_lock[ii]); pjdlog_debug(2, "remote_guard: Reconnecting to %s.", res->hr_remoteaddr); in = out = NULL; if (init_remote(res, &in, &out)) { rw_wlock(&hio_remote_lock[ii]); assert(res->hr_remotein == NULL); assert(res->hr_remoteout == NULL); assert(in != NULL && out != NULL); res->hr_remotein = in; res->hr_remoteout = out; rw_unlock(&hio_remote_lock[ii]); pjdlog_info("Successfully reconnected to %s.", res->hr_remoteaddr); sync_start(); } else { /* Both connections should be NULL. */ assert(res->hr_remotein == NULL); assert(res->hr_remoteout == NULL); assert(in == NULL && out == NULL); pjdlog_debug(2, "remote_guard: Reconnect to %s failed.", res->hr_remoteaddr); timeout = RECONNECT_SLEEP; } } } (void)cv_timedwait(&hio_guard_cond, &hio_guard_lock, timeout); mtx_unlock(&hio_guard_lock); } #undef ISREMOTE /* NOTREACHED */ return (NULL); } Index: user/imp/tbemd/sbin/ifconfig/ifconfig.c =================================================================== --- user/imp/tbemd/sbin/ifconfig/ifconfig.c (revision 206684) +++ user/imp/tbemd/sbin/ifconfig/ifconfig.c (revision 206685) @@ -1,1179 +1,1180 @@ /* * Copyright (c) 1983, 1993 * The Regents of the University of California. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 4. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #ifndef lint static const char copyright[] = "@(#) Copyright (c) 1983, 1993\n\ The Regents of the University of California. All rights reserved.\n"; #endif /* not lint */ #ifndef lint #if 0 static char sccsid[] = "@(#)ifconfig.c 8.2 (Berkeley) 2/16/94"; #endif static const char rcsid[] = "$FreeBSD$"; #endif /* not lint */ #include #include #include #include #include #include #include #include #include #include #include #include /* IP */ #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include "ifconfig.h" /* * Since "struct ifreq" is composed of various union members, callers * should pay special attention to interprete the value. * (.e.g. little/big endian difference in the structure.) */ struct ifreq ifr; char name[IFNAMSIZ]; char *descr = NULL; size_t descrlen = 64; int setaddr; int setmask; int doalias; int clearaddr; int newaddr = 1; int verbose; int noload; int supmedia = 0; int printkeys = 0; /* Print keying material for interfaces. */ static int ifconfig(int argc, char *const *argv, int iscreate, const struct afswtch *afp); static void status(const struct afswtch *afp, const struct sockaddr_dl *sdl, struct ifaddrs *ifa); static void tunnel_status(int s); static void usage(void); static struct afswtch *af_getbyname(const char *name); static struct afswtch *af_getbyfamily(int af); static void af_other_status(int); static struct option *opts = NULL; void opt_register(struct option *p) { p->next = opts; opts = p; } static void usage(void) { char options[1024]; struct option *p; /* XXX not right but close enough for now */ options[0] = '\0'; for (p = opts; p != NULL; p = p->next) { strlcat(options, p->opt_usage, sizeof(options)); strlcat(options, " ", sizeof(options)); } fprintf(stderr, "usage: ifconfig %sinterface address_family [address [dest_address]]\n" " [parameters]\n" " ifconfig interface create\n" " ifconfig -a %s[-d] [-m] [-u] [-v] [address_family]\n" " ifconfig -l [-d] [-u] [address_family]\n" " ifconfig %s[-d] [-m] [-u] [-v]\n", options, options, options); exit(1); } int main(int argc, char *argv[]) { int c, all, namesonly, downonly, uponly; const struct afswtch *afp = NULL; int ifindex; struct ifaddrs *ifap, *ifa; struct ifreq paifr; const struct sockaddr_dl *sdl; char options[1024], *cp, *namecp = NULL; const char *ifname; struct option *p; size_t iflen; all = downonly = uponly = namesonly = noload = verbose = 0; /* Parse leading line options */ strlcpy(options, "adklmnuv", sizeof(options)); for (p = opts; p != NULL; p = p->next) strlcat(options, p->opt, sizeof(options)); while ((c = getopt(argc, argv, options)) != -1) { switch (c) { case 'a': /* scan all interfaces */ all++; break; case 'd': /* restrict scan to "down" interfaces */ downonly++; break; case 'k': printkeys++; break; case 'l': /* scan interface names only */ namesonly++; break; case 'm': /* show media choices in status */ supmedia = 1; break; case 'n': /* suppress module loading */ noload++; break; case 'u': /* restrict scan to "up" interfaces */ uponly++; break; case 'v': verbose++; break; default: for (p = opts; p != NULL; p = p->next) if (p->opt[0] == c) { p->cb(optarg); break; } if (p == NULL) usage(); break; } } argc -= optind; argv += optind; /* -l cannot be used with -a or -m */ if (namesonly && (all || supmedia)) usage(); /* nonsense.. */ if (uponly && downonly) usage(); /* no arguments is equivalent to '-a' */ if (!namesonly && argc < 1) all = 1; /* -a and -l allow an address family arg to limit the output */ if (all || namesonly) { if (argc > 1) usage(); ifname = NULL; ifindex = 0; if (argc == 1) { afp = af_getbyname(*argv); if (afp == NULL) usage(); if (afp->af_name != NULL) argc--, argv++; /* leave with afp non-zero */ } } else { /* not listing, need an argument */ if (argc < 1) usage(); ifname = *argv; argc--, argv++; /* check and maybe load support for this interface */ ifmaybeload(ifname); ifindex = if_nametoindex(ifname); if (ifindex == 0) { /* * NOTE: We must special-case the `create' command * right here as we would otherwise fail when trying * to find the interface. */ if (argc > 0 && (strcmp(argv[0], "create") == 0 || strcmp(argv[0], "plumb") == 0)) { iflen = strlcpy(name, ifname, sizeof(name)); if (iflen >= sizeof(name)) errx(1, "%s: cloning name too long", ifname); ifconfig(argc, argv, 1, NULL); exit(0); } /* * NOTE: We have to special-case the `-vnet' command * right here as we would otherwise fail when trying * to find the interface as it lives in another vnet. */ if (argc > 0 && (strcmp(argv[0], "-vnet") == 0)) { iflen = strlcpy(name, ifname, sizeof(name)); if (iflen >= sizeof(name)) errx(1, "%s: interface name too long", ifname); ifconfig(argc, argv, 0, NULL); exit(0); } errx(1, "interface %s does not exist", ifname); } } /* Check for address family */ if (argc > 0) { afp = af_getbyname(*argv); if (afp != NULL) argc--, argv++; } if (getifaddrs(&ifap) != 0) err(EXIT_FAILURE, "getifaddrs"); cp = NULL; ifindex = 0; for (ifa = ifap; ifa; ifa = ifa->ifa_next) { memset(&paifr, 0, sizeof(paifr)); strncpy(paifr.ifr_name, ifa->ifa_name, sizeof(paifr.ifr_name)); if (sizeof(paifr.ifr_addr) >= ifa->ifa_addr->sa_len) { memcpy(&paifr.ifr_addr, ifa->ifa_addr, ifa->ifa_addr->sa_len); } if (ifname != NULL && strcmp(ifname, ifa->ifa_name) != 0) continue; if (ifa->ifa_addr->sa_family == AF_LINK) sdl = (const struct sockaddr_dl *) ifa->ifa_addr; else sdl = NULL; if (cp != NULL && strcmp(cp, ifa->ifa_name) == 0 && !namesonly) continue; iflen = strlcpy(name, ifa->ifa_name, sizeof(name)); if (iflen >= sizeof(name)) { warnx("%s: interface name too long, skipping", ifa->ifa_name); continue; } cp = ifa->ifa_name; if (downonly && (ifa->ifa_flags & IFF_UP) != 0) continue; if (uponly && (ifa->ifa_flags & IFF_UP) == 0) continue; /* * Are we just listing the interfaces? */ if (namesonly) { if (namecp == cp) continue; if (afp != NULL) { /* special case for "ether" address family */ if (!strcmp(afp->af_name, "ether")) { if (sdl == NULL || sdl->sdl_type != IFT_ETHER || sdl->sdl_alen != ETHER_ADDR_LEN) continue; } else { if (ifa->ifa_addr->sa_family != afp->af_af) continue; } } namecp = cp; ifindex++; if (ifindex > 1) printf(" "); fputs(name, stdout); continue; } ifindex++; if (argc > 0) ifconfig(argc, argv, 0, afp); else status(afp, sdl, ifa); } if (namesonly) printf("\n"); freeifaddrs(ifap); exit(0); } static struct afswtch *afs = NULL; void af_register(struct afswtch *p) { p->af_next = afs; afs = p; } static struct afswtch * af_getbyname(const char *name) { struct afswtch *afp; for (afp = afs; afp != NULL; afp = afp->af_next) if (strcmp(afp->af_name, name) == 0) return afp; return NULL; } static struct afswtch * af_getbyfamily(int af) { struct afswtch *afp; for (afp = afs; afp != NULL; afp = afp->af_next) if (afp->af_af == af) return afp; return NULL; } static void af_other_status(int s) { struct afswtch *afp; uint8_t afmask[howmany(AF_MAX, NBBY)]; memset(afmask, 0, sizeof(afmask)); for (afp = afs; afp != NULL; afp = afp->af_next) { if (afp->af_other_status == NULL) continue; if (afp->af_af != AF_UNSPEC && isset(afmask, afp->af_af)) continue; afp->af_other_status(s); setbit(afmask, afp->af_af); } } static void af_all_tunnel_status(int s) { struct afswtch *afp; uint8_t afmask[howmany(AF_MAX, NBBY)]; memset(afmask, 0, sizeof(afmask)); for (afp = afs; afp != NULL; afp = afp->af_next) { if (afp->af_status_tunnel == NULL) continue; if (afp->af_af != AF_UNSPEC && isset(afmask, afp->af_af)) continue; afp->af_status_tunnel(s); setbit(afmask, afp->af_af); } } static struct cmd *cmds = NULL; void cmd_register(struct cmd *p) { p->c_next = cmds; cmds = p; } static const struct cmd * cmd_lookup(const char *name, int iscreate) { #define N(a) (sizeof(a)/sizeof(a[0])) const struct cmd *p; for (p = cmds; p != NULL; p = p->c_next) if (strcmp(name, p->c_name) == 0) { if (iscreate) { if (p->c_iscloneop) return p; } else { if (!p->c_iscloneop) return p; } } return NULL; #undef N } struct callback { callback_func *cb_func; void *cb_arg; struct callback *cb_next; }; static struct callback *callbacks = NULL; void callback_register(callback_func *func, void *arg) { struct callback *cb; cb = malloc(sizeof(struct callback)); if (cb == NULL) errx(1, "unable to allocate memory for callback"); cb->cb_func = func; cb->cb_arg = arg; cb->cb_next = callbacks; callbacks = cb; } /* specially-handled commands */ static void setifaddr(const char *, int, int, const struct afswtch *); static const struct cmd setifaddr_cmd = DEF_CMD("ifaddr", 0, setifaddr); static void setifdstaddr(const char *, int, int, const struct afswtch *); static const struct cmd setifdstaddr_cmd = DEF_CMD("ifdstaddr", 0, setifdstaddr); static int ifconfig(int argc, char *const *argv, int iscreate, const struct afswtch *uafp) { const struct afswtch *afp, *nafp; const struct cmd *p; struct callback *cb; int s; strncpy(ifr.ifr_name, name, sizeof ifr.ifr_name); afp = uafp != NULL ? uafp : af_getbyname("inet"); top: ifr.ifr_addr.sa_family = afp->af_af == AF_LINK || afp->af_af == AF_UNSPEC ? AF_LOCAL : afp->af_af; if ((s = socket(ifr.ifr_addr.sa_family, SOCK_DGRAM, 0)) < 0 && (uafp != NULL || errno != EPROTONOSUPPORT || (s = socket(AF_LOCAL, SOCK_DGRAM, 0)) < 0)) err(1, "socket(family %u,SOCK_DGRAM", ifr.ifr_addr.sa_family); while (argc > 0) { p = cmd_lookup(*argv, iscreate); if (iscreate && p == NULL) { /* * Push the clone create callback so the new * device is created and can be used for any * remaining arguments. */ cb = callbacks; if (cb == NULL) errx(1, "internal error, no callback"); callbacks = cb->cb_next; cb->cb_func(s, cb->cb_arg); iscreate = 0; /* * Handle any address family spec that * immediately follows and potentially * recreate the socket. */ nafp = af_getbyname(*argv); if (nafp != NULL) { argc--, argv++; if (nafp != afp) { close(s); afp = nafp; goto top; } } /* * Look for a normal parameter. */ continue; } if (p == NULL) { /* * Not a recognized command, choose between setting * the interface address and the dst address. */ p = (setaddr ? &setifdstaddr_cmd : &setifaddr_cmd); } if (p->c_u.c_func || p->c_u.c_func2) { if (p->c_parameter == NEXTARG) { if (argv[1] == NULL) errx(1, "'%s' requires argument", p->c_name); p->c_u.c_func(argv[1], 0, s, afp); argc--, argv++; } else if (p->c_parameter == OPTARG) { p->c_u.c_func(argv[1], 0, s, afp); if (argv[1] != NULL) argc--, argv++; } else if (p->c_parameter == NEXTARG2) { if (argc < 3) errx(1, "'%s' requires 2 arguments", p->c_name); p->c_u.c_func2(argv[1], argv[2], s, afp); argc -= 2, argv += 2; } else p->c_u.c_func(*argv, p->c_parameter, s, afp); } argc--, argv++; } /* * Do any post argument processing required by the address family. */ if (afp->af_postproc != NULL) afp->af_postproc(s, afp); /* * Do deferred callbacks registered while processing * command-line arguments. */ for (cb = callbacks; cb != NULL; cb = cb->cb_next) cb->cb_func(s, cb->cb_arg); /* * Do deferred operations. */ if (clearaddr) { if (afp->af_ridreq == NULL || afp->af_difaddr == 0) { warnx("interface %s cannot change %s addresses!", name, afp->af_name); clearaddr = 0; } } if (clearaddr) { int ret; strncpy(afp->af_ridreq, name, sizeof ifr.ifr_name); ret = ioctl(s, afp->af_difaddr, afp->af_ridreq); if (ret < 0) { if (errno == EADDRNOTAVAIL && (doalias >= 0)) { /* means no previous address for interface */ } else Perror("ioctl (SIOCDIFADDR)"); } } if (newaddr) { if (afp->af_addreq == NULL || afp->af_aifaddr == 0) { warnx("interface %s cannot change %s addresses!", name, afp->af_name); newaddr = 0; } } if (newaddr && (setaddr || setmask)) { strncpy(afp->af_addreq, name, sizeof ifr.ifr_name); if (ioctl(s, afp->af_aifaddr, afp->af_addreq) < 0) Perror("ioctl (SIOCAIFADDR)"); } close(s); return(0); } /*ARGSUSED*/ static void setifaddr(const char *addr, int param, int s, const struct afswtch *afp) { if (afp->af_getaddr == NULL) return; /* * Delay the ioctl to set the interface addr until flags are all set. * The address interpretation may depend on the flags, * and the flags may change when the address is set. */ setaddr++; if (doalias == 0 && afp->af_af != AF_LINK) clearaddr = 1; afp->af_getaddr(addr, (doalias >= 0 ? ADDR : RIDADDR)); } static void settunnel(const char *src, const char *dst, int s, const struct afswtch *afp) { struct addrinfo *srcres, *dstres; int ecode; if (afp->af_settunnel == NULL) { warn("address family %s does not support tunnel setup", afp->af_name); return; } if ((ecode = getaddrinfo(src, NULL, NULL, &srcres)) != 0) errx(1, "error in parsing address string: %s", gai_strerror(ecode)); if ((ecode = getaddrinfo(dst, NULL, NULL, &dstres)) != 0) errx(1, "error in parsing address string: %s", gai_strerror(ecode)); if (srcres->ai_addr->sa_family != dstres->ai_addr->sa_family) errx(1, "source and destination address families do not match"); afp->af_settunnel(s, srcres, dstres); freeaddrinfo(srcres); freeaddrinfo(dstres); } /* ARGSUSED */ static void deletetunnel(const char *vname, int param, int s, const struct afswtch *afp) { if (ioctl(s, SIOCDIFPHYADDR, &ifr) < 0) err(1, "SIOCDIFPHYADDR"); } static void setifvnet(const char *jname, int dummy __unused, int s, const struct afswtch *afp) { struct ifreq my_ifr; memcpy(&my_ifr, &ifr, sizeof(my_ifr)); my_ifr.ifr_jid = jail_getid(jname); if (my_ifr.ifr_jid < 0) errx(1, "%s", jail_errmsg); if (ioctl(s, SIOCSIFVNET, &my_ifr) < 0) err(1, "SIOCSIFVNET"); } static void setifrvnet(const char *jname, int dummy __unused, int s, const struct afswtch *afp) { struct ifreq my_ifr; memcpy(&my_ifr, &ifr, sizeof(my_ifr)); my_ifr.ifr_jid = jail_getid(jname); if (my_ifr.ifr_jid < 0) errx(1, "%s", jail_errmsg); if (ioctl(s, SIOCSIFRVNET, &my_ifr) < 0) err(1, "SIOCSIFRVNET(%d, %s)", my_ifr.ifr_jid, my_ifr.ifr_name); } static void setifnetmask(const char *addr, int dummy __unused, int s, const struct afswtch *afp) { if (afp->af_getaddr != NULL) { setmask++; afp->af_getaddr(addr, MASK); } } static void setifbroadaddr(const char *addr, int dummy __unused, int s, const struct afswtch *afp) { if (afp->af_getaddr != NULL) afp->af_getaddr(addr, DSTADDR); } static void setifipdst(const char *addr, int dummy __unused, int s, const struct afswtch *afp) { const struct afswtch *inet; inet = af_getbyname("inet"); if (inet == NULL) return; inet->af_getaddr(addr, DSTADDR); clearaddr = 0; newaddr = 0; } static void notealias(const char *addr, int param, int s, const struct afswtch *afp) { #define rqtosa(x) (&(((struct ifreq *)(afp->x))->ifr_addr)) if (setaddr && doalias == 0 && param < 0) if (afp->af_addreq != NULL && afp->af_ridreq != NULL) bcopy((caddr_t)rqtosa(af_addreq), (caddr_t)rqtosa(af_ridreq), rqtosa(af_addreq)->sa_len); doalias = param; if (param < 0) { clearaddr = 1; newaddr = 0; } else clearaddr = 0; #undef rqtosa } /*ARGSUSED*/ static void setifdstaddr(const char *addr, int param __unused, int s, const struct afswtch *afp) { if (afp->af_getaddr != NULL) afp->af_getaddr(addr, DSTADDR); } /* * Note: doing an SIOCIGIFFLAGS scribbles on the union portion * of the ifreq structure, which may confuse other parts of ifconfig. * Make a private copy so we can avoid that. */ static void setifflags(const char *vname, int value, int s, const struct afswtch *afp) { struct ifreq my_ifr; int flags; memset(&my_ifr, 0, sizeof(my_ifr)); (void) strlcpy(my_ifr.ifr_name, name, sizeof(my_ifr.ifr_name)); if (ioctl(s, SIOCGIFFLAGS, (caddr_t)&my_ifr) < 0) { Perror("ioctl (SIOCGIFFLAGS)"); exit(1); } flags = (my_ifr.ifr_flags & 0xffff) | (my_ifr.ifr_flagshigh << 16); if (value < 0) { value = -value; flags &= ~value; } else flags |= value; my_ifr.ifr_flags = flags & 0xffff; my_ifr.ifr_flagshigh = flags >> 16; if (ioctl(s, SIOCSIFFLAGS, (caddr_t)&my_ifr) < 0) Perror(vname); } void setifcap(const char *vname, int value, int s, const struct afswtch *afp) { int flags; if (ioctl(s, SIOCGIFCAP, (caddr_t)&ifr) < 0) { Perror("ioctl (SIOCGIFCAP)"); exit(1); } flags = ifr.ifr_curcap; if (value < 0) { value = -value; flags &= ~value; } else flags |= value; flags &= ifr.ifr_reqcap; ifr.ifr_reqcap = flags; if (ioctl(s, SIOCSIFCAP, (caddr_t)&ifr) < 0) Perror(vname); } static void setifmetric(const char *val, int dummy __unused, int s, const struct afswtch *afp) { strncpy(ifr.ifr_name, name, sizeof (ifr.ifr_name)); ifr.ifr_metric = atoi(val); if (ioctl(s, SIOCSIFMETRIC, (caddr_t)&ifr) < 0) warn("ioctl (set metric)"); } static void setifmtu(const char *val, int dummy __unused, int s, const struct afswtch *afp) { strncpy(ifr.ifr_name, name, sizeof (ifr.ifr_name)); ifr.ifr_mtu = atoi(val); if (ioctl(s, SIOCSIFMTU, (caddr_t)&ifr) < 0) warn("ioctl (set mtu)"); } static void setifname(const char *val, int dummy __unused, int s, const struct afswtch *afp) { char *newname; newname = strdup(val); if (newname == NULL) { warn("no memory to set ifname"); return; } ifr.ifr_data = newname; if (ioctl(s, SIOCSIFNAME, (caddr_t)&ifr) < 0) { warn("ioctl (set name)"); free(newname); return; } strlcpy(name, newname, sizeof(name)); free(newname); } /* ARGSUSED */ static void setifdescr(const char *val, int dummy __unused, int s, const struct afswtch *afp) { char *newdescr; ifr.ifr_buffer.length = strlen(val) + 1; if (ifr.ifr_buffer.length == 1) { ifr.ifr_buffer.buffer = newdescr = NULL; ifr.ifr_buffer.length = 0; } else { newdescr = strdup(val); ifr.ifr_buffer.buffer = newdescr; if (newdescr == NULL) { warn("no memory to set ifdescr"); return; } } if (ioctl(s, SIOCSIFDESCR, (caddr_t)&ifr) < 0) warn("ioctl (set descr)"); free(newdescr); } /* ARGSUSED */ static void unsetifdescr(const char *val, int value, int s, const struct afswtch *afp) { setifdescr("", 0, s, 0); } #define IFFBITS \ "\020\1UP\2BROADCAST\3DEBUG\4LOOPBACK\5POINTOPOINT\6SMART\7RUNNING" \ "\10NOARP\11PROMISC\12ALLMULTI\13OACTIVE\14SIMPLEX\15LINK0\16LINK1\17LINK2" \ "\20MULTICAST\22PPROMISC\23MONITOR\24STATICARP" #define IFCAPBITS \ "\020\1RXCSUM\2TXCSUM\3NETCONS\4VLAN_MTU\5VLAN_HWTAGGING\6JUMBO_MTU\7POLLING" \ "\10VLAN_HWCSUM\11TSO4\12TSO6\13LRO\14WOL_UCAST\15WOL_MCAST\16WOL_MAGIC" \ "\21VLAN_HWFILTER\23VLAN_HWTSO\24LINKSTATE" /* * Print the status of the interface. If an address family was * specified, show only it; otherwise, show them all. */ static void status(const struct afswtch *afp, const struct sockaddr_dl *sdl, struct ifaddrs *ifa) { struct ifaddrs *ift; int allfamilies, s; struct ifstat ifs; if (afp == NULL) { allfamilies = 1; ifr.ifr_addr.sa_family = AF_LOCAL; } else { allfamilies = 0; ifr.ifr_addr.sa_family = afp->af_af == AF_LINK ? AF_LOCAL : afp->af_af; } strncpy(ifr.ifr_name, name, sizeof(ifr.ifr_name)); s = socket(ifr.ifr_addr.sa_family, SOCK_DGRAM, 0); if (s < 0) err(1, "socket(family %u,SOCK_DGRAM)", ifr.ifr_addr.sa_family); printf("%s: ", name); printb("flags", ifa->ifa_flags, IFFBITS); if (ioctl(s, SIOCGIFMETRIC, &ifr) != -1) printf(" metric %d", ifr.ifr_metric); if (ioctl(s, SIOCGIFMTU, &ifr) != -1) printf(" mtu %d", ifr.ifr_mtu); putchar('\n'); for (;;) { if ((descr = reallocf(descr, descrlen)) != NULL) { ifr.ifr_buffer.buffer = descr; ifr.ifr_buffer.length = descrlen; if (ioctl(s, SIOCGIFDESCR, &ifr) == 0) { - if (strlen(descr) > 0) - printf("\tdescription: %s\n", descr); - break; - } else if (errno == ENAMETOOLONG) - descrlen = ifr.ifr_buffer.length; - else - break; - } else { + if (ifr.ifr_buffer.buffer == descr) { + if (strlen(descr) > 0) + printf("\tdescription: %s\n", + descr); + } else if (ifr.ifr_buffer.length > descrlen) { + descrlen = ifr.ifr_buffer.length; + continue; + } + } + } else warn("unable to allocate memory for interface" "description"); - break; - } - }; + break; + } if (ioctl(s, SIOCGIFCAP, (caddr_t)&ifr) == 0) { if (ifr.ifr_curcap != 0) { printb("\toptions", ifr.ifr_curcap, IFCAPBITS); putchar('\n'); } if (supmedia && ifr.ifr_reqcap != 0) { printb("\tcapabilities", ifr.ifr_reqcap, IFCAPBITS); putchar('\n'); } } tunnel_status(s); for (ift = ifa; ift != NULL; ift = ift->ifa_next) { if (ift->ifa_addr == NULL) continue; if (strcmp(ifa->ifa_name, ift->ifa_name) != 0) continue; if (allfamilies) { const struct afswtch *p; p = af_getbyfamily(ift->ifa_addr->sa_family); if (p != NULL && p->af_status != NULL) p->af_status(s, ift); } else if (afp->af_af == ift->ifa_addr->sa_family) afp->af_status(s, ift); } #if 0 if (allfamilies || afp->af_af == AF_LINK) { const struct afswtch *lafp; /* * Hack; the link level address is received separately * from the routing information so any address is not * handled above. Cobble together an entry and invoke * the status method specially. */ lafp = af_getbyname("lladdr"); if (lafp != NULL) { info.rti_info[RTAX_IFA] = (struct sockaddr *)sdl; lafp->af_status(s, &info); } } #endif if (allfamilies) af_other_status(s); else if (afp->af_other_status != NULL) afp->af_other_status(s); strncpy(ifs.ifs_name, name, sizeof ifs.ifs_name); if (ioctl(s, SIOCGIFSTATUS, &ifs) == 0) printf("%s", ifs.ascii); close(s); return; } static void tunnel_status(int s) { af_all_tunnel_status(s); } void Perror(const char *cmd) { switch (errno) { case ENXIO: errx(1, "%s: no such interface", cmd); break; case EPERM: errx(1, "%s: permission denied", cmd); break; default: err(1, "%s", cmd); } } /* * Print a value a la the %b format of the kernel's printf */ void printb(const char *s, unsigned v, const char *bits) { int i, any = 0; char c; if (bits && *bits == 8) printf("%s=%o", s, v); else printf("%s=%x", s, v); bits++; if (bits) { putchar('<'); while ((i = *bits++) != '\0') { if (v & (1 << (i-1))) { if (any) putchar(','); any = 1; for (; (c = *bits) > 32; bits++) putchar(c); } else for (; *bits > 32; bits++) ; } putchar('>'); } } void ifmaybeload(const char *name) { #define MOD_PREFIX_LEN 3 /* "if_" */ struct module_stat mstat; int fileid, modid; char ifkind[IFNAMSIZ + MOD_PREFIX_LEN], ifname[IFNAMSIZ], *dp; const char *cp; /* loading suppressed by the user */ if (noload) return; /* trim the interface number off the end */ strlcpy(ifname, name, sizeof(ifname)); for (dp = ifname; *dp != 0; dp++) if (isdigit(*dp)) { *dp = 0; break; } /* turn interface and unit into module name */ strcpy(ifkind, "if_"); strlcpy(ifkind + MOD_PREFIX_LEN, ifname, sizeof(ifkind) - MOD_PREFIX_LEN); /* scan files in kernel */ mstat.version = sizeof(struct module_stat); for (fileid = kldnext(0); fileid > 0; fileid = kldnext(fileid)) { /* scan modules in file */ for (modid = kldfirstmod(fileid); modid > 0; modid = modfnext(modid)) { if (modstat(modid, &mstat) < 0) continue; /* strip bus name if present */ if ((cp = strchr(mstat.name, '/')) != NULL) { cp++; } else { cp = mstat.name; } /* already loaded? */ if (strncmp(ifname, cp, strlen(ifname) + 1) == 0 || strncmp(ifkind, cp, strlen(ifkind) + 1) == 0) return; } } /* not present, we should try to load it */ kldload(ifkind); } static struct cmd basic_cmds[] = { DEF_CMD("up", IFF_UP, setifflags), DEF_CMD("down", -IFF_UP, setifflags), DEF_CMD("arp", -IFF_NOARP, setifflags), DEF_CMD("-arp", IFF_NOARP, setifflags), DEF_CMD("debug", IFF_DEBUG, setifflags), DEF_CMD("-debug", -IFF_DEBUG, setifflags), DEF_CMD_ARG("description", setifdescr), DEF_CMD_ARG("descr", setifdescr), DEF_CMD("-description", 0, unsetifdescr), DEF_CMD("-descr", 0, unsetifdescr), DEF_CMD("promisc", IFF_PPROMISC, setifflags), DEF_CMD("-promisc", -IFF_PPROMISC, setifflags), DEF_CMD("add", IFF_UP, notealias), DEF_CMD("alias", IFF_UP, notealias), DEF_CMD("-alias", -IFF_UP, notealias), DEF_CMD("delete", -IFF_UP, notealias), DEF_CMD("remove", -IFF_UP, notealias), #ifdef notdef #define EN_SWABIPS 0x1000 DEF_CMD("swabips", EN_SWABIPS, setifflags), DEF_CMD("-swabips", -EN_SWABIPS, setifflags), #endif DEF_CMD_ARG("netmask", setifnetmask), DEF_CMD_ARG("metric", setifmetric), DEF_CMD_ARG("broadcast", setifbroadaddr), DEF_CMD_ARG("ipdst", setifipdst), DEF_CMD_ARG2("tunnel", settunnel), DEF_CMD("-tunnel", 0, deletetunnel), DEF_CMD("deletetunnel", 0, deletetunnel), DEF_CMD_ARG("vnet", setifvnet), DEF_CMD_ARG("-vnet", setifrvnet), DEF_CMD("link0", IFF_LINK0, setifflags), DEF_CMD("-link0", -IFF_LINK0, setifflags), DEF_CMD("link1", IFF_LINK1, setifflags), DEF_CMD("-link1", -IFF_LINK1, setifflags), DEF_CMD("link2", IFF_LINK2, setifflags), DEF_CMD("-link2", -IFF_LINK2, setifflags), DEF_CMD("monitor", IFF_MONITOR, setifflags), DEF_CMD("-monitor", -IFF_MONITOR, setifflags), DEF_CMD("staticarp", IFF_STATICARP, setifflags), DEF_CMD("-staticarp", -IFF_STATICARP, setifflags), DEF_CMD("rxcsum", IFCAP_RXCSUM, setifcap), DEF_CMD("-rxcsum", -IFCAP_RXCSUM, setifcap), DEF_CMD("txcsum", IFCAP_TXCSUM, setifcap), DEF_CMD("-txcsum", -IFCAP_TXCSUM, setifcap), DEF_CMD("netcons", IFCAP_NETCONS, setifcap), DEF_CMD("-netcons", -IFCAP_NETCONS, setifcap), DEF_CMD("polling", IFCAP_POLLING, setifcap), DEF_CMD("-polling", -IFCAP_POLLING, setifcap), DEF_CMD("tso", IFCAP_TSO, setifcap), DEF_CMD("-tso", -IFCAP_TSO, setifcap), DEF_CMD("lro", IFCAP_LRO, setifcap), DEF_CMD("-lro", -IFCAP_LRO, setifcap), DEF_CMD("wol", IFCAP_WOL, setifcap), DEF_CMD("-wol", -IFCAP_WOL, setifcap), DEF_CMD("wol_ucast", IFCAP_WOL_UCAST, setifcap), DEF_CMD("-wol_ucast", -IFCAP_WOL_UCAST, setifcap), DEF_CMD("wol_mcast", IFCAP_WOL_MCAST, setifcap), DEF_CMD("-wol_mcast", -IFCAP_WOL_MCAST, setifcap), DEF_CMD("wol_magic", IFCAP_WOL_MAGIC, setifcap), DEF_CMD("-wol_magic", -IFCAP_WOL_MAGIC, setifcap), DEF_CMD("normal", -IFF_LINK0, setifflags), DEF_CMD("compress", IFF_LINK0, setifflags), DEF_CMD("noicmp", IFF_LINK1, setifflags), DEF_CMD_ARG("mtu", setifmtu), DEF_CMD_ARG("name", setifname), }; static __constructor void ifconfig_ctor(void) { #define N(a) (sizeof(a) / sizeof(a[0])) size_t i; for (i = 0; i < N(basic_cmds); i++) cmd_register(&basic_cmds[i]); #undef N } Index: user/imp/tbemd/share/man/man4/Makefile =================================================================== --- user/imp/tbemd/share/man/man4/Makefile (revision 206684) +++ user/imp/tbemd/share/man/man4/Makefile (revision 206685) @@ -1,678 +1,679 @@ # @(#)Makefile 8.1 (Berkeley) 6/18/93 # $FreeBSD$ MAN= aac.4 \ acpi.4 \ ${_acpi_aiboost.4} \ ${_acpi_asus.4} \ ${_acpi_dock.4} \ ${_acpi_fujitsu.4} \ ${_acpi_hp.4} \ ${_acpi_ibm.4} \ ${_acpi_panasonic.4} \ ${_acpi_sony.4} \ acpi_thermal.4 \ ${_acpi_toshiba.4} \ acpi_video.4 \ ${_acpi_wmi.4} \ ada.4 \ adv.4 \ adw.4 \ ae.4 \ age.4 \ agp.4 \ aha.4 \ ahb.4 \ ahc.4 \ ahci.4 \ ahd.4 \ aio.4 \ alc.4 \ ale.4 \ alpm.4 \ altq.4 \ amd.4 \ amdpm.4 \ ${_amdsbwd.4} \ ${_amdsmb.4} \ ${_amdtemp.4} \ amr.4 \ an.4 \ arcmsr.4 \ ${_asmc.4} \ asr.4 \ ata.4 \ atapicam.4 \ ataraid.4 \ ath.4 \ ath_hal.4 \ atkbd.4 \ atkbdc.4 \ ${_atp.4} \ audit.4 \ auditpipe.4 \ aue.4 \ axe.4 \ bce.4 \ bfe.4 \ bge.4 \ bktr.4 \ blackhole.4 \ bpf.4 \ bridge.4 \ bt.4 \ bwi.4 \ bwn.4 \ cardbus.4 \ carp.4 \ cas.4 \ ccd.4 \ cd.4 \ cdce.4 \ ch.4 \ ciss.4 \ cm.4 \ cmx.4 \ coda.4 \ ${_coretemp.4} \ ${_cpuctl.4} \ cpufreq.4 \ crypto.4 \ cue.4 \ cxgb.4 \ cy.4 \ da.4 \ dc.4 \ dcons.4 \ dcons_crom.4 \ ddb.4 \ de.4 \ devctl.4 \ digi.4 \ disc.4 \ divert.4 \ ${_dpms.4} \ dpt.4 \ dummynet.4 \ ed.4 \ edsc.4 \ ef.4 \ ehci.4 \ em.4 \ en.4 \ enc.4 \ epair.4 \ esp.4 \ et.4 \ exca.4 \ faith.4 \ fatm.4 \ fd.4 \ fdc.4 \ firewire.4 \ fpa.4 \ fwe.4 \ fwip.4 \ fwohci.4 \ fxp.4 \ gbde.4 \ gdb.4 \ gem.4 \ geom.4 \ geom_fox.4 \ geom_linux_lvm.4 \ geom_uzip.4 \ gif.4 \ gpib.4 \ gre.4 \ harp.4 \ hatm.4 \ hfa.4 \ hifn.4 \ hme.4 \ ${_hptiop.4} \ ${_hptmv.4} \ ${_hptrr.4} \ hwpmc.4 \ ichsmb.4 \ ${_ichwd.4} \ icmp.4 \ icmp6.4 \ ida.4 \ idt.4 \ ifmib.4 \ igb.4 \ igmp.4 \ iic.4 \ iicbb.4 \ iicbus.4 \ iicsmb.4 \ iir.4 \ inet.4 \ inet6.4 \ intpm.4 \ intro.4 \ ${_io.4} \ ip.4 \ ip6.4 \ ipfirewall.4 \ ${_ipmi.4} \ ips.4 \ ipsec.4 \ ipw.4 \ ipwfw.4 \ iscsi_initiator.4 \ isp.4 \ ispfw.4 \ iwi.4 \ iwifw.4 \ iwn.4 \ iwnfw.4 \ ixgb.4 \ ixgbe.4 \ jme.4 \ joy.4 \ kbdmux.4 \ keyboard.4 \ kld.4 \ ksyms.4 \ ktr.4 \ kue.4 \ lagg.4 \ le.4 \ led.4 \ lge.4 \ ${_lindev.4} \ ${_linux.4} \ lmc.4 \ lo.4 \ lp.4 \ lpbb.4 \ lpt.4 \ mac.4 \ mac_biba.4 \ mac_bsdextended.4 \ mac_ifoff.4 \ mac_lomac.4 \ mac_mls.4 \ mac_none.4 \ mac_partition.4 \ mac_portacl.4 \ mac_seeotheruids.4 \ mac_stub.4 \ mac_test.4 \ malo.4 \ mcd.4 \ md.4 \ mem.4 \ meteor.4 \ mfi.4 \ miibus.4 \ mk48txx.4 \ mld.4 \ mlx.4 \ mly.4 \ mmc.4 \ mmcsd.4 \ mn.4 \ mouse.4 \ mpt.4 \ msk.4 \ mtio.4 \ multicast.4 \ mwl.4 \ mwlfw.4 \ mxge.4 \ my.4 \ natm.4 \ natmip.4 \ ncr.4 \ ncv.4 \ ${_ndis.4} \ net80211.4 \ netgraph.4 \ netintro.4 \ ${_nfe.4} \ ${_nfsmb.4} \ ng_async.4 \ ng_atm.4 \ ngatmbase.4 \ ng_atmllc.4 \ ng_bluetooth.4 \ ng_bpf.4 \ ng_bridge.4 \ ng_bt3c.4 \ ng_btsocket.4 \ ng_car.4 \ ng_ccatm.4 \ ng_cisco.4 \ ng_deflate.4 \ ng_device.4 \ nge.4 \ ng_echo.4 \ ng_eiface.4 \ ng_etf.4 \ ng_ether.4 \ ng_ether_echo.4 \ ng_fec.4 \ ng_frame_relay.4 \ ng_gif.4 \ ng_gif_demux.4 \ ng_h4.4 \ ng_hci.4 \ ng_hole.4 \ ng_hub.4 \ ng_iface.4 \ ng_ipfw.4 \ ng_ip_input.4 \ ng_ksocket.4 \ ng_l2cap.4 \ ng_l2tp.4 \ ng_lmi.4 \ ng_mppc.4 \ ng_nat.4 \ ng_netflow.4 \ ng_one2many.4 \ ng_ppp.4 \ ng_pppoe.4 \ ng_pptpgre.4 \ ng_pred1.4 \ ng_rfc1490.4 \ ng_socket.4 \ ng_source.4 \ ng_split.4 \ ng_sppp.4 \ ng_sscfu.4 \ ng_sscop.4 \ ng_tag.4 \ ng_tcpmss.4 \ ng_tee.4 \ ng_tty.4 \ ng_ubt.4 \ ng_UI.4 \ ng_uni.4 \ ng_vjc.4 \ ng_vlan.4 \ nmdm.4 \ nsp.4 \ null.4 \ ${_nve.4} \ ${_nvram.4} \ ${_nxge.4} \ ohci.4 \ orm.4 \ ${_padlock.4} \ pass.4 \ patm.4 \ pccard.4 \ pccbb.4 \ pcf.4 \ pci.4 \ pcib.4 \ pcic.4 \ pcii.4 \ pcm.4 \ pcn.4 \ pim.4 \ polling.4 \ ppbus.4 \ ppc.4 \ ppi.4 \ psm.4 \ pst.4 \ pt.4 \ pts.4 \ pty.4 \ puc.4 \ ral.4 \ random.4 \ rc.4 \ re.4 \ rl.4 \ rndtest.4 \ route.4 \ rp.4 \ rue.4 \ rum.4 \ run.4 \ sa.4 \ safe.4 \ sbp.4 \ sbp_targ.4 \ scc.4 \ scd.4 \ sched_4bsd.4 \ sched_ule.4 \ screen.4 \ scsi.4 \ sctp.4 \ sdhci.4 \ sem.4 \ ses.4 \ sf.4 \ + sge.4 \ si.4 \ siba.4 \ sio.4 \ siis.4 \ sis.4 \ sk.4 \ smb.4 \ smbus.4 \ smp.4 \ sn.4 \ snd_ad1816.4 \ snd_als4000.4 \ snd_atiixp.4 \ snd_cmi.4 \ snd_cs4281.4 \ snd_csa.4 \ snd_ds1.4 \ snd_emu10k1.4 \ snd_emu10kx.4 \ snd_envy24.4 \ snd_envy24ht.4 \ snd_es137x.4 \ snd_ess.4 \ snd_fm801.4 \ snd_gusc.4 \ snd_hda.4 \ snd_ich.4 \ snd_maestro3.4 \ snd_maestro.4 \ snd_mss.4 \ snd_neomagic.4 \ snd_sbc.4 \ snd_solo.4 \ snd_spicds.4 \ snd_t4dwave.4 \ snd_uaudio.4 \ snd_via8233.4 \ snd_via82c686.4 \ snd_vibes.4 \ snp.4 \ spic.4 \ ${_spkr.4} \ splash.4 \ sppp.4 \ ste.4 \ stf.4 \ stg.4 \ stge.4 \ sym.4 \ syncache.4 \ syncer.4 \ syscons.4 \ sysmouse.4 \ tap.4 \ targ.4 \ tcp.4 \ tdfx.4 \ termios.4 \ textdump.4 \ ti.4 \ tl.4 \ tnt4882.4 \ trm.4 \ tty.4 \ tun.4 \ twa.4 \ twe.4 \ tx.4 \ txp.4 \ u3g.4 \ uark.4 \ uart.4 \ uath.4 \ ubsa.4 \ ubsec.4 \ ubser.4 \ ubtbcmfw.4 \ uchcom.4 \ ucom.4 \ ucycom.4 \ udav.4 \ udbp.4 \ udp.4 \ ufm.4 \ ufoma.4 \ uftdi.4 \ ugen.4 \ uhci.4 \ uhid.4 \ uhso.4 \ uipaq.4 \ ukbd.4 \ ulpt.4 \ umass.4 \ umct.4 \ umodem.4 \ ums.4 \ unix.4 \ upgt.4 \ uplcom.4 \ ural.4 \ urio.4 \ ${_urtw.4} \ usb.4 \ uslcom.4 \ utopia.4 \ uvisor.4 \ uvscom.4 \ vga.4 \ vge.4 \ viapm.4 \ vinum.4 \ vkbd.4 \ vlan.4 \ vpo.4 \ vr.4 \ watchdog.4 \ wb.4 \ wi.4 \ witness.4 \ wlan.4 \ wlan_acl.4 \ wlan_amrr.4 \ wlan_ccmp.4 \ wlan_tkip.4 \ wlan_wep.4 \ wlan_xauth.4 \ ${_wpi.4} \ xe.4 \ xl.4 \ xpt.4 \ zero.4 \ zyd.4 MLINKS= ae.4 if_ae.4 MLINKS+=age.4 if_age.4 MLINKS+=agp.4 agpgart.4 MLINKS+=ale.4 if_ale.4 MLINKS+=altq.4 ALTQ.4 MLINKS+=an.4 if_an.4 MLINKS+=ata.4 acd.4 \ ata.4 ad.4 \ ata.4 afd.4 \ ata.4 ast.4 MLINKS+=aue.4 if_aue.4 MLINKS+=axe.4 if_axe.4 MLINKS+=bce.4 if_bce.4 MLINKS+=bfe.4 if_bfe.4 MLINKS+=bge.4 if_bge.4 MLINKS+=bktr.4 brooktree.4 MLINKS+=bridge.4 if_bridge.4 MLINKS+=bwi.4 if_bwi.4 MLINKS+=cas.4 if_cas.4 MLINKS+=crypto.4 cryptodev.4 MLINKS+=cue.4 if_cue.4 MLINKS+=dc.4 if_dc.4 MLINKS+=de.4 if_de.4 MLINKS+=disc.4 if_disc.4 MLINKS+=ed.4 if_ed.4 MLINKS+=edsc.4 if_edsc.4 MLINKS+=ef.4 if_ef.4 MLINKS+=em.4 if_em.4 MLINKS+=en.4 if_en.4 MLINKS+=enc.4 if_enc.4 MLINKS+=epair.4 if_epair.4 MLINKS+=et.4 if_et.4 MLINKS+=faith.4 if_faith.4 MLINKS+=fatm.4 if_fatm.4 MLINKS+=fd.4 stderr.4 \ fd.4 stdin.4 \ fd.4 stdout.4 MLINKS+=firewire.4 ieee1394.4 MLINKS+=fpa.4 fea.4 MLINKS+=fwe.4 if_fwe.4 MLINKS+=fwip.4 if_fwip.4 MLINKS+=fxp.4 if_fxp.4 MLINKS+=gem.4 if_gem.4 MLINKS+=geom.4 GEOM.4 MLINKS+=gif.4 if_gif.4 MLINKS+=gre.4 if_gre.4 MLINKS+=hatm.4 if_hatm.4 MLINKS+=hme.4 if_hme.4 MLINKS+=${_hptrr.4} ${_rr232x.4} MLINKS+=idt.4 if_idt.4 MLINKS+=igb.4 if_igb.4 MLINKS+=ip.4 rawip.4 MLINKS+=ipfirewall.4 ipaccounting.4 \ ipfirewall.4 ipacct.4 \ ipfirewall.4 ipfw.4 MLINKS+=ipw.4 if_ipw.4 MLINKS+=iwi.4 if_iwi.4 MLINKS+=iwn.4 if_iwn.4 MLINKS+=ixgb.4 if_ixgb.4 MLINKS+=ixgbe.4 if_ixgbe.4 MLINKS+=jme.4 if_jme.4 MLINKS+=kue.4 if_kue.4 MLINKS+=lagg.4 trunk.4 MLINKS+=le.4 if_le.4 MLINKS+=lge.4 if_lge.4 MLINKS+=lo.4 loop.4 MLINKS+=lp.4 plip.4 MLINKS+=malo.4 if_malo.4 MLINKS+=md.4 vn.4 MLINKS+=mem.4 kmem.4 MLINKS+=mn.4 if_mn.4 MLINKS+=mxge.4 if_mxge.4 MLINKS+=my.4 if_my.4 MLINKS+=${_ndis.4} ${_if_ndis.4} MLINKS+=netintro.4 net.4 \ netintro.4 networking.4 MLINKS+=${_nfe.4} ${_if_nfe.4} MLINKS+=nge.4 if_nge.4 MLINKS+=${_nve.4} ${_if_nve.4} MLINKS+=${_nxge.4} ${_if_nxge.4} MLINKS+=patm.4 if_patm.4 MLINKS+=pccbb.4 cbb.4 MLINKS+=pcm.4 snd.4 \ pcm.4 sound.4 MLINKS+=pcn.4 if_pcn.4 MLINKS+=ral.4 if_ral.4 MLINKS+=re.4 if_re.4 MLINKS+=rl.4 if_rl.4 MLINKS+=rue.4 if_rue.4 MLINKS+=rum.4 if_rum.4 MLINKS+=scsi.4 CAM.4 \ scsi.4 cam.4 \ scsi.4 scbus.4 \ scsi.4 SCSI.4 MLINKS+=sf.4 if_sf.4 MLINKS+=sis.4 if_sis.4 MLINKS+=sk.4 if_sk.4 MLINKS+=smp.4 SMP.4 MLINKS+=sn.4 if_sn.4 MLINKS+=snd_envy24.4 snd_ak452x.4 MLINKS+=snd_sbc.4 snd_sb16.4 \ snd_sbc.4 snd_sb8.4 MLINKS+=${_spkr.4} ${_speaker.4} MLINKS+=splash.4 screensaver.4 MLINKS+=ste.4 if_ste.4 MLINKS+=stf.4 if_stf.4 MLINKS+=stge.4 if_stge.4 MLINKS+=syncache.4 syncookies.4 MLINKS+=syscons.4 sc.4 MLINKS+=tap.4 if_tap.4 MLINKS+=tdfx.4 tdfx_linux.4 MLINKS+=ti.4 if_ti.4 MLINKS+=tl.4 if_tl.4 MLINKS+=tun.4 if_tun.4 MLINKS+=tx.4 if_tx.4 MLINKS+=txp.4 if_txp.4 MLINKS+=u3g.4 u3gstub.4 MLINKS+=uath.4 if_uath.4 MLINKS+=udav.4 if_udav.4 MLINKS+=upgt.4 if_upgt.4 MLINKS+=ural.4 if_ural.4 MLINKS+=${_urtw.4} ${_if_urtw.4} MLINKS+=vge.4 if_vge.4 MLINKS+=vlan.4 if_vlan.4 MLINKS+=vpo.4 imm.4 MLINKS+=vr.4 if_vr.4 MLINKS+=watchdog.4 SW_WATCHDOG.4 MLINKS+=wb.4 if_wb.4 MLINKS+=wi.4 if_wi.4 MLINKS+=${_wpi.4} ${_if_wpi.4} MLINKS+=xe.4 if_xe.4 MLINKS+=xl.4 if_xl.4 MLINKS+=zyd.4 if_zyd.4 .if ${MACHINE_CPUARCH} == "amd64" || ${MACHINE_CPUARCH} == "i386" _acpi_aiboost.4=acpi_aiboost.4 _acpi_asus.4= acpi_asus.4 _acpi_dock.4= acpi_dock.4 _acpi_fujitsu.4=acpi_fujitsu.4 _acpi_hp.4= acpi_hp.4 _acpi_ibm.4= acpi_ibm.4 _acpi_panasonic.4=acpi_panasonic.4 _acpi_sony.4= acpi_sony.4 _acpi_toshiba.4=acpi_toshiba.4 _acpi_wmi.4= acpi_wmi.4 _amdsbwd.4= amdsbwd.4 _amdsmb.4= amdsmb.4 _amdtemp.4= amdtemp.4 _asmc.4= asmc.4 _atp.4= atp.4 _coretemp.4= coretemp.4 _cpuctl.4= cpuctl.4 _dpms.4= dpms.4 _hptiop.4= hptiop.4 _hptmv.4= hptmv.4 _hptrr.4= hptrr.4 _ichwd.4= ichwd.4 _if_ndis.4= if_ndis.4 _if_nfe.4= if_nfe.4 _if_nve.4= if_nve.4 _if_nxge.4= if_nxge.4 _if_urtw.4= if_urtw.4 _if_wpi.4= if_wpi.4 _ipmi.4= ipmi.4 _io.4= io.4 _lindev.4= lindev.4 _linux.4= linux.4 _ndis.4= ndis.4 _nfe.4= nfe.4 _nfsmb.4= nfsmb.4 _nve.4= nve.4 _nvram.4= nvram.4 _nxge.4= nxge.4 _padlock.4= padlock.4 _rr232x.4= rr232x.4 _speaker.4= speaker.4 _spkr.4= spkr.4 _urtw.4= urtw.4 _wpi.4= wpi.4 MLINKS+=lindev.4 full.4 .endif .if ${MACHINE_CPUARCH} == "powerpc" _atp.4= atp.4 .endif .if exists(${.CURDIR}/man4.${MACHINE_CPUARCH}) SUBDIR= man4.${MACHINE_CPUARCH} .endif .include Index: user/imp/tbemd/share/man/man4/altq.4 =================================================================== --- user/imp/tbemd/share/man/man4/altq.4 (revision 206684) +++ user/imp/tbemd/share/man/man4/altq.4 (revision 206685) @@ -1,184 +1,185 @@ .\" .\" Copyright (c) 2004 Max Laier .\" All rights reserved. .\" .\" Redistribution and use in source and binary forms, with or without .\" modification, are permitted provided that the following conditions .\" are met: .\" 1. Redistributions of source code must retain the above copyright .\" notice, this list of conditions and the following disclaimer. .\" 2. Redistributions in binary form must reproduce the above copyright .\" notice, this list of conditions and the following disclaimer in the .\" documentation and/or other materials provided with the distribution. .\" .\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND .\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE .\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE .\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE .\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL .\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS .\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) .\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT .\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY .\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF .\" SUCH DAMAGE. .\" .\" $FreeBSD$ .\" -.Dd July 26, 2009 +.Dd April 14, 2010 .Dt ALTQ 4 .Os .Sh NAME .Nm ALTQ .Nd "alternate queuing of network packets" .Sh SYNOPSIS .Cd options ALTQ .Pp .Cd options ALTQ_CBQ .Cd options ALTQ_RED .Cd options ALTQ_RIO .Cd options ALTQ_HFSC .Cd options ALTQ_CDNR .Cd options ALTQ_PRIQ .Sh DESCRIPTION The .Nm framework provides several disciplines for queuing outgoing network packets. This is done by modifications to the interface packet queues. See .Xr altq 9 for details. .Pp The user interface for .Nm is implemented by the .Xr pfctl 8 utility, so please refer to the .Xr pfctl 8 and the .Xr pf.conf 5 manpages for a complete description of the .Nm capabilities and how to use it. .Ss Kernel Options The following options in the kernel configuration file are related to .Nm operation: .Pp .Bl -tag -width ".Dv ALTQ_DEBUG" -compact .It Dv ALTQ Enable .Nm . .It Dv ALTQ_CBQ Build the .Dq "Class Based Queuing" discipline. .It Dv ALTQ_RED Build the .Dq "Random Early Detection" extension. .It Dv ALTQ_RIO Build .Dq "Random Early Drop" for input and output. .It Dv ALTQ_HFSC Build the .Dq "Hierarchical Packet Scheduler" discipline. .It Dv ALTQ_CDNR Build the traffic conditioner. This option is meaningless at the moment as the conditioner is not used by any of the available disciplines or consumers. .It Dv ALTQ_PRIQ Build the .Dq "Priority Queuing" discipline. .It Dv ALTQ_NOPCC Required if the TSC is unusable. .It Dv ALTQ_DEBUG Enable additional debugging facilities. .El .Pp Note that .Nm Ns -disciplines cannot be loaded as kernel modules. In order to use a certain discipline you have to build it into a custom kernel. The .Xr pf 4 interface, that is required for the configuration process of .Nm can be loaded as a module. .Sh SUPPORTED DEVICES The driver modifications described in .Xr altq 9 are required to use a certain network card with .Nm . They have been applied to the following hardware drivers: .Xr age 4 , .Xr alc 4 , .Xr ale 4 , .Xr an 4 , .Xr ath 4 , .Xr aue 4 , .Xr axe 4 , .Xr bce 4 , .Xr bfe 4 , .Xr bge 4 , .Xr cas 4 , .Xr dc 4 , .Xr de 4 , .Xr ed 4 , .Xr em 4 , .Xr ep 4 , .Xr epair 4 , .Xr fxp 4 , .Xr gem 4 , .Xr hme 4 , .Xr ipw 4 , .Xr iwi 4 , .Xr jme 4 , .Xr le 4 , .Xr msk 4 , .Xr mxge 4 , .Xr my 4 , .Xr nfe 4 , .Xr nge 4 , .Xr npe 4 , .Xr nve 4 , .Xr ral 4 , .Xr re 4 , .Xr rl 4 , .Xr rum 4 , .Xr sf 4 , +.Xr sge 4 , .Xr sis 4 , .Xr sk 4 , .Xr ste 4 , .Xr stge 4 , .Xr txp 4 , .Xr udav 4 , .Xr ural 4 , .Xr vge 4 , .Xr vr 4 , .Xr wi 4 , and .Xr xl 4 . .Pp The .Xr ndis 4 framework also has support for .Nm and thus all encapsulated drivers. .Pp The .Xr tun 4 and .Xr ng_iface 4 pseudo drivers also do support .Nm . .Sh SEE ALSO .Xr pf 4 , .Xr pf.conf 5 , .Xr ipfw 8 , .Xr pfctl 8 , .Xr altq 9 Index: user/imp/tbemd/share/man/man4/ktr.4 =================================================================== --- user/imp/tbemd/share/man/man4/ktr.4 (revision 206684) +++ user/imp/tbemd/share/man/man4/ktr.4 (revision 206685) @@ -1,200 +1,206 @@ .\" Copyright (c) 2001 John H. Baldwin .\" All rights reserved. .\" .\" Redistribution and use in source and binary forms, with or without .\" modification, are permitted provided that the following conditions .\" are met: .\" 1. Redistributions of source code must retain the above copyright .\" notice, this list of conditions and the following disclaimer. .\" 2. Redistributions in binary form must reproduce the above copyright .\" notice, this list of conditions and the following disclaimer in the .\" documentation and/or other materials provided with the distribution. .\" .\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND .\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE .\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE .\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE .\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL .\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS .\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) .\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT .\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY .\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF .\" SUCH DAMAGE. .\" .\" $FreeBSD$ .\" .Dd January 25, 2005 .Dt KTR 4 .Os .Sh NAME .Nm ktr .Nd kernel tracing facility .Sh SYNOPSIS .Cd options KTR .Cd options ALQ .Cd options KTR_ALQ .Cd options KTR_COMPILE=(KTR_LOCK|KTR_INTR|KTR_PROC) .Cd options KTR_CPUMASK=0x3 .Cd options KTR_ENTRIES=8192 .Cd options KTR_MASK=(KTR_INTR|KTR_PROC) .Cd options KTR_VERBOSE .Sh DESCRIPTION The .Nm facility allows kernel events to be logged while the kernel executes so that they can be examined later when debugging. The only mandatory option to enable .Nm is .Dq Li options KTR . .Pp The .Dv KTR_ENTRIES option sets the size of the buffer of events. It must be a power of two. The size of the buffer in the currently running kernel can be found via the read-only sysctl .Va debug.ktr.entries . By default the buffer contains 1024 entries. .Ss Event Masking Event levels can be enabled or disabled to trim excessive and overly verbose logging. First, a mask of events is specified at compile time via the .Dv KTR_COMPILE option to limit which events are actually compiled into the kernel. The default value for this option is for all events to be enabled. .Pp Secondly, the actual events logged while the kernel runs can be further masked via the run time event mask. The .Dv KTR_MASK option sets the default value of the run time event mask. The runtime event mask can also be set by the .Xr loader 8 via the .Va debug.ktr.mask environment variable. It can also be examined and set after booting via the .Va debug.ktr.mask sysctl. By default the run time mask is set to log only .Dv KTR_GEN events. The definitions of the event mask bits can be found in .In sys/ktr.h . .Pp Furthermore, there is a CPU event mask whose default value can be changed via the .Dv KTR_CPUMASK option. A CPU must have the bit corresponding to its logical id set in this bitmask for events that occur on it to be logged. This mask can be set by the .Xr loader 8 via the .Va debug.ktr.cpumask environment variable. It can also be examined and set after booting via the .Va debug.ktr.cpumask sysctl. By default events on all CPUs are enabled. .Ss Verbose Mode By default, events are only logged to the internal buffer for examination later, but if the verbose flag is set then they are dumped to the kernel console as well. This flag can also be set from the loader via the .Va debug.ktr.verbose environment variable, or it can be examined and set after booting via the .Va debug.ktr.verbose sysctl. If the flag is set to zero, which is the default, then verbose output is disabled. If the flag is set to one, then the contents of the log message and the CPU number are printed to the kernel console. If the flag is greater than one, then the filename and line number of the event are output to the console in addition to the log message and the CPU number. The .Dv KTR_VERBOSE option sets the flag to one. .Ss Examining the Events The KTR buffer can be examined from within .Xr ddb 4 via the -.Ic show ktr Op Cm /v +.Ic show ktr Op Cm /vV command. This command displays the contents of the trace buffer one page at a time. At the .Dq Li --more-- prompt, the Enter key displays one more entry and prompts again. The spacebar displays another page of entries. Any other key quits. By default the timestamp, filename, and line number are not displayed with each log entry. If the .Cm /v modifier is specified, then they are displayed in addition to the normal output. +If the +.Cm /V +modifier is specified, then just the timestamp is displayed in +addition to the normal output. Note that the events are displayed in reverse chronological order. That is, the most recent events are displayed first. .Ss Logging ktr to Disk The .Dv KTR_ALQ option can be used to log .Nm entries to disk for post analysis using the .Xr ktrdump 8 utility. This option depends on the .Dv ALQ option. Due to the potentially high volume of trace messages the trace mask should be selected carefully. This feature is configured through a group of sysctls. .Bl -tag -width ".Va debug.ktr.alq_enable" .It Va debug.ktr.alq_file displays or sets the file that .Nm will log to. By default its value is .Pa /tmp/ktr.out . If the file name is changed while .Nm is enabled it will not take effect until the next invocation. .It Va debug.ktr.alq_enable enables logging of .Nm entries to disk if it is set to one. -Setting this to 0 will terminate logging. +Setting this to 0 will terminate logging to disk and revert to +logging to the normal ktr ring buffer. +Data is not sent to the ring buffer while logging to disk. .It Va debug.ktr.alq_max is the maximum number of entries that will be recorded to disk, or 0 for infinite. This is helpful for limiting the number of particularly high frequency entries that are recorded. .It Va debug.ktr.alq_depth determines the number of entries in the write buffer. This is the buffer that holds entries before they are written to disk and defaults to the value of the .Dv KTR_ENTRIES option. .It Va debug.ktr.alq_failed records the number of times we failed to write an entry due to overflowing the write buffer. This may happen if the frequency of the logged .Nm messages outpaces the depth of the queue. .It Va debug.ktr.alq_cnt records the number of entries that have currently been written to disk. .El .Sh SEE ALSO .Xr ktrdump 8 , .Xr alq 9 , .Xr ktr 9 .Sh HISTORY The KTR kernel tracing facility first appeared in .Bsx 3.0 and was imported into .Fx 5.0 . Index: user/imp/tbemd/share/man/man4/miibus.4 =================================================================== --- user/imp/tbemd/share/man/man4/miibus.4 (revision 206684) +++ user/imp/tbemd/share/man/man4/miibus.4 (revision 206685) @@ -1,181 +1,184 @@ .\" Written by Tom Rhodes for the FreeBSD Project. .\" Please see the /usr/src/COPYRIGHT file for copyright information. .\" .\" This document takes information from the IEEE 802.3 Standard .\" along with various comments from Peter Wemm, Robert Watson, and Bill Paul. .\" Originally this file looked much like the NetBSD mii(4) manual page, but .\" I doubt you would ever notice due to large differences. .\" .\" $FreeBSD$ .\" -.Dd June 14, 2009 +.Dd April 14, 2010 .Dt MIIBUS 4 .Os .Sh NAME .Nm miibus .Nd IEEE 802.3 Media Independent Interface network bus .Sh SYNOPSIS For most network interface cards (NIC): .Cd "device miibus" .Sh DESCRIPTION The .Nm driver provides an interconnection between the Media Access Control (MAC) sublayer, the Physical Layer entities (PHY), Station Management (STA) entities, and the PHY Layer as defined by the IEEE 802.3 Standard. .Pp The .Nm layer allows network device drivers to share common support code for various external PHY devices. Most 10/100 network interface cards either use an MII transceiver or have built-in transceivers that can be programmed using an MII interface. The .Nm driver currently handles all of the media detection, selection, and reporting using the ifmedia interface. A generic driver has been included for all PHYs that are not handled by a specific driver, this is possible because all 10/100 PHYs implement the same general register set along with their vendor specific register set. .Pp The following network device drivers use the .Nm interface: .Pp .Bl -tag -compact -width ".Xr fxp 4" .It Xr age 4 Attansic/Atheros L1 Gigabit Ethernet .It Xr alc 4 Atheros AR8131/AR8132 PCIe Ethernet .It Xr ale 4 Atheros AR8121/AR8113/AR8114 PCIe Ethernet .It Xr aue 4 ADMtek USB Ethernet .It Xr axe 4 ASIX Electronics AX88172 USB Ethernet .It Xr bce 4 Broadcom NetXtreme II Gigabit Ethernet .It Xr bfe 4 Broadcom BCM4401 Ethernet .It Xr bge 4 Broadcom BCM570xx Gigabit Ethernet .It Xr cas 4 Sun Cassini/Cassini+ and National Semiconductor DP83065 Saturn .It Xr dc 4 DEC/Intel 21143 and various workalikes .It Xr ed 4 NE[12]000, SMC Ultra, 3c503, DS8390 cards .It Xr et 4 Agere ET1310 Gigabit Ethernet .It Xr fxp 4 Intel EtherExpress PRO/100B .It Xr gem 4 Sun ERI, Sun GEM and Apple GMAC Ethernet .It Xr hme 4 Sun HME Ethernet .It Xr jme 4 JMicron JMC250 Gigabit/JMC260 Fast Ethernet .It Xr lge 4 Level 1 LXT1001 NetCellerator Gigabit Ethernet .It Xr msk 4 Marvell/SysKonnect Yukon II Gigabit Ethernet .It Xr nfe 4 NVIDIA nForce MCP Networking Adapter .It Xr nge 4 National Semiconductor DP83820/DP83821 Gigabit Ethernet .It Xr nve 4 NVIDIA nForce MCP Networking Adapter .It Xr pcn 4 AMD Am79C97x PCI 10/100 .It Xr re 4 RealTek 8139C+/8169/8169S/8110S .It Xr rl 4 RealTek 8129/8139 .It Xr rue 4 RealTek RTL8150 USB To Fast Ethernet .It Xr sf 4 Adaptec AIC-6915 +.It Xr sge 4 +Silicon Integrated Systems SiS190/191 Ethernet .It Xr sis 4 Silicon Integrated Systems SiS 900/SiS 7016 .It Xr sk 4 SysKonnect SK-984x and SK-982x Gigabit Ethernet .It Xr ste 4 Sundance ST201 (D-Link DFE-550TX) .It Xr stge 4 Sundance/Tamarack TC9021 Gigabit Ethernet .It Xr tl 4 Texas Instruments ThunderLAN .It Xr tx 4 SMC EtherPower II (83c170) .It Xr udav 4 Davicom DM9601 USB Ethernet .It Xr vge 4 VIA VT612x PCI Gigabit Ethernet .It Xr vr 4 VIA Rhine, Rhine II .It Xr wb 4 Winbond W89C840F .It Xr xl 4 3Com 3c90x .El .Sh COMPATIBILITY The implementation of .Nm was originally intended to have similar API interfaces to .Bsx 3.0 and .Nx , but as a result are not well behaved newbus device drivers. .Sh SEE ALSO .Xr age 4 , .Xr alc 4 , .Xr ale 4 , .Xr arp 4 , .Xr aue 4 , .Xr axe 4 , .Xr bce 4 , .Xr bfe 4 , .Xr bge 4 , .Xr cas 4 , .Xr dc 4 , .Xr ed 4 , .Xr et 4 , .Xr fxp 4 , .Xr gem 4 , .Xr hme 4 , .Xr jme 4 , .Xr lge 4 , .Xr msk 4 , .Xr netintro 4 , .Xr nfe 4 , .Xr nge 4 , .Xr nve 4 , .Xr pcn 4 , .Xr re 4 , .Xr rl 4 , .Xr rue 4 , .Xr sf 4 , +.Xr sge 4 , .Xr sis 4 , .Xr sk 4 , .Xr ste 4 , .Xr stge 4 , .Xr tl 4 , .Xr tx 4 , .Xr udav 4 , .Xr vge 4 , .Xr vr 4 , .Xr wb 4 , .Xr xl 4 .Sh STANDARDS More information on MII can be found in the IEEE 802.3 Standard. .Sh HISTORY The .Nm driver first appeared in .Fx 3.3 . .Sh AUTHORS This manual page was written by .An Tom Rhodes Aq trhodes@FreeBSD.org . Index: user/imp/tbemd/share/man/man4/netintro.4 =================================================================== --- user/imp/tbemd/share/man/man4/netintro.4 (revision 206684) +++ user/imp/tbemd/share/man/man4/netintro.4 (revision 206685) @@ -1,454 +1,457 @@ .\" Copyright (c) 1983, 1990, 1991, 1993 .\" The Regents of the University of California. All rights reserved. .\" .\" Redistribution and use in source and binary forms, with or without .\" modification, are permitted provided that the following conditions .\" are met: .\" 1. Redistributions of source code must retain the above copyright .\" notice, this list of conditions and the following disclaimer. .\" 2. Redistributions in binary form must reproduce the above copyright .\" notice, this list of conditions and the following disclaimer in the .\" documentation and/or other materials provided with the distribution. .\" 3. All advertising materials mentioning features or use of this software .\" must display the following acknowledgement: .\" This product includes software developed by the University of .\" California, Berkeley and its contributors. .\" 4. Neither the name of the University nor the names of its contributors .\" may be used to endorse or promote products derived from this software .\" without specific prior written permission. .\" .\" THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND .\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE .\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE .\" ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE .\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL .\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS .\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) .\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT .\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY .\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF .\" SUCH DAMAGE. .\" .\" @(#)netintro.4 8.2 (Berkeley) 11/30/93 .\" $FreeBSD$ .\" -.Dd January 26, 2010 +.Dd April 14, 2010 .Dt NETINTRO 4 .Os .Sh NAME .Nm networking .Nd introduction to networking facilities .Sh SYNOPSIS .In sys/types.h .In sys/time.h .In sys/socket.h .In net/if.h .In net/route.h .Sh DESCRIPTION This section is a general introduction to the networking facilities available in the system. Documentation in this part of section 4 is broken up into three areas: .Em protocol families (domains), .Em protocols , and .Em network interfaces . .Pp All network protocols are associated with a specific .Em protocol family . A protocol family provides basic services to the protocol implementation to allow it to function within a specific network environment. These services may include packet fragmentation and reassembly, routing, addressing, and basic transport. A protocol family may support multiple methods of addressing, though the current protocol implementations do not. A protocol family is normally comprised of a number of protocols, one per .Xr socket 2 type. It is not required that a protocol family support all socket types. A protocol family may contain multiple protocols supporting the same socket abstraction. .Pp A protocol supports one of the socket abstractions detailed in .Xr socket 2 . A specific protocol may be accessed either by creating a socket of the appropriate type and protocol family, or by requesting the protocol explicitly when creating a socket. Protocols normally accept only one type of address format, usually determined by the addressing structure inherent in the design of the protocol family/network architecture. Certain semantics of the basic socket abstractions are protocol specific. All protocols are expected to support the basic model for their particular socket type, but may, in addition, provide non-standard facilities or extensions to a mechanism. For example, a protocol supporting the .Dv SOCK_STREAM abstraction may allow more than one byte of out-of-band data to be transmitted per out-of-band message. .Pp A network interface is similar to a device interface. Network interfaces comprise the lowest layer of the networking subsystem, interacting with the actual transport hardware. An interface may support one or more protocol families and/or address formats. The SYNOPSIS section of each network interface entry gives a sample specification of the related drivers for use in providing a system description to the .Xr config 8 program. The DIAGNOSTICS section lists messages which may appear on the console and/or in the system error log, .Pa /var/log/messages (see .Xr syslogd 8 ) , due to errors in device operation. .Sh PROTOCOLS The system currently supports the Internet protocols, the Xerox Network Systems(tm) protocols, and some of the .Tn ISO OSI protocols. Raw socket interfaces are provided to the .Tn IP protocol layer of the Internet, and to the .Tn IDP protocol of Xerox .Tn NS . Consult the appropriate manual pages in this section for more information regarding the support for each protocol family. .Sh ADDRESSING Associated with each protocol family is an address format. All network addresses adhere to a general structure, called a sockaddr, described below. However, each protocol imposes finer and more specific structure, generally renaming the variant, which is discussed in the protocol family manual page alluded to above. .Bd -literal -offset indent struct sockaddr { u_char sa_len; u_char sa_family; char sa_data[14]; }; .Ed .Pp The field .Va sa_len contains the total length of the structure, which may exceed 16 bytes. The following address values for .Va sa_family are known to the system (and additional formats are defined for possible future implementation): .Bd -literal #define AF_UNIX 1 /* local to host (pipes, portals) */ #define AF_INET 2 /* internetwork: UDP, TCP, etc. */ #define AF_NS 6 /* Xerox NS protocols */ #define AF_CCITT 10 /* CCITT protocols, X.25 etc */ #define AF_HYLINK 15 /* NSC Hyperchannel */ #define AF_ISO 18 /* ISO protocols */ .Ed .Sh ROUTING .Fx provides some packet routing facilities. The kernel maintains a routing information database, which is used in selecting the appropriate network interface when transmitting packets. .Pp A user process (or possibly multiple co-operating processes) maintains this database by sending messages over a special kind of socket. This supplants fixed size .Xr ioctl 2 used in earlier releases. .Pp This facility is described in .Xr route 4 . .Sh INTERFACES Each network interface in a system corresponds to a path through which messages may be sent and received. A network interface usually has a hardware device associated with it, though certain interfaces such as the loopback interface, .Xr lo 4 , do not. .Pp The following .Xr ioctl 2 calls may be used to manipulate network interfaces. The .Fn ioctl is made on a socket (typically of type .Dv SOCK_DGRAM ) in the desired domain. Most of the requests supported in earlier releases take an .Vt ifreq structure as its parameter. This structure has the form .Bd -literal struct ifreq { #define IFNAMSIZ 16 char ifr_name[IFNAMSIZ]; /* if name, e.g. "en0" */ union { struct sockaddr ifru_addr; struct sockaddr ifru_dstaddr; struct sockaddr ifru_broadaddr; struct ifreq_buffer ifru_buffer; short ifru_flags[2]; short ifru_index; int ifru_metric; int ifru_mtu; int ifru_phys; int ifru_media; caddr_t ifru_data; int ifru_cap[2]; } ifr_ifru; #define ifr_addr ifr_ifru.ifru_addr /* address */ #define ifr_dstaddr ifr_ifru.ifru_dstaddr /* other end of p-to-p link */ #define ifr_broadaddr ifr_ifru.ifru_broadaddr /* broadcast address */ #define ifr_buffer ifr_ifru.ifru_buffer /* user supplied buffer with its length */ #define ifr_flags ifr_ifru.ifru_flags[0] /* flags (low 16 bits) */ #define ifr_flagshigh ifr_ifru.ifru_flags[1] /* flags (high 16 bits) */ #define ifr_metric ifr_ifru.ifru_metric /* metric */ #define ifr_mtu ifr_ifru.ifru_mtu /* mtu */ #define ifr_phys ifr_ifru.ifru_phys /* physical wire */ #define ifr_media ifr_ifru.ifru_media /* physical media */ #define ifr_data ifr_ifru.ifru_data /* for use by interface */ #define ifr_reqcap ifr_ifru.ifru_cap[0] /* requested capabilities */ #define ifr_curcap ifr_ifru.ifru_cap[1] /* current capabilities */ #define ifr_index ifr_ifru.ifru_index /* interface index */ }; .Ed .Pp Calls which are now deprecated are: .Bl -tag -width SIOCGIFBRDADDR .It Dv SIOCSIFADDR Set interface address for protocol family. Following the address assignment, the .Dq initialization routine for the interface is called. .It Dv SIOCSIFDSTADDR Set point to point address for protocol family and interface. .It Dv SIOCSIFBRDADDR Set broadcast address for protocol family and interface. .El .Pp .Fn Ioctl requests to obtain addresses and requests both to set and retrieve other data are still fully supported and use the .Vt ifreq structure: .Bl -tag -width SIOCGIFBRDADDR .It Dv SIOCGIFADDR Get interface address for protocol family. .It Dv SIOCGIFDSTADDR Get point to point address for protocol family and interface. .It Dv SIOCGIFBRDADDR Get broadcast address for protocol family and interface. .It Dv SIOCSIFCAP Attempt to set the enabled capabilities field for the interface to the value of the .Va ifr_reqcap field of the .Vt ifreq structure. Note that, depending on the particular interface features, some capabilities may appear hard-coded to enabled, or toggling a capability may affect the status of other ones. The supported capabilities field is read-only, and the .Va ifr_curcap field is unused by this call. .It Dv SIOCGIFCAP Get the interface capabilities fields. The values for supported and enabled capabilities will be returned in the .Va ifr_reqcap and .Va ifr_curcap fields of the .Vt ifreq structure, respectively. .It Dv SIOCGIFDESCR Get the interface description, returned in the .Va buffer field of .Va ifru_buffer struct. The user supplied buffer length should be defined in the .Va length field of .Va ifru_buffer struct passed in as parameter, and the length would include the terminating nul character. If there is not enough space to hold the interface length, -no copy would be done and an -error would be returned. +no copy would be done and the +.Va buffer +field of +.Va ifru_buffer +would be set to NULL. The kernel will store the buffer length in the .Va length field upon return, regardless whether the buffer itself is sufficient to hold the data. .It Dv SIOCSIFDESCR Set the interface description to the value of the .Va buffer field of .Va ifru_buffer struct, with .Va length field specifying its length (counting the terminating nul). .It Dv SIOCSIFFLAGS Set interface flags field. If the interface is marked down, any processes currently routing packets through the interface are notified; some interfaces may be reset so that incoming packets are no longer received. When marked up again, the interface is reinitialized. .It Dv SIOCGIFFLAGS Get interface flags. .It Dv SIOCSIFMETRIC Set interface routing metric. The metric is used only by user-level routers. .It Dv SIOCGIFMETRIC Get interface metric. .It Dv SIOCIFCREATE Attempt to create the specified interface. If the interface name is given without a unit number the system will attempt to create a new interface with an arbitrary unit number. On successful return the .Va ifr_name field will contain the new interface name. .It Dv SIOCIFDESTROY Attempt to destroy the specified interface. .El .Pp There are two requests that make use of a new structure: .Bl -tag -width SIOCGIFBRDADDR .It Dv SIOCAIFADDR An interface may have more than one address associated with it in some protocols. This request provides a means to add additional addresses (or modify characteristics of the primary address if the default address for the address family is specified). Rather than making separate calls to set destination or broadcast addresses, or network masks (now an integral feature of multiple protocols) a separate structure is used to specify all three facets simultaneously (see below). One would use a slightly tailored version of this struct specific to each family (replacing each sockaddr by one of the family-specific type). Where the sockaddr itself is larger than the default size, one needs to modify the .Fn ioctl identifier itself to include the total size, as described in .Fn ioctl . .It Dv SIOCDIFADDR This requests deletes the specified address from the list associated with an interface. It also uses the .Vt ifaliasreq structure to allow for the possibility of protocols allowing multiple masks or destination addresses, and also adopts the convention that specification of the default address means to delete the first address for the interface belonging to the address family in which the original socket was opened. .It Dv SIOCGIFCONF Get interface configuration list. This request takes an .Vt ifconf structure (see below) as a value-result parameter. The .Va ifc_len field should be initially set to the size of the buffer pointed to by .Va ifc_buf . On return it will contain the length, in bytes, of the configuration list. .It Dv SIOCIFGCLONERS Get list of clonable interfaces. This request takes an .Vt if_clonereq structure (see below) as a value-result parameter. The .Va ifcr_count field should be set to the number of .Dv IFNAMSIZ sized strings that can be fit in the buffer pointed to by .Va ifcr_buffer . On return, .Va ifcr_total will be set to the number of clonable interfaces and the buffer pointed to by .Va ifcr_buffer will be filled with the names of clonable interfaces aligned on .Dv IFNAMSIZ boundaries. .El .Bd -literal /* * Structure used in SIOCAIFCONF request. */ struct ifaliasreq { char ifra_name[IFNAMSIZ]; /* if name, e.g. "en0" */ struct sockaddr ifra_addr; struct sockaddr ifra_broadaddr; struct sockaddr ifra_mask; }; .Ed .Pp .Bd -literal /* * Structure used in SIOCGIFCONF request. * Used to retrieve interface configuration * for machine (useful for programs which * must know all networks accessible). */ struct ifconf { int ifc_len; /* size of associated buffer */ union { caddr_t ifcu_buf; struct ifreq *ifcu_req; } ifc_ifcu; #define ifc_buf ifc_ifcu.ifcu_buf /* buffer address */ #define ifc_req ifc_ifcu.ifcu_req /* array of structures returned */ }; .Ed .Pp .Bd -literal /* Structure used in SIOCIFGCLONERS request. */ struct if_clonereq { int ifcr_total; /* total cloners (out) */ int ifcr_count; /* room for this many in user buffer */ char *ifcr_buffer; /* buffer for cloner names */ }; .Ed .Bd -literal /* Structure used in SIOCGIFDESCR and SIOCSIFDESCR requests */ struct ifreq_buffer { size_t length; /* length of the buffer */ void *buffer; /* pointer to userland space buffer */ }; .Ed .Sh SEE ALSO .Xr ioctl 2 , .Xr socket 2 , .Xr intro 4 , .Xr config 8 , .Xr routed 8 , .Xr ifnet 9 .Sh HISTORY The .Nm netintro manual appeared in .Bx 4.3 tahoe . Index: user/imp/tbemd/share/man/man4/sge.4 =================================================================== --- user/imp/tbemd/share/man/man4/sge.4 (nonexistent) +++ user/imp/tbemd/share/man/man4/sge.4 (revision 206685) @@ -0,0 +1,120 @@ +.\" Copyright (c) 2010 Pyun YongHyeon +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions +.\" are met: +.\" 1. Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" 2. Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND +.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +.\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE +.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS +.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) +.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT +.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY +.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF +.\" SUCH DAMAGE. +.\" +.\" $FreeBSD$ +.\" +.Dd April 14, 2010 +.Dt SGE 4 +.Os +.Sh NAME +.Nm sge +.Nd Silicon Integrated Systems SiS190/191 Fast/Gigabit Ethernet driver +.Sh SYNOPSIS +To compile this driver into the kernel, +place the following lines in your +kernel configuration file: +.Bd -ragged -offset indent +.Cd "device miibus" +.Cd "device sge" +.Ed +.Pp +Alternatively, to load the driver as a +module at boot time, place the following line in +.Xr loader.conf 5 : +.Bd -literal -offset indent +if_sge="YES" +.Ed +.Sh DESCRIPTION +The +.Nm +device driver provides support for SiS190 Fast Ethernet +controllers and SiS191 Fast/Gigabit Ethernet controllers. +.Pp +All LOMs supported by the +.Nm +driver have TCP/UDP/IP checksum offload for transmit and receive. +Due to lack of documentation more offloading features like TCP +segmentation offload (TSO), hardware VLAN tag stripping/insertion +features, Wake On Lan (WOL), Jumbo frame and an interrupt moderation +mechanism are not supported yet. +.Pp +The +.Nm +driver supports the following media types: +.Bl -tag -width ".Cm 10baseT/UTP" +.It Cm autoselect +Enable autoselection of the media type and options. +The user can manually override +the autoselected mode by adding media options to +.Xr rc.conf 5 . +.It Cm 10baseT/UTP +Set 10Mbps operation. +.It Cm 100baseTX +Set 100Mbps (Fast Ethernet) operation. +.It Cm 1000baseTX +Set 1000baseTX operation over twisted pair. +.El +.Pp +The +.Nm +driver supports the following media options: +.Bl -tag -width ".Cm full-duplex" +.It Cm full-duplex +Force full duplex operation. +.It Cm half-duplex +Force half duplex operation. +.El +.Pp +For more information on configuring this device, see +.Xr ifconfig 8 . +.Sh HARDWARE +The +.Nm +device driver provides support for the following Ethernet controllers: +.Pp +.Bl -bullet -compact +.It +SiS190 Fast Ethernet controller +.It +SiS191 Fast/Gigabit Ethernet controller +.El +.Sh SEE ALSO +.Xr altq 4 , +.Xr arp 4 , +.Xr miibus 4 , +.Xr netintro 4 , +.Xr ng_ether 4 , +.Xr vlan 4 , +.Xr ifconfig 8 +.Sh HISTORY +The +.Nm +driver was written by +.An Alexander Pohoyda +.Aq alexander.pohoyda@gmx.net . +And enhanced by +.An Nikolay Denev +.Aq ndenev@gmail.com . +It first appeared in +.Fx 8.1 . Property changes on: user/imp/tbemd/share/man/man4/sge.4 ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/imp/tbemd/share/man/man4/vlan.4 =================================================================== --- user/imp/tbemd/share/man/man4/vlan.4 (revision 206684) +++ user/imp/tbemd/share/man/man4/vlan.4 (revision 206685) @@ -1,200 +1,201 @@ .\" .\" Copyright (c) 2001 Yar Tikhiy .\" All rights reserved. .\" .\" Redistribution and use in source and binary forms, with or without .\" modification, are permitted provided that the following conditions .\" are met: .\" 1. Redistributions of source code must retain the above copyright .\" notice, this list of conditions and the following disclaimer. .\" 2. Redistributions in binary form must reproduce the above copyright .\" notice, this list of conditions and the following disclaimer in the .\" documentation and/or other materials provided with the distribution. .\" .\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND .\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE .\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE .\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE .\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL .\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS .\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) .\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT .\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY .\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF .\" SUCH DAMAGE. .\" .\" $FreeBSD$ .\" -.Dd June 14, 2009 +.Dd April 14, 2010 .Dt VLAN 4 .Os .Sh NAME .Nm vlan .Nd "IEEE 802.1Q VLAN network interface" .Sh SYNOPSIS To compile this driver into the kernel, place the following lines in your kernel configuration file: .Bd -ragged -offset indent .Cd "device vlan" .Ed .Pp Alternatively, to load the driver as a module at boot time, place the following line in .Xr loader.conf 5 : .Bd -literal -offset indent if_vlan_load="YES" .Ed .Sh DESCRIPTION The .Nm driver demultiplexes frames tagged according to the IEEE 802.1Q standard into logical .Nm network interfaces, which allows routing/bridging between multiple VLANs through a single switch trunk port. .Pp Each .Nm interface is created at runtime using interface cloning. This is most easily done with the .Xr ifconfig 8 .Cm create command or using the .Va cloned_interfaces variable in .Xr rc.conf 5 . .Pp To function, a .Nm interface must be assigned a parent interface and numeric VLAN tag using .Xr ifconfig 8 . A single parent can be assigned to multiple .Nm interfaces provided they have different tags. The parent interface is likely to be an Ethernet card connected to a properly configured switch port. The VLAN tag should match one of those set up in the switched network. .Pp Initially .Nm assumes the same minimum length for tagged and untagged frames. This mode is selected by the .Xr sysctl 8 variable .Va net.link.vlan.soft_pad set to 0 (default). However, there are network devices that fail to adjust frame length, should it fall below the allowed minimum due to untagging. Such devices should be able to interoperate with .Nm after changing the value of .Va net.link.vlan.soft_pad to 1. In the latter mode, .Nm will pad short frames before tagging them so that their length stays not less than the minimum value after untagging by the non-compliant devices. .Sh HARDWARE The .Nm driver supports efficient operation over parent interfaces that can provide help in processing VLANs. Such interfaces are automatically recognized by their capabilities. Depending on the level of sophistication found in a physical interface, it may do full VLAN processing or just be able to receive and transmit long frames (up to 1522 bytes including an Ethernet header and FCS). The capabilities may be user-controlled by the respective parameters to .Xr ifconfig 8 , .Cm vlanhwtag and .Cm vlanmtu . However, a physical interface is not obliged to react to them: It may have either capability enabled permanently without a way to turn it off. The whole issue is very specific to a particular device and its driver. .Pp By now, the list of physical interfaces able of full VLAN processing in the hardware is limited to the following devices: .Xr ae 4 , .Xr age 4 , .Xr alc 4 , .Xr ale 4 , .Xr bce 4 , .Xr bge 4 , .Xr cxgb 4 , .Xr em 4 , .Xr ixgb 4 , .Xr jme 4 , .Xr msk 4 , .Xr nge 4 , .Xr re 4 , .Xr stge 4 , .Xr ti 4 , .Xr txp 4 , and .Xr vge 4 . .Pp The rest of the Ethernet interfaces can run VLANs using software emulation in the .Nm driver. However, some of them lack the capability of transmitting and receiving long frames. Assigning such an interface as the parent to .Nm will result in a reduced MTU on the corresponding .Nm interfaces. In the modern Internet, this is likely to cause .Xr tcp 4 connectivity problems due to massive, inadequate .Xr icmp 4 filtering that breaks the Path MTU Discovery mechanism. .Pp The following interfaces support long frames for .Nm natively: .Xr bfe 4 , .Xr cas 4 , .Xr dc 4 , .Xr fwe 4 , .Xr fxp 4 , .Xr gem 4 , .Xr hme 4 , .Xr le 4 , .Xr nfe 4 , .Xr nve 4 , .Xr rl 4 , .Xr sf 4 , +.Xr sge 4 , .Xr sis 4 , .Xr sk 4 , .Xr ste 4 , .Xr tl 4 , .Xr tx 4 , .Xr vr 4 , and .Xr xl 4 . .Pp The .Nm driver automatically recognizes devices that natively support long frames for .Nm use and calculates the appropriate frame MTU based on the capabilities of the parent interface. Some other interfaces not listed above may handle long frames, but they do not advertise this ability of theirs. The MTU setting on .Nm can be corrected manually if used in conjunction with such a parent interface. .Sh SEE ALSO .Xr ifconfig 8 , .Xr sysctl 8 .Sh BUGS No 802.1Q features except VLAN tagging are implemented. Index: user/imp/tbemd/sys/cam/scsi/scsi_cd.c =================================================================== --- user/imp/tbemd/sys/cam/scsi/scsi_cd.c (revision 206684) +++ user/imp/tbemd/sys/cam/scsi/scsi_cd.c (revision 206685) @@ -1,4236 +1,4237 @@ /*- * Copyright (c) 1997 Justin T. Gibbs. * Copyright (c) 1997, 1998, 1999, 2000, 2001, 2002, 2003 Kenneth D. Merry. * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions, and the following disclaimer, * without modification, immediately at the beginning of the file. * 2. The name of the author may not be used to endorse or promote products * derived from this software without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE FOR * ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ /*- * Portions of this driver taken from the original FreeBSD cd driver. * Written by Julian Elischer (julian@tfs.com) * for TRW Financial Systems for use under the MACH(2.5) operating system. * * TRW Financial Systems, in accordance with their agreement with Carnegie * Mellon University, makes this software available to CMU to distribute * or use in any manner that they see fit as long as this message is kept with * the software. For this reason TFS also grants any other persons or * organisations permission to use or modify this software. * * TFS supplies this software to be publicly redistributed * on the understanding that TFS is not responsible for the correct * functioning of this software in any circumstances. * * Ported to run under 386BSD by Julian Elischer (julian@tfs.com) Sept 1992 * * from: cd.c,v 1.83 1997/05/04 15:24:22 joerg Exp $ */ #include __FBSDID("$FreeBSD$"); #include "opt_cd.h" #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #define LEADOUT 0xaa /* leadout toc entry */ struct cd_params { u_int32_t blksize; u_long disksize; }; typedef enum { CD_Q_NONE = 0x00, CD_Q_NO_TOUCH = 0x01, CD_Q_BCD_TRACKS = 0x02, CD_Q_NO_CHANGER = 0x04, CD_Q_CHANGER = 0x08, CD_Q_10_BYTE_ONLY = 0x10 } cd_quirks; typedef enum { CD_FLAG_INVALID = 0x0001, CD_FLAG_NEW_DISC = 0x0002, CD_FLAG_DISC_LOCKED = 0x0004, CD_FLAG_DISC_REMOVABLE = 0x0008, CD_FLAG_TAGGED_QUEUING = 0x0010, CD_FLAG_CHANGER = 0x0040, CD_FLAG_ACTIVE = 0x0080, CD_FLAG_SCHED_ON_COMP = 0x0100, CD_FLAG_RETRY_UA = 0x0200, CD_FLAG_VALID_MEDIA = 0x0400, CD_FLAG_VALID_TOC = 0x0800, CD_FLAG_SCTX_INIT = 0x1000, CD_FLAG_OPEN = 0x2000 } cd_flags; typedef enum { CD_CCB_PROBE = 0x01, CD_CCB_BUFFER_IO = 0x02, CD_CCB_WAITING = 0x03, CD_CCB_TYPE_MASK = 0x0F, CD_CCB_RETRY_UA = 0x10 } cd_ccb_state; typedef enum { CHANGER_TIMEOUT_SCHED = 0x01, CHANGER_SHORT_TMOUT_SCHED = 0x02, CHANGER_MANUAL_CALL = 0x04, CHANGER_NEED_TIMEOUT = 0x08 } cd_changer_flags; #define ccb_state ppriv_field0 #define ccb_bp ppriv_ptr1 struct cd_tocdata { struct ioc_toc_header header; struct cd_toc_entry entries[100]; }; struct cd_toc_single { struct ioc_toc_header header; struct cd_toc_entry entry; }; typedef enum { CD_STATE_PROBE, CD_STATE_NORMAL } cd_state; struct cd_softc { cam_pinfo pinfo; cd_state state; volatile cd_flags flags; struct bio_queue_head bio_queue; LIST_HEAD(, ccb_hdr) pending_ccbs; struct cd_params params; union ccb saved_ccb; cd_quirks quirks; STAILQ_ENTRY(cd_softc) changer_links; struct cdchanger *changer; int bufs_left; struct cam_periph *periph; int minimum_command_size; int outstanding_cmds; struct task sysctl_task; struct sysctl_ctx_list sysctl_ctx; struct sysctl_oid *sysctl_tree; STAILQ_HEAD(, cd_mode_params) mode_queue; struct cd_tocdata toc; struct disk *disk; }; struct cd_page_sizes { int page; int page_size; }; static struct cd_page_sizes cd_page_size_table[] = { { AUDIO_PAGE, sizeof(struct cd_audio_page)} }; struct cd_quirk_entry { struct scsi_inquiry_pattern inq_pat; cd_quirks quirks; }; /* * The changer quirk entries aren't strictly necessary. Basically, what * they do is tell cdregister() up front that a device is a changer. * Otherwise, it will figure that fact out once it sees a LUN on the device * that is greater than 0. If it is known up front that a device is a changer, * all I/O to the device will go through the changer scheduling routines, as * opposed to the "normal" CD code. * * NOTE ON 10_BYTE_ONLY quirks: Any 10_BYTE_ONLY quirks MUST be because * your device hangs when it gets a 10 byte command. Adding a quirk just * to get rid of the informative diagnostic message is not acceptable. All * 10_BYTE_ONLY quirks must be documented in full in a PR (which should be * referenced in a comment along with the quirk) , and must be approved by * ken@FreeBSD.org. Any quirks added that don't adhere to this policy may * be removed until the submitter can explain why they are needed. * 10_BYTE_ONLY quirks will be removed (as they will no longer be necessary) * when the CAM_NEW_TRAN_CODE work is done. */ static struct cd_quirk_entry cd_quirk_table[] = { { { T_CDROM, SIP_MEDIA_REMOVABLE, "NRC", "MBR-7", "*"}, /*quirks*/ CD_Q_CHANGER }, { { T_CDROM, SIP_MEDIA_REMOVABLE, "PIONEER", "CD-ROM DRM*", "*"}, /* quirks */ CD_Q_CHANGER }, { { T_CDROM, SIP_MEDIA_REMOVABLE, "NAKAMICH", "MJ-*", "*"}, /* quirks */ CD_Q_CHANGER }, { { T_CDROM, SIP_MEDIA_REMOVABLE, "CHINON", "CD-ROM CDS-535","*"}, /* quirks */ CD_Q_BCD_TRACKS } }; static disk_open_t cdopen; static disk_close_t cdclose; static disk_ioctl_t cdioctl; static disk_strategy_t cdstrategy; static periph_init_t cdinit; static periph_ctor_t cdregister; static periph_dtor_t cdcleanup; static periph_start_t cdstart; static periph_oninv_t cdoninvalidate; static void cdasync(void *callback_arg, u_int32_t code, struct cam_path *path, void *arg); static int cdcmdsizesysctl(SYSCTL_HANDLER_ARGS); static void cdshorttimeout(void *arg); static void cdschedule(struct cam_periph *periph, int priority); static void cdrunchangerqueue(void *arg); static void cdchangerschedule(struct cd_softc *softc); static int cdrunccb(union ccb *ccb, int (*error_routine)(union ccb *ccb, u_int32_t cam_flags, u_int32_t sense_flags), u_int32_t cam_flags, u_int32_t sense_flags); static union ccb *cdgetccb(struct cam_periph *periph, u_int32_t priority); static void cddone(struct cam_periph *periph, union ccb *start_ccb); static union cd_pages *cdgetpage(struct cd_mode_params *mode_params); static int cdgetpagesize(int page_num); static void cdprevent(struct cam_periph *periph, int action); static int cdcheckmedia(struct cam_periph *periph); static int cdsize(struct cam_periph *periph, u_int32_t *size); static int cd6byteworkaround(union ccb *ccb); static int cderror(union ccb *ccb, u_int32_t cam_flags, u_int32_t sense_flags); static int cdreadtoc(struct cam_periph *periph, u_int32_t mode, u_int32_t start, u_int8_t *data, u_int32_t len, u_int32_t sense_flags); static int cdgetmode(struct cam_periph *periph, struct cd_mode_params *data, u_int32_t page); static int cdsetmode(struct cam_periph *periph, struct cd_mode_params *data); static int cdplay(struct cam_periph *periph, u_int32_t blk, u_int32_t len); static int cdreadsubchannel(struct cam_periph *periph, u_int32_t mode, u_int32_t format, int track, struct cd_sub_channel_info *data, u_int32_t len); static int cdplaymsf(struct cam_periph *periph, u_int32_t startm, u_int32_t starts, u_int32_t startf, u_int32_t endm, u_int32_t ends, u_int32_t endf); static int cdplaytracks(struct cam_periph *periph, u_int32_t strack, u_int32_t sindex, u_int32_t etrack, u_int32_t eindex); static int cdpause(struct cam_periph *periph, u_int32_t go); static int cdstopunit(struct cam_periph *periph, u_int32_t eject); static int cdstartunit(struct cam_periph *periph, int load); static int cdsetspeed(struct cam_periph *periph, u_int32_t rdspeed, u_int32_t wrspeed); static int cdreportkey(struct cam_periph *periph, struct dvd_authinfo *authinfo); static int cdsendkey(struct cam_periph *periph, struct dvd_authinfo *authinfo); static int cdreaddvdstructure(struct cam_periph *periph, struct dvd_struct *dvdstruct); static struct periph_driver cddriver = { cdinit, "cd", TAILQ_HEAD_INITIALIZER(cddriver.units), /* generation */ 0 }; PERIPHDRIVER_DECLARE(cd, cddriver); #ifndef CD_DEFAULT_RETRY #define CD_DEFAULT_RETRY 4 #endif #ifndef CHANGER_MIN_BUSY_SECONDS #define CHANGER_MIN_BUSY_SECONDS 5 #endif #ifndef CHANGER_MAX_BUSY_SECONDS #define CHANGER_MAX_BUSY_SECONDS 15 #endif static int cd_retry_count = CD_DEFAULT_RETRY; static int changer_min_busy_seconds = CHANGER_MIN_BUSY_SECONDS; static int changer_max_busy_seconds = CHANGER_MAX_BUSY_SECONDS; SYSCTL_NODE(_kern_cam, OID_AUTO, cd, CTLFLAG_RD, 0, "CAM CDROM driver"); SYSCTL_NODE(_kern_cam_cd, OID_AUTO, changer, CTLFLAG_RD, 0, "CD Changer"); SYSCTL_INT(_kern_cam_cd, OID_AUTO, retry_count, CTLFLAG_RW, &cd_retry_count, 0, "Normal I/O retry count"); TUNABLE_INT("kern.cam.cd.retry_count", &cd_retry_count); SYSCTL_INT(_kern_cam_cd_changer, OID_AUTO, min_busy_seconds, CTLFLAG_RW, &changer_min_busy_seconds, 0, "Minimum changer scheduling quantum"); TUNABLE_INT("kern.cam.cd.changer.min_busy_seconds", &changer_min_busy_seconds); SYSCTL_INT(_kern_cam_cd_changer, OID_AUTO, max_busy_seconds, CTLFLAG_RW, &changer_max_busy_seconds, 0, "Maximum changer scheduling quantum"); TUNABLE_INT("kern.cam.cd.changer.max_busy_seconds", &changer_max_busy_seconds); struct cdchanger { path_id_t path_id; target_id_t target_id; int num_devices; struct camq devq; struct timeval start_time; struct cd_softc *cur_device; struct callout short_handle; struct callout long_handle; volatile cd_changer_flags flags; STAILQ_ENTRY(cdchanger) changer_links; STAILQ_HEAD(chdevlist, cd_softc) chluns; }; static struct mtx changerq_mtx; static STAILQ_HEAD(changerlist, cdchanger) changerq; static int num_changers; MALLOC_DEFINE(M_SCSICD, "scsi_cd", "scsi_cd buffers"); static void cdinit(void) { cam_status status; mtx_init(&changerq_mtx, "cdchangerq", "SCSI CD Changer List", MTX_DEF); STAILQ_INIT(&changerq); /* * Install a global async callback. This callback will * receive async callbacks like "new device found". */ status = xpt_register_async(AC_FOUND_DEVICE, cdasync, NULL, NULL); if (status != CAM_REQ_CMP) { printf("cd: Failed to attach master async callback " "due to status 0x%x!\n", status); } } static void cdoninvalidate(struct cam_periph *periph) { struct cd_softc *softc; softc = (struct cd_softc *)periph->softc; /* * De-register any async callbacks. */ xpt_register_async(0, cdasync, periph, periph->path); softc->flags |= CD_FLAG_INVALID; /* * Return all queued I/O with ENXIO. * XXX Handle any transactions queued to the card * with XPT_ABORT_CCB. */ bioq_flush(&softc->bio_queue, NULL, ENXIO); /* * If this device is part of a changer, and it was scheduled * to run, remove it from the run queue since we just nuked * all of its scheduled I/O. */ if ((softc->flags & CD_FLAG_CHANGER) && (softc->pinfo.index != CAM_UNQUEUED_INDEX)) camq_remove(&softc->changer->devq, softc->pinfo.index); disk_gone(softc->disk); xpt_print(periph->path, "lost device\n"); } static void cdcleanup(struct cam_periph *periph) { struct cd_softc *softc; softc = (struct cd_softc *)periph->softc; xpt_print(periph->path, "removing device entry\n"); /* * In the queued, non-active case, the device in question * has already been removed from the changer run queue. Since this * device is active, we need to de-activate it, and schedule * another device to run. (if there is another one to run) */ if ((softc->flags & CD_FLAG_CHANGER) && (softc->flags & CD_FLAG_ACTIVE)) { /* * The purpose of the short timeout is soley to determine * whether the current device has finished or not. Well, * since we're removing the active device, we know that it * is finished. So, get rid of the short timeout. * Otherwise, if we're in the time period before the short * timeout fires, and there are no other devices in the * queue to run, there won't be any other device put in the * active slot. i.e., when we call cdrunchangerqueue() * below, it won't do anything. Then, when the short * timeout fires, it'll look at the "current device", which * we are free below, and possibly panic the kernel on a * bogus pointer reference. * * The long timeout doesn't really matter, since we * decrement the qfrozen_cnt to indicate that there is * nothing in the active slot now. Therefore, there won't * be any bogus pointer references there. */ if (softc->changer->flags & CHANGER_SHORT_TMOUT_SCHED) { callout_stop(&softc->changer->short_handle); softc->changer->flags &= ~CHANGER_SHORT_TMOUT_SCHED; } softc->changer->devq.qfrozen_cnt[0]--; softc->changer->flags |= CHANGER_MANUAL_CALL; cdrunchangerqueue(softc->changer); } /* * If we're removing the last device on the changer, go ahead and * remove the changer device structure. */ if ((softc->flags & CD_FLAG_CHANGER) && (--softc->changer->num_devices == 0)) { /* * Theoretically, there shouldn't be any timeouts left, but * I'm not completely sure that that will be the case. So, * it won't hurt to check and see if there are any left. */ if (softc->changer->flags & CHANGER_TIMEOUT_SCHED) { callout_stop(&softc->changer->long_handle); softc->changer->flags &= ~CHANGER_TIMEOUT_SCHED; } if (softc->changer->flags & CHANGER_SHORT_TMOUT_SCHED) { callout_stop(&softc->changer->short_handle); softc->changer->flags &= ~CHANGER_SHORT_TMOUT_SCHED; } mtx_lock(&changerq_mtx); STAILQ_REMOVE(&changerq, softc->changer, cdchanger, changer_links); num_changers--; mtx_unlock(&changerq_mtx); xpt_print(periph->path, "removing changer entry\n"); free(softc->changer, M_DEVBUF); } cam_periph_unlock(periph); if ((softc->flags & CD_FLAG_SCTX_INIT) != 0 && sysctl_ctx_free(&softc->sysctl_ctx) != 0) { xpt_print(periph->path, "can't remove sysctl context\n"); } disk_destroy(softc->disk); free(softc, M_DEVBUF); cam_periph_lock(periph); } static void cdasync(void *callback_arg, u_int32_t code, struct cam_path *path, void *arg) { struct cam_periph *periph; periph = (struct cam_periph *)callback_arg; switch (code) { case AC_FOUND_DEVICE: { struct ccb_getdev *cgd; cam_status status; cgd = (struct ccb_getdev *)arg; if (cgd == NULL) break; if (cgd->protocol != PROTO_SCSI) break; if (SID_TYPE(&cgd->inq_data) != T_CDROM && SID_TYPE(&cgd->inq_data) != T_WORM) break; /* * Allocate a peripheral instance for * this device and start the probe * process. */ status = cam_periph_alloc(cdregister, cdoninvalidate, cdcleanup, cdstart, "cd", CAM_PERIPH_BIO, cgd->ccb_h.path, cdasync, AC_FOUND_DEVICE, cgd); if (status != CAM_REQ_CMP && status != CAM_REQ_INPROG) printf("cdasync: Unable to attach new device " "due to status 0x%x\n", status); break; } case AC_SENT_BDR: case AC_BUS_RESET: { struct cd_softc *softc; struct ccb_hdr *ccbh; softc = (struct cd_softc *)periph->softc; /* * Don't fail on the expected unit attention * that will occur. */ softc->flags |= CD_FLAG_RETRY_UA; LIST_FOREACH(ccbh, &softc->pending_ccbs, periph_links.le) ccbh->ccb_state |= CD_CCB_RETRY_UA; /* FALLTHROUGH */ } default: cam_periph_async(periph, code, path, arg); break; } } static void cdsysctlinit(void *context, int pending) { struct cam_periph *periph; struct cd_softc *softc; char tmpstr[80], tmpstr2[80]; periph = (struct cam_periph *)context; if (cam_periph_acquire(periph) != CAM_REQ_CMP) return; softc = (struct cd_softc *)periph->softc; snprintf(tmpstr, sizeof(tmpstr), "CAM CD unit %d", periph->unit_number); snprintf(tmpstr2, sizeof(tmpstr2), "%d", periph->unit_number); sysctl_ctx_init(&softc->sysctl_ctx); softc->flags |= CD_FLAG_SCTX_INIT; softc->sysctl_tree = SYSCTL_ADD_NODE(&softc->sysctl_ctx, SYSCTL_STATIC_CHILDREN(_kern_cam_cd), OID_AUTO, tmpstr2, CTLFLAG_RD, 0, tmpstr); if (softc->sysctl_tree == NULL) { printf("cdsysctlinit: unable to allocate sysctl tree\n"); cam_periph_release(periph); return; } /* * Now register the sysctl handler, so the user can the value on * the fly. */ SYSCTL_ADD_PROC(&softc->sysctl_ctx,SYSCTL_CHILDREN(softc->sysctl_tree), OID_AUTO, "minimum_cmd_size", CTLTYPE_INT | CTLFLAG_RW, &softc->minimum_command_size, 0, cdcmdsizesysctl, "I", "Minimum CDB size"); cam_periph_release(periph); } /* * We have a handler function for this so we can check the values when the * user sets them, instead of every time we look at them. */ static int cdcmdsizesysctl(SYSCTL_HANDLER_ARGS) { int error, value; value = *(int *)arg1; error = sysctl_handle_int(oidp, &value, 0, req); if ((error != 0) || (req->newptr == NULL)) return (error); /* * The only real values we can have here are 6 or 10. I don't * really forsee having 12 be an option at any time in the future. * So if the user sets something less than or equal to 6, we'll set * it to 6. If he sets something greater than 6, we'll set it to 10. * * I suppose we could just return an error here for the wrong values, * but I don't think it's necessary to do so, as long as we can * determine the user's intent without too much trouble. */ if (value < 6) value = 6; else if (value > 6) value = 10; *(int *)arg1 = value; return (0); } static cam_status cdregister(struct cam_periph *periph, void *arg) { struct cd_softc *softc; struct ccb_pathinq cpi; struct ccb_getdev *cgd; char tmpstr[80]; caddr_t match; cgd = (struct ccb_getdev *)arg; if (periph == NULL) { printf("cdregister: periph was NULL!!\n"); return(CAM_REQ_CMP_ERR); } if (cgd == NULL) { printf("cdregister: no getdev CCB, can't register device\n"); return(CAM_REQ_CMP_ERR); } softc = (struct cd_softc *)malloc(sizeof(*softc),M_DEVBUF, M_NOWAIT | M_ZERO); if (softc == NULL) { printf("cdregister: Unable to probe new device. " "Unable to allocate softc\n"); return(CAM_REQ_CMP_ERR); } LIST_INIT(&softc->pending_ccbs); STAILQ_INIT(&softc->mode_queue); softc->state = CD_STATE_PROBE; bioq_init(&softc->bio_queue); if (SID_IS_REMOVABLE(&cgd->inq_data)) softc->flags |= CD_FLAG_DISC_REMOVABLE; if ((cgd->inq_data.flags & SID_CmdQue) != 0) softc->flags |= CD_FLAG_TAGGED_QUEUING; periph->softc = softc; softc->periph = periph; /* * See if this device has any quirks. */ match = cam_quirkmatch((caddr_t)&cgd->inq_data, (caddr_t)cd_quirk_table, sizeof(cd_quirk_table)/sizeof(*cd_quirk_table), sizeof(*cd_quirk_table), scsi_inquiry_match); if (match != NULL) softc->quirks = ((struct cd_quirk_entry *)match)->quirks; else softc->quirks = CD_Q_NONE; /* Check if the SIM does not want 6 byte commands */ bzero(&cpi, sizeof(cpi)); xpt_setup_ccb(&cpi.ccb_h, periph->path, CAM_PRIORITY_NORMAL); cpi.ccb_h.func_code = XPT_PATH_INQ; xpt_action((union ccb *)&cpi); if (cpi.ccb_h.status == CAM_REQ_CMP && (cpi.hba_misc & PIM_NO_6_BYTE)) softc->quirks |= CD_Q_10_BYTE_ONLY; TASK_INIT(&softc->sysctl_task, 0, cdsysctlinit, periph); /* The default is 6 byte commands, unless quirked otherwise */ if (softc->quirks & CD_Q_10_BYTE_ONLY) softc->minimum_command_size = 10; else softc->minimum_command_size = 6; /* * Load the user's default, if any. */ snprintf(tmpstr, sizeof(tmpstr), "kern.cam.cd.%d.minimum_cmd_size", periph->unit_number); TUNABLE_INT_FETCH(tmpstr, &softc->minimum_command_size); /* 6 and 10 are the only permissible values here. */ if (softc->minimum_command_size < 6) softc->minimum_command_size = 6; else if (softc->minimum_command_size > 6) softc->minimum_command_size = 10; /* * We need to register the statistics structure for this device, * but we don't have the blocksize yet for it. So, we register * the structure and indicate that we don't have the blocksize * yet. Unlike other SCSI peripheral drivers, we explicitly set * the device type here to be CDROM, rather than just ORing in * the device type. This is because this driver can attach to either * CDROM or WORM devices, and we want this peripheral driver to * show up in the devstat list as a CD peripheral driver, not a * WORM peripheral driver. WORM drives will also have the WORM * driver attached to them. */ cam_periph_unlock(periph); softc->disk = disk_alloc(); softc->disk->d_devstat = devstat_new_entry("cd", periph->unit_number, 0, DEVSTAT_BS_UNAVAILABLE, DEVSTAT_TYPE_CDROM | DEVSTAT_TYPE_IF_SCSI, DEVSTAT_PRIORITY_CD); softc->disk->d_open = cdopen; softc->disk->d_close = cdclose; softc->disk->d_strategy = cdstrategy; softc->disk->d_ioctl = cdioctl; softc->disk->d_name = "cd"; softc->disk->d_unit = periph->unit_number; softc->disk->d_drv1 = periph; if (cpi.maxio == 0) softc->disk->d_maxsize = DFLTPHYS; /* traditional default */ else if (cpi.maxio > MAXPHYS) softc->disk->d_maxsize = MAXPHYS; /* for safety */ else softc->disk->d_maxsize = cpi.maxio; softc->disk->d_flags = 0; disk_create(softc->disk, DISK_VERSION); cam_periph_lock(periph); /* * Add an async callback so that we get * notified if this device goes away. */ xpt_register_async(AC_SENT_BDR | AC_BUS_RESET | AC_LOST_DEVICE, cdasync, periph, periph->path); /* * If the target lun is greater than 0, we most likely have a CD * changer device. Check the quirk entries as well, though, just * in case someone has a CD tower with one lun per drive or * something like that. Also, if we know up front that a * particular device is a changer, we can mark it as such starting * with lun 0, instead of lun 1. It shouldn't be necessary to have * a quirk entry to define something as a changer, however. */ if (((cgd->ccb_h.target_lun > 0) && ((softc->quirks & CD_Q_NO_CHANGER) == 0)) || ((softc->quirks & CD_Q_CHANGER) != 0)) { struct cdchanger *nchanger; struct cam_periph *nperiph; struct cam_path *path; cam_status status; int found; /* Set the changer flag in the current device's softc */ softc->flags |= CD_FLAG_CHANGER; /* * Now, look around for an existing changer device with the * same path and target ID as the current device. */ mtx_lock(&changerq_mtx); for (found = 0, nchanger = (struct cdchanger *)STAILQ_FIRST(&changerq); nchanger != NULL; nchanger = STAILQ_NEXT(nchanger, changer_links)){ if ((nchanger->path_id == cgd->ccb_h.path_id) && (nchanger->target_id == cgd->ccb_h.target_id)) { found = 1; break; } } mtx_unlock(&changerq_mtx); /* * If we found a matching entry, just add this device to * the list of devices on this changer. */ if (found == 1) { struct chdevlist *chlunhead; chlunhead = &nchanger->chluns; /* * XXX KDM look at consolidating this code with the * code below in a separate function. */ /* * Create a path with lun id 0, and see if we can * find a matching device */ status = xpt_create_path(&path, /*periph*/ periph, cgd->ccb_h.path_id, cgd->ccb_h.target_id, 0); if ((status == CAM_REQ_CMP) && ((nperiph = cam_periph_find(path, "cd")) != NULL)){ struct cd_softc *nsoftc; nsoftc = (struct cd_softc *)nperiph->softc; if ((nsoftc->flags & CD_FLAG_CHANGER) == 0){ nsoftc->flags |= CD_FLAG_CHANGER; nchanger->num_devices++; if (camq_resize(&nchanger->devq, nchanger->num_devices)!=CAM_REQ_CMP){ printf("cdregister: " "camq_resize " "failed, changer " "support may " "be messed up\n"); } nsoftc->changer = nchanger; nsoftc->pinfo.index =CAM_UNQUEUED_INDEX; STAILQ_INSERT_TAIL(&nchanger->chluns, nsoftc,changer_links); } xpt_free_path(path); } else if (status == CAM_REQ_CMP) xpt_free_path(path); else { printf("cdregister: unable to allocate path\n" "cdregister: changer support may be " "broken\n"); } nchanger->num_devices++; softc->changer = nchanger; softc->pinfo.index = CAM_UNQUEUED_INDEX; if (camq_resize(&nchanger->devq, nchanger->num_devices) != CAM_REQ_CMP) { printf("cdregister: camq_resize " "failed, changer support may " "be messed up\n"); } STAILQ_INSERT_TAIL(chlunhead, softc, changer_links); } /* * In this case, we don't already have an entry for this * particular changer, so we need to create one, add it to * the queue, and queue this device on the list for this * changer. Before we queue this device, however, we need * to search for lun id 0 on this target, and add it to the * queue first, if it exists. (and if it hasn't already * been marked as part of the changer.) */ else { nchanger = malloc(sizeof(struct cdchanger), M_DEVBUF, M_NOWAIT | M_ZERO); if (nchanger == NULL) { softc->flags &= ~CD_FLAG_CHANGER; printf("cdregister: unable to malloc " "changer structure\ncdregister: " "changer support disabled\n"); /* * Yes, gotos can be gross but in this case * I think it's justified.. */ goto cdregisterexit; } if (camq_init(&nchanger->devq, 1) != 0) { softc->flags &= ~CD_FLAG_CHANGER; printf("cdregister: changer support " "disabled\n"); goto cdregisterexit; } nchanger->path_id = cgd->ccb_h.path_id; nchanger->target_id = cgd->ccb_h.target_id; /* this is superfluous, but it makes things clearer */ nchanger->num_devices = 0; STAILQ_INIT(&nchanger->chluns); callout_init_mtx(&nchanger->long_handle, periph->sim->mtx, 0); callout_init_mtx(&nchanger->short_handle, periph->sim->mtx, 0); mtx_lock(&changerq_mtx); num_changers++; STAILQ_INSERT_TAIL(&changerq, nchanger, changer_links); mtx_unlock(&changerq_mtx); /* * Create a path with lun id 0, and see if we can * find a matching device */ status = xpt_create_path(&path, /*periph*/ periph, cgd->ccb_h.path_id, cgd->ccb_h.target_id, 0); /* * If we were able to allocate the path, and if we * find a matching device and it isn't already * marked as part of a changer, then we add it to * the current changer. */ if ((status == CAM_REQ_CMP) && ((nperiph = cam_periph_find(path, "cd")) != NULL) && ((((struct cd_softc *)periph->softc)->flags & CD_FLAG_CHANGER) == 0)) { struct cd_softc *nsoftc; nsoftc = (struct cd_softc *)nperiph->softc; nsoftc->flags |= CD_FLAG_CHANGER; nchanger->num_devices++; if (camq_resize(&nchanger->devq, nchanger->num_devices) != CAM_REQ_CMP) { printf("cdregister: camq_resize " "failed, changer support may " "be messed up\n"); } nsoftc->changer = nchanger; nsoftc->pinfo.index = CAM_UNQUEUED_INDEX; STAILQ_INSERT_TAIL(&nchanger->chluns, nsoftc, changer_links); xpt_free_path(path); } else if (status == CAM_REQ_CMP) xpt_free_path(path); else { printf("cdregister: unable to allocate path\n" "cdregister: changer support may be " "broken\n"); } softc->changer = nchanger; softc->pinfo.index = CAM_UNQUEUED_INDEX; nchanger->num_devices++; if (camq_resize(&nchanger->devq, nchanger->num_devices) != CAM_REQ_CMP) { printf("cdregister: camq_resize " "failed, changer support may " "be messed up\n"); } STAILQ_INSERT_TAIL(&nchanger->chluns, softc, changer_links); } } cdregisterexit: /* * Refcount and block open attempts until we are setup * Can't block */ (void)cam_periph_hold(periph, PRIBIO); if ((softc->flags & CD_FLAG_CHANGER) == 0) xpt_schedule(periph, CAM_PRIORITY_DEV); else cdschedule(periph, CAM_PRIORITY_DEV); return(CAM_REQ_CMP); } static int cdopen(struct disk *dp) { struct cam_periph *periph; struct cd_softc *softc; int error; periph = (struct cam_periph *)dp->d_drv1; if (periph == NULL) return (ENXIO); softc = (struct cd_softc *)periph->softc; if (cam_periph_acquire(periph) != CAM_REQ_CMP) return(ENXIO); cam_periph_lock(periph); if (softc->flags & CD_FLAG_INVALID) { cam_periph_unlock(periph); cam_periph_release(periph); return(ENXIO); } if ((error = cam_periph_hold(periph, PRIBIO | PCATCH)) != 0) { cam_periph_unlock(periph); cam_periph_release(periph); return (error); } /* * Check for media, and set the appropriate flags. We don't bail * if we don't have media, but then we don't allow anything but the * CDIOCEJECT/CDIOCCLOSE ioctls if there is no media. */ cdcheckmedia(periph); CAM_DEBUG(periph->path, CAM_DEBUG_TRACE, ("leaving cdopen\n")); cam_periph_unhold(periph); /* Closes aren't symmetrical with opens, so fix up the refcounting. */ if ((softc->flags & CD_FLAG_OPEN) == 0) { softc->flags |= CD_FLAG_OPEN; cam_periph_unlock(periph); } else { cam_periph_unlock(periph); cam_periph_release(periph); } return (0); } static int cdclose(struct disk *dp) { struct cam_periph *periph; struct cd_softc *softc; periph = (struct cam_periph *)dp->d_drv1; if (periph == NULL) return (ENXIO); softc = (struct cd_softc *)periph->softc; cam_periph_lock(periph); cam_periph_hold(periph, PRIBIO); if ((softc->flags & CD_FLAG_DISC_REMOVABLE) != 0) cdprevent(periph, PR_ALLOW); /* * Since we're closing this CD, mark the blocksize as unavailable. * It will be marked as available when the CD is opened again. */ softc->disk->d_devstat->flags |= DEVSTAT_BS_UNAVAILABLE; /* * We'll check the media and toc again at the next open(). */ softc->flags &= ~(CD_FLAG_VALID_MEDIA|CD_FLAG_VALID_TOC|CD_FLAG_OPEN); cam_periph_unhold(periph); cam_periph_unlock(periph); cam_periph_release(periph); return (0); } static void cdshorttimeout(void *arg) { struct cdchanger *changer; changer = (struct cdchanger *)arg; /* Always clear the short timeout flag, since that's what we're in */ changer->flags &= ~CHANGER_SHORT_TMOUT_SCHED; /* * Check to see if there is any more pending or outstanding I/O for * this device. If not, move it out of the active slot. */ if ((bioq_first(&changer->cur_device->bio_queue) == NULL) && (changer->cur_device->outstanding_cmds == 0)) { changer->flags |= CHANGER_MANUAL_CALL; cdrunchangerqueue(changer); } } /* * This is a wrapper for xpt_schedule. It only applies to changers. */ static void cdschedule(struct cam_periph *periph, int priority) { struct cd_softc *softc; softc = (struct cd_softc *)periph->softc; /* * If this device isn't currently queued, and if it isn't * the active device, then we queue this device and run the * changer queue if there is no timeout scheduled to do it. * If this device is the active device, just schedule it * to run again. If this device is queued, there should be * a timeout in place already that will make sure it runs. */ if ((softc->pinfo.index == CAM_UNQUEUED_INDEX) && ((softc->flags & CD_FLAG_ACTIVE) == 0)) { /* * We don't do anything with the priority here. * This is strictly a fifo queue. */ softc->pinfo.priority = CAM_PRIORITY_NORMAL; softc->pinfo.generation = ++softc->changer->devq.generation; camq_insert(&softc->changer->devq, (cam_pinfo *)softc); /* * Since we just put a device in the changer queue, * check and see if there is a timeout scheduled for * this changer. If so, let the timeout handle * switching this device into the active slot. If * not, manually call the timeout routine to * bootstrap things. */ if (((softc->changer->flags & CHANGER_TIMEOUT_SCHED)==0) && ((softc->changer->flags & CHANGER_NEED_TIMEOUT)==0) && ((softc->changer->flags & CHANGER_SHORT_TMOUT_SCHED)==0)){ softc->changer->flags |= CHANGER_MANUAL_CALL; cdrunchangerqueue(softc->changer); } } else if ((softc->flags & CD_FLAG_ACTIVE) && ((softc->flags & CD_FLAG_SCHED_ON_COMP) == 0)) xpt_schedule(periph, priority); } static void cdrunchangerqueue(void *arg) { struct cd_softc *softc; struct cdchanger *changer; int called_from_timeout; changer = (struct cdchanger *)arg; /* * If we have NOT been called from cdstrategy() or cddone(), and * instead from a timeout routine, go ahead and clear the * timeout flag. */ if ((changer->flags & CHANGER_MANUAL_CALL) == 0) { changer->flags &= ~CHANGER_TIMEOUT_SCHED; called_from_timeout = 1; } else called_from_timeout = 0; /* Always clear the manual call flag */ changer->flags &= ~CHANGER_MANUAL_CALL; /* nothing to do if the queue is empty */ if (changer->devq.entries <= 0) { return; } /* * If the changer queue is frozen, that means we have an active * device. */ if (changer->devq.qfrozen_cnt[0] > 0) { /* * We always need to reset the frozen count and clear the * active flag. */ changer->devq.qfrozen_cnt[0]--; changer->cur_device->flags &= ~CD_FLAG_ACTIVE; changer->cur_device->flags &= ~CD_FLAG_SCHED_ON_COMP; if (changer->cur_device->outstanding_cmds > 0) { changer->cur_device->flags |= CD_FLAG_SCHED_ON_COMP; changer->cur_device->bufs_left = changer->cur_device->outstanding_cmds; if (called_from_timeout) { callout_reset(&changer->long_handle, changer_max_busy_seconds * hz, cdrunchangerqueue, changer); changer->flags |= CHANGER_TIMEOUT_SCHED; } return; } /* * Check to see whether the current device has any I/O left * to do. If so, requeue it at the end of the queue. If * not, there is no need to requeue it. */ if (bioq_first(&changer->cur_device->bio_queue) != NULL) { changer->cur_device->pinfo.generation = ++changer->devq.generation; camq_insert(&changer->devq, (cam_pinfo *)changer->cur_device); } } softc = (struct cd_softc *)camq_remove(&changer->devq, CAMQ_HEAD); changer->cur_device = softc; changer->devq.qfrozen_cnt[0]++; softc->flags |= CD_FLAG_ACTIVE; /* Just in case this device is waiting */ wakeup(&softc->changer); xpt_schedule(softc->periph, CAM_PRIORITY_NORMAL); /* * Get rid of any pending timeouts, and set a flag to schedule new * ones so this device gets its full time quantum. */ if (changer->flags & CHANGER_TIMEOUT_SCHED) { callout_stop(&changer->long_handle); changer->flags &= ~CHANGER_TIMEOUT_SCHED; } if (changer->flags & CHANGER_SHORT_TMOUT_SCHED) { callout_stop(&changer->short_handle); changer->flags &= ~CHANGER_SHORT_TMOUT_SCHED; } /* * We need to schedule timeouts, but we only do this after the * first transaction has completed. This eliminates the changer * switch time. */ changer->flags |= CHANGER_NEED_TIMEOUT; } static void cdchangerschedule(struct cd_softc *softc) { struct cdchanger *changer; changer = softc->changer; /* * If this is a changer, and this is the current device, * and this device has at least the minimum time quantum to * run, see if we can switch it out. */ if ((softc->flags & CD_FLAG_ACTIVE) && ((changer->flags & CHANGER_SHORT_TMOUT_SCHED) == 0) && ((changer->flags & CHANGER_NEED_TIMEOUT) == 0)) { /* * We try three things here. The first is that we * check to see whether the schedule on completion * flag is set. If it is, we decrement the number * of buffers left, and if it's zero, we reschedule. * Next, we check to see whether the pending buffer * queue is empty and whether there are no * outstanding transactions. If so, we reschedule. * Next, we see if the pending buffer queue is empty. * If it is, we set the number of buffers left to * the current active buffer count and set the * schedule on complete flag. */ if (softc->flags & CD_FLAG_SCHED_ON_COMP) { if (--softc->bufs_left == 0) { softc->changer->flags |= CHANGER_MANUAL_CALL; softc->flags &= ~CD_FLAG_SCHED_ON_COMP; cdrunchangerqueue(softc->changer); } } else if ((bioq_first(&softc->bio_queue) == NULL) && (softc->outstanding_cmds == 0)) { softc->changer->flags |= CHANGER_MANUAL_CALL; cdrunchangerqueue(softc->changer); } } else if ((softc->changer->flags & CHANGER_NEED_TIMEOUT) && (softc->flags & CD_FLAG_ACTIVE)) { /* * Now that the first transaction to this * particular device has completed, we can go ahead * and schedule our timeouts. */ if ((changer->flags & CHANGER_TIMEOUT_SCHED) == 0) { callout_reset(&changer->long_handle, changer_max_busy_seconds * hz, cdrunchangerqueue, changer); changer->flags |= CHANGER_TIMEOUT_SCHED; } else printf("cdchangerschedule: already have a long" " timeout!\n"); if ((changer->flags & CHANGER_SHORT_TMOUT_SCHED) == 0) { callout_reset(&changer->short_handle, changer_min_busy_seconds * hz, cdshorttimeout, changer); changer->flags |= CHANGER_SHORT_TMOUT_SCHED; } else printf("cdchangerschedule: already have a short " "timeout!\n"); /* * We just scheduled timeouts, no need to schedule * more. */ changer->flags &= ~CHANGER_NEED_TIMEOUT; } } static int cdrunccb(union ccb *ccb, int (*error_routine)(union ccb *ccb, u_int32_t cam_flags, u_int32_t sense_flags), u_int32_t cam_flags, u_int32_t sense_flags) { struct cd_softc *softc; struct cam_periph *periph; int error; periph = xpt_path_periph(ccb->ccb_h.path); softc = (struct cd_softc *)periph->softc; error = cam_periph_runccb(ccb, error_routine, cam_flags, sense_flags, softc->disk->d_devstat); if (softc->flags & CD_FLAG_CHANGER) cdchangerschedule(softc); return(error); } static union ccb * cdgetccb(struct cam_periph *periph, u_int32_t priority) { struct cd_softc *softc; softc = (struct cd_softc *)periph->softc; if (softc->flags & CD_FLAG_CHANGER) { /* * This should work the first time this device is woken up, * but just in case it doesn't, we use a while loop. */ while ((softc->flags & CD_FLAG_ACTIVE) == 0) { /* * If this changer isn't already queued, queue it up. */ if (softc->pinfo.index == CAM_UNQUEUED_INDEX) { softc->pinfo.priority = CAM_PRIORITY_NORMAL; softc->pinfo.generation = ++softc->changer->devq.generation; camq_insert(&softc->changer->devq, (cam_pinfo *)softc); } if (((softc->changer->flags & CHANGER_TIMEOUT_SCHED)==0) && ((softc->changer->flags & CHANGER_NEED_TIMEOUT)==0) && ((softc->changer->flags & CHANGER_SHORT_TMOUT_SCHED)==0)) { softc->changer->flags |= CHANGER_MANUAL_CALL; cdrunchangerqueue(softc->changer); } else msleep(&softc->changer, periph->sim->mtx, PRIBIO, "cgticb", 0); } } return(cam_periph_getccb(periph, priority)); } /* * Actually translate the requested transfer into one the physical driver * can understand. The transfer is described by a buf and will include * only one physical transfer. */ static void cdstrategy(struct bio *bp) { struct cam_periph *periph; struct cd_softc *softc; periph = (struct cam_periph *)bp->bio_disk->d_drv1; if (periph == NULL) { biofinish(bp, NULL, ENXIO); return; } cam_periph_lock(periph); CAM_DEBUG(periph->path, CAM_DEBUG_TRACE, ("entering cdstrategy\n")); softc = (struct cd_softc *)periph->softc; /* * If the device has been made invalid, error out */ if ((softc->flags & CD_FLAG_INVALID)) { cam_periph_unlock(periph); biofinish(bp, NULL, ENXIO); return; } /* * If we don't have valid media, look for it before trying to * schedule the I/O. */ if ((softc->flags & CD_FLAG_VALID_MEDIA) == 0) { int error; error = cdcheckmedia(periph); if (error != 0) { cam_periph_unlock(periph); biofinish(bp, NULL, error); return; } } /* * Place it in the queue of disk activities for this disk */ bioq_disksort(&softc->bio_queue, bp); /* * Schedule ourselves for performing the work. We do things * differently for changers. */ if ((softc->flags & CD_FLAG_CHANGER) == 0) xpt_schedule(periph, CAM_PRIORITY_NORMAL); else cdschedule(periph, CAM_PRIORITY_NORMAL); cam_periph_unlock(periph); return; } static void cdstart(struct cam_periph *periph, union ccb *start_ccb) { struct cd_softc *softc; struct bio *bp; struct ccb_scsiio *csio; struct scsi_read_capacity_data *rcap; softc = (struct cd_softc *)periph->softc; CAM_DEBUG(periph->path, CAM_DEBUG_TRACE, ("entering cdstart\n")); switch (softc->state) { case CD_STATE_NORMAL: { bp = bioq_first(&softc->bio_queue); if (periph->immediate_priority <= periph->pinfo.priority) { start_ccb->ccb_h.ccb_state = CD_CCB_WAITING; SLIST_INSERT_HEAD(&periph->ccb_list, &start_ccb->ccb_h, periph_links.sle); periph->immediate_priority = CAM_PRIORITY_NONE; wakeup(&periph->ccb_list); } else if (bp == NULL) { xpt_release_ccb(start_ccb); } else { bioq_remove(&softc->bio_queue, bp); scsi_read_write(&start_ccb->csio, /*retries*/ cd_retry_count, /* cbfcnp */ cddone, MSG_SIMPLE_Q_TAG, /* read */bp->bio_cmd == BIO_READ, /* byte2 */ 0, /* minimum_cmd_size */ 10, /* lba */ bp->bio_offset / softc->params.blksize, bp->bio_bcount / softc->params.blksize, /* data_ptr */ bp->bio_data, /* dxfer_len */ bp->bio_bcount, /* sense_len */ SSD_FULL_SIZE, /* timeout */ 30000); start_ccb->ccb_h.ccb_state = CD_CCB_BUFFER_IO; LIST_INSERT_HEAD(&softc->pending_ccbs, &start_ccb->ccb_h, periph_links.le); softc->outstanding_cmds++; /* We expect a unit attention from this device */ if ((softc->flags & CD_FLAG_RETRY_UA) != 0) { start_ccb->ccb_h.ccb_state |= CD_CCB_RETRY_UA; softc->flags &= ~CD_FLAG_RETRY_UA; } start_ccb->ccb_h.ccb_bp = bp; bp = bioq_first(&softc->bio_queue); xpt_action(start_ccb); } if (bp != NULL) { /* Have more work to do, so ensure we stay scheduled */ xpt_schedule(periph, CAM_PRIORITY_NORMAL); } break; } case CD_STATE_PROBE: { rcap = (struct scsi_read_capacity_data *)malloc(sizeof(*rcap), M_SCSICD, M_NOWAIT | M_ZERO); if (rcap == NULL) { xpt_print(periph->path, "cdstart: Couldn't malloc read_capacity data\n"); /* cd_free_periph??? */ break; } csio = &start_ccb->csio; scsi_read_capacity(csio, /*retries*/ cd_retry_count, cddone, MSG_SIMPLE_Q_TAG, rcap, SSD_FULL_SIZE, /*timeout*/20000); start_ccb->ccb_h.ccb_bp = NULL; start_ccb->ccb_h.ccb_state = CD_CCB_PROBE; xpt_action(start_ccb); break; } } } static void cddone(struct cam_periph *periph, union ccb *done_ccb) { struct cd_softc *softc; struct ccb_scsiio *csio; CAM_DEBUG(periph->path, CAM_DEBUG_TRACE, ("entering cddone\n")); softc = (struct cd_softc *)periph->softc; csio = &done_ccb->csio; switch (csio->ccb_h.ccb_state & CD_CCB_TYPE_MASK) { case CD_CCB_BUFFER_IO: { struct bio *bp; int error; bp = (struct bio *)done_ccb->ccb_h.ccb_bp; error = 0; if ((done_ccb->ccb_h.status & CAM_STATUS_MASK) != CAM_REQ_CMP) { int sf; if ((done_ccb->ccb_h.ccb_state & CD_CCB_RETRY_UA) != 0) sf = SF_RETRY_UA; else sf = 0; error = cderror(done_ccb, CAM_RETRY_SELTO, sf); if (error == ERESTART) { /* * A retry was scheuled, so * just return. */ return; } } if (error != 0) { xpt_print(periph->path, "cddone: got error %#x back\n", error); bioq_flush(&softc->bio_queue, NULL, EIO); bp->bio_resid = bp->bio_bcount; bp->bio_error = error; bp->bio_flags |= BIO_ERROR; if ((done_ccb->ccb_h.status & CAM_DEV_QFRZN) != 0) cam_release_devq(done_ccb->ccb_h.path, /*relsim_flags*/0, /*reduction*/0, /*timeout*/0, /*getcount_only*/0); } else { bp->bio_resid = csio->resid; bp->bio_error = 0; if (bp->bio_resid != 0) { /* * Short transfer ??? * XXX: not sure this is correct for partial * transfers at EOM */ bp->bio_flags |= BIO_ERROR; } } LIST_REMOVE(&done_ccb->ccb_h, periph_links.le); softc->outstanding_cmds--; if (softc->flags & CD_FLAG_CHANGER) cdchangerschedule(softc); biofinish(bp, NULL, 0); break; } case CD_CCB_PROBE: { struct scsi_read_capacity_data *rdcap; char announce_buf[120]; /* * Currently (9/30/97) the * longest possible announce * buffer is 108 bytes, for the * first error case below. * That is 39 bytes for the * basic string, 16 bytes for the * biggest sense key (hardware * error), 52 bytes for the * text of the largest sense * qualifier valid for a CDROM, * (0x72, 0x03 or 0x04, * 0x03), and one byte for the * null terminating character. * To allow for longer strings, * the announce buffer is 120 * bytes. */ struct cd_params *cdp; cdp = &softc->params; rdcap = (struct scsi_read_capacity_data *)csio->data_ptr; cdp->disksize = scsi_4btoul (rdcap->addr) + 1; cdp->blksize = scsi_4btoul (rdcap->length); if ((csio->ccb_h.status & CAM_STATUS_MASK) == CAM_REQ_CMP) { snprintf(announce_buf, sizeof(announce_buf), "cd present [%lu x %lu byte records]", cdp->disksize, (u_long)cdp->blksize); } else { int error; /* * Retry any UNIT ATTENTION type errors. They * are expected at boot. */ error = cderror(done_ccb, CAM_RETRY_SELTO, SF_RETRY_UA | SF_NO_PRINT); if (error == ERESTART) { /* * A retry was scheuled, so * just return. */ return; } else if (error != 0) { struct scsi_sense_data *sense; int asc, ascq; int sense_key, error_code; int have_sense; cam_status status; struct ccb_getdev cgd; /* Don't wedge this device's queue */ if ((done_ccb->ccb_h.status & CAM_DEV_QFRZN) != 0) cam_release_devq(done_ccb->ccb_h.path, /*relsim_flags*/0, /*reduction*/0, /*timeout*/0, /*getcount_only*/0); status = done_ccb->ccb_h.status; xpt_setup_ccb(&cgd.ccb_h, done_ccb->ccb_h.path, CAM_PRIORITY_NORMAL); cgd.ccb_h.func_code = XPT_GDEV_TYPE; xpt_action((union ccb *)&cgd); if (((csio->ccb_h.flags & CAM_SENSE_PHYS) != 0) || ((csio->ccb_h.flags & CAM_SENSE_PTR) != 0) || ((status & CAM_AUTOSNS_VALID) == 0)) have_sense = FALSE; else have_sense = TRUE; if (have_sense) { sense = &csio->sense_data; scsi_extract_sense(sense, &error_code, &sense_key, &asc, &ascq); } /* * Attach to anything that claims to be a * CDROM or WORM device, as long as it * doesn't return a "Logical unit not * supported" (0x25) error. */ if ((have_sense) && (asc != 0x25) && (error_code == SSD_CURRENT_ERROR)) { const char *sense_key_desc; const char *asc_desc; scsi_sense_desc(sense_key, asc, ascq, &cgd.inq_data, &sense_key_desc, &asc_desc); snprintf(announce_buf, sizeof(announce_buf), "Attempt to query device " "size failed: %s, %s", sense_key_desc, asc_desc); } else if ((have_sense == 0) && ((status & CAM_STATUS_MASK) == CAM_SCSI_STATUS_ERROR) && (csio->scsi_status == SCSI_STATUS_BUSY)) { snprintf(announce_buf, sizeof(announce_buf), "Attempt to query device " "size failed: SCSI Status: %s", scsi_status_string(csio)); } else if (SID_TYPE(&cgd.inq_data) == T_CDROM) { /* * We only print out an error for * CDROM type devices. For WORM * devices, we don't print out an * error since a few WORM devices * don't support CDROM commands. * If we have sense information, go * ahead and print it out. * Otherwise, just say that we * couldn't attach. */ /* * Just print out the error, not * the full probe message, when we * don't attach. */ if (have_sense) scsi_sense_print( &done_ccb->csio); else { xpt_print(periph->path, "got CAM status %#x\n", done_ccb->ccb_h.status); } xpt_print(periph->path, "fatal error, " "failed to attach to device\n"); /* * Invalidate this peripheral. */ cam_periph_invalidate(periph); announce_buf[0] = '\0'; } else { /* * Invalidate this peripheral. */ cam_periph_invalidate(periph); announce_buf[0] = '\0'; } } } free(rdcap, M_SCSICD); if (announce_buf[0] != '\0') { xpt_announce_periph(periph, announce_buf); if (softc->flags & CD_FLAG_CHANGER) cdchangerschedule(softc); /* * Create our sysctl variables, now that we know * we have successfully attached. */ taskqueue_enqueue(taskqueue_thread,&softc->sysctl_task); } softc->state = CD_STATE_NORMAL; /* * Since our peripheral may be invalidated by an error * above or an external event, we must release our CCB * before releasing the probe lock on the peripheral. * The peripheral will only go away once the last lock * is removed, and we need it around for the CCB release * operation. */ xpt_release_ccb(done_ccb); cam_periph_unhold(periph); return; } case CD_CCB_WAITING: { /* Caller will release the CCB */ CAM_DEBUG(periph->path, CAM_DEBUG_TRACE, ("trying to wakeup ccbwait\n")); wakeup(&done_ccb->ccb_h.cbfcnp); return; } default: break; } xpt_release_ccb(done_ccb); } static union cd_pages * cdgetpage(struct cd_mode_params *mode_params) { union cd_pages *page; if (mode_params->cdb_size == 10) page = (union cd_pages *)find_mode_page_10( (struct scsi_mode_header_10 *)mode_params->mode_buf); else page = (union cd_pages *)find_mode_page_6( (struct scsi_mode_header_6 *)mode_params->mode_buf); return (page); } static int cdgetpagesize(int page_num) { int i; for (i = 0; i < (sizeof(cd_page_size_table)/ sizeof(cd_page_size_table[0])); i++) { if (cd_page_size_table[i].page == page_num) return (cd_page_size_table[i].page_size); } return (-1); } static int cdioctl(struct disk *dp, u_long cmd, void *addr, int flag, struct thread *td) { struct cam_periph *periph; struct cd_softc *softc; int nocopyout, error = 0; periph = (struct cam_periph *)dp->d_drv1; if (periph == NULL) return(ENXIO); cam_periph_lock(periph); CAM_DEBUG(periph->path, CAM_DEBUG_TRACE, ("entering cdioctl\n")); softc = (struct cd_softc *)periph->softc; CAM_DEBUG(periph->path, CAM_DEBUG_TRACE, ("trying to do ioctl %#lx\n", cmd)); if ((error = cam_periph_hold(periph, PRIBIO | PCATCH)) != 0) { cam_periph_unlock(periph); cam_periph_release(periph); return (error); } /* * If we don't have media loaded, check for it. If still don't * have media loaded, we can only do a load or eject. * * We only care whether media is loaded if this is a cd-specific ioctl * (thus the IOCGROUP check below). Note that this will break if * anyone adds any ioctls into the switch statement below that don't * have their ioctl group set to 'c'. */ if (((softc->flags & CD_FLAG_VALID_MEDIA) == 0) && ((cmd != CDIOCCLOSE) && (cmd != CDIOCEJECT)) && (IOCGROUP(cmd) == 'c')) { error = cdcheckmedia(periph); if (error != 0) { cam_periph_unhold(periph); cam_periph_unlock(periph); return (error); } } /* * Drop the lock here so later mallocs can use WAITOK. The periph * is essentially locked still with the cam_periph_hold call above. */ cam_periph_unlock(periph); nocopyout = 0; switch (cmd) { case CDIOCPLAYTRACKS: { struct ioc_play_track *args = (struct ioc_play_track *) addr; struct cd_mode_params params; union cd_pages *page; params.alloc_len = sizeof(union cd_mode_data_6_10); params.mode_buf = malloc(params.alloc_len, M_SCSICD, M_WAITOK | M_ZERO); cam_periph_lock(periph); CAM_DEBUG(periph->path, CAM_DEBUG_SUBTRACE, ("trying to do CDIOCPLAYTRACKS\n")); error = cdgetmode(periph, ¶ms, AUDIO_PAGE); if (error) { free(params.mode_buf, M_SCSICD); cam_periph_unlock(periph); break; } page = cdgetpage(¶ms); page->audio.flags &= ~CD_PA_SOTC; page->audio.flags |= CD_PA_IMMED; error = cdsetmode(periph, ¶ms); free(params.mode_buf, M_SCSICD); if (error) { cam_periph_unlock(periph); break; } /* * This was originally implemented with the PLAY * AUDIO TRACK INDEX command, but that command was * deprecated after SCSI-2. Most (all?) SCSI CDROM * drives support it but ATAPI and ATAPI-derivative * drives don't seem to support it. So we keep a * cache of the table of contents and translate * track numbers to MSF format. */ if (softc->flags & CD_FLAG_VALID_TOC) { union msf_lba *sentry, *eentry; int st, et; if (args->end_track < softc->toc.header.ending_track + 1) args->end_track++; if (args->end_track > softc->toc.header.ending_track + 1) args->end_track = softc->toc.header.ending_track + 1; st = args->start_track - softc->toc.header.starting_track; et = args->end_track - softc->toc.header.starting_track; if ((st < 0) || (et < 0) || (st > (softc->toc.header.ending_track - softc->toc.header.starting_track))) { error = EINVAL; break; } sentry = &softc->toc.entries[st].addr; eentry = &softc->toc.entries[et].addr; error = cdplaymsf(periph, sentry->msf.minute, sentry->msf.second, sentry->msf.frame, eentry->msf.minute, eentry->msf.second, eentry->msf.frame); } else { /* * If we don't have a valid TOC, try the * play track index command. It is part of * the SCSI-2 spec, but was removed in the * MMC specs. ATAPI and ATAPI-derived * drives don't support it. */ if (softc->quirks & CD_Q_BCD_TRACKS) { args->start_track = bin2bcd(args->start_track); args->end_track = bin2bcd(args->end_track); } error = cdplaytracks(periph, args->start_track, args->start_index, args->end_track, args->end_index); } cam_periph_unlock(periph); } break; case CDIOCPLAYMSF: { struct ioc_play_msf *args = (struct ioc_play_msf *) addr; struct cd_mode_params params; union cd_pages *page; params.alloc_len = sizeof(union cd_mode_data_6_10); params.mode_buf = malloc(params.alloc_len, M_SCSICD, M_WAITOK | M_ZERO); cam_periph_lock(periph); CAM_DEBUG(periph->path, CAM_DEBUG_SUBTRACE, ("trying to do CDIOCPLAYMSF\n")); error = cdgetmode(periph, ¶ms, AUDIO_PAGE); if (error) { free(params.mode_buf, M_SCSICD); cam_periph_unlock(periph); break; } page = cdgetpage(¶ms); page->audio.flags &= ~CD_PA_SOTC; page->audio.flags |= CD_PA_IMMED; error = cdsetmode(periph, ¶ms); free(params.mode_buf, M_SCSICD); if (error) { cam_periph_unlock(periph); break; } error = cdplaymsf(periph, args->start_m, args->start_s, args->start_f, args->end_m, args->end_s, args->end_f); cam_periph_unlock(periph); } break; case CDIOCPLAYBLOCKS: { struct ioc_play_blocks *args = (struct ioc_play_blocks *) addr; struct cd_mode_params params; union cd_pages *page; params.alloc_len = sizeof(union cd_mode_data_6_10); params.mode_buf = malloc(params.alloc_len, M_SCSICD, M_WAITOK | M_ZERO); cam_periph_lock(periph); CAM_DEBUG(periph->path, CAM_DEBUG_SUBTRACE, ("trying to do CDIOCPLAYBLOCKS\n")); error = cdgetmode(periph, ¶ms, AUDIO_PAGE); if (error) { free(params.mode_buf, M_SCSICD); cam_periph_unlock(periph); break; } page = cdgetpage(¶ms); page->audio.flags &= ~CD_PA_SOTC; page->audio.flags |= CD_PA_IMMED; error = cdsetmode(periph, ¶ms); free(params.mode_buf, M_SCSICD); if (error) { cam_periph_unlock(periph); break; } error = cdplay(periph, args->blk, args->len); cam_periph_unlock(periph); } break; case CDIOCREADSUBCHANNEL_SYSSPACE: nocopyout = 1; /* Fallthrough */ case CDIOCREADSUBCHANNEL: { struct ioc_read_subchannel *args = (struct ioc_read_subchannel *) addr; struct cd_sub_channel_info *data; u_int32_t len = args->data_len; data = malloc(sizeof(struct cd_sub_channel_info), M_SCSICD, M_WAITOK | M_ZERO); cam_periph_lock(periph); CAM_DEBUG(periph->path, CAM_DEBUG_SUBTRACE, ("trying to do CDIOCREADSUBCHANNEL\n")); if ((len > sizeof(struct cd_sub_channel_info)) || (len < sizeof(struct cd_sub_channel_header))) { printf( "scsi_cd: cdioctl: " "cdioreadsubchannel: error, len=%d\n", len); error = EINVAL; free(data, M_SCSICD); cam_periph_unlock(periph); break; } if (softc->quirks & CD_Q_BCD_TRACKS) args->track = bin2bcd(args->track); error = cdreadsubchannel(periph, args->address_format, args->data_format, args->track, data, len); if (error) { free(data, M_SCSICD); cam_periph_unlock(periph); break; } if (softc->quirks & CD_Q_BCD_TRACKS) data->what.track_info.track_number = bcd2bin(data->what.track_info.track_number); len = min(len, ((data->header.data_len[0] << 8) + data->header.data_len[1] + sizeof(struct cd_sub_channel_header))); cam_periph_unlock(periph); if (nocopyout == 0) { if (copyout(data, args->data, len) != 0) { error = EFAULT; } } else { bcopy(data, args->data, len); } free(data, M_SCSICD); } break; case CDIOREADTOCHEADER: { struct ioc_toc_header *th; th = malloc(sizeof(struct ioc_toc_header), M_SCSICD, M_WAITOK | M_ZERO); cam_periph_lock(periph); CAM_DEBUG(periph->path, CAM_DEBUG_SUBTRACE, ("trying to do CDIOREADTOCHEADER\n")); error = cdreadtoc(periph, 0, 0, (u_int8_t *)th, sizeof (*th), /*sense_flags*/0); if (error) { free(th, M_SCSICD); cam_periph_unlock(periph); break; } if (softc->quirks & CD_Q_BCD_TRACKS) { /* we are going to have to convert the BCD * encoding on the cd to what is expected */ th->starting_track = bcd2bin(th->starting_track); th->ending_track = bcd2bin(th->ending_track); } th->len = ntohs(th->len); bcopy(th, addr, sizeof(*th)); free(th, M_SCSICD); cam_periph_unlock(periph); } break; case CDIOREADTOCENTRYS: { struct cd_tocdata *data; struct cd_toc_single *lead; struct ioc_read_toc_entry *te = (struct ioc_read_toc_entry *) addr; struct ioc_toc_header *th; u_int32_t len, readlen, idx, num; u_int32_t starting_track = te->starting_track; data = malloc(sizeof(*data), M_SCSICD, M_WAITOK | M_ZERO); lead = malloc(sizeof(*lead), M_SCSICD, M_WAITOK | M_ZERO); cam_periph_lock(periph); CAM_DEBUG(periph->path, CAM_DEBUG_SUBTRACE, ("trying to do CDIOREADTOCENTRYS\n")); if (te->data_len < sizeof(struct cd_toc_entry) || (te->data_len % sizeof(struct cd_toc_entry)) != 0 || (te->address_format != CD_MSF_FORMAT && te->address_format != CD_LBA_FORMAT)) { error = EINVAL; printf("scsi_cd: error in readtocentries, " "returning EINVAL\n"); free(data, M_SCSICD); free(lead, M_SCSICD); cam_periph_unlock(periph); break; } th = &data->header; error = cdreadtoc(periph, 0, 0, (u_int8_t *)th, sizeof (*th), /*sense_flags*/0); if (error) { free(data, M_SCSICD); free(lead, M_SCSICD); cam_periph_unlock(periph); break; } if (softc->quirks & CD_Q_BCD_TRACKS) { /* we are going to have to convert the BCD * encoding on the cd to what is expected */ th->starting_track = bcd2bin(th->starting_track); th->ending_track = bcd2bin(th->ending_track); } if (starting_track == 0) starting_track = th->starting_track; else if (starting_track == LEADOUT) starting_track = th->ending_track + 1; else if (starting_track < th->starting_track || starting_track > th->ending_track + 1) { printf("scsi_cd: error in readtocentries, " "returning EINVAL\n"); free(data, M_SCSICD); free(lead, M_SCSICD); cam_periph_unlock(periph); error = EINVAL; break; } /* calculate reading length without leadout entry */ readlen = (th->ending_track - starting_track + 1) * sizeof(struct cd_toc_entry); /* and with leadout entry */ len = readlen + sizeof(struct cd_toc_entry); if (te->data_len < len) { len = te->data_len; if (readlen > len) readlen = len; } if (len > sizeof(data->entries)) { printf("scsi_cd: error in readtocentries, " "returning EINVAL\n"); error = EINVAL; free(data, M_SCSICD); free(lead, M_SCSICD); cam_periph_unlock(periph); break; } num = len / sizeof(struct cd_toc_entry); if (readlen > 0) { error = cdreadtoc(periph, te->address_format, starting_track, (u_int8_t *)data, readlen + sizeof (*th), /*sense_flags*/0); if (error) { free(data, M_SCSICD); free(lead, M_SCSICD); cam_periph_unlock(periph); break; } } /* make leadout entry if needed */ idx = starting_track + num - 1; if (softc->quirks & CD_Q_BCD_TRACKS) th->ending_track = bcd2bin(th->ending_track); if (idx == th->ending_track + 1) { error = cdreadtoc(periph, te->address_format, LEADOUT, (u_int8_t *)lead, sizeof(*lead), /*sense_flags*/0); if (error) { free(data, M_SCSICD); free(lead, M_SCSICD); cam_periph_unlock(periph); break; } data->entries[idx - starting_track] = lead->entry; } if (softc->quirks & CD_Q_BCD_TRACKS) { for (idx = 0; idx < num - 1; idx++) { data->entries[idx].track = bcd2bin(data->entries[idx].track); } } cam_periph_unlock(periph); error = copyout(data->entries, te->data, len); free(data, M_SCSICD); free(lead, M_SCSICD); } break; case CDIOREADTOCENTRY: { struct cd_toc_single *data; struct ioc_read_toc_single_entry *te = (struct ioc_read_toc_single_entry *) addr; struct ioc_toc_header *th; u_int32_t track; data = malloc(sizeof(*data), M_SCSICD, M_WAITOK | M_ZERO); cam_periph_lock(periph); CAM_DEBUG(periph->path, CAM_DEBUG_SUBTRACE, ("trying to do CDIOREADTOCENTRY\n")); if (te->address_format != CD_MSF_FORMAT && te->address_format != CD_LBA_FORMAT) { printf("error in readtocentry, " " returning EINVAL\n"); free(data, M_SCSICD); error = EINVAL; cam_periph_unlock(periph); break; } th = &data->header; error = cdreadtoc(periph, 0, 0, (u_int8_t *)th, sizeof (*th), /*sense_flags*/0); if (error) { free(data, M_SCSICD); cam_periph_unlock(periph); break; } if (softc->quirks & CD_Q_BCD_TRACKS) { /* we are going to have to convert the BCD * encoding on the cd to what is expected */ th->starting_track = bcd2bin(th->starting_track); th->ending_track = bcd2bin(th->ending_track); } track = te->track; if (track == 0) track = th->starting_track; else if (track == LEADOUT) /* OK */; else if (track < th->starting_track || track > th->ending_track + 1) { printf("error in readtocentry, " " returning EINVAL\n"); free(data, M_SCSICD); error = EINVAL; cam_periph_unlock(periph); break; } error = cdreadtoc(periph, te->address_format, track, (u_int8_t *)data, sizeof(*data), /*sense_flags*/0); if (error) { free(data, M_SCSICD); cam_periph_unlock(periph); break; } if (softc->quirks & CD_Q_BCD_TRACKS) data->entry.track = bcd2bin(data->entry.track); bcopy(&data->entry, &te->entry, sizeof(struct cd_toc_entry)); free(data, M_SCSICD); cam_periph_unlock(periph); } break; case CDIOCSETPATCH: { struct ioc_patch *arg = (struct ioc_patch *)addr; struct cd_mode_params params; union cd_pages *page; params.alloc_len = sizeof(union cd_mode_data_6_10); params.mode_buf = malloc(params.alloc_len, M_SCSICD, M_WAITOK | M_ZERO); cam_periph_lock(periph); CAM_DEBUG(periph->path, CAM_DEBUG_SUBTRACE, ("trying to do CDIOCSETPATCH\n")); error = cdgetmode(periph, ¶ms, AUDIO_PAGE); if (error) { free(params.mode_buf, M_SCSICD); cam_periph_unlock(periph); break; } page = cdgetpage(¶ms); page->audio.port[LEFT_PORT].channels = arg->patch[0]; page->audio.port[RIGHT_PORT].channels = arg->patch[1]; page->audio.port[2].channels = arg->patch[2]; page->audio.port[3].channels = arg->patch[3]; error = cdsetmode(periph, ¶ms); free(params.mode_buf, M_SCSICD); cam_periph_unlock(periph); } break; case CDIOCGETVOL: { struct ioc_vol *arg = (struct ioc_vol *) addr; struct cd_mode_params params; union cd_pages *page; params.alloc_len = sizeof(union cd_mode_data_6_10); params.mode_buf = malloc(params.alloc_len, M_SCSICD, M_WAITOK | M_ZERO); cam_periph_lock(periph); CAM_DEBUG(periph->path, CAM_DEBUG_SUBTRACE, ("trying to do CDIOCGETVOL\n")); error = cdgetmode(periph, ¶ms, AUDIO_PAGE); if (error) { free(params.mode_buf, M_SCSICD); cam_periph_unlock(periph); break; } page = cdgetpage(¶ms); arg->vol[LEFT_PORT] = page->audio.port[LEFT_PORT].volume; arg->vol[RIGHT_PORT] = page->audio.port[RIGHT_PORT].volume; arg->vol[2] = page->audio.port[2].volume; arg->vol[3] = page->audio.port[3].volume; free(params.mode_buf, M_SCSICD); cam_periph_unlock(periph); } break; case CDIOCSETVOL: { struct ioc_vol *arg = (struct ioc_vol *) addr; struct cd_mode_params params; union cd_pages *page; params.alloc_len = sizeof(union cd_mode_data_6_10); params.mode_buf = malloc(params.alloc_len, M_SCSICD, M_WAITOK | M_ZERO); cam_periph_lock(periph); CAM_DEBUG(periph->path, CAM_DEBUG_SUBTRACE, ("trying to do CDIOCSETVOL\n")); error = cdgetmode(periph, ¶ms, AUDIO_PAGE); if (error) { free(params.mode_buf, M_SCSICD); cam_periph_unlock(periph); break; } page = cdgetpage(¶ms); page->audio.port[LEFT_PORT].channels = CHANNEL_0; page->audio.port[LEFT_PORT].volume = arg->vol[LEFT_PORT]; page->audio.port[RIGHT_PORT].channels = CHANNEL_1; page->audio.port[RIGHT_PORT].volume = arg->vol[RIGHT_PORT]; page->audio.port[2].volume = arg->vol[2]; page->audio.port[3].volume = arg->vol[3]; error = cdsetmode(periph, ¶ms); cam_periph_unlock(periph); free(params.mode_buf, M_SCSICD); } break; case CDIOCSETMONO: { struct cd_mode_params params; union cd_pages *page; params.alloc_len = sizeof(union cd_mode_data_6_10); params.mode_buf = malloc(params.alloc_len, M_SCSICD, M_WAITOK | M_ZERO); cam_periph_lock(periph); CAM_DEBUG(periph->path, CAM_DEBUG_SUBTRACE, ("trying to do CDIOCSETMONO\n")); error = cdgetmode(periph, ¶ms, AUDIO_PAGE); if (error) { free(params.mode_buf, M_SCSICD); cam_periph_unlock(periph); break; } page = cdgetpage(¶ms); page->audio.port[LEFT_PORT].channels = LEFT_CHANNEL | RIGHT_CHANNEL; page->audio.port[RIGHT_PORT].channels = LEFT_CHANNEL | RIGHT_CHANNEL; page->audio.port[2].channels = 0; page->audio.port[3].channels = 0; error = cdsetmode(periph, ¶ms); cam_periph_unlock(periph); free(params.mode_buf, M_SCSICD); } break; case CDIOCSETSTEREO: { struct cd_mode_params params; union cd_pages *page; params.alloc_len = sizeof(union cd_mode_data_6_10); params.mode_buf = malloc(params.alloc_len, M_SCSICD, M_WAITOK | M_ZERO); cam_periph_lock(periph); CAM_DEBUG(periph->path, CAM_DEBUG_SUBTRACE, ("trying to do CDIOCSETSTEREO\n")); error = cdgetmode(periph, ¶ms, AUDIO_PAGE); if (error) { free(params.mode_buf, M_SCSICD); cam_periph_unlock(periph); break; } page = cdgetpage(¶ms); page->audio.port[LEFT_PORT].channels = LEFT_CHANNEL; page->audio.port[RIGHT_PORT].channels = RIGHT_CHANNEL; page->audio.port[2].channels = 0; page->audio.port[3].channels = 0; error = cdsetmode(periph, ¶ms); free(params.mode_buf, M_SCSICD); cam_periph_unlock(periph); } break; case CDIOCSETMUTE: { struct cd_mode_params params; union cd_pages *page; params.alloc_len = sizeof(union cd_mode_data_6_10); params.mode_buf = malloc(params.alloc_len, M_SCSICD, M_WAITOK | M_ZERO); cam_periph_lock(periph); CAM_DEBUG(periph->path, CAM_DEBUG_SUBTRACE, ("trying to do CDIOCSETMUTE\n")); error = cdgetmode(periph, ¶ms, AUDIO_PAGE); if (error) { free(¶ms.mode_buf, M_SCSICD); cam_periph_unlock(periph); break; } page = cdgetpage(¶ms); page->audio.port[LEFT_PORT].channels = 0; page->audio.port[RIGHT_PORT].channels = 0; page->audio.port[2].channels = 0; page->audio.port[3].channels = 0; error = cdsetmode(periph, ¶ms); free(params.mode_buf, M_SCSICD); cam_periph_unlock(periph); } break; case CDIOCSETLEFT: { struct cd_mode_params params; union cd_pages *page; params.alloc_len = sizeof(union cd_mode_data_6_10); params.mode_buf = malloc(params.alloc_len, M_SCSICD, M_WAITOK | M_ZERO); cam_periph_lock(periph); CAM_DEBUG(periph->path, CAM_DEBUG_SUBTRACE, ("trying to do CDIOCSETLEFT\n")); error = cdgetmode(periph, ¶ms, AUDIO_PAGE); if (error) { free(params.mode_buf, M_SCSICD); cam_periph_unlock(periph); break; } page = cdgetpage(¶ms); page->audio.port[LEFT_PORT].channels = LEFT_CHANNEL; page->audio.port[RIGHT_PORT].channels = LEFT_CHANNEL; page->audio.port[2].channels = 0; page->audio.port[3].channels = 0; error = cdsetmode(periph, ¶ms); free(params.mode_buf, M_SCSICD); cam_periph_unlock(periph); } break; case CDIOCSETRIGHT: { struct cd_mode_params params; union cd_pages *page; params.alloc_len = sizeof(union cd_mode_data_6_10); params.mode_buf = malloc(params.alloc_len, M_SCSICD, M_WAITOK | M_ZERO); cam_periph_lock(periph); CAM_DEBUG(periph->path, CAM_DEBUG_SUBTRACE, ("trying to do CDIOCSETRIGHT\n")); error = cdgetmode(periph, ¶ms, AUDIO_PAGE); if (error) { free(params.mode_buf, M_SCSICD); cam_periph_unlock(periph); break; } page = cdgetpage(¶ms); page->audio.port[LEFT_PORT].channels = RIGHT_CHANNEL; page->audio.port[RIGHT_PORT].channels = RIGHT_CHANNEL; page->audio.port[2].channels = 0; page->audio.port[3].channels = 0; error = cdsetmode(periph, ¶ms); free(params.mode_buf, M_SCSICD); cam_periph_unlock(periph); } break; case CDIOCRESUME: cam_periph_lock(periph); error = cdpause(periph, 1); cam_periph_unlock(periph); break; case CDIOCPAUSE: cam_periph_lock(periph); error = cdpause(periph, 0); cam_periph_unlock(periph); break; case CDIOCSTART: cam_periph_lock(periph); error = cdstartunit(periph, 0); cam_periph_unlock(periph); break; case CDIOCCLOSE: cam_periph_lock(periph); error = cdstartunit(periph, 1); cam_periph_unlock(periph); break; case CDIOCSTOP: cam_periph_lock(periph); error = cdstopunit(periph, 0); cam_periph_unlock(periph); break; case CDIOCEJECT: cam_periph_lock(periph); error = cdstopunit(periph, 1); cam_periph_unlock(periph); break; case CDIOCALLOW: cam_periph_lock(periph); cdprevent(periph, PR_ALLOW); cam_periph_unlock(periph); break; case CDIOCPREVENT: cam_periph_lock(periph); cdprevent(periph, PR_PREVENT); cam_periph_unlock(periph); break; case CDIOCSETDEBUG: /* sc_link->flags |= (SDEV_DB1 | SDEV_DB2); */ error = ENOTTY; break; case CDIOCCLRDEBUG: /* sc_link->flags &= ~(SDEV_DB1 | SDEV_DB2); */ error = ENOTTY; break; case CDIOCRESET: /* return (cd_reset(periph)); */ error = ENOTTY; break; case CDRIOCREADSPEED: cam_periph_lock(periph); error = cdsetspeed(periph, *(u_int32_t *)addr, CDR_MAX_SPEED); cam_periph_unlock(periph); break; case CDRIOCWRITESPEED: cam_periph_lock(periph); error = cdsetspeed(periph, CDR_MAX_SPEED, *(u_int32_t *)addr); cam_periph_unlock(periph); break; case DVDIOCSENDKEY: case DVDIOCREPORTKEY: { struct dvd_authinfo *authinfo; authinfo = (struct dvd_authinfo *)addr; if (cmd == DVDIOCREPORTKEY) error = cdreportkey(periph, authinfo); else error = cdsendkey(periph, authinfo); break; } case DVDIOCREADSTRUCTURE: { struct dvd_struct *dvdstruct; dvdstruct = (struct dvd_struct *)addr; error = cdreaddvdstructure(periph, dvdstruct); break; } default: cam_periph_lock(periph); error = cam_periph_ioctl(periph, cmd, addr, cderror); cam_periph_unlock(periph); break; } cam_periph_lock(periph); cam_periph_unhold(periph); CAM_DEBUG(periph->path, CAM_DEBUG_TRACE, ("leaving cdioctl\n")); if (error && bootverbose) { printf("scsi_cd.c::ioctl cmd=%08lx error=%d\n", cmd, error); } cam_periph_unlock(periph); return (error); } static void cdprevent(struct cam_periph *periph, int action) { union ccb *ccb; struct cd_softc *softc; int error; CAM_DEBUG(periph->path, CAM_DEBUG_TRACE, ("entering cdprevent\n")); softc = (struct cd_softc *)periph->softc; if (((action == PR_ALLOW) && (softc->flags & CD_FLAG_DISC_LOCKED) == 0) || ((action == PR_PREVENT) && (softc->flags & CD_FLAG_DISC_LOCKED) != 0)) { return; } ccb = cdgetccb(periph, CAM_PRIORITY_NORMAL); scsi_prevent(&ccb->csio, /*retries*/ cd_retry_count, cddone, MSG_SIMPLE_Q_TAG, action, SSD_FULL_SIZE, /* timeout */60000); error = cdrunccb(ccb, cderror, /*cam_flags*/CAM_RETRY_SELTO, /*sense_flags*/SF_RETRY_UA|SF_NO_PRINT); xpt_release_ccb(ccb); if (error == 0) { if (action == PR_ALLOW) softc->flags &= ~CD_FLAG_DISC_LOCKED; else softc->flags |= CD_FLAG_DISC_LOCKED; } } /* * XXX: the disk media and sector size is only really able to change * XXX: while the device is closed. */ static int cdcheckmedia(struct cam_periph *periph) { struct cd_softc *softc; struct ioc_toc_header *toch; struct cd_toc_single leadout; u_int32_t size, toclen; int error, num_entries, cdindex; softc = (struct cd_softc *)periph->softc; cdprevent(periph, PR_PREVENT); softc->disk->d_sectorsize = 2048; softc->disk->d_mediasize = 0; /* * Get the disc size and block size. If we can't get it, we don't * have media, most likely. */ if ((error = cdsize(periph, &size)) != 0) { softc->flags &= ~(CD_FLAG_VALID_MEDIA|CD_FLAG_VALID_TOC); cdprevent(periph, PR_ALLOW); return (error); - } else + } else { softc->flags |= CD_FLAG_VALID_MEDIA; + softc->disk->d_sectorsize = softc->params.blksize; + softc->disk->d_mediasize = + (off_t)softc->params.blksize * softc->params.disksize; + } /* * Now we check the table of contents. This (currently) is only * used for the CDIOCPLAYTRACKS ioctl. It may be used later to do * things like present a separate entry in /dev for each track, * like that acd(4) driver does. */ bzero(&softc->toc, sizeof(softc->toc)); toch = &softc->toc.header; /* * We will get errors here for media that doesn't have a table of * contents. According to the MMC-3 spec: "When a Read TOC/PMA/ATIP * command is presented for a DDCD/CD-R/RW media, where the first TOC * has not been recorded (no complete session) and the Format codes * 0000b, 0001b, or 0010b are specified, this command shall be rejected * with an INVALID FIELD IN CDB. Devices that are not capable of * reading an incomplete session on DDC/CD-R/RW media shall report * CANNOT READ MEDIUM - INCOMPATIBLE FORMAT." * * So this isn't fatal if we can't read the table of contents, it * just means that the user won't be able to issue the play tracks * ioctl, and likely lots of other stuff won't work either. They * need to burn the CD before we can do a whole lot with it. So * we don't print anything here if we get an error back. */ error = cdreadtoc(periph, 0, 0, (u_int8_t *)toch, sizeof(*toch), SF_NO_PRINT); /* * Errors in reading the table of contents aren't fatal, we just * won't have a valid table of contents cached. */ if (error != 0) { error = 0; bzero(&softc->toc, sizeof(softc->toc)); goto bailout; } if (softc->quirks & CD_Q_BCD_TRACKS) { toch->starting_track = bcd2bin(toch->starting_track); toch->ending_track = bcd2bin(toch->ending_track); } /* Number of TOC entries, plus leadout */ num_entries = (toch->ending_track - toch->starting_track) + 2; if (num_entries <= 0) goto bailout; toclen = num_entries * sizeof(struct cd_toc_entry); error = cdreadtoc(periph, CD_MSF_FORMAT, toch->starting_track, (u_int8_t *)&softc->toc, toclen + sizeof(*toch), SF_NO_PRINT); if (error != 0) { error = 0; bzero(&softc->toc, sizeof(softc->toc)); goto bailout; } if (softc->quirks & CD_Q_BCD_TRACKS) { toch->starting_track = bcd2bin(toch->starting_track); toch->ending_track = bcd2bin(toch->ending_track); } /* * XXX KDM is this necessary? Probably only if the drive doesn't * return leadout information with the table of contents. */ cdindex = toch->starting_track + num_entries -1; if (cdindex == toch->ending_track + 1) { error = cdreadtoc(periph, CD_MSF_FORMAT, LEADOUT, (u_int8_t *)&leadout, sizeof(leadout), SF_NO_PRINT); if (error != 0) { error = 0; goto bailout; } softc->toc.entries[cdindex - toch->starting_track] = leadout.entry; } if (softc->quirks & CD_Q_BCD_TRACKS) { for (cdindex = 0; cdindex < num_entries - 1; cdindex++) { softc->toc.entries[cdindex].track = bcd2bin(softc->toc.entries[cdindex].track); } } softc->flags |= CD_FLAG_VALID_TOC; - softc->disk->d_sectorsize = softc->params.blksize; - softc->disk->d_mediasize = - (off_t)softc->params.blksize * softc->params.disksize; bailout: /* * We unconditionally (re)set the blocksize each time the * CD device is opened. This is because the CD can change, * and therefore the blocksize might change. * XXX problems here if some slice or partition is still * open with the old size? */ if ((softc->disk->d_devstat->flags & DEVSTAT_BS_UNAVAILABLE) != 0) softc->disk->d_devstat->flags &= ~DEVSTAT_BS_UNAVAILABLE; softc->disk->d_devstat->block_size = softc->params.blksize; return (error); } static int cdsize(struct cam_periph *periph, u_int32_t *size) { struct cd_softc *softc; union ccb *ccb; struct scsi_read_capacity_data *rcap_buf; int error; CAM_DEBUG(periph->path, CAM_DEBUG_TRACE, ("entering cdsize\n")); softc = (struct cd_softc *)periph->softc; ccb = cdgetccb(periph, CAM_PRIORITY_NORMAL); /* XXX Should be M_WAITOK */ rcap_buf = malloc(sizeof(struct scsi_read_capacity_data), M_SCSICD, M_NOWAIT | M_ZERO); if (rcap_buf == NULL) return (ENOMEM); scsi_read_capacity(&ccb->csio, /*retries*/ cd_retry_count, cddone, MSG_SIMPLE_Q_TAG, rcap_buf, SSD_FULL_SIZE, /* timeout */20000); error = cdrunccb(ccb, cderror, /*cam_flags*/CAM_RETRY_SELTO, /*sense_flags*/SF_RETRY_UA|SF_NO_PRINT); xpt_release_ccb(ccb); softc->params.disksize = scsi_4btoul(rcap_buf->addr) + 1; softc->params.blksize = scsi_4btoul(rcap_buf->length); /* Make sure we got at least some block size. */ if (error == 0 && softc->params.blksize == 0) error = EIO; /* * SCSI-3 mandates that the reported blocksize shall be 2048. * Older drives sometimes report funny values, trim it down to * 2048, or other parts of the kernel will get confused. * * XXX we leave drives alone that might report 512 bytes, as * well as drives reporting more weird sizes like perhaps 4K. */ if (softc->params.blksize > 2048 && softc->params.blksize <= 2352) softc->params.blksize = 2048; free(rcap_buf, M_SCSICD); *size = softc->params.disksize; return (error); } static int cd6byteworkaround(union ccb *ccb) { u_int8_t *cdb; struct cam_periph *periph; struct cd_softc *softc; struct cd_mode_params *params; int frozen, found; periph = xpt_path_periph(ccb->ccb_h.path); softc = (struct cd_softc *)periph->softc; cdb = ccb->csio.cdb_io.cdb_bytes; if ((ccb->ccb_h.flags & CAM_CDB_POINTER) || ((cdb[0] != MODE_SENSE_6) && (cdb[0] != MODE_SELECT_6))) return (0); /* * Because there is no convenient place to stash the overall * cd_mode_params structure pointer, we have to grab it like this. * This means that ALL MODE_SENSE and MODE_SELECT requests in the * cd(4) driver MUST go through cdgetmode() and cdsetmode()! * * XXX It would be nice if, at some point, we could increase the * number of available peripheral private pointers. Both pointers * are currently used in most every peripheral driver. */ found = 0; STAILQ_FOREACH(params, &softc->mode_queue, links) { if (params->mode_buf == ccb->csio.data_ptr) { found = 1; break; } } /* * This shouldn't happen. All mode sense and mode select * operations in the cd(4) driver MUST go through cdgetmode() and * cdsetmode()! */ if (found == 0) { xpt_print(periph->path, "mode buffer not found in mode queue!\n"); return (0); } params->cdb_size = 10; softc->minimum_command_size = 10; xpt_print(ccb->ccb_h.path, "%s(6) failed, increasing minimum CDB size to 10 bytes\n", (cdb[0] == MODE_SENSE_6) ? "MODE_SENSE" : "MODE_SELECT"); if (cdb[0] == MODE_SENSE_6) { struct scsi_mode_sense_10 ms10; struct scsi_mode_sense_6 *ms6; int len; ms6 = (struct scsi_mode_sense_6 *)cdb; bzero(&ms10, sizeof(ms10)); ms10.opcode = MODE_SENSE_10; ms10.byte2 = ms6->byte2; ms10.page = ms6->page; /* * 10 byte mode header, block descriptor, * sizeof(union cd_pages) */ len = sizeof(struct cd_mode_data_10); ccb->csio.dxfer_len = len; scsi_ulto2b(len, ms10.length); ms10.control = ms6->control; bcopy(&ms10, cdb, 10); ccb->csio.cdb_len = 10; } else { struct scsi_mode_select_10 ms10; struct scsi_mode_select_6 *ms6; struct scsi_mode_header_6 *header6; struct scsi_mode_header_10 *header10; struct scsi_mode_page_header *page_header; int blk_desc_len, page_num, page_size, len; ms6 = (struct scsi_mode_select_6 *)cdb; bzero(&ms10, sizeof(ms10)); ms10.opcode = MODE_SELECT_10; ms10.byte2 = ms6->byte2; header6 = (struct scsi_mode_header_6 *)params->mode_buf; header10 = (struct scsi_mode_header_10 *)params->mode_buf; page_header = find_mode_page_6(header6); page_num = page_header->page_code; blk_desc_len = header6->blk_desc_len; page_size = cdgetpagesize(page_num); if (page_size != (page_header->page_length + sizeof(*page_header))) page_size = page_header->page_length + sizeof(*page_header); len = sizeof(*header10) + blk_desc_len + page_size; len = min(params->alloc_len, len); /* * Since the 6 byte parameter header is shorter than the 10 * byte parameter header, we need to copy the actual mode * page data, and the block descriptor, if any, so things wind * up in the right place. The regions will overlap, but * bcopy() does the right thing. */ bcopy(params->mode_buf + sizeof(*header6), params->mode_buf + sizeof(*header10), len - sizeof(*header10)); /* Make sure these fields are set correctly. */ scsi_ulto2b(0, header10->data_length); header10->medium_type = 0; scsi_ulto2b(blk_desc_len, header10->blk_desc_len); ccb->csio.dxfer_len = len; scsi_ulto2b(len, ms10.length); ms10.control = ms6->control; bcopy(&ms10, cdb, 10); ccb->csio.cdb_len = 10; } frozen = (ccb->ccb_h.status & CAM_DEV_QFRZN) != 0; ccb->ccb_h.status = CAM_REQUEUE_REQ; xpt_action(ccb); if (frozen) { cam_release_devq(ccb->ccb_h.path, /*relsim_flags*/0, /*openings*/0, /*timeout*/0, /*getcount_only*/0); } return (ERESTART); } static int cderror(union ccb *ccb, u_int32_t cam_flags, u_int32_t sense_flags) { struct cd_softc *softc; struct cam_periph *periph; int error; periph = xpt_path_periph(ccb->ccb_h.path); softc = (struct cd_softc *)periph->softc; error = 0; /* * We use a status of CAM_REQ_INVALID as shorthand -- if a 6 byte * CDB comes back with this particular error, try transforming it * into the 10 byte version. */ if ((ccb->ccb_h.status & CAM_STATUS_MASK) == CAM_REQ_INVALID) { error = cd6byteworkaround(ccb); } else if (((ccb->ccb_h.status & CAM_STATUS_MASK) == CAM_SCSI_STATUS_ERROR) && (ccb->ccb_h.status & CAM_AUTOSNS_VALID) && (ccb->csio.scsi_status == SCSI_STATUS_CHECK_COND) && ((ccb->ccb_h.flags & CAM_SENSE_PHYS) == 0) && ((ccb->ccb_h.flags & CAM_SENSE_PTR) == 0)) { int sense_key, error_code, asc, ascq; scsi_extract_sense(&ccb->csio.sense_data, &error_code, &sense_key, &asc, &ascq); if (sense_key == SSD_KEY_ILLEGAL_REQUEST) error = cd6byteworkaround(ccb); } if (error == ERESTART) return (error); /* * XXX * Until we have a better way of doing pack validation, * don't treat UAs as errors. */ sense_flags |= SF_RETRY_UA; return (cam_periph_error(ccb, cam_flags, sense_flags, &softc->saved_ccb)); } /* * Read table of contents */ static int cdreadtoc(struct cam_periph *periph, u_int32_t mode, u_int32_t start, u_int8_t *data, u_int32_t len, u_int32_t sense_flags) { struct scsi_read_toc *scsi_cmd; u_int32_t ntoc; struct ccb_scsiio *csio; union ccb *ccb; int error; ntoc = len; error = 0; ccb = cdgetccb(periph, CAM_PRIORITY_NORMAL); csio = &ccb->csio; cam_fill_csio(csio, /* retries */ cd_retry_count, /* cbfcnp */ cddone, /* flags */ CAM_DIR_IN, /* tag_action */ MSG_SIMPLE_Q_TAG, /* data_ptr */ data, /* dxfer_len */ len, /* sense_len */ SSD_FULL_SIZE, sizeof(struct scsi_read_toc), /* timeout */ 50000); scsi_cmd = (struct scsi_read_toc *)&csio->cdb_io.cdb_bytes; bzero (scsi_cmd, sizeof(*scsi_cmd)); if (mode == CD_MSF_FORMAT) scsi_cmd->byte2 |= CD_MSF; scsi_cmd->from_track = start; /* scsi_ulto2b(ntoc, (u_int8_t *)scsi_cmd->data_len); */ scsi_cmd->data_len[0] = (ntoc) >> 8; scsi_cmd->data_len[1] = (ntoc) & 0xff; scsi_cmd->op_code = READ_TOC; error = cdrunccb(ccb, cderror, /*cam_flags*/CAM_RETRY_SELTO, /*sense_flags*/SF_RETRY_UA | sense_flags); xpt_release_ccb(ccb); return(error); } static int cdreadsubchannel(struct cam_periph *periph, u_int32_t mode, u_int32_t format, int track, struct cd_sub_channel_info *data, u_int32_t len) { struct scsi_read_subchannel *scsi_cmd; struct ccb_scsiio *csio; union ccb *ccb; int error; error = 0; ccb = cdgetccb(periph, CAM_PRIORITY_NORMAL); csio = &ccb->csio; cam_fill_csio(csio, /* retries */ cd_retry_count, /* cbfcnp */ cddone, /* flags */ CAM_DIR_IN, /* tag_action */ MSG_SIMPLE_Q_TAG, /* data_ptr */ (u_int8_t *)data, /* dxfer_len */ len, /* sense_len */ SSD_FULL_SIZE, sizeof(struct scsi_read_subchannel), /* timeout */ 50000); scsi_cmd = (struct scsi_read_subchannel *)&csio->cdb_io.cdb_bytes; bzero (scsi_cmd, sizeof(*scsi_cmd)); scsi_cmd->op_code = READ_SUBCHANNEL; if (mode == CD_MSF_FORMAT) scsi_cmd->byte1 |= CD_MSF; scsi_cmd->byte2 = SRS_SUBQ; scsi_cmd->subchan_format = format; scsi_cmd->track = track; scsi_ulto2b(len, (u_int8_t *)scsi_cmd->data_len); scsi_cmd->control = 0; error = cdrunccb(ccb, cderror, /*cam_flags*/CAM_RETRY_SELTO, /*sense_flags*/SF_RETRY_UA); xpt_release_ccb(ccb); return(error); } /* * All MODE_SENSE requests in the cd(4) driver MUST go through this * routine. See comments in cd6byteworkaround() for details. */ static int cdgetmode(struct cam_periph *periph, struct cd_mode_params *data, u_int32_t page) { struct ccb_scsiio *csio; struct cd_softc *softc; union ccb *ccb; int param_len; int error; softc = (struct cd_softc *)periph->softc; ccb = cdgetccb(periph, CAM_PRIORITY_NORMAL); csio = &ccb->csio; data->cdb_size = softc->minimum_command_size; if (data->cdb_size < 10) param_len = sizeof(struct cd_mode_data); else param_len = sizeof(struct cd_mode_data_10); /* Don't say we've got more room than we actually allocated */ param_len = min(param_len, data->alloc_len); scsi_mode_sense_len(csio, /* retries */ cd_retry_count, /* cbfcnp */ cddone, /* tag_action */ MSG_SIMPLE_Q_TAG, /* dbd */ 0, /* page_code */ SMS_PAGE_CTRL_CURRENT, /* page */ page, /* param_buf */ data->mode_buf, /* param_len */ param_len, /* minimum_cmd_size */ softc->minimum_command_size, /* sense_len */ SSD_FULL_SIZE, /* timeout */ 50000); /* * It would be nice not to have to do this, but there's no * available pointer in the CCB that would allow us to stuff the * mode params structure in there and retrieve it in * cd6byteworkaround(), so we can set the cdb size. The cdb size * lets the caller know what CDB size we ended up using, so they * can find the actual mode page offset. */ STAILQ_INSERT_TAIL(&softc->mode_queue, data, links); error = cdrunccb(ccb, cderror, /*cam_flags*/CAM_RETRY_SELTO, /*sense_flags*/SF_RETRY_UA); xpt_release_ccb(ccb); STAILQ_REMOVE(&softc->mode_queue, data, cd_mode_params, links); /* * This is a bit of belt-and-suspenders checking, but if we run * into a situation where the target sends back multiple block * descriptors, we might not have enough space in the buffer to * see the whole mode page. Better to return an error than * potentially access memory beyond our malloced region. */ if (error == 0) { u_int32_t data_len; if (data->cdb_size == 10) { struct scsi_mode_header_10 *hdr10; hdr10 = (struct scsi_mode_header_10 *)data->mode_buf; data_len = scsi_2btoul(hdr10->data_length); data_len += sizeof(hdr10->data_length); } else { struct scsi_mode_header_6 *hdr6; hdr6 = (struct scsi_mode_header_6 *)data->mode_buf; data_len = hdr6->data_length; data_len += sizeof(hdr6->data_length); } /* * Complain if there is more mode data available than we * allocated space for. This could potentially happen if * we miscalculated the page length for some reason, if the * drive returns multiple block descriptors, or if it sets * the data length incorrectly. */ if (data_len > data->alloc_len) { xpt_print(periph->path, "allocated modepage %d length " "%d < returned length %d\n", page, data->alloc_len, data_len); error = ENOSPC; } } return (error); } /* * All MODE_SELECT requests in the cd(4) driver MUST go through this * routine. See comments in cd6byteworkaround() for details. */ static int cdsetmode(struct cam_periph *periph, struct cd_mode_params *data) { struct ccb_scsiio *csio; struct cd_softc *softc; union ccb *ccb; int cdb_size, param_len; int error; softc = (struct cd_softc *)periph->softc; ccb = cdgetccb(periph, CAM_PRIORITY_NORMAL); csio = &ccb->csio; error = 0; /* * If the data is formatted for the 10 byte version of the mode * select parameter list, we need to use the 10 byte CDB. * Otherwise, we use whatever the stored minimum command size. */ if (data->cdb_size == 10) cdb_size = data->cdb_size; else cdb_size = softc->minimum_command_size; if (cdb_size >= 10) { struct scsi_mode_header_10 *mode_header; u_int32_t data_len; mode_header = (struct scsi_mode_header_10 *)data->mode_buf; data_len = scsi_2btoul(mode_header->data_length); scsi_ulto2b(0, mode_header->data_length); /* * SONY drives do not allow a mode select with a medium_type * value that has just been returned by a mode sense; use a * medium_type of 0 (Default) instead. */ mode_header->medium_type = 0; /* * Pass back whatever the drive passed to us, plus the size * of the data length field. */ param_len = data_len + sizeof(mode_header->data_length); } else { struct scsi_mode_header_6 *mode_header; mode_header = (struct scsi_mode_header_6 *)data->mode_buf; param_len = mode_header->data_length + 1; mode_header->data_length = 0; /* * SONY drives do not allow a mode select with a medium_type * value that has just been returned by a mode sense; use a * medium_type of 0 (Default) instead. */ mode_header->medium_type = 0; } /* Don't say we've got more room than we actually allocated */ param_len = min(param_len, data->alloc_len); scsi_mode_select_len(csio, /* retries */ cd_retry_count, /* cbfcnp */ cddone, /* tag_action */ MSG_SIMPLE_Q_TAG, /* scsi_page_fmt */ 1, /* save_pages */ 0, /* param_buf */ data->mode_buf, /* param_len */ param_len, /* minimum_cmd_size */ cdb_size, /* sense_len */ SSD_FULL_SIZE, /* timeout */ 50000); /* See comments in cdgetmode() and cd6byteworkaround(). */ STAILQ_INSERT_TAIL(&softc->mode_queue, data, links); error = cdrunccb(ccb, cderror, /*cam_flags*/CAM_RETRY_SELTO, /*sense_flags*/SF_RETRY_UA); xpt_release_ccb(ccb); STAILQ_REMOVE(&softc->mode_queue, data, cd_mode_params, links); return (error); } static int cdplay(struct cam_periph *periph, u_int32_t blk, u_int32_t len) { struct ccb_scsiio *csio; union ccb *ccb; int error; u_int8_t cdb_len; error = 0; ccb = cdgetccb(periph, CAM_PRIORITY_NORMAL); csio = &ccb->csio; /* * Use the smallest possible command to perform the operation. */ if ((len & 0xffff0000) == 0) { /* * We can fit in a 10 byte cdb. */ struct scsi_play_10 *scsi_cmd; scsi_cmd = (struct scsi_play_10 *)&csio->cdb_io.cdb_bytes; bzero (scsi_cmd, sizeof(*scsi_cmd)); scsi_cmd->op_code = PLAY_10; scsi_ulto4b(blk, (u_int8_t *)scsi_cmd->blk_addr); scsi_ulto2b(len, (u_int8_t *)scsi_cmd->xfer_len); cdb_len = sizeof(*scsi_cmd); } else { struct scsi_play_12 *scsi_cmd; scsi_cmd = (struct scsi_play_12 *)&csio->cdb_io.cdb_bytes; bzero (scsi_cmd, sizeof(*scsi_cmd)); scsi_cmd->op_code = PLAY_12; scsi_ulto4b(blk, (u_int8_t *)scsi_cmd->blk_addr); scsi_ulto4b(len, (u_int8_t *)scsi_cmd->xfer_len); cdb_len = sizeof(*scsi_cmd); } cam_fill_csio(csio, /*retries*/ cd_retry_count, cddone, /*flags*/CAM_DIR_NONE, MSG_SIMPLE_Q_TAG, /*dataptr*/NULL, /*datalen*/0, /*sense_len*/SSD_FULL_SIZE, cdb_len, /*timeout*/50 * 1000); error = cdrunccb(ccb, cderror, /*cam_flags*/CAM_RETRY_SELTO, /*sense_flags*/SF_RETRY_UA); xpt_release_ccb(ccb); return(error); } static int cdplaymsf(struct cam_periph *periph, u_int32_t startm, u_int32_t starts, u_int32_t startf, u_int32_t endm, u_int32_t ends, u_int32_t endf) { struct scsi_play_msf *scsi_cmd; struct ccb_scsiio *csio; union ccb *ccb; int error; error = 0; ccb = cdgetccb(periph, CAM_PRIORITY_NORMAL); csio = &ccb->csio; cam_fill_csio(csio, /* retries */ cd_retry_count, /* cbfcnp */ cddone, /* flags */ CAM_DIR_NONE, /* tag_action */ MSG_SIMPLE_Q_TAG, /* data_ptr */ NULL, /* dxfer_len */ 0, /* sense_len */ SSD_FULL_SIZE, sizeof(struct scsi_play_msf), /* timeout */ 50000); scsi_cmd = (struct scsi_play_msf *)&csio->cdb_io.cdb_bytes; bzero (scsi_cmd, sizeof(*scsi_cmd)); scsi_cmd->op_code = PLAY_MSF; scsi_cmd->start_m = startm; scsi_cmd->start_s = starts; scsi_cmd->start_f = startf; scsi_cmd->end_m = endm; scsi_cmd->end_s = ends; scsi_cmd->end_f = endf; error = cdrunccb(ccb, cderror, /*cam_flags*/CAM_RETRY_SELTO, /*sense_flags*/SF_RETRY_UA); xpt_release_ccb(ccb); return(error); } static int cdplaytracks(struct cam_periph *periph, u_int32_t strack, u_int32_t sindex, u_int32_t etrack, u_int32_t eindex) { struct scsi_play_track *scsi_cmd; struct ccb_scsiio *csio; union ccb *ccb; int error; error = 0; ccb = cdgetccb(periph, CAM_PRIORITY_NORMAL); csio = &ccb->csio; cam_fill_csio(csio, /* retries */ cd_retry_count, /* cbfcnp */ cddone, /* flags */ CAM_DIR_NONE, /* tag_action */ MSG_SIMPLE_Q_TAG, /* data_ptr */ NULL, /* dxfer_len */ 0, /* sense_len */ SSD_FULL_SIZE, sizeof(struct scsi_play_track), /* timeout */ 50000); scsi_cmd = (struct scsi_play_track *)&csio->cdb_io.cdb_bytes; bzero (scsi_cmd, sizeof(*scsi_cmd)); scsi_cmd->op_code = PLAY_TRACK; scsi_cmd->start_track = strack; scsi_cmd->start_index = sindex; scsi_cmd->end_track = etrack; scsi_cmd->end_index = eindex; error = cdrunccb(ccb, cderror, /*cam_flags*/CAM_RETRY_SELTO, /*sense_flags*/SF_RETRY_UA); xpt_release_ccb(ccb); return(error); } static int cdpause(struct cam_periph *periph, u_int32_t go) { struct scsi_pause *scsi_cmd; struct ccb_scsiio *csio; union ccb *ccb; int error; error = 0; ccb = cdgetccb(periph, CAM_PRIORITY_NORMAL); csio = &ccb->csio; cam_fill_csio(csio, /* retries */ cd_retry_count, /* cbfcnp */ cddone, /* flags */ CAM_DIR_NONE, /* tag_action */ MSG_SIMPLE_Q_TAG, /* data_ptr */ NULL, /* dxfer_len */ 0, /* sense_len */ SSD_FULL_SIZE, sizeof(struct scsi_pause), /* timeout */ 50000); scsi_cmd = (struct scsi_pause *)&csio->cdb_io.cdb_bytes; bzero (scsi_cmd, sizeof(*scsi_cmd)); scsi_cmd->op_code = PAUSE; scsi_cmd->resume = go; error = cdrunccb(ccb, cderror, /*cam_flags*/CAM_RETRY_SELTO, /*sense_flags*/SF_RETRY_UA); xpt_release_ccb(ccb); return(error); } static int cdstartunit(struct cam_periph *periph, int load) { union ccb *ccb; int error; error = 0; ccb = cdgetccb(periph, CAM_PRIORITY_NORMAL); scsi_start_stop(&ccb->csio, /* retries */ cd_retry_count, /* cbfcnp */ cddone, /* tag_action */ MSG_SIMPLE_Q_TAG, /* start */ TRUE, /* load_eject */ load, /* immediate */ FALSE, /* sense_len */ SSD_FULL_SIZE, /* timeout */ 50000); error = cdrunccb(ccb, cderror, /*cam_flags*/CAM_RETRY_SELTO, /*sense_flags*/SF_RETRY_UA); xpt_release_ccb(ccb); return(error); } static int cdstopunit(struct cam_periph *periph, u_int32_t eject) { union ccb *ccb; int error; error = 0; ccb = cdgetccb(periph, CAM_PRIORITY_NORMAL); scsi_start_stop(&ccb->csio, /* retries */ cd_retry_count, /* cbfcnp */ cddone, /* tag_action */ MSG_SIMPLE_Q_TAG, /* start */ FALSE, /* load_eject */ eject, /* immediate */ FALSE, /* sense_len */ SSD_FULL_SIZE, /* timeout */ 50000); error = cdrunccb(ccb, cderror, /*cam_flags*/CAM_RETRY_SELTO, /*sense_flags*/SF_RETRY_UA); xpt_release_ccb(ccb); return(error); } static int cdsetspeed(struct cam_periph *periph, u_int32_t rdspeed, u_int32_t wrspeed) { struct scsi_set_speed *scsi_cmd; struct ccb_scsiio *csio; union ccb *ccb; int error; error = 0; ccb = cdgetccb(periph, CAM_PRIORITY_NORMAL); csio = &ccb->csio; /* Preserve old behavior: units in multiples of CDROM speed */ if (rdspeed < 177) rdspeed *= 177; if (wrspeed < 177) wrspeed *= 177; cam_fill_csio(csio, /* retries */ cd_retry_count, /* cbfcnp */ cddone, /* flags */ CAM_DIR_NONE, /* tag_action */ MSG_SIMPLE_Q_TAG, /* data_ptr */ NULL, /* dxfer_len */ 0, /* sense_len */ SSD_FULL_SIZE, sizeof(struct scsi_set_speed), /* timeout */ 50000); scsi_cmd = (struct scsi_set_speed *)&csio->cdb_io.cdb_bytes; bzero(scsi_cmd, sizeof(*scsi_cmd)); scsi_cmd->opcode = SET_CD_SPEED; scsi_ulto2b(rdspeed, scsi_cmd->readspeed); scsi_ulto2b(wrspeed, scsi_cmd->writespeed); error = cdrunccb(ccb, cderror, /*cam_flags*/CAM_RETRY_SELTO, /*sense_flags*/SF_RETRY_UA); xpt_release_ccb(ccb); return(error); } static int cdreportkey(struct cam_periph *periph, struct dvd_authinfo *authinfo) { union ccb *ccb; u_int8_t *databuf; u_int32_t lba; int error; int length; error = 0; databuf = NULL; lba = 0; switch (authinfo->format) { case DVD_REPORT_AGID: length = sizeof(struct scsi_report_key_data_agid); break; case DVD_REPORT_CHALLENGE: length = sizeof(struct scsi_report_key_data_challenge); break; case DVD_REPORT_KEY1: length = sizeof(struct scsi_report_key_data_key1_key2); break; case DVD_REPORT_TITLE_KEY: length = sizeof(struct scsi_report_key_data_title); /* The lba field is only set for the title key */ lba = authinfo->lba; break; case DVD_REPORT_ASF: length = sizeof(struct scsi_report_key_data_asf); break; case DVD_REPORT_RPC: length = sizeof(struct scsi_report_key_data_rpc); break; case DVD_INVALIDATE_AGID: length = 0; break; default: return (EINVAL); } if (length != 0) { databuf = malloc(length, M_DEVBUF, M_WAITOK | M_ZERO); } else databuf = NULL; cam_periph_lock(periph); ccb = cdgetccb(periph, CAM_PRIORITY_NORMAL); scsi_report_key(&ccb->csio, /* retries */ cd_retry_count, /* cbfcnp */ cddone, /* tag_action */ MSG_SIMPLE_Q_TAG, /* lba */ lba, /* agid */ authinfo->agid, /* key_format */ authinfo->format, /* data_ptr */ databuf, /* dxfer_len */ length, /* sense_len */ SSD_FULL_SIZE, /* timeout */ 50000); error = cdrunccb(ccb, cderror, /*cam_flags*/CAM_RETRY_SELTO, /*sense_flags*/SF_RETRY_UA); if (error != 0) goto bailout; if (ccb->csio.resid != 0) { xpt_print(periph->path, "warning, residual for report key " "command is %d\n", ccb->csio.resid); } switch(authinfo->format) { case DVD_REPORT_AGID: { struct scsi_report_key_data_agid *agid_data; agid_data = (struct scsi_report_key_data_agid *)databuf; authinfo->agid = (agid_data->agid & RKD_AGID_MASK) >> RKD_AGID_SHIFT; break; } case DVD_REPORT_CHALLENGE: { struct scsi_report_key_data_challenge *chal_data; chal_data = (struct scsi_report_key_data_challenge *)databuf; bcopy(chal_data->challenge_key, authinfo->keychal, min(sizeof(chal_data->challenge_key), sizeof(authinfo->keychal))); break; } case DVD_REPORT_KEY1: { struct scsi_report_key_data_key1_key2 *key1_data; key1_data = (struct scsi_report_key_data_key1_key2 *)databuf; bcopy(key1_data->key1, authinfo->keychal, min(sizeof(key1_data->key1), sizeof(authinfo->keychal))); break; } case DVD_REPORT_TITLE_KEY: { struct scsi_report_key_data_title *title_data; title_data = (struct scsi_report_key_data_title *)databuf; authinfo->cpm = (title_data->byte0 & RKD_TITLE_CPM) >> RKD_TITLE_CPM_SHIFT; authinfo->cp_sec = (title_data->byte0 & RKD_TITLE_CP_SEC) >> RKD_TITLE_CP_SEC_SHIFT; authinfo->cgms = (title_data->byte0 & RKD_TITLE_CMGS_MASK) >> RKD_TITLE_CMGS_SHIFT; bcopy(title_data->title_key, authinfo->keychal, min(sizeof(title_data->title_key), sizeof(authinfo->keychal))); break; } case DVD_REPORT_ASF: { struct scsi_report_key_data_asf *asf_data; asf_data = (struct scsi_report_key_data_asf *)databuf; authinfo->asf = asf_data->success & RKD_ASF_SUCCESS; break; } case DVD_REPORT_RPC: { struct scsi_report_key_data_rpc *rpc_data; rpc_data = (struct scsi_report_key_data_rpc *)databuf; authinfo->reg_type = (rpc_data->byte4 & RKD_RPC_TYPE_MASK) >> RKD_RPC_TYPE_SHIFT; authinfo->vend_rsts = (rpc_data->byte4 & RKD_RPC_VENDOR_RESET_MASK) >> RKD_RPC_VENDOR_RESET_SHIFT; authinfo->user_rsts = rpc_data->byte4 & RKD_RPC_USER_RESET_MASK; authinfo->region = rpc_data->region_mask; authinfo->rpc_scheme = rpc_data->rpc_scheme1; break; } case DVD_INVALIDATE_AGID: break; default: /* This should be impossible, since we checked above */ error = EINVAL; goto bailout; break; /* NOTREACHED */ } bailout: xpt_release_ccb(ccb); cam_periph_unlock(periph); if (databuf != NULL) free(databuf, M_DEVBUF); return(error); } static int cdsendkey(struct cam_periph *periph, struct dvd_authinfo *authinfo) { union ccb *ccb; u_int8_t *databuf; int length; int error; error = 0; databuf = NULL; switch(authinfo->format) { case DVD_SEND_CHALLENGE: { struct scsi_report_key_data_challenge *challenge_data; length = sizeof(*challenge_data); challenge_data = malloc(length, M_DEVBUF, M_WAITOK | M_ZERO); databuf = (u_int8_t *)challenge_data; scsi_ulto2b(length - sizeof(challenge_data->data_len), challenge_data->data_len); bcopy(authinfo->keychal, challenge_data->challenge_key, min(sizeof(authinfo->keychal), sizeof(challenge_data->challenge_key))); break; } case DVD_SEND_KEY2: { struct scsi_report_key_data_key1_key2 *key2_data; length = sizeof(*key2_data); key2_data = malloc(length, M_DEVBUF, M_WAITOK | M_ZERO); databuf = (u_int8_t *)key2_data; scsi_ulto2b(length - sizeof(key2_data->data_len), key2_data->data_len); bcopy(authinfo->keychal, key2_data->key1, min(sizeof(authinfo->keychal), sizeof(key2_data->key1))); break; } case DVD_SEND_RPC: { struct scsi_send_key_data_rpc *rpc_data; length = sizeof(*rpc_data); rpc_data = malloc(length, M_DEVBUF, M_WAITOK | M_ZERO); databuf = (u_int8_t *)rpc_data; scsi_ulto2b(length - sizeof(rpc_data->data_len), rpc_data->data_len); rpc_data->region_code = authinfo->region; break; } default: return (EINVAL); } cam_periph_lock(periph); ccb = cdgetccb(periph, CAM_PRIORITY_NORMAL); scsi_send_key(&ccb->csio, /* retries */ cd_retry_count, /* cbfcnp */ cddone, /* tag_action */ MSG_SIMPLE_Q_TAG, /* agid */ authinfo->agid, /* key_format */ authinfo->format, /* data_ptr */ databuf, /* dxfer_len */ length, /* sense_len */ SSD_FULL_SIZE, /* timeout */ 50000); error = cdrunccb(ccb, cderror, /*cam_flags*/CAM_RETRY_SELTO, /*sense_flags*/SF_RETRY_UA); xpt_release_ccb(ccb); cam_periph_unlock(periph); if (databuf != NULL) free(databuf, M_DEVBUF); return(error); } static int cdreaddvdstructure(struct cam_periph *periph, struct dvd_struct *dvdstruct) { union ccb *ccb; u_int8_t *databuf; u_int32_t address; int error; int length; error = 0; databuf = NULL; /* The address is reserved for many of the formats */ address = 0; switch(dvdstruct->format) { case DVD_STRUCT_PHYSICAL: length = sizeof(struct scsi_read_dvd_struct_data_physical); break; case DVD_STRUCT_COPYRIGHT: length = sizeof(struct scsi_read_dvd_struct_data_copyright); break; case DVD_STRUCT_DISCKEY: length = sizeof(struct scsi_read_dvd_struct_data_disc_key); break; case DVD_STRUCT_BCA: length = sizeof(struct scsi_read_dvd_struct_data_bca); break; case DVD_STRUCT_MANUFACT: length = sizeof(struct scsi_read_dvd_struct_data_manufacturer); break; case DVD_STRUCT_CMI: return (ENODEV); case DVD_STRUCT_PROTDISCID: length = sizeof(struct scsi_read_dvd_struct_data_prot_discid); break; case DVD_STRUCT_DISCKEYBLOCK: length = sizeof(struct scsi_read_dvd_struct_data_disc_key_blk); break; case DVD_STRUCT_DDS: length = sizeof(struct scsi_read_dvd_struct_data_dds); break; case DVD_STRUCT_MEDIUM_STAT: length = sizeof(struct scsi_read_dvd_struct_data_medium_status); break; case DVD_STRUCT_SPARE_AREA: length = sizeof(struct scsi_read_dvd_struct_data_spare_area); break; case DVD_STRUCT_RMD_LAST: return (ENODEV); case DVD_STRUCT_RMD_RMA: return (ENODEV); case DVD_STRUCT_PRERECORDED: length = sizeof(struct scsi_read_dvd_struct_data_leadin); break; case DVD_STRUCT_UNIQUEID: length = sizeof(struct scsi_read_dvd_struct_data_disc_id); break; case DVD_STRUCT_DCB: return (ENODEV); case DVD_STRUCT_LIST: /* * This is the maximum allocation length for the READ DVD * STRUCTURE command. There's nothing in the MMC3 spec * that indicates a limit in the amount of data that can * be returned from this call, other than the limits * imposed by the 2-byte length variables. */ length = 65535; break; default: return (EINVAL); } if (length != 0) { databuf = malloc(length, M_DEVBUF, M_WAITOK | M_ZERO); } else databuf = NULL; cam_periph_lock(periph); ccb = cdgetccb(periph, CAM_PRIORITY_NORMAL); scsi_read_dvd_structure(&ccb->csio, /* retries */ cd_retry_count, /* cbfcnp */ cddone, /* tag_action */ MSG_SIMPLE_Q_TAG, /* lba */ address, /* layer_number */ dvdstruct->layer_num, /* key_format */ dvdstruct->format, /* agid */ dvdstruct->agid, /* data_ptr */ databuf, /* dxfer_len */ length, /* sense_len */ SSD_FULL_SIZE, /* timeout */ 50000); error = cdrunccb(ccb, cderror, /*cam_flags*/CAM_RETRY_SELTO, /*sense_flags*/SF_RETRY_UA); if (error != 0) goto bailout; switch(dvdstruct->format) { case DVD_STRUCT_PHYSICAL: { struct scsi_read_dvd_struct_data_layer_desc *inlayer; struct dvd_layer *outlayer; struct scsi_read_dvd_struct_data_physical *phys_data; phys_data = (struct scsi_read_dvd_struct_data_physical *)databuf; inlayer = &phys_data->layer_desc; outlayer = (struct dvd_layer *)&dvdstruct->data; dvdstruct->length = sizeof(*inlayer); outlayer->book_type = (inlayer->book_type_version & RDSD_BOOK_TYPE_MASK) >> RDSD_BOOK_TYPE_SHIFT; outlayer->book_version = (inlayer->book_type_version & RDSD_BOOK_VERSION_MASK); outlayer->disc_size = (inlayer->disc_size_max_rate & RDSD_DISC_SIZE_MASK) >> RDSD_DISC_SIZE_SHIFT; outlayer->max_rate = (inlayer->disc_size_max_rate & RDSD_MAX_RATE_MASK); outlayer->nlayers = (inlayer->layer_info & RDSD_NUM_LAYERS_MASK) >> RDSD_NUM_LAYERS_SHIFT; outlayer->track_path = (inlayer->layer_info & RDSD_TRACK_PATH_MASK) >> RDSD_TRACK_PATH_SHIFT; outlayer->layer_type = (inlayer->layer_info & RDSD_LAYER_TYPE_MASK); outlayer->linear_density = (inlayer->density & RDSD_LIN_DENSITY_MASK) >> RDSD_LIN_DENSITY_SHIFT; outlayer->track_density = (inlayer->density & RDSD_TRACK_DENSITY_MASK); outlayer->bca = (inlayer->bca & RDSD_BCA_MASK) >> RDSD_BCA_SHIFT; outlayer->start_sector = scsi_3btoul(inlayer->main_data_start); outlayer->end_sector = scsi_3btoul(inlayer->main_data_end); outlayer->end_sector_l0 = scsi_3btoul(inlayer->end_sector_layer0); break; } case DVD_STRUCT_COPYRIGHT: { struct scsi_read_dvd_struct_data_copyright *copy_data; copy_data = (struct scsi_read_dvd_struct_data_copyright *) databuf; dvdstruct->cpst = copy_data->cps_type; dvdstruct->rmi = copy_data->region_info; dvdstruct->length = 0; break; } default: /* * Tell the user what the overall length is, no matter * what we can actually fit in the data buffer. */ dvdstruct->length = length - ccb->csio.resid - sizeof(struct scsi_read_dvd_struct_data_header); /* * But only actually copy out the smaller of what we read * in or what the structure can take. */ bcopy(databuf + sizeof(struct scsi_read_dvd_struct_data_header), dvdstruct->data, min(sizeof(dvdstruct->data), dvdstruct->length)); break; } bailout: xpt_release_ccb(ccb); cam_periph_unlock(periph); if (databuf != NULL) free(databuf, M_DEVBUF); return(error); } void scsi_report_key(struct ccb_scsiio *csio, u_int32_t retries, void (*cbfcnp)(struct cam_periph *, union ccb *), u_int8_t tag_action, u_int32_t lba, u_int8_t agid, u_int8_t key_format, u_int8_t *data_ptr, u_int32_t dxfer_len, u_int8_t sense_len, u_int32_t timeout) { struct scsi_report_key *scsi_cmd; scsi_cmd = (struct scsi_report_key *)&csio->cdb_io.cdb_bytes; bzero(scsi_cmd, sizeof(*scsi_cmd)); scsi_cmd->opcode = REPORT_KEY; scsi_ulto4b(lba, scsi_cmd->lba); scsi_ulto2b(dxfer_len, scsi_cmd->alloc_len); scsi_cmd->agid_keyformat = (agid << RK_KF_AGID_SHIFT) | (key_format & RK_KF_KEYFORMAT_MASK); cam_fill_csio(csio, retries, cbfcnp, /*flags*/ (dxfer_len == 0) ? CAM_DIR_NONE : CAM_DIR_IN, tag_action, /*data_ptr*/ data_ptr, /*dxfer_len*/ dxfer_len, sense_len, sizeof(*scsi_cmd), timeout); } void scsi_send_key(struct ccb_scsiio *csio, u_int32_t retries, void (*cbfcnp)(struct cam_periph *, union ccb *), u_int8_t tag_action, u_int8_t agid, u_int8_t key_format, u_int8_t *data_ptr, u_int32_t dxfer_len, u_int8_t sense_len, u_int32_t timeout) { struct scsi_send_key *scsi_cmd; scsi_cmd = (struct scsi_send_key *)&csio->cdb_io.cdb_bytes; bzero(scsi_cmd, sizeof(*scsi_cmd)); scsi_cmd->opcode = SEND_KEY; scsi_ulto2b(dxfer_len, scsi_cmd->param_len); scsi_cmd->agid_keyformat = (agid << RK_KF_AGID_SHIFT) | (key_format & RK_KF_KEYFORMAT_MASK); cam_fill_csio(csio, retries, cbfcnp, /*flags*/ CAM_DIR_OUT, tag_action, /*data_ptr*/ data_ptr, /*dxfer_len*/ dxfer_len, sense_len, sizeof(*scsi_cmd), timeout); } void scsi_read_dvd_structure(struct ccb_scsiio *csio, u_int32_t retries, void (*cbfcnp)(struct cam_periph *, union ccb *), u_int8_t tag_action, u_int32_t address, u_int8_t layer_number, u_int8_t format, u_int8_t agid, u_int8_t *data_ptr, u_int32_t dxfer_len, u_int8_t sense_len, u_int32_t timeout) { struct scsi_read_dvd_structure *scsi_cmd; scsi_cmd = (struct scsi_read_dvd_structure *)&csio->cdb_io.cdb_bytes; bzero(scsi_cmd, sizeof(*scsi_cmd)); scsi_cmd->opcode = READ_DVD_STRUCTURE; scsi_ulto4b(address, scsi_cmd->address); scsi_cmd->layer_number = layer_number; scsi_cmd->format = format; scsi_ulto2b(dxfer_len, scsi_cmd->alloc_len); /* The AGID is the top two bits of this byte */ scsi_cmd->agid = agid << 6; cam_fill_csio(csio, retries, cbfcnp, /*flags*/ CAM_DIR_IN, tag_action, /*data_ptr*/ data_ptr, /*dxfer_len*/ dxfer_len, sense_len, sizeof(*scsi_cmd), timeout); } Index: user/imp/tbemd/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vfsops.c =================================================================== --- user/imp/tbemd/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vfsops.c (revision 206684) +++ user/imp/tbemd/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vfsops.c (revision 206685) @@ -1,1516 +1,1519 @@ /* * CDDL HEADER START * * The contents of this file are subject to the terms of the * Common Development and Distribution License (the "License"). * You may not use this file except in compliance with the License. * * You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE * or http://www.opensolaris.org/os/licensing. * See the License for the specific language governing permissions * and limitations under the License. * * When distributing Covered Code, include this CDDL HEADER in each * file and include the License file at usr/src/OPENSOLARIS.LICENSE. * If applicable, add the following below this CDDL HEADER, with the * fields enclosed by brackets "[]" replaced with your own identifying * information: Portions Copyright [yyyy] [name of copyright owner] * * CDDL HEADER END */ /* * Copyright 2008 Sun Microsystems, Inc. All rights reserved. * Use is subject to license terms. */ #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include /* VDEV_BOOT_VERSION */ struct mtx zfs_debug_mtx; MTX_SYSINIT(zfs_debug_mtx, &zfs_debug_mtx, "zfs_debug", MTX_DEF); SYSCTL_NODE(_vfs, OID_AUTO, zfs, CTLFLAG_RW, 0, "ZFS file system"); int zfs_super_owner = 0; SYSCTL_INT(_vfs_zfs, OID_AUTO, super_owner, CTLFLAG_RW, &zfs_super_owner, 0, "File system owner can perform privileged operation on his file systems"); int zfs_debug_level = 0; TUNABLE_INT("vfs.zfs.debug", &zfs_debug_level); SYSCTL_INT(_vfs_zfs, OID_AUTO, debug, CTLFLAG_RW, &zfs_debug_level, 0, "Debug level"); SYSCTL_NODE(_vfs_zfs, OID_AUTO, version, CTLFLAG_RD, 0, "ZFS versions"); static int zfs_version_acl = ZFS_ACL_VERSION; SYSCTL_INT(_vfs_zfs_version, OID_AUTO, acl, CTLFLAG_RD, &zfs_version_acl, 0, "ZFS_ACL_VERSION"); static int zfs_version_dmu_backup_header = DMU_BACKUP_HEADER_VERSION; SYSCTL_INT(_vfs_zfs_version, OID_AUTO, dmu_backup_header, CTLFLAG_RD, &zfs_version_dmu_backup_header, 0, "DMU_BACKUP_HEADER_VERSION"); static int zfs_version_dmu_backup_stream = DMU_BACKUP_STREAM_VERSION; SYSCTL_INT(_vfs_zfs_version, OID_AUTO, dmu_backup_stream, CTLFLAG_RD, &zfs_version_dmu_backup_stream, 0, "DMU_BACKUP_STREAM_VERSION"); static int zfs_version_spa = SPA_VERSION; SYSCTL_INT(_vfs_zfs_version, OID_AUTO, spa, CTLFLAG_RD, &zfs_version_spa, 0, "SPA_VERSION"); static int zfs_version_vdev_boot = VDEV_BOOT_VERSION; SYSCTL_INT(_vfs_zfs_version, OID_AUTO, vdev_boot, CTLFLAG_RD, &zfs_version_vdev_boot, 0, "VDEV_BOOT_VERSION"); static int zfs_version_zpl = ZPL_VERSION; SYSCTL_INT(_vfs_zfs_version, OID_AUTO, zpl, CTLFLAG_RD, &zfs_version_zpl, 0, "ZPL_VERSION"); static int zfs_mount(vfs_t *vfsp); static int zfs_umount(vfs_t *vfsp, int fflag); static int zfs_root(vfs_t *vfsp, int flags, vnode_t **vpp); static int zfs_statfs(vfs_t *vfsp, struct statfs *statp); static int zfs_vget(vfs_t *vfsp, ino_t ino, int flags, vnode_t **vpp); static int zfs_sync(vfs_t *vfsp, int waitfor); static int zfs_checkexp(vfs_t *vfsp, struct sockaddr *nam, int *extflagsp, struct ucred **credanonp, int *numsecflavors, int **secflavors); static int zfs_fhtovp(vfs_t *vfsp, fid_t *fidp, vnode_t **vpp); static void zfs_objset_close(zfsvfs_t *zfsvfs); static void zfs_freevfs(vfs_t *vfsp); static struct vfsops zfs_vfsops = { .vfs_mount = zfs_mount, .vfs_unmount = zfs_umount, .vfs_root = zfs_root, .vfs_statfs = zfs_statfs, .vfs_vget = zfs_vget, .vfs_sync = zfs_sync, .vfs_checkexp = zfs_checkexp, .vfs_fhtovp = zfs_fhtovp, }; VFS_SET(zfs_vfsops, zfs, VFCF_JAIL | VFCF_DELEGADMIN); /* * We need to keep a count of active fs's. * This is necessary to prevent our module * from being unloaded after a umount -f */ static uint32_t zfs_active_fs_count = 0; /*ARGSUSED*/ static int zfs_sync(vfs_t *vfsp, int waitfor) { /* * Data integrity is job one. We don't want a compromised kernel * writing to the storage pool, so we never sync during panic. */ if (panicstr) return (0); if (vfsp != NULL) { /* * Sync a specific filesystem. */ zfsvfs_t *zfsvfs = vfsp->vfs_data; int error; error = vfs_stdsync(vfsp, waitfor); if (error != 0) return (error); ZFS_ENTER(zfsvfs); if (zfsvfs->z_log != NULL) zil_commit(zfsvfs->z_log, UINT64_MAX, 0); else txg_wait_synced(dmu_objset_pool(zfsvfs->z_os), 0); ZFS_EXIT(zfsvfs); } else { /* * Sync all ZFS filesystems. This is what happens when you * run sync(1M). Unlike other filesystems, ZFS honors the * request by waiting for all pools to commit all dirty data. */ spa_sync_allpools(); } return (0); } static void atime_changed_cb(void *arg, uint64_t newval) { zfsvfs_t *zfsvfs = arg; if (newval == TRUE) { zfsvfs->z_atime = TRUE; zfsvfs->z_vfs->vfs_flag &= ~MNT_NOATIME; vfs_clearmntopt(zfsvfs->z_vfs, MNTOPT_NOATIME); vfs_setmntopt(zfsvfs->z_vfs, MNTOPT_ATIME, NULL, 0); } else { zfsvfs->z_atime = FALSE; zfsvfs->z_vfs->vfs_flag |= MNT_NOATIME; vfs_clearmntopt(zfsvfs->z_vfs, MNTOPT_ATIME); vfs_setmntopt(zfsvfs->z_vfs, MNTOPT_NOATIME, NULL, 0); } } static void xattr_changed_cb(void *arg, uint64_t newval) { zfsvfs_t *zfsvfs = arg; if (newval == TRUE) { /* XXX locking on vfs_flag? */ #ifdef TODO zfsvfs->z_vfs->vfs_flag |= VFS_XATTR; #endif vfs_clearmntopt(zfsvfs->z_vfs, MNTOPT_NOXATTR); vfs_setmntopt(zfsvfs->z_vfs, MNTOPT_XATTR, NULL, 0); } else { /* XXX locking on vfs_flag? */ #ifdef TODO zfsvfs->z_vfs->vfs_flag &= ~VFS_XATTR; #endif vfs_clearmntopt(zfsvfs->z_vfs, MNTOPT_XATTR); vfs_setmntopt(zfsvfs->z_vfs, MNTOPT_NOXATTR, NULL, 0); } } static void blksz_changed_cb(void *arg, uint64_t newval) { zfsvfs_t *zfsvfs = arg; if (newval < SPA_MINBLOCKSIZE || newval > SPA_MAXBLOCKSIZE || !ISP2(newval)) newval = SPA_MAXBLOCKSIZE; zfsvfs->z_max_blksz = newval; zfsvfs->z_vfs->mnt_stat.f_iosize = newval; } static void readonly_changed_cb(void *arg, uint64_t newval) { zfsvfs_t *zfsvfs = arg; if (newval) { /* XXX locking on vfs_flag? */ zfsvfs->z_vfs->vfs_flag |= VFS_RDONLY; vfs_clearmntopt(zfsvfs->z_vfs, MNTOPT_RW); vfs_setmntopt(zfsvfs->z_vfs, MNTOPT_RO, NULL, 0); } else { /* XXX locking on vfs_flag? */ zfsvfs->z_vfs->vfs_flag &= ~VFS_RDONLY; vfs_clearmntopt(zfsvfs->z_vfs, MNTOPT_RO); vfs_setmntopt(zfsvfs->z_vfs, MNTOPT_RW, NULL, 0); } } static void setuid_changed_cb(void *arg, uint64_t newval) { zfsvfs_t *zfsvfs = arg; if (newval == FALSE) { zfsvfs->z_vfs->vfs_flag |= VFS_NOSETUID; vfs_clearmntopt(zfsvfs->z_vfs, MNTOPT_SETUID); vfs_setmntopt(zfsvfs->z_vfs, MNTOPT_NOSETUID, NULL, 0); } else { zfsvfs->z_vfs->vfs_flag &= ~VFS_NOSETUID; vfs_clearmntopt(zfsvfs->z_vfs, MNTOPT_NOSETUID); vfs_setmntopt(zfsvfs->z_vfs, MNTOPT_SETUID, NULL, 0); } } static void exec_changed_cb(void *arg, uint64_t newval) { zfsvfs_t *zfsvfs = arg; if (newval == FALSE) { zfsvfs->z_vfs->vfs_flag |= VFS_NOEXEC; vfs_clearmntopt(zfsvfs->z_vfs, MNTOPT_EXEC); vfs_setmntopt(zfsvfs->z_vfs, MNTOPT_NOEXEC, NULL, 0); } else { zfsvfs->z_vfs->vfs_flag &= ~VFS_NOEXEC; vfs_clearmntopt(zfsvfs->z_vfs, MNTOPT_NOEXEC); vfs_setmntopt(zfsvfs->z_vfs, MNTOPT_EXEC, NULL, 0); } } /* * The nbmand mount option can be changed at mount time. * We can't allow it to be toggled on live file systems or incorrect * behavior may be seen from cifs clients * * This property isn't registered via dsl_prop_register(), but this callback * will be called when a file system is first mounted */ static void nbmand_changed_cb(void *arg, uint64_t newval) { zfsvfs_t *zfsvfs = arg; if (newval == FALSE) { vfs_clearmntopt(zfsvfs->z_vfs, MNTOPT_NBMAND); vfs_setmntopt(zfsvfs->z_vfs, MNTOPT_NONBMAND, NULL, 0); } else { vfs_clearmntopt(zfsvfs->z_vfs, MNTOPT_NONBMAND); vfs_setmntopt(zfsvfs->z_vfs, MNTOPT_NBMAND, NULL, 0); } } static void snapdir_changed_cb(void *arg, uint64_t newval) { zfsvfs_t *zfsvfs = arg; zfsvfs->z_show_ctldir = newval; } static void vscan_changed_cb(void *arg, uint64_t newval) { zfsvfs_t *zfsvfs = arg; zfsvfs->z_vscan = newval; } static void acl_mode_changed_cb(void *arg, uint64_t newval) { zfsvfs_t *zfsvfs = arg; zfsvfs->z_acl_mode = newval; } static void acl_inherit_changed_cb(void *arg, uint64_t newval) { zfsvfs_t *zfsvfs = arg; zfsvfs->z_acl_inherit = newval; } static int zfs_register_callbacks(vfs_t *vfsp) { struct dsl_dataset *ds = NULL; objset_t *os = NULL; zfsvfs_t *zfsvfs = NULL; uint64_t nbmand; int readonly, do_readonly = FALSE; int setuid, do_setuid = FALSE; int exec, do_exec = FALSE; int xattr, do_xattr = FALSE; int atime, do_atime = FALSE; int error = 0; ASSERT(vfsp); zfsvfs = vfsp->vfs_data; ASSERT(zfsvfs); os = zfsvfs->z_os; /* * This function can be called for a snapshot when we update snapshot's * mount point, which isn't really supported. */ if (dmu_objset_is_snapshot(os)) return (EOPNOTSUPP); /* * The act of registering our callbacks will destroy any mount * options we may have. In order to enable temporary overrides * of mount options, we stash away the current values and * restore them after we register the callbacks. */ if (vfs_optionisset(vfsp, MNTOPT_RO, NULL)) { readonly = B_TRUE; do_readonly = B_TRUE; } else if (vfs_optionisset(vfsp, MNTOPT_RW, NULL)) { readonly = B_FALSE; do_readonly = B_TRUE; } if (vfs_optionisset(vfsp, MNTOPT_NOSUID, NULL)) { setuid = B_FALSE; do_setuid = B_TRUE; } else { if (vfs_optionisset(vfsp, MNTOPT_NOSETUID, NULL)) { setuid = B_FALSE; do_setuid = B_TRUE; } else if (vfs_optionisset(vfsp, MNTOPT_SETUID, NULL)) { setuid = B_TRUE; do_setuid = B_TRUE; } } if (vfs_optionisset(vfsp, MNTOPT_NOEXEC, NULL)) { exec = B_FALSE; do_exec = B_TRUE; } else if (vfs_optionisset(vfsp, MNTOPT_EXEC, NULL)) { exec = B_TRUE; do_exec = B_TRUE; } if (vfs_optionisset(vfsp, MNTOPT_NOXATTR, NULL)) { xattr = B_FALSE; do_xattr = B_TRUE; } else if (vfs_optionisset(vfsp, MNTOPT_XATTR, NULL)) { xattr = B_TRUE; do_xattr = B_TRUE; } if (vfs_optionisset(vfsp, MNTOPT_NOATIME, NULL)) { atime = B_FALSE; do_atime = B_TRUE; } else if (vfs_optionisset(vfsp, MNTOPT_ATIME, NULL)) { atime = B_TRUE; do_atime = B_TRUE; } /* * nbmand is a special property. It can only be changed at * mount time. * * This is weird, but it is documented to only be changeable * at mount time. */ if (vfs_optionisset(vfsp, MNTOPT_NONBMAND, NULL)) { nbmand = B_FALSE; } else if (vfs_optionisset(vfsp, MNTOPT_NBMAND, NULL)) { nbmand = B_TRUE; } else { char osname[MAXNAMELEN]; dmu_objset_name(os, osname); if (error = dsl_prop_get_integer(osname, "nbmand", &nbmand, NULL)) { return (error); } } /* * Register property callbacks. * * It would probably be fine to just check for i/o error from * the first prop_register(), but I guess I like to go * overboard... */ ds = dmu_objset_ds(os); error = dsl_prop_register(ds, "atime", atime_changed_cb, zfsvfs); error = error ? error : dsl_prop_register(ds, "xattr", xattr_changed_cb, zfsvfs); error = error ? error : dsl_prop_register(ds, "recordsize", blksz_changed_cb, zfsvfs); error = error ? error : dsl_prop_register(ds, "readonly", readonly_changed_cb, zfsvfs); error = error ? error : dsl_prop_register(ds, "setuid", setuid_changed_cb, zfsvfs); error = error ? error : dsl_prop_register(ds, "exec", exec_changed_cb, zfsvfs); error = error ? error : dsl_prop_register(ds, "snapdir", snapdir_changed_cb, zfsvfs); error = error ? error : dsl_prop_register(ds, "aclmode", acl_mode_changed_cb, zfsvfs); error = error ? error : dsl_prop_register(ds, "aclinherit", acl_inherit_changed_cb, zfsvfs); error = error ? error : dsl_prop_register(ds, "vscan", vscan_changed_cb, zfsvfs); if (error) goto unregister; /* * Invoke our callbacks to restore temporary mount options. */ if (do_readonly) readonly_changed_cb(zfsvfs, readonly); if (do_setuid) setuid_changed_cb(zfsvfs, setuid); if (do_exec) exec_changed_cb(zfsvfs, exec); if (do_xattr) xattr_changed_cb(zfsvfs, xattr); if (do_atime) atime_changed_cb(zfsvfs, atime); nbmand_changed_cb(zfsvfs, nbmand); return (0); unregister: /* * We may attempt to unregister some callbacks that are not * registered, but this is OK; it will simply return ENOMSG, * which we will ignore. */ (void) dsl_prop_unregister(ds, "atime", atime_changed_cb, zfsvfs); (void) dsl_prop_unregister(ds, "xattr", xattr_changed_cb, zfsvfs); (void) dsl_prop_unregister(ds, "recordsize", blksz_changed_cb, zfsvfs); (void) dsl_prop_unregister(ds, "readonly", readonly_changed_cb, zfsvfs); (void) dsl_prop_unregister(ds, "setuid", setuid_changed_cb, zfsvfs); (void) dsl_prop_unregister(ds, "exec", exec_changed_cb, zfsvfs); (void) dsl_prop_unregister(ds, "snapdir", snapdir_changed_cb, zfsvfs); (void) dsl_prop_unregister(ds, "aclmode", acl_mode_changed_cb, zfsvfs); (void) dsl_prop_unregister(ds, "aclinherit", acl_inherit_changed_cb, zfsvfs); (void) dsl_prop_unregister(ds, "vscan", vscan_changed_cb, zfsvfs); return (error); } static int zfsvfs_setup(zfsvfs_t *zfsvfs, boolean_t mounting) { int error; error = zfs_register_callbacks(zfsvfs->z_vfs); if (error) return (error); /* * Set the objset user_ptr to track its zfsvfs. */ mutex_enter(&zfsvfs->z_os->os->os_user_ptr_lock); dmu_objset_set_user(zfsvfs->z_os, zfsvfs); mutex_exit(&zfsvfs->z_os->os->os_user_ptr_lock); /* * If we are not mounting (ie: online recv), then we don't * have to worry about replaying the log as we blocked all * operations out since we closed the ZIL. */ if (mounting) { boolean_t readonly; /* * During replay we remove the read only flag to * allow replays to succeed. */ readonly = zfsvfs->z_vfs->vfs_flag & VFS_RDONLY; zfsvfs->z_vfs->vfs_flag &= ~VFS_RDONLY; /* * Parse and replay the intent log. */ zil_replay(zfsvfs->z_os, zfsvfs, &zfsvfs->z_assign, zfs_replay_vector, zfs_unlinked_drain); zfs_unlinked_drain(zfsvfs); zfsvfs->z_vfs->vfs_flag |= readonly; /* restore readonly bit */ } if (!zil_disable) zfsvfs->z_log = zil_open(zfsvfs->z_os, zfs_get_data); return (0); } static void zfs_freezfsvfs(zfsvfs_t *zfsvfs) { mutex_destroy(&zfsvfs->z_znodes_lock); mutex_destroy(&zfsvfs->z_online_recv_lock); list_destroy(&zfsvfs->z_all_znodes); rrw_destroy(&zfsvfs->z_teardown_lock); rw_destroy(&zfsvfs->z_teardown_inactive_lock); rw_destroy(&zfsvfs->z_fuid_lock); kmem_free(zfsvfs, sizeof (zfsvfs_t)); } static int zfs_domount(vfs_t *vfsp, char *osname) { uint64_t recordsize, readonly; int error = 0; int mode; zfsvfs_t *zfsvfs; znode_t *zp = NULL; ASSERT(vfsp); ASSERT(osname); /* * Initialize the zfs-specific filesystem structure. * Should probably make this a kmem cache, shuffle fields, * and just bzero up to z_hold_mtx[]. */ zfsvfs = kmem_zalloc(sizeof (zfsvfs_t), KM_SLEEP); zfsvfs->z_vfs = vfsp; zfsvfs->z_parent = zfsvfs; zfsvfs->z_assign = TXG_NOWAIT; zfsvfs->z_max_blksz = SPA_MAXBLOCKSIZE; zfsvfs->z_show_ctldir = ZFS_SNAPDIR_VISIBLE; mutex_init(&zfsvfs->z_znodes_lock, NULL, MUTEX_DEFAULT, NULL); mutex_init(&zfsvfs->z_online_recv_lock, NULL, MUTEX_DEFAULT, NULL); list_create(&zfsvfs->z_all_znodes, sizeof (znode_t), offsetof(znode_t, z_link_node)); rrw_init(&zfsvfs->z_teardown_lock); rw_init(&zfsvfs->z_teardown_inactive_lock, NULL, RW_DEFAULT, NULL); rw_init(&zfsvfs->z_fuid_lock, NULL, RW_DEFAULT, NULL); if (error = dsl_prop_get_integer(osname, "recordsize", &recordsize, NULL)) goto out; zfsvfs->z_vfs->vfs_bsize = SPA_MINBLOCKSIZE; zfsvfs->z_vfs->mnt_stat.f_iosize = recordsize; vfsp->vfs_data = zfsvfs; vfsp->mnt_flag |= MNT_LOCAL; vfsp->mnt_kern_flag |= MNTK_MPSAFE; vfsp->mnt_kern_flag |= MNTK_LOOKUP_SHARED; vfsp->mnt_kern_flag |= MNTK_SHARED_WRITES; if (error = dsl_prop_get_integer(osname, "readonly", &readonly, NULL)) goto out; mode = DS_MODE_OWNER; if (readonly) mode |= DS_MODE_READONLY; error = dmu_objset_open(osname, DMU_OST_ZFS, mode, &zfsvfs->z_os); if (error == EROFS) { mode = DS_MODE_OWNER | DS_MODE_READONLY; error = dmu_objset_open(osname, DMU_OST_ZFS, mode, &zfsvfs->z_os); } if (error) goto out; if (error = zfs_init_fs(zfsvfs, &zp)) goto out; /* * Set features for file system. */ zfsvfs->z_use_fuids = USE_FUIDS(zfsvfs->z_version, zfsvfs->z_os); if (zfsvfs->z_use_fuids) { vfs_set_feature(vfsp, VFSFT_XVATTR); vfs_set_feature(vfsp, VFSFT_SYSATTR_VIEWS); vfs_set_feature(vfsp, VFSFT_ACEMASKONACCESS); vfs_set_feature(vfsp, VFSFT_ACLONCREATE); } if (zfsvfs->z_case == ZFS_CASE_INSENSITIVE) { vfs_set_feature(vfsp, VFSFT_DIRENTFLAGS); vfs_set_feature(vfsp, VFSFT_CASEINSENSITIVE); vfs_set_feature(vfsp, VFSFT_NOCASESENSITIVE); } else if (zfsvfs->z_case == ZFS_CASE_MIXED) { vfs_set_feature(vfsp, VFSFT_DIRENTFLAGS); vfs_set_feature(vfsp, VFSFT_CASEINSENSITIVE); } if (dmu_objset_is_snapshot(zfsvfs->z_os)) { uint64_t pval; ASSERT(mode & DS_MODE_READONLY); atime_changed_cb(zfsvfs, B_FALSE); readonly_changed_cb(zfsvfs, B_TRUE); if (error = dsl_prop_get_integer(osname, "xattr", &pval, NULL)) goto out; xattr_changed_cb(zfsvfs, pval); zfsvfs->z_issnap = B_TRUE; } else { error = zfsvfs_setup(zfsvfs, B_TRUE); } vfs_mountedfrom(vfsp, osname); if (!zfsvfs->z_issnap) zfsctl_create(zfsvfs); out: if (error) { if (zfsvfs->z_os) dmu_objset_close(zfsvfs->z_os); zfs_freezfsvfs(zfsvfs); } else { atomic_add_32(&zfs_active_fs_count, 1); } return (error); } void zfs_unregister_callbacks(zfsvfs_t *zfsvfs) { objset_t *os = zfsvfs->z_os; struct dsl_dataset *ds; /* * Unregister properties. */ if (!dmu_objset_is_snapshot(os)) { ds = dmu_objset_ds(os); VERIFY(dsl_prop_unregister(ds, "atime", atime_changed_cb, zfsvfs) == 0); VERIFY(dsl_prop_unregister(ds, "xattr", xattr_changed_cb, zfsvfs) == 0); VERIFY(dsl_prop_unregister(ds, "recordsize", blksz_changed_cb, zfsvfs) == 0); VERIFY(dsl_prop_unregister(ds, "readonly", readonly_changed_cb, zfsvfs) == 0); VERIFY(dsl_prop_unregister(ds, "setuid", setuid_changed_cb, zfsvfs) == 0); VERIFY(dsl_prop_unregister(ds, "exec", exec_changed_cb, zfsvfs) == 0); VERIFY(dsl_prop_unregister(ds, "snapdir", snapdir_changed_cb, zfsvfs) == 0); VERIFY(dsl_prop_unregister(ds, "aclmode", acl_mode_changed_cb, zfsvfs) == 0); VERIFY(dsl_prop_unregister(ds, "aclinherit", acl_inherit_changed_cb, zfsvfs) == 0); VERIFY(dsl_prop_unregister(ds, "vscan", vscan_changed_cb, zfsvfs) == 0); } } /*ARGSUSED*/ static int zfs_mount(vfs_t *vfsp) { kthread_t *td = curthread; vnode_t *mvp = vfsp->mnt_vnodecovered; cred_t *cr = td->td_ucred; char *osname; int error = 0; int canwrite; if (vfs_getopt(vfsp->mnt_optnew, "from", (void **)&osname, NULL)) return (EINVAL); /* * If full-owner-access is enabled and delegated administration is * turned on, we must set nosuid. */ if (zfs_super_owner && dsl_deleg_access(osname, ZFS_DELEG_PERM_MOUNT, cr) != ECANCELED) { secpolicy_fs_mount_clearopts(cr, vfsp); } /* * Check for mount privilege? * * If we don't have privilege then see if * we have local permission to allow it */ error = secpolicy_fs_mount(cr, mvp, vfsp); if (error) { error = dsl_deleg_access(osname, ZFS_DELEG_PERM_MOUNT, cr); if (error != 0) goto out; if (!(vfsp->vfs_flag & MS_REMOUNT)) { vattr_t vattr; /* * Make sure user is the owner of the mount point * or has sufficient privileges. */ vattr.va_mask = AT_UID; vn_lock(mvp, LK_SHARED | LK_RETRY); if (error = VOP_GETATTR(mvp, &vattr, cr)) { VOP_UNLOCK(mvp, 0); goto out; } #if 0 /* CHECK THIS! Is probably needed for zfs_suser. */ if (secpolicy_vnode_owner(mvp, cr, vattr.va_uid) != 0 && VOP_ACCESS(mvp, VWRITE, cr, td) != 0) { error = EPERM; goto out; } #else if (error = secpolicy_vnode_owner(mvp, cr, vattr.va_uid)) { VOP_UNLOCK(mvp, 0); goto out; } if (error = VOP_ACCESS(mvp, VWRITE, cr, td)) { VOP_UNLOCK(mvp, 0); goto out; } VOP_UNLOCK(mvp, 0); #endif } secpolicy_fs_mount_clearopts(cr, vfsp); } /* * Refuse to mount a filesystem if we are in a local zone and the * dataset is not visible. */ if (!INGLOBALZONE(curthread) && (!zone_dataset_visible(osname, &canwrite) || !canwrite)) { error = EPERM; goto out; } /* * When doing a remount, we simply refresh our temporary properties * according to those options set in the current VFS options. */ if (vfsp->vfs_flag & MS_REMOUNT) { /* refresh mount options */ zfs_unregister_callbacks(vfsp->vfs_data); error = zfs_register_callbacks(vfsp); goto out; } DROP_GIANT(); error = zfs_domount(vfsp, osname); PICKUP_GIANT(); out: return (error); } static int zfs_statfs(vfs_t *vfsp, struct statfs *statp) { zfsvfs_t *zfsvfs = vfsp->vfs_data; uint64_t refdbytes, availbytes, usedobjs, availobjs; statp->f_version = STATFS_VERSION; ZFS_ENTER(zfsvfs); dmu_objset_space(zfsvfs->z_os, &refdbytes, &availbytes, &usedobjs, &availobjs); /* * The underlying storage pool actually uses multiple block sizes. * We report the fragsize as the smallest block size we support, * and we report our blocksize as the filesystem's maximum blocksize. */ statp->f_bsize = SPA_MINBLOCKSIZE; statp->f_iosize = zfsvfs->z_vfs->mnt_stat.f_iosize; /* * The following report "total" blocks of various kinds in the * file system, but reported in terms of f_frsize - the * "fragment" size. */ statp->f_blocks = (refdbytes + availbytes) >> SPA_MINBLOCKSHIFT; statp->f_bfree = availbytes / statp->f_bsize; statp->f_bavail = statp->f_bfree; /* no root reservation */ /* * statvfs() should really be called statufs(), because it assumes * static metadata. ZFS doesn't preallocate files, so the best * we can do is report the max that could possibly fit in f_files, * and that minus the number actually used in f_ffree. * For f_ffree, report the smaller of the number of object available * and the number of blocks (each object will take at least a block). */ statp->f_ffree = MIN(availobjs, statp->f_bfree); statp->f_files = statp->f_ffree + usedobjs; /* * We're a zfs filesystem. */ (void) strlcpy(statp->f_fstypename, "zfs", sizeof(statp->f_fstypename)); strlcpy(statp->f_mntfromname, vfsp->mnt_stat.f_mntfromname, sizeof(statp->f_mntfromname)); strlcpy(statp->f_mntonname, vfsp->mnt_stat.f_mntonname, sizeof(statp->f_mntonname)); statp->f_namemax = ZFS_MAXNAMELEN; ZFS_EXIT(zfsvfs); return (0); } static int zfs_root(vfs_t *vfsp, int flags, vnode_t **vpp) { zfsvfs_t *zfsvfs = vfsp->vfs_data; znode_t *rootzp; int error; ZFS_ENTER_NOERROR(zfsvfs); error = zfs_zget(zfsvfs, zfsvfs->z_root, &rootzp); + + ZFS_EXIT(zfsvfs); + if (error == 0) { *vpp = ZTOV(rootzp); error = vn_lock(*vpp, flags); (*vpp)->v_vflag |= VV_ROOT; } - ZFS_EXIT(zfsvfs); return (error); } /* * Teardown the zfsvfs::z_os. * * Note, if 'unmounting' if FALSE, we return with the 'z_teardown_lock' * and 'z_teardown_inactive_lock' held. */ static int zfsvfs_teardown(zfsvfs_t *zfsvfs, boolean_t unmounting) { znode_t *zp; rrw_enter(&zfsvfs->z_teardown_lock, RW_WRITER, FTAG); if (!unmounting) { /* * We purge the parent filesystem's vfsp as the parent * filesystem and all of its snapshots have their vnode's * v_vfsp set to the parent's filesystem's vfsp. Note, * 'z_parent' is self referential for non-snapshots. */ (void) dnlc_purge_vfsp(zfsvfs->z_parent->z_vfs, 0); #ifdef FREEBSD_NAMECACHE cache_purgevfs(zfsvfs->z_parent->z_vfs); #endif } /* * Close the zil. NB: Can't close the zil while zfs_inactive * threads are blocked as zil_close can call zfs_inactive. */ if (zfsvfs->z_log) { zil_close(zfsvfs->z_log); zfsvfs->z_log = NULL; } rw_enter(&zfsvfs->z_teardown_inactive_lock, RW_WRITER); /* * If we are not unmounting (ie: online recv) and someone already * unmounted this file system while we were doing the switcheroo, * or a reopen of z_os failed then just bail out now. */ if (!unmounting && (zfsvfs->z_unmounted || zfsvfs->z_os == NULL)) { rw_exit(&zfsvfs->z_teardown_inactive_lock); rrw_exit(&zfsvfs->z_teardown_lock, FTAG); return (EIO); } /* * At this point there are no vops active, and any new vops will * fail with EIO since we have z_teardown_lock for writer (only * relavent for forced unmount). * * Release all holds on dbufs. */ mutex_enter(&zfsvfs->z_znodes_lock); for (zp = list_head(&zfsvfs->z_all_znodes); zp != NULL; zp = list_next(&zfsvfs->z_all_znodes, zp)) if (zp->z_dbuf) { ASSERT(ZTOV(zp)->v_count >= 0); zfs_znode_dmu_fini(zp); } mutex_exit(&zfsvfs->z_znodes_lock); /* * If we are unmounting, set the unmounted flag and let new vops * unblock. zfs_inactive will have the unmounted behavior, and all * other vops will fail with EIO. */ if (unmounting) { zfsvfs->z_unmounted = B_TRUE; rrw_exit(&zfsvfs->z_teardown_lock, FTAG); rw_exit(&zfsvfs->z_teardown_inactive_lock); #ifdef __FreeBSD__ /* * Some znodes might not be fully reclaimed, wait for them. */ mutex_enter(&zfsvfs->z_znodes_lock); while (list_head(&zfsvfs->z_all_znodes) != NULL) { msleep(zfsvfs, &zfsvfs->z_znodes_lock, 0, "zteardown", 0); } mutex_exit(&zfsvfs->z_znodes_lock); #endif } /* * z_os will be NULL if there was an error in attempting to reopen * zfsvfs, so just return as the properties had already been * unregistered and cached data had been evicted before. */ if (zfsvfs->z_os == NULL) return (0); /* * Unregister properties. */ zfs_unregister_callbacks(zfsvfs); /* * Evict cached data */ if (dmu_objset_evict_dbufs(zfsvfs->z_os)) { txg_wait_synced(dmu_objset_pool(zfsvfs->z_os), 0); (void) dmu_objset_evict_dbufs(zfsvfs->z_os); } return (0); } /*ARGSUSED*/ static int zfs_umount(vfs_t *vfsp, int fflag) { zfsvfs_t *zfsvfs = vfsp->vfs_data; objset_t *os; cred_t *cr = curthread->td_ucred; int ret; ret = secpolicy_fs_unmount(cr, vfsp); if (ret) { ret = dsl_deleg_access((char *)refstr_value(vfsp->vfs_resource), ZFS_DELEG_PERM_MOUNT, cr); if (ret) return (ret); } /* * We purge the parent filesystem's vfsp as the parent filesystem * and all of its snapshots have their vnode's v_vfsp set to the * parent's filesystem's vfsp. Note, 'z_parent' is self * referential for non-snapshots. */ (void) dnlc_purge_vfsp(zfsvfs->z_parent->z_vfs, 0); /* * Unmount any snapshots mounted under .zfs before unmounting the * dataset itself. */ if (zfsvfs->z_ctldir != NULL) { if ((ret = zfsctl_umount_snapshots(vfsp, fflag, cr)) != 0) return (ret); ret = vflush(vfsp, 0, 0, curthread); ASSERT(ret == EBUSY); if (!(fflag & MS_FORCE)) { if (zfsvfs->z_ctldir->v_count > 1) return (EBUSY); ASSERT(zfsvfs->z_ctldir->v_count == 1); } zfsctl_destroy(zfsvfs); ASSERT(zfsvfs->z_ctldir == NULL); } if (fflag & MS_FORCE) { /* * Mark file system as unmounted before calling * vflush(FORCECLOSE). This way we ensure no future vnops * will be called and risk operating on DOOMED vnodes. */ rrw_enter(&zfsvfs->z_teardown_lock, RW_WRITER, FTAG); zfsvfs->z_unmounted = B_TRUE; rrw_exit(&zfsvfs->z_teardown_lock, FTAG); } /* * Flush all the files. */ ret = vflush(vfsp, 1, (fflag & MS_FORCE) ? FORCECLOSE : 0, curthread); if (ret != 0) { if (!zfsvfs->z_issnap) { zfsctl_create(zfsvfs); ASSERT(zfsvfs->z_ctldir != NULL); } return (ret); } if (!(fflag & MS_FORCE)) { /* * Check the number of active vnodes in the file system. * Our count is maintained in the vfs structure, but the * number is off by 1 to indicate a hold on the vfs * structure itself. * * The '.zfs' directory maintains a reference of its * own, and any active references underneath are * reflected in the vnode count. */ if (zfsvfs->z_ctldir == NULL) { if (vfsp->vfs_count > 1) return (EBUSY); } else { if (vfsp->vfs_count > 2 || zfsvfs->z_ctldir->v_count > 1) return (EBUSY); } } else { MNT_ILOCK(vfsp); vfsp->mnt_kern_flag |= MNTK_UNMOUNTF; MNT_IUNLOCK(vfsp); } VERIFY(zfsvfs_teardown(zfsvfs, B_TRUE) == 0); os = zfsvfs->z_os; /* * z_os will be NULL if there was an error in * attempting to reopen zfsvfs. */ if (os != NULL) { /* * Unset the objset user_ptr. */ mutex_enter(&os->os->os_user_ptr_lock); dmu_objset_set_user(os, NULL); mutex_exit(&os->os->os_user_ptr_lock); /* * Finally release the objset */ dmu_objset_close(os); } /* * We can now safely destroy the '.zfs' directory node. */ if (zfsvfs->z_ctldir != NULL) zfsctl_destroy(zfsvfs); if (zfsvfs->z_issnap) { vnode_t *svp = vfsp->mnt_vnodecovered; if (svp->v_count >= 2) VN_RELE(svp); } zfs_freevfs(vfsp); return (0); } static int zfs_vget(vfs_t *vfsp, ino_t ino, int flags, vnode_t **vpp) { zfsvfs_t *zfsvfs = vfsp->vfs_data; znode_t *zp; int err; /* * XXXPJD: zfs_zget() can't operate on virtual entires like .zfs/ or * .zfs/snapshot/ directories, so for now just return EOPNOTSUPP. * This will make NFS to fall back to using READDIR instead of * READDIRPLUS. * Also snapshots are stored in AVL tree, but based on their names, * not inode numbers, so it will be very inefficient to iterate * over all snapshots to find the right one. * Note that OpenSolaris READDIRPLUS implementation does LOOKUP on * d_name, and not VGET on d_fileno as we do. */ if (ino == ZFSCTL_INO_ROOT || ino == ZFSCTL_INO_SNAPDIR) return (EOPNOTSUPP); ZFS_ENTER(zfsvfs); err = zfs_zget(zfsvfs, ino, &zp); if (err == 0 && zp->z_unlinked) { VN_RELE(ZTOV(zp)); err = EINVAL; } + ZFS_EXIT(zfsvfs); if (err != 0) *vpp = NULL; else { *vpp = ZTOV(zp); vn_lock(*vpp, flags); } - ZFS_EXIT(zfsvfs); return (err); } static int zfs_checkexp(vfs_t *vfsp, struct sockaddr *nam, int *extflagsp, struct ucred **credanonp, int *numsecflavors, int **secflavors) { zfsvfs_t *zfsvfs = vfsp->vfs_data; /* * If this is regular file system vfsp is the same as * zfsvfs->z_parent->z_vfs, but if it is snapshot, * zfsvfs->z_parent->z_vfs represents parent file system * which we have to use here, because only this file system * has mnt_export configured. */ vfsp = zfsvfs->z_parent->z_vfs; return (vfs_stdcheckexp(zfsvfs->z_parent->z_vfs, nam, extflagsp, credanonp, numsecflavors, secflavors)); } CTASSERT(SHORT_FID_LEN <= sizeof(struct fid)); CTASSERT(LONG_FID_LEN <= sizeof(struct fid)); static int zfs_fhtovp(vfs_t *vfsp, fid_t *fidp, vnode_t **vpp) { zfsvfs_t *zfsvfs = vfsp->vfs_data; znode_t *zp; uint64_t object = 0; uint64_t fid_gen = 0; uint64_t gen_mask; uint64_t zp_gen; int i, err; *vpp = NULL; ZFS_ENTER(zfsvfs); /* * On FreeBSD we can get snapshot's mount point or its parent file * system mount point depending if snapshot is already mounted or not. */ if (zfsvfs->z_parent == zfsvfs && fidp->fid_len == LONG_FID_LEN) { zfid_long_t *zlfid = (zfid_long_t *)fidp; uint64_t objsetid = 0; uint64_t setgen = 0; for (i = 0; i < sizeof (zlfid->zf_setid); i++) objsetid |= ((uint64_t)zlfid->zf_setid[i]) << (8 * i); for (i = 0; i < sizeof (zlfid->zf_setgen); i++) setgen |= ((uint64_t)zlfid->zf_setgen[i]) << (8 * i); ZFS_EXIT(zfsvfs); err = zfsctl_lookup_objset(vfsp, objsetid, &zfsvfs); if (err) return (EINVAL); ZFS_ENTER(zfsvfs); } if (fidp->fid_len == SHORT_FID_LEN || fidp->fid_len == LONG_FID_LEN) { zfid_short_t *zfid = (zfid_short_t *)fidp; for (i = 0; i < sizeof (zfid->zf_object); i++) object |= ((uint64_t)zfid->zf_object[i]) << (8 * i); for (i = 0; i < sizeof (zfid->zf_gen); i++) fid_gen |= ((uint64_t)zfid->zf_gen[i]) << (8 * i); } else { ZFS_EXIT(zfsvfs); return (EINVAL); } /* A zero fid_gen means we are in the .zfs control directories */ if (fid_gen == 0 && (object == ZFSCTL_INO_ROOT || object == ZFSCTL_INO_SNAPDIR)) { *vpp = zfsvfs->z_ctldir; ASSERT(*vpp != NULL); if (object == ZFSCTL_INO_SNAPDIR) { VERIFY(zfsctl_root_lookup(*vpp, "snapshot", vpp, NULL, 0, NULL, NULL, NULL, NULL, NULL) == 0); } else { VN_HOLD(*vpp); } - vn_lock(*vpp, LK_EXCLUSIVE | LK_RETRY); ZFS_EXIT(zfsvfs); + vn_lock(*vpp, LK_EXCLUSIVE | LK_RETRY); return (0); } gen_mask = -1ULL >> (64 - 8 * i); dprintf("getting %llu [%u mask %llx]\n", object, fid_gen, gen_mask); if (err = zfs_zget(zfsvfs, object, &zp)) { ZFS_EXIT(zfsvfs); return (err); } zp_gen = zp->z_phys->zp_gen & gen_mask; if (zp_gen == 0) zp_gen = 1; if (zp->z_unlinked || zp_gen != fid_gen) { dprintf("znode gen (%u) != fid gen (%u)\n", zp_gen, fid_gen); VN_RELE(ZTOV(zp)); ZFS_EXIT(zfsvfs); return (EINVAL); } + ZFS_EXIT(zfsvfs); + *vpp = ZTOV(zp); vn_lock(*vpp, LK_EXCLUSIVE | LK_RETRY); vnode_create_vobject(*vpp, zp->z_phys->zp_size, curthread); - ZFS_EXIT(zfsvfs); return (0); } /* * Block out VOPs and close zfsvfs_t::z_os * * Note, if successful, then we return with the 'z_teardown_lock' and * 'z_teardown_inactive_lock' write held. */ int zfs_suspend_fs(zfsvfs_t *zfsvfs, char *name, int *mode) { int error; if ((error = zfsvfs_teardown(zfsvfs, B_FALSE)) != 0) return (error); *mode = zfsvfs->z_os->os_mode; dmu_objset_name(zfsvfs->z_os, name); dmu_objset_close(zfsvfs->z_os); return (0); } /* * Reopen zfsvfs_t::z_os and release VOPs. */ int zfs_resume_fs(zfsvfs_t *zfsvfs, const char *osname, int mode) { int err; ASSERT(RRW_WRITE_HELD(&zfsvfs->z_teardown_lock)); ASSERT(RW_WRITE_HELD(&zfsvfs->z_teardown_inactive_lock)); err = dmu_objset_open(osname, DMU_OST_ZFS, mode, &zfsvfs->z_os); if (err) { zfsvfs->z_os = NULL; } else { znode_t *zp; VERIFY(zfsvfs_setup(zfsvfs, B_FALSE) == 0); /* * Attempt to re-establish all the active znodes with * their dbufs. If a zfs_rezget() fails, then we'll let * any potential callers discover that via ZFS_ENTER_VERIFY_VP * when they try to use their znode. */ mutex_enter(&zfsvfs->z_znodes_lock); for (zp = list_head(&zfsvfs->z_all_znodes); zp; zp = list_next(&zfsvfs->z_all_znodes, zp)) { (void) zfs_rezget(zp); } mutex_exit(&zfsvfs->z_znodes_lock); } /* release the VOPs */ rw_exit(&zfsvfs->z_teardown_inactive_lock); rrw_exit(&zfsvfs->z_teardown_lock, FTAG); if (err) { /* * Since we couldn't reopen zfsvfs::z_os, force * unmount this file system. */ if (vn_vfswlock(zfsvfs->z_vfs->vfs_vnodecovered) == 0) (void) dounmount(zfsvfs->z_vfs, MS_FORCE, curthread); } return (err); } static void zfs_freevfs(vfs_t *vfsp) { zfsvfs_t *zfsvfs = vfsp->vfs_data; int i; for (i = 0; i != ZFS_OBJ_MTX_SZ; i++) mutex_destroy(&zfsvfs->z_hold_mtx[i]); zfs_fuid_destroy(zfsvfs); zfs_freezfsvfs(zfsvfs); atomic_add_32(&zfs_active_fs_count, -1); } #ifdef __i386__ static int desiredvnodes_backup; #endif static void zfs_vnodes_adjust(void) { #ifdef __i386__ int newdesiredvnodes; desiredvnodes_backup = desiredvnodes; /* * We calculate newdesiredvnodes the same way it is done in * vntblinit(). If it is equal to desiredvnodes, it means that * it wasn't tuned by the administrator and we can tune it down. */ newdesiredvnodes = min(maxproc + cnt.v_page_count / 4, 2 * vm_kmem_size / (5 * (sizeof(struct vm_object) + sizeof(struct vnode)))); if (newdesiredvnodes == desiredvnodes) desiredvnodes = (3 * newdesiredvnodes) / 4; #endif } static void zfs_vnodes_adjust_back(void) { #ifdef __i386__ desiredvnodes = desiredvnodes_backup; #endif } void zfs_init(void) { printf("ZFS filesystem version " ZPL_VERSION_STRING "\n"); /* * Initialize znode cache, vnode ops, etc... */ zfs_znode_init(); /* * Initialize .zfs directory structures */ zfsctl_init(); /* * Reduce number of vnode. Originally number of vnodes is calculated * with UFS inode in mind. We reduce it here, because it's too big for * ZFS/i386. */ zfs_vnodes_adjust(); } void zfs_fini(void) { zfsctl_fini(); zfs_znode_fini(); zfs_vnodes_adjust_back(); } int zfs_busy(void) { return (zfs_active_fs_count != 0); } int zfs_set_version(const char *name, uint64_t newvers) { int error; objset_t *os; dmu_tx_t *tx; uint64_t curvers; /* * XXX for now, require that the filesystem be unmounted. Would * be nice to find the zfsvfs_t and just update that if * possible. */ if (newvers < ZPL_VERSION_INITIAL || newvers > ZPL_VERSION) return (EINVAL); error = dmu_objset_open(name, DMU_OST_ZFS, DS_MODE_OWNER, &os); if (error) return (error); error = zap_lookup(os, MASTER_NODE_OBJ, ZPL_VERSION_STR, 8, 1, &curvers); if (error) goto out; if (newvers < curvers) { error = EINVAL; goto out; } tx = dmu_tx_create(os); dmu_tx_hold_zap(tx, MASTER_NODE_OBJ, 0, ZPL_VERSION_STR); error = dmu_tx_assign(tx, TXG_WAIT); if (error) { dmu_tx_abort(tx); goto out; } error = zap_update(os, MASTER_NODE_OBJ, ZPL_VERSION_STR, 8, 1, &newvers, tx); spa_history_internal_log(LOG_DS_UPGRADE, dmu_objset_spa(os), tx, CRED(), "oldver=%llu newver=%llu dataset = %llu", curvers, newvers, dmu_objset_id(os)); dmu_tx_commit(tx); out: dmu_objset_close(os); return (error); } /* * Read a property stored within the master node. */ int zfs_get_zplprop(objset_t *os, zfs_prop_t prop, uint64_t *value) { const char *pname; int error = ENOENT; /* * Look up the file system's value for the property. For the * version property, we look up a slightly different string. */ if (prop == ZFS_PROP_VERSION) pname = ZPL_VERSION_STR; else pname = zfs_prop_to_name(prop); if (os != NULL) error = zap_lookup(os, MASTER_NODE_OBJ, pname, 8, 1, value); if (error == ENOENT) { /* No value set, use the default value */ switch (prop) { case ZFS_PROP_VERSION: *value = ZPL_VERSION; break; case ZFS_PROP_NORMALIZE: case ZFS_PROP_UTF8ONLY: *value = 0; break; case ZFS_PROP_CASE: *value = ZFS_CASE_SENSITIVE; break; default: return (error); } error = 0; } return (error); } Index: user/imp/tbemd/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c =================================================================== --- user/imp/tbemd/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c (revision 206684) +++ user/imp/tbemd/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c (revision 206685) @@ -1,5025 +1,5025 @@ /* * CDDL HEADER START * * The contents of this file are subject to the terms of the * Common Development and Distribution License (the "License"). * You may not use this file except in compliance with the License. * * You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE * or http://www.opensolaris.org/os/licensing. * See the License for the specific language governing permissions * and limitations under the License. * * When distributing Covered Code, include this CDDL HEADER in each * file and include the License file at usr/src/OPENSOLARIS.LICENSE. * If applicable, add the following below this CDDL HEADER, with the * fields enclosed by brackets "[]" replaced with your own identifying * information: Portions Copyright [yyyy] [name of copyright owner] * * CDDL HEADER END */ /* * Copyright 2008 Sun Microsystems, Inc. All rights reserved. * Use is subject to license terms. */ /* Portions Copyright 2007 Jeremy Teo */ #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include /* * Programming rules. * * Each vnode op performs some logical unit of work. To do this, the ZPL must * properly lock its in-core state, create a DMU transaction, do the work, * record this work in the intent log (ZIL), commit the DMU transaction, * and wait for the intent log to commit if it is a synchronous operation. * Moreover, the vnode ops must work in both normal and log replay context. * The ordering of events is important to avoid deadlocks and references * to freed memory. The example below illustrates the following Big Rules: * * (1) A check must be made in each zfs thread for a mounted file system. * This is done avoiding races using ZFS_ENTER(zfsvfs). * A ZFS_EXIT(zfsvfs) is needed before all returns. Any znodes * must be checked with ZFS_VERIFY_ZP(zp). Both of these macros * can return EIO from the calling function. * * (2) VN_RELE() should always be the last thing except for zil_commit() * (if necessary) and ZFS_EXIT(). This is for 3 reasons: * First, if it's the last reference, the vnode/znode * can be freed, so the zp may point to freed memory. Second, the last * reference will call zfs_zinactive(), which may induce a lot of work -- * pushing cached pages (which acquires range locks) and syncing out * cached atime changes. Third, zfs_zinactive() may require a new tx, * which could deadlock the system if you were already holding one. * If you must call VN_RELE() within a tx then use VN_RELE_ASYNC(). * * (3) All range locks must be grabbed before calling dmu_tx_assign(), * as they can span dmu_tx_assign() calls. * * (4) Always pass zfsvfs->z_assign as the second argument to dmu_tx_assign(). * In normal operation, this will be TXG_NOWAIT. During ZIL replay, * it will be a specific txg. Either way, dmu_tx_assign() never blocks. * This is critical because we don't want to block while holding locks. * Note, in particular, that if a lock is sometimes acquired before * the tx assigns, and sometimes after (e.g. z_lock), then failing to * use a non-blocking assign can deadlock the system. The scenario: * * Thread A has grabbed a lock before calling dmu_tx_assign(). * Thread B is in an already-assigned tx, and blocks for this lock. * Thread A calls dmu_tx_assign(TXG_WAIT) and blocks in txg_wait_open() * forever, because the previous txg can't quiesce until B's tx commits. * * If dmu_tx_assign() returns ERESTART and zfsvfs->z_assign is TXG_NOWAIT, * then drop all locks, call dmu_tx_wait(), and try again. * * (5) If the operation succeeded, generate the intent log entry for it * before dropping locks. This ensures that the ordering of events * in the intent log matches the order in which they actually occurred. * * (6) At the end of each vnode op, the DMU tx must always commit, * regardless of whether there were any errors. * * (7) After dropping all locks, invoke zil_commit(zilog, seq, foid) * to ensure that synchronous semantics are provided when necessary. * * In general, this is how things should be ordered in each vnode op: * * ZFS_ENTER(zfsvfs); // exit if unmounted * top: * zfs_dirent_lock(&dl, ...) // lock directory entry (may VN_HOLD()) * rw_enter(...); // grab any other locks you need * tx = dmu_tx_create(...); // get DMU tx * dmu_tx_hold_*(); // hold each object you might modify * error = dmu_tx_assign(tx, zfsvfs->z_assign); // try to assign * if (error) { * rw_exit(...); // drop locks * zfs_dirent_unlock(dl); // unlock directory entry * VN_RELE(...); // release held vnodes * if (error == ERESTART && zfsvfs->z_assign == TXG_NOWAIT) { * dmu_tx_wait(tx); * dmu_tx_abort(tx); * goto top; * } * dmu_tx_abort(tx); // abort DMU tx * ZFS_EXIT(zfsvfs); // finished in zfs * return (error); // really out of space * } * error = do_real_work(); // do whatever this VOP does * if (error == 0) * zfs_log_*(...); // on success, make ZIL entry * dmu_tx_commit(tx); // commit DMU tx -- error or not * rw_exit(...); // drop locks * zfs_dirent_unlock(dl); // unlock directory entry * VN_RELE(...); // release held vnodes * zil_commit(zilog, seq, foid); // synchronous when necessary * ZFS_EXIT(zfsvfs); // finished in zfs * return (error); // done, report error */ /* ARGSUSED */ static int zfs_open(vnode_t **vpp, int flag, cred_t *cr, caller_context_t *ct) { znode_t *zp = VTOZ(*vpp); if ((flag & FWRITE) && (zp->z_phys->zp_flags & ZFS_APPENDONLY) && ((flag & FAPPEND) == 0)) { return (EPERM); } if (!zfs_has_ctldir(zp) && zp->z_zfsvfs->z_vscan && ZTOV(zp)->v_type == VREG && !(zp->z_phys->zp_flags & ZFS_AV_QUARANTINED) && zp->z_phys->zp_size > 0) if (fs_vscan(*vpp, cr, 0) != 0) return (EACCES); /* Keep a count of the synchronous opens in the znode */ if (flag & (FSYNC | FDSYNC)) atomic_inc_32(&zp->z_sync_cnt); return (0); } /* ARGSUSED */ static int zfs_close(vnode_t *vp, int flag, int count, offset_t offset, cred_t *cr, caller_context_t *ct) { znode_t *zp = VTOZ(vp); /* Decrement the synchronous opens in the znode */ if ((flag & (FSYNC | FDSYNC)) && (count == 1)) atomic_dec_32(&zp->z_sync_cnt); /* * Clean up any locks held by this process on the vp. */ cleanlocks(vp, ddi_get_pid(), 0); cleanshares(vp, ddi_get_pid()); if (!zfs_has_ctldir(zp) && zp->z_zfsvfs->z_vscan && ZTOV(zp)->v_type == VREG && !(zp->z_phys->zp_flags & ZFS_AV_QUARANTINED) && zp->z_phys->zp_size > 0) VERIFY(fs_vscan(vp, cr, 1) == 0); return (0); } /* * Lseek support for finding holes (cmd == _FIO_SEEK_HOLE) and * data (cmd == _FIO_SEEK_DATA). "off" is an in/out parameter. */ static int zfs_holey(vnode_t *vp, u_long cmd, offset_t *off) { znode_t *zp = VTOZ(vp); uint64_t noff = (uint64_t)*off; /* new offset */ uint64_t file_sz; int error; boolean_t hole; file_sz = zp->z_phys->zp_size; if (noff >= file_sz) { return (ENXIO); } if (cmd == _FIO_SEEK_HOLE) hole = B_TRUE; else hole = B_FALSE; error = dmu_offset_next(zp->z_zfsvfs->z_os, zp->z_id, hole, &noff); /* end of file? */ if ((error == ESRCH) || (noff > file_sz)) { /* * Handle the virtual hole at the end of file. */ if (hole) { *off = file_sz; return (0); } return (ENXIO); } if (noff < *off) return (error); *off = noff; return (error); } /* ARGSUSED */ static int zfs_ioctl(vnode_t *vp, u_long com, intptr_t data, int flag, cred_t *cred, int *rvalp, caller_context_t *ct) { offset_t off; int error; zfsvfs_t *zfsvfs; znode_t *zp; switch (com) { case _FIOFFS: return (0); /* * The following two ioctls are used by bfu. Faking out, * necessary to avoid bfu errors. */ case _FIOGDIO: case _FIOSDIO: return (0); case _FIO_SEEK_DATA: case _FIO_SEEK_HOLE: if (ddi_copyin((void *)data, &off, sizeof (off), flag)) return (EFAULT); zp = VTOZ(vp); zfsvfs = zp->z_zfsvfs; ZFS_ENTER(zfsvfs); ZFS_VERIFY_ZP(zp); /* offset parameter is in/out */ error = zfs_holey(vp, com, &off); ZFS_EXIT(zfsvfs); if (error) return (error); if (ddi_copyout(&off, (void *)data, sizeof (off), flag)) return (EFAULT); return (0); } return (ENOTTY); } /* * When a file is memory mapped, we must keep the IO data synchronized * between the DMU cache and the memory mapped pages. What this means: * * On Write: If we find a memory mapped page, we write to *both* * the page and the dmu buffer. * * NOTE: We will always "break up" the IO into PAGESIZE uiomoves when * the file is memory mapped. */ static int mappedwrite(vnode_t *vp, int nbytes, uio_t *uio, dmu_tx_t *tx) { znode_t *zp = VTOZ(vp); objset_t *os = zp->z_zfsvfs->z_os; vm_object_t obj; vm_page_t m; struct sf_buf *sf; int64_t start, off; int len = nbytes; int error = 0; uint64_t dirbytes; ASSERT(vp->v_mount != NULL); obj = vp->v_object; ASSERT(obj != NULL); start = uio->uio_loffset; off = start & PAGEOFFSET; dirbytes = 0; VM_OBJECT_LOCK(obj); for (start &= PAGEMASK; len > 0; start += PAGESIZE) { uint64_t bytes = MIN(PAGESIZE - off, len); uint64_t fsize; again: if ((m = vm_page_lookup(obj, OFF_TO_IDX(start))) != NULL && vm_page_is_valid(m, (vm_offset_t)off, bytes)) { uint64_t woff; caddr_t va; if (vm_page_sleep_if_busy(m, FALSE, "zfsmwb")) goto again; fsize = obj->un_pager.vnp.vnp_size; vm_page_busy(m); vm_page_lock_queues(); vm_page_undirty(m); vm_page_unlock_queues(); VM_OBJECT_UNLOCK(obj); if (dirbytes > 0) { error = dmu_write_uio(os, zp->z_id, uio, dirbytes, tx); dirbytes = 0; } if (error == 0) { sched_pin(); sf = sf_buf_alloc(m, SFB_CPUPRIVATE); va = (caddr_t)sf_buf_kva(sf); woff = uio->uio_loffset - off; error = uiomove(va + off, bytes, UIO_WRITE, uio); /* * The uiomove() above could have been partially * successful, that's why we call dmu_write() * below unconditionally. The page was marked * non-dirty above and we would lose the changes * without doing so. If the uiomove() failed * entirely, well, we just write what we got * before one more time. */ dmu_write(os, zp->z_id, woff, MIN(PAGESIZE, fsize - woff), va, tx); sf_buf_free(sf); sched_unpin(); } VM_OBJECT_LOCK(obj); vm_page_wakeup(m); } else { if (__predict_false(obj->cache != NULL)) { vm_page_cache_free(obj, OFF_TO_IDX(start), OFF_TO_IDX(start) + 1); } dirbytes += bytes; } len -= bytes; off = 0; if (error) break; } VM_OBJECT_UNLOCK(obj); if (error == 0 && dirbytes > 0) error = dmu_write_uio(os, zp->z_id, uio, dirbytes, tx); return (error); } /* * When a file is memory mapped, we must keep the IO data synchronized * between the DMU cache and the memory mapped pages. What this means: * * On Read: We "read" preferentially from memory mapped pages, * else we default from the dmu buffer. * * NOTE: We will always "break up" the IO into PAGESIZE uiomoves when * the file is memory mapped. */ static int mappedread(vnode_t *vp, int nbytes, uio_t *uio) { znode_t *zp = VTOZ(vp); objset_t *os = zp->z_zfsvfs->z_os; vm_object_t obj; vm_page_t m; struct sf_buf *sf; int64_t start, off; caddr_t va; int len = nbytes; int error = 0; uint64_t dirbytes; ASSERT(vp->v_mount != NULL); obj = vp->v_object; ASSERT(obj != NULL); start = uio->uio_loffset; off = start & PAGEOFFSET; dirbytes = 0; VM_OBJECT_LOCK(obj); for (start &= PAGEMASK; len > 0; start += PAGESIZE) { uint64_t bytes = MIN(PAGESIZE - off, len); again: if ((m = vm_page_lookup(obj, OFF_TO_IDX(start))) != NULL && vm_page_is_valid(m, (vm_offset_t)off, bytes)) { if (vm_page_sleep_if_busy(m, FALSE, "zfsmrb")) goto again; vm_page_busy(m); VM_OBJECT_UNLOCK(obj); if (dirbytes > 0) { error = dmu_read_uio(os, zp->z_id, uio, dirbytes); dirbytes = 0; } if (error == 0) { sched_pin(); sf = sf_buf_alloc(m, SFB_CPUPRIVATE); va = (caddr_t)sf_buf_kva(sf); error = uiomove(va + off, bytes, UIO_READ, uio); sf_buf_free(sf); sched_unpin(); } VM_OBJECT_LOCK(obj); vm_page_wakeup(m); } else if (m != NULL && uio->uio_segflg == UIO_NOCOPY) { /* * The code below is here to make sendfile(2) work * correctly with ZFS. As pointed out by ups@ * sendfile(2) should be changed to use VOP_GETPAGES(), * but it pessimize performance of sendfile/UFS, that's * why I handle this special case in ZFS code. */ if (vm_page_sleep_if_busy(m, FALSE, "zfsmrb")) goto again; vm_page_busy(m); VM_OBJECT_UNLOCK(obj); if (dirbytes > 0) { error = dmu_read_uio(os, zp->z_id, uio, dirbytes); dirbytes = 0; } if (error == 0) { sched_pin(); sf = sf_buf_alloc(m, SFB_CPUPRIVATE); va = (caddr_t)sf_buf_kva(sf); error = dmu_read(os, zp->z_id, start + off, bytes, (void *)(va + off)); sf_buf_free(sf); sched_unpin(); } VM_OBJECT_LOCK(obj); vm_page_wakeup(m); if (error == 0) uio->uio_resid -= bytes; } else { dirbytes += bytes; } len -= bytes; off = 0; if (error) break; } VM_OBJECT_UNLOCK(obj); if (error == 0 && dirbytes > 0) error = dmu_read_uio(os, zp->z_id, uio, dirbytes); return (error); } offset_t zfs_read_chunk_size = 1024 * 1024; /* Tunable */ /* * Read bytes from specified file into supplied buffer. * * IN: vp - vnode of file to be read from. * uio - structure supplying read location, range info, * and return buffer. * ioflag - SYNC flags; used to provide FRSYNC semantics. * cr - credentials of caller. * ct - caller context * * OUT: uio - updated offset and range, buffer filled. * * RETURN: 0 if success * error code if failure * * Side Effects: * vp - atime updated if byte count > 0 */ /* ARGSUSED */ static int zfs_read(vnode_t *vp, uio_t *uio, int ioflag, cred_t *cr, caller_context_t *ct) { znode_t *zp = VTOZ(vp); zfsvfs_t *zfsvfs = zp->z_zfsvfs; objset_t *os; ssize_t n, nbytes; int error; rl_t *rl; ZFS_ENTER(zfsvfs); ZFS_VERIFY_ZP(zp); os = zfsvfs->z_os; if (zp->z_phys->zp_flags & ZFS_AV_QUARANTINED) { ZFS_EXIT(zfsvfs); return (EACCES); } /* * Validate file offset */ if (uio->uio_loffset < (offset_t)0) { ZFS_EXIT(zfsvfs); return (EINVAL); } /* * Fasttrack empty reads */ if (uio->uio_resid == 0) { ZFS_EXIT(zfsvfs); return (0); } /* * Check for mandatory locks */ if (MANDMODE((mode_t)zp->z_phys->zp_mode)) { if (error = chklock(vp, FREAD, uio->uio_loffset, uio->uio_resid, uio->uio_fmode, ct)) { ZFS_EXIT(zfsvfs); return (error); } } /* * If we're in FRSYNC mode, sync out this znode before reading it. */ if (ioflag & FRSYNC) zil_commit(zfsvfs->z_log, zp->z_last_itx, zp->z_id); /* * Lock the range against changes. */ rl = zfs_range_lock(zp, uio->uio_loffset, uio->uio_resid, RL_READER); /* * If we are reading past end-of-file we can skip * to the end; but we might still need to set atime. */ if (uio->uio_loffset >= zp->z_phys->zp_size) { error = 0; goto out; } ASSERT(uio->uio_loffset < zp->z_phys->zp_size); n = MIN(uio->uio_resid, zp->z_phys->zp_size - uio->uio_loffset); while (n > 0) { nbytes = MIN(n, zfs_read_chunk_size - P2PHASE(uio->uio_loffset, zfs_read_chunk_size)); if (vn_has_cached_data(vp)) error = mappedread(vp, nbytes, uio); else error = dmu_read_uio(os, zp->z_id, uio, nbytes); if (error) { /* convert checksum errors into IO errors */ if (error == ECKSUM) error = EIO; break; } n -= nbytes; } out: zfs_range_unlock(rl); ZFS_ACCESSTIME_STAMP(zfsvfs, zp); ZFS_EXIT(zfsvfs); return (error); } /* * Fault in the pages of the first n bytes specified by the uio structure. * 1 byte in each page is touched and the uio struct is unmodified. * Any error will exit this routine as this is only a best * attempt to get the pages resident. This is a copy of ufs_trans_touch(). */ static void zfs_prefault_write(ssize_t n, struct uio *uio) { struct iovec *iov; ulong_t cnt, incr; caddr_t p; if (uio->uio_segflg != UIO_USERSPACE) return; iov = uio->uio_iov; while (n) { cnt = MIN(iov->iov_len, n); if (cnt == 0) { /* empty iov entry */ iov++; continue; } n -= cnt; /* * touch each page in this segment. */ p = iov->iov_base; while (cnt) { if (fubyte(p) == -1) return; incr = MIN(cnt, PAGESIZE); p += incr; cnt -= incr; } /* * touch the last byte in case it straddles a page. */ p--; if (fubyte(p) == -1) return; iov++; } } /* * Write the bytes to a file. * * IN: vp - vnode of file to be written to. * uio - structure supplying write location, range info, * and data buffer. * ioflag - IO_APPEND flag set if in append mode. * cr - credentials of caller. * ct - caller context (NFS/CIFS fem monitor only) * * OUT: uio - updated offset and range. * * RETURN: 0 if success * error code if failure * * Timestamps: * vp - ctime|mtime updated if byte count > 0 */ /* ARGSUSED */ static int zfs_write(vnode_t *vp, uio_t *uio, int ioflag, cred_t *cr, caller_context_t *ct) { znode_t *zp = VTOZ(vp); rlim64_t limit = MAXOFFSET_T; ssize_t start_resid = uio->uio_resid; ssize_t tx_bytes; uint64_t end_size; dmu_tx_t *tx; zfsvfs_t *zfsvfs = zp->z_zfsvfs; zilog_t *zilog; offset_t woff; ssize_t n, nbytes; rl_t *rl; int max_blksz = zfsvfs->z_max_blksz; uint64_t pflags; int error; /* * Fasttrack empty write */ n = start_resid; if (n == 0) return (0); if (limit == RLIM64_INFINITY || limit > MAXOFFSET_T) limit = MAXOFFSET_T; ZFS_ENTER(zfsvfs); ZFS_VERIFY_ZP(zp); /* * If immutable or not appending then return EPERM */ pflags = zp->z_phys->zp_flags; if ((pflags & (ZFS_IMMUTABLE | ZFS_READONLY)) || ((pflags & ZFS_APPENDONLY) && !(ioflag & FAPPEND) && (uio->uio_loffset < zp->z_phys->zp_size))) { ZFS_EXIT(zfsvfs); return (EPERM); } zilog = zfsvfs->z_log; /* * Pre-fault the pages to ensure slow (eg NFS) pages * don't hold up txg. */ zfs_prefault_write(n, uio); /* * If in append mode, set the io offset pointer to eof. */ if (ioflag & IO_APPEND) { /* * Range lock for a file append: * The value for the start of range will be determined by * zfs_range_lock() (to guarantee append semantics). * If this write will cause the block size to increase, * zfs_range_lock() will lock the entire file, so we must * later reduce the range after we grow the block size. */ rl = zfs_range_lock(zp, 0, n, RL_APPEND); if (rl->r_len == UINT64_MAX) { /* overlocked, zp_size can't change */ woff = uio->uio_loffset = zp->z_phys->zp_size; } else { woff = uio->uio_loffset = rl->r_off; } } else { woff = uio->uio_loffset; /* * Validate file offset */ if (woff < 0) { ZFS_EXIT(zfsvfs); return (EINVAL); } /* * If we need to grow the block size then zfs_range_lock() * will lock a wider range than we request here. * Later after growing the block size we reduce the range. */ rl = zfs_range_lock(zp, woff, n, RL_WRITER); } if (woff >= limit) { zfs_range_unlock(rl); ZFS_EXIT(zfsvfs); return (EFBIG); } if ((woff + n) > limit || woff > (limit - n)) n = limit - woff; /* * Check for mandatory locks */ if (MANDMODE((mode_t)zp->z_phys->zp_mode) && (error = chklock(vp, FWRITE, woff, n, uio->uio_fmode, ct)) != 0) { zfs_range_unlock(rl); ZFS_EXIT(zfsvfs); return (error); } end_size = MAX(zp->z_phys->zp_size, woff + n); /* * Write the file in reasonable size chunks. Each chunk is written * in a separate transaction; this keeps the intent log records small * and allows us to do more fine-grained space accounting. */ while (n > 0) { /* * Start a transaction. */ woff = uio->uio_loffset; tx = dmu_tx_create(zfsvfs->z_os); dmu_tx_hold_bonus(tx, zp->z_id); dmu_tx_hold_write(tx, zp->z_id, woff, MIN(n, max_blksz)); error = dmu_tx_assign(tx, zfsvfs->z_assign); if (error) { if (error == ERESTART && zfsvfs->z_assign == TXG_NOWAIT) { dmu_tx_wait(tx); dmu_tx_abort(tx); continue; } dmu_tx_abort(tx); break; } /* * If zfs_range_lock() over-locked we grow the blocksize * and then reduce the lock range. This will only happen * on the first iteration since zfs_range_reduce() will * shrink down r_len to the appropriate size. */ if (rl->r_len == UINT64_MAX) { uint64_t new_blksz; if (zp->z_blksz > max_blksz) { ASSERT(!ISP2(zp->z_blksz)); new_blksz = MIN(end_size, SPA_MAXBLOCKSIZE); } else { new_blksz = MIN(end_size, max_blksz); } zfs_grow_blocksize(zp, new_blksz, tx); zfs_range_reduce(rl, woff, n); } /* * XXX - should we really limit each write to z_max_blksz? * Perhaps we should use SPA_MAXBLOCKSIZE chunks? */ nbytes = MIN(n, max_blksz - P2PHASE(woff, max_blksz)); if (woff + nbytes > zp->z_phys->zp_size) vnode_pager_setsize(vp, woff + nbytes); rw_enter(&zp->z_map_lock, RW_READER); tx_bytes = uio->uio_resid; if (vn_has_cached_data(vp)) { rw_exit(&zp->z_map_lock); error = mappedwrite(vp, nbytes, uio, tx); } else { error = dmu_write_uio(zfsvfs->z_os, zp->z_id, uio, nbytes, tx); rw_exit(&zp->z_map_lock); } tx_bytes -= uio->uio_resid; /* * If we made no progress, we're done. If we made even * partial progress, update the znode and ZIL accordingly. */ if (tx_bytes == 0) { dmu_tx_commit(tx); ASSERT(error != 0); break; } /* * Clear Set-UID/Set-GID bits on successful write if not * privileged and at least one of the excute bits is set. * * It would be nice to to this after all writes have * been done, but that would still expose the ISUID/ISGID * to another app after the partial write is committed. * * Note: we don't call zfs_fuid_map_id() here because * user 0 is not an ephemeral uid. */ mutex_enter(&zp->z_acl_lock); if ((zp->z_phys->zp_mode & (S_IXUSR | (S_IXUSR >> 3) | (S_IXUSR >> 6))) != 0 && (zp->z_phys->zp_mode & (S_ISUID | S_ISGID)) != 0 && secpolicy_vnode_setid_retain(vp, cr, (zp->z_phys->zp_mode & S_ISUID) != 0 && zp->z_phys->zp_uid == 0) != 0) { zp->z_phys->zp_mode &= ~(S_ISUID | S_ISGID); } mutex_exit(&zp->z_acl_lock); /* * Update time stamp. NOTE: This marks the bonus buffer as * dirty, so we don't have to do it again for zp_size. */ zfs_time_stamper(zp, CONTENT_MODIFIED, tx); /* * Update the file size (zp_size) if it has changed; * account for possible concurrent updates. */ while ((end_size = zp->z_phys->zp_size) < uio->uio_loffset) (void) atomic_cas_64(&zp->z_phys->zp_size, end_size, uio->uio_loffset); zfs_log_write(zilog, tx, TX_WRITE, zp, woff, tx_bytes, ioflag); dmu_tx_commit(tx); if (error != 0) break; ASSERT(tx_bytes == nbytes); n -= nbytes; } zfs_range_unlock(rl); /* * If we're in replay mode, or we made no progress, return error. * Otherwise, it's at least a partial write, so it's successful. */ if (zfsvfs->z_assign >= TXG_INITIAL || uio->uio_resid == start_resid) { ZFS_EXIT(zfsvfs); return (error); } if (ioflag & (FSYNC | FDSYNC)) zil_commit(zilog, zp->z_last_itx, zp->z_id); ZFS_EXIT(zfsvfs); return (0); } void zfs_get_done(dmu_buf_t *db, void *vzgd) { zgd_t *zgd = (zgd_t *)vzgd; rl_t *rl = zgd->zgd_rl; vnode_t *vp = ZTOV(rl->r_zp); objset_t *os = rl->r_zp->z_zfsvfs->z_os; int vfslocked; vfslocked = VFS_LOCK_GIANT(vp->v_vfsp); dmu_buf_rele(db, vzgd); zfs_range_unlock(rl); /* * Release the vnode asynchronously as we currently have the * txg stopped from syncing. */ VN_RELE_ASYNC(vp, dsl_pool_vnrele_taskq(dmu_objset_pool(os))); zil_add_block(zgd->zgd_zilog, zgd->zgd_bp); kmem_free(zgd, sizeof (zgd_t)); VFS_UNLOCK_GIANT(vfslocked); } /* * Get data to generate a TX_WRITE intent log record. */ int zfs_get_data(void *arg, lr_write_t *lr, char *buf, zio_t *zio) { zfsvfs_t *zfsvfs = arg; objset_t *os = zfsvfs->z_os; znode_t *zp; uint64_t off = lr->lr_offset; dmu_buf_t *db; rl_t *rl; zgd_t *zgd; int dlen = lr->lr_length; /* length of user data */ int error = 0; ASSERT(zio); ASSERT(dlen != 0); /* * Nothing to do if the file has been removed */ if (zfs_zget(zfsvfs, lr->lr_foid, &zp) != 0) return (ENOENT); if (zp->z_unlinked) { /* * Release the vnode asynchronously as we currently have the * txg stopped from syncing. */ VN_RELE_ASYNC(ZTOV(zp), dsl_pool_vnrele_taskq(dmu_objset_pool(os))); return (ENOENT); } /* * Write records come in two flavors: immediate and indirect. * For small writes it's cheaper to store the data with the * log record (immediate); for large writes it's cheaper to * sync the data and get a pointer to it (indirect) so that * we don't have to write the data twice. */ if (buf != NULL) { /* immediate write */ rl = zfs_range_lock(zp, off, dlen, RL_READER); /* test for truncation needs to be done while range locked */ if (off >= zp->z_phys->zp_size) { error = ENOENT; goto out; } VERIFY(0 == dmu_read(os, lr->lr_foid, off, dlen, buf)); } else { /* indirect write */ uint64_t boff; /* block starting offset */ /* * Have to lock the whole block to ensure when it's * written out and it's checksum is being calculated * that no one can change the data. We need to re-check * blocksize after we get the lock in case it's changed! */ for (;;) { if (ISP2(zp->z_blksz)) { boff = P2ALIGN_TYPED(off, zp->z_blksz, uint64_t); } else { boff = 0; } dlen = zp->z_blksz; rl = zfs_range_lock(zp, boff, dlen, RL_READER); if (zp->z_blksz == dlen) break; zfs_range_unlock(rl); } /* test for truncation needs to be done while range locked */ if (off >= zp->z_phys->zp_size) { error = ENOENT; goto out; } zgd = (zgd_t *)kmem_alloc(sizeof (zgd_t), KM_SLEEP); zgd->zgd_rl = rl; zgd->zgd_zilog = zfsvfs->z_log; zgd->zgd_bp = &lr->lr_blkptr; VERIFY(0 == dmu_buf_hold(os, lr->lr_foid, boff, zgd, &db)); ASSERT(boff == db->db_offset); lr->lr_blkoff = off - boff; error = dmu_sync(zio, db, &lr->lr_blkptr, lr->lr_common.lrc_txg, zfs_get_done, zgd); ASSERT((error && error != EINPROGRESS) || lr->lr_length <= zp->z_blksz); if (error == 0) zil_add_block(zfsvfs->z_log, &lr->lr_blkptr); /* * If we get EINPROGRESS, then we need to wait for a * write IO initiated by dmu_sync() to complete before * we can release this dbuf. We will finish everything * up in the zfs_get_done() callback. */ if (error == EINPROGRESS) return (0); dmu_buf_rele(db, zgd); kmem_free(zgd, sizeof (zgd_t)); } out: zfs_range_unlock(rl); /* * Release the vnode asynchronously as we currently have the * txg stopped from syncing. */ VN_RELE_ASYNC(ZTOV(zp), dsl_pool_vnrele_taskq(dmu_objset_pool(os))); return (error); } /*ARGSUSED*/ static int zfs_access(vnode_t *vp, int mode, int flag, cred_t *cr, caller_context_t *ct) { znode_t *zp = VTOZ(vp); zfsvfs_t *zfsvfs = zp->z_zfsvfs; int error; ZFS_ENTER(zfsvfs); ZFS_VERIFY_ZP(zp); if (flag & V_ACE_MASK) error = zfs_zaccess(zp, mode, flag, B_FALSE, cr); else error = zfs_zaccess_rwx(zp, mode, flag, cr); ZFS_EXIT(zfsvfs); return (error); } /* * Lookup an entry in a directory, or an extended attribute directory. * If it exists, return a held vnode reference for it. * * IN: dvp - vnode of directory to search. * nm - name of entry to lookup. * pnp - full pathname to lookup [UNUSED]. * flags - LOOKUP_XATTR set if looking for an attribute. * rdir - root directory vnode [UNUSED]. * cr - credentials of caller. * ct - caller context * direntflags - directory lookup flags * realpnp - returned pathname. * * OUT: vpp - vnode of located entry, NULL if not found. * * RETURN: 0 if success * error code if failure * * Timestamps: * NA */ /* ARGSUSED */ static int zfs_lookup(vnode_t *dvp, char *nm, vnode_t **vpp, struct componentname *cnp, int nameiop, cred_t *cr, kthread_t *td, int flags) { znode_t *zdp = VTOZ(dvp); zfsvfs_t *zfsvfs = zdp->z_zfsvfs; int error; int *direntflags = NULL; void *realpnp = NULL; ZFS_ENTER(zfsvfs); ZFS_VERIFY_ZP(zdp); *vpp = NULL; if (flags & LOOKUP_XATTR) { #ifdef TODO /* * If the xattr property is off, refuse the lookup request. */ if (!(zfsvfs->z_vfs->vfs_flag & VFS_XATTR)) { ZFS_EXIT(zfsvfs); return (EINVAL); } #endif /* * We don't allow recursive attributes.. * Maybe someday we will. */ if (zdp->z_phys->zp_flags & ZFS_XATTR) { ZFS_EXIT(zfsvfs); return (EINVAL); } if (error = zfs_get_xattrdir(VTOZ(dvp), vpp, cr, flags)) { ZFS_EXIT(zfsvfs); return (error); } /* * Do we have permission to get into attribute directory? */ if (error = zfs_zaccess(VTOZ(*vpp), ACE_EXECUTE, 0, B_FALSE, cr)) { VN_RELE(*vpp); *vpp = NULL; } ZFS_EXIT(zfsvfs); return (error); } if (dvp->v_type != VDIR) { ZFS_EXIT(zfsvfs); return (ENOTDIR); } /* * Check accessibility of directory. */ if (error = zfs_zaccess(zdp, ACE_EXECUTE, 0, B_FALSE, cr)) { ZFS_EXIT(zfsvfs); return (error); } if (zfsvfs->z_utf8 && u8_validate(nm, strlen(nm), NULL, U8_VALIDATE_ENTIRE, &error) < 0) { ZFS_EXIT(zfsvfs); return (EILSEQ); } error = zfs_dirlook(zdp, nm, vpp, flags, direntflags, realpnp); if (error == 0) { /* * Convert device special files */ if (IS_DEVVP(*vpp)) { vnode_t *svp; svp = specvp(*vpp, (*vpp)->v_rdev, (*vpp)->v_type, cr); VN_RELE(*vpp); if (svp == NULL) error = ENOSYS; else *vpp = svp; } } /* Translate errors and add SAVENAME when needed. */ if (cnp->cn_flags & ISLASTCN) { switch (nameiop) { case CREATE: case RENAME: if (error == ENOENT) { error = EJUSTRETURN; cnp->cn_flags |= SAVENAME; break; } /* FALLTHROUGH */ case DELETE: if (error == 0) cnp->cn_flags |= SAVENAME; break; } } if (error == 0 && (nm[0] != '.' || nm[1] != '\0')) { int ltype = 0; if (cnp->cn_flags & ISDOTDOT) { ltype = VOP_ISLOCKED(dvp); VOP_UNLOCK(dvp, 0); } + ZFS_EXIT(zfsvfs); error = vn_lock(*vpp, cnp->cn_lkflags); if (cnp->cn_flags & ISDOTDOT) vn_lock(dvp, ltype | LK_RETRY); if (error != 0) { VN_RELE(*vpp); *vpp = NULL; - ZFS_EXIT(zfsvfs); return (error); } + } else { + ZFS_EXIT(zfsvfs); } #ifdef FREEBSD_NAMECACHE /* * Insert name into cache (as non-existent) if appropriate. */ if (error == ENOENT && (cnp->cn_flags & MAKEENTRY) && nameiop != CREATE) cache_enter(dvp, *vpp, cnp); /* * Insert name into cache if appropriate. */ if (error == 0 && (cnp->cn_flags & MAKEENTRY)) { if (!(cnp->cn_flags & ISLASTCN) || (nameiop != DELETE && nameiop != RENAME)) { cache_enter(dvp, *vpp, cnp); } } #endif - - ZFS_EXIT(zfsvfs); return (error); } /* * Attempt to create a new entry in a directory. If the entry * already exists, truncate the file if permissible, else return * an error. Return the vp of the created or trunc'd file. * * IN: dvp - vnode of directory to put new file entry in. * name - name of new file entry. * vap - attributes of new file. * excl - flag indicating exclusive or non-exclusive mode. * mode - mode to open file with. * cr - credentials of caller. * flag - large file flag [UNUSED]. * ct - caller context * vsecp - ACL to be set * * OUT: vpp - vnode of created or trunc'd entry. * * RETURN: 0 if success * error code if failure * * Timestamps: * dvp - ctime|mtime updated if new entry created * vp - ctime|mtime always, atime if new */ /* ARGSUSED */ static int zfs_create(vnode_t *dvp, char *name, vattr_t *vap, int excl, int mode, vnode_t **vpp, cred_t *cr, kthread_t *td) { znode_t *zp, *dzp = VTOZ(dvp); zfsvfs_t *zfsvfs = dzp->z_zfsvfs; zilog_t *zilog; objset_t *os; zfs_dirlock_t *dl; dmu_tx_t *tx; int error; zfs_acl_t *aclp = NULL; zfs_fuid_info_t *fuidp = NULL; void *vsecp = NULL; int flag = 0; /* * If we have an ephemeral id, ACL, or XVATTR then * make sure file system is at proper version */ if (zfsvfs->z_use_fuids == B_FALSE && (vsecp || (vap->va_mask & AT_XVATTR) || IS_EPHEMERAL(crgetuid(cr)) || IS_EPHEMERAL(crgetgid(cr)))) return (EINVAL); ZFS_ENTER(zfsvfs); ZFS_VERIFY_ZP(dzp); os = zfsvfs->z_os; zilog = zfsvfs->z_log; if (zfsvfs->z_utf8 && u8_validate(name, strlen(name), NULL, U8_VALIDATE_ENTIRE, &error) < 0) { ZFS_EXIT(zfsvfs); return (EILSEQ); } if (vap->va_mask & AT_XVATTR) { if ((error = secpolicy_xvattr(dvp, (xvattr_t *)vap, crgetuid(cr), cr, vap->va_type)) != 0) { ZFS_EXIT(zfsvfs); return (error); } } top: *vpp = NULL; if ((vap->va_mode & S_ISVTX) && secpolicy_vnode_stky_modify(cr)) vap->va_mode &= ~S_ISVTX; if (*name == '\0') { /* * Null component name refers to the directory itself. */ VN_HOLD(dvp); zp = dzp; dl = NULL; error = 0; } else { /* possible VN_HOLD(zp) */ int zflg = 0; if (flag & FIGNORECASE) zflg |= ZCILOOK; error = zfs_dirent_lock(&dl, dzp, name, &zp, zflg, NULL, NULL); if (error) { if (strcmp(name, "..") == 0) error = EISDIR; ZFS_EXIT(zfsvfs); if (aclp) zfs_acl_free(aclp); return (error); } } if (vsecp && aclp == NULL) { error = zfs_vsec_2_aclp(zfsvfs, vap->va_type, vsecp, &aclp); if (error) { ZFS_EXIT(zfsvfs); if (dl) zfs_dirent_unlock(dl); return (error); } } if (zp == NULL) { uint64_t txtype; /* * Create a new file object and update the directory * to reference it. */ if (error = zfs_zaccess(dzp, ACE_ADD_FILE, 0, B_FALSE, cr)) { goto out; } /* * We only support the creation of regular files in * extended attribute directories. */ if ((dzp->z_phys->zp_flags & ZFS_XATTR) && (vap->va_type != VREG)) { error = EINVAL; goto out; } tx = dmu_tx_create(os); dmu_tx_hold_bonus(tx, DMU_NEW_OBJECT); if ((aclp && aclp->z_has_fuids) || IS_EPHEMERAL(crgetuid(cr)) || IS_EPHEMERAL(crgetgid(cr))) { if (zfsvfs->z_fuid_obj == 0) { dmu_tx_hold_bonus(tx, DMU_NEW_OBJECT); dmu_tx_hold_write(tx, DMU_NEW_OBJECT, 0, FUID_SIZE_ESTIMATE(zfsvfs)); dmu_tx_hold_zap(tx, MASTER_NODE_OBJ, FALSE, NULL); } else { dmu_tx_hold_bonus(tx, zfsvfs->z_fuid_obj); dmu_tx_hold_write(tx, zfsvfs->z_fuid_obj, 0, FUID_SIZE_ESTIMATE(zfsvfs)); } } dmu_tx_hold_bonus(tx, dzp->z_id); dmu_tx_hold_zap(tx, dzp->z_id, TRUE, name); if ((dzp->z_phys->zp_flags & ZFS_INHERIT_ACE) || aclp) { dmu_tx_hold_write(tx, DMU_NEW_OBJECT, 0, SPA_MAXBLOCKSIZE); } error = dmu_tx_assign(tx, zfsvfs->z_assign); if (error) { zfs_dirent_unlock(dl); if (error == ERESTART && zfsvfs->z_assign == TXG_NOWAIT) { dmu_tx_wait(tx); dmu_tx_abort(tx); goto top; } dmu_tx_abort(tx); ZFS_EXIT(zfsvfs); if (aclp) zfs_acl_free(aclp); return (error); } zfs_mknode(dzp, vap, tx, cr, 0, &zp, 0, aclp, &fuidp); (void) zfs_link_create(dl, zp, tx, ZNEW); txtype = zfs_log_create_txtype(Z_FILE, vsecp, vap); if (flag & FIGNORECASE) txtype |= TX_CI; zfs_log_create(zilog, tx, txtype, dzp, zp, name, vsecp, fuidp, vap); if (fuidp) zfs_fuid_info_free(fuidp); dmu_tx_commit(tx); } else { int aflags = (flag & FAPPEND) ? V_APPEND : 0; /* * A directory entry already exists for this name. */ /* * Can't truncate an existing file if in exclusive mode. */ if (excl == EXCL) { error = EEXIST; goto out; } /* * Can't open a directory for writing. */ if ((ZTOV(zp)->v_type == VDIR) && (mode & S_IWRITE)) { error = EISDIR; goto out; } /* * Verify requested access to file. */ if (mode && (error = zfs_zaccess_rwx(zp, mode, aflags, cr))) { goto out; } mutex_enter(&dzp->z_lock); dzp->z_seq++; mutex_exit(&dzp->z_lock); /* * Truncate regular files if requested. */ if ((ZTOV(zp)->v_type == VREG) && (vap->va_mask & AT_SIZE) && (vap->va_size == 0)) { /* we can't hold any locks when calling zfs_freesp() */ zfs_dirent_unlock(dl); dl = NULL; error = zfs_freesp(zp, 0, 0, mode, TRUE); if (error == 0) { vnevent_create(ZTOV(zp), ct); } } } out: if (dl) zfs_dirent_unlock(dl); if (error) { if (zp) VN_RELE(ZTOV(zp)); } else { *vpp = ZTOV(zp); /* * If vnode is for a device return a specfs vnode instead. */ if (IS_DEVVP(*vpp)) { struct vnode *svp; svp = specvp(*vpp, (*vpp)->v_rdev, (*vpp)->v_type, cr); VN_RELE(*vpp); if (svp == NULL) { error = ENOSYS; } *vpp = svp; } } if (aclp) zfs_acl_free(aclp); ZFS_EXIT(zfsvfs); return (error); } /* * Remove an entry from a directory. * * IN: dvp - vnode of directory to remove entry from. * name - name of entry to remove. * cr - credentials of caller. * ct - caller context * flags - case flags * * RETURN: 0 if success * error code if failure * * Timestamps: * dvp - ctime|mtime * vp - ctime (if nlink > 0) */ /*ARGSUSED*/ static int zfs_remove(vnode_t *dvp, char *name, cred_t *cr, caller_context_t *ct, int flags) { znode_t *zp, *dzp = VTOZ(dvp); znode_t *xzp = NULL; vnode_t *vp; zfsvfs_t *zfsvfs = dzp->z_zfsvfs; zilog_t *zilog; uint64_t acl_obj, xattr_obj; zfs_dirlock_t *dl; dmu_tx_t *tx; boolean_t may_delete_now, delete_now = FALSE; boolean_t unlinked, toobig = FALSE; uint64_t txtype; pathname_t *realnmp = NULL; pathname_t realnm; int error; int zflg = ZEXISTS; ZFS_ENTER(zfsvfs); ZFS_VERIFY_ZP(dzp); zilog = zfsvfs->z_log; if (flags & FIGNORECASE) { zflg |= ZCILOOK; pn_alloc(&realnm); realnmp = &realnm; } top: /* * Attempt to lock directory; fail if entry doesn't exist. */ if (error = zfs_dirent_lock(&dl, dzp, name, &zp, zflg, NULL, realnmp)) { if (realnmp) pn_free(realnmp); ZFS_EXIT(zfsvfs); return (error); } vp = ZTOV(zp); if (error = zfs_zaccess_delete(dzp, zp, cr)) { goto out; } /* * Need to use rmdir for removing directories. */ if (vp->v_type == VDIR) { error = EPERM; goto out; } vnevent_remove(vp, dvp, name, ct); if (realnmp) dnlc_remove(dvp, realnmp->pn_buf); else dnlc_remove(dvp, name); may_delete_now = FALSE; /* * We may delete the znode now, or we may put it in the unlinked set; * it depends on whether we're the last link, and on whether there are * other holds on the vnode. So we dmu_tx_hold() the right things to * allow for either case. */ tx = dmu_tx_create(zfsvfs->z_os); dmu_tx_hold_zap(tx, dzp->z_id, FALSE, name); dmu_tx_hold_bonus(tx, zp->z_id); if (may_delete_now) { toobig = zp->z_phys->zp_size > zp->z_blksz * DMU_MAX_DELETEBLKCNT; /* if the file is too big, only hold_free a token amount */ dmu_tx_hold_free(tx, zp->z_id, 0, (toobig ? DMU_MAX_ACCESS : DMU_OBJECT_END)); } /* are there any extended attributes? */ if ((xattr_obj = zp->z_phys->zp_xattr) != 0) { /* XXX - do we need this if we are deleting? */ dmu_tx_hold_bonus(tx, xattr_obj); } /* are there any additional acls */ if ((acl_obj = zp->z_phys->zp_acl.z_acl_extern_obj) != 0 && may_delete_now) dmu_tx_hold_free(tx, acl_obj, 0, DMU_OBJECT_END); /* charge as an update -- would be nice not to charge at all */ dmu_tx_hold_zap(tx, zfsvfs->z_unlinkedobj, FALSE, NULL); error = dmu_tx_assign(tx, zfsvfs->z_assign); if (error) { zfs_dirent_unlock(dl); VN_RELE(vp); if (error == ERESTART && zfsvfs->z_assign == TXG_NOWAIT) { dmu_tx_wait(tx); dmu_tx_abort(tx); goto top; } if (realnmp) pn_free(realnmp); dmu_tx_abort(tx); ZFS_EXIT(zfsvfs); return (error); } /* * Remove the directory entry. */ error = zfs_link_destroy(dl, zp, tx, zflg, &unlinked); if (error) { dmu_tx_commit(tx); goto out; } if (0 && unlinked) { VI_LOCK(vp); delete_now = may_delete_now && !toobig && vp->v_count == 1 && !vn_has_cached_data(vp) && zp->z_phys->zp_xattr == xattr_obj && zp->z_phys->zp_acl.z_acl_extern_obj == acl_obj; VI_UNLOCK(vp); } if (delete_now) { if (zp->z_phys->zp_xattr) { error = zfs_zget(zfsvfs, zp->z_phys->zp_xattr, &xzp); ASSERT3U(error, ==, 0); ASSERT3U(xzp->z_phys->zp_links, ==, 2); dmu_buf_will_dirty(xzp->z_dbuf, tx); mutex_enter(&xzp->z_lock); xzp->z_unlinked = 1; xzp->z_phys->zp_links = 0; mutex_exit(&xzp->z_lock); zfs_unlinked_add(xzp, tx); zp->z_phys->zp_xattr = 0; /* probably unnecessary */ } mutex_enter(&zp->z_lock); VI_LOCK(vp); vp->v_count--; ASSERT3U(vp->v_count, ==, 0); VI_UNLOCK(vp); mutex_exit(&zp->z_lock); zfs_znode_delete(zp, tx); } else if (unlinked) { zfs_unlinked_add(zp, tx); } txtype = TX_REMOVE; if (flags & FIGNORECASE) txtype |= TX_CI; zfs_log_remove(zilog, tx, txtype, dzp, name); dmu_tx_commit(tx); out: if (realnmp) pn_free(realnmp); zfs_dirent_unlock(dl); if (!delete_now) { VN_RELE(vp); } else if (xzp) { /* this rele is delayed to prevent nesting transactions */ VN_RELE(ZTOV(xzp)); } ZFS_EXIT(zfsvfs); return (error); } /* * Create a new directory and insert it into dvp using the name * provided. Return a pointer to the inserted directory. * * IN: dvp - vnode of directory to add subdir to. * dirname - name of new directory. * vap - attributes of new directory. * cr - credentials of caller. * ct - caller context * vsecp - ACL to be set * * OUT: vpp - vnode of created directory. * * RETURN: 0 if success * error code if failure * * Timestamps: * dvp - ctime|mtime updated * vp - ctime|mtime|atime updated */ /*ARGSUSED*/ static int zfs_mkdir(vnode_t *dvp, char *dirname, vattr_t *vap, vnode_t **vpp, cred_t *cr, caller_context_t *ct, int flags, vsecattr_t *vsecp) { znode_t *zp, *dzp = VTOZ(dvp); zfsvfs_t *zfsvfs = dzp->z_zfsvfs; zilog_t *zilog; zfs_dirlock_t *dl; uint64_t txtype; dmu_tx_t *tx; int error; zfs_acl_t *aclp = NULL; zfs_fuid_info_t *fuidp = NULL; int zf = ZNEW; ASSERT(vap->va_type == VDIR); /* * If we have an ephemeral id, ACL, or XVATTR then * make sure file system is at proper version */ if (zfsvfs->z_use_fuids == B_FALSE && (vsecp || (vap->va_mask & AT_XVATTR) || IS_EPHEMERAL(crgetuid(cr))|| IS_EPHEMERAL(crgetgid(cr)))) return (EINVAL); ZFS_ENTER(zfsvfs); ZFS_VERIFY_ZP(dzp); zilog = zfsvfs->z_log; if (dzp->z_phys->zp_flags & ZFS_XATTR) { ZFS_EXIT(zfsvfs); return (EINVAL); } if (zfsvfs->z_utf8 && u8_validate(dirname, strlen(dirname), NULL, U8_VALIDATE_ENTIRE, &error) < 0) { ZFS_EXIT(zfsvfs); return (EILSEQ); } if (flags & FIGNORECASE) zf |= ZCILOOK; if (vap->va_mask & AT_XVATTR) if ((error = secpolicy_xvattr(dvp, (xvattr_t *)vap, crgetuid(cr), cr, vap->va_type)) != 0) { ZFS_EXIT(zfsvfs); return (error); } /* * First make sure the new directory doesn't exist. */ top: *vpp = NULL; if (error = zfs_dirent_lock(&dl, dzp, dirname, &zp, zf, NULL, NULL)) { ZFS_EXIT(zfsvfs); return (error); } if (error = zfs_zaccess(dzp, ACE_ADD_SUBDIRECTORY, 0, B_FALSE, cr)) { zfs_dirent_unlock(dl); ZFS_EXIT(zfsvfs); return (error); } if (vsecp && aclp == NULL) { error = zfs_vsec_2_aclp(zfsvfs, vap->va_type, vsecp, &aclp); if (error) { zfs_dirent_unlock(dl); ZFS_EXIT(zfsvfs); return (error); } } /* * Add a new entry to the directory. */ tx = dmu_tx_create(zfsvfs->z_os); dmu_tx_hold_zap(tx, dzp->z_id, TRUE, dirname); dmu_tx_hold_zap(tx, DMU_NEW_OBJECT, FALSE, NULL); if ((aclp && aclp->z_has_fuids) || IS_EPHEMERAL(crgetuid(cr)) || IS_EPHEMERAL(crgetgid(cr))) { if (zfsvfs->z_fuid_obj == 0) { dmu_tx_hold_bonus(tx, DMU_NEW_OBJECT); dmu_tx_hold_write(tx, DMU_NEW_OBJECT, 0, FUID_SIZE_ESTIMATE(zfsvfs)); dmu_tx_hold_zap(tx, MASTER_NODE_OBJ, FALSE, NULL); } else { dmu_tx_hold_bonus(tx, zfsvfs->z_fuid_obj); dmu_tx_hold_write(tx, zfsvfs->z_fuid_obj, 0, FUID_SIZE_ESTIMATE(zfsvfs)); } } if ((dzp->z_phys->zp_flags & ZFS_INHERIT_ACE) || aclp) dmu_tx_hold_write(tx, DMU_NEW_OBJECT, 0, SPA_MAXBLOCKSIZE); error = dmu_tx_assign(tx, zfsvfs->z_assign); if (error) { zfs_dirent_unlock(dl); if (error == ERESTART && zfsvfs->z_assign == TXG_NOWAIT) { dmu_tx_wait(tx); dmu_tx_abort(tx); goto top; } dmu_tx_abort(tx); ZFS_EXIT(zfsvfs); if (aclp) zfs_acl_free(aclp); return (error); } /* * Create new node. */ zfs_mknode(dzp, vap, tx, cr, 0, &zp, 0, aclp, &fuidp); if (aclp) zfs_acl_free(aclp); /* * Now put new name in parent dir. */ (void) zfs_link_create(dl, zp, tx, ZNEW); *vpp = ZTOV(zp); txtype = zfs_log_create_txtype(Z_DIR, vsecp, vap); if (flags & FIGNORECASE) txtype |= TX_CI; zfs_log_create(zilog, tx, txtype, dzp, zp, dirname, vsecp, fuidp, vap); if (fuidp) zfs_fuid_info_free(fuidp); dmu_tx_commit(tx); zfs_dirent_unlock(dl); ZFS_EXIT(zfsvfs); return (0); } /* * Remove a directory subdir entry. If the current working * directory is the same as the subdir to be removed, the * remove will fail. * * IN: dvp - vnode of directory to remove from. * name - name of directory to be removed. * cwd - vnode of current working directory. * cr - credentials of caller. * ct - caller context * flags - case flags * * RETURN: 0 if success * error code if failure * * Timestamps: * dvp - ctime|mtime updated */ /*ARGSUSED*/ static int zfs_rmdir(vnode_t *dvp, char *name, vnode_t *cwd, cred_t *cr, caller_context_t *ct, int flags) { znode_t *dzp = VTOZ(dvp); znode_t *zp; vnode_t *vp; zfsvfs_t *zfsvfs = dzp->z_zfsvfs; zilog_t *zilog; zfs_dirlock_t *dl; dmu_tx_t *tx; int error; int zflg = ZEXISTS; ZFS_ENTER(zfsvfs); ZFS_VERIFY_ZP(dzp); zilog = zfsvfs->z_log; if (flags & FIGNORECASE) zflg |= ZCILOOK; top: zp = NULL; /* * Attempt to lock directory; fail if entry doesn't exist. */ if (error = zfs_dirent_lock(&dl, dzp, name, &zp, zflg, NULL, NULL)) { ZFS_EXIT(zfsvfs); return (error); } vp = ZTOV(zp); if (error = zfs_zaccess_delete(dzp, zp, cr)) { goto out; } if (vp->v_type != VDIR) { error = ENOTDIR; goto out; } if (vp == cwd) { error = EINVAL; goto out; } vnevent_rmdir(vp, dvp, name, ct); /* * Grab a lock on the directory to make sure that noone is * trying to add (or lookup) entries while we are removing it. */ rw_enter(&zp->z_name_lock, RW_WRITER); /* * Grab a lock on the parent pointer to make sure we play well * with the treewalk and directory rename code. */ rw_enter(&zp->z_parent_lock, RW_WRITER); tx = dmu_tx_create(zfsvfs->z_os); dmu_tx_hold_zap(tx, dzp->z_id, FALSE, name); dmu_tx_hold_bonus(tx, zp->z_id); dmu_tx_hold_zap(tx, zfsvfs->z_unlinkedobj, FALSE, NULL); error = dmu_tx_assign(tx, zfsvfs->z_assign); if (error) { rw_exit(&zp->z_parent_lock); rw_exit(&zp->z_name_lock); zfs_dirent_unlock(dl); VN_RELE(vp); if (error == ERESTART && zfsvfs->z_assign == TXG_NOWAIT) { dmu_tx_wait(tx); dmu_tx_abort(tx); goto top; } dmu_tx_abort(tx); ZFS_EXIT(zfsvfs); return (error); } #ifdef FREEBSD_NAMECACHE cache_purge(dvp); #endif error = zfs_link_destroy(dl, zp, tx, zflg, NULL); if (error == 0) { uint64_t txtype = TX_RMDIR; if (flags & FIGNORECASE) txtype |= TX_CI; zfs_log_remove(zilog, tx, txtype, dzp, name); } dmu_tx_commit(tx); rw_exit(&zp->z_parent_lock); rw_exit(&zp->z_name_lock); #ifdef FREEBSD_NAMECACHE cache_purge(vp); #endif out: zfs_dirent_unlock(dl); VN_RELE(vp); ZFS_EXIT(zfsvfs); return (error); } /* * Read as many directory entries as will fit into the provided * buffer from the given directory cursor position (specified in * the uio structure. * * IN: vp - vnode of directory to read. * uio - structure supplying read location, range info, * and return buffer. * cr - credentials of caller. * ct - caller context * flags - case flags * * OUT: uio - updated offset and range, buffer filled. * eofp - set to true if end-of-file detected. * * RETURN: 0 if success * error code if failure * * Timestamps: * vp - atime updated * * Note that the low 4 bits of the cookie returned by zap is always zero. * This allows us to use the low range for "special" directory entries: * We use 0 for '.', and 1 for '..'. If this is the root of the filesystem, * we use the offset 2 for the '.zfs' directory. */ /* ARGSUSED */ static int zfs_readdir(vnode_t *vp, uio_t *uio, cred_t *cr, int *eofp, int *ncookies, u_long **cookies) { znode_t *zp = VTOZ(vp); iovec_t *iovp; edirent_t *eodp; dirent64_t *odp; zfsvfs_t *zfsvfs = zp->z_zfsvfs; objset_t *os; caddr_t outbuf; size_t bufsize; zap_cursor_t zc; zap_attribute_t zap; uint_t bytes_wanted; uint64_t offset; /* must be unsigned; checks for < 1 */ int local_eof; int outcount; int error; uint8_t prefetch; boolean_t check_sysattrs; uint8_t type; int ncooks; u_long *cooks = NULL; int flags = 0; ZFS_ENTER(zfsvfs); ZFS_VERIFY_ZP(zp); /* * If we are not given an eof variable, * use a local one. */ if (eofp == NULL) eofp = &local_eof; /* * Check for valid iov_len. */ if (uio->uio_iov->iov_len <= 0) { ZFS_EXIT(zfsvfs); return (EINVAL); } /* * Quit if directory has been removed (posix) */ if ((*eofp = zp->z_unlinked) != 0) { ZFS_EXIT(zfsvfs); return (0); } error = 0; os = zfsvfs->z_os; offset = uio->uio_loffset; prefetch = zp->z_zn_prefetch; /* * Initialize the iterator cursor. */ if (offset <= 3) { /* * Start iteration from the beginning of the directory. */ zap_cursor_init(&zc, os, zp->z_id); } else { /* * The offset is a serialized cursor. */ zap_cursor_init_serialized(&zc, os, zp->z_id, offset); } /* * Get space to change directory entries into fs independent format. */ iovp = uio->uio_iov; bytes_wanted = iovp->iov_len; if (uio->uio_segflg != UIO_SYSSPACE || uio->uio_iovcnt != 1) { bufsize = bytes_wanted; outbuf = kmem_alloc(bufsize, KM_SLEEP); odp = (struct dirent64 *)outbuf; } else { bufsize = bytes_wanted; odp = (struct dirent64 *)iovp->iov_base; } eodp = (struct edirent *)odp; if (ncookies != NULL) { /* * Minimum entry size is dirent size and 1 byte for a file name. */ ncooks = uio->uio_resid / (sizeof(struct dirent) - sizeof(((struct dirent *)NULL)->d_name) + 1); cooks = malloc(ncooks * sizeof(u_long), M_TEMP, M_WAITOK); *cookies = cooks; *ncookies = ncooks; } /* * If this VFS supports the system attribute view interface; and * we're looking at an extended attribute directory; and we care * about normalization conflicts on this vfs; then we must check * for normalization conflicts with the sysattr name space. */ #ifdef TODO check_sysattrs = vfs_has_feature(vp->v_vfsp, VFSFT_SYSATTR_VIEWS) && (vp->v_flag & V_XATTRDIR) && zfsvfs->z_norm && (flags & V_RDDIR_ENTFLAGS); #else check_sysattrs = 0; #endif /* * Transform to file-system independent format */ outcount = 0; while (outcount < bytes_wanted) { ino64_t objnum; ushort_t reclen; off64_t *next; /* * Special case `.', `..', and `.zfs'. */ if (offset == 0) { (void) strcpy(zap.za_name, "."); zap.za_normalization_conflict = 0; objnum = zp->z_id; type = DT_DIR; } else if (offset == 1) { (void) strcpy(zap.za_name, ".."); zap.za_normalization_conflict = 0; objnum = zp->z_phys->zp_parent; type = DT_DIR; } else if (offset == 2 && zfs_show_ctldir(zp)) { (void) strcpy(zap.za_name, ZFS_CTLDIR_NAME); zap.za_normalization_conflict = 0; objnum = ZFSCTL_INO_ROOT; type = DT_DIR; } else { /* * Grab next entry. */ if (error = zap_cursor_retrieve(&zc, &zap)) { if ((*eofp = (error == ENOENT)) != 0) break; else goto update; } if (zap.za_integer_length != 8 || zap.za_num_integers != 1) { cmn_err(CE_WARN, "zap_readdir: bad directory " "entry, obj = %lld, offset = %lld\n", (u_longlong_t)zp->z_id, (u_longlong_t)offset); error = ENXIO; goto update; } objnum = ZFS_DIRENT_OBJ(zap.za_first_integer); /* * MacOS X can extract the object type here such as: * uint8_t type = ZFS_DIRENT_TYPE(zap.za_first_integer); */ type = ZFS_DIRENT_TYPE(zap.za_first_integer); if (check_sysattrs && !zap.za_normalization_conflict) { #ifdef TODO zap.za_normalization_conflict = xattr_sysattr_casechk(zap.za_name); #else panic("%s:%u: TODO", __func__, __LINE__); #endif } } if (flags & V_RDDIR_ENTFLAGS) reclen = EDIRENT_RECLEN(strlen(zap.za_name)); else reclen = DIRENT64_RECLEN(strlen(zap.za_name)); /* * Will this entry fit in the buffer? */ if (outcount + reclen > bufsize) { /* * Did we manage to fit anything in the buffer? */ if (!outcount) { error = EINVAL; goto update; } break; } if (flags & V_RDDIR_ENTFLAGS) { /* * Add extended flag entry: */ eodp->ed_ino = objnum; eodp->ed_reclen = reclen; /* NOTE: ed_off is the offset for the *next* entry */ next = &(eodp->ed_off); eodp->ed_eflags = zap.za_normalization_conflict ? ED_CASE_CONFLICT : 0; (void) strncpy(eodp->ed_name, zap.za_name, EDIRENT_NAMELEN(reclen)); eodp = (edirent_t *)((intptr_t)eodp + reclen); } else { /* * Add normal entry: */ odp->d_ino = objnum; odp->d_reclen = reclen; odp->d_namlen = strlen(zap.za_name); (void) strlcpy(odp->d_name, zap.za_name, odp->d_namlen + 1); odp->d_type = type; odp = (dirent64_t *)((intptr_t)odp + reclen); } outcount += reclen; ASSERT(outcount <= bufsize); /* Prefetch znode */ if (prefetch) dmu_prefetch(os, objnum, 0, 0); /* * Move to the next entry, fill in the previous offset. */ if (offset > 2 || (offset == 2 && !zfs_show_ctldir(zp))) { zap_cursor_advance(&zc); offset = zap_cursor_serialize(&zc); } else { offset += 1; } if (cooks != NULL) { *cooks++ = offset; ncooks--; KASSERT(ncooks >= 0, ("ncookies=%d", ncooks)); } } zp->z_zn_prefetch = B_FALSE; /* a lookup will re-enable pre-fetching */ /* Subtract unused cookies */ if (ncookies != NULL) *ncookies -= ncooks; if (uio->uio_segflg == UIO_SYSSPACE && uio->uio_iovcnt == 1) { iovp->iov_base += outcount; iovp->iov_len -= outcount; uio->uio_resid -= outcount; } else if (error = uiomove(outbuf, (long)outcount, UIO_READ, uio)) { /* * Reset the pointer. */ offset = uio->uio_loffset; } update: zap_cursor_fini(&zc); if (uio->uio_segflg != UIO_SYSSPACE || uio->uio_iovcnt != 1) kmem_free(outbuf, bufsize); if (error == ENOENT) error = 0; ZFS_ACCESSTIME_STAMP(zfsvfs, zp); uio->uio_loffset = offset; ZFS_EXIT(zfsvfs); if (error != 0 && cookies != NULL) { free(*cookies, M_TEMP); *cookies = NULL; *ncookies = 0; } return (error); } ulong_t zfs_fsync_sync_cnt = 4; static int zfs_fsync(vnode_t *vp, int syncflag, cred_t *cr, caller_context_t *ct) { znode_t *zp = VTOZ(vp); zfsvfs_t *zfsvfs = zp->z_zfsvfs; (void) tsd_set(zfs_fsyncer_key, (void *)zfs_fsync_sync_cnt); ZFS_ENTER(zfsvfs); ZFS_VERIFY_ZP(zp); zil_commit(zfsvfs->z_log, zp->z_last_itx, zp->z_id); ZFS_EXIT(zfsvfs); return (0); } /* * Get the requested file attributes and place them in the provided * vattr structure. * * IN: vp - vnode of file. * vap - va_mask identifies requested attributes. * If AT_XVATTR set, then optional attrs are requested * flags - ATTR_NOACLCHECK (CIFS server context) * cr - credentials of caller. * ct - caller context * * OUT: vap - attribute values. * * RETURN: 0 (always succeeds) */ /* ARGSUSED */ static int zfs_getattr(vnode_t *vp, vattr_t *vap, int flags, cred_t *cr, caller_context_t *ct) { znode_t *zp = VTOZ(vp); zfsvfs_t *zfsvfs = zp->z_zfsvfs; znode_phys_t *pzp; int error = 0; uint32_t blksize; u_longlong_t nblocks; uint64_t links; xvattr_t *xvap = (xvattr_t *)vap; /* vap may be an xvattr_t * */ xoptattr_t *xoap = NULL; boolean_t skipaclchk = (flags & ATTR_NOACLCHECK) ? B_TRUE : B_FALSE; ZFS_ENTER(zfsvfs); ZFS_VERIFY_ZP(zp); pzp = zp->z_phys; mutex_enter(&zp->z_lock); /* * If ACL is trivial don't bother looking for ACE_READ_ATTRIBUTES. * Also, if we are the owner don't bother, since owner should * always be allowed to read basic attributes of file. */ if (!(pzp->zp_flags & ZFS_ACL_TRIVIAL) && (pzp->zp_uid != crgetuid(cr))) { if (error = zfs_zaccess(zp, ACE_READ_ATTRIBUTES, 0, skipaclchk, cr)) { mutex_exit(&zp->z_lock); ZFS_EXIT(zfsvfs); return (error); } } /* * Return all attributes. It's cheaper to provide the answer * than to determine whether we were asked the question. */ vap->va_type = IFTOVT(pzp->zp_mode); vap->va_mode = pzp->zp_mode & ~S_IFMT; zfs_fuid_map_ids(zp, cr, &vap->va_uid, &vap->va_gid); // vap->va_fsid = zp->z_zfsvfs->z_vfs->vfs_dev; vap->va_nodeid = zp->z_id; if ((vp->v_flag & VROOT) && zfs_show_ctldir(zp)) links = pzp->zp_links + 1; else links = pzp->zp_links; vap->va_nlink = MIN(links, UINT32_MAX); /* nlink_t limit! */ vap->va_size = pzp->zp_size; vap->va_fsid = vp->v_mount->mnt_stat.f_fsid.val[0]; vap->va_rdev = zfs_cmpldev(pzp->zp_rdev); vap->va_seq = zp->z_seq; vap->va_flags = 0; /* FreeBSD: Reset chflags(2) flags. */ /* * Add in any requested optional attributes and the create time. * Also set the corresponding bits in the returned attribute bitmap. */ if ((xoap = xva_getxoptattr(xvap)) != NULL && zfsvfs->z_use_fuids) { if (XVA_ISSET_REQ(xvap, XAT_ARCHIVE)) { xoap->xoa_archive = ((pzp->zp_flags & ZFS_ARCHIVE) != 0); XVA_SET_RTN(xvap, XAT_ARCHIVE); } if (XVA_ISSET_REQ(xvap, XAT_READONLY)) { xoap->xoa_readonly = ((pzp->zp_flags & ZFS_READONLY) != 0); XVA_SET_RTN(xvap, XAT_READONLY); } if (XVA_ISSET_REQ(xvap, XAT_SYSTEM)) { xoap->xoa_system = ((pzp->zp_flags & ZFS_SYSTEM) != 0); XVA_SET_RTN(xvap, XAT_SYSTEM); } if (XVA_ISSET_REQ(xvap, XAT_HIDDEN)) { xoap->xoa_hidden = ((pzp->zp_flags & ZFS_HIDDEN) != 0); XVA_SET_RTN(xvap, XAT_HIDDEN); } if (XVA_ISSET_REQ(xvap, XAT_NOUNLINK)) { xoap->xoa_nounlink = ((pzp->zp_flags & ZFS_NOUNLINK) != 0); XVA_SET_RTN(xvap, XAT_NOUNLINK); } if (XVA_ISSET_REQ(xvap, XAT_IMMUTABLE)) { xoap->xoa_immutable = ((pzp->zp_flags & ZFS_IMMUTABLE) != 0); XVA_SET_RTN(xvap, XAT_IMMUTABLE); } if (XVA_ISSET_REQ(xvap, XAT_APPENDONLY)) { xoap->xoa_appendonly = ((pzp->zp_flags & ZFS_APPENDONLY) != 0); XVA_SET_RTN(xvap, XAT_APPENDONLY); } if (XVA_ISSET_REQ(xvap, XAT_NODUMP)) { xoap->xoa_nodump = ((pzp->zp_flags & ZFS_NODUMP) != 0); XVA_SET_RTN(xvap, XAT_NODUMP); } if (XVA_ISSET_REQ(xvap, XAT_OPAQUE)) { xoap->xoa_opaque = ((pzp->zp_flags & ZFS_OPAQUE) != 0); XVA_SET_RTN(xvap, XAT_OPAQUE); } if (XVA_ISSET_REQ(xvap, XAT_AV_QUARANTINED)) { xoap->xoa_av_quarantined = ((pzp->zp_flags & ZFS_AV_QUARANTINED) != 0); XVA_SET_RTN(xvap, XAT_AV_QUARANTINED); } if (XVA_ISSET_REQ(xvap, XAT_AV_MODIFIED)) { xoap->xoa_av_modified = ((pzp->zp_flags & ZFS_AV_MODIFIED) != 0); XVA_SET_RTN(xvap, XAT_AV_MODIFIED); } if (XVA_ISSET_REQ(xvap, XAT_AV_SCANSTAMP) && vp->v_type == VREG && (pzp->zp_flags & ZFS_BONUS_SCANSTAMP)) { size_t len; dmu_object_info_t doi; /* * Only VREG files have anti-virus scanstamps, so we * won't conflict with symlinks in the bonus buffer. */ dmu_object_info_from_db(zp->z_dbuf, &doi); len = sizeof (xoap->xoa_av_scanstamp) + sizeof (znode_phys_t); if (len <= doi.doi_bonus_size) { /* * pzp points to the start of the * znode_phys_t. pzp + 1 points to the * first byte after the znode_phys_t. */ (void) memcpy(xoap->xoa_av_scanstamp, pzp + 1, sizeof (xoap->xoa_av_scanstamp)); XVA_SET_RTN(xvap, XAT_AV_SCANSTAMP); } } if (XVA_ISSET_REQ(xvap, XAT_CREATETIME)) { ZFS_TIME_DECODE(&xoap->xoa_createtime, pzp->zp_crtime); XVA_SET_RTN(xvap, XAT_CREATETIME); } } ZFS_TIME_DECODE(&vap->va_atime, pzp->zp_atime); ZFS_TIME_DECODE(&vap->va_mtime, pzp->zp_mtime); ZFS_TIME_DECODE(&vap->va_ctime, pzp->zp_ctime); ZFS_TIME_DECODE(&vap->va_birthtime, pzp->zp_crtime); mutex_exit(&zp->z_lock); dmu_object_size_from_db(zp->z_dbuf, &blksize, &nblocks); vap->va_blksize = blksize; vap->va_bytes = nblocks << 9; /* nblocks * 512 */ if (zp->z_blksz == 0) { /* * Block size hasn't been set; suggest maximal I/O transfers. */ vap->va_blksize = zfsvfs->z_max_blksz; } ZFS_EXIT(zfsvfs); return (0); } /* * Set the file attributes to the values contained in the * vattr structure. * * IN: vp - vnode of file to be modified. * vap - new attribute values. * If AT_XVATTR set, then optional attrs are being set * flags - ATTR_UTIME set if non-default time values provided. * - ATTR_NOACLCHECK (CIFS context only). * cr - credentials of caller. * ct - caller context * * RETURN: 0 if success * error code if failure * * Timestamps: * vp - ctime updated, mtime updated if size changed. */ /* ARGSUSED */ static int zfs_setattr(vnode_t *vp, vattr_t *vap, int flags, cred_t *cr, caller_context_t *ct) { znode_t *zp = VTOZ(vp); znode_phys_t *pzp; zfsvfs_t *zfsvfs = zp->z_zfsvfs; zilog_t *zilog; dmu_tx_t *tx; vattr_t oldva; uint_t mask = vap->va_mask; uint_t saved_mask; uint64_t saved_mode; int trim_mask = 0; uint64_t new_mode; znode_t *attrzp; int need_policy = FALSE; int err; zfs_fuid_info_t *fuidp = NULL; xvattr_t *xvap = (xvattr_t *)vap; /* vap may be an xvattr_t * */ xoptattr_t *xoap; zfs_acl_t *aclp = NULL; boolean_t skipaclchk = (flags & ATTR_NOACLCHECK) ? B_TRUE : B_FALSE; if (mask == 0) return (0); if (mask & AT_NOSET) return (EINVAL); ZFS_ENTER(zfsvfs); ZFS_VERIFY_ZP(zp); pzp = zp->z_phys; zilog = zfsvfs->z_log; /* * Make sure that if we have ephemeral uid/gid or xvattr specified * that file system is at proper version level */ if (zfsvfs->z_use_fuids == B_FALSE && (((mask & AT_UID) && IS_EPHEMERAL(vap->va_uid)) || ((mask & AT_GID) && IS_EPHEMERAL(vap->va_gid)) || (mask & AT_XVATTR))) { ZFS_EXIT(zfsvfs); return (EINVAL); } if (mask & AT_SIZE && vp->v_type == VDIR) { ZFS_EXIT(zfsvfs); return (EISDIR); } if (mask & AT_SIZE && vp->v_type != VREG && vp->v_type != VFIFO) { ZFS_EXIT(zfsvfs); return (EINVAL); } /* * If this is an xvattr_t, then get a pointer to the structure of * optional attributes. If this is NULL, then we have a vattr_t. */ xoap = xva_getxoptattr(xvap); /* * Immutable files can only alter immutable bit and atime */ if ((pzp->zp_flags & ZFS_IMMUTABLE) && ((mask & (AT_SIZE|AT_UID|AT_GID|AT_MTIME|AT_MODE)) || ((mask & AT_XVATTR) && XVA_ISSET_REQ(xvap, XAT_CREATETIME)))) { ZFS_EXIT(zfsvfs); return (EPERM); } if ((mask & AT_SIZE) && (pzp->zp_flags & ZFS_READONLY)) { ZFS_EXIT(zfsvfs); return (EPERM); } /* * Verify timestamps doesn't overflow 32 bits. * ZFS can handle large timestamps, but 32bit syscalls can't * handle times greater than 2039. This check should be removed * once large timestamps are fully supported. */ if (mask & (AT_ATIME | AT_MTIME)) { if (((mask & AT_ATIME) && TIMESPEC_OVERFLOW(&vap->va_atime)) || ((mask & AT_MTIME) && TIMESPEC_OVERFLOW(&vap->va_mtime))) { ZFS_EXIT(zfsvfs); return (EOVERFLOW); } } top: attrzp = NULL; if (zfsvfs->z_vfs->vfs_flag & VFS_RDONLY) { ZFS_EXIT(zfsvfs); return (EROFS); } /* * First validate permissions */ if (mask & AT_SIZE) { err = zfs_zaccess(zp, ACE_WRITE_DATA, 0, skipaclchk, cr); if (err) { ZFS_EXIT(zfsvfs); return (err); } /* * XXX - Note, we are not providing any open * mode flags here (like FNDELAY), so we may * block if there are locks present... this * should be addressed in openat(). */ /* XXX - would it be OK to generate a log record here? */ err = zfs_freesp(zp, vap->va_size, 0, 0, FALSE); if (err) { ZFS_EXIT(zfsvfs); return (err); } } if (mask & (AT_ATIME|AT_MTIME) || ((mask & AT_XVATTR) && (XVA_ISSET_REQ(xvap, XAT_HIDDEN) || XVA_ISSET_REQ(xvap, XAT_READONLY) || XVA_ISSET_REQ(xvap, XAT_ARCHIVE) || XVA_ISSET_REQ(xvap, XAT_CREATETIME) || XVA_ISSET_REQ(xvap, XAT_SYSTEM)))) need_policy = zfs_zaccess(zp, ACE_WRITE_ATTRIBUTES, 0, skipaclchk, cr); if (mask & (AT_UID|AT_GID)) { int idmask = (mask & (AT_UID|AT_GID)); int take_owner; int take_group; /* * NOTE: even if a new mode is being set, * we may clear S_ISUID/S_ISGID bits. */ if (!(mask & AT_MODE)) vap->va_mode = pzp->zp_mode; /* * Take ownership or chgrp to group we are a member of */ take_owner = (mask & AT_UID) && (vap->va_uid == crgetuid(cr)); take_group = (mask & AT_GID) && zfs_groupmember(zfsvfs, vap->va_gid, cr); /* * If both AT_UID and AT_GID are set then take_owner and * take_group must both be set in order to allow taking * ownership. * * Otherwise, send the check through secpolicy_vnode_setattr() * */ if (((idmask == (AT_UID|AT_GID)) && take_owner && take_group) || ((idmask == AT_UID) && take_owner) || ((idmask == AT_GID) && take_group)) { if (zfs_zaccess(zp, ACE_WRITE_OWNER, 0, skipaclchk, cr) == 0) { /* * Remove setuid/setgid for non-privileged users */ secpolicy_setid_clear(vap, vp, cr); trim_mask = (mask & (AT_UID|AT_GID)); } else { need_policy = TRUE; } } else { need_policy = TRUE; } } mutex_enter(&zp->z_lock); oldva.va_mode = pzp->zp_mode; zfs_fuid_map_ids(zp, cr, &oldva.va_uid, &oldva.va_gid); if (mask & AT_XVATTR) { if ((need_policy == FALSE) && (XVA_ISSET_REQ(xvap, XAT_APPENDONLY) && xoap->xoa_appendonly != ((pzp->zp_flags & ZFS_APPENDONLY) != 0)) || (XVA_ISSET_REQ(xvap, XAT_NOUNLINK) && xoap->xoa_nounlink != ((pzp->zp_flags & ZFS_NOUNLINK) != 0)) || (XVA_ISSET_REQ(xvap, XAT_IMMUTABLE) && xoap->xoa_immutable != ((pzp->zp_flags & ZFS_IMMUTABLE) != 0)) || (XVA_ISSET_REQ(xvap, XAT_NODUMP) && xoap->xoa_nodump != ((pzp->zp_flags & ZFS_NODUMP) != 0)) || (XVA_ISSET_REQ(xvap, XAT_AV_MODIFIED) && xoap->xoa_av_modified != ((pzp->zp_flags & ZFS_AV_MODIFIED) != 0)) || ((XVA_ISSET_REQ(xvap, XAT_AV_QUARANTINED) && ((vp->v_type != VREG && xoap->xoa_av_quarantined) || xoap->xoa_av_quarantined != ((pzp->zp_flags & ZFS_AV_QUARANTINED) != 0)))) || (XVA_ISSET_REQ(xvap, XAT_AV_SCANSTAMP)) || (XVA_ISSET_REQ(xvap, XAT_OPAQUE))) { need_policy = TRUE; } } mutex_exit(&zp->z_lock); if (mask & AT_MODE) { if (zfs_zaccess(zp, ACE_WRITE_ACL, 0, skipaclchk, cr) == 0) { err = secpolicy_setid_setsticky_clear(vp, vap, &oldva, cr); if (err) { ZFS_EXIT(zfsvfs); return (err); } trim_mask |= AT_MODE; } else { need_policy = TRUE; } } if (need_policy) { /* * If trim_mask is set then take ownership * has been granted or write_acl is present and user * has the ability to modify mode. In that case remove * UID|GID and or MODE from mask so that * secpolicy_vnode_setattr() doesn't revoke it. */ if (trim_mask) { saved_mask = vap->va_mask; vap->va_mask &= ~trim_mask; if (trim_mask & AT_MODE) { /* * Save the mode, as secpolicy_vnode_setattr() * will overwrite it with ova.va_mode. */ saved_mode = vap->va_mode; } } err = secpolicy_vnode_setattr(cr, vp, vap, &oldva, flags, (int (*)(void *, int, cred_t *))zfs_zaccess_unix, zp); if (err) { ZFS_EXIT(zfsvfs); return (err); } if (trim_mask) { vap->va_mask |= saved_mask; if (trim_mask & AT_MODE) { /* * Recover the mode after * secpolicy_vnode_setattr(). */ vap->va_mode = saved_mode; } } } /* * secpolicy_vnode_setattr, or take ownership may have * changed va_mask */ mask = vap->va_mask; tx = dmu_tx_create(zfsvfs->z_os); dmu_tx_hold_bonus(tx, zp->z_id); if (((mask & AT_UID) && IS_EPHEMERAL(vap->va_uid)) || ((mask & AT_GID) && IS_EPHEMERAL(vap->va_gid))) { if (zfsvfs->z_fuid_obj == 0) { dmu_tx_hold_bonus(tx, DMU_NEW_OBJECT); dmu_tx_hold_write(tx, DMU_NEW_OBJECT, 0, FUID_SIZE_ESTIMATE(zfsvfs)); dmu_tx_hold_zap(tx, MASTER_NODE_OBJ, FALSE, NULL); } else { dmu_tx_hold_bonus(tx, zfsvfs->z_fuid_obj); dmu_tx_hold_write(tx, zfsvfs->z_fuid_obj, 0, FUID_SIZE_ESTIMATE(zfsvfs)); } } if (mask & AT_MODE) { uint64_t pmode = pzp->zp_mode; new_mode = (pmode & S_IFMT) | (vap->va_mode & ~S_IFMT); if (err = zfs_acl_chmod_setattr(zp, &aclp, new_mode)) { dmu_tx_abort(tx); ZFS_EXIT(zfsvfs); return (err); } if (pzp->zp_acl.z_acl_extern_obj) { /* Are we upgrading ACL from old V0 format to new V1 */ if (zfsvfs->z_version <= ZPL_VERSION_FUID && pzp->zp_acl.z_acl_version == ZFS_ACL_VERSION_INITIAL) { dmu_tx_hold_free(tx, pzp->zp_acl.z_acl_extern_obj, 0, DMU_OBJECT_END); dmu_tx_hold_write(tx, DMU_NEW_OBJECT, 0, aclp->z_acl_bytes); } else { dmu_tx_hold_write(tx, pzp->zp_acl.z_acl_extern_obj, 0, aclp->z_acl_bytes); } } else if (aclp->z_acl_bytes > ZFS_ACE_SPACE) { dmu_tx_hold_write(tx, DMU_NEW_OBJECT, 0, aclp->z_acl_bytes); } } if ((mask & (AT_UID | AT_GID)) && pzp->zp_xattr != 0) { err = zfs_zget(zp->z_zfsvfs, pzp->zp_xattr, &attrzp); if (err) { dmu_tx_abort(tx); ZFS_EXIT(zfsvfs); if (aclp) zfs_acl_free(aclp); return (err); } dmu_tx_hold_bonus(tx, attrzp->z_id); } err = dmu_tx_assign(tx, zfsvfs->z_assign); if (err) { if (attrzp) VN_RELE(ZTOV(attrzp)); if (aclp) { zfs_acl_free(aclp); aclp = NULL; } if (err == ERESTART && zfsvfs->z_assign == TXG_NOWAIT) { dmu_tx_wait(tx); dmu_tx_abort(tx); goto top; } dmu_tx_abort(tx); ZFS_EXIT(zfsvfs); return (err); } dmu_buf_will_dirty(zp->z_dbuf, tx); /* * Set each attribute requested. * We group settings according to the locks they need to acquire. * * Note: you cannot set ctime directly, although it will be * updated as a side-effect of calling this function. */ mutex_enter(&zp->z_lock); if (mask & AT_MODE) { mutex_enter(&zp->z_acl_lock); zp->z_phys->zp_mode = new_mode; err = zfs_aclset_common(zp, aclp, cr, &fuidp, tx); ASSERT3U(err, ==, 0); mutex_exit(&zp->z_acl_lock); } if (attrzp) mutex_enter(&attrzp->z_lock); if (mask & AT_UID) { pzp->zp_uid = zfs_fuid_create(zfsvfs, vap->va_uid, cr, ZFS_OWNER, tx, &fuidp); if (attrzp) { attrzp->z_phys->zp_uid = zfs_fuid_create(zfsvfs, vap->va_uid, cr, ZFS_OWNER, tx, &fuidp); } } if (mask & AT_GID) { pzp->zp_gid = zfs_fuid_create(zfsvfs, vap->va_gid, cr, ZFS_GROUP, tx, &fuidp); if (attrzp) attrzp->z_phys->zp_gid = zfs_fuid_create(zfsvfs, vap->va_gid, cr, ZFS_GROUP, tx, &fuidp); } if (aclp) zfs_acl_free(aclp); if (attrzp) mutex_exit(&attrzp->z_lock); if (mask & AT_ATIME) ZFS_TIME_ENCODE(&vap->va_atime, pzp->zp_atime); if (mask & AT_MTIME) ZFS_TIME_ENCODE(&vap->va_mtime, pzp->zp_mtime); /* XXX - shouldn't this be done *before* the ATIME/MTIME checks? */ if (mask & AT_SIZE) zfs_time_stamper_locked(zp, CONTENT_MODIFIED, tx); else if (mask != 0) zfs_time_stamper_locked(zp, STATE_CHANGED, tx); /* * Do this after setting timestamps to prevent timestamp * update from toggling bit */ if (xoap && (mask & AT_XVATTR)) { if (XVA_ISSET_REQ(xvap, XAT_AV_SCANSTAMP)) { size_t len; dmu_object_info_t doi; ASSERT(vp->v_type == VREG); /* Grow the bonus buffer if necessary. */ dmu_object_info_from_db(zp->z_dbuf, &doi); len = sizeof (xoap->xoa_av_scanstamp) + sizeof (znode_phys_t); if (len > doi.doi_bonus_size) VERIFY(dmu_set_bonus(zp->z_dbuf, len, tx) == 0); } zfs_xvattr_set(zp, xvap); } if (mask != 0) zfs_log_setattr(zilog, tx, TX_SETATTR, zp, vap, mask, fuidp); if (fuidp) zfs_fuid_info_free(fuidp); mutex_exit(&zp->z_lock); if (attrzp) VN_RELE(ZTOV(attrzp)); dmu_tx_commit(tx); ZFS_EXIT(zfsvfs); return (err); } typedef struct zfs_zlock { krwlock_t *zl_rwlock; /* lock we acquired */ znode_t *zl_znode; /* znode we held */ struct zfs_zlock *zl_next; /* next in list */ } zfs_zlock_t; /* * Drop locks and release vnodes that were held by zfs_rename_lock(). */ static void zfs_rename_unlock(zfs_zlock_t **zlpp) { zfs_zlock_t *zl; while ((zl = *zlpp) != NULL) { if (zl->zl_znode != NULL) VN_RELE(ZTOV(zl->zl_znode)); rw_exit(zl->zl_rwlock); *zlpp = zl->zl_next; kmem_free(zl, sizeof (*zl)); } } /* * Search back through the directory tree, using the ".." entries. * Lock each directory in the chain to prevent concurrent renames. * Fail any attempt to move a directory into one of its own descendants. * XXX - z_parent_lock can overlap with map or grow locks */ static int zfs_rename_lock(znode_t *szp, znode_t *tdzp, znode_t *sdzp, zfs_zlock_t **zlpp) { zfs_zlock_t *zl; znode_t *zp = tdzp; uint64_t rootid = zp->z_zfsvfs->z_root; uint64_t *oidp = &zp->z_id; krwlock_t *rwlp = &szp->z_parent_lock; krw_t rw = RW_WRITER; /* * First pass write-locks szp and compares to zp->z_id. * Later passes read-lock zp and compare to zp->z_parent. */ do { if (!rw_tryenter(rwlp, rw)) { /* * Another thread is renaming in this path. * Note that if we are a WRITER, we don't have any * parent_locks held yet. */ if (rw == RW_READER && zp->z_id > szp->z_id) { /* * Drop our locks and restart */ zfs_rename_unlock(&zl); *zlpp = NULL; zp = tdzp; oidp = &zp->z_id; rwlp = &szp->z_parent_lock; rw = RW_WRITER; continue; } else { /* * Wait for other thread to drop its locks */ rw_enter(rwlp, rw); } } zl = kmem_alloc(sizeof (*zl), KM_SLEEP); zl->zl_rwlock = rwlp; zl->zl_znode = NULL; zl->zl_next = *zlpp; *zlpp = zl; if (*oidp == szp->z_id) /* We're a descendant of szp */ return (EINVAL); if (*oidp == rootid) /* We've hit the top */ return (0); if (rw == RW_READER) { /* i.e. not the first pass */ int error = zfs_zget(zp->z_zfsvfs, *oidp, &zp); if (error) return (error); zl->zl_znode = zp; } oidp = &zp->z_phys->zp_parent; rwlp = &zp->z_parent_lock; rw = RW_READER; } while (zp->z_id != sdzp->z_id); return (0); } /* * Move an entry from the provided source directory to the target * directory. Change the entry name as indicated. * * IN: sdvp - Source directory containing the "old entry". * snm - Old entry name. * tdvp - Target directory to contain the "new entry". * tnm - New entry name. * cr - credentials of caller. * ct - caller context * flags - case flags * * RETURN: 0 if success * error code if failure * * Timestamps: * sdvp,tdvp - ctime|mtime updated */ /*ARGSUSED*/ static int zfs_rename(vnode_t *sdvp, char *snm, vnode_t *tdvp, char *tnm, cred_t *cr, caller_context_t *ct, int flags) { znode_t *tdzp, *szp, *tzp; znode_t *sdzp = VTOZ(sdvp); zfsvfs_t *zfsvfs = sdzp->z_zfsvfs; zilog_t *zilog; vnode_t *realvp; zfs_dirlock_t *sdl, *tdl; dmu_tx_t *tx; zfs_zlock_t *zl; int cmp, serr, terr; int error = 0; int zflg = 0; ZFS_ENTER(zfsvfs); ZFS_VERIFY_ZP(sdzp); zilog = zfsvfs->z_log; /* * Make sure we have the real vp for the target directory. */ if (VOP_REALVP(tdvp, &realvp, ct) == 0) tdvp = realvp; if (tdvp->v_vfsp != sdvp->v_vfsp) { ZFS_EXIT(zfsvfs); return (EXDEV); } tdzp = VTOZ(tdvp); ZFS_VERIFY_ZP(tdzp); if (zfsvfs->z_utf8 && u8_validate(tnm, strlen(tnm), NULL, U8_VALIDATE_ENTIRE, &error) < 0) { ZFS_EXIT(zfsvfs); return (EILSEQ); } if (flags & FIGNORECASE) zflg |= ZCILOOK; top: szp = NULL; tzp = NULL; zl = NULL; /* * This is to prevent the creation of links into attribute space * by renaming a linked file into/outof an attribute directory. * See the comment in zfs_link() for why this is considered bad. */ if ((tdzp->z_phys->zp_flags & ZFS_XATTR) != (sdzp->z_phys->zp_flags & ZFS_XATTR)) { ZFS_EXIT(zfsvfs); return (EINVAL); } /* * Lock source and target directory entries. To prevent deadlock, * a lock ordering must be defined. We lock the directory with * the smallest object id first, or if it's a tie, the one with * the lexically first name. */ if (sdzp->z_id < tdzp->z_id) { cmp = -1; } else if (sdzp->z_id > tdzp->z_id) { cmp = 1; } else { /* * First compare the two name arguments without * considering any case folding. */ int nofold = (zfsvfs->z_norm & ~U8_TEXTPREP_TOUPPER); cmp = u8_strcmp(snm, tnm, 0, nofold, U8_UNICODE_LATEST, &error); ASSERT(error == 0 || !zfsvfs->z_utf8); if (cmp == 0) { /* * POSIX: "If the old argument and the new argument * both refer to links to the same existing file, * the rename() function shall return successfully * and perform no other action." */ ZFS_EXIT(zfsvfs); return (0); } /* * If the file system is case-folding, then we may * have some more checking to do. A case-folding file * system is either supporting mixed case sensitivity * access or is completely case-insensitive. Note * that the file system is always case preserving. * * In mixed sensitivity mode case sensitive behavior * is the default. FIGNORECASE must be used to * explicitly request case insensitive behavior. * * If the source and target names provided differ only * by case (e.g., a request to rename 'tim' to 'Tim'), * we will treat this as a special case in the * case-insensitive mode: as long as the source name * is an exact match, we will allow this to proceed as * a name-change request. */ if ((zfsvfs->z_case == ZFS_CASE_INSENSITIVE || (zfsvfs->z_case == ZFS_CASE_MIXED && flags & FIGNORECASE)) && u8_strcmp(snm, tnm, 0, zfsvfs->z_norm, U8_UNICODE_LATEST, &error) == 0) { /* * case preserving rename request, require exact * name matches */ zflg |= ZCIEXACT; zflg &= ~ZCILOOK; } } if (cmp < 0) { serr = zfs_dirent_lock(&sdl, sdzp, snm, &szp, ZEXISTS | zflg, NULL, NULL); terr = zfs_dirent_lock(&tdl, tdzp, tnm, &tzp, ZRENAMING | zflg, NULL, NULL); } else { terr = zfs_dirent_lock(&tdl, tdzp, tnm, &tzp, zflg, NULL, NULL); serr = zfs_dirent_lock(&sdl, sdzp, snm, &szp, ZEXISTS | ZRENAMING | zflg, NULL, NULL); } if (serr) { /* * Source entry invalid or not there. */ if (!terr) { zfs_dirent_unlock(tdl); if (tzp) VN_RELE(ZTOV(tzp)); } if (strcmp(snm, ".") == 0 || strcmp(snm, "..") == 0) serr = EINVAL; ZFS_EXIT(zfsvfs); return (serr); } if (terr) { zfs_dirent_unlock(sdl); VN_RELE(ZTOV(szp)); if (strcmp(tnm, "..") == 0) terr = EINVAL; ZFS_EXIT(zfsvfs); return (terr); } /* * Must have write access at the source to remove the old entry * and write access at the target to create the new entry. * Note that if target and source are the same, this can be * done in a single check. */ if (error = zfs_zaccess_rename(sdzp, szp, tdzp, tzp, cr)) goto out; if (ZTOV(szp)->v_type == VDIR) { /* * Check to make sure rename is valid. * Can't do a move like this: /usr/a/b to /usr/a/b/c/d */ if (error = zfs_rename_lock(szp, tdzp, sdzp, &zl)) goto out; } /* * Does target exist? */ if (tzp) { /* * Source and target must be the same type. */ if (ZTOV(szp)->v_type == VDIR) { if (ZTOV(tzp)->v_type != VDIR) { error = ENOTDIR; goto out; } } else { if (ZTOV(tzp)->v_type == VDIR) { error = EISDIR; goto out; } } /* * POSIX dictates that when the source and target * entries refer to the same file object, rename * must do nothing and exit without error. */ if (szp->z_id == tzp->z_id) { error = 0; goto out; } } vnevent_rename_src(ZTOV(szp), sdvp, snm, ct); if (tzp) vnevent_rename_dest(ZTOV(tzp), tdvp, tnm, ct); /* * notify the target directory if it is not the same * as source directory. */ if (tdvp != sdvp) { vnevent_rename_dest_dir(tdvp, ct); } tx = dmu_tx_create(zfsvfs->z_os); dmu_tx_hold_bonus(tx, szp->z_id); /* nlink changes */ dmu_tx_hold_bonus(tx, sdzp->z_id); /* nlink changes */ dmu_tx_hold_zap(tx, sdzp->z_id, FALSE, snm); dmu_tx_hold_zap(tx, tdzp->z_id, TRUE, tnm); if (sdzp != tdzp) dmu_tx_hold_bonus(tx, tdzp->z_id); /* nlink changes */ if (tzp) dmu_tx_hold_bonus(tx, tzp->z_id); /* parent changes */ dmu_tx_hold_zap(tx, zfsvfs->z_unlinkedobj, FALSE, NULL); error = dmu_tx_assign(tx, zfsvfs->z_assign); if (error) { if (zl != NULL) zfs_rename_unlock(&zl); zfs_dirent_unlock(sdl); zfs_dirent_unlock(tdl); VN_RELE(ZTOV(szp)); if (tzp) VN_RELE(ZTOV(tzp)); if (error == ERESTART && zfsvfs->z_assign == TXG_NOWAIT) { dmu_tx_wait(tx); dmu_tx_abort(tx); goto top; } dmu_tx_abort(tx); ZFS_EXIT(zfsvfs); return (error); } if (tzp) /* Attempt to remove the existing target */ error = zfs_link_destroy(tdl, tzp, tx, zflg, NULL); if (error == 0) { error = zfs_link_create(tdl, szp, tx, ZRENAMING); if (error == 0) { szp->z_phys->zp_flags |= ZFS_AV_MODIFIED; error = zfs_link_destroy(sdl, szp, tx, ZRENAMING, NULL); ASSERT(error == 0); zfs_log_rename(zilog, tx, TX_RENAME | (flags & FIGNORECASE ? TX_CI : 0), sdzp, sdl->dl_name, tdzp, tdl->dl_name, szp); /* Update path information for the target vnode */ vn_renamepath(tdvp, ZTOV(szp), tnm, strlen(tnm)); } #ifdef FREEBSD_NAMECACHE if (error == 0) { cache_purge(sdvp); cache_purge(tdvp); } #endif } dmu_tx_commit(tx); out: if (zl != NULL) zfs_rename_unlock(&zl); zfs_dirent_unlock(sdl); zfs_dirent_unlock(tdl); VN_RELE(ZTOV(szp)); if (tzp) VN_RELE(ZTOV(tzp)); ZFS_EXIT(zfsvfs); return (error); } /* * Insert the indicated symbolic reference entry into the directory. * * IN: dvp - Directory to contain new symbolic link. * link - Name for new symlink entry. * vap - Attributes of new entry. * target - Target path of new symlink. * cr - credentials of caller. * ct - caller context * flags - case flags * * RETURN: 0 if success * error code if failure * * Timestamps: * dvp - ctime|mtime updated */ /*ARGSUSED*/ static int zfs_symlink(vnode_t *dvp, vnode_t **vpp, char *name, vattr_t *vap, char *link, cred_t *cr, kthread_t *td) { znode_t *zp, *dzp = VTOZ(dvp); zfs_dirlock_t *dl; dmu_tx_t *tx; zfsvfs_t *zfsvfs = dzp->z_zfsvfs; zilog_t *zilog; int len = strlen(link); int error; int zflg = ZNEW; zfs_fuid_info_t *fuidp = NULL; int flags = 0; ASSERT(vap->va_type == VLNK); ZFS_ENTER(zfsvfs); ZFS_VERIFY_ZP(dzp); zilog = zfsvfs->z_log; if (zfsvfs->z_utf8 && u8_validate(name, strlen(name), NULL, U8_VALIDATE_ENTIRE, &error) < 0) { ZFS_EXIT(zfsvfs); return (EILSEQ); } if (flags & FIGNORECASE) zflg |= ZCILOOK; top: if (error = zfs_zaccess(dzp, ACE_ADD_FILE, 0, B_FALSE, cr)) { ZFS_EXIT(zfsvfs); return (error); } if (len > MAXPATHLEN) { ZFS_EXIT(zfsvfs); return (ENAMETOOLONG); } /* * Attempt to lock directory; fail if entry already exists. */ error = zfs_dirent_lock(&dl, dzp, name, &zp, zflg, NULL, NULL); if (error) { ZFS_EXIT(zfsvfs); return (error); } tx = dmu_tx_create(zfsvfs->z_os); dmu_tx_hold_write(tx, DMU_NEW_OBJECT, 0, MAX(1, len)); dmu_tx_hold_bonus(tx, dzp->z_id); dmu_tx_hold_zap(tx, dzp->z_id, TRUE, name); if (dzp->z_phys->zp_flags & ZFS_INHERIT_ACE) dmu_tx_hold_write(tx, DMU_NEW_OBJECT, 0, SPA_MAXBLOCKSIZE); if (IS_EPHEMERAL(crgetuid(cr)) || IS_EPHEMERAL(crgetgid(cr))) { if (zfsvfs->z_fuid_obj == 0) { dmu_tx_hold_bonus(tx, DMU_NEW_OBJECT); dmu_tx_hold_write(tx, DMU_NEW_OBJECT, 0, FUID_SIZE_ESTIMATE(zfsvfs)); dmu_tx_hold_zap(tx, MASTER_NODE_OBJ, FALSE, NULL); } else { dmu_tx_hold_bonus(tx, zfsvfs->z_fuid_obj); dmu_tx_hold_write(tx, zfsvfs->z_fuid_obj, 0, FUID_SIZE_ESTIMATE(zfsvfs)); } } error = dmu_tx_assign(tx, zfsvfs->z_assign); if (error) { zfs_dirent_unlock(dl); if (error == ERESTART && zfsvfs->z_assign == TXG_NOWAIT) { dmu_tx_wait(tx); dmu_tx_abort(tx); goto top; } dmu_tx_abort(tx); ZFS_EXIT(zfsvfs); return (error); } dmu_buf_will_dirty(dzp->z_dbuf, tx); /* * Create a new object for the symlink. * Put the link content into bonus buffer if it will fit; * otherwise, store it just like any other file data. */ if (sizeof (znode_phys_t) + len <= dmu_bonus_max()) { zfs_mknode(dzp, vap, tx, cr, 0, &zp, len, NULL, &fuidp); if (len != 0) bcopy(link, zp->z_phys + 1, len); } else { dmu_buf_t *dbp; zfs_mknode(dzp, vap, tx, cr, 0, &zp, 0, NULL, &fuidp); /* * Nothing can access the znode yet so no locking needed * for growing the znode's blocksize. */ zfs_grow_blocksize(zp, len, tx); VERIFY(0 == dmu_buf_hold(zfsvfs->z_os, zp->z_id, 0, FTAG, &dbp)); dmu_buf_will_dirty(dbp, tx); ASSERT3U(len, <=, dbp->db_size); bcopy(link, dbp->db_data, len); dmu_buf_rele(dbp, FTAG); } zp->z_phys->zp_size = len; /* * Insert the new object into the directory. */ (void) zfs_link_create(dl, zp, tx, ZNEW); out: if (error == 0) { uint64_t txtype = TX_SYMLINK; if (flags & FIGNORECASE) txtype |= TX_CI; zfs_log_symlink(zilog, tx, txtype, dzp, zp, name, link); *vpp = ZTOV(zp); } if (fuidp) zfs_fuid_info_free(fuidp); dmu_tx_commit(tx); zfs_dirent_unlock(dl); ZFS_EXIT(zfsvfs); return (error); } /* * Return, in the buffer contained in the provided uio structure, * the symbolic path referred to by vp. * * IN: vp - vnode of symbolic link. * uoip - structure to contain the link path. * cr - credentials of caller. * ct - caller context * * OUT: uio - structure to contain the link path. * * RETURN: 0 if success * error code if failure * * Timestamps: * vp - atime updated */ /* ARGSUSED */ static int zfs_readlink(vnode_t *vp, uio_t *uio, cred_t *cr, caller_context_t *ct) { znode_t *zp = VTOZ(vp); zfsvfs_t *zfsvfs = zp->z_zfsvfs; size_t bufsz; int error; ZFS_ENTER(zfsvfs); ZFS_VERIFY_ZP(zp); bufsz = (size_t)zp->z_phys->zp_size; if (bufsz + sizeof (znode_phys_t) <= zp->z_dbuf->db_size) { error = uiomove(zp->z_phys + 1, MIN((size_t)bufsz, uio->uio_resid), UIO_READ, uio); } else { dmu_buf_t *dbp; error = dmu_buf_hold(zfsvfs->z_os, zp->z_id, 0, FTAG, &dbp); if (error) { ZFS_EXIT(zfsvfs); return (error); } error = uiomove(dbp->db_data, MIN((size_t)bufsz, uio->uio_resid), UIO_READ, uio); dmu_buf_rele(dbp, FTAG); } ZFS_ACCESSTIME_STAMP(zfsvfs, zp); ZFS_EXIT(zfsvfs); return (error); } /* * Insert a new entry into directory tdvp referencing svp. * * IN: tdvp - Directory to contain new entry. * svp - vnode of new entry. * name - name of new entry. * cr - credentials of caller. * ct - caller context * * RETURN: 0 if success * error code if failure * * Timestamps: * tdvp - ctime|mtime updated * svp - ctime updated */ /* ARGSUSED */ static int zfs_link(vnode_t *tdvp, vnode_t *svp, char *name, cred_t *cr, caller_context_t *ct, int flags) { znode_t *dzp = VTOZ(tdvp); znode_t *tzp, *szp; zfsvfs_t *zfsvfs = dzp->z_zfsvfs; zilog_t *zilog; zfs_dirlock_t *dl; dmu_tx_t *tx; vnode_t *realvp; int error; int zf = ZNEW; uid_t owner; ASSERT(tdvp->v_type == VDIR); ZFS_ENTER(zfsvfs); ZFS_VERIFY_ZP(dzp); zilog = zfsvfs->z_log; if (VOP_REALVP(svp, &realvp, ct) == 0) svp = realvp; if (svp->v_vfsp != tdvp->v_vfsp) { ZFS_EXIT(zfsvfs); return (EXDEV); } szp = VTOZ(svp); ZFS_VERIFY_ZP(szp); if (zfsvfs->z_utf8 && u8_validate(name, strlen(name), NULL, U8_VALIDATE_ENTIRE, &error) < 0) { ZFS_EXIT(zfsvfs); return (EILSEQ); } if (flags & FIGNORECASE) zf |= ZCILOOK; top: /* * We do not support links between attributes and non-attributes * because of the potential security risk of creating links * into "normal" file space in order to circumvent restrictions * imposed in attribute space. */ if ((szp->z_phys->zp_flags & ZFS_XATTR) != (dzp->z_phys->zp_flags & ZFS_XATTR)) { ZFS_EXIT(zfsvfs); return (EINVAL); } /* * POSIX dictates that we return EPERM here. * Better choices include ENOTSUP or EISDIR. */ if (svp->v_type == VDIR) { ZFS_EXIT(zfsvfs); return (EPERM); } owner = zfs_fuid_map_id(zfsvfs, szp->z_phys->zp_uid, cr, ZFS_OWNER); if (owner != crgetuid(cr) && secpolicy_basic_link(svp, cr) != 0) { ZFS_EXIT(zfsvfs); return (EPERM); } if (error = zfs_zaccess(dzp, ACE_ADD_FILE, 0, B_FALSE, cr)) { ZFS_EXIT(zfsvfs); return (error); } /* * Attempt to lock directory; fail if entry already exists. */ error = zfs_dirent_lock(&dl, dzp, name, &tzp, zf, NULL, NULL); if (error) { ZFS_EXIT(zfsvfs); return (error); } tx = dmu_tx_create(zfsvfs->z_os); dmu_tx_hold_bonus(tx, szp->z_id); dmu_tx_hold_zap(tx, dzp->z_id, TRUE, name); error = dmu_tx_assign(tx, zfsvfs->z_assign); if (error) { zfs_dirent_unlock(dl); if (error == ERESTART && zfsvfs->z_assign == TXG_NOWAIT) { dmu_tx_wait(tx); dmu_tx_abort(tx); goto top; } dmu_tx_abort(tx); ZFS_EXIT(zfsvfs); return (error); } error = zfs_link_create(dl, szp, tx, 0); if (error == 0) { uint64_t txtype = TX_LINK; if (flags & FIGNORECASE) txtype |= TX_CI; zfs_log_link(zilog, tx, txtype, dzp, szp, name); } dmu_tx_commit(tx); zfs_dirent_unlock(dl); if (error == 0) { vnevent_link(svp, ct); } ZFS_EXIT(zfsvfs); return (error); } /*ARGSUSED*/ void zfs_inactive(vnode_t *vp, cred_t *cr, caller_context_t *ct) { znode_t *zp = VTOZ(vp); zfsvfs_t *zfsvfs = zp->z_zfsvfs; int error; rw_enter(&zfsvfs->z_teardown_inactive_lock, RW_READER); if (zp->z_dbuf == NULL) { /* * The fs has been unmounted, or we did a * suspend/resume and this file no longer exists. */ VI_LOCK(vp); vp->v_count = 0; /* count arrives as 1 */ VI_UNLOCK(vp); vrecycle(vp, curthread); rw_exit(&zfsvfs->z_teardown_inactive_lock); return; } if (zp->z_atime_dirty && zp->z_unlinked == 0) { dmu_tx_t *tx = dmu_tx_create(zfsvfs->z_os); dmu_tx_hold_bonus(tx, zp->z_id); error = dmu_tx_assign(tx, TXG_WAIT); if (error) { dmu_tx_abort(tx); } else { dmu_buf_will_dirty(zp->z_dbuf, tx); mutex_enter(&zp->z_lock); zp->z_atime_dirty = 0; mutex_exit(&zp->z_lock); dmu_tx_commit(tx); } } zfs_zinactive(zp); rw_exit(&zfsvfs->z_teardown_inactive_lock); } CTASSERT(sizeof(struct zfid_short) <= sizeof(struct fid)); CTASSERT(sizeof(struct zfid_long) <= sizeof(struct fid)); /*ARGSUSED*/ static int zfs_fid(vnode_t *vp, fid_t *fidp, caller_context_t *ct) { znode_t *zp = VTOZ(vp); zfsvfs_t *zfsvfs = zp->z_zfsvfs; uint32_t gen; uint64_t object = zp->z_id; zfid_short_t *zfid; int size, i; ZFS_ENTER(zfsvfs); ZFS_VERIFY_ZP(zp); gen = (uint32_t)zp->z_gen; size = (zfsvfs->z_parent != zfsvfs) ? LONG_FID_LEN : SHORT_FID_LEN; fidp->fid_len = size; zfid = (zfid_short_t *)fidp; zfid->zf_len = size; for (i = 0; i < sizeof (zfid->zf_object); i++) zfid->zf_object[i] = (uint8_t)(object >> (8 * i)); /* Must have a non-zero generation number to distinguish from .zfs */ if (gen == 0) gen = 1; for (i = 0; i < sizeof (zfid->zf_gen); i++) zfid->zf_gen[i] = (uint8_t)(gen >> (8 * i)); if (size == LONG_FID_LEN) { uint64_t objsetid = dmu_objset_id(zfsvfs->z_os); zfid_long_t *zlfid; zlfid = (zfid_long_t *)fidp; for (i = 0; i < sizeof (zlfid->zf_setid); i++) zlfid->zf_setid[i] = (uint8_t)(objsetid >> (8 * i)); /* XXX - this should be the generation number for the objset */ for (i = 0; i < sizeof (zlfid->zf_setgen); i++) zlfid->zf_setgen[i] = 0; } ZFS_EXIT(zfsvfs); return (0); } static int zfs_pathconf(vnode_t *vp, int cmd, ulong_t *valp, cred_t *cr, caller_context_t *ct) { znode_t *zp, *xzp; zfsvfs_t *zfsvfs; zfs_dirlock_t *dl; int error; switch (cmd) { case _PC_LINK_MAX: *valp = INT_MAX; return (0); case _PC_FILESIZEBITS: *valp = 64; return (0); #if 0 case _PC_XATTR_EXISTS: zp = VTOZ(vp); zfsvfs = zp->z_zfsvfs; ZFS_ENTER(zfsvfs); ZFS_VERIFY_ZP(zp); *valp = 0; error = zfs_dirent_lock(&dl, zp, "", &xzp, ZXATTR | ZEXISTS | ZSHARED, NULL, NULL); if (error == 0) { zfs_dirent_unlock(dl); if (!zfs_dirempty(xzp)) *valp = 1; VN_RELE(ZTOV(xzp)); } else if (error == ENOENT) { /* * If there aren't extended attributes, it's the * same as having zero of them. */ error = 0; } ZFS_EXIT(zfsvfs); return (error); #endif case _PC_ACL_EXTENDED: *valp = 0; return (0); case _PC_ACL_NFS4: *valp = 1; return (0); case _PC_ACL_PATH_MAX: *valp = ACL_MAX_ENTRIES; return (0); case _PC_MIN_HOLE_SIZE: *valp = (int)SPA_MINBLOCKSIZE; return (0); default: return (EOPNOTSUPP); } } /*ARGSUSED*/ static int zfs_getsecattr(vnode_t *vp, vsecattr_t *vsecp, int flag, cred_t *cr, caller_context_t *ct) { znode_t *zp = VTOZ(vp); zfsvfs_t *zfsvfs = zp->z_zfsvfs; int error; boolean_t skipaclchk = (flag & ATTR_NOACLCHECK) ? B_TRUE : B_FALSE; ZFS_ENTER(zfsvfs); ZFS_VERIFY_ZP(zp); error = zfs_getacl(zp, vsecp, skipaclchk, cr); ZFS_EXIT(zfsvfs); return (error); } /*ARGSUSED*/ static int zfs_setsecattr(vnode_t *vp, vsecattr_t *vsecp, int flag, cred_t *cr, caller_context_t *ct) { znode_t *zp = VTOZ(vp); zfsvfs_t *zfsvfs = zp->z_zfsvfs; int error; boolean_t skipaclchk = (flag & ATTR_NOACLCHECK) ? B_TRUE : B_FALSE; ZFS_ENTER(zfsvfs); ZFS_VERIFY_ZP(zp); error = zfs_setacl(zp, vsecp, skipaclchk, cr); ZFS_EXIT(zfsvfs); return (error); } static int zfs_freebsd_open(ap) struct vop_open_args /* { struct vnode *a_vp; int a_mode; struct ucred *a_cred; struct thread *a_td; } */ *ap; { vnode_t *vp = ap->a_vp; znode_t *zp = VTOZ(vp); int error; error = zfs_open(&vp, ap->a_mode, ap->a_cred, NULL); if (error == 0) vnode_create_vobject(vp, zp->z_phys->zp_size, ap->a_td); return (error); } static int zfs_freebsd_close(ap) struct vop_close_args /* { struct vnode *a_vp; int a_fflag; struct ucred *a_cred; struct thread *a_td; } */ *ap; { return (zfs_close(ap->a_vp, ap->a_fflag, 0, 0, ap->a_cred, NULL)); } static int zfs_freebsd_ioctl(ap) struct vop_ioctl_args /* { struct vnode *a_vp; u_long a_command; caddr_t a_data; int a_fflag; struct ucred *cred; struct thread *td; } */ *ap; { return (zfs_ioctl(ap->a_vp, ap->a_command, (intptr_t)ap->a_data, ap->a_fflag, ap->a_cred, NULL, NULL)); } static int zfs_freebsd_read(ap) struct vop_read_args /* { struct vnode *a_vp; struct uio *a_uio; int a_ioflag; struct ucred *a_cred; } */ *ap; { return (zfs_read(ap->a_vp, ap->a_uio, ap->a_ioflag, ap->a_cred, NULL)); } static int zfs_freebsd_write(ap) struct vop_write_args /* { struct vnode *a_vp; struct uio *a_uio; int a_ioflag; struct ucred *a_cred; } */ *ap; { return (zfs_write(ap->a_vp, ap->a_uio, ap->a_ioflag, ap->a_cred, NULL)); } static int zfs_freebsd_access(ap) struct vop_access_args /* { struct vnode *a_vp; accmode_t a_accmode; struct ucred *a_cred; struct thread *a_td; } */ *ap; { accmode_t accmode; int error = 0; /* * ZFS itself only knowns about VREAD, VWRITE, VEXEC and VAPPEND, */ accmode = ap->a_accmode & (VREAD|VWRITE|VEXEC|VAPPEND); if (accmode != 0) error = zfs_access(ap->a_vp, accmode, 0, ap->a_cred, NULL); /* * VADMIN has to be handled by vaccess(). */ if (error == 0) { accmode = ap->a_accmode & ~(VREAD|VWRITE|VEXEC|VAPPEND); if (accmode != 0) { vnode_t *vp = ap->a_vp; znode_t *zp = VTOZ(vp); znode_phys_t *zphys = zp->z_phys; error = vaccess(vp->v_type, zphys->zp_mode, zphys->zp_uid, zphys->zp_gid, accmode, ap->a_cred, NULL); } } return (error); } static int zfs_freebsd_lookup(ap) struct vop_lookup_args /* { struct vnode *a_dvp; struct vnode **a_vpp; struct componentname *a_cnp; } */ *ap; { struct componentname *cnp = ap->a_cnp; char nm[NAME_MAX + 1]; ASSERT(cnp->cn_namelen < sizeof(nm)); strlcpy(nm, cnp->cn_nameptr, MIN(cnp->cn_namelen + 1, sizeof(nm))); return (zfs_lookup(ap->a_dvp, nm, ap->a_vpp, cnp, cnp->cn_nameiop, cnp->cn_cred, cnp->cn_thread, 0)); } static int zfs_freebsd_create(ap) struct vop_create_args /* { struct vnode *a_dvp; struct vnode **a_vpp; struct componentname *a_cnp; struct vattr *a_vap; } */ *ap; { struct componentname *cnp = ap->a_cnp; vattr_t *vap = ap->a_vap; int mode; ASSERT(cnp->cn_flags & SAVENAME); vattr_init_mask(vap); mode = vap->va_mode & ALLPERMS; return (zfs_create(ap->a_dvp, cnp->cn_nameptr, vap, !EXCL, mode, ap->a_vpp, cnp->cn_cred, cnp->cn_thread)); } static int zfs_freebsd_remove(ap) struct vop_remove_args /* { struct vnode *a_dvp; struct vnode *a_vp; struct componentname *a_cnp; } */ *ap; { ASSERT(ap->a_cnp->cn_flags & SAVENAME); return (zfs_remove(ap->a_dvp, ap->a_cnp->cn_nameptr, ap->a_cnp->cn_cred, NULL, 0)); } static int zfs_freebsd_mkdir(ap) struct vop_mkdir_args /* { struct vnode *a_dvp; struct vnode **a_vpp; struct componentname *a_cnp; struct vattr *a_vap; } */ *ap; { vattr_t *vap = ap->a_vap; ASSERT(ap->a_cnp->cn_flags & SAVENAME); vattr_init_mask(vap); return (zfs_mkdir(ap->a_dvp, ap->a_cnp->cn_nameptr, vap, ap->a_vpp, ap->a_cnp->cn_cred, NULL, 0, NULL)); } static int zfs_freebsd_rmdir(ap) struct vop_rmdir_args /* { struct vnode *a_dvp; struct vnode *a_vp; struct componentname *a_cnp; } */ *ap; { struct componentname *cnp = ap->a_cnp; ASSERT(cnp->cn_flags & SAVENAME); return (zfs_rmdir(ap->a_dvp, cnp->cn_nameptr, NULL, cnp->cn_cred, NULL, 0)); } static int zfs_freebsd_readdir(ap) struct vop_readdir_args /* { struct vnode *a_vp; struct uio *a_uio; struct ucred *a_cred; int *a_eofflag; int *a_ncookies; u_long **a_cookies; } */ *ap; { return (zfs_readdir(ap->a_vp, ap->a_uio, ap->a_cred, ap->a_eofflag, ap->a_ncookies, ap->a_cookies)); } static int zfs_freebsd_fsync(ap) struct vop_fsync_args /* { struct vnode *a_vp; int a_waitfor; struct thread *a_td; } */ *ap; { vop_stdfsync(ap); return (zfs_fsync(ap->a_vp, 0, ap->a_td->td_ucred, NULL)); } static int zfs_freebsd_getattr(ap) struct vop_getattr_args /* { struct vnode *a_vp; struct vattr *a_vap; struct ucred *a_cred; struct thread *a_td; } */ *ap; { vattr_t *vap = ap->a_vap; xvattr_t xvap; u_long fflags = 0; int error; xva_init(&xvap); xvap.xva_vattr = *vap; xvap.xva_vattr.va_mask |= AT_XVATTR; /* Convert chflags into ZFS-type flags. */ /* XXX: what about SF_SETTABLE?. */ XVA_SET_REQ(&xvap, XAT_IMMUTABLE); XVA_SET_REQ(&xvap, XAT_APPENDONLY); XVA_SET_REQ(&xvap, XAT_NOUNLINK); XVA_SET_REQ(&xvap, XAT_NODUMP); error = zfs_getattr(ap->a_vp, (vattr_t *)&xvap, 0, ap->a_cred, NULL); if (error != 0) return (error); /* Convert ZFS xattr into chflags. */ #define FLAG_CHECK(fflag, xflag, xfield) do { \ if (XVA_ISSET_RTN(&xvap, (xflag)) && (xfield) != 0) \ fflags |= (fflag); \ } while (0) FLAG_CHECK(SF_IMMUTABLE, XAT_IMMUTABLE, xvap.xva_xoptattrs.xoa_immutable); FLAG_CHECK(SF_APPEND, XAT_APPENDONLY, xvap.xva_xoptattrs.xoa_appendonly); FLAG_CHECK(SF_NOUNLINK, XAT_NOUNLINK, xvap.xva_xoptattrs.xoa_nounlink); FLAG_CHECK(UF_NODUMP, XAT_NODUMP, xvap.xva_xoptattrs.xoa_nodump); #undef FLAG_CHECK *vap = xvap.xva_vattr; vap->va_flags = fflags; return (0); } static int zfs_freebsd_setattr(ap) struct vop_setattr_args /* { struct vnode *a_vp; struct vattr *a_vap; struct ucred *a_cred; struct thread *a_td; } */ *ap; { vnode_t *vp = ap->a_vp; vattr_t *vap = ap->a_vap; cred_t *cred = ap->a_cred; xvattr_t xvap; u_long fflags; uint64_t zflags; vattr_init_mask(vap); vap->va_mask &= ~AT_NOSET; xva_init(&xvap); xvap.xva_vattr = *vap; zflags = VTOZ(vp)->z_phys->zp_flags; if (vap->va_flags != VNOVAL) { zfsvfs_t *zfsvfs = VTOZ(vp)->z_zfsvfs; int error; if (zfsvfs->z_use_fuids == B_FALSE) return (EOPNOTSUPP); fflags = vap->va_flags; if ((fflags & ~(SF_IMMUTABLE|SF_APPEND|SF_NOUNLINK|UF_NODUMP)) != 0) return (EOPNOTSUPP); /* * Unprivileged processes are not permitted to unset system * flags, or modify flags if any system flags are set. * Privileged non-jail processes may not modify system flags * if securelevel > 0 and any existing system flags are set. * Privileged jail processes behave like privileged non-jail * processes if the security.jail.chflags_allowed sysctl is * is non-zero; otherwise, they behave like unprivileged * processes. */ if (secpolicy_fs_owner(vp->v_mount, cred) == 0 || priv_check_cred(cred, PRIV_VFS_SYSFLAGS, 0) == 0) { if (zflags & (ZFS_IMMUTABLE | ZFS_APPENDONLY | ZFS_NOUNLINK)) { error = securelevel_gt(cred, 0); if (error != 0) return (error); } } else { /* * Callers may only modify the file flags on objects they * have VADMIN rights for. */ if ((error = VOP_ACCESS(vp, VADMIN, cred, curthread)) != 0) return (error); if (zflags & (ZFS_IMMUTABLE | ZFS_APPENDONLY | ZFS_NOUNLINK)) { return (EPERM); } if (fflags & (SF_IMMUTABLE | SF_APPEND | SF_NOUNLINK)) { return (EPERM); } } #define FLAG_CHANGE(fflag, zflag, xflag, xfield) do { \ if (((fflags & (fflag)) && !(zflags & (zflag))) || \ ((zflags & (zflag)) && !(fflags & (fflag)))) { \ XVA_SET_REQ(&xvap, (xflag)); \ (xfield) = ((fflags & (fflag)) != 0); \ } \ } while (0) /* Convert chflags into ZFS-type flags. */ /* XXX: what about SF_SETTABLE?. */ FLAG_CHANGE(SF_IMMUTABLE, ZFS_IMMUTABLE, XAT_IMMUTABLE, xvap.xva_xoptattrs.xoa_immutable); FLAG_CHANGE(SF_APPEND, ZFS_APPENDONLY, XAT_APPENDONLY, xvap.xva_xoptattrs.xoa_appendonly); FLAG_CHANGE(SF_NOUNLINK, ZFS_NOUNLINK, XAT_NOUNLINK, xvap.xva_xoptattrs.xoa_nounlink); FLAG_CHANGE(UF_NODUMP, ZFS_NODUMP, XAT_NODUMP, xvap.xva_xoptattrs.xoa_nodump); #undef FLAG_CHANGE } return (zfs_setattr(vp, (vattr_t *)&xvap, 0, cred, NULL)); } static int zfs_freebsd_rename(ap) struct vop_rename_args /* { struct vnode *a_fdvp; struct vnode *a_fvp; struct componentname *a_fcnp; struct vnode *a_tdvp; struct vnode *a_tvp; struct componentname *a_tcnp; } */ *ap; { vnode_t *fdvp = ap->a_fdvp; vnode_t *fvp = ap->a_fvp; vnode_t *tdvp = ap->a_tdvp; vnode_t *tvp = ap->a_tvp; int error; ASSERT(ap->a_fcnp->cn_flags & (SAVENAME|SAVESTART)); ASSERT(ap->a_tcnp->cn_flags & (SAVENAME|SAVESTART)); error = zfs_rename(fdvp, ap->a_fcnp->cn_nameptr, tdvp, ap->a_tcnp->cn_nameptr, ap->a_fcnp->cn_cred, NULL, 0); if (tdvp == tvp) VN_RELE(tdvp); else VN_URELE(tdvp); if (tvp) VN_URELE(tvp); VN_RELE(fdvp); VN_RELE(fvp); return (error); } static int zfs_freebsd_symlink(ap) struct vop_symlink_args /* { struct vnode *a_dvp; struct vnode **a_vpp; struct componentname *a_cnp; struct vattr *a_vap; char *a_target; } */ *ap; { struct componentname *cnp = ap->a_cnp; vattr_t *vap = ap->a_vap; ASSERT(cnp->cn_flags & SAVENAME); vap->va_type = VLNK; /* FreeBSD: Syscall only sets va_mode. */ vattr_init_mask(vap); return (zfs_symlink(ap->a_dvp, ap->a_vpp, cnp->cn_nameptr, vap, ap->a_target, cnp->cn_cred, cnp->cn_thread)); } static int zfs_freebsd_readlink(ap) struct vop_readlink_args /* { struct vnode *a_vp; struct uio *a_uio; struct ucred *a_cred; } */ *ap; { return (zfs_readlink(ap->a_vp, ap->a_uio, ap->a_cred, NULL)); } static int zfs_freebsd_link(ap) struct vop_link_args /* { struct vnode *a_tdvp; struct vnode *a_vp; struct componentname *a_cnp; } */ *ap; { struct componentname *cnp = ap->a_cnp; ASSERT(cnp->cn_flags & SAVENAME); return (zfs_link(ap->a_tdvp, ap->a_vp, cnp->cn_nameptr, cnp->cn_cred, NULL, 0)); } static int zfs_freebsd_inactive(ap) struct vop_inactive_args /* { struct vnode *a_vp; struct thread *a_td; } */ *ap; { vnode_t *vp = ap->a_vp; zfs_inactive(vp, ap->a_td->td_ucred, NULL); return (0); } static void zfs_reclaim_complete(void *arg, int pending) { znode_t *zp = arg; zfsvfs_t *zfsvfs = zp->z_zfsvfs; rw_enter(&zfsvfs->z_teardown_inactive_lock, RW_READER); if (zp->z_dbuf != NULL) { ZFS_OBJ_HOLD_ENTER(zfsvfs, zp->z_id); zfs_znode_dmu_fini(zp); ZFS_OBJ_HOLD_EXIT(zfsvfs, zp->z_id); } zfs_znode_free(zp); rw_exit(&zfsvfs->z_teardown_inactive_lock); /* * If the file system is being unmounted, there is a process waiting * for us, wake it up. */ if (zfsvfs->z_unmounted) wakeup_one(zfsvfs); } static int zfs_freebsd_reclaim(ap) struct vop_reclaim_args /* { struct vnode *a_vp; struct thread *a_td; } */ *ap; { vnode_t *vp = ap->a_vp; znode_t *zp = VTOZ(vp); zfsvfs_t *zfsvfs = zp->z_zfsvfs; rw_enter(&zfsvfs->z_teardown_inactive_lock, RW_READER); ASSERT(zp != NULL); /* * Destroy the vm object and flush associated pages. */ vnode_destroy_vobject(vp); mutex_enter(&zp->z_lock); ASSERT(zp->z_phys != NULL); zp->z_vnode = NULL; mutex_exit(&zp->z_lock); if (zp->z_unlinked) ; /* Do nothing. */ else if (zp->z_dbuf == NULL) zfs_znode_free(zp); else /* if (!zp->z_unlinked && zp->z_dbuf != NULL) */ { int locked; locked = MUTEX_HELD(ZFS_OBJ_MUTEX(zfsvfs, zp->z_id)) ? 2 : ZFS_OBJ_HOLD_TRYENTER(zfsvfs, zp->z_id); if (locked == 0) { /* * Lock can't be obtained due to deadlock possibility, * so defer znode destruction. */ TASK_INIT(&zp->z_task, 0, zfs_reclaim_complete, zp); taskqueue_enqueue(taskqueue_thread, &zp->z_task); } else { zfs_znode_dmu_fini(zp); if (locked == 1) ZFS_OBJ_HOLD_EXIT(zfsvfs, zp->z_id); zfs_znode_free(zp); } } VI_LOCK(vp); vp->v_data = NULL; ASSERT(vp->v_holdcnt >= 1); VI_UNLOCK(vp); rw_exit(&zfsvfs->z_teardown_inactive_lock); return (0); } static int zfs_freebsd_fid(ap) struct vop_fid_args /* { struct vnode *a_vp; struct fid *a_fid; } */ *ap; { return (zfs_fid(ap->a_vp, (void *)ap->a_fid, NULL)); } static int zfs_freebsd_pathconf(ap) struct vop_pathconf_args /* { struct vnode *a_vp; int a_name; register_t *a_retval; } */ *ap; { ulong_t val; int error; error = zfs_pathconf(ap->a_vp, ap->a_name, &val, curthread->td_ucred, NULL); if (error == 0) *ap->a_retval = val; else if (error == EOPNOTSUPP) error = vop_stdpathconf(ap); return (error); } static int zfs_freebsd_fifo_pathconf(ap) struct vop_pathconf_args /* { struct vnode *a_vp; int a_name; register_t *a_retval; } */ *ap; { switch (ap->a_name) { case _PC_ACL_EXTENDED: case _PC_ACL_NFS4: case _PC_ACL_PATH_MAX: case _PC_MAC_PRESENT: return (zfs_freebsd_pathconf(ap)); default: return (fifo_specops.vop_pathconf(ap)); } } /* * FreeBSD's extended attributes namespace defines file name prefix for ZFS' * extended attribute name: * * NAMESPACE PREFIX * system freebsd:system: * user (none, can be used to access ZFS fsattr(5) attributes * created on Solaris) */ static int zfs_create_attrname(int attrnamespace, const char *name, char *attrname, size_t size) { const char *namespace, *prefix, *suffix; /* We don't allow '/' character in attribute name. */ if (strchr(name, '/') != NULL) return (EINVAL); /* We don't allow attribute names that start with "freebsd:" string. */ if (strncmp(name, "freebsd:", 8) == 0) return (EINVAL); bzero(attrname, size); switch (attrnamespace) { case EXTATTR_NAMESPACE_USER: #if 0 prefix = "freebsd:"; namespace = EXTATTR_NAMESPACE_USER_STRING; suffix = ":"; #else /* * This is the default namespace by which we can access all * attributes created on Solaris. */ prefix = namespace = suffix = ""; #endif break; case EXTATTR_NAMESPACE_SYSTEM: prefix = "freebsd:"; namespace = EXTATTR_NAMESPACE_SYSTEM_STRING; suffix = ":"; break; case EXTATTR_NAMESPACE_EMPTY: default: return (EINVAL); } if (snprintf(attrname, size, "%s%s%s%s", prefix, namespace, suffix, name) >= size) { return (ENAMETOOLONG); } return (0); } /* * Vnode operating to retrieve a named extended attribute. */ static int zfs_getextattr(struct vop_getextattr_args *ap) /* vop_getextattr { IN struct vnode *a_vp; IN int a_attrnamespace; IN const char *a_name; INOUT struct uio *a_uio; OUT size_t *a_size; IN struct ucred *a_cred; IN struct thread *a_td; }; */ { zfsvfs_t *zfsvfs = VTOZ(ap->a_vp)->z_zfsvfs; struct thread *td = ap->a_td; struct nameidata nd; char attrname[255]; struct vattr va; vnode_t *xvp = NULL, *vp; int error, flags; error = extattr_check_cred(ap->a_vp, ap->a_attrnamespace, ap->a_cred, ap->a_td, VREAD); if (error != 0) return (error); error = zfs_create_attrname(ap->a_attrnamespace, ap->a_name, attrname, sizeof(attrname)); if (error != 0) return (error); ZFS_ENTER(zfsvfs); error = zfs_lookup(ap->a_vp, NULL, &xvp, NULL, 0, ap->a_cred, td, LOOKUP_XATTR); if (error != 0) { ZFS_EXIT(zfsvfs); return (error); } flags = FREAD; NDINIT_ATVP(&nd, LOOKUP, NOFOLLOW | MPSAFE, UIO_SYSSPACE, attrname, xvp, td); error = vn_open_cred(&nd, &flags, 0, 0, ap->a_cred, NULL); vp = nd.ni_vp; NDFREE(&nd, NDF_ONLY_PNBUF); if (error != 0) { ZFS_EXIT(zfsvfs); if (error == ENOENT) error = ENOATTR; return (error); } if (ap->a_size != NULL) { error = VOP_GETATTR(vp, &va, ap->a_cred); if (error == 0) *ap->a_size = (size_t)va.va_size; } else if (ap->a_uio != NULL) error = VOP_READ(vp, ap->a_uio, IO_UNIT | IO_SYNC, ap->a_cred); VOP_UNLOCK(vp, 0); vn_close(vp, flags, ap->a_cred, td); ZFS_EXIT(zfsvfs); return (error); } /* * Vnode operation to remove a named attribute. */ int zfs_deleteextattr(struct vop_deleteextattr_args *ap) /* vop_deleteextattr { IN struct vnode *a_vp; IN int a_attrnamespace; IN const char *a_name; IN struct ucred *a_cred; IN struct thread *a_td; }; */ { zfsvfs_t *zfsvfs = VTOZ(ap->a_vp)->z_zfsvfs; struct thread *td = ap->a_td; struct nameidata nd; char attrname[255]; struct vattr va; vnode_t *xvp = NULL, *vp; int error, flags; error = extattr_check_cred(ap->a_vp, ap->a_attrnamespace, ap->a_cred, ap->a_td, VWRITE); if (error != 0) return (error); error = zfs_create_attrname(ap->a_attrnamespace, ap->a_name, attrname, sizeof(attrname)); if (error != 0) return (error); ZFS_ENTER(zfsvfs); error = zfs_lookup(ap->a_vp, NULL, &xvp, NULL, 0, ap->a_cred, td, LOOKUP_XATTR); if (error != 0) { ZFS_EXIT(zfsvfs); return (error); } NDINIT_ATVP(&nd, DELETE, NOFOLLOW | LOCKPARENT | LOCKLEAF | MPSAFE, UIO_SYSSPACE, attrname, xvp, td); error = namei(&nd); vp = nd.ni_vp; NDFREE(&nd, NDF_ONLY_PNBUF); if (error != 0) { ZFS_EXIT(zfsvfs); if (error == ENOENT) error = ENOATTR; return (error); } error = VOP_REMOVE(nd.ni_dvp, vp, &nd.ni_cnd); vput(nd.ni_dvp); if (vp == nd.ni_dvp) vrele(vp); else vput(vp); ZFS_EXIT(zfsvfs); return (error); } /* * Vnode operation to set a named attribute. */ static int zfs_setextattr(struct vop_setextattr_args *ap) /* vop_setextattr { IN struct vnode *a_vp; IN int a_attrnamespace; IN const char *a_name; INOUT struct uio *a_uio; IN struct ucred *a_cred; IN struct thread *a_td; }; */ { zfsvfs_t *zfsvfs = VTOZ(ap->a_vp)->z_zfsvfs; struct thread *td = ap->a_td; struct nameidata nd; char attrname[255]; struct vattr va; vnode_t *xvp = NULL, *vp; int error, flags; error = extattr_check_cred(ap->a_vp, ap->a_attrnamespace, ap->a_cred, ap->a_td, VWRITE); if (error != 0) return (error); error = zfs_create_attrname(ap->a_attrnamespace, ap->a_name, attrname, sizeof(attrname)); if (error != 0) return (error); ZFS_ENTER(zfsvfs); error = zfs_lookup(ap->a_vp, NULL, &xvp, NULL, 0, ap->a_cred, td, LOOKUP_XATTR | CREATE_XATTR_DIR); if (error != 0) { ZFS_EXIT(zfsvfs); return (error); } flags = FFLAGS(O_WRONLY | O_CREAT); NDINIT_ATVP(&nd, LOOKUP, NOFOLLOW | MPSAFE, UIO_SYSSPACE, attrname, xvp, td); error = vn_open_cred(&nd, &flags, 0600, 0, ap->a_cred, NULL); vp = nd.ni_vp; NDFREE(&nd, NDF_ONLY_PNBUF); if (error != 0) { ZFS_EXIT(zfsvfs); return (error); } VATTR_NULL(&va); va.va_size = 0; error = VOP_SETATTR(vp, &va, ap->a_cred); if (error == 0) VOP_WRITE(vp, ap->a_uio, IO_UNIT | IO_SYNC, ap->a_cred); VOP_UNLOCK(vp, 0); vn_close(vp, flags, ap->a_cred, td); ZFS_EXIT(zfsvfs); return (error); } /* * Vnode operation to retrieve extended attributes on a vnode. */ static int zfs_listextattr(struct vop_listextattr_args *ap) /* vop_listextattr { IN struct vnode *a_vp; IN int a_attrnamespace; INOUT struct uio *a_uio; OUT size_t *a_size; IN struct ucred *a_cred; IN struct thread *a_td; }; */ { zfsvfs_t *zfsvfs = VTOZ(ap->a_vp)->z_zfsvfs; struct thread *td = ap->a_td; struct nameidata nd; char attrprefix[16]; u_char dirbuf[sizeof(struct dirent)]; struct dirent *dp; struct iovec aiov; struct uio auio, *uio = ap->a_uio; size_t *sizep = ap->a_size; size_t plen; vnode_t *xvp = NULL, *vp; int done, error, eof, pos; error = extattr_check_cred(ap->a_vp, ap->a_attrnamespace, ap->a_cred, ap->a_td, VREAD); if (error != 0) return (error); error = zfs_create_attrname(ap->a_attrnamespace, "", attrprefix, sizeof(attrprefix)); if (error != 0) return (error); plen = strlen(attrprefix); ZFS_ENTER(zfsvfs); if (sizep != NULL) *sizep = 0; error = zfs_lookup(ap->a_vp, NULL, &xvp, NULL, 0, ap->a_cred, td, LOOKUP_XATTR); if (error != 0) { ZFS_EXIT(zfsvfs); /* * ENOATTR means that the EA directory does not yet exist, * i.e. there are no extended attributes there. */ if (error == ENOATTR) error = 0; return (error); } NDINIT_ATVP(&nd, LOOKUP, NOFOLLOW | LOCKLEAF | LOCKSHARED | MPSAFE, UIO_SYSSPACE, ".", xvp, td); error = namei(&nd); vp = nd.ni_vp; NDFREE(&nd, NDF_ONLY_PNBUF); if (error != 0) { ZFS_EXIT(zfsvfs); return (error); } auio.uio_iov = &aiov; auio.uio_iovcnt = 1; auio.uio_segflg = UIO_SYSSPACE; auio.uio_td = td; auio.uio_rw = UIO_READ; auio.uio_offset = 0; do { u_char nlen; aiov.iov_base = (void *)dirbuf; aiov.iov_len = sizeof(dirbuf); auio.uio_resid = sizeof(dirbuf); error = VOP_READDIR(vp, &auio, ap->a_cred, &eof, NULL, NULL); done = sizeof(dirbuf) - auio.uio_resid; if (error != 0) break; for (pos = 0; pos < done;) { dp = (struct dirent *)(dirbuf + pos); pos += dp->d_reclen; /* * XXX: Temporarily we also accept DT_UNKNOWN, as this * is what we get when attribute was created on Solaris. */ if (dp->d_type != DT_REG && dp->d_type != DT_UNKNOWN) continue; if (plen == 0 && strncmp(dp->d_name, "freebsd:", 8) == 0) continue; else if (strncmp(dp->d_name, attrprefix, plen) != 0) continue; nlen = dp->d_namlen - plen; if (sizep != NULL) *sizep += 1 + nlen; else if (uio != NULL) { /* * Format of extattr name entry is one byte for * length and the rest for name. */ error = uiomove(&nlen, 1, uio->uio_rw, uio); if (error == 0) { error = uiomove(dp->d_name + plen, nlen, uio->uio_rw, uio); } if (error != 0) break; } } } while (!eof && error == 0); vput(vp); ZFS_EXIT(zfsvfs); return (error); } int zfs_freebsd_getacl(ap) struct vop_getacl_args /* { struct vnode *vp; acl_type_t type; struct acl *aclp; struct ucred *cred; struct thread *td; } */ *ap; { int error; vsecattr_t vsecattr; if (ap->a_type != ACL_TYPE_NFS4) return (EINVAL); vsecattr.vsa_mask = VSA_ACE | VSA_ACECNT; if (error = zfs_getsecattr(ap->a_vp, &vsecattr, 0, ap->a_cred, NULL)) return (error); error = acl_from_aces(ap->a_aclp, vsecattr.vsa_aclentp, vsecattr.vsa_aclcnt); if (vsecattr.vsa_aclentp != NULL) kmem_free(vsecattr.vsa_aclentp, vsecattr.vsa_aclentsz); return (error); } int zfs_freebsd_setacl(ap) struct vop_setacl_args /* { struct vnode *vp; acl_type_t type; struct acl *aclp; struct ucred *cred; struct thread *td; } */ *ap; { int error; vsecattr_t vsecattr; int aclbsize; /* size of acl list in bytes */ aclent_t *aaclp; if (ap->a_type != ACL_TYPE_NFS4) return (EINVAL); if (ap->a_aclp->acl_cnt < 1 || ap->a_aclp->acl_cnt > MAX_ACL_ENTRIES) return (EINVAL); /* * With NFSv4 ACLs, chmod(2) may need to add additional entries, * splitting every entry into two and appending "canonical six" * entries at the end. Don't allow for setting an ACL that would * cause chmod(2) to run out of ACL entries. */ if (ap->a_aclp->acl_cnt * 2 + 6 > ACL_MAX_ENTRIES) return (ENOSPC); vsecattr.vsa_mask = VSA_ACE; aclbsize = ap->a_aclp->acl_cnt * sizeof(ace_t); vsecattr.vsa_aclentp = kmem_alloc(aclbsize, KM_SLEEP); aaclp = vsecattr.vsa_aclentp; vsecattr.vsa_aclentsz = aclbsize; aces_from_acl(vsecattr.vsa_aclentp, &vsecattr.vsa_aclcnt, ap->a_aclp); error = zfs_setsecattr(ap->a_vp, &vsecattr, 0, ap->a_cred, NULL); kmem_free(aaclp, aclbsize); return (error); } int zfs_freebsd_aclcheck(ap) struct vop_aclcheck_args /* { struct vnode *vp; acl_type_t type; struct acl *aclp; struct ucred *cred; struct thread *td; } */ *ap; { return (EOPNOTSUPP); } struct vop_vector zfs_vnodeops; struct vop_vector zfs_fifoops; struct vop_vector zfs_vnodeops = { .vop_default = &default_vnodeops, .vop_inactive = zfs_freebsd_inactive, .vop_reclaim = zfs_freebsd_reclaim, .vop_access = zfs_freebsd_access, #ifdef FREEBSD_NAMECACHE .vop_lookup = vfs_cache_lookup, .vop_cachedlookup = zfs_freebsd_lookup, #else .vop_lookup = zfs_freebsd_lookup, #endif .vop_getattr = zfs_freebsd_getattr, .vop_setattr = zfs_freebsd_setattr, .vop_create = zfs_freebsd_create, .vop_mknod = zfs_freebsd_create, .vop_mkdir = zfs_freebsd_mkdir, .vop_readdir = zfs_freebsd_readdir, .vop_fsync = zfs_freebsd_fsync, .vop_open = zfs_freebsd_open, .vop_close = zfs_freebsd_close, .vop_rmdir = zfs_freebsd_rmdir, .vop_ioctl = zfs_freebsd_ioctl, .vop_link = zfs_freebsd_link, .vop_symlink = zfs_freebsd_symlink, .vop_readlink = zfs_freebsd_readlink, .vop_read = zfs_freebsd_read, .vop_write = zfs_freebsd_write, .vop_remove = zfs_freebsd_remove, .vop_rename = zfs_freebsd_rename, .vop_pathconf = zfs_freebsd_pathconf, .vop_bmap = VOP_EOPNOTSUPP, .vop_fid = zfs_freebsd_fid, .vop_getextattr = zfs_getextattr, .vop_deleteextattr = zfs_deleteextattr, .vop_setextattr = zfs_setextattr, .vop_listextattr = zfs_listextattr, .vop_getacl = zfs_freebsd_getacl, .vop_setacl = zfs_freebsd_setacl, .vop_aclcheck = zfs_freebsd_aclcheck, }; struct vop_vector zfs_fifoops = { .vop_default = &fifo_specops, .vop_fsync = zfs_freebsd_fsync, .vop_access = zfs_freebsd_access, .vop_getattr = zfs_freebsd_getattr, .vop_inactive = zfs_freebsd_inactive, .vop_read = VOP_PANIC, .vop_reclaim = zfs_freebsd_reclaim, .vop_setattr = zfs_freebsd_setattr, .vop_write = VOP_PANIC, .vop_pathconf = zfs_freebsd_fifo_pathconf, .vop_fid = zfs_freebsd_fid, .vop_getacl = zfs_freebsd_getacl, .vop_setacl = zfs_freebsd_setacl, .vop_aclcheck = zfs_freebsd_aclcheck, }; Index: user/imp/tbemd/sys/contrib/x86emu =================================================================== --- user/imp/tbemd/sys/contrib/x86emu (revision 206684) +++ user/imp/tbemd/sys/contrib/x86emu (revision 206685) Property changes on: user/imp/tbemd/sys/contrib/x86emu ___________________________________________________________________ Modified: svn:mergeinfo ## -0,0 +0,1 ## Merged /head/sys/contrib/x86emu:r206628-206684 Index: user/imp/tbemd/sys/ddb/db_sym.c =================================================================== --- user/imp/tbemd/sys/ddb/db_sym.c (revision 206684) +++ user/imp/tbemd/sys/ddb/db_sym.c (revision 206685) @@ -1,511 +1,505 @@ /*- * Mach Operating System * Copyright (c) 1991,1990 Carnegie Mellon University * All Rights Reserved. * * Permission to use, copy, modify and distribute this software and its * documentation is hereby granted, provided that both the copyright * notice and this permission notice appear in all copies of the * software, derivative works or modified versions, and any portions * thereof, and that both notices appear in supporting documentation. * * CARNEGIE MELLON ALLOWS FREE USE OF THIS SOFTWARE IN ITS * CONDITION. CARNEGIE MELLON DISCLAIMS ANY LIABILITY OF ANY KIND FOR * ANY DAMAGES WHATSOEVER RESULTING FROM THE USE OF THIS SOFTWARE. * * Carnegie Mellon requests users of this software to return to * * Software Distribution Coordinator or Software.Distribution@CS.CMU.EDU * School of Computer Science * Carnegie Mellon University * Pittsburgh PA 15213-3890 * * any improvements or extensions that they make and grant Carnegie the * rights to redistribute these changes. */ /* * Author: David B. Golub, Carnegie Mellon University * Date: 7/90 */ #include __FBSDID("$FreeBSD$"); #include #include #include #include #include #include #include #include #include /* * Multiple symbol tables */ #ifndef MAXNOSYMTABS #define MAXNOSYMTABS 3 /* mach, ux, emulator */ #endif static db_symtab_t db_symtabs[MAXNOSYMTABS] = {{0,},}; static int db_nsymtab = 0; static db_symtab_t *db_last_symtab; /* where last symbol was found */ static c_db_sym_t db_lookup( const char *symstr); static char *db_qualify(c_db_sym_t sym, char *symtabname); static boolean_t db_symbol_is_ambiguous(c_db_sym_t sym); static boolean_t db_line_at_pc(c_db_sym_t, char **, int *, db_expr_t); static int db_cpu = -1; #ifdef VIMAGE -extern uintptr_t *__start_set_vnet; -extern uintptr_t *__stop_set_vnet; - -#define VNET_START (uintptr_t)&__start_set_vnet -#define VNET_STOP (uintptr_t)&__stop_set_vnet - static void *db_vnet = NULL; #endif /* * Validate the CPU number used to interpret per-CPU variables so we can * avoid later confusion if an invalid CPU is requested. */ int db_var_db_cpu(struct db_variable *vp, db_expr_t *valuep, int op) { switch (op) { case DB_VAR_GET: *valuep = db_cpu; return (1); case DB_VAR_SET: if (*(int *)valuep < -1 && *(int *)valuep > mp_maxid) { db_printf("Invalid value: %d", *(int*)valuep); return (0); } db_cpu = *(int *)valuep; return (1); default: db_printf("db_var_db_cpu: unknown operation\n"); return (0); } } /* * Read-only variable reporting the current CPU, which is what we use when * db_cpu is set to -1. */ int db_var_curcpu(struct db_variable *vp, db_expr_t *valuep, int op) { switch (op) { case DB_VAR_GET: *valuep = curcpu; return (1); case DB_VAR_SET: db_printf("Read-only variable.\n"); return (0); default: db_printf("db_var_curcpu: unknown operation\n"); return (0); } } #ifdef VIMAGE /* * Validate the virtual network pointer used to interpret per-vnet global * variable expansion. Right now we don't do much here, really we should * walk the global vnet list to check it's an OK pointer. */ int db_var_db_vnet(struct db_variable *vp, db_expr_t *valuep, int op) { switch (op) { case DB_VAR_GET: *valuep = (db_expr_t)db_vnet; return (1); case DB_VAR_SET: db_vnet = *(void **)valuep; return (1); default: db_printf("db_var_db_vnet: unknown operation\n"); return (0); } } /* * Read-only variable reporting the current vnet, which is what we use when * db_vnet is set to NULL. */ int db_var_curvnet(struct db_variable *vp, db_expr_t *valuep, int op) { switch (op) { case DB_VAR_GET: *valuep = (db_expr_t)curvnet; return (1); case DB_VAR_SET: db_printf("Read-only variable.\n"); return (0); default: db_printf("db_var_curcpu: unknown operation\n"); return (0); } } #endif /* * Add symbol table, with given name, to list of symbol tables. */ void db_add_symbol_table(start, end, name, ref) char *start; char *end; char *name; char *ref; { if (db_nsymtab >= MAXNOSYMTABS) { printf ("No slots left for %s symbol table", name); panic ("db_sym.c: db_add_symbol_table"); } db_symtabs[db_nsymtab].start = start; db_symtabs[db_nsymtab].end = end; db_symtabs[db_nsymtab].name = name; db_symtabs[db_nsymtab].private = ref; db_nsymtab++; } /* * db_qualify("vm_map", "ux") returns "unix:vm_map". * * Note: return value points to static data whose content is * overwritten by each call... but in practice this seems okay. */ static char * db_qualify(sym, symtabname) c_db_sym_t sym; register char *symtabname; { const char *symname; static char tmp[256]; db_symbol_values(sym, &symname, 0); snprintf(tmp, sizeof(tmp), "%s:%s", symtabname, symname); return tmp; } boolean_t db_eqname(src, dst, c) const char *src; const char *dst; int c; { if (!strcmp(src, dst)) return (TRUE); if (src[0] == c) return (!strcmp(src+1,dst)); return (FALSE); } boolean_t db_value_of_name(name, valuep) const char *name; db_expr_t *valuep; { c_db_sym_t sym; sym = db_lookup(name); if (sym == C_DB_SYM_NULL) return (FALSE); db_symbol_values(sym, &name, valuep); return (TRUE); } boolean_t db_value_of_name_pcpu(name, valuep) const char *name; db_expr_t *valuep; { static char tmp[256]; db_expr_t value; c_db_sym_t sym; int cpu; if (db_cpu != -1) cpu = db_cpu; else cpu = curcpu; snprintf(tmp, sizeof(tmp), "pcpu_entry_%s", name); sym = db_lookup(tmp); if (sym == C_DB_SYM_NULL) return (FALSE); db_symbol_values(sym, &name, &value); if (value < DPCPU_START || value >= DPCPU_STOP) return (FALSE); *valuep = (db_expr_t)((uintptr_t)value + dpcpu_off[cpu]); return (TRUE); } boolean_t db_value_of_name_vnet(name, valuep) const char *name; db_expr_t *valuep; { #ifdef VIMAGE static char tmp[256]; db_expr_t value; c_db_sym_t sym; struct vnet *vnet; if (db_vnet != NULL) vnet = db_vnet; else vnet = curvnet; snprintf(tmp, sizeof(tmp), "vnet_entry_%s", name); sym = db_lookup(tmp); if (sym == C_DB_SYM_NULL) return (FALSE); db_symbol_values(sym, &name, &value); if (value < VNET_START || value >= VNET_STOP) return (FALSE); *valuep = (db_expr_t)((uintptr_t)value + vnet->vnet_data_base); return (TRUE); #else return (FALSE); #endif } /* * Lookup a symbol. * If the symbol has a qualifier (e.g., ux:vm_map), * then only the specified symbol table will be searched; * otherwise, all symbol tables will be searched. */ static c_db_sym_t db_lookup(symstr) const char *symstr; { c_db_sym_t sp; register int i; int symtab_start = 0; int symtab_end = db_nsymtab; register const char *cp; /* * Look for, remove, and remember any symbol table specifier. */ for (cp = symstr; *cp; cp++) { if (*cp == ':') { for (i = 0; i < db_nsymtab; i++) { int n = strlen(db_symtabs[i].name); if ( n == (cp - symstr) && strncmp(symstr, db_symtabs[i].name, n) == 0 ) { symtab_start = i; symtab_end = i + 1; break; } } if (i == db_nsymtab) { db_error("invalid symbol table name"); } symstr = cp+1; } } /* * Look in the specified set of symbol tables. * Return on first match. */ for (i = symtab_start; i < symtab_end; i++) { sp = X_db_lookup(&db_symtabs[i], symstr); if (sp) { db_last_symtab = &db_symtabs[i]; return sp; } } return 0; } /* * If TRUE, check across symbol tables for multiple occurrences * of a name. Might slow things down quite a bit. */ static volatile boolean_t db_qualify_ambiguous_names = FALSE; /* * Does this symbol name appear in more than one symbol table? * Used by db_symbol_values to decide whether to qualify a symbol. */ static boolean_t db_symbol_is_ambiguous(sym) c_db_sym_t sym; { const char *sym_name; register int i; register boolean_t found_once = FALSE; if (!db_qualify_ambiguous_names) return FALSE; db_symbol_values(sym, &sym_name, 0); for (i = 0; i < db_nsymtab; i++) { if (X_db_lookup(&db_symtabs[i], sym_name)) { if (found_once) return TRUE; found_once = TRUE; } } return FALSE; } /* * Find the closest symbol to val, and return its name * and the difference between val and the symbol found. */ c_db_sym_t db_search_symbol( val, strategy, offp) register db_addr_t val; db_strategy_t strategy; db_expr_t *offp; { register unsigned int diff; size_t newdiff; register int i; c_db_sym_t ret = C_DB_SYM_NULL, sym; newdiff = diff = ~0; for (i = 0; i < db_nsymtab; i++) { sym = X_db_search_symbol(&db_symtabs[i], val, strategy, &newdiff); if (newdiff < diff) { db_last_symtab = &db_symtabs[i]; diff = newdiff; ret = sym; } } *offp = diff; return ret; } /* * Return name and value of a symbol */ void db_symbol_values(sym, namep, valuep) c_db_sym_t sym; const char **namep; db_expr_t *valuep; { db_expr_t value; if (sym == DB_SYM_NULL) { *namep = 0; return; } X_db_symbol_values(db_last_symtab, sym, namep, &value); if (db_symbol_is_ambiguous(sym)) *namep = db_qualify(sym, db_last_symtab->name); if (valuep) *valuep = value; } /* * Print a the closest symbol to value * * After matching the symbol according to the given strategy * we print it in the name+offset format, provided the symbol's * value is close enough (eg smaller than db_maxoff). * We also attempt to print [filename:linenum] when applicable * (eg for procedure names). * * If we could not find a reasonable name+offset representation, * then we just print the value in hex. Small values might get * bogus symbol associations, e.g. 3 might get some absolute * value like _INCLUDE_VERSION or something, therefore we do * not accept symbols whose value is "small" (and use plain hex). */ db_expr_t db_maxoff = 0x10000; void db_printsym(off, strategy) db_expr_t off; db_strategy_t strategy; { db_expr_t d; char *filename; const char *name; db_expr_t value; int linenum; c_db_sym_t cursym; cursym = db_search_symbol(off, strategy, &d); db_symbol_values(cursym, &name, &value); if (name == 0) value = off; if (value >= DB_SMALL_VALUE_MIN && value <= DB_SMALL_VALUE_MAX) { db_printf("%+#lr", (long)off); return; } if (name == 0 || d >= (unsigned long)db_maxoff) { db_printf("%#lr", (unsigned long)off); return; } #ifdef DDB_NUMSYM db_printf("%#lr = %s", (unsigned long)off, name); #else db_printf("%s", name); #endif if (d) db_printf("+%+#lr", (long)d); if (strategy == DB_STGY_PROC) { if (db_line_at_pc(cursym, &filename, &linenum, off)) db_printf(" [%s:%d]", filename, linenum); } } static boolean_t db_line_at_pc( sym, filename, linenum, pc) c_db_sym_t sym; char **filename; int *linenum; db_expr_t pc; { return X_db_line_at_pc( db_last_symtab, sym, filename, linenum, pc); } int db_sym_numargs(sym, nargp, argnames) c_db_sym_t sym; int *nargp; char **argnames; { return X_db_sym_numargs(db_last_symtab, sym, nargp, argnames); } Index: user/imp/tbemd/sys/dev/e1000/if_em.c =================================================================== --- user/imp/tbemd/sys/dev/e1000/if_em.c (revision 206684) +++ user/imp/tbemd/sys/dev/e1000/if_em.c (revision 206685) @@ -1,5119 +1,5124 @@ /****************************************************************************** Copyright (c) 2001-2010, Intel Corporation All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. 3. Neither the name of the Intel Corporation nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. ******************************************************************************/ /*$FreeBSD$*/ #ifdef HAVE_KERNEL_OPTION_HEADERS #include "opt_device_polling.h" #include "opt_inet.h" #endif #include #include #if __FreeBSD_version >= 800000 #include #endif #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include "e1000_api.h" #include "e1000_82571.h" #include "if_em.h" /********************************************************************* * Set this to one to display debug statistics *********************************************************************/ int em_display_debug_stats = 0; /********************************************************************* * Driver version: *********************************************************************/ -char em_driver_version[] = "7.0.4"; +char em_driver_version[] = "7.0.5"; /********************************************************************* * PCI Device ID Table * * Used by probe to select devices to load on * Last field stores an index into e1000_strings * Last entry must be all 0s * * { Vendor ID, Device ID, SubVendor ID, SubDevice ID, String Index } *********************************************************************/ static em_vendor_info_t em_vendor_info_array[] = { /* Intel(R) PRO/1000 Network Connection */ { 0x8086, E1000_DEV_ID_82571EB_COPPER, PCI_ANY_ID, PCI_ANY_ID, 0}, { 0x8086, E1000_DEV_ID_82571EB_FIBER, PCI_ANY_ID, PCI_ANY_ID, 0}, { 0x8086, E1000_DEV_ID_82571EB_SERDES, PCI_ANY_ID, PCI_ANY_ID, 0}, { 0x8086, E1000_DEV_ID_82571EB_SERDES_DUAL, PCI_ANY_ID, PCI_ANY_ID, 0}, { 0x8086, E1000_DEV_ID_82571EB_SERDES_QUAD, PCI_ANY_ID, PCI_ANY_ID, 0}, { 0x8086, E1000_DEV_ID_82571EB_QUAD_COPPER, PCI_ANY_ID, PCI_ANY_ID, 0}, { 0x8086, E1000_DEV_ID_82571EB_QUAD_COPPER_LP, PCI_ANY_ID, PCI_ANY_ID, 0}, { 0x8086, E1000_DEV_ID_82571EB_QUAD_FIBER, PCI_ANY_ID, PCI_ANY_ID, 0}, { 0x8086, E1000_DEV_ID_82571PT_QUAD_COPPER, PCI_ANY_ID, PCI_ANY_ID, 0}, { 0x8086, E1000_DEV_ID_82572EI_COPPER, PCI_ANY_ID, PCI_ANY_ID, 0}, { 0x8086, E1000_DEV_ID_82572EI_FIBER, PCI_ANY_ID, PCI_ANY_ID, 0}, { 0x8086, E1000_DEV_ID_82572EI_SERDES, PCI_ANY_ID, PCI_ANY_ID, 0}, { 0x8086, E1000_DEV_ID_82572EI, PCI_ANY_ID, PCI_ANY_ID, 0}, { 0x8086, E1000_DEV_ID_82573E, PCI_ANY_ID, PCI_ANY_ID, 0}, { 0x8086, E1000_DEV_ID_82573E_IAMT, PCI_ANY_ID, PCI_ANY_ID, 0}, { 0x8086, E1000_DEV_ID_82573L, PCI_ANY_ID, PCI_ANY_ID, 0}, { 0x8086, E1000_DEV_ID_82583V, PCI_ANY_ID, PCI_ANY_ID, 0}, { 0x8086, E1000_DEV_ID_80003ES2LAN_COPPER_SPT, PCI_ANY_ID, PCI_ANY_ID, 0}, { 0x8086, E1000_DEV_ID_80003ES2LAN_SERDES_SPT, PCI_ANY_ID, PCI_ANY_ID, 0}, { 0x8086, E1000_DEV_ID_80003ES2LAN_COPPER_DPT, PCI_ANY_ID, PCI_ANY_ID, 0}, { 0x8086, E1000_DEV_ID_80003ES2LAN_SERDES_DPT, PCI_ANY_ID, PCI_ANY_ID, 0}, { 0x8086, E1000_DEV_ID_ICH8_IGP_M_AMT, PCI_ANY_ID, PCI_ANY_ID, 0}, { 0x8086, E1000_DEV_ID_ICH8_IGP_AMT, PCI_ANY_ID, PCI_ANY_ID, 0}, { 0x8086, E1000_DEV_ID_ICH8_IGP_C, PCI_ANY_ID, PCI_ANY_ID, 0}, { 0x8086, E1000_DEV_ID_ICH8_IFE, PCI_ANY_ID, PCI_ANY_ID, 0}, { 0x8086, E1000_DEV_ID_ICH8_IFE_GT, PCI_ANY_ID, PCI_ANY_ID, 0}, { 0x8086, E1000_DEV_ID_ICH8_IFE_G, PCI_ANY_ID, PCI_ANY_ID, 0}, { 0x8086, E1000_DEV_ID_ICH8_IGP_M, PCI_ANY_ID, PCI_ANY_ID, 0}, { 0x8086, E1000_DEV_ID_ICH8_82567V_3, PCI_ANY_ID, PCI_ANY_ID, 0}, { 0x8086, E1000_DEV_ID_ICH9_IGP_M_AMT, PCI_ANY_ID, PCI_ANY_ID, 0}, { 0x8086, E1000_DEV_ID_ICH9_IGP_AMT, PCI_ANY_ID, PCI_ANY_ID, 0}, { 0x8086, E1000_DEV_ID_ICH9_IGP_C, PCI_ANY_ID, PCI_ANY_ID, 0}, { 0x8086, E1000_DEV_ID_ICH9_IGP_M, PCI_ANY_ID, PCI_ANY_ID, 0}, { 0x8086, E1000_DEV_ID_ICH9_IGP_M_V, PCI_ANY_ID, PCI_ANY_ID, 0}, { 0x8086, E1000_DEV_ID_ICH9_IFE, PCI_ANY_ID, PCI_ANY_ID, 0}, { 0x8086, E1000_DEV_ID_ICH9_IFE_GT, PCI_ANY_ID, PCI_ANY_ID, 0}, { 0x8086, E1000_DEV_ID_ICH9_IFE_G, PCI_ANY_ID, PCI_ANY_ID, 0}, { 0x8086, E1000_DEV_ID_ICH9_BM, PCI_ANY_ID, PCI_ANY_ID, 0}, { 0x8086, E1000_DEV_ID_82574L, PCI_ANY_ID, PCI_ANY_ID, 0}, { 0x8086, E1000_DEV_ID_82574LA, PCI_ANY_ID, PCI_ANY_ID, 0}, { 0x8086, E1000_DEV_ID_ICH10_R_BM_LM, PCI_ANY_ID, PCI_ANY_ID, 0}, { 0x8086, E1000_DEV_ID_ICH10_R_BM_LF, PCI_ANY_ID, PCI_ANY_ID, 0}, { 0x8086, E1000_DEV_ID_ICH10_R_BM_V, PCI_ANY_ID, PCI_ANY_ID, 0}, { 0x8086, E1000_DEV_ID_ICH10_D_BM_LM, PCI_ANY_ID, PCI_ANY_ID, 0}, { 0x8086, E1000_DEV_ID_ICH10_D_BM_LF, PCI_ANY_ID, PCI_ANY_ID, 0}, { 0x8086, E1000_DEV_ID_PCH_M_HV_LM, PCI_ANY_ID, PCI_ANY_ID, 0}, { 0x8086, E1000_DEV_ID_PCH_M_HV_LC, PCI_ANY_ID, PCI_ANY_ID, 0}, { 0x8086, E1000_DEV_ID_PCH_D_HV_DM, PCI_ANY_ID, PCI_ANY_ID, 0}, { 0x8086, E1000_DEV_ID_PCH_D_HV_DC, PCI_ANY_ID, PCI_ANY_ID, 0}, /* required last entry */ { 0, 0, 0, 0, 0} }; /********************************************************************* * Table of branding strings for all supported NICs. *********************************************************************/ static char *em_strings[] = { "Intel(R) PRO/1000 Network Connection" }; /********************************************************************* * Function prototypes *********************************************************************/ static int em_probe(device_t); static int em_attach(device_t); static int em_detach(device_t); static int em_shutdown(device_t); static int em_suspend(device_t); static int em_resume(device_t); static void em_start(struct ifnet *); static void em_start_locked(struct ifnet *, struct tx_ring *); #ifdef EM_MULTIQUEUE static int em_mq_start(struct ifnet *, struct mbuf *); static int em_mq_start_locked(struct ifnet *, struct tx_ring *, struct mbuf *); static void em_qflush(struct ifnet *); #endif static int em_ioctl(struct ifnet *, u_long, caddr_t); static void em_init(void *); static void em_init_locked(struct adapter *); static void em_stop(void *); static void em_media_status(struct ifnet *, struct ifmediareq *); static int em_media_change(struct ifnet *); static void em_identify_hardware(struct adapter *); static int em_allocate_pci_resources(struct adapter *); static int em_allocate_legacy(struct adapter *); static int em_allocate_msix(struct adapter *); static int em_allocate_queues(struct adapter *); static int em_setup_msix(struct adapter *); static void em_free_pci_resources(struct adapter *); static void em_local_timer(void *); static void em_reset(struct adapter *); static void em_setup_interface(device_t, struct adapter *); static void em_setup_transmit_structures(struct adapter *); static void em_initialize_transmit_unit(struct adapter *); static int em_allocate_transmit_buffers(struct tx_ring *); static void em_free_transmit_structures(struct adapter *); static void em_free_transmit_buffers(struct tx_ring *); static int em_setup_receive_structures(struct adapter *); static int em_allocate_receive_buffers(struct rx_ring *); static void em_initialize_receive_unit(struct adapter *); static void em_free_receive_structures(struct adapter *); static void em_free_receive_buffers(struct rx_ring *); static void em_enable_intr(struct adapter *); static void em_disable_intr(struct adapter *); static void em_update_stats_counters(struct adapter *); static bool em_txeof(struct tx_ring *); static int em_rxeof(struct rx_ring *, int); #ifndef __NO_STRICT_ALIGNMENT static int em_fixup_rx(struct rx_ring *); #endif static void em_receive_checksum(struct e1000_rx_desc *, struct mbuf *); static void em_transmit_checksum_setup(struct tx_ring *, struct mbuf *, u32 *, u32 *); static bool em_tso_setup(struct tx_ring *, struct mbuf *, u32 *, u32 *); static void em_set_promisc(struct adapter *); static void em_disable_promisc(struct adapter *); static void em_set_multi(struct adapter *); static void em_print_hw_stats(struct adapter *); static void em_update_link_status(struct adapter *); static void em_refresh_mbufs(struct rx_ring *, int); static void em_register_vlan(void *, struct ifnet *, u16); static void em_unregister_vlan(void *, struct ifnet *, u16); static void em_setup_vlan_hw_support(struct adapter *); static int em_xmit(struct tx_ring *, struct mbuf **); static int em_dma_malloc(struct adapter *, bus_size_t, struct em_dma_alloc *, int); static void em_dma_free(struct adapter *, struct em_dma_alloc *); static void em_print_debug_info(struct adapter *); static void em_print_nvm_info(struct adapter *); static int em_is_valid_ether_addr(u8 *); static int em_sysctl_stats(SYSCTL_HANDLER_ARGS); static int em_sysctl_debug_info(SYSCTL_HANDLER_ARGS); static int em_sysctl_int_delay(SYSCTL_HANDLER_ARGS); static void em_add_int_delay_sysctl(struct adapter *, const char *, const char *, struct em_int_delay_info *, int, int); /* Management and WOL Support */ static void em_init_manageability(struct adapter *); static void em_release_manageability(struct adapter *); static void em_get_hw_control(struct adapter *); static void em_release_hw_control(struct adapter *); static void em_get_wakeup(device_t); static void em_enable_wakeup(device_t); static int em_enable_phy_wakeup(struct adapter *); static void em_led_func(void *, int); static int em_irq_fast(void *); /* MSIX handlers */ static void em_msix_tx(void *); static void em_msix_rx(void *); static void em_msix_link(void *); static void em_handle_tx(void *context, int pending); static void em_handle_rx(void *context, int pending); static void em_handle_link(void *context, int pending); static void em_add_rx_process_limit(struct adapter *, const char *, const char *, int *, int); #ifdef DEVICE_POLLING static poll_handler_t em_poll; #endif /* POLLING */ /********************************************************************* * FreeBSD Device Interface Entry Points *********************************************************************/ static device_method_t em_methods[] = { /* Device interface */ DEVMETHOD(device_probe, em_probe), DEVMETHOD(device_attach, em_attach), DEVMETHOD(device_detach, em_detach), DEVMETHOD(device_shutdown, em_shutdown), DEVMETHOD(device_suspend, em_suspend), DEVMETHOD(device_resume, em_resume), {0, 0} }; static driver_t em_driver = { "em", em_methods, sizeof(struct adapter), }; devclass_t em_devclass; DRIVER_MODULE(em, pci, em_driver, em_devclass, 0, 0); MODULE_DEPEND(em, pci, 1, 1, 1); MODULE_DEPEND(em, ether, 1, 1, 1); /********************************************************************* * Tunable default values. *********************************************************************/ #define EM_TICKS_TO_USECS(ticks) ((1024 * (ticks) + 500) / 1000) #define EM_USECS_TO_TICKS(usecs) ((1000 * (usecs) + 512) / 1024) #define M_TSO_LEN 66 /* Allow common code without TSO */ #ifndef CSUM_TSO #define CSUM_TSO 0 #endif static int em_tx_int_delay_dflt = EM_TICKS_TO_USECS(EM_TIDV); static int em_rx_int_delay_dflt = EM_TICKS_TO_USECS(EM_RDTR); TUNABLE_INT("hw.em.tx_int_delay", &em_tx_int_delay_dflt); TUNABLE_INT("hw.em.rx_int_delay", &em_rx_int_delay_dflt); static int em_tx_abs_int_delay_dflt = EM_TICKS_TO_USECS(EM_TADV); static int em_rx_abs_int_delay_dflt = EM_TICKS_TO_USECS(EM_RADV); TUNABLE_INT("hw.em.tx_abs_int_delay", &em_tx_abs_int_delay_dflt); TUNABLE_INT("hw.em.rx_abs_int_delay", &em_rx_abs_int_delay_dflt); static int em_rxd = EM_DEFAULT_RXD; static int em_txd = EM_DEFAULT_TXD; TUNABLE_INT("hw.em.rxd", &em_rxd); TUNABLE_INT("hw.em.txd", &em_txd); static int em_smart_pwr_down = FALSE; TUNABLE_INT("hw.em.smart_pwr_down", &em_smart_pwr_down); /* Controls whether promiscuous also shows bad packets */ static int em_debug_sbp = FALSE; TUNABLE_INT("hw.em.sbp", &em_debug_sbp); /* Local controls for MSI/MSIX */ static int em_enable_msix = TRUE; static int em_msix_queues = 2; /* for 82574, can be 1 or 2 */ TUNABLE_INT("hw.em.enable_msix", &em_enable_msix); TUNABLE_INT("hw.em.msix_queues", &em_msix_queues); /* How many packets rxeof tries to clean at a time */ static int em_rx_process_limit = 100; TUNABLE_INT("hw.em.rx_process_limit", &em_rx_process_limit); /* Flow control setting - default to FULL */ static int em_fc_setting = e1000_fc_full; TUNABLE_INT("hw.em.fc_setting", &em_fc_setting); /* ** Shadow VFTA table, this is needed because ** the real vlan filter table gets cleared during ** a soft reset and the driver needs to be able ** to repopulate it. */ static u32 em_shadow_vfta[EM_VFTA_SIZE]; /* Global used in WOL setup with multiport cards */ static int global_quad_port_a = 0; /********************************************************************* * Device identification routine * * em_probe determines if the driver should be loaded on * adapter based on PCI vendor/device id of the adapter. * * return BUS_PROBE_DEFAULT on success, positive on failure *********************************************************************/ static int em_probe(device_t dev) { char adapter_name[60]; u16 pci_vendor_id = 0; u16 pci_device_id = 0; u16 pci_subvendor_id = 0; u16 pci_subdevice_id = 0; em_vendor_info_t *ent; INIT_DEBUGOUT("em_probe: begin"); pci_vendor_id = pci_get_vendor(dev); if (pci_vendor_id != EM_VENDOR_ID) return (ENXIO); pci_device_id = pci_get_device(dev); pci_subvendor_id = pci_get_subvendor(dev); pci_subdevice_id = pci_get_subdevice(dev); ent = em_vendor_info_array; while (ent->vendor_id != 0) { if ((pci_vendor_id == ent->vendor_id) && (pci_device_id == ent->device_id) && ((pci_subvendor_id == ent->subvendor_id) || (ent->subvendor_id == PCI_ANY_ID)) && ((pci_subdevice_id == ent->subdevice_id) || (ent->subdevice_id == PCI_ANY_ID))) { sprintf(adapter_name, "%s %s", em_strings[ent->index], em_driver_version); device_set_desc_copy(dev, adapter_name); return (BUS_PROBE_DEFAULT); } ent++; } return (ENXIO); } /********************************************************************* * Device initialization routine * * The attach entry point is called when the driver is being loaded. * This routine identifies the type of hardware, allocates all resources * and initializes the hardware. * * return 0 on success, positive on failure *********************************************************************/ static int em_attach(device_t dev) { struct adapter *adapter; int error = 0; INIT_DEBUGOUT("em_attach: begin"); adapter = device_get_softc(dev); adapter->dev = adapter->osdep.dev = dev; EM_CORE_LOCK_INIT(adapter, device_get_nameunit(dev)); /* SYSCTL stuff */ SYSCTL_ADD_PROC(device_get_sysctl_ctx(dev), SYSCTL_CHILDREN(device_get_sysctl_tree(dev)), OID_AUTO, "debug", CTLTYPE_INT|CTLFLAG_RW, adapter, 0, em_sysctl_debug_info, "I", "Debug Information"); SYSCTL_ADD_PROC(device_get_sysctl_ctx(dev), SYSCTL_CHILDREN(device_get_sysctl_tree(dev)), OID_AUTO, "stats", CTLTYPE_INT|CTLFLAG_RW, adapter, 0, em_sysctl_stats, "I", "Statistics"); callout_init_mtx(&adapter->timer, &adapter->core_mtx, 0); /* Determine hardware and mac info */ em_identify_hardware(adapter); /* Setup PCI resources */ if (em_allocate_pci_resources(adapter)) { device_printf(dev, "Allocation of PCI resources failed\n"); error = ENXIO; goto err_pci; } /* ** For ICH8 and family we need to ** map the flash memory, and this ** must happen after the MAC is ** identified */ if ((adapter->hw.mac.type == e1000_ich8lan) || (adapter->hw.mac.type == e1000_pchlan) || (adapter->hw.mac.type == e1000_ich9lan) || (adapter->hw.mac.type == e1000_ich10lan)) { int rid = EM_BAR_TYPE_FLASH; adapter->flash = bus_alloc_resource_any(dev, SYS_RES_MEMORY, &rid, RF_ACTIVE); if (adapter->flash == NULL) { device_printf(dev, "Mapping of Flash failed\n"); error = ENXIO; goto err_pci; } /* This is used in the shared code */ adapter->hw.flash_address = (u8 *)adapter->flash; adapter->osdep.flash_bus_space_tag = rman_get_bustag(adapter->flash); adapter->osdep.flash_bus_space_handle = rman_get_bushandle(adapter->flash); } /* Do Shared Code initialization */ if (e1000_setup_init_funcs(&adapter->hw, TRUE)) { device_printf(dev, "Setup of Shared code failed\n"); error = ENXIO; goto err_pci; } e1000_get_bus_info(&adapter->hw); /* Set up some sysctls for the tunable interrupt delays */ em_add_int_delay_sysctl(adapter, "rx_int_delay", "receive interrupt delay in usecs", &adapter->rx_int_delay, E1000_REGISTER(&adapter->hw, E1000_RDTR), em_rx_int_delay_dflt); em_add_int_delay_sysctl(adapter, "tx_int_delay", "transmit interrupt delay in usecs", &adapter->tx_int_delay, E1000_REGISTER(&adapter->hw, E1000_TIDV), em_tx_int_delay_dflt); em_add_int_delay_sysctl(adapter, "rx_abs_int_delay", "receive interrupt delay limit in usecs", &adapter->rx_abs_int_delay, E1000_REGISTER(&adapter->hw, E1000_RADV), em_rx_abs_int_delay_dflt); em_add_int_delay_sysctl(adapter, "tx_abs_int_delay", "transmit interrupt delay limit in usecs", &adapter->tx_abs_int_delay, E1000_REGISTER(&adapter->hw, E1000_TADV), em_tx_abs_int_delay_dflt); /* Sysctls for limiting the amount of work done in the taskqueue */ em_add_rx_process_limit(adapter, "rx_processing_limit", "max number of rx packets to process", &adapter->rx_process_limit, em_rx_process_limit); /* * Validate number of transmit and receive descriptors. It * must not exceed hardware maximum, and must be multiple * of E1000_DBA_ALIGN. */ if (((em_txd * sizeof(struct e1000_tx_desc)) % EM_DBA_ALIGN) != 0 || (em_txd > EM_MAX_TXD) || (em_txd < EM_MIN_TXD)) { device_printf(dev, "Using %d TX descriptors instead of %d!\n", EM_DEFAULT_TXD, em_txd); adapter->num_tx_desc = EM_DEFAULT_TXD; } else adapter->num_tx_desc = em_txd; if (((em_rxd * sizeof(struct e1000_rx_desc)) % EM_DBA_ALIGN) != 0 || (em_rxd > EM_MAX_RXD) || (em_rxd < EM_MIN_RXD)) { device_printf(dev, "Using %d RX descriptors instead of %d!\n", EM_DEFAULT_RXD, em_rxd); adapter->num_rx_desc = EM_DEFAULT_RXD; } else adapter->num_rx_desc = em_rxd; adapter->hw.mac.autoneg = DO_AUTO_NEG; adapter->hw.phy.autoneg_wait_to_complete = FALSE; adapter->hw.phy.autoneg_advertised = AUTONEG_ADV_DEFAULT; /* Copper options */ if (adapter->hw.phy.media_type == e1000_media_type_copper) { adapter->hw.phy.mdix = AUTO_ALL_MODES; adapter->hw.phy.disable_polarity_correction = FALSE; adapter->hw.phy.ms_type = EM_MASTER_SLAVE; } /* * Set the frame limits assuming * standard ethernet sized frames. */ adapter->max_frame_size = ETHERMTU + ETHER_HDR_LEN + ETHERNET_FCS_SIZE; adapter->min_frame_size = ETH_ZLEN + ETHERNET_FCS_SIZE; /* * This controls when hardware reports transmit completion * status. */ adapter->hw.mac.report_tx_early = 1; /* ** Get queue/ring memory */ if (em_allocate_queues(adapter)) { error = ENOMEM; goto err_pci; } /* ** Start from a known state, this is ** important in reading the nvm and ** mac from that. */ e1000_reset_hw(&adapter->hw); /* Make sure we have a good EEPROM before we read from it */ if (e1000_validate_nvm_checksum(&adapter->hw) < 0) { /* ** Some PCI-E parts fail the first check due to ** the link being in sleep state, call it again, ** if it fails a second time its a real issue. */ if (e1000_validate_nvm_checksum(&adapter->hw) < 0) { device_printf(dev, "The EEPROM Checksum Is Not Valid\n"); error = EIO; goto err_late; } } /* Copy the permanent MAC address out of the EEPROM */ if (e1000_read_mac_addr(&adapter->hw) < 0) { device_printf(dev, "EEPROM read error while reading MAC" " address\n"); error = EIO; goto err_late; } if (!em_is_valid_ether_addr(adapter->hw.mac.addr)) { device_printf(dev, "Invalid MAC address\n"); error = EIO; goto err_late; } /* ** Do interrupt configuration */ if (adapter->msix > 1) /* Do MSIX */ error = em_allocate_msix(adapter); else /* MSI or Legacy */ error = em_allocate_legacy(adapter); if (error) goto err_late; /* * Get Wake-on-Lan and Management info for later use */ em_get_wakeup(dev); /* Setup OS specific network interface */ em_setup_interface(dev, adapter); em_reset(adapter); /* Initialize statistics */ em_update_stats_counters(adapter); adapter->hw.mac.get_link_status = 1; em_update_link_status(adapter); /* Indicate SOL/IDER usage */ if (e1000_check_reset_block(&adapter->hw)) device_printf(dev, "PHY reset is blocked due to SOL/IDER session.\n"); /* Register for VLAN events */ adapter->vlan_attach = EVENTHANDLER_REGISTER(vlan_config, em_register_vlan, adapter, EVENTHANDLER_PRI_FIRST); adapter->vlan_detach = EVENTHANDLER_REGISTER(vlan_unconfig, em_unregister_vlan, adapter, EVENTHANDLER_PRI_FIRST); /* Non-AMT based hardware can now take control from firmware */ if (adapter->has_manage && !adapter->has_amt) em_get_hw_control(adapter); /* Tell the stack that the interface is not active */ adapter->ifp->if_drv_flags &= ~(IFF_DRV_RUNNING | IFF_DRV_OACTIVE); adapter->led_dev = led_create(em_led_func, adapter, device_get_nameunit(dev)); INIT_DEBUGOUT("em_attach: end"); return (0); err_late: em_free_transmit_structures(adapter); em_free_receive_structures(adapter); em_release_hw_control(adapter); err_pci: em_free_pci_resources(adapter); EM_CORE_LOCK_DESTROY(adapter); return (error); } /********************************************************************* * Device removal routine * * The detach entry point is called when the driver is being removed. * This routine stops the adapter and deallocates all the resources * that were allocated for driver operation. * * return 0 on success, positive on failure *********************************************************************/ static int em_detach(device_t dev) { struct adapter *adapter = device_get_softc(dev); struct ifnet *ifp = adapter->ifp; INIT_DEBUGOUT("em_detach: begin"); /* Make sure VLANS are not using driver */ if (adapter->ifp->if_vlantrunk != NULL) { device_printf(dev,"Vlan in use, detach first\n"); return (EBUSY); } #ifdef DEVICE_POLLING if (ifp->if_capenable & IFCAP_POLLING) ether_poll_deregister(ifp); #endif EM_CORE_LOCK(adapter); adapter->in_detach = 1; em_stop(adapter); EM_CORE_UNLOCK(adapter); EM_CORE_LOCK_DESTROY(adapter); e1000_phy_hw_reset(&adapter->hw); em_release_manageability(adapter); em_release_hw_control(adapter); /* Unregister VLAN events */ if (adapter->vlan_attach != NULL) EVENTHANDLER_DEREGISTER(vlan_config, adapter->vlan_attach); if (adapter->vlan_detach != NULL) EVENTHANDLER_DEREGISTER(vlan_unconfig, adapter->vlan_detach); ether_ifdetach(adapter->ifp); callout_drain(&adapter->timer); em_free_pci_resources(adapter); bus_generic_detach(dev); if_free(ifp); em_free_transmit_structures(adapter); em_free_receive_structures(adapter); em_release_hw_control(adapter); return (0); } /********************************************************************* * * Shutdown entry point * **********************************************************************/ static int em_shutdown(device_t dev) { return em_suspend(dev); } /* * Suspend/resume device methods. */ static int em_suspend(device_t dev) { struct adapter *adapter = device_get_softc(dev); EM_CORE_LOCK(adapter); em_release_manageability(adapter); em_release_hw_control(adapter); em_enable_wakeup(dev); EM_CORE_UNLOCK(adapter); return bus_generic_suspend(dev); } static int em_resume(device_t dev) { struct adapter *adapter = device_get_softc(dev); struct ifnet *ifp = adapter->ifp; if (adapter->led_dev != NULL) led_destroy(adapter->led_dev); EM_CORE_LOCK(adapter); em_init_locked(adapter); em_init_manageability(adapter); EM_CORE_UNLOCK(adapter); em_start(ifp); return bus_generic_resume(dev); } /********************************************************************* * Transmit entry point * * em_start is called by the stack to initiate a transmit. * The driver will remain in this routine as long as there are * packets to transmit and transmit resources are available. * In case resources are not available stack is notified and * the packet is requeued. **********************************************************************/ #ifdef EM_MULTIQUEUE static int em_mq_start_locked(struct ifnet *ifp, struct tx_ring *txr, struct mbuf *m) { struct adapter *adapter = txr->adapter; struct mbuf *next; int err = 0, enq = 0; if ((ifp->if_drv_flags & (IFF_DRV_RUNNING | IFF_DRV_OACTIVE)) != IFF_DRV_RUNNING || adapter->link_active == 0) { if (m != NULL) err = drbr_enqueue(ifp, txr->br, m); return (err); } /* Call cleanup if number of TX descriptors low */ if (txr->tx_avail <= EM_TX_CLEANUP_THRESHOLD) em_txeof(txr); enq = 0; if (m == NULL) { next = drbr_dequeue(ifp, txr->br); } else if (drbr_needs_enqueue(ifp, txr->br)) { if ((err = drbr_enqueue(ifp, txr->br, m)) != 0) return (err); next = drbr_dequeue(ifp, txr->br); } else next = m; /* Process the queue */ while (next != NULL) { if ((err = em_xmit(txr, &next)) != 0) { if (next != NULL) err = drbr_enqueue(ifp, txr->br, next); break; } enq++; drbr_stats_update(ifp, next->m_pkthdr.len, next->m_flags); ETHER_BPF_MTAP(ifp, next); if ((ifp->if_drv_flags & IFF_DRV_RUNNING) == 0) break; if (txr->tx_avail < EM_MAX_SCATTER) { ifp->if_drv_flags |= IFF_DRV_OACTIVE; break; } next = drbr_dequeue(ifp, txr->br); } if (enq > 0) { /* Set the watchdog */ txr->watchdog_check = TRUE; txr->watchdog_time = ticks; } return (err); } /* ** Multiqueue capable stack interface, this is not ** yet truely multiqueue, but that is coming... */ static int em_mq_start(struct ifnet *ifp, struct mbuf *m) { struct adapter *adapter = ifp->if_softc; struct tx_ring *txr; int i, error = 0; /* Which queue to use */ if ((m->m_flags & M_FLOWID) != 0) i = m->m_pkthdr.flowid % adapter->num_queues; else i = curcpu % adapter->num_queues; txr = &adapter->tx_rings[i]; if (EM_TX_TRYLOCK(txr)) { error = em_mq_start_locked(ifp, txr, m); EM_TX_UNLOCK(txr); } else error = drbr_enqueue(ifp, txr->br, m); return (error); } /* ** Flush all ring buffers */ static void em_qflush(struct ifnet *ifp) { struct adapter *adapter = ifp->if_softc; struct tx_ring *txr = adapter->tx_rings; struct mbuf *m; for (int i = 0; i < adapter->num_queues; i++, txr++) { EM_TX_LOCK(txr); while ((m = buf_ring_dequeue_sc(txr->br)) != NULL) m_freem(m); EM_TX_UNLOCK(txr); } if_qflush(ifp); } #endif /* EM_MULTIQUEUE */ static void em_start_locked(struct ifnet *ifp, struct tx_ring *txr) { struct adapter *adapter = ifp->if_softc; struct mbuf *m_head; EM_TX_LOCK_ASSERT(txr); if ((ifp->if_drv_flags & (IFF_DRV_RUNNING|IFF_DRV_OACTIVE)) != IFF_DRV_RUNNING) return; if (!adapter->link_active) return; /* Call cleanup if number of TX descriptors low */ if (txr->tx_avail <= EM_TX_CLEANUP_THRESHOLD) em_txeof(txr); while (!IFQ_DRV_IS_EMPTY(&ifp->if_snd)) { if (txr->tx_avail < EM_MAX_SCATTER) { ifp->if_drv_flags |= IFF_DRV_OACTIVE; break; } IFQ_DRV_DEQUEUE(&ifp->if_snd, m_head); if (m_head == NULL) break; /* * Encapsulation can modify our pointer, and or make it * NULL on failure. In that event, we can't requeue. */ if (em_xmit(txr, &m_head)) { if (m_head == NULL) break; ifp->if_drv_flags |= IFF_DRV_OACTIVE; IFQ_DRV_PREPEND(&ifp->if_snd, m_head); break; } /* Send a copy of the frame to the BPF listener */ ETHER_BPF_MTAP(ifp, m_head); /* Set timeout in case hardware has problems transmitting. */ txr->watchdog_time = ticks; txr->watchdog_check = TRUE; } return; } static void em_start(struct ifnet *ifp) { struct adapter *adapter = ifp->if_softc; struct tx_ring *txr = adapter->tx_rings; if (ifp->if_drv_flags & IFF_DRV_RUNNING) { EM_TX_LOCK(txr); em_start_locked(ifp, txr); EM_TX_UNLOCK(txr); } return; } /********************************************************************* * Ioctl entry point * * em_ioctl is called when the user wants to configure the * interface. * * return 0 on success, positive on failure **********************************************************************/ static int em_ioctl(struct ifnet *ifp, u_long command, caddr_t data) { struct adapter *adapter = ifp->if_softc; struct ifreq *ifr = (struct ifreq *)data; #ifdef INET struct ifaddr *ifa = (struct ifaddr *)data; #endif int error = 0; if (adapter->in_detach) return (error); switch (command) { case SIOCSIFADDR: #ifdef INET if (ifa->ifa_addr->sa_family == AF_INET) { /* * XXX * Since resetting hardware takes a very long time * and results in link renegotiation we only * initialize the hardware only when it is absolutely * required. */ ifp->if_flags |= IFF_UP; if (!(ifp->if_drv_flags & IFF_DRV_RUNNING)) { EM_CORE_LOCK(adapter); em_init_locked(adapter); EM_CORE_UNLOCK(adapter); } arp_ifinit(ifp, ifa); } else #endif error = ether_ioctl(ifp, command, data); break; case SIOCSIFMTU: { int max_frame_size; IOCTL_DEBUGOUT("ioctl rcv'd: SIOCSIFMTU (Set Interface MTU)"); EM_CORE_LOCK(adapter); switch (adapter->hw.mac.type) { case e1000_82571: case e1000_82572: case e1000_ich9lan: case e1000_ich10lan: case e1000_82574: case e1000_80003es2lan: /* 9K Jumbo Frame size */ max_frame_size = 9234; break; case e1000_pchlan: max_frame_size = 4096; break; /* Adapters that do not support jumbo frames */ case e1000_82583: case e1000_ich8lan: max_frame_size = ETHER_MAX_LEN; break; default: max_frame_size = MAX_JUMBO_FRAME_SIZE; } if (ifr->ifr_mtu > max_frame_size - ETHER_HDR_LEN - ETHER_CRC_LEN) { EM_CORE_UNLOCK(adapter); error = EINVAL; break; } ifp->if_mtu = ifr->ifr_mtu; adapter->max_frame_size = ifp->if_mtu + ETHER_HDR_LEN + ETHER_CRC_LEN; em_init_locked(adapter); EM_CORE_UNLOCK(adapter); break; } case SIOCSIFFLAGS: IOCTL_DEBUGOUT("ioctl rcv'd:\ SIOCSIFFLAGS (Set Interface Flags)"); EM_CORE_LOCK(adapter); if (ifp->if_flags & IFF_UP) { if ((ifp->if_drv_flags & IFF_DRV_RUNNING)) { if ((ifp->if_flags ^ adapter->if_flags) & (IFF_PROMISC | IFF_ALLMULTI)) { em_disable_promisc(adapter); em_set_promisc(adapter); } } else em_init_locked(adapter); } else if (ifp->if_drv_flags & IFF_DRV_RUNNING) em_stop(adapter); adapter->if_flags = ifp->if_flags; EM_CORE_UNLOCK(adapter); break; case SIOCADDMULTI: case SIOCDELMULTI: IOCTL_DEBUGOUT("ioctl rcv'd: SIOC(ADD|DEL)MULTI"); if (ifp->if_drv_flags & IFF_DRV_RUNNING) { EM_CORE_LOCK(adapter); em_disable_intr(adapter); em_set_multi(adapter); #ifdef DEVICE_POLLING if (!(ifp->if_capenable & IFCAP_POLLING)) #endif em_enable_intr(adapter); EM_CORE_UNLOCK(adapter); } break; case SIOCSIFMEDIA: /* Check SOL/IDER usage */ EM_CORE_LOCK(adapter); if (e1000_check_reset_block(&adapter->hw)) { EM_CORE_UNLOCK(adapter); device_printf(adapter->dev, "Media change is" " blocked due to SOL/IDER session.\n"); break; } EM_CORE_UNLOCK(adapter); case SIOCGIFMEDIA: IOCTL_DEBUGOUT("ioctl rcv'd: \ SIOCxIFMEDIA (Get/Set Interface Media)"); error = ifmedia_ioctl(ifp, ifr, &adapter->media, command); break; case SIOCSIFCAP: { int mask, reinit; IOCTL_DEBUGOUT("ioctl rcv'd: SIOCSIFCAP (Set Capabilities)"); reinit = 0; mask = ifr->ifr_reqcap ^ ifp->if_capenable; #ifdef DEVICE_POLLING if (mask & IFCAP_POLLING) { if (ifr->ifr_reqcap & IFCAP_POLLING) { error = ether_poll_register(em_poll, ifp); if (error) return (error); EM_CORE_LOCK(adapter); em_disable_intr(adapter); ifp->if_capenable |= IFCAP_POLLING; EM_CORE_UNLOCK(adapter); } else { error = ether_poll_deregister(ifp); /* Enable interrupt even in error case */ EM_CORE_LOCK(adapter); em_enable_intr(adapter); ifp->if_capenable &= ~IFCAP_POLLING; EM_CORE_UNLOCK(adapter); } } #endif if (mask & IFCAP_HWCSUM) { ifp->if_capenable ^= IFCAP_HWCSUM; reinit = 1; } if (mask & IFCAP_TSO4) { ifp->if_capenable ^= IFCAP_TSO4; reinit = 1; } if (mask & IFCAP_VLAN_HWTAGGING) { ifp->if_capenable ^= IFCAP_VLAN_HWTAGGING; reinit = 1; } if (mask & IFCAP_VLAN_HWFILTER) { ifp->if_capenable ^= IFCAP_VLAN_HWFILTER; reinit = 1; } if ((mask & IFCAP_WOL) && (ifp->if_capabilities & IFCAP_WOL) != 0) { if (mask & IFCAP_WOL_MCAST) ifp->if_capenable ^= IFCAP_WOL_MCAST; if (mask & IFCAP_WOL_MAGIC) ifp->if_capenable ^= IFCAP_WOL_MAGIC; } if (reinit && (ifp->if_drv_flags & IFF_DRV_RUNNING)) em_init(adapter); VLAN_CAPABILITIES(ifp); break; } default: error = ether_ioctl(ifp, command, data); break; } return (error); } /********************************************************************* * Init entry point * * This routine is used in two ways. It is used by the stack as * init entry point in network interface structure. It is also used * by the driver as a hw/sw initialization routine to get to a * consistent state. * * return 0 on success, positive on failure **********************************************************************/ static void em_init_locked(struct adapter *adapter) { struct ifnet *ifp = adapter->ifp; device_t dev = adapter->dev; u32 pba; INIT_DEBUGOUT("em_init: begin"); EM_CORE_LOCK_ASSERT(adapter); em_disable_intr(adapter); callout_stop(&adapter->timer); /* * Packet Buffer Allocation (PBA) * Writing PBA sets the receive portion of the buffer * the remainder is used for the transmit buffer. */ switch (adapter->hw.mac.type) { /* Total Packet Buffer on these is 48K */ case e1000_82571: case e1000_82572: case e1000_80003es2lan: pba = E1000_PBA_32K; /* 32K for Rx, 16K for Tx */ break; case e1000_82573: /* 82573: Total Packet Buffer is 32K */ pba = E1000_PBA_12K; /* 12K for Rx, 20K for Tx */ break; case e1000_82574: case e1000_82583: pba = E1000_PBA_20K; /* 20K for Rx, 20K for Tx */ break; case e1000_ich9lan: case e1000_ich10lan: case e1000_pchlan: pba = E1000_PBA_10K; break; case e1000_ich8lan: pba = E1000_PBA_8K; break; default: if (adapter->max_frame_size > 8192) pba = E1000_PBA_40K; /* 40K for Rx, 24K for Tx */ else pba = E1000_PBA_48K; /* 48K for Rx, 16K for Tx */ } INIT_DEBUGOUT1("em_init: pba=%dK",pba); E1000_WRITE_REG(&adapter->hw, E1000_PBA, pba); /* Get the latest mac address, User can use a LAA */ bcopy(IF_LLADDR(adapter->ifp), adapter->hw.mac.addr, ETHER_ADDR_LEN); /* Put the address into the Receive Address Array */ e1000_rar_set(&adapter->hw, adapter->hw.mac.addr, 0); /* * With the 82571 adapter, RAR[0] may be overwritten * when the other port is reset, we make a duplicate * in RAR[14] for that eventuality, this assures * the interface continues to function. */ if (adapter->hw.mac.type == e1000_82571) { e1000_set_laa_state_82571(&adapter->hw, TRUE); e1000_rar_set(&adapter->hw, adapter->hw.mac.addr, E1000_RAR_ENTRIES - 1); } /* Initialize the hardware */ em_reset(adapter); em_update_link_status(adapter); /* Setup VLAN support, basic and offload if available */ E1000_WRITE_REG(&adapter->hw, E1000_VET, ETHERTYPE_VLAN); /* Use real VLAN Filter support? */ if (ifp->if_capenable & IFCAP_VLAN_HWTAGGING) { if (ifp->if_capenable & IFCAP_VLAN_HWFILTER) /* Use real VLAN Filter support */ em_setup_vlan_hw_support(adapter); else { u32 ctrl; ctrl = E1000_READ_REG(&adapter->hw, E1000_CTRL); ctrl |= E1000_CTRL_VME; E1000_WRITE_REG(&adapter->hw, E1000_CTRL, ctrl); } } /* Set hardware offload abilities */ ifp->if_hwassist = 0; if (ifp->if_capenable & IFCAP_TXCSUM) ifp->if_hwassist |= (CSUM_TCP | CSUM_UDP); if (ifp->if_capenable & IFCAP_TSO4) ifp->if_hwassist |= CSUM_TSO; /* Configure for OS presence */ em_init_manageability(adapter); /* Prepare transmit descriptors and buffers */ em_setup_transmit_structures(adapter); em_initialize_transmit_unit(adapter); /* Setup Multicast table */ em_set_multi(adapter); /* Prepare receive descriptors and buffers */ if (em_setup_receive_structures(adapter)) { device_printf(dev, "Could not setup receive structures\n"); em_stop(adapter); return; } em_initialize_receive_unit(adapter); /* Don't lose promiscuous settings */ em_set_promisc(adapter); ifp->if_drv_flags |= IFF_DRV_RUNNING; ifp->if_drv_flags &= ~IFF_DRV_OACTIVE; callout_reset(&adapter->timer, hz, em_local_timer, adapter); e1000_clear_hw_cntrs_base_generic(&adapter->hw); /* MSI/X configuration for 82574 */ if (adapter->hw.mac.type == e1000_82574) { int tmp; tmp = E1000_READ_REG(&adapter->hw, E1000_CTRL_EXT); tmp |= E1000_CTRL_EXT_PBA_CLR; E1000_WRITE_REG(&adapter->hw, E1000_CTRL_EXT, tmp); /* Set the IVAR - interrupt vector routing. */ E1000_WRITE_REG(&adapter->hw, E1000_IVAR, adapter->ivars); } #ifdef DEVICE_POLLING /* * Only enable interrupts if we are not polling, make sure * they are off otherwise. */ if (ifp->if_capenable & IFCAP_POLLING) em_disable_intr(adapter); else #endif /* DEVICE_POLLING */ em_enable_intr(adapter); /* AMT based hardware can now take control from firmware */ if (adapter->has_manage && adapter->has_amt) em_get_hw_control(adapter); /* Don't reset the phy next time init gets called */ adapter->hw.phy.reset_disable = TRUE; } static void em_init(void *arg) { struct adapter *adapter = arg; EM_CORE_LOCK(adapter); em_init_locked(adapter); EM_CORE_UNLOCK(adapter); } #ifdef DEVICE_POLLING /********************************************************************* * * Legacy polling routine: note this only works with single queue * *********************************************************************/ static int em_poll(struct ifnet *ifp, enum poll_cmd cmd, int count) { struct adapter *adapter = ifp->if_softc; struct tx_ring *txr = adapter->tx_rings; struct rx_ring *rxr = adapter->rx_rings; u32 reg_icr, rx_done = 0; EM_CORE_LOCK(adapter); if ((ifp->if_drv_flags & IFF_DRV_RUNNING) == 0) { EM_CORE_UNLOCK(adapter); return (rx_done); } if (cmd == POLL_AND_CHECK_STATUS) { reg_icr = E1000_READ_REG(&adapter->hw, E1000_ICR); if (reg_icr & (E1000_ICR_RXSEQ | E1000_ICR_LSC)) { callout_stop(&adapter->timer); adapter->hw.mac.get_link_status = 1; em_update_link_status(adapter); callout_reset(&adapter->timer, hz, em_local_timer, adapter); } } EM_CORE_UNLOCK(adapter); EM_RX_LOCK(rxr); rx_done = em_rxeof(rxr, count); EM_RX_UNLOCK(rxr); EM_TX_LOCK(txr); em_txeof(txr); #ifdef EM_MULTIQUEUE if (!drbr_empty(ifp, txr->br)) em_mq_start_locked(ifp, txr, NULL); #else if (!IFQ_DRV_IS_EMPTY(&ifp->if_snd)) em_start_locked(ifp, txr); #endif EM_TX_UNLOCK(txr); return (rx_done); } #endif /* DEVICE_POLLING */ /********************************************************************* * * Fast Legacy/MSI Combined Interrupt Service routine * *********************************************************************/ static int em_irq_fast(void *arg) { struct adapter *adapter = arg; struct ifnet *ifp; u32 reg_icr; ifp = adapter->ifp; reg_icr = E1000_READ_REG(&adapter->hw, E1000_ICR); /* Hot eject? */ if (reg_icr == 0xffffffff) return FILTER_STRAY; /* Definitely not our interrupt. */ if (reg_icr == 0x0) return FILTER_STRAY; /* * Starting with the 82571 chip, bit 31 should be used to * determine whether the interrupt belongs to us. */ if (adapter->hw.mac.type >= e1000_82571 && (reg_icr & E1000_ICR_INT_ASSERTED) == 0) return FILTER_STRAY; em_disable_intr(adapter); taskqueue_enqueue(adapter->tq, &adapter->que_task); /* Link status change */ if (reg_icr & (E1000_ICR_RXSEQ | E1000_ICR_LSC)) { adapter->hw.mac.get_link_status = 1; taskqueue_enqueue(taskqueue_fast, &adapter->link_task); } if (reg_icr & E1000_ICR_RXO) adapter->rx_overruns++; return FILTER_HANDLED; } /* Combined RX/TX handler, used by Legacy and MSI */ static void em_handle_que(void *context, int pending) { struct adapter *adapter = context; struct ifnet *ifp = adapter->ifp; struct tx_ring *txr = adapter->tx_rings; struct rx_ring *rxr = adapter->rx_rings; bool more_rx; if (ifp->if_drv_flags & IFF_DRV_RUNNING) { EM_RX_LOCK(rxr); more_rx = em_rxeof(rxr, adapter->rx_process_limit); EM_RX_UNLOCK(rxr); EM_TX_LOCK(txr); em_txeof(txr); #ifdef EM_MULTIQUEUE if (!drbr_empty(ifp, txr->br)) em_mq_start_locked(ifp, txr, NULL); #else if (!IFQ_DRV_IS_EMPTY(&ifp->if_snd)) em_start_locked(ifp, txr); #endif EM_TX_UNLOCK(txr); if (more_rx) { taskqueue_enqueue(adapter->tq, &adapter->que_task); return; } } em_enable_intr(adapter); return; } /********************************************************************* * * MSIX Interrupt Service Routines * **********************************************************************/ static void em_msix_tx(void *arg) { struct tx_ring *txr = arg; struct adapter *adapter = txr->adapter; + bool more; ++txr->tx_irq; EM_TX_LOCK(txr); - em_txeof(txr); + more = em_txeof(txr); EM_TX_UNLOCK(txr); - E1000_WRITE_REG(&adapter->hw, E1000_IMS, txr->ims); + if (more) + taskqueue_enqueue(txr->tq, &txr->tx_task); + else + /* Reenable this interrupt */ + E1000_WRITE_REG(&adapter->hw, E1000_IMS, txr->ims); return; } /********************************************************************* * * MSIX RX Interrupt Service routine * **********************************************************************/ static void em_msix_rx(void *arg) { struct rx_ring *rxr = arg; struct adapter *adapter = rxr->adapter; bool more; EM_RX_LOCK(rxr); ++rxr->rx_irq; more = em_rxeof(rxr, adapter->rx_process_limit); EM_RX_UNLOCK(rxr); if (more) taskqueue_enqueue(rxr->tq, &rxr->rx_task); else /* Reenable this interrupt */ E1000_WRITE_REG(&adapter->hw, E1000_IMS, rxr->ims); return; } /********************************************************************* * * MSIX Link Fast Interrupt Service routine * **********************************************************************/ static void em_msix_link(void *arg) { struct adapter *adapter = arg; u32 reg_icr; ++adapter->link_irq; reg_icr = E1000_READ_REG(&adapter->hw, E1000_ICR); if (reg_icr & (E1000_ICR_RXSEQ | E1000_ICR_LSC)) { adapter->hw.mac.get_link_status = 1; taskqueue_enqueue(taskqueue_fast, &adapter->link_task); } else E1000_WRITE_REG(&adapter->hw, E1000_IMS, EM_MSIX_LINK | E1000_IMS_LSC); return; } static void em_handle_rx(void *context, int pending) { struct rx_ring *rxr = context; struct adapter *adapter = rxr->adapter; bool more; EM_RX_LOCK(rxr); more = em_rxeof(rxr, adapter->rx_process_limit); EM_RX_UNLOCK(rxr); if (more) taskqueue_enqueue(rxr->tq, &rxr->rx_task); else /* Reenable this interrupt */ E1000_WRITE_REG(&adapter->hw, E1000_IMS, rxr->ims); } static void em_handle_tx(void *context, int pending) { struct tx_ring *txr = context; struct adapter *adapter = txr->adapter; struct ifnet *ifp = adapter->ifp; if (!EM_TX_TRYLOCK(txr)) return; em_txeof(txr); #ifdef EM_MULTIQUEUE if (!drbr_empty(ifp, txr->br)) em_mq_start_locked(ifp, txr, NULL); #else if (!IFQ_DRV_IS_EMPTY(&ifp->if_snd)) em_start_locked(ifp, txr); #endif E1000_WRITE_REG(&adapter->hw, E1000_IMS, txr->ims); EM_TX_UNLOCK(txr); } static void em_handle_link(void *context, int pending) { struct adapter *adapter = context; struct ifnet *ifp = adapter->ifp; if (!(ifp->if_drv_flags & IFF_DRV_RUNNING)) return; EM_CORE_LOCK(adapter); callout_stop(&adapter->timer); em_update_link_status(adapter); callout_reset(&adapter->timer, hz, em_local_timer, adapter); E1000_WRITE_REG(&adapter->hw, E1000_IMS, EM_MSIX_LINK | E1000_IMS_LSC); EM_CORE_UNLOCK(adapter); } /********************************************************************* * * Media Ioctl callback * * This routine is called whenever the user queries the status of * the interface using ifconfig. * **********************************************************************/ static void em_media_status(struct ifnet *ifp, struct ifmediareq *ifmr) { struct adapter *adapter = ifp->if_softc; u_char fiber_type = IFM_1000_SX; INIT_DEBUGOUT("em_media_status: begin"); EM_CORE_LOCK(adapter); em_update_link_status(adapter); ifmr->ifm_status = IFM_AVALID; ifmr->ifm_active = IFM_ETHER; if (!adapter->link_active) { EM_CORE_UNLOCK(adapter); return; } ifmr->ifm_status |= IFM_ACTIVE; if ((adapter->hw.phy.media_type == e1000_media_type_fiber) || (adapter->hw.phy.media_type == e1000_media_type_internal_serdes)) { ifmr->ifm_active |= fiber_type | IFM_FDX; } else { switch (adapter->link_speed) { case 10: ifmr->ifm_active |= IFM_10_T; break; case 100: ifmr->ifm_active |= IFM_100_TX; break; case 1000: ifmr->ifm_active |= IFM_1000_T; break; } if (adapter->link_duplex == FULL_DUPLEX) ifmr->ifm_active |= IFM_FDX; else ifmr->ifm_active |= IFM_HDX; } EM_CORE_UNLOCK(adapter); } /********************************************************************* * * Media Ioctl callback * * This routine is called when the user changes speed/duplex using * media/mediopt option with ifconfig. * **********************************************************************/ static int em_media_change(struct ifnet *ifp) { struct adapter *adapter = ifp->if_softc; struct ifmedia *ifm = &adapter->media; INIT_DEBUGOUT("em_media_change: begin"); if (IFM_TYPE(ifm->ifm_media) != IFM_ETHER) return (EINVAL); EM_CORE_LOCK(adapter); switch (IFM_SUBTYPE(ifm->ifm_media)) { case IFM_AUTO: adapter->hw.mac.autoneg = DO_AUTO_NEG; adapter->hw.phy.autoneg_advertised = AUTONEG_ADV_DEFAULT; break; case IFM_1000_LX: case IFM_1000_SX: case IFM_1000_T: adapter->hw.mac.autoneg = DO_AUTO_NEG; adapter->hw.phy.autoneg_advertised = ADVERTISE_1000_FULL; break; case IFM_100_TX: adapter->hw.mac.autoneg = FALSE; adapter->hw.phy.autoneg_advertised = 0; if ((ifm->ifm_media & IFM_GMASK) == IFM_FDX) adapter->hw.mac.forced_speed_duplex = ADVERTISE_100_FULL; else adapter->hw.mac.forced_speed_duplex = ADVERTISE_100_HALF; break; case IFM_10_T: adapter->hw.mac.autoneg = FALSE; adapter->hw.phy.autoneg_advertised = 0; if ((ifm->ifm_media & IFM_GMASK) == IFM_FDX) adapter->hw.mac.forced_speed_duplex = ADVERTISE_10_FULL; else adapter->hw.mac.forced_speed_duplex = ADVERTISE_10_HALF; break; default: device_printf(adapter->dev, "Unsupported media type\n"); } /* As the speed/duplex settings my have changed we need to * reset the PHY. */ adapter->hw.phy.reset_disable = FALSE; em_init_locked(adapter); EM_CORE_UNLOCK(adapter); return (0); } /********************************************************************* * * This routine maps the mbufs to tx descriptors. * * return 0 on success, positive on failure **********************************************************************/ static int em_xmit(struct tx_ring *txr, struct mbuf **m_headp) { struct adapter *adapter = txr->adapter; bus_dma_segment_t segs[EM_MAX_SCATTER]; bus_dmamap_t map; struct em_buffer *tx_buffer, *tx_buffer_mapped; struct e1000_tx_desc *ctxd = NULL; struct mbuf *m_head; u32 txd_upper, txd_lower, txd_used, txd_saved; int nsegs, i, j, first, last = 0; int error, do_tso, tso_desc = 0; m_head = *m_headp; txd_upper = txd_lower = txd_used = txd_saved = 0; do_tso = ((m_head->m_pkthdr.csum_flags & CSUM_TSO) != 0); /* * TSO workaround: * If an mbuf is only header we need * to pull 4 bytes of data into it. */ if (do_tso && (m_head->m_len <= M_TSO_LEN)) { m_head = m_pullup(m_head, M_TSO_LEN + 4); *m_headp = m_head; if (m_head == NULL) return (ENOBUFS); } /* * Map the packet for DMA * * Capture the first descriptor index, * this descriptor will have the index * of the EOP which is the only one that * now gets a DONE bit writeback. */ first = txr->next_avail_desc; tx_buffer = &txr->tx_buffers[first]; tx_buffer_mapped = tx_buffer; map = tx_buffer->map; error = bus_dmamap_load_mbuf_sg(txr->txtag, map, *m_headp, segs, &nsegs, BUS_DMA_NOWAIT); /* * There are two types of errors we can (try) to handle: * - EFBIG means the mbuf chain was too long and bus_dma ran * out of segments. Defragment the mbuf chain and try again. * - ENOMEM means bus_dma could not obtain enough bounce buffers * at this point in time. Defer sending and try again later. * All other errors, in particular EINVAL, are fatal and prevent the * mbuf chain from ever going through. Drop it and report error. */ if (error == EFBIG) { struct mbuf *m; m = m_defrag(*m_headp, M_DONTWAIT); if (m == NULL) { adapter->mbuf_alloc_failed++; m_freem(*m_headp); *m_headp = NULL; return (ENOBUFS); } *m_headp = m; /* Try it again */ error = bus_dmamap_load_mbuf_sg(txr->txtag, map, *m_headp, segs, &nsegs, BUS_DMA_NOWAIT); if (error) { adapter->no_tx_dma_setup++; m_freem(*m_headp); *m_headp = NULL; return (error); } } else if (error != 0) { adapter->no_tx_dma_setup++; return (error); } /* * TSO Hardware workaround, if this packet is not * TSO, and is only a single descriptor long, and * it follows a TSO burst, then we need to add a * sentinel descriptor to prevent premature writeback. */ if ((do_tso == 0) && (txr->tx_tso == TRUE)) { if (nsegs == 1) tso_desc = TRUE; txr->tx_tso = FALSE; } if (nsegs > (txr->tx_avail - 2)) { txr->no_desc_avail++; bus_dmamap_unload(txr->txtag, map); return (ENOBUFS); } m_head = *m_headp; /* Do hardware assists */ #if __FreeBSD_version >= 700000 if (m_head->m_pkthdr.csum_flags & CSUM_TSO) { error = em_tso_setup(txr, m_head, &txd_upper, &txd_lower); if (error != TRUE) return (ENXIO); /* something foobar */ /* we need to make a final sentinel transmit desc */ tso_desc = TRUE; } else #endif if (m_head->m_pkthdr.csum_flags & CSUM_OFFLOAD) em_transmit_checksum_setup(txr, m_head, &txd_upper, &txd_lower); i = txr->next_avail_desc; /* Set up our transmit descriptors */ for (j = 0; j < nsegs; j++) { bus_size_t seg_len; bus_addr_t seg_addr; tx_buffer = &txr->tx_buffers[i]; ctxd = &txr->tx_base[i]; seg_addr = segs[j].ds_addr; seg_len = segs[j].ds_len; /* ** TSO Workaround: ** If this is the last descriptor, we want to ** split it so we have a small final sentinel */ if (tso_desc && (j == (nsegs -1)) && (seg_len > 8)) { seg_len -= 4; ctxd->buffer_addr = htole64(seg_addr); ctxd->lower.data = htole32( adapter->txd_cmd | txd_lower | seg_len); ctxd->upper.data = htole32(txd_upper); if (++i == adapter->num_tx_desc) i = 0; /* Now make the sentinel */ ++txd_used; /* using an extra txd */ ctxd = &txr->tx_base[i]; tx_buffer = &txr->tx_buffers[i]; ctxd->buffer_addr = htole64(seg_addr + seg_len); ctxd->lower.data = htole32( adapter->txd_cmd | txd_lower | 4); ctxd->upper.data = htole32(txd_upper); last = i; if (++i == adapter->num_tx_desc) i = 0; } else { ctxd->buffer_addr = htole64(seg_addr); ctxd->lower.data = htole32( adapter->txd_cmd | txd_lower | seg_len); ctxd->upper.data = htole32(txd_upper); last = i; if (++i == adapter->num_tx_desc) i = 0; } tx_buffer->m_head = NULL; tx_buffer->next_eop = -1; } txr->next_avail_desc = i; txr->tx_avail -= nsegs; if (tso_desc) /* TSO used an extra for sentinel */ txr->tx_avail -= txd_used; if (m_head->m_flags & M_VLANTAG) { /* Set the vlan id. */ ctxd->upper.fields.special = htole16(m_head->m_pkthdr.ether_vtag); /* Tell hardware to add tag */ ctxd->lower.data |= htole32(E1000_TXD_CMD_VLE); } tx_buffer->m_head = m_head; tx_buffer_mapped->map = tx_buffer->map; tx_buffer->map = map; bus_dmamap_sync(txr->txtag, map, BUS_DMASYNC_PREWRITE); /* * Last Descriptor of Packet * needs End Of Packet (EOP) * and Report Status (RS) */ ctxd->lower.data |= htole32(E1000_TXD_CMD_EOP | E1000_TXD_CMD_RS); /* * Keep track in the first buffer which * descriptor will be written back */ tx_buffer = &txr->tx_buffers[first]; tx_buffer->next_eop = last; /* * Advance the Transmit Descriptor Tail (TDT), this tells the E1000 * that this frame is available to transmit. */ bus_dmamap_sync(txr->txdma.dma_tag, txr->txdma.dma_map, BUS_DMASYNC_PREREAD | BUS_DMASYNC_PREWRITE); E1000_WRITE_REG(&adapter->hw, E1000_TDT(txr->me), i); return (0); } static void em_set_promisc(struct adapter *adapter) { struct ifnet *ifp = adapter->ifp; u32 reg_rctl; reg_rctl = E1000_READ_REG(&adapter->hw, E1000_RCTL); if (ifp->if_flags & IFF_PROMISC) { reg_rctl |= (E1000_RCTL_UPE | E1000_RCTL_MPE); /* Turn this on if you want to see bad packets */ if (em_debug_sbp) reg_rctl |= E1000_RCTL_SBP; E1000_WRITE_REG(&adapter->hw, E1000_RCTL, reg_rctl); } else if (ifp->if_flags & IFF_ALLMULTI) { reg_rctl |= E1000_RCTL_MPE; reg_rctl &= ~E1000_RCTL_UPE; E1000_WRITE_REG(&adapter->hw, E1000_RCTL, reg_rctl); } } static void em_disable_promisc(struct adapter *adapter) { u32 reg_rctl; reg_rctl = E1000_READ_REG(&adapter->hw, E1000_RCTL); reg_rctl &= (~E1000_RCTL_UPE); reg_rctl &= (~E1000_RCTL_MPE); reg_rctl &= (~E1000_RCTL_SBP); E1000_WRITE_REG(&adapter->hw, E1000_RCTL, reg_rctl); } /********************************************************************* * Multicast Update * * This routine is called whenever multicast address list is updated. * **********************************************************************/ static void em_set_multi(struct adapter *adapter) { struct ifnet *ifp = adapter->ifp; struct ifmultiaddr *ifma; u32 reg_rctl = 0; u8 *mta; /* Multicast array memory */ int mcnt = 0; IOCTL_DEBUGOUT("em_set_multi: begin"); if (adapter->hw.mac.type == e1000_82542 && adapter->hw.revision_id == E1000_REVISION_2) { reg_rctl = E1000_READ_REG(&adapter->hw, E1000_RCTL); if (adapter->hw.bus.pci_cmd_word & CMD_MEM_WRT_INVALIDATE) e1000_pci_clear_mwi(&adapter->hw); reg_rctl |= E1000_RCTL_RST; E1000_WRITE_REG(&adapter->hw, E1000_RCTL, reg_rctl); msec_delay(5); } /* Allocate temporary memory to setup array */ mta = malloc(sizeof(u8) * (ETH_ADDR_LEN * MAX_NUM_MULTICAST_ADDRESSES), M_DEVBUF, M_NOWAIT | M_ZERO); if (mta == NULL) panic("em_set_multi memory failure\n"); #if __FreeBSD_version < 800000 IF_ADDR_LOCK(ifp); #else if_maddr_rlock(ifp); #endif TAILQ_FOREACH(ifma, &ifp->if_multiaddrs, ifma_link) { if (ifma->ifma_addr->sa_family != AF_LINK) continue; if (mcnt == MAX_NUM_MULTICAST_ADDRESSES) break; bcopy(LLADDR((struct sockaddr_dl *)ifma->ifma_addr), &mta[mcnt * ETH_ADDR_LEN], ETH_ADDR_LEN); mcnt++; } #if __FreeBSD_version < 800000 IF_ADDR_UNLOCK(ifp); #else if_maddr_runlock(ifp); #endif if (mcnt >= MAX_NUM_MULTICAST_ADDRESSES) { reg_rctl = E1000_READ_REG(&adapter->hw, E1000_RCTL); reg_rctl |= E1000_RCTL_MPE; E1000_WRITE_REG(&adapter->hw, E1000_RCTL, reg_rctl); } else e1000_update_mc_addr_list(&adapter->hw, mta, mcnt); if (adapter->hw.mac.type == e1000_82542 && adapter->hw.revision_id == E1000_REVISION_2) { reg_rctl = E1000_READ_REG(&adapter->hw, E1000_RCTL); reg_rctl &= ~E1000_RCTL_RST; E1000_WRITE_REG(&adapter->hw, E1000_RCTL, reg_rctl); msec_delay(5); if (adapter->hw.bus.pci_cmd_word & CMD_MEM_WRT_INVALIDATE) e1000_pci_set_mwi(&adapter->hw); } free(mta, M_DEVBUF); } /********************************************************************* * Timer routine * * This routine checks for link status and updates statistics. * **********************************************************************/ static void em_local_timer(void *arg) { struct adapter *adapter = arg; struct ifnet *ifp = adapter->ifp; struct tx_ring *txr = adapter->tx_rings; EM_CORE_LOCK_ASSERT(adapter); em_update_link_status(adapter); em_update_stats_counters(adapter); /* Reset LAA into RAR[0] on 82571 */ if (e1000_get_laa_state_82571(&adapter->hw) == TRUE) e1000_rar_set(&adapter->hw, adapter->hw.mac.addr, 0); if (em_display_debug_stats && ifp->if_drv_flags & IFF_DRV_RUNNING) em_print_hw_stats(adapter); /* ** Check for time since any descriptor was cleaned */ for (int i = 0; i < adapter->num_queues; i++, txr++) { EM_TX_LOCK(txr); if (txr->watchdog_check == FALSE) { EM_TX_UNLOCK(txr); continue; } if ((ticks - txr->watchdog_time) > EM_WATCHDOG) goto hung; EM_TX_UNLOCK(txr); } callout_reset(&adapter->timer, hz, em_local_timer, adapter); return; hung: device_printf(adapter->dev, "Watchdog timeout -- resetting\n"); ifp->if_drv_flags &= ~IFF_DRV_RUNNING; adapter->watchdog_events++; EM_TX_UNLOCK(txr); em_init_locked(adapter); } static void em_update_link_status(struct adapter *adapter) { struct e1000_hw *hw = &adapter->hw; struct ifnet *ifp = adapter->ifp; device_t dev = adapter->dev; u32 link_check = 0; /* Get the cached link value or read phy for real */ switch (hw->phy.media_type) { case e1000_media_type_copper: if (hw->mac.get_link_status) { /* Do the work to read phy */ e1000_check_for_link(hw); link_check = !hw->mac.get_link_status; if (link_check) /* ESB2 fix */ e1000_cfg_on_link_up(hw); } else link_check = TRUE; break; case e1000_media_type_fiber: e1000_check_for_link(hw); link_check = (E1000_READ_REG(hw, E1000_STATUS) & E1000_STATUS_LU); break; case e1000_media_type_internal_serdes: e1000_check_for_link(hw); link_check = adapter->hw.mac.serdes_has_link; break; default: case e1000_media_type_unknown: break; } /* Now check for a transition */ if (link_check && (adapter->link_active == 0)) { e1000_get_speed_and_duplex(hw, &adapter->link_speed, &adapter->link_duplex); /* Check if we must disable SPEED_MODE bit on PCI-E */ if ((adapter->link_speed != SPEED_1000) && ((hw->mac.type == e1000_82571) || (hw->mac.type == e1000_82572))) { int tarc0; tarc0 = E1000_READ_REG(hw, E1000_TARC(0)); tarc0 &= ~SPEED_MODE_BIT; E1000_WRITE_REG(hw, E1000_TARC(0), tarc0); } if (bootverbose) device_printf(dev, "Link is up %d Mbps %s\n", adapter->link_speed, ((adapter->link_duplex == FULL_DUPLEX) ? "Full Duplex" : "Half Duplex")); adapter->link_active = 1; adapter->smartspeed = 0; ifp->if_baudrate = adapter->link_speed * 1000000; if_link_state_change(ifp, LINK_STATE_UP); } else if (!link_check && (adapter->link_active == 1)) { ifp->if_baudrate = adapter->link_speed = 0; adapter->link_duplex = 0; if (bootverbose) device_printf(dev, "Link is Down\n"); adapter->link_active = 0; /* Link down, disable watchdog */ // JFV change later //adapter->watchdog_check = FALSE; if_link_state_change(ifp, LINK_STATE_DOWN); } } /********************************************************************* * * This routine disables all traffic on the adapter by issuing a * global reset on the MAC and deallocates TX/RX buffers. * * This routine should always be called with BOTH the CORE * and TX locks. **********************************************************************/ static void em_stop(void *arg) { struct adapter *adapter = arg; struct ifnet *ifp = adapter->ifp; struct tx_ring *txr = adapter->tx_rings; EM_CORE_LOCK_ASSERT(adapter); INIT_DEBUGOUT("em_stop: begin"); em_disable_intr(adapter); callout_stop(&adapter->timer); /* Tell the stack that the interface is no longer active */ ifp->if_drv_flags &= ~(IFF_DRV_RUNNING | IFF_DRV_OACTIVE); /* Unarm watchdog timer. */ for (int i = 0; i < adapter->num_queues; i++, txr++) { EM_TX_LOCK(txr); txr->watchdog_check = FALSE; EM_TX_UNLOCK(txr); } e1000_reset_hw(&adapter->hw); E1000_WRITE_REG(&adapter->hw, E1000_WUC, 0); e1000_led_off(&adapter->hw); e1000_cleanup_led(&adapter->hw); } /********************************************************************* * * Determine hardware revision. * **********************************************************************/ static void em_identify_hardware(struct adapter *adapter) { device_t dev = adapter->dev; /* Make sure our PCI config space has the necessary stuff set */ adapter->hw.bus.pci_cmd_word = pci_read_config(dev, PCIR_COMMAND, 2); if (!((adapter->hw.bus.pci_cmd_word & PCIM_CMD_BUSMASTEREN) && (adapter->hw.bus.pci_cmd_word & PCIM_CMD_MEMEN))) { device_printf(dev, "Memory Access and/or Bus Master bits " "were not set!\n"); adapter->hw.bus.pci_cmd_word |= (PCIM_CMD_BUSMASTEREN | PCIM_CMD_MEMEN); pci_write_config(dev, PCIR_COMMAND, adapter->hw.bus.pci_cmd_word, 2); } /* Save off the information about this board */ adapter->hw.vendor_id = pci_get_vendor(dev); adapter->hw.device_id = pci_get_device(dev); adapter->hw.revision_id = pci_read_config(dev, PCIR_REVID, 1); adapter->hw.subsystem_vendor_id = pci_read_config(dev, PCIR_SUBVEND_0, 2); adapter->hw.subsystem_device_id = pci_read_config(dev, PCIR_SUBDEV_0, 2); /* Do Shared Code Init and Setup */ if (e1000_set_mac_type(&adapter->hw)) { device_printf(dev, "Setup init failure\n"); return; } } static int em_allocate_pci_resources(struct adapter *adapter) { device_t dev = adapter->dev; int rid; rid = PCIR_BAR(0); adapter->memory = bus_alloc_resource_any(dev, SYS_RES_MEMORY, &rid, RF_ACTIVE); if (adapter->memory == NULL) { device_printf(dev, "Unable to allocate bus resource: memory\n"); return (ENXIO); } adapter->osdep.mem_bus_space_tag = rman_get_bustag(adapter->memory); adapter->osdep.mem_bus_space_handle = rman_get_bushandle(adapter->memory); adapter->hw.hw_addr = (u8 *)&adapter->osdep.mem_bus_space_handle; /* Default to a single queue */ adapter->num_queues = 1; /* * Setup MSI/X or MSI if PCI Express */ adapter->msix = em_setup_msix(adapter); adapter->hw.back = &adapter->osdep; return (0); } /********************************************************************* * * Setup the Legacy or MSI Interrupt handler * **********************************************************************/ int em_allocate_legacy(struct adapter *adapter) { device_t dev = adapter->dev; int error, rid = 0; /* Manually turn off all interrupts */ E1000_WRITE_REG(&adapter->hw, E1000_IMC, 0xffffffff); if (adapter->msix == 1) /* using MSI */ rid = 1; /* We allocate a single interrupt resource */ adapter->res = bus_alloc_resource_any(dev, SYS_RES_IRQ, &rid, RF_SHAREABLE | RF_ACTIVE); if (adapter->res == NULL) { device_printf(dev, "Unable to allocate bus resource: " "interrupt\n"); return (ENXIO); } /* * Allocate a fast interrupt and the associated * deferred processing contexts. */ TASK_INIT(&adapter->que_task, 0, em_handle_que, adapter); TASK_INIT(&adapter->link_task, 0, em_handle_link, adapter); adapter->tq = taskqueue_create_fast("em_taskq", M_NOWAIT, taskqueue_thread_enqueue, &adapter->tq); taskqueue_start_threads(&adapter->tq, 1, PI_NET, "%s taskq", device_get_nameunit(adapter->dev)); if ((error = bus_setup_intr(dev, adapter->res, INTR_TYPE_NET, em_irq_fast, NULL, adapter, &adapter->tag)) != 0) { device_printf(dev, "Failed to register fast interrupt " "handler: %d\n", error); taskqueue_free(adapter->tq); adapter->tq = NULL; return (error); } return (0); } /********************************************************************* * * Setup the MSIX Interrupt handlers * This is not really Multiqueue, rather * its just multiple interrupt vectors. * **********************************************************************/ int em_allocate_msix(struct adapter *adapter) { device_t dev = adapter->dev; struct tx_ring *txr = adapter->tx_rings; struct rx_ring *rxr = adapter->rx_rings; int error, rid, vector = 0; /* Make sure all interrupts are disabled */ E1000_WRITE_REG(&adapter->hw, E1000_IMC, 0xffffffff); /* First set up ring resources */ for (int i = 0; i < adapter->num_queues; i++, txr++, rxr++) { /* RX ring */ rid = vector + 1; rxr->res = bus_alloc_resource_any(dev, SYS_RES_IRQ, &rid, RF_ACTIVE); if (rxr->res == NULL) { device_printf(dev, "Unable to allocate bus resource: " "RX MSIX Interrupt %d\n", i); return (ENXIO); } if ((error = bus_setup_intr(dev, rxr->res, INTR_TYPE_NET | INTR_MPSAFE, NULL, em_msix_rx, rxr, &rxr->tag)) != 0) { device_printf(dev, "Failed to register RX handler"); return (error); } rxr->msix = vector++; /* NOTE increment vector for TX */ TASK_INIT(&rxr->rx_task, 0, em_handle_rx, rxr); rxr->tq = taskqueue_create_fast("em_rxq", M_NOWAIT, taskqueue_thread_enqueue, &rxr->tq); taskqueue_start_threads(&rxr->tq, 1, PI_NET, "%s rxq", device_get_nameunit(adapter->dev)); /* ** Set the bit to enable interrupt ** in E1000_IMS -- bits 20 and 21 ** are for RX0 and RX1, note this has ** NOTHING to do with the MSIX vector */ rxr->ims = 1 << (20 + i); adapter->ivars |= (8 | rxr->msix) << (i * 4); /* TX ring */ rid = vector + 1; txr->res = bus_alloc_resource_any(dev, SYS_RES_IRQ, &rid, RF_ACTIVE); if (txr->res == NULL) { device_printf(dev, "Unable to allocate bus resource: " "TX MSIX Interrupt %d\n", i); return (ENXIO); } if ((error = bus_setup_intr(dev, txr->res, INTR_TYPE_NET | INTR_MPSAFE, NULL, em_msix_tx, txr, &txr->tag)) != 0) { device_printf(dev, "Failed to register TX handler"); return (error); } txr->msix = vector++; /* Increment vector for next pass */ TASK_INIT(&txr->tx_task, 0, em_handle_tx, txr); txr->tq = taskqueue_create_fast("em_txq", M_NOWAIT, taskqueue_thread_enqueue, &txr->tq); taskqueue_start_threads(&txr->tq, 1, PI_NET, "%s txq", device_get_nameunit(adapter->dev)); /* ** Set the bit to enable interrupt ** in E1000_IMS -- bits 22 and 23 ** are for TX0 and TX1, note this has ** NOTHING to do with the MSIX vector */ txr->ims = 1 << (22 + i); adapter->ivars |= (8 | txr->msix) << (8 + (i * 4)); } /* Link interrupt */ ++rid; adapter->res = bus_alloc_resource_any(dev, SYS_RES_IRQ, &rid, RF_ACTIVE); if (!adapter->res) { device_printf(dev,"Unable to allocate " "bus resource: Link interrupt [%d]\n", rid); return (ENXIO); } /* Set the link handler function */ error = bus_setup_intr(dev, adapter->res, INTR_TYPE_NET | INTR_MPSAFE, NULL, em_msix_link, adapter, &adapter->tag); if (error) { adapter->res = NULL; device_printf(dev, "Failed to register LINK handler"); return (error); } adapter->linkvec = vector; adapter->ivars |= (8 | vector) << 16; adapter->ivars |= 0x80000000; TASK_INIT(&adapter->link_task, 0, em_handle_link, adapter); adapter->tq = taskqueue_create_fast("em_link", M_NOWAIT, taskqueue_thread_enqueue, &adapter->tq); taskqueue_start_threads(&adapter->tq, 1, PI_NET, "%s linkq", device_get_nameunit(adapter->dev)); return (0); } static void em_free_pci_resources(struct adapter *adapter) { device_t dev = adapter->dev; struct tx_ring *txr; struct rx_ring *rxr; int rid; /* ** Release all the queue interrupt resources: */ for (int i = 0; i < adapter->num_queues; i++) { txr = &adapter->tx_rings[i]; rxr = &adapter->rx_rings[i]; rid = txr->msix +1; if (txr->tag != NULL) { bus_teardown_intr(dev, txr->res, txr->tag); txr->tag = NULL; } if (txr->res != NULL) bus_release_resource(dev, SYS_RES_IRQ, rid, txr->res); rid = rxr->msix +1; if (rxr->tag != NULL) { bus_teardown_intr(dev, rxr->res, rxr->tag); rxr->tag = NULL; } if (rxr->res != NULL) bus_release_resource(dev, SYS_RES_IRQ, rid, rxr->res); } if (adapter->linkvec) /* we are doing MSIX */ rid = adapter->linkvec + 1; else (adapter->msix != 0) ? (rid = 1):(rid = 0); if (adapter->tag != NULL) { bus_teardown_intr(dev, adapter->res, adapter->tag); adapter->tag = NULL; } if (adapter->res != NULL) bus_release_resource(dev, SYS_RES_IRQ, rid, adapter->res); if (adapter->msix) pci_release_msi(dev); if (adapter->msix_mem != NULL) bus_release_resource(dev, SYS_RES_MEMORY, PCIR_BAR(EM_MSIX_BAR), adapter->msix_mem); if (adapter->memory != NULL) bus_release_resource(dev, SYS_RES_MEMORY, PCIR_BAR(0), adapter->memory); if (adapter->flash != NULL) bus_release_resource(dev, SYS_RES_MEMORY, EM_FLASH, adapter->flash); } /* * Setup MSI or MSI/X */ static int em_setup_msix(struct adapter *adapter) { device_t dev = adapter->dev; int val = 0; /* Setup MSI/X for Hartwell */ if ((adapter->hw.mac.type == e1000_82574) && (em_enable_msix == TRUE)) { /* Map the MSIX BAR */ int rid = PCIR_BAR(EM_MSIX_BAR); adapter->msix_mem = bus_alloc_resource_any(dev, SYS_RES_MEMORY, &rid, RF_ACTIVE); if (!adapter->msix_mem) { /* May not be enabled */ device_printf(adapter->dev, "Unable to map MSIX table \n"); goto msi; } val = pci_msix_count(dev); if (val != 5) { bus_release_resource(dev, SYS_RES_MEMORY, PCIR_BAR(EM_MSIX_BAR), adapter->msix_mem); adapter->msix_mem = NULL; device_printf(adapter->dev, "MSIX vectors wrong, using MSI \n"); goto msi; } if (em_msix_queues == 2) { val = 5; adapter->num_queues = 2; } else { val = 3; adapter->num_queues = 1; } if (pci_alloc_msix(dev, &val) == 0) { device_printf(adapter->dev, "Using MSIX interrupts " "with %d vectors\n", val); } return (val); } msi: val = pci_msi_count(dev); if (val == 1 && pci_alloc_msi(dev, &val) == 0) { adapter->msix = 1; device_printf(adapter->dev,"Using MSI interrupt\n"); return (val); } /* Should only happen due to manual invention */ device_printf(adapter->dev,"Setup MSIX failure\n"); return (0); } /********************************************************************* * * Initialize the hardware to a configuration * as specified by the adapter structure. * **********************************************************************/ static void em_reset(struct adapter *adapter) { device_t dev = adapter->dev; struct e1000_hw *hw = &adapter->hw; u16 rx_buffer_size; INIT_DEBUGOUT("em_reset: begin"); /* Set up smart power down as default off on newer adapters. */ if (!em_smart_pwr_down && (hw->mac.type == e1000_82571 || hw->mac.type == e1000_82572)) { u16 phy_tmp = 0; /* Speed up time to link by disabling smart power down. */ e1000_read_phy_reg(hw, IGP02E1000_PHY_POWER_MGMT, &phy_tmp); phy_tmp &= ~IGP02E1000_PM_SPD; e1000_write_phy_reg(hw, IGP02E1000_PHY_POWER_MGMT, phy_tmp); } /* * These parameters control the automatic generation (Tx) and * response (Rx) to Ethernet PAUSE frames. * - High water mark should allow for at least two frames to be * received after sending an XOFF. * - Low water mark works best when it is very near the high water mark. * This allows the receiver to restart by sending XON when it has * drained a bit. Here we use an arbitary value of 1500 which will * restart after one full frame is pulled from the buffer. There * could be several smaller frames in the buffer and if so they will * not trigger the XON until their total number reduces the buffer * by 1500. * - The pause time is fairly large at 1000 x 512ns = 512 usec. */ rx_buffer_size = ((E1000_READ_REG(hw, E1000_PBA) & 0xffff) << 10 ); hw->fc.high_water = rx_buffer_size - roundup2(adapter->max_frame_size, 1024); hw->fc.low_water = hw->fc.high_water - 1500; if (hw->mac.type == e1000_80003es2lan) hw->fc.pause_time = 0xFFFF; else hw->fc.pause_time = EM_FC_PAUSE_TIME; hw->fc.send_xon = TRUE; /* Set Flow control, use the tunable location if sane */ if ((em_fc_setting >= 0) || (em_fc_setting < 4)) hw->fc.requested_mode = em_fc_setting; else hw->fc.requested_mode = e1000_fc_none; /* Override - workaround for PCHLAN issue */ if (hw->mac.type == e1000_pchlan) hw->fc.requested_mode = e1000_fc_rx_pause; /* Issue a global reset */ e1000_reset_hw(hw); E1000_WRITE_REG(hw, E1000_WUC, 0); if (e1000_init_hw(hw) < 0) { device_printf(dev, "Hardware Initialization Failed\n"); return; } E1000_WRITE_REG(hw, E1000_VET, ETHERTYPE_VLAN); e1000_get_phy_info(hw); e1000_check_for_link(hw); return; } /********************************************************************* * * Setup networking device structure and register an interface. * **********************************************************************/ static void em_setup_interface(device_t dev, struct adapter *adapter) { struct ifnet *ifp; INIT_DEBUGOUT("em_setup_interface: begin"); ifp = adapter->ifp = if_alloc(IFT_ETHER); if (ifp == NULL) panic("%s: can not if_alloc()", device_get_nameunit(dev)); if_initname(ifp, device_get_name(dev), device_get_unit(dev)); ifp->if_mtu = ETHERMTU; ifp->if_init = em_init; ifp->if_softc = adapter; ifp->if_flags = IFF_BROADCAST | IFF_SIMPLEX | IFF_MULTICAST; ifp->if_ioctl = em_ioctl; ifp->if_start = em_start; IFQ_SET_MAXLEN(&ifp->if_snd, adapter->num_tx_desc - 1); ifp->if_snd.ifq_drv_maxlen = adapter->num_tx_desc - 1; IFQ_SET_READY(&ifp->if_snd); ether_ifattach(ifp, adapter->hw.mac.addr); ifp->if_capabilities = ifp->if_capenable = 0; #ifdef EM_MULTIQUEUE /* Multiqueue tx functions */ ifp->if_transmit = em_mq_start; ifp->if_qflush = em_qflush; #endif ifp->if_capabilities |= IFCAP_HWCSUM | IFCAP_VLAN_HWCSUM; ifp->if_capenable |= IFCAP_HWCSUM | IFCAP_VLAN_HWCSUM; /* Enable TSO by default, can disable with ifconfig */ ifp->if_capabilities |= IFCAP_TSO4; ifp->if_capenable |= IFCAP_TSO4; /* * Tell the upper layer(s) we * support full VLAN capability */ ifp->if_data.ifi_hdrlen = sizeof(struct ether_vlan_header); ifp->if_capabilities |= IFCAP_VLAN_HWTAGGING | IFCAP_VLAN_MTU; ifp->if_capenable |= IFCAP_VLAN_HWTAGGING | IFCAP_VLAN_MTU; /* ** Dont turn this on by default, if vlans are ** created on another pseudo device (eg. lagg) ** then vlan events are not passed thru, breaking ** operation, but with HW FILTER off it works. If ** using vlans directly on the em driver you can ** enable this and get full hardware tag filtering. */ ifp->if_capabilities |= IFCAP_VLAN_HWFILTER; #ifdef DEVICE_POLLING ifp->if_capabilities |= IFCAP_POLLING; #endif /* Enable All WOL methods by default */ if (adapter->wol) { ifp->if_capabilities |= IFCAP_WOL; ifp->if_capenable |= IFCAP_WOL; } /* * Specify the media types supported by this adapter and register * callbacks to update media and link information */ ifmedia_init(&adapter->media, IFM_IMASK, em_media_change, em_media_status); if ((adapter->hw.phy.media_type == e1000_media_type_fiber) || (adapter->hw.phy.media_type == e1000_media_type_internal_serdes)) { u_char fiber_type = IFM_1000_SX; /* default type */ ifmedia_add(&adapter->media, IFM_ETHER | fiber_type | IFM_FDX, 0, NULL); ifmedia_add(&adapter->media, IFM_ETHER | fiber_type, 0, NULL); } else { ifmedia_add(&adapter->media, IFM_ETHER | IFM_10_T, 0, NULL); ifmedia_add(&adapter->media, IFM_ETHER | IFM_10_T | IFM_FDX, 0, NULL); ifmedia_add(&adapter->media, IFM_ETHER | IFM_100_TX, 0, NULL); ifmedia_add(&adapter->media, IFM_ETHER | IFM_100_TX | IFM_FDX, 0, NULL); if (adapter->hw.phy.type != e1000_phy_ife) { ifmedia_add(&adapter->media, IFM_ETHER | IFM_1000_T | IFM_FDX, 0, NULL); ifmedia_add(&adapter->media, IFM_ETHER | IFM_1000_T, 0, NULL); } } ifmedia_add(&adapter->media, IFM_ETHER | IFM_AUTO, 0, NULL); ifmedia_set(&adapter->media, IFM_ETHER | IFM_AUTO); } /* * Manage DMA'able memory. */ static void em_dmamap_cb(void *arg, bus_dma_segment_t *segs, int nseg, int error) { if (error) return; *(bus_addr_t *) arg = segs[0].ds_addr; } static int em_dma_malloc(struct adapter *adapter, bus_size_t size, struct em_dma_alloc *dma, int mapflags) { int error; error = bus_dma_tag_create(bus_get_dma_tag(adapter->dev), /* parent */ EM_DBA_ALIGN, 0, /* alignment, bounds */ BUS_SPACE_MAXADDR, /* lowaddr */ BUS_SPACE_MAXADDR, /* highaddr */ NULL, NULL, /* filter, filterarg */ size, /* maxsize */ 1, /* nsegments */ size, /* maxsegsize */ 0, /* flags */ NULL, /* lockfunc */ NULL, /* lockarg */ &dma->dma_tag); if (error) { device_printf(adapter->dev, "%s: bus_dma_tag_create failed: %d\n", __func__, error); goto fail_0; } error = bus_dmamem_alloc(dma->dma_tag, (void**) &dma->dma_vaddr, BUS_DMA_NOWAIT | BUS_DMA_COHERENT, &dma->dma_map); if (error) { device_printf(adapter->dev, "%s: bus_dmamem_alloc(%ju) failed: %d\n", __func__, (uintmax_t)size, error); goto fail_2; } dma->dma_paddr = 0; error = bus_dmamap_load(dma->dma_tag, dma->dma_map, dma->dma_vaddr, size, em_dmamap_cb, &dma->dma_paddr, mapflags | BUS_DMA_NOWAIT); if (error || dma->dma_paddr == 0) { device_printf(adapter->dev, "%s: bus_dmamap_load failed: %d\n", __func__, error); goto fail_3; } return (0); fail_3: bus_dmamap_unload(dma->dma_tag, dma->dma_map); fail_2: bus_dmamem_free(dma->dma_tag, dma->dma_vaddr, dma->dma_map); bus_dma_tag_destroy(dma->dma_tag); fail_0: dma->dma_map = NULL; dma->dma_tag = NULL; return (error); } static void em_dma_free(struct adapter *adapter, struct em_dma_alloc *dma) { if (dma->dma_tag == NULL) return; if (dma->dma_map != NULL) { bus_dmamap_sync(dma->dma_tag, dma->dma_map, BUS_DMASYNC_POSTREAD | BUS_DMASYNC_POSTWRITE); bus_dmamap_unload(dma->dma_tag, dma->dma_map); bus_dmamem_free(dma->dma_tag, dma->dma_vaddr, dma->dma_map); dma->dma_map = NULL; } bus_dma_tag_destroy(dma->dma_tag); dma->dma_tag = NULL; } /********************************************************************* * * Allocate memory for the transmit and receive rings, and then * the descriptors associated with each, called only once at attach. * **********************************************************************/ static int em_allocate_queues(struct adapter *adapter) { device_t dev = adapter->dev; struct tx_ring *txr = NULL; struct rx_ring *rxr = NULL; int rsize, tsize, error = E1000_SUCCESS; int txconf = 0, rxconf = 0; /* Allocate the TX ring struct memory */ if (!(adapter->tx_rings = (struct tx_ring *) malloc(sizeof(struct tx_ring) * adapter->num_queues, M_DEVBUF, M_NOWAIT | M_ZERO))) { device_printf(dev, "Unable to allocate TX ring memory\n"); error = ENOMEM; goto fail; } /* Now allocate the RX */ if (!(adapter->rx_rings = (struct rx_ring *) malloc(sizeof(struct rx_ring) * adapter->num_queues, M_DEVBUF, M_NOWAIT | M_ZERO))) { device_printf(dev, "Unable to allocate RX ring memory\n"); error = ENOMEM; goto rx_fail; } tsize = roundup2(adapter->num_tx_desc * sizeof(struct e1000_tx_desc), EM_DBA_ALIGN); /* * Now set up the TX queues, txconf is needed to handle the * possibility that things fail midcourse and we need to * undo memory gracefully */ for (int i = 0; i < adapter->num_queues; i++, txconf++) { /* Set up some basics */ txr = &adapter->tx_rings[i]; txr->adapter = adapter; txr->me = i; /* Initialize the TX lock */ snprintf(txr->mtx_name, sizeof(txr->mtx_name), "%s:tx(%d)", device_get_nameunit(dev), txr->me); mtx_init(&txr->tx_mtx, txr->mtx_name, NULL, MTX_DEF); if (em_dma_malloc(adapter, tsize, &txr->txdma, BUS_DMA_NOWAIT)) { device_printf(dev, "Unable to allocate TX Descriptor memory\n"); error = ENOMEM; goto err_tx_desc; } txr->tx_base = (struct e1000_tx_desc *)txr->txdma.dma_vaddr; bzero((void *)txr->tx_base, tsize); if (em_allocate_transmit_buffers(txr)) { device_printf(dev, "Critical Failure setting up transmit buffers\n"); error = ENOMEM; goto err_tx_desc; } #if __FreeBSD_version >= 800000 /* Allocate a buf ring */ txr->br = buf_ring_alloc(4096, M_DEVBUF, M_WAITOK, &txr->tx_mtx); #endif } /* * Next the RX queues... */ rsize = roundup2(adapter->num_rx_desc * sizeof(struct e1000_rx_desc), EM_DBA_ALIGN); for (int i = 0; i < adapter->num_queues; i++, rxconf++) { rxr = &adapter->rx_rings[i]; rxr->adapter = adapter; rxr->me = i; /* Initialize the RX lock */ snprintf(rxr->mtx_name, sizeof(rxr->mtx_name), "%s:rx(%d)", device_get_nameunit(dev), txr->me); mtx_init(&rxr->rx_mtx, rxr->mtx_name, NULL, MTX_DEF); if (em_dma_malloc(adapter, rsize, &rxr->rxdma, BUS_DMA_NOWAIT)) { device_printf(dev, "Unable to allocate RxDescriptor memory\n"); error = ENOMEM; goto err_rx_desc; } rxr->rx_base = (struct e1000_rx_desc *)rxr->rxdma.dma_vaddr; bzero((void *)rxr->rx_base, rsize); /* Allocate receive buffers for the ring*/ if (em_allocate_receive_buffers(rxr)) { device_printf(dev, "Critical Failure setting up receive buffers\n"); error = ENOMEM; goto err_rx_desc; } } return (0); err_rx_desc: for (rxr = adapter->rx_rings; rxconf > 0; rxr++, rxconf--) em_dma_free(adapter, &rxr->rxdma); err_tx_desc: for (txr = adapter->tx_rings; txconf > 0; txr++, txconf--) em_dma_free(adapter, &txr->txdma); free(adapter->rx_rings, M_DEVBUF); rx_fail: buf_ring_free(txr->br, M_DEVBUF); free(adapter->tx_rings, M_DEVBUF); fail: return (error); } /********************************************************************* * * Allocate memory for tx_buffer structures. The tx_buffer stores all * the information needed to transmit a packet on the wire. This is * called only once at attach, setup is done every reset. * **********************************************************************/ static int em_allocate_transmit_buffers(struct tx_ring *txr) { struct adapter *adapter = txr->adapter; device_t dev = adapter->dev; struct em_buffer *txbuf; int error, i; /* * Setup DMA descriptor areas. */ if ((error = bus_dma_tag_create(bus_get_dma_tag(dev), 1, 0, /* alignment, bounds */ BUS_SPACE_MAXADDR, /* lowaddr */ BUS_SPACE_MAXADDR, /* highaddr */ NULL, NULL, /* filter, filterarg */ EM_TSO_SIZE, /* maxsize */ EM_MAX_SCATTER, /* nsegments */ PAGE_SIZE, /* maxsegsize */ 0, /* flags */ NULL, /* lockfunc */ NULL, /* lockfuncarg */ &txr->txtag))) { device_printf(dev,"Unable to allocate TX DMA tag\n"); goto fail; } if (!(txr->tx_buffers = (struct em_buffer *) malloc(sizeof(struct em_buffer) * adapter->num_tx_desc, M_DEVBUF, M_NOWAIT | M_ZERO))) { device_printf(dev, "Unable to allocate tx_buffer memory\n"); error = ENOMEM; goto fail; } /* Create the descriptor buffer dma maps */ txbuf = txr->tx_buffers; for (i = 0; i < adapter->num_tx_desc; i++, txbuf++) { error = bus_dmamap_create(txr->txtag, 0, &txbuf->map); if (error != 0) { device_printf(dev, "Unable to create TX DMA map\n"); goto fail; } } return 0; fail: /* We free all, it handles case where we are in the middle */ em_free_transmit_structures(adapter); return (error); } /********************************************************************* * * Initialize a transmit ring. * **********************************************************************/ static void em_setup_transmit_ring(struct tx_ring *txr) { struct adapter *adapter = txr->adapter; struct em_buffer *txbuf; int i; /* Clear the old descriptor contents */ EM_TX_LOCK(txr); bzero((void *)txr->tx_base, (sizeof(struct e1000_tx_desc)) * adapter->num_tx_desc); /* Reset indices */ txr->next_avail_desc = 0; txr->next_to_clean = 0; /* Free any existing tx buffers. */ txbuf = txr->tx_buffers; for (i = 0; i < adapter->num_tx_desc; i++, txbuf++) { if (txbuf->m_head != NULL) { bus_dmamap_sync(txr->txtag, txbuf->map, BUS_DMASYNC_POSTWRITE); bus_dmamap_unload(txr->txtag, txbuf->map); m_freem(txbuf->m_head); txbuf->m_head = NULL; } /* clear the watch index */ txbuf->next_eop = -1; } /* Set number of descriptors available */ txr->tx_avail = adapter->num_tx_desc; bus_dmamap_sync(txr->txdma.dma_tag, txr->txdma.dma_map, BUS_DMASYNC_PREREAD | BUS_DMASYNC_PREWRITE); EM_TX_UNLOCK(txr); } /********************************************************************* * * Initialize all transmit rings. * **********************************************************************/ static void em_setup_transmit_structures(struct adapter *adapter) { struct tx_ring *txr = adapter->tx_rings; for (int i = 0; i < adapter->num_queues; i++, txr++) em_setup_transmit_ring(txr); return; } /********************************************************************* * * Enable transmit unit. * **********************************************************************/ static void em_initialize_transmit_unit(struct adapter *adapter) { struct tx_ring *txr = adapter->tx_rings; struct e1000_hw *hw = &adapter->hw; u32 tctl, tarc, tipg = 0; INIT_DEBUGOUT("em_initialize_transmit_unit: begin"); for (int i = 0; i < adapter->num_queues; i++, txr++) { u64 bus_addr = txr->txdma.dma_paddr; /* Base and Len of TX Ring */ E1000_WRITE_REG(hw, E1000_TDLEN(i), adapter->num_tx_desc * sizeof(struct e1000_tx_desc)); E1000_WRITE_REG(hw, E1000_TDBAH(i), (u32)(bus_addr >> 32)); E1000_WRITE_REG(hw, E1000_TDBAL(i), (u32)bus_addr); /* Init the HEAD/TAIL indices */ E1000_WRITE_REG(hw, E1000_TDT(i), 0); E1000_WRITE_REG(hw, E1000_TDH(i), 0); HW_DEBUGOUT2("Base = %x, Length = %x\n", E1000_READ_REG(&adapter->hw, E1000_TDBAL(i)), E1000_READ_REG(&adapter->hw, E1000_TDLEN(i))); txr->watchdog_check = FALSE; } /* Set the default values for the Tx Inter Packet Gap timer */ switch (adapter->hw.mac.type) { case e1000_82542: tipg = DEFAULT_82542_TIPG_IPGT; tipg |= DEFAULT_82542_TIPG_IPGR1 << E1000_TIPG_IPGR1_SHIFT; tipg |= DEFAULT_82542_TIPG_IPGR2 << E1000_TIPG_IPGR2_SHIFT; break; case e1000_80003es2lan: tipg = DEFAULT_82543_TIPG_IPGR1; tipg |= DEFAULT_80003ES2LAN_TIPG_IPGR2 << E1000_TIPG_IPGR2_SHIFT; break; default: if ((adapter->hw.phy.media_type == e1000_media_type_fiber) || (adapter->hw.phy.media_type == e1000_media_type_internal_serdes)) tipg = DEFAULT_82543_TIPG_IPGT_FIBER; else tipg = DEFAULT_82543_TIPG_IPGT_COPPER; tipg |= DEFAULT_82543_TIPG_IPGR1 << E1000_TIPG_IPGR1_SHIFT; tipg |= DEFAULT_82543_TIPG_IPGR2 << E1000_TIPG_IPGR2_SHIFT; } E1000_WRITE_REG(&adapter->hw, E1000_TIPG, tipg); E1000_WRITE_REG(&adapter->hw, E1000_TIDV, adapter->tx_int_delay.value); if(adapter->hw.mac.type >= e1000_82540) E1000_WRITE_REG(&adapter->hw, E1000_TADV, adapter->tx_abs_int_delay.value); if ((adapter->hw.mac.type == e1000_82571) || (adapter->hw.mac.type == e1000_82572)) { tarc = E1000_READ_REG(&adapter->hw, E1000_TARC(0)); tarc |= SPEED_MODE_BIT; E1000_WRITE_REG(&adapter->hw, E1000_TARC(0), tarc); } else if (adapter->hw.mac.type == e1000_80003es2lan) { tarc = E1000_READ_REG(&adapter->hw, E1000_TARC(0)); tarc |= 1; E1000_WRITE_REG(&adapter->hw, E1000_TARC(0), tarc); tarc = E1000_READ_REG(&adapter->hw, E1000_TARC(1)); tarc |= 1; E1000_WRITE_REG(&adapter->hw, E1000_TARC(1), tarc); } adapter->txd_cmd = E1000_TXD_CMD_IFCS; if (adapter->tx_int_delay.value > 0) adapter->txd_cmd |= E1000_TXD_CMD_IDE; /* Program the Transmit Control Register */ tctl = E1000_READ_REG(&adapter->hw, E1000_TCTL); tctl &= ~E1000_TCTL_CT; tctl |= (E1000_TCTL_PSP | E1000_TCTL_RTLC | E1000_TCTL_EN | (E1000_COLLISION_THRESHOLD << E1000_CT_SHIFT)); if (adapter->hw.mac.type >= e1000_82571) tctl |= E1000_TCTL_MULR; /* This write will effectively turn on the transmit unit. */ E1000_WRITE_REG(&adapter->hw, E1000_TCTL, tctl); } /********************************************************************* * * Free all transmit rings. * **********************************************************************/ static void em_free_transmit_structures(struct adapter *adapter) { struct tx_ring *txr = adapter->tx_rings; for (int i = 0; i < adapter->num_queues; i++, txr++) { EM_TX_LOCK(txr); em_free_transmit_buffers(txr); em_dma_free(adapter, &txr->txdma); EM_TX_UNLOCK(txr); EM_TX_LOCK_DESTROY(txr); } free(adapter->tx_rings, M_DEVBUF); } /********************************************************************* * * Free transmit ring related data structures. * **********************************************************************/ static void em_free_transmit_buffers(struct tx_ring *txr) { struct adapter *adapter = txr->adapter; struct em_buffer *txbuf; INIT_DEBUGOUT("free_transmit_ring: begin"); if (txr->tx_buffers == NULL) return; for (int i = 0; i < adapter->num_tx_desc; i++) { txbuf = &txr->tx_buffers[i]; if (txbuf->m_head != NULL) { bus_dmamap_sync(txr->txtag, txbuf->map, BUS_DMASYNC_POSTWRITE); bus_dmamap_unload(txr->txtag, txbuf->map); m_freem(txbuf->m_head); txbuf->m_head = NULL; if (txbuf->map != NULL) { bus_dmamap_destroy(txr->txtag, txbuf->map); txbuf->map = NULL; } } else if (txbuf->map != NULL) { bus_dmamap_unload(txr->txtag, txbuf->map); bus_dmamap_destroy(txr->txtag, txbuf->map); txbuf->map = NULL; } } #if __FreeBSD_version >= 800000 if (txr->br != NULL) buf_ring_free(txr->br, M_DEVBUF); #endif if (txr->tx_buffers != NULL) { free(txr->tx_buffers, M_DEVBUF); txr->tx_buffers = NULL; } if (txr->txtag != NULL) { bus_dma_tag_destroy(txr->txtag); txr->txtag = NULL; } return; } /********************************************************************* * * The offload context needs to be set when we transfer the first * packet of a particular protocol (TCP/UDP). This routine has been * enhanced to deal with inserted VLAN headers, and IPV6 (not complete) * * Added back the old method of keeping the current context type * and not setting if unnecessary, as this is reported to be a * big performance win. -jfv **********************************************************************/ static void em_transmit_checksum_setup(struct tx_ring *txr, struct mbuf *mp, u32 *txd_upper, u32 *txd_lower) { struct adapter *adapter = txr->adapter; struct e1000_context_desc *TXD = NULL; struct em_buffer *tx_buffer; struct ether_vlan_header *eh; struct ip *ip = NULL; struct ip6_hdr *ip6; int cur, ehdrlen; u32 cmd, hdr_len, ip_hlen; u16 etype; u8 ipproto; cmd = hdr_len = ipproto = 0; cur = txr->next_avail_desc; /* * Determine where frame payload starts. * Jump over vlan headers if already present, * helpful for QinQ too. */ eh = mtod(mp, struct ether_vlan_header *); if (eh->evl_encap_proto == htons(ETHERTYPE_VLAN)) { etype = ntohs(eh->evl_proto); ehdrlen = ETHER_HDR_LEN + ETHER_VLAN_ENCAP_LEN; } else { etype = ntohs(eh->evl_encap_proto); ehdrlen = ETHER_HDR_LEN; } /* * We only support TCP/UDP for IPv4 and IPv6 for the moment. * TODO: Support SCTP too when it hits the tree. */ switch (etype) { case ETHERTYPE_IP: ip = (struct ip *)(mp->m_data + ehdrlen); ip_hlen = ip->ip_hl << 2; /* Setup of IP header checksum. */ if (mp->m_pkthdr.csum_flags & CSUM_IP) { /* * Start offset for header checksum calculation. * End offset for header checksum calculation. * Offset of place to put the checksum. */ TXD = (struct e1000_context_desc *) &txr->tx_base[cur]; TXD->lower_setup.ip_fields.ipcss = ehdrlen; TXD->lower_setup.ip_fields.ipcse = htole16(ehdrlen + ip_hlen); TXD->lower_setup.ip_fields.ipcso = ehdrlen + offsetof(struct ip, ip_sum); cmd |= E1000_TXD_CMD_IP; *txd_upper |= E1000_TXD_POPTS_IXSM << 8; } if (mp->m_len < ehdrlen + ip_hlen) return; /* failure */ hdr_len = ehdrlen + ip_hlen; ipproto = ip->ip_p; break; case ETHERTYPE_IPV6: ip6 = (struct ip6_hdr *)(mp->m_data + ehdrlen); ip_hlen = sizeof(struct ip6_hdr); /* XXX: No header stacking. */ if (mp->m_len < ehdrlen + ip_hlen) return; /* failure */ /* IPv6 doesn't have a header checksum. */ hdr_len = ehdrlen + ip_hlen; ipproto = ip6->ip6_nxt; break; default: *txd_upper = 0; *txd_lower = 0; return; } switch (ipproto) { case IPPROTO_TCP: if (mp->m_pkthdr.csum_flags & CSUM_TCP) { *txd_lower = E1000_TXD_CMD_DEXT | E1000_TXD_DTYP_D; *txd_upper |= E1000_TXD_POPTS_TXSM << 8; /* no need for context if already set */ if (txr->last_hw_offload == CSUM_TCP) return; txr->last_hw_offload = CSUM_TCP; /* * Start offset for payload checksum calculation. * End offset for payload checksum calculation. * Offset of place to put the checksum. */ TXD = (struct e1000_context_desc *) &txr->tx_base[cur]; TXD->upper_setup.tcp_fields.tucss = hdr_len; TXD->upper_setup.tcp_fields.tucse = htole16(0); TXD->upper_setup.tcp_fields.tucso = hdr_len + offsetof(struct tcphdr, th_sum); cmd |= E1000_TXD_CMD_TCP; } break; case IPPROTO_UDP: { if (mp->m_pkthdr.csum_flags & CSUM_UDP) { *txd_lower = E1000_TXD_CMD_DEXT | E1000_TXD_DTYP_D; *txd_upper |= E1000_TXD_POPTS_TXSM << 8; /* no need for context if already set */ if (txr->last_hw_offload == CSUM_UDP) return; txr->last_hw_offload = CSUM_UDP; /* * Start offset for header checksum calculation. * End offset for header checksum calculation. * Offset of place to put the checksum. */ TXD = (struct e1000_context_desc *) &txr->tx_base[cur]; TXD->upper_setup.tcp_fields.tucss = hdr_len; TXD->upper_setup.tcp_fields.tucse = htole16(0); TXD->upper_setup.tcp_fields.tucso = hdr_len + offsetof(struct udphdr, uh_sum); } /* Fall Thru */ } default: break; } TXD->tcp_seg_setup.data = htole32(0); TXD->cmd_and_length = htole32(adapter->txd_cmd | E1000_TXD_CMD_DEXT | cmd); tx_buffer = &txr->tx_buffers[cur]; tx_buffer->m_head = NULL; tx_buffer->next_eop = -1; if (++cur == adapter->num_tx_desc) cur = 0; txr->tx_avail--; txr->next_avail_desc = cur; } /********************************************************************** * * Setup work for hardware segmentation offload (TSO) * **********************************************************************/ static bool em_tso_setup(struct tx_ring *txr, struct mbuf *mp, u32 *txd_upper, u32 *txd_lower) { struct adapter *adapter = txr->adapter; struct e1000_context_desc *TXD; struct em_buffer *tx_buffer; struct ether_vlan_header *eh; struct ip *ip; struct ip6_hdr *ip6; struct tcphdr *th; int cur, ehdrlen, hdr_len, ip_hlen, isip6; u16 etype; /* * This function could/should be extended to support IP/IPv6 * fragmentation as well. But as they say, one step at a time. */ /* * Determine where frame payload starts. * Jump over vlan headers if already present, * helpful for QinQ too. */ eh = mtod(mp, struct ether_vlan_header *); if (eh->evl_encap_proto == htons(ETHERTYPE_VLAN)) { etype = ntohs(eh->evl_proto); ehdrlen = ETHER_HDR_LEN + ETHER_VLAN_ENCAP_LEN; } else { etype = ntohs(eh->evl_encap_proto); ehdrlen = ETHER_HDR_LEN; } /* Ensure we have at least the IP+TCP header in the first mbuf. */ if (mp->m_len < ehdrlen + sizeof(struct ip) + sizeof(struct tcphdr)) return FALSE; /* -1 */ /* * We only support TCP for IPv4 and IPv6 (notyet) for the moment. * TODO: Support SCTP too when it hits the tree. */ switch (etype) { case ETHERTYPE_IP: isip6 = 0; ip = (struct ip *)(mp->m_data + ehdrlen); if (ip->ip_p != IPPROTO_TCP) return FALSE; /* 0 */ ip->ip_len = 0; ip->ip_sum = 0; ip_hlen = ip->ip_hl << 2; if (mp->m_len < ehdrlen + ip_hlen + sizeof(struct tcphdr)) return FALSE; /* -1 */ th = (struct tcphdr *)((caddr_t)ip + ip_hlen); #if 1 th->th_sum = in_pseudo(ip->ip_src.s_addr, ip->ip_dst.s_addr, htons(IPPROTO_TCP)); #else th->th_sum = mp->m_pkthdr.csum_data; #endif break; case ETHERTYPE_IPV6: isip6 = 1; return FALSE; /* Not supported yet. */ ip6 = (struct ip6_hdr *)(mp->m_data + ehdrlen); if (ip6->ip6_nxt != IPPROTO_TCP) return FALSE; /* 0 */ ip6->ip6_plen = 0; ip_hlen = sizeof(struct ip6_hdr); /* XXX: no header stacking. */ if (mp->m_len < ehdrlen + ip_hlen + sizeof(struct tcphdr)) return FALSE; /* -1 */ th = (struct tcphdr *)((caddr_t)ip6 + ip_hlen); #if 0 th->th_sum = in6_pseudo(ip6->ip6_src, ip->ip6_dst, htons(IPPROTO_TCP)); /* XXX: function notyet. */ #else th->th_sum = mp->m_pkthdr.csum_data; #endif break; default: return FALSE; } hdr_len = ehdrlen + ip_hlen + (th->th_off << 2); *txd_lower = (E1000_TXD_CMD_DEXT | /* Extended descr type */ E1000_TXD_DTYP_D | /* Data descr type */ E1000_TXD_CMD_TSE); /* Do TSE on this packet */ /* IP and/or TCP header checksum calculation and insertion. */ *txd_upper = ((isip6 ? 0 : E1000_TXD_POPTS_IXSM) | E1000_TXD_POPTS_TXSM) << 8; cur = txr->next_avail_desc; tx_buffer = &txr->tx_buffers[cur]; TXD = (struct e1000_context_desc *) &txr->tx_base[cur]; /* IPv6 doesn't have a header checksum. */ if (!isip6) { /* * Start offset for header checksum calculation. * End offset for header checksum calculation. * Offset of place put the checksum. */ TXD->lower_setup.ip_fields.ipcss = ehdrlen; TXD->lower_setup.ip_fields.ipcse = htole16(ehdrlen + ip_hlen - 1); TXD->lower_setup.ip_fields.ipcso = ehdrlen + offsetof(struct ip, ip_sum); } /* * Start offset for payload checksum calculation. * End offset for payload checksum calculation. * Offset of place to put the checksum. */ TXD->upper_setup.tcp_fields.tucss = ehdrlen + ip_hlen; TXD->upper_setup.tcp_fields.tucse = 0; TXD->upper_setup.tcp_fields.tucso = ehdrlen + ip_hlen + offsetof(struct tcphdr, th_sum); /* * Payload size per packet w/o any headers. * Length of all headers up to payload. */ TXD->tcp_seg_setup.fields.mss = htole16(mp->m_pkthdr.tso_segsz); TXD->tcp_seg_setup.fields.hdr_len = hdr_len; TXD->cmd_and_length = htole32(adapter->txd_cmd | E1000_TXD_CMD_DEXT | /* Extended descr */ E1000_TXD_CMD_TSE | /* TSE context */ (isip6 ? 0 : E1000_TXD_CMD_IP) | E1000_TXD_CMD_TCP | /* Do TCP checksum */ (mp->m_pkthdr.len - (hdr_len))); /* Total len */ tx_buffer->m_head = NULL; tx_buffer->next_eop = -1; if (++cur == adapter->num_tx_desc) cur = 0; txr->tx_avail--; txr->next_avail_desc = cur; txr->tx_tso = TRUE; return TRUE; } /********************************************************************** * * Examine each tx_buffer in the used queue. If the hardware is done * processing the packet then free associated resources. The * tx_buffer is put back on the free queue. * **********************************************************************/ static bool em_txeof(struct tx_ring *txr) { struct adapter *adapter = txr->adapter; int first, last, done, num_avail; struct em_buffer *tx_buffer; struct e1000_tx_desc *tx_desc, *eop_desc; struct ifnet *ifp = adapter->ifp; EM_TX_LOCK_ASSERT(txr); if (txr->tx_avail == adapter->num_tx_desc) return (FALSE); num_avail = txr->tx_avail; first = txr->next_to_clean; tx_desc = &txr->tx_base[first]; tx_buffer = &txr->tx_buffers[first]; last = tx_buffer->next_eop; eop_desc = &txr->tx_base[last]; /* * What this does is get the index of the * first descriptor AFTER the EOP of the * first packet, that way we can do the * simple comparison on the inner while loop. */ if (++last == adapter->num_tx_desc) last = 0; done = last; bus_dmamap_sync(txr->txdma.dma_tag, txr->txdma.dma_map, BUS_DMASYNC_POSTREAD); while (eop_desc->upper.fields.status & E1000_TXD_STAT_DD) { /* We clean the range of the packet */ while (first != done) { tx_desc->upper.data = 0; tx_desc->lower.data = 0; tx_desc->buffer_addr = 0; ++num_avail; if (tx_buffer->m_head) { ifp->if_opackets++; bus_dmamap_sync(txr->txtag, tx_buffer->map, BUS_DMASYNC_POSTWRITE); bus_dmamap_unload(txr->txtag, tx_buffer->map); m_freem(tx_buffer->m_head); tx_buffer->m_head = NULL; } tx_buffer->next_eop = -1; txr->watchdog_time = ticks; if (++first == adapter->num_tx_desc) first = 0; tx_buffer = &txr->tx_buffers[first]; tx_desc = &txr->tx_base[first]; } /* See if we can continue to the next packet */ last = tx_buffer->next_eop; if (last != -1) { eop_desc = &txr->tx_base[last]; /* Get new done point */ if (++last == adapter->num_tx_desc) last = 0; done = last; } else break; } bus_dmamap_sync(txr->txdma.dma_tag, txr->txdma.dma_map, BUS_DMASYNC_PREREAD | BUS_DMASYNC_PREWRITE); txr->next_to_clean = first; /* * If we have enough room, clear IFF_DRV_OACTIVE to * tell the stack that it is OK to send packets. * If there are no pending descriptors, clear the watchdog. */ if (num_avail > EM_TX_CLEANUP_THRESHOLD) { ifp->if_drv_flags &= ~IFF_DRV_OACTIVE; if (num_avail == adapter->num_tx_desc) { txr->watchdog_check = FALSE; txr->tx_avail = num_avail; return (FALSE); } } txr->tx_avail = num_avail; return (TRUE); } /********************************************************************* * * Refresh RX descriptor mbufs from system mbuf buffer pool. * **********************************************************************/ static void em_refresh_mbufs(struct rx_ring *rxr, int limit) { struct adapter *adapter = rxr->adapter; struct mbuf *m; bus_dma_segment_t segs[1]; bus_dmamap_t map; struct em_buffer *rxbuf; int i, error, nsegs, cleaned; i = rxr->next_to_refresh; cleaned = -1; while (i != limit) { m = m_getcl(M_DONTWAIT, MT_DATA, M_PKTHDR); if (m == NULL) goto update; m->m_len = m->m_pkthdr.len = MCLBYTES; if (adapter->max_frame_size <= (MCLBYTES - ETHER_ALIGN)) m_adj(m, ETHER_ALIGN); /* * Using memory from the mbuf cluster pool, invoke the * bus_dma machinery to arrange the memory mapping. */ error = bus_dmamap_load_mbuf_sg(rxr->rxtag, rxr->rx_sparemap, m, segs, &nsegs, BUS_DMA_NOWAIT); if (error != 0) { m_free(m); goto update; } /* If nsegs is wrong then the stack is corrupt. */ KASSERT(nsegs == 1, ("Too many segments returned!")); rxbuf = &rxr->rx_buffers[i]; if (rxbuf->m_head != NULL) bus_dmamap_unload(rxr->rxtag, rxbuf->map); map = rxbuf->map; rxbuf->map = rxr->rx_sparemap; rxr->rx_sparemap = map; bus_dmamap_sync(rxr->rxtag, rxbuf->map, BUS_DMASYNC_PREREAD); rxbuf->m_head = m; rxr->rx_base[i].buffer_addr = htole64(segs[0].ds_addr); cleaned = i; /* Calculate next index */ if (++i == adapter->num_rx_desc) i = 0; /* This is the work marker for refresh */ rxr->next_to_refresh = i; } update: bus_dmamap_sync(rxr->rxdma.dma_tag, rxr->rxdma.dma_map, BUS_DMASYNC_PREREAD | BUS_DMASYNC_PREWRITE); if (cleaned != -1) /* Update tail index */ E1000_WRITE_REG(&adapter->hw, E1000_RDT(rxr->me), cleaned); return; } /********************************************************************* * * Allocate memory for rx_buffer structures. Since we use one * rx_buffer per received packet, the maximum number of rx_buffer's * that we'll need is equal to the number of receive descriptors * that we've allocated. * **********************************************************************/ static int em_allocate_receive_buffers(struct rx_ring *rxr) { struct adapter *adapter = rxr->adapter; device_t dev = adapter->dev; struct em_buffer *rxbuf; int error; rxr->rx_buffers = malloc(sizeof(struct em_buffer) * adapter->num_rx_desc, M_DEVBUF, M_NOWAIT | M_ZERO); if (rxr->rx_buffers == NULL) { device_printf(dev, "Unable to allocate rx_buffer memory\n"); return (ENOMEM); } error = bus_dma_tag_create(bus_get_dma_tag(dev), /* parent */ 1, 0, /* alignment, bounds */ BUS_SPACE_MAXADDR, /* lowaddr */ BUS_SPACE_MAXADDR, /* highaddr */ NULL, NULL, /* filter, filterarg */ MCLBYTES, /* maxsize */ 1, /* nsegments */ MCLBYTES, /* maxsegsize */ 0, /* flags */ NULL, /* lockfunc */ NULL, /* lockarg */ &rxr->rxtag); if (error) { device_printf(dev, "%s: bus_dma_tag_create failed %d\n", __func__, error); goto fail; } /* Create the spare map (used by getbuf) */ error = bus_dmamap_create(rxr->rxtag, BUS_DMA_NOWAIT, &rxr->rx_sparemap); if (error) { device_printf(dev, "%s: bus_dmamap_create failed: %d\n", __func__, error); goto fail; } rxbuf = rxr->rx_buffers; for (int i = 0; i < adapter->num_rx_desc; i++, rxbuf++) { rxbuf = &rxr->rx_buffers[i]; error = bus_dmamap_create(rxr->rxtag, BUS_DMA_NOWAIT, &rxbuf->map); if (error) { device_printf(dev, "%s: bus_dmamap_create failed: %d\n", __func__, error); goto fail; } } return (0); fail: em_free_receive_structures(adapter); return (error); } /********************************************************************* * * Initialize a receive ring and its buffers. * **********************************************************************/ static int em_setup_receive_ring(struct rx_ring *rxr) { struct adapter *adapter = rxr->adapter; struct em_buffer *rxbuf; bus_dma_segment_t seg[1]; int rsize, nsegs, error; /* Clear the ring contents */ EM_RX_LOCK(rxr); rsize = roundup2(adapter->num_rx_desc * sizeof(struct e1000_rx_desc), EM_DBA_ALIGN); bzero((void *)rxr->rx_base, rsize); /* ** Free current RX buffer structs and their mbufs */ for (int i = 0; i < adapter->num_rx_desc; i++) { rxbuf = &rxr->rx_buffers[i]; if (rxbuf->m_head != NULL) { bus_dmamap_sync(rxr->rxtag, rxbuf->map, BUS_DMASYNC_POSTREAD); bus_dmamap_unload(rxr->rxtag, rxbuf->map); m_freem(rxbuf->m_head); } } /* Now replenish the mbufs */ for (int j = 0; j != adapter->num_rx_desc; ++j) { rxbuf = &rxr->rx_buffers[j]; rxbuf->m_head = m_getcl(M_DONTWAIT, MT_DATA, M_PKTHDR); if (rxbuf->m_head == NULL) panic("RX ring hdr initialization failed!\n"); rxbuf->m_head->m_len = MCLBYTES; rxbuf->m_head->m_flags &= ~M_HASFCS; /* we strip it */ rxbuf->m_head->m_pkthdr.len = MCLBYTES; /* Get the memory mapping */ error = bus_dmamap_load_mbuf_sg(rxr->rxtag, rxbuf->map, rxbuf->m_head, seg, &nsegs, BUS_DMA_NOWAIT); if (error != 0) panic("RX ring dma initialization failed!\n"); bus_dmamap_sync(rxr->rxtag, rxbuf->map, BUS_DMASYNC_PREREAD); /* Update descriptor */ rxr->rx_base[j].buffer_addr = htole64(seg[0].ds_addr); } /* Setup our descriptor indices */ rxr->next_to_check = 0; rxr->next_to_refresh = 0; bus_dmamap_sync(rxr->rxdma.dma_tag, rxr->rxdma.dma_map, BUS_DMASYNC_PREREAD | BUS_DMASYNC_PREWRITE); EM_RX_UNLOCK(rxr); return (0); } /********************************************************************* * * Initialize all receive rings. * **********************************************************************/ static int em_setup_receive_structures(struct adapter *adapter) { struct rx_ring *rxr = adapter->rx_rings; int j; for (j = 0; j < adapter->num_queues; j++, rxr++) if (em_setup_receive_ring(rxr)) goto fail; return (0); fail: /* * Free RX buffers allocated so far, we will only handle * the rings that completed, the failing case will have * cleaned up for itself. 'j' failed, so its the terminus. */ for (int i = 0; i < j; ++i) { rxr = &adapter->rx_rings[i]; for (int n = 0; n < adapter->num_rx_desc; n++) { struct em_buffer *rxbuf; rxbuf = &rxr->rx_buffers[n]; if (rxbuf->m_head != NULL) { bus_dmamap_sync(rxr->rxtag, rxbuf->map, BUS_DMASYNC_POSTREAD); bus_dmamap_unload(rxr->rxtag, rxbuf->map); m_freem(rxbuf->m_head); rxbuf->m_head = NULL; } } } return (ENOBUFS); } /********************************************************************* * * Free all receive rings. * **********************************************************************/ static void em_free_receive_structures(struct adapter *adapter) { struct rx_ring *rxr = adapter->rx_rings; for (int i = 0; i < adapter->num_queues; i++, rxr++) { em_free_receive_buffers(rxr); /* Free the ring memory as well */ em_dma_free(adapter, &rxr->rxdma); EM_RX_LOCK_DESTROY(rxr); } free(adapter->rx_rings, M_DEVBUF); } /********************************************************************* * * Free receive ring data structures * **********************************************************************/ static void em_free_receive_buffers(struct rx_ring *rxr) { struct adapter *adapter = rxr->adapter; struct em_buffer *rxbuf = NULL; INIT_DEBUGOUT("free_receive_buffers: begin"); if (rxr->rx_sparemap) { bus_dmamap_destroy(rxr->rxtag, rxr->rx_sparemap); rxr->rx_sparemap = NULL; } if (rxr->rx_buffers != NULL) { for (int i = 0; i < adapter->num_rx_desc; i++) { rxbuf = &rxr->rx_buffers[i]; if (rxbuf->map != NULL) { bus_dmamap_sync(rxr->rxtag, rxbuf->map, BUS_DMASYNC_POSTREAD); bus_dmamap_unload(rxr->rxtag, rxbuf->map); bus_dmamap_destroy(rxr->rxtag, rxbuf->map); } if (rxbuf->m_head != NULL) { m_freem(rxbuf->m_head); rxbuf->m_head = NULL; } } free(rxr->rx_buffers, M_DEVBUF); rxr->rx_buffers = NULL; } if (rxr->rxtag != NULL) { bus_dma_tag_destroy(rxr->rxtag); rxr->rxtag = NULL; } return; } /********************************************************************* * * Enable receive unit. * **********************************************************************/ #define MAX_INTS_PER_SEC 8000 #define DEFAULT_ITR 1000000000/(MAX_INTS_PER_SEC * 256) static void em_initialize_receive_unit(struct adapter *adapter) { struct rx_ring *rxr = adapter->rx_rings; struct ifnet *ifp = adapter->ifp; struct e1000_hw *hw = &adapter->hw; u64 bus_addr; u32 rctl, rxcsum; INIT_DEBUGOUT("em_initialize_receive_units: begin"); /* * Make sure receives are disabled while setting * up the descriptor ring */ rctl = E1000_READ_REG(hw, E1000_RCTL); E1000_WRITE_REG(hw, E1000_RCTL, rctl & ~E1000_RCTL_EN); E1000_WRITE_REG(&adapter->hw, E1000_RADV, adapter->rx_abs_int_delay.value); /* * Set the interrupt throttling rate. Value is calculated * as DEFAULT_ITR = 1/(MAX_INTS_PER_SEC * 256ns) */ E1000_WRITE_REG(hw, E1000_ITR, DEFAULT_ITR); /* ** When using MSIX interrupts we need to throttle ** using the EITR register (82574 only) */ if (hw->mac.type == e1000_82574) for (int i = 0; i < 4; i++) E1000_WRITE_REG(hw, E1000_EITR_82574(i), DEFAULT_ITR); /* Disable accelerated ackknowledge */ if (adapter->hw.mac.type == e1000_82574) E1000_WRITE_REG(hw, E1000_RFCTL, E1000_RFCTL_ACK_DIS); if (ifp->if_capenable & IFCAP_RXCSUM) { rxcsum = E1000_READ_REG(hw, E1000_RXCSUM); rxcsum |= (E1000_RXCSUM_IPOFL | E1000_RXCSUM_TUOFL); E1000_WRITE_REG(hw, E1000_RXCSUM, rxcsum); } /* ** XXX TEMPORARY WORKAROUND: on some systems with 82573 ** long latencies are observed, like Lenovo X60. This ** change eliminates the problem, but since having positive ** values in RDTR is a known source of problems on other ** platforms another solution is being sought. */ if (hw->mac.type == e1000_82573) E1000_WRITE_REG(hw, E1000_RDTR, 0x20); for (int i = 0; i < adapter->num_queues; i++, rxr++) { /* Setup the Base and Length of the Rx Descriptor Ring */ bus_addr = rxr->rxdma.dma_paddr; E1000_WRITE_REG(hw, E1000_RDLEN(i), adapter->num_rx_desc * sizeof(struct e1000_rx_desc)); E1000_WRITE_REG(hw, E1000_RDBAH(i), (u32)(bus_addr >> 32)); E1000_WRITE_REG(hw, E1000_RDBAL(i), (u32)bus_addr); /* Setup the Head and Tail Descriptor Pointers */ E1000_WRITE_REG(hw, E1000_RDH(i), 0); E1000_WRITE_REG(hw, E1000_RDT(i), adapter->num_rx_desc - 1); } /* Setup the Receive Control Register */ rctl &= ~(3 << E1000_RCTL_MO_SHIFT); rctl |= E1000_RCTL_EN | E1000_RCTL_BAM | E1000_RCTL_LBM_NO | E1000_RCTL_RDMTS_HALF | (hw->mac.mc_filter_type << E1000_RCTL_MO_SHIFT); /* Strip the CRC */ rctl |= E1000_RCTL_SECRC; /* Make sure VLAN Filters are off */ rctl &= ~E1000_RCTL_VFE; rctl &= ~E1000_RCTL_SBP; rctl |= E1000_RCTL_SZ_2048; if (ifp->if_mtu > ETHERMTU) rctl |= E1000_RCTL_LPE; else rctl &= ~E1000_RCTL_LPE; /* Write out the settings */ E1000_WRITE_REG(hw, E1000_RCTL, rctl); return; } /********************************************************************* * * This routine executes in interrupt context. It replenishes * the mbufs in the descriptor and sends data which has been * dma'ed into host memory to upper layer. * * We loop at most count times if count is > 0, or until done if * count < 0. * * For polling we also now return the number of cleaned packets *********************************************************************/ static int em_rxeof(struct rx_ring *rxr, int count) { struct adapter *adapter = rxr->adapter; struct ifnet *ifp = adapter->ifp; struct mbuf *mp, *sendmp; u8 status = 0; u16 len; int i, processed, rxdone = 0; bool eop; struct e1000_rx_desc *cur; EM_RX_LOCK_ASSERT(rxr); for (i = rxr->next_to_check, processed = 0; count != 0;) { if ((ifp->if_drv_flags & IFF_DRV_RUNNING) == 0) break; bus_dmamap_sync(rxr->rxdma.dma_tag, rxr->rxdma.dma_map, BUS_DMASYNC_POSTREAD | BUS_DMASYNC_POSTWRITE); cur = &rxr->rx_base[i]; status = cur->status; mp = sendmp = NULL; if ((status & E1000_RXD_STAT_DD) == 0) break; len = le16toh(cur->length); eop = (status & E1000_RXD_STAT_EOP) != 0; count--; if ((cur->errors & E1000_RXD_ERR_FRAME_ERR_MASK) == 0) { /* Assign correct length to the current fragment */ mp = rxr->rx_buffers[i].m_head; mp->m_len = len; if (rxr->fmp == NULL) { mp->m_pkthdr.len = len; rxr->fmp = mp; /* Store the first mbuf */ rxr->lmp = mp; } else { /* Chain mbuf's together */ mp->m_flags &= ~M_PKTHDR; rxr->lmp->m_next = mp; rxr->lmp = rxr->lmp->m_next; rxr->fmp->m_pkthdr.len += len; } if (eop) { rxr->fmp->m_pkthdr.rcvif = ifp; ifp->if_ipackets++; em_receive_checksum(cur, rxr->fmp); #ifndef __NO_STRICT_ALIGNMENT if (adapter->max_frame_size > (MCLBYTES - ETHER_ALIGN) && em_fixup_rx(rxr) != 0) goto skip; #endif if (status & E1000_RXD_STAT_VP) { rxr->fmp->m_pkthdr.ether_vtag = (le16toh(cur->special) & E1000_RXD_SPC_VLAN_MASK); rxr->fmp->m_flags |= M_VLANTAG; } #ifdef EM_MULTIQUEUE rxr->fmp->m_pkthdr.flowid = curcpu; rxr->fmp->m_flags |= M_FLOWID; #endif #ifndef __NO_STRICT_ALIGNMENT skip: #endif sendmp = rxr->fmp; rxr->fmp = NULL; rxr->lmp = NULL; } } else { ifp->if_ierrors++; /* Reuse loaded DMA map and just update mbuf chain */ mp = rxr->rx_buffers[i].m_head; mp->m_len = mp->m_pkthdr.len = MCLBYTES; mp->m_data = mp->m_ext.ext_buf; mp->m_next = NULL; if (adapter->max_frame_size <= (MCLBYTES - ETHER_ALIGN)) m_adj(mp, ETHER_ALIGN); if (rxr->fmp != NULL) { m_freem(rxr->fmp); rxr->fmp = NULL; rxr->lmp = NULL; } sendmp = NULL; } /* Zero out the receive descriptors status. */ cur->status = 0; ++rxdone; /* cumulative for POLL */ ++processed; /* Advance our pointers to the next descriptor. */ if (++i == adapter->num_rx_desc) i = 0; /* Send to the stack */ if (sendmp != NULL) (*ifp->if_input)(ifp, sendmp); /* Only refresh mbufs every 8 descriptors */ if (processed == 8) { em_refresh_mbufs(rxr, i); processed = 0; } } /* Catch any remaining refresh work */ if (processed != 0) { em_refresh_mbufs(rxr, i); processed = 0; } rxr->next_to_check = i; #ifdef DEVICE_POLLING return (rxdone); #else return ((status & E1000_RXD_STAT_DD) ? TRUE : FALSE); #endif } #ifndef __NO_STRICT_ALIGNMENT /* * When jumbo frames are enabled we should realign entire payload on * architecures with strict alignment. This is serious design mistake of 8254x * as it nullifies DMA operations. 8254x just allows RX buffer size to be * 2048/4096/8192/16384. What we really want is 2048 - ETHER_ALIGN to align its * payload. On architecures without strict alignment restrictions 8254x still * performs unaligned memory access which would reduce the performance too. * To avoid copying over an entire frame to align, we allocate a new mbuf and * copy ethernet header to the new mbuf. The new mbuf is prepended into the * existing mbuf chain. * * Be aware, best performance of the 8254x is achived only when jumbo frame is * not used at all on architectures with strict alignment. */ static int em_fixup_rx(struct rx_ring *rxr) { struct adapter *adapter = rxr->adapter; struct mbuf *m, *n; int error; error = 0; m = rxr->fmp; if (m->m_len <= (MCLBYTES - ETHER_HDR_LEN)) { bcopy(m->m_data, m->m_data + ETHER_HDR_LEN, m->m_len); m->m_data += ETHER_HDR_LEN; } else { MGETHDR(n, M_DONTWAIT, MT_DATA); if (n != NULL) { bcopy(m->m_data, n->m_data, ETHER_HDR_LEN); m->m_data += ETHER_HDR_LEN; m->m_len -= ETHER_HDR_LEN; n->m_len = ETHER_HDR_LEN; M_MOVE_PKTHDR(n, m); n->m_next = m; rxr->fmp = n; } else { adapter->dropped_pkts++; m_freem(rxr->fmp); rxr->fmp = NULL; error = ENOMEM; } } return (error); } #endif /********************************************************************* * * Verify that the hardware indicated that the checksum is valid. * Inform the stack about the status of checksum so that stack * doesn't spend time verifying the checksum. * *********************************************************************/ static void em_receive_checksum(struct e1000_rx_desc *rx_desc, struct mbuf *mp) { /* Ignore Checksum bit is set */ if (rx_desc->status & E1000_RXD_STAT_IXSM) { mp->m_pkthdr.csum_flags = 0; return; } if (rx_desc->status & E1000_RXD_STAT_IPCS) { /* Did it pass? */ if (!(rx_desc->errors & E1000_RXD_ERR_IPE)) { /* IP Checksum Good */ mp->m_pkthdr.csum_flags = CSUM_IP_CHECKED; mp->m_pkthdr.csum_flags |= CSUM_IP_VALID; } else { mp->m_pkthdr.csum_flags = 0; } } if (rx_desc->status & E1000_RXD_STAT_TCPCS) { /* Did it pass? */ if (!(rx_desc->errors & E1000_RXD_ERR_TCPE)) { mp->m_pkthdr.csum_flags |= (CSUM_DATA_VALID | CSUM_PSEUDO_HDR); mp->m_pkthdr.csum_data = htons(0xffff); } } } /* * This routine is run via an vlan * config EVENT */ static void em_register_vlan(void *arg, struct ifnet *ifp, u16 vtag) { struct adapter *adapter = ifp->if_softc; u32 index, bit; if (ifp->if_softc != arg) /* Not our event */ return; if ((vtag == 0) || (vtag > 4095)) /* Invalid ID */ return; index = (vtag >> 5) & 0x7F; bit = vtag & 0x1F; em_shadow_vfta[index] |= (1 << bit); ++adapter->num_vlans; /* Re-init to load the changes */ em_init(adapter); } /* * This routine is run via an vlan * unconfig EVENT */ static void em_unregister_vlan(void *arg, struct ifnet *ifp, u16 vtag) { struct adapter *adapter = ifp->if_softc; u32 index, bit; if (ifp->if_softc != arg) return; if ((vtag == 0) || (vtag > 4095)) /* Invalid */ return; index = (vtag >> 5) & 0x7F; bit = vtag & 0x1F; em_shadow_vfta[index] &= ~(1 << bit); --adapter->num_vlans; /* Re-init to load the changes */ em_init(adapter); } static void em_setup_vlan_hw_support(struct adapter *adapter) { struct e1000_hw *hw = &adapter->hw; u32 reg; /* ** We get here thru init_locked, meaning ** a soft reset, this has already cleared ** the VFTA and other state, so if there ** have been no vlan's registered do nothing. */ if (adapter->num_vlans == 0) return; /* ** A soft reset zero's out the VFTA, so ** we need to repopulate it now. */ for (int i = 0; i < EM_VFTA_SIZE; i++) if (em_shadow_vfta[i] != 0) E1000_WRITE_REG_ARRAY(hw, E1000_VFTA, i, em_shadow_vfta[i]); reg = E1000_READ_REG(hw, E1000_CTRL); reg |= E1000_CTRL_VME; E1000_WRITE_REG(hw, E1000_CTRL, reg); /* Enable the Filter Table */ reg = E1000_READ_REG(hw, E1000_RCTL); reg &= ~E1000_RCTL_CFIEN; reg |= E1000_RCTL_VFE; E1000_WRITE_REG(hw, E1000_RCTL, reg); /* Update the frame size */ E1000_WRITE_REG(&adapter->hw, E1000_RLPML, adapter->max_frame_size + VLAN_TAG_SIZE); } static void em_enable_intr(struct adapter *adapter) { struct e1000_hw *hw = &adapter->hw; u32 ims_mask = IMS_ENABLE_MASK; if (hw->mac.type == e1000_82574) { E1000_WRITE_REG(hw, EM_EIAC, EM_MSIX_MASK); ims_mask |= EM_MSIX_MASK; } E1000_WRITE_REG(hw, E1000_IMS, ims_mask); } static void em_disable_intr(struct adapter *adapter) { struct e1000_hw *hw = &adapter->hw; if (hw->mac.type == e1000_82574) E1000_WRITE_REG(hw, EM_EIAC, 0); E1000_WRITE_REG(&adapter->hw, E1000_IMC, 0xffffffff); } /* * Bit of a misnomer, what this really means is * to enable OS management of the system... aka * to disable special hardware management features */ static void em_init_manageability(struct adapter *adapter) { /* A shared code workaround */ #define E1000_82542_MANC2H E1000_MANC2H if (adapter->has_manage) { int manc2h = E1000_READ_REG(&adapter->hw, E1000_MANC2H); int manc = E1000_READ_REG(&adapter->hw, E1000_MANC); /* disable hardware interception of ARP */ manc &= ~(E1000_MANC_ARP_EN); /* enable receiving management packets to the host */ manc |= E1000_MANC_EN_MNG2HOST; #define E1000_MNG2HOST_PORT_623 (1 << 5) #define E1000_MNG2HOST_PORT_664 (1 << 6) manc2h |= E1000_MNG2HOST_PORT_623; manc2h |= E1000_MNG2HOST_PORT_664; E1000_WRITE_REG(&adapter->hw, E1000_MANC2H, manc2h); E1000_WRITE_REG(&adapter->hw, E1000_MANC, manc); } } /* * Give control back to hardware management * controller if there is one. */ static void em_release_manageability(struct adapter *adapter) { if (adapter->has_manage) { int manc = E1000_READ_REG(&adapter->hw, E1000_MANC); /* re-enable hardware interception of ARP */ manc |= E1000_MANC_ARP_EN; manc &= ~E1000_MANC_EN_MNG2HOST; E1000_WRITE_REG(&adapter->hw, E1000_MANC, manc); } } /* * em_get_hw_control sets the {CTRL_EXT|FWSM}:DRV_LOAD bit. * For ASF and Pass Through versions of f/w this means * that the driver is loaded. For AMT version type f/w * this means that the network i/f is open. */ static void em_get_hw_control(struct adapter *adapter) { u32 ctrl_ext, swsm; if (adapter->hw.mac.type == e1000_82573) { swsm = E1000_READ_REG(&adapter->hw, E1000_SWSM); E1000_WRITE_REG(&adapter->hw, E1000_SWSM, swsm | E1000_SWSM_DRV_LOAD); return; } /* else */ ctrl_ext = E1000_READ_REG(&adapter->hw, E1000_CTRL_EXT); E1000_WRITE_REG(&adapter->hw, E1000_CTRL_EXT, ctrl_ext | E1000_CTRL_EXT_DRV_LOAD); return; } /* * em_release_hw_control resets {CTRL_EXT|FWSM}:DRV_LOAD bit. * For ASF and Pass Through versions of f/w this means that * the driver is no longer loaded. For AMT versions of the * f/w this means that the network i/f is closed. */ static void em_release_hw_control(struct adapter *adapter) { u32 ctrl_ext, swsm; if (!adapter->has_manage) return; if (adapter->hw.mac.type == e1000_82573) { swsm = E1000_READ_REG(&adapter->hw, E1000_SWSM); E1000_WRITE_REG(&adapter->hw, E1000_SWSM, swsm & ~E1000_SWSM_DRV_LOAD); return; } /* else */ ctrl_ext = E1000_READ_REG(&adapter->hw, E1000_CTRL_EXT); E1000_WRITE_REG(&adapter->hw, E1000_CTRL_EXT, ctrl_ext & ~E1000_CTRL_EXT_DRV_LOAD); return; } static int em_is_valid_ether_addr(u8 *addr) { char zero_addr[6] = { 0, 0, 0, 0, 0, 0 }; if ((addr[0] & 1) || (!bcmp(addr, zero_addr, ETHER_ADDR_LEN))) { return (FALSE); } return (TRUE); } /* ** Parse the interface capabilities with regard ** to both system management and wake-on-lan for ** later use. */ static void em_get_wakeup(device_t dev) { struct adapter *adapter = device_get_softc(dev); u16 eeprom_data = 0, device_id, apme_mask; adapter->has_manage = e1000_enable_mng_pass_thru(&adapter->hw); apme_mask = EM_EEPROM_APME; switch (adapter->hw.mac.type) { case e1000_82573: case e1000_82583: adapter->has_amt = TRUE; /* Falls thru */ case e1000_82571: case e1000_82572: case e1000_80003es2lan: if (adapter->hw.bus.func == 1) { e1000_read_nvm(&adapter->hw, NVM_INIT_CONTROL3_PORT_B, 1, &eeprom_data); break; } else e1000_read_nvm(&adapter->hw, NVM_INIT_CONTROL3_PORT_A, 1, &eeprom_data); break; case e1000_ich8lan: case e1000_ich9lan: case e1000_ich10lan: case e1000_pchlan: apme_mask = E1000_WUC_APME; adapter->has_amt = TRUE; eeprom_data = E1000_READ_REG(&adapter->hw, E1000_WUC); break; default: e1000_read_nvm(&adapter->hw, NVM_INIT_CONTROL3_PORT_A, 1, &eeprom_data); break; } if (eeprom_data & apme_mask) adapter->wol = (E1000_WUFC_MAG | E1000_WUFC_MC); /* * We have the eeprom settings, now apply the special cases * where the eeprom may be wrong or the board won't support * wake on lan on a particular port */ device_id = pci_get_device(dev); switch (device_id) { case E1000_DEV_ID_82571EB_FIBER: /* Wake events only supported on port A for dual fiber * regardless of eeprom setting */ if (E1000_READ_REG(&adapter->hw, E1000_STATUS) & E1000_STATUS_FUNC_1) adapter->wol = 0; break; case E1000_DEV_ID_82571EB_QUAD_COPPER: case E1000_DEV_ID_82571EB_QUAD_FIBER: case E1000_DEV_ID_82571EB_QUAD_COPPER_LP: /* if quad port adapter, disable WoL on all but port A */ if (global_quad_port_a != 0) adapter->wol = 0; /* Reset for multiple quad port adapters */ if (++global_quad_port_a == 4) global_quad_port_a = 0; break; } return; } /* * Enable PCI Wake On Lan capability */ static void em_enable_wakeup(device_t dev) { struct adapter *adapter = device_get_softc(dev); struct ifnet *ifp = adapter->ifp; u32 pmc, ctrl, ctrl_ext, rctl; u16 status; if ((pci_find_extcap(dev, PCIY_PMG, &pmc) != 0)) return; /* Advertise the wakeup capability */ ctrl = E1000_READ_REG(&adapter->hw, E1000_CTRL); ctrl |= (E1000_CTRL_SWDPIN2 | E1000_CTRL_SWDPIN3); E1000_WRITE_REG(&adapter->hw, E1000_CTRL, ctrl); E1000_WRITE_REG(&adapter->hw, E1000_WUC, E1000_WUC_PME_EN); if ((adapter->hw.mac.type == e1000_ich8lan) || (adapter->hw.mac.type == e1000_pchlan) || (adapter->hw.mac.type == e1000_ich9lan) || (adapter->hw.mac.type == e1000_ich10lan)) { e1000_disable_gig_wol_ich8lan(&adapter->hw); e1000_hv_phy_powerdown_workaround_ich8lan(&adapter->hw); } /* Keep the laser running on Fiber adapters */ if (adapter->hw.phy.media_type == e1000_media_type_fiber || adapter->hw.phy.media_type == e1000_media_type_internal_serdes) { ctrl_ext = E1000_READ_REG(&adapter->hw, E1000_CTRL_EXT); ctrl_ext |= E1000_CTRL_EXT_SDP3_DATA; E1000_WRITE_REG(&adapter->hw, E1000_CTRL_EXT, ctrl_ext); } /* ** Determine type of Wakeup: note that wol ** is set with all bits on by default. */ if ((ifp->if_capenable & IFCAP_WOL_MAGIC) == 0) adapter->wol &= ~E1000_WUFC_MAG; if ((ifp->if_capenable & IFCAP_WOL_MCAST) == 0) adapter->wol &= ~E1000_WUFC_MC; else { rctl = E1000_READ_REG(&adapter->hw, E1000_RCTL); rctl |= E1000_RCTL_MPE; E1000_WRITE_REG(&adapter->hw, E1000_RCTL, rctl); } if (adapter->hw.mac.type == e1000_pchlan) { if (em_enable_phy_wakeup(adapter)) return; } else { E1000_WRITE_REG(&adapter->hw, E1000_WUC, E1000_WUC_PME_EN); E1000_WRITE_REG(&adapter->hw, E1000_WUFC, adapter->wol); } if (adapter->hw.phy.type == e1000_phy_igp_3) e1000_igp3_phy_powerdown_workaround_ich8lan(&adapter->hw); /* Request PME */ status = pci_read_config(dev, pmc + PCIR_POWER_STATUS, 2); status &= ~(PCIM_PSTAT_PME | PCIM_PSTAT_PMEENABLE); if (ifp->if_capenable & IFCAP_WOL) status |= PCIM_PSTAT_PME | PCIM_PSTAT_PMEENABLE; pci_write_config(dev, pmc + PCIR_POWER_STATUS, status, 2); return; } /* ** WOL in the newer chipset interfaces (pchlan) ** require thing to be copied into the phy */ static int em_enable_phy_wakeup(struct adapter *adapter) { struct e1000_hw *hw = &adapter->hw; u32 mreg, ret = 0; u16 preg; /* copy MAC RARs to PHY RARs */ for (int i = 0; i < adapter->hw.mac.rar_entry_count; i++) { mreg = E1000_READ_REG(hw, E1000_RAL(i)); e1000_write_phy_reg(hw, BM_RAR_L(i), (u16)(mreg & 0xFFFF)); e1000_write_phy_reg(hw, BM_RAR_M(i), (u16)((mreg >> 16) & 0xFFFF)); mreg = E1000_READ_REG(hw, E1000_RAH(i)); e1000_write_phy_reg(hw, BM_RAR_H(i), (u16)(mreg & 0xFFFF)); e1000_write_phy_reg(hw, BM_RAR_CTRL(i), (u16)((mreg >> 16) & 0xFFFF)); } /* copy MAC MTA to PHY MTA */ for (int i = 0; i < adapter->hw.mac.mta_reg_count; i++) { mreg = E1000_READ_REG_ARRAY(hw, E1000_MTA, i); e1000_write_phy_reg(hw, BM_MTA(i), (u16)(mreg & 0xFFFF)); e1000_write_phy_reg(hw, BM_MTA(i) + 1, (u16)((mreg >> 16) & 0xFFFF)); } /* configure PHY Rx Control register */ e1000_read_phy_reg(&adapter->hw, BM_RCTL, &preg); mreg = E1000_READ_REG(hw, E1000_RCTL); if (mreg & E1000_RCTL_UPE) preg |= BM_RCTL_UPE; if (mreg & E1000_RCTL_MPE) preg |= BM_RCTL_MPE; preg &= ~(BM_RCTL_MO_MASK); if (mreg & E1000_RCTL_MO_3) preg |= (((mreg & E1000_RCTL_MO_3) >> E1000_RCTL_MO_SHIFT) << BM_RCTL_MO_SHIFT); if (mreg & E1000_RCTL_BAM) preg |= BM_RCTL_BAM; if (mreg & E1000_RCTL_PMCF) preg |= BM_RCTL_PMCF; mreg = E1000_READ_REG(hw, E1000_CTRL); if (mreg & E1000_CTRL_RFCE) preg |= BM_RCTL_RFCE; e1000_write_phy_reg(&adapter->hw, BM_RCTL, preg); /* enable PHY wakeup in MAC register */ E1000_WRITE_REG(hw, E1000_WUC, E1000_WUC_PHY_WAKE | E1000_WUC_PME_EN); E1000_WRITE_REG(hw, E1000_WUFC, adapter->wol); /* configure and enable PHY wakeup in PHY registers */ e1000_write_phy_reg(&adapter->hw, BM_WUFC, adapter->wol); e1000_write_phy_reg(&adapter->hw, BM_WUC, E1000_WUC_PME_EN); /* activate PHY wakeup */ ret = hw->phy.ops.acquire(hw); if (ret) { printf("Could not acquire PHY\n"); return ret; } e1000_write_phy_reg_mdic(hw, IGP01E1000_PHY_PAGE_SELECT, (BM_WUC_ENABLE_PAGE << IGP_PAGE_SHIFT)); ret = e1000_read_phy_reg_mdic(hw, BM_WUC_ENABLE_REG, &preg); if (ret) { printf("Could not read PHY page 769\n"); goto out; } preg |= BM_WUC_ENABLE_BIT | BM_WUC_HOST_WU_BIT; ret = e1000_write_phy_reg_mdic(hw, BM_WUC_ENABLE_REG, preg); if (ret) printf("Could not set PHY Host Wakeup bit\n"); out: hw->phy.ops.release(hw); return ret; } static void em_led_func(void *arg, int onoff) { struct adapter *adapter = arg; EM_CORE_LOCK(adapter); if (onoff) { e1000_setup_led(&adapter->hw); e1000_led_on(&adapter->hw); } else { e1000_led_off(&adapter->hw); e1000_cleanup_led(&adapter->hw); } EM_CORE_UNLOCK(adapter); } /********************************************************************** * * Update the board statistics counters. * **********************************************************************/ static void em_update_stats_counters(struct adapter *adapter) { struct ifnet *ifp; if(adapter->hw.phy.media_type == e1000_media_type_copper || (E1000_READ_REG(&adapter->hw, E1000_STATUS) & E1000_STATUS_LU)) { adapter->stats.symerrs += E1000_READ_REG(&adapter->hw, E1000_SYMERRS); adapter->stats.sec += E1000_READ_REG(&adapter->hw, E1000_SEC); } adapter->stats.crcerrs += E1000_READ_REG(&adapter->hw, E1000_CRCERRS); adapter->stats.mpc += E1000_READ_REG(&adapter->hw, E1000_MPC); adapter->stats.scc += E1000_READ_REG(&adapter->hw, E1000_SCC); adapter->stats.ecol += E1000_READ_REG(&adapter->hw, E1000_ECOL); adapter->stats.mcc += E1000_READ_REG(&adapter->hw, E1000_MCC); adapter->stats.latecol += E1000_READ_REG(&adapter->hw, E1000_LATECOL); adapter->stats.colc += E1000_READ_REG(&adapter->hw, E1000_COLC); adapter->stats.dc += E1000_READ_REG(&adapter->hw, E1000_DC); adapter->stats.rlec += E1000_READ_REG(&adapter->hw, E1000_RLEC); adapter->stats.xonrxc += E1000_READ_REG(&adapter->hw, E1000_XONRXC); adapter->stats.xontxc += E1000_READ_REG(&adapter->hw, E1000_XONTXC); adapter->stats.xoffrxc += E1000_READ_REG(&adapter->hw, E1000_XOFFRXC); adapter->stats.xofftxc += E1000_READ_REG(&adapter->hw, E1000_XOFFTXC); adapter->stats.fcruc += E1000_READ_REG(&adapter->hw, E1000_FCRUC); adapter->stats.prc64 += E1000_READ_REG(&adapter->hw, E1000_PRC64); adapter->stats.prc127 += E1000_READ_REG(&adapter->hw, E1000_PRC127); adapter->stats.prc255 += E1000_READ_REG(&adapter->hw, E1000_PRC255); adapter->stats.prc511 += E1000_READ_REG(&adapter->hw, E1000_PRC511); adapter->stats.prc1023 += E1000_READ_REG(&adapter->hw, E1000_PRC1023); adapter->stats.prc1522 += E1000_READ_REG(&adapter->hw, E1000_PRC1522); adapter->stats.gprc += E1000_READ_REG(&adapter->hw, E1000_GPRC); adapter->stats.bprc += E1000_READ_REG(&adapter->hw, E1000_BPRC); adapter->stats.mprc += E1000_READ_REG(&adapter->hw, E1000_MPRC); adapter->stats.gptc += E1000_READ_REG(&adapter->hw, E1000_GPTC); /* For the 64-bit byte counters the low dword must be read first. */ /* Both registers clear on the read of the high dword */ adapter->stats.gorc += E1000_READ_REG(&adapter->hw, E1000_GORCH); adapter->stats.gotc += E1000_READ_REG(&adapter->hw, E1000_GOTCH); adapter->stats.rnbc += E1000_READ_REG(&adapter->hw, E1000_RNBC); adapter->stats.ruc += E1000_READ_REG(&adapter->hw, E1000_RUC); adapter->stats.rfc += E1000_READ_REG(&adapter->hw, E1000_RFC); adapter->stats.roc += E1000_READ_REG(&adapter->hw, E1000_ROC); adapter->stats.rjc += E1000_READ_REG(&adapter->hw, E1000_RJC); adapter->stats.tor += E1000_READ_REG(&adapter->hw, E1000_TORH); adapter->stats.tot += E1000_READ_REG(&adapter->hw, E1000_TOTH); adapter->stats.tpr += E1000_READ_REG(&adapter->hw, E1000_TPR); adapter->stats.tpt += E1000_READ_REG(&adapter->hw, E1000_TPT); adapter->stats.ptc64 += E1000_READ_REG(&adapter->hw, E1000_PTC64); adapter->stats.ptc127 += E1000_READ_REG(&adapter->hw, E1000_PTC127); adapter->stats.ptc255 += E1000_READ_REG(&adapter->hw, E1000_PTC255); adapter->stats.ptc511 += E1000_READ_REG(&adapter->hw, E1000_PTC511); adapter->stats.ptc1023 += E1000_READ_REG(&adapter->hw, E1000_PTC1023); adapter->stats.ptc1522 += E1000_READ_REG(&adapter->hw, E1000_PTC1522); adapter->stats.mptc += E1000_READ_REG(&adapter->hw, E1000_MPTC); adapter->stats.bptc += E1000_READ_REG(&adapter->hw, E1000_BPTC); if (adapter->hw.mac.type >= e1000_82543) { adapter->stats.algnerrc += E1000_READ_REG(&adapter->hw, E1000_ALGNERRC); adapter->stats.rxerrc += E1000_READ_REG(&adapter->hw, E1000_RXERRC); adapter->stats.tncrs += E1000_READ_REG(&adapter->hw, E1000_TNCRS); adapter->stats.cexterr += E1000_READ_REG(&adapter->hw, E1000_CEXTERR); adapter->stats.tsctc += E1000_READ_REG(&adapter->hw, E1000_TSCTC); adapter->stats.tsctfc += E1000_READ_REG(&adapter->hw, E1000_TSCTFC); } ifp = adapter->ifp; ifp->if_collisions = adapter->stats.colc; /* Rx Errors */ ifp->if_ierrors = adapter->dropped_pkts + adapter->stats.rxerrc + adapter->stats.crcerrs + adapter->stats.algnerrc + adapter->stats.ruc + adapter->stats.roc + adapter->stats.mpc + adapter->stats.cexterr; /* Tx Errors */ ifp->if_oerrors = adapter->stats.ecol + adapter->stats.latecol + adapter->watchdog_events; } /********************************************************************** * * This routine is called only when em_display_debug_stats is enabled. * This routine provides a way to take a look at important statistics * maintained by the driver and hardware. * **********************************************************************/ static void em_print_debug_info(struct adapter *adapter) { device_t dev = adapter->dev; u8 *hw_addr = adapter->hw.hw_addr; struct rx_ring *rxr = adapter->rx_rings; struct tx_ring *txr = adapter->tx_rings; device_printf(dev, "Adapter hardware address = %p \n", hw_addr); device_printf(dev, "CTRL = 0x%x RCTL = 0x%x \n", E1000_READ_REG(&adapter->hw, E1000_CTRL), E1000_READ_REG(&adapter->hw, E1000_RCTL)); device_printf(dev, "Packet buffer = Tx=%dk Rx=%dk \n", ((E1000_READ_REG(&adapter->hw, E1000_PBA) & 0xffff0000) >> 16),\ (E1000_READ_REG(&adapter->hw, E1000_PBA) & 0xffff) ); device_printf(dev, "Flow control watermarks high = %d low = %d\n", adapter->hw.fc.high_water, adapter->hw.fc.low_water); device_printf(dev, "tx_int_delay = %d, tx_abs_int_delay = %d\n", E1000_READ_REG(&adapter->hw, E1000_TIDV), E1000_READ_REG(&adapter->hw, E1000_TADV)); device_printf(dev, "rx_int_delay = %d, rx_abs_int_delay = %d\n", E1000_READ_REG(&adapter->hw, E1000_RDTR), E1000_READ_REG(&adapter->hw, E1000_RADV)); for (int i = 0; i < adapter->num_queues; i++, txr++) { device_printf(dev, "Queue(%d) tdh = %d, tdt = %d\n", i, E1000_READ_REG(&adapter->hw, E1000_TDH(i)), E1000_READ_REG(&adapter->hw, E1000_TDT(i))); device_printf(dev, "TX(%d) no descriptors avail event = %ld\n", txr->me, txr->no_desc_avail); device_printf(dev, "TX(%d) MSIX IRQ Handled = %ld\n", txr->me, txr->tx_irq); device_printf(dev, "Num Tx descriptors avail = %d\n", txr->tx_avail); device_printf(dev, "Tx Descriptors not avail1 = %ld\n", txr->no_desc_avail); } for (int i = 0; i < adapter->num_queues; i++, rxr++) { device_printf(dev, "RX(%d) MSIX IRQ Handled = %ld\n", rxr->me, rxr->rx_irq); device_printf(dev, "hw rdh = %d, hw rdt = %d\n", E1000_READ_REG(&adapter->hw, E1000_RDH(i)), E1000_READ_REG(&adapter->hw, E1000_RDT(i))); } device_printf(dev, "Std mbuf failed = %ld\n", adapter->mbuf_alloc_failed); device_printf(dev, "Std mbuf cluster failed = %ld\n", adapter->mbuf_cluster_failed); device_printf(dev, "Driver dropped packets = %ld\n", adapter->dropped_pkts); } static void em_print_hw_stats(struct adapter *adapter) { device_t dev = adapter->dev; device_printf(dev, "Excessive collisions = %lld\n", (long long)adapter->stats.ecol); #if (DEBUG_HW > 0) /* Dont output these errors normally */ device_printf(dev, "Symbol errors = %lld\n", (long long)adapter->stats.symerrs); #endif device_printf(dev, "Sequence errors = %lld\n", (long long)adapter->stats.sec); device_printf(dev, "Defer count = %lld\n", (long long)adapter->stats.dc); device_printf(dev, "Missed Packets = %lld\n", (long long)adapter->stats.mpc); device_printf(dev, "Receive No Buffers = %lld\n", (long long)adapter->stats.rnbc); /* RLEC is inaccurate on some hardware, calculate our own. */ device_printf(dev, "Receive Length Errors = %lld\n", ((long long)adapter->stats.roc + (long long)adapter->stats.ruc)); device_printf(dev, "Receive errors = %lld\n", (long long)adapter->stats.rxerrc); device_printf(dev, "Crc errors = %lld\n", (long long)adapter->stats.crcerrs); device_printf(dev, "Alignment errors = %lld\n", (long long)adapter->stats.algnerrc); device_printf(dev, "Collision/Carrier extension errors = %lld\n", (long long)adapter->stats.cexterr); device_printf(dev, "watchdog timeouts = %ld\n", adapter->watchdog_events); device_printf(dev, "XON Rcvd = %lld\n", (long long)adapter->stats.xonrxc); device_printf(dev, "XON Xmtd = %lld\n", (long long)adapter->stats.xontxc); device_printf(dev, "XOFF Rcvd = %lld\n", (long long)adapter->stats.xoffrxc); device_printf(dev, "XOFF Xmtd = %lld\n", (long long)adapter->stats.xofftxc); device_printf(dev, "Good Packets Rcvd = %lld\n", (long long)adapter->stats.gprc); device_printf(dev, "Good Packets Xmtd = %lld\n", (long long)adapter->stats.gptc); device_printf(dev, "TSO Contexts Xmtd = %lld\n", (long long)adapter->stats.tsctc); device_printf(dev, "TSO Contexts Failed = %lld\n", (long long)adapter->stats.tsctfc); } /********************************************************************** * * This routine provides a way to dump out the adapter eeprom, * often a useful debug/service tool. This only dumps the first * 32 words, stuff that matters is in that extent. * **********************************************************************/ static void em_print_nvm_info(struct adapter *adapter) { u16 eeprom_data; int i, j, row = 0; /* Its a bit crude, but it gets the job done */ printf("\nInterface EEPROM Dump:\n"); printf("Offset\n0x0000 "); for (i = 0, j = 0; i < 32; i++, j++) { if (j == 8) { /* Make the offset block */ j = 0; ++row; printf("\n0x00%x0 ",row); } e1000_read_nvm(&adapter->hw, i, 1, &eeprom_data); printf("%04x ", eeprom_data); } printf("\n"); } static int em_sysctl_debug_info(SYSCTL_HANDLER_ARGS) { struct adapter *adapter; int error; int result; result = -1; error = sysctl_handle_int(oidp, &result, 0, req); if (error || !req->newptr) return (error); if (result == 1) { adapter = (struct adapter *)arg1; em_print_debug_info(adapter); } /* * This value will cause a hex dump of the * first 32 16-bit words of the EEPROM to * the screen. */ if (result == 2) { adapter = (struct adapter *)arg1; em_print_nvm_info(adapter); } return (error); } static int em_sysctl_stats(SYSCTL_HANDLER_ARGS) { struct adapter *adapter; int error; int result; result = -1; error = sysctl_handle_int(oidp, &result, 0, req); if (error || !req->newptr) return (error); if (result == 1) { adapter = (struct adapter *)arg1; em_print_hw_stats(adapter); } return (error); } static int em_sysctl_int_delay(SYSCTL_HANDLER_ARGS) { struct em_int_delay_info *info; struct adapter *adapter; u32 regval; int error, usecs, ticks; info = (struct em_int_delay_info *)arg1; usecs = info->value; error = sysctl_handle_int(oidp, &usecs, 0, req); if (error != 0 || req->newptr == NULL) return (error); if (usecs < 0 || usecs > EM_TICKS_TO_USECS(65535)) return (EINVAL); info->value = usecs; ticks = EM_USECS_TO_TICKS(usecs); adapter = info->adapter; EM_CORE_LOCK(adapter); regval = E1000_READ_OFFSET(&adapter->hw, info->offset); regval = (regval & ~0xffff) | (ticks & 0xffff); /* Handle a few special cases. */ switch (info->offset) { case E1000_RDTR: break; case E1000_TIDV: if (ticks == 0) { adapter->txd_cmd &= ~E1000_TXD_CMD_IDE; /* Don't write 0 into the TIDV register. */ regval++; } else adapter->txd_cmd |= E1000_TXD_CMD_IDE; break; } E1000_WRITE_OFFSET(&adapter->hw, info->offset, regval); EM_CORE_UNLOCK(adapter); return (0); } static void em_add_int_delay_sysctl(struct adapter *adapter, const char *name, const char *description, struct em_int_delay_info *info, int offset, int value) { info->adapter = adapter; info->offset = offset; info->value = value; SYSCTL_ADD_PROC(device_get_sysctl_ctx(adapter->dev), SYSCTL_CHILDREN(device_get_sysctl_tree(adapter->dev)), OID_AUTO, name, CTLTYPE_INT|CTLFLAG_RW, info, 0, em_sysctl_int_delay, "I", description); } static void em_add_rx_process_limit(struct adapter *adapter, const char *name, const char *description, int *limit, int value) { *limit = value; SYSCTL_ADD_INT(device_get_sysctl_ctx(adapter->dev), SYSCTL_CHILDREN(device_get_sysctl_tree(adapter->dev)), OID_AUTO, name, CTLTYPE_INT|CTLFLAG_RW, limit, value, description); } Index: user/imp/tbemd/sys/dev/e1000/if_igb.c =================================================================== --- user/imp/tbemd/sys/dev/e1000/if_igb.c (revision 206684) +++ user/imp/tbemd/sys/dev/e1000/if_igb.c (revision 206685) @@ -1,5014 +1,5024 @@ /****************************************************************************** Copyright (c) 2001-2010, Intel Corporation All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. 3. Neither the name of the Intel Corporation nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. ******************************************************************************/ /*$FreeBSD$*/ #ifdef HAVE_KERNEL_OPTION_HEADERS #include "opt_device_polling.h" #include "opt_inet.h" #include "opt_altq.h" #endif #include #include #if __FreeBSD_version >= 800000 #include #endif #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include "e1000_api.h" #include "e1000_82575.h" #include "if_igb.h" /********************************************************************* * Set this to one to display debug statistics *********************************************************************/ int igb_display_debug_stats = 0; /********************************************************************* * Driver version: *********************************************************************/ -char igb_driver_version[] = "version - 1.9.4"; +char igb_driver_version[] = "version - 1.9.5"; /********************************************************************* * PCI Device ID Table * * Used by probe to select devices to load on * Last field stores an index into e1000_strings * Last entry must be all 0s * * { Vendor ID, Device ID, SubVendor ID, SubDevice ID, String Index } *********************************************************************/ static igb_vendor_info_t igb_vendor_info_array[] = { { 0x8086, E1000_DEV_ID_82575EB_COPPER, PCI_ANY_ID, PCI_ANY_ID, 0}, { 0x8086, E1000_DEV_ID_82575EB_FIBER_SERDES, PCI_ANY_ID, PCI_ANY_ID, 0}, { 0x8086, E1000_DEV_ID_82575GB_QUAD_COPPER, PCI_ANY_ID, PCI_ANY_ID, 0}, { 0x8086, E1000_DEV_ID_82576, PCI_ANY_ID, PCI_ANY_ID, 0}, { 0x8086, E1000_DEV_ID_82576_NS, PCI_ANY_ID, PCI_ANY_ID, 0}, { 0x8086, E1000_DEV_ID_82576_NS_SERDES, PCI_ANY_ID, PCI_ANY_ID, 0}, { 0x8086, E1000_DEV_ID_82576_FIBER, PCI_ANY_ID, PCI_ANY_ID, 0}, { 0x8086, E1000_DEV_ID_82576_SERDES, PCI_ANY_ID, PCI_ANY_ID, 0}, { 0x8086, E1000_DEV_ID_82576_SERDES_QUAD, PCI_ANY_ID, PCI_ANY_ID, 0}, { 0x8086, E1000_DEV_ID_82576_QUAD_COPPER, PCI_ANY_ID, PCI_ANY_ID, 0}, { 0x8086, E1000_DEV_ID_82580_COPPER, PCI_ANY_ID, PCI_ANY_ID, 0}, { 0x8086, E1000_DEV_ID_82580_FIBER, PCI_ANY_ID, PCI_ANY_ID, 0}, { 0x8086, E1000_DEV_ID_82580_SERDES, PCI_ANY_ID, PCI_ANY_ID, 0}, { 0x8086, E1000_DEV_ID_82580_SGMII, PCI_ANY_ID, PCI_ANY_ID, 0}, { 0x8086, E1000_DEV_ID_82580_COPPER_DUAL, PCI_ANY_ID, PCI_ANY_ID, 0}, /* required last entry */ { 0, 0, 0, 0, 0} }; /********************************************************************* * Table of branding strings for all supported NICs. *********************************************************************/ static char *igb_strings[] = { "Intel(R) PRO/1000 Network Connection" }; /********************************************************************* * Function prototypes *********************************************************************/ static int igb_probe(device_t); static int igb_attach(device_t); static int igb_detach(device_t); static int igb_shutdown(device_t); static int igb_suspend(device_t); static int igb_resume(device_t); static void igb_start(struct ifnet *); static void igb_start_locked(struct tx_ring *, struct ifnet *ifp); #if __FreeBSD_version >= 800000 static int igb_mq_start(struct ifnet *, struct mbuf *); static int igb_mq_start_locked(struct ifnet *, struct tx_ring *, struct mbuf *); static void igb_qflush(struct ifnet *); #endif static int igb_ioctl(struct ifnet *, u_long, caddr_t); static void igb_init(void *); static void igb_init_locked(struct adapter *); static void igb_stop(void *); static void igb_media_status(struct ifnet *, struct ifmediareq *); static int igb_media_change(struct ifnet *); static void igb_identify_hardware(struct adapter *); static int igb_allocate_pci_resources(struct adapter *); static int igb_allocate_msix(struct adapter *); static int igb_allocate_legacy(struct adapter *); static int igb_setup_msix(struct adapter *); static void igb_free_pci_resources(struct adapter *); static void igb_local_timer(void *); static void igb_reset(struct adapter *); static void igb_setup_interface(device_t, struct adapter *); static int igb_allocate_queues(struct adapter *); static void igb_configure_queues(struct adapter *); static int igb_allocate_transmit_buffers(struct tx_ring *); static void igb_setup_transmit_structures(struct adapter *); static void igb_setup_transmit_ring(struct tx_ring *); static void igb_initialize_transmit_units(struct adapter *); static void igb_free_transmit_structures(struct adapter *); static void igb_free_transmit_buffers(struct tx_ring *); static int igb_allocate_receive_buffers(struct rx_ring *); static int igb_setup_receive_structures(struct adapter *); static int igb_setup_receive_ring(struct rx_ring *); static void igb_initialize_receive_units(struct adapter *); static void igb_free_receive_structures(struct adapter *); static void igb_free_receive_buffers(struct rx_ring *); static void igb_free_receive_ring(struct rx_ring *); static void igb_enable_intr(struct adapter *); static void igb_disable_intr(struct adapter *); static void igb_update_stats_counters(struct adapter *); static bool igb_txeof(struct tx_ring *); static __inline void igb_rx_discard(struct rx_ring *, int); static __inline void igb_rx_input(struct rx_ring *, struct ifnet *, struct mbuf *, u32); static bool igb_rxeof(struct igb_queue *, int); static void igb_rx_checksum(u32, struct mbuf *, u32); static int igb_tx_ctx_setup(struct tx_ring *, struct mbuf *); static bool igb_tso_setup(struct tx_ring *, struct mbuf *, u32 *); static void igb_set_promisc(struct adapter *); static void igb_disable_promisc(struct adapter *); static void igb_set_multi(struct adapter *); static void igb_print_hw_stats(struct adapter *); static void igb_update_link_status(struct adapter *); static void igb_refresh_mbufs(struct rx_ring *, int); static void igb_register_vlan(void *, struct ifnet *, u16); static void igb_unregister_vlan(void *, struct ifnet *, u16); static void igb_setup_vlan_hw_support(struct adapter *); static int igb_xmit(struct tx_ring *, struct mbuf **); static int igb_dma_malloc(struct adapter *, bus_size_t, struct igb_dma_alloc *, int); static void igb_dma_free(struct adapter *, struct igb_dma_alloc *); static void igb_print_debug_info(struct adapter *); static void igb_print_nvm_info(struct adapter *); static int igb_is_valid_ether_addr(u8 *); static int igb_sysctl_stats(SYSCTL_HANDLER_ARGS); static int igb_sysctl_debug_info(SYSCTL_HANDLER_ARGS); /* Management and WOL Support */ static void igb_init_manageability(struct adapter *); static void igb_release_manageability(struct adapter *); static void igb_get_hw_control(struct adapter *); static void igb_release_hw_control(struct adapter *); static void igb_enable_wakeup(device_t); static void igb_led_func(void *, int); static int igb_irq_fast(void *); static void igb_add_rx_process_limit(struct adapter *, const char *, const char *, int *, int); static void igb_handle_rxtx(void *context, int pending); static void igb_handle_que(void *context, int pending); static void igb_handle_link(void *context, int pending); /* These are MSIX only irq handlers */ static void igb_msix_que(void *); static void igb_msix_link(void *); #ifdef DEVICE_POLLING static poll_handler_t igb_poll; #endif /* POLLING */ /********************************************************************* * FreeBSD Device Interface Entry Points *********************************************************************/ static device_method_t igb_methods[] = { /* Device interface */ DEVMETHOD(device_probe, igb_probe), DEVMETHOD(device_attach, igb_attach), DEVMETHOD(device_detach, igb_detach), DEVMETHOD(device_shutdown, igb_shutdown), DEVMETHOD(device_suspend, igb_suspend), DEVMETHOD(device_resume, igb_resume), {0, 0} }; static driver_t igb_driver = { "igb", igb_methods, sizeof(struct adapter), }; static devclass_t igb_devclass; DRIVER_MODULE(igb, pci, igb_driver, igb_devclass, 0, 0); MODULE_DEPEND(igb, pci, 1, 1, 1); MODULE_DEPEND(igb, ether, 1, 1, 1); /********************************************************************* * Tunable default values. *********************************************************************/ /* Descriptor defaults */ static int igb_rxd = IGB_DEFAULT_RXD; static int igb_txd = IGB_DEFAULT_TXD; TUNABLE_INT("hw.igb.rxd", &igb_rxd); TUNABLE_INT("hw.igb.txd", &igb_txd); /* ** AIM: Adaptive Interrupt Moderation ** which means that the interrupt rate ** is varied over time based on the ** traffic for that interrupt vector */ static int igb_enable_aim = TRUE; TUNABLE_INT("hw.igb.enable_aim", &igb_enable_aim); /* * MSIX should be the default for best performance, * but this allows it to be forced off for testing. */ static int igb_enable_msix = 1; TUNABLE_INT("hw.igb.enable_msix", &igb_enable_msix); /* * Header split has seemed to be beneficial in * many circumstances tested, however there have * been some stability issues, so the default is * off. */ static bool igb_header_split = FALSE; TUNABLE_INT("hw.igb.hdr_split", &igb_header_split); /* ** This will autoconfigure based on ** the number of CPUs if left at 0. */ static int igb_num_queues = 0; TUNABLE_INT("hw.igb.num_queues", &igb_num_queues); /* How many packets rxeof tries to clean at a time */ static int igb_rx_process_limit = 100; TUNABLE_INT("hw.igb.rx_process_limit", &igb_rx_process_limit); /* Flow control setting - default to FULL */ static int igb_fc_setting = e1000_fc_full; TUNABLE_INT("hw.igb.fc_setting", &igb_fc_setting); /* ** Shadow VFTA table, this is needed because ** the real filter table gets cleared during ** a soft reset and the driver needs to be able ** to repopulate it. */ static u32 igb_shadow_vfta[IGB_VFTA_SIZE]; /********************************************************************* * Device identification routine * * igb_probe determines if the driver should be loaded on * adapter based on PCI vendor/device id of the adapter. * * return BUS_PROBE_DEFAULT on success, positive on failure *********************************************************************/ static int igb_probe(device_t dev) { char adapter_name[60]; uint16_t pci_vendor_id = 0; uint16_t pci_device_id = 0; uint16_t pci_subvendor_id = 0; uint16_t pci_subdevice_id = 0; igb_vendor_info_t *ent; INIT_DEBUGOUT("igb_probe: begin"); pci_vendor_id = pci_get_vendor(dev); if (pci_vendor_id != IGB_VENDOR_ID) return (ENXIO); pci_device_id = pci_get_device(dev); pci_subvendor_id = pci_get_subvendor(dev); pci_subdevice_id = pci_get_subdevice(dev); ent = igb_vendor_info_array; while (ent->vendor_id != 0) { if ((pci_vendor_id == ent->vendor_id) && (pci_device_id == ent->device_id) && ((pci_subvendor_id == ent->subvendor_id) || (ent->subvendor_id == PCI_ANY_ID)) && ((pci_subdevice_id == ent->subdevice_id) || (ent->subdevice_id == PCI_ANY_ID))) { sprintf(adapter_name, "%s %s", igb_strings[ent->index], igb_driver_version); device_set_desc_copy(dev, adapter_name); return (BUS_PROBE_DEFAULT); } ent++; } return (ENXIO); } /********************************************************************* * Device initialization routine * * The attach entry point is called when the driver is being loaded. * This routine identifies the type of hardware, allocates all resources * and initializes the hardware. * * return 0 on success, positive on failure *********************************************************************/ static int igb_attach(device_t dev) { struct adapter *adapter; int error = 0; u16 eeprom_data; INIT_DEBUGOUT("igb_attach: begin"); adapter = device_get_softc(dev); adapter->dev = adapter->osdep.dev = dev; IGB_CORE_LOCK_INIT(adapter, device_get_nameunit(dev)); /* SYSCTL stuff */ SYSCTL_ADD_PROC(device_get_sysctl_ctx(dev), SYSCTL_CHILDREN(device_get_sysctl_tree(dev)), OID_AUTO, "debug", CTLTYPE_INT|CTLFLAG_RW, adapter, 0, igb_sysctl_debug_info, "I", "Debug Information"); SYSCTL_ADD_PROC(device_get_sysctl_ctx(dev), SYSCTL_CHILDREN(device_get_sysctl_tree(dev)), OID_AUTO, "stats", CTLTYPE_INT|CTLFLAG_RW, adapter, 0, igb_sysctl_stats, "I", "Statistics"); SYSCTL_ADD_INT(device_get_sysctl_ctx(adapter->dev), SYSCTL_CHILDREN(device_get_sysctl_tree(adapter->dev)), OID_AUTO, "flow_control", CTLTYPE_INT|CTLFLAG_RW, &igb_fc_setting, 0, "Flow Control"); SYSCTL_ADD_INT(device_get_sysctl_ctx(dev), SYSCTL_CHILDREN(device_get_sysctl_tree(dev)), OID_AUTO, "enable_aim", CTLTYPE_INT|CTLFLAG_RW, &igb_enable_aim, 1, "Interrupt Moderation"); callout_init_mtx(&adapter->timer, &adapter->core_mtx, 0); /* Determine hardware and mac info */ igb_identify_hardware(adapter); /* Setup PCI resources */ if (igb_allocate_pci_resources(adapter)) { device_printf(dev, "Allocation of PCI resources failed\n"); error = ENXIO; goto err_pci; } /* Do Shared Code initialization */ if (e1000_setup_init_funcs(&adapter->hw, TRUE)) { device_printf(dev, "Setup of Shared code failed\n"); error = ENXIO; goto err_pci; } e1000_get_bus_info(&adapter->hw); /* Sysctls for limiting the amount of work done in the taskqueue */ igb_add_rx_process_limit(adapter, "rx_processing_limit", "max number of rx packets to process", &adapter->rx_process_limit, igb_rx_process_limit); /* * Validate number of transmit and receive descriptors. It * must not exceed hardware maximum, and must be multiple * of E1000_DBA_ALIGN. */ if (((igb_txd * sizeof(struct e1000_tx_desc)) % IGB_DBA_ALIGN) != 0 || (igb_txd > IGB_MAX_TXD) || (igb_txd < IGB_MIN_TXD)) { device_printf(dev, "Using %d TX descriptors instead of %d!\n", IGB_DEFAULT_TXD, igb_txd); adapter->num_tx_desc = IGB_DEFAULT_TXD; } else adapter->num_tx_desc = igb_txd; if (((igb_rxd * sizeof(struct e1000_rx_desc)) % IGB_DBA_ALIGN) != 0 || (igb_rxd > IGB_MAX_RXD) || (igb_rxd < IGB_MIN_RXD)) { device_printf(dev, "Using %d RX descriptors instead of %d!\n", IGB_DEFAULT_RXD, igb_rxd); adapter->num_rx_desc = IGB_DEFAULT_RXD; } else adapter->num_rx_desc = igb_rxd; adapter->hw.mac.autoneg = DO_AUTO_NEG; adapter->hw.phy.autoneg_wait_to_complete = FALSE; adapter->hw.phy.autoneg_advertised = AUTONEG_ADV_DEFAULT; /* Copper options */ if (adapter->hw.phy.media_type == e1000_media_type_copper) { adapter->hw.phy.mdix = AUTO_ALL_MODES; adapter->hw.phy.disable_polarity_correction = FALSE; adapter->hw.phy.ms_type = IGB_MASTER_SLAVE; } /* * Set the frame limits assuming * standard ethernet sized frames. */ adapter->max_frame_size = ETHERMTU + ETHER_HDR_LEN + ETHERNET_FCS_SIZE; adapter->min_frame_size = ETH_ZLEN + ETHERNET_FCS_SIZE; /* ** Allocate and Setup Queues */ if (igb_allocate_queues(adapter)) { error = ENOMEM; goto err_pci; } /* ** Start from a known state, this is ** important in reading the nvm and ** mac from that. */ e1000_reset_hw(&adapter->hw); /* Make sure we have a good EEPROM before we read from it */ if (e1000_validate_nvm_checksum(&adapter->hw) < 0) { /* ** Some PCI-E parts fail the first check due to ** the link being in sleep state, call it again, ** if it fails a second time its a real issue. */ if (e1000_validate_nvm_checksum(&adapter->hw) < 0) { device_printf(dev, "The EEPROM Checksum Is Not Valid\n"); error = EIO; goto err_late; } } /* ** Copy the permanent MAC address out of the EEPROM */ if (e1000_read_mac_addr(&adapter->hw) < 0) { device_printf(dev, "EEPROM read error while reading MAC" " address\n"); error = EIO; goto err_late; } /* Check its sanity */ if (!igb_is_valid_ether_addr(adapter->hw.mac.addr)) { device_printf(dev, "Invalid MAC address\n"); error = EIO; goto err_late; } /* ** Configure Interrupts */ if ((adapter->msix > 1) && (igb_enable_msix)) error = igb_allocate_msix(adapter); else /* MSI or Legacy */ error = igb_allocate_legacy(adapter); if (error) goto err_late; /* Setup OS specific network interface */ igb_setup_interface(dev, adapter); /* Now get a good starting state */ igb_reset(adapter); /* Initialize statistics */ igb_update_stats_counters(adapter); adapter->hw.mac.get_link_status = 1; igb_update_link_status(adapter); /* Indicate SOL/IDER usage */ if (e1000_check_reset_block(&adapter->hw)) device_printf(dev, "PHY reset is blocked due to SOL/IDER session.\n"); /* Determine if we have to control management hardware */ adapter->has_manage = e1000_enable_mng_pass_thru(&adapter->hw); /* * Setup Wake-on-Lan */ /* APME bit in EEPROM is mapped to WUC.APME */ eeprom_data = E1000_READ_REG(&adapter->hw, E1000_WUC) & E1000_WUC_APME; if (eeprom_data) adapter->wol = E1000_WUFC_MAG; /* Register for VLAN events */ adapter->vlan_attach = EVENTHANDLER_REGISTER(vlan_config, igb_register_vlan, adapter, EVENTHANDLER_PRI_FIRST); adapter->vlan_detach = EVENTHANDLER_REGISTER(vlan_unconfig, igb_unregister_vlan, adapter, EVENTHANDLER_PRI_FIRST); /* Tell the stack that the interface is not active */ adapter->ifp->if_drv_flags &= ~(IFF_DRV_RUNNING | IFF_DRV_OACTIVE); adapter->led_dev = led_create(igb_led_func, adapter, device_get_nameunit(dev)); INIT_DEBUGOUT("igb_attach: end"); return (0); err_late: igb_free_transmit_structures(adapter); igb_free_receive_structures(adapter); igb_release_hw_control(adapter); err_pci: igb_free_pci_resources(adapter); IGB_CORE_LOCK_DESTROY(adapter); return (error); } /********************************************************************* * Device removal routine * * The detach entry point is called when the driver is being removed. * This routine stops the adapter and deallocates all the resources * that were allocated for driver operation. * * return 0 on success, positive on failure *********************************************************************/ static int igb_detach(device_t dev) { struct adapter *adapter = device_get_softc(dev); struct ifnet *ifp = adapter->ifp; INIT_DEBUGOUT("igb_detach: begin"); /* Make sure VLANS are not using driver */ if (adapter->ifp->if_vlantrunk != NULL) { device_printf(dev,"Vlan in use, detach first\n"); return (EBUSY); } if (adapter->led_dev != NULL) led_destroy(adapter->led_dev); #ifdef DEVICE_POLLING if (ifp->if_capenable & IFCAP_POLLING) ether_poll_deregister(ifp); #endif IGB_CORE_LOCK(adapter); adapter->in_detach = 1; igb_stop(adapter); IGB_CORE_UNLOCK(adapter); e1000_phy_hw_reset(&adapter->hw); /* Give control back to firmware */ igb_release_manageability(adapter); igb_release_hw_control(adapter); if (adapter->wol) { E1000_WRITE_REG(&adapter->hw, E1000_WUC, E1000_WUC_PME_EN); E1000_WRITE_REG(&adapter->hw, E1000_WUFC, adapter->wol); igb_enable_wakeup(dev); } /* Unregister VLAN events */ if (adapter->vlan_attach != NULL) EVENTHANDLER_DEREGISTER(vlan_config, adapter->vlan_attach); if (adapter->vlan_detach != NULL) EVENTHANDLER_DEREGISTER(vlan_unconfig, adapter->vlan_detach); ether_ifdetach(adapter->ifp); callout_drain(&adapter->timer); igb_free_pci_resources(adapter); bus_generic_detach(dev); if_free(ifp); igb_free_transmit_structures(adapter); igb_free_receive_structures(adapter); IGB_CORE_LOCK_DESTROY(adapter); return (0); } /********************************************************************* * * Shutdown entry point * **********************************************************************/ static int igb_shutdown(device_t dev) { return igb_suspend(dev); } /* * Suspend/resume device methods. */ static int igb_suspend(device_t dev) { struct adapter *adapter = device_get_softc(dev); IGB_CORE_LOCK(adapter); igb_stop(adapter); igb_release_manageability(adapter); igb_release_hw_control(adapter); if (adapter->wol) { E1000_WRITE_REG(&adapter->hw, E1000_WUC, E1000_WUC_PME_EN); E1000_WRITE_REG(&adapter->hw, E1000_WUFC, adapter->wol); igb_enable_wakeup(dev); } IGB_CORE_UNLOCK(adapter); return bus_generic_suspend(dev); } static int igb_resume(device_t dev) { struct adapter *adapter = device_get_softc(dev); struct ifnet *ifp = adapter->ifp; IGB_CORE_LOCK(adapter); igb_init_locked(adapter); igb_init_manageability(adapter); if ((ifp->if_flags & IFF_UP) && (ifp->if_drv_flags & IFF_DRV_RUNNING)) igb_start(ifp); IGB_CORE_UNLOCK(adapter); return bus_generic_resume(dev); } /********************************************************************* * Transmit entry point * * igb_start is called by the stack to initiate a transmit. * The driver will remain in this routine as long as there are * packets to transmit and transmit resources are available. * In case resources are not available stack is notified and * the packet is requeued. **********************************************************************/ static void igb_start_locked(struct tx_ring *txr, struct ifnet *ifp) { struct adapter *adapter = ifp->if_softc; struct mbuf *m_head; IGB_TX_LOCK_ASSERT(txr); if ((ifp->if_drv_flags & (IFF_DRV_RUNNING|IFF_DRV_OACTIVE)) != IFF_DRV_RUNNING) return; if (!adapter->link_active) return; - while (!IFQ_DRV_IS_EMPTY(&ifp->if_snd)) { + /* Call cleanup if number of TX descriptors low */ + if (txr->tx_avail <= IGB_TX_CLEANUP_THRESHOLD) + igb_txeof(txr); + while (!IFQ_DRV_IS_EMPTY(&ifp->if_snd)) { + if (txr->tx_avail <= IGB_TX_OP_THRESHOLD) { + ifp->if_drv_flags |= IFF_DRV_OACTIVE; + break; + } IFQ_DRV_DEQUEUE(&ifp->if_snd, m_head); if (m_head == NULL) break; /* * Encapsulation can modify our pointer, and or make it * NULL on failure. In that event, we can't requeue. */ if (igb_xmit(txr, &m_head)) { if (m_head == NULL) break; ifp->if_drv_flags |= IFF_DRV_OACTIVE; IFQ_DRV_PREPEND(&ifp->if_snd, m_head); break; } /* Send a copy of the frame to the BPF listener */ ETHER_BPF_MTAP(ifp, m_head); /* Set watchdog on */ + txr->watchdog_time = ticks; txr->watchdog_check = TRUE; } } /* * Legacy TX driver routine, called from the * stack, always uses tx[0], and spins for it. * Should not be used with multiqueue tx */ static void igb_start(struct ifnet *ifp) { struct adapter *adapter = ifp->if_softc; struct tx_ring *txr = adapter->tx_rings; if (ifp->if_drv_flags & IFF_DRV_RUNNING) { IGB_TX_LOCK(txr); igb_start_locked(txr, ifp); IGB_TX_UNLOCK(txr); } return; } #if __FreeBSD_version >= 800000 /* ** Multiqueue Transmit driver ** */ static int igb_mq_start(struct ifnet *ifp, struct mbuf *m) { struct adapter *adapter = ifp->if_softc; struct tx_ring *txr; int i = 0, err = 0; /* Which queue to use */ if ((m->m_flags & M_FLOWID) != 0) i = m->m_pkthdr.flowid % adapter->num_queues; - else - i = curcpu % adapter->num_queues; txr = &adapter->tx_rings[i]; if (IGB_TX_TRYLOCK(txr)) { err = igb_mq_start_locked(ifp, txr, m); IGB_TX_UNLOCK(txr); } else err = drbr_enqueue(ifp, txr->br, m); return (err); } static int igb_mq_start_locked(struct ifnet *ifp, struct tx_ring *txr, struct mbuf *m) { struct adapter *adapter = txr->adapter; struct mbuf *next; int err = 0, enq; IGB_TX_LOCK_ASSERT(txr); if ((ifp->if_drv_flags & (IFF_DRV_RUNNING | IFF_DRV_OACTIVE)) != IFF_DRV_RUNNING || adapter->link_active == 0) { if (m != NULL) err = drbr_enqueue(ifp, txr->br, m); return (err); } + /* Call cleanup if number of TX descriptors low */ + if (txr->tx_avail <= IGB_TX_CLEANUP_THRESHOLD) + igb_txeof(txr); + enq = 0; if (m == NULL) { next = drbr_dequeue(ifp, txr->br); } else if (drbr_needs_enqueue(ifp, txr->br)) { if ((err = drbr_enqueue(ifp, txr->br, m)) != 0) return (err); next = drbr_dequeue(ifp, txr->br); } else next = m; + /* Process the queue */ while (next != NULL) { if ((err = igb_xmit(txr, &next)) != 0) { if (next != NULL) err = drbr_enqueue(ifp, txr->br, next); break; } enq++; drbr_stats_update(ifp, next->m_pkthdr.len, next->m_flags); ETHER_BPF_MTAP(ifp, next); if ((ifp->if_drv_flags & IFF_DRV_RUNNING) == 0) break; if (txr->tx_avail <= IGB_TX_OP_THRESHOLD) { ifp->if_drv_flags |= IFF_DRV_OACTIVE; break; } next = drbr_dequeue(ifp, txr->br); } if (enq > 0) { /* Set the watchdog */ txr->watchdog_check = TRUE; + txr->watchdog_time = ticks; } return (err); } /* ** Flush all ring buffers */ static void igb_qflush(struct ifnet *ifp) { struct adapter *adapter = ifp->if_softc; struct tx_ring *txr = adapter->tx_rings; struct mbuf *m; for (int i = 0; i < adapter->num_queues; i++, txr++) { IGB_TX_LOCK(txr); while ((m = buf_ring_dequeue_sc(txr->br)) != NULL) m_freem(m); IGB_TX_UNLOCK(txr); } if_qflush(ifp); } #endif /* __FreeBSD_version >= 800000 */ /********************************************************************* * Ioctl entry point * * igb_ioctl is called when the user wants to configure the * interface. * * return 0 on success, positive on failure **********************************************************************/ static int igb_ioctl(struct ifnet *ifp, u_long command, caddr_t data) { struct adapter *adapter = ifp->if_softc; struct ifreq *ifr = (struct ifreq *)data; #ifdef INET struct ifaddr *ifa = (struct ifaddr *)data; #endif int error = 0; if (adapter->in_detach) return (error); switch (command) { case SIOCSIFADDR: #ifdef INET if (ifa->ifa_addr->sa_family == AF_INET) { /* * XXX * Since resetting hardware takes a very long time * and results in link renegotiation we only * initialize the hardware only when it is absolutely * required. */ ifp->if_flags |= IFF_UP; if (!(ifp->if_drv_flags & IFF_DRV_RUNNING)) { IGB_CORE_LOCK(adapter); igb_init_locked(adapter); IGB_CORE_UNLOCK(adapter); } if (!(ifp->if_flags & IFF_NOARP)) arp_ifinit(ifp, ifa); } else #endif error = ether_ioctl(ifp, command, data); break; case SIOCSIFMTU: { int max_frame_size; IOCTL_DEBUGOUT("ioctl rcv'd: SIOCSIFMTU (Set Interface MTU)"); IGB_CORE_LOCK(adapter); max_frame_size = 9234; if (ifr->ifr_mtu > max_frame_size - ETHER_HDR_LEN - ETHER_CRC_LEN) { IGB_CORE_UNLOCK(adapter); error = EINVAL; break; } ifp->if_mtu = ifr->ifr_mtu; adapter->max_frame_size = ifp->if_mtu + ETHER_HDR_LEN + ETHER_CRC_LEN; igb_init_locked(adapter); IGB_CORE_UNLOCK(adapter); break; } case SIOCSIFFLAGS: IOCTL_DEBUGOUT("ioctl rcv'd:\ SIOCSIFFLAGS (Set Interface Flags)"); IGB_CORE_LOCK(adapter); if (ifp->if_flags & IFF_UP) { if ((ifp->if_drv_flags & IFF_DRV_RUNNING)) { if ((ifp->if_flags ^ adapter->if_flags) & (IFF_PROMISC | IFF_ALLMULTI)) { igb_disable_promisc(adapter); igb_set_promisc(adapter); } } else igb_init_locked(adapter); } else if (ifp->if_drv_flags & IFF_DRV_RUNNING) igb_stop(adapter); adapter->if_flags = ifp->if_flags; IGB_CORE_UNLOCK(adapter); break; case SIOCADDMULTI: case SIOCDELMULTI: IOCTL_DEBUGOUT("ioctl rcv'd: SIOC(ADD|DEL)MULTI"); if (ifp->if_drv_flags & IFF_DRV_RUNNING) { IGB_CORE_LOCK(adapter); igb_disable_intr(adapter); igb_set_multi(adapter); #ifdef DEVICE_POLLING if (!(ifp->if_capenable & IFCAP_POLLING)) #endif igb_enable_intr(adapter); IGB_CORE_UNLOCK(adapter); } break; case SIOCSIFMEDIA: /* Check SOL/IDER usage */ IGB_CORE_LOCK(adapter); if (e1000_check_reset_block(&adapter->hw)) { IGB_CORE_UNLOCK(adapter); device_printf(adapter->dev, "Media change is" " blocked due to SOL/IDER session.\n"); break; } IGB_CORE_UNLOCK(adapter); case SIOCGIFMEDIA: IOCTL_DEBUGOUT("ioctl rcv'd: \ SIOCxIFMEDIA (Get/Set Interface Media)"); error = ifmedia_ioctl(ifp, ifr, &adapter->media, command); break; case SIOCSIFCAP: { int mask, reinit; IOCTL_DEBUGOUT("ioctl rcv'd: SIOCSIFCAP (Set Capabilities)"); reinit = 0; mask = ifr->ifr_reqcap ^ ifp->if_capenable; #ifdef DEVICE_POLLING if (mask & IFCAP_POLLING) { if (ifr->ifr_reqcap & IFCAP_POLLING) { error = ether_poll_register(igb_poll, ifp); if (error) return (error); IGB_CORE_LOCK(adapter); igb_disable_intr(adapter); ifp->if_capenable |= IFCAP_POLLING; IGB_CORE_UNLOCK(adapter); } else { error = ether_poll_deregister(ifp); /* Enable interrupt even in error case */ IGB_CORE_LOCK(adapter); igb_enable_intr(adapter); ifp->if_capenable &= ~IFCAP_POLLING; IGB_CORE_UNLOCK(adapter); } } #endif if (mask & IFCAP_HWCSUM) { ifp->if_capenable ^= IFCAP_HWCSUM; reinit = 1; } if (mask & IFCAP_TSO4) { ifp->if_capenable ^= IFCAP_TSO4; reinit = 1; } if (mask & IFCAP_VLAN_HWTAGGING) { ifp->if_capenable ^= IFCAP_VLAN_HWTAGGING; reinit = 1; } if (mask & IFCAP_VLAN_HWFILTER) { ifp->if_capenable ^= IFCAP_VLAN_HWFILTER; reinit = 1; } if (mask & IFCAP_LRO) { ifp->if_capenable ^= IFCAP_LRO; reinit = 1; } if (reinit && (ifp->if_drv_flags & IFF_DRV_RUNNING)) igb_init(adapter); VLAN_CAPABILITIES(ifp); break; } default: error = ether_ioctl(ifp, command, data); break; } return (error); } /********************************************************************* * Init entry point * * This routine is used in two ways. It is used by the stack as * init entry point in network interface structure. It is also used * by the driver as a hw/sw initialization routine to get to a * consistent state. * * return 0 on success, positive on failure **********************************************************************/ static void igb_init_locked(struct adapter *adapter) { struct ifnet *ifp = adapter->ifp; device_t dev = adapter->dev; INIT_DEBUGOUT("igb_init: begin"); IGB_CORE_LOCK_ASSERT(adapter); igb_disable_intr(adapter); callout_stop(&adapter->timer); /* Get the latest mac address, User can use a LAA */ bcopy(IF_LLADDR(adapter->ifp), adapter->hw.mac.addr, ETHER_ADDR_LEN); /* Put the address into the Receive Address Array */ e1000_rar_set(&adapter->hw, adapter->hw.mac.addr, 0); igb_reset(adapter); igb_update_link_status(adapter); E1000_WRITE_REG(&adapter->hw, E1000_VET, ETHERTYPE_VLAN); /* Use real VLAN Filter support? */ if (ifp->if_capenable & IFCAP_VLAN_HWTAGGING) { if (ifp->if_capenable & IFCAP_VLAN_HWFILTER) /* Use real VLAN Filter support */ igb_setup_vlan_hw_support(adapter); else { u32 ctrl; ctrl = E1000_READ_REG(&adapter->hw, E1000_CTRL); ctrl |= E1000_CTRL_VME; E1000_WRITE_REG(&adapter->hw, E1000_CTRL, ctrl); } } /* Set hardware offload abilities */ ifp->if_hwassist = 0; if (ifp->if_capenable & IFCAP_TXCSUM) { ifp->if_hwassist |= (CSUM_TCP | CSUM_UDP); #if __FreeBSD_version >= 800000 if (adapter->hw.mac.type == e1000_82576) ifp->if_hwassist |= CSUM_SCTP; #endif } if (ifp->if_capenable & IFCAP_TSO4) ifp->if_hwassist |= CSUM_TSO; /* Configure for OS presence */ igb_init_manageability(adapter); /* Prepare transmit descriptors and buffers */ igb_setup_transmit_structures(adapter); igb_initialize_transmit_units(adapter); /* Setup Multicast table */ igb_set_multi(adapter); /* ** Figure out the desired mbuf pool ** for doing jumbo/packetsplit */ if (ifp->if_mtu > ETHERMTU) adapter->rx_mbuf_sz = MJUMPAGESIZE; else adapter->rx_mbuf_sz = MCLBYTES; /* Prepare receive descriptors and buffers */ if (igb_setup_receive_structures(adapter)) { device_printf(dev, "Could not setup receive structures\n"); return; } igb_initialize_receive_units(adapter); /* Don't lose promiscuous settings */ igb_set_promisc(adapter); ifp->if_drv_flags |= IFF_DRV_RUNNING; ifp->if_drv_flags &= ~IFF_DRV_OACTIVE; callout_reset(&adapter->timer, hz, igb_local_timer, adapter); e1000_clear_hw_cntrs_base_generic(&adapter->hw); if (adapter->msix > 1) /* Set up queue routing */ igb_configure_queues(adapter); /* Set up VLAN tag offload and filter */ igb_setup_vlan_hw_support(adapter); /* this clears any pending interrupts */ E1000_READ_REG(&adapter->hw, E1000_ICR); #ifdef DEVICE_POLLING /* * Only enable interrupts if we are not polling, make sure * they are off otherwise. */ if (ifp->if_capenable & IFCAP_POLLING) igb_disable_intr(adapter); else #endif /* DEVICE_POLLING */ { igb_enable_intr(adapter); E1000_WRITE_REG(&adapter->hw, E1000_ICS, E1000_ICS_LSC); } /* Don't reset the phy next time init gets called */ adapter->hw.phy.reset_disable = TRUE; } static void igb_init(void *arg) { struct adapter *adapter = arg; IGB_CORE_LOCK(adapter); igb_init_locked(adapter); IGB_CORE_UNLOCK(adapter); } static void igb_handle_rxtx(void *context, int pending) { struct igb_queue *que = context; struct adapter *adapter = que->adapter; struct tx_ring *txr = adapter->tx_rings; struct ifnet *ifp; ifp = adapter->ifp; if (ifp->if_drv_flags & IFF_DRV_RUNNING) { if (igb_rxeof(que, adapter->rx_process_limit)) taskqueue_enqueue(adapter->tq, &adapter->rxtx_task); IGB_TX_LOCK(txr); igb_txeof(txr); #if __FreeBSD_version >= 800000 if (!drbr_empty(ifp, txr->br)) igb_mq_start_locked(ifp, txr, NULL); #else if (!IFQ_DRV_IS_EMPTY(&ifp->if_snd)) igb_start_locked(txr, ifp); #endif IGB_TX_UNLOCK(txr); } igb_enable_intr(adapter); } static void igb_handle_que(void *context, int pending) { struct igb_queue *que = context; struct adapter *adapter = que->adapter; struct tx_ring *txr = que->txr; struct ifnet *ifp = adapter->ifp; - u32 loop = IGB_MAX_LOOP; bool more; - /* RX first */ - do { + if (ifp->if_drv_flags & IFF_DRV_RUNNING) { more = igb_rxeof(que, -1); - } while (loop-- && more); - if (IGB_TX_TRYLOCK(txr)) { - loop = IGB_MAX_LOOP; - do { - more = igb_txeof(txr); - } while (loop-- && more); + IGB_TX_LOCK(txr); + igb_txeof(txr); #if __FreeBSD_version >= 800000 igb_mq_start_locked(ifp, txr, NULL); #else if (!IFQ_DRV_IS_EMPTY(&ifp->if_snd)) igb_start_locked(txr, ifp); #endif IGB_TX_UNLOCK(txr); + if (more) { + taskqueue_enqueue(que->tq, &que->que_task); + return; + } } /* Reenable this interrupt */ #ifdef DEVICE_POLLING if (!(ifp->if_capenable & IFCAP_POLLING)) #endif E1000_WRITE_REG(&adapter->hw, E1000_EIMS, que->eims); } /* Deal with link in a sleepable context */ static void igb_handle_link(void *context, int pending) { struct adapter *adapter = context; adapter->hw.mac.get_link_status = 1; igb_update_link_status(adapter); } /********************************************************************* * * MSI/Legacy Deferred * Interrupt Service routine * *********************************************************************/ static int igb_irq_fast(void *arg) { struct adapter *adapter = arg; uint32_t reg_icr; reg_icr = E1000_READ_REG(&adapter->hw, E1000_ICR); /* Hot eject? */ if (reg_icr == 0xffffffff) return FILTER_STRAY; /* Definitely not our interrupt. */ if (reg_icr == 0x0) return FILTER_STRAY; if ((reg_icr & E1000_ICR_INT_ASSERTED) == 0) return FILTER_STRAY; /* * Mask interrupts until the taskqueue is finished running. This is * cheap, just assume that it is needed. This also works around the * MSI message reordering errata on certain systems. */ igb_disable_intr(adapter); taskqueue_enqueue(adapter->tq, &adapter->rxtx_task); /* Link status change */ if (reg_icr & (E1000_ICR_RXSEQ | E1000_ICR_LSC)) taskqueue_enqueue(adapter->tq, &adapter->link_task); if (reg_icr & E1000_ICR_RXO) adapter->rx_overruns++; return FILTER_HANDLED; } #ifdef DEVICE_POLLING /********************************************************************* * * Legacy polling routine : if using this code you MUST be sure that * multiqueue is not defined, ie, set igb_num_queues to 1. * *********************************************************************/ #if __FreeBSD_version >= 800000 #define POLL_RETURN_COUNT(a) (a) static int #else #define POLL_RETURN_COUNT(a) static void #endif igb_poll(struct ifnet *ifp, enum poll_cmd cmd, int count) { struct adapter *adapter = ifp->if_softc; struct igb_queue *que = adapter->queues; struct tx_ring *txr = adapter->tx_rings; u32 reg_icr, rx_done = 0; u32 loop = IGB_MAX_LOOP; bool more; IGB_CORE_LOCK(adapter); if ((ifp->if_drv_flags & IFF_DRV_RUNNING) == 0) { IGB_CORE_UNLOCK(adapter); return POLL_RETURN_COUNT(rx_done); } if (cmd == POLL_AND_CHECK_STATUS) { reg_icr = E1000_READ_REG(&adapter->hw, E1000_ICR); /* Link status change */ if (reg_icr & (E1000_ICR_RXSEQ | E1000_ICR_LSC)) taskqueue_enqueue(adapter->tq, &adapter->link_task); if (reg_icr & E1000_ICR_RXO) adapter->rx_overruns++; } IGB_CORE_UNLOCK(adapter); /* TODO: rx_count */ rx_done = igb_rxeof(que, count) ? 1 : 0; IGB_TX_LOCK(txr); do { more = igb_txeof(txr); } while (loop-- && more); #if __FreeBSD_version >= 800000 if (!drbr_empty(ifp, txr->br)) igb_mq_start_locked(ifp, txr, NULL); #else if (!IFQ_DRV_IS_EMPTY(&ifp->if_snd)) igb_start_locked(txr, ifp); #endif IGB_TX_UNLOCK(txr); return POLL_RETURN_COUNT(rx_done); } #endif /* DEVICE_POLLING */ /********************************************************************* * * MSIX TX Interrupt Service routine * **********************************************************************/ static void igb_msix_que(void *arg) { struct igb_queue *que = arg; struct adapter *adapter = que->adapter; struct tx_ring *txr = que->txr; struct rx_ring *rxr = que->rxr; u32 newitr = 0; bool more_tx, more_rx; E1000_WRITE_REG(&adapter->hw, E1000_EIMC, que->eims); ++que->irqs; IGB_TX_LOCK(txr); more_tx = igb_txeof(txr); IGB_TX_UNLOCK(txr); more_rx = igb_rxeof(que, adapter->rx_process_limit); if (igb_enable_aim == FALSE) goto no_calc; /* ** Do Adaptive Interrupt Moderation: ** - Write out last calculated setting ** - Calculate based on average size over ** the last interval. */ if (que->eitr_setting) E1000_WRITE_REG(&adapter->hw, E1000_EITR(que->msix), que->eitr_setting); que->eitr_setting = 0; /* Idle, do nothing */ if ((txr->bytes == 0) && (rxr->bytes == 0)) goto no_calc; /* Used half Default if sub-gig */ if (adapter->link_speed != 1000) newitr = IGB_DEFAULT_ITR / 2; else { if ((txr->bytes) && (txr->packets)) newitr = txr->bytes/txr->packets; if ((rxr->bytes) && (rxr->packets)) newitr = max(newitr, (rxr->bytes / rxr->packets)); newitr += 24; /* account for hardware frame, crc */ /* set an upper boundary */ newitr = min(newitr, 3000); /* Be nice to the mid range */ if ((newitr > 300) && (newitr < 1200)) newitr = (newitr / 3); else newitr = (newitr / 2); } newitr &= 0x7FFC; /* Mask invalid bits */ if (adapter->hw.mac.type == e1000_82575) newitr |= newitr << 16; else newitr |= E1000_EITR_CNT_IGNR; /* save for next interrupt */ que->eitr_setting = newitr; /* Reset state */ txr->bytes = 0; txr->packets = 0; rxr->bytes = 0; rxr->packets = 0; no_calc: /* Schedule a clean task if needed*/ if (more_tx || more_rx) taskqueue_enqueue(que->tq, &que->que_task); else /* Reenable this interrupt */ E1000_WRITE_REG(&adapter->hw, E1000_EIMS, que->eims); return; } /********************************************************************* * * MSIX Link Interrupt Service routine * **********************************************************************/ static void igb_msix_link(void *arg) { struct adapter *adapter = arg; u32 icr; ++adapter->link_irq; icr = E1000_READ_REG(&adapter->hw, E1000_ICR); if (!(icr & E1000_ICR_LSC)) goto spurious; taskqueue_enqueue(adapter->tq, &adapter->link_task); spurious: /* Rearm */ E1000_WRITE_REG(&adapter->hw, E1000_IMS, E1000_IMS_LSC); E1000_WRITE_REG(&adapter->hw, E1000_EIMS, adapter->link_mask); return; } /********************************************************************* * * Media Ioctl callback * * This routine is called whenever the user queries the status of * the interface using ifconfig. * **********************************************************************/ static void igb_media_status(struct ifnet *ifp, struct ifmediareq *ifmr) { struct adapter *adapter = ifp->if_softc; u_char fiber_type = IFM_1000_SX; INIT_DEBUGOUT("igb_media_status: begin"); IGB_CORE_LOCK(adapter); igb_update_link_status(adapter); ifmr->ifm_status = IFM_AVALID; ifmr->ifm_active = IFM_ETHER; if (!adapter->link_active) { IGB_CORE_UNLOCK(adapter); return; } ifmr->ifm_status |= IFM_ACTIVE; if ((adapter->hw.phy.media_type == e1000_media_type_fiber) || (adapter->hw.phy.media_type == e1000_media_type_internal_serdes)) ifmr->ifm_active |= fiber_type | IFM_FDX; else { switch (adapter->link_speed) { case 10: ifmr->ifm_active |= IFM_10_T; break; case 100: ifmr->ifm_active |= IFM_100_TX; break; case 1000: ifmr->ifm_active |= IFM_1000_T; break; } if (adapter->link_duplex == FULL_DUPLEX) ifmr->ifm_active |= IFM_FDX; else ifmr->ifm_active |= IFM_HDX; } IGB_CORE_UNLOCK(adapter); } /********************************************************************* * * Media Ioctl callback * * This routine is called when the user changes speed/duplex using * media/mediopt option with ifconfig. * **********************************************************************/ static int igb_media_change(struct ifnet *ifp) { struct adapter *adapter = ifp->if_softc; struct ifmedia *ifm = &adapter->media; INIT_DEBUGOUT("igb_media_change: begin"); if (IFM_TYPE(ifm->ifm_media) != IFM_ETHER) return (EINVAL); IGB_CORE_LOCK(adapter); switch (IFM_SUBTYPE(ifm->ifm_media)) { case IFM_AUTO: adapter->hw.mac.autoneg = DO_AUTO_NEG; adapter->hw.phy.autoneg_advertised = AUTONEG_ADV_DEFAULT; break; case IFM_1000_LX: case IFM_1000_SX: case IFM_1000_T: adapter->hw.mac.autoneg = DO_AUTO_NEG; adapter->hw.phy.autoneg_advertised = ADVERTISE_1000_FULL; break; case IFM_100_TX: adapter->hw.mac.autoneg = FALSE; adapter->hw.phy.autoneg_advertised = 0; if ((ifm->ifm_media & IFM_GMASK) == IFM_FDX) adapter->hw.mac.forced_speed_duplex = ADVERTISE_100_FULL; else adapter->hw.mac.forced_speed_duplex = ADVERTISE_100_HALF; break; case IFM_10_T: adapter->hw.mac.autoneg = FALSE; adapter->hw.phy.autoneg_advertised = 0; if ((ifm->ifm_media & IFM_GMASK) == IFM_FDX) adapter->hw.mac.forced_speed_duplex = ADVERTISE_10_FULL; else adapter->hw.mac.forced_speed_duplex = ADVERTISE_10_HALF; break; default: device_printf(adapter->dev, "Unsupported media type\n"); } /* As the speed/duplex settings my have changed we need to * reset the PHY. */ adapter->hw.phy.reset_disable = FALSE; igb_init_locked(adapter); IGB_CORE_UNLOCK(adapter); return (0); } /********************************************************************* * * This routine maps the mbufs to Advanced TX descriptors. * used by the 82575 adapter. * **********************************************************************/ static int igb_xmit(struct tx_ring *txr, struct mbuf **m_headp) { struct adapter *adapter = txr->adapter; bus_dma_segment_t segs[IGB_MAX_SCATTER]; bus_dmamap_t map; struct igb_tx_buffer *tx_buffer, *tx_buffer_mapped; union e1000_adv_tx_desc *txd = NULL; struct mbuf *m_head; u32 olinfo_status = 0, cmd_type_len = 0; int nsegs, i, j, error, first, last = 0; u32 hdrlen = 0; m_head = *m_headp; /* Set basic descriptor constants */ cmd_type_len |= E1000_ADVTXD_DTYP_DATA; cmd_type_len |= E1000_ADVTXD_DCMD_IFCS | E1000_ADVTXD_DCMD_DEXT; if (m_head->m_flags & M_VLANTAG) cmd_type_len |= E1000_ADVTXD_DCMD_VLE; /* * Force a cleanup if number of TX descriptors * available hits the threshold */ if (txr->tx_avail <= IGB_TX_CLEANUP_THRESHOLD) { igb_txeof(txr); /* Now do we at least have a minimal? */ if (txr->tx_avail <= IGB_TX_OP_THRESHOLD) { txr->no_desc_avail++; return (ENOBUFS); } } /* * Map the packet for DMA. * * Capture the first descriptor index, * this descriptor will have the index * of the EOP which is the only one that * now gets a DONE bit writeback. */ first = txr->next_avail_desc; tx_buffer = &txr->tx_buffers[first]; tx_buffer_mapped = tx_buffer; map = tx_buffer->map; error = bus_dmamap_load_mbuf_sg(txr->txtag, map, *m_headp, segs, &nsegs, BUS_DMA_NOWAIT); if (error == EFBIG) { struct mbuf *m; m = m_defrag(*m_headp, M_DONTWAIT); if (m == NULL) { adapter->mbuf_defrag_failed++; m_freem(*m_headp); *m_headp = NULL; return (ENOBUFS); } *m_headp = m; /* Try it again */ error = bus_dmamap_load_mbuf_sg(txr->txtag, map, *m_headp, segs, &nsegs, BUS_DMA_NOWAIT); if (error == ENOMEM) { adapter->no_tx_dma_setup++; return (error); } else if (error != 0) { adapter->no_tx_dma_setup++; m_freem(*m_headp); *m_headp = NULL; return (error); } } else if (error == ENOMEM) { adapter->no_tx_dma_setup++; return (error); } else if (error != 0) { adapter->no_tx_dma_setup++; m_freem(*m_headp); *m_headp = NULL; return (error); } /* Check again to be sure we have enough descriptors */ if (nsegs > (txr->tx_avail - 2)) { txr->no_desc_avail++; bus_dmamap_unload(txr->txtag, map); return (ENOBUFS); } m_head = *m_headp; /* * Set up the context descriptor: * used when any hardware offload is done. * This includes CSUM, VLAN, and TSO. It * will use the first descriptor. */ if (m_head->m_pkthdr.csum_flags & CSUM_TSO) { if (igb_tso_setup(txr, m_head, &hdrlen)) { cmd_type_len |= E1000_ADVTXD_DCMD_TSE; olinfo_status |= E1000_TXD_POPTS_IXSM << 8; olinfo_status |= E1000_TXD_POPTS_TXSM << 8; } else return (ENXIO); } else if (igb_tx_ctx_setup(txr, m_head)) olinfo_status |= E1000_TXD_POPTS_TXSM << 8; /* Calculate payload length */ olinfo_status |= ((m_head->m_pkthdr.len - hdrlen) << E1000_ADVTXD_PAYLEN_SHIFT); /* 82575 needs the queue index added */ if (adapter->hw.mac.type == e1000_82575) olinfo_status |= txr->me << 4; /* Set up our transmit descriptors */ i = txr->next_avail_desc; for (j = 0; j < nsegs; j++) { bus_size_t seg_len; bus_addr_t seg_addr; tx_buffer = &txr->tx_buffers[i]; txd = (union e1000_adv_tx_desc *)&txr->tx_base[i]; seg_addr = segs[j].ds_addr; seg_len = segs[j].ds_len; txd->read.buffer_addr = htole64(seg_addr); txd->read.cmd_type_len = htole32(cmd_type_len | seg_len); txd->read.olinfo_status = htole32(olinfo_status); last = i; if (++i == adapter->num_tx_desc) i = 0; tx_buffer->m_head = NULL; tx_buffer->next_eop = -1; } txr->next_avail_desc = i; txr->tx_avail -= nsegs; tx_buffer->m_head = m_head; tx_buffer_mapped->map = tx_buffer->map; tx_buffer->map = map; bus_dmamap_sync(txr->txtag, map, BUS_DMASYNC_PREWRITE); /* * Last Descriptor of Packet * needs End Of Packet (EOP) * and Report Status (RS) */ txd->read.cmd_type_len |= htole32(E1000_ADVTXD_DCMD_EOP | E1000_ADVTXD_DCMD_RS); /* * Keep track in the first buffer which * descriptor will be written back */ tx_buffer = &txr->tx_buffers[first]; tx_buffer->next_eop = last; txr->watchdog_time = ticks; /* * Advance the Transmit Descriptor Tail (TDT), this tells the E1000 * that this frame is available to transmit. */ bus_dmamap_sync(txr->txdma.dma_tag, txr->txdma.dma_map, BUS_DMASYNC_PREREAD | BUS_DMASYNC_PREWRITE); E1000_WRITE_REG(&adapter->hw, E1000_TDT(txr->me), i); ++txr->tx_packets; return (0); } static void igb_set_promisc(struct adapter *adapter) { struct ifnet *ifp = adapter->ifp; uint32_t reg_rctl; reg_rctl = E1000_READ_REG(&adapter->hw, E1000_RCTL); if (ifp->if_flags & IFF_PROMISC) { reg_rctl |= (E1000_RCTL_UPE | E1000_RCTL_MPE); E1000_WRITE_REG(&adapter->hw, E1000_RCTL, reg_rctl); } else if (ifp->if_flags & IFF_ALLMULTI) { reg_rctl |= E1000_RCTL_MPE; reg_rctl &= ~E1000_RCTL_UPE; E1000_WRITE_REG(&adapter->hw, E1000_RCTL, reg_rctl); } } static void igb_disable_promisc(struct adapter *adapter) { uint32_t reg_rctl; reg_rctl = E1000_READ_REG(&adapter->hw, E1000_RCTL); reg_rctl &= (~E1000_RCTL_UPE); reg_rctl &= (~E1000_RCTL_MPE); E1000_WRITE_REG(&adapter->hw, E1000_RCTL, reg_rctl); } /********************************************************************* * Multicast Update * * This routine is called whenever multicast address list is updated. * **********************************************************************/ static void igb_set_multi(struct adapter *adapter) { struct ifnet *ifp = adapter->ifp; struct ifmultiaddr *ifma; u32 reg_rctl = 0; u8 mta[MAX_NUM_MULTICAST_ADDRESSES * ETH_ADDR_LEN]; int mcnt = 0; IOCTL_DEBUGOUT("igb_set_multi: begin"); #if __FreeBSD_version < 800000 IF_ADDR_LOCK(ifp); #else if_maddr_rlock(ifp); #endif TAILQ_FOREACH(ifma, &ifp->if_multiaddrs, ifma_link) { if (ifma->ifma_addr->sa_family != AF_LINK) continue; if (mcnt == MAX_NUM_MULTICAST_ADDRESSES) break; bcopy(LLADDR((struct sockaddr_dl *)ifma->ifma_addr), &mta[mcnt * ETH_ADDR_LEN], ETH_ADDR_LEN); mcnt++; } #if __FreeBSD_version < 800000 IF_ADDR_UNLOCK(ifp); #else if_maddr_runlock(ifp); #endif if (mcnt >= MAX_NUM_MULTICAST_ADDRESSES) { reg_rctl = E1000_READ_REG(&adapter->hw, E1000_RCTL); reg_rctl |= E1000_RCTL_MPE; E1000_WRITE_REG(&adapter->hw, E1000_RCTL, reg_rctl); } else e1000_update_mc_addr_list(&adapter->hw, mta, mcnt); } /********************************************************************* * Timer routine: * This routine checks for link status, * updates statistics, and does the watchdog. * **********************************************************************/ static void igb_local_timer(void *arg) { struct adapter *adapter = arg; struct ifnet *ifp = adapter->ifp; device_t dev = adapter->dev; struct tx_ring *txr = adapter->tx_rings; IGB_CORE_LOCK_ASSERT(adapter); igb_update_link_status(adapter); igb_update_stats_counters(adapter); if (igb_display_debug_stats && ifp->if_drv_flags & IFF_DRV_RUNNING) igb_print_hw_stats(adapter); /* ** Watchdog: check for time since any descriptor was cleaned */ for (int i = 0; i < adapter->num_queues; i++, txr++) { if (txr->watchdog_check == FALSE) continue; if ((ticks - txr->watchdog_time) > IGB_WATCHDOG) goto timeout; } /* Trigger an RX interrupt on all queues */ #ifdef DEVICE_POLLING if (!(ifp->if_capenable & IFCAP_POLLING)) #endif E1000_WRITE_REG(&adapter->hw, E1000_EICS, adapter->rx_mask); callout_reset(&adapter->timer, hz, igb_local_timer, adapter); return; timeout: device_printf(adapter->dev, "Watchdog timeout -- resetting\n"); device_printf(dev,"Queue(%d) tdh = %d, hw tdt = %d\n", txr->me, E1000_READ_REG(&adapter->hw, E1000_TDH(txr->me)), E1000_READ_REG(&adapter->hw, E1000_TDT(txr->me))); device_printf(dev,"TX(%d) desc avail = %d," "Next TX to Clean = %d\n", txr->me, txr->tx_avail, txr->next_to_clean); adapter->ifp->if_drv_flags &= ~IFF_DRV_RUNNING; adapter->watchdog_events++; igb_init_locked(adapter); } static void igb_update_link_status(struct adapter *adapter) { struct e1000_hw *hw = &adapter->hw; struct ifnet *ifp = adapter->ifp; device_t dev = adapter->dev; struct tx_ring *txr = adapter->tx_rings; u32 link_check = 0; /* Get the cached link value or read for real */ switch (hw->phy.media_type) { case e1000_media_type_copper: if (hw->mac.get_link_status) { /* Do the work to read phy */ e1000_check_for_link(hw); link_check = !hw->mac.get_link_status; } else link_check = TRUE; break; case e1000_media_type_fiber: e1000_check_for_link(hw); link_check = (E1000_READ_REG(hw, E1000_STATUS) & E1000_STATUS_LU); break; case e1000_media_type_internal_serdes: e1000_check_for_link(hw); link_check = adapter->hw.mac.serdes_has_link; break; default: case e1000_media_type_unknown: break; } /* Now we check if a transition has happened */ if (link_check && (adapter->link_active == 0)) { e1000_get_speed_and_duplex(&adapter->hw, &adapter->link_speed, &adapter->link_duplex); if (bootverbose) device_printf(dev, "Link is up %d Mbps %s\n", adapter->link_speed, ((adapter->link_duplex == FULL_DUPLEX) ? "Full Duplex" : "Half Duplex")); adapter->link_active = 1; ifp->if_baudrate = adapter->link_speed * 1000000; /* This can sleep */ if_link_state_change(ifp, LINK_STATE_UP); } else if (!link_check && (adapter->link_active == 1)) { ifp->if_baudrate = adapter->link_speed = 0; adapter->link_duplex = 0; if (bootverbose) device_printf(dev, "Link is Down\n"); adapter->link_active = 0; /* This can sleep */ if_link_state_change(ifp, LINK_STATE_DOWN); /* Turn off watchdogs */ for (int i = 0; i < adapter->num_queues; i++, txr++) txr->watchdog_check = FALSE; } } /********************************************************************* * * This routine disables all traffic on the adapter by issuing a * global reset on the MAC and deallocates TX/RX buffers. * **********************************************************************/ static void igb_stop(void *arg) { struct adapter *adapter = arg; struct ifnet *ifp = adapter->ifp; struct tx_ring *txr = adapter->tx_rings; IGB_CORE_LOCK_ASSERT(adapter); INIT_DEBUGOUT("igb_stop: begin"); igb_disable_intr(adapter); callout_stop(&adapter->timer); /* Tell the stack that the interface is no longer active */ ifp->if_drv_flags &= ~(IFF_DRV_RUNNING | IFF_DRV_OACTIVE); /* Unarm watchdog timer. */ for (int i = 0; i < adapter->num_queues; i++, txr++) { IGB_TX_LOCK(txr); txr->watchdog_check = FALSE; IGB_TX_UNLOCK(txr); } e1000_reset_hw(&adapter->hw); E1000_WRITE_REG(&adapter->hw, E1000_WUC, 0); e1000_led_off(&adapter->hw); e1000_cleanup_led(&adapter->hw); } /********************************************************************* * * Determine hardware revision. * **********************************************************************/ static void igb_identify_hardware(struct adapter *adapter) { device_t dev = adapter->dev; /* Make sure our PCI config space has the necessary stuff set */ adapter->hw.bus.pci_cmd_word = pci_read_config(dev, PCIR_COMMAND, 2); if (!((adapter->hw.bus.pci_cmd_word & PCIM_CMD_BUSMASTEREN) && (adapter->hw.bus.pci_cmd_word & PCIM_CMD_MEMEN))) { device_printf(dev, "Memory Access and/or Bus Master bits " "were not set!\n"); adapter->hw.bus.pci_cmd_word |= (PCIM_CMD_BUSMASTEREN | PCIM_CMD_MEMEN); pci_write_config(dev, PCIR_COMMAND, adapter->hw.bus.pci_cmd_word, 2); } /* Save off the information about this board */ adapter->hw.vendor_id = pci_get_vendor(dev); adapter->hw.device_id = pci_get_device(dev); adapter->hw.revision_id = pci_read_config(dev, PCIR_REVID, 1); adapter->hw.subsystem_vendor_id = pci_read_config(dev, PCIR_SUBVEND_0, 2); adapter->hw.subsystem_device_id = pci_read_config(dev, PCIR_SUBDEV_0, 2); /* Do Shared Code Init and Setup */ if (e1000_set_mac_type(&adapter->hw)) { device_printf(dev, "Setup init failure\n"); return; } } static int igb_allocate_pci_resources(struct adapter *adapter) { device_t dev = adapter->dev; int rid; rid = PCIR_BAR(0); adapter->pci_mem = bus_alloc_resource_any(dev, SYS_RES_MEMORY, &rid, RF_ACTIVE); if (adapter->pci_mem == NULL) { device_printf(dev, "Unable to allocate bus resource: memory\n"); return (ENXIO); } adapter->osdep.mem_bus_space_tag = rman_get_bustag(adapter->pci_mem); adapter->osdep.mem_bus_space_handle = rman_get_bushandle(adapter->pci_mem); adapter->hw.hw_addr = (u8 *)&adapter->osdep.mem_bus_space_handle; adapter->num_queues = 1; /* Defaults for Legacy or MSI */ /* This will setup either MSI/X or MSI */ adapter->msix = igb_setup_msix(adapter); adapter->hw.back = &adapter->osdep; return (0); } /********************************************************************* * * Setup the Legacy or MSI Interrupt handler * **********************************************************************/ static int igb_allocate_legacy(struct adapter *adapter) { device_t dev = adapter->dev; struct igb_queue *que = adapter->queues; int error, rid = 0; /* Turn off all interrupts */ E1000_WRITE_REG(&adapter->hw, E1000_IMC, 0xffffffff); /* MSI RID is 1 */ if (adapter->msix == 1) rid = 1; /* We allocate a single interrupt resource */ adapter->res = bus_alloc_resource_any(dev, SYS_RES_IRQ, &rid, RF_SHAREABLE | RF_ACTIVE); if (adapter->res == NULL) { device_printf(dev, "Unable to allocate bus resource: " "interrupt\n"); return (ENXIO); } /* * Try allocating a fast interrupt and the associated deferred * processing contexts. */ TASK_INIT(&adapter->rxtx_task, 0, igb_handle_rxtx, que); /* Make tasklet for deferred link handling */ TASK_INIT(&adapter->link_task, 0, igb_handle_link, adapter); adapter->tq = taskqueue_create_fast("igb_taskq", M_NOWAIT, taskqueue_thread_enqueue, &adapter->tq); taskqueue_start_threads(&adapter->tq, 1, PI_NET, "%s taskq", device_get_nameunit(adapter->dev)); if ((error = bus_setup_intr(dev, adapter->res, INTR_TYPE_NET | INTR_MPSAFE, igb_irq_fast, NULL, adapter, &adapter->tag)) != 0) { device_printf(dev, "Failed to register fast interrupt " "handler: %d\n", error); taskqueue_free(adapter->tq); adapter->tq = NULL; return (error); } return (0); } /********************************************************************* * * Setup the MSIX Queue Interrupt handlers: * **********************************************************************/ static int igb_allocate_msix(struct adapter *adapter) { device_t dev = adapter->dev; struct igb_queue *que = adapter->queues; int error, rid, vector = 0; for (int i = 0; i < adapter->num_queues; i++, vector++, que++) { rid = vector +1; que->res = bus_alloc_resource_any(dev, SYS_RES_IRQ, &rid, RF_SHAREABLE | RF_ACTIVE); if (que->res == NULL) { device_printf(dev, "Unable to allocate bus resource: " "MSIX Queue Interrupt\n"); return (ENXIO); } error = bus_setup_intr(dev, que->res, INTR_TYPE_NET | INTR_MPSAFE, NULL, igb_msix_que, que, &que->tag); if (error) { que->res = NULL; device_printf(dev, "Failed to register Queue handler"); return (error); } que->msix = vector; if (adapter->hw.mac.type == e1000_82575) que->eims = E1000_EICR_TX_QUEUE0 << i; else que->eims = 1 << vector; /* ** Bind the msix vector, and thus the ** rings to the corresponding cpu. */ if (adapter->num_queues > 1) bus_bind_intr(dev, que->res, i); /* Make tasklet for deferred handling */ TASK_INIT(&que->que_task, 0, igb_handle_que, que); que->tq = taskqueue_create_fast("igb_que", M_NOWAIT, taskqueue_thread_enqueue, &que->tq); taskqueue_start_threads(&que->tq, 1, PI_NET, "%s que", device_get_nameunit(adapter->dev)); } /* And Link */ rid = vector + 1; adapter->res = bus_alloc_resource_any(dev, SYS_RES_IRQ, &rid, RF_SHAREABLE | RF_ACTIVE); if (adapter->res == NULL) { device_printf(dev, "Unable to allocate bus resource: " "MSIX Link Interrupt\n"); return (ENXIO); } if ((error = bus_setup_intr(dev, adapter->res, INTR_TYPE_NET | INTR_MPSAFE, NULL, igb_msix_link, adapter, &adapter->tag)) != 0) { device_printf(dev, "Failed to register Link handler"); return (error); } adapter->linkvec = vector; /* Make tasklet for deferred handling */ TASK_INIT(&adapter->link_task, 0, igb_handle_link, adapter); adapter->tq = taskqueue_create_fast("igb_link", M_NOWAIT, taskqueue_thread_enqueue, &adapter->tq); taskqueue_start_threads(&adapter->tq, 1, PI_NET, "%s link", device_get_nameunit(adapter->dev)); return (0); } static void igb_configure_queues(struct adapter *adapter) { struct e1000_hw *hw = &adapter->hw; struct igb_queue *que; u32 tmp, ivar = 0; u32 newitr = IGB_DEFAULT_ITR; /* First turn on RSS capability */ if (adapter->hw.mac.type > e1000_82575) E1000_WRITE_REG(hw, E1000_GPIE, E1000_GPIE_MSIX_MODE | E1000_GPIE_EIAME | E1000_GPIE_PBA | E1000_GPIE_NSICR); /* Turn on MSIX */ switch (adapter->hw.mac.type) { case e1000_82580: /* RX entries */ for (int i = 0; i < adapter->num_queues; i++) { u32 index = i >> 1; ivar = E1000_READ_REG_ARRAY(hw, E1000_IVAR0, index); que = &adapter->queues[i]; if (i & 1) { ivar &= 0xFF00FFFF; ivar |= (que->msix | E1000_IVAR_VALID) << 16; } else { ivar &= 0xFFFFFF00; ivar |= que->msix | E1000_IVAR_VALID; } E1000_WRITE_REG_ARRAY(hw, E1000_IVAR0, index, ivar); } /* TX entries */ for (int i = 0; i < adapter->num_queues; i++) { u32 index = i >> 1; ivar = E1000_READ_REG_ARRAY(hw, E1000_IVAR0, index); que = &adapter->queues[i]; if (i & 1) { ivar &= 0x00FFFFFF; ivar |= (que->msix | E1000_IVAR_VALID) << 24; } else { ivar &= 0xFFFF00FF; ivar |= (que->msix | E1000_IVAR_VALID) << 8; } E1000_WRITE_REG_ARRAY(hw, E1000_IVAR0, index, ivar); adapter->eims_mask |= que->eims; } /* And for the link interrupt */ ivar = (adapter->linkvec | E1000_IVAR_VALID) << 8; adapter->link_mask = 1 << adapter->linkvec; adapter->eims_mask |= adapter->link_mask; E1000_WRITE_REG(hw, E1000_IVAR_MISC, ivar); break; case e1000_82576: /* RX entries */ for (int i = 0; i < adapter->num_queues; i++) { u32 index = i & 0x7; /* Each IVAR has two entries */ ivar = E1000_READ_REG_ARRAY(hw, E1000_IVAR0, index); que = &adapter->queues[i]; if (i < 8) { ivar &= 0xFFFFFF00; ivar |= que->msix | E1000_IVAR_VALID; } else { ivar &= 0xFF00FFFF; ivar |= (que->msix | E1000_IVAR_VALID) << 16; } E1000_WRITE_REG_ARRAY(hw, E1000_IVAR0, index, ivar); adapter->eims_mask |= que->eims; } /* TX entries */ for (int i = 0; i < adapter->num_queues; i++) { u32 index = i & 0x7; /* Each IVAR has two entries */ ivar = E1000_READ_REG_ARRAY(hw, E1000_IVAR0, index); que = &adapter->queues[i]; if (i < 8) { ivar &= 0xFFFF00FF; ivar |= (que->msix | E1000_IVAR_VALID) << 8; } else { ivar &= 0x00FFFFFF; ivar |= (que->msix | E1000_IVAR_VALID) << 24; } E1000_WRITE_REG_ARRAY(hw, E1000_IVAR0, index, ivar); adapter->eims_mask |= que->eims; } /* And for the link interrupt */ ivar = (adapter->linkvec | E1000_IVAR_VALID) << 8; adapter->link_mask = 1 << adapter->linkvec; adapter->eims_mask |= adapter->link_mask; E1000_WRITE_REG(hw, E1000_IVAR_MISC, ivar); break; case e1000_82575: /* enable MSI-X support*/ tmp = E1000_READ_REG(hw, E1000_CTRL_EXT); tmp |= E1000_CTRL_EXT_PBA_CLR; /* Auto-Mask interrupts upon ICR read. */ tmp |= E1000_CTRL_EXT_EIAME; tmp |= E1000_CTRL_EXT_IRCA; E1000_WRITE_REG(hw, E1000_CTRL_EXT, tmp); /* Queues */ for (int i = 0; i < adapter->num_queues; i++) { que = &adapter->queues[i]; tmp = E1000_EICR_RX_QUEUE0 << i; tmp |= E1000_EICR_TX_QUEUE0 << i; que->eims = tmp; E1000_WRITE_REG_ARRAY(hw, E1000_MSIXBM(0), i, que->eims); adapter->eims_mask |= que->eims; } /* Link */ E1000_WRITE_REG(hw, E1000_MSIXBM(adapter->linkvec), E1000_EIMS_OTHER); adapter->link_mask |= E1000_EIMS_OTHER; adapter->eims_mask |= adapter->link_mask; default: break; } /* Set the starting interrupt rate */ if (hw->mac.type == e1000_82575) newitr |= newitr << 16; else newitr |= E1000_EITR_CNT_IGNR; for (int i = 0; i < adapter->num_queues; i++) { que = &adapter->queues[i]; E1000_WRITE_REG(hw, E1000_EITR(que->msix), newitr); } return; } static void igb_free_pci_resources(struct adapter *adapter) { struct igb_queue *que = adapter->queues; device_t dev = adapter->dev; int rid; /* ** There is a slight possibility of a failure mode ** in attach that will result in entering this function ** before interrupt resources have been initialized, and ** in that case we do not want to execute the loops below ** We can detect this reliably by the state of the adapter ** res pointer. */ if (adapter->res == NULL) goto mem; /* * First release all the interrupt resources: */ for (int i = 0; i < adapter->num_queues; i++, que++) { rid = que->msix + 1; if (que->tag != NULL) { bus_teardown_intr(dev, que->res, que->tag); que->tag = NULL; } if (que->res != NULL) bus_release_resource(dev, SYS_RES_IRQ, rid, que->res); } /* Clean the Legacy or Link interrupt last */ if (adapter->linkvec) /* we are doing MSIX */ rid = adapter->linkvec + 1; else (adapter->msix != 0) ? (rid = 1):(rid = 0); if (adapter->tag != NULL) { bus_teardown_intr(dev, adapter->res, adapter->tag); adapter->tag = NULL; } if (adapter->res != NULL) bus_release_resource(dev, SYS_RES_IRQ, rid, adapter->res); mem: if (adapter->msix) pci_release_msi(dev); if (adapter->msix_mem != NULL) bus_release_resource(dev, SYS_RES_MEMORY, PCIR_BAR(IGB_MSIX_BAR), adapter->msix_mem); if (adapter->pci_mem != NULL) bus_release_resource(dev, SYS_RES_MEMORY, PCIR_BAR(0), adapter->pci_mem); } /* * Setup Either MSI/X or MSI */ static int igb_setup_msix(struct adapter *adapter) { device_t dev = adapter->dev; int rid, want, queues, msgs; /* tuneable override */ if (igb_enable_msix == 0) goto msi; /* First try MSI/X */ rid = PCIR_BAR(IGB_MSIX_BAR); adapter->msix_mem = bus_alloc_resource_any(dev, SYS_RES_MEMORY, &rid, RF_ACTIVE); if (!adapter->msix_mem) { /* May not be enabled */ device_printf(adapter->dev, "Unable to map MSIX table \n"); goto msi; } msgs = pci_msix_count(dev); if (msgs == 0) { /* system has msix disabled */ bus_release_resource(dev, SYS_RES_MEMORY, PCIR_BAR(IGB_MSIX_BAR), adapter->msix_mem); adapter->msix_mem = NULL; goto msi; } /* Figure out a reasonable auto config value */ queues = (mp_ncpus > (msgs-1)) ? (msgs-1) : mp_ncpus; /* Manual override */ if (igb_num_queues != 0) queues = igb_num_queues; /* Can have max of 4 queues on 82575 */ if ((adapter->hw.mac.type == e1000_82575) && (queues > 4)) queues = 4; /* ** One vector (RX/TX pair) per queue ** plus an additional for Link interrupt */ want = queues + 1; if (msgs >= want) msgs = want; else { device_printf(adapter->dev, "MSIX Configuration Problem, " "%d vectors configured, but %d queues wanted!\n", msgs, want); return (ENXIO); } if ((msgs) && pci_alloc_msix(dev, &msgs) == 0) { device_printf(adapter->dev, "Using MSIX interrupts with %d vectors\n", msgs); adapter->num_queues = queues; return (msgs); } msi: msgs = pci_msi_count(dev); if (msgs == 1 && pci_alloc_msi(dev, &msgs) == 0) device_printf(adapter->dev,"Using MSI interrupt\n"); return (msgs); } /********************************************************************* * * Set up an fresh starting state * **********************************************************************/ static void igb_reset(struct adapter *adapter) { device_t dev = adapter->dev; struct e1000_hw *hw = &adapter->hw; struct e1000_fc_info *fc = &hw->fc; struct ifnet *ifp = adapter->ifp; u32 pba = 0; u16 hwm; INIT_DEBUGOUT("igb_reset: begin"); /* Let the firmware know the OS is in control */ igb_get_hw_control(adapter); /* * Packet Buffer Allocation (PBA) * Writing PBA sets the receive portion of the buffer * the remainder is used for the transmit buffer. */ switch (hw->mac.type) { case e1000_82575: pba = E1000_PBA_32K; break; case e1000_82576: pba = E1000_PBA_64K; break; case e1000_82580: pba = E1000_PBA_35K; default: break; } /* Special needs in case of Jumbo frames */ if ((hw->mac.type == e1000_82575) && (ifp->if_mtu > ETHERMTU)) { u32 tx_space, min_tx, min_rx; pba = E1000_READ_REG(hw, E1000_PBA); tx_space = pba >> 16; pba &= 0xffff; min_tx = (adapter->max_frame_size + sizeof(struct e1000_tx_desc) - ETHERNET_FCS_SIZE) * 2; min_tx = roundup2(min_tx, 1024); min_tx >>= 10; min_rx = adapter->max_frame_size; min_rx = roundup2(min_rx, 1024); min_rx >>= 10; if (tx_space < min_tx && ((min_tx - tx_space) < pba)) { pba = pba - (min_tx - tx_space); /* * if short on rx space, rx wins * and must trump tx adjustment */ if (pba < min_rx) pba = min_rx; } E1000_WRITE_REG(hw, E1000_PBA, pba); } INIT_DEBUGOUT1("igb_init: pba=%dK",pba); /* * These parameters control the automatic generation (Tx) and * response (Rx) to Ethernet PAUSE frames. * - High water mark should allow for at least two frames to be * received after sending an XOFF. * - Low water mark works best when it is very near the high water mark. * This allows the receiver to restart by sending XON when it has * drained a bit. */ hwm = min(((pba << 10) * 9 / 10), ((pba << 10) - 2 * adapter->max_frame_size)); if (hw->mac.type < e1000_82576) { fc->high_water = hwm & 0xFFF8; /* 8-byte granularity */ fc->low_water = fc->high_water - 8; } else { fc->high_water = hwm & 0xFFF0; /* 16-byte granularity */ fc->low_water = fc->high_water - 16; } fc->pause_time = IGB_FC_PAUSE_TIME; fc->send_xon = TRUE; /* Set Flow control, use the tunable location if sane */ if ((igb_fc_setting >= 0) || (igb_fc_setting < 4)) fc->requested_mode = igb_fc_setting; else fc->requested_mode = e1000_fc_none; fc->current_mode = fc->requested_mode; /* Issue a global reset */ e1000_reset_hw(hw); E1000_WRITE_REG(hw, E1000_WUC, 0); if (e1000_init_hw(hw) < 0) device_printf(dev, "Hardware Initialization Failed\n"); if (hw->mac.type == e1000_82580) { u32 reg; hwm = (pba << 10) - (2 * adapter->max_frame_size); /* * 0x80000000 - enable DMA COAL * 0x10000000 - use L0s as low power * 0x20000000 - use L1 as low power * X << 16 - exit dma coal when rx data exceeds X kB * Y - upper limit to stay in dma coal in units of 32usecs */ E1000_WRITE_REG(hw, E1000_DMACR, 0xA0000006 | ((hwm << 6) & 0x00FF0000)); /* set hwm to PBA - 2 * max frame size */ E1000_WRITE_REG(hw, E1000_FCRTC, hwm); /* * This sets the time to wait before requesting transition to * low power state to number of usecs needed to receive 1 512 * byte frame at gigabit line rate */ E1000_WRITE_REG(hw, E1000_DMCTLX, 4); /* free space in tx packet buffer to wake from DMA coal */ E1000_WRITE_REG(hw, E1000_DMCTXTH, (20480 - (2 * adapter->max_frame_size)) >> 6); /* make low power state decision controlled by DMA coal */ reg = E1000_READ_REG(hw, E1000_PCIEMISC); E1000_WRITE_REG(hw, E1000_PCIEMISC, reg | E1000_PCIEMISC_LX_DECISION); } E1000_WRITE_REG(&adapter->hw, E1000_VET, ETHERTYPE_VLAN); e1000_get_phy_info(hw); e1000_check_for_link(hw); return; } /********************************************************************* * * Setup networking device structure and register an interface. * **********************************************************************/ static void igb_setup_interface(device_t dev, struct adapter *adapter) { struct ifnet *ifp; INIT_DEBUGOUT("igb_setup_interface: begin"); ifp = adapter->ifp = if_alloc(IFT_ETHER); if (ifp == NULL) panic("%s: can not if_alloc()", device_get_nameunit(dev)); if_initname(ifp, device_get_name(dev), device_get_unit(dev)); ifp->if_mtu = ETHERMTU; ifp->if_init = igb_init; ifp->if_softc = adapter; ifp->if_flags = IFF_BROADCAST | IFF_SIMPLEX | IFF_MULTICAST; ifp->if_ioctl = igb_ioctl; ifp->if_start = igb_start; #if __FreeBSD_version >= 800000 ifp->if_transmit = igb_mq_start; ifp->if_qflush = igb_qflush; #endif IFQ_SET_MAXLEN(&ifp->if_snd, adapter->num_tx_desc - 1); ifp->if_snd.ifq_drv_maxlen = adapter->num_tx_desc - 1; IFQ_SET_READY(&ifp->if_snd); ether_ifattach(ifp, adapter->hw.mac.addr); ifp->if_capabilities = ifp->if_capenable = 0; ifp->if_capabilities = IFCAP_HWCSUM | IFCAP_VLAN_MTU; ifp->if_capabilities |= IFCAP_TSO4; ifp->if_capabilities |= IFCAP_JUMBO_MTU; if (igb_header_split) ifp->if_capabilities |= IFCAP_LRO; ifp->if_capenable = ifp->if_capabilities; #ifdef DEVICE_POLLING ifp->if_capabilities |= IFCAP_POLLING; #endif /* * Tell the upper layer(s) we * support full VLAN capability. */ ifp->if_data.ifi_hdrlen = sizeof(struct ether_vlan_header); ifp->if_capabilities |= IFCAP_VLAN_HWTAGGING | IFCAP_VLAN_MTU; ifp->if_capenable |= IFCAP_VLAN_HWTAGGING | IFCAP_VLAN_MTU; /* ** Dont turn this on by default, if vlans are ** created on another pseudo device (eg. lagg) ** then vlan events are not passed thru, breaking ** operation, but with HW FILTER off it works. If ** using vlans directly on the em driver you can ** enable this and get full hardware tag filtering. */ ifp->if_capabilities |= IFCAP_VLAN_HWFILTER; /* * Specify the media types supported by this adapter and register * callbacks to update media and link information */ ifmedia_init(&adapter->media, IFM_IMASK, igb_media_change, igb_media_status); if ((adapter->hw.phy.media_type == e1000_media_type_fiber) || (adapter->hw.phy.media_type == e1000_media_type_internal_serdes)) { ifmedia_add(&adapter->media, IFM_ETHER | IFM_1000_SX | IFM_FDX, 0, NULL); ifmedia_add(&adapter->media, IFM_ETHER | IFM_1000_SX, 0, NULL); } else { ifmedia_add(&adapter->media, IFM_ETHER | IFM_10_T, 0, NULL); ifmedia_add(&adapter->media, IFM_ETHER | IFM_10_T | IFM_FDX, 0, NULL); ifmedia_add(&adapter->media, IFM_ETHER | IFM_100_TX, 0, NULL); ifmedia_add(&adapter->media, IFM_ETHER | IFM_100_TX | IFM_FDX, 0, NULL); if (adapter->hw.phy.type != e1000_phy_ife) { ifmedia_add(&adapter->media, IFM_ETHER | IFM_1000_T | IFM_FDX, 0, NULL); ifmedia_add(&adapter->media, IFM_ETHER | IFM_1000_T, 0, NULL); } } ifmedia_add(&adapter->media, IFM_ETHER | IFM_AUTO, 0, NULL); ifmedia_set(&adapter->media, IFM_ETHER | IFM_AUTO); } /* * Manage DMA'able memory. */ static void igb_dmamap_cb(void *arg, bus_dma_segment_t *segs, int nseg, int error) { if (error) return; *(bus_addr_t *) arg = segs[0].ds_addr; } static int igb_dma_malloc(struct adapter *adapter, bus_size_t size, struct igb_dma_alloc *dma, int mapflags) { int error; error = bus_dma_tag_create(bus_get_dma_tag(adapter->dev), /* parent */ IGB_DBA_ALIGN, 0, /* alignment, bounds */ BUS_SPACE_MAXADDR, /* lowaddr */ BUS_SPACE_MAXADDR, /* highaddr */ NULL, NULL, /* filter, filterarg */ size, /* maxsize */ 1, /* nsegments */ size, /* maxsegsize */ 0, /* flags */ NULL, /* lockfunc */ NULL, /* lockarg */ &dma->dma_tag); if (error) { device_printf(adapter->dev, "%s: bus_dma_tag_create failed: %d\n", __func__, error); goto fail_0; } error = bus_dmamem_alloc(dma->dma_tag, (void**) &dma->dma_vaddr, BUS_DMA_NOWAIT, &dma->dma_map); if (error) { device_printf(adapter->dev, "%s: bus_dmamem_alloc(%ju) failed: %d\n", __func__, (uintmax_t)size, error); goto fail_2; } dma->dma_paddr = 0; error = bus_dmamap_load(dma->dma_tag, dma->dma_map, dma->dma_vaddr, size, igb_dmamap_cb, &dma->dma_paddr, mapflags | BUS_DMA_NOWAIT); if (error || dma->dma_paddr == 0) { device_printf(adapter->dev, "%s: bus_dmamap_load failed: %d\n", __func__, error); goto fail_3; } return (0); fail_3: bus_dmamap_unload(dma->dma_tag, dma->dma_map); fail_2: bus_dmamem_free(dma->dma_tag, dma->dma_vaddr, dma->dma_map); bus_dma_tag_destroy(dma->dma_tag); fail_0: dma->dma_map = NULL; dma->dma_tag = NULL; return (error); } static void igb_dma_free(struct adapter *adapter, struct igb_dma_alloc *dma) { if (dma->dma_tag == NULL) return; if (dma->dma_map != NULL) { bus_dmamap_sync(dma->dma_tag, dma->dma_map, BUS_DMASYNC_POSTREAD | BUS_DMASYNC_POSTWRITE); bus_dmamap_unload(dma->dma_tag, dma->dma_map); bus_dmamem_free(dma->dma_tag, dma->dma_vaddr, dma->dma_map); dma->dma_map = NULL; } bus_dma_tag_destroy(dma->dma_tag); dma->dma_tag = NULL; } /********************************************************************* * * Allocate memory for the transmit and receive rings, and then * the descriptors associated with each, called only once at attach. * **********************************************************************/ static int igb_allocate_queues(struct adapter *adapter) { device_t dev = adapter->dev; struct igb_queue *que = NULL; struct tx_ring *txr = NULL; struct rx_ring *rxr = NULL; int rsize, tsize, error = E1000_SUCCESS; int txconf = 0, rxconf = 0; /* First allocate the top level queue structs */ if (!(adapter->queues = (struct igb_queue *) malloc(sizeof(struct igb_queue) * adapter->num_queues, M_DEVBUF, M_NOWAIT | M_ZERO))) { device_printf(dev, "Unable to allocate queue memory\n"); error = ENOMEM; goto fail; } /* Next allocate the TX ring struct memory */ if (!(adapter->tx_rings = (struct tx_ring *) malloc(sizeof(struct tx_ring) * adapter->num_queues, M_DEVBUF, M_NOWAIT | M_ZERO))) { device_printf(dev, "Unable to allocate TX ring memory\n"); error = ENOMEM; goto tx_fail; } /* Now allocate the RX */ if (!(adapter->rx_rings = (struct rx_ring *) malloc(sizeof(struct rx_ring) * adapter->num_queues, M_DEVBUF, M_NOWAIT | M_ZERO))) { device_printf(dev, "Unable to allocate RX ring memory\n"); error = ENOMEM; goto rx_fail; } tsize = roundup2(adapter->num_tx_desc * sizeof(union e1000_adv_tx_desc), IGB_DBA_ALIGN); /* * Now set up the TX queues, txconf is needed to handle the * possibility that things fail midcourse and we need to * undo memory gracefully */ for (int i = 0; i < adapter->num_queues; i++, txconf++) { /* Set up some basics */ txr = &adapter->tx_rings[i]; txr->adapter = adapter; txr->me = i; /* Initialize the TX lock */ snprintf(txr->mtx_name, sizeof(txr->mtx_name), "%s:tx(%d)", device_get_nameunit(dev), txr->me); mtx_init(&txr->tx_mtx, txr->mtx_name, NULL, MTX_DEF); if (igb_dma_malloc(adapter, tsize, &txr->txdma, BUS_DMA_NOWAIT)) { device_printf(dev, "Unable to allocate TX Descriptor memory\n"); error = ENOMEM; goto err_tx_desc; } txr->tx_base = (struct e1000_tx_desc *)txr->txdma.dma_vaddr; bzero((void *)txr->tx_base, tsize); /* Now allocate transmit buffers for the ring */ if (igb_allocate_transmit_buffers(txr)) { device_printf(dev, "Critical Failure setting up transmit buffers\n"); error = ENOMEM; goto err_tx_desc; } #if __FreeBSD_version >= 800000 /* Allocate a buf ring */ txr->br = buf_ring_alloc(IGB_BR_SIZE, M_DEVBUF, M_WAITOK, &txr->tx_mtx); #endif } /* * Next the RX queues... */ rsize = roundup2(adapter->num_rx_desc * sizeof(union e1000_adv_rx_desc), IGB_DBA_ALIGN); for (int i = 0; i < adapter->num_queues; i++, rxconf++) { rxr = &adapter->rx_rings[i]; rxr->adapter = adapter; rxr->me = i; /* Initialize the RX lock */ snprintf(rxr->mtx_name, sizeof(rxr->mtx_name), "%s:rx(%d)", device_get_nameunit(dev), txr->me); mtx_init(&rxr->rx_mtx, rxr->mtx_name, NULL, MTX_DEF); if (igb_dma_malloc(adapter, rsize, &rxr->rxdma, BUS_DMA_NOWAIT)) { device_printf(dev, "Unable to allocate RxDescriptor memory\n"); error = ENOMEM; goto err_rx_desc; } rxr->rx_base = (union e1000_adv_rx_desc *)rxr->rxdma.dma_vaddr; bzero((void *)rxr->rx_base, rsize); /* Allocate receive buffers for the ring*/ if (igb_allocate_receive_buffers(rxr)) { device_printf(dev, "Critical Failure setting up receive buffers\n"); error = ENOMEM; goto err_rx_desc; } } /* ** Finally set up the queue holding structs */ for (int i = 0; i < adapter->num_queues; i++) { que = &adapter->queues[i]; que->adapter = adapter; que->txr = &adapter->tx_rings[i]; que->rxr = &adapter->rx_rings[i]; } return (0); err_rx_desc: for (rxr = adapter->rx_rings; rxconf > 0; rxr++, rxconf--) igb_dma_free(adapter, &rxr->rxdma); err_tx_desc: for (txr = adapter->tx_rings; txconf > 0; txr++, txconf--) igb_dma_free(adapter, &txr->txdma); free(adapter->rx_rings, M_DEVBUF); rx_fail: buf_ring_free(txr->br, M_DEVBUF); free(adapter->tx_rings, M_DEVBUF); tx_fail: free(adapter->queues, M_DEVBUF); fail: return (error); } /********************************************************************* * * Allocate memory for tx_buffer structures. The tx_buffer stores all * the information needed to transmit a packet on the wire. This is * called only once at attach, setup is done every reset. * **********************************************************************/ static int igb_allocate_transmit_buffers(struct tx_ring *txr) { struct adapter *adapter = txr->adapter; device_t dev = adapter->dev; struct igb_tx_buffer *txbuf; int error, i; /* * Setup DMA descriptor areas. */ if ((error = bus_dma_tag_create(bus_get_dma_tag(dev), 1, 0, /* alignment, bounds */ BUS_SPACE_MAXADDR, /* lowaddr */ BUS_SPACE_MAXADDR, /* highaddr */ NULL, NULL, /* filter, filterarg */ IGB_TSO_SIZE, /* maxsize */ IGB_MAX_SCATTER, /* nsegments */ PAGE_SIZE, /* maxsegsize */ 0, /* flags */ NULL, /* lockfunc */ NULL, /* lockfuncarg */ &txr->txtag))) { device_printf(dev,"Unable to allocate TX DMA tag\n"); goto fail; } if (!(txr->tx_buffers = (struct igb_tx_buffer *) malloc(sizeof(struct igb_tx_buffer) * adapter->num_tx_desc, M_DEVBUF, M_NOWAIT | M_ZERO))) { device_printf(dev, "Unable to allocate tx_buffer memory\n"); error = ENOMEM; goto fail; } /* Create the descriptor buffer dma maps */ txbuf = txr->tx_buffers; for (i = 0; i < adapter->num_tx_desc; i++, txbuf++) { error = bus_dmamap_create(txr->txtag, 0, &txbuf->map); if (error != 0) { device_printf(dev, "Unable to create TX DMA map\n"); goto fail; } } return 0; fail: /* We free all, it handles case where we are in the middle */ igb_free_transmit_structures(adapter); return (error); } /********************************************************************* * * Initialize a transmit ring. * **********************************************************************/ static void igb_setup_transmit_ring(struct tx_ring *txr) { struct adapter *adapter = txr->adapter; struct igb_tx_buffer *txbuf; int i; /* Clear the old descriptor contents */ IGB_TX_LOCK(txr); bzero((void *)txr->tx_base, (sizeof(union e1000_adv_tx_desc)) * adapter->num_tx_desc); /* Reset indices */ txr->next_avail_desc = 0; txr->next_to_clean = 0; /* Free any existing tx buffers. */ txbuf = txr->tx_buffers; for (i = 0; i < adapter->num_tx_desc; i++, txbuf++) { if (txbuf->m_head != NULL) { bus_dmamap_sync(txr->txtag, txbuf->map, BUS_DMASYNC_POSTWRITE); bus_dmamap_unload(txr->txtag, txbuf->map); m_freem(txbuf->m_head); txbuf->m_head = NULL; } /* clear the watch index */ txbuf->next_eop = -1; } /* Set number of descriptors available */ txr->tx_avail = adapter->num_tx_desc; bus_dmamap_sync(txr->txdma.dma_tag, txr->txdma.dma_map, BUS_DMASYNC_PREREAD | BUS_DMASYNC_PREWRITE); IGB_TX_UNLOCK(txr); } /********************************************************************* * * Initialize all transmit rings. * **********************************************************************/ static void igb_setup_transmit_structures(struct adapter *adapter) { struct tx_ring *txr = adapter->tx_rings; for (int i = 0; i < adapter->num_queues; i++, txr++) igb_setup_transmit_ring(txr); return; } /********************************************************************* * * Enable transmit unit. * **********************************************************************/ static void igb_initialize_transmit_units(struct adapter *adapter) { struct tx_ring *txr = adapter->tx_rings; struct e1000_hw *hw = &adapter->hw; u32 tctl, txdctl; INIT_DEBUGOUT("igb_initialize_transmit_units: begin"); /* Setup the Tx Descriptor Rings */ for (int i = 0; i < adapter->num_queues; i++, txr++) { u64 bus_addr = txr->txdma.dma_paddr; E1000_WRITE_REG(hw, E1000_TDLEN(i), adapter->num_tx_desc * sizeof(struct e1000_tx_desc)); E1000_WRITE_REG(hw, E1000_TDBAH(i), (uint32_t)(bus_addr >> 32)); E1000_WRITE_REG(hw, E1000_TDBAL(i), (uint32_t)bus_addr); /* Setup the HW Tx Head and Tail descriptor pointers */ E1000_WRITE_REG(hw, E1000_TDT(i), 0); E1000_WRITE_REG(hw, E1000_TDH(i), 0); HW_DEBUGOUT2("Base = %x, Length = %x\n", E1000_READ_REG(hw, E1000_TDBAL(i)), E1000_READ_REG(hw, E1000_TDLEN(i))); txr->watchdog_check = FALSE; txdctl = E1000_READ_REG(hw, E1000_TXDCTL(i)); txdctl |= IGB_TX_PTHRESH; txdctl |= IGB_TX_HTHRESH << 8; txdctl |= IGB_TX_WTHRESH << 16; txdctl |= E1000_TXDCTL_QUEUE_ENABLE; E1000_WRITE_REG(hw, E1000_TXDCTL(i), txdctl); } /* Program the Transmit Control Register */ tctl = E1000_READ_REG(hw, E1000_TCTL); tctl &= ~E1000_TCTL_CT; tctl |= (E1000_TCTL_PSP | E1000_TCTL_RTLC | E1000_TCTL_EN | (E1000_COLLISION_THRESHOLD << E1000_CT_SHIFT)); e1000_config_collision_dist(hw); /* This write will effectively turn on the transmit unit. */ E1000_WRITE_REG(hw, E1000_TCTL, tctl); } /********************************************************************* * * Free all transmit rings. * **********************************************************************/ static void igb_free_transmit_structures(struct adapter *adapter) { struct tx_ring *txr = adapter->tx_rings; for (int i = 0; i < adapter->num_queues; i++, txr++) { IGB_TX_LOCK(txr); igb_free_transmit_buffers(txr); igb_dma_free(adapter, &txr->txdma); IGB_TX_UNLOCK(txr); IGB_TX_LOCK_DESTROY(txr); } free(adapter->tx_rings, M_DEVBUF); } /********************************************************************* * * Free transmit ring related data structures. * **********************************************************************/ static void igb_free_transmit_buffers(struct tx_ring *txr) { struct adapter *adapter = txr->adapter; struct igb_tx_buffer *tx_buffer; int i; INIT_DEBUGOUT("free_transmit_ring: begin"); if (txr->tx_buffers == NULL) return; tx_buffer = txr->tx_buffers; for (i = 0; i < adapter->num_tx_desc; i++, tx_buffer++) { if (tx_buffer->m_head != NULL) { bus_dmamap_sync(txr->txtag, tx_buffer->map, BUS_DMASYNC_POSTWRITE); bus_dmamap_unload(txr->txtag, tx_buffer->map); m_freem(tx_buffer->m_head); tx_buffer->m_head = NULL; if (tx_buffer->map != NULL) { bus_dmamap_destroy(txr->txtag, tx_buffer->map); tx_buffer->map = NULL; } } else if (tx_buffer->map != NULL) { bus_dmamap_unload(txr->txtag, tx_buffer->map); bus_dmamap_destroy(txr->txtag, tx_buffer->map); tx_buffer->map = NULL; } } #if __FreeBSD_version >= 800000 if (txr->br != NULL) buf_ring_free(txr->br, M_DEVBUF); #endif if (txr->tx_buffers != NULL) { free(txr->tx_buffers, M_DEVBUF); txr->tx_buffers = NULL; } if (txr->txtag != NULL) { bus_dma_tag_destroy(txr->txtag); txr->txtag = NULL; } return; } /********************************************************************** * * Setup work for hardware segmentation offload (TSO) * **********************************************************************/ static boolean_t igb_tso_setup(struct tx_ring *txr, struct mbuf *mp, u32 *hdrlen) { struct adapter *adapter = txr->adapter; struct e1000_adv_tx_context_desc *TXD; struct igb_tx_buffer *tx_buffer; u32 vlan_macip_lens = 0, type_tucmd_mlhl = 0; u32 mss_l4len_idx = 0; u16 vtag = 0; int ctxd, ehdrlen, ip_hlen, tcp_hlen; struct ether_vlan_header *eh; struct ip *ip; struct tcphdr *th; /* * Determine where frame payload starts. * Jump over vlan headers if already present */ eh = mtod(mp, struct ether_vlan_header *); if (eh->evl_encap_proto == htons(ETHERTYPE_VLAN)) ehdrlen = ETHER_HDR_LEN + ETHER_VLAN_ENCAP_LEN; else ehdrlen = ETHER_HDR_LEN; /* Ensure we have at least the IP+TCP header in the first mbuf. */ if (mp->m_len < ehdrlen + sizeof(struct ip) + sizeof(struct tcphdr)) return FALSE; /* Only supports IPV4 for now */ ctxd = txr->next_avail_desc; tx_buffer = &txr->tx_buffers[ctxd]; TXD = (struct e1000_adv_tx_context_desc *) &txr->tx_base[ctxd]; ip = (struct ip *)(mp->m_data + ehdrlen); if (ip->ip_p != IPPROTO_TCP) return FALSE; /* 0 */ ip->ip_sum = 0; ip_hlen = ip->ip_hl << 2; th = (struct tcphdr *)((caddr_t)ip + ip_hlen); th->th_sum = in_pseudo(ip->ip_src.s_addr, ip->ip_dst.s_addr, htons(IPPROTO_TCP)); tcp_hlen = th->th_off << 2; /* * Calculate header length, this is used * in the transmit desc in igb_xmit */ *hdrlen = ehdrlen + ip_hlen + tcp_hlen; /* VLAN MACLEN IPLEN */ if (mp->m_flags & M_VLANTAG) { vtag = htole16(mp->m_pkthdr.ether_vtag); vlan_macip_lens |= (vtag << E1000_ADVTXD_VLAN_SHIFT); } vlan_macip_lens |= (ehdrlen << E1000_ADVTXD_MACLEN_SHIFT); vlan_macip_lens |= ip_hlen; TXD->vlan_macip_lens |= htole32(vlan_macip_lens); /* ADV DTYPE TUCMD */ type_tucmd_mlhl |= E1000_ADVTXD_DCMD_DEXT | E1000_ADVTXD_DTYP_CTXT; type_tucmd_mlhl |= E1000_ADVTXD_TUCMD_L4T_TCP; type_tucmd_mlhl |= E1000_ADVTXD_TUCMD_IPV4; TXD->type_tucmd_mlhl |= htole32(type_tucmd_mlhl); /* MSS L4LEN IDX */ mss_l4len_idx |= (mp->m_pkthdr.tso_segsz << E1000_ADVTXD_MSS_SHIFT); mss_l4len_idx |= (tcp_hlen << E1000_ADVTXD_L4LEN_SHIFT); /* 82575 needs the queue index added */ if (adapter->hw.mac.type == e1000_82575) mss_l4len_idx |= txr->me << 4; TXD->mss_l4len_idx = htole32(mss_l4len_idx); TXD->seqnum_seed = htole32(0); tx_buffer->m_head = NULL; tx_buffer->next_eop = -1; if (++ctxd == adapter->num_tx_desc) ctxd = 0; txr->tx_avail--; txr->next_avail_desc = ctxd; return TRUE; } /********************************************************************* * * Context Descriptor setup for VLAN or CSUM * **********************************************************************/ static bool igb_tx_ctx_setup(struct tx_ring *txr, struct mbuf *mp) { struct adapter *adapter = txr->adapter; struct e1000_adv_tx_context_desc *TXD; struct igb_tx_buffer *tx_buffer; u32 vlan_macip_lens, type_tucmd_mlhl, mss_l4len_idx; struct ether_vlan_header *eh; struct ip *ip = NULL; struct ip6_hdr *ip6; int ehdrlen, ctxd, ip_hlen = 0; u16 etype, vtag = 0; u8 ipproto = 0; bool offload = TRUE; if ((mp->m_pkthdr.csum_flags & CSUM_OFFLOAD) == 0) offload = FALSE; vlan_macip_lens = type_tucmd_mlhl = mss_l4len_idx = 0; ctxd = txr->next_avail_desc; tx_buffer = &txr->tx_buffers[ctxd]; TXD = (struct e1000_adv_tx_context_desc *) &txr->tx_base[ctxd]; /* ** In advanced descriptors the vlan tag must ** be placed into the context descriptor, thus ** we need to be here just for that setup. */ if (mp->m_flags & M_VLANTAG) { vtag = htole16(mp->m_pkthdr.ether_vtag); vlan_macip_lens |= (vtag << E1000_ADVTXD_VLAN_SHIFT); } else if (offload == FALSE) return FALSE; /* * Determine where frame payload starts. * Jump over vlan headers if already present, * helpful for QinQ too. */ eh = mtod(mp, struct ether_vlan_header *); if (eh->evl_encap_proto == htons(ETHERTYPE_VLAN)) { etype = ntohs(eh->evl_proto); ehdrlen = ETHER_HDR_LEN + ETHER_VLAN_ENCAP_LEN; } else { etype = ntohs(eh->evl_encap_proto); ehdrlen = ETHER_HDR_LEN; } /* Set the ether header length */ vlan_macip_lens |= ehdrlen << E1000_ADVTXD_MACLEN_SHIFT; switch (etype) { case ETHERTYPE_IP: ip = (struct ip *)(mp->m_data + ehdrlen); ip_hlen = ip->ip_hl << 2; if (mp->m_len < ehdrlen + ip_hlen) { offload = FALSE; break; } ipproto = ip->ip_p; type_tucmd_mlhl |= E1000_ADVTXD_TUCMD_IPV4; break; case ETHERTYPE_IPV6: ip6 = (struct ip6_hdr *)(mp->m_data + ehdrlen); ip_hlen = sizeof(struct ip6_hdr); if (mp->m_len < ehdrlen + ip_hlen) return (FALSE); ipproto = ip6->ip6_nxt; type_tucmd_mlhl |= E1000_ADVTXD_TUCMD_IPV6; break; default: offload = FALSE; break; } vlan_macip_lens |= ip_hlen; type_tucmd_mlhl |= E1000_ADVTXD_DCMD_DEXT | E1000_ADVTXD_DTYP_CTXT; switch (ipproto) { case IPPROTO_TCP: if (mp->m_pkthdr.csum_flags & CSUM_TCP) type_tucmd_mlhl |= E1000_ADVTXD_TUCMD_L4T_TCP; break; case IPPROTO_UDP: if (mp->m_pkthdr.csum_flags & CSUM_UDP) type_tucmd_mlhl |= E1000_ADVTXD_TUCMD_L4T_UDP; break; #if __FreeBSD_version >= 800000 case IPPROTO_SCTP: if (mp->m_pkthdr.csum_flags & CSUM_SCTP) type_tucmd_mlhl |= E1000_ADVTXD_TUCMD_L4T_SCTP; break; #endif default: offload = FALSE; break; } /* 82575 needs the queue index added */ if (adapter->hw.mac.type == e1000_82575) mss_l4len_idx = txr->me << 4; /* Now copy bits into descriptor */ TXD->vlan_macip_lens |= htole32(vlan_macip_lens); TXD->type_tucmd_mlhl |= htole32(type_tucmd_mlhl); TXD->seqnum_seed = htole32(0); TXD->mss_l4len_idx = htole32(mss_l4len_idx); tx_buffer->m_head = NULL; tx_buffer->next_eop = -1; /* We've consumed the first desc, adjust counters */ if (++ctxd == adapter->num_tx_desc) ctxd = 0; txr->next_avail_desc = ctxd; --txr->tx_avail; return (offload); } /********************************************************************** * * Examine each tx_buffer in the used queue. If the hardware is done * processing the packet then free associated resources. The * tx_buffer is put back on the free queue. * * TRUE return means there's work in the ring to clean, FALSE its empty. **********************************************************************/ static bool igb_txeof(struct tx_ring *txr) { struct adapter *adapter = txr->adapter; int first, last, done; struct igb_tx_buffer *tx_buffer; struct e1000_tx_desc *tx_desc, *eop_desc; struct ifnet *ifp = adapter->ifp; IGB_TX_LOCK_ASSERT(txr); if (txr->tx_avail == adapter->num_tx_desc) return FALSE; first = txr->next_to_clean; tx_desc = &txr->tx_base[first]; tx_buffer = &txr->tx_buffers[first]; last = tx_buffer->next_eop; eop_desc = &txr->tx_base[last]; /* * What this does is get the index of the * first descriptor AFTER the EOP of the * first packet, that way we can do the * simple comparison on the inner while loop. */ if (++last == adapter->num_tx_desc) last = 0; done = last; bus_dmamap_sync(txr->txdma.dma_tag, txr->txdma.dma_map, BUS_DMASYNC_POSTREAD | BUS_DMASYNC_POSTWRITE); while (eop_desc->upper.fields.status & E1000_TXD_STAT_DD) { /* We clean the range of the packet */ while (first != done) { tx_desc->upper.data = 0; tx_desc->lower.data = 0; tx_desc->buffer_addr = 0; ++txr->tx_avail; if (tx_buffer->m_head) { txr->bytes += tx_buffer->m_head->m_pkthdr.len; bus_dmamap_sync(txr->txtag, tx_buffer->map, BUS_DMASYNC_POSTWRITE); bus_dmamap_unload(txr->txtag, tx_buffer->map); m_freem(tx_buffer->m_head); tx_buffer->m_head = NULL; } tx_buffer->next_eop = -1; txr->watchdog_time = ticks; if (++first == adapter->num_tx_desc) first = 0; tx_buffer = &txr->tx_buffers[first]; tx_desc = &txr->tx_base[first]; } ++txr->packets; ++ifp->if_opackets; /* See if we can continue to the next packet */ last = tx_buffer->next_eop; if (last != -1) { eop_desc = &txr->tx_base[last]; /* Get new done point */ if (++last == adapter->num_tx_desc) last = 0; done = last; } else break; } bus_dmamap_sync(txr->txdma.dma_tag, txr->txdma.dma_map, BUS_DMASYNC_PREREAD | BUS_DMASYNC_PREWRITE); txr->next_to_clean = first; /* * If we have enough room, clear IFF_DRV_OACTIVE * to tell the stack that it is OK to send packets. */ if (txr->tx_avail > IGB_TX_CLEANUP_THRESHOLD) { ifp->if_drv_flags &= ~IFF_DRV_OACTIVE; /* All clean, turn off the watchdog */ if (txr->tx_avail == adapter->num_tx_desc) { txr->watchdog_check = FALSE; return FALSE; } } return (TRUE); } /********************************************************************* * * Refresh mbuf buffers for RX descriptor rings * - now keeps its own state so discards due to resource * exhaustion are unnecessary, if an mbuf cannot be obtained * it just returns, keeping its placeholder, thus it can simply * be recalled to try again. * **********************************************************************/ static void igb_refresh_mbufs(struct rx_ring *rxr, int limit) { struct adapter *adapter = rxr->adapter; bus_dma_segment_t hseg[1]; bus_dma_segment_t pseg[1]; struct igb_rx_buf *rxbuf; struct mbuf *mh, *mp; int i, nsegs, error, cleaned; i = rxr->next_to_refresh; cleaned = -1; /* Signify no completions */ while (i != limit) { rxbuf = &rxr->rx_buffers[i]; if (rxbuf->m_head == NULL) { mh = m_gethdr(M_DONTWAIT, MT_DATA); if (mh == NULL) goto update; mh->m_pkthdr.len = mh->m_len = MHLEN; mh->m_len = MHLEN; mh->m_flags |= M_PKTHDR; m_adj(mh, ETHER_ALIGN); /* Get the memory mapping */ error = bus_dmamap_load_mbuf_sg(rxr->htag, rxbuf->hmap, mh, hseg, &nsegs, BUS_DMA_NOWAIT); if (error != 0) { printf("GET BUF: dmamap load" " failure - %d\n", error); m_free(mh); goto update; } rxbuf->m_head = mh; bus_dmamap_sync(rxr->htag, rxbuf->hmap, BUS_DMASYNC_PREREAD); rxr->rx_base[i].read.hdr_addr = htole64(hseg[0].ds_addr); } if (rxbuf->m_pack == NULL) { mp = m_getjcl(M_DONTWAIT, MT_DATA, M_PKTHDR, adapter->rx_mbuf_sz); if (mp == NULL) goto update; mp->m_pkthdr.len = mp->m_len = adapter->rx_mbuf_sz; /* Get the memory mapping */ error = bus_dmamap_load_mbuf_sg(rxr->ptag, rxbuf->pmap, mp, pseg, &nsegs, BUS_DMA_NOWAIT); if (error != 0) { printf("GET BUF: dmamap load" " failure - %d\n", error); m_free(mp); goto update; } rxbuf->m_pack = mp; bus_dmamap_sync(rxr->ptag, rxbuf->pmap, BUS_DMASYNC_PREREAD); rxr->rx_base[i].read.pkt_addr = htole64(pseg[0].ds_addr); } cleaned = i; /* Calculate next index */ if (++i == adapter->num_rx_desc) i = 0; /* This is the work marker for refresh */ rxr->next_to_refresh = i; } update: if (cleaned != -1) /* If we refreshed some, bump tail */ E1000_WRITE_REG(&adapter->hw, E1000_RDT(rxr->me), cleaned); return; } /********************************************************************* * * Allocate memory for rx_buffer structures. Since we use one * rx_buffer per received packet, the maximum number of rx_buffer's * that we'll need is equal to the number of receive descriptors * that we've allocated. * **********************************************************************/ static int igb_allocate_receive_buffers(struct rx_ring *rxr) { struct adapter *adapter = rxr->adapter; device_t dev = adapter->dev; struct igb_rx_buf *rxbuf; int i, bsize, error; bsize = sizeof(struct igb_rx_buf) * adapter->num_rx_desc; if (!(rxr->rx_buffers = (struct igb_rx_buf *) malloc(bsize, M_DEVBUF, M_NOWAIT | M_ZERO))) { device_printf(dev, "Unable to allocate rx_buffer memory\n"); error = ENOMEM; goto fail; } if ((error = bus_dma_tag_create(bus_get_dma_tag(dev), 1, 0, /* alignment, bounds */ BUS_SPACE_MAXADDR, /* lowaddr */ BUS_SPACE_MAXADDR, /* highaddr */ NULL, NULL, /* filter, filterarg */ MSIZE, /* maxsize */ 1, /* nsegments */ MSIZE, /* maxsegsize */ 0, /* flags */ NULL, /* lockfunc */ NULL, /* lockfuncarg */ &rxr->htag))) { device_printf(dev, "Unable to create RX DMA tag\n"); goto fail; } if ((error = bus_dma_tag_create(bus_get_dma_tag(dev), 1, 0, /* alignment, bounds */ BUS_SPACE_MAXADDR, /* lowaddr */ BUS_SPACE_MAXADDR, /* highaddr */ NULL, NULL, /* filter, filterarg */ MJUMPAGESIZE, /* maxsize */ 1, /* nsegments */ MJUMPAGESIZE, /* maxsegsize */ 0, /* flags */ NULL, /* lockfunc */ NULL, /* lockfuncarg */ &rxr->ptag))) { device_printf(dev, "Unable to create RX payload DMA tag\n"); goto fail; } for (i = 0; i < adapter->num_rx_desc; i++) { rxbuf = &rxr->rx_buffers[i]; error = bus_dmamap_create(rxr->htag, BUS_DMA_NOWAIT, &rxbuf->hmap); if (error) { device_printf(dev, "Unable to create RX head DMA maps\n"); goto fail; } error = bus_dmamap_create(rxr->ptag, BUS_DMA_NOWAIT, &rxbuf->pmap); if (error) { device_printf(dev, "Unable to create RX packet DMA maps\n"); goto fail; } } return (0); fail: /* Frees all, but can handle partial completion */ igb_free_receive_structures(adapter); return (error); } static void igb_free_receive_ring(struct rx_ring *rxr) { struct adapter *adapter; struct igb_rx_buf *rxbuf; int i; adapter = rxr->adapter; for (i = 0; i < adapter->num_rx_desc; i++) { rxbuf = &rxr->rx_buffers[i]; if (rxbuf->m_head != NULL) { bus_dmamap_sync(rxr->htag, rxbuf->hmap, BUS_DMASYNC_POSTREAD); bus_dmamap_unload(rxr->htag, rxbuf->hmap); rxbuf->m_head->m_flags |= M_PKTHDR; m_freem(rxbuf->m_head); } if (rxbuf->m_pack != NULL) { bus_dmamap_sync(rxr->ptag, rxbuf->pmap, BUS_DMASYNC_POSTREAD); bus_dmamap_unload(rxr->ptag, rxbuf->pmap); rxbuf->m_pack->m_flags |= M_PKTHDR; m_freem(rxbuf->m_pack); } rxbuf->m_head = NULL; rxbuf->m_pack = NULL; } } /********************************************************************* * * Initialize a receive ring and its buffers. * **********************************************************************/ static int igb_setup_receive_ring(struct rx_ring *rxr) { struct adapter *adapter; struct ifnet *ifp; device_t dev; struct igb_rx_buf *rxbuf; bus_dma_segment_t pseg[1], hseg[1]; struct lro_ctrl *lro = &rxr->lro; int rsize, nsegs, error = 0; adapter = rxr->adapter; dev = adapter->dev; ifp = adapter->ifp; /* Clear the ring contents */ IGB_RX_LOCK(rxr); rsize = roundup2(adapter->num_rx_desc * sizeof(union e1000_adv_rx_desc), IGB_DBA_ALIGN); bzero((void *)rxr->rx_base, rsize); /* ** Free current RX buffer structures and their mbufs */ igb_free_receive_ring(rxr); /* Now replenish the ring mbufs */ for (int j = 0; j != adapter->num_rx_desc; ++j) { struct mbuf *mh, *mp; rxbuf = &rxr->rx_buffers[j]; /* First the header */ rxbuf->m_head = m_gethdr(M_DONTWAIT, MT_DATA); if (rxbuf->m_head == NULL) goto fail; m_adj(rxbuf->m_head, ETHER_ALIGN); mh = rxbuf->m_head; mh->m_len = mh->m_pkthdr.len = MHLEN; mh->m_flags |= M_PKTHDR; /* Get the memory mapping */ error = bus_dmamap_load_mbuf_sg(rxr->htag, rxbuf->hmap, rxbuf->m_head, hseg, &nsegs, BUS_DMA_NOWAIT); if (error != 0) /* Nothing elegant to do here */ goto fail; bus_dmamap_sync(rxr->htag, rxbuf->hmap, BUS_DMASYNC_PREREAD); /* Update descriptor */ rxr->rx_base[j].read.hdr_addr = htole64(hseg[0].ds_addr); /* Now the payload cluster */ rxbuf->m_pack = m_getjcl(M_DONTWAIT, MT_DATA, M_PKTHDR, adapter->rx_mbuf_sz); if (rxbuf->m_pack == NULL) goto fail; mp = rxbuf->m_pack; mp->m_pkthdr.len = mp->m_len = adapter->rx_mbuf_sz; /* Get the memory mapping */ error = bus_dmamap_load_mbuf_sg(rxr->ptag, rxbuf->pmap, mp, pseg, &nsegs, BUS_DMA_NOWAIT); if (error != 0) goto fail; bus_dmamap_sync(rxr->ptag, rxbuf->pmap, BUS_DMASYNC_PREREAD); /* Update descriptor */ rxr->rx_base[j].read.pkt_addr = htole64(pseg[0].ds_addr); } /* Setup our descriptor indices */ rxr->next_to_check = 0; rxr->next_to_refresh = 0; rxr->lro_enabled = FALSE; if (igb_header_split) rxr->hdr_split = TRUE; else ifp->if_capabilities &= ~IFCAP_LRO; rxr->fmp = NULL; rxr->lmp = NULL; rxr->discard = FALSE; bus_dmamap_sync(rxr->rxdma.dma_tag, rxr->rxdma.dma_map, BUS_DMASYNC_PREREAD | BUS_DMASYNC_PREWRITE); /* ** Now set up the LRO interface, we ** also only do head split when LRO ** is enabled, since so often they ** are undesireable in similar setups. */ if (ifp->if_capenable & IFCAP_LRO) { int err = tcp_lro_init(lro); if (err) { device_printf(dev, "LRO Initialization failed!\n"); goto fail; } INIT_DEBUGOUT("RX LRO Initialized\n"); rxr->lro_enabled = TRUE; lro->ifp = adapter->ifp; } IGB_RX_UNLOCK(rxr); return (0); fail: igb_free_receive_ring(rxr); IGB_RX_UNLOCK(rxr); return (error); } /********************************************************************* * * Initialize all receive rings. * **********************************************************************/ static int igb_setup_receive_structures(struct adapter *adapter) { struct rx_ring *rxr = adapter->rx_rings; int i, j; for (i = 0; i < adapter->num_queues; i++, rxr++) if (igb_setup_receive_ring(rxr)) goto fail; return (0); fail: /* * Free RX buffers allocated so far, we will only handle * the rings that completed, the failing case will have * cleaned up for itself. The value of 'i' will be the * failed ring so we must pre-decrement it. */ rxr = adapter->rx_rings; for (--i; i > 0; i--, rxr++) { for (j = 0; j < adapter->num_rx_desc; j++) igb_free_receive_ring(rxr); } return (ENOBUFS); } /********************************************************************* * * Enable receive unit. * **********************************************************************/ static void igb_initialize_receive_units(struct adapter *adapter) { struct rx_ring *rxr = adapter->rx_rings; struct ifnet *ifp = adapter->ifp; struct e1000_hw *hw = &adapter->hw; u32 rctl, rxcsum, psize, srrctl = 0; INIT_DEBUGOUT("igb_initialize_receive_unit: begin"); /* * Make sure receives are disabled while setting * up the descriptor ring */ rctl = E1000_READ_REG(hw, E1000_RCTL); E1000_WRITE_REG(hw, E1000_RCTL, rctl & ~E1000_RCTL_EN); /* ** Set up for header split */ if (rxr->hdr_split) { /* Use a standard mbuf for the header */ srrctl |= IGB_HDR_BUF << E1000_SRRCTL_BSIZEHDRSIZE_SHIFT; srrctl |= E1000_SRRCTL_DESCTYPE_HDR_SPLIT_ALWAYS; } else srrctl |= E1000_SRRCTL_DESCTYPE_ADV_ONEBUF; /* ** Set up for jumbo frames */ if (ifp->if_mtu > ETHERMTU) { rctl |= E1000_RCTL_LPE; srrctl |= 4096 >> E1000_SRRCTL_BSIZEPKT_SHIFT; rctl |= E1000_RCTL_SZ_4096 | E1000_RCTL_BSEX; /* Set maximum packet len */ psize = adapter->max_frame_size; /* are we on a vlan? */ if (adapter->ifp->if_vlantrunk != NULL) psize += VLAN_TAG_SIZE; E1000_WRITE_REG(&adapter->hw, E1000_RLPML, psize); } else { rctl &= ~E1000_RCTL_LPE; srrctl |= 2048 >> E1000_SRRCTL_BSIZEPKT_SHIFT; rctl |= E1000_RCTL_SZ_2048; } /* Setup the Base and Length of the Rx Descriptor Rings */ for (int i = 0; i < adapter->num_queues; i++, rxr++) { u64 bus_addr = rxr->rxdma.dma_paddr; u32 rxdctl; E1000_WRITE_REG(hw, E1000_RDLEN(i), adapter->num_rx_desc * sizeof(struct e1000_rx_desc)); E1000_WRITE_REG(hw, E1000_RDBAH(i), (uint32_t)(bus_addr >> 32)); E1000_WRITE_REG(hw, E1000_RDBAL(i), (uint32_t)bus_addr); E1000_WRITE_REG(hw, E1000_SRRCTL(i), srrctl); /* Enable this Queue */ rxdctl = E1000_READ_REG(hw, E1000_RXDCTL(i)); rxdctl |= E1000_RXDCTL_QUEUE_ENABLE; rxdctl &= 0xFFF00000; rxdctl |= IGB_RX_PTHRESH; rxdctl |= IGB_RX_HTHRESH << 8; rxdctl |= IGB_RX_WTHRESH << 16; E1000_WRITE_REG(hw, E1000_RXDCTL(i), rxdctl); } /* ** Setup for RX MultiQueue */ rxcsum = E1000_READ_REG(hw, E1000_RXCSUM); if (adapter->num_queues >1) { u32 random[10], mrqc, shift = 0; union igb_reta { u32 dword; u8 bytes[4]; } reta; arc4rand(&random, sizeof(random), 0); if (adapter->hw.mac.type == e1000_82575) shift = 6; /* Warning FM follows */ for (int i = 0; i < 128; i++) { reta.bytes[i & 3] = (i % adapter->num_queues) << shift; if ((i & 3) == 3) E1000_WRITE_REG(hw, E1000_RETA(i >> 2), reta.dword); } /* Now fill in hash table */ mrqc = E1000_MRQC_ENABLE_RSS_4Q; for (int i = 0; i < 10; i++) E1000_WRITE_REG_ARRAY(hw, E1000_RSSRK(0), i, random[i]); mrqc |= (E1000_MRQC_RSS_FIELD_IPV4 | E1000_MRQC_RSS_FIELD_IPV4_TCP); mrqc |= (E1000_MRQC_RSS_FIELD_IPV6 | E1000_MRQC_RSS_FIELD_IPV6_TCP); mrqc |=( E1000_MRQC_RSS_FIELD_IPV4_UDP | E1000_MRQC_RSS_FIELD_IPV6_UDP); mrqc |=( E1000_MRQC_RSS_FIELD_IPV6_UDP_EX | E1000_MRQC_RSS_FIELD_IPV6_TCP_EX); E1000_WRITE_REG(hw, E1000_MRQC, mrqc); /* ** NOTE: Receive Full-Packet Checksum Offload ** is mutually exclusive with Multiqueue. However ** this is not the same as TCP/IP checksums which ** still work. */ rxcsum |= E1000_RXCSUM_PCSD; #if __FreeBSD_version >= 800000 /* For SCTP Offload */ if ((hw->mac.type == e1000_82576) && (ifp->if_capenable & IFCAP_RXCSUM)) rxcsum |= E1000_RXCSUM_CRCOFL; #endif } else { /* Non RSS setup */ if (ifp->if_capenable & IFCAP_RXCSUM) { rxcsum |= E1000_RXCSUM_IPPCSE; #if __FreeBSD_version >= 800000 if (adapter->hw.mac.type == e1000_82576) rxcsum |= E1000_RXCSUM_CRCOFL; #endif } else rxcsum &= ~E1000_RXCSUM_TUOFL; } E1000_WRITE_REG(hw, E1000_RXCSUM, rxcsum); /* Setup the Receive Control Register */ rctl &= ~(3 << E1000_RCTL_MO_SHIFT); rctl |= E1000_RCTL_EN | E1000_RCTL_BAM | E1000_RCTL_LBM_NO | E1000_RCTL_RDMTS_HALF | (hw->mac.mc_filter_type << E1000_RCTL_MO_SHIFT); /* Strip CRC bytes. */ rctl |= E1000_RCTL_SECRC; /* Make sure VLAN Filters are off */ rctl &= ~E1000_RCTL_VFE; /* Don't store bad packets */ rctl &= ~E1000_RCTL_SBP; /* Enable Receives */ E1000_WRITE_REG(hw, E1000_RCTL, rctl); /* * Setup the HW Rx Head and Tail Descriptor Pointers * - needs to be after enable */ for (int i = 0; i < adapter->num_queues; i++) { E1000_WRITE_REG(hw, E1000_RDH(i), 0); E1000_WRITE_REG(hw, E1000_RDT(i), adapter->num_rx_desc - 1); } return; } /********************************************************************* * * Free receive rings. * **********************************************************************/ static void igb_free_receive_structures(struct adapter *adapter) { struct rx_ring *rxr = adapter->rx_rings; for (int i = 0; i < adapter->num_queues; i++, rxr++) { struct lro_ctrl *lro = &rxr->lro; igb_free_receive_buffers(rxr); tcp_lro_free(lro); igb_dma_free(adapter, &rxr->rxdma); } free(adapter->rx_rings, M_DEVBUF); } /********************************************************************* * * Free receive ring data structures. * **********************************************************************/ static void igb_free_receive_buffers(struct rx_ring *rxr) { struct adapter *adapter = rxr->adapter; struct igb_rx_buf *rxbuf; int i; INIT_DEBUGOUT("free_receive_structures: begin"); /* Cleanup any existing buffers */ if (rxr->rx_buffers != NULL) { for (i = 0; i < adapter->num_rx_desc; i++) { rxbuf = &rxr->rx_buffers[i]; if (rxbuf->m_head != NULL) { bus_dmamap_sync(rxr->htag, rxbuf->hmap, BUS_DMASYNC_POSTREAD); bus_dmamap_unload(rxr->htag, rxbuf->hmap); rxbuf->m_head->m_flags |= M_PKTHDR; m_freem(rxbuf->m_head); } if (rxbuf->m_pack != NULL) { bus_dmamap_sync(rxr->ptag, rxbuf->pmap, BUS_DMASYNC_POSTREAD); bus_dmamap_unload(rxr->ptag, rxbuf->pmap); rxbuf->m_pack->m_flags |= M_PKTHDR; m_freem(rxbuf->m_pack); } rxbuf->m_head = NULL; rxbuf->m_pack = NULL; if (rxbuf->hmap != NULL) { bus_dmamap_destroy(rxr->htag, rxbuf->hmap); rxbuf->hmap = NULL; } if (rxbuf->pmap != NULL) { bus_dmamap_destroy(rxr->ptag, rxbuf->pmap); rxbuf->pmap = NULL; } } if (rxr->rx_buffers != NULL) { free(rxr->rx_buffers, M_DEVBUF); rxr->rx_buffers = NULL; } } if (rxr->htag != NULL) { bus_dma_tag_destroy(rxr->htag); rxr->htag = NULL; } if (rxr->ptag != NULL) { bus_dma_tag_destroy(rxr->ptag); rxr->ptag = NULL; } } static __inline void igb_rx_discard(struct rx_ring *rxr, int i) { struct adapter *adapter = rxr->adapter; struct igb_rx_buf *rbuf; struct mbuf *mh, *mp; rbuf = &rxr->rx_buffers[i]; if (rxr->fmp != NULL) { rxr->fmp->m_flags |= M_PKTHDR; m_freem(rxr->fmp); rxr->fmp = NULL; rxr->lmp = NULL; } mh = rbuf->m_head; mp = rbuf->m_pack; /* Reuse loaded DMA map and just update mbuf chain */ mh->m_len = MHLEN; mh->m_flags |= M_PKTHDR; mh->m_next = NULL; mp->m_len = mp->m_pkthdr.len = adapter->rx_mbuf_sz; mp->m_data = mp->m_ext.ext_buf; mp->m_next = NULL; return; } static __inline void igb_rx_input(struct rx_ring *rxr, struct ifnet *ifp, struct mbuf *m, u32 ptype) { /* * ATM LRO is only for IPv4/TCP packets and TCP checksum of the packet * should be computed by hardware. Also it should not have VLAN tag in * ethernet header. */ if (rxr->lro_enabled && (ifp->if_capenable & IFCAP_VLAN_HWTAGGING) != 0 && (ptype & E1000_RXDADV_PKTTYPE_ETQF) == 0 && (ptype & (E1000_RXDADV_PKTTYPE_IPV4 | E1000_RXDADV_PKTTYPE_TCP)) == (E1000_RXDADV_PKTTYPE_IPV4 | E1000_RXDADV_PKTTYPE_TCP) && (m->m_pkthdr.csum_flags & (CSUM_DATA_VALID | CSUM_PSEUDO_HDR)) == (CSUM_DATA_VALID | CSUM_PSEUDO_HDR)) { /* * Send to the stack if: ** - LRO not enabled, or ** - no LRO resources, or ** - lro enqueue fails */ if (rxr->lro.lro_cnt != 0) if (tcp_lro_rx(&rxr->lro, m, 0) == 0) return; } (*ifp->if_input)(ifp, m); } /********************************************************************* * * This routine executes in interrupt context. It replenishes * the mbufs in the descriptor and sends data which has been * dma'ed into host memory to upper layer. * * We loop at most count times if count is > 0, or until done if * count < 0. * * Return TRUE if more to clean, FALSE otherwise *********************************************************************/ static bool igb_rxeof(struct igb_queue *que, int count) { struct adapter *adapter = que->adapter; struct rx_ring *rxr = que->rxr; struct ifnet *ifp = adapter->ifp; struct lro_ctrl *lro = &rxr->lro; struct lro_entry *queued; int i, processed = 0; u32 ptype, staterr = 0; union e1000_adv_rx_desc *cur; IGB_RX_LOCK(rxr); /* Sync the ring. */ bus_dmamap_sync(rxr->rxdma.dma_tag, rxr->rxdma.dma_map, BUS_DMASYNC_POSTREAD | BUS_DMASYNC_POSTWRITE); /* Main clean loop */ for (i = rxr->next_to_check; count != 0;) { struct mbuf *sendmp, *mh, *mp; struct igb_rx_buf *rxbuf; u16 hlen, plen, hdr, vtag; bool eop = FALSE; cur = &rxr->rx_base[i]; staterr = le32toh(cur->wb.upper.status_error); if ((staterr & E1000_RXD_STAT_DD) == 0) break; if ((ifp->if_drv_flags & IFF_DRV_RUNNING) == 0) break; count--; sendmp = mh = mp = NULL; cur->wb.upper.status_error = 0; rxbuf = &rxr->rx_buffers[i]; plen = le16toh(cur->wb.upper.length); ptype = le32toh(cur->wb.lower.lo_dword.data) & IGB_PKTTYPE_MASK; vtag = le16toh(cur->wb.upper.vlan); hdr = le16toh(cur->wb.lower.lo_dword.hs_rss.hdr_info); eop = ((staterr & E1000_RXD_STAT_EOP) == E1000_RXD_STAT_EOP); /* Make sure all segments of a bad packet are discarded */ if (((staterr & E1000_RXDEXT_ERR_FRAME_ERR_MASK) != 0) || (rxr->discard)) { ifp->if_ierrors++; ++rxr->rx_discarded; if (!eop) /* Catch subsequent segs */ rxr->discard = TRUE; else rxr->discard = FALSE; igb_rx_discard(rxr, i); goto next_desc; } /* ** The way the hardware is configured to ** split, it will ONLY use the header buffer ** when header split is enabled, otherwise we ** get normal behavior, ie, both header and ** payload are DMA'd into the payload buffer. ** ** The fmp test is to catch the case where a ** packet spans multiple descriptors, in that ** case only the first header is valid. */ if (rxr->hdr_split && rxr->fmp == NULL) { hlen = (hdr & E1000_RXDADV_HDRBUFLEN_MASK) >> E1000_RXDADV_HDRBUFLEN_SHIFT; if (hlen > IGB_HDR_BUF) hlen = IGB_HDR_BUF; /* Handle the header mbuf */ mh = rxr->rx_buffers[i].m_head; mh->m_len = hlen; /* clear buf info for refresh */ rxbuf->m_head = NULL; /* ** Get the payload length, this ** could be zero if its a small ** packet. */ if (plen > 0) { mp = rxr->rx_buffers[i].m_pack; mp->m_len = plen; mh->m_next = mp; /* clear buf info for refresh */ rxbuf->m_pack = NULL; rxr->rx_split_packets++; } } else { /* ** Either no header split, or a ** secondary piece of a fragmented ** split packet. */ mh = rxr->rx_buffers[i].m_pack; mh->m_len = plen; /* clear buf info for refresh */ rxbuf->m_pack = NULL; } ++processed; /* So we know when to refresh */ /* Initial frame - setup */ if (rxr->fmp == NULL) { mh->m_pkthdr.len = mh->m_len; /* Store the first mbuf */ rxr->fmp = mh; rxr->lmp = mh; if (mp != NULL) { /* Add payload if split */ mh->m_pkthdr.len += mp->m_len; rxr->lmp = mh->m_next; } } else { /* Chain mbuf's together */ rxr->lmp->m_next = mh; rxr->lmp = rxr->lmp->m_next; rxr->fmp->m_pkthdr.len += mh->m_len; } if (eop) { rxr->fmp->m_pkthdr.rcvif = ifp; ifp->if_ipackets++; rxr->rx_packets++; /* capture data for AIM */ rxr->packets++; rxr->bytes += rxr->fmp->m_pkthdr.len; rxr->rx_bytes += rxr->fmp->m_pkthdr.len; if ((ifp->if_capenable & IFCAP_RXCSUM) != 0) igb_rx_checksum(staterr, rxr->fmp, ptype); if ((ifp->if_capenable & IFCAP_VLAN_HWTAGGING) != 0 && (staterr & E1000_RXD_STAT_VP) != 0) { rxr->fmp->m_pkthdr.ether_vtag = vtag; rxr->fmp->m_flags |= M_VLANTAG; } #if __FreeBSD_version >= 800000 rxr->fmp->m_pkthdr.flowid = que->msix; rxr->fmp->m_flags |= M_FLOWID; #endif sendmp = rxr->fmp; /* Make sure to set M_PKTHDR. */ sendmp->m_flags |= M_PKTHDR; rxr->fmp = NULL; rxr->lmp = NULL; } next_desc: bus_dmamap_sync(rxr->rxdma.dma_tag, rxr->rxdma.dma_map, BUS_DMASYNC_PREREAD | BUS_DMASYNC_PREWRITE); /* Advance our pointers to the next descriptor. */ if (++i == adapter->num_rx_desc) i = 0; /* ** Send to the stack or LRO */ if (sendmp != NULL) igb_rx_input(rxr, ifp, sendmp, ptype); /* Every 8 descriptors we go to refresh mbufs */ if (processed == 8) { igb_refresh_mbufs(rxr, i); processed = 0; } } /* Catch any remainders */ if (processed != 0) { igb_refresh_mbufs(rxr, i); processed = 0; } rxr->next_to_check = i; /* * Flush any outstanding LRO work */ while ((queued = SLIST_FIRST(&lro->lro_active)) != NULL) { SLIST_REMOVE_HEAD(&lro->lro_active, next); tcp_lro_flush(lro, queued); } IGB_RX_UNLOCK(rxr); /* ** We still have cleaning to do? ** Schedule another interrupt if so. */ if ((staterr & E1000_RXD_STAT_DD) != 0) return (TRUE); return (FALSE); } /********************************************************************* * * Verify that the hardware indicated that the checksum is valid. * Inform the stack about the status of checksum so that stack * doesn't spend time verifying the checksum. * *********************************************************************/ static void igb_rx_checksum(u32 staterr, struct mbuf *mp, u32 ptype) { u16 status = (u16)staterr; u8 errors = (u8) (staterr >> 24); int sctp; /* Ignore Checksum bit is set */ if (status & E1000_RXD_STAT_IXSM) { mp->m_pkthdr.csum_flags = 0; return; } if ((ptype & E1000_RXDADV_PKTTYPE_ETQF) == 0 && (ptype & E1000_RXDADV_PKTTYPE_SCTP) != 0) sctp = 1; else sctp = 0; if (status & E1000_RXD_STAT_IPCS) { /* Did it pass? */ if (!(errors & E1000_RXD_ERR_IPE)) { /* IP Checksum Good */ mp->m_pkthdr.csum_flags = CSUM_IP_CHECKED; mp->m_pkthdr.csum_flags |= CSUM_IP_VALID; } else mp->m_pkthdr.csum_flags = 0; } if (status & (E1000_RXD_STAT_TCPCS | E1000_RXD_STAT_UDPCS)) { u16 type = (CSUM_DATA_VALID | CSUM_PSEUDO_HDR); #if __FreeBSD_version >= 800000 if (sctp) /* reassign */ type = CSUM_SCTP_VALID; #endif /* Did it pass? */ if (!(errors & E1000_RXD_ERR_TCPE)) { mp->m_pkthdr.csum_flags |= type; if (sctp == 0) mp->m_pkthdr.csum_data = htons(0xffff); } } return; } /* * This routine is run via an vlan * config EVENT */ static void igb_register_vlan(void *arg, struct ifnet *ifp, u16 vtag) { struct adapter *adapter = ifp->if_softc; u32 index, bit; if (ifp->if_softc != arg) /* Not our event */ return; if ((vtag == 0) || (vtag > 4095)) /* Invalid */ return; index = (vtag >> 5) & 0x7F; bit = vtag & 0x1F; igb_shadow_vfta[index] |= (1 << bit); ++adapter->num_vlans; /* Re-init to load the changes */ igb_init(adapter); } /* * This routine is run via an vlan * unconfig EVENT */ static void igb_unregister_vlan(void *arg, struct ifnet *ifp, u16 vtag) { struct adapter *adapter = ifp->if_softc; u32 index, bit; if (ifp->if_softc != arg) return; if ((vtag == 0) || (vtag > 4095)) /* Invalid */ return; index = (vtag >> 5) & 0x7F; bit = vtag & 0x1F; igb_shadow_vfta[index] &= ~(1 << bit); --adapter->num_vlans; /* Re-init to load the changes */ igb_init(adapter); } static void igb_setup_vlan_hw_support(struct adapter *adapter) { struct e1000_hw *hw = &adapter->hw; u32 reg; /* ** We get here thru init_locked, meaning ** a soft reset, this has already cleared ** the VFTA and other state, so if there ** have been no vlan's registered do nothing. */ if (adapter->num_vlans == 0) return; /* ** A soft reset zero's out the VFTA, so ** we need to repopulate it now. */ for (int i = 0; i < IGB_VFTA_SIZE; i++) if (igb_shadow_vfta[i] != 0) E1000_WRITE_REG_ARRAY(hw, E1000_VFTA, i, igb_shadow_vfta[i]); reg = E1000_READ_REG(hw, E1000_CTRL); reg |= E1000_CTRL_VME; E1000_WRITE_REG(hw, E1000_CTRL, reg); /* Enable the Filter Table */ reg = E1000_READ_REG(hw, E1000_RCTL); reg &= ~E1000_RCTL_CFIEN; reg |= E1000_RCTL_VFE; E1000_WRITE_REG(hw, E1000_RCTL, reg); /* Update the frame size */ E1000_WRITE_REG(&adapter->hw, E1000_RLPML, adapter->max_frame_size + VLAN_TAG_SIZE); } static void igb_enable_intr(struct adapter *adapter) { /* With RSS set up what to auto clear */ if (adapter->msix_mem) { E1000_WRITE_REG(&adapter->hw, E1000_EIAC, adapter->eims_mask); E1000_WRITE_REG(&adapter->hw, E1000_EIAM, adapter->eims_mask); E1000_WRITE_REG(&adapter->hw, E1000_EIMS, adapter->eims_mask); E1000_WRITE_REG(&adapter->hw, E1000_IMS, E1000_IMS_LSC); } else { E1000_WRITE_REG(&adapter->hw, E1000_IMS, IMS_ENABLE_MASK); } E1000_WRITE_FLUSH(&adapter->hw); return; } static void igb_disable_intr(struct adapter *adapter) { if (adapter->msix_mem) { E1000_WRITE_REG(&adapter->hw, E1000_EIMC, ~0); E1000_WRITE_REG(&adapter->hw, E1000_EIAC, 0); } E1000_WRITE_REG(&adapter->hw, E1000_IMC, ~0); E1000_WRITE_FLUSH(&adapter->hw); return; } /* * Bit of a misnomer, what this really means is * to enable OS management of the system... aka * to disable special hardware management features */ static void igb_init_manageability(struct adapter *adapter) { if (adapter->has_manage) { int manc2h = E1000_READ_REG(&adapter->hw, E1000_MANC2H); int manc = E1000_READ_REG(&adapter->hw, E1000_MANC); /* disable hardware interception of ARP */ manc &= ~(E1000_MANC_ARP_EN); /* enable receiving management packets to the host */ manc |= E1000_MANC_EN_MNG2HOST; manc2h |= 1 << 5; /* Mng Port 623 */ manc2h |= 1 << 6; /* Mng Port 664 */ E1000_WRITE_REG(&adapter->hw, E1000_MANC2H, manc2h); E1000_WRITE_REG(&adapter->hw, E1000_MANC, manc); } } /* * Give control back to hardware management * controller if there is one. */ static void igb_release_manageability(struct adapter *adapter) { if (adapter->has_manage) { int manc = E1000_READ_REG(&adapter->hw, E1000_MANC); /* re-enable hardware interception of ARP */ manc |= E1000_MANC_ARP_EN; manc &= ~E1000_MANC_EN_MNG2HOST; E1000_WRITE_REG(&adapter->hw, E1000_MANC, manc); } } /* * igb_get_hw_control sets CTRL_EXT:DRV_LOAD bit. * For ASF and Pass Through versions of f/w this means that * the driver is loaded. * */ static void igb_get_hw_control(struct adapter *adapter) { u32 ctrl_ext; /* Let firmware know the driver has taken over */ ctrl_ext = E1000_READ_REG(&adapter->hw, E1000_CTRL_EXT); E1000_WRITE_REG(&adapter->hw, E1000_CTRL_EXT, ctrl_ext | E1000_CTRL_EXT_DRV_LOAD); } /* * igb_release_hw_control resets CTRL_EXT:DRV_LOAD bit. * For ASF and Pass Through versions of f/w this means that the * driver is no longer loaded. * */ static void igb_release_hw_control(struct adapter *adapter) { u32 ctrl_ext; /* Let firmware taken over control of h/w */ ctrl_ext = E1000_READ_REG(&adapter->hw, E1000_CTRL_EXT); E1000_WRITE_REG(&adapter->hw, E1000_CTRL_EXT, ctrl_ext & ~E1000_CTRL_EXT_DRV_LOAD); } static int igb_is_valid_ether_addr(uint8_t *addr) { char zero_addr[6] = { 0, 0, 0, 0, 0, 0 }; if ((addr[0] & 1) || (!bcmp(addr, zero_addr, ETHER_ADDR_LEN))) { return (FALSE); } return (TRUE); } /* * Enable PCI Wake On Lan capability */ static void igb_enable_wakeup(device_t dev) { u16 cap, status; u8 id; /* First find the capabilities pointer*/ cap = pci_read_config(dev, PCIR_CAP_PTR, 2); /* Read the PM Capabilities */ id = pci_read_config(dev, cap, 1); if (id != PCIY_PMG) /* Something wrong */ return; /* OK, we have the power capabilities, so now get the status register */ cap += PCIR_POWER_STATUS; status = pci_read_config(dev, cap, 2); status |= PCIM_PSTAT_PME | PCIM_PSTAT_PMEENABLE; pci_write_config(dev, cap, status, 2); return; } static void igb_led_func(void *arg, int onoff) { struct adapter *adapter = arg; IGB_CORE_LOCK(adapter); if (onoff) { e1000_setup_led(&adapter->hw); e1000_led_on(&adapter->hw); } else { e1000_led_off(&adapter->hw); e1000_cleanup_led(&adapter->hw); } IGB_CORE_UNLOCK(adapter); } /********************************************************************** * * Update the board statistics counters. * **********************************************************************/ static void igb_update_stats_counters(struct adapter *adapter) { struct ifnet *ifp; if (adapter->hw.phy.media_type == e1000_media_type_copper || (E1000_READ_REG(&adapter->hw, E1000_STATUS) & E1000_STATUS_LU)) { adapter->stats.symerrs += E1000_READ_REG(&adapter->hw, E1000_SYMERRS); adapter->stats.sec += E1000_READ_REG(&adapter->hw, E1000_SEC); } adapter->stats.crcerrs += E1000_READ_REG(&adapter->hw, E1000_CRCERRS); adapter->stats.mpc += E1000_READ_REG(&adapter->hw, E1000_MPC); adapter->stats.scc += E1000_READ_REG(&adapter->hw, E1000_SCC); adapter->stats.ecol += E1000_READ_REG(&adapter->hw, E1000_ECOL); adapter->stats.mcc += E1000_READ_REG(&adapter->hw, E1000_MCC); adapter->stats.latecol += E1000_READ_REG(&adapter->hw, E1000_LATECOL); adapter->stats.colc += E1000_READ_REG(&adapter->hw, E1000_COLC); adapter->stats.dc += E1000_READ_REG(&adapter->hw, E1000_DC); adapter->stats.rlec += E1000_READ_REG(&adapter->hw, E1000_RLEC); adapter->stats.xonrxc += E1000_READ_REG(&adapter->hw, E1000_XONRXC); adapter->stats.xontxc += E1000_READ_REG(&adapter->hw, E1000_XONTXC); adapter->stats.xoffrxc += E1000_READ_REG(&adapter->hw, E1000_XOFFRXC); adapter->stats.xofftxc += E1000_READ_REG(&adapter->hw, E1000_XOFFTXC); adapter->stats.fcruc += E1000_READ_REG(&adapter->hw, E1000_FCRUC); adapter->stats.prc64 += E1000_READ_REG(&adapter->hw, E1000_PRC64); adapter->stats.prc127 += E1000_READ_REG(&adapter->hw, E1000_PRC127); adapter->stats.prc255 += E1000_READ_REG(&adapter->hw, E1000_PRC255); adapter->stats.prc511 += E1000_READ_REG(&adapter->hw, E1000_PRC511); adapter->stats.prc1023 += E1000_READ_REG(&adapter->hw, E1000_PRC1023); adapter->stats.prc1522 += E1000_READ_REG(&adapter->hw, E1000_PRC1522); adapter->stats.gprc += E1000_READ_REG(&adapter->hw, E1000_GPRC); adapter->stats.bprc += E1000_READ_REG(&adapter->hw, E1000_BPRC); adapter->stats.mprc += E1000_READ_REG(&adapter->hw, E1000_MPRC); adapter->stats.gptc += E1000_READ_REG(&adapter->hw, E1000_GPTC); /* For the 64-bit byte counters the low dword must be read first. */ /* Both registers clear on the read of the high dword */ adapter->stats.gorc += E1000_READ_REG(&adapter->hw, E1000_GORCH); adapter->stats.gotc += E1000_READ_REG(&adapter->hw, E1000_GOTCH); adapter->stats.rnbc += E1000_READ_REG(&adapter->hw, E1000_RNBC); adapter->stats.ruc += E1000_READ_REG(&adapter->hw, E1000_RUC); adapter->stats.rfc += E1000_READ_REG(&adapter->hw, E1000_RFC); adapter->stats.roc += E1000_READ_REG(&adapter->hw, E1000_ROC); adapter->stats.rjc += E1000_READ_REG(&adapter->hw, E1000_RJC); adapter->stats.tor += E1000_READ_REG(&adapter->hw, E1000_TORH); adapter->stats.tot += E1000_READ_REG(&adapter->hw, E1000_TOTH); adapter->stats.tpr += E1000_READ_REG(&adapter->hw, E1000_TPR); adapter->stats.tpt += E1000_READ_REG(&adapter->hw, E1000_TPT); adapter->stats.ptc64 += E1000_READ_REG(&adapter->hw, E1000_PTC64); adapter->stats.ptc127 += E1000_READ_REG(&adapter->hw, E1000_PTC127); adapter->stats.ptc255 += E1000_READ_REG(&adapter->hw, E1000_PTC255); adapter->stats.ptc511 += E1000_READ_REG(&adapter->hw, E1000_PTC511); adapter->stats.ptc1023 += E1000_READ_REG(&adapter->hw, E1000_PTC1023); adapter->stats.ptc1522 += E1000_READ_REG(&adapter->hw, E1000_PTC1522); adapter->stats.mptc += E1000_READ_REG(&adapter->hw, E1000_MPTC); adapter->stats.bptc += E1000_READ_REG(&adapter->hw, E1000_BPTC); adapter->stats.algnerrc += E1000_READ_REG(&adapter->hw, E1000_ALGNERRC); adapter->stats.rxerrc += E1000_READ_REG(&adapter->hw, E1000_RXERRC); adapter->stats.tncrs += E1000_READ_REG(&adapter->hw, E1000_TNCRS); adapter->stats.cexterr += E1000_READ_REG(&adapter->hw, E1000_CEXTERR); adapter->stats.tsctc += E1000_READ_REG(&adapter->hw, E1000_TSCTC); adapter->stats.tsctfc += E1000_READ_REG(&adapter->hw, E1000_TSCTFC); ifp = adapter->ifp; ifp->if_collisions = adapter->stats.colc; /* Rx Errors */ ifp->if_ierrors = adapter->dropped_pkts + adapter->stats.rxerrc + adapter->stats.crcerrs + adapter->stats.algnerrc + adapter->stats.ruc + adapter->stats.roc + adapter->stats.mpc + adapter->stats.cexterr; /* Tx Errors */ ifp->if_oerrors = adapter->stats.ecol + adapter->stats.latecol + adapter->watchdog_events; } /********************************************************************** * * This routine is called only when igb_display_debug_stats is enabled. * This routine provides a way to take a look at important statistics * maintained by the driver and hardware. * **********************************************************************/ static void igb_print_debug_info(struct adapter *adapter) { device_t dev = adapter->dev; struct igb_queue *que = adapter->queues; struct rx_ring *rxr = adapter->rx_rings; struct tx_ring *txr = adapter->tx_rings; uint8_t *hw_addr = adapter->hw.hw_addr; device_printf(dev, "Adapter hardware address = %p \n", hw_addr); device_printf(dev, "CTRL = 0x%x RCTL = 0x%x \n", E1000_READ_REG(&adapter->hw, E1000_CTRL), E1000_READ_REG(&adapter->hw, E1000_RCTL)); #if (DEBUG_HW > 0) /* Dont output these errors normally */ device_printf(dev, "IMS = 0x%x EIMS = 0x%x \n", E1000_READ_REG(&adapter->hw, E1000_IMS), E1000_READ_REG(&adapter->hw, E1000_EIMS)); #endif device_printf(dev, "Packet buffer = Tx=%dk Rx=%dk \n", ((E1000_READ_REG(&adapter->hw, E1000_PBA) & 0xffff0000) >> 16),\ (E1000_READ_REG(&adapter->hw, E1000_PBA) & 0xffff) ); device_printf(dev, "Flow control watermarks high = %d low = %d\n", adapter->hw.fc.high_water, adapter->hw.fc.low_water); for (int i = 0; i < adapter->num_queues; i++, rxr++, txr++) { device_printf(dev, "Queue(%d) tdh = %d, tdt = %d ", i, E1000_READ_REG(&adapter->hw, E1000_TDH(i)), E1000_READ_REG(&adapter->hw, E1000_TDT(i))); device_printf(dev, "rdh = %d, rdt = %d\n", E1000_READ_REG(&adapter->hw, E1000_RDH(i)), E1000_READ_REG(&adapter->hw, E1000_RDT(i))); device_printf(dev, "TX(%d) no descriptors avail event = %lld\n", txr->me, (long long)txr->no_desc_avail); device_printf(dev, "TX(%d) Packets sent = %lld\n", txr->me, (long long)txr->tx_packets); device_printf(dev, "RX(%d) Packets received = %lld ", rxr->me, (long long)rxr->rx_packets); } for (int i = 0; i < adapter->num_queues; i++, rxr++) { struct lro_ctrl *lro = &rxr->lro; device_printf(dev, "Queue(%d) rdh = %d, rdt = %d\n", i, E1000_READ_REG(&adapter->hw, E1000_RDH(i)), E1000_READ_REG(&adapter->hw, E1000_RDT(i))); device_printf(dev, "RX(%d) Packets received = %lld\n", rxr->me, (long long)rxr->rx_packets); device_printf(dev, " Split Packets = %lld ", (long long)rxr->rx_split_packets); device_printf(dev, " Byte count = %lld\n", (long long)rxr->rx_bytes); device_printf(dev,"RX(%d) LRO Queued= %d ", i, lro->lro_queued); device_printf(dev,"LRO Flushed= %d\n",lro->lro_flushed); } for (int i = 0; i < adapter->num_queues; i++, que++) device_printf(dev,"QUE(%d) IRQs = %llx\n", i, (long long)que->irqs); device_printf(dev, "LINK MSIX IRQ Handled = %u\n", adapter->link_irq); device_printf(dev, "Mbuf defrag failed = %ld\n", adapter->mbuf_defrag_failed); device_printf(dev, "Std mbuf header failed = %ld\n", adapter->mbuf_header_failed); device_printf(dev, "Std mbuf packet failed = %ld\n", adapter->mbuf_packet_failed); device_printf(dev, "Driver dropped packets = %ld\n", adapter->dropped_pkts); device_printf(dev, "Driver tx dma failure in xmit = %ld\n", adapter->no_tx_dma_setup); } static void igb_print_hw_stats(struct adapter *adapter) { device_t dev = adapter->dev; device_printf(dev, "Excessive collisions = %lld\n", (long long)adapter->stats.ecol); #if (DEBUG_HW > 0) /* Dont output these errors normally */ device_printf(dev, "Symbol errors = %lld\n", (long long)adapter->stats.symerrs); #endif device_printf(dev, "Sequence errors = %lld\n", (long long)adapter->stats.sec); device_printf(dev, "Defer count = %lld\n", (long long)adapter->stats.dc); device_printf(dev, "Missed Packets = %lld\n", (long long)adapter->stats.mpc); device_printf(dev, "Receive No Buffers = %lld\n", (long long)adapter->stats.rnbc); /* RLEC is inaccurate on some hardware, calculate our own. */ device_printf(dev, "Receive Length Errors = %lld\n", ((long long)adapter->stats.roc + (long long)adapter->stats.ruc)); device_printf(dev, "Receive errors = %lld\n", (long long)adapter->stats.rxerrc); device_printf(dev, "Crc errors = %lld\n", (long long)adapter->stats.crcerrs); device_printf(dev, "Alignment errors = %lld\n", (long long)adapter->stats.algnerrc); /* On 82575 these are collision counts */ device_printf(dev, "Collision/Carrier extension errors = %lld\n", (long long)adapter->stats.cexterr); device_printf(dev, "RX overruns = %ld\n", adapter->rx_overruns); device_printf(dev, "watchdog timeouts = %ld\n", adapter->watchdog_events); device_printf(dev, "XON Rcvd = %lld\n", (long long)adapter->stats.xonrxc); device_printf(dev, "XON Xmtd = %lld\n", (long long)adapter->stats.xontxc); device_printf(dev, "XOFF Rcvd = %lld\n", (long long)adapter->stats.xoffrxc); device_printf(dev, "XOFF Xmtd = %lld\n", (long long)adapter->stats.xofftxc); device_printf(dev, "Good Packets Rcvd = %lld\n", (long long)adapter->stats.gprc); device_printf(dev, "Good Packets Xmtd = %lld\n", (long long)adapter->stats.gptc); device_printf(dev, "TSO Contexts Xmtd = %lld\n", (long long)adapter->stats.tsctc); device_printf(dev, "TSO Contexts Failed = %lld\n", (long long)adapter->stats.tsctfc); } /********************************************************************** * * This routine provides a way to dump out the adapter eeprom, * often a useful debug/service tool. This only dumps the first * 32 words, stuff that matters is in that extent. * **********************************************************************/ static void igb_print_nvm_info(struct adapter *adapter) { u16 eeprom_data; int i, j, row = 0; /* Its a bit crude, but it gets the job done */ printf("\nInterface EEPROM Dump:\n"); printf("Offset\n0x0000 "); for (i = 0, j = 0; i < 32; i++, j++) { if (j == 8) { /* Make the offset block */ j = 0; ++row; printf("\n0x00%x0 ",row); } e1000_read_nvm(&adapter->hw, i, 1, &eeprom_data); printf("%04x ", eeprom_data); } printf("\n"); } static int igb_sysctl_debug_info(SYSCTL_HANDLER_ARGS) { struct adapter *adapter; int error; int result; result = -1; error = sysctl_handle_int(oidp, &result, 0, req); if (error || !req->newptr) return (error); if (result == 1) { adapter = (struct adapter *)arg1; igb_print_debug_info(adapter); } /* * This value will cause a hex dump of the * first 32 16-bit words of the EEPROM to * the screen. */ if (result == 2) { adapter = (struct adapter *)arg1; igb_print_nvm_info(adapter); } return (error); } static int igb_sysctl_stats(SYSCTL_HANDLER_ARGS) { struct adapter *adapter; int error; int result; result = -1; error = sysctl_handle_int(oidp, &result, 0, req); if (error || !req->newptr) return (error); if (result == 1) { adapter = (struct adapter *)arg1; igb_print_hw_stats(adapter); } return (error); } static void igb_add_rx_process_limit(struct adapter *adapter, const char *name, const char *description, int *limit, int value) { *limit = value; SYSCTL_ADD_INT(device_get_sysctl_ctx(adapter->dev), SYSCTL_CHILDREN(device_get_sysctl_tree(adapter->dev)), OID_AUTO, name, CTLTYPE_INT|CTLFLAG_RW, limit, value, description); } Index: user/imp/tbemd/sys/dev/hwpmc/hwpmc_core.c =================================================================== --- user/imp/tbemd/sys/dev/hwpmc/hwpmc_core.c (revision 206684) +++ user/imp/tbemd/sys/dev/hwpmc/hwpmc_core.c (revision 206685) @@ -1,2254 +1,2255 @@ /*- * Copyright (c) 2008 Joseph Koshy * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ /* * Intel Core, Core 2 and Atom PMCs. */ #include __FBSDID("$FreeBSD$"); #include #include #include #include #include #include #include #include #include #include #define CORE_CPUID_REQUEST 0xA #define CORE_CPUID_REQUEST_SIZE 0x4 #define CORE_CPUID_EAX 0x0 #define CORE_CPUID_EBX 0x1 #define CORE_CPUID_ECX 0x2 #define CORE_CPUID_EDX 0x3 #define IAF_PMC_CAPS \ (PMC_CAP_READ | PMC_CAP_WRITE | PMC_CAP_INTERRUPT) #define IAF_RI_TO_MSR(RI) ((RI) + (1 << 30)) #define IAP_PMC_CAPS (PMC_CAP_INTERRUPT | PMC_CAP_USER | PMC_CAP_SYSTEM | \ PMC_CAP_EDGE | PMC_CAP_THRESHOLD | PMC_CAP_READ | PMC_CAP_WRITE | \ PMC_CAP_INVERT | PMC_CAP_QUALIFIER | PMC_CAP_PRECISE) /* * "Architectural" events defined by Intel. The values of these * symbols correspond to positions in the bitmask returned by * the CPUID.0AH instruction. */ enum core_arch_events { CORE_AE_BRANCH_INSTRUCTION_RETIRED = 5, CORE_AE_BRANCH_MISSES_RETIRED = 6, CORE_AE_INSTRUCTION_RETIRED = 1, CORE_AE_LLC_MISSES = 4, CORE_AE_LLC_REFERENCE = 3, CORE_AE_UNHALTED_REFERENCE_CYCLES = 2, CORE_AE_UNHALTED_CORE_CYCLES = 0 }; static enum pmc_cputype core_cputype; struct core_cpu { volatile uint32_t pc_resync; volatile uint32_t pc_iafctrl; /* Fixed function control. */ volatile uint64_t pc_globalctrl; /* Global control register. */ struct pmc_hw pc_corepmcs[]; }; static struct core_cpu **core_pcpu; static uint32_t core_architectural_events; static uint64_t core_pmcmask; static int core_iaf_ri; /* relative index of fixed counters */ static int core_iaf_width; static int core_iaf_npmc; static int core_iap_width; static int core_iap_npmc; static int core_pcpu_noop(struct pmc_mdep *md, int cpu) { (void) md; (void) cpu; return (0); } static int core_pcpu_init(struct pmc_mdep *md, int cpu) { struct pmc_cpu *pc; struct core_cpu *cc; struct pmc_hw *phw; int core_ri, n, npmc; KASSERT(cpu >= 0 && cpu < pmc_cpu_max(), ("[iaf,%d] insane cpu number %d", __LINE__, cpu)); PMCDBG(MDP,INI,1,"core-init cpu=%d", cpu); core_ri = md->pmd_classdep[PMC_MDEP_CLASS_INDEX_IAP].pcd_ri; npmc = md->pmd_classdep[PMC_MDEP_CLASS_INDEX_IAP].pcd_num; if (core_cputype != PMC_CPU_INTEL_CORE) npmc += md->pmd_classdep[PMC_MDEP_CLASS_INDEX_IAF].pcd_num; cc = malloc(sizeof(struct core_cpu) + npmc * sizeof(struct pmc_hw), M_PMC, M_WAITOK | M_ZERO); core_pcpu[cpu] = cc; pc = pmc_pcpu[cpu]; KASSERT(pc != NULL && cc != NULL, ("[core,%d] NULL per-cpu structures cpu=%d", __LINE__, cpu)); for (n = 0, phw = cc->pc_corepmcs; n < npmc; n++, phw++) { phw->phw_state = PMC_PHW_FLAG_IS_ENABLED | PMC_PHW_CPU_TO_STATE(cpu) | PMC_PHW_INDEX_TO_STATE(n + core_ri); phw->phw_pmc = NULL; pc->pc_hwpmcs[n + core_ri] = phw; } return (0); } static int core_pcpu_fini(struct pmc_mdep *md, int cpu) { int core_ri, n, npmc; struct pmc_cpu *pc; struct core_cpu *cc; KASSERT(cpu >= 0 && cpu < pmc_cpu_max(), ("[core,%d] insane cpu number (%d)", __LINE__, cpu)); PMCDBG(MDP,INI,1,"core-pcpu-fini cpu=%d", cpu); if ((cc = core_pcpu[cpu]) == NULL) return (0); core_pcpu[cpu] = NULL; pc = pmc_pcpu[cpu]; KASSERT(pc != NULL, ("[core,%d] NULL per-cpu %d state", __LINE__, cpu)); npmc = md->pmd_classdep[PMC_MDEP_CLASS_INDEX_IAP].pcd_num; core_ri = md->pmd_classdep[PMC_MDEP_CLASS_INDEX_IAP].pcd_ri; for (n = 0; n < npmc; n++) wrmsr(IAP_EVSEL0 + n, 0); if (core_cputype != PMC_CPU_INTEL_CORE) { wrmsr(IAF_CTRL, 0); npmc += md->pmd_classdep[PMC_MDEP_CLASS_INDEX_IAF].pcd_num; } for (n = 0; n < npmc; n++) pc->pc_hwpmcs[n + core_ri] = NULL; free(cc, M_PMC); return (0); } /* * Fixed function counters. */ static pmc_value_t iaf_perfctr_value_to_reload_count(pmc_value_t v) { v &= (1ULL << core_iaf_width) - 1; return (1ULL << core_iaf_width) - v; } static pmc_value_t iaf_reload_count_to_perfctr_value(pmc_value_t rlc) { return (1ULL << core_iaf_width) - rlc; } static int iaf_allocate_pmc(int cpu, int ri, struct pmc *pm, const struct pmc_op_pmcallocate *a) { enum pmc_event ev; uint32_t caps, flags, validflags; KASSERT(cpu >= 0 && cpu < pmc_cpu_max(), ("[core,%d] illegal CPU %d", __LINE__, cpu)); PMCDBG(MDP,ALL,1, "iaf-allocate ri=%d reqcaps=0x%x", ri, pm->pm_caps); if (ri < 0 || ri > core_iaf_npmc) return (EINVAL); caps = a->pm_caps; if (a->pm_class != PMC_CLASS_IAF || (caps & IAF_PMC_CAPS) != caps) return (EINVAL); ev = pm->pm_event; if (ev < PMC_EV_IAF_FIRST || ev > PMC_EV_IAF_LAST) return (EINVAL); if (ev == PMC_EV_IAF_INSTR_RETIRED_ANY && ri != 0) return (EINVAL); if (ev == PMC_EV_IAF_CPU_CLK_UNHALTED_CORE && ri != 1) return (EINVAL); if (ev == PMC_EV_IAF_CPU_CLK_UNHALTED_REF && ri != 2) return (EINVAL); flags = a->pm_md.pm_iaf.pm_iaf_flags; validflags = IAF_MASK; if (core_cputype != PMC_CPU_INTEL_ATOM) validflags &= ~IAF_ANY; if ((flags & ~validflags) != 0) return (EINVAL); if (caps & PMC_CAP_INTERRUPT) flags |= IAF_PMI; if (caps & PMC_CAP_SYSTEM) flags |= IAF_OS; if (caps & PMC_CAP_USER) flags |= IAF_USR; if ((caps & (PMC_CAP_USER | PMC_CAP_SYSTEM)) == 0) flags |= (IAF_OS | IAF_USR); pm->pm_md.pm_iaf.pm_iaf_ctrl = (flags << (ri * 4)); PMCDBG(MDP,ALL,2, "iaf-allocate config=0x%jx", (uintmax_t) pm->pm_md.pm_iaf.pm_iaf_ctrl); return (0); } static int iaf_config_pmc(int cpu, int ri, struct pmc *pm) { KASSERT(cpu >= 0 && cpu < pmc_cpu_max(), ("[core,%d] illegal CPU %d", __LINE__, cpu)); KASSERT(ri >= 0 && ri < core_iaf_npmc, ("[core,%d] illegal row-index %d", __LINE__, ri)); PMCDBG(MDP,CFG,1, "iaf-config cpu=%d ri=%d pm=%p", cpu, ri, pm); KASSERT(core_pcpu[cpu] != NULL, ("[core,%d] null per-cpu %d", __LINE__, cpu)); core_pcpu[cpu]->pc_corepmcs[ri + core_iaf_ri].phw_pmc = pm; return (0); } static int iaf_describe(int cpu, int ri, struct pmc_info *pi, struct pmc **ppmc) { int error; struct pmc_hw *phw; char iaf_name[PMC_NAME_MAX]; phw = &core_pcpu[cpu]->pc_corepmcs[ri + core_iaf_ri]; (void) snprintf(iaf_name, sizeof(iaf_name), "IAF-%d", ri); if ((error = copystr(iaf_name, pi->pm_name, PMC_NAME_MAX, NULL)) != 0) return (error); pi->pm_class = PMC_CLASS_IAF; if (phw->phw_state & PMC_PHW_FLAG_IS_ENABLED) { pi->pm_enabled = TRUE; *ppmc = phw->phw_pmc; } else { pi->pm_enabled = FALSE; *ppmc = NULL; } return (0); } static int iaf_get_config(int cpu, int ri, struct pmc **ppm) { *ppm = core_pcpu[cpu]->pc_corepmcs[ri + core_iaf_ri].phw_pmc; return (0); } static int iaf_get_msr(int ri, uint32_t *msr) { KASSERT(ri >= 0 && ri < core_iaf_npmc, ("[iaf,%d] ri %d out of range", __LINE__, ri)); *msr = IAF_RI_TO_MSR(ri); return (0); } static int iaf_read_pmc(int cpu, int ri, pmc_value_t *v) { struct pmc *pm; pmc_value_t tmp; KASSERT(cpu >= 0 && cpu < pmc_cpu_max(), ("[core,%d] illegal cpu value %d", __LINE__, cpu)); KASSERT(ri >= 0 && ri < core_iaf_npmc, ("[core,%d] illegal row-index %d", __LINE__, ri)); pm = core_pcpu[cpu]->pc_corepmcs[ri + core_iaf_ri].phw_pmc; KASSERT(pm, ("[core,%d] cpu %d ri %d(%d) pmc not configured", __LINE__, cpu, ri, ri + core_iaf_ri)); tmp = rdpmc(IAF_RI_TO_MSR(ri)); if (PMC_IS_SAMPLING_MODE(PMC_TO_MODE(pm))) *v = iaf_perfctr_value_to_reload_count(tmp); else *v = tmp; PMCDBG(MDP,REA,1, "iaf-read cpu=%d ri=%d msr=0x%x -> v=%jx", cpu, ri, IAF_RI_TO_MSR(ri), *v); return (0); } static int iaf_release_pmc(int cpu, int ri, struct pmc *pmc) { PMCDBG(MDP,REL,1, "iaf-release cpu=%d ri=%d pm=%p", cpu, ri, pmc); KASSERT(cpu >= 0 && cpu < pmc_cpu_max(), ("[core,%d] illegal CPU value %d", __LINE__, cpu)); KASSERT(ri >= 0 && ri < core_iaf_npmc, ("[core,%d] illegal row-index %d", __LINE__, ri)); KASSERT(core_pcpu[cpu]->pc_corepmcs[ri + core_iaf_ri].phw_pmc == NULL, ("[core,%d] PHW pmc non-NULL", __LINE__)); return (0); } static int iaf_start_pmc(int cpu, int ri) { struct pmc *pm; struct core_cpu *iafc; KASSERT(cpu >= 0 && cpu < pmc_cpu_max(), ("[core,%d] illegal CPU value %d", __LINE__, cpu)); KASSERT(ri >= 0 && ri < core_iaf_npmc, ("[core,%d] illegal row-index %d", __LINE__, ri)); PMCDBG(MDP,STA,1,"iaf-start cpu=%d ri=%d", cpu, ri); iafc = core_pcpu[cpu]; pm = iafc->pc_corepmcs[ri + core_iaf_ri].phw_pmc; iafc->pc_iafctrl |= pm->pm_md.pm_iaf.pm_iaf_ctrl; wrmsr(IAF_CTRL, iafc->pc_iafctrl); do { iafc->pc_resync = 0; iafc->pc_globalctrl |= (1ULL << (ri + IAF_OFFSET)); wrmsr(IA_GLOBAL_CTRL, iafc->pc_globalctrl); } while (iafc->pc_resync != 0); PMCDBG(MDP,STA,1,"iafctrl=%x(%x) globalctrl=%jx(%jx)", iafc->pc_iafctrl, (uint32_t) rdmsr(IAF_CTRL), iafc->pc_globalctrl, rdmsr(IA_GLOBAL_CTRL)); return (0); } static int iaf_stop_pmc(int cpu, int ri) { uint32_t fc; struct core_cpu *iafc; PMCDBG(MDP,STO,1,"iaf-stop cpu=%d ri=%d", cpu, ri); iafc = core_pcpu[cpu]; KASSERT(cpu >= 0 && cpu < pmc_cpu_max(), ("[core,%d] illegal CPU value %d", __LINE__, cpu)); KASSERT(ri >= 0 && ri < core_iaf_npmc, ("[core,%d] illegal row-index %d", __LINE__, ri)); fc = (IAF_MASK << (ri * 4)); if (core_cputype != PMC_CPU_INTEL_ATOM) fc &= ~IAF_ANY; iafc->pc_iafctrl &= ~fc; PMCDBG(MDP,STO,1,"iaf-stop iafctrl=%x", iafc->pc_iafctrl); wrmsr(IAF_CTRL, iafc->pc_iafctrl); do { iafc->pc_resync = 0; iafc->pc_globalctrl &= ~(1ULL << (ri + IAF_OFFSET)); wrmsr(IA_GLOBAL_CTRL, iafc->pc_globalctrl); } while (iafc->pc_resync != 0); PMCDBG(MDP,STO,1,"iafctrl=%x(%x) globalctrl=%jx(%jx)", iafc->pc_iafctrl, (uint32_t) rdmsr(IAF_CTRL), iafc->pc_globalctrl, rdmsr(IA_GLOBAL_CTRL)); return (0); } static int iaf_write_pmc(int cpu, int ri, pmc_value_t v) { struct core_cpu *cc; struct pmc *pm; KASSERT(cpu >= 0 && cpu < pmc_cpu_max(), ("[core,%d] illegal cpu value %d", __LINE__, cpu)); KASSERT(ri >= 0 && ri < core_iaf_npmc, ("[core,%d] illegal row-index %d", __LINE__, ri)); cc = core_pcpu[cpu]; pm = cc->pc_corepmcs[ri + core_iaf_ri].phw_pmc; KASSERT(pm, ("[core,%d] cpu %d ri %d pmc not configured", __LINE__, cpu, ri)); if (PMC_IS_SAMPLING_MODE(PMC_TO_MODE(pm))) v = iaf_reload_count_to_perfctr_value(v); wrmsr(IAF_CTRL, 0); /* Turn off fixed counters */ wrmsr(IAF_CTR0 + ri, v); wrmsr(IAF_CTRL, cc->pc_iafctrl); PMCDBG(MDP,WRI,1, "iaf-write cpu=%d ri=%d msr=0x%x v=%jx iafctrl=%jx " "pmc=%jx", cpu, ri, IAF_RI_TO_MSR(ri), v, (uintmax_t) rdmsr(IAF_CTRL), (uintmax_t) rdpmc(IAF_RI_TO_MSR(ri))); return (0); } static void iaf_initialize(struct pmc_mdep *md, int maxcpu, int npmc, int pmcwidth) { struct pmc_classdep *pcd; KASSERT(md != NULL, ("[iaf,%d] md is NULL", __LINE__)); PMCDBG(MDP,INI,1, "%s", "iaf-initialize"); pcd = &md->pmd_classdep[PMC_MDEP_CLASS_INDEX_IAF]; pcd->pcd_caps = IAF_PMC_CAPS; pcd->pcd_class = PMC_CLASS_IAF; pcd->pcd_num = npmc; pcd->pcd_ri = md->pmd_npmc; pcd->pcd_width = pmcwidth; pcd->pcd_allocate_pmc = iaf_allocate_pmc; pcd->pcd_config_pmc = iaf_config_pmc; pcd->pcd_describe = iaf_describe; pcd->pcd_get_config = iaf_get_config; pcd->pcd_get_msr = iaf_get_msr; pcd->pcd_pcpu_fini = core_pcpu_noop; pcd->pcd_pcpu_init = core_pcpu_noop; pcd->pcd_read_pmc = iaf_read_pmc; pcd->pcd_release_pmc = iaf_release_pmc; pcd->pcd_start_pmc = iaf_start_pmc; pcd->pcd_stop_pmc = iaf_stop_pmc; pcd->pcd_write_pmc = iaf_write_pmc; md->pmd_npmc += npmc; } /* * Intel programmable PMCs. */ /* * Event descriptor tables. * * For each event id, we track: * * 1. The CPUs that the event is valid for. * * 2. If the event uses a fixed UMASK, the value of the umask field. * If the event doesn't use a fixed UMASK, a mask of legal bits * to check against. */ struct iap_event_descr { enum pmc_event iap_ev; unsigned char iap_evcode; unsigned char iap_umask; unsigned char iap_flags; }; #define IAP_F_CC (1 << 0) /* CPU: Core */ #define IAP_F_CC2 (1 << 1) /* CPU: Core2 family */ #define IAP_F_CC2E (1 << 2) /* CPU: Core2 Extreme only */ #define IAP_F_CA (1 << 3) /* CPU: Atom */ #define IAP_F_I7 (1 << 4) /* CPU: Core i7 */ #define IAP_F_I7O (1 << 4) /* CPU: Core i7 (old) */ #define IAP_F_WM (1 << 5) /* CPU: Westmere */ #define IAP_F_FM (1 << 6) /* Fixed mask */ #define IAP_F_ALLCPUSCORE2 \ (IAP_F_CC | IAP_F_CC2 | IAP_F_CC2E | IAP_F_CA) /* Sub fields of UMASK that this event supports. */ #define IAP_M_CORE (1 << 0) /* Core specificity */ #define IAP_M_AGENT (1 << 1) /* Agent specificity */ #define IAP_M_PREFETCH (1 << 2) /* Prefetch */ #define IAP_M_MESI (1 << 3) /* MESI */ #define IAP_M_SNOOPRESPONSE (1 << 4) /* Snoop response */ #define IAP_M_SNOOPTYPE (1 << 5) /* Snoop type */ #define IAP_M_TRANSITION (1 << 6) /* Transition */ #define IAP_F_CORE (0x3 << 14) /* Core specificity */ #define IAP_F_AGENT (0x1 << 13) /* Agent specificity */ #define IAP_F_PREFETCH (0x3 << 12) /* Prefetch */ #define IAP_F_MESI (0xF << 8) /* MESI */ #define IAP_F_SNOOPRESPONSE (0xB << 8) /* Snoop response */ #define IAP_F_SNOOPTYPE (0x3 << 8) /* Snoop type */ #define IAP_F_TRANSITION (0x1 << 12) /* Transition */ #define IAP_PREFETCH_RESERVED (0x2 << 12) #define IAP_CORE_THIS (0x1 << 14) #define IAP_CORE_ALL (0x3 << 14) #define IAP_F_CMASK 0xFF000000 static struct iap_event_descr iap_events[] = { #undef IAPDESCR #define IAPDESCR(N,EV,UM,FLAGS) { \ .iap_ev = PMC_EV_IAP_EVENT_##N, \ .iap_evcode = (EV), \ .iap_umask = (UM), \ .iap_flags = (FLAGS) \ } IAPDESCR(02H_01H, 0x02, 0x01, IAP_F_FM | IAP_F_I7O), IAPDESCR(02H_81H, 0x02, 0x81, IAP_F_FM | IAP_F_CA), IAPDESCR(03H_00H, 0x03, 0x00, IAP_F_FM | IAP_F_CC), IAPDESCR(03H_01H, 0x03, 0x01, IAP_F_FM | IAP_F_I7O), IAPDESCR(03H_02H, 0x03, 0x02, IAP_F_FM | IAP_F_CA | IAP_F_CC2 | IAP_F_WM), IAPDESCR(03H_04H, 0x03, 0x04, IAP_F_FM | IAP_F_CA | IAP_F_CC2 | IAP_F_I7O), IAPDESCR(03H_08H, 0x03, 0x08, IAP_F_FM | IAP_F_CA | IAP_F_CC2), IAPDESCR(03H_10H, 0x03, 0x10, IAP_F_FM | IAP_F_CA | IAP_F_CC2), IAPDESCR(03H_20H, 0x03, 0x20, IAP_F_FM | IAP_F_CA | IAP_F_CC2), IAPDESCR(04H_00H, 0x04, 0x00, IAP_F_FM | IAP_F_CC), IAPDESCR(04H_01H, 0x04, 0x01, IAP_F_FM | IAP_F_CA | IAP_F_CC2 | IAP_F_I7O), IAPDESCR(04H_02H, 0x04, 0x02, IAP_F_FM | IAP_F_CA | IAP_F_CC2), IAPDESCR(04H_07H, 0x04, 0x07, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(04H_08H, 0x04, 0x08, IAP_F_FM | IAP_F_CA | IAP_F_CC2), IAPDESCR(05H_00H, 0x05, 0x00, IAP_F_FM | IAP_F_CC), IAPDESCR(05H_01H, 0x05, 0x01, IAP_F_FM | IAP_F_I7O), IAPDESCR(05H_02H, 0x05, 0x02, IAP_F_FM | IAP_F_I7O | IAP_F_WM), IAPDESCR(05H_03H, 0x05, 0x03, IAP_F_FM | IAP_F_I7O), IAPDESCR(06H_00H, 0x06, 0x00, IAP_F_FM | IAP_F_CC | IAP_F_CC2 | IAP_F_CC2E | IAP_F_CA), IAPDESCR(06H_01H, 0x06, 0x01, IAP_F_FM | IAP_F_I7O), IAPDESCR(06H_02H, 0x06, 0x02, IAP_F_FM | IAP_F_I7O), IAPDESCR(06H_04H, 0x06, 0x04, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(06H_08H, 0x06, 0x08, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(06H_0FH, 0x06, 0x0F, IAP_F_FM | IAP_F_I7O), IAPDESCR(07H_00H, 0x07, 0x00, IAP_F_FM | IAP_F_CC | IAP_F_CC2), - IAPDESCR(07H_01H, 0x07, 0x01, IAP_F_FM | IAP_F_ALLCPUSCORE2 | IAP_F_WM), + IAPDESCR(07H_01H, 0x07, 0x01, IAP_F_FM | IAP_F_ALLCPUSCORE2 | IAP_F_I7 | IAP_F_WM), IAPDESCR(07H_02H, 0x07, 0x02, IAP_F_FM | IAP_F_ALLCPUSCORE2), IAPDESCR(07H_03H, 0x07, 0x03, IAP_F_FM | IAP_F_ALLCPUSCORE2), IAPDESCR(07H_06H, 0x07, 0x06, IAP_F_FM | IAP_F_CA), IAPDESCR(07H_08H, 0x07, 0x08, IAP_F_FM | IAP_F_CA), IAPDESCR(08H_01H, 0x08, 0x01, IAP_F_FM | IAP_F_CA | IAP_F_CC2 | IAP_F_I7 | IAP_F_WM), IAPDESCR(08H_02H, 0x08, 0x02, IAP_F_FM | IAP_F_CA | IAP_F_CC2 | IAP_F_I7 | IAP_F_WM), IAPDESCR(08H_04H, 0x08, 0x04, IAP_F_FM | IAP_F_CA | IAP_F_CC2 | IAP_F_WM), IAPDESCR(08H_05H, 0x08, 0x05, IAP_F_FM | IAP_F_CA), IAPDESCR(08H_06H, 0x08, 0x06, IAP_F_FM | IAP_F_CA), IAPDESCR(08H_07H, 0x08, 0x07, IAP_F_FM | IAP_F_CA), IAPDESCR(08H_08H, 0x08, 0x08, IAP_F_FM | IAP_F_CA | IAP_F_CC2), IAPDESCR(08H_09H, 0x08, 0x09, IAP_F_FM | IAP_F_CA), IAPDESCR(08H_10H, 0x08, 0x10, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(08H_20H, 0x08, 0x20, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(08H_40H, 0x08, 0x40, IAP_F_FM | IAP_F_I7), IAPDESCR(08H_80H, 0x08, 0x80, IAP_F_FM | IAP_F_I7), IAPDESCR(09H_01H, 0x09, 0x01, IAP_F_FM | IAP_F_CA | IAP_F_CC2 | IAP_F_I7O), IAPDESCR(09H_02H, 0x09, 0x02, IAP_F_FM | IAP_F_CA | IAP_F_CC2 | IAP_F_I7O), IAPDESCR(09H_04H, 0x09, 0x04, IAP_F_FM | IAP_F_I7O), IAPDESCR(09H_08H, 0x09, 0x08, IAP_F_FM | IAP_F_I7O), IAPDESCR(0BH_01H, 0x0B, 0x01, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(0BH_02H, 0x0B, 0x02, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(0BH_10H, 0x0B, 0x10, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(0CH_01H, 0x0C, 0x01, IAP_F_FM | IAP_F_CC2 | IAP_F_I7 | IAP_F_WM), IAPDESCR(0CH_02H, 0x0C, 0x02, IAP_F_FM | IAP_F_CC2), IAPDESCR(0CH_03H, 0x0C, 0x03, IAP_F_FM | IAP_F_CA), IAPDESCR(0EH_01H, 0x0E, 0x01, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(0EH_02H, 0x0E, 0x02, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(0FH_01H, 0x0F, 0x01, IAP_F_FM | IAP_F_I7), IAPDESCR(0FH_02H, 0x0F, 0x02, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(0FH_08H, 0x0F, 0x08, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(0FH_10H, 0x0F, 0x10, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(0FH_20H, 0x0F, 0x20, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(0FH_80H, 0x0F, 0x80, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(10H_00H, 0x10, 0x00, IAP_F_FM | IAP_F_ALLCPUSCORE2), IAPDESCR(10H_01H, 0x10, 0x01, IAP_F_FM | IAP_F_CA | IAP_F_I7 | IAP_F_WM), IAPDESCR(10H_02H, 0x10, 0x02, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(10H_04H, 0x10, 0x04, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(10H_08H, 0x10, 0x08, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(10H_10H, 0x10, 0x10, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(10H_20H, 0x10, 0x20, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(10H_40H, 0x10, 0x40, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(10H_80H, 0x10, 0x80, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(10H_81H, 0x10, 0x81, IAP_F_FM | IAP_F_CA), IAPDESCR(11H_00H, 0x11, 0x00, IAP_F_FM | IAP_F_CC | IAP_F_CC2), IAPDESCR(11H_01H, 0x11, 0x01, IAP_F_FM | IAP_F_CA), IAPDESCR(11H_81H, 0x11, 0x81, IAP_F_FM | IAP_F_CA), IAPDESCR(12H_00H, 0x12, 0x00, IAP_F_FM | IAP_F_ALLCPUSCORE2), IAPDESCR(12H_01H, 0x12, 0x01, IAP_F_FM | IAP_F_CA | IAP_F_I7 | IAP_F_WM), IAPDESCR(12H_02H, 0x12, 0x02, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(12H_04H, 0x12, 0x04, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(12H_08H, 0x12, 0x08, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(12H_10H, 0x12, 0x10, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(12H_20H, 0x12, 0x20, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(12H_40H, 0x12, 0x40, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(12H_81H, 0x12, 0x81, IAP_F_FM | IAP_F_CA), IAPDESCR(13H_00H, 0x13, 0x00, IAP_F_FM | IAP_F_ALLCPUSCORE2), IAPDESCR(13H_01H, 0x13, 0x01, IAP_F_FM | IAP_F_CA | IAP_F_I7 | IAP_F_WM), IAPDESCR(13H_02H, 0x13, 0x02, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(13H_04H, 0x13, 0x04, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(13H_07H, 0x13, 0x07, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(13H_81H, 0x13, 0x81, IAP_F_FM | IAP_F_CA), IAPDESCR(14H_00H, 0x14, 0x00, IAP_F_FM | IAP_F_CC | IAP_F_CC2), IAPDESCR(14H_01H, 0x14, 0x01, IAP_F_FM | IAP_F_CA | IAP_F_I7 | IAP_F_WM), IAPDESCR(14H_02H, 0x14, 0x02, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(17H_01H, 0x17, 0x01, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(18H_00H, 0x18, 0x00, IAP_F_FM | IAP_F_CA | IAP_F_CC2), IAPDESCR(18H_01H, 0x18, 0x01, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(19H_00H, 0x19, 0x00, IAP_F_FM | IAP_F_CA | IAP_F_CC2), IAPDESCR(19H_01H, 0x19, 0x01, IAP_F_FM | IAP_F_CA | IAP_F_CC2 | IAP_F_I7 | IAP_F_WM), IAPDESCR(19H_02H, 0x19, 0x02, IAP_F_FM | IAP_F_CA | IAP_F_CC2), IAPDESCR(1DH_01H, 0x1D, 0x01, IAP_F_FM | IAP_F_I7O), IAPDESCR(1DH_02H, 0x1D, 0x02, IAP_F_FM | IAP_F_I7O), IAPDESCR(1DH_04H, 0x1D, 0x04, IAP_F_FM | IAP_F_I7O), IAPDESCR(1EH_01H, 0x1E, 0x01, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(20H_01H, 0x20, 0x01, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(21H, 0x21, IAP_M_CORE, IAP_F_ALLCPUSCORE2), IAPDESCR(22H, 0x22, IAP_M_CORE, IAP_F_CC2), IAPDESCR(23H, 0x23, IAP_M_CORE, IAP_F_ALLCPUSCORE2), IAPDESCR(24H, 0x24, IAP_M_CORE | IAP_M_PREFETCH, IAP_F_ALLCPUSCORE2), IAPDESCR(24H_01H, 0x24, 0x01, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(24H_02H, 0x24, 0x02, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(24H_03H, 0x24, 0x03, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(24H_04H, 0x24, 0x04, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(24H_08H, 0x24, 0x08, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(24H_0CH, 0x24, 0x0C, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(24H_10H, 0x24, 0x10, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(24H_20H, 0x24, 0x20, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(24H_30H, 0x24, 0x30, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(24H_40H, 0x24, 0x40, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(24H_80H, 0x24, 0x80, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(24H_C0H, 0x24, 0xC0, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(24H_AAH, 0x24, 0xAA, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(24H_FFH, 0x24, 0xFF, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(25H, 0x25, IAP_M_CORE, IAP_F_ALLCPUSCORE2), IAPDESCR(26H, 0x26, IAP_M_CORE | IAP_M_PREFETCH, IAP_F_ALLCPUSCORE2), IAPDESCR(26H_01H, 0x26, 0x01, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(26H_02H, 0x26, 0x02, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(26H_04H, 0x26, 0x04, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(26H_08H, 0x26, 0x08, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(26H_0FH, 0x26, 0x0F, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(26H_10H, 0x26, 0x10, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(26H_20H, 0x26, 0x20, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(26H_40H, 0x26, 0x40, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(26H_80H, 0x26, 0x80, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(26H_F0H, 0x26, 0xF0, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(26H_FFH, 0x26, 0xFF, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(27H, 0x27, IAP_M_CORE | IAP_M_PREFETCH, IAP_F_ALLCPUSCORE2), IAPDESCR(27H_01H, 0x27, 0x01, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(27H_02H, 0x27, 0x02, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(27H_04H, 0x27, 0x04, IAP_F_FM | IAP_F_I7O), IAPDESCR(27H_08H, 0x27, 0x08, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(27H_0EH, 0x27, 0x0E, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(27H_0FH, 0x27, 0x0F, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(27H_10H, 0x27, 0x10, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(27H_20H, 0x27, 0x20, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(27H_40H, 0x27, 0x40, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(27H_80H, 0x27, 0x80, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(27H_E0H, 0x27, 0xE0, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(27H_F0H, 0x27, 0xF0, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(28H, 0x28, IAP_M_CORE | IAP_M_MESI, IAP_F_ALLCPUSCORE2), IAPDESCR(28H_01H, 0x28, 0x01, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(28H_02H, 0x28, 0x02, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(28H_04H, 0x28, 0x04, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(28H_08H, 0x28, 0x08, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(28H_0FH, 0x28, 0x0F, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(29H, 0x29, IAP_M_CORE | IAP_M_MESI, IAP_F_CC), IAPDESCR(29H, 0x29, IAP_M_CORE | IAP_M_MESI | IAP_M_PREFETCH, IAP_F_CA | IAP_F_CC2), IAPDESCR(2AH, 0x2A, IAP_M_CORE | IAP_M_MESI, IAP_F_ALLCPUSCORE2), IAPDESCR(2BH, 0x2B, IAP_M_CORE | IAP_M_MESI, IAP_F_CA | IAP_F_CC2), IAPDESCR(2EH, 0x2E, IAP_M_CORE | IAP_M_MESI | IAP_M_PREFETCH, IAP_F_ALLCPUSCORE2), IAPDESCR(2EH_01H, 0x2E, 0x01, IAP_F_FM | IAP_F_WM), IAPDESCR(2EH_02H, 0x2E, 0x02, IAP_F_FM | IAP_F_WM), IAPDESCR(2EH_41H, 0x2E, 0x41, IAP_F_FM | IAP_F_ALLCPUSCORE2 | IAP_F_I7), IAPDESCR(2EH_4FH, 0x2E, 0x4F, IAP_F_FM | IAP_F_ALLCPUSCORE2 | IAP_F_I7), IAPDESCR(30H, 0x30, IAP_M_CORE | IAP_M_MESI | IAP_M_PREFETCH, IAP_F_ALLCPUSCORE2), IAPDESCR(32H, 0x32, IAP_M_CORE | IAP_M_MESI | IAP_M_PREFETCH, IAP_F_CC), IAPDESCR(32H, 0x32, IAP_M_CORE, IAP_F_CA | IAP_F_CC2), IAPDESCR(3AH, 0x3A, IAP_M_TRANSITION, IAP_F_CC), IAPDESCR(3AH_00H, 0x3A, 0x00, IAP_F_FM | IAP_F_CA | IAP_F_CC2), IAPDESCR(3BH_C0H, 0x3B, 0xC0, IAP_F_FM | IAP_F_ALLCPUSCORE2), IAPDESCR(3CH_00H, 0x3C, 0x00, IAP_F_FM | IAP_F_ALLCPUSCORE2 | IAP_F_I7 | IAP_F_WM), IAPDESCR(3CH_01H, 0x3C, 0x01, IAP_F_FM | IAP_F_ALLCPUSCORE2 | IAP_F_I7 | IAP_F_WM), IAPDESCR(3CH_02H, 0x3C, 0x02, IAP_F_FM | IAP_F_ALLCPUSCORE2), IAPDESCR(3DH_01H, 0x3D, 0x01, IAP_F_FM | IAP_F_I7O), IAPDESCR(40H, 0x40, IAP_M_MESI, IAP_F_CC | IAP_F_CC2), IAPDESCR(40H_01H, 0x40, 0x01, IAP_F_FM | IAP_F_I7), IAPDESCR(40H_02H, 0x40, 0x02, IAP_F_FM | IAP_F_I7), IAPDESCR(40H_04H, 0x40, 0x04, IAP_F_FM | IAP_F_I7), IAPDESCR(40H_08H, 0x40, 0x08, IAP_F_FM | IAP_F_I7), IAPDESCR(40H_0FH, 0x40, 0x0F, IAP_F_FM | IAP_F_I7), IAPDESCR(40H_21H, 0x40, 0x21, IAP_F_FM | IAP_F_CA), IAPDESCR(41H, 0x41, IAP_M_MESI, IAP_F_CC | IAP_F_CC2), IAPDESCR(41H_01H, 0x41, 0x01, IAP_F_FM | IAP_F_I7O), IAPDESCR(41H_02H, 0x41, 0x02, IAP_F_FM | IAP_F_I7), IAPDESCR(41H_04H, 0x41, 0x04, IAP_F_FM | IAP_F_I7), IAPDESCR(41H_08H, 0x41, 0x08, IAP_F_FM | IAP_F_I7), IAPDESCR(41H_0FH, 0x41, 0x0F, IAP_F_FM | IAP_F_I7O), IAPDESCR(41H_22H, 0x41, 0x22, IAP_F_FM | IAP_F_CA), IAPDESCR(42H, 0x42, IAP_M_MESI, IAP_F_ALLCPUSCORE2), IAPDESCR(42H_01H, 0x42, 0x01, IAP_F_FM | IAP_F_I7), IAPDESCR(42H_02H, 0x42, 0x02, IAP_F_FM | IAP_F_I7), IAPDESCR(42H_04H, 0x42, 0x04, IAP_F_FM | IAP_F_I7), IAPDESCR(42H_08H, 0x42, 0x08, IAP_F_FM | IAP_F_I7), IAPDESCR(42H_10H, 0x42, 0x10, IAP_F_FM | IAP_F_CA | IAP_F_CC2), IAPDESCR(43H_01H, 0x43, 0x01, IAP_F_FM | IAP_F_ALLCPUSCORE2 | IAP_F_I7), IAPDESCR(43H_02H, 0x43, 0x02, IAP_F_FM | IAP_F_CA | IAP_F_CC2 | IAP_F_I7), IAPDESCR(44H_02H, 0x44, 0x02, IAP_F_FM | IAP_F_CC), IAPDESCR(45H_0FH, 0x45, 0x0F, IAP_F_FM | IAP_F_ALLCPUSCORE2), IAPDESCR(46H_00H, 0x46, 0x00, IAP_F_FM | IAP_F_ALLCPUSCORE2), IAPDESCR(47H_00H, 0x47, 0x00, IAP_F_FM | IAP_F_ALLCPUSCORE2), IAPDESCR(48H_00H, 0x48, 0x00, IAP_F_FM | IAP_F_ALLCPUSCORE2), IAPDESCR(48H_02H, 0x48, 0x02, IAP_F_FM | IAP_F_I7), IAPDESCR(49H_00H, 0x49, 0x00, IAP_F_FM | IAP_F_CC), IAPDESCR(49H_01H, 0x49, 0x01, IAP_F_FM | IAP_F_CA | IAP_F_CC2 | IAP_F_I7 | IAP_F_WM), IAPDESCR(49H_02H, 0x49, 0x02, IAP_F_FM | IAP_F_CA | IAP_F_CC2 | IAP_F_I7 | IAP_F_WM), IAPDESCR(49H_04H, 0x49, 0x04, IAP_F_FM | IAP_F_WM), IAPDESCR(49H_10H, 0x49, 0x10, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(49H_20H, 0x49, 0x20, IAP_F_FM | IAP_F_I7O), IAPDESCR(49H_40H, 0x49, 0x40, IAP_F_FM | IAP_F_I7O), IAPDESCR(49H_80H, 0x49, 0x80, IAP_F_FM | IAP_F_WM | IAP_F_I7O), IAPDESCR(4BH_00H, 0x4B, 0x00, IAP_F_FM | IAP_F_ALLCPUSCORE2), IAPDESCR(4BH_01H, 0x4B, 0x01, IAP_F_FM | IAP_F_ALLCPUSCORE2 | IAP_F_I7O), IAPDESCR(4BH_02H, 0x4B, 0x02, IAP_F_FM | IAP_F_ALLCPUSCORE2), IAPDESCR(4BH_03H, 0x4B, 0x03, IAP_F_FM | IAP_F_CC), IAPDESCR(4BH_08H, 0x4B, 0x08, IAP_F_FM | IAP_F_I7O), IAPDESCR(4CH_00H, 0x4C, 0x00, IAP_F_FM | IAP_F_CA | IAP_F_CC2), IAPDESCR(4CH_01H, 0x4C, 0x01, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(4DH_01H, 0x4D, 0x01, IAP_F_FM | IAP_F_I7O), IAPDESCR(4EH_01H, 0x4E, 0x01, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(4EH_02H, 0x4E, 0x02, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(4EH_04H, 0x4E, 0x04, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(4EH_10H, 0x4E, 0x10, IAP_F_FM | IAP_F_CA | IAP_F_CC2), IAPDESCR(4FH_00H, 0x4F, 0x00, IAP_F_FM | IAP_F_CC), IAPDESCR(4FH_02H, 0x4F, 0x02, IAP_F_FM | IAP_F_I7O), IAPDESCR(4FH_04H, 0x4F, 0x04, IAP_F_FM | IAP_F_I7O), IAPDESCR(4FH_08H, 0x4F, 0x08, IAP_F_FM | IAP_F_I7O), IAPDESCR(4FH_10H, 0x4F, 0x10, IAP_F_FM | IAP_F_WM), IAPDESCR(51H_01H, 0x51, 0x01, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(51H_02H, 0x51, 0x02, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(51H_04H, 0x51, 0x04, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(51H_08H, 0x51, 0x08, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(52H_01H, 0x52, 0x01, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(53H_01H, 0x53, 0x01, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(60H, 0x60, IAP_M_AGENT | IAP_M_CORE, IAP_F_ALLCPUSCORE2), IAPDESCR(60H_01H, 0x60, 0x01, IAP_F_FM | IAP_F_WM | IAP_F_I7O), IAPDESCR(60H_02H, 0x60, 0x02, IAP_F_FM | IAP_F_WM | IAP_F_I7O), IAPDESCR(60H_04H, 0x60, 0x04, IAP_F_FM | IAP_F_WM | IAP_F_I7O), IAPDESCR(60H_08H, 0x60, 0x08, IAP_F_FM | IAP_F_WM | IAP_F_I7O), IAPDESCR(61H, 0x61, IAP_M_AGENT, IAP_F_CA | IAP_F_CC2), IAPDESCR(61H_00H, 0x61, 0x00, IAP_F_FM | IAP_F_CC), IAPDESCR(62H, 0x62, IAP_M_AGENT, IAP_F_ALLCPUSCORE2), IAPDESCR(62H_00H, 0x62, 0x00, IAP_F_FM | IAP_F_CC), IAPDESCR(63H, 0x63, IAP_M_AGENT | IAP_M_CORE, IAP_F_CA | IAP_F_CC2), IAPDESCR(63H, 0x63, IAP_M_CORE, IAP_F_CC), IAPDESCR(63H_01H, 0x63, 0x01, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(63H_02H, 0x63, 0x02, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(64H, 0x64, IAP_M_CORE, IAP_F_CA | IAP_F_CC2), IAPDESCR(64H_40H, 0x64, 0x40, IAP_F_FM | IAP_F_CC), IAPDESCR(65H, 0x65, IAP_M_AGENT | IAP_M_CORE, IAP_F_CA | IAP_F_CC2), IAPDESCR(65H, 0x65, IAP_M_CORE, IAP_F_CC), IAPDESCR(66H, 0x66, IAP_M_AGENT | IAP_M_CORE, IAP_F_ALLCPUSCORE2), IAPDESCR(67H, 0x67, IAP_M_AGENT | IAP_M_CORE, IAP_F_CA | IAP_F_CC2), IAPDESCR(67H, 0x67, IAP_M_AGENT, IAP_F_CC), IAPDESCR(68H, 0x68, IAP_M_AGENT | IAP_M_CORE, IAP_F_ALLCPUSCORE2), IAPDESCR(69H, 0x69, IAP_M_AGENT | IAP_M_CORE, IAP_F_ALLCPUSCORE2), IAPDESCR(6AH, 0x6A, IAP_M_AGENT | IAP_M_CORE, IAP_F_ALLCPUSCORE2), IAPDESCR(6BH, 0x6B, IAP_M_AGENT | IAP_M_CORE, IAP_F_ALLCPUSCORE2), IAPDESCR(6CH, 0x6C, IAP_M_AGENT | IAP_M_CORE, IAP_F_ALLCPUSCORE2), IAPDESCR(6CH_01H, 0x6C, 0x01, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(6DH, 0x6D, IAP_M_AGENT | IAP_M_CORE, IAP_F_CA | IAP_F_CC2), IAPDESCR(6DH, 0x6D, IAP_M_CORE, IAP_F_CC), IAPDESCR(6EH, 0x6E, IAP_M_AGENT | IAP_M_CORE, IAP_F_CA | IAP_F_CC2), IAPDESCR(6EH, 0x6E, IAP_M_CORE, IAP_F_CC), IAPDESCR(6FH, 0x6F, IAP_M_AGENT | IAP_M_CORE, IAP_F_CA | IAP_F_CC2), IAPDESCR(6FH, 0x6F, IAP_M_CORE, IAP_F_CC), IAPDESCR(70H, 0x70, IAP_M_AGENT | IAP_M_CORE, IAP_F_CA | IAP_F_CC2), IAPDESCR(70H, 0x70, IAP_M_CORE, IAP_F_CC), IAPDESCR(77H, 0x77, IAP_M_AGENT | IAP_M_SNOOPRESPONSE, IAP_F_CA | IAP_F_CC2), IAPDESCR(77H, 0x77, IAP_M_AGENT | IAP_M_MESI, IAP_F_CC), IAPDESCR(78H, 0x78, IAP_M_CORE, IAP_F_CC), IAPDESCR(78H, 0x78, IAP_M_CORE | IAP_M_SNOOPTYPE, IAP_F_CA | IAP_F_CC2), IAPDESCR(7AH, 0x7A, IAP_M_AGENT, IAP_F_CA | IAP_F_CC2), IAPDESCR(7BH, 0x7B, IAP_M_AGENT, IAP_F_CA | IAP_F_CC2), IAPDESCR(7DH, 0x7D, IAP_M_CORE, IAP_F_ALLCPUSCORE2), IAPDESCR(7EH, 0x7E, IAP_M_AGENT | IAP_M_CORE, IAP_F_CA | IAP_F_CC2), IAPDESCR(7EH_00H, 0x7E, 0x00, IAP_F_FM | IAP_F_CC), IAPDESCR(7FH, 0x7F, IAP_M_CORE, IAP_F_CA | IAP_F_CC2), IAPDESCR(80H_00H, 0x80, 0x00, IAP_F_FM | IAP_F_ALLCPUSCORE2), IAPDESCR(80H_01H, 0x80, 0x01, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(80H_02H, 0x80, 0x02, IAP_F_FM | IAP_F_CA | IAP_F_I7 | IAP_F_WM), IAPDESCR(80H_03H, 0x80, 0x03, IAP_F_FM | IAP_F_CA | IAP_F_I7 | IAP_F_WM), IAPDESCR(80H_04H, 0x80, 0x04, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(81H_00H, 0x81, 0x00, IAP_F_FM | IAP_F_ALLCPUSCORE2), IAPDESCR(81H_01H, 0x81, 0x01, IAP_F_FM | IAP_F_I7O), IAPDESCR(81H_02H, 0x81, 0x02, IAP_F_FM | IAP_F_I7O), IAPDESCR(82H_01H, 0x82, 0x01, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(82H_02H, 0x82, 0x02, IAP_F_FM | IAP_F_CA | IAP_F_CC2), IAPDESCR(82H_04H, 0x82, 0x04, IAP_F_FM | IAP_F_CA), IAPDESCR(82H_10H, 0x82, 0x10, IAP_F_FM | IAP_F_CA | IAP_F_CC2), IAPDESCR(82H_12H, 0x82, 0x12, IAP_F_FM | IAP_F_CC2), IAPDESCR(82H_40H, 0x82, 0x40, IAP_F_FM | IAP_F_CC2), IAPDESCR(83H_01H, 0x83, 0x01, IAP_F_FM | IAP_F_I7O), IAPDESCR(83H_02H, 0x83, 0x02, IAP_F_FM | IAP_F_CA | IAP_F_CC2), IAPDESCR(85H_00H, 0x85, 0x00, IAP_F_FM | IAP_F_CC), IAPDESCR(85H_01H, 0x85, 0x01, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(85H_02H, 0x85, 0x02, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(85H_04H, 0x85, 0x04, IAP_F_FM | IAP_F_WM | IAP_F_I7O), IAPDESCR(85H_10H, 0x85, 0x10, IAP_F_FM | IAP_F_I7O), IAPDESCR(85H_20H, 0x85, 0x20, IAP_F_FM | IAP_F_I7O), IAPDESCR(85H_40H, 0x85, 0x40, IAP_F_FM | IAP_F_I7O), IAPDESCR(85H_80H, 0x85, 0x80, IAP_F_FM | IAP_F_WM | IAP_F_I7O), IAPDESCR(86H_00H, 0x86, 0x00, IAP_F_FM | IAP_F_ALLCPUSCORE2), IAPDESCR(87H_00H, 0x87, 0x00, IAP_F_FM | IAP_F_ALLCPUSCORE2), IAPDESCR(87H_01H, 0x87, 0x01, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(87H_02H, 0x87, 0x02, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(87H_04H, 0x87, 0x04, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(87H_08H, 0x87, 0x08, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(87H_0FH, 0x87, 0x0F, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(88H_00H, 0x88, 0x00, IAP_F_FM | IAP_F_ALLCPUSCORE2), IAPDESCR(88H_01H, 0x88, 0x01, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(88H_02H, 0x88, 0x02, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(88H_04H, 0x88, 0x04, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(88H_07H, 0x88, 0x07, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(88H_08H, 0x88, 0x08, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(88H_10H, 0x88, 0x10, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(88H_20H, 0x88, 0x20, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(88H_30H, 0x88, 0x30, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(88H_40H, 0x88, 0x40, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(88H_7FH, 0x88, 0x7F, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(89H_00H, 0x89, 0x00, IAP_F_FM | IAP_F_ALLCPUSCORE2), IAPDESCR(89H_01H, 0x89, 0x01, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(89H_02H, 0x89, 0x02, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(89H_04H, 0x89, 0x04, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(89H_07H, 0x89, 0x07, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(89H_08H, 0x89, 0x08, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(89H_10H, 0x89, 0x10, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(89H_20H, 0x89, 0x20, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(89H_30H, 0x89, 0x30, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(89H_40H, 0x89, 0x40, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(89H_7FH, 0x89, 0x7F, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(8AH_00H, 0x8A, 0x00, IAP_F_FM | IAP_F_ALLCPUSCORE2), IAPDESCR(8BH_00H, 0x8B, 0x00, IAP_F_FM | IAP_F_ALLCPUSCORE2), IAPDESCR(8CH_00H, 0x8C, 0x00, IAP_F_FM | IAP_F_ALLCPUSCORE2), IAPDESCR(8DH_00H, 0x8D, 0x00, IAP_F_FM | IAP_F_ALLCPUSCORE2), IAPDESCR(8EH_00H, 0x8E, 0x00, IAP_F_FM | IAP_F_ALLCPUSCORE2), IAPDESCR(8FH_00H, 0x8F, 0x00, IAP_F_FM | IAP_F_ALLCPUSCORE2), IAPDESCR(90H_00H, 0x90, 0x00, IAP_F_FM | IAP_F_ALLCPUSCORE2), IAPDESCR(91H_00H, 0x91, 0x00, IAP_F_FM | IAP_F_ALLCPUSCORE2), IAPDESCR(92H_00H, 0x92, 0x00, IAP_F_FM | IAP_F_ALLCPUSCORE2), IAPDESCR(93H_00H, 0x93, 0x00, IAP_F_FM | IAP_F_ALLCPUSCORE2), IAPDESCR(94H_00H, 0x94, 0x00, IAP_F_FM | IAP_F_ALLCPUSCORE2), IAPDESCR(97H_00H, 0x97, 0x00, IAP_F_FM | IAP_F_CA | IAP_F_CC2), IAPDESCR(98H_00H, 0x98, 0x00, IAP_F_FM | IAP_F_CA | IAP_F_CC2), IAPDESCR(A0H_00H, 0xA0, 0x00, IAP_F_FM | IAP_F_CA | IAP_F_CC2), IAPDESCR(A1H_01H, 0xA1, 0x01, IAP_F_FM | IAP_F_CA | IAP_F_CC2), IAPDESCR(A1H_02H, 0xA1, 0x02, IAP_F_FM | IAP_F_CA | IAP_F_CC2), IAPDESCR(A1H_04H, 0xA1, 0x04, IAP_F_FM | IAP_F_CA | IAP_F_CC2), IAPDESCR(A1H_08H, 0xA1, 0x08, IAP_F_FM | IAP_F_CA | IAP_F_CC2), IAPDESCR(A1H_10H, 0xA1, 0x10, IAP_F_FM | IAP_F_CA | IAP_F_CC2), IAPDESCR(A1H_20H, 0xA1, 0x20, IAP_F_FM | IAP_F_CA | IAP_F_CC2), IAPDESCR(A2H_00H, 0xA2, 0x00, IAP_F_FM | IAP_F_CC), IAPDESCR(A2H_01H, 0xA2, 0x01, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(A2H_02H, 0xA2, 0x02, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(A2H_04H, 0xA2, 0x04, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(A2H_08H, 0xA2, 0x08, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(A2H_10H, 0xA2, 0x10, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(A2H_20H, 0xA2, 0x20, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(A2H_40H, 0xA2, 0x40, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(A2H_80H, 0xA2, 0x80, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(A6H_01H, 0xA6, 0x01, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(A7H_01H, 0xA7, 0x01, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(A8H_01H, 0xA8, 0x01, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(AAH_01H, 0xAA, 0x01, IAP_F_FM | IAP_F_CC2), IAPDESCR(AAH_02H, 0xAA, 0x02, IAP_F_FM | IAP_F_CA), IAPDESCR(AAH_03H, 0xAA, 0x03, IAP_F_FM | IAP_F_CA), IAPDESCR(AAH_08H, 0xAA, 0x08, IAP_F_FM | IAP_F_CC2), IAPDESCR(ABH_01H, 0xAB, 0x01, IAP_F_FM | IAP_F_CA | IAP_F_CC2), IAPDESCR(ABH_02H, 0xAB, 0x02, IAP_F_FM | IAP_F_CA | IAP_F_CC2), IAPDESCR(AEH_01H, 0xAE, 0x01, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(B0H_00H, 0xB0, 0x00, IAP_F_FM | IAP_F_ALLCPUSCORE2), IAPDESCR(B0H_01H, 0xB0, 0x01, IAP_F_FM | IAP_F_WM | IAP_F_I7O), IAPDESCR(B0H_02H, 0xB0, 0x02, IAP_F_FM | IAP_F_WM | IAP_F_I7O), IAPDESCR(B0H_04H, 0xB0, 0x04, IAP_F_FM | IAP_F_WM | IAP_F_I7O), IAPDESCR(B0H_08H, 0xB0, 0x08, IAP_F_FM | IAP_F_WM | IAP_F_I7O), + IAPDESCR(B0H_10H, 0xB0, 0x10, IAP_F_FM | IAP_F_WM | IAP_F_I7O), IAPDESCR(B0H_20H, 0xB0, 0x20, IAP_F_FM | IAP_F_I7O), IAPDESCR(B0H_40H, 0xB0, 0x40, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(B0H_80H, 0xB0, 0x80, IAP_F_FM | IAP_F_CA | IAP_F_WM | IAP_F_I7O), IAPDESCR(B1H_00H, 0xB1, 0x00, IAP_F_FM | IAP_F_ALLCPUSCORE2), IAPDESCR(B1H_01H, 0xB1, 0x01, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(B1H_02H, 0xB1, 0x02, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(B1H_04H, 0xB1, 0x04, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(B1H_08H, 0xB1, 0x08, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(B1H_10H, 0xB1, 0x10, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(B1H_1FH, 0xB1, 0x1F, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(B1H_20H, 0xB1, 0x20, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(B1H_3FH, 0xB1, 0x3F, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(B1H_40H, 0xB1, 0x40, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(B1H_80H, 0xB1, 0x80, IAP_F_FM | IAP_F_CA | IAP_F_I7 | IAP_F_WM), IAPDESCR(B2H_01H, 0xB2, 0x01, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(B3H_01H, 0xB3, 0x01, IAP_F_FM | IAP_F_ALLCPUSCORE2 | IAP_F_WM | IAP_F_I7O), IAPDESCR(B3H_02H, 0xB3, 0x02, IAP_F_FM | IAP_F_ALLCPUSCORE2 | IAP_F_WM | IAP_F_I7O), IAPDESCR(B3H_04H, 0xB3, 0x04, IAP_F_FM | IAP_F_ALLCPUSCORE2 | IAP_F_WM | IAP_F_I7O), IAPDESCR(B3H_08H, 0xB3, 0x08, IAP_F_FM | IAP_F_ALLCPUSCORE2), IAPDESCR(B3H_10H, 0xB3, 0x10, IAP_F_FM | IAP_F_ALLCPUSCORE2), IAPDESCR(B3H_20H, 0xB3, 0x20, IAP_F_FM | IAP_F_ALLCPUSCORE2), IAPDESCR(B3H_81H, 0xB3, 0x81, IAP_F_FM | IAP_F_CA), IAPDESCR(B3H_82H, 0xB3, 0x82, IAP_F_FM | IAP_F_CA), IAPDESCR(B3H_84H, 0xB3, 0x84, IAP_F_FM | IAP_F_CA), IAPDESCR(B3H_88H, 0xB3, 0x88, IAP_F_FM | IAP_F_CA), IAPDESCR(B3H_90H, 0xB3, 0x90, IAP_F_FM | IAP_F_CA), IAPDESCR(B3H_A0H, 0xB3, 0xA0, IAP_F_FM | IAP_F_CA), IAPDESCR(B4H_01H, 0xB4, 0x01, IAP_F_FM | IAP_F_WM), IAPDESCR(B4H_02H, 0xB4, 0x02, IAP_F_FM | IAP_F_WM), IAPDESCR(B4H_04H, 0xB4, 0x04, IAP_F_FM | IAP_F_WM), IAPDESCR(B7H_01H, 0xB7, 0x01, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(B8H_01H, 0xB8, 0x01, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(B8H_02H, 0xB8, 0x02, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(B8H_04H, 0xB8, 0x04, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(BAH_01H, 0xBA, 0x01, IAP_F_FM | IAP_F_I7O), IAPDESCR(BAH_02H, 0xBA, 0x02, IAP_F_FM | IAP_F_I7O), IAPDESCR(BBH_01H, 0xBB, 0x01, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(C0H_00H, 0xC0, 0x00, IAP_F_FM | IAP_F_ALLCPUSCORE2), IAPDESCR(C0H_01H, 0xC0, 0x01, IAP_F_FM | IAP_F_CA | IAP_F_CC2 | IAP_F_I7 | IAP_F_WM), IAPDESCR(C0H_02H, 0xC0, 0x02, IAP_F_FM | IAP_F_CA | IAP_F_CC2 | IAP_F_I7 | IAP_F_WM), IAPDESCR(C0H_04H, 0xC0, 0x04, IAP_F_FM | IAP_F_CA | IAP_F_CC2 | IAP_F_I7 | IAP_F_WM), IAPDESCR(C0H_08H, 0xC0, 0x08, IAP_F_FM | IAP_F_CC2E), IAPDESCR(C1H_00H, 0xC1, 0x00, IAP_F_FM | IAP_F_CC), IAPDESCR(C1H_01H, 0xC1, 0x01, IAP_F_FM | IAP_F_CA | IAP_F_CC2), IAPDESCR(C1H_FEH, 0xC1, 0xFE, IAP_F_FM | IAP_F_CA | IAP_F_CC2), IAPDESCR(C2H_00H, 0xC2, 0x00, IAP_F_FM | IAP_F_CC), IAPDESCR(C2H_01H, 0xC2, 0x01, IAP_F_FM | IAP_F_CA | IAP_F_CC2 | IAP_F_I7 | IAP_F_WM), IAPDESCR(C2H_02H, 0xC2, 0x02, IAP_F_FM | IAP_F_CA | IAP_F_CC2 | IAP_F_I7 | IAP_F_WM), IAPDESCR(C2H_04H, 0xC2, 0x04, IAP_F_FM | IAP_F_CA | IAP_F_CC2 | IAP_F_I7 | IAP_F_WM), IAPDESCR(C2H_07H, 0xC2, 0x07, IAP_F_FM | IAP_F_CA | IAP_F_CC2), IAPDESCR(C2H_08H, 0xC2, 0x08, IAP_F_FM | IAP_F_CA | IAP_F_CC2), IAPDESCR(C2H_0FH, 0xC2, 0x0F, IAP_F_FM | IAP_F_CC2), IAPDESCR(C2H_10H, 0xC2, 0x10, IAP_F_FM | IAP_F_CA), IAPDESCR(C3H_00H, 0xC3, 0x00, IAP_F_FM | IAP_F_CC), IAPDESCR(C3H_01H, 0xC3, 0x01, IAP_F_FM | IAP_F_CA | IAP_F_CC2 | IAP_F_I7 | IAP_F_WM), IAPDESCR(C3H_02H, 0xC3, 0x02, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(C3H_04H, 0xC3, 0x04, IAP_F_FM | IAP_F_CA | IAP_F_CC2 | IAP_F_I7 | IAP_F_WM), IAPDESCR(C3H_10H, 0xC3, 0x10, IAP_F_FM | IAP_F_I7O), IAPDESCR(C4H_00H, 0xC4, 0x00, IAP_F_FM | IAP_F_ALLCPUSCORE2 | IAP_F_I7 | IAP_F_WM), IAPDESCR(C4H_01H, 0xC4, 0x01, IAP_F_FM | IAP_F_CA | IAP_F_CC2 | IAP_F_I7 | IAP_F_WM), IAPDESCR(C4H_02H, 0xC4, 0x02, IAP_F_FM | IAP_F_CA | IAP_F_CC2 | IAP_F_I7 | IAP_F_WM), IAPDESCR(C4H_04H, 0xC4, 0x04, IAP_F_FM | IAP_F_CA | IAP_F_CC2 | IAP_F_I7 | IAP_F_WM), IAPDESCR(C4H_08H, 0xC4, 0x08, IAP_F_FM | IAP_F_CA | IAP_F_CC2), IAPDESCR(C4H_0CH, 0xC4, 0x0C, IAP_F_FM | IAP_F_CA | IAP_F_CC2), IAPDESCR(C4H_0FH, 0xC4, 0x0F, IAP_F_FM | IAP_F_CA), IAPDESCR(C5H_00H, 0xC5, 0x00, IAP_F_FM | IAP_F_ALLCPUSCORE2 | IAP_F_I7 | IAP_F_WM), IAPDESCR(C5H_01H, 0xC5, 0x01, IAP_F_FM | IAP_F_WM), IAPDESCR(C5H_02H, 0xC5, 0x02, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(C5H_04H, 0xC5, 0x04, IAP_F_FM | IAP_F_WM), IAPDESCR(C6H_00H, 0xC6, 0x00, IAP_F_FM | IAP_F_CC), IAPDESCR(C6H_01H, 0xC6, 0x01, IAP_F_FM | IAP_F_CA | IAP_F_CC2), IAPDESCR(C6H_02H, 0xC6, 0x02, IAP_F_FM | IAP_F_CA | IAP_F_CC2), IAPDESCR(C7H_00H, 0xC7, 0x00, IAP_F_FM | IAP_F_CC), IAPDESCR(C7H_01H, 0xC7, 0x01, IAP_F_FM | IAP_F_CA | IAP_F_CC2 | IAP_F_I7 | IAP_F_WM), IAPDESCR(C7H_02H, 0xC7, 0x02, IAP_F_FM | IAP_F_CA | IAP_F_CC2 | IAP_F_I7 | IAP_F_WM), IAPDESCR(C7H_04H, 0xC7, 0x04, IAP_F_FM | IAP_F_CA | IAP_F_CC2 | IAP_F_I7 | IAP_F_WM), IAPDESCR(C7H_08H, 0xC7, 0x08, IAP_F_FM | IAP_F_CA | IAP_F_CC2 | IAP_F_I7 | IAP_F_WM), IAPDESCR(C7H_10H, 0xC7, 0x10, IAP_F_FM | IAP_F_CA | IAP_F_CC2 | IAP_F_I7 | IAP_F_WM), IAPDESCR(C7H_1FH, 0xC7, 0x1F, IAP_F_FM | IAP_F_CA | IAP_F_CC2), IAPDESCR(C8H_00H, 0xC8, 0x00, IAP_F_FM | IAP_F_ALLCPUSCORE2), IAPDESCR(C8H_20H, 0xC8, 0x20, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(C9H_00H, 0xC9, 0x00, IAP_F_FM | IAP_F_ALLCPUSCORE2), IAPDESCR(CAH_00H, 0xCA, 0x00, IAP_F_FM | IAP_F_CC), IAPDESCR(CAH_01H, 0xCA, 0x01, IAP_F_FM | IAP_F_CA | IAP_F_CC2), IAPDESCR(CAH_02H, 0xCA, 0x02, IAP_F_FM | IAP_F_CA | IAP_F_CC2), IAPDESCR(CAH_04H, 0xCA, 0x04, IAP_F_FM | IAP_F_CA | IAP_F_CC2), IAPDESCR(CAH_08H, 0xCA, 0x08, IAP_F_FM | IAP_F_CA | IAP_F_CC2), IAPDESCR(CBH_01H, 0xCB, 0x01, IAP_F_FM | IAP_F_CA | IAP_F_CC2 | IAP_F_I7 | IAP_F_WM), IAPDESCR(CBH_02H, 0xCB, 0x02, IAP_F_FM | IAP_F_CA | IAP_F_CC2 | IAP_F_I7 | IAP_F_WM), IAPDESCR(CBH_04H, 0xCB, 0x04, IAP_F_FM | IAP_F_CA | IAP_F_CC2 | IAP_F_I7 | IAP_F_WM), IAPDESCR(CBH_08H, 0xCB, 0x08, IAP_F_FM | IAP_F_CA | IAP_F_CC2 | IAP_F_I7 | IAP_F_WM), IAPDESCR(CBH_10H, 0xCB, 0x10, IAP_F_FM | IAP_F_CC2 | IAP_F_I7 | IAP_F_WM), IAPDESCR(CBH_40H, 0xCB, 0x40, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(CBH_80H, 0xCB, 0x80, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(CCH_00H, 0xCC, 0x00, IAP_F_FM | IAP_F_CC), IAPDESCR(CCH_01H, 0xCC, 0x01, IAP_F_FM | IAP_F_ALLCPUSCORE2 | IAP_F_I7 | IAP_F_WM), IAPDESCR(CCH_02H, 0xCC, 0x02, IAP_F_FM | IAP_F_CA | IAP_F_CC2 | IAP_F_I7 | IAP_F_WM), IAPDESCR(CCH_03H, 0xCC, 0x03, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(CDH_00H, 0xCD, 0x00, IAP_F_FM | IAP_F_ALLCPUSCORE2), IAPDESCR(CEH_00H, 0xCE, 0x00, IAP_F_FM | IAP_F_ALLCPUSCORE2), IAPDESCR(CFH_00H, 0xCF, 0x00, IAP_F_FM | IAP_F_CA | IAP_F_CC2), IAPDESCR(D0H_00H, 0xD0, 0x00, IAP_F_FM | IAP_F_CC), IAPDESCR(D0H_01H, 0xD0, 0x01, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(D1H_01H, 0xD1, 0x01, IAP_F_FM | IAP_F_WM), IAPDESCR(D1H_02H, 0xD1, 0x02, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(D1H_04H, 0xD1, 0x04, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(D1H_08H, 0xD1, 0x08, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(D2H_01H, 0xD2, 0x01, IAP_F_FM | IAP_F_CA | IAP_F_CC2 | IAP_F_I7 | IAP_F_WM), IAPDESCR(D2H_02H, 0xD2, 0x02, IAP_F_FM | IAP_F_CA | IAP_F_CC2 | IAP_F_I7 | IAP_F_WM), IAPDESCR(D2H_04H, 0xD2, 0x04, IAP_F_FM | IAP_F_CA | IAP_F_CC2 | IAP_F_I7 | IAP_F_WM), IAPDESCR(D2H_08H, 0xD2, 0x08, IAP_F_FM | IAP_F_CA | IAP_F_CC2 | IAP_F_I7 | IAP_F_WM), IAPDESCR(D2H_0FH, 0xD2, 0x0F, IAP_F_FM | IAP_F_CA | IAP_F_CC2 | IAP_F_I7 | IAP_F_WM), IAPDESCR(D2H_10H, 0xD2, 0x10, IAP_F_FM | IAP_F_CC2E), IAPDESCR(D4H_01H, 0xD4, 0x01, IAP_F_FM | IAP_F_CA | IAP_F_CC2 | IAP_F_I7 | IAP_F_WM), IAPDESCR(D4H_02H, 0xD4, 0x02, IAP_F_FM | IAP_F_CA | IAP_F_CC2), IAPDESCR(D4H_04H, 0xD4, 0x04, IAP_F_FM | IAP_F_CA | IAP_F_CC2), IAPDESCR(D4H_08H, 0xD4, 0x08, IAP_F_FM | IAP_F_CA | IAP_F_CC2), IAPDESCR(D4H_0FH, 0xD4, 0x0F, IAP_F_FM | IAP_F_CA | IAP_F_CC2), IAPDESCR(D5H_01H, 0xD5, 0x01, IAP_F_FM | IAP_F_CA | IAP_F_CC2 | IAP_F_I7 | IAP_F_WM), IAPDESCR(D5H_02H, 0xD5, 0x02, IAP_F_FM | IAP_F_CA | IAP_F_CC2), IAPDESCR(D5H_04H, 0xD5, 0x04, IAP_F_FM | IAP_F_CA | IAP_F_CC2), IAPDESCR(D5H_08H, 0xD5, 0x08, IAP_F_FM | IAP_F_CA | IAP_F_CC2), IAPDESCR(D5H_0FH, 0xD5, 0x0F, IAP_F_FM | IAP_F_CA | IAP_F_CC2), IAPDESCR(D7H_00H, 0xD7, 0x00, IAP_F_FM | IAP_F_CC), IAPDESCR(D8H_00H, 0xD8, 0x00, IAP_F_FM | IAP_F_CC), IAPDESCR(D8H_01H, 0xD8, 0x01, IAP_F_FM | IAP_F_CC), IAPDESCR(D8H_02H, 0xD8, 0x02, IAP_F_FM | IAP_F_CC), IAPDESCR(D8H_03H, 0xD8, 0x03, IAP_F_FM | IAP_F_CC), IAPDESCR(D8H_04H, 0xD8, 0x04, IAP_F_FM | IAP_F_CC), IAPDESCR(D9H_00H, 0xD9, 0x00, IAP_F_FM | IAP_F_CC), IAPDESCR(D9H_01H, 0xD9, 0x01, IAP_F_FM | IAP_F_CC), IAPDESCR(D9H_02H, 0xD9, 0x02, IAP_F_FM | IAP_F_CC), IAPDESCR(D9H_03H, 0xD9, 0x03, IAP_F_FM | IAP_F_CC), IAPDESCR(DAH_00H, 0xDA, 0x00, IAP_F_FM | IAP_F_CC), IAPDESCR(DAH_01H, 0xDA, 0x01, IAP_F_FM | IAP_F_CC), IAPDESCR(DAH_02H, 0xDA, 0x02, IAP_F_FM | IAP_F_CC), IAPDESCR(DBH_00H, 0xDB, 0x00, IAP_F_FM | IAP_F_CC), IAPDESCR(DBH_01H, 0xDB, 0x01, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(DCH_01H, 0xDC, 0x01, IAP_F_FM | IAP_F_CA | IAP_F_CC2), IAPDESCR(DCH_02H, 0xDC, 0x02, IAP_F_FM | IAP_F_CA | IAP_F_CC2), IAPDESCR(DCH_04H, 0xDC, 0x04, IAP_F_FM | IAP_F_CA | IAP_F_CC2), IAPDESCR(DCH_08H, 0xDC, 0x08, IAP_F_FM | IAP_F_CA | IAP_F_CC2), IAPDESCR(DCH_10H, 0xDC, 0x10, IAP_F_FM | IAP_F_CA | IAP_F_CC2), IAPDESCR(DCH_1FH, 0xDC, 0x1F, IAP_F_FM | IAP_F_CA | IAP_F_CC2), IAPDESCR(E0H_00H, 0xE0, 0x00, IAP_F_FM | IAP_F_CC | IAP_F_CC2), IAPDESCR(E0H_01H, 0xE0, 0x01, IAP_F_FM | IAP_F_CA | IAP_F_I7 | IAP_F_WM), IAPDESCR(E2H_00H, 0xE2, 0x00, IAP_F_FM | IAP_F_CC), IAPDESCR(E4H_00H, 0xE4, 0x00, IAP_F_FM | IAP_F_ALLCPUSCORE2), IAPDESCR(E4H_01H, 0xE4, 0x01, IAP_F_FM | IAP_F_I7O), IAPDESCR(E5H_01H, 0xE5, 0x01, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(E6H_00H, 0xE6, 0x00, IAP_F_FM | IAP_F_CC | IAP_F_CC2), IAPDESCR(E6H_01H, 0xE6, 0x01, IAP_F_FM | IAP_F_CA | IAP_F_I7 | IAP_F_WM), IAPDESCR(E6H_02H, 0xE6, 0x02, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(E8H_01H, 0xE8, 0x01, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(E8H_02H, 0xE8, 0x02, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(E8H_03H, 0xE8, 0x03, IAP_F_FM | IAP_F_I7), IAPDESCR(ECH_01H, 0xEC, 0x01, IAP_F_FM | IAP_F_WM), IAPDESCR(F0H_00H, 0xF0, 0x00, IAP_F_FM | IAP_F_ALLCPUSCORE2), IAPDESCR(F0H_01H, 0xF0, 0x01, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(F0H_02H, 0xF0, 0x02, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(F0H_04H, 0xF0, 0x04, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(F0H_08H, 0xF0, 0x08, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(F0H_10H, 0xF0, 0x10, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(F0H_20H, 0xF0, 0x20, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(F0H_40H, 0xF0, 0x40, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(F0H_80H, 0xF0, 0x80, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(F1H_02H, 0xF1, 0x02, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(F1H_04H, 0xF1, 0x04, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(F1H_07H, 0xF1, 0x07, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(F2H_01H, 0xF2, 0x01, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(F2H_02H, 0xF2, 0x02, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(F2H_04H, 0xF2, 0x04, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(F2H_08H, 0xF2, 0x08, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(F2H_0FH, 0xF2, 0x0F, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(F3H_01H, 0xF3, 0x01, IAP_F_FM | IAP_F_I7O), IAPDESCR(F3H_02H, 0xF3, 0x02, IAP_F_FM | IAP_F_I7O), IAPDESCR(F3H_04H, 0xF3, 0x04, IAP_F_FM | IAP_F_I7O), IAPDESCR(F3H_08H, 0xF3, 0x08, IAP_F_FM | IAP_F_I7O), IAPDESCR(F3H_10H, 0xF3, 0x10, IAP_F_FM | IAP_F_I7O), IAPDESCR(F3H_20H, 0xF3, 0x20, IAP_F_FM | IAP_F_I7O), IAPDESCR(F4H_01H, 0xF4, 0x01, IAP_F_FM | IAP_F_I7O), IAPDESCR(F4H_02H, 0xF4, 0x02, IAP_F_FM | IAP_F_I7O), IAPDESCR(F4H_04H, 0xF4, 0x04, IAP_F_FM | IAP_F_WM | IAP_F_I7O), IAPDESCR(F4H_08H, 0xF4, 0x08, IAP_F_FM | IAP_F_I7O), IAPDESCR(F4H_10H, 0xF4, 0x10, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(F6H_01H, 0xF6, 0x01, IAP_F_FM | IAP_F_I7 | IAP_F_WM), IAPDESCR(F7H_01H, 0xF7, 0x01, IAP_F_FM | IAP_F_WM | IAP_F_I7), IAPDESCR(F7H_02H, 0xF7, 0x02, IAP_F_FM | IAP_F_WM | IAP_F_I7), IAPDESCR(F7H_04H, 0xF7, 0x04, IAP_F_FM | IAP_F_WM | IAP_F_I7), IAPDESCR(F8H_00H, 0xF8, 0x00, IAP_F_FM | IAP_F_ALLCPUSCORE2), IAPDESCR(F8H_01H, 0xF8, 0x01, IAP_F_FM | IAP_F_I7O), IAPDESCR(FDH_01H, 0xFD, 0x01, IAP_F_FM | IAP_F_WM | IAP_F_I7), IAPDESCR(FDH_02H, 0xFD, 0x02, IAP_F_FM | IAP_F_WM | IAP_F_I7), IAPDESCR(FDH_04H, 0xFD, 0x04, IAP_F_FM | IAP_F_WM | IAP_F_I7), IAPDESCR(FDH_08H, 0xFD, 0x08, IAP_F_FM | IAP_F_WM | IAP_F_I7), IAPDESCR(FDH_10H, 0xFD, 0x10, IAP_F_FM | IAP_F_WM | IAP_F_I7), IAPDESCR(FDH_20H, 0xFD, 0x20, IAP_F_FM | IAP_F_WM | IAP_F_I7), IAPDESCR(FDH_40H, 0xFD, 0x40, IAP_F_FM | IAP_F_WM | IAP_F_I7), }; static const int niap_events = sizeof(iap_events) / sizeof(iap_events[0]); static pmc_value_t iap_perfctr_value_to_reload_count(pmc_value_t v) { v &= (1ULL << core_iap_width) - 1; return (1ULL << core_iap_width) - v; } static pmc_value_t iap_reload_count_to_perfctr_value(pmc_value_t rlc) { return (1ULL << core_iap_width) - rlc; } static int iap_pmc_has_overflowed(int ri) { uint64_t v; /* * We treat a Core (i.e., Intel architecture v1) PMC as has * having overflowed if its MSB is zero. */ v = rdpmc(ri); return ((v & (1ULL << (core_iap_width - 1))) == 0); } /* * Check an event against the set of supported architectural events. * * Returns 1 if the event is architectural and unsupported on this * CPU. Returns 0 otherwise. */ static int iap_architectural_event_is_unsupported(enum pmc_event pe) { enum core_arch_events ae; switch (pe) { case PMC_EV_IAP_EVENT_3CH_00H: ae = CORE_AE_UNHALTED_CORE_CYCLES; break; case PMC_EV_IAP_EVENT_C0H_00H: ae = CORE_AE_INSTRUCTION_RETIRED; break; case PMC_EV_IAP_EVENT_3CH_01H: ae = CORE_AE_UNHALTED_REFERENCE_CYCLES; break; case PMC_EV_IAP_EVENT_2EH_4FH: ae = CORE_AE_LLC_REFERENCE; break; case PMC_EV_IAP_EVENT_2EH_41H: ae = CORE_AE_LLC_MISSES; break; case PMC_EV_IAP_EVENT_C4H_00H: ae = CORE_AE_BRANCH_INSTRUCTION_RETIRED; break; case PMC_EV_IAP_EVENT_C5H_00H: ae = CORE_AE_BRANCH_MISSES_RETIRED; break; default: /* Non architectural event. */ return (0); } return ((core_architectural_events & (1 << ae)) == 0); } static int iap_event_corei7_ok_on_counter(enum pmc_event pe, int ri) { uint32_t mask; switch (pe) { /* * Events valid only on counter 0, 1. */ case PMC_EV_IAP_EVENT_40H_01H: case PMC_EV_IAP_EVENT_40H_02H: case PMC_EV_IAP_EVENT_40H_04H: case PMC_EV_IAP_EVENT_40H_08H: case PMC_EV_IAP_EVENT_40H_0FH: case PMC_EV_IAP_EVENT_41H_02H: case PMC_EV_IAP_EVENT_41H_04H: case PMC_EV_IAP_EVENT_41H_08H: case PMC_EV_IAP_EVENT_42H_01H: case PMC_EV_IAP_EVENT_42H_02H: case PMC_EV_IAP_EVENT_42H_04H: case PMC_EV_IAP_EVENT_42H_08H: case PMC_EV_IAP_EVENT_43H_01H: case PMC_EV_IAP_EVENT_43H_02H: case PMC_EV_IAP_EVENT_48H_02H: case PMC_EV_IAP_EVENT_51H_01H: case PMC_EV_IAP_EVENT_51H_02H: case PMC_EV_IAP_EVENT_51H_04H: case PMC_EV_IAP_EVENT_51H_08H: case PMC_EV_IAP_EVENT_63H_01H: case PMC_EV_IAP_EVENT_63H_02H: mask = 0x3; break; default: mask = ~0; /* Any row index is ok. */ } return (mask & (1 << ri)); } static int iap_event_westmere_ok_on_counter(enum pmc_event pe, int ri) { uint32_t mask; switch (pe) { /* * Events valid only on counter 0. */ case PMC_EV_IAP_EVENT_B3H_01H: case PMC_EV_IAP_EVENT_B3H_02H: case PMC_EV_IAP_EVENT_B3H_04H: mask = 0x1; break; /* * Events valid only on counter 0, 1. */ case PMC_EV_IAP_EVENT_51H_01H: case PMC_EV_IAP_EVENT_51H_02H: case PMC_EV_IAP_EVENT_51H_04H: case PMC_EV_IAP_EVENT_51H_08H: case PMC_EV_IAP_EVENT_63H_01H: case PMC_EV_IAP_EVENT_63H_02H: mask = 0x3; break; default: mask = ~0; /* Any row index is ok. */ } return (mask & (1 << ri)); } static int iap_event_ok_on_counter(enum pmc_event pe, int ri) { uint32_t mask; switch (pe) { /* * Events valid only on counter 0. */ case PMC_EV_IAP_EVENT_10H_00H: case PMC_EV_IAP_EVENT_14H_00H: case PMC_EV_IAP_EVENT_18H_00H: case PMC_EV_IAP_EVENT_B3H_01H: case PMC_EV_IAP_EVENT_B3H_02H: case PMC_EV_IAP_EVENT_B3H_04H: case PMC_EV_IAP_EVENT_C1H_00H: case PMC_EV_IAP_EVENT_CBH_01H: case PMC_EV_IAP_EVENT_CBH_02H: mask = (1 << 0); break; /* * Events valid only on counter 1. */ case PMC_EV_IAP_EVENT_11H_00H: case PMC_EV_IAP_EVENT_12H_00H: case PMC_EV_IAP_EVENT_13H_00H: mask = (1 << 1); break; default: mask = ~0; /* Any row index is ok. */ } return (mask & (1 << ri)); } static int iap_allocate_pmc(int cpu, int ri, struct pmc *pm, const struct pmc_op_pmcallocate *a) { int n; enum pmc_event ev; struct iap_event_descr *ie; uint32_t c, caps, config, cpuflag, evsel, mask; KASSERT(cpu >= 0 && cpu < pmc_cpu_max(), ("[core,%d] illegal CPU %d", __LINE__, cpu)); KASSERT(ri >= 0 && ri < core_iap_npmc, ("[core,%d] illegal row-index value %d", __LINE__, ri)); /* check requested capabilities */ caps = a->pm_caps; if ((IAP_PMC_CAPS & caps) != caps) return (EPERM); ev = pm->pm_event; if (iap_architectural_event_is_unsupported(ev)) return (EOPNOTSUPP); switch (core_cputype) { case PMC_CPU_INTEL_COREI7: if (iap_event_corei7_ok_on_counter(ev, ri) == 0) return (EINVAL); break; case PMC_CPU_INTEL_WESTMERE: if (iap_event_westmere_ok_on_counter(ev, ri) == 0) return (EINVAL); break; default: if (iap_event_ok_on_counter(ev, ri) == 0) return (EINVAL); } /* * Look for an event descriptor with matching CPU and event id * fields. */ switch (core_cputype) { default: case PMC_CPU_INTEL_ATOM: cpuflag = IAP_F_CA; break; case PMC_CPU_INTEL_CORE: cpuflag = IAP_F_CC; break; case PMC_CPU_INTEL_CORE2: cpuflag = IAP_F_CC2; break; case PMC_CPU_INTEL_CORE2EXTREME: cpuflag = IAP_F_CC2 | IAP_F_CC2E; break; case PMC_CPU_INTEL_COREI7: cpuflag = IAP_F_I7; break; case PMC_CPU_INTEL_WESTMERE: cpuflag = IAP_F_WM; break; } for (n = 0, ie = iap_events; n < niap_events; n++, ie++) if (ie->iap_ev == ev && ie->iap_flags & cpuflag) break; if (n == niap_events) return (EINVAL); /* * A matching event descriptor has been found, so start * assembling the contents of the event select register. */ evsel = ie->iap_evcode; config = a->pm_md.pm_iap.pm_iap_config & ~IAP_F_CMASK; /* * If the event uses a fixed umask value, reject any umask * bits set by the user. */ if (ie->iap_flags & IAP_F_FM) { if (IAP_UMASK(config) != 0) return (EINVAL); evsel |= (ie->iap_umask << 8); } else { /* * Otherwise, the UMASK value needs to be taken from * the MD fields of the allocation request. Reject * requests that specify reserved bits. */ mask = 0; if (ie->iap_umask & IAP_M_CORE) { if ((c = (config & IAP_F_CORE)) != IAP_CORE_ALL && c != IAP_CORE_THIS) return (EINVAL); mask |= IAP_F_CORE; } if (ie->iap_umask & IAP_M_AGENT) mask |= IAP_F_AGENT; if (ie->iap_umask & IAP_M_PREFETCH) { if ((c = (config & IAP_F_PREFETCH)) == IAP_PREFETCH_RESERVED) return (EINVAL); mask |= IAP_F_PREFETCH; } if (ie->iap_umask & IAP_M_MESI) mask |= IAP_F_MESI; if (ie->iap_umask & IAP_M_SNOOPRESPONSE) mask |= IAP_F_SNOOPRESPONSE; if (ie->iap_umask & IAP_M_SNOOPTYPE) mask |= IAP_F_SNOOPTYPE; if (ie->iap_umask & IAP_M_TRANSITION) mask |= IAP_F_TRANSITION; /* * If bits outside of the allowed set of umask bits * are set, reject the request. */ if (config & ~mask) return (EINVAL); evsel |= (config & mask); } /* * Only Atom CPUs support the 'ANY' qualifier. */ if (core_cputype == PMC_CPU_INTEL_ATOM) evsel |= (config & IAP_ANY); else if (config & IAP_ANY) return (EINVAL); /* * Check offcore response configuration. */ if (a->pm_md.pm_iap.pm_iap_rsp != 0) { if (ev != PMC_EV_IAP_EVENT_B7H_01H && ev != PMC_EV_IAP_EVENT_BBH_01H) return (EINVAL); if (core_cputype == PMC_CPU_INTEL_COREI7 && ev == PMC_EV_IAP_EVENT_BBH_01H) return (EINVAL); if ( a->pm_md.pm_iap.pm_iap_rsp & ~IA_OFFCORE_RSP_MASK) return (EINVAL); pm->pm_md.pm_iap.pm_iap_rsp = a->pm_md.pm_iap.pm_iap_rsp & IA_OFFCORE_RSP_MASK; } if (caps & PMC_CAP_THRESHOLD) evsel |= (a->pm_md.pm_iap.pm_iap_config & IAP_F_CMASK); if (caps & PMC_CAP_USER) evsel |= IAP_USR; if (caps & PMC_CAP_SYSTEM) evsel |= IAP_OS; if ((caps & (PMC_CAP_USER | PMC_CAP_SYSTEM)) == 0) evsel |= (IAP_OS | IAP_USR); if (caps & PMC_CAP_EDGE) evsel |= IAP_EDGE; if (caps & PMC_CAP_INVERT) evsel |= IAP_INV; if (caps & PMC_CAP_INTERRUPT) evsel |= IAP_INT; pm->pm_md.pm_iap.pm_iap_evsel = evsel; return (0); } static int iap_config_pmc(int cpu, int ri, struct pmc *pm) { KASSERT(cpu >= 0 && cpu < pmc_cpu_max(), ("[core,%d] illegal CPU %d", __LINE__, cpu)); KASSERT(ri >= 0 && ri < core_iap_npmc, ("[core,%d] illegal row-index %d", __LINE__, ri)); PMCDBG(MDP,CFG,1, "iap-config cpu=%d ri=%d pm=%p", cpu, ri, pm); KASSERT(core_pcpu[cpu] != NULL, ("[core,%d] null per-cpu %d", __LINE__, cpu)); core_pcpu[cpu]->pc_corepmcs[ri].phw_pmc = pm; return (0); } static int iap_describe(int cpu, int ri, struct pmc_info *pi, struct pmc **ppmc) { int error; struct pmc_hw *phw; char iap_name[PMC_NAME_MAX]; phw = &core_pcpu[cpu]->pc_corepmcs[ri]; (void) snprintf(iap_name, sizeof(iap_name), "IAP-%d", ri); if ((error = copystr(iap_name, pi->pm_name, PMC_NAME_MAX, NULL)) != 0) return (error); pi->pm_class = PMC_CLASS_IAP; if (phw->phw_state & PMC_PHW_FLAG_IS_ENABLED) { pi->pm_enabled = TRUE; *ppmc = phw->phw_pmc; } else { pi->pm_enabled = FALSE; *ppmc = NULL; } return (0); } static int iap_get_config(int cpu, int ri, struct pmc **ppm) { *ppm = core_pcpu[cpu]->pc_corepmcs[ri].phw_pmc; return (0); } static int iap_get_msr(int ri, uint32_t *msr) { KASSERT(ri >= 0 && ri < core_iap_npmc, ("[iap,%d] ri %d out of range", __LINE__, ri)); *msr = ri; return (0); } static int iap_read_pmc(int cpu, int ri, pmc_value_t *v) { struct pmc *pm; pmc_value_t tmp; KASSERT(cpu >= 0 && cpu < pmc_cpu_max(), ("[core,%d] illegal cpu value %d", __LINE__, cpu)); KASSERT(ri >= 0 && ri < core_iap_npmc, ("[core,%d] illegal row-index %d", __LINE__, ri)); pm = core_pcpu[cpu]->pc_corepmcs[ri].phw_pmc; KASSERT(pm, ("[core,%d] cpu %d ri %d pmc not configured", __LINE__, cpu, ri)); tmp = rdpmc(ri); if (PMC_IS_SAMPLING_MODE(PMC_TO_MODE(pm))) *v = iap_perfctr_value_to_reload_count(tmp); else *v = tmp; PMCDBG(MDP,REA,1, "iap-read cpu=%d ri=%d msr=0x%x -> v=%jx", cpu, ri, ri, *v); return (0); } static int iap_release_pmc(int cpu, int ri, struct pmc *pm) { (void) pm; PMCDBG(MDP,REL,1, "iap-release cpu=%d ri=%d pm=%p", cpu, ri, pm); KASSERT(cpu >= 0 && cpu < pmc_cpu_max(), ("[core,%d] illegal CPU value %d", __LINE__, cpu)); KASSERT(ri >= 0 && ri < core_iap_npmc, ("[core,%d] illegal row-index %d", __LINE__, ri)); KASSERT(core_pcpu[cpu]->pc_corepmcs[ri].phw_pmc == NULL, ("[core,%d] PHW pmc non-NULL", __LINE__)); return (0); } static int iap_start_pmc(int cpu, int ri) { struct pmc *pm; uint32_t evsel; struct core_cpu *cc; KASSERT(cpu >= 0 && cpu < pmc_cpu_max(), ("[core,%d] illegal CPU value %d", __LINE__, cpu)); KASSERT(ri >= 0 && ri < core_iap_npmc, ("[core,%d] illegal row-index %d", __LINE__, ri)); cc = core_pcpu[cpu]; pm = cc->pc_corepmcs[ri].phw_pmc; KASSERT(pm, ("[core,%d] starting cpu%d,ri%d with no pmc configured", __LINE__, cpu, ri)); PMCDBG(MDP,STA,1, "iap-start cpu=%d ri=%d", cpu, ri); evsel = pm->pm_md.pm_iap.pm_iap_evsel; PMCDBG(MDP,STA,2, "iap-start/2 cpu=%d ri=%d evselmsr=0x%x evsel=0x%x", cpu, ri, IAP_EVSEL0 + ri, evsel); /* Event specific configuration. */ switch (pm->pm_event) { case PMC_EV_IAP_EVENT_B7H_01H: wrmsr(IA_OFFCORE_RSP0, pm->pm_md.pm_iap.pm_iap_rsp); break; case PMC_EV_IAP_EVENT_BBH_01H: wrmsr(IA_OFFCORE_RSP1, pm->pm_md.pm_iap.pm_iap_rsp); break; default: break; } wrmsr(IAP_EVSEL0 + ri, evsel | IAP_EN); if (core_cputype == PMC_CPU_INTEL_CORE) return (0); do { cc->pc_resync = 0; cc->pc_globalctrl |= (1ULL << ri); wrmsr(IA_GLOBAL_CTRL, cc->pc_globalctrl); } while (cc->pc_resync != 0); return (0); } static int iap_stop_pmc(int cpu, int ri) { struct pmc *pm; struct core_cpu *cc; KASSERT(cpu >= 0 && cpu < pmc_cpu_max(), ("[core,%d] illegal cpu value %d", __LINE__, cpu)); KASSERT(ri >= 0 && ri < core_iap_npmc, ("[core,%d] illegal row index %d", __LINE__, ri)); cc = core_pcpu[cpu]; pm = cc->pc_corepmcs[ri].phw_pmc; KASSERT(pm, ("[core,%d] cpu%d ri%d no configured PMC to stop", __LINE__, cpu, ri)); PMCDBG(MDP,STO,1, "iap-stop cpu=%d ri=%d", cpu, ri); wrmsr(IAP_EVSEL0 + ri, 0); /* stop hw */ if (core_cputype == PMC_CPU_INTEL_CORE) return (0); do { cc->pc_resync = 0; cc->pc_globalctrl &= ~(1ULL << ri); wrmsr(IA_GLOBAL_CTRL, cc->pc_globalctrl); } while (cc->pc_resync != 0); return (0); } static int iap_write_pmc(int cpu, int ri, pmc_value_t v) { struct pmc *pm; struct core_cpu *cc; KASSERT(cpu >= 0 && cpu < pmc_cpu_max(), ("[core,%d] illegal cpu value %d", __LINE__, cpu)); KASSERT(ri >= 0 && ri < core_iap_npmc, ("[core,%d] illegal row index %d", __LINE__, ri)); cc = core_pcpu[cpu]; pm = cc->pc_corepmcs[ri].phw_pmc; KASSERT(pm, ("[core,%d] cpu%d ri%d no configured PMC to stop", __LINE__, cpu, ri)); PMCDBG(MDP,WRI,1, "iap-write cpu=%d ri=%d msr=0x%x v=%jx", cpu, ri, IAP_PMC0 + ri, v); if (PMC_IS_SAMPLING_MODE(PMC_TO_MODE(pm))) v = iap_reload_count_to_perfctr_value(v); /* * Write the new value to the counter. The counter will be in * a stopped state when the pcd_write() entry point is called. */ wrmsr(IAP_PMC0 + ri, v); return (0); } static void iap_initialize(struct pmc_mdep *md, int maxcpu, int npmc, int pmcwidth, int flags) { struct pmc_classdep *pcd; KASSERT(md != NULL, ("[iap,%d] md is NULL", __LINE__)); PMCDBG(MDP,INI,1, "%s", "iap-initialize"); /* Remember the set of architectural events supported. */ core_architectural_events = ~flags; pcd = &md->pmd_classdep[PMC_MDEP_CLASS_INDEX_IAP]; pcd->pcd_caps = IAP_PMC_CAPS; pcd->pcd_class = PMC_CLASS_IAP; pcd->pcd_num = npmc; pcd->pcd_ri = md->pmd_npmc; pcd->pcd_width = pmcwidth; pcd->pcd_allocate_pmc = iap_allocate_pmc; pcd->pcd_config_pmc = iap_config_pmc; pcd->pcd_describe = iap_describe; pcd->pcd_get_config = iap_get_config; pcd->pcd_get_msr = iap_get_msr; pcd->pcd_pcpu_fini = core_pcpu_fini; pcd->pcd_pcpu_init = core_pcpu_init; pcd->pcd_read_pmc = iap_read_pmc; pcd->pcd_release_pmc = iap_release_pmc; pcd->pcd_start_pmc = iap_start_pmc; pcd->pcd_stop_pmc = iap_stop_pmc; pcd->pcd_write_pmc = iap_write_pmc; md->pmd_npmc += npmc; } static int core_intr(int cpu, struct trapframe *tf) { pmc_value_t v; struct pmc *pm; struct core_cpu *cc; int error, found_interrupt, ri; PMCDBG(MDP,INT, 1, "cpu=%d tf=0x%p um=%d", cpu, (void *) tf, TRAPF_USERMODE(tf)); found_interrupt = 0; cc = core_pcpu[cpu]; for (ri = 0; ri < core_iap_npmc; ri++) { if ((pm = cc->pc_corepmcs[ri].phw_pmc) == NULL || !PMC_IS_SAMPLING_MODE(PMC_TO_MODE(pm))) continue; if (!iap_pmc_has_overflowed(ri)) continue; found_interrupt = 1; if (pm->pm_state != PMC_STATE_RUNNING) continue; error = pmc_process_interrupt(cpu, pm, tf, TRAPF_USERMODE(tf)); v = pm->pm_sc.pm_reloadcount; v = iaf_reload_count_to_perfctr_value(v); /* * Stop the counter, reload it but only restart it if * the PMC is not stalled. */ wrmsr(IAP_EVSEL0 + ri, 0); wrmsr(IAP_PMC0 + ri, v); if (error) continue; wrmsr(IAP_EVSEL0 + ri, pm->pm_md.pm_iap.pm_iap_evsel | IAP_EN); } if (found_interrupt) lapic_reenable_pmc(); atomic_add_int(found_interrupt ? &pmc_stats.pm_intr_processed : &pmc_stats.pm_intr_ignored, 1); return (found_interrupt); } static int core2_intr(int cpu, struct trapframe *tf) { int error, found_interrupt, n; uint64_t flag, intrstatus, intrenable; struct pmc *pm; struct core_cpu *cc; pmc_value_t v; PMCDBG(MDP,INT, 1, "cpu=%d tf=0x%p um=%d", cpu, (void *) tf, TRAPF_USERMODE(tf)); /* * The IA_GLOBAL_STATUS (MSR 0x38E) register indicates which * PMCs have a pending PMI interrupt. We take a 'snapshot' of * the current set of interrupting PMCs and process these * after stopping them. */ intrstatus = rdmsr(IA_GLOBAL_STATUS); intrenable = intrstatus & core_pmcmask; PMCDBG(MDP,INT, 1, "cpu=%d intrstatus=%jx", cpu, (uintmax_t) intrstatus); found_interrupt = 0; cc = core_pcpu[cpu]; KASSERT(cc != NULL, ("[core,%d] null pcpu", __LINE__)); cc->pc_globalctrl &= ~intrenable; cc->pc_resync = 1; /* MSRs now potentially out of sync. */ /* * Stop PMCs and clear overflow status bits. */ wrmsr(IA_GLOBAL_CTRL, 0); wrmsr(IA_GLOBAL_OVF_CTRL, intrenable | IA_GLOBAL_STATUS_FLAG_OVFBUF | IA_GLOBAL_STATUS_FLAG_CONDCHG); /* * Look for interrupts from fixed function PMCs. */ for (n = 0, flag = (1ULL << IAF_OFFSET); n < core_iaf_npmc; n++, flag <<= 1) { if ((intrstatus & flag) == 0) continue; found_interrupt = 1; pm = cc->pc_corepmcs[n + core_iaf_ri].phw_pmc; if (pm == NULL || pm->pm_state != PMC_STATE_RUNNING || !PMC_IS_SAMPLING_MODE(PMC_TO_MODE(pm))) continue; error = pmc_process_interrupt(cpu, pm, tf, TRAPF_USERMODE(tf)); if (error) intrenable &= ~flag; v = iaf_reload_count_to_perfctr_value(pm->pm_sc.pm_reloadcount); /* Reload sampling count. */ wrmsr(IAF_CTR0 + n, v); PMCDBG(MDP,INT, 1, "iaf-intr cpu=%d error=%d v=%jx(%jx)", cpu, error, (uintmax_t) v, (uintmax_t) rdpmc(IAF_RI_TO_MSR(n))); } /* * Process interrupts from the programmable counters. */ for (n = 0, flag = 1; n < core_iap_npmc; n++, flag <<= 1) { if ((intrstatus & flag) == 0) continue; found_interrupt = 1; pm = cc->pc_corepmcs[n].phw_pmc; if (pm == NULL || pm->pm_state != PMC_STATE_RUNNING || !PMC_IS_SAMPLING_MODE(PMC_TO_MODE(pm))) continue; error = pmc_process_interrupt(cpu, pm, tf, TRAPF_USERMODE(tf)); if (error) intrenable &= ~flag; v = iap_reload_count_to_perfctr_value(pm->pm_sc.pm_reloadcount); PMCDBG(MDP,INT, 1, "iap-intr cpu=%d error=%d v=%jx", cpu, error, (uintmax_t) v); /* Reload sampling count. */ wrmsr(IAP_PMC0 + n, v); } /* * Reenable all non-stalled PMCs. */ PMCDBG(MDP,INT, 1, "cpu=%d intrenable=%jx", cpu, (uintmax_t) intrenable); cc->pc_globalctrl |= intrenable; wrmsr(IA_GLOBAL_CTRL, cc->pc_globalctrl); PMCDBG(MDP,INT, 1, "cpu=%d fixedctrl=%jx globalctrl=%jx status=%jx " "ovf=%jx", cpu, (uintmax_t) rdmsr(IAF_CTRL), (uintmax_t) rdmsr(IA_GLOBAL_CTRL), (uintmax_t) rdmsr(IA_GLOBAL_STATUS), (uintmax_t) rdmsr(IA_GLOBAL_OVF_CTRL)); if (found_interrupt) lapic_reenable_pmc(); atomic_add_int(found_interrupt ? &pmc_stats.pm_intr_processed : &pmc_stats.pm_intr_ignored, 1); return (found_interrupt); } int pmc_core_initialize(struct pmc_mdep *md, int maxcpu) { int cpuid[CORE_CPUID_REQUEST_SIZE]; int ipa_version, flags, nflags; do_cpuid(CORE_CPUID_REQUEST, cpuid); ipa_version = cpuid[CORE_CPUID_EAX] & 0xFF; PMCDBG(MDP,INI,1,"core-init cputype=%d ncpu=%d ipa-version=%d", md->pmd_cputype, maxcpu, ipa_version); if (ipa_version < 1 || ipa_version > 3) /* Unknown PMC architecture. */ return (EPROGMISMATCH); core_cputype = md->pmd_cputype; core_pmcmask = 0; /* * Initialize programmable counters. */ KASSERT(ipa_version >= 1, ("[core,%d] ipa_version %d too small", __LINE__, ipa_version)); core_iap_npmc = (cpuid[CORE_CPUID_EAX] >> 8) & 0xFF; core_iap_width = (cpuid[CORE_CPUID_EAX] >> 16) & 0xFF; core_pmcmask |= ((1ULL << core_iap_npmc) - 1); nflags = (cpuid[CORE_CPUID_EAX] >> 24) & 0xFF; flags = cpuid[CORE_CPUID_EBX] & ((1 << nflags) - 1); iap_initialize(md, maxcpu, core_iap_npmc, core_iap_width, flags); /* * Initialize fixed function counters, if present. */ if (core_cputype != PMC_CPU_INTEL_CORE) { KASSERT(ipa_version >= 2, ("[core,%d] ipa_version %d too small", __LINE__, ipa_version)); core_iaf_ri = core_iap_npmc; core_iaf_npmc = cpuid[CORE_CPUID_EDX] & 0x1F; core_iaf_width = (cpuid[CORE_CPUID_EDX] >> 5) & 0xFF; if (core_iaf_npmc > 0) { iaf_initialize(md, maxcpu, core_iaf_npmc, core_iaf_width); core_pmcmask |= ((1ULL << core_iaf_npmc) - 1) << IAF_OFFSET; } else { /* * Adjust the number of classes exported to * user space. */ md->pmd_nclass--; KASSERT(md->pmd_nclass == 2, ("[core,%d] unexpected nclass %d", __LINE__, md->pmd_nclass)); } } PMCDBG(MDP,INI,1,"core-init pmcmask=0x%jx iafri=%d", core_pmcmask, core_iaf_ri); core_pcpu = malloc(sizeof(struct core_cpu **) * maxcpu, M_PMC, M_ZERO | M_WAITOK); /* * Choose the appropriate interrupt handler. */ if (ipa_version == 1) md->pmd_intr = core_intr; else md->pmd_intr = core2_intr; md->pmd_pcpu_fini = NULL; md->pmd_pcpu_init = NULL; return (0); } void pmc_core_finalize(struct pmc_mdep *md) { PMCDBG(MDP,INI,1, "%s", "core-finalize"); free(core_pcpu, M_PMC); core_pcpu = NULL; } Index: user/imp/tbemd/sys/dev/hwpmc/pmc_events.h =================================================================== --- user/imp/tbemd/sys/dev/hwpmc/pmc_events.h (revision 206684) +++ user/imp/tbemd/sys/dev/hwpmc/pmc_events.h (revision 206685) @@ -1,3145 +1,3145 @@ /*- * Copyright (c) 2005 Joseph Koshy * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * $FreeBSD$ */ #ifndef _DEV_HWPMC_PMC_EVENTS_H_ #define _DEV_HWPMC_PMC_EVENTS_H_ /* * Note: Documentation on adding events can be found both in * the source tree at src/share/doc/papers/hwpmc/hwpmc.ms * as well as on-line at: * * http://wiki.freebsd.org/PmcTools/PmcHardwareHowTo * * Please refer to those resources before you attempt to modify * this file or the hwpmc driver/subsystem. */ /* * PMC event codes. * * __PMC_EV(CLASS, SYMBOLIC-NAME) * */ /* * AMD K7 Events, from "The AMD Athlon(tm) Processor x86 Code * Optimization Guide" [Doc#22007K, Feb 2002] */ #define __PMC_EV_K7() \ __PMC_EV(K7, DC_ACCESSES) \ __PMC_EV(K7, DC_MISSES) \ __PMC_EV(K7, DC_REFILLS_FROM_L2) \ __PMC_EV(K7, DC_REFILLS_FROM_SYSTEM) \ __PMC_EV(K7, DC_WRITEBACKS) \ __PMC_EV(K7, L1_DTLB_MISS_AND_L2_DTLB_HITS) \ __PMC_EV(K7, L1_AND_L2_DTLB_MISSES) \ __PMC_EV(K7, MISALIGNED_REFERENCES) \ __PMC_EV(K7, IC_FETCHES) \ __PMC_EV(K7, IC_MISSES) \ __PMC_EV(K7, L1_ITLB_MISSES) \ __PMC_EV(K7, L1_L2_ITLB_MISSES) \ __PMC_EV(K7, RETIRED_INSTRUCTIONS) \ __PMC_EV(K7, RETIRED_OPS) \ __PMC_EV(K7, RETIRED_BRANCHES) \ __PMC_EV(K7, RETIRED_BRANCHES_MISPREDICTED) \ __PMC_EV(K7, RETIRED_TAKEN_BRANCHES) \ __PMC_EV(K7, RETIRED_TAKEN_BRANCHES_MISPREDICTED) \ __PMC_EV(K7, RETIRED_FAR_CONTROL_TRANSFERS) \ __PMC_EV(K7, RETIRED_RESYNC_BRANCHES) \ __PMC_EV(K7, INTERRUPTS_MASKED_CYCLES) \ __PMC_EV(K7, INTERRUPTS_MASKED_WHILE_PENDING_CYCLES) \ __PMC_EV(K7, HARDWARE_INTERRUPTS) #define PMC_EV_K7_FIRST PMC_EV_K7_DC_ACCESSES #define PMC_EV_K7_LAST PMC_EV_K7_HARDWARE_INTERRUPTS /* * Intel P4 Events, from "IA-32 Intel(r) Architecture Software * Developer's Manual, Volume 3: System Programming Guide" [245472-012] */ #define __PMC_EV_P4() \ __PMC_EV(P4, TC_DELIVER_MODE) \ __PMC_EV(P4, BPU_FETCH_REQUEST) \ __PMC_EV(P4, ITLB_REFERENCE) \ __PMC_EV(P4, MEMORY_CANCEL) \ __PMC_EV(P4, MEMORY_COMPLETE) \ __PMC_EV(P4, LOAD_PORT_REPLAY) \ __PMC_EV(P4, STORE_PORT_REPLAY) \ __PMC_EV(P4, MOB_LOAD_REPLAY) \ __PMC_EV(P4, PAGE_WALK_TYPE) \ __PMC_EV(P4, BSQ_CACHE_REFERENCE) \ __PMC_EV(P4, IOQ_ALLOCATION) \ __PMC_EV(P4, IOQ_ACTIVE_ENTRIES) \ __PMC_EV(P4, FSB_DATA_ACTIVITY) \ __PMC_EV(P4, BSQ_ALLOCATION) \ __PMC_EV(P4, BSQ_ACTIVE_ENTRIES) \ __PMC_EV(P4, SSE_INPUT_ASSIST) \ __PMC_EV(P4, PACKED_SP_UOP) \ __PMC_EV(P4, PACKED_DP_UOP) \ __PMC_EV(P4, SCALAR_SP_UOP) \ __PMC_EV(P4, SCALAR_DP_UOP) \ __PMC_EV(P4, 64BIT_MMX_UOP) \ __PMC_EV(P4, 128BIT_MMX_UOP) \ __PMC_EV(P4, X87_FP_UOP) \ __PMC_EV(P4, X87_SIMD_MOVES_UOP) \ __PMC_EV(P4, GLOBAL_POWER_EVENTS) \ __PMC_EV(P4, TC_MS_XFER) \ __PMC_EV(P4, UOP_QUEUE_WRITES) \ __PMC_EV(P4, RETIRED_MISPRED_BRANCH_TYPE) \ __PMC_EV(P4, RETIRED_BRANCH_TYPE) \ __PMC_EV(P4, RESOURCE_STALL) \ __PMC_EV(P4, WC_BUFFER) \ __PMC_EV(P4, B2B_CYCLES) \ __PMC_EV(P4, BNR) \ __PMC_EV(P4, SNOOP) \ __PMC_EV(P4, RESPONSE) \ __PMC_EV(P4, FRONT_END_EVENT) \ __PMC_EV(P4, EXECUTION_EVENT) \ __PMC_EV(P4, REPLAY_EVENT) \ __PMC_EV(P4, INSTR_RETIRED) \ __PMC_EV(P4, UOPS_RETIRED) \ __PMC_EV(P4, UOP_TYPE) \ __PMC_EV(P4, BRANCH_RETIRED) \ __PMC_EV(P4, MISPRED_BRANCH_RETIRED) \ __PMC_EV(P4, X87_ASSIST) \ __PMC_EV(P4, MACHINE_CLEAR) #define PMC_EV_P4_FIRST PMC_EV_P4_TC_DELIVER_MODE #define PMC_EV_P4_LAST PMC_EV_P4_MACHINE_CLEAR /* Intel Pentium Pro, P-II, P-III and Pentium-M style events */ #define __PMC_EV_P6() \ __PMC_EV(P6, DATA_MEM_REFS) \ __PMC_EV(P6, DCU_LINES_IN) \ __PMC_EV(P6, DCU_M_LINES_IN) \ __PMC_EV(P6, DCU_M_LINES_OUT) \ __PMC_EV(P6, DCU_MISS_OUTSTANDING) \ __PMC_EV(P6, IFU_FETCH) \ __PMC_EV(P6, IFU_FETCH_MISS) \ __PMC_EV(P6, ITLB_MISS) \ __PMC_EV(P6, IFU_MEM_STALL) \ __PMC_EV(P6, ILD_STALL) \ __PMC_EV(P6, L2_IFETCH) \ __PMC_EV(P6, L2_LD) \ __PMC_EV(P6, L2_ST) \ __PMC_EV(P6, L2_LINES_IN) \ __PMC_EV(P6, L2_LINES_OUT) \ __PMC_EV(P6, L2_M_LINES_INM) \ __PMC_EV(P6, L2_M_LINES_OUTM) \ __PMC_EV(P6, L2_RQSTS) \ __PMC_EV(P6, L2_ADS) \ __PMC_EV(P6, L2_DBUS_BUSY) \ __PMC_EV(P6, L2_DBUS_BUSY_RD) \ __PMC_EV(P6, BUS_DRDY_CLOCKS) \ __PMC_EV(P6, BUS_LOCK_CLOCKS) \ __PMC_EV(P6, BUS_REQ_OUTSTANDING) \ __PMC_EV(P6, BUS_TRAN_BRD) \ __PMC_EV(P6, BUS_TRAN_RFO) \ __PMC_EV(P6, BUS_TRANS_WB) \ __PMC_EV(P6, BUS_TRAN_IFETCH) \ __PMC_EV(P6, BUS_TRAN_INVAL) \ __PMC_EV(P6, BUS_TRAN_PWR) \ __PMC_EV(P6, BUS_TRANS_P) \ __PMC_EV(P6, BUS_TRANS_IO) \ __PMC_EV(P6, BUS_TRAN_DEF) \ __PMC_EV(P6, BUS_TRAN_BURST) \ __PMC_EV(P6, BUS_TRAN_ANY) \ __PMC_EV(P6, BUS_TRAN_MEM) \ __PMC_EV(P6, BUS_DATA_RCV) \ __PMC_EV(P6, BUS_BNR_DRV) \ __PMC_EV(P6, BUS_HIT_DRV) \ __PMC_EV(P6, BUS_HITM_DRV) \ __PMC_EV(P6, BUS_SNOOP_STALL) \ __PMC_EV(P6, FLOPS) \ __PMC_EV(P6, FP_COMPS_OPS_EXE) \ __PMC_EV(P6, FP_ASSIST) \ __PMC_EV(P6, MUL) \ __PMC_EV(P6, DIV) \ __PMC_EV(P6, CYCLES_DIV_BUSY) \ __PMC_EV(P6, LD_BLOCKS) \ __PMC_EV(P6, SB_DRAINS) \ __PMC_EV(P6, MISALIGN_MEM_REF) \ __PMC_EV(P6, EMON_KNI_PREF_DISPATCHED) \ __PMC_EV(P6, EMON_KNI_PREF_MISS) \ __PMC_EV(P6, INST_RETIRED) \ __PMC_EV(P6, UOPS_RETIRED) \ __PMC_EV(P6, INST_DECODED) \ __PMC_EV(P6, EMON_KNI_INST_RETIRED) \ __PMC_EV(P6, EMON_KNI_COMP_INST_RET) \ __PMC_EV(P6, HW_INT_RX) \ __PMC_EV(P6, CYCLES_INT_MASKED) \ __PMC_EV(P6, CYCLES_INT_PENDING_AND_MASKED) \ __PMC_EV(P6, BR_INST_RETIRED) \ __PMC_EV(P6, BR_MISS_PRED_RETIRED) \ __PMC_EV(P6, BR_TAKEN_RETIRED) \ __PMC_EV(P6, BR_MISS_PRED_TAKEN_RET) \ __PMC_EV(P6, BR_INST_DECODED) \ __PMC_EV(P6, BTB_MISSES) \ __PMC_EV(P6, BR_BOGUS) \ __PMC_EV(P6, BACLEARS) \ __PMC_EV(P6, RESOURCE_STALLS) \ __PMC_EV(P6, PARTIAL_RAT_STALLS) \ __PMC_EV(P6, SEGMENT_REG_LOADS) \ __PMC_EV(P6, CPU_CLK_UNHALTED) \ __PMC_EV(P6, MMX_INSTR_EXEC) \ __PMC_EV(P6, MMX_SAT_INSTR_EXEC) \ __PMC_EV(P6, MMX_UOPS_EXEC) \ __PMC_EV(P6, MMX_INSTR_TYPE_EXEC) \ __PMC_EV(P6, FP_MMX_TRANS) \ __PMC_EV(P6, MMX_ASSIST) \ __PMC_EV(P6, MMX_INSTR_RET) \ __PMC_EV(P6, SEG_RENAME_STALLS) \ __PMC_EV(P6, SEG_REG_RENAMES) \ __PMC_EV(P6, RET_SEG_RENAMES) \ __PMC_EV(P6, EMON_EST_TRANS) \ __PMC_EV(P6, EMON_THERMAL_TRIP) \ __PMC_EV(P6, BR_INST_EXEC) \ __PMC_EV(P6, BR_MISSP_EXEC) \ __PMC_EV(P6, BR_BAC_MISSP_EXEC) \ __PMC_EV(P6, BR_CND_EXEC) \ __PMC_EV(P6, BR_CND_MISSP_EXEC) \ __PMC_EV(P6, BR_IND_EXEC) \ __PMC_EV(P6, BR_IND_MISSP_EXEC) \ __PMC_EV(P6, BR_RET_EXEC) \ __PMC_EV(P6, BR_RET_MISSP_EXEC) \ __PMC_EV(P6, BR_RET_BAC_MISSP_EXEC) \ __PMC_EV(P6, BR_CALL_EXEC) \ __PMC_EV(P6, BR_CALL_MISSP_EXEC) \ __PMC_EV(P6, BR_IND_CALL_EXEC) \ __PMC_EV(P6, EMON_SIMD_INSTR_RETIRED) \ __PMC_EV(P6, EMON_SYNCH_UOPS) \ __PMC_EV(P6, EMON_ESP_UOPS) \ __PMC_EV(P6, EMON_FUSED_UOPS_RET) \ __PMC_EV(P6, EMON_UNFUSION) \ __PMC_EV(P6, EMON_PREF_RQSTS_UP) \ __PMC_EV(P6, EMON_PREF_RQSTS_DN) \ __PMC_EV(P6, EMON_SSE_SSE2_INST_RETIRED) \ __PMC_EV(P6, EMON_SSE_SSE2_COMP_INST_RETIRED) #define PMC_EV_P6_FIRST PMC_EV_P6_DATA_MEM_REFS #define PMC_EV_P6_LAST PMC_EV_P6_EMON_SSE_SSE2_COMP_INST_RETIRED /* AMD K8 PMCs */ #define __PMC_EV_K8() \ __PMC_EV(K8, FP_DISPATCHED_FPU_OPS) \ __PMC_EV(K8, FP_CYCLES_WITH_NO_FPU_OPS_RETIRED) \ __PMC_EV(K8, FP_DISPATCHED_FPU_FAST_FLAG_OPS) \ __PMC_EV(K8, LS_SEGMENT_REGISTER_LOAD) \ __PMC_EV(K8, LS_MICROARCHITECTURAL_RESYNC_BY_SELF_MODIFYING_CODE) \ __PMC_EV(K8, LS_MICROARCHITECTURAL_RESYNC_BY_SNOOP) \ __PMC_EV(K8, LS_BUFFER2_FULL) \ __PMC_EV(K8, LS_LOCKED_OPERATION) \ __PMC_EV(K8, LS_MICROARCHITECTURAL_LATE_CANCEL) \ __PMC_EV(K8, LS_RETIRED_CFLUSH_INSTRUCTIONS) \ __PMC_EV(K8, LS_RETIRED_CPUID_INSTRUCTIONS) \ __PMC_EV(K8, DC_ACCESS) \ __PMC_EV(K8, DC_MISS) \ __PMC_EV(K8, DC_REFILL_FROM_L2) \ __PMC_EV(K8, DC_REFILL_FROM_SYSTEM) \ __PMC_EV(K8, DC_COPYBACK) \ __PMC_EV(K8, DC_L1_DTLB_MISS_AND_L2_DTLB_HIT) \ __PMC_EV(K8, DC_L1_DTLB_MISS_AND_L2_DTLB_MISS) \ __PMC_EV(K8, DC_MISALIGNED_DATA_REFERENCE) \ __PMC_EV(K8, DC_MICROARCHITECTURAL_LATE_CANCEL) \ __PMC_EV(K8, DC_MICROARCHITECTURAL_EARLY_CANCEL) \ __PMC_EV(K8, DC_ONE_BIT_ECC_ERROR) \ __PMC_EV(K8, DC_DISPATCHED_PREFETCH_INSTRUCTIONS) \ __PMC_EV(K8, DC_DCACHE_ACCESSES_BY_LOCKS) \ __PMC_EV(K8, BU_CPU_CLK_UNHALTED) \ __PMC_EV(K8, BU_INTERNAL_L2_REQUEST) \ __PMC_EV(K8, BU_FILL_REQUEST_L2_MISS) \ __PMC_EV(K8, BU_FILL_INTO_L2) \ __PMC_EV(K8, IC_FETCH) \ __PMC_EV(K8, IC_MISS) \ __PMC_EV(K8, IC_REFILL_FROM_L2) \ __PMC_EV(K8, IC_REFILL_FROM_SYSTEM) \ __PMC_EV(K8, IC_L1_ITLB_MISS_AND_L2_ITLB_HIT) \ __PMC_EV(K8, IC_L1_ITLB_MISS_AND_L2_ITLB_MISS) \ __PMC_EV(K8, IC_MICROARCHITECTURAL_RESYNC_BY_SNOOP) \ __PMC_EV(K8, IC_INSTRUCTION_FETCH_STALL) \ __PMC_EV(K8, IC_RETURN_STACK_HIT) \ __PMC_EV(K8, IC_RETURN_STACK_OVERFLOW) \ __PMC_EV(K8, FR_RETIRED_X86_INSTRUCTIONS) \ __PMC_EV(K8, FR_RETIRED_UOPS) \ __PMC_EV(K8, FR_RETIRED_BRANCHES) \ __PMC_EV(K8, FR_RETIRED_BRANCHES_MISPREDICTED) \ __PMC_EV(K8, FR_RETIRED_TAKEN_BRANCHES) \ __PMC_EV(K8, FR_RETIRED_TAKEN_BRANCHES_MISPREDICTED) \ __PMC_EV(K8, FR_RETIRED_FAR_CONTROL_TRANSFERS) \ __PMC_EV(K8, FR_RETIRED_RESYNCS) \ __PMC_EV(K8, FR_RETIRED_NEAR_RETURNS) \ __PMC_EV(K8, FR_RETIRED_NEAR_RETURNS_MISPREDICTED) \ __PMC_EV(K8, FR_RETIRED_TAKEN_BRANCHES_MISPREDICTED_BY_ADDR_MISCOMPARE) \ __PMC_EV(K8, FR_RETIRED_FPU_INSTRUCTIONS) \ __PMC_EV(K8, FR_RETIRED_FASTPATH_DOUBLE_OP_INSTRUCTIONS) \ __PMC_EV(K8, FR_INTERRUPTS_MASKED_CYCLES) \ __PMC_EV(K8, FR_INTERRUPTS_MASKED_WHILE_PENDING_CYCLES) \ __PMC_EV(K8, FR_TAKEN_HARDWARE_INTERRUPTS) \ __PMC_EV(K8, FR_DECODER_EMPTY) \ __PMC_EV(K8, FR_DISPATCH_STALLS) \ __PMC_EV(K8, FR_DISPATCH_STALL_FROM_BRANCH_ABORT_TO_RETIRE) \ __PMC_EV(K8, FR_DISPATCH_STALL_FOR_SERIALIZATION) \ __PMC_EV(K8, FR_DISPATCH_STALL_FOR_SEGMENT_LOAD) \ __PMC_EV(K8, FR_DISPATCH_STALL_WHEN_REORDER_BUFFER_IS_FULL) \ __PMC_EV(K8, FR_DISPATCH_STALL_WHEN_RESERVATION_STATIONS_ARE_FULL) \ __PMC_EV(K8, FR_DISPATCH_STALL_WHEN_FPU_IS_FULL) \ __PMC_EV(K8, FR_DISPATCH_STALL_WHEN_LS_IS_FULL) \ __PMC_EV(K8, FR_DISPATCH_STALL_WHEN_WAITING_FOR_ALL_TO_BE_QUIET) \ __PMC_EV(K8, FR_DISPATCH_STALL_WHEN_FAR_XFER_OR_RESYNC_BRANCH_PENDING) \ __PMC_EV(K8, FR_FPU_EXCEPTIONS) \ __PMC_EV(K8, FR_NUMBER_OF_BREAKPOINTS_FOR_DR0) \ __PMC_EV(K8, FR_NUMBER_OF_BREAKPOINTS_FOR_DR1) \ __PMC_EV(K8, FR_NUMBER_OF_BREAKPOINTS_FOR_DR2) \ __PMC_EV(K8, FR_NUMBER_OF_BREAKPOINTS_FOR_DR3) \ __PMC_EV(K8, NB_MEMORY_CONTROLLER_PAGE_ACCESS_EVENT) \ __PMC_EV(K8, NB_MEMORY_CONTROLLER_PAGE_TABLE_OVERFLOW) \ __PMC_EV(K8, NB_MEMORY_CONTROLLER_DRAM_COMMAND_SLOTS_MISSED) \ __PMC_EV(K8, NB_MEMORY_CONTROLLER_TURNAROUND) \ __PMC_EV(K8, NB_MEMORY_CONTROLLER_BYPASS_SATURATION) \ __PMC_EV(K8, NB_SIZED_COMMANDS) \ __PMC_EV(K8, NB_PROBE_RESULT) \ __PMC_EV(K8, NB_HT_BUS0_BANDWIDTH) \ __PMC_EV(K8, NB_HT_BUS1_BANDWIDTH) \ __PMC_EV(K8, NB_HT_BUS2_BANDWIDTH) #define PMC_EV_K8_FIRST PMC_EV_K8_FP_DISPATCHED_FPU_OPS #define PMC_EV_K8_LAST PMC_EV_K8_NB_HT_BUS2_BANDWIDTH /* * Intel Pentium and Pentium MMX events, from the "Intel 64 and IA-32 * Architectures Software Developer's Manual, Volume 3B: System Programming * Guide, Part 2, August 2007". */ #define __PMC_EV_P5() \ __PMC_EV(P5, DATA_READ) \ __PMC_EV(P5, DATA_WRITE) \ __PMC_EV(P5, DATA_TLB_MISS) \ __PMC_EV(P5, DATA_READ_MISS) \ __PMC_EV(P5, DATA_WRITE_MISS) \ __PMC_EV(P5, WRITE_HIT_TO_M_OR_E_STATE_LINES) \ __PMC_EV(P5, DATA_CACHE_LINES_WRITTEN_BACK) \ __PMC_EV(P5, EXTERNAL_SNOOPS) \ __PMC_EV(P5, EXTERNAL_DATA_CACHE_SNOOP_HITS) \ __PMC_EV(P5, MEMORY_ACCESSES_IN_BOTH_PIPES) \ __PMC_EV(P5, BANK_CONFLICTS) \ __PMC_EV(P5, MISALIGNED_DATA_OR_IO_REFERENCES) \ __PMC_EV(P5, CODE_READ) \ __PMC_EV(P5, CODE_TLB_MISS) \ __PMC_EV(P5, CODE_CACHE_MISS) \ __PMC_EV(P5, ANY_SEGMENT_REGISTER_LOADED) \ __PMC_EV(P5, BRANCHES) \ __PMC_EV(P5, BTB_HITS) \ __PMC_EV(P5, TAKEN_BRANCH_OR_BTB_HIT) \ __PMC_EV(P5, PIPELINE_FLUSHES) \ __PMC_EV(P5, INSTRUCTIONS_EXECUTED) \ __PMC_EV(P5, INSTRUCTIONS_EXECUTED_V_PIPE) \ __PMC_EV(P5, BUS_CYCLE_DURATION) \ __PMC_EV(P5, WRITE_BUFFER_FULL_STALL_DURATION) \ __PMC_EV(P5, WAITING_FOR_DATA_MEMORY_READ_STALL_DURATION) \ __PMC_EV(P5, STALL_ON_WRITE_TO_AN_E_OR_M_STATE_LINE) \ __PMC_EV(P5, LOCKED_BUS_CYCLE) \ __PMC_EV(P5, IO_READ_OR_WRITE_CYCLE) \ __PMC_EV(P5, NONCACHEABLE_MEMORY_READS) \ __PMC_EV(P5, PIPELINE_AGI_STALLS) \ __PMC_EV(P5, FLOPS) \ __PMC_EV(P5, BREAKPOINT_MATCH_ON_DR0_REGISTER) \ __PMC_EV(P5, BREAKPOINT_MATCH_ON_DR1_REGISTER) \ __PMC_EV(P5, BREAKPOINT_MATCH_ON_DR2_REGISTER) \ __PMC_EV(P5, BREAKPOINT_MATCH_ON_DR3_REGISTER) \ __PMC_EV(P5, HARDWARE_INTERRUPTS) \ __PMC_EV(P5, DATA_READ_OR_WRITE) \ __PMC_EV(P5, DATA_READ_MISS_OR_WRITE_MISS) \ __PMC_EV(P5, BUS_OWNERSHIP_LATENCY) \ __PMC_EV(P5, BUS_OWNERSHIP_TRANSFERS) \ __PMC_EV(P5, MMX_INSTRUCTIONS_EXECUTED_U_PIPE) \ __PMC_EV(P5, MMX_INSTRUCTIONS_EXECUTED_V_PIPE) \ __PMC_EV(P5, CACHE_M_LINE_SHARING) \ __PMC_EV(P5, CACHE_LINE_SHARING) \ __PMC_EV(P5, EMMS_INSTRUCTIONS_EXECUTED) \ __PMC_EV(P5, TRANSITIONS_BETWEEN_MMX_AND_FP_INSTRUCTIONS) \ __PMC_EV(P5, BUS_UTILIZATION_DUE_TO_PROCESSOR_ACTIVITY) \ __PMC_EV(P5, WRITES_TO_NONCACHEABLE_MEMORY) \ __PMC_EV(P5, SATURATING_MMX_INSTRUCTIONS_EXECUTED) \ __PMC_EV(P5, SATURATIONS_PERFORMED) \ __PMC_EV(P5, NUMBER_OF_CYCLES_NOT_IN_HALT_STATE) \ __PMC_EV(P5, DATA_CACHE_TLB_MISS_STALL_DURATION) \ __PMC_EV(P5, MMX_INSTRUCTION_DATA_READS) \ __PMC_EV(P5, MMX_INSTRUCTION_DATA_READ_MISSES) \ __PMC_EV(P5, FLOATING_POINT_STALLS_DURATION) \ __PMC_EV(P5, TAKEN_BRANCHES) \ __PMC_EV(P5, D1_STARVATION_AND_FIFO_IS_EMPTY) \ __PMC_EV(P5, D1_STARVATION_AND_ONLY_ONE_INSTRUCTION_IN_FIFO) \ __PMC_EV(P5, MMX_INSTRUCTION_DATA_WRITES) \ __PMC_EV(P5, MMX_INSTRUCTION_DATA_WRITE_MISSES) \ __PMC_EV(P5, PIPELINE_FLUSHES_DUE_TO_WRONG_BRANCH_PREDICTIONS) \ __PMC_EV(P5, \ PIPELINE_FLUSHES_DUE_TO_WRONG_BRANCH_PREDICTIONS_RESOLVED_IN_WB_STAGE) \ __PMC_EV(P5, MISALIGNED_DATA_MEMORY_REFERENCE_ON_MMX_INSTRUCTIONS) \ __PMC_EV(P5, PIPELINE_STALL_FOR_MMX_INSTRUCTION_DATA_MEMORY_READS) \ __PMC_EV(P5, MISPREDICTED_OR_UNPREDICTED_RETURNS) \ __PMC_EV(P5, PREDICTED_RETURNS) \ __PMC_EV(P5, MMX_MULTIPLY_UNIT_INTERLOCK) \ __PMC_EV(P5, MOVD_MOVQ_STORE_STALL_DUE_TO_PREVIOUS_MMX_OPERATION) \ __PMC_EV(P5, RETURNS) \ __PMC_EV(P5, BTB_FALSE_ENTRIES) \ __PMC_EV(P5, BTB_MISS_PREDICTION_ON_NOT_TAKEN_BRANCH) \ __PMC_EV(P5, \ FULL_WRITE_BUFFER_STALL_DURATION_WHILE_EXECUTING_MMX_INSTRUCTIONS) \ __PMC_EV(P5, STALL_ON_MMX_INSTRUCTION_WRITE_TO_E_OR_M_STATE_LINE) #define PMC_EV_P5_FIRST PMC_EV_P5_DATA_READ #define PMC_EV_P5_LAST \ PMC_EV_P5_STALL_ON_MMX_INSTRUCTION_WRITE_TO_E_OR_M_STATE_LINE /* * Events supported by Intel architectural fixed function counters, * from the "Intel 64 and IA-32 Architectures Software Developer's * Manual Volume 3B: System Programming Guide, Part 2", July 2008. */ #define __PMC_EV_IAF() \ __PMC_EV(IAF, INSTR_RETIRED_ANY) \ __PMC_EV(IAF, CPU_CLK_UNHALTED_CORE) \ __PMC_EV(IAF, CPU_CLK_UNHALTED_REF) #define PMC_EV_IAF_FIRST PMC_EV_IAF_INSTR_RETIRED_ANY #define PMC_EV_IAF_LAST PMC_EV_IAF_CPU_CLK_UNHALTED_REF #define __PMC_EV_ALIAS_IAF() \ __PMC_EV_ALIAS("instruction-retired", IAF_INSTR_RETIRED_ANY) \ __PMC_EV_ALIAS("unhalted-core-cycles", IAF_CPU_CLK_UNHALTED_CORE) \ __PMC_EV_ALIAS("unhalted-reference-cycles", IAF_CPU_CLK_UNHALTED_REF) /* * Events supported by programmable function counters present in * Intel Atom, Core and Core2 CPUs, from the "Intel 64 and IA-32 * Architectures Software Developer's Manual Volume 3B: System Programming * Guide, Part 2", July 2008. * * These PMCs select events with a combination of an event code and * unit mask. Quirks that need to be taken care of include: * - The set of (event code, umask) combinations supported by a processor * varies according to the processor model. * - A given (event code, umask) combination need not measure the same * hardware event in all processor models. * - Event names in vendor documentation for an (event code, umask) pair * may vary according to the CPU model. * - Identically named events can map to different (event code, umask) * pairs on different CPUs. * - New (event code, umask) combinations continue to be added as CPUs * evolve. The interface between hwpmc(4) and libpmc(3) needs to be * robust with respect to ABI changes. * * The IAP_EVENT_* symbols below define the ABI between userland and kernel. * New (event code, * umask) combinations used in new CPUs would be added * to the end of the list. Vendor names for events are mapped to IAP_EVENT_* * symbols using aliases. The final disambiguation of semantics based on * the CPU model happens inside hwpmc(4). */ #define __PMC_EV_IAP() \ __PMC_EV(IAP, EVENT_02H_01H) \ __PMC_EV(IAP, EVENT_02H_81H) \ __PMC_EV(IAP, EVENT_03H_00H) \ __PMC_EV(IAP, EVENT_03H_01H) \ __PMC_EV(IAP, EVENT_03H_02H) \ __PMC_EV(IAP, EVENT_03H_04H) \ __PMC_EV(IAP, EVENT_03H_08H) \ __PMC_EV(IAP, EVENT_03H_10H) \ __PMC_EV(IAP, EVENT_03H_20H) \ __PMC_EV(IAP, EVENT_04H_00H) \ __PMC_EV(IAP, EVENT_04H_01H) \ __PMC_EV(IAP, EVENT_04H_02H) \ __PMC_EV(IAP, EVENT_04H_07H) \ __PMC_EV(IAP, EVENT_04H_08H) \ __PMC_EV(IAP, EVENT_05H_00H) \ __PMC_EV(IAP, EVENT_05H_01H) \ __PMC_EV(IAP, EVENT_05H_02H) \ __PMC_EV(IAP, EVENT_05H_03H) \ __PMC_EV(IAP, EVENT_06H_00H) \ __PMC_EV(IAP, EVENT_06H_01H) \ __PMC_EV(IAP, EVENT_06H_02H) \ __PMC_EV(IAP, EVENT_06H_04H) \ __PMC_EV(IAP, EVENT_06H_08H) \ __PMC_EV(IAP, EVENT_06H_0FH) \ __PMC_EV(IAP, EVENT_07H_00H) \ __PMC_EV(IAP, EVENT_07H_01H) \ __PMC_EV(IAP, EVENT_07H_02H) \ __PMC_EV(IAP, EVENT_07H_03H) \ __PMC_EV(IAP, EVENT_07H_06H) \ __PMC_EV(IAP, EVENT_07H_08H) \ __PMC_EV(IAP, EVENT_08H_01H) \ __PMC_EV(IAP, EVENT_08H_02H) \ __PMC_EV(IAP, EVENT_08H_04H) \ __PMC_EV(IAP, EVENT_08H_05H) \ __PMC_EV(IAP, EVENT_08H_06H) \ __PMC_EV(IAP, EVENT_08H_07H) \ __PMC_EV(IAP, EVENT_08H_08H) \ __PMC_EV(IAP, EVENT_08H_09H) \ __PMC_EV(IAP, EVENT_08H_10H) \ __PMC_EV(IAP, EVENT_08H_20H) \ __PMC_EV(IAP, EVENT_08H_40H) \ __PMC_EV(IAP, EVENT_08H_80H) \ __PMC_EV(IAP, EVENT_09H_01H) \ __PMC_EV(IAP, EVENT_09H_02H) \ __PMC_EV(IAP, EVENT_09H_04H) \ __PMC_EV(IAP, EVENT_09H_08H) \ __PMC_EV(IAP, EVENT_0BH_01H) \ __PMC_EV(IAP, EVENT_0BH_02H) \ __PMC_EV(IAP, EVENT_0BH_10H) \ __PMC_EV(IAP, EVENT_0CH_01H) \ __PMC_EV(IAP, EVENT_0CH_02H) \ __PMC_EV(IAP, EVENT_0CH_03H) \ __PMC_EV(IAP, EVENT_0EH_01H) \ __PMC_EV(IAP, EVENT_0EH_02H) \ __PMC_EV(IAP, EVENT_0FH_01H) \ __PMC_EV(IAP, EVENT_0FH_02H) \ __PMC_EV(IAP, EVENT_0FH_08H) \ __PMC_EV(IAP, EVENT_0FH_10H) \ __PMC_EV(IAP, EVENT_0FH_20H) \ __PMC_EV(IAP, EVENT_0FH_80H) \ __PMC_EV(IAP, EVENT_10H_00H) \ __PMC_EV(IAP, EVENT_10H_01H) \ __PMC_EV(IAP, EVENT_10H_02H) \ __PMC_EV(IAP, EVENT_10H_04H) \ __PMC_EV(IAP, EVENT_10H_08H) \ __PMC_EV(IAP, EVENT_10H_10H) \ __PMC_EV(IAP, EVENT_10H_20H) \ __PMC_EV(IAP, EVENT_10H_40H) \ __PMC_EV(IAP, EVENT_10H_80H) \ __PMC_EV(IAP, EVENT_10H_81H) \ __PMC_EV(IAP, EVENT_11H_00H) \ __PMC_EV(IAP, EVENT_11H_01H) \ __PMC_EV(IAP, EVENT_11H_81H) \ __PMC_EV(IAP, EVENT_12H_00H) \ __PMC_EV(IAP, EVENT_12H_01H) \ __PMC_EV(IAP, EVENT_12H_02H) \ __PMC_EV(IAP, EVENT_12H_04H) \ __PMC_EV(IAP, EVENT_12H_08H) \ __PMC_EV(IAP, EVENT_12H_10H) \ __PMC_EV(IAP, EVENT_12H_20H) \ __PMC_EV(IAP, EVENT_12H_40H) \ __PMC_EV(IAP, EVENT_12H_81H) \ __PMC_EV(IAP, EVENT_13H_00H) \ __PMC_EV(IAP, EVENT_13H_01H) \ __PMC_EV(IAP, EVENT_13H_02H) \ __PMC_EV(IAP, EVENT_13H_04H) \ __PMC_EV(IAP, EVENT_13H_07H) \ __PMC_EV(IAP, EVENT_13H_81H) \ __PMC_EV(IAP, EVENT_14H_00H) \ __PMC_EV(IAP, EVENT_14H_01H) \ __PMC_EV(IAP, EVENT_14H_02H) \ __PMC_EV(IAP, EVENT_17H_01H) \ __PMC_EV(IAP, EVENT_18H_00H) \ __PMC_EV(IAP, EVENT_18H_01H) \ __PMC_EV(IAP, EVENT_19H_00H) \ __PMC_EV(IAP, EVENT_19H_01H) \ __PMC_EV(IAP, EVENT_19H_02H) \ __PMC_EV(IAP, EVENT_1DH_01H) \ __PMC_EV(IAP, EVENT_1DH_02H) \ __PMC_EV(IAP, EVENT_1DH_04H) \ __PMC_EV(IAP, EVENT_1EH_01H) \ __PMC_EV(IAP, EVENT_20H_01H) \ __PMC_EV(IAP, EVENT_21H) \ __PMC_EV(IAP, EVENT_22H) \ __PMC_EV(IAP, EVENT_23H) \ __PMC_EV(IAP, EVENT_24H) \ __PMC_EV(IAP, EVENT_24H_01H) \ __PMC_EV(IAP, EVENT_24H_02H) \ __PMC_EV(IAP, EVENT_24H_03H) \ __PMC_EV(IAP, EVENT_24H_04H) \ __PMC_EV(IAP, EVENT_24H_08H) \ __PMC_EV(IAP, EVENT_24H_0CH) \ __PMC_EV(IAP, EVENT_24H_10H) \ __PMC_EV(IAP, EVENT_24H_20H) \ __PMC_EV(IAP, EVENT_24H_30H) \ __PMC_EV(IAP, EVENT_24H_40H) \ __PMC_EV(IAP, EVENT_24H_80H) \ __PMC_EV(IAP, EVENT_24H_AAH) \ __PMC_EV(IAP, EVENT_24H_C0H) \ __PMC_EV(IAP, EVENT_24H_FFH) \ __PMC_EV(IAP, EVENT_25H) \ __PMC_EV(IAP, EVENT_26H) \ __PMC_EV(IAP, EVENT_26H_01H) \ __PMC_EV(IAP, EVENT_26H_02H) \ __PMC_EV(IAP, EVENT_26H_04H) \ __PMC_EV(IAP, EVENT_26H_08H) \ __PMC_EV(IAP, EVENT_26H_0FH) \ __PMC_EV(IAP, EVENT_26H_10H) \ __PMC_EV(IAP, EVENT_26H_20H) \ __PMC_EV(IAP, EVENT_26H_40H) \ __PMC_EV(IAP, EVENT_26H_80H) \ __PMC_EV(IAP, EVENT_26H_F0H) \ __PMC_EV(IAP, EVENT_26H_FFH) \ __PMC_EV(IAP, EVENT_27H) \ __PMC_EV(IAP, EVENT_27H_01H) \ __PMC_EV(IAP, EVENT_27H_02H) \ __PMC_EV(IAP, EVENT_27H_04H) \ __PMC_EV(IAP, EVENT_27H_08H) \ __PMC_EV(IAP, EVENT_27H_0EH) \ __PMC_EV(IAP, EVENT_27H_0FH) \ __PMC_EV(IAP, EVENT_27H_10H) \ __PMC_EV(IAP, EVENT_27H_20H) \ __PMC_EV(IAP, EVENT_27H_40H) \ __PMC_EV(IAP, EVENT_27H_80H) \ __PMC_EV(IAP, EVENT_27H_E0H) \ __PMC_EV(IAP, EVENT_27H_F0H) \ __PMC_EV(IAP, EVENT_28H) \ __PMC_EV(IAP, EVENT_28H_01H) \ __PMC_EV(IAP, EVENT_28H_02H) \ __PMC_EV(IAP, EVENT_28H_04H) \ __PMC_EV(IAP, EVENT_28H_08H) \ __PMC_EV(IAP, EVENT_28H_0FH) \ __PMC_EV(IAP, EVENT_29H) \ __PMC_EV(IAP, EVENT_2AH) \ __PMC_EV(IAP, EVENT_2BH) \ __PMC_EV(IAP, EVENT_2EH) \ __PMC_EV(IAP, EVENT_2EH_01H) \ __PMC_EV(IAP, EVENT_2EH_02H) \ __PMC_EV(IAP, EVENT_2EH_41H) \ __PMC_EV(IAP, EVENT_2EH_4FH) \ __PMC_EV(IAP, EVENT_30H) \ __PMC_EV(IAP, EVENT_32H) \ __PMC_EV(IAP, EVENT_3AH) \ __PMC_EV(IAP, EVENT_3AH_00H) \ __PMC_EV(IAP, EVENT_3BH_C0H) \ __PMC_EV(IAP, EVENT_3CH_00H) \ __PMC_EV(IAP, EVENT_3CH_01H) \ __PMC_EV(IAP, EVENT_3CH_02H) \ __PMC_EV(IAP, EVENT_3DH_01H) \ __PMC_EV(IAP, EVENT_40H) \ __PMC_EV(IAP, EVENT_40H_01H) \ __PMC_EV(IAP, EVENT_40H_02H) \ __PMC_EV(IAP, EVENT_40H_04H) \ __PMC_EV(IAP, EVENT_40H_08H) \ __PMC_EV(IAP, EVENT_40H_0FH) \ __PMC_EV(IAP, EVENT_40H_21H) \ __PMC_EV(IAP, EVENT_41H) \ __PMC_EV(IAP, EVENT_41H_01H) \ __PMC_EV(IAP, EVENT_41H_02H) \ __PMC_EV(IAP, EVENT_41H_04H) \ __PMC_EV(IAP, EVENT_41H_08H) \ __PMC_EV(IAP, EVENT_41H_0FH) \ __PMC_EV(IAP, EVENT_41H_22H) \ __PMC_EV(IAP, EVENT_42H) \ __PMC_EV(IAP, EVENT_42H_01H) \ __PMC_EV(IAP, EVENT_42H_02H) \ __PMC_EV(IAP, EVENT_42H_04H) \ __PMC_EV(IAP, EVENT_42H_08H) \ __PMC_EV(IAP, EVENT_42H_10H) \ __PMC_EV(IAP, EVENT_43H_01H) \ __PMC_EV(IAP, EVENT_43H_02H) \ __PMC_EV(IAP, EVENT_44H_02H) \ __PMC_EV(IAP, EVENT_45H_0FH) \ __PMC_EV(IAP, EVENT_46H_00H) \ __PMC_EV(IAP, EVENT_47H_00H) \ __PMC_EV(IAP, EVENT_48H_00H) \ __PMC_EV(IAP, EVENT_48H_02H) \ __PMC_EV(IAP, EVENT_49H_00H) \ __PMC_EV(IAP, EVENT_49H_01H) \ __PMC_EV(IAP, EVENT_49H_02H) \ __PMC_EV(IAP, EVENT_49H_04H) \ __PMC_EV(IAP, EVENT_49H_10H) \ __PMC_EV(IAP, EVENT_49H_20H) \ __PMC_EV(IAP, EVENT_49H_40H) \ __PMC_EV(IAP, EVENT_49H_80H) \ __PMC_EV(IAP, EVENT_4BH_00H) \ __PMC_EV(IAP, EVENT_4BH_01H) \ __PMC_EV(IAP, EVENT_4BH_02H) \ __PMC_EV(IAP, EVENT_4BH_03H) \ __PMC_EV(IAP, EVENT_4BH_08H) \ __PMC_EV(IAP, EVENT_4CH_00H) \ __PMC_EV(IAP, EVENT_4CH_01H) \ __PMC_EV(IAP, EVENT_4DH_01H) \ __PMC_EV(IAP, EVENT_4EH_01H) \ __PMC_EV(IAP, EVENT_4EH_02H) \ __PMC_EV(IAP, EVENT_4EH_04H) \ __PMC_EV(IAP, EVENT_4EH_10H) \ __PMC_EV(IAP, EVENT_4FH_00H) \ __PMC_EV(IAP, EVENT_4FH_02H) \ __PMC_EV(IAP, EVENT_4FH_04H) \ __PMC_EV(IAP, EVENT_4FH_08H) \ __PMC_EV(IAP, EVENT_4FH_10H) \ __PMC_EV(IAP, EVENT_51H_01H) \ __PMC_EV(IAP, EVENT_51H_02H) \ __PMC_EV(IAP, EVENT_51H_04H) \ __PMC_EV(IAP, EVENT_51H_08H) \ __PMC_EV(IAP, EVENT_52H_01H) \ __PMC_EV(IAP, EVENT_53H_01H) \ __PMC_EV(IAP, EVENT_60H) \ __PMC_EV(IAP, EVENT_60H_01H) \ __PMC_EV(IAP, EVENT_60H_02H) \ __PMC_EV(IAP, EVENT_60H_04H) \ __PMC_EV(IAP, EVENT_60H_08H) \ __PMC_EV(IAP, EVENT_61H) \ __PMC_EV(IAP, EVENT_61H_00H) \ __PMC_EV(IAP, EVENT_62H) \ __PMC_EV(IAP, EVENT_62H_00H) \ __PMC_EV(IAP, EVENT_63H) \ __PMC_EV(IAP, EVENT_63H_01H) \ __PMC_EV(IAP, EVENT_63H_02H) \ __PMC_EV(IAP, EVENT_64H) \ __PMC_EV(IAP, EVENT_64H_40H) \ __PMC_EV(IAP, EVENT_65H) \ __PMC_EV(IAP, EVENT_66H) \ __PMC_EV(IAP, EVENT_67H) \ __PMC_EV(IAP, EVENT_68H) \ __PMC_EV(IAP, EVENT_69H) \ __PMC_EV(IAP, EVENT_6AH) \ __PMC_EV(IAP, EVENT_6BH) \ __PMC_EV(IAP, EVENT_6CH) \ __PMC_EV(IAP, EVENT_6CH_01H) \ __PMC_EV(IAP, EVENT_6DH) \ __PMC_EV(IAP, EVENT_6EH) \ __PMC_EV(IAP, EVENT_6FH) \ __PMC_EV(IAP, EVENT_70H) \ __PMC_EV(IAP, EVENT_77H) \ __PMC_EV(IAP, EVENT_78H) \ __PMC_EV(IAP, EVENT_7AH) \ __PMC_EV(IAP, EVENT_7BH) \ __PMC_EV(IAP, EVENT_7DH) \ __PMC_EV(IAP, EVENT_7EH) \ __PMC_EV(IAP, EVENT_7EH_00H) \ __PMC_EV(IAP, EVENT_7FH) \ __PMC_EV(IAP, EVENT_80H_00H) \ __PMC_EV(IAP, EVENT_80H_01H) \ __PMC_EV(IAP, EVENT_80H_02H) \ __PMC_EV(IAP, EVENT_80H_03H) \ __PMC_EV(IAP, EVENT_80H_04H) \ __PMC_EV(IAP, EVENT_80H_10H) \ __PMC_EV(IAP, EVENT_81H_00H) \ __PMC_EV(IAP, EVENT_81H_01H) \ __PMC_EV(IAP, EVENT_81H_02H) \ __PMC_EV(IAP, EVENT_82H_01H) \ __PMC_EV(IAP, EVENT_82H_02H) \ __PMC_EV(IAP, EVENT_82H_04H) \ __PMC_EV(IAP, EVENT_82H_10H) \ __PMC_EV(IAP, EVENT_82H_12H) \ __PMC_EV(IAP, EVENT_82H_40H) \ __PMC_EV(IAP, EVENT_83H_01H) \ __PMC_EV(IAP, EVENT_83H_02H) \ __PMC_EV(IAP, EVENT_85H_00H) \ __PMC_EV(IAP, EVENT_85H_01H) \ __PMC_EV(IAP, EVENT_85H_02H) \ __PMC_EV(IAP, EVENT_85H_04H) \ __PMC_EV(IAP, EVENT_85H_10H) \ __PMC_EV(IAP, EVENT_85H_20H) \ __PMC_EV(IAP, EVENT_85H_40H) \ __PMC_EV(IAP, EVENT_85H_80H) \ __PMC_EV(IAP, EVENT_86H_00H) \ __PMC_EV(IAP, EVENT_87H_00H) \ __PMC_EV(IAP, EVENT_87H_01H) \ __PMC_EV(IAP, EVENT_87H_02H) \ __PMC_EV(IAP, EVENT_87H_04H) \ __PMC_EV(IAP, EVENT_87H_08H) \ __PMC_EV(IAP, EVENT_87H_0FH) \ __PMC_EV(IAP, EVENT_88H_00H) \ __PMC_EV(IAP, EVENT_88H_01H) \ __PMC_EV(IAP, EVENT_88H_02H) \ __PMC_EV(IAP, EVENT_88H_04H) \ __PMC_EV(IAP, EVENT_88H_07H) \ __PMC_EV(IAP, EVENT_88H_08H) \ __PMC_EV(IAP, EVENT_88H_10H) \ __PMC_EV(IAP, EVENT_88H_20H) \ __PMC_EV(IAP, EVENT_88H_30H) \ __PMC_EV(IAP, EVENT_88H_40H) \ __PMC_EV(IAP, EVENT_88H_7FH) \ __PMC_EV(IAP, EVENT_89H_00H) \ __PMC_EV(IAP, EVENT_89H_01H) \ __PMC_EV(IAP, EVENT_89H_02H) \ __PMC_EV(IAP, EVENT_89H_04H) \ __PMC_EV(IAP, EVENT_89H_07H) \ __PMC_EV(IAP, EVENT_89H_08H) \ __PMC_EV(IAP, EVENT_89H_10H) \ __PMC_EV(IAP, EVENT_89H_20H) \ __PMC_EV(IAP, EVENT_89H_30H) \ __PMC_EV(IAP, EVENT_89H_40H) \ __PMC_EV(IAP, EVENT_89H_7FH) \ __PMC_EV(IAP, EVENT_8AH_00H) \ __PMC_EV(IAP, EVENT_8BH_00H) \ __PMC_EV(IAP, EVENT_8CH_00H) \ __PMC_EV(IAP, EVENT_8DH_00H) \ __PMC_EV(IAP, EVENT_8EH_00H) \ __PMC_EV(IAP, EVENT_8FH_00H) \ __PMC_EV(IAP, EVENT_90H_00H) \ __PMC_EV(IAP, EVENT_91H_00H) \ __PMC_EV(IAP, EVENT_92H_00H) \ __PMC_EV(IAP, EVENT_93H_00H) \ __PMC_EV(IAP, EVENT_94H_00H) \ __PMC_EV(IAP, EVENT_97H_00H) \ __PMC_EV(IAP, EVENT_98H_00H) \ __PMC_EV(IAP, EVENT_A0H_00H) \ __PMC_EV(IAP, EVENT_A1H_01H) \ __PMC_EV(IAP, EVENT_A1H_02H) \ __PMC_EV(IAP, EVENT_A1H_04H) \ __PMC_EV(IAP, EVENT_A1H_08H) \ __PMC_EV(IAP, EVENT_A1H_10H) \ __PMC_EV(IAP, EVENT_A1H_20H) \ __PMC_EV(IAP, EVENT_A2H_00H) \ __PMC_EV(IAP, EVENT_A2H_01H) \ __PMC_EV(IAP, EVENT_A2H_02H) \ __PMC_EV(IAP, EVENT_A2H_04H) \ __PMC_EV(IAP, EVENT_A2H_08H) \ __PMC_EV(IAP, EVENT_A2H_10H) \ __PMC_EV(IAP, EVENT_A2H_20H) \ __PMC_EV(IAP, EVENT_A2H_40H) \ __PMC_EV(IAP, EVENT_A2H_80H) \ __PMC_EV(IAP, EVENT_A6H_01H) \ __PMC_EV(IAP, EVENT_A7H_01H) \ __PMC_EV(IAP, EVENT_A8H_01H) \ __PMC_EV(IAP, EVENT_AAH_01H) \ __PMC_EV(IAP, EVENT_AAH_02H) \ __PMC_EV(IAP, EVENT_AAH_03H) \ __PMC_EV(IAP, EVENT_AAH_08H) \ __PMC_EV(IAP, EVENT_ABH_01H) \ __PMC_EV(IAP, EVENT_ABH_02H) \ __PMC_EV(IAP, EVENT_AEH_01H) \ __PMC_EV(IAP, EVENT_B0H_00H) \ __PMC_EV(IAP, EVENT_B0H_01H) \ __PMC_EV(IAP, EVENT_B0H_02H) \ __PMC_EV(IAP, EVENT_B0H_04H) \ __PMC_EV(IAP, EVENT_B0H_08H) \ __PMC_EV(IAP, EVENT_B0H_10H) \ __PMC_EV(IAP, EVENT_B0H_20H) \ __PMC_EV(IAP, EVENT_B0H_40H) \ __PMC_EV(IAP, EVENT_B0H_80H) \ __PMC_EV(IAP, EVENT_B1H_00H) \ __PMC_EV(IAP, EVENT_B1H_01H) \ __PMC_EV(IAP, EVENT_B1H_02H) \ __PMC_EV(IAP, EVENT_B1H_04H) \ __PMC_EV(IAP, EVENT_B1H_08H) \ __PMC_EV(IAP, EVENT_B1H_10H) \ __PMC_EV(IAP, EVENT_B1H_1FH) \ __PMC_EV(IAP, EVENT_B1H_20H) \ __PMC_EV(IAP, EVENT_B1H_3FH) \ __PMC_EV(IAP, EVENT_B1H_40H) \ __PMC_EV(IAP, EVENT_B1H_80H) \ __PMC_EV(IAP, EVENT_B2H_01H) \ __PMC_EV(IAP, EVENT_B3H_01H) \ __PMC_EV(IAP, EVENT_B3H_02H) \ __PMC_EV(IAP, EVENT_B3H_04H) \ __PMC_EV(IAP, EVENT_B3H_08H) \ __PMC_EV(IAP, EVENT_B3H_10H) \ __PMC_EV(IAP, EVENT_B3H_20H) \ __PMC_EV(IAP, EVENT_B3H_81H) \ __PMC_EV(IAP, EVENT_B3H_82H) \ __PMC_EV(IAP, EVENT_B3H_84H) \ __PMC_EV(IAP, EVENT_B3H_88H) \ __PMC_EV(IAP, EVENT_B3H_90H) \ __PMC_EV(IAP, EVENT_B3H_A0H) \ __PMC_EV(IAP, EVENT_B4H_01H) \ __PMC_EV(IAP, EVENT_B4H_02H) \ __PMC_EV(IAP, EVENT_B4H_04H) \ __PMC_EV(IAP, EVENT_B7H_01H) \ __PMC_EV(IAP, EVENT_B8H_01H) \ __PMC_EV(IAP, EVENT_B8H_02H) \ __PMC_EV(IAP, EVENT_B8H_04H) \ __PMC_EV(IAP, EVENT_BAH_01H) \ __PMC_EV(IAP, EVENT_BAH_02H) \ __PMC_EV(IAP, EVENT_BBH_01H) \ __PMC_EV(IAP, EVENT_C0H_00H) \ __PMC_EV(IAP, EVENT_C0H_01H) \ __PMC_EV(IAP, EVENT_C0H_02H) \ __PMC_EV(IAP, EVENT_C0H_04H) \ __PMC_EV(IAP, EVENT_C0H_08H) \ __PMC_EV(IAP, EVENT_C1H_00H) \ __PMC_EV(IAP, EVENT_C1H_01H) \ __PMC_EV(IAP, EVENT_C1H_FEH) \ __PMC_EV(IAP, EVENT_C2H_00H) \ __PMC_EV(IAP, EVENT_C2H_01H) \ __PMC_EV(IAP, EVENT_C2H_02H) \ __PMC_EV(IAP, EVENT_C2H_04H) \ __PMC_EV(IAP, EVENT_C2H_07H) \ __PMC_EV(IAP, EVENT_C2H_08H) \ __PMC_EV(IAP, EVENT_C2H_0FH) \ __PMC_EV(IAP, EVENT_C2H_10H) \ __PMC_EV(IAP, EVENT_C3H_00H) \ __PMC_EV(IAP, EVENT_C3H_01H) \ __PMC_EV(IAP, EVENT_C3H_02H) \ __PMC_EV(IAP, EVENT_C3H_04H) \ __PMC_EV(IAP, EVENT_C3H_10H) \ __PMC_EV(IAP, EVENT_C4H_00H) \ __PMC_EV(IAP, EVENT_C4H_01H) \ __PMC_EV(IAP, EVENT_C4H_02H) \ __PMC_EV(IAP, EVENT_C4H_04H) \ __PMC_EV(IAP, EVENT_C4H_08H) \ __PMC_EV(IAP, EVENT_C4H_0CH) \ __PMC_EV(IAP, EVENT_C4H_0FH) \ __PMC_EV(IAP, EVENT_C5H_00H) \ __PMC_EV(IAP, EVENT_C5H_01H) \ __PMC_EV(IAP, EVENT_C5H_02H) \ __PMC_EV(IAP, EVENT_C5H_04H) \ __PMC_EV(IAP, EVENT_C6H_00H) \ __PMC_EV(IAP, EVENT_C6H_01H) \ __PMC_EV(IAP, EVENT_C6H_02H) \ __PMC_EV(IAP, EVENT_C7H_00H) \ __PMC_EV(IAP, EVENT_C7H_01H) \ __PMC_EV(IAP, EVENT_C7H_02H) \ __PMC_EV(IAP, EVENT_C7H_04H) \ __PMC_EV(IAP, EVENT_C7H_08H) \ __PMC_EV(IAP, EVENT_C7H_10H) \ __PMC_EV(IAP, EVENT_C7H_1FH) \ __PMC_EV(IAP, EVENT_C8H_00H) \ __PMC_EV(IAP, EVENT_C8H_20H) \ __PMC_EV(IAP, EVENT_C9H_00H) \ __PMC_EV(IAP, EVENT_CAH_00H) \ __PMC_EV(IAP, EVENT_CAH_01H) \ __PMC_EV(IAP, EVENT_CAH_02H) \ __PMC_EV(IAP, EVENT_CAH_04H) \ __PMC_EV(IAP, EVENT_CAH_08H) \ __PMC_EV(IAP, EVENT_CBH_01H) \ __PMC_EV(IAP, EVENT_CBH_02H) \ __PMC_EV(IAP, EVENT_CBH_04H) \ __PMC_EV(IAP, EVENT_CBH_08H) \ __PMC_EV(IAP, EVENT_CBH_10H) \ __PMC_EV(IAP, EVENT_CBH_40H) \ __PMC_EV(IAP, EVENT_CBH_80H) \ __PMC_EV(IAP, EVENT_CCH_00H) \ __PMC_EV(IAP, EVENT_CCH_01H) \ __PMC_EV(IAP, EVENT_CCH_02H) \ __PMC_EV(IAP, EVENT_CCH_03H) \ __PMC_EV(IAP, EVENT_CDH_00H) \ __PMC_EV(IAP, EVENT_CEH_00H) \ __PMC_EV(IAP, EVENT_CFH_00H) \ __PMC_EV(IAP, EVENT_D0H_00H) \ __PMC_EV(IAP, EVENT_D0H_01H) \ __PMC_EV(IAP, EVENT_D1H_01H) \ __PMC_EV(IAP, EVENT_D1H_02H) \ __PMC_EV(IAP, EVENT_D1H_04H) \ __PMC_EV(IAP, EVENT_D1H_08H) \ __PMC_EV(IAP, EVENT_D2H_01H) \ __PMC_EV(IAP, EVENT_D2H_02H) \ __PMC_EV(IAP, EVENT_D2H_04H) \ __PMC_EV(IAP, EVENT_D2H_08H) \ __PMC_EV(IAP, EVENT_D2H_0FH) \ __PMC_EV(IAP, EVENT_D2H_10H) \ __PMC_EV(IAP, EVENT_D4H_01H) \ __PMC_EV(IAP, EVENT_D4H_02H) \ __PMC_EV(IAP, EVENT_D4H_04H) \ __PMC_EV(IAP, EVENT_D4H_08H) \ __PMC_EV(IAP, EVENT_D4H_0FH) \ __PMC_EV(IAP, EVENT_D5H_01H) \ __PMC_EV(IAP, EVENT_D5H_02H) \ __PMC_EV(IAP, EVENT_D5H_04H) \ __PMC_EV(IAP, EVENT_D5H_08H) \ __PMC_EV(IAP, EVENT_D5H_0FH) \ __PMC_EV(IAP, EVENT_D7H_00H) \ __PMC_EV(IAP, EVENT_D8H_00H) \ __PMC_EV(IAP, EVENT_D8H_01H) \ __PMC_EV(IAP, EVENT_D8H_02H) \ __PMC_EV(IAP, EVENT_D8H_03H) \ __PMC_EV(IAP, EVENT_D8H_04H) \ __PMC_EV(IAP, EVENT_D9H_00H) \ __PMC_EV(IAP, EVENT_D9H_01H) \ __PMC_EV(IAP, EVENT_D9H_02H) \ __PMC_EV(IAP, EVENT_D9H_03H) \ __PMC_EV(IAP, EVENT_DAH_00H) \ __PMC_EV(IAP, EVENT_DAH_01H) \ __PMC_EV(IAP, EVENT_DAH_02H) \ __PMC_EV(IAP, EVENT_DBH_00H) \ __PMC_EV(IAP, EVENT_DBH_01H) \ __PMC_EV(IAP, EVENT_DCH_01H) \ __PMC_EV(IAP, EVENT_DCH_02H) \ __PMC_EV(IAP, EVENT_DCH_04H) \ __PMC_EV(IAP, EVENT_DCH_08H) \ __PMC_EV(IAP, EVENT_DCH_10H) \ __PMC_EV(IAP, EVENT_DCH_1FH) \ __PMC_EV(IAP, EVENT_E0H_00H) \ __PMC_EV(IAP, EVENT_E0H_01H) \ __PMC_EV(IAP, EVENT_E2H_00H) \ __PMC_EV(IAP, EVENT_E4H_00H) \ __PMC_EV(IAP, EVENT_E4H_01H) \ __PMC_EV(IAP, EVENT_E5H_01H) \ __PMC_EV(IAP, EVENT_E6H_00H) \ __PMC_EV(IAP, EVENT_E6H_01H) \ __PMC_EV(IAP, EVENT_E6H_02H) \ __PMC_EV(IAP, EVENT_E8H_01H) \ __PMC_EV(IAP, EVENT_E8H_02H) \ __PMC_EV(IAP, EVENT_E8H_03H) \ __PMC_EV(IAP, EVENT_ECH_01H) \ __PMC_EV(IAP, EVENT_F0H_00H) \ __PMC_EV(IAP, EVENT_F0H_01H) \ __PMC_EV(IAP, EVENT_F0H_02H) \ __PMC_EV(IAP, EVENT_F0H_04H) \ __PMC_EV(IAP, EVENT_F0H_08H) \ __PMC_EV(IAP, EVENT_F0H_10H) \ __PMC_EV(IAP, EVENT_F0H_20H) \ __PMC_EV(IAP, EVENT_F0H_40H) \ __PMC_EV(IAP, EVENT_F0H_80H) \ __PMC_EV(IAP, EVENT_F1H_02H) \ __PMC_EV(IAP, EVENT_F1H_04H) \ __PMC_EV(IAP, EVENT_F1H_07H) \ __PMC_EV(IAP, EVENT_F2H_01H) \ __PMC_EV(IAP, EVENT_F2H_02H) \ __PMC_EV(IAP, EVENT_F2H_04H) \ __PMC_EV(IAP, EVENT_F2H_08H) \ __PMC_EV(IAP, EVENT_F2H_0FH) \ __PMC_EV(IAP, EVENT_F3H_01H) \ __PMC_EV(IAP, EVENT_F3H_02H) \ __PMC_EV(IAP, EVENT_F3H_04H) \ __PMC_EV(IAP, EVENT_F3H_08H) \ __PMC_EV(IAP, EVENT_F3H_10H) \ __PMC_EV(IAP, EVENT_F3H_20H) \ __PMC_EV(IAP, EVENT_F4H_01H) \ __PMC_EV(IAP, EVENT_F4H_02H) \ __PMC_EV(IAP, EVENT_F4H_04H) \ __PMC_EV(IAP, EVENT_F4H_08H) \ __PMC_EV(IAP, EVENT_F4H_10H) \ __PMC_EV(IAP, EVENT_F6H_01H) \ __PMC_EV(IAP, EVENT_F7H_01H) \ __PMC_EV(IAP, EVENT_F7H_02H) \ __PMC_EV(IAP, EVENT_F7H_04H) \ __PMC_EV(IAP, EVENT_F8H_00H) \ __PMC_EV(IAP, EVENT_F8H_01H) \ __PMC_EV(IAP, EVENT_FDH_01H) \ __PMC_EV(IAP, EVENT_FDH_02H) \ __PMC_EV(IAP, EVENT_FDH_04H) \ __PMC_EV(IAP, EVENT_FDH_08H) \ __PMC_EV(IAP, EVENT_FDH_10H) \ __PMC_EV(IAP, EVENT_FDH_20H) \ __PMC_EV(IAP, EVENT_FDH_40H) #define PMC_EV_IAP_FIRST PMC_EV_IAP_EVENT_02H_01H #define PMC_EV_IAP_LAST PMC_EV_IAP_EVENT_FDH_40H /* * Map "architectural" event names to event ids. */ #define __PMC_EV_ALIAS_INTEL_ARCHITECTURAL() \ __PMC_EV_ALIAS("branch-instruction-retired", IAP_EVENT_C4H_00H) \ __PMC_EV_ALIAS("branch-misses-retired", IAP_EVENT_C5H_00H) \ __PMC_EV_ALIAS("instruction-retired", IAP_EVENT_C0H_00H) \ __PMC_EV_ALIAS("llc-misses", IAP_EVENT_2EH_41H) \ __PMC_EV_ALIAS("llc-reference", IAP_EVENT_2EH_4FH) \ __PMC_EV_ALIAS("unhalted-reference-cycles", IAP_EVENT_3CH_01H) \ __PMC_EV_ALIAS("unhalted-core-cycles", IAP_EVENT_3CH_00H) /* * Aliases for Atom PMCs. */ #define __PMC_EV_ALIAS_ATOM() \ __PMC_EV_ALIAS_INTEL_ARCHITECTURAL() \ __PMC_EV_ALIAS("BACLEARS", IAP_EVENT_E6H_01H) \ __PMC_EV_ALIAS("BOGUS_BR", IAP_EVENT_E4H_00H) \ __PMC_EV_ALIAS("BR_BAC_MISSP_EXEC", IAP_EVENT_8AH_00H) \ __PMC_EV_ALIAS("BR_CALL_EXEC", IAP_EVENT_92H_00H) \ __PMC_EV_ALIAS("BR_CALL_MISSP_EXEC", IAP_EVENT_93H_00H) \ __PMC_EV_ALIAS("BR_CND_EXEC", IAP_EVENT_8BH_00H) \ __PMC_EV_ALIAS("BR_CND_MISSP_EXEC", IAP_EVENT_8CH_00H) \ __PMC_EV_ALIAS("BR_IND_CALL_EXEC", IAP_EVENT_94H_00H) \ __PMC_EV_ALIAS("BR_IND_EXEC", IAP_EVENT_8DH_00H) \ __PMC_EV_ALIAS("BR_IND_MISSP_EXEC", IAP_EVENT_8EH_00H) \ __PMC_EV_ALIAS("BR_INST_DECODED", IAP_EVENT_E0H_01H) \ __PMC_EV_ALIAS("BR_INST_EXEC", IAP_EVENT_88H_00H) \ __PMC_EV_ALIAS("BR_INST_RETIRED.ANY", IAP_EVENT_C4H_00H) \ __PMC_EV_ALIAS("BR_INST_RETIRED.ANY1", IAP_EVENT_C4H_0FH) \ __PMC_EV_ALIAS("BR_INST_RETIRED.MISPRED", IAP_EVENT_C5H_00H) \ __PMC_EV_ALIAS("BR_INST_RETIRED.MISPRED_NOT_TAKEN", \ IAP_EVENT_C4H_02H) \ __PMC_EV_ALIAS("BR_INST_RETIRED.MISPRED_TAKEN", IAP_EVENT_C4H_08H) \ __PMC_EV_ALIAS("BR_INST_RETIRED.PRED_NOT_TAKEN",IAP_EVENT_C4H_01H) \ __PMC_EV_ALIAS("BR_INST_RETIRED.PRED_TAKEN", IAP_EVENT_C4H_04H) \ __PMC_EV_ALIAS("BR_INST_RETIRED.TAKEN", IAP_EVENT_C4H_0CH) \ __PMC_EV_ALIAS("BR_MISSP_EXEC", IAP_EVENT_89H_00H) \ __PMC_EV_ALIAS("BR_RET_BAC_MISSP_EXEC", IAP_EVENT_91H_00H) \ __PMC_EV_ALIAS("BR_RET_EXEC", IAP_EVENT_8FH_00H) \ __PMC_EV_ALIAS("BR_RET_MISSP_EXEC", IAP_EVENT_90H_00H) \ __PMC_EV_ALIAS("BR_TKN_BUBBLE_1", IAP_EVENT_97H_00H) \ __PMC_EV_ALIAS("BR_TKN_BUBBLE_2", IAP_EVENT_98H_00H) \ __PMC_EV_ALIAS("BUSQ_EMPTY", IAP_EVENT_7DH) \ __PMC_EV_ALIAS("BUS_BNR_DRV", IAP_EVENT_61H) \ __PMC_EV_ALIAS("BUS_DATA_RCV", IAP_EVENT_64H) \ __PMC_EV_ALIAS("BUS_DRDY_CLOCKS", IAP_EVENT_62H) \ __PMC_EV_ALIAS("BUS_HITM_DRV", IAP_EVENT_7BH) \ __PMC_EV_ALIAS("BUS_HIT_DRV", IAP_EVENT_7AH) \ __PMC_EV_ALIAS("BUS_IO_WAIT", IAP_EVENT_7FH) \ __PMC_EV_ALIAS("BUS_LOCK_CLOCKS", IAP_EVENT_63H) \ __PMC_EV_ALIAS("BUS_REQUEST_OUTSTANDING", IAP_EVENT_60H) \ __PMC_EV_ALIAS("BUS_TRANS_ANY", IAP_EVENT_70H) \ __PMC_EV_ALIAS("BUS_TRANS_BRD", IAP_EVENT_65H) \ __PMC_EV_ALIAS("BUS_TRANS_BURST", IAP_EVENT_6EH) \ __PMC_EV_ALIAS("BUS_TRANS_DEF", IAP_EVENT_6DH) \ __PMC_EV_ALIAS("BUS_TRANS_IFETCH", IAP_EVENT_68H) \ __PMC_EV_ALIAS("BUS_TRANS_INVAL", IAP_EVENT_69H) \ __PMC_EV_ALIAS("BUS_TRANS_IO", IAP_EVENT_6CH) \ __PMC_EV_ALIAS("BUS_TRANS_MEM", IAP_EVENT_6FH) \ __PMC_EV_ALIAS("BUS_TRANS_P", IAP_EVENT_6BH) \ __PMC_EV_ALIAS("BUS_TRANS_PWR", IAP_EVENT_6AH) \ __PMC_EV_ALIAS("BUS_TRANS_RFO", IAP_EVENT_66H) \ __PMC_EV_ALIAS("BUS_TRANS_WB", IAP_EVENT_67H) \ __PMC_EV_ALIAS("CMP_SNOOP", IAP_EVENT_78H) \ __PMC_EV_ALIAS("CPU_CLK_UNHALTED.BUS", IAP_EVENT_3CH_01H) \ __PMC_EV_ALIAS("CPU_CLK_UNHALTED.CORE_P", IAP_EVENT_3CH_00H) \ __PMC_EV_ALIAS("CPU_CLK_UNHALTED.NO_OTHER", IAP_EVENT_3CH_02H) \ __PMC_EV_ALIAS("CYCLES_DIV_BUSY", IAP_EVENT_14H_01H) \ __PMC_EV_ALIAS("CYCLES_INT_MASKED.CYCLES_INT_MASKED", \ IAP_EVENT_C6H_01H) \ __PMC_EV_ALIAS("CYCLES_INT_MASKED.CYCLES_INT_PENDING_AND_MASKED", \ IAP_EVENT_C6H_02H) \ __PMC_EV_ALIAS("CYCLES_L1I_MEM_STALLED", IAP_EVENT_86H_00H) \ __PMC_EV_ALIAS("DATA_TLB_MISSES.DTLB_MISS", IAP_EVENT_08H_07H) \ __PMC_EV_ALIAS("DATA_TLB_MISSES.DTLB_MISS_LD", IAP_EVENT_08H_05H) \ __PMC_EV_ALIAS("DATA_TLB_MISSES.DTLB_MISS_ST", IAP_EVENT_08H_06H) \ __PMC_EV_ALIAS("DATA_TLB_MISSES.UTLB_MISS_LD", IAP_EVENT_08H_09H) \ __PMC_EV_ALIAS("DELAYED_BYPASS.FP", IAP_EVENT_19H_00H) \ __PMC_EV_ALIAS("DELAYED_BYPASS.LOAD", IAP_EVENT_19H_01H) \ __PMC_EV_ALIAS("DELAYED_BYPASS.SIMD", IAP_EVENT_19H_02H) \ __PMC_EV_ALIAS("DIV", IAP_EVENT_13H_00H) \ __PMC_EV_ALIAS("DIV.AR", IAP_EVENT_13H_81H) \ __PMC_EV_ALIAS("DIV.S", IAP_EVENT_13H_01H) \ __PMC_EV_ALIAS("DTLB_MISSES.ANY", IAP_EVENT_08H_01H) \ __PMC_EV_ALIAS("DTLB_MISSES.L0_MISS_LD", IAP_EVENT_08H_04H) \ __PMC_EV_ALIAS("DTLB_MISSES.MISS_LD", IAP_EVENT_08H_02H) \ __PMC_EV_ALIAS("DTLB_MISSES.MISS_ST", IAP_EVENT_08H_08H) \ __PMC_EV_ALIAS("EIST_TRANS", IAP_EVENT_3AH_00H) \ __PMC_EV_ALIAS("ESP.ADDITIONS", IAP_EVENT_ABH_02H) \ __PMC_EV_ALIAS("ESP.SYNCH", IAP_EVENT_ABH_01H) \ __PMC_EV_ALIAS("EXT_SNOOP", IAP_EVENT_77H) \ __PMC_EV_ALIAS("FP_ASSIST", IAP_EVENT_11H_01H) \ __PMC_EV_ALIAS("FP_ASSIST.AR", IAP_EVENT_11H_81H) \ __PMC_EV_ALIAS("FP_COMP_OPS_EXE", IAP_EVENT_10H_00H) \ __PMC_EV_ALIAS("FP_MMX_TRANS_TO_FP", IAP_EVENT_CCH_02H) \ __PMC_EV_ALIAS("FP_MMX_TRANS_TO_MMX", IAP_EVENT_CCH_01H) \ __PMC_EV_ALIAS("HW_INT_RCV", IAP_EVENT_C8H_00H) \ __PMC_EV_ALIAS("ICACHE.ACCESSES", IAP_EVENT_80H_03H) \ __PMC_EV_ALIAS("ICACHE.MISSES", IAP_EVENT_80H_02H) \ __PMC_EV_ALIAS("IDLE_DURING_DIV", IAP_EVENT_18H_00H) \ __PMC_EV_ALIAS("ILD_STALL", IAP_EVENT_87H_00H) \ __PMC_EV_ALIAS("INST_QUEUE.FULL", IAP_EVENT_83H_02H) \ __PMC_EV_ALIAS("INST_RETIRED.ANY_P", IAP_EVENT_C0H_00H) \ __PMC_EV_ALIAS("INST_RETIRED.LOADS", IAP_EVENT_C0H_01H) \ __PMC_EV_ALIAS("INST_RETIRED.OTHER", IAP_EVENT_C0H_04H) \ __PMC_EV_ALIAS("INST_RETIRED.STORES", IAP_EVENT_C0H_02H) \ __PMC_EV_ALIAS("ITLB.FLUSH", IAP_EVENT_82H_04H) \ __PMC_EV_ALIAS("ITLB.LARGE_MISS", IAP_EVENT_82H_10H) \ __PMC_EV_ALIAS("ITLB.MISSES", IAP_EVENT_82H_02H) \ __PMC_EV_ALIAS("ITLB.SMALL_MISS", IAP_EVENT_82H_02H) \ __PMC_EV_ALIAS("ITLB_MISS_RETIRED", IAP_EVENT_C9H_00H) \ __PMC_EV_ALIAS("L1D_ALL_CACHE_REF", IAP_EVENT_43H_02H) \ __PMC_EV_ALIAS("L1D_ALL_REF", IAP_EVENT_43H_01H) \ __PMC_EV_ALIAS("L1D_CACHE.LD", IAP_EVENT_40H_21H) \ __PMC_EV_ALIAS("L1D_CACHE.ST", IAP_EVENT_41H_22H) \ __PMC_EV_ALIAS("L1D_CACHE_LOCK", IAP_EVENT_42H) \ __PMC_EV_ALIAS("L1D_CACHE_LOCK_DURATION", IAP_EVENT_42H_10H) \ __PMC_EV_ALIAS("L1D_M_EVICT", IAP_EVENT_47H_00H) \ __PMC_EV_ALIAS("L1D_M_REPL", IAP_EVENT_46H_00H) \ __PMC_EV_ALIAS("L1D_PEND_MISS", IAP_EVENT_48H_00H) \ __PMC_EV_ALIAS("L1D_PREFETCH.REQUESTS", IAP_EVENT_4EH_10H) \ __PMC_EV_ALIAS("L1D_REPL", IAP_EVENT_45H_0FH) \ __PMC_EV_ALIAS("L1D_SPLIT.LOADS", IAP_EVENT_49H_01H) \ __PMC_EV_ALIAS("L1D_SPLIT.STORES", IAP_EVENT_49H_02H) \ __PMC_EV_ALIAS("L1I_MISSES", IAP_EVENT_81H_00H) \ __PMC_EV_ALIAS("L1I_READS", IAP_EVENT_80H_00H) \ __PMC_EV_ALIAS("L2_ADS", IAP_EVENT_21H) \ __PMC_EV_ALIAS("L2_DBUS_BUSY_RD", IAP_EVENT_23H) \ __PMC_EV_ALIAS("L2_IFETCH", IAP_EVENT_28H) \ __PMC_EV_ALIAS("L2_LD", IAP_EVENT_29H) \ __PMC_EV_ALIAS("L2_LINES_IN", IAP_EVENT_24H) \ __PMC_EV_ALIAS("L2_LINES_OUT", IAP_EVENT_26H) \ __PMC_EV_ALIAS("L2_LOCK", IAP_EVENT_2BH) \ __PMC_EV_ALIAS("L2_M_LINES_IN", IAP_EVENT_25H) \ __PMC_EV_ALIAS("L2_M_LINES_OUT", IAP_EVENT_27H) \ __PMC_EV_ALIAS("L2_NO_REQ", IAP_EVENT_32H) \ __PMC_EV_ALIAS("L2_REJECT_BUSQ", IAP_EVENT_30H) \ __PMC_EV_ALIAS("L2_RQSTS", IAP_EVENT_2EH) \ __PMC_EV_ALIAS("L2_RQSTS.SELF.DEMAND.I_STATE", IAP_EVENT_2EH_41H) \ __PMC_EV_ALIAS("L2_RQSTS.SELF.DEMAND.MESI", IAP_EVENT_2EH_4FH) \ __PMC_EV_ALIAS("L2_ST", IAP_EVENT_2AH) \ __PMC_EV_ALIAS("LOAD_BLOCK.L1D", IAP_EVENT_03H_20H) \ __PMC_EV_ALIAS("LOAD_BLOCK.OVERLAP_STORE", IAP_EVENT_03H_08H) \ __PMC_EV_ALIAS("LOAD_BLOCK.STA", IAP_EVENT_03H_02H) \ __PMC_EV_ALIAS("LOAD_BLOCK.STD", IAP_EVENT_03H_04H) \ __PMC_EV_ALIAS("LOAD_BLOCK.UNTIL_RETIRE", IAP_EVENT_03H_10H) \ __PMC_EV_ALIAS("LOAD_HIT_PRE", IAP_EVENT_4CH_00H) \ __PMC_EV_ALIAS("MACHINE_CLEARS.SMC", IAP_EVENT_C3H_01H) \ __PMC_EV_ALIAS("MACHINE_NUKES.MEM_ORDER", IAP_EVENT_C3H_04H) \ __PMC_EV_ALIAS("MACRO_INSTS.ALL_DECODED", IAP_EVENT_AAH_03H) \ __PMC_EV_ALIAS("MACRO_INSTS.CISC_DECODED", IAP_EVENT_AAH_02H) \ __PMC_EV_ALIAS("MEMORY_DISAMBIGUATION.RESET", IAP_EVENT_09H_01H) \ __PMC_EV_ALIAS("MEMORY_DISAMBIGUATION.SUCCESS", IAP_EVENT_09H_02H) \ __PMC_EV_ALIAS("MEM_LOAD_RETIRED.DTLB_MISS", IAP_EVENT_CBH_04H) \ __PMC_EV_ALIAS("MEM_LOAD_RETIRED.L2_HIT", IAP_EVENT_CBH_01H) \ __PMC_EV_ALIAS("MEM_LOAD_RETIRED.L2_LINE_MISS", IAP_EVENT_CBH_08H) \ __PMC_EV_ALIAS("MEM_LOAD_RETIRED.L2_MISS", IAP_EVENT_CBH_02H) \ __PMC_EV_ALIAS("MUL", IAP_EVENT_12H_00H) \ __PMC_EV_ALIAS("MUL.AR", IAP_EVENT_12H_81H) \ __PMC_EV_ALIAS("MUL.S", IAP_EVENT_12H_01H) \ __PMC_EV_ALIAS("PAGE_WALKS.CYCLES", IAP_EVENT_0CH_03H) \ __PMC_EV_ALIAS("PAGE_WALKS.WALKS", IAP_EVENT_0CH_03H) \ __PMC_EV_ALIAS("PREFETCH.PREFETCHNTA", IAP_EVENT_07H_08H) \ __PMC_EV_ALIAS("PREFETCH.PREFETCHT0", IAP_EVENT_07H_01H) \ __PMC_EV_ALIAS("PREFETCH.SW_L2", IAP_EVENT_07H_06H) \ __PMC_EV_ALIAS("PREF_RQSTS_DN", IAP_EVENT_F8H_00H) \ __PMC_EV_ALIAS("PREF_RQSTS_UP", IAP_EVENT_F0H_00H) \ __PMC_EV_ALIAS("RAT_STALLS.ANY", IAP_EVENT_D2H_0FH) \ __PMC_EV_ALIAS("RAT_STALLS.FLAGS", IAP_EVENT_D2H_04H) \ __PMC_EV_ALIAS("RAT_STALLS.FPSW", IAP_EVENT_D2H_08H) \ __PMC_EV_ALIAS("RAT_STALLS.PARTIAL_CYCLES", IAP_EVENT_D2H_02H) \ __PMC_EV_ALIAS("RAT_STALLS.ROB_READ_PORT", IAP_EVENT_D2H_01H) \ __PMC_EV_ALIAS("RESOURCE_STALLS.ANY", IAP_EVENT_DCH_1FH) \ __PMC_EV_ALIAS("RESOURCE_STALLS.BR_MISS_CLEAR", IAP_EVENT_DCH_10H) \ __PMC_EV_ALIAS("RESOURCE_STALLS.FPCW", IAP_EVENT_DCH_08H) \ __PMC_EV_ALIAS("RESOURCE_STALLS.LD_ST", IAP_EVENT_DCH_04H) \ __PMC_EV_ALIAS("RESOURCE_STALLS.ROB_FULL", IAP_EVENT_DCH_01H) \ __PMC_EV_ALIAS("RESOURCE_STALLS.RS_FULL", IAP_EVENT_DCH_02H) \ __PMC_EV_ALIAS("RS_UOPS_DISPATCHED", IAP_EVENT_A0H_00H) \ __PMC_EV_ALIAS("RS_UOPS_DISPATCHED.PORT0", IAP_EVENT_A1H_01H) \ __PMC_EV_ALIAS("RS_UOPS_DISPATCHED.PORT1", IAP_EVENT_A1H_02H) \ __PMC_EV_ALIAS("RS_UOPS_DISPATCHED.PORT2", IAP_EVENT_A1H_04H) \ __PMC_EV_ALIAS("RS_UOPS_DISPATCHED.PORT3", IAP_EVENT_A1H_08H) \ __PMC_EV_ALIAS("RS_UOPS_DISPATCHED.PORT4", IAP_EVENT_A1H_10H) \ __PMC_EV_ALIAS("RS_UOPS_DISPATCHED.PORT5", IAP_EVENT_A1H_20H) \ __PMC_EV_ALIAS("SB_DRAIN_CYCLES", IAP_EVENT_04H_01H) \ __PMC_EV_ALIAS("SEGMENT_REG_LOADS.ANY", IAP_EVENT_06H_00H) \ __PMC_EV_ALIAS("SEG_REG_RENAMES.ANY", IAP_EVENT_D5H_0FH) \ __PMC_EV_ALIAS("SEG_REG_RENAMES.DS", IAP_EVENT_D5H_02H) \ __PMC_EV_ALIAS("SEG_REG_RENAMES.ES", IAP_EVENT_D5H_01H) \ __PMC_EV_ALIAS("SEG_REG_RENAMES.FS", IAP_EVENT_D5H_04H) \ __PMC_EV_ALIAS("SEG_REG_RENAMES.GS", IAP_EVENT_D5H_08H) \ __PMC_EV_ALIAS("SEG_RENAME_STALLS.ANY", IAP_EVENT_D4H_0FH) \ __PMC_EV_ALIAS("SEG_RENAME_STALLS.DS", IAP_EVENT_D4H_02H) \ __PMC_EV_ALIAS("SEG_RENAME_STALLS.ES", IAP_EVENT_D4H_01H) \ __PMC_EV_ALIAS("SEG_RENAME_STALLS.FS", IAP_EVENT_D4H_04H) \ __PMC_EV_ALIAS("SEG_RENAME_STALLS.GS", IAP_EVENT_D4H_08H) \ __PMC_EV_ALIAS("SIMD_ASSIST", IAP_EVENT_CDH_00H) \ __PMC_EV_ALIAS("SIMD_COMP_INST_RETIRED.PACKED_DOUBLE", \ IAP_EVENT_CAH_04H) \ __PMC_EV_ALIAS("SIMD_COMP_INST_RETIRED.PACKED_SINGLE", \ IAP_EVENT_CAH_01H) \ __PMC_EV_ALIAS("SIMD_COMP_INST_RETIRED.SCALAR_DOUBLE", \ IAP_EVENT_CAH_08H) \ __PMC_EV_ALIAS("SIMD_COMP_INST_RETIRED.SCALAR_SINGLE", \ IAP_EVENT_CAH_02H) \ __PMC_EV_ALIAS("SIMD_INSTR_RETIRED", IAP_EVENT_CEH_00H) \ __PMC_EV_ALIAS("SIMD_INST_RETIRED.ANY", IAP_EVENT_C7H_1FH) \ __PMC_EV_ALIAS("SIMD_INST_RETIRED.PACKED_DOUBLE", IAP_EVENT_C7H_04H) \ __PMC_EV_ALIAS("SIMD_INST_RETIRED.PACKED_SINGLE", IAP_EVENT_C7H_01H) \ __PMC_EV_ALIAS("SIMD_INST_RETIRED.SCALAR_DOUBLE", IAP_EVENT_C7H_08H) \ __PMC_EV_ALIAS("SIMD_INST_RETIRED.SCALAR_SINGLE", IAP_EVENT_C7H_02H) \ __PMC_EV_ALIAS("SIMD_INST_RETIRED.VECTOR", IAP_EVENT_C7H_10H) \ __PMC_EV_ALIAS("SIMD_SAT_INSTR_RETIRED", IAP_EVENT_CFH_00H) \ __PMC_EV_ALIAS("SIMD_SAT_UOP_EXEC.AR", IAP_EVENT_B1H_80H) \ __PMC_EV_ALIAS("SIMD_SAT_UOP_EXEC.S", IAP_EVENT_B1H_00H) \ __PMC_EV_ALIAS("SIMD_UOPS_EXEC.AR", IAP_EVENT_B0H_80H) \ __PMC_EV_ALIAS("SIMD_UOPS_EXEC.S", IAP_EVENT_B0H_00H) \ __PMC_EV_ALIAS("SIMD_UOP_TYPE_EXEC.ARITHMETIC.AR", IAP_EVENT_B3H_A0H) \ __PMC_EV_ALIAS("SIMD_UOP_TYPE_EXEC.ARITHMETIC.S", IAP_EVENT_B3H_20H) \ __PMC_EV_ALIAS("SIMD_UOP_TYPE_EXEC.LOGICAL.AR", IAP_EVENT_B3H_90H) \ __PMC_EV_ALIAS("SIMD_UOP_TYPE_EXEC.LOGICAL.S", IAP_EVENT_B3H_10H) \ __PMC_EV_ALIAS("SIMD_UOP_TYPE_EXEC.MUL.AR", IAP_EVENT_B3H_81H) \ __PMC_EV_ALIAS("SIMD_UOP_TYPE_EXEC.MUL.S", IAP_EVENT_B3H_01H) \ __PMC_EV_ALIAS("SIMD_UOP_TYPE_EXEC.PACK.AR", IAP_EVENT_B3H_84H) \ __PMC_EV_ALIAS("SIMD_UOP_TYPE_EXEC.PACK.S", IAP_EVENT_B3H_04H) \ __PMC_EV_ALIAS("SIMD_UOP_TYPE_EXEC.SHIFT.AR", IAP_EVENT_B3H_82H) \ __PMC_EV_ALIAS("SIMD_UOP_TYPE_EXEC.SHIFT.S", IAP_EVENT_B3H_02H) \ __PMC_EV_ALIAS("SIMD_UOP_TYPE_EXEC.UNPACK.AR", IAP_EVENT_B3H_88H) \ __PMC_EV_ALIAS("SIMD_UOP_TYPE_EXEC.UNPACK.S", IAP_EVENT_B3H_08H) \ __PMC_EV_ALIAS("SNOOP_STALL_DRV", IAP_EVENT_7EH) \ __PMC_EV_ALIAS("SSE_PRE_EXEC.L2", IAP_EVENT_07H_02H) \ __PMC_EV_ALIAS("SSE_PRE_EXEC.STORES", IAP_EVENT_07H_03H) \ __PMC_EV_ALIAS("SSE_PRE_MISS.L1", IAP_EVENT_4BH_01H) \ __PMC_EV_ALIAS("SSE_PRE_MISS.L2", IAP_EVENT_4BH_02H) \ __PMC_EV_ALIAS("SSE_PRE_MISS.NTA", IAP_EVENT_4BH_00H) \ __PMC_EV_ALIAS("STORE_BLOCK.ORDER", IAP_EVENT_04H_02H) \ __PMC_EV_ALIAS("STORE_BLOCK.SNOOP", IAP_EVENT_04H_08H) \ __PMC_EV_ALIAS("STORE_FORWARDS.GOOD", IAP_EVENT_02H_81H) \ __PMC_EV_ALIAS("THERMAL_TRIP", IAP_EVENT_3BH_C0H) \ __PMC_EV_ALIAS("UOPS_RETIRED.ANY", IAP_EVENT_C2H_10H) \ __PMC_EV_ALIAS("UOPS_RETIRED.FUSED", IAP_EVENT_C2H_07H) \ __PMC_EV_ALIAS("UOPS_RETIRED.LD_IND_BR", IAP_EVENT_C2H_01H) \ __PMC_EV_ALIAS("UOPS_RETIRED.MACRO_FUSION", IAP_EVENT_C2H_04H) \ __PMC_EV_ALIAS("UOPS_RETIRED.NON_FUSED", IAP_EVENT_C2H_08H) \ __PMC_EV_ALIAS("UOPS_RETIRED.STD_STA", IAP_EVENT_C2H_02H) \ __PMC_EV_ALIAS("X87_COMP_OPS_EXE.ANY.AR", IAP_EVENT_10H_81H) \ __PMC_EV_ALIAS("X87_COMP_OPS_EXE.ANY.S", IAP_EVENT_10H_01H) \ __PMC_EV_ALIAS("X87_OPS_RETIRED.ANY", IAP_EVENT_C1H_FEH) \ __PMC_EV_ALIAS("X87_OPS_RETIRED.FXCH", IAP_EVENT_C1H_01H) /* * Aliases for Core PMC events. */ #define __PMC_EV_ALIAS_CORE() \ __PMC_EV_ALIAS_INTEL_ARCHITECTURAL() \ __PMC_EV_ALIAS("BAClears", IAP_EVENT_E6H_00H) \ __PMC_EV_ALIAS("BTB_Misses", IAP_EVENT_E2H_00H) \ __PMC_EV_ALIAS("Br_BAC_Missp_Exec", IAP_EVENT_8AH_00H) \ __PMC_EV_ALIAS("Br_Bogus", IAP_EVENT_E4H_00H) \ __PMC_EV_ALIAS("Br_Call_Exec", IAP_EVENT_92H_00H) \ __PMC_EV_ALIAS("Br_Call_Missp_Exec", IAP_EVENT_93H_00H) \ __PMC_EV_ALIAS("Br_Cnd_Exec", IAP_EVENT_8BH_00H) \ __PMC_EV_ALIAS("Br_Cnd_Missp_Exec", IAP_EVENT_8CH_00H) \ __PMC_EV_ALIAS("Br_Ind_Call_Exec", IAP_EVENT_94H_00H) \ __PMC_EV_ALIAS("Br_Ind_Exec", IAP_EVENT_8DH_00H) \ __PMC_EV_ALIAS("Br_Ind_Missp_Exec", IAP_EVENT_8EH_00H) \ __PMC_EV_ALIAS("Br_Inst_Exec", IAP_EVENT_88H_00H) \ __PMC_EV_ALIAS("Br_Instr_Decoded", IAP_EVENT_E0H_00H) \ __PMC_EV_ALIAS("Br_Instr_Ret", IAP_EVENT_C4H_00H) \ __PMC_EV_ALIAS("Br_MisPred_Ret", IAP_EVENT_C5H_00H) \ __PMC_EV_ALIAS("Br_MisPred_Taken_Ret", IAP_EVENT_CAH_00H) \ __PMC_EV_ALIAS("Br_Missp_Exec", IAP_EVENT_89H_00H) \ __PMC_EV_ALIAS("Br_Ret_BAC_Missp_Exec", IAP_EVENT_91H_00H) \ __PMC_EV_ALIAS("Br_Ret_Exec", IAP_EVENT_8FH_00H) \ __PMC_EV_ALIAS("Br_Ret_Missp_Exec", IAP_EVENT_90H_00H) \ __PMC_EV_ALIAS("Br_Taken_Ret", IAP_EVENT_C9H_00H) \ __PMC_EV_ALIAS("Bus_BNR_Clocks", IAP_EVENT_61H_00H) \ __PMC_EV_ALIAS("Bus_DRDY_Clocks", IAP_EVENT_62H_00H) \ __PMC_EV_ALIAS("Bus_Data_Rcv", IAP_EVENT_64H_40H) \ __PMC_EV_ALIAS("Bus_Locks_Clocks", IAP_EVENT_63H) \ __PMC_EV_ALIAS("Bus_Not_In_Use", IAP_EVENT_7DH) \ __PMC_EV_ALIAS("Bus_Req_Outstanding", IAP_EVENT_60H) \ __PMC_EV_ALIAS("Bus_Snoop_Stall", IAP_EVENT_7EH_00H) \ __PMC_EV_ALIAS("Bus_Snoops", IAP_EVENT_77H) \ __PMC_EV_ALIAS("Bus_Trans_Any", IAP_EVENT_70H) \ __PMC_EV_ALIAS("Bus_Trans_Brd", IAP_EVENT_65H) \ __PMC_EV_ALIAS("Bus_Trans_Burst", IAP_EVENT_6EH) \ __PMC_EV_ALIAS("Bus_Trans_Def", IAP_EVENT_6DH) \ __PMC_EV_ALIAS("Bus_Trans_IO", IAP_EVENT_6CH) \ __PMC_EV_ALIAS("Bus_Trans_Ifetch", IAP_EVENT_68H) \ __PMC_EV_ALIAS("Bus_Trans_Inval", IAP_EVENT_69H) \ __PMC_EV_ALIAS("Bus_Trans_Mem", IAP_EVENT_6FH) \ __PMC_EV_ALIAS("Bus_Trans_P", IAP_EVENT_6BH) \ __PMC_EV_ALIAS("Bus_Trans_Pwr", IAP_EVENT_6AH) \ __PMC_EV_ALIAS("Bus_Trans_RFO", IAP_EVENT_66H) \ __PMC_EV_ALIAS("Bus_Trans_WB", IAP_EVENT_67H) \ __PMC_EV_ALIAS("Cycles_Div_Busy", IAP_EVENT_14H_00H) \ __PMC_EV_ALIAS("Cycles_Int_Masked", IAP_EVENT_C6H_00H) \ __PMC_EV_ALIAS("Cycles_Int_Pending_Masked", IAP_EVENT_C7H_00H) \ __PMC_EV_ALIAS("DCU_Snoop_To_Share", IAP_EVENT_78H) \ __PMC_EV_ALIAS("DCache_Cache_LD", IAP_EVENT_40H) \ __PMC_EV_ALIAS("DCache_Cache_Lock", IAP_EVENT_42H) \ __PMC_EV_ALIAS("DCache_Cache_ST", IAP_EVENT_41H) \ __PMC_EV_ALIAS("DCache_M_Evict", IAP_EVENT_47H_00H) \ __PMC_EV_ALIAS("DCache_M_Repl", IAP_EVENT_46H_00H) \ __PMC_EV_ALIAS("DCache_Pend_Miss", IAP_EVENT_48H_00H) \ __PMC_EV_ALIAS("DCache_Repl", IAP_EVENT_45H_0FH) \ __PMC_EV_ALIAS("Data_Mem_Cache_Ref", IAP_EVENT_44H_02H) \ __PMC_EV_ALIAS("Data_Mem_Ref", IAP_EVENT_43H_01H) \ __PMC_EV_ALIAS("Dbus_Busy", IAP_EVENT_22H) \ __PMC_EV_ALIAS("Dbus_Busy_Rd", IAP_EVENT_23H) \ __PMC_EV_ALIAS("Div", IAP_EVENT_13H_00H) \ __PMC_EV_ALIAS("Dtlb_Miss", IAP_EVENT_49H_00H) \ __PMC_EV_ALIAS("ESP_Uops", IAP_EVENT_D7H_00H) \ __PMC_EV_ALIAS("EST_Trans", IAP_EVENT_3AH) \ __PMC_EV_ALIAS("FP_Assist", IAP_EVENT_11H_00H) \ __PMC_EV_ALIAS("FP_Comp_Instr_Ret", IAP_EVENT_C1H_00H) \ __PMC_EV_ALIAS("FP_Comps_Op_Exe", IAP_EVENT_10H_00H) \ __PMC_EV_ALIAS("FP_MMX_Trans", IAP_EVENT_CCH_01H) \ __PMC_EV_ALIAS("Fused_Ld_Uops_Ret", IAP_EVENT_DAH_01H) \ __PMC_EV_ALIAS("Fused_St_Uops_Ret", IAP_EVENT_DAH_02H) \ __PMC_EV_ALIAS("Fused_Uops_Ret", IAP_EVENT_DAH_00H) \ __PMC_EV_ALIAS("HW_Int_Rx", IAP_EVENT_C8H_00H) \ __PMC_EV_ALIAS("ICache_Misses", IAP_EVENT_81H_00H) \ __PMC_EV_ALIAS("ICache_Reads", IAP_EVENT_80H_00H) \ __PMC_EV_ALIAS("IFU_Mem_Stall", IAP_EVENT_86H_00H) \ __PMC_EV_ALIAS("ILD_Stall", IAP_EVENT_87H_00H) \ __PMC_EV_ALIAS("ITLB_Misses", IAP_EVENT_85H_00H) \ __PMC_EV_ALIAS("Instr_Decoded", IAP_EVENT_D0H_00H) \ __PMC_EV_ALIAS("Instr_Ret", IAP_EVENT_C0H_00H) \ __PMC_EV_ALIAS("L1_Pref_Req", IAP_EVENT_4FH_00H) \ __PMC_EV_ALIAS("L2_ADS", IAP_EVENT_21H) \ __PMC_EV_ALIAS("L2_IFetch", IAP_EVENT_28H) \ __PMC_EV_ALIAS("L2_LD", IAP_EVENT_29H) \ __PMC_EV_ALIAS("L2_Lines_In", IAP_EVENT_24H) \ __PMC_EV_ALIAS("L2_Lines_Out", IAP_EVENT_26H) \ __PMC_EV_ALIAS("L2_M_Lines_In", IAP_EVENT_25H) \ __PMC_EV_ALIAS("L2_M_Lines_Out", IAP_EVENT_27H) \ __PMC_EV_ALIAS("L2_No_Request_Cycles", IAP_EVENT_32H) \ __PMC_EV_ALIAS("L2_Reject_Cycles", IAP_EVENT_30H) \ __PMC_EV_ALIAS("L2_Rqsts", IAP_EVENT_2EH) \ __PMC_EV_ALIAS("L2_ST", IAP_EVENT_2AH) \ __PMC_EV_ALIAS("LD_Blocks", IAP_EVENT_03H_00H) \ __PMC_EV_ALIAS("LLC_Misses", IAP_EVENT_2EH_41H) \ __PMC_EV_ALIAS("LLC_Reference", IAP_EVENT_2EH_4FH) \ __PMC_EV_ALIAS("MMX_Assist", IAP_EVENT_CDH_00H) \ __PMC_EV_ALIAS("MMX_FP_Trans", IAP_EVENT_CCH_00H) \ __PMC_EV_ALIAS("MMX_Instr_Exec", IAP_EVENT_B0H_00H) \ __PMC_EV_ALIAS("MMX_Instr_Ret", IAP_EVENT_CEH_00H) \ __PMC_EV_ALIAS("Misalign_Mem_Ref", IAP_EVENT_05H_00H) \ __PMC_EV_ALIAS("Mul", IAP_EVENT_12H_00H) \ __PMC_EV_ALIAS("NonHlt_Ref_Cycles", IAP_EVENT_3CH_01H) \ __PMC_EV_ALIAS("Pref_Rqsts_Dn", IAP_EVENT_F8H_00H) \ __PMC_EV_ALIAS("Pref_Rqsts_Up", IAP_EVENT_F0H_00H) \ __PMC_EV_ALIAS("Resource_Stall", IAP_EVENT_A2H_00H) \ __PMC_EV_ALIAS("SD_Drains", IAP_EVENT_04H_00H) \ __PMC_EV_ALIAS("SIMD_FP_DP_P_Comp_Ret", IAP_EVENT_D9H_02H) \ __PMC_EV_ALIAS("SIMD_FP_DP_P_Ret", IAP_EVENT_D8H_02H) \ __PMC_EV_ALIAS("SIMD_FP_DP_S_Comp_Ret", IAP_EVENT_D9H_03H) \ __PMC_EV_ALIAS("SIMD_FP_DP_S_Ret", IAP_EVENT_D8H_03H) \ __PMC_EV_ALIAS("SIMD_FP_SP_P_Comp_Ret", IAP_EVENT_D9H_00H) \ __PMC_EV_ALIAS("SIMD_FP_SP_Ret", IAP_EVENT_D8H_00H) \ __PMC_EV_ALIAS("SIMD_FP_SP_S_Comp_Ret", IAP_EVENT_D9H_01H) \ __PMC_EV_ALIAS("SIMD_FP_SP_S_Ret", IAP_EVENT_D8H_01H) \ __PMC_EV_ALIAS("SIMD_Int_128_Ret", IAP_EVENT_D8H_04H) \ __PMC_EV_ALIAS("SIMD_Int_Pari_Exec", IAP_EVENT_B3H_20H) \ __PMC_EV_ALIAS("SIMD_Int_Pck_Exec", IAP_EVENT_B3H_04H) \ __PMC_EV_ALIAS("SIMD_Int_Plog_Exec", IAP_EVENT_B3H_10H) \ __PMC_EV_ALIAS("SIMD_Int_Pmul_Exec", IAP_EVENT_B3H_01H) \ __PMC_EV_ALIAS("SIMD_Int_Psft_Exec", IAP_EVENT_B3H_02H) \ __PMC_EV_ALIAS("SIMD_Int_Sat_Exec", IAP_EVENT_B1H_00H) \ __PMC_EV_ALIAS("SIMD_Int_Upck_Exec", IAP_EVENT_B3H_08H) \ __PMC_EV_ALIAS("SMC_Detected", IAP_EVENT_C3H_00H) \ __PMC_EV_ALIAS("SSE_NTStores_Miss", IAP_EVENT_4BH_03H) \ __PMC_EV_ALIAS("SSE_NTStores_Ret", IAP_EVENT_07H_03H) \ __PMC_EV_ALIAS("SSE_PrefNta_Miss", IAP_EVENT_4BH_00H) \ __PMC_EV_ALIAS("SSE_PrefNta_Ret", IAP_EVENT_07H_00H) \ __PMC_EV_ALIAS("SSE_PrefT1_Miss", IAP_EVENT_4BH_01H) \ __PMC_EV_ALIAS("SSE_PrefT1_Ret", IAP_EVENT_07H_01H) \ __PMC_EV_ALIAS("SSE_PrefT2_Miss", IAP_EVENT_4BH_02H) \ __PMC_EV_ALIAS("SSE_PrefT2_Ret", IAP_EVENT_07H_02H) \ __PMC_EV_ALIAS("Seg_Reg_Loads", IAP_EVENT_06H_00H) \ __PMC_EV_ALIAS("Serial_Execution_Cycles", IAP_EVENT_3CH_02H) \ __PMC_EV_ALIAS("Thermal_Trip", IAP_EVENT_3BH_C0H) \ __PMC_EV_ALIAS("Unfusion", IAP_EVENT_DBH_00H) \ __PMC_EV_ALIAS("Unhalted_Core_Cycles", IAP_EVENT_3CH_00H) \ __PMC_EV_ALIAS("Uops_Ret", IAP_EVENT_C2H_00H) /* * Aliases for Core2 PMC events. */ #define __PMC_EV_ALIAS_CORE2() \ __PMC_EV_ALIAS_INTEL_ARCHITECTURAL() \ __PMC_EV_ALIAS("BACLEARS", IAP_EVENT_E6H_00H) \ __PMC_EV_ALIAS("BOGUS_BR", IAP_EVENT_E4H_00H) \ __PMC_EV_ALIAS("BR_BAC_MISSP_EXEC", IAP_EVENT_8AH_00H) \ __PMC_EV_ALIAS("BR_CALL_EXEC", IAP_EVENT_92H_00H) \ __PMC_EV_ALIAS("BR_CALL_MISSP_EXEC", IAP_EVENT_93H_00H) \ __PMC_EV_ALIAS("BR_CND_EXEC", IAP_EVENT_8BH_00H) \ __PMC_EV_ALIAS("BR_CND_MISSP_EXEC", IAP_EVENT_8CH_00H) \ __PMC_EV_ALIAS("BR_IND_CALL_EXEC", IAP_EVENT_94H_00H) \ __PMC_EV_ALIAS("BR_IND_EXEC", IAP_EVENT_8DH_00H) \ __PMC_EV_ALIAS("BR_IND_MISSP_EXEC", IAP_EVENT_8EH_00H) \ __PMC_EV_ALIAS("BR_INST_DECODED", IAP_EVENT_E0H_00H) \ __PMC_EV_ALIAS("BR_INST_EXEC", IAP_EVENT_88H_00H) \ __PMC_EV_ALIAS("BR_INST_RETIRED.ANY", IAP_EVENT_C4H_00H) \ __PMC_EV_ALIAS("BR_INST_RETIRED.MISPRED", IAP_EVENT_C5H_00H) \ __PMC_EV_ALIAS("BR_INST_RETIRED.MISPRED_NOT_TAKEN", \ IAP_EVENT_C4H_02H) \ __PMC_EV_ALIAS("BR_INST_RETIRED.MISPRED_TAKEN", \ IAP_EVENT_C4H_08H) \ __PMC_EV_ALIAS("BR_INST_RETIRED.PRED_NOT_TAKEN", \ IAP_EVENT_C4H_01H) \ __PMC_EV_ALIAS("BR_INST_RETIRED.PRED_TAKEN", \ IAP_EVENT_C4H_04H) \ __PMC_EV_ALIAS("BR_INST_RETIRED.TAKEN", IAP_EVENT_C4H_0CH) \ __PMC_EV_ALIAS("BR_MISSP_EXEC", IAP_EVENT_89H_00H) \ __PMC_EV_ALIAS("BR_RET_BAC_MISSP_EXEC", IAP_EVENT_91H_00H) \ __PMC_EV_ALIAS("BR_RET_EXEC", IAP_EVENT_8FH_00H) \ __PMC_EV_ALIAS("BR_RET_MISSP_EXEC", IAP_EVENT_90H_00H) \ __PMC_EV_ALIAS("BR_TKN_BUBBLE_1", IAP_EVENT_97H_00H) \ __PMC_EV_ALIAS("BR_TKN_BUBBLE_2", IAP_EVENT_98H_00H) \ __PMC_EV_ALIAS("BUSQ_EMPTY", IAP_EVENT_7DH) \ __PMC_EV_ALIAS("BUS_BNR_DRV", IAP_EVENT_61H) \ __PMC_EV_ALIAS("BUS_DATA_RCV", IAP_EVENT_64H) \ __PMC_EV_ALIAS("BUS_DRDY_CLOCKS", IAP_EVENT_62H) \ __PMC_EV_ALIAS("BUS_HITM_DRV", IAP_EVENT_7BH) \ __PMC_EV_ALIAS("BUS_HIT_DRV", IAP_EVENT_7AH) \ __PMC_EV_ALIAS("BUS_IO_WAIT", IAP_EVENT_7FH) \ __PMC_EV_ALIAS("BUS_LOCK_CLOCKS", IAP_EVENT_63H) \ __PMC_EV_ALIAS("BUS_REQUEST_OUTSTANDING", \ IAP_EVENT_60H) \ __PMC_EV_ALIAS("BUS_TRANS_ANY", IAP_EVENT_70H) \ __PMC_EV_ALIAS("BUS_TRANS_BRD", IAP_EVENT_65H) \ __PMC_EV_ALIAS("BUS_TRANS_BURST", IAP_EVENT_6EH) \ __PMC_EV_ALIAS("BUS_TRANS_DEF", IAP_EVENT_6DH) \ __PMC_EV_ALIAS("BUS_TRANS_IFETCH", IAP_EVENT_68H) \ __PMC_EV_ALIAS("BUS_TRANS_INVAL", IAP_EVENT_69H) \ __PMC_EV_ALIAS("BUS_TRANS_IO", IAP_EVENT_6CH) \ __PMC_EV_ALIAS("BUS_TRANS_MEM", IAP_EVENT_6FH) \ __PMC_EV_ALIAS("BUS_TRANS_P", IAP_EVENT_6BH) \ __PMC_EV_ALIAS("BUS_TRANS_PWR", IAP_EVENT_6AH) \ __PMC_EV_ALIAS("BUS_TRANS_RFO", IAP_EVENT_66H) \ __PMC_EV_ALIAS("BUS_TRANS_WB", IAP_EVENT_67H) \ __PMC_EV_ALIAS("CMP_SNOOP", IAP_EVENT_78H) \ __PMC_EV_ALIAS("CPU_CLK_UNHALTED.BUS", IAP_EVENT_3CH_01H) \ __PMC_EV_ALIAS("CPU_CLK_UNHALTED.CORE_P", \ IAP_EVENT_3CH_00H) \ __PMC_EV_ALIAS("CPU_CLK_UNHALTED.NO_OTHER", \ IAP_EVENT_3CH_02H) \ __PMC_EV_ALIAS("CYCLES_DIV_BUSY", IAP_EVENT_14H_00H) \ __PMC_EV_ALIAS("CYCLES_INT_MASKED", IAP_EVENT_C6H_01H) \ __PMC_EV_ALIAS("CYCLES_INT_PENDING_AND_MASKED", \ IAP_EVENT_C6H_02H) \ __PMC_EV_ALIAS("CYCLES_L1I_MEM_STALLED", IAP_EVENT_86H_00H) \ __PMC_EV_ALIAS("DELAYED_BYPASS.FP", IAP_EVENT_19H_00H) \ __PMC_EV_ALIAS("DELAYED_BYPASS.LOAD", IAP_EVENT_19H_01H) \ __PMC_EV_ALIAS("DELAYED_BYPASS.SIMD", IAP_EVENT_19H_02H) \ __PMC_EV_ALIAS("DIV", IAP_EVENT_13H_00H) \ __PMC_EV_ALIAS("DTLB_MISSES.ANY", IAP_EVENT_08H_01H) \ __PMC_EV_ALIAS("DTLB_MISSES.L0_MISS_LD", IAP_EVENT_08H_04H) \ __PMC_EV_ALIAS("DTLB_MISSES.MISS_LD", IAP_EVENT_08H_02H) \ __PMC_EV_ALIAS("DTLB_MISSES.MISS_ST", IAP_EVENT_08H_08H) \ __PMC_EV_ALIAS("EIST_TRANS", IAP_EVENT_3AH_00H) \ __PMC_EV_ALIAS("ESP.ADDITIONS", IAP_EVENT_ABH_02H) \ __PMC_EV_ALIAS("ESP.SYNCH", IAP_EVENT_ABH_01H) \ __PMC_EV_ALIAS("EXT_SNOOP", IAP_EVENT_77H) \ __PMC_EV_ALIAS("FP_ASSIST", IAP_EVENT_11H_00H) \ __PMC_EV_ALIAS("FP_COMP_OPS_EXE", IAP_EVENT_10H_00H) \ __PMC_EV_ALIAS("FP_MMX_TRANS_TO_FP", IAP_EVENT_CCH_02H) \ __PMC_EV_ALIAS("FP_MMX_TRANS_TO_MMX", IAP_EVENT_CCH_01H) \ __PMC_EV_ALIAS("HW_INT_RCV", IAP_EVENT_C8H_00H) \ __PMC_EV_ALIAS("IDLE_DURING_DIV", IAP_EVENT_18H_00H) \ __PMC_EV_ALIAS("ILD_STALL", IAP_EVENT_87H_00H) \ __PMC_EV_ALIAS("INST_QUEUE.FULL", IAP_EVENT_83H_02H) \ __PMC_EV_ALIAS("INST_RETIRED.ANY_P", IAP_EVENT_C0H_00H) \ __PMC_EV_ALIAS("INST_RETIRED.LOADS", IAP_EVENT_C0H_01H) \ __PMC_EV_ALIAS("INST_RETIRED.OTHER", IAP_EVENT_C0H_04H) \ __PMC_EV_ALIAS("INST_RETIRED.STORES", IAP_EVENT_C0H_02H) \ __PMC_EV_ALIAS("INST_RETIRED.VM_H", IAP_EVENT_C0H_08H) \ __PMC_EV_ALIAS("ITLB.FLUSH", IAP_EVENT_82H_40H) \ __PMC_EV_ALIAS("ITLB.LARGE_MISS", IAP_EVENT_82H_10H) \ __PMC_EV_ALIAS("ITLB.MISSES", IAP_EVENT_82H_12H) \ __PMC_EV_ALIAS("ITLB.SMALL_MISS", IAP_EVENT_82H_02H) \ __PMC_EV_ALIAS("ITLB_MISS_RETIRED", IAP_EVENT_C9H_00H) \ __PMC_EV_ALIAS("L1D_ALL_CACHE_REF", IAP_EVENT_43H_02H) \ __PMC_EV_ALIAS("L1D_ALL_REF", IAP_EVENT_43H_01H) \ __PMC_EV_ALIAS("L1D_CACHE_LD", IAP_EVENT_40H) \ __PMC_EV_ALIAS("L1D_CACHE_LOCK", IAP_EVENT_42H) \ __PMC_EV_ALIAS("L1D_CACHE_LOCK_DURATION", IAP_EVENT_42H_10H) \ __PMC_EV_ALIAS("L1D_CACHE_ST", IAP_EVENT_41H) \ __PMC_EV_ALIAS("L1D_M_EVICT", IAP_EVENT_47H_00H) \ __PMC_EV_ALIAS("L1D_M_REPL", IAP_EVENT_46H_00H) \ __PMC_EV_ALIAS("L1D_PEND_MISS", IAP_EVENT_48H_00H) \ __PMC_EV_ALIAS("L1D_PREFETCH.REQUESTS", IAP_EVENT_4EH_10H) \ __PMC_EV_ALIAS("L1D_REPL", IAP_EVENT_45H_0FH) \ __PMC_EV_ALIAS("L1D_SPLIT.LOADS", IAP_EVENT_49H_01H) \ __PMC_EV_ALIAS("L1D_SPLIT.STORES", IAP_EVENT_49H_02H) \ __PMC_EV_ALIAS("L1I_MISSES", IAP_EVENT_81H_00H) \ __PMC_EV_ALIAS("L1I_READS", IAP_EVENT_80H_00H) \ __PMC_EV_ALIAS("L2_ADS", IAP_EVENT_21H) \ __PMC_EV_ALIAS("L2_DBUS_BUSY_RD", IAP_EVENT_23H) \ __PMC_EV_ALIAS("L2_IFETCH", IAP_EVENT_28H) \ __PMC_EV_ALIAS("L2_LD", IAP_EVENT_29H) \ __PMC_EV_ALIAS("L2_LINES_IN", IAP_EVENT_24H) \ __PMC_EV_ALIAS("L2_LINES_OUT", IAP_EVENT_26H) \ __PMC_EV_ALIAS("L2_LOCK", IAP_EVENT_2BH) \ __PMC_EV_ALIAS("L2_M_LINES_IN", IAP_EVENT_25H) \ __PMC_EV_ALIAS("L2_M_LINES_OUT", IAP_EVENT_27H) \ __PMC_EV_ALIAS("L2_NO_REQ", IAP_EVENT_32H) \ __PMC_EV_ALIAS("L2_REJECT_BUSQ", IAP_EVENT_30H) \ __PMC_EV_ALIAS("L2_RQSTS", IAP_EVENT_2EH) \ __PMC_EV_ALIAS("L2_RQSTS.SELF.DEMAND.I_STATE", \ IAP_EVENT_2EH_41H) \ __PMC_EV_ALIAS("L2_RQSTS.SELF.DEMAND.MESI", \ IAP_EVENT_2EH_4FH) \ __PMC_EV_ALIAS("L2_ST", IAP_EVENT_2AH) \ __PMC_EV_ALIAS("LOAD_BLOCK.L1D", IAP_EVENT_03H_20H) \ __PMC_EV_ALIAS("LOAD_BLOCK.OVERLAP_STORE", \ IAP_EVENT_03H_08H) \ __PMC_EV_ALIAS("LOAD_BLOCK.STA", IAP_EVENT_03H_02H) \ __PMC_EV_ALIAS("LOAD_BLOCK.STD", IAP_EVENT_03H_04H) \ __PMC_EV_ALIAS("LOAD_BLOCK.UNTIL_RETIRE", IAP_EVENT_03H_10H) \ __PMC_EV_ALIAS("LOAD_HIT_PRE", IAP_EVENT_4CH_00H) \ __PMC_EV_ALIAS("MACHINE_NUKES.MEM_ORDER", IAP_EVENT_C3H_04H) \ __PMC_EV_ALIAS("MACHINE_NUKES.SMC", IAP_EVENT_C3H_01H) \ __PMC_EV_ALIAS("MACRO_INSTS.CISC_DECODED", IAP_EVENT_AAH_08H) \ __PMC_EV_ALIAS("MACRO_INSTS.DECODED", IAP_EVENT_AAH_01H) \ __PMC_EV_ALIAS("MEMORY_DISAMBIGUATION.RESET", \ IAP_EVENT_09H_01H) \ __PMC_EV_ALIAS("MEMORY_DISAMBIGUATION.SUCCESS", \ IAP_EVENT_09H_02H) \ __PMC_EV_ALIAS("MEM_LOAD_RETIRED.DTLB_MISS", \ IAP_EVENT_CBH_10H) \ __PMC_EV_ALIAS("MEM_LOAD_RETIRED.L1D_LINE_MISS", \ IAP_EVENT_CBH_02H) \ __PMC_EV_ALIAS("MEM_LOAD_RETIRED.L1D_MISS", \ IAP_EVENT_CBH_01H) \ __PMC_EV_ALIAS("MEM_LOAD_RETIRED.L2_LINE_MISS", \ IAP_EVENT_CBH_08H) \ __PMC_EV_ALIAS("MEM_LOAD_RETIRED.L2_MISS", \ IAP_EVENT_CBH_04H) \ __PMC_EV_ALIAS("MUL", IAP_EVENT_12H_00H) \ __PMC_EV_ALIAS("PAGE_WALKS.COUNT", IAP_EVENT_0CH_01H) \ __PMC_EV_ALIAS("PAGE_WALKS.CYCLES", IAP_EVENT_0CH_02H) \ __PMC_EV_ALIAS("PREF_RQSTS_DN", IAP_EVENT_F8H_00H) \ __PMC_EV_ALIAS("PREF_RQSTS_UP", IAP_EVENT_F0H_00H) \ __PMC_EV_ALIAS("RAT_STALLS.ANY", IAP_EVENT_D2H_0FH) \ __PMC_EV_ALIAS("RAT_STALLS.FLAGS", IAP_EVENT_D2H_04H) \ __PMC_EV_ALIAS("RAT_STALLS.FPSW", IAP_EVENT_D2H_08H) \ __PMC_EV_ALIAS("RAT_STALLS.OTHER_SERIALIZATION_STALLS", \ IAP_EVENT_D2H_10H) \ __PMC_EV_ALIAS("RAT_STALLS.PARTIAL_CYCLES", \ IAP_EVENT_D2H_02H) \ __PMC_EV_ALIAS("RAT_STALLS.ROB_READ_PORT", \ IAP_EVENT_D2H_01H) \ __PMC_EV_ALIAS("RESOURCE_STALLS.ANY", IAP_EVENT_DCH_1FH) \ __PMC_EV_ALIAS("RESOURCE_STALLS.BR_MISS_CLEAR", \ IAP_EVENT_DCH_10H) \ __PMC_EV_ALIAS("RESOURCE_STALLS.FPCW", IAP_EVENT_DCH_08H) \ __PMC_EV_ALIAS("RESOURCE_STALLS.LD_ST", IAP_EVENT_DCH_04H) \ __PMC_EV_ALIAS("RESOURCE_STALLS.ROB_FULL", \ IAP_EVENT_DCH_01H) \ __PMC_EV_ALIAS("RESOURCE_STALLS.RS_FULL", IAP_EVENT_DCH_02H) \ __PMC_EV_ALIAS("RS_UOPS_DISPATCHED", IAP_EVENT_A0H_00H) \ __PMC_EV_ALIAS("RS_UOPS_DISPATCHED.PORT0", IAP_EVENT_A1H_01H) \ __PMC_EV_ALIAS("RS_UOPS_DISPATCHED.PORT1", IAP_EVENT_A1H_02H) \ __PMC_EV_ALIAS("RS_UOPS_DISPATCHED.PORT2", IAP_EVENT_A1H_04H) \ __PMC_EV_ALIAS("RS_UOPS_DISPATCHED.PORT3", IAP_EVENT_A1H_08H) \ __PMC_EV_ALIAS("RS_UOPS_DISPATCHED.PORT4", IAP_EVENT_A1H_10H) \ __PMC_EV_ALIAS("RS_UOPS_DISPATCHED.PORT5", IAP_EVENT_A1H_20H) \ __PMC_EV_ALIAS("SB_DRAIN_CYCLES", IAP_EVENT_04H_01H) \ __PMC_EV_ALIAS("SEGMENT_REG_LOADS", IAP_EVENT_06H_00H) \ __PMC_EV_ALIAS("SEG_REG_RENAMES.ANY", IAP_EVENT_D5H_0FH) \ __PMC_EV_ALIAS("SEG_REG_RENAMES.DS", IAP_EVENT_D5H_02H) \ __PMC_EV_ALIAS("SEG_REG_RENAMES.ES", IAP_EVENT_D5H_01H) \ __PMC_EV_ALIAS("SEG_REG_RENAMES.FS", IAP_EVENT_D5H_04H) \ __PMC_EV_ALIAS("SEG_REG_RENAMES.GS", IAP_EVENT_D5H_08H) \ __PMC_EV_ALIAS("SEG_RENAME_STALLS.ANY", IAP_EVENT_D4H_0FH) \ __PMC_EV_ALIAS("SEG_RENAME_STALLS.DS", IAP_EVENT_D4H_02H) \ __PMC_EV_ALIAS("SEG_RENAME_STALLS.ES", IAP_EVENT_D4H_01H) \ __PMC_EV_ALIAS("SEG_RENAME_STALLS.FS", IAP_EVENT_D4H_04H) \ __PMC_EV_ALIAS("SEG_RENAME_STALLS.GS", IAP_EVENT_D4H_08H) \ __PMC_EV_ALIAS("SIMD_ASSIST", IAP_EVENT_CDH_00H) \ __PMC_EV_ALIAS("SIMD_COMP_INST_RETIRED.PACKED_DOUBLE", \ IAP_EVENT_CAH_04H) \ __PMC_EV_ALIAS("SIMD_COMP_INST_RETIRED.PACKED_SINGLE", \ IAP_EVENT_CAH_01H) \ __PMC_EV_ALIAS("SIMD_COMP_INST_RETIRED.SCALAR_DOUBLE", \ IAP_EVENT_CAH_08H) \ __PMC_EV_ALIAS("SIMD_COMP_INST_RETIRED.SCALAR_SINGLE", \ IAP_EVENT_CAH_02H) \ __PMC_EV_ALIAS("SIMD_INSTR_RETIRED", IAP_EVENT_CEH_00H) \ __PMC_EV_ALIAS("SIMD_INST_RETIRED.ANY", IAP_EVENT_C7H_1FH) \ __PMC_EV_ALIAS("SIMD_INST_RETIRED.PACKED_DOUBLE", \ IAP_EVENT_C7H_04H) \ __PMC_EV_ALIAS("SIMD_INST_RETIRED.PACKED_SINGLE", \ IAP_EVENT_C7H_01H) \ __PMC_EV_ALIAS("SIMD_INST_RETIRED.SCALAR_DOUBLE", \ IAP_EVENT_C7H_08H) \ __PMC_EV_ALIAS("SIMD_INST_RETIRED.SCALAR_SINGLE", \ IAP_EVENT_C7H_02H) \ __PMC_EV_ALIAS("SIMD_INST_RETIRED.VECTOR", IAP_EVENT_C7H_10H) \ __PMC_EV_ALIAS("SIMD_SAT_INSTR_RETIRED", IAP_EVENT_CFH_00H) \ __PMC_EV_ALIAS("SIMD_SAT_UOP_EXEC", IAP_EVENT_B1H_00H) \ __PMC_EV_ALIAS("SIMD_UOPS_EXEC", IAP_EVENT_B0H_00H) \ __PMC_EV_ALIAS("SIMD_UOP_TYPE_EXEC.ARITHMETIC", IAP_EVENT_B3H_20H) \ __PMC_EV_ALIAS("SIMD_UOP_TYPE_EXEC.LOGICAL", IAP_EVENT_B3H_10H) \ __PMC_EV_ALIAS("SIMD_UOP_TYPE_EXEC.MUL", IAP_EVENT_B3H_01H) \ __PMC_EV_ALIAS("SIMD_UOP_TYPE_EXEC.PACK", IAP_EVENT_B3H_04H) \ __PMC_EV_ALIAS("SIMD_UOP_TYPE_EXEC.SHIFT", IAP_EVENT_B3H_02H) \ __PMC_EV_ALIAS("SIMD_UOP_TYPE_EXEC.UNPACK", IAP_EVENT_B3H_08H) \ __PMC_EV_ALIAS("SNOOP_STALL_DRV", IAP_EVENT_7EH) \ __PMC_EV_ALIAS("SSE_PRE_EXEC.L1", IAP_EVENT_07H_01H) \ __PMC_EV_ALIAS("SSE_PRE_EXEC.L2", IAP_EVENT_07H_02H) \ __PMC_EV_ALIAS("SSE_PRE_EXEC.NTA", IAP_EVENT_07H_00H) \ __PMC_EV_ALIAS("SSE_PRE_EXEC.STORES", IAP_EVENT_07H_03H) \ __PMC_EV_ALIAS("SSE_PRE_MISS.L1", IAP_EVENT_4BH_01H) \ __PMC_EV_ALIAS("SSE_PRE_MISS.L2", IAP_EVENT_4BH_02H) \ __PMC_EV_ALIAS("SSE_PRE_MISS.NTA", IAP_EVENT_4BH_00H) \ __PMC_EV_ALIAS("STORE_BLOCK.ORDER", IAP_EVENT_04H_02H) \ __PMC_EV_ALIAS("STORE_BLOCK.SNOOP", IAP_EVENT_04H_08H) \ __PMC_EV_ALIAS("THERMAL_TRIP", IAP_EVENT_3BH_C0H) \ __PMC_EV_ALIAS("UOPS_RETIRED.ANY", IAP_EVENT_C2H_0FH) \ __PMC_EV_ALIAS("UOPS_RETIRED.FUSED", IAP_EVENT_C2H_07H) \ __PMC_EV_ALIAS("UOPS_RETIRED.LD_IND_BR", IAP_EVENT_C2H_01H) \ __PMC_EV_ALIAS("UOPS_RETIRED.MACRO_FUSION", IAP_EVENT_C2H_04H) \ __PMC_EV_ALIAS("UOPS_RETIRED.NON_FUSED", IAP_EVENT_C2H_08H) \ __PMC_EV_ALIAS("UOPS_RETIRED.STD_STA", IAP_EVENT_C2H_02H) \ __PMC_EV_ALIAS("X87_OPS_RETIRED.ANY", IAP_EVENT_C1H_FEH) \ __PMC_EV_ALIAS("X87_OPS_RETIRED.FXCH", IAP_EVENT_C1H_01H) /* * Core i7 and Xeon 5500 events removed between 253669-031US June 2009 * and 253669-033US December 2009. */ #define __PMC_EV_ALIAS_COREI7_OLD() \ __PMC_EV_ALIAS("SB_FORWARD.ANY", IAP_EVENT_02H_01H) \ __PMC_EV_ALIAS("LOAD_BLOCK.STD", IAP_EVENT_03H_01H) \ __PMC_EV_ALIAS("LOAD_BLOCK.ADDRESS_OFFSET", IAP_EVENT_03H_04H) \ __PMC_EV_ALIAS("SB_DRAIN.CYCLES", IAP_EVENT_04H_01H) \ __PMC_EV_ALIAS("MISALIGN_MEM_REF.LOAD", IAP_EVENT_05H_01H) \ __PMC_EV_ALIAS("MISALIGN_MEM_REF.STORE", IAP_EVENT_05H_02H) \ __PMC_EV_ALIAS("MISALIGN_MEM_REF.ANY", IAP_EVENT_05H_03H) \ __PMC_EV_ALIAS("STORE_BLOCKS.NOT_STA", IAP_EVENT_06H_01H) \ __PMC_EV_ALIAS("STORE_BLOCKS.STA", IAP_EVENT_06H_02H) \ __PMC_EV_ALIAS("STORE_BLOCKS.ANY", IAP_EVENT_06H_0FH) \ __PMC_EV_ALIAS("MEMORY_DISAMBIGURATION.RESET", IAP_EVENT_09H_01H) \ __PMC_EV_ALIAS("MEMORY_DISAMBIGURATION.SUCCESS", IAP_EVENT_09H_02H) \ __PMC_EV_ALIAS("MEMORY_DISAMBIGURATION.WATCHDOG", IAP_EVENT_09H_04H) \ __PMC_EV_ALIAS("MEMORY_DISAMBIGURATION.WATCH_CYCLES", IAP_EVENT_09H_08H)\ __PMC_EV_ALIAS("HW_INT.RCV", IAP_EVENT_1DH_01H) \ __PMC_EV_ALIAS("HW_INT.CYCLES_MASKED", IAP_EVENT_1DH_02H) \ __PMC_EV_ALIAS("HW_INT.CYCLES_PENDING_AND_MASKED", IAP_EVENT_1DH_04H) \ __PMC_EV_ALIAS("L2_WRITE.RFO.E_STATE", IAP_EVENT_27H_04H) \ __PMC_EV_ALIAS("UOPS_DECODED.DEC0", IAP_EVENT_3DH_01H) \ __PMC_EV_ALIAS("L1D_CACHE_ST.I_STATE", IAP_EVENT_41H_01H) \ __PMC_EV_ALIAS("L1D_CACHE_ST.MESI", IAP_EVENT_41H_0FH) \ __PMC_EV_ALIAS("DTLB_MISSES.PDE_MISS", IAP_EVENT_49H_20H) \ __PMC_EV_ALIAS("DTLB_MISSES.PDP_MISS", IAP_EVENT_49H_40H) \ __PMC_EV_ALIAS("DTLB_MISSES.LARGE_WALK_COMPLETED", IAP_EVENT_49H_80H) \ __PMC_EV_ALIAS("SSE_MEM_EXEC.NTA", IAP_EVENT_4BH_01H) \ __PMC_EV_ALIAS("SSE_MEM_EXEC.STREAMING_STORES", IAP_EVENT_4BH_08H) \ __PMC_EV_ALIAS("SFENCE_CYCLES", IAP_EVENT_4DH_01H) \ __PMC_EV_ALIAS("EPT.EPDE_MISS", IAP_EVENT_4FH_02H) \ __PMC_EV_ALIAS("EPT.EPDPE_HIT", IAP_EVENT_4FH_04H) \ __PMC_EV_ALIAS("EPT.EPDPE_MISS", IAP_EVENT_4FH_08H) \ __PMC_EV_ALIAS("OFFCORE_REQUESTS_OUTSTANDING.DEMAND.READ_DATA", \ IAP_EVENT_60H_01H) \ __PMC_EV_ALIAS("OFFCORE_REQUESTS_OUTSTANDING.DEMAND.READ_CODE", \ IAP_EVENT_60H_02H) \ __PMC_EV_ALIAS("OFFCORE_REQUESTS_OUTSTANDING.DEMAND.RFO", \ IAP_EVENT_60H_04H) \ __PMC_EV_ALIAS("OFFCORE_REQUESTS_OUTSTANDING.ANY.READ", \ IAP_EVENT_60H_08H) \ __PMC_EV_ALIAS("IFU_IVC.FULL", IAP_EVENT_81H_01H) \ __PMC_EV_ALIAS("IFU_IVC.L1I_EVICTION", IAP_EVENT_81H_02H) \ __PMC_EV_ALIAS("L1I_OPPORTUNISTIC_HITS", IAP_EVENT_83H_01H) \ __PMC_EV_ALIAS("ITLB_MISSES.WALK_CYCLES", IAP_EVENT_85H_04H) \ __PMC_EV_ALIAS("ITLB_MISSES.PMH_BUSY_CYCLES", IAP_EVENT_85H_04H) \ __PMC_EV_ALIAS("ITLB_MISSES.STLB_HIT", IAP_EVENT_85H_10H) \ __PMC_EV_ALIAS("ITLB_MISSES.PDE_MISS", IAP_EVENT_85H_20H) \ __PMC_EV_ALIAS("ITLB_MISSES.PDP_MISS", IAP_EVENT_85H_40H) \ __PMC_EV_ALIAS("ITLB_MISSES.LARGE_WALK_COMPLETED", IAP_EVENT_85H_80H) \ __PMC_EV_ALIAS("OFFCORE_REQUESTS.DEMAND.READ_DATA", IAP_EVENT_B0H_01H) \ __PMC_EV_ALIAS("OFFCORE_REQUESTS.DEMAND.READ_CODE", IAP_EVENT_B0H_02H) \ __PMC_EV_ALIAS("OFFCORE_REQUESTS.DEMAND.RFO", IAP_EVENT_B0H_04H) \ __PMC_EV_ALIAS("OFFCORE_REQUESTS.ANY.READ", IAP_EVENT_B0H_08H) \ __PMC_EV_ALIAS("OFFCORE_REQUESTS.ANY.RFO", IAP_EVENT_B0H_10H) \ __PMC_EV_ALIAS("OFFCORE_REQUESTS.UNCACHED_MEM", IAP_EVENT_B0H_20H) \ __PMC_EV_ALIAS("OFFCORE_REQUESTS.ANY", IAP_EVENT_B0H_80H) \ __PMC_EV_ALIAS("SNOOPQ_REQUESTS_OUTSTANDING.DATA", IAP_EVENT_B3H_01H) \ __PMC_EV_ALIAS("SNOOPQ_REQUESTS_OUTSTANDING.INVALIDATE", \ IAP_EVENT_B3H_02H) \ __PMC_EV_ALIAS("SNOOPQ_REQUESTS_OUTSTANDING.CODE", IAP_EVENT_B3H_04H) \ __PMC_EV_ALIAS("PIC_ACCESSES.TPR_READS", IAP_EVENT_BAH_01H) \ __PMC_EV_ALIAS("PIC_ACCESSES.TPR_WRITES", IAP_EVENT_BAH_02H) \ __PMC_EV_ALIAS("MACHINE_CLEARS.FUSION_ASSIST", IAP_EVENT_C3H_10H) \ __PMC_EV_ALIAS("BOGUS_BR", IAP_EVENT_E4H_01H) \ __PMC_EV_ALIAS("L2_HW_PREFETCH.HIT", IAP_EVENT_F3H_01H) \ __PMC_EV_ALIAS("L2_HW_PREFETCH.ALLOC", IAP_EVENT_F3H_02H) \ __PMC_EV_ALIAS("L2_HW_PREFETCH.DATA_TRIGGER", IAP_EVENT_F3H_04H) \ __PMC_EV_ALIAS("L2_HW_PREFETCH.CODE_TRIGGER", IAP_EVENT_F3H_08H) \ __PMC_EV_ALIAS("L2_HW_PREFETCH.DCA_TRIGGER", IAP_EVENT_F3H_10H) \ __PMC_EV_ALIAS("L2_HW_PREFETCH.KICK_START", IAP_EVENT_F3H_20H) \ __PMC_EV_ALIAS("SQ_MISC.PROMOTION", IAP_EVENT_F4H_01H) \ __PMC_EV_ALIAS("SQ_MISC.PROMOTION_POST_GO", IAP_EVENT_F4H_02H) \ __PMC_EV_ALIAS("SQ_MISC.LRU_HINTS", IAP_EVENT_F4H_04H) \ __PMC_EV_ALIAS("SQ_MISC.FILL_DROPPED", IAP_EVENT_F4H_08H) \ __PMC_EV_ALIAS("SEGMENT_REG_LOADS", IAP_EVENT_F8H_01H) /* * Aliases for Core i7 and Xeon 5500 PMC events (253669-033US December 2009) */ #define __PMC_EV_ALIAS_COREI7() \ __PMC_EV_ALIAS_INTEL_ARCHITECTURAL() \ __PMC_EV_ALIAS("SB_DRAIN.ANY", IAP_EVENT_04H_07H) \ __PMC_EV_ALIAS("STORE_BLOCKS.AT_RET", IAP_EVENT_06H_04H) \ __PMC_EV_ALIAS("STORE_BLOCKS.L1D_BLOCK", IAP_EVENT_06H_08H) \ __PMC_EV_ALIAS("PARTIAL_ADDRESS_ALIAS", IAP_EVENT_07H_01H) \ __PMC_EV_ALIAS("DTLB_LOAD_MISSES.ANY", IAP_EVENT_08H_01H) \ __PMC_EV_ALIAS("DTLB_LOAD_MISSES.WALK_COMPLETED", IAP_EVENT_08H_02H) \ __PMC_EV_ALIAS("DTLB_LOAD_MISSES.STLB_HIT", IAP_EVENT_08H_10H) \ __PMC_EV_ALIAS("DTLB_LOAD_MISSES.PDE_MISS", IAP_EVENT_08H_20H) \ __PMC_EV_ALIAS("DTLB_LOAD_MISSES.PDP_MISS", IAP_EVENT_08H_40H) \ __PMC_EV_ALIAS("DTLB_LOAD_MISSES.LARGE_WALK_COMPLETED", \ IAP_EVENT_08H_80H) \ __PMC_EV_ALIAS("MEM_INST_RETIRED.LOADS", IAP_EVENT_0BH_01H) \ __PMC_EV_ALIAS("MEM_INST_RETIRED.STORES", IAP_EVENT_0BH_02H) \ __PMC_EV_ALIAS("MEM_INST_RETIRED.LATENCY_ABOVE_THRESHOLD", \ IAP_EVENT_0BH_10H) \ __PMC_EV_ALIAS("MEM_STORE_RETIRED.DTLB_MISS", IAP_EVENT_0CH_01H) \ __PMC_EV_ALIAS("UOPS_ISSUED.ANY", IAP_EVENT_0EH_01H) \ __PMC_EV_ALIAS("UOPS_ISSUED.STALLED_CYCLES", IAP_EVENT_0EH_01H) \ __PMC_EV_ALIAS("UOPS_ISSUED.FUSED", IAP_EVENT_0EH_02H) \ __PMC_EV_ALIAS("MEM_UNCORE_RETIRED.L3_DATA_MISS_UNKNOWN", \ IAP_EVENT_0FH_01H) \ __PMC_EV_ALIAS("MEM_UNCORE_RETIRED.OTHER_CORE_L2_HITM", \ IAP_EVENT_0FH_02H) \ __PMC_EV_ALIAS("MEM_UNCORE_RETIRED.REMOTE_CACHE_LOCAL_HOME_HIT", \ IAP_EVENT_0FH_08H) \ __PMC_EV_ALIAS("MEM_UNCORE_RETIRED.REMOTE_DRAM", \ IAP_EVENT_0FH_10H) \ __PMC_EV_ALIAS("MEM_UNCORE_RETIRED.LOCAL_DRAM", IAP_EVENT_0FH_20H) \ __PMC_EV_ALIAS("MEM_UNCORE_RETIRED.UNCACHEABLE", IAP_EVENT_0FH_80H) \ __PMC_EV_ALIAS("FP_COMP_OPS_EXE.X87", IAP_EVENT_10H_01H) \ __PMC_EV_ALIAS("FP_COMP_OPS_EXE.MMX", IAP_EVENT_10H_02H) \ __PMC_EV_ALIAS("FP_COMP_OPS_EXE.SSE_FP", IAP_EVENT_10H_04H) \ __PMC_EV_ALIAS("FP_COMP_OPS_EXE.SSE2_INTEGER", IAP_EVENT_10H_08H) \ __PMC_EV_ALIAS("FP_COMP_OPS_EXE.SSE_FP_PACKED", IAP_EVENT_10H_10H) \ __PMC_EV_ALIAS("FP_COMP_OPS_EXE.SSE_FP_SCALAR", IAP_EVENT_10H_20H) \ __PMC_EV_ALIAS("FP_COMP_OPS_EXE.SSE_SINGLE_PRECISION", \ IAP_EVENT_10H_40H) \ __PMC_EV_ALIAS("FP_COMP_OPS_EXE.SSE_DOUBLE_PRECISION", \ IAP_EVENT_10H_80H) \ __PMC_EV_ALIAS("SIMD_INT_128.PACKED_MPY", IAP_EVENT_12H_01H) \ __PMC_EV_ALIAS("SIMD_INT_128.PACKED_SHIFT", IAP_EVENT_12H_02H) \ __PMC_EV_ALIAS("SIMD_INT_128.PACK", IAP_EVENT_12H_04H) \ __PMC_EV_ALIAS("SIMD_INT_128.UNPACK", IAP_EVENT_12H_08H) \ __PMC_EV_ALIAS("SIMD_INT_128.PACKED_LOGICAL", IAP_EVENT_12H_10H) \ __PMC_EV_ALIAS("SIMD_INT_128.PACKED_ARITH", IAP_EVENT_12H_20H) \ __PMC_EV_ALIAS("SIMD_INT_128.SHUFFLE_MOVE", IAP_EVENT_12H_40H) \ __PMC_EV_ALIAS("LOAD_DISPATCH.RS", IAP_EVENT_13H_01H) \ __PMC_EV_ALIAS("LOAD_DISPATCH.RS_DELAYED", IAP_EVENT_13H_02H) \ __PMC_EV_ALIAS("LOAD_DISPATCH.MOB", IAP_EVENT_13H_04H) \ __PMC_EV_ALIAS("LOAD_DISPATCH.ANY", IAP_EVENT_13H_07H) \ __PMC_EV_ALIAS("ARITH.CYCLES_DIV_BUSY", IAP_EVENT_14H_01H) \ __PMC_EV_ALIAS("ARITH.MUL", IAP_EVENT_14H_02H) \ __PMC_EV_ALIAS("INST_QUEUE_WRITES", IAP_EVENT_17H_01H) \ __PMC_EV_ALIAS("INST_DECODED.DEC0", IAP_EVENT_18H_01H) \ __PMC_EV_ALIAS("TWO_UOP_INSTS_DECODED", IAP_EVENT_19H_01H) \ __PMC_EV_ALIAS("INST_QUEUE_WRITE_CYCLES", IAP_EVENT_1EH_01H) \ __PMC_EV_ALIAS("LSD_OVERFLOW", IAP_EVENT_20H_01H) \ __PMC_EV_ALIAS("L2_RQSTS.LD_HIT", IAP_EVENT_24H_01H) \ __PMC_EV_ALIAS("L2_RQSTS.LD_MISS", IAP_EVENT_24H_02H) \ __PMC_EV_ALIAS("L2_RQSTS.LOADS", IAP_EVENT_24H_03H) \ __PMC_EV_ALIAS("L2_RQSTS.RFO_HIT", IAP_EVENT_24H_04H) \ __PMC_EV_ALIAS("L2_RQSTS.RFO_MISS", IAP_EVENT_24H_08H) \ __PMC_EV_ALIAS("L2_RQSTS.RFOS", IAP_EVENT_24H_0CH) \ __PMC_EV_ALIAS("L2_RQSTS.IFETCH_HIT", IAP_EVENT_24H_10H) \ __PMC_EV_ALIAS("L2_RQSTS.IFETCH_MISS", IAP_EVENT_24H_20H) \ __PMC_EV_ALIAS("L2_RQSTS.IFETCHES", IAP_EVENT_24H_30H) \ __PMC_EV_ALIAS("L2_RQSTS.PREFETCH_HIT", IAP_EVENT_24H_40H) \ __PMC_EV_ALIAS("L2_RQSTS.PREFETCH_MISS", IAP_EVENT_24H_80H) \ __PMC_EV_ALIAS("L2_RQSTS.PREFETCHES", IAP_EVENT_24H_C0H) \ __PMC_EV_ALIAS("L2_RQSTS.MISS", IAP_EVENT_24H_AAH) \ __PMC_EV_ALIAS("L2_RQSTS.REFERENCES", IAP_EVENT_24H_FFH) \ __PMC_EV_ALIAS("L2_DATA_RQSTS.DEMAND.I_STATE", IAP_EVENT_26H_01H) \ __PMC_EV_ALIAS("L2_DATA_RQSTS.DEMAND.S_STATE", IAP_EVENT_26H_02H) \ __PMC_EV_ALIAS("L2_DATA_RQSTS.DEMAND.E_STATE", IAP_EVENT_26H_04H) \ __PMC_EV_ALIAS("L2_DATA_RQSTS.DEMAND.M_STATE", IAP_EVENT_26H_08H) \ __PMC_EV_ALIAS("L2_DATA_RQSTS.DEMAND.MESI", IAP_EVENT_26H_0FH) \ __PMC_EV_ALIAS("L2_DATA_RQSTS.PREFETCH.I_STATE", IAP_EVENT_26H_10H) \ __PMC_EV_ALIAS("L2_DATA_RQSTS.PREFETCH.S_STATE", IAP_EVENT_26H_20H) \ __PMC_EV_ALIAS("L2_DATA_RQSTS.PREFETCH.E_STATE", IAP_EVENT_26H_40H) \ __PMC_EV_ALIAS("L2_DATA_RQSTS.PREFETCH.M_STATE", IAP_EVENT_26H_80H) \ __PMC_EV_ALIAS("L2_DATA_RQSTS.PREFETCH.MESI", IAP_EVENT_26H_F0H) \ __PMC_EV_ALIAS("L2_DATA_RQSTS.ANY", IAP_EVENT_26H_FFH) \ __PMC_EV_ALIAS("L2_WRITE.RFO.I_STATE", IAP_EVENT_27H_01H) \ __PMC_EV_ALIAS("L2_WRITE.RFO.S_STATE", IAP_EVENT_27H_02H) \ __PMC_EV_ALIAS("L2_WRITE.RFO.M_STATE", IAP_EVENT_27H_08H) \ __PMC_EV_ALIAS("L2_WRITE.RFO.HIT", IAP_EVENT_27H_0EH) \ __PMC_EV_ALIAS("L2_WRITE.RFO.MESI", IAP_EVENT_27H_0FH) \ __PMC_EV_ALIAS("L2_WRITE.LOCK.I_STATE", IAP_EVENT_27H_10H) \ __PMC_EV_ALIAS("L2_WRITE.LOCK.S_STATE", IAP_EVENT_27H_20H) \ __PMC_EV_ALIAS("L2_WRITE.LOCK.E_STATE", IAP_EVENT_27H_40H) \ __PMC_EV_ALIAS("L2_WRITE.LOCK.M_STATE", IAP_EVENT_27H_80H) \ __PMC_EV_ALIAS("L2_WRITE.LOCK.HIT", IAP_EVENT_27H_E0H) \ __PMC_EV_ALIAS("L2_WRITE.LOCK.MESI", IAP_EVENT_27H_F0H) \ __PMC_EV_ALIAS("L1D_WB_L2.I_STATE", IAP_EVENT_28H_01H) \ __PMC_EV_ALIAS("L1D_WB_L2.S_STATE", IAP_EVENT_28H_02H) \ __PMC_EV_ALIAS("L1D_WB_L2.E_STATE", IAP_EVENT_28H_04H) \ __PMC_EV_ALIAS("L1D_WB_L2.M_STATE", IAP_EVENT_28H_08H) \ __PMC_EV_ALIAS("L1D_WB_L2.MESI", IAP_EVENT_28H_0FH) \ __PMC_EV_ALIAS("L3_LAT_CACHE.REFERENCE", IAP_EVENT_2EH_4FH) \ __PMC_EV_ALIAS("L3_LAT_CACHE.MISS", IAP_EVENT_2EH_41H) \ __PMC_EV_ALIAS("CPU_CLK_UNHALTED.THREAD_P", IAP_EVENT_3CH_00H) \ __PMC_EV_ALIAS("CPU_CLK_UNHALTED.REF_P", IAP_EVENT_3CH_01H) \ __PMC_EV_ALIAS("L1D_CACHE_LD.I_STATE", IAP_EVENT_40H_01H) \ __PMC_EV_ALIAS("L1D_CACHE_LD.S_STATE", IAP_EVENT_40H_02H) \ __PMC_EV_ALIAS("L1D_CACHE_LD.E_STATE", IAP_EVENT_40H_04H) \ __PMC_EV_ALIAS("L1D_CACHE_LD.M_STATE", IAP_EVENT_40H_08H) \ __PMC_EV_ALIAS("L1D_CACHE_LD.MESI", IAP_EVENT_40H_0FH) \ __PMC_EV_ALIAS("L1D_CACHE_ST.S_STATE", IAP_EVENT_41H_02H) \ __PMC_EV_ALIAS("L1D_CACHE_ST.E_STATE", IAP_EVENT_41H_04H) \ __PMC_EV_ALIAS("L1D_CACHE_ST.M_STATE", IAP_EVENT_41H_08H) \ __PMC_EV_ALIAS("L1D_CACHE_LOCK.HIT", IAP_EVENT_42H_01H) \ __PMC_EV_ALIAS("L1D_CACHE_LOCK.S_STATE", IAP_EVENT_42H_02H) \ __PMC_EV_ALIAS("L1D_CACHE_LOCK.E_STATE", IAP_EVENT_42H_04H) \ __PMC_EV_ALIAS("L1D_CACHE_LOCK.M_STATE", IAP_EVENT_42H_08H) \ __PMC_EV_ALIAS("L1D_ALL_REF.ANY", IAP_EVENT_43H_01H) \ __PMC_EV_ALIAS("L1D_ALL_REF.CACHEABLE", IAP_EVENT_43H_02H) \ __PMC_EV_ALIAS("L1D_PEND_MISS.LOAD_BUFFERS_FULL", IAP_EVENT_48H_02H) \ __PMC_EV_ALIAS("DTLB_MISSES.ANY", IAP_EVENT_49H_01H) \ __PMC_EV_ALIAS("DTLB_MISSES.WALK_COMPLETED", IAP_EVENT_49H_02H) \ __PMC_EV_ALIAS("DTLB_MISSES.STLB_HIT", IAP_EVENT_49H_10H) \ __PMC_EV_ALIAS("LOAD_HIT_PRE", IAP_EVENT_4CH_01H) \ __PMC_EV_ALIAS("L1D_PREFETCH.REQUESTS", IAP_EVENT_4EH_01H) \ __PMC_EV_ALIAS("L1D_PREFETCH.MISS", IAP_EVENT_4EH_02H) \ __PMC_EV_ALIAS("L1D_PREFETCH.TRIGGERS", IAP_EVENT_4EH_04H) \ __PMC_EV_ALIAS("L1D.REPL", IAP_EVENT_51H_01H) \ __PMC_EV_ALIAS("L1D.M_REPL", IAP_EVENT_51H_02H) \ __PMC_EV_ALIAS("L1D.M_EVICT", IAP_EVENT_51H_04H) \ __PMC_EV_ALIAS("L1D.M_SNOOP_EVICT", IAP_EVENT_51H_08H) \ __PMC_EV_ALIAS("L1D_CACHE_PREFETCH_LOCK_FB_HIT", IAP_EVENT_52H_01H) \ __PMC_EV_ALIAS("L1D_CACHE_LOCK_FB_HIT", IAP_EVENT_53H_01H) \ __PMC_EV_ALIAS("CACHE_LOCK_CYCLES.L1D_L2", IAP_EVENT_63H_01H) \ __PMC_EV_ALIAS("CACHE_LOCK_CYCLES.L1D", IAP_EVENT_63H_02H) \ __PMC_EV_ALIAS("IO_TRANSACTIONS", IAP_EVENT_6CH_01H) \ __PMC_EV_ALIAS("L1I.HITS", IAP_EVENT_80H_01H) \ __PMC_EV_ALIAS("L1I.MISSES", IAP_EVENT_80H_02H) \ __PMC_EV_ALIAS("L1I.READS", IAP_EVENT_80H_03H) \ __PMC_EV_ALIAS("L1I.CYCLES_STALLED", IAP_EVENT_80H_04H) \ __PMC_EV_ALIAS("LARGE_ITLB.HIT", IAP_EVENT_82H_01H) \ __PMC_EV_ALIAS("ITLB_MISSES.ANY", IAP_EVENT_85H_01H) \ __PMC_EV_ALIAS("ITLB_MISSES.WALK_COMPLETED", IAP_EVENT_85H_02H) \ __PMC_EV_ALIAS("ILD_STALL.LCP", IAP_EVENT_87H_01H) \ __PMC_EV_ALIAS("ILD_STALL.MRU", IAP_EVENT_87H_02H) \ __PMC_EV_ALIAS("ILD_STALL.IQ_FULL", IAP_EVENT_87H_04H) \ __PMC_EV_ALIAS("ILD_STALL.REGEN", IAP_EVENT_87H_08H) \ __PMC_EV_ALIAS("ILD_STALL.ANY", IAP_EVENT_87H_0FH) \ __PMC_EV_ALIAS("BR_INST_EXEC.COND", IAP_EVENT_88H_01H) \ __PMC_EV_ALIAS("BR_INST_EXEC.DIRECT", IAP_EVENT_88H_02H) \ __PMC_EV_ALIAS("BR_INST_EXEC.INDIRECT_NON_CALL", IAP_EVENT_88H_04H) \ __PMC_EV_ALIAS("BR_INST_EXEC.NON_CALLS", IAP_EVENT_88H_07H) \ __PMC_EV_ALIAS("BR_INST_EXEC.RETURN_NEAR", IAP_EVENT_88H_08H) \ __PMC_EV_ALIAS("BR_INST_EXEC.DIRECT_NEAR_CALL", IAP_EVENT_88H_10H) \ __PMC_EV_ALIAS("BR_INST_EXEC.INDIRECT_NEAR_CALL", IAP_EVENT_88H_20H) \ __PMC_EV_ALIAS("BR_INST_EXEC.NEAR_CALLS", IAP_EVENT_88H_30H) \ __PMC_EV_ALIAS("BR_INST_EXEC.TAKEN", IAP_EVENT_88H_40H) \ __PMC_EV_ALIAS("BR_INST_EXEC.ANY", IAP_EVENT_88H_7FH) \ __PMC_EV_ALIAS("BR_MISP_EXEC.COND", IAP_EVENT_89H_01H) \ __PMC_EV_ALIAS("BR_MISP_EXEC.DIRECT", IAP_EVENT_89H_02H) \ __PMC_EV_ALIAS("BR_MISP_EXEC.INDIRECT_NON_CALL", IAP_EVENT_89H_04H) \ __PMC_EV_ALIAS("BR_MISP_EXEC.NON_CALLS", IAP_EVENT_89H_07H) \ __PMC_EV_ALIAS("BR_MISP_EXEC.RETURN_NEAR", IAP_EVENT_89H_08H) \ __PMC_EV_ALIAS("BR_MISP_EXEC.DIRECT_NEAR_CALL", IAP_EVENT_89H_10H) \ __PMC_EV_ALIAS("BR_MISP_EXEC.INDIRECT_NEAR_CALL", IAP_EVENT_89H_20H) \ __PMC_EV_ALIAS("BR_MISP_EXEC.NEAR_CALLS", IAP_EVENT_89H_30H) \ __PMC_EV_ALIAS("BR_MISP_EXEC.TAKEN", IAP_EVENT_89H_40H) \ __PMC_EV_ALIAS("BR_MISP_EXEC.ANY", IAP_EVENT_89H_7FH) \ __PMC_EV_ALIAS("RESOURCE_STALLS.ANY", IAP_EVENT_A2H_01H) \ __PMC_EV_ALIAS("RESOURCE_STALLS.LOAD", IAP_EVENT_A2H_02H) \ __PMC_EV_ALIAS("RESOURCE_STALLS.RS_FULL", IAP_EVENT_A2H_04H) \ __PMC_EV_ALIAS("RESOURCE_STALLS.STORE", IAP_EVENT_A2H_08H) \ __PMC_EV_ALIAS("RESOURCE_STALLS.ROB_FULL", IAP_EVENT_A2H_10H) \ __PMC_EV_ALIAS("RESOURCE_STALLS.FPCW", IAP_EVENT_A2H_20H) \ __PMC_EV_ALIAS("RESOURCE_STALLS.MXCSR", IAP_EVENT_A2H_40H) \ __PMC_EV_ALIAS("RESOURCE_STALLS.OTHER", IAP_EVENT_A2H_80H) \ __PMC_EV_ALIAS("MACRO_INSTS.FUSIONS_DECODED", IAP_EVENT_A6H_01H) \ __PMC_EV_ALIAS("BACLEAR_FORCE_IQ", IAP_EVENT_A7H_01H) \ __PMC_EV_ALIAS("LSD.UOPS", IAP_EVENT_A8H_01H) \ __PMC_EV_ALIAS("ITLB_FLUSH", IAP_EVENT_AEH_01H) \ __PMC_EV_ALIAS("OFFCORE_REQUESTS.L1D_WRITEBACK", IAP_EVENT_B0H_40H) \ __PMC_EV_ALIAS("UOPS_EXECUTED.PORT0", IAP_EVENT_B1H_01H) \ __PMC_EV_ALIAS("UOPS_EXECUTED.PORT1", IAP_EVENT_B1H_02H) \ __PMC_EV_ALIAS("UOPS_EXECUTED.PORT2_CORE", IAP_EVENT_B1H_04H) \ __PMC_EV_ALIAS("UOPS_EXECUTED.PORT3_CORE", IAP_EVENT_B1H_08H) \ __PMC_EV_ALIAS("UOPS_EXECUTED.PORT4_CORE", IAP_EVENT_B1H_10H) \ __PMC_EV_ALIAS("UOPS_EXECUTED.CORE_ACTIVE_CYCLES_NO_PORT5", \ IAP_EVENT_B1H_1FH) \ __PMC_EV_ALIAS("UOPS_EXECUTED.PORT5", IAP_EVENT_B1H_20H) \ __PMC_EV_ALIAS("UOPS_EXECUTED.CORE_ACTIVE_CYCLES", IAP_EVENT_B1H_3FH) \ __PMC_EV_ALIAS("UOPS_EXECUTED.PORT015", IAP_EVENT_B1H_40H) \ __PMC_EV_ALIAS("UOPS_EXECUTED.PORT234", IAP_EVENT_B1H_80H) \ __PMC_EV_ALIAS("OFFCORE_REQUESTS_SQ_FULL", IAP_EVENT_B2H_01H) \ __PMC_EV_ALIAS("OFF_CORE_RESPONSE_0", IAP_EVENT_B7H_01H) \ __PMC_EV_ALIAS("SNOOP_RESPONSE.HIT", IAP_EVENT_B8H_01H) \ __PMC_EV_ALIAS("SNOOP_RESPONSE.HITE", IAP_EVENT_B8H_02H) \ __PMC_EV_ALIAS("SNOOP_RESPONSE.HITM", IAP_EVENT_B8H_04H) \ __PMC_EV_ALIAS("OFF_CORE_RESPONSE_1", IAP_EVENT_BBH_01H) \ __PMC_EV_ALIAS("INST_RETIRED.ANY_P", IAP_EVENT_C0H_01H) \ __PMC_EV_ALIAS("INST_RETIRED.X87", IAP_EVENT_C0H_02H) \ __PMC_EV_ALIAS("INST_RETIRED.MMX", IAP_EVENT_C0H_04H) \ __PMC_EV_ALIAS("UOPS_RETIRED.ANY", IAP_EVENT_C2H_01H) \ __PMC_EV_ALIAS("UOPS_RETIRED.RETIRE_SLOTS", IAP_EVENT_C2H_02H) \ __PMC_EV_ALIAS("UOPS_RETIRED.MACRO_FUSED", IAP_EVENT_C2H_04H) \ __PMC_EV_ALIAS("MACHINE_CLEARS.CYCLES", IAP_EVENT_C3H_01H) \ __PMC_EV_ALIAS("MACHINE_CLEARS.MEM_ORDER", IAP_EVENT_C3H_02H) \ __PMC_EV_ALIAS("MACHINE_CLEARS.SMC", IAP_EVENT_C3H_04H) \ __PMC_EV_ALIAS("BR_INST_RETIRED.ALL_BRANCHES", IAP_EVENT_C4H_00H) \ __PMC_EV_ALIAS("BR_INST_RETIRED.CONDITIONAL", IAP_EVENT_C4H_01H) \ __PMC_EV_ALIAS("BR_INST_RETIRED.NEAR_CALL", IAP_EVENT_C4H_02H) \ __PMC_EV_ALIAS("BR_INST_RETIRED.ALL_BRANCHES", IAP_EVENT_C4H_04H) \ __PMC_EV_ALIAS("BR_MISP_RETIRED.ALL_BRANCHES", IAP_EVENT_C5H_00H) \ __PMC_EV_ALIAS("BR_MISP_RETIRED.NEAR_CALL", IAP_EVENT_C5H_02H) \ __PMC_EV_ALIAS("SSEX_UOPS_RETIRED.PACKED_SINGLE", IAP_EVENT_C7H_01H) \ __PMC_EV_ALIAS("SSEX_UOPS_RETIRED.SCALAR_SINGLE", IAP_EVENT_C7H_02H) \ __PMC_EV_ALIAS("SSEX_UOPS_RETIRED.PACKED_DOUBLE", IAP_EVENT_C7H_04H) \ __PMC_EV_ALIAS("SSEX_UOPS_RETIRED.SCALAR_DOUBLE", IAP_EVENT_C7H_08H) \ __PMC_EV_ALIAS("SSEX_UOPS_RETIRED.VECTOR_INTEGER", IAP_EVENT_C7H_10H) \ __PMC_EV_ALIAS("ITLB_MISS_RETIRED", IAP_EVENT_C8H_20H) \ __PMC_EV_ALIAS("MEM_LOAD_RETIRED.L1D_HIT", IAP_EVENT_CBH_01H) \ __PMC_EV_ALIAS("MEM_LOAD_RETIRED.L2_HIT", IAP_EVENT_CBH_02H) \ __PMC_EV_ALIAS("MEM_LOAD_RETIRED.L3_UNSHARED_HIT", IAP_EVENT_CBH_04H) \ __PMC_EV_ALIAS("MEM_LOAD_RETIRED.OTHER_CORE_L2_HIT_HITM", \ IAP_EVENT_CBH_08H) \ __PMC_EV_ALIAS("MEM_LOAD_RETIRED.L3_MISS", IAP_EVENT_CBH_10H) \ __PMC_EV_ALIAS("MEM_LOAD_RETIRED.HIT_LFB", IAP_EVENT_CBH_40H) \ __PMC_EV_ALIAS("MEM_LOAD_RETIRED.DTLB_MISS", IAP_EVENT_CBH_80H) \ __PMC_EV_ALIAS("FP_MMX_TRANS.TO_FP", IAP_EVENT_CCH_01H) \ __PMC_EV_ALIAS("FP_MMX_TRANS.TO_MMX", IAP_EVENT_CCH_02H) \ __PMC_EV_ALIAS("FP_MMX_TRANS.ANY", IAP_EVENT_CCH_03H) \ __PMC_EV_ALIAS("MACRO_INSTS.DECODED", IAP_EVENT_D0H_01H) \ __PMC_EV_ALIAS("UOPS_DECODED.MS", IAP_EVENT_D1H_02H) \ __PMC_EV_ALIAS("UOPS_DECODED.ESP_FOLDING", IAP_EVENT_D1H_04H) \ __PMC_EV_ALIAS("UOPS_DECODED.ESP_SYNC", IAP_EVENT_D1H_08H) \ __PMC_EV_ALIAS("RAT_STALLS.FLAGS", IAP_EVENT_D2H_01H) \ __PMC_EV_ALIAS("RAT_STALLS.REGISTERS", IAP_EVENT_D2H_02H) \ __PMC_EV_ALIAS("RAT_STALLS.ROB_READ_PORT", IAP_EVENT_D2H_04H) \ __PMC_EV_ALIAS("RAT_STALLS.SCOREBOARD", IAP_EVENT_D2H_08H) \ __PMC_EV_ALIAS("RAT_STALLS.ANY", IAP_EVENT_D2H_0FH) \ __PMC_EV_ALIAS("SEG_RENAME_STALLS", IAP_EVENT_D4H_01H) \ __PMC_EV_ALIAS("ES_REG_RENAMES", IAP_EVENT_D5H_01H) \ __PMC_EV_ALIAS("UOP_UNFUSION", IAP_EVENT_DBH_01H) \ __PMC_EV_ALIAS("BR_INST_DECODED", IAP_EVENT_E0H_01H) \ __PMC_EV_ALIAS("BPU_MISSED_CALL_RET", IAP_EVENT_E5H_01H) \ __PMC_EV_ALIAS("BACLEAR.CLEAR", IAP_EVENT_E6H_01H) \ __PMC_EV_ALIAS("BACLEAR.BAD_TARGET", IAP_EVENT_E6H_02H) \ __PMC_EV_ALIAS("BPU_CLEARS.EARLY", IAP_EVENT_E8H_01H) \ __PMC_EV_ALIAS("BPU_CLEARS.LATE", IAP_EVENT_E8H_02H) \ __PMC_EV_ALIAS("BPU_CLEARS.ANY", IAP_EVENT_E8H_03H) \ __PMC_EV_ALIAS("L2_TRANSACTIONS.LOAD", IAP_EVENT_F0H_01H) \ __PMC_EV_ALIAS("L2_TRANSACTIONS.RFO", IAP_EVENT_F0H_02H) \ __PMC_EV_ALIAS("L2_TRANSACTIONS.IFETCH", IAP_EVENT_F0H_04H) \ __PMC_EV_ALIAS("L2_TRANSACTIONS.PREFETCH", IAP_EVENT_F0H_08H) \ __PMC_EV_ALIAS("L2_TRANSACTIONS.L1D_WB", IAP_EVENT_F0H_10H) \ __PMC_EV_ALIAS("L2_TRANSACTIONS.FILL", IAP_EVENT_F0H_20H) \ __PMC_EV_ALIAS("L2_TRANSACTIONS.WB", IAP_EVENT_F0H_40H) \ __PMC_EV_ALIAS("L2_TRANSACTIONS.ANY", IAP_EVENT_F0H_80H) \ __PMC_EV_ALIAS("L2_LINES_IN.S_STATE", IAP_EVENT_F1H_02H) \ __PMC_EV_ALIAS("L2_LINES_IN.E_STATE", IAP_EVENT_F1H_04H) \ __PMC_EV_ALIAS("L2_LINES_IN.ANY", IAP_EVENT_F1H_07H) \ __PMC_EV_ALIAS("L2_LINES_OUT.DEMAND_CLEAN", IAP_EVENT_F2H_01H) \ __PMC_EV_ALIAS("L2_LINES_OUT.DEMAND_DIRTY", IAP_EVENT_F2H_02H) \ __PMC_EV_ALIAS("L2_LINES_OUT.PREFETCH_CLEAN", IAP_EVENT_F2H_04H) \ __PMC_EV_ALIAS("L2_LINES_OUT.PREFETCH_DIRTY", IAP_EVENT_F2H_08H) \ __PMC_EV_ALIAS("L2_LINES_OUT.ANY", IAP_EVENT_F2H_0FH) \ __PMC_EV_ALIAS("SQ_MISC.SPLIT_LOCK", IAP_EVENT_F4H_10H) \ __PMC_EV_ALIAS("SQ_FULL_STALL_CYCLES", IAP_EVENT_F6H_01H) \ __PMC_EV_ALIAS("FP_ASSIST.ALL", IAP_EVENT_F7H_01H) \ __PMC_EV_ALIAS("FP_ASSIST.OUTPUT", IAP_EVENT_F7H_02H) \ __PMC_EV_ALIAS("FP_ASSIST.INPUT", IAP_EVENT_F7H_04H) \ __PMC_EV_ALIAS("SIMD_INT_64.PACKED_MPY", IAP_EVENT_FDH_01H) \ __PMC_EV_ALIAS("SIMD_INT_64.PACKED_SHIFT", IAP_EVENT_FDH_02H) \ __PMC_EV_ALIAS("SIMD_INT_64.PACK", IAP_EVENT_FDH_04H) \ __PMC_EV_ALIAS("SIMD_INT_64.UNPACK", IAP_EVENT_FDH_08H) \ __PMC_EV_ALIAS("SIMD_INT_64.PACKED_LOGICAL", IAP_EVENT_FDH_10H) \ __PMC_EV_ALIAS("SIMD_INT_64.PACKED_ARITH", IAP_EVENT_FDH_20H) \ __PMC_EV_ALIAS("SIMD_INT_64.SHUFFLE_MOVE", IAP_EVENT_FDH_40H) \ __PMC_EV_ALIAS_COREI7_OLD() /* * Aliases for Westmere PMC events (253669-033US December 2009) */ #define __PMC_EV_ALIAS_WESTMERE() \ __PMC_EV_ALIAS_INTEL_ARCHITECTURAL() \ __PMC_EV_ALIAS("LOAD_BLOCK.OVERLAP_STORE", IAP_EVENT_03H_02H) \ __PMC_EV_ALIAS("SB_DRAIN.ANY", IAP_EVENT_04H_07H) \ __PMC_EV_ALIAS("MISALIGN_MEMORY.STORE", IAP_EVENT_05H_02H) \ __PMC_EV_ALIAS("STORE_BLOCKS.AT_RET", IAP_EVENT_06H_04H) \ __PMC_EV_ALIAS("STORE_BLOCKS.L1D_BLOCK", IAP_EVENT_06H_08H) \ __PMC_EV_ALIAS("PARTIAL_ADDRESS_ALIAS", IAP_EVENT_07H_01H) \ __PMC_EV_ALIAS("DTLB_LOAD_MISSES.ANY", IAP_EVENT_08H_01H) \ __PMC_EV_ALIAS("DTLB_LOAD_MISSES.WALK_COMPLETED", IAP_EVENT_08H_02H) \ __PMC_EV_ALIAS("DTLB_LOAD_MISSES.WALK_CYCLES", IAP_EVENT_08H_04H) \ __PMC_EV_ALIAS("DTLB_LOAD_MISSES.STLB_HIT", IAP_EVENT_08H_10H) \ __PMC_EV_ALIAS("DTLB_LOAD_MISSES.PDE_MISS", IAP_EVENT_08H_20H) \ __PMC_EV_ALIAS("MEM_INST_RETIRED.LOADS", IAP_EVENT_0BH_01H) \ __PMC_EV_ALIAS("MEM_INST_RETIRED.STORES", IAP_EVENT_0BH_02H) \ __PMC_EV_ALIAS("MEM_INST_RETIRED.LATENCY_ABOVE_THRESHOLD", \ IAP_EVENT_0BH_10H) \ __PMC_EV_ALIAS("MEM_STORE_RETIRED.DTLB_MISS", IAP_EVENT_0CH_01H) \ __PMC_EV_ALIAS("UOPS_ISSUED.ANY", IAP_EVENT_0EH_01H) \ __PMC_EV_ALIAS("UOPS_ISSUED.STALLED_CYCLES", IAP_EVENT_0EH_01H) \ __PMC_EV_ALIAS("UOPS_ISSUED.FUSED", IAP_EVENT_0EH_02H) \ __PMC_EV_ALIAS("MEM_UNCORE_RETIRED.LOCAL_HITM", IAP_EVENT_0FH_02H) \ __PMC_EV_ALIAS("MEM_UNCORE_RETIRED.LOCAL_DRAM_AND_REMOTE_CACHE_HIT", \ IAP_EVENT_0FH_08H) \ __PMC_EV_ALIAS("MEM_UNCORE_RETIRED.LOCAL_DRAM", IAP_EVENT_0FH_10H) \ __PMC_EV_ALIAS("MEM_UNCORE_RETIRED.REMOTE_DRAM", IAP_EVENT_0FH_20H) \ __PMC_EV_ALIAS("MEM_UNCORE_RETIRED.UNCACHEABLE", IAP_EVENT_0FH_80H) \ __PMC_EV_ALIAS("FP_COMP_OPS_EXE.X87", IAP_EVENT_10H_01H) \ __PMC_EV_ALIAS("FP_COMP_OPS_EXE.MMX", IAP_EVENT_10H_02H) \ __PMC_EV_ALIAS("FP_COMP_OPS_EXE.SSE_FP", IAP_EVENT_10H_04H) \ __PMC_EV_ALIAS("FP_COMP_OPS_EXE.SSE2_INTEGER", IAP_EVENT_10H_08H) \ __PMC_EV_ALIAS("FP_COMP_OPS_EXE.SSE_FP_PACKED", IAP_EVENT_10H_10H) \ __PMC_EV_ALIAS("FP_COMP_OPS_EXE.SSE_FP_SCALAR", IAP_EVENT_10H_20H) \ __PMC_EV_ALIAS("FP_COMP_OPS_EXE.SSE_SINGLE_PRECISION", \ IAP_EVENT_10H_40H) \ __PMC_EV_ALIAS("FP_COMP_OPS_EXE.SSE_DOUBLE_PRECISION", \ IAP_EVENT_10H_80H) \ __PMC_EV_ALIAS("SIMD_INT_128.PACKED_MPY", IAP_EVENT_12H_01H) \ __PMC_EV_ALIAS("SIMD_INT_128.PACKED_SHIFT", IAP_EVENT_12H_02H) \ __PMC_EV_ALIAS("SIMD_INT_128.PACK", IAP_EVENT_12H_04H) \ __PMC_EV_ALIAS("SIMD_INT_128.UNPACK", IAP_EVENT_12H_08H) \ __PMC_EV_ALIAS("SIMD_INT_128.PACKED_LOGICAL", IAP_EVENT_12H_10H) \ __PMC_EV_ALIAS("SIMD_INT_128.PACKED_ARITH", IAP_EVENT_12H_20H) \ __PMC_EV_ALIAS("SIMD_INT_128.SHUFFLE_MOVE", IAP_EVENT_12H_40H) \ __PMC_EV_ALIAS("LOAD_DISPATCH.RS", IAP_EVENT_13H_01H) \ __PMC_EV_ALIAS("LOAD_DISPATCH.RS_DELAYED", IAP_EVENT_13H_02H) \ __PMC_EV_ALIAS("LOAD_DISPATCH.MOB", IAP_EVENT_13H_04H) \ __PMC_EV_ALIAS("LOAD_DISPATCH.ANY", IAP_EVENT_13H_07H) \ __PMC_EV_ALIAS("ARITH.CYCLES_DIV_BUSY", IAP_EVENT_14H_01H) \ __PMC_EV_ALIAS("ARITH.MUL", IAP_EVENT_14H_02H) \ __PMC_EV_ALIAS("INST_QUEUE_WRITES", IAP_EVENT_17H_01H) \ __PMC_EV_ALIAS("INST_DECODED.DEC0", IAP_EVENT_18H_01H) \ __PMC_EV_ALIAS("TWO_UOP_INSTS_DECODED", IAP_EVENT_19H_01H) \ __PMC_EV_ALIAS("INST_QUEUE_WRITE_CYCLES", IAP_EVENT_1EH_01H) \ __PMC_EV_ALIAS("LSD_OVERFLOW", IAP_EVENT_20H_01H) \ __PMC_EV_ALIAS("L2_RQSTS.LD_HIT", IAP_EVENT_24H_01H) \ __PMC_EV_ALIAS("L2_RQSTS.LD_MISS", IAP_EVENT_24H_02H) \ __PMC_EV_ALIAS("L2_RQSTS.LOADS", IAP_EVENT_24H_03H) \ __PMC_EV_ALIAS("L2_RQSTS.RFO_HIT", IAP_EVENT_24H_04H) \ __PMC_EV_ALIAS("L2_RQSTS.RFO_MISS", IAP_EVENT_24H_08H) \ __PMC_EV_ALIAS("L2_RQSTS.RFOS", IAP_EVENT_24H_0CH) \ __PMC_EV_ALIAS("L2_RQSTS.IFETCH_HIT", IAP_EVENT_24H_10H) \ __PMC_EV_ALIAS("L2_RQSTS.IFETCH_MISS", IAP_EVENT_24H_20H) \ __PMC_EV_ALIAS("L2_RQSTS.IFETCHES", IAP_EVENT_24H_30H) \ __PMC_EV_ALIAS("L2_RQSTS.PREFETCH_HIT", IAP_EVENT_24H_40H) \ __PMC_EV_ALIAS("L2_RQSTS.PREFETCH_MISS", IAP_EVENT_24H_80H) \ __PMC_EV_ALIAS("L2_RQSTS.PREFETCHES", IAP_EVENT_24H_C0H) \ __PMC_EV_ALIAS("L2_RQSTS.MISS", IAP_EVENT_24H_AAH) \ __PMC_EV_ALIAS("L2_RQSTS.REFERENCES", IAP_EVENT_24H_FFH) \ __PMC_EV_ALIAS("L2_DATA_RQSTS.DEMAND.I_STATE", IAP_EVENT_26H_01H) \ __PMC_EV_ALIAS("L2_DATA_RQSTS.DEMAND.S_STATE", IAP_EVENT_26H_02H) \ __PMC_EV_ALIAS("L2_DATA_RQSTS.DEMAND.E_STATE", IAP_EVENT_26H_04H) \ __PMC_EV_ALIAS("L2_DATA_RQSTS.DEMAND.M_STATE", IAP_EVENT_26H_08H) \ __PMC_EV_ALIAS("L2_DATA_RQSTS.DEMAND.MESI", IAP_EVENT_26H_0FH) \ __PMC_EV_ALIAS("L2_DATA_RQSTS.PREFETCH.I_STATE", IAP_EVENT_26H_10H) \ __PMC_EV_ALIAS("L2_DATA_RQSTS.PREFETCH.S_STATE", IAP_EVENT_26H_20H) \ __PMC_EV_ALIAS("L2_DATA_RQSTS.PREFETCH.E_STATE", IAP_EVENT_26H_40H) \ __PMC_EV_ALIAS("L2_DATA_RQSTS.PREFETCH.M_STATE", IAP_EVENT_26H_80H) \ __PMC_EV_ALIAS("L2_DATA_RQSTS.PREFETCH.MESI", IAP_EVENT_26H_F0H) \ __PMC_EV_ALIAS("L2_DATA_RQSTS.ANY", IAP_EVENT_26H_FFH) \ __PMC_EV_ALIAS("L2_WRITE.RFO.I_STATE", IAP_EVENT_27H_01H) \ __PMC_EV_ALIAS("L2_WRITE.RFO.S_STATE", IAP_EVENT_27H_02H) \ __PMC_EV_ALIAS("L2_WRITE.RFO.M_STATE", IAP_EVENT_27H_08H) \ __PMC_EV_ALIAS("L2_WRITE.RFO.HIT", IAP_EVENT_27H_0EH) \ __PMC_EV_ALIAS("L2_WRITE.RFO.MESI", IAP_EVENT_27H_0FH) \ __PMC_EV_ALIAS("L2_WRITE.LOCK.I_STATE", IAP_EVENT_27H_10H) \ __PMC_EV_ALIAS("L2_WRITE.LOCK.S_STATE", IAP_EVENT_27H_20H) \ __PMC_EV_ALIAS("L2_WRITE.LOCK.E_STATE", IAP_EVENT_27H_40H) \ __PMC_EV_ALIAS("L2_WRITE.LOCK.M_STATE", IAP_EVENT_27H_80H) \ __PMC_EV_ALIAS("L2_WRITE.LOCK.HIT", IAP_EVENT_27H_E0H) \ __PMC_EV_ALIAS("L2_WRITE.LOCK.MESI", IAP_EVENT_27H_F0H) \ __PMC_EV_ALIAS("L1D_WB_L2.I_STATE", IAP_EVENT_28H_01H) \ __PMC_EV_ALIAS("L1D_WB_L2.S_STATE", IAP_EVENT_28H_02H) \ __PMC_EV_ALIAS("L1D_WB_L2.E_STATE", IAP_EVENT_28H_04H) \ __PMC_EV_ALIAS("L1D_WB_L2.M_STATE", IAP_EVENT_28H_08H) \ __PMC_EV_ALIAS("L1D_WB_L2.MESI", IAP_EVENT_28H_0FH) \ __PMC_EV_ALIAS("L3_LAT_CACHE.REFERENCE", IAP_EVENT_2EH_02H) \ __PMC_EV_ALIAS("L3_LAT_CACHE.MISS", IAP_EVENT_2EH_01H) \ __PMC_EV_ALIAS("CPU_CLK_UNHALTED.THREAD_P", IAP_EVENT_3CH_00H) \ __PMC_EV_ALIAS("CPU_CLK_UNHALTED.REF_P", IAP_EVENT_3CH_01H) \ __PMC_EV_ALIAS("DTLB_MISSES.ANY", IAP_EVENT_49H_01H) \ __PMC_EV_ALIAS("DTLB_MISSES.WALK_COMPLETED", IAP_EVENT_49H_02H) \ __PMC_EV_ALIAS("DTLB_MISSES.WALK_CYCLES", IAP_EVENT_49H_04H) \ __PMC_EV_ALIAS("DTLB_MISSES.STLB_HIT", IAP_EVENT_49H_10H) \ __PMC_EV_ALIAS("DTLB_MISSES.LARGE_WALK_COMPLETED", IAP_EVENT_49H_80H) \ __PMC_EV_ALIAS("LOAD_HIT_PRE", IAP_EVENT_4CH_01H) \ __PMC_EV_ALIAS("L1D_PREFETCH.REQUESTS", IAP_EVENT_4EH_01H) \ __PMC_EV_ALIAS("L1D_PREFETCH.MISS", IAP_EVENT_4EH_02H) \ __PMC_EV_ALIAS("L1D_PREFETCH.TRIGGERS", IAP_EVENT_4EH_04H) \ __PMC_EV_ALIAS("EPT.WALK_CYCLES", IAP_EVENT_4FH_10H) \ __PMC_EV_ALIAS("L1D.REPL", IAP_EVENT_51H_01H) \ __PMC_EV_ALIAS("L1D.M_REPL", IAP_EVENT_51H_02H) \ __PMC_EV_ALIAS("L1D.M_EVICT", IAP_EVENT_51H_04H) \ __PMC_EV_ALIAS("L1D.M_SNOOP_EVICT", IAP_EVENT_51H_08H) \ __PMC_EV_ALIAS("L1D_CACHE_PREFETCH_LOCK_FB_HIT", IAP_EVENT_52H_01H) \ __PMC_EV_ALIAS("L1D_CACHE_LOCK_FB_HIT", IAP_EVENT_53H_01H) \ __PMC_EV_ALIAS("OFFCORE_REQUESTS_OUTSTANDING.DEMAND.READ_DATA", \ IAP_EVENT_60H_01H) \ __PMC_EV_ALIAS("OFFCORE_REQUESTS_OUTSTANDING.DEMAND.READ_CODE", \ IAP_EVENT_60H_02H) \ __PMC_EV_ALIAS("OFFCORE_REQUESTS_OUTSTANDING.DEMAND.RFO", \ IAP_EVENT_60H_04H) \ __PMC_EV_ALIAS("OFFCORE_REQUESTS_OUTSTANDING.ANY.READ", \ IAP_EVENT_60H_08H) \ __PMC_EV_ALIAS("CACHE_LOCK_CYCLES.L1D_L2", IAP_EVENT_63H_01H) \ __PMC_EV_ALIAS("CACHE_LOCK_CYCLES.L1D", IAP_EVENT_63H_02H) \ __PMC_EV_ALIAS("IO_TRANSACTIONS", IAP_EVENT_6CH_01H) \ __PMC_EV_ALIAS("L1I.HITS", IAP_EVENT_80H_01H) \ __PMC_EV_ALIAS("L1I.MISSES", IAP_EVENT_80H_02H) \ __PMC_EV_ALIAS("L1I.READS", IAP_EVENT_80H_03H) \ __PMC_EV_ALIAS("L1I.CYCLES_STALLED", IAP_EVENT_80H_04H) \ __PMC_EV_ALIAS("LARGE_ITLB.HIT", IAP_EVENT_82H_01H) \ __PMC_EV_ALIAS("ITLB_MISSES.ANY", IAP_EVENT_85H_01H) \ __PMC_EV_ALIAS("ITLB_MISSES.WALK_COMPLETED", IAP_EVENT_85H_02H) \ __PMC_EV_ALIAS("ITLB_MISSES.WALK_CYCLES", IAP_EVENT_85H_04H) \ __PMC_EV_ALIAS("ITLB_MISSES.LARGE_WALK_COMPLETED", IAP_EVENT_85H_80H) \ __PMC_EV_ALIAS("ILD_STALL.LCP", IAP_EVENT_87H_01H) \ __PMC_EV_ALIAS("ILD_STALL.MRU", IAP_EVENT_87H_02H) \ __PMC_EV_ALIAS("ILD_STALL.IQ_FULL", IAP_EVENT_87H_04H) \ __PMC_EV_ALIAS("ILD_STALL.REGEN", IAP_EVENT_87H_08H) \ __PMC_EV_ALIAS("ILD_STALL.ANY", IAP_EVENT_87H_0FH) \ __PMC_EV_ALIAS("BR_INST_EXEC.COND", IAP_EVENT_88H_01H) \ __PMC_EV_ALIAS("BR_INST_EXEC.DIRECT", IAP_EVENT_88H_02H) \ __PMC_EV_ALIAS("BR_INST_EXEC.INDIRECT_NON_CALL", IAP_EVENT_88H_04H) \ __PMC_EV_ALIAS("BR_INST_EXEC.NON_CALLS", IAP_EVENT_88H_07H) \ __PMC_EV_ALIAS("BR_INST_EXEC.RETURN_NEAR", IAP_EVENT_88H_08H) \ __PMC_EV_ALIAS("BR_INST_EXEC.DIRECT_NEAR_CALL", IAP_EVENT_88H_10H) \ __PMC_EV_ALIAS("BR_INST_EXEC.INDIRECT_NEAR_CALL", IAP_EVENT_88H_20H) \ __PMC_EV_ALIAS("BR_INST_EXEC.NEAR_CALLS", IAP_EVENT_88H_30H) \ __PMC_EV_ALIAS("BR_INST_EXEC.TAKEN", IAP_EVENT_88H_40H) \ __PMC_EV_ALIAS("BR_INST_EXEC.ANY", IAP_EVENT_88H_7FH) \ __PMC_EV_ALIAS("BR_MISP_EXEC.COND", IAP_EVENT_89H_01H) \ __PMC_EV_ALIAS("BR_MISP_EXEC.DIRECT", IAP_EVENT_89H_02H) \ __PMC_EV_ALIAS("BR_MISP_EXEC.INDIRECT_NON_CALL", IAP_EVENT_89H_04H) \ __PMC_EV_ALIAS("BR_MISP_EXEC.NON_CALLS", IAP_EVENT_89H_07H) \ __PMC_EV_ALIAS("BR_MISP_EXEC.RETURN_NEAR", IAP_EVENT_89H_08H) \ __PMC_EV_ALIAS("BR_MISP_EXEC.DIRECT_NEAR_CALL", IAP_EVENT_89H_10H) \ __PMC_EV_ALIAS("BR_MISP_EXEC.INDIRECT_NEAR_CALL", IAP_EVENT_89H_20H) \ __PMC_EV_ALIAS("BR_MISP_EXEC.NEAR_CALLS", IAP_EVENT_89H_30H) \ __PMC_EV_ALIAS("BR_MISP_EXEC.TAKEN", IAP_EVENT_89H_40H) \ __PMC_EV_ALIAS("BR_MISP_EXEC.ANY", IAP_EVENT_89H_7FH) \ __PMC_EV_ALIAS("RESOURCE_STALLS.ANY", IAP_EVENT_A2H_01H) \ __PMC_EV_ALIAS("RESOURCE_STALLS.LOAD", IAP_EVENT_A2H_02H) \ __PMC_EV_ALIAS("RESOURCE_STALLS.RS_FULL", IAP_EVENT_A2H_04H) \ __PMC_EV_ALIAS("RESOURCE_STALLS.STORE", IAP_EVENT_A2H_08H) \ __PMC_EV_ALIAS("RESOURCE_STALLS.ROB_FULL", IAP_EVENT_A2H_10H) \ __PMC_EV_ALIAS("RESOURCE_STALLS.FPCW", IAP_EVENT_A2H_20H) \ __PMC_EV_ALIAS("RESOURCE_STALLS.MXCSR", IAP_EVENT_A2H_40H) \ __PMC_EV_ALIAS("RESOURCE_STALLS.OTHER", IAP_EVENT_A2H_80H) \ __PMC_EV_ALIAS("MACRO_INSTS.FUSIONS_DECODED", IAP_EVENT_A6H_01H) \ __PMC_EV_ALIAS("BACLEAR_FORCE_IQ", IAP_EVENT_A7H_01H) \ __PMC_EV_ALIAS("LSD.UOPS", IAP_EVENT_A8H_01H) \ __PMC_EV_ALIAS("ITLB_FLUSH", IAP_EVENT_AEH_01H) \ __PMC_EV_ALIAS("OFFCORE_REQUESTS.DEMAND.READ_DATA", IAP_EVENT_B0H_01H) \ __PMC_EV_ALIAS("OFFCORE_REQUESTS.DEMAND.READ_CODE", IAP_EVENT_B0H_02H) \ __PMC_EV_ALIAS("OFFCORE_REQUESTS.DEMAND.RFO", IAP_EVENT_B0H_04H) \ __PMC_EV_ALIAS("OFFCORE_REQUESTS.ANY.READ", IAP_EVENT_B0H_08H) \ -__PMC_EV_ALIAS("OFFCORE_REQUESTS.ANY.RFO", IAP_EVENT_80H_10H) \ +__PMC_EV_ALIAS("OFFCORE_REQUESTS.ANY.RFO", IAP_EVENT_B0H_10H) \ __PMC_EV_ALIAS("OFFCORE_REQUESTS.L1D_WRITEBACK", IAP_EVENT_B0H_40H) \ __PMC_EV_ALIAS("OFFCORE_REQUESTS.ANY", IAP_EVENT_B0H_80H) \ __PMC_EV_ALIAS("UOPS_EXECUTED.PORT0", IAP_EVENT_B1H_01H) \ __PMC_EV_ALIAS("UOPS_EXECUTED.PORT1", IAP_EVENT_B1H_02H) \ __PMC_EV_ALIAS("UOPS_EXECUTED.PORT2_CORE", IAP_EVENT_B1H_04H) \ __PMC_EV_ALIAS("UOPS_EXECUTED.PORT3_CORE", IAP_EVENT_B1H_08H) \ __PMC_EV_ALIAS("UOPS_EXECUTED.PORT4_CORE", IAP_EVENT_B1H_10H) \ __PMC_EV_ALIAS("UOPS_EXECUTED.CORE_ACTIVE_CYCLES_NO_PORT5", \ IAP_EVENT_B1H_1FH) \ __PMC_EV_ALIAS("UOPS_EXECUTED.PORT5", IAP_EVENT_B1H_20H) \ __PMC_EV_ALIAS("UOPS_EXECUTED.CORE_ACTIVE_CYCLES", IAP_EVENT_B1H_3FH) \ __PMC_EV_ALIAS("UOPS_EXECUTED.PORT015", IAP_EVENT_B1H_40H) \ __PMC_EV_ALIAS("UOPS_EXECUTED.PORT234", IAP_EVENT_B1H_80H) \ __PMC_EV_ALIAS("OFFCORE_REQUESTS_SQ_FULL", IAP_EVENT_B2H_01H) \ __PMC_EV_ALIAS("SNOOPQ_REQUESTS_OUTSTANDING.DATA", IAP_EVENT_B3H_01H) \ __PMC_EV_ALIAS("SNOOPQ_REQUESTS_OUTSTANDING.INVALIDATE", \ IAP_EVENT_B3H_02H) \ __PMC_EV_ALIAS("SNOOPQ_REQUESTS_OUTSTANDING.CODE", IAP_EVENT_B3H_04H) \ __PMC_EV_ALIAS("SNOOPQ_REQUESTS.CODE", IAP_EVENT_B4H_01H) \ __PMC_EV_ALIAS("SNOOPQ_REQUESTS.DATA", IAP_EVENT_B4H_02H) \ __PMC_EV_ALIAS("SNOOPQ_REQUESTS.INVALIDATE", IAP_EVENT_B4H_04H) \ __PMC_EV_ALIAS("OFF_CORE_RESPONSE_0", IAP_EVENT_B7H_01H) \ __PMC_EV_ALIAS("SNOOP_RESPONSE.HIT", IAP_EVENT_B8H_01H) \ __PMC_EV_ALIAS("SNOOP_RESPONSE.HITE", IAP_EVENT_B8H_02H) \ __PMC_EV_ALIAS("SNOOP_RESPONSE.HITM", IAP_EVENT_B8H_04H) \ __PMC_EV_ALIAS("OFF_CORE_RESPONSE_1", IAP_EVENT_BBH_01H) \ __PMC_EV_ALIAS("INST_RETIRED.ANY_P", IAP_EVENT_C0H_01H) \ __PMC_EV_ALIAS("INST_RETIRED.X87", IAP_EVENT_C0H_02H) \ __PMC_EV_ALIAS("INST_RETIRED.MMX", IAP_EVENT_C0H_04H) \ __PMC_EV_ALIAS("UOPS_RETIRED.ANY", IAP_EVENT_C2H_01H) \ __PMC_EV_ALIAS("UOPS_RETIRED.RETIRE_SLOTS", IAP_EVENT_C2H_02H) \ __PMC_EV_ALIAS("UOPS_RETIRED.MACRO_FUSED", IAP_EVENT_C2H_04H) \ __PMC_EV_ALIAS("MACHINE_CLEARS.CYCLES", IAP_EVENT_C3H_01H) \ __PMC_EV_ALIAS("MACHINE_CLEARS.MEM_ORDER", IAP_EVENT_C3H_02H) \ __PMC_EV_ALIAS("MACHINE_CLEARS.SMC", IAP_EVENT_C3H_04H) \ __PMC_EV_ALIAS("BR_INST_RETIRED.ALL_BRANCHES", IAP_EVENT_C4H_00H) \ __PMC_EV_ALIAS("BR_INST_RETIRED.CONDITIONAL", IAP_EVENT_C4H_01H) \ __PMC_EV_ALIAS("BR_INST_RETIRED.NEAR_CALL", IAP_EVENT_C4H_02H) \ __PMC_EV_ALIAS("BR_INST_RETIRED.ALL_BRANCHES", IAP_EVENT_C4H_04H) \ __PMC_EV_ALIAS("BR_MISP_RETIRED.ALL_BRANCHES", IAP_EVENT_C5H_00H) \ __PMC_EV_ALIAS("BR_MISP_RETIRED.CONDITIONAL", IAP_EVENT_C5H_01H) \ __PMC_EV_ALIAS("BR_MISP_RETIRED.NEAR_CALL", IAP_EVENT_C5H_02H) \ __PMC_EV_ALIAS("BR_MISP_RETIRED.ALL_BRANCHES", IAP_EVENT_C5H_04H) \ __PMC_EV_ALIAS("SSEX_UOPS_RETIRED.PACKED_SINGLE", IAP_EVENT_C7H_01H) \ __PMC_EV_ALIAS("SSEX_UOPS_RETIRED.SCALAR_SINGLE", IAP_EVENT_C7H_02H) \ __PMC_EV_ALIAS("SSEX_UOPS_RETIRED.PACKED_DOUBLE", IAP_EVENT_C7H_04H) \ __PMC_EV_ALIAS("SSEX_UOPS_RETIRED.SCALAR_DOUBLE", IAP_EVENT_C7H_08H) \ __PMC_EV_ALIAS("SSEX_UOPS_RETIRED.VECTOR_INTEGER", IAP_EVENT_C7H_10H) \ __PMC_EV_ALIAS("ITLB_MISS_RETIRED", IAP_EVENT_C8H_20H) \ __PMC_EV_ALIAS("MEM_LOAD_RETIRED.L1D_HIT", IAP_EVENT_CBH_01H) \ __PMC_EV_ALIAS("MEM_LOAD_RETIRED.L2_HIT", IAP_EVENT_CBH_02H) \ __PMC_EV_ALIAS("MEM_LOAD_RETIRED.L3_UNSHARED_HIT", IAP_EVENT_CBH_04H) \ __PMC_EV_ALIAS("MEM_LOAD_RETIRED.OTHER_CORE_L2_HIT_HITM", \ IAP_EVENT_CBH_08H) \ __PMC_EV_ALIAS("MEM_LOAD_RETIRED.L3_MISS", IAP_EVENT_CBH_10H) \ __PMC_EV_ALIAS("MEM_LOAD_RETIRED.HIT_LFB", IAP_EVENT_CBH_40H) \ __PMC_EV_ALIAS("MEM_LOAD_RETIRED.DTLB_MISS", IAP_EVENT_CBH_80H) \ __PMC_EV_ALIAS("FP_MMX_TRANS.TO_FP", IAP_EVENT_CCH_01H) \ __PMC_EV_ALIAS("FP_MMX_TRANS.TO_MMX", IAP_EVENT_CCH_02H) \ __PMC_EV_ALIAS("FP_MMX_TRANS.ANY", IAP_EVENT_CCH_03H) \ __PMC_EV_ALIAS("MACRO_INSTS.DECODED", IAP_EVENT_D0H_01H) \ __PMC_EV_ALIAS("UOPS_DECODED.STALL_CYCLES", IAP_EVENT_D1H_01H) \ __PMC_EV_ALIAS("UOPS_DECODED.MS", IAP_EVENT_D1H_02H) \ __PMC_EV_ALIAS("UOPS_DECODED.ESP_FOLDING", IAP_EVENT_D1H_04H) \ __PMC_EV_ALIAS("UOPS_DECODED.ESP_SYNC", IAP_EVENT_D1H_08H) \ __PMC_EV_ALIAS("RAT_STALLS.FLAGS", IAP_EVENT_D2H_01H) \ __PMC_EV_ALIAS("RAT_STALLS.REGISTERS", IAP_EVENT_D2H_02H) \ __PMC_EV_ALIAS("RAT_STALLS.ROB_READ_PORT", IAP_EVENT_D2H_04H) \ __PMC_EV_ALIAS("RAT_STALLS.SCOREBOARD", IAP_EVENT_D2H_08H) \ __PMC_EV_ALIAS("RAT_STALLS.ANY", IAP_EVENT_D2H_0FH) \ __PMC_EV_ALIAS("SEG_RENAME_STALLS", IAP_EVENT_D4H_01H) \ __PMC_EV_ALIAS("ES_REG_RENAMES", IAP_EVENT_D5H_01H) \ __PMC_EV_ALIAS("UOP_UNFUSION", IAP_EVENT_DBH_01H) \ __PMC_EV_ALIAS("BR_INST_DECODED", IAP_EVENT_E0H_01H) \ __PMC_EV_ALIAS("BPU_MISSED_CALL_RET", IAP_EVENT_E5H_01H) \ __PMC_EV_ALIAS("BACLEAR.CLEAR", IAP_EVENT_E6H_01H) \ __PMC_EV_ALIAS("BACLEAR.BAD_TARGET", IAP_EVENT_E6H_02H) \ __PMC_EV_ALIAS("BPU_CLEARS.EARLY", IAP_EVENT_E8H_01H) \ __PMC_EV_ALIAS("BPU_CLEARS.LATE", IAP_EVENT_E8H_02H) \ __PMC_EV_ALIAS("THREAD_ACTIVE", IAP_EVENT_ECH_01H) \ __PMC_EV_ALIAS("L2_TRANSACTIONS.LOAD", IAP_EVENT_F0H_01H) \ __PMC_EV_ALIAS("L2_TRANSACTIONS.RFO", IAP_EVENT_F0H_02H) \ __PMC_EV_ALIAS("L2_TRANSACTIONS.IFETCH", IAP_EVENT_F0H_04H) \ __PMC_EV_ALIAS("L2_TRANSACTIONS.PREFETCH", IAP_EVENT_F0H_08H) \ __PMC_EV_ALIAS("L2_TRANSACTIONS.L1D_WB", IAP_EVENT_F0H_10H) \ __PMC_EV_ALIAS("L2_TRANSACTIONS.FILL", IAP_EVENT_F0H_20H) \ __PMC_EV_ALIAS("L2_TRANSACTIONS.WB", IAP_EVENT_F0H_40H) \ __PMC_EV_ALIAS("L2_TRANSACTIONS.ANY", IAP_EVENT_F0H_80H) \ __PMC_EV_ALIAS("L2_LINES_IN.S_STATE", IAP_EVENT_F1H_02H) \ __PMC_EV_ALIAS("L2_LINES_IN.E_STATE", IAP_EVENT_F1H_04H) \ __PMC_EV_ALIAS("L2_LINES_IN.ANY", IAP_EVENT_F1H_07H) \ __PMC_EV_ALIAS("L2_LINES_OUT.DEMAND_CLEAN", IAP_EVENT_F2H_01H) \ __PMC_EV_ALIAS("L2_LINES_OUT.DEMAND_DIRTY", IAP_EVENT_F2H_02H) \ __PMC_EV_ALIAS("L2_LINES_OUT.PREFETCH_CLEAN", IAP_EVENT_F2H_04H) \ __PMC_EV_ALIAS("L2_LINES_OUT.PREFETCH_DIRTY", IAP_EVENT_F2H_08H) \ __PMC_EV_ALIAS("L2_LINES_OUT.ANY", IAP_EVENT_F2H_0FH) \ __PMC_EV_ALIAS("SQ_MISC.LRU_HINTS", IAP_EVENT_F4H_04H) \ __PMC_EV_ALIAS("SQ_MISC.SPLIT_LOCK", IAP_EVENT_F4H_10H) \ __PMC_EV_ALIAS("SQ_FULL_STALL_CYCLES", IAP_EVENT_F6H_01H) \ __PMC_EV_ALIAS("FP_ASSIST.ALL", IAP_EVENT_F7H_01H) \ __PMC_EV_ALIAS("FP_ASSIST.OUTPUT", IAP_EVENT_F7H_02H) \ __PMC_EV_ALIAS("FP_ASSIST.INPUT", IAP_EVENT_F7H_04H) \ __PMC_EV_ALIAS("SIMD_INT_64.PACKED_MPY", IAP_EVENT_FDH_01H) \ __PMC_EV_ALIAS("SIMD_INT_64.PACKED_SHIFT", IAP_EVENT_FDH_02H) \ __PMC_EV_ALIAS("SIMD_INT_64.PACK", IAP_EVENT_FDH_04H) \ __PMC_EV_ALIAS("SIMD_INT_64.UNPACK", IAP_EVENT_FDH_08H) \ __PMC_EV_ALIAS("SIMD_INT_64.PACKED_LOGICAL", IAP_EVENT_FDH_10H) \ __PMC_EV_ALIAS("SIMD_INT_64.PACKED_ARITH", IAP_EVENT_FDH_20H) \ __PMC_EV_ALIAS("SIMD_INT_64.SHUFFLE_MOVE", IAP_EVENT_FDH_40H) /* timestamp counters. */ #define __PMC_EV_TSC() \ __PMC_EV(TSC, TSC) #define PMC_EV_TSC_FIRST PMC_EV_TSC_TSC #define PMC_EV_TSC_LAST PMC_EV_TSC_TSC #define __PMC_EV_UCF() \ __PMC_EV(UCF, UCLOCK) #define PMC_EV_UCF_FIRST PMC_EV_UCF_UCLOCK #define PMC_EV_UCF_LAST PMC_EV_UCF_UCLOCK #define __PMC_EV_UCP() \ __PMC_EV(UCP, EVENT_00H_01H) \ __PMC_EV(UCP, EVENT_00H_02H) \ __PMC_EV(UCP, EVENT_00H_04H) \ __PMC_EV(UCP, EVENT_01H_01H) \ __PMC_EV(UCP, EVENT_01H_02H) \ __PMC_EV(UCP, EVENT_01H_04H) \ __PMC_EV(UCP, EVENT_02H_01H) \ __PMC_EV(UCP, EVENT_03H_01H) \ __PMC_EV(UCP, EVENT_03H_02H) \ __PMC_EV(UCP, EVENT_03H_04H) \ __PMC_EV(UCP, EVENT_03H_08H) \ __PMC_EV(UCP, EVENT_03H_10H) \ __PMC_EV(UCP, EVENT_03H_20H) \ __PMC_EV(UCP, EVENT_03H_40H) \ __PMC_EV(UCP, EVENT_04H_01H) \ __PMC_EV(UCP, EVENT_04H_02H) \ __PMC_EV(UCP, EVENT_04H_04H) \ __PMC_EV(UCP, EVENT_04H_08H) \ __PMC_EV(UCP, EVENT_04H_10H) \ __PMC_EV(UCP, EVENT_05H_01H) \ __PMC_EV(UCP, EVENT_05H_02H) \ __PMC_EV(UCP, EVENT_05H_04H) \ __PMC_EV(UCP, EVENT_06H_01H) \ __PMC_EV(UCP, EVENT_06H_02H) \ __PMC_EV(UCP, EVENT_06H_04H) \ __PMC_EV(UCP, EVENT_06H_08H) \ __PMC_EV(UCP, EVENT_06H_10H) \ __PMC_EV(UCP, EVENT_06H_20H) \ __PMC_EV(UCP, EVENT_07H_01H) \ __PMC_EV(UCP, EVENT_07H_02H) \ __PMC_EV(UCP, EVENT_07H_04H) \ __PMC_EV(UCP, EVENT_07H_08H) \ __PMC_EV(UCP, EVENT_07H_10H) \ __PMC_EV(UCP, EVENT_07H_20H) \ __PMC_EV(UCP, EVENT_07H_24H) \ __PMC_EV(UCP, EVENT_08H_01H) \ __PMC_EV(UCP, EVENT_08H_02H) \ __PMC_EV(UCP, EVENT_08H_04H) \ __PMC_EV(UCP, EVENT_08H_03H) \ __PMC_EV(UCP, EVENT_09H_01H) \ __PMC_EV(UCP, EVENT_09H_02H) \ __PMC_EV(UCP, EVENT_09H_04H) \ __PMC_EV(UCP, EVENT_09H_03H) \ __PMC_EV(UCP, EVENT_0AH_01H) \ __PMC_EV(UCP, EVENT_0AH_02H) \ __PMC_EV(UCP, EVENT_0AH_04H) \ __PMC_EV(UCP, EVENT_0AH_08H) \ __PMC_EV(UCP, EVENT_0AH_0FH) \ __PMC_EV(UCP, EVENT_0BH_01H) \ __PMC_EV(UCP, EVENT_0BH_02H) \ __PMC_EV(UCP, EVENT_0BH_04H) \ __PMC_EV(UCP, EVENT_0BH_08H) \ __PMC_EV(UCP, EVENT_0BH_10H) \ __PMC_EV(UCP, EVENT_0BH_1FH) \ __PMC_EV(UCP, EVENT_0CH_01H) \ __PMC_EV(UCP, EVENT_0CH_02H) \ __PMC_EV(UCP, EVENT_0CH_04H) \ __PMC_EV(UCP, EVENT_0CH_08H) \ __PMC_EV(UCP, EVENT_20H_01H) \ __PMC_EV(UCP, EVENT_20H_02H) \ __PMC_EV(UCP, EVENT_20H_04H) \ __PMC_EV(UCP, EVENT_20H_08H) \ __PMC_EV(UCP, EVENT_20H_10H) \ __PMC_EV(UCP, EVENT_20H_20H) \ __PMC_EV(UCP, EVENT_21H_01H) \ __PMC_EV(UCP, EVENT_21H_02H) \ __PMC_EV(UCP, EVENT_21H_04H) \ __PMC_EV(UCP, EVENT_22H_01H) \ __PMC_EV(UCP, EVENT_22H_02H) \ __PMC_EV(UCP, EVENT_22H_04H) \ __PMC_EV(UCP, EVENT_23H_01H) \ __PMC_EV(UCP, EVENT_23H_02H) \ __PMC_EV(UCP, EVENT_23H_04H) \ __PMC_EV(UCP, EVENT_24H_02H) \ __PMC_EV(UCP, EVENT_24H_04H) \ __PMC_EV(UCP, EVENT_25H_01H) \ __PMC_EV(UCP, EVENT_25H_02H) \ __PMC_EV(UCP, EVENT_25H_04H) \ __PMC_EV(UCP, EVENT_26H_01H) \ __PMC_EV(UCP, EVENT_27H_01H) \ __PMC_EV(UCP, EVENT_27H_02H) \ __PMC_EV(UCP, EVENT_27H_04H) \ __PMC_EV(UCP, EVENT_27H_08H) \ __PMC_EV(UCP, EVENT_27H_10H) \ __PMC_EV(UCP, EVENT_27H_20H) \ __PMC_EV(UCP, EVENT_28H_01H) \ __PMC_EV(UCP, EVENT_28H_02H) \ __PMC_EV(UCP, EVENT_28H_04H) \ __PMC_EV(UCP, EVENT_28H_08H) \ __PMC_EV(UCP, EVENT_28H_10H) \ __PMC_EV(UCP, EVENT_28H_20H) \ __PMC_EV(UCP, EVENT_29H_01H) \ __PMC_EV(UCP, EVENT_29H_02H) \ __PMC_EV(UCP, EVENT_29H_04H) \ __PMC_EV(UCP, EVENT_29H_08H) \ __PMC_EV(UCP, EVENT_29H_10H) \ __PMC_EV(UCP, EVENT_29H_20H) \ __PMC_EV(UCP, EVENT_2AH_01H) \ __PMC_EV(UCP, EVENT_2AH_02H) \ __PMC_EV(UCP, EVENT_2AH_04H) \ __PMC_EV(UCP, EVENT_2AH_07H) \ __PMC_EV(UCP, EVENT_2BH_01H) \ __PMC_EV(UCP, EVENT_2BH_02H) \ __PMC_EV(UCP, EVENT_2BH_04H) \ __PMC_EV(UCP, EVENT_2BH_07H) \ __PMC_EV(UCP, EVENT_2CH_01H) \ __PMC_EV(UCP, EVENT_2CH_02H) \ __PMC_EV(UCP, EVENT_2CH_04H) \ __PMC_EV(UCP, EVENT_2CH_07H) \ __PMC_EV(UCP, EVENT_2DH_01H) \ __PMC_EV(UCP, EVENT_2DH_02H) \ __PMC_EV(UCP, EVENT_2DH_04H) \ __PMC_EV(UCP, EVENT_2DH_07H) \ __PMC_EV(UCP, EVENT_2EH_01H) \ __PMC_EV(UCP, EVENT_2EH_02H) \ __PMC_EV(UCP, EVENT_2EH_04H) \ __PMC_EV(UCP, EVENT_2EH_07H) \ __PMC_EV(UCP, EVENT_2FH_01H) \ __PMC_EV(UCP, EVENT_2FH_02H) \ __PMC_EV(UCP, EVENT_2FH_04H) \ __PMC_EV(UCP, EVENT_2FH_07H) \ __PMC_EV(UCP, EVENT_2FH_08H) \ __PMC_EV(UCP, EVENT_2FH_10H) \ __PMC_EV(UCP, EVENT_2FH_20H) \ __PMC_EV(UCP, EVENT_2FH_38H) \ __PMC_EV(UCP, EVENT_30H_01H) \ __PMC_EV(UCP, EVENT_30H_02H) \ __PMC_EV(UCP, EVENT_30H_04H) \ __PMC_EV(UCP, EVENT_30H_07H) \ __PMC_EV(UCP, EVENT_31H_01H) \ __PMC_EV(UCP, EVENT_31H_02H) \ __PMC_EV(UCP, EVENT_31H_04H) \ __PMC_EV(UCP, EVENT_31H_07H) \ __PMC_EV(UCP, EVENT_32H_01H) \ __PMC_EV(UCP, EVENT_32H_02H) \ __PMC_EV(UCP, EVENT_32H_04H) \ __PMC_EV(UCP, EVENT_32H_07H) \ __PMC_EV(UCP, EVENT_33H_01H) \ __PMC_EV(UCP, EVENT_33H_02H) \ __PMC_EV(UCP, EVENT_33H_04H) \ __PMC_EV(UCP, EVENT_33H_07H) \ __PMC_EV(UCP, EVENT_34H_01H) \ __PMC_EV(UCP, EVENT_34H_02H) \ __PMC_EV(UCP, EVENT_34H_04H) \ __PMC_EV(UCP, EVENT_34H_08H) \ __PMC_EV(UCP, EVENT_34H_10H) \ __PMC_EV(UCP, EVENT_34H_20H) \ __PMC_EV(UCP, EVENT_35H_01H) \ __PMC_EV(UCP, EVENT_35H_02H) \ __PMC_EV(UCP, EVENT_35H_04H) \ __PMC_EV(UCP, EVENT_40H_01H) \ __PMC_EV(UCP, EVENT_40H_02H) \ __PMC_EV(UCP, EVENT_40H_04H) \ __PMC_EV(UCP, EVENT_40H_08H) \ __PMC_EV(UCP, EVENT_40H_10H) \ __PMC_EV(UCP, EVENT_40H_20H) \ __PMC_EV(UCP, EVENT_40H_07H) \ __PMC_EV(UCP, EVENT_40H_38H) \ __PMC_EV(UCP, EVENT_41H_01H) \ __PMC_EV(UCP, EVENT_41H_02H) \ __PMC_EV(UCP, EVENT_41H_04H) \ __PMC_EV(UCP, EVENT_41H_08H) \ __PMC_EV(UCP, EVENT_41H_10H) \ __PMC_EV(UCP, EVENT_41H_20H) \ __PMC_EV(UCP, EVENT_41H_07H) \ __PMC_EV(UCP, EVENT_41H_38H) \ __PMC_EV(UCP, EVENT_42H_01H) \ __PMC_EV(UCP, EVENT_42H_02H) \ __PMC_EV(UCP, EVENT_42H_04H) \ __PMC_EV(UCP, EVENT_42H_08H) \ __PMC_EV(UCP, EVENT_43H_01H) \ __PMC_EV(UCP, EVENT_43H_02H) \ __PMC_EV(UCP, EVENT_60H_01H) \ __PMC_EV(UCP, EVENT_60H_02H) \ __PMC_EV(UCP, EVENT_60H_04H) \ __PMC_EV(UCP, EVENT_61H_01H) \ __PMC_EV(UCP, EVENT_61H_02H) \ __PMC_EV(UCP, EVENT_61H_04H) \ __PMC_EV(UCP, EVENT_62H_01H) \ __PMC_EV(UCP, EVENT_62H_02H) \ __PMC_EV(UCP, EVENT_62H_04H) \ __PMC_EV(UCP, EVENT_63H_01H) \ __PMC_EV(UCP, EVENT_63H_02H) \ __PMC_EV(UCP, EVENT_63H_04H) \ __PMC_EV(UCP, EVENT_63H_08H) \ __PMC_EV(UCP, EVENT_63H_10H) \ __PMC_EV(UCP, EVENT_63H_20H) \ __PMC_EV(UCP, EVENT_64H_01H) \ __PMC_EV(UCP, EVENT_64H_02H) \ __PMC_EV(UCP, EVENT_64H_04H) \ __PMC_EV(UCP, EVENT_64H_08H) \ __PMC_EV(UCP, EVENT_64H_10H) \ __PMC_EV(UCP, EVENT_64H_20H) \ __PMC_EV(UCP, EVENT_65H_01H) \ __PMC_EV(UCP, EVENT_65H_02H) \ __PMC_EV(UCP, EVENT_65H_04H) \ __PMC_EV(UCP, EVENT_66H_01H) \ __PMC_EV(UCP, EVENT_66H_02H) \ __PMC_EV(UCP, EVENT_66H_04H) \ __PMC_EV(UCP, EVENT_67H_01H) \ __PMC_EV(UCP, EVENT_80H_01H) \ __PMC_EV(UCP, EVENT_80H_02H) \ __PMC_EV(UCP, EVENT_80H_04H) \ __PMC_EV(UCP, EVENT_80H_08H) \ __PMC_EV(UCP, EVENT_81H_01H) \ __PMC_EV(UCP, EVENT_81H_02H) \ __PMC_EV(UCP, EVENT_81H_04H) \ __PMC_EV(UCP, EVENT_81H_08H) \ __PMC_EV(UCP, EVENT_82H_01H) \ __PMC_EV(UCP, EVENT_83H_01H) \ __PMC_EV(UCP, EVENT_83H_02H) \ __PMC_EV(UCP, EVENT_83H_04H) \ __PMC_EV(UCP, EVENT_83H_08H) \ __PMC_EV(UCP, EVENT_84H_01H) \ __PMC_EV(UCP, EVENT_84H_02H) \ __PMC_EV(UCP, EVENT_84H_04H) \ __PMC_EV(UCP, EVENT_84H_08H) \ __PMC_EV(UCP, EVENT_85H_02H) \ __PMC_EV(UCP, EVENT_86H_01H) #define PMC_EV_UCP_FIRST PMC_EV_UCP_EVENT_00H_01H #define PMC_EV_UCP_LAST PMC_EV_UCP_EVENT_86H_01H #define __PMC_EV_ALIAS_COREI7UC() \ __PMC_EV_ALIAS("GQ_CYCLES_FULL.READ_TRACKER", UCP_EVENT_00H_01H) \ __PMC_EV_ALIAS("GQ_CYCLES_FULL.WRITE_TRACKER", UCP_EVENT_00H_02H) \ __PMC_EV_ALIAS("GQ_CYCLES_FULL.PEER_PROBE_TRACKER", UCP_EVENT_00H_04H) \ __PMC_EV_ALIAS("GQ_CYCLES_NOT_EMPTY.READ_TRACKER", UCP_EVENT_01H_01H) \ __PMC_EV_ALIAS("GQ_CYCLES_NOT_EMPTY.WRITE_TRACKER", UCP_EVENT_01H_02H) \ __PMC_EV_ALIAS("GQ_CYCLES_NOT_EMPTY.PEER_PROBE_TRACKER", UCP_EVENT_01H_04H) \ __PMC_EV_ALIAS("GQ_ALLOC.READ_TRACKER", UCP_EVENT_03H_01H) \ __PMC_EV_ALIAS("GQ_ALLOC.RT_L3_MISS", UCP_EVENT_03H_02H) \ __PMC_EV_ALIAS("GQ_ALLOC.RT_TO_L3_RESP", UCP_EVENT_03H_04H) \ __PMC_EV_ALIAS("GQ_ALLOC.RT_TO_RTID_ACQUIRED", UCP_EVENT_03H_08H) \ __PMC_EV_ALIAS("GQ_ALLOC.WT_TO_RTID_ACQUIRED", UCP_EVENT_03H_10H) \ __PMC_EV_ALIAS("GQ_ALLOC.WRITE_TRACKER", UCP_EVENT_03H_20H) \ __PMC_EV_ALIAS("GQ_ALLOC.PEER_PROBE_TRACKER", UCP_EVENT_03H_40H) \ __PMC_EV_ALIAS("GQ_DATA.FROM_QPI", UCP_EVENT_04H_01H) \ __PMC_EV_ALIAS("GQ_DATA.FROM_QMC", UCP_EVENT_04H_02H) \ __PMC_EV_ALIAS("GQ_DATA.FROM_L3", UCP_EVENT_04H_04H) \ __PMC_EV_ALIAS("GQ_DATA.FROM_CORES_02", UCP_EVENT_04H_08H) \ __PMC_EV_ALIAS("GQ_DATA.FROM_CORES_13", UCP_EVENT_04H_10H) \ __PMC_EV_ALIAS("GQ_DATA.TO_QPI_QMC", UCP_EVENT_05H_01H) \ __PMC_EV_ALIAS("GQ_DATA.TO_L3", UCP_EVENT_05H_02H) \ __PMC_EV_ALIAS("GQ_DATA.TO_CORES", UCP_EVENT_05H_04H) \ __PMC_EV_ALIAS("SNP_RESP_TO_LOCAL_HOME.I_STATE", UCP_EVENT_06H_01H) \ __PMC_EV_ALIAS("SNP_RESP_TO_LOCAL_HOME.S_STATE", UCP_EVENT_06H_02H) \ __PMC_EV_ALIAS("SNP_RESP_TO_LOCAL_HOME.FWD_S_STATE", UCP_EVENT_06H_04H) \ __PMC_EV_ALIAS("SNP_RESP_TO_LOCAL_HOME.FWD_I_STATE", UCP_EVENT_06H_08H) \ __PMC_EV_ALIAS("SNP_RESP_TO_LOCAL_HOME.CONFLICT", UCP_EVENT_06H_10H) \ __PMC_EV_ALIAS("SNP_RESP_TO_LOCAL_HOME.WB", UCP_EVENT_06H_20H) \ __PMC_EV_ALIAS("SNP_RESP_TO_REMOTE_HOME.I_STATE", UCP_EVENT_07H_01H) \ __PMC_EV_ALIAS("SNP_RESP_TO_REMOTE_HOME.S_STATE", UCP_EVENT_07H_02H) \ __PMC_EV_ALIAS("SNP_RESP_TO_REMOTE_HOME.FWD_S_STATE", UCP_EVENT_07H_04H) \ __PMC_EV_ALIAS("SNP_RESP_TO_REMOTE_HOME.FWD_I_STATE", UCP_EVENT_07H_08H) \ __PMC_EV_ALIAS("SNP_RESP_TO_REMOTE_HOME.CONFLICT", UCP_EVENT_07H_10H) \ __PMC_EV_ALIAS("SNP_RESP_TO_REMOTE_HOME.WB", UCP_EVENT_07H_20H) \ __PMC_EV_ALIAS("SNP_RESP_TO_REMOTE_HOME.HITM", UCP_EVENT_07H_24H) \ __PMC_EV_ALIAS("L3_HITS.READ", UCP_EVENT_08H_01H) \ __PMC_EV_ALIAS("L3_HITS.WRITE", UCP_EVENT_08H_02H) \ __PMC_EV_ALIAS("L3_HITS.PROBE", UCP_EVENT_08H_04H) \ __PMC_EV_ALIAS("L3_HITS.ANY", UCP_EVENT_08H_03H) \ __PMC_EV_ALIAS("L3_MISS.READ", UCP_EVENT_09H_01H) \ __PMC_EV_ALIAS("L3_MISS.WRITE", UCP_EVENT_09H_02H) \ __PMC_EV_ALIAS("L3_MISS.PROBE", UCP_EVENT_09H_04H) \ __PMC_EV_ALIAS("L3_MISS.ANY", UCP_EVENT_09H_03H) \ __PMC_EV_ALIAS("L3_LINES_IN.M_STATE", UCP_EVENT_0AH_01H) \ __PMC_EV_ALIAS("L3_LINES_IN.E_STATE", UCP_EVENT_0AH_02H) \ __PMC_EV_ALIAS("L3_LINES_IN.S_STATE", UCP_EVENT_0AH_04H) \ __PMC_EV_ALIAS("L3_LINES_IN.F_STATE", UCP_EVENT_0AH_08H) \ __PMC_EV_ALIAS("L3_LINES_IN.ANY", UCP_EVENT_0AH_0FH) \ __PMC_EV_ALIAS("L3_LINES_OUT.M_STATE", UCP_EVENT_0BH_01H) \ __PMC_EV_ALIAS("L3_LINES_OUT.E_STATE", UCP_EVENT_0BH_02H) \ __PMC_EV_ALIAS("L3_LINES_OUT.S_STATE", UCP_EVENT_0BH_04H) \ __PMC_EV_ALIAS("L3_LINES_OUT.I_STATE", UCP_EVENT_0BH_08H) \ __PMC_EV_ALIAS("L3_LINES_OUT.F_STATE", UCP_EVENT_0BH_10H) \ __PMC_EV_ALIAS("L3_LINES_OUT.ANY", UCP_EVENT_0BH_1FH) \ __PMC_EV_ALIAS("QHL_REQUESTS.IOH_READS", UCP_EVENT_20H_01H) \ __PMC_EV_ALIAS("QHL_REQUESTS.IOH_WRITES", UCP_EVENT_20H_02H) \ __PMC_EV_ALIAS("QHL_REQUESTS.REMOTE_READS", UCP_EVENT_20H_04H) \ __PMC_EV_ALIAS("QHL_REQUESTS.REMOTE_WRITES", UCP_EVENT_20H_08H) \ __PMC_EV_ALIAS("QHL_REQUESTS.LOCAL_READS", UCP_EVENT_20H_10H) \ __PMC_EV_ALIAS("QHL_REQUESTS.LOCAL_WRITES", UCP_EVENT_20H_20H) \ __PMC_EV_ALIAS("QHL_CYCLES_FULL.IOH", UCP_EVENT_21H_01H) \ __PMC_EV_ALIAS("QHL_CYCLES_FULL.REMOTE", UCP_EVENT_21H_02H) \ __PMC_EV_ALIAS("QHL_CYCLES_FULL.LOCAL", UCP_EVENT_21H_04H) \ __PMC_EV_ALIAS("QHL_CYCLES_NOT_EMPTY.IOH", UCP_EVENT_22H_01H) \ __PMC_EV_ALIAS("QHL_CYCLES_NOT_EMPTY.REMOTE", UCP_EVENT_22H_02H) \ __PMC_EV_ALIAS("QHL_CYCLES_NOT_EMPTY.LOCAL", UCP_EVENT_22H_04H) \ __PMC_EV_ALIAS("QHL_OCCUPANCY.IOH", UCP_EVENT_23H_01H) \ __PMC_EV_ALIAS("QHL_OCCUPANCY.REMOTE", UCP_EVENT_23H_02H) \ __PMC_EV_ALIAS("QHL_OCCUPANCY.LOCAL", UCP_EVENT_23H_04H) \ __PMC_EV_ALIAS("QHL_ADDRESS_CONFLICTS.2WAY", UCP_EVENT_24H_02H) \ __PMC_EV_ALIAS("QHL_ADDRESS_CONFLICTS.3WAY", UCP_EVENT_24H_04H) \ __PMC_EV_ALIAS("QHL_CONFLICT_CYCLES.IOH", UCP_EVENT_25H_01H) \ __PMC_EV_ALIAS("QHL_CONFLICT_CYCLES.REMOTE", UCP_EVENT_25H_02H) \ __PMC_EV_ALIAS("QHL_CONFLICT_CYCLES.LOCAL", UCP_EVENT_25H_04H) \ __PMC_EV_ALIAS("QHL_TO_QMC_BYPASS", UCP_EVENT_26H_01H) \ __PMC_EV_ALIAS("QMC_NORMAL_FULL.READ.CH0", UCP_EVENT_27H_01H) \ __PMC_EV_ALIAS("QMC_NORMAL_FULL.READ.CH1", UCP_EVENT_27H_02H) \ __PMC_EV_ALIAS("QMC_NORMAL_FULL.READ.CH2", UCP_EVENT_27H_04H) \ __PMC_EV_ALIAS("QMC_NORMAL_FULL.WRITE.CH0", UCP_EVENT_27H_08H) \ __PMC_EV_ALIAS("QMC_NORMAL_FULL.WRITE.CH1", UCP_EVENT_27H_10H) \ __PMC_EV_ALIAS("QMC_NORMAL_FULL.WRITE.CH2", UCP_EVENT_27H_20H) \ __PMC_EV_ALIAS("QMC_ISOC_FULL.READ.CH0", UCP_EVENT_28H_01H) \ __PMC_EV_ALIAS("QMC_ISOC_FULL.READ.CH1", UCP_EVENT_28H_02H) \ __PMC_EV_ALIAS("QMC_ISOC_FULL.READ.CH2", UCP_EVENT_28H_04H) \ __PMC_EV_ALIAS("QMC_ISOC_FULL.WRITE.CH0", UCP_EVENT_28H_08H) \ __PMC_EV_ALIAS("QMC_ISOC_FULL.WRITE.CH1", UCP_EVENT_28H_10H) \ __PMC_EV_ALIAS("QMC_ISOC_FULL.WRITE.CH2", UCP_EVENT_28H_20H) \ __PMC_EV_ALIAS("QMC_BUSY.READ.CH0", UCP_EVENT_29H_01H) \ __PMC_EV_ALIAS("QMC_BUSY.READ.CH1", UCP_EVENT_29H_02H) \ __PMC_EV_ALIAS("QMC_BUSY.READ.CH2", UCP_EVENT_29H_04H) \ __PMC_EV_ALIAS("QMC_BUSY.WRITE.CH0", UCP_EVENT_29H_08H) \ __PMC_EV_ALIAS("QMC_BUSY.WRITE.CH1", UCP_EVENT_29H_10H) \ __PMC_EV_ALIAS("QMC_BUSY.WRITE.CH2", UCP_EVENT_29H_20H) \ __PMC_EV_ALIAS("QMC_OCCUPANCY.CH0", UCP_EVENT_2AH_01H) \ __PMC_EV_ALIAS("QMC_OCCUPANCY.CH1", UCP_EVENT_2AH_02H) \ __PMC_EV_ALIAS("QMC_OCCUPANCY.CH2", UCP_EVENT_2AH_04H) \ __PMC_EV_ALIAS("QMC_ISSOC_OCCUPANCY.CH0", UCP_EVENT_2BH_01H) \ __PMC_EV_ALIAS("QMC_ISSOC_OCCUPANCY.CH1", UCP_EVENT_2BH_02H) \ __PMC_EV_ALIAS("QMC_ISSOC_OCCUPANCY.CH2", UCP_EVENT_2BH_04H) \ __PMC_EV_ALIAS("QMC_ISSOC_READS.ANY", UCP_EVENT_2BH_07H) \ __PMC_EV_ALIAS("QMC_NORMAL_READS.CH0", UCP_EVENT_2CH_01H) \ __PMC_EV_ALIAS("QMC_NORMAL_READS.CH1", UCP_EVENT_2CH_02H) \ __PMC_EV_ALIAS("QMC_NORMAL_READS.CH2", UCP_EVENT_2CH_04H) \ __PMC_EV_ALIAS("QMC_NORMAL_READS.ANY", UCP_EVENT_2CH_07H) \ __PMC_EV_ALIAS("QMC_HIGH_PRIORITY_READS.CH0", UCP_EVENT_2DH_01H) \ __PMC_EV_ALIAS("QMC_HIGH_PRIORITY_READS.CH1", UCP_EVENT_2DH_02H) \ __PMC_EV_ALIAS("QMC_HIGH_PRIORITY_READS.CH2", UCP_EVENT_2DH_04H) \ __PMC_EV_ALIAS("QMC_HIGH_PRIORITY_READS.ANY", UCP_EVENT_2DH_07H) \ __PMC_EV_ALIAS("QMC_CRITICAL_PRIORITY_READS.CH0", UCP_EVENT_2EH_01H) \ __PMC_EV_ALIAS("QMC_CRITICAL_PRIORITY_READS.CH1", UCP_EVENT_2EH_02H) \ __PMC_EV_ALIAS("QMC_CRITICAL_PRIORITY_READS.CH2", UCP_EVENT_2EH_04H) \ __PMC_EV_ALIAS("QMC_CRITICAL_PRIORITY_READS.ANY", UCP_EVENT_2EH_07H) \ __PMC_EV_ALIAS("QMC_WRITES.FULL.CH0", UCP_EVENT_2FH_01H) \ __PMC_EV_ALIAS("QMC_WRITES.FULL.CH1", UCP_EVENT_2FH_02H) \ __PMC_EV_ALIAS("QMC_WRITES.FULL.CH2", UCP_EVENT_2FH_04H) \ __PMC_EV_ALIAS("QMC_WRITES.FULL.ANY", UCP_EVENT_2FH_07H) \ __PMC_EV_ALIAS("QMC_WRITES.PARTIAL.CH0", UCP_EVENT_2FH_08H) \ __PMC_EV_ALIAS("QMC_WRITES.PARTIAL.CH1", UCP_EVENT_2FH_10H) \ __PMC_EV_ALIAS("QMC_WRITES.PARTIAL.CH2", UCP_EVENT_2FH_20H) \ __PMC_EV_ALIAS("QMC_WRITES.PARTIAL.ANY", UCP_EVENT_2FH_38H) \ __PMC_EV_ALIAS("QMC_CANCEL.CH0", UCP_EVENT_30H_01H) \ __PMC_EV_ALIAS("QMC_CANCEL.CH1", UCP_EVENT_30H_02H) \ __PMC_EV_ALIAS("QMC_CANCEL.CH2", UCP_EVENT_30H_04H) \ __PMC_EV_ALIAS("QMC_CANCEL.ANY", UCP_EVENT_30H_07H) \ __PMC_EV_ALIAS("QMC_PRIORITY_UPDATES.CH0", UCP_EVENT_31H_01H) \ __PMC_EV_ALIAS("QMC_PRIORITY_UPDATES.CH1", UCP_EVENT_31H_02H) \ __PMC_EV_ALIAS("QMC_PRIORITY_UPDATES.CH2", UCP_EVENT_31H_04H) \ __PMC_EV_ALIAS("QMC_PRIORITY_UPDATES.ANY", UCP_EVENT_31H_07H) \ __PMC_EV_ALIAS("QHL_FRC_ACK_CNFLTS.LOCAL", UCP_EVENT_33H_04H) \ __PMC_EV_ALIAS("QPI_TX_STALLED_SINGLE_FLIT.HOME.LINK_0", UCP_EVENT_40H_01H) \ __PMC_EV_ALIAS("QPI_TX_STALLED_SINGLE_FLIT.SNOOP.LINK_0", UCP_EVENT_40H_02H) \ __PMC_EV_ALIAS("QPI_TX_STALLED_SINGLE_FLIT.NDR.LINK_0", UCP_EVENT_40H_04H) \ __PMC_EV_ALIAS("QPI_TX_STALLED_SINGLE_FLIT.HOME.LINK_1", UCP_EVENT_40H_08H) \ __PMC_EV_ALIAS("QPI_TX_STALLED_SINGLE_FLIT.SNOOP.LINK_1", UCP_EVENT_40H_10H) \ __PMC_EV_ALIAS("QPI_TX_STALLED_SINGLE_FLIT.NDR.LINK_1", UCP_EVENT_40H_20H) \ __PMC_EV_ALIAS("QPI_TX_STALLED_SINGLE_FLIT.LINK_0", UCP_EVENT_40H_07H) \ __PMC_EV_ALIAS("QPI_TX_STALLED_SINGLE_FLIT.LINK_1", UCP_EVENT_40H_38H) \ __PMC_EV_ALIAS("QPI_TX_STALLED_MULTI_FLIT.DRS.LINK_0", UCP_EVENT_41H_01H) \ __PMC_EV_ALIAS("QPI_TX_STALLED_MULTI_FLIT.NCB.LINK_0", UCP_EVENT_41H_02H) \ __PMC_EV_ALIAS("QPI_TX_STALLED_MULTI_FLIT.NCS.LINK_0", UCP_EVENT_41H_04H) \ __PMC_EV_ALIAS("QPI_TX_STALLED_MULTI_FLIT.DRS.LINK_1", UCP_EVENT_41H_08H) \ __PMC_EV_ALIAS("QPI_TX_STALLED_MULTI_FLIT.NCB.LINK_1", UCP_EVENT_41H_10H) \ __PMC_EV_ALIAS("QPI_TX_STALLED_MULTI_FLIT.NCS.LINK_1", UCP_EVENT_41H_20H) \ __PMC_EV_ALIAS("QPI_TX_STALLED_MULTI_FLIT.LINK_0", UCP_EVENT_41H_07H) \ __PMC_EV_ALIAS("QPI_TX_STALLED_MULTI_FLIT.LINK_1", UCP_EVENT_41H_38H) \ __PMC_EV_ALIAS("QPI_TX_HEADER.BUSY.LINK_0", UCP_EVENT_42H_02H) \ __PMC_EV_ALIAS("QPI_TX_HEADER.BUSY.LINK_1", UCP_EVENT_42H_08H) \ __PMC_EV_ALIAS("QPI_RX_NO_PPT_CREDIT.STALLS.LINK_0", UCP_EVENT_43H_01H) \ __PMC_EV_ALIAS("QPI_RX_NO_PPT_CREDIT.STALLS.LINK_1", UCP_EVENT_43H_02H) \ __PMC_EV_ALIAS("DRAM_OPEN.CH0", UCP_EVENT_60H_01H) \ __PMC_EV_ALIAS("DRAM_OPEN.CH1", UCP_EVENT_60H_02H) \ __PMC_EV_ALIAS("DRAM_OPEN.CH2", UCP_EVENT_60H_04H) \ __PMC_EV_ALIAS("DRAM_PAGE_CLOSE.CH0", UCP_EVENT_61H_01H) \ __PMC_EV_ALIAS("DRAM_PAGE_CLOSE.CH1", UCP_EVENT_61H_02H) \ __PMC_EV_ALIAS("DRAM_PAGE_CLOSE.CH2", UCP_EVENT_61H_04H) \ __PMC_EV_ALIAS("DRAM_PAGE_MISS.CH0", UCP_EVENT_62H_01H) \ __PMC_EV_ALIAS("DRAM_PAGE_MISS.CH1", UCP_EVENT_62H_02H) \ __PMC_EV_ALIAS("DRAM_PAGE_MISS.CH2", UCP_EVENT_62H_04H) \ __PMC_EV_ALIAS("DRAM_READ_CAS.CH0", UCP_EVENT_63H_01H) \ __PMC_EV_ALIAS("DRAM_READ_CAS.AUTOPRE_CH0", UCP_EVENT_63H_02H) \ __PMC_EV_ALIAS("DRAM_READ_CAS.CH1", UCP_EVENT_63H_04H) \ __PMC_EV_ALIAS("DRAM_READ_CAS.AUTOPRE_CH1", UCP_EVENT_63H_08H) \ __PMC_EV_ALIAS("DRAM_READ_CAS.CH2", UCP_EVENT_63H_10H) \ __PMC_EV_ALIAS("DRAM_READ_CAS.AUTOPRE_CH2", UCP_EVENT_63H_20H) \ __PMC_EV_ALIAS("DRAM_WRITE_CAS.CH0", UCP_EVENT_64H_01H) \ __PMC_EV_ALIAS("DRAM_WRITE_CAS.AUTOPRE_CH0", UCP_EVENT_64H_02H) \ __PMC_EV_ALIAS("DRAM_WRITE_CAS.CH1", UCP_EVENT_64H_04H) \ __PMC_EV_ALIAS("DRAM_WRITE_CAS.AUTOPRE_CH1", UCP_EVENT_64H_08H) \ __PMC_EV_ALIAS("DRAM_WRITE_CAS.CH2", UCP_EVENT_64H_10H) \ __PMC_EV_ALIAS("DRAM_WRITE_CAS.AUTOPRE_CH2", UCP_EVENT_64H_20H) \ __PMC_EV_ALIAS("DRAM_REFRESH.CH0", UCP_EVENT_65H_01H) \ __PMC_EV_ALIAS("DRAM_REFRESH.CH1", UCP_EVENT_65H_02H) \ __PMC_EV_ALIAS("DRAM_REFRESH.CH2", UCP_EVENT_65H_04H) \ __PMC_EV_ALIAS("DRAM_PRE_ALL.CH0", UCP_EVENT_66H_01H) \ __PMC_EV_ALIAS("DRAM_PRE_ALL.CH1", UCP_EVENT_66H_02H) \ __PMC_EV_ALIAS("DRAM_PRE_ALL.CH2", UCP_EVENT_66H_04H) #define __PMC_EV_ALIAS_WESTMEREUC() \ __PMC_EV_ALIAS("GQ_CYCLES_FULL.READ_TRACKER", UCP_EVENT_00H_01H) \ __PMC_EV_ALIAS("GQ_CYCLES_FULL.WRITE_TRACKER", UCP_EVENT_00H_02H) \ __PMC_EV_ALIAS("GQ_CYCLES_FULL.PEER_PROBE_TRACKER", UCP_EVENT_00H_04H) \ __PMC_EV_ALIAS("GQ_CYCLES_NOT_EMPTY.READ_TRACKER", UCP_EVENT_01H_01H) \ __PMC_EV_ALIAS("GQ_CYCLES_NOT_EMPTY.WRITE_TRACKER", UCP_EVENT_01H_02H) \ __PMC_EV_ALIAS("GQ_CYCLES_NOT_EMPTY.PEER_PROBE_TRACKER", UCP_EVENT_01H_04H) \ __PMC_EV_ALIAS("GQ_OCCUPANCY.READ_TRACKER", UCP_EVENT_02H_01H) \ __PMC_EV_ALIAS("GQ_ALLOC.READ_TRACKER", UCP_EVENT_03H_01H) \ __PMC_EV_ALIAS("GQ_ALLOC.RT_L3_MISS", UCP_EVENT_03H_02H) \ __PMC_EV_ALIAS("GQ_ALLOC.RT_TO_L3_RESP", UCP_EVENT_03H_04H) \ __PMC_EV_ALIAS("GQ_ALLOC.RT_TO_RTID_ACQUIRED", UCP_EVENT_03H_08H) \ __PMC_EV_ALIAS("GQ_ALLOC.WT_TO_RTID_ACQUIRED", UCP_EVENT_03H_10H) \ __PMC_EV_ALIAS("GQ_ALLOC.WRITE_TRACKER", UCP_EVENT_03H_20H) \ __PMC_EV_ALIAS("GQ_ALLOC.PEER_PROBE_TRACKER", UCP_EVENT_03H_40H) \ __PMC_EV_ALIAS("GQ_DATA.FROM_QPI", UCP_EVENT_04H_01H) \ __PMC_EV_ALIAS("GQ_DATA.FROM_QMC", UCP_EVENT_04H_02H) \ __PMC_EV_ALIAS("GQ_DATA.FROM_L3", UCP_EVENT_04H_04H) \ __PMC_EV_ALIAS("GQ_DATA.FROM_CORES_02", UCP_EVENT_04H_08H) \ __PMC_EV_ALIAS("GQ_DATA.FROM_CORES_13", UCP_EVENT_04H_10H) \ __PMC_EV_ALIAS("GQ_DATA.TO_QPI_QMC", UCP_EVENT_05H_01H) \ __PMC_EV_ALIAS("GQ_DATA.TO_L3", UCP_EVENT_05H_02H) \ __PMC_EV_ALIAS("GQ_DATA.TO_CORES", UCP_EVENT_05H_04H) \ __PMC_EV_ALIAS("SNP_RESP_TO_LOCAL_HOME.I_STATE", UCP_EVENT_06H_01H) \ __PMC_EV_ALIAS("SNP_RESP_TO_LOCAL_HOME.S_STATE", UCP_EVENT_06H_02H) \ __PMC_EV_ALIAS("SNP_RESP_TO_LOCAL_HOME.FWD_S_STATE", UCP_EVENT_06H_04H) \ __PMC_EV_ALIAS("SNP_RESP_TO_LOCAL_HOME.FWD_I_STATE", UCP_EVENT_06H_08H) \ __PMC_EV_ALIAS("SNP_RESP_TO_LOCAL_HOME.CONFLICT", UCP_EVENT_06H_10H) \ __PMC_EV_ALIAS("SNP_RESP_TO_LOCAL_HOME.WB", UCP_EVENT_06H_20H) \ __PMC_EV_ALIAS("SNP_RESP_TO_REMOTE_HOME.I_STATE", UCP_EVENT_07H_01H) \ __PMC_EV_ALIAS("SNP_RESP_TO_REMOTE_HOME.S_STATE", UCP_EVENT_07H_02H) \ __PMC_EV_ALIAS("SNP_RESP_TO_REMOTE_HOME.FWD_S_STATE", UCP_EVENT_07H_04H) \ __PMC_EV_ALIAS("SNP_RESP_TO_REMOTE_HOME.FWD_I_STATE", UCP_EVENT_07H_08H) \ __PMC_EV_ALIAS("SNP_RESP_TO_REMOTE_HOME.CONFLICT", UCP_EVENT_07H_10H) \ __PMC_EV_ALIAS("SNP_RESP_TO_REMOTE_HOME.WB", UCP_EVENT_07H_20H) \ __PMC_EV_ALIAS("SNP_RESP_TO_REMOTE_HOME.HITM", UCP_EVENT_07H_24H) \ __PMC_EV_ALIAS("L3_HITS.READ", UCP_EVENT_08H_01H) \ __PMC_EV_ALIAS("L3_HITS.WRITE", UCP_EVENT_08H_02H) \ __PMC_EV_ALIAS("L3_HITS.PROBE", UCP_EVENT_08H_04H) \ __PMC_EV_ALIAS("L3_HITS.ANY", UCP_EVENT_08H_03H) \ __PMC_EV_ALIAS("L3_MISS.READ", UCP_EVENT_09H_01H) \ __PMC_EV_ALIAS("L3_MISS.WRITE", UCP_EVENT_09H_02H) \ __PMC_EV_ALIAS("L3_MISS.PROBE", UCP_EVENT_09H_04H) \ __PMC_EV_ALIAS("L3_MISS.ANY", UCP_EVENT_09H_03H) \ __PMC_EV_ALIAS("L3_LINES_IN.M_STATE", UCP_EVENT_0AH_01H) \ __PMC_EV_ALIAS("L3_LINES_IN.E_STATE", UCP_EVENT_0AH_02H) \ __PMC_EV_ALIAS("L3_LINES_IN.S_STATE", UCP_EVENT_0AH_04H) \ __PMC_EV_ALIAS("L3_LINES_IN.F_STATE", UCP_EVENT_0AH_08H) \ __PMC_EV_ALIAS("L3_LINES_IN.ANY", UCP_EVENT_0AH_0FH) \ __PMC_EV_ALIAS("L3_LINES_OUT.M_STATE", UCP_EVENT_0BH_01H) \ __PMC_EV_ALIAS("L3_LINES_OUT.E_STATE", UCP_EVENT_0BH_02H) \ __PMC_EV_ALIAS("L3_LINES_OUT.S_STATE", UCP_EVENT_0BH_04H) \ __PMC_EV_ALIAS("L3_LINES_OUT.I_STATE", UCP_EVENT_0BH_08H) \ __PMC_EV_ALIAS("L3_LINES_OUT.F_STATE", UCP_EVENT_0BH_10H) \ __PMC_EV_ALIAS("L3_LINES_OUT.ANY", UCP_EVENT_0BH_1FH) \ __PMC_EV_ALIAS("GQ_SNOOP.GOTO_S", UCP_EVENT_0CH_01H) \ __PMC_EV_ALIAS("GQ_SNOOP.GOTO_I", UCP_EVENT_0CH_02H) \ __PMC_EV_ALIAS("GQ_SNOOP.GOTO_S_HIT", UCP_EVENT_0CH_04H) \ __PMC_EV_ALIAS("GQ_SNOOP.GOTO_I_HIT", UCP_EVENT_0CH_08H) \ __PMC_EV_ALIAS("QHL_REQUESTS.IOH_READS", UCP_EVENT_20H_01H) \ __PMC_EV_ALIAS("QHL_REQUESTS.IOH_WRITES", UCP_EVENT_20H_02H) \ __PMC_EV_ALIAS("QHL_REQUESTS.REMOTE_READS", UCP_EVENT_20H_04H) \ __PMC_EV_ALIAS("QHL_REQUESTS.REMOTE_WRITES", UCP_EVENT_20H_08H) \ __PMC_EV_ALIAS("QHL_REQUESTS.LOCAL_READS", UCP_EVENT_20H_10H) \ __PMC_EV_ALIAS("QHL_REQUESTS.LOCAL_WRITES", UCP_EVENT_20H_20H) \ __PMC_EV_ALIAS("QHL_CYCLES_FULL.IOH", UCP_EVENT_21H_01H) \ __PMC_EV_ALIAS("QHL_CYCLES_FULL.REMOTE", UCP_EVENT_21H_02H) \ __PMC_EV_ALIAS("QHL_CYCLES_FULL.LOCAL", UCP_EVENT_21H_04H) \ __PMC_EV_ALIAS("QHL_CYCLES_NOT_EMPTY.IOH", UCP_EVENT_22H_01H) \ __PMC_EV_ALIAS("QHL_CYCLES_NOT_EMPTY.REMOTE", UCP_EVENT_22H_02H) \ __PMC_EV_ALIAS("QHL_CYCLES_NOT_EMPTY.LOCAL", UCP_EVENT_22H_04H) \ __PMC_EV_ALIAS("QHL_OCCUPANCY.IOH", UCP_EVENT_23H_01H) \ __PMC_EV_ALIAS("QHL_OCCUPANCY.REMOTE", UCP_EVENT_23H_02H) \ __PMC_EV_ALIAS("QHL_OCCUPANCY.LOCAL", UCP_EVENT_23H_04H) \ __PMC_EV_ALIAS("QHL_ADDRESS_CONFLICTS.2WAY", UCP_EVENT_24H_02H) \ __PMC_EV_ALIAS("QHL_ADDRESS_CONFLICTS.3WAY", UCP_EVENT_24H_04H) \ __PMC_EV_ALIAS("QHL_CONFLICT_CYCLES.IOH", UCP_EVENT_25H_01H) \ __PMC_EV_ALIAS("QHL_CONFLICT_CYCLES.REMOTE", UCP_EVENT_25H_02H) \ __PMC_EV_ALIAS("QHL_CONFLICT_CYCLES.LOCAL", UCP_EVENT_25H_04H) \ __PMC_EV_ALIAS("QHL_TO_QMC_BYPASS", UCP_EVENT_26H_01H) \ __PMC_EV_ALIAS("QMC_ISOC_FULL.READ.CH0", UCP_EVENT_28H_01H) \ __PMC_EV_ALIAS("QMC_ISOC_FULL.READ.CH1", UCP_EVENT_28H_02H) \ __PMC_EV_ALIAS("QMC_ISOC_FULL.READ.CH2", UCP_EVENT_28H_04H) \ __PMC_EV_ALIAS("QMC_ISOC_FULL.WRITE.CH0", UCP_EVENT_28H_08H) \ __PMC_EV_ALIAS("QMC_ISOC_FULL.WRITE.CH1", UCP_EVENT_28H_10H) \ __PMC_EV_ALIAS("QMC_ISOC_FULL.WRITE.CH2", UCP_EVENT_28H_20H) \ __PMC_EV_ALIAS("QMC_BUSY.READ.CH0", UCP_EVENT_29H_01H) \ __PMC_EV_ALIAS("QMC_BUSY.READ.CH1", UCP_EVENT_29H_02H) \ __PMC_EV_ALIAS("QMC_BUSY.READ.CH2", UCP_EVENT_29H_04H) \ __PMC_EV_ALIAS("QMC_BUSY.WRITE.CH0", UCP_EVENT_29H_08H) \ __PMC_EV_ALIAS("QMC_BUSY.WRITE.CH1", UCP_EVENT_29H_10H) \ __PMC_EV_ALIAS("QMC_BUSY.WRITE.CH2", UCP_EVENT_29H_20H) \ __PMC_EV_ALIAS("QMC_OCCUPANCY.CH0", UCP_EVENT_2AH_01H) \ __PMC_EV_ALIAS("QMC_OCCUPANCY.CH1", UCP_EVENT_2AH_02H) \ __PMC_EV_ALIAS("QMC_OCCUPANCY.CH2", UCP_EVENT_2AH_04H) \ __PMC_EV_ALIAS("QMC_OCCUPANCY.ANY", UCP_EVENT_2AH_07H) \ __PMC_EV_ALIAS("QMC_ISSOC_OCCUPANCY.CH0", UCP_EVENT_2BH_01H) \ __PMC_EV_ALIAS("QMC_ISSOC_OCCUPANCY.CH1", UCP_EVENT_2BH_02H) \ __PMC_EV_ALIAS("QMC_ISSOC_OCCUPANCY.CH2", UCP_EVENT_2BH_04H) \ __PMC_EV_ALIAS("QMC_ISSOC_READS.ANY", UCP_EVENT_2BH_07H) \ __PMC_EV_ALIAS("QMC_NORMAL_READS.CH0", UCP_EVENT_2CH_01H) \ __PMC_EV_ALIAS("QMC_NORMAL_READS.CH1", UCP_EVENT_2CH_02H) \ __PMC_EV_ALIAS("QMC_NORMAL_READS.CH2", UCP_EVENT_2CH_04H) \ __PMC_EV_ALIAS("QMC_NORMAL_READS.ANY", UCP_EVENT_2CH_07H) \ __PMC_EV_ALIAS("QMC_HIGH_PRIORITY_READS.CH0", UCP_EVENT_2DH_01H) \ __PMC_EV_ALIAS("QMC_HIGH_PRIORITY_READS.CH1", UCP_EVENT_2DH_02H) \ __PMC_EV_ALIAS("QMC_HIGH_PRIORITY_READS.CH2", UCP_EVENT_2DH_04H) \ __PMC_EV_ALIAS("QMC_HIGH_PRIORITY_READS.ANY", UCP_EVENT_2DH_07H) \ __PMC_EV_ALIAS("QMC_CRITICAL_PRIORITY_READS.CH0", UCP_EVENT_2EH_01H) \ __PMC_EV_ALIAS("QMC_CRITICAL_PRIORITY_READS.CH1", UCP_EVENT_2EH_02H) \ __PMC_EV_ALIAS("QMC_CRITICAL_PRIORITY_READS.CH2", UCP_EVENT_2EH_04H) \ __PMC_EV_ALIAS("QMC_CRITICAL_PRIORITY_READS.ANY", UCP_EVENT_2EH_07H) \ __PMC_EV_ALIAS("QMC_WRITES.FULL.CH0", UCP_EVENT_2FH_01H) \ __PMC_EV_ALIAS("QMC_WRITES.FULL.CH1", UCP_EVENT_2FH_02H) \ __PMC_EV_ALIAS("QMC_WRITES.FULL.CH2", UCP_EVENT_2FH_04H) \ __PMC_EV_ALIAS("QMC_WRITES.FULL.ANY", UCP_EVENT_2FH_07H) \ __PMC_EV_ALIAS("QMC_WRITES.PARTIAL.CH0", UCP_EVENT_2FH_08H) \ __PMC_EV_ALIAS("QMC_WRITES.PARTIAL.CH1", UCP_EVENT_2FH_10H) \ __PMC_EV_ALIAS("QMC_WRITES.PARTIAL.CH2", UCP_EVENT_2FH_20H) \ __PMC_EV_ALIAS("QMC_WRITES.PARTIAL.ANY", UCP_EVENT_2FH_38H) \ __PMC_EV_ALIAS("QMC_CANCEL.CH0", UCP_EVENT_30H_01H) \ __PMC_EV_ALIAS("QMC_CANCEL.CH1", UCP_EVENT_30H_02H) \ __PMC_EV_ALIAS("QMC_CANCEL.CH2", UCP_EVENT_30H_04H) \ __PMC_EV_ALIAS("QMC_CANCEL.ANY", UCP_EVENT_30H_07H) \ __PMC_EV_ALIAS("QMC_PRIORITY_UPDATES.CH0", UCP_EVENT_31H_01H) \ __PMC_EV_ALIAS("QMC_PRIORITY_UPDATES.CH1", UCP_EVENT_31H_02H) \ __PMC_EV_ALIAS("QMC_PRIORITY_UPDATES.CH2", UCP_EVENT_31H_04H) \ __PMC_EV_ALIAS("QMC_PRIORITY_UPDATES.ANY", UCP_EVENT_31H_07H) \ __PMC_EV_ALIAS("IMC_RETRY.CH0", UCP_EVENT_32H_01H) \ __PMC_EV_ALIAS("IMC_RETRY.CH1", UCP_EVENT_32H_02H) \ __PMC_EV_ALIAS("IMC_RETRY.CH2", UCP_EVENT_32H_04H) \ __PMC_EV_ALIAS("IMC_RETRY.ANY", UCP_EVENT_32H_07H) \ __PMC_EV_ALIAS("QHL_FRC_ACK_CNFLTS.IOH", UCP_EVENT_33H_01H) \ __PMC_EV_ALIAS("QHL_FRC_ACK_CNFLTS.REMOTE", UCP_EVENT_33H_02H) \ __PMC_EV_ALIAS("QHL_FRC_ACK_CNFLTS.LOCAL", UCP_EVENT_33H_04H) \ __PMC_EV_ALIAS("QHL_FRC_ACK_CNFLTS.ANY", UCP_EVENT_33H_07H) \ __PMC_EV_ALIAS("QHL_SLEEPS.IOH_ORDER", UCP_EVENT_34H_01H) \ __PMC_EV_ALIAS("QHL_SLEEPS.REMOTE_ORDER", UCP_EVENT_34H_02H) \ __PMC_EV_ALIAS("QHL_SLEEPS.LOCAL_ORDER", UCP_EVENT_34H_04H) \ __PMC_EV_ALIAS("QHL_SLEEPS.IOH_CONFLICT", UCP_EVENT_34H_08H) \ __PMC_EV_ALIAS("QHL_SLEEPS.REMOTE_CONFLICT", UCP_EVENT_34H_10H) \ __PMC_EV_ALIAS("QHL_SLEEPS.LOCAL_CONFLICT", UCP_EVENT_34H_20H) \ __PMC_EV_ALIAS("ADDR_OPCODE_MATCH.IOH", UCP_EVENT_35H_01H) \ __PMC_EV_ALIAS("ADDR_OPCODE_MATCH.REMOTE", UCP_EVENT_35H_02H) \ __PMC_EV_ALIAS("ADDR_OPCODE_MATCH.LOCAL", UCP_EVENT_35H_04H) \ __PMC_EV_ALIAS("QPI_TX_STALLED_SINGLE_FLIT.HOME.LINK_0", UCP_EVENT_40H_01H) \ __PMC_EV_ALIAS("QPI_TX_STALLED_SINGLE_FLIT.SNOOP.LINK_0", UCP_EVENT_40H_02H) \ __PMC_EV_ALIAS("QPI_TX_STALLED_SINGLE_FLIT.NDR.LINK_0", UCP_EVENT_40H_04H) \ __PMC_EV_ALIAS("QPI_TX_STALLED_SINGLE_FLIT.HOME.LINK_1", UCP_EVENT_40H_08H) \ __PMC_EV_ALIAS("QPI_TX_STALLED_SINGLE_FLIT.SNOOP.LINK_1", UCP_EVENT_40H_10H) \ __PMC_EV_ALIAS("QPI_TX_STALLED_SINGLE_FLIT.NDR.LINK_1", UCP_EVENT_40H_20H) \ __PMC_EV_ALIAS("QPI_TX_STALLED_SINGLE_FLIT.LINK_0", UCP_EVENT_40H_07H) \ __PMC_EV_ALIAS("QPI_TX_STALLED_SINGLE_FLIT.LINK_1", UCP_EVENT_40H_38H) \ __PMC_EV_ALIAS("QPI_TX_STALLED_MULTI_FLIT.DRS.LINK_0", UCP_EVENT_41H_01H) \ __PMC_EV_ALIAS("QPI_TX_STALLED_MULTI_FLIT.NCB.LINK_0", UCP_EVENT_41H_02H) \ __PMC_EV_ALIAS("QPI_TX_STALLED_MULTI_FLIT.NCS.LINK_0", UCP_EVENT_41H_04H) \ __PMC_EV_ALIAS("QPI_TX_STALLED_MULTI_FLIT.DRS.LINK_1", UCP_EVENT_41H_08H) \ __PMC_EV_ALIAS("QPI_TX_STALLED_MULTI_FLIT.NCB.LINK_1", UCP_EVENT_41H_10H) \ __PMC_EV_ALIAS("QPI_TX_STALLED_MULTI_FLIT.NCS.LINK_1", UCP_EVENT_41H_20H) \ __PMC_EV_ALIAS("QPI_TX_STALLED_MULTI_FLIT.LINK_0", UCP_EVENT_41H_07H) \ __PMC_EV_ALIAS("QPI_TX_STALLED_MULTI_FLIT.LINK_1", UCP_EVENT_41H_38H) \ __PMC_EV_ALIAS("QPI_TX_HEADER.FULL.LINK_0", UCP_EVENT_42H_01H) \ __PMC_EV_ALIAS("QPI_TX_HEADER.BUSY.LINK_0", UCP_EVENT_42H_02H) \ __PMC_EV_ALIAS("QPI_TX_HEADER.FULL.LINK_1", UCP_EVENT_42H_04H) \ __PMC_EV_ALIAS("QPI_TX_HEADER.BUSY.LINK_1", UCP_EVENT_42H_08H) \ __PMC_EV_ALIAS("QPI_RX_NO_PPT_CREDIT.STALLS.LINK_0", UCP_EVENT_43H_01H) \ __PMC_EV_ALIAS("QPI_RX_NO_PPT_CREDIT.STALLS.LINK_1", UCP_EVENT_43H_02H) \ __PMC_EV_ALIAS("DRAM_OPEN.CH0", UCP_EVENT_60H_01H) \ __PMC_EV_ALIAS("DRAM_OPEN.CH1", UCP_EVENT_60H_02H) \ __PMC_EV_ALIAS("DRAM_OPEN.CH2", UCP_EVENT_60H_04H) \ __PMC_EV_ALIAS("DRAM_PAGE_CLOSE.CH0", UCP_EVENT_61H_01H) \ __PMC_EV_ALIAS("DRAM_PAGE_CLOSE.CH1", UCP_EVENT_61H_02H) \ __PMC_EV_ALIAS("DRAM_PAGE_CLOSE.CH2", UCP_EVENT_61H_04H) \ __PMC_EV_ALIAS("DRAM_PAGE_MISS.CH0", UCP_EVENT_62H_01H) \ __PMC_EV_ALIAS("DRAM_PAGE_MISS.CH1", UCP_EVENT_62H_02H) \ __PMC_EV_ALIAS("DRAM_PAGE_MISS.CH2", UCP_EVENT_62H_04H) \ __PMC_EV_ALIAS("DRAM_READ_CAS.CH0", UCP_EVENT_63H_01H) \ __PMC_EV_ALIAS("DRAM_READ_CAS.AUTOPRE_CH0", UCP_EVENT_63H_02H) \ __PMC_EV_ALIAS("DRAM_READ_CAS.CH1", UCP_EVENT_63H_04H) \ __PMC_EV_ALIAS("DRAM_READ_CAS.AUTOPRE_CH1", UCP_EVENT_63H_08H) \ __PMC_EV_ALIAS("DRAM_READ_CAS.CH2", UCP_EVENT_63H_10H) \ __PMC_EV_ALIAS("DRAM_READ_CAS.AUTOPRE_CH2", UCP_EVENT_63H_20H) \ __PMC_EV_ALIAS("DRAM_WRITE_CAS.CH0", UCP_EVENT_64H_01H) \ __PMC_EV_ALIAS("DRAM_WRITE_CAS.AUTOPRE_CH0", UCP_EVENT_64H_02H) \ __PMC_EV_ALIAS("DRAM_WRITE_CAS.CH1", UCP_EVENT_64H_04H) \ __PMC_EV_ALIAS("DRAM_WRITE_CAS.AUTOPRE_CH1", UCP_EVENT_64H_08H) \ __PMC_EV_ALIAS("DRAM_WRITE_CAS.CH2", UCP_EVENT_64H_10H) \ __PMC_EV_ALIAS("DRAM_WRITE_CAS.AUTOPRE_CH2", UCP_EVENT_64H_20H) \ __PMC_EV_ALIAS("DRAM_REFRESH.CH0", UCP_EVENT_65H_01H) \ __PMC_EV_ALIAS("DRAM_REFRESH.CH1", UCP_EVENT_65H_02H) \ __PMC_EV_ALIAS("DRAM_REFRESH.CH2", UCP_EVENT_65H_04H) \ __PMC_EV_ALIAS("DRAM_PRE_ALL.CH0", UCP_EVENT_66H_01H) \ __PMC_EV_ALIAS("DRAM_PRE_ALL.CH1", UCP_EVENT_66H_02H) \ __PMC_EV_ALIAS("DRAM_PRE_ALL.CH2", UCP_EVENT_66H_04H) \ __PMC_EV_ALIAS("DRAM_THERMAL_THROTTLED", UCP_EVENT_67H_01H) \ __PMC_EV_ALIAS("THERMAL_THROTTLING_TEMP.CORE_0", UCP_EVENT_80H_01H) \ __PMC_EV_ALIAS("THERMAL_THROTTLING_TEMP.CORE_1", UCP_EVENT_80H_02H) \ __PMC_EV_ALIAS("THERMAL_THROTTLING_TEMP.CORE_2", UCP_EVENT_80H_04H) \ __PMC_EV_ALIAS("THERMAL_THROTTLING_TEMP.CORE_3", UCP_EVENT_80H_08H) \ __PMC_EV_ALIAS("THERMAL_THROTTLED_TEMP.CORE_0", UCP_EVENT_81H_01H) \ __PMC_EV_ALIAS("THERMAL_THROTTLED_TEMP.CORE_1", UCP_EVENT_81H_02H) \ __PMC_EV_ALIAS("THERMAL_THROTTLED_TEMP.CORE_2", UCP_EVENT_81H_04H) \ __PMC_EV_ALIAS("THERMAL_THROTTLED_TEMP.CORE_3", UCP_EVENT_81H_08H) \ __PMC_EV_ALIAS("PROCHOT_ASSERTION", UCP_EVENT_82H_01H) \ __PMC_EV_ALIAS("THERMAL_THROTTLING_PROCHOT.CORE_0", UCP_EVENT_83H_01H) \ __PMC_EV_ALIAS("THERMAL_THROTTLING_PROCHOT.CORE_1", UCP_EVENT_83H_02H) \ __PMC_EV_ALIAS("THERMAL_THROTTLING_PROCHOT.CORE_2", UCP_EVENT_83H_04H) \ __PMC_EV_ALIAS("THERMAL_THROTTLING_PROCHOT.CORE_3", UCP_EVENT_83H_08H) \ __PMC_EV_ALIAS("TURBO_MODE.CORE_0", UCP_EVENT_84H_01H) \ __PMC_EV_ALIAS("TURBO_MODE.CORE_1", UCP_EVENT_84H_02H) \ __PMC_EV_ALIAS("TURBO_MODE.CORE_2", UCP_EVENT_84H_04H) \ __PMC_EV_ALIAS("TURBO_MODE.CORE_3", UCP_EVENT_84H_08H) \ __PMC_EV_ALIAS("CYCLES_UNHALTED_L3_FLL_ENABLE", UCP_EVENT_85H_02H) \ __PMC_EV_ALIAS("CYCLES_UNHALTED_L3_FLL_DISABLE", UCP_EVENT_86H_01H) /* * Intel XScale events from: * * Intel XScale Core Developer's Manual * January, 2004, #27347302 * * 3rd Generation Intel XScale Microarchitecture * Developer's Manual * May 2007, #31628302 * * First 14 events are for 1st and 2nd Generation Intel XScale cores. The * remaining are available only on 3rd Generation Intel XScale cores. */ #define __PMC_EV_XSCALE() \ __PMC_EV(XSCALE, IC_FETCH) \ __PMC_EV(XSCALE, IC_MISS) \ __PMC_EV(XSCALE, DATA_DEPENDENCY_STALLED) \ __PMC_EV(XSCALE, ITLB_MISS) \ __PMC_EV(XSCALE, DTLB_MISS) \ __PMC_EV(XSCALE, BRANCH_RETIRED) \ __PMC_EV(XSCALE, BRANCH_MISPRED) \ __PMC_EV(XSCALE, INSTR_RETIRED) \ __PMC_EV(XSCALE, DC_FULL_CYCLE) \ __PMC_EV(XSCALE, DC_FULL_CONTIG) \ __PMC_EV(XSCALE, DC_ACCESS) \ __PMC_EV(XSCALE, DC_MISS) \ __PMC_EV(XSCALE, DC_WRITEBACK) \ __PMC_EV(XSCALE, PC_CHANGE) \ __PMC_EV(XSCALE, BRANCH_RETIRED_ALL) \ __PMC_EV(XSCALE, INSTR_CYCLE) \ __PMC_EV(XSCALE, CP_STALL) \ __PMC_EV(XSCALE, PC_CHANGE_ALL) \ __PMC_EV(XSCALE, PIPELINE_FLUSH) \ __PMC_EV(XSCALE, BACKEND_STALL) \ __PMC_EV(XSCALE, MULTIPLIER_USE) \ __PMC_EV(XSCALE, MULTIPLIER_STALLED) \ __PMC_EV(XSCALE, DATA_CACHE_STALLED) \ __PMC_EV(XSCALE, L2_CACHE_REQ) \ __PMC_EV(XSCALE, L2_CACHE_MISS) \ __PMC_EV(XSCALE, ADDRESS_BUS_TRANS) \ __PMC_EV(XSCALE, SELF_ADDRESS_BUS_TRANS) \ __PMC_EV(XSCALE, DATA_BUS_TRANS) #define PMC_EV_XSCALE_FIRST PMC_EV_XSCALE_IC_FETCH #define PMC_EV_XSCALE_LAST PMC_EV_XSCALE_DATA_BUS_TRANS /* * MIPS Events from "Programming the MIPS32 24K Core Family", * Document Number: MD00355 Revision 04.63 December 19, 2008 * These events are kept in the order found in Table 7.4. * For counters which are different between the left hand * column (0/2) and the right hand column (1/3) the left * hand is given first, e.g. BRANCH_COMPLETED and BRANCH_MISPRED * in the definition below. */ #define __PMC_EV_MIPS24K() \ __PMC_EV(MIPS24K, CYCLE) \ __PMC_EV(MIPS24K, INSTR_EXECUTED) \ __PMC_EV(MIPS24K, BRANCH_COMPLETED) \ __PMC_EV(MIPS24K, BRANCH_MISPRED) \ __PMC_EV(MIPS24K, RETURN) \ __PMC_EV(MIPS24K, RETURN_MISPRED) \ __PMC_EV(MIPS24K, RETURN_NOT_31) \ __PMC_EV(MIPS24K, RETURN_NOTPRED) \ __PMC_EV(MIPS24K, ITLB_ACCESS) \ __PMC_EV(MIPS24K, ITLB_MISS) \ __PMC_EV(MIPS24K, DTLB_ACCESS) \ __PMC_EV(MIPS24K, DTLB_MISS) \ __PMC_EV(MIPS24K, JTLB_IACCESS) \ __PMC_EV(MIPS24K, JTLB_IMISS) \ __PMC_EV(MIPS24K, JTLB_DACCESS) \ __PMC_EV(MIPS24K, JTLB_DMISS) \ __PMC_EV(MIPS24K, IC_FETCH) \ __PMC_EV(MIPS24K, IC_MISS) \ __PMC_EV(MIPS24K, DC_LOADSTORE) \ __PMC_EV(MIPS24K, DC_WRITEBACK) \ __PMC_EV(MIPS24K, DC_MISS) \ __PMC_EV(MIPS24K, STORE_MISS) \ __PMC_EV(MIPS24K, LOAD_MISS) \ __PMC_EV(MIPS24K, INTEGER_COMPLETED) \ __PMC_EV(MIPS24K, FP_COMPLETED) \ __PMC_EV(MIPS24K, LOAD_COMPLETED) \ __PMC_EV(MIPS24K, STORE_COMPLETED) \ __PMC_EV(MIPS24K, BARRIER_COMPLETED) \ __PMC_EV(MIPS24K, MIPS16_COMPLETED) \ __PMC_EV(MIPS24K, NOP_COMPLETED) \ __PMC_EV(MIPS24K, INTEGER_MULDIV_COMPLETED)\ __PMC_EV(MIPS24K, RF_STALL) \ __PMC_EV(MIPS24K, INSTR_REFETCH) \ __PMC_EV(MIPS24K, STORE_COND_COMPLETED) \ __PMC_EV(MIPS24K, STORE_COND_FAILED) \ __PMC_EV(MIPS24K, ICACHE_REQUESTS) \ __PMC_EV(MIPS24K, ICACHE_HIT) \ __PMC_EV(MIPS24K, L2_WRITEBACK) \ __PMC_EV(MIPS24K, L2_ACCESS) \ __PMC_EV(MIPS24K, L2_MISS) \ __PMC_EV(MIPS24K, L2_ERR_CORRECTED) \ __PMC_EV(MIPS24K, EXCEPTIONS) \ __PMC_EV(MIPS24K, RF_CYCLES_STALLED) \ __PMC_EV(MIPS24K, IFU_CYCLES_STALLED) \ __PMC_EV(MIPS24K, ALU_CYCLES_STALLED) \ __PMC_EV(MIPS24K, UNCACHED_LOAD) \ __PMC_EV(MIPS24K, UNCACHED_STORE) \ __PMC_EV(MIPS24K, CP2_REG_TO_REG_COMPLETED)\ __PMC_EV(MIPS24K, MFTC_COMPLETED) \ __PMC_EV(MIPS24K, IC_BLOCKED_CYCLES) \ __PMC_EV(MIPS24K, DC_BLOCKED_CYCLES) \ __PMC_EV(MIPS24K, L2_IMISS_STALL_CYCLES) \ __PMC_EV(MIPS24K, L2_DMISS_STALL_CYCLES) \ __PMC_EV(MIPS24K, DMISS_CYCLES) \ __PMC_EV(MIPS24K, L2_MISS_CYCLES) \ __PMC_EV(MIPS24K, UNCACHED_BLOCK_CYCLES) \ __PMC_EV(MIPS24K, MDU_STALL_CYCLES) \ __PMC_EV(MIPS24K, FPU_STALL_CYCLES) \ __PMC_EV(MIPS24K, CP2_STALL_CYCLES) \ __PMC_EV(MIPS24K, COREXTEND_STALL_CYCLES) \ __PMC_EV(MIPS24K, ISPRAM_STALL_CYCLES) \ __PMC_EV(MIPS24K, DSPRAM_STALL_CYCLES) \ __PMC_EV(MIPS24K, CACHE_STALL_CYCLES) \ __PMC_EV(MIPS24K, LOAD_TO_USE_STALLS) \ __PMC_EV(MIPS24K, BASE_MISPRED_STALLS) \ __PMC_EV(MIPS24K, CPO_READ_STALLS) \ __PMC_EV(MIPS24K, BRANCH_MISPRED_CYCLES) \ __PMC_EV(MIPS24K, IFETCH_BUFFER_FULL) \ __PMC_EV(MIPS24K, FETCH_BUFFER_ALLOCATED) \ __PMC_EV(MIPS24K, EJTAG_ITRIGGER) \ __PMC_EV(MIPS24K, EJTAG_DTRIGGER) \ __PMC_EV(MIPS24K, FSB_LT_QUARTER) \ __PMC_EV(MIPS24K, FSB_QUARTER_TO_HALF) \ __PMC_EV(MIPS24K, FSB_GT_HALF) \ __PMC_EV(MIPS24K, FSB_FULL_PIPELINE_STALLS)\ __PMC_EV(MIPS24K, LDQ_LT_QUARTER) \ __PMC_EV(MIPS24K, LDQ_QUARTER_TO_HALF) \ __PMC_EV(MIPS24K, LDQ_GT_HALF) \ __PMC_EV(MIPS24K, LDQ_FULL_PIPELINE_STALLS)\ __PMC_EV(MIPS24K, WBB_LT_QUARTER) \ __PMC_EV(MIPS24K, WBB_QUARTER_TO_HALF) \ __PMC_EV(MIPS24K, WBB_GT_HALF) \ __PMC_EV(MIPS24K, WBB_FULL_PIPELINE_STALLS) \ __PMC_EV(MIPS24K, REQUEST_LATENCY) \ __PMC_EV(MIPS24K, REQUEST_COUNT) #define PMC_EV_MIPS24K_FIRST PMC_EV_MIPS24K_CYCLE #define PMC_EV_MIPS24K_LAST PMC_EV_MIPS24K_WBB_FULL_PIPELINE_STALLS /* * All known PMC events. * * PMC event numbers are allocated sparsely to allow new PMC events to * be added to a PMC class without breaking ABI compatibility. The * current allocation scheme is: * * START #EVENTS DESCRIPTION * 0 0x1000 Reserved * 0x1000 0x0001 TSC * 0x2000 0x0080 AMD K7 events * 0x2080 0x0100 AMD K8 events * 0x10000 0x0080 INTEL architectural fixed-function events * 0x10080 0x0F80 INTEL architectural programmable events * 0x11000 0x0080 INTEL Pentium 4 events * 0x11080 0x0080 INTEL Pentium MMX events * 0x11100 0x0100 INTEL Pentium Pro/P-II/P-III/Pentium-M events * 0x11200 0x00FF INTEL XScale events * 0x11300 0x00FF MIPS 24K events */ #define __PMC_EVENTS() \ __PMC_EV_BLOCK(TSC, 0x01000) \ __PMC_EV_TSC() \ __PMC_EV_BLOCK(K7, 0x2000) \ __PMC_EV_K7() \ __PMC_EV_BLOCK(K8, 0x2080) \ __PMC_EV_K8() \ __PMC_EV_BLOCK(IAF, 0x10000) \ __PMC_EV_IAF() \ __PMC_EV_BLOCK(IAP, 0x10080) \ __PMC_EV_IAP() \ __PMC_EV_BLOCK(P4, 0x11000) \ __PMC_EV_P4() \ __PMC_EV_BLOCK(P5, 0x11080) \ __PMC_EV_P5() \ __PMC_EV_BLOCK(P6, 0x11100) \ __PMC_EV_P6() \ __PMC_EV_BLOCK(XSCALE, 0x11200) \ __PMC_EV_XSCALE() \ __PMC_EV_BLOCK(MIPS24K, 0x11300) \ __PMC_EV_MIPS24K() \ __PMC_EV_BLOCK(UCF, 0x12000) \ __PMC_EV_UCF() \ __PMC_EV_BLOCK(UCP, 0x12080) \ __PMC_EV_UCP() \ #define PMC_EVENT_FIRST PMC_EV_TSC_TSC #define PMC_EVENT_LAST PMC_EV_UCP_LAST #endif /* _DEV_HWPMC_PMC_EVENTS_H_ */ Index: user/imp/tbemd/sys/dev/mxge/if_mxge.c =================================================================== --- user/imp/tbemd/sys/dev/mxge/if_mxge.c (revision 206684) +++ user/imp/tbemd/sys/dev/mxge/if_mxge.c (revision 206685) @@ -1,4869 +1,4896 @@ /****************************************************************************** Copyright (c) 2006-2009, Myricom Inc. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Neither the name of the Myricom Inc, nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. ***************************************************************************/ #include __FBSDID("$FreeBSD$"); #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include /* count xmits ourselves, rather than via drbr */ #define NO_SLOW_STATS #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include /* XXX for pci_cfg_restore */ #include /* for pmap_mapdev() */ #include #if defined(__i386) || defined(__amd64) #include #endif #include #include /*#define MXGE_FAKE_IFP*/ #include #ifdef IFNET_BUF_RING #include #endif #include "opt_inet.h" /* tunable params */ static int mxge_nvidia_ecrc_enable = 1; static int mxge_force_firmware = 0; static int mxge_intr_coal_delay = 30; static int mxge_deassert_wait = 1; static int mxge_flow_control = 1; static int mxge_verbose = 0; static int mxge_lro_cnt = 8; static int mxge_ticks; static int mxge_max_slices = 1; static int mxge_rss_hash_type = MXGEFW_RSS_HASH_TYPE_SRC_DST_PORT; static int mxge_always_promisc = 0; static int mxge_initial_mtu = ETHERMTU_JUMBO; static int mxge_throttle = 0; static char *mxge_fw_unaligned = "mxge_ethp_z8e"; static char *mxge_fw_aligned = "mxge_eth_z8e"; static char *mxge_fw_rss_aligned = "mxge_rss_eth_z8e"; static char *mxge_fw_rss_unaligned = "mxge_rss_ethp_z8e"; static int mxge_probe(device_t dev); static int mxge_attach(device_t dev); static int mxge_detach(device_t dev); static int mxge_shutdown(device_t dev); static void mxge_intr(void *arg); static device_method_t mxge_methods[] = { /* Device interface */ DEVMETHOD(device_probe, mxge_probe), DEVMETHOD(device_attach, mxge_attach), DEVMETHOD(device_detach, mxge_detach), DEVMETHOD(device_shutdown, mxge_shutdown), {0, 0} }; static driver_t mxge_driver = { "mxge", mxge_methods, sizeof(mxge_softc_t), }; static devclass_t mxge_devclass; /* Declare ourselves to be a child of the PCI bus.*/ DRIVER_MODULE(mxge, pci, mxge_driver, mxge_devclass, 0, 0); MODULE_DEPEND(mxge, firmware, 1, 1, 1); MODULE_DEPEND(mxge, zlib, 1, 1, 1); static int mxge_load_firmware(mxge_softc_t *sc, int adopt); static int mxge_send_cmd(mxge_softc_t *sc, uint32_t cmd, mxge_cmd_t *data); static int mxge_close(mxge_softc_t *sc, int down); static int mxge_open(mxge_softc_t *sc); static void mxge_tick(void *arg); static int mxge_probe(device_t dev) { int rev; if ((pci_get_vendor(dev) == MXGE_PCI_VENDOR_MYRICOM) && ((pci_get_device(dev) == MXGE_PCI_DEVICE_Z8E) || (pci_get_device(dev) == MXGE_PCI_DEVICE_Z8E_9))) { rev = pci_get_revid(dev); switch (rev) { case MXGE_PCI_REV_Z8E: device_set_desc(dev, "Myri10G-PCIE-8A"); break; case MXGE_PCI_REV_Z8ES: device_set_desc(dev, "Myri10G-PCIE-8B"); break; default: device_set_desc(dev, "Myri10G-PCIE-8??"); device_printf(dev, "Unrecognized rev %d NIC\n", rev); break; } return 0; } return ENXIO; } static void mxge_enable_wc(mxge_softc_t *sc) { #if defined(__i386) || defined(__amd64) vm_offset_t len; int err; sc->wc = 1; len = rman_get_size(sc->mem_res); err = pmap_change_attr((vm_offset_t) sc->sram, len, PAT_WRITE_COMBINING); if (err != 0) { device_printf(sc->dev, "pmap_change_attr failed, %d\n", err); sc->wc = 0; } #endif } /* callback to get our DMA address */ static void mxge_dmamap_callback(void *arg, bus_dma_segment_t *segs, int nsegs, int error) { if (error == 0) { *(bus_addr_t *) arg = segs->ds_addr; } } static int mxge_dma_alloc(mxge_softc_t *sc, mxge_dma_t *dma, size_t bytes, bus_size_t alignment) { int err; device_t dev = sc->dev; bus_size_t boundary, maxsegsize; if (bytes > 4096 && alignment == 4096) { boundary = 0; maxsegsize = bytes; } else { boundary = 4096; maxsegsize = 4096; } /* allocate DMAable memory tags */ err = bus_dma_tag_create(sc->parent_dmat, /* parent */ alignment, /* alignment */ boundary, /* boundary */ BUS_SPACE_MAXADDR, /* low */ BUS_SPACE_MAXADDR, /* high */ NULL, NULL, /* filter */ bytes, /* maxsize */ 1, /* num segs */ maxsegsize, /* maxsegsize */ BUS_DMA_COHERENT, /* flags */ NULL, NULL, /* lock */ &dma->dmat); /* tag */ if (err != 0) { device_printf(dev, "couldn't alloc tag (err = %d)\n", err); return err; } /* allocate DMAable memory & map */ err = bus_dmamem_alloc(dma->dmat, &dma->addr, (BUS_DMA_WAITOK | BUS_DMA_COHERENT | BUS_DMA_ZERO), &dma->map); if (err != 0) { device_printf(dev, "couldn't alloc mem (err = %d)\n", err); goto abort_with_dmat; } /* load the memory */ err = bus_dmamap_load(dma->dmat, dma->map, dma->addr, bytes, mxge_dmamap_callback, (void *)&dma->bus_addr, 0); if (err != 0) { device_printf(dev, "couldn't load map (err = %d)\n", err); goto abort_with_mem; } return 0; abort_with_mem: bus_dmamem_free(dma->dmat, dma->addr, dma->map); abort_with_dmat: (void)bus_dma_tag_destroy(dma->dmat); return err; } static void mxge_dma_free(mxge_dma_t *dma) { bus_dmamap_unload(dma->dmat, dma->map); bus_dmamem_free(dma->dmat, dma->addr, dma->map); (void)bus_dma_tag_destroy(dma->dmat); } /* * The eeprom strings on the lanaiX have the format * SN=x\0 * MAC=x:x:x:x:x:x\0 * PC=text\0 */ static int mxge_parse_strings(mxge_softc_t *sc) { #define MXGE_NEXT_STRING(p) while(ptr < limit && *ptr++) char *ptr, *limit; int i, found_mac; ptr = sc->eeprom_strings; limit = sc->eeprom_strings + MXGE_EEPROM_STRINGS_SIZE; found_mac = 0; while (ptr < limit && *ptr != '\0') { if (memcmp(ptr, "MAC=", 4) == 0) { ptr += 1; sc->mac_addr_string = ptr; for (i = 0; i < 6; i++) { ptr += 3; if ((ptr + 2) > limit) goto abort; sc->mac_addr[i] = strtoul(ptr, NULL, 16); found_mac = 1; } } else if (memcmp(ptr, "PC=", 3) == 0) { ptr += 3; strncpy(sc->product_code_string, ptr, sizeof (sc->product_code_string) - 1); } else if (memcmp(ptr, "SN=", 3) == 0) { ptr += 3; strncpy(sc->serial_number_string, ptr, sizeof (sc->serial_number_string) - 1); } MXGE_NEXT_STRING(ptr); } if (found_mac) return 0; abort: device_printf(sc->dev, "failed to parse eeprom_strings\n"); return ENXIO; } #if defined __i386 || defined i386 || defined __i386__ || defined __x86_64__ static void mxge_enable_nvidia_ecrc(mxge_softc_t *sc) { uint32_t val; unsigned long base, off; char *va, *cfgptr; device_t pdev, mcp55; uint16_t vendor_id, device_id, word; uintptr_t bus, slot, func, ivend, idev; uint32_t *ptr32; if (!mxge_nvidia_ecrc_enable) return; pdev = device_get_parent(device_get_parent(sc->dev)); if (pdev == NULL) { device_printf(sc->dev, "could not find parent?\n"); return; } vendor_id = pci_read_config(pdev, PCIR_VENDOR, 2); device_id = pci_read_config(pdev, PCIR_DEVICE, 2); if (vendor_id != 0x10de) return; base = 0; if (device_id == 0x005d) { /* ck804, base address is magic */ base = 0xe0000000UL; } else if (device_id >= 0x0374 && device_id <= 0x378) { /* mcp55, base address stored in chipset */ mcp55 = pci_find_bsf(0, 0, 0); if (mcp55 && 0x10de == pci_read_config(mcp55, PCIR_VENDOR, 2) && 0x0369 == pci_read_config(mcp55, PCIR_DEVICE, 2)) { word = pci_read_config(mcp55, 0x90, 2); base = ((unsigned long)word & 0x7ffeU) << 25; } } if (!base) return; /* XXXX Test below is commented because it is believed that doing config read/write beyond 0xff will access the config space for the next larger function. Uncomment this and remove the hacky pmap_mapdev() way of accessing config space when FreeBSD grows support for extended pcie config space access */ #if 0 /* See if we can, by some miracle, access the extended config space */ val = pci_read_config(pdev, 0x178, 4); if (val != 0xffffffff) { val |= 0x40; pci_write_config(pdev, 0x178, val, 4); return; } #endif /* Rather than using normal pci config space writes, we must * map the Nvidia config space ourselves. This is because on * opteron/nvidia class machine the 0xe000000 mapping is * handled by the nvidia chipset, that means the internal PCI * device (the on-chip northbridge), or the amd-8131 bridge * and things behind them are not visible by this method. */ BUS_READ_IVAR(device_get_parent(pdev), pdev, PCI_IVAR_BUS, &bus); BUS_READ_IVAR(device_get_parent(pdev), pdev, PCI_IVAR_SLOT, &slot); BUS_READ_IVAR(device_get_parent(pdev), pdev, PCI_IVAR_FUNCTION, &func); BUS_READ_IVAR(device_get_parent(pdev), pdev, PCI_IVAR_VENDOR, &ivend); BUS_READ_IVAR(device_get_parent(pdev), pdev, PCI_IVAR_DEVICE, &idev); off = base + 0x00100000UL * (unsigned long)bus + 0x00001000UL * (unsigned long)(func + 8 * slot); /* map it into the kernel */ va = pmap_mapdev(trunc_page((vm_paddr_t)off), PAGE_SIZE); if (va == NULL) { device_printf(sc->dev, "pmap_kenter_temporary didn't\n"); return; } /* get a pointer to the config space mapped into the kernel */ cfgptr = va + (off & PAGE_MASK); /* make sure that we can really access it */ vendor_id = *(uint16_t *)(cfgptr + PCIR_VENDOR); device_id = *(uint16_t *)(cfgptr + PCIR_DEVICE); if (! (vendor_id == ivend && device_id == idev)) { device_printf(sc->dev, "mapping failed: 0x%x:0x%x\n", vendor_id, device_id); pmap_unmapdev((vm_offset_t)va, PAGE_SIZE); return; } ptr32 = (uint32_t*)(cfgptr + 0x178); val = *ptr32; if (val == 0xffffffff) { device_printf(sc->dev, "extended mapping failed\n"); pmap_unmapdev((vm_offset_t)va, PAGE_SIZE); return; } *ptr32 = val | 0x40; pmap_unmapdev((vm_offset_t)va, PAGE_SIZE); if (mxge_verbose) device_printf(sc->dev, "Enabled ECRC on upstream Nvidia bridge " "at %d:%d:%d\n", (int)bus, (int)slot, (int)func); return; } #else static void mxge_enable_nvidia_ecrc(mxge_softc_t *sc) { device_printf(sc->dev, "Nforce 4 chipset on non-x86/amd64!?!?!\n"); return; } #endif static int mxge_dma_test(mxge_softc_t *sc, int test_type) { mxge_cmd_t cmd; bus_addr_t dmatest_bus = sc->dmabench_dma.bus_addr; int status; uint32_t len; char *test = " "; /* Run a small DMA test. * The magic multipliers to the length tell the firmware * to do DMA read, write, or read+write tests. The * results are returned in cmd.data0. The upper 16 * bits of the return is the number of transfers completed. * The lower 16 bits is the time in 0.5us ticks that the * transfers took to complete. */ len = sc->tx_boundary; cmd.data0 = MXGE_LOWPART_TO_U32(dmatest_bus); cmd.data1 = MXGE_HIGHPART_TO_U32(dmatest_bus); cmd.data2 = len * 0x10000; status = mxge_send_cmd(sc, test_type, &cmd); if (status != 0) { test = "read"; goto abort; } sc->read_dma = ((cmd.data0>>16) * len * 2) / (cmd.data0 & 0xffff); cmd.data0 = MXGE_LOWPART_TO_U32(dmatest_bus); cmd.data1 = MXGE_HIGHPART_TO_U32(dmatest_bus); cmd.data2 = len * 0x1; status = mxge_send_cmd(sc, test_type, &cmd); if (status != 0) { test = "write"; goto abort; } sc->write_dma = ((cmd.data0>>16) * len * 2) / (cmd.data0 & 0xffff); cmd.data0 = MXGE_LOWPART_TO_U32(dmatest_bus); cmd.data1 = MXGE_HIGHPART_TO_U32(dmatest_bus); cmd.data2 = len * 0x10001; status = mxge_send_cmd(sc, test_type, &cmd); if (status != 0) { test = "read/write"; goto abort; } sc->read_write_dma = ((cmd.data0>>16) * len * 2 * 2) / (cmd.data0 & 0xffff); abort: if (status != 0 && test_type != MXGEFW_CMD_UNALIGNED_TEST) device_printf(sc->dev, "DMA %s benchmark failed: %d\n", test, status); return status; } /* * The Lanai Z8E PCI-E interface achieves higher Read-DMA throughput * when the PCI-E Completion packets are aligned on an 8-byte * boundary. Some PCI-E chip sets always align Completion packets; on * the ones that do not, the alignment can be enforced by enabling * ECRC generation (if supported). * * When PCI-E Completion packets are not aligned, it is actually more * efficient to limit Read-DMA transactions to 2KB, rather than 4KB. * * If the driver can neither enable ECRC nor verify that it has * already been enabled, then it must use a firmware image which works * around unaligned completion packets (ethp_z8e.dat), and it should * also ensure that it never gives the device a Read-DMA which is * larger than 2KB by setting the tx_boundary to 2KB. If ECRC is * enabled, then the driver should use the aligned (eth_z8e.dat) * firmware image, and set tx_boundary to 4KB. */ static int mxge_firmware_probe(mxge_softc_t *sc) { device_t dev = sc->dev; int reg, status; uint16_t pectl; sc->tx_boundary = 4096; /* * Verify the max read request size was set to 4KB * before trying the test with 4KB. */ if (pci_find_extcap(dev, PCIY_EXPRESS, ®) == 0) { pectl = pci_read_config(dev, reg + 0x8, 2); if ((pectl & (5 << 12)) != (5 << 12)) { device_printf(dev, "Max Read Req. size != 4k (0x%x\n", pectl); sc->tx_boundary = 2048; } } /* * load the optimized firmware (which assumes aligned PCIe * completions) in order to see if it works on this host. */ sc->fw_name = mxge_fw_aligned; status = mxge_load_firmware(sc, 1); if (status != 0) { return status; } /* * Enable ECRC if possible */ mxge_enable_nvidia_ecrc(sc); /* * Run a DMA test which watches for unaligned completions and * aborts on the first one seen. */ status = mxge_dma_test(sc, MXGEFW_CMD_UNALIGNED_TEST); if (status == 0) return 0; /* keep the aligned firmware */ if (status != E2BIG) device_printf(dev, "DMA test failed: %d\n", status); if (status == ENOSYS) device_printf(dev, "Falling back to ethp! " "Please install up to date fw\n"); return status; } static int mxge_select_firmware(mxge_softc_t *sc) { int aligned = 0; int force_firmware = mxge_force_firmware; if (sc->throttle) force_firmware = sc->throttle; if (force_firmware != 0) { if (force_firmware == 1) aligned = 1; else aligned = 0; if (mxge_verbose) device_printf(sc->dev, "Assuming %s completions (forced)\n", aligned ? "aligned" : "unaligned"); goto abort; } /* if the PCIe link width is 4 or less, we can use the aligned firmware and skip any checks */ if (sc->link_width != 0 && sc->link_width <= 4) { device_printf(sc->dev, "PCIe x%d Link, expect reduced performance\n", sc->link_width); aligned = 1; goto abort; } if (0 == mxge_firmware_probe(sc)) return 0; abort: if (aligned) { sc->fw_name = mxge_fw_aligned; sc->tx_boundary = 4096; } else { sc->fw_name = mxge_fw_unaligned; sc->tx_boundary = 2048; } return (mxge_load_firmware(sc, 0)); } union qualhack { const char *ro_char; char *rw_char; }; static int mxge_validate_firmware(mxge_softc_t *sc, const mcp_gen_header_t *hdr) { if (be32toh(hdr->mcp_type) != MCP_TYPE_ETH) { device_printf(sc->dev, "Bad firmware type: 0x%x\n", be32toh(hdr->mcp_type)); return EIO; } /* save firmware version for sysctl */ strncpy(sc->fw_version, hdr->version, sizeof (sc->fw_version)); if (mxge_verbose) device_printf(sc->dev, "firmware id: %s\n", hdr->version); sscanf(sc->fw_version, "%d.%d.%d", &sc->fw_ver_major, &sc->fw_ver_minor, &sc->fw_ver_tiny); if (!(sc->fw_ver_major == MXGEFW_VERSION_MAJOR && sc->fw_ver_minor == MXGEFW_VERSION_MINOR)) { device_printf(sc->dev, "Found firmware version %s\n", sc->fw_version); device_printf(sc->dev, "Driver needs %d.%d\n", MXGEFW_VERSION_MAJOR, MXGEFW_VERSION_MINOR); return EINVAL; } return 0; } static void * z_alloc(void *nil, u_int items, u_int size) { void *ptr; ptr = malloc(items * size, M_TEMP, M_NOWAIT); return ptr; } static void z_free(void *nil, void *ptr) { free(ptr, M_TEMP); } static int mxge_load_firmware_helper(mxge_softc_t *sc, uint32_t *limit) { z_stream zs; char *inflate_buffer; const struct firmware *fw; const mcp_gen_header_t *hdr; unsigned hdr_offset; int status; unsigned int i; char dummy; size_t fw_len; fw = firmware_get(sc->fw_name); if (fw == NULL) { device_printf(sc->dev, "Could not find firmware image %s\n", sc->fw_name); return ENOENT; } /* setup zlib and decompress f/w */ bzero(&zs, sizeof (zs)); zs.zalloc = z_alloc; zs.zfree = z_free; status = inflateInit(&zs); if (status != Z_OK) { status = EIO; goto abort_with_fw; } /* the uncompressed size is stored as the firmware version, which would otherwise go unused */ fw_len = (size_t) fw->version; inflate_buffer = malloc(fw_len, M_TEMP, M_NOWAIT); if (inflate_buffer == NULL) goto abort_with_zs; zs.avail_in = fw->datasize; zs.next_in = __DECONST(char *, fw->data); zs.avail_out = fw_len; zs.next_out = inflate_buffer; status = inflate(&zs, Z_FINISH); if (status != Z_STREAM_END) { device_printf(sc->dev, "zlib %d\n", status); status = EIO; goto abort_with_buffer; } /* check id */ hdr_offset = htobe32(*(const uint32_t *) (inflate_buffer + MCP_HEADER_PTR_OFFSET)); if ((hdr_offset & 3) || hdr_offset + sizeof(*hdr) > fw_len) { device_printf(sc->dev, "Bad firmware file"); status = EIO; goto abort_with_buffer; } hdr = (const void*)(inflate_buffer + hdr_offset); status = mxge_validate_firmware(sc, hdr); if (status != 0) goto abort_with_buffer; /* Copy the inflated firmware to NIC SRAM. */ for (i = 0; i < fw_len; i += 256) { mxge_pio_copy(sc->sram + MXGE_FW_OFFSET + i, inflate_buffer + i, min(256U, (unsigned)(fw_len - i))); wmb(); dummy = *sc->sram; wmb(); } *limit = fw_len; status = 0; abort_with_buffer: free(inflate_buffer, M_TEMP); abort_with_zs: inflateEnd(&zs); abort_with_fw: firmware_put(fw, FIRMWARE_UNLOAD); return status; } /* * Enable or disable periodic RDMAs from the host to make certain * chipsets resend dropped PCIe messages */ static void mxge_dummy_rdma(mxge_softc_t *sc, int enable) { char buf_bytes[72]; volatile uint32_t *confirm; volatile char *submit; uint32_t *buf, dma_low, dma_high; int i; buf = (uint32_t *)((unsigned long)(buf_bytes + 7) & ~7UL); /* clear confirmation addr */ confirm = (volatile uint32_t *)sc->cmd; *confirm = 0; wmb(); /* send an rdma command to the PCIe engine, and wait for the response in the confirmation address. The firmware should write a -1 there to indicate it is alive and well */ dma_low = MXGE_LOWPART_TO_U32(sc->cmd_dma.bus_addr); dma_high = MXGE_HIGHPART_TO_U32(sc->cmd_dma.bus_addr); buf[0] = htobe32(dma_high); /* confirm addr MSW */ buf[1] = htobe32(dma_low); /* confirm addr LSW */ buf[2] = htobe32(0xffffffff); /* confirm data */ dma_low = MXGE_LOWPART_TO_U32(sc->zeropad_dma.bus_addr); dma_high = MXGE_HIGHPART_TO_U32(sc->zeropad_dma.bus_addr); buf[3] = htobe32(dma_high); /* dummy addr MSW */ buf[4] = htobe32(dma_low); /* dummy addr LSW */ buf[5] = htobe32(enable); /* enable? */ submit = (volatile char *)(sc->sram + MXGEFW_BOOT_DUMMY_RDMA); mxge_pio_copy(submit, buf, 64); wmb(); DELAY(1000); wmb(); i = 0; while (*confirm != 0xffffffff && i < 20) { DELAY(1000); i++; } if (*confirm != 0xffffffff) { device_printf(sc->dev, "dummy rdma %s failed (%p = 0x%x)", (enable ? "enable" : "disable"), confirm, *confirm); } return; } static int mxge_send_cmd(mxge_softc_t *sc, uint32_t cmd, mxge_cmd_t *data) { mcp_cmd_t *buf; char buf_bytes[sizeof(*buf) + 8]; volatile mcp_cmd_response_t *response = sc->cmd; volatile char *cmd_addr = sc->sram + MXGEFW_ETH_CMD; uint32_t dma_low, dma_high; int err, sleep_total = 0; /* ensure buf is aligned to 8 bytes */ buf = (mcp_cmd_t *)((unsigned long)(buf_bytes + 7) & ~7UL); buf->data0 = htobe32(data->data0); buf->data1 = htobe32(data->data1); buf->data2 = htobe32(data->data2); buf->cmd = htobe32(cmd); dma_low = MXGE_LOWPART_TO_U32(sc->cmd_dma.bus_addr); dma_high = MXGE_HIGHPART_TO_U32(sc->cmd_dma.bus_addr); buf->response_addr.low = htobe32(dma_low); buf->response_addr.high = htobe32(dma_high); mtx_lock(&sc->cmd_mtx); response->result = 0xffffffff; wmb(); mxge_pio_copy((volatile void *)cmd_addr, buf, sizeof (*buf)); /* wait up to 20ms */ err = EAGAIN; for (sleep_total = 0; sleep_total < 20; sleep_total++) { bus_dmamap_sync(sc->cmd_dma.dmat, sc->cmd_dma.map, BUS_DMASYNC_POSTREAD); wmb(); switch (be32toh(response->result)) { case 0: data->data0 = be32toh(response->data); err = 0; break; case 0xffffffff: DELAY(1000); break; case MXGEFW_CMD_UNKNOWN: err = ENOSYS; break; case MXGEFW_CMD_ERROR_UNALIGNED: err = E2BIG; break; case MXGEFW_CMD_ERROR_BUSY: err = EBUSY; break; + case MXGEFW_CMD_ERROR_I2C_ABSENT: + err = ENXIO; + break; default: device_printf(sc->dev, "mxge: command %d " "failed, result = %d\n", cmd, be32toh(response->result)); err = ENXIO; break; } if (err != EAGAIN) break; } if (err == EAGAIN) device_printf(sc->dev, "mxge: command %d timed out" "result = %d\n", cmd, be32toh(response->result)); mtx_unlock(&sc->cmd_mtx); return err; } static int mxge_adopt_running_firmware(mxge_softc_t *sc) { struct mcp_gen_header *hdr; const size_t bytes = sizeof (struct mcp_gen_header); size_t hdr_offset; int status; /* find running firmware header */ hdr_offset = htobe32(*(volatile uint32_t *) (sc->sram + MCP_HEADER_PTR_OFFSET)); if ((hdr_offset & 3) || hdr_offset + sizeof(*hdr) > sc->sram_size) { device_printf(sc->dev, "Running firmware has bad header offset (%d)\n", (int)hdr_offset); return EIO; } /* copy header of running firmware from SRAM to host memory to * validate firmware */ hdr = malloc(bytes, M_DEVBUF, M_NOWAIT); if (hdr == NULL) { device_printf(sc->dev, "could not malloc firmware hdr\n"); return ENOMEM; } bus_space_read_region_1(rman_get_bustag(sc->mem_res), rman_get_bushandle(sc->mem_res), hdr_offset, (char *)hdr, bytes); status = mxge_validate_firmware(sc, hdr); free(hdr, M_DEVBUF); /* * check to see if adopted firmware has bug where adopting * it will cause broadcasts to be filtered unless the NIC * is kept in ALLMULTI mode */ if (sc->fw_ver_major == 1 && sc->fw_ver_minor == 4 && sc->fw_ver_tiny >= 4 && sc->fw_ver_tiny <= 11) { sc->adopted_rx_filter_bug = 1; device_printf(sc->dev, "Adopting fw %d.%d.%d: " "working around rx filter bug\n", sc->fw_ver_major, sc->fw_ver_minor, sc->fw_ver_tiny); } return status; } static int mxge_load_firmware(mxge_softc_t *sc, int adopt) { volatile uint32_t *confirm; volatile char *submit; char buf_bytes[72]; uint32_t *buf, size, dma_low, dma_high; int status, i; buf = (uint32_t *)((unsigned long)(buf_bytes + 7) & ~7UL); size = sc->sram_size; status = mxge_load_firmware_helper(sc, &size); if (status) { if (!adopt) return status; /* Try to use the currently running firmware, if it is new enough */ status = mxge_adopt_running_firmware(sc); if (status) { device_printf(sc->dev, "failed to adopt running firmware\n"); return status; } device_printf(sc->dev, "Successfully adopted running firmware\n"); if (sc->tx_boundary == 4096) { device_printf(sc->dev, "Using firmware currently running on NIC" ". For optimal\n"); device_printf(sc->dev, "performance consider loading optimized " "firmware\n"); } sc->fw_name = mxge_fw_unaligned; sc->tx_boundary = 2048; return 0; } /* clear confirmation addr */ confirm = (volatile uint32_t *)sc->cmd; *confirm = 0; wmb(); /* send a reload command to the bootstrap MCP, and wait for the response in the confirmation address. The firmware should write a -1 there to indicate it is alive and well */ dma_low = MXGE_LOWPART_TO_U32(sc->cmd_dma.bus_addr); dma_high = MXGE_HIGHPART_TO_U32(sc->cmd_dma.bus_addr); buf[0] = htobe32(dma_high); /* confirm addr MSW */ buf[1] = htobe32(dma_low); /* confirm addr LSW */ buf[2] = htobe32(0xffffffff); /* confirm data */ /* FIX: All newest firmware should un-protect the bottom of the sram before handoff. However, the very first interfaces do not. Therefore the handoff copy must skip the first 8 bytes */ /* where the code starts*/ buf[3] = htobe32(MXGE_FW_OFFSET + 8); buf[4] = htobe32(size - 8); /* length of code */ buf[5] = htobe32(8); /* where to copy to */ buf[6] = htobe32(0); /* where to jump to */ submit = (volatile char *)(sc->sram + MXGEFW_BOOT_HANDOFF); mxge_pio_copy(submit, buf, 64); wmb(); DELAY(1000); wmb(); i = 0; while (*confirm != 0xffffffff && i < 20) { DELAY(1000*10); i++; bus_dmamap_sync(sc->cmd_dma.dmat, sc->cmd_dma.map, BUS_DMASYNC_POSTREAD); } if (*confirm != 0xffffffff) { device_printf(sc->dev,"handoff failed (%p = 0x%x)", confirm, *confirm); return ENXIO; } return 0; } static int mxge_update_mac_address(mxge_softc_t *sc) { mxge_cmd_t cmd; uint8_t *addr = sc->mac_addr; int status; cmd.data0 = ((addr[0] << 24) | (addr[1] << 16) | (addr[2] << 8) | addr[3]); cmd.data1 = ((addr[4] << 8) | (addr[5])); status = mxge_send_cmd(sc, MXGEFW_SET_MAC_ADDRESS, &cmd); return status; } static int mxge_change_pause(mxge_softc_t *sc, int pause) { mxge_cmd_t cmd; int status; if (pause) status = mxge_send_cmd(sc, MXGEFW_ENABLE_FLOW_CONTROL, &cmd); else status = mxge_send_cmd(sc, MXGEFW_DISABLE_FLOW_CONTROL, &cmd); if (status) { device_printf(sc->dev, "Failed to set flow control mode\n"); return ENXIO; } sc->pause = pause; return 0; } static void mxge_change_promisc(mxge_softc_t *sc, int promisc) { mxge_cmd_t cmd; int status; if (mxge_always_promisc) promisc = 1; if (promisc) status = mxge_send_cmd(sc, MXGEFW_ENABLE_PROMISC, &cmd); else status = mxge_send_cmd(sc, MXGEFW_DISABLE_PROMISC, &cmd); if (status) { device_printf(sc->dev, "Failed to set promisc mode\n"); } } static void mxge_set_multicast_list(mxge_softc_t *sc) { mxge_cmd_t cmd; struct ifmultiaddr *ifma; struct ifnet *ifp = sc->ifp; int err; /* This firmware is known to not support multicast */ if (!sc->fw_multicast_support) return; /* Disable multicast filtering while we play with the lists*/ err = mxge_send_cmd(sc, MXGEFW_ENABLE_ALLMULTI, &cmd); if (err != 0) { device_printf(sc->dev, "Failed MXGEFW_ENABLE_ALLMULTI," " error status: %d\n", err); return; } if (sc->adopted_rx_filter_bug) return; if (ifp->if_flags & IFF_ALLMULTI) /* request to disable multicast filtering, so quit here */ return; /* Flush all the filters */ err = mxge_send_cmd(sc, MXGEFW_LEAVE_ALL_MULTICAST_GROUPS, &cmd); if (err != 0) { device_printf(sc->dev, "Failed MXGEFW_LEAVE_ALL_MULTICAST_GROUPS" ", error status: %d\n", err); return; } /* Walk the multicast list, and add each address */ if_maddr_rlock(ifp); TAILQ_FOREACH(ifma, &ifp->if_multiaddrs, ifma_link) { if (ifma->ifma_addr->sa_family != AF_LINK) continue; bcopy(LLADDR((struct sockaddr_dl *)ifma->ifma_addr), &cmd.data0, 4); bcopy(LLADDR((struct sockaddr_dl *)ifma->ifma_addr) + 4, &cmd.data1, 2); cmd.data0 = htonl(cmd.data0); cmd.data1 = htonl(cmd.data1); err = mxge_send_cmd(sc, MXGEFW_JOIN_MULTICAST_GROUP, &cmd); if (err != 0) { device_printf(sc->dev, "Failed " "MXGEFW_JOIN_MULTICAST_GROUP, error status:" "%d\t", err); /* abort, leaving multicast filtering off */ if_maddr_runlock(ifp); return; } } if_maddr_runlock(ifp); /* Enable multicast filtering */ err = mxge_send_cmd(sc, MXGEFW_DISABLE_ALLMULTI, &cmd); if (err != 0) { device_printf(sc->dev, "Failed MXGEFW_DISABLE_ALLMULTI" ", error status: %d\n", err); } } static int mxge_max_mtu(mxge_softc_t *sc) { mxge_cmd_t cmd; int status; if (MJUMPAGESIZE - MXGEFW_PAD > MXGEFW_MAX_MTU) return MXGEFW_MAX_MTU - MXGEFW_PAD; /* try to set nbufs to see if it we can use virtually contiguous jumbos */ cmd.data0 = 0; status = mxge_send_cmd(sc, MXGEFW_CMD_ALWAYS_USE_N_BIG_BUFFERS, &cmd); if (status == 0) return MXGEFW_MAX_MTU - MXGEFW_PAD; /* otherwise, we're limited to MJUMPAGESIZE */ return MJUMPAGESIZE - MXGEFW_PAD; } static int mxge_reset(mxge_softc_t *sc, int interrupts_setup) { struct mxge_slice_state *ss; mxge_rx_done_t *rx_done; volatile uint32_t *irq_claim; mxge_cmd_t cmd; int slice, status; /* try to send a reset command to the card to see if it is alive */ memset(&cmd, 0, sizeof (cmd)); status = mxge_send_cmd(sc, MXGEFW_CMD_RESET, &cmd); if (status != 0) { device_printf(sc->dev, "failed reset\n"); return ENXIO; } mxge_dummy_rdma(sc, 1); /* set the intrq size */ cmd.data0 = sc->rx_ring_size; status = mxge_send_cmd(sc, MXGEFW_CMD_SET_INTRQ_SIZE, &cmd); /* * Even though we already know how many slices are supported * via mxge_slice_probe(), MXGEFW_CMD_GET_MAX_RSS_QUEUES * has magic side effects, and must be called after a reset. * It must be called prior to calling any RSS related cmds, * including assigning an interrupt queue for anything but * slice 0. It must also be called *after* * MXGEFW_CMD_SET_INTRQ_SIZE, since the intrq size is used by * the firmware to compute offsets. */ if (sc->num_slices > 1) { /* ask the maximum number of slices it supports */ status = mxge_send_cmd(sc, MXGEFW_CMD_GET_MAX_RSS_QUEUES, &cmd); if (status != 0) { device_printf(sc->dev, "failed to get number of slices\n"); return status; } /* * MXGEFW_CMD_ENABLE_RSS_QUEUES must be called prior * to setting up the interrupt queue DMA */ cmd.data0 = sc->num_slices; cmd.data1 = MXGEFW_SLICE_INTR_MODE_ONE_PER_SLICE; #ifdef IFNET_BUF_RING cmd.data1 |= MXGEFW_SLICE_ENABLE_MULTIPLE_TX_QUEUES; #endif status = mxge_send_cmd(sc, MXGEFW_CMD_ENABLE_RSS_QUEUES, &cmd); if (status != 0) { device_printf(sc->dev, "failed to set number of slices\n"); return status; } } if (interrupts_setup) { /* Now exchange information about interrupts */ for (slice = 0; slice < sc->num_slices; slice++) { rx_done = &sc->ss[slice].rx_done; memset(rx_done->entry, 0, sc->rx_ring_size); cmd.data0 = MXGE_LOWPART_TO_U32(rx_done->dma.bus_addr); cmd.data1 = MXGE_HIGHPART_TO_U32(rx_done->dma.bus_addr); cmd.data2 = slice; status |= mxge_send_cmd(sc, MXGEFW_CMD_SET_INTRQ_DMA, &cmd); } } status |= mxge_send_cmd(sc, MXGEFW_CMD_GET_INTR_COAL_DELAY_OFFSET, &cmd); sc->intr_coal_delay_ptr = (volatile uint32_t *)(sc->sram + cmd.data0); status |= mxge_send_cmd(sc, MXGEFW_CMD_GET_IRQ_ACK_OFFSET, &cmd); irq_claim = (volatile uint32_t *)(sc->sram + cmd.data0); status |= mxge_send_cmd(sc, MXGEFW_CMD_GET_IRQ_DEASSERT_OFFSET, &cmd); sc->irq_deassert = (volatile uint32_t *)(sc->sram + cmd.data0); if (status != 0) { device_printf(sc->dev, "failed set interrupt parameters\n"); return status; } *sc->intr_coal_delay_ptr = htobe32(sc->intr_coal_delay); /* run a DMA benchmark */ (void) mxge_dma_test(sc, MXGEFW_DMA_TEST); for (slice = 0; slice < sc->num_slices; slice++) { ss = &sc->ss[slice]; ss->irq_claim = irq_claim + (2 * slice); /* reset mcp/driver shared state back to 0 */ ss->rx_done.idx = 0; ss->rx_done.cnt = 0; ss->tx.req = 0; ss->tx.done = 0; ss->tx.pkt_done = 0; ss->tx.queue_active = 0; ss->tx.activate = 0; ss->tx.deactivate = 0; ss->tx.wake = 0; ss->tx.defrag = 0; ss->tx.stall = 0; ss->rx_big.cnt = 0; ss->rx_small.cnt = 0; ss->lro_bad_csum = 0; ss->lro_queued = 0; ss->lro_flushed = 0; if (ss->fw_stats != NULL) { bzero(ss->fw_stats, sizeof *ss->fw_stats); } } sc->rdma_tags_available = 15; status = mxge_update_mac_address(sc); mxge_change_promisc(sc, sc->ifp->if_flags & IFF_PROMISC); mxge_change_pause(sc, sc->pause); mxge_set_multicast_list(sc); if (sc->throttle) { cmd.data0 = sc->throttle; if (mxge_send_cmd(sc, MXGEFW_CMD_SET_THROTTLE_FACTOR, &cmd)) { device_printf(sc->dev, "can't enable throttle\n"); } } return status; } static int mxge_change_throttle(SYSCTL_HANDLER_ARGS) { mxge_cmd_t cmd; mxge_softc_t *sc; int err; unsigned int throttle; sc = arg1; throttle = sc->throttle; err = sysctl_handle_int(oidp, &throttle, arg2, req); if (err != 0) { return err; } if (throttle == sc->throttle) return 0; if (throttle < MXGE_MIN_THROTTLE || throttle > MXGE_MAX_THROTTLE) return EINVAL; mtx_lock(&sc->driver_mtx); cmd.data0 = throttle; err = mxge_send_cmd(sc, MXGEFW_CMD_SET_THROTTLE_FACTOR, &cmd); if (err == 0) sc->throttle = throttle; mtx_unlock(&sc->driver_mtx); return err; } static int mxge_change_intr_coal(SYSCTL_HANDLER_ARGS) { mxge_softc_t *sc; unsigned int intr_coal_delay; int err; sc = arg1; intr_coal_delay = sc->intr_coal_delay; err = sysctl_handle_int(oidp, &intr_coal_delay, arg2, req); if (err != 0) { return err; } if (intr_coal_delay == sc->intr_coal_delay) return 0; if (intr_coal_delay == 0 || intr_coal_delay > 1000*1000) return EINVAL; mtx_lock(&sc->driver_mtx); *sc->intr_coal_delay_ptr = htobe32(intr_coal_delay); sc->intr_coal_delay = intr_coal_delay; mtx_unlock(&sc->driver_mtx); return err; } static int mxge_change_flow_control(SYSCTL_HANDLER_ARGS) { mxge_softc_t *sc; unsigned int enabled; int err; sc = arg1; enabled = sc->pause; err = sysctl_handle_int(oidp, &enabled, arg2, req); if (err != 0) { return err; } if (enabled == sc->pause) return 0; mtx_lock(&sc->driver_mtx); err = mxge_change_pause(sc, enabled); mtx_unlock(&sc->driver_mtx); return err; } static int mxge_change_lro_locked(mxge_softc_t *sc, int lro_cnt) { struct ifnet *ifp; int err = 0; ifp = sc->ifp; if (lro_cnt == 0) ifp->if_capenable &= ~IFCAP_LRO; else ifp->if_capenable |= IFCAP_LRO; sc->lro_cnt = lro_cnt; if (ifp->if_drv_flags & IFF_DRV_RUNNING) { mxge_close(sc, 0); err = mxge_open(sc); } return err; } static int mxge_change_lro(SYSCTL_HANDLER_ARGS) { mxge_softc_t *sc; unsigned int lro_cnt; int err; sc = arg1; lro_cnt = sc->lro_cnt; err = sysctl_handle_int(oidp, &lro_cnt, arg2, req); if (err != 0) return err; if (lro_cnt == sc->lro_cnt) return 0; if (lro_cnt > 128) return EINVAL; mtx_lock(&sc->driver_mtx); err = mxge_change_lro_locked(sc, lro_cnt); mtx_unlock(&sc->driver_mtx); return err; } static int mxge_handle_be32(SYSCTL_HANDLER_ARGS) { int err; if (arg1 == NULL) return EFAULT; arg2 = be32toh(*(int *)arg1); arg1 = NULL; err = sysctl_handle_int(oidp, arg1, arg2, req); return err; } static void mxge_rem_sysctls(mxge_softc_t *sc) { struct mxge_slice_state *ss; int slice; if (sc->slice_sysctl_tree == NULL) return; for (slice = 0; slice < sc->num_slices; slice++) { ss = &sc->ss[slice]; if (ss == NULL || ss->sysctl_tree == NULL) continue; sysctl_ctx_free(&ss->sysctl_ctx); ss->sysctl_tree = NULL; } sysctl_ctx_free(&sc->slice_sysctl_ctx); sc->slice_sysctl_tree = NULL; } static void mxge_add_sysctls(mxge_softc_t *sc) { struct sysctl_ctx_list *ctx; struct sysctl_oid_list *children; mcp_irq_data_t *fw; struct mxge_slice_state *ss; int slice; char slice_num[8]; ctx = device_get_sysctl_ctx(sc->dev); children = SYSCTL_CHILDREN(device_get_sysctl_tree(sc->dev)); fw = sc->ss[0].fw_stats; /* random information */ SYSCTL_ADD_STRING(ctx, children, OID_AUTO, "firmware_version", CTLFLAG_RD, &sc->fw_version, 0, "firmware version"); SYSCTL_ADD_STRING(ctx, children, OID_AUTO, "serial_number", CTLFLAG_RD, &sc->serial_number_string, 0, "serial number"); SYSCTL_ADD_STRING(ctx, children, OID_AUTO, "product_code", CTLFLAG_RD, &sc->product_code_string, 0, "product_code"); SYSCTL_ADD_INT(ctx, children, OID_AUTO, "pcie_link_width", CTLFLAG_RD, &sc->link_width, 0, "tx_boundary"); SYSCTL_ADD_INT(ctx, children, OID_AUTO, "tx_boundary", CTLFLAG_RD, &sc->tx_boundary, 0, "tx_boundary"); SYSCTL_ADD_INT(ctx, children, OID_AUTO, "write_combine", CTLFLAG_RD, &sc->wc, 0, "write combining PIO?"); SYSCTL_ADD_INT(ctx, children, OID_AUTO, "read_dma_MBs", CTLFLAG_RD, &sc->read_dma, 0, "DMA Read speed in MB/s"); SYSCTL_ADD_INT(ctx, children, OID_AUTO, "write_dma_MBs", CTLFLAG_RD, &sc->write_dma, 0, "DMA Write speed in MB/s"); SYSCTL_ADD_INT(ctx, children, OID_AUTO, "read_write_dma_MBs", CTLFLAG_RD, &sc->read_write_dma, 0, "DMA concurrent Read/Write speed in MB/s"); SYSCTL_ADD_INT(ctx, children, OID_AUTO, "watchdog_resets", CTLFLAG_RD, &sc->watchdog_resets, 0, "Number of times NIC was reset"); /* performance related tunables */ SYSCTL_ADD_PROC(ctx, children, OID_AUTO, "intr_coal_delay", CTLTYPE_INT|CTLFLAG_RW, sc, 0, mxge_change_intr_coal, "I", "interrupt coalescing delay in usecs"); SYSCTL_ADD_PROC(ctx, children, OID_AUTO, "throttle", CTLTYPE_INT|CTLFLAG_RW, sc, 0, mxge_change_throttle, "I", "transmit throttling"); SYSCTL_ADD_PROC(ctx, children, OID_AUTO, "flow_control_enabled", CTLTYPE_INT|CTLFLAG_RW, sc, 0, mxge_change_flow_control, "I", "interrupt coalescing delay in usecs"); SYSCTL_ADD_INT(ctx, children, OID_AUTO, "deassert_wait", CTLFLAG_RW, &mxge_deassert_wait, 0, "Wait for IRQ line to go low in ihandler"); /* stats block from firmware is in network byte order. Need to swap it */ SYSCTL_ADD_PROC(ctx, children, OID_AUTO, "link_up", CTLTYPE_INT|CTLFLAG_RD, &fw->link_up, 0, mxge_handle_be32, "I", "link up"); SYSCTL_ADD_PROC(ctx, children, OID_AUTO, "rdma_tags_available", CTLTYPE_INT|CTLFLAG_RD, &fw->rdma_tags_available, 0, mxge_handle_be32, "I", "rdma_tags_available"); SYSCTL_ADD_PROC(ctx, children, OID_AUTO, "dropped_bad_crc32", CTLTYPE_INT|CTLFLAG_RD, &fw->dropped_bad_crc32, 0, mxge_handle_be32, "I", "dropped_bad_crc32"); SYSCTL_ADD_PROC(ctx, children, OID_AUTO, "dropped_bad_phy", CTLTYPE_INT|CTLFLAG_RD, &fw->dropped_bad_phy, 0, mxge_handle_be32, "I", "dropped_bad_phy"); SYSCTL_ADD_PROC(ctx, children, OID_AUTO, "dropped_link_error_or_filtered", CTLTYPE_INT|CTLFLAG_RD, &fw->dropped_link_error_or_filtered, 0, mxge_handle_be32, "I", "dropped_link_error_or_filtered"); SYSCTL_ADD_PROC(ctx, children, OID_AUTO, "dropped_link_overflow", CTLTYPE_INT|CTLFLAG_RD, &fw->dropped_link_overflow, 0, mxge_handle_be32, "I", "dropped_link_overflow"); SYSCTL_ADD_PROC(ctx, children, OID_AUTO, "dropped_multicast_filtered", CTLTYPE_INT|CTLFLAG_RD, &fw->dropped_multicast_filtered, 0, mxge_handle_be32, "I", "dropped_multicast_filtered"); SYSCTL_ADD_PROC(ctx, children, OID_AUTO, "dropped_no_big_buffer", CTLTYPE_INT|CTLFLAG_RD, &fw->dropped_no_big_buffer, 0, mxge_handle_be32, "I", "dropped_no_big_buffer"); SYSCTL_ADD_PROC(ctx, children, OID_AUTO, "dropped_no_small_buffer", CTLTYPE_INT|CTLFLAG_RD, &fw->dropped_no_small_buffer, 0, mxge_handle_be32, "I", "dropped_no_small_buffer"); SYSCTL_ADD_PROC(ctx, children, OID_AUTO, "dropped_overrun", CTLTYPE_INT|CTLFLAG_RD, &fw->dropped_overrun, 0, mxge_handle_be32, "I", "dropped_overrun"); SYSCTL_ADD_PROC(ctx, children, OID_AUTO, "dropped_pause", CTLTYPE_INT|CTLFLAG_RD, &fw->dropped_pause, 0, mxge_handle_be32, "I", "dropped_pause"); SYSCTL_ADD_PROC(ctx, children, OID_AUTO, "dropped_runt", CTLTYPE_INT|CTLFLAG_RD, &fw->dropped_runt, 0, mxge_handle_be32, "I", "dropped_runt"); SYSCTL_ADD_PROC(ctx, children, OID_AUTO, "dropped_unicast_filtered", CTLTYPE_INT|CTLFLAG_RD, &fw->dropped_unicast_filtered, 0, mxge_handle_be32, "I", "dropped_unicast_filtered"); /* verbose printing? */ SYSCTL_ADD_INT(ctx, children, OID_AUTO, "verbose", CTLFLAG_RW, &mxge_verbose, 0, "verbose printing"); /* lro */ SYSCTL_ADD_PROC(ctx, children, OID_AUTO, "lro_cnt", CTLTYPE_INT|CTLFLAG_RW, sc, 0, mxge_change_lro, "I", "number of lro merge queues"); /* add counters exported for debugging from all slices */ sysctl_ctx_init(&sc->slice_sysctl_ctx); sc->slice_sysctl_tree = SYSCTL_ADD_NODE(&sc->slice_sysctl_ctx, children, OID_AUTO, "slice", CTLFLAG_RD, 0, ""); for (slice = 0; slice < sc->num_slices; slice++) { ss = &sc->ss[slice]; sysctl_ctx_init(&ss->sysctl_ctx); ctx = &ss->sysctl_ctx; children = SYSCTL_CHILDREN(sc->slice_sysctl_tree); sprintf(slice_num, "%d", slice); ss->sysctl_tree = SYSCTL_ADD_NODE(ctx, children, OID_AUTO, slice_num, CTLFLAG_RD, 0, ""); children = SYSCTL_CHILDREN(ss->sysctl_tree); SYSCTL_ADD_INT(ctx, children, OID_AUTO, "rx_small_cnt", CTLFLAG_RD, &ss->rx_small.cnt, 0, "rx_small_cnt"); SYSCTL_ADD_INT(ctx, children, OID_AUTO, "rx_big_cnt", CTLFLAG_RD, &ss->rx_big.cnt, 0, "rx_small_cnt"); SYSCTL_ADD_INT(ctx, children, OID_AUTO, "lro_flushed", CTLFLAG_RD, &ss->lro_flushed, 0, "number of lro merge queues flushed"); SYSCTL_ADD_INT(ctx, children, OID_AUTO, "lro_queued", CTLFLAG_RD, &ss->lro_queued, 0, "number of frames appended to lro merge" "queues"); #ifndef IFNET_BUF_RING /* only transmit from slice 0 for now */ if (slice > 0) continue; #endif SYSCTL_ADD_INT(ctx, children, OID_AUTO, "tx_req", CTLFLAG_RD, &ss->tx.req, 0, "tx_req"); SYSCTL_ADD_INT(ctx, children, OID_AUTO, "tx_done", CTLFLAG_RD, &ss->tx.done, 0, "tx_done"); SYSCTL_ADD_INT(ctx, children, OID_AUTO, "tx_pkt_done", CTLFLAG_RD, &ss->tx.pkt_done, 0, "tx_done"); SYSCTL_ADD_INT(ctx, children, OID_AUTO, "tx_stall", CTLFLAG_RD, &ss->tx.stall, 0, "tx_stall"); SYSCTL_ADD_INT(ctx, children, OID_AUTO, "tx_wake", CTLFLAG_RD, &ss->tx.wake, 0, "tx_wake"); SYSCTL_ADD_INT(ctx, children, OID_AUTO, "tx_defrag", CTLFLAG_RD, &ss->tx.defrag, 0, "tx_defrag"); SYSCTL_ADD_INT(ctx, children, OID_AUTO, "tx_queue_active", CTLFLAG_RD, &ss->tx.queue_active, 0, "tx_queue_active"); SYSCTL_ADD_INT(ctx, children, OID_AUTO, "tx_activate", CTLFLAG_RD, &ss->tx.activate, 0, "tx_activate"); SYSCTL_ADD_INT(ctx, children, OID_AUTO, "tx_deactivate", CTLFLAG_RD, &ss->tx.deactivate, 0, "tx_deactivate"); } } /* copy an array of mcp_kreq_ether_send_t's to the mcp. Copy backwards one at a time and handle ring wraps */ static inline void mxge_submit_req_backwards(mxge_tx_ring_t *tx, mcp_kreq_ether_send_t *src, int cnt) { int idx, starting_slot; starting_slot = tx->req; while (cnt > 1) { cnt--; idx = (starting_slot + cnt) & tx->mask; mxge_pio_copy(&tx->lanai[idx], &src[cnt], sizeof(*src)); wmb(); } } /* * copy an array of mcp_kreq_ether_send_t's to the mcp. Copy * at most 32 bytes at a time, so as to avoid involving the software * pio handler in the nic. We re-write the first segment's flags * to mark them valid only after writing the entire chain */ static inline void mxge_submit_req(mxge_tx_ring_t *tx, mcp_kreq_ether_send_t *src, int cnt) { int idx, i; uint32_t *src_ints; volatile uint32_t *dst_ints; mcp_kreq_ether_send_t *srcp; volatile mcp_kreq_ether_send_t *dstp, *dst; uint8_t last_flags; idx = tx->req & tx->mask; last_flags = src->flags; src->flags = 0; wmb(); dst = dstp = &tx->lanai[idx]; srcp = src; if ((idx + cnt) < tx->mask) { for (i = 0; i < (cnt - 1); i += 2) { mxge_pio_copy(dstp, srcp, 2 * sizeof(*src)); wmb(); /* force write every 32 bytes */ srcp += 2; dstp += 2; } } else { /* submit all but the first request, and ensure that it is submitted below */ mxge_submit_req_backwards(tx, src, cnt); i = 0; } if (i < cnt) { /* submit the first request */ mxge_pio_copy(dstp, srcp, sizeof(*src)); wmb(); /* barrier before setting valid flag */ } /* re-write the last 32-bits with the valid flags */ src->flags = last_flags; src_ints = (uint32_t *)src; src_ints+=3; dst_ints = (volatile uint32_t *)dst; dst_ints+=3; *dst_ints = *src_ints; tx->req += cnt; wmb(); } #if IFCAP_TSO4 static void mxge_encap_tso(struct mxge_slice_state *ss, struct mbuf *m, int busdma_seg_cnt, int ip_off) { mxge_tx_ring_t *tx; mcp_kreq_ether_send_t *req; bus_dma_segment_t *seg; struct ip *ip; struct tcphdr *tcp; uint32_t low, high_swapped; int len, seglen, cum_len, cum_len_next; int next_is_first, chop, cnt, rdma_count, small; uint16_t pseudo_hdr_offset, cksum_offset, mss; uint8_t flags, flags_next; static int once; mss = m->m_pkthdr.tso_segsz; /* negative cum_len signifies to the * send loop that we are still in the * header portion of the TSO packet. */ /* ensure we have the ethernet, IP and TCP header together in the first mbuf, copy it to a scratch buffer if not */ if (__predict_false(m->m_len < ip_off + sizeof (*ip))) { m_copydata(m, 0, ip_off + sizeof (*ip), ss->scratch); ip = (struct ip *)(ss->scratch + ip_off); } else { ip = (struct ip *)(mtod(m, char *) + ip_off); } if (__predict_false(m->m_len < ip_off + (ip->ip_hl << 2) + sizeof (*tcp))) { m_copydata(m, 0, ip_off + (ip->ip_hl << 2) + sizeof (*tcp), ss->scratch); ip = (struct ip *)(mtod(m, char *) + ip_off); } tcp = (struct tcphdr *)((char *)ip + (ip->ip_hl << 2)); cum_len = -(ip_off + ((ip->ip_hl + tcp->th_off) << 2)); /* TSO implies checksum offload on this hardware */ cksum_offset = ip_off + (ip->ip_hl << 2); flags = MXGEFW_FLAGS_TSO_HDR | MXGEFW_FLAGS_FIRST; /* for TSO, pseudo_hdr_offset holds mss. * The firmware figures out where to put * the checksum by parsing the header. */ pseudo_hdr_offset = htobe16(mss); tx = &ss->tx; req = tx->req_list; seg = tx->seg_list; cnt = 0; rdma_count = 0; /* "rdma_count" is the number of RDMAs belonging to the * current packet BEFORE the current send request. For * non-TSO packets, this is equal to "count". * For TSO packets, rdma_count needs to be reset * to 0 after a segment cut. * * The rdma_count field of the send request is * the number of RDMAs of the packet starting at * that request. For TSO send requests with one ore more cuts * in the middle, this is the number of RDMAs starting * after the last cut in the request. All previous * segments before the last cut implicitly have 1 RDMA. * * Since the number of RDMAs is not known beforehand, * it must be filled-in retroactively - after each * segmentation cut or at the end of the entire packet. */ while (busdma_seg_cnt) { /* Break the busdma segment up into pieces*/ low = MXGE_LOWPART_TO_U32(seg->ds_addr); high_swapped = htobe32(MXGE_HIGHPART_TO_U32(seg->ds_addr)); len = seg->ds_len; while (len) { flags_next = flags & ~MXGEFW_FLAGS_FIRST; seglen = len; cum_len_next = cum_len + seglen; (req-rdma_count)->rdma_count = rdma_count + 1; if (__predict_true(cum_len >= 0)) { /* payload */ chop = (cum_len_next > mss); cum_len_next = cum_len_next % mss; next_is_first = (cum_len_next == 0); flags |= chop * MXGEFW_FLAGS_TSO_CHOP; flags_next |= next_is_first * MXGEFW_FLAGS_FIRST; rdma_count |= -(chop | next_is_first); rdma_count += chop & !next_is_first; } else if (cum_len_next >= 0) { /* header ends */ rdma_count = -1; cum_len_next = 0; seglen = -cum_len; small = (mss <= MXGEFW_SEND_SMALL_SIZE); flags_next = MXGEFW_FLAGS_TSO_PLD | MXGEFW_FLAGS_FIRST | (small * MXGEFW_FLAGS_SMALL); } req->addr_high = high_swapped; req->addr_low = htobe32(low); req->pseudo_hdr_offset = pseudo_hdr_offset; req->pad = 0; req->rdma_count = 1; req->length = htobe16(seglen); req->cksum_offset = cksum_offset; req->flags = flags | ((cum_len & 1) * MXGEFW_FLAGS_ALIGN_ODD); low += seglen; len -= seglen; cum_len = cum_len_next; flags = flags_next; req++; cnt++; rdma_count++; if (__predict_false(cksum_offset > seglen)) cksum_offset -= seglen; else cksum_offset = 0; if (__predict_false(cnt > tx->max_desc)) goto drop; } busdma_seg_cnt--; seg++; } (req-rdma_count)->rdma_count = rdma_count; do { req--; req->flags |= MXGEFW_FLAGS_TSO_LAST; } while (!(req->flags & (MXGEFW_FLAGS_TSO_CHOP | MXGEFW_FLAGS_FIRST))); tx->info[((cnt - 1) + tx->req) & tx->mask].flag = 1; mxge_submit_req(tx, tx->req_list, cnt); #ifdef IFNET_BUF_RING if ((ss->sc->num_slices > 1) && tx->queue_active == 0) { /* tell the NIC to start polling this slice */ *tx->send_go = 1; tx->queue_active = 1; tx->activate++; wmb(); } #endif return; drop: bus_dmamap_unload(tx->dmat, tx->info[tx->req & tx->mask].map); m_freem(m); ss->oerrors++; if (!once) { printf("tx->max_desc exceeded via TSO!\n"); printf("mss = %d, %ld, %d!\n", mss, (long)seg - (long)tx->seg_list, tx->max_desc); once = 1; } return; } #endif /* IFCAP_TSO4 */ #ifdef MXGE_NEW_VLAN_API /* * We reproduce the software vlan tag insertion from * net/if_vlan.c:vlan_start() here so that we can advertise "hardware" * vlan tag insertion. We need to advertise this in order to have the * vlan interface respect our csum offload flags. */ static struct mbuf * mxge_vlan_tag_insert(struct mbuf *m) { struct ether_vlan_header *evl; M_PREPEND(m, ETHER_VLAN_ENCAP_LEN, M_DONTWAIT); if (__predict_false(m == NULL)) return NULL; if (m->m_len < sizeof(*evl)) { m = m_pullup(m, sizeof(*evl)); if (__predict_false(m == NULL)) return NULL; } /* * Transform the Ethernet header into an Ethernet header * with 802.1Q encapsulation. */ evl = mtod(m, struct ether_vlan_header *); bcopy((char *)evl + ETHER_VLAN_ENCAP_LEN, (char *)evl, ETHER_HDR_LEN - ETHER_TYPE_LEN); evl->evl_encap_proto = htons(ETHERTYPE_VLAN); evl->evl_tag = htons(m->m_pkthdr.ether_vtag); m->m_flags &= ~M_VLANTAG; return m; } #endif /* MXGE_NEW_VLAN_API */ static void mxge_encap(struct mxge_slice_state *ss, struct mbuf *m) { mxge_softc_t *sc; mcp_kreq_ether_send_t *req; bus_dma_segment_t *seg; struct mbuf *m_tmp; struct ifnet *ifp; mxge_tx_ring_t *tx; struct ip *ip; int cnt, cum_len, err, i, idx, odd_flag, ip_off; uint16_t pseudo_hdr_offset; uint8_t flags, cksum_offset; sc = ss->sc; ifp = sc->ifp; tx = &ss->tx; ip_off = sizeof (struct ether_header); #ifdef MXGE_NEW_VLAN_API if (m->m_flags & M_VLANTAG) { m = mxge_vlan_tag_insert(m); if (__predict_false(m == NULL)) goto drop; ip_off += ETHER_VLAN_ENCAP_LEN; } #endif /* (try to) map the frame for DMA */ idx = tx->req & tx->mask; err = bus_dmamap_load_mbuf_sg(tx->dmat, tx->info[idx].map, m, tx->seg_list, &cnt, BUS_DMA_NOWAIT); if (__predict_false(err == EFBIG)) { /* Too many segments in the chain. Try to defrag */ m_tmp = m_defrag(m, M_NOWAIT); if (m_tmp == NULL) { goto drop; } ss->tx.defrag++; m = m_tmp; err = bus_dmamap_load_mbuf_sg(tx->dmat, tx->info[idx].map, m, tx->seg_list, &cnt, BUS_DMA_NOWAIT); } if (__predict_false(err != 0)) { device_printf(sc->dev, "bus_dmamap_load_mbuf_sg returned %d" " packet len = %d\n", err, m->m_pkthdr.len); goto drop; } bus_dmamap_sync(tx->dmat, tx->info[idx].map, BUS_DMASYNC_PREWRITE); tx->info[idx].m = m; #if IFCAP_TSO4 /* TSO is different enough, we handle it in another routine */ if (m->m_pkthdr.csum_flags & (CSUM_TSO)) { mxge_encap_tso(ss, m, cnt, ip_off); return; } #endif req = tx->req_list; cksum_offset = 0; pseudo_hdr_offset = 0; flags = MXGEFW_FLAGS_NO_TSO; /* checksum offloading? */ if (m->m_pkthdr.csum_flags & (CSUM_DELAY_DATA)) { /* ensure ip header is in first mbuf, copy it to a scratch buffer if not */ if (__predict_false(m->m_len < ip_off + sizeof (*ip))) { m_copydata(m, 0, ip_off + sizeof (*ip), ss->scratch); ip = (struct ip *)(ss->scratch + ip_off); } else { ip = (struct ip *)(mtod(m, char *) + ip_off); } cksum_offset = ip_off + (ip->ip_hl << 2); pseudo_hdr_offset = cksum_offset + m->m_pkthdr.csum_data; pseudo_hdr_offset = htobe16(pseudo_hdr_offset); req->cksum_offset = cksum_offset; flags |= MXGEFW_FLAGS_CKSUM; odd_flag = MXGEFW_FLAGS_ALIGN_ODD; } else { odd_flag = 0; } if (m->m_pkthdr.len < MXGEFW_SEND_SMALL_SIZE) flags |= MXGEFW_FLAGS_SMALL; /* convert segments into a request list */ cum_len = 0; seg = tx->seg_list; req->flags = MXGEFW_FLAGS_FIRST; for (i = 0; i < cnt; i++) { req->addr_low = htobe32(MXGE_LOWPART_TO_U32(seg->ds_addr)); req->addr_high = htobe32(MXGE_HIGHPART_TO_U32(seg->ds_addr)); req->length = htobe16(seg->ds_len); req->cksum_offset = cksum_offset; if (cksum_offset > seg->ds_len) cksum_offset -= seg->ds_len; else cksum_offset = 0; req->pseudo_hdr_offset = pseudo_hdr_offset; req->pad = 0; /* complete solid 16-byte block */ req->rdma_count = 1; req->flags |= flags | ((cum_len & 1) * odd_flag); cum_len += seg->ds_len; seg++; req++; req->flags = 0; } req--; /* pad runts to 60 bytes */ if (cum_len < 60) { req++; req->addr_low = htobe32(MXGE_LOWPART_TO_U32(sc->zeropad_dma.bus_addr)); req->addr_high = htobe32(MXGE_HIGHPART_TO_U32(sc->zeropad_dma.bus_addr)); req->length = htobe16(60 - cum_len); req->cksum_offset = 0; req->pseudo_hdr_offset = pseudo_hdr_offset; req->pad = 0; /* complete solid 16-byte block */ req->rdma_count = 1; req->flags |= flags | ((cum_len & 1) * odd_flag); cnt++; } tx->req_list[0].rdma_count = cnt; #if 0 /* print what the firmware will see */ for (i = 0; i < cnt; i++) { printf("%d: addr: 0x%x 0x%x len:%d pso%d," "cso:%d, flags:0x%x, rdma:%d\n", i, (int)ntohl(tx->req_list[i].addr_high), (int)ntohl(tx->req_list[i].addr_low), (int)ntohs(tx->req_list[i].length), (int)ntohs(tx->req_list[i].pseudo_hdr_offset), tx->req_list[i].cksum_offset, tx->req_list[i].flags, tx->req_list[i].rdma_count); } printf("--------------\n"); #endif tx->info[((cnt - 1) + tx->req) & tx->mask].flag = 1; mxge_submit_req(tx, tx->req_list, cnt); #ifdef IFNET_BUF_RING if ((ss->sc->num_slices > 1) && tx->queue_active == 0) { /* tell the NIC to start polling this slice */ *tx->send_go = 1; tx->queue_active = 1; tx->activate++; wmb(); } #endif return; drop: m_freem(m); ss->oerrors++; return; } #ifdef IFNET_BUF_RING static void mxge_qflush(struct ifnet *ifp) { mxge_softc_t *sc = ifp->if_softc; mxge_tx_ring_t *tx; struct mbuf *m; int slice; for (slice = 0; slice < sc->num_slices; slice++) { tx = &sc->ss[slice].tx; mtx_lock(&tx->mtx); while ((m = buf_ring_dequeue_sc(tx->br)) != NULL) m_freem(m); mtx_unlock(&tx->mtx); } if_qflush(ifp); } static inline void mxge_start_locked(struct mxge_slice_state *ss) { mxge_softc_t *sc; struct mbuf *m; struct ifnet *ifp; mxge_tx_ring_t *tx; sc = ss->sc; ifp = sc->ifp; tx = &ss->tx; while ((tx->mask - (tx->req - tx->done)) > tx->max_desc) { m = drbr_dequeue(ifp, tx->br); if (m == NULL) { return; } /* let BPF see it */ BPF_MTAP(ifp, m); /* give it to the nic */ mxge_encap(ss, m); } /* ran out of transmit slots */ if (((ss->if_drv_flags & IFF_DRV_OACTIVE) == 0) && (!drbr_empty(ifp, tx->br))) { ss->if_drv_flags |= IFF_DRV_OACTIVE; tx->stall++; } } static int mxge_transmit_locked(struct mxge_slice_state *ss, struct mbuf *m) { mxge_softc_t *sc; struct ifnet *ifp; mxge_tx_ring_t *tx; int err; sc = ss->sc; ifp = sc->ifp; tx = &ss->tx; if ((ss->if_drv_flags & (IFF_DRV_RUNNING|IFF_DRV_OACTIVE)) != IFF_DRV_RUNNING) { err = drbr_enqueue(ifp, tx->br, m); return (err); } if (!drbr_needs_enqueue(ifp, tx->br) && ((tx->mask - (tx->req - tx->done)) > tx->max_desc)) { /* let BPF see it */ BPF_MTAP(ifp, m); /* give it to the nic */ mxge_encap(ss, m); } else if ((err = drbr_enqueue(ifp, tx->br, m)) != 0) { return (err); } if (!drbr_empty(ifp, tx->br)) mxge_start_locked(ss); return (0); } static int mxge_transmit(struct ifnet *ifp, struct mbuf *m) { mxge_softc_t *sc = ifp->if_softc; struct mxge_slice_state *ss; mxge_tx_ring_t *tx; int err = 0; int slice; slice = m->m_pkthdr.flowid; slice &= (sc->num_slices - 1); /* num_slices always power of 2 */ ss = &sc->ss[slice]; tx = &ss->tx; if (mtx_trylock(&tx->mtx)) { err = mxge_transmit_locked(ss, m); mtx_unlock(&tx->mtx); } else { err = drbr_enqueue(ifp, tx->br, m); } return (err); } #else static inline void mxge_start_locked(struct mxge_slice_state *ss) { mxge_softc_t *sc; struct mbuf *m; struct ifnet *ifp; mxge_tx_ring_t *tx; sc = ss->sc; ifp = sc->ifp; tx = &ss->tx; while ((tx->mask - (tx->req - tx->done)) > tx->max_desc) { IFQ_DRV_DEQUEUE(&ifp->if_snd, m); if (m == NULL) { return; } /* let BPF see it */ BPF_MTAP(ifp, m); /* give it to the nic */ mxge_encap(ss, m); } /* ran out of transmit slots */ if ((sc->ifp->if_drv_flags & IFF_DRV_OACTIVE) == 0) { sc->ifp->if_drv_flags |= IFF_DRV_OACTIVE; tx->stall++; } } #endif static void mxge_start(struct ifnet *ifp) { mxge_softc_t *sc = ifp->if_softc; struct mxge_slice_state *ss; /* only use the first slice for now */ ss = &sc->ss[0]; mtx_lock(&ss->tx.mtx); mxge_start_locked(ss); mtx_unlock(&ss->tx.mtx); } /* * copy an array of mcp_kreq_ether_recv_t's to the mcp. Copy * at most 32 bytes at a time, so as to avoid involving the software * pio handler in the nic. We re-write the first segment's low * DMA address to mark it valid only after we write the entire chunk * in a burst */ static inline void mxge_submit_8rx(volatile mcp_kreq_ether_recv_t *dst, mcp_kreq_ether_recv_t *src) { uint32_t low; low = src->addr_low; src->addr_low = 0xffffffff; mxge_pio_copy(dst, src, 4 * sizeof (*src)); wmb(); mxge_pio_copy(dst + 4, src + 4, 4 * sizeof (*src)); wmb(); src->addr_low = low; dst->addr_low = low; wmb(); } static int mxge_get_buf_small(struct mxge_slice_state *ss, bus_dmamap_t map, int idx) { bus_dma_segment_t seg; struct mbuf *m; mxge_rx_ring_t *rx = &ss->rx_small; int cnt, err; m = m_gethdr(M_DONTWAIT, MT_DATA); if (m == NULL) { rx->alloc_fail++; err = ENOBUFS; goto done; } m->m_len = MHLEN; err = bus_dmamap_load_mbuf_sg(rx->dmat, map, m, &seg, &cnt, BUS_DMA_NOWAIT); if (err != 0) { m_free(m); goto done; } rx->info[idx].m = m; rx->shadow[idx].addr_low = htobe32(MXGE_LOWPART_TO_U32(seg.ds_addr)); rx->shadow[idx].addr_high = htobe32(MXGE_HIGHPART_TO_U32(seg.ds_addr)); done: if ((idx & 7) == 7) mxge_submit_8rx(&rx->lanai[idx - 7], &rx->shadow[idx - 7]); return err; } static int mxge_get_buf_big(struct mxge_slice_state *ss, bus_dmamap_t map, int idx) { bus_dma_segment_t seg[3]; struct mbuf *m; mxge_rx_ring_t *rx = &ss->rx_big; int cnt, err, i; if (rx->cl_size == MCLBYTES) m = m_getcl(M_DONTWAIT, MT_DATA, M_PKTHDR); else m = m_getjcl(M_DONTWAIT, MT_DATA, M_PKTHDR, rx->cl_size); if (m == NULL) { rx->alloc_fail++; err = ENOBUFS; goto done; } m->m_len = rx->mlen; err = bus_dmamap_load_mbuf_sg(rx->dmat, map, m, seg, &cnt, BUS_DMA_NOWAIT); if (err != 0) { m_free(m); goto done; } rx->info[idx].m = m; rx->shadow[idx].addr_low = htobe32(MXGE_LOWPART_TO_U32(seg->ds_addr)); rx->shadow[idx].addr_high = htobe32(MXGE_HIGHPART_TO_U32(seg->ds_addr)); #if MXGE_VIRT_JUMBOS for (i = 1; i < cnt; i++) { rx->shadow[idx + i].addr_low = htobe32(MXGE_LOWPART_TO_U32(seg[i].ds_addr)); rx->shadow[idx + i].addr_high = htobe32(MXGE_HIGHPART_TO_U32(seg[i].ds_addr)); } #endif done: for (i = 0; i < rx->nbufs; i++) { if ((idx & 7) == 7) { mxge_submit_8rx(&rx->lanai[idx - 7], &rx->shadow[idx - 7]); } idx++; } return err; } /* * Myri10GE hardware checksums are not valid if the sender * padded the frame with non-zero padding. This is because * the firmware just does a simple 16-bit 1s complement * checksum across the entire frame, excluding the first 14 * bytes. It is best to simply to check the checksum and * tell the stack about it only if the checksum is good */ static inline uint16_t mxge_rx_csum(struct mbuf *m, int csum) { struct ether_header *eh; struct ip *ip; uint16_t c; eh = mtod(m, struct ether_header *); /* only deal with IPv4 TCP & UDP for now */ if (__predict_false(eh->ether_type != htons(ETHERTYPE_IP))) return 1; ip = (struct ip *)(eh + 1); if (__predict_false(ip->ip_p != IPPROTO_TCP && ip->ip_p != IPPROTO_UDP)) return 1; #ifdef INET c = in_pseudo(ip->ip_src.s_addr, ip->ip_dst.s_addr, htonl(ntohs(csum) + ntohs(ip->ip_len) + - (ip->ip_hl << 2) + ip->ip_p)); #else c = 1; #endif c ^= 0xffff; return (c); } static void mxge_vlan_tag_remove(struct mbuf *m, uint32_t *csum) { struct ether_vlan_header *evl; struct ether_header *eh; uint32_t partial; evl = mtod(m, struct ether_vlan_header *); eh = mtod(m, struct ether_header *); /* * fix checksum by subtracting ETHER_VLAN_ENCAP_LEN bytes * after what the firmware thought was the end of the ethernet * header. */ /* put checksum into host byte order */ *csum = ntohs(*csum); partial = ntohl(*(uint32_t *)(mtod(m, char *) + ETHER_HDR_LEN)); (*csum) += ~partial; (*csum) += ((*csum) < ~partial); (*csum) = ((*csum) >> 16) + ((*csum) & 0xFFFF); (*csum) = ((*csum) >> 16) + ((*csum) & 0xFFFF); /* restore checksum to network byte order; later consumers expect this */ *csum = htons(*csum); /* save the tag */ #ifdef MXGE_NEW_VLAN_API m->m_pkthdr.ether_vtag = ntohs(evl->evl_tag); #else { struct m_tag *mtag; mtag = m_tag_alloc(MTAG_VLAN, MTAG_VLAN_TAG, sizeof(u_int), M_NOWAIT); if (mtag == NULL) return; VLAN_TAG_VALUE(mtag) = ntohs(evl->evl_tag); m_tag_prepend(m, mtag); } #endif m->m_flags |= M_VLANTAG; /* * Remove the 802.1q header by copying the Ethernet * addresses over it and adjusting the beginning of * the data in the mbuf. The encapsulated Ethernet * type field is already in place. */ bcopy((char *)evl, (char *)evl + ETHER_VLAN_ENCAP_LEN, ETHER_HDR_LEN - ETHER_TYPE_LEN); m_adj(m, ETHER_VLAN_ENCAP_LEN); } static inline void mxge_rx_done_big(struct mxge_slice_state *ss, uint32_t len, uint32_t csum) { mxge_softc_t *sc; struct ifnet *ifp; struct mbuf *m; struct ether_header *eh; mxge_rx_ring_t *rx; bus_dmamap_t old_map; int idx; uint16_t tcpudp_csum; sc = ss->sc; ifp = sc->ifp; rx = &ss->rx_big; idx = rx->cnt & rx->mask; rx->cnt += rx->nbufs; /* save a pointer to the received mbuf */ m = rx->info[idx].m; /* try to replace the received mbuf */ if (mxge_get_buf_big(ss, rx->extra_map, idx)) { /* drop the frame -- the old mbuf is re-cycled */ ifp->if_ierrors++; return; } /* unmap the received buffer */ old_map = rx->info[idx].map; bus_dmamap_sync(rx->dmat, old_map, BUS_DMASYNC_POSTREAD); bus_dmamap_unload(rx->dmat, old_map); /* swap the bus_dmamap_t's */ rx->info[idx].map = rx->extra_map; rx->extra_map = old_map; /* mcp implicitly skips 1st 2 bytes so that packet is properly * aligned */ m->m_data += MXGEFW_PAD; m->m_pkthdr.rcvif = ifp; m->m_len = m->m_pkthdr.len = len; ss->ipackets++; eh = mtod(m, struct ether_header *); if (eh->ether_type == htons(ETHERTYPE_VLAN)) { mxge_vlan_tag_remove(m, &csum); } /* if the checksum is valid, mark it in the mbuf header */ if (sc->csum_flag && (0 == (tcpudp_csum = mxge_rx_csum(m, csum)))) { if (sc->lro_cnt && (0 == mxge_lro_rx(ss, m, csum))) return; /* otherwise, it was a UDP frame, or a TCP frame which we could not do LRO on. Tell the stack that the checksum is good */ m->m_pkthdr.csum_data = 0xffff; m->m_pkthdr.csum_flags = CSUM_PSEUDO_HDR | CSUM_DATA_VALID; } /* flowid only valid if RSS hashing is enabled */ if (sc->num_slices > 1) { m->m_pkthdr.flowid = (ss - sc->ss); m->m_flags |= M_FLOWID; } /* pass the frame up the stack */ (*ifp->if_input)(ifp, m); } static inline void mxge_rx_done_small(struct mxge_slice_state *ss, uint32_t len, uint32_t csum) { mxge_softc_t *sc; struct ifnet *ifp; struct ether_header *eh; struct mbuf *m; mxge_rx_ring_t *rx; bus_dmamap_t old_map; int idx; uint16_t tcpudp_csum; sc = ss->sc; ifp = sc->ifp; rx = &ss->rx_small; idx = rx->cnt & rx->mask; rx->cnt++; /* save a pointer to the received mbuf */ m = rx->info[idx].m; /* try to replace the received mbuf */ if (mxge_get_buf_small(ss, rx->extra_map, idx)) { /* drop the frame -- the old mbuf is re-cycled */ ifp->if_ierrors++; return; } /* unmap the received buffer */ old_map = rx->info[idx].map; bus_dmamap_sync(rx->dmat, old_map, BUS_DMASYNC_POSTREAD); bus_dmamap_unload(rx->dmat, old_map); /* swap the bus_dmamap_t's */ rx->info[idx].map = rx->extra_map; rx->extra_map = old_map; /* mcp implicitly skips 1st 2 bytes so that packet is properly * aligned */ m->m_data += MXGEFW_PAD; m->m_pkthdr.rcvif = ifp; m->m_len = m->m_pkthdr.len = len; ss->ipackets++; eh = mtod(m, struct ether_header *); if (eh->ether_type == htons(ETHERTYPE_VLAN)) { mxge_vlan_tag_remove(m, &csum); } /* if the checksum is valid, mark it in the mbuf header */ if (sc->csum_flag && (0 == (tcpudp_csum = mxge_rx_csum(m, csum)))) { if (sc->lro_cnt && (0 == mxge_lro_rx(ss, m, csum))) return; /* otherwise, it was a UDP frame, or a TCP frame which we could not do LRO on. Tell the stack that the checksum is good */ m->m_pkthdr.csum_data = 0xffff; m->m_pkthdr.csum_flags = CSUM_PSEUDO_HDR | CSUM_DATA_VALID; } /* flowid only valid if RSS hashing is enabled */ if (sc->num_slices > 1) { m->m_pkthdr.flowid = (ss - sc->ss); m->m_flags |= M_FLOWID; } /* pass the frame up the stack */ (*ifp->if_input)(ifp, m); } static inline void mxge_clean_rx_done(struct mxge_slice_state *ss) { mxge_rx_done_t *rx_done = &ss->rx_done; int limit = 0; uint16_t length; uint16_t checksum; while (rx_done->entry[rx_done->idx].length != 0) { length = ntohs(rx_done->entry[rx_done->idx].length); rx_done->entry[rx_done->idx].length = 0; checksum = rx_done->entry[rx_done->idx].checksum; if (length <= (MHLEN - MXGEFW_PAD)) mxge_rx_done_small(ss, length, checksum); else mxge_rx_done_big(ss, length, checksum); rx_done->cnt++; rx_done->idx = rx_done->cnt & rx_done->mask; /* limit potential for livelock */ if (__predict_false(++limit > rx_done->mask / 2)) break; } #ifdef INET while (!SLIST_EMPTY(&ss->lro_active)) { struct lro_entry *lro = SLIST_FIRST(&ss->lro_active); SLIST_REMOVE_HEAD(&ss->lro_active, next); mxge_lro_flush(ss, lro); } #endif } static inline void mxge_tx_done(struct mxge_slice_state *ss, uint32_t mcp_idx) { struct ifnet *ifp; mxge_tx_ring_t *tx; struct mbuf *m; bus_dmamap_t map; int idx; int *flags; tx = &ss->tx; ifp = ss->sc->ifp; while (tx->pkt_done != mcp_idx) { idx = tx->done & tx->mask; tx->done++; m = tx->info[idx].m; /* mbuf and DMA map only attached to the first segment per-mbuf */ if (m != NULL) { ss->obytes += m->m_pkthdr.len; if (m->m_flags & M_MCAST) ss->omcasts++; ss->opackets++; tx->info[idx].m = NULL; map = tx->info[idx].map; bus_dmamap_unload(tx->dmat, map); m_freem(m); } if (tx->info[idx].flag) { tx->info[idx].flag = 0; tx->pkt_done++; } } /* If we have space, clear IFF_OACTIVE to tell the stack that its OK to send packets */ #ifdef IFNET_BUF_RING flags = &ss->if_drv_flags; #else flags = &ifp->if_drv_flags; #endif mtx_lock(&ss->tx.mtx); if ((*flags) & IFF_DRV_OACTIVE && tx->req - tx->done < (tx->mask + 1)/4) { *(flags) &= ~IFF_DRV_OACTIVE; ss->tx.wake++; mxge_start_locked(ss); } #ifdef IFNET_BUF_RING if ((ss->sc->num_slices > 1) && (tx->req == tx->done)) { /* let the NIC stop polling this queue, since there * are no more transmits pending */ if (tx->req == tx->done) { *tx->send_stop = 1; tx->queue_active = 0; tx->deactivate++; wmb(); } } #endif mtx_unlock(&ss->tx.mtx); } static struct mxge_media_type mxge_xfp_media_types[] = { {IFM_10G_CX4, 0x7f, "10GBASE-CX4 (module)"}, {IFM_10G_SR, (1 << 7), "10GBASE-SR"}, {IFM_10G_LR, (1 << 6), "10GBASE-LR"}, {0, (1 << 5), "10GBASE-ER"}, {IFM_10G_LRM, (1 << 4), "10GBASE-LRM"}, {0, (1 << 3), "10GBASE-SW"}, {0, (1 << 2), "10GBASE-LW"}, {0, (1 << 1), "10GBASE-EW"}, {0, (1 << 0), "Reserved"} }; static struct mxge_media_type mxge_sfp_media_types[] = { {IFM_10G_TWINAX, 0, "10GBASE-Twinax"}, {0, (1 << 7), "Reserved"}, {IFM_10G_LRM, (1 << 6), "10GBASE-LRM"}, {IFM_10G_LR, (1 << 5), "10GBASE-LR"}, {IFM_10G_SR, (1 << 4), "10GBASE-SR"} }; static void -mxge_set_media(mxge_softc_t *sc, int type) +mxge_media_set(mxge_softc_t *sc, int media_type) { - sc->media_flags |= type; - ifmedia_add(&sc->media, sc->media_flags, 0, NULL); - ifmedia_set(&sc->media, sc->media_flags); + + + ifmedia_add(&sc->media, IFM_ETHER | IFM_FDX | media_type, + 0, NULL); + ifmedia_set(&sc->media, IFM_ETHER | IFM_FDX | media_type); + sc->current_media = media_type; + sc->media.ifm_media = sc->media.ifm_cur->ifm_media; } - -/* - * Determine the media type for a NIC. Some XFPs will identify - * themselves only when their link is up, so this is initiated via a - * link up interrupt. However, this can potentially take up to - * several milliseconds, so it is run via the watchdog routine, rather - * than in the interrupt handler itself. This need only be done - * once, not each time the link is up. - */ static void -mxge_media_probe(mxge_softc_t *sc) +mxge_media_init(mxge_softc_t *sc) { - mxge_cmd_t cmd; - char *cage_type; char *ptr; - struct mxge_media_type *mxge_media_types = NULL; - int i, err, ms, mxge_media_type_entries; - uint32_t byte; + int i; - sc->need_media_probe = 0; + ifmedia_removeall(&sc->media); + mxge_media_set(sc, IFM_AUTO); - /* if we've already set a media type, we're done */ - if (sc->media_flags != (IFM_ETHER | IFM_AUTO)) - return; - /* * parse the product code to deterimine the interface type * (CX4, XFP, Quad Ribbon Fiber) by looking at the character * after the 3rd dash in the driver's cached copy of the * EEPROM's product code string. */ ptr = sc->product_code_string; if (ptr == NULL) { device_printf(sc->dev, "Missing product code\n"); + return; } for (i = 0; i < 3; i++, ptr++) { ptr = index(ptr, '-'); if (ptr == NULL) { device_printf(sc->dev, "only %d dashes in PC?!?\n", i); return; } } if (*ptr == 'C') { /* -C is CX4 */ - mxge_set_media(sc, IFM_10G_CX4); - return; - } - else if (*ptr == 'Q') { + sc->connector = MXGE_CX4; + mxge_media_set(sc, IFM_10G_CX4); + } else if (*ptr == 'Q') { /* -Q is Quad Ribbon Fiber */ + sc->connector = MXGE_QRF; device_printf(sc->dev, "Quad Ribbon Fiber Media\n"); /* FreeBSD has no media type for Quad ribbon fiber */ - return; + } else if (*ptr == 'R') { + /* -R is XFP */ + sc->connector = MXGE_XFP; + } else if (*ptr == 'S' || *(ptr +1) == 'S') { + /* -S or -2S is SFP+ */ + sc->connector = MXGE_SFP; + } else { + device_printf(sc->dev, "Unknown media type: %c\n", *ptr); } +} - if (*ptr == 'R') { +/* + * Determine the media type for a NIC. Some XFPs will identify + * themselves only when their link is up, so this is initiated via a + * link up interrupt. However, this can potentially take up to + * several milliseconds, so it is run via the watchdog routine, rather + * than in the interrupt handler itself. + */ +static void +mxge_media_probe(mxge_softc_t *sc) +{ + mxge_cmd_t cmd; + char *cage_type; + + struct mxge_media_type *mxge_media_types = NULL; + int i, err, ms, mxge_media_type_entries; + uint32_t byte; + + sc->need_media_probe = 0; + + if (sc->connector == MXGE_XFP) { /* -R is XFP */ mxge_media_types = mxge_xfp_media_types; mxge_media_type_entries = sizeof (mxge_xfp_media_types) / sizeof (mxge_xfp_media_types[0]); byte = MXGE_XFP_COMPLIANCE_BYTE; cage_type = "XFP"; - } - - if (*ptr == 'S' || *(ptr +1) == 'S') { + } else if (sc->connector == MXGE_SFP) { /* -S or -2S is SFP+ */ mxge_media_types = mxge_sfp_media_types; mxge_media_type_entries = sizeof (mxge_sfp_media_types) / sizeof (mxge_sfp_media_types[0]); cage_type = "SFP+"; byte = 3; - } - - if (mxge_media_types == NULL) { - device_printf(sc->dev, "Unknown media type: %c\n", *ptr); + } else { + /* nothing to do; media type cannot change */ return; } /* * At this point we know the NIC has an XFP cage, so now we * try to determine what is in the cage by using the * firmware's XFP I2C commands to read the XFP 10GbE compilance * register. We read just one byte, which may take over * a millisecond */ cmd.data0 = 0; /* just fetch 1 byte, not all 256 */ cmd.data1 = byte; err = mxge_send_cmd(sc, MXGEFW_CMD_I2C_READ, &cmd); if (err == MXGEFW_CMD_ERROR_I2C_FAILURE) { device_printf(sc->dev, "failed to read XFP\n"); } if (err == MXGEFW_CMD_ERROR_I2C_ABSENT) { device_printf(sc->dev, "Type R/S with no XFP!?!?\n"); } if (err != MXGEFW_CMD_OK) { return; } /* now we wait for the data to be cached */ cmd.data0 = byte; err = mxge_send_cmd(sc, MXGEFW_CMD_I2C_BYTE, &cmd); for (ms = 0; (err == EBUSY) && (ms < 50); ms++) { DELAY(1000); cmd.data0 = byte; err = mxge_send_cmd(sc, MXGEFW_CMD_I2C_BYTE, &cmd); } if (err != MXGEFW_CMD_OK) { device_printf(sc->dev, "failed to read %s (%d, %dms)\n", cage_type, err, ms); return; } if (cmd.data0 == mxge_media_types[0].bitmask) { if (mxge_verbose) device_printf(sc->dev, "%s:%s\n", cage_type, mxge_media_types[0].name); - mxge_set_media(sc, mxge_media_types[0].flag); + if (sc->current_media != mxge_media_types[0].flag) { + mxge_media_init(sc); + mxge_media_set(sc, mxge_media_types[0].flag); + } return; } for (i = 1; i < mxge_media_type_entries; i++) { if (cmd.data0 & mxge_media_types[i].bitmask) { if (mxge_verbose) device_printf(sc->dev, "%s:%s\n", cage_type, mxge_media_types[i].name); - mxge_set_media(sc, mxge_media_types[i].flag); + if (sc->current_media != mxge_media_types[i].flag) { + mxge_media_init(sc); + mxge_media_set(sc, mxge_media_types[i].flag); + } return; } } - device_printf(sc->dev, "%s media 0x%x unknown\n", cage_type, - cmd.data0); + if (mxge_verbose) + device_printf(sc->dev, "%s media 0x%x unknown\n", + cage_type, cmd.data0); return; } static void mxge_intr(void *arg) { struct mxge_slice_state *ss = arg; mxge_softc_t *sc = ss->sc; mcp_irq_data_t *stats = ss->fw_stats; mxge_tx_ring_t *tx = &ss->tx; mxge_rx_done_t *rx_done = &ss->rx_done; uint32_t send_done_count; uint8_t valid; #ifndef IFNET_BUF_RING /* an interrupt on a non-zero slice is implicitly valid since MSI-X irqs are not shared */ if (ss != sc->ss) { mxge_clean_rx_done(ss); *ss->irq_claim = be32toh(3); return; } #endif /* make sure the DMA has finished */ if (!stats->valid) { return; } valid = stats->valid; if (sc->legacy_irq) { /* lower legacy IRQ */ *sc->irq_deassert = 0; if (!mxge_deassert_wait) /* don't wait for conf. that irq is low */ stats->valid = 0; } else { stats->valid = 0; } /* loop while waiting for legacy irq deassertion */ do { /* check for transmit completes and receives */ send_done_count = be32toh(stats->send_done_count); while ((send_done_count != tx->pkt_done) || (rx_done->entry[rx_done->idx].length != 0)) { if (send_done_count != tx->pkt_done) mxge_tx_done(ss, (int)send_done_count); mxge_clean_rx_done(ss); send_done_count = be32toh(stats->send_done_count); } if (sc->legacy_irq && mxge_deassert_wait) wmb(); } while (*((volatile uint8_t *) &stats->valid)); /* fw link & error stats meaningful only on the first slice */ if (__predict_false((ss == sc->ss) && stats->stats_updated)) { if (sc->link_state != stats->link_up) { sc->link_state = stats->link_up; if (sc->link_state) { if_link_state_change(sc->ifp, LINK_STATE_UP); + sc->ifp->if_baudrate = IF_Gbps(10UL); if (mxge_verbose) device_printf(sc->dev, "link up\n"); } else { if_link_state_change(sc->ifp, LINK_STATE_DOWN); + sc->ifp->if_baudrate = 0; if (mxge_verbose) device_printf(sc->dev, "link down\n"); } sc->need_media_probe = 1; } if (sc->rdma_tags_available != be32toh(stats->rdma_tags_available)) { sc->rdma_tags_available = be32toh(stats->rdma_tags_available); device_printf(sc->dev, "RDMA timed out! %d tags " "left\n", sc->rdma_tags_available); } if (stats->link_down) { sc->down_cnt += stats->link_down; sc->link_state = 0; if_link_state_change(sc->ifp, LINK_STATE_DOWN); } } /* check to see if we have rx token to pass back */ if (valid & 0x1) *ss->irq_claim = be32toh(3); *(ss->irq_claim + 1) = be32toh(3); } static void mxge_init(void *arg) { } static void mxge_free_slice_mbufs(struct mxge_slice_state *ss) { struct lro_entry *lro_entry; int i; while (!SLIST_EMPTY(&ss->lro_free)) { lro_entry = SLIST_FIRST(&ss->lro_free); SLIST_REMOVE_HEAD(&ss->lro_free, next); free(lro_entry, M_DEVBUF); } for (i = 0; i <= ss->rx_big.mask; i++) { if (ss->rx_big.info[i].m == NULL) continue; bus_dmamap_unload(ss->rx_big.dmat, ss->rx_big.info[i].map); m_freem(ss->rx_big.info[i].m); ss->rx_big.info[i].m = NULL; } for (i = 0; i <= ss->rx_small.mask; i++) { if (ss->rx_small.info[i].m == NULL) continue; bus_dmamap_unload(ss->rx_small.dmat, ss->rx_small.info[i].map); m_freem(ss->rx_small.info[i].m); ss->rx_small.info[i].m = NULL; } /* transmit ring used only on the first slice */ if (ss->tx.info == NULL) return; for (i = 0; i <= ss->tx.mask; i++) { ss->tx.info[i].flag = 0; if (ss->tx.info[i].m == NULL) continue; bus_dmamap_unload(ss->tx.dmat, ss->tx.info[i].map); m_freem(ss->tx.info[i].m); ss->tx.info[i].m = NULL; } } static void mxge_free_mbufs(mxge_softc_t *sc) { int slice; for (slice = 0; slice < sc->num_slices; slice++) mxge_free_slice_mbufs(&sc->ss[slice]); } static void mxge_free_slice_rings(struct mxge_slice_state *ss) { int i; if (ss->rx_done.entry != NULL) mxge_dma_free(&ss->rx_done.dma); ss->rx_done.entry = NULL; if (ss->tx.req_bytes != NULL) free(ss->tx.req_bytes, M_DEVBUF); ss->tx.req_bytes = NULL; if (ss->tx.seg_list != NULL) free(ss->tx.seg_list, M_DEVBUF); ss->tx.seg_list = NULL; if (ss->rx_small.shadow != NULL) free(ss->rx_small.shadow, M_DEVBUF); ss->rx_small.shadow = NULL; if (ss->rx_big.shadow != NULL) free(ss->rx_big.shadow, M_DEVBUF); ss->rx_big.shadow = NULL; if (ss->tx.info != NULL) { if (ss->tx.dmat != NULL) { for (i = 0; i <= ss->tx.mask; i++) { bus_dmamap_destroy(ss->tx.dmat, ss->tx.info[i].map); } bus_dma_tag_destroy(ss->tx.dmat); } free(ss->tx.info, M_DEVBUF); } ss->tx.info = NULL; if (ss->rx_small.info != NULL) { if (ss->rx_small.dmat != NULL) { for (i = 0; i <= ss->rx_small.mask; i++) { bus_dmamap_destroy(ss->rx_small.dmat, ss->rx_small.info[i].map); } bus_dmamap_destroy(ss->rx_small.dmat, ss->rx_small.extra_map); bus_dma_tag_destroy(ss->rx_small.dmat); } free(ss->rx_small.info, M_DEVBUF); } ss->rx_small.info = NULL; if (ss->rx_big.info != NULL) { if (ss->rx_big.dmat != NULL) { for (i = 0; i <= ss->rx_big.mask; i++) { bus_dmamap_destroy(ss->rx_big.dmat, ss->rx_big.info[i].map); } bus_dmamap_destroy(ss->rx_big.dmat, ss->rx_big.extra_map); bus_dma_tag_destroy(ss->rx_big.dmat); } free(ss->rx_big.info, M_DEVBUF); } ss->rx_big.info = NULL; } static void mxge_free_rings(mxge_softc_t *sc) { int slice; for (slice = 0; slice < sc->num_slices; slice++) mxge_free_slice_rings(&sc->ss[slice]); } static int mxge_alloc_slice_rings(struct mxge_slice_state *ss, int rx_ring_entries, int tx_ring_entries) { mxge_softc_t *sc = ss->sc; size_t bytes; int err, i; err = ENOMEM; /* allocate per-slice receive resources */ ss->rx_small.mask = ss->rx_big.mask = rx_ring_entries - 1; ss->rx_done.mask = (2 * rx_ring_entries) - 1; /* allocate the rx shadow rings */ bytes = rx_ring_entries * sizeof (*ss->rx_small.shadow); ss->rx_small.shadow = malloc(bytes, M_DEVBUF, M_ZERO|M_WAITOK); if (ss->rx_small.shadow == NULL) return err; bytes = rx_ring_entries * sizeof (*ss->rx_big.shadow); ss->rx_big.shadow = malloc(bytes, M_DEVBUF, M_ZERO|M_WAITOK); if (ss->rx_big.shadow == NULL) return err; /* allocate the rx host info rings */ bytes = rx_ring_entries * sizeof (*ss->rx_small.info); ss->rx_small.info = malloc(bytes, M_DEVBUF, M_ZERO|M_WAITOK); if (ss->rx_small.info == NULL) return err; bytes = rx_ring_entries * sizeof (*ss->rx_big.info); ss->rx_big.info = malloc(bytes, M_DEVBUF, M_ZERO|M_WAITOK); if (ss->rx_big.info == NULL) return err; /* allocate the rx busdma resources */ err = bus_dma_tag_create(sc->parent_dmat, /* parent */ 1, /* alignment */ 4096, /* boundary */ BUS_SPACE_MAXADDR, /* low */ BUS_SPACE_MAXADDR, /* high */ NULL, NULL, /* filter */ MHLEN, /* maxsize */ 1, /* num segs */ MHLEN, /* maxsegsize */ BUS_DMA_ALLOCNOW, /* flags */ NULL, NULL, /* lock */ &ss->rx_small.dmat); /* tag */ if (err != 0) { device_printf(sc->dev, "Err %d allocating rx_small dmat\n", err); return err; } err = bus_dma_tag_create(sc->parent_dmat, /* parent */ 1, /* alignment */ #if MXGE_VIRT_JUMBOS 4096, /* boundary */ #else 0, /* boundary */ #endif BUS_SPACE_MAXADDR, /* low */ BUS_SPACE_MAXADDR, /* high */ NULL, NULL, /* filter */ 3*4096, /* maxsize */ #if MXGE_VIRT_JUMBOS 3, /* num segs */ 4096, /* maxsegsize*/ #else 1, /* num segs */ MJUM9BYTES, /* maxsegsize*/ #endif BUS_DMA_ALLOCNOW, /* flags */ NULL, NULL, /* lock */ &ss->rx_big.dmat); /* tag */ if (err != 0) { device_printf(sc->dev, "Err %d allocating rx_big dmat\n", err); return err; } for (i = 0; i <= ss->rx_small.mask; i++) { err = bus_dmamap_create(ss->rx_small.dmat, 0, &ss->rx_small.info[i].map); if (err != 0) { device_printf(sc->dev, "Err %d rx_small dmamap\n", err); return err; } } err = bus_dmamap_create(ss->rx_small.dmat, 0, &ss->rx_small.extra_map); if (err != 0) { device_printf(sc->dev, "Err %d extra rx_small dmamap\n", err); return err; } for (i = 0; i <= ss->rx_big.mask; i++) { err = bus_dmamap_create(ss->rx_big.dmat, 0, &ss->rx_big.info[i].map); if (err != 0) { device_printf(sc->dev, "Err %d rx_big dmamap\n", err); return err; } } err = bus_dmamap_create(ss->rx_big.dmat, 0, &ss->rx_big.extra_map); if (err != 0) { device_printf(sc->dev, "Err %d extra rx_big dmamap\n", err); return err; } /* now allocate TX resouces */ #ifndef IFNET_BUF_RING /* only use a single TX ring for now */ if (ss != ss->sc->ss) return 0; #endif ss->tx.mask = tx_ring_entries - 1; ss->tx.max_desc = MIN(MXGE_MAX_SEND_DESC, tx_ring_entries / 4); /* allocate the tx request copy block */ bytes = 8 + sizeof (*ss->tx.req_list) * (ss->tx.max_desc + 4); ss->tx.req_bytes = malloc(bytes, M_DEVBUF, M_WAITOK); if (ss->tx.req_bytes == NULL) return err; /* ensure req_list entries are aligned to 8 bytes */ ss->tx.req_list = (mcp_kreq_ether_send_t *) ((unsigned long)(ss->tx.req_bytes + 7) & ~7UL); /* allocate the tx busdma segment list */ bytes = sizeof (*ss->tx.seg_list) * ss->tx.max_desc; ss->tx.seg_list = (bus_dma_segment_t *) malloc(bytes, M_DEVBUF, M_WAITOK); if (ss->tx.seg_list == NULL) return err; /* allocate the tx host info ring */ bytes = tx_ring_entries * sizeof (*ss->tx.info); ss->tx.info = malloc(bytes, M_DEVBUF, M_ZERO|M_WAITOK); if (ss->tx.info == NULL) return err; /* allocate the tx busdma resources */ err = bus_dma_tag_create(sc->parent_dmat, /* parent */ 1, /* alignment */ sc->tx_boundary, /* boundary */ BUS_SPACE_MAXADDR, /* low */ BUS_SPACE_MAXADDR, /* high */ NULL, NULL, /* filter */ 65536 + 256, /* maxsize */ ss->tx.max_desc - 2, /* num segs */ sc->tx_boundary, /* maxsegsz */ BUS_DMA_ALLOCNOW, /* flags */ NULL, NULL, /* lock */ &ss->tx.dmat); /* tag */ if (err != 0) { device_printf(sc->dev, "Err %d allocating tx dmat\n", err); return err; } /* now use these tags to setup dmamaps for each slot in the ring */ for (i = 0; i <= ss->tx.mask; i++) { err = bus_dmamap_create(ss->tx.dmat, 0, &ss->tx.info[i].map); if (err != 0) { device_printf(sc->dev, "Err %d tx dmamap\n", err); return err; } } return 0; } static int mxge_alloc_rings(mxge_softc_t *sc) { mxge_cmd_t cmd; int tx_ring_size; int tx_ring_entries, rx_ring_entries; int err, slice; /* get ring sizes */ err = mxge_send_cmd(sc, MXGEFW_CMD_GET_SEND_RING_SIZE, &cmd); tx_ring_size = cmd.data0; if (err != 0) { device_printf(sc->dev, "Cannot determine tx ring sizes\n"); goto abort; } tx_ring_entries = tx_ring_size / sizeof (mcp_kreq_ether_send_t); rx_ring_entries = sc->rx_ring_size / sizeof (mcp_dma_addr_t); IFQ_SET_MAXLEN(&sc->ifp->if_snd, tx_ring_entries - 1); sc->ifp->if_snd.ifq_drv_maxlen = sc->ifp->if_snd.ifq_maxlen; IFQ_SET_READY(&sc->ifp->if_snd); for (slice = 0; slice < sc->num_slices; slice++) { err = mxge_alloc_slice_rings(&sc->ss[slice], rx_ring_entries, tx_ring_entries); if (err != 0) goto abort; } return 0; abort: mxge_free_rings(sc); return err; } static void mxge_choose_params(int mtu, int *big_buf_size, int *cl_size, int *nbufs) { int bufsize = mtu + ETHER_HDR_LEN + ETHER_VLAN_ENCAP_LEN + MXGEFW_PAD; if (bufsize < MCLBYTES) { /* easy, everything fits in a single buffer */ *big_buf_size = MCLBYTES; *cl_size = MCLBYTES; *nbufs = 1; return; } if (bufsize < MJUMPAGESIZE) { /* still easy, everything still fits in a single buffer */ *big_buf_size = MJUMPAGESIZE; *cl_size = MJUMPAGESIZE; *nbufs = 1; return; } #if MXGE_VIRT_JUMBOS /* now we need to use virtually contiguous buffers */ *cl_size = MJUM9BYTES; *big_buf_size = 4096; *nbufs = mtu / 4096 + 1; /* needs to be a power of two, so round up */ if (*nbufs == 3) *nbufs = 4; #else *cl_size = MJUM9BYTES; *big_buf_size = MJUM9BYTES; *nbufs = 1; #endif } static int mxge_slice_open(struct mxge_slice_state *ss, int nbufs, int cl_size) { mxge_softc_t *sc; mxge_cmd_t cmd; bus_dmamap_t map; struct lro_entry *lro_entry; int err, i, slice; sc = ss->sc; slice = ss - sc->ss; SLIST_INIT(&ss->lro_free); SLIST_INIT(&ss->lro_active); for (i = 0; i < sc->lro_cnt; i++) { lro_entry = (struct lro_entry *) malloc(sizeof (*lro_entry), M_DEVBUF, M_NOWAIT | M_ZERO); if (lro_entry == NULL) { sc->lro_cnt = i; break; } SLIST_INSERT_HEAD(&ss->lro_free, lro_entry, next); } /* get the lanai pointers to the send and receive rings */ err = 0; #ifndef IFNET_BUF_RING /* We currently only send from the first slice */ if (slice == 0) { #endif cmd.data0 = slice; err = mxge_send_cmd(sc, MXGEFW_CMD_GET_SEND_OFFSET, &cmd); ss->tx.lanai = (volatile mcp_kreq_ether_send_t *)(sc->sram + cmd.data0); ss->tx.send_go = (volatile uint32_t *) (sc->sram + MXGEFW_ETH_SEND_GO + 64 * slice); ss->tx.send_stop = (volatile uint32_t *) (sc->sram + MXGEFW_ETH_SEND_STOP + 64 * slice); #ifndef IFNET_BUF_RING } #endif cmd.data0 = slice; err |= mxge_send_cmd(sc, MXGEFW_CMD_GET_SMALL_RX_OFFSET, &cmd); ss->rx_small.lanai = (volatile mcp_kreq_ether_recv_t *)(sc->sram + cmd.data0); cmd.data0 = slice; err |= mxge_send_cmd(sc, MXGEFW_CMD_GET_BIG_RX_OFFSET, &cmd); ss->rx_big.lanai = (volatile mcp_kreq_ether_recv_t *)(sc->sram + cmd.data0); if (err != 0) { device_printf(sc->dev, "failed to get ring sizes or locations\n"); return EIO; } /* stock receive rings */ for (i = 0; i <= ss->rx_small.mask; i++) { map = ss->rx_small.info[i].map; err = mxge_get_buf_small(ss, map, i); if (err) { device_printf(sc->dev, "alloced %d/%d smalls\n", i, ss->rx_small.mask + 1); return ENOMEM; } } for (i = 0; i <= ss->rx_big.mask; i++) { ss->rx_big.shadow[i].addr_low = 0xffffffff; ss->rx_big.shadow[i].addr_high = 0xffffffff; } ss->rx_big.nbufs = nbufs; ss->rx_big.cl_size = cl_size; ss->rx_big.mlen = ss->sc->ifp->if_mtu + ETHER_HDR_LEN + ETHER_VLAN_ENCAP_LEN + MXGEFW_PAD; for (i = 0; i <= ss->rx_big.mask; i += ss->rx_big.nbufs) { map = ss->rx_big.info[i].map; err = mxge_get_buf_big(ss, map, i); if (err) { device_printf(sc->dev, "alloced %d/%d bigs\n", i, ss->rx_big.mask + 1); return ENOMEM; } } return 0; } static int mxge_open(mxge_softc_t *sc) { mxge_cmd_t cmd; int err, big_bytes, nbufs, slice, cl_size, i; bus_addr_t bus; volatile uint8_t *itable; struct mxge_slice_state *ss; /* Copy the MAC address in case it was overridden */ bcopy(IF_LLADDR(sc->ifp), sc->mac_addr, ETHER_ADDR_LEN); err = mxge_reset(sc, 1); if (err != 0) { device_printf(sc->dev, "failed to reset\n"); return EIO; } if (sc->num_slices > 1) { /* setup the indirection table */ cmd.data0 = sc->num_slices; err = mxge_send_cmd(sc, MXGEFW_CMD_SET_RSS_TABLE_SIZE, &cmd); err |= mxge_send_cmd(sc, MXGEFW_CMD_GET_RSS_TABLE_OFFSET, &cmd); if (err != 0) { device_printf(sc->dev, "failed to setup rss tables\n"); return err; } /* just enable an identity mapping */ itable = sc->sram + cmd.data0; for (i = 0; i < sc->num_slices; i++) itable[i] = (uint8_t)i; cmd.data0 = 1; cmd.data1 = mxge_rss_hash_type; err = mxge_send_cmd(sc, MXGEFW_CMD_SET_RSS_ENABLE, &cmd); if (err != 0) { device_printf(sc->dev, "failed to enable slices\n"); return err; } } mxge_choose_params(sc->ifp->if_mtu, &big_bytes, &cl_size, &nbufs); cmd.data0 = nbufs; err = mxge_send_cmd(sc, MXGEFW_CMD_ALWAYS_USE_N_BIG_BUFFERS, &cmd); /* error is only meaningful if we're trying to set MXGEFW_CMD_ALWAYS_USE_N_BIG_BUFFERS > 1 */ if (err && nbufs > 1) { device_printf(sc->dev, "Failed to set alway-use-n to %d\n", nbufs); return EIO; } /* Give the firmware the mtu and the big and small buffer sizes. The firmware wants the big buf size to be a power of two. Luckily, FreeBSD's clusters are powers of two */ cmd.data0 = sc->ifp->if_mtu + ETHER_HDR_LEN + ETHER_VLAN_ENCAP_LEN; err = mxge_send_cmd(sc, MXGEFW_CMD_SET_MTU, &cmd); cmd.data0 = MHLEN - MXGEFW_PAD; err |= mxge_send_cmd(sc, MXGEFW_CMD_SET_SMALL_BUFFER_SIZE, &cmd); cmd.data0 = big_bytes; err |= mxge_send_cmd(sc, MXGEFW_CMD_SET_BIG_BUFFER_SIZE, &cmd); if (err != 0) { device_printf(sc->dev, "failed to setup params\n"); goto abort; } /* Now give him the pointer to the stats block */ for (slice = 0; #ifdef IFNET_BUF_RING slice < sc->num_slices; #else slice < 1; #endif slice++) { ss = &sc->ss[slice]; cmd.data0 = MXGE_LOWPART_TO_U32(ss->fw_stats_dma.bus_addr); cmd.data1 = MXGE_HIGHPART_TO_U32(ss->fw_stats_dma.bus_addr); cmd.data2 = sizeof(struct mcp_irq_data); cmd.data2 |= (slice << 16); err |= mxge_send_cmd(sc, MXGEFW_CMD_SET_STATS_DMA_V2, &cmd); } if (err != 0) { bus = sc->ss->fw_stats_dma.bus_addr; bus += offsetof(struct mcp_irq_data, send_done_count); cmd.data0 = MXGE_LOWPART_TO_U32(bus); cmd.data1 = MXGE_HIGHPART_TO_U32(bus); err = mxge_send_cmd(sc, MXGEFW_CMD_SET_STATS_DMA_OBSOLETE, &cmd); /* Firmware cannot support multicast without STATS_DMA_V2 */ sc->fw_multicast_support = 0; } else { sc->fw_multicast_support = 1; } if (err != 0) { device_printf(sc->dev, "failed to setup params\n"); goto abort; } for (slice = 0; slice < sc->num_slices; slice++) { err = mxge_slice_open(&sc->ss[slice], nbufs, cl_size); if (err != 0) { device_printf(sc->dev, "couldn't open slice %d\n", slice); goto abort; } } /* Finally, start the firmware running */ err = mxge_send_cmd(sc, MXGEFW_CMD_ETHERNET_UP, &cmd); if (err) { device_printf(sc->dev, "Couldn't bring up link\n"); goto abort; } #ifdef IFNET_BUF_RING for (slice = 0; slice < sc->num_slices; slice++) { ss = &sc->ss[slice]; ss->if_drv_flags |= IFF_DRV_RUNNING; ss->if_drv_flags &= ~IFF_DRV_OACTIVE; } #endif sc->ifp->if_drv_flags |= IFF_DRV_RUNNING; sc->ifp->if_drv_flags &= ~IFF_DRV_OACTIVE; return 0; abort: mxge_free_mbufs(sc); return err; } static int mxge_close(mxge_softc_t *sc, int down) { mxge_cmd_t cmd; int err, old_down_cnt; #ifdef IFNET_BUF_RING struct mxge_slice_state *ss; int slice; #endif #ifdef IFNET_BUF_RING for (slice = 0; slice < sc->num_slices; slice++) { ss = &sc->ss[slice]; ss->if_drv_flags &= ~IFF_DRV_RUNNING; } #endif sc->ifp->if_drv_flags &= ~IFF_DRV_RUNNING; if (!down) { old_down_cnt = sc->down_cnt; wmb(); err = mxge_send_cmd(sc, MXGEFW_CMD_ETHERNET_DOWN, &cmd); if (err) { device_printf(sc->dev, "Couldn't bring down link\n"); } if (old_down_cnt == sc->down_cnt) { /* wait for down irq */ DELAY(10 * sc->intr_coal_delay); } wmb(); if (old_down_cnt == sc->down_cnt) { device_printf(sc->dev, "never got down irq\n"); } } mxge_free_mbufs(sc); return 0; } static void mxge_setup_cfg_space(mxge_softc_t *sc) { device_t dev = sc->dev; int reg; uint16_t cmd, lnk, pectl; /* find the PCIe link width and set max read request to 4KB*/ if (pci_find_extcap(dev, PCIY_EXPRESS, ®) == 0) { lnk = pci_read_config(dev, reg + 0x12, 2); sc->link_width = (lnk >> 4) & 0x3f; if (sc->pectl == 0) { pectl = pci_read_config(dev, reg + 0x8, 2); pectl = (pectl & ~0x7000) | (5 << 12); pci_write_config(dev, reg + 0x8, pectl, 2); sc->pectl = pectl; } else { /* restore saved pectl after watchdog reset */ pci_write_config(dev, reg + 0x8, sc->pectl, 2); } } /* Enable DMA and Memory space access */ pci_enable_busmaster(dev); cmd = pci_read_config(dev, PCIR_COMMAND, 2); cmd |= PCIM_CMD_MEMEN; pci_write_config(dev, PCIR_COMMAND, cmd, 2); } static uint32_t mxge_read_reboot(mxge_softc_t *sc) { device_t dev = sc->dev; uint32_t vs; /* find the vendor specific offset */ if (pci_find_extcap(dev, PCIY_VENDOR, &vs) != 0) { device_printf(sc->dev, "could not find vendor specific offset\n"); return (uint32_t)-1; } /* enable read32 mode */ pci_write_config(dev, vs + 0x10, 0x3, 1); /* tell NIC which register to read */ pci_write_config(dev, vs + 0x18, 0xfffffff0, 4); return (pci_read_config(dev, vs + 0x14, 4)); } static void mxge_watchdog_reset(mxge_softc_t *sc) { struct pci_devinfo *dinfo; struct mxge_slice_state *ss; int err, running, s, num_tx_slices = 1; uint32_t reboot; uint16_t cmd; err = ENXIO; device_printf(sc->dev, "Watchdog reset!\n"); /* * check to see if the NIC rebooted. If it did, then all of * PCI config space has been reset, and things like the * busmaster bit will be zero. If this is the case, then we * must restore PCI config space before the NIC can be used * again */ cmd = pci_read_config(sc->dev, PCIR_COMMAND, 2); if (cmd == 0xffff) { /* * maybe the watchdog caught the NIC rebooting; wait * up to 100ms for it to finish. If it does not come * back, then give up */ DELAY(1000*100); cmd = pci_read_config(sc->dev, PCIR_COMMAND, 2); if (cmd == 0xffff) { device_printf(sc->dev, "NIC disappeared!\n"); } } if ((cmd & PCIM_CMD_BUSMASTEREN) == 0) { /* print the reboot status */ reboot = mxge_read_reboot(sc); device_printf(sc->dev, "NIC rebooted, status = 0x%x\n", reboot); running = sc->ifp->if_drv_flags & IFF_DRV_RUNNING; if (running) { /* * quiesce NIC so that TX routines will not try to * xmit after restoration of BAR */ /* Mark the link as down */ if (sc->link_state) { sc->link_state = 0; if_link_state_change(sc->ifp, LINK_STATE_DOWN); } #ifdef IFNET_BUF_RING num_tx_slices = sc->num_slices; #endif /* grab all TX locks to ensure no tx */ for (s = 0; s < num_tx_slices; s++) { ss = &sc->ss[s]; mtx_lock(&ss->tx.mtx); } mxge_close(sc, 1); } /* restore PCI configuration space */ dinfo = device_get_ivars(sc->dev); pci_cfg_restore(sc->dev, dinfo); /* and redo any changes we made to our config space */ mxge_setup_cfg_space(sc); /* reload f/w */ err = mxge_load_firmware(sc, 0); if (err) { device_printf(sc->dev, "Unable to re-load f/w\n"); } if (running) { if (!err) err = mxge_open(sc); /* release all TX locks */ for (s = 0; s < num_tx_slices; s++) { ss = &sc->ss[s]; #ifdef IFNET_BUF_RING mxge_start_locked(ss); #endif mtx_unlock(&ss->tx.mtx); } } sc->watchdog_resets++; } else { device_printf(sc->dev, "NIC did not reboot, not resetting\n"); err = 0; } if (err) { device_printf(sc->dev, "watchdog reset failed\n"); } else { if (sc->dying == 2) sc->dying = 0; callout_reset(&sc->co_hdl, mxge_ticks, mxge_tick, sc); } } static void mxge_watchdog_task(void *arg, int pending) { mxge_softc_t *sc = arg; mtx_lock(&sc->driver_mtx); mxge_watchdog_reset(sc); mtx_unlock(&sc->driver_mtx); } static void mxge_warn_stuck(mxge_softc_t *sc, mxge_tx_ring_t *tx, int slice) { tx = &sc->ss[slice].tx; device_printf(sc->dev, "slice %d struck? ring state:\n", slice); device_printf(sc->dev, "tx.req=%d tx.done=%d, tx.queue_active=%d\n", tx->req, tx->done, tx->queue_active); device_printf(sc->dev, "tx.activate=%d tx.deactivate=%d\n", tx->activate, tx->deactivate); device_printf(sc->dev, "pkt_done=%d fw=%d\n", tx->pkt_done, be32toh(sc->ss->fw_stats->send_done_count)); } static int mxge_watchdog(mxge_softc_t *sc) { mxge_tx_ring_t *tx; uint32_t rx_pause = be32toh(sc->ss->fw_stats->dropped_pause); int i, err = 0; /* see if we have outstanding transmits, which have been pending for more than mxge_ticks */ for (i = 0; #ifdef IFNET_BUF_RING (i < sc->num_slices) && (err == 0); #else (i < 1) && (err == 0); #endif i++) { tx = &sc->ss[i].tx; if (tx->req != tx->done && tx->watchdog_req != tx->watchdog_done && tx->done == tx->watchdog_done) { /* check for pause blocking before resetting */ if (tx->watchdog_rx_pause == rx_pause) { mxge_warn_stuck(sc, tx, i); taskqueue_enqueue(sc->tq, &sc->watchdog_task); return (ENXIO); } else device_printf(sc->dev, "Flow control blocking " "xmits, check link partner\n"); } tx->watchdog_req = tx->req; tx->watchdog_done = tx->done; tx->watchdog_rx_pause = rx_pause; } if (sc->need_media_probe) mxge_media_probe(sc); return (err); } static u_long mxge_update_stats(mxge_softc_t *sc) { struct mxge_slice_state *ss; u_long pkts = 0; u_long ipackets = 0; u_long opackets = 0; #ifdef IFNET_BUF_RING u_long obytes = 0; u_long omcasts = 0; u_long odrops = 0; #endif u_long oerrors = 0; int slice; for (slice = 0; slice < sc->num_slices; slice++) { ss = &sc->ss[slice]; ipackets += ss->ipackets; opackets += ss->opackets; #ifdef IFNET_BUF_RING obytes += ss->obytes; omcasts += ss->omcasts; odrops += ss->tx.br->br_drops; #endif oerrors += ss->oerrors; } pkts = (ipackets - sc->ifp->if_ipackets); pkts += (opackets - sc->ifp->if_opackets); sc->ifp->if_ipackets = ipackets; sc->ifp->if_opackets = opackets; #ifdef IFNET_BUF_RING sc->ifp->if_obytes = obytes; sc->ifp->if_omcasts = omcasts; sc->ifp->if_snd.ifq_drops = odrops; #endif sc->ifp->if_oerrors = oerrors; return pkts; } static void mxge_tick(void *arg) { mxge_softc_t *sc = arg; u_long pkts = 0; int err = 0; int running, ticks; uint16_t cmd; ticks = mxge_ticks; running = sc->ifp->if_drv_flags & IFF_DRV_RUNNING; if (running) { /* aggregate stats from different slices */ pkts = mxge_update_stats(sc); if (!sc->watchdog_countdown) { err = mxge_watchdog(sc); sc->watchdog_countdown = 4; } sc->watchdog_countdown--; } if (pkts == 0) { /* ensure NIC did not suffer h/w fault while idle */ cmd = pci_read_config(sc->dev, PCIR_COMMAND, 2); if ((cmd & PCIM_CMD_BUSMASTEREN) == 0) { sc->dying = 2; taskqueue_enqueue(sc->tq, &sc->watchdog_task); err = ENXIO; } /* look less often if NIC is idle */ ticks *= 4; } if (err == 0) callout_reset(&sc->co_hdl, ticks, mxge_tick, sc); } static int mxge_media_change(struct ifnet *ifp) { return EINVAL; } static int mxge_change_mtu(mxge_softc_t *sc, int mtu) { struct ifnet *ifp = sc->ifp; int real_mtu, old_mtu; int err = 0; real_mtu = mtu + ETHER_HDR_LEN + ETHER_VLAN_ENCAP_LEN; if ((real_mtu > sc->max_mtu) || real_mtu < 60) return EINVAL; mtx_lock(&sc->driver_mtx); old_mtu = ifp->if_mtu; ifp->if_mtu = mtu; if (ifp->if_drv_flags & IFF_DRV_RUNNING) { mxge_close(sc, 0); err = mxge_open(sc); if (err != 0) { ifp->if_mtu = old_mtu; mxge_close(sc, 0); (void) mxge_open(sc); } } mtx_unlock(&sc->driver_mtx); return err; } static void mxge_media_status(struct ifnet *ifp, struct ifmediareq *ifmr) { mxge_softc_t *sc = ifp->if_softc; if (sc == NULL) return; ifmr->ifm_status = IFM_AVALID; + ifmr->ifm_active = IFM_ETHER | IFM_FDX; ifmr->ifm_status |= sc->link_state ? IFM_ACTIVE : 0; - ifmr->ifm_active = IFM_AUTO | IFM_ETHER; - ifmr->ifm_active |= sc->link_state ? IFM_FDX : 0; + ifmr->ifm_active |= sc->current_media; } static int mxge_ioctl(struct ifnet *ifp, u_long command, caddr_t data) { mxge_softc_t *sc = ifp->if_softc; struct ifreq *ifr = (struct ifreq *)data; int err, mask; err = 0; switch (command) { case SIOCSIFADDR: case SIOCGIFADDR: err = ether_ioctl(ifp, command, data); break; case SIOCSIFMTU: err = mxge_change_mtu(sc, ifr->ifr_mtu); break; case SIOCSIFFLAGS: mtx_lock(&sc->driver_mtx); if (sc->dying) { mtx_unlock(&sc->driver_mtx); return EINVAL; } if (ifp->if_flags & IFF_UP) { if (!(ifp->if_drv_flags & IFF_DRV_RUNNING)) { err = mxge_open(sc); } else { /* take care of promis can allmulti flag chages */ mxge_change_promisc(sc, ifp->if_flags & IFF_PROMISC); mxge_set_multicast_list(sc); } } else { if (ifp->if_drv_flags & IFF_DRV_RUNNING) { mxge_close(sc, 0); } } mtx_unlock(&sc->driver_mtx); break; case SIOCADDMULTI: case SIOCDELMULTI: mtx_lock(&sc->driver_mtx); mxge_set_multicast_list(sc); mtx_unlock(&sc->driver_mtx); break; case SIOCSIFCAP: mtx_lock(&sc->driver_mtx); mask = ifr->ifr_reqcap ^ ifp->if_capenable; if (mask & IFCAP_TXCSUM) { if (IFCAP_TXCSUM & ifp->if_capenable) { ifp->if_capenable &= ~(IFCAP_TXCSUM|IFCAP_TSO4); ifp->if_hwassist &= ~(CSUM_TCP | CSUM_UDP | CSUM_TSO); } else { ifp->if_capenable |= IFCAP_TXCSUM; ifp->if_hwassist |= (CSUM_TCP | CSUM_UDP); } } else if (mask & IFCAP_RXCSUM) { if (IFCAP_RXCSUM & ifp->if_capenable) { ifp->if_capenable &= ~IFCAP_RXCSUM; sc->csum_flag = 0; } else { ifp->if_capenable |= IFCAP_RXCSUM; sc->csum_flag = 1; } } if (mask & IFCAP_TSO4) { if (IFCAP_TSO4 & ifp->if_capenable) { ifp->if_capenable &= ~IFCAP_TSO4; ifp->if_hwassist &= ~CSUM_TSO; } else if (IFCAP_TXCSUM & ifp->if_capenable) { ifp->if_capenable |= IFCAP_TSO4; ifp->if_hwassist |= CSUM_TSO; } else { printf("mxge requires tx checksum offload" " be enabled to use TSO\n"); err = EINVAL; } } if (mask & IFCAP_LRO) { if (IFCAP_LRO & ifp->if_capenable) err = mxge_change_lro_locked(sc, 0); else err = mxge_change_lro_locked(sc, mxge_lro_cnt); } if (mask & IFCAP_VLAN_HWTAGGING) ifp->if_capenable ^= IFCAP_VLAN_HWTAGGING; if (mask & IFCAP_VLAN_HWTSO) ifp->if_capenable ^= IFCAP_VLAN_HWTSO; if (!(ifp->if_capabilities & IFCAP_VLAN_HWTSO) || !(ifp->if_capenable & IFCAP_VLAN_HWTAGGING)) ifp->if_capenable &= ~IFCAP_VLAN_HWTSO; mtx_unlock(&sc->driver_mtx); VLAN_CAPABILITIES(ifp); break; case SIOCGIFMEDIA: + mtx_lock(&sc->driver_mtx); + mxge_media_probe(sc); + mtx_unlock(&sc->driver_mtx); err = ifmedia_ioctl(ifp, (struct ifreq *)data, &sc->media, command); break; default: err = ENOTTY; } return err; } static void mxge_fetch_tunables(mxge_softc_t *sc) { TUNABLE_INT_FETCH("hw.mxge.max_slices", &mxge_max_slices); TUNABLE_INT_FETCH("hw.mxge.flow_control_enabled", &mxge_flow_control); TUNABLE_INT_FETCH("hw.mxge.intr_coal_delay", &mxge_intr_coal_delay); TUNABLE_INT_FETCH("hw.mxge.nvidia_ecrc_enable", &mxge_nvidia_ecrc_enable); TUNABLE_INT_FETCH("hw.mxge.force_firmware", &mxge_force_firmware); TUNABLE_INT_FETCH("hw.mxge.deassert_wait", &mxge_deassert_wait); TUNABLE_INT_FETCH("hw.mxge.verbose", &mxge_verbose); TUNABLE_INT_FETCH("hw.mxge.ticks", &mxge_ticks); TUNABLE_INT_FETCH("hw.mxge.lro_cnt", &sc->lro_cnt); TUNABLE_INT_FETCH("hw.mxge.always_promisc", &mxge_always_promisc); TUNABLE_INT_FETCH("hw.mxge.rss_hash_type", &mxge_rss_hash_type); TUNABLE_INT_FETCH("hw.mxge.rss_hashtype", &mxge_rss_hash_type); TUNABLE_INT_FETCH("hw.mxge.initial_mtu", &mxge_initial_mtu); TUNABLE_INT_FETCH("hw.mxge.throttle", &mxge_throttle); if (sc->lro_cnt != 0) mxge_lro_cnt = sc->lro_cnt; if (bootverbose) mxge_verbose = 1; if (mxge_intr_coal_delay < 0 || mxge_intr_coal_delay > 10*1000) mxge_intr_coal_delay = 30; if (mxge_ticks == 0) mxge_ticks = hz / 2; sc->pause = mxge_flow_control; if (mxge_rss_hash_type < MXGEFW_RSS_HASH_TYPE_IPV4 || mxge_rss_hash_type > MXGEFW_RSS_HASH_TYPE_MAX) { mxge_rss_hash_type = MXGEFW_RSS_HASH_TYPE_SRC_DST_PORT; } if (mxge_initial_mtu > ETHERMTU_JUMBO || mxge_initial_mtu < ETHER_MIN_LEN) mxge_initial_mtu = ETHERMTU_JUMBO; if (mxge_throttle && mxge_throttle > MXGE_MAX_THROTTLE) mxge_throttle = MXGE_MAX_THROTTLE; if (mxge_throttle && mxge_throttle < MXGE_MIN_THROTTLE) mxge_throttle = MXGE_MIN_THROTTLE; sc->throttle = mxge_throttle; } static void mxge_free_slices(mxge_softc_t *sc) { struct mxge_slice_state *ss; int i; if (sc->ss == NULL) return; for (i = 0; i < sc->num_slices; i++) { ss = &sc->ss[i]; if (ss->fw_stats != NULL) { mxge_dma_free(&ss->fw_stats_dma); ss->fw_stats = NULL; #ifdef IFNET_BUF_RING if (ss->tx.br != NULL) { drbr_free(ss->tx.br, M_DEVBUF); ss->tx.br = NULL; } #endif mtx_destroy(&ss->tx.mtx); } if (ss->rx_done.entry != NULL) { mxge_dma_free(&ss->rx_done.dma); ss->rx_done.entry = NULL; } } free(sc->ss, M_DEVBUF); sc->ss = NULL; } static int mxge_alloc_slices(mxge_softc_t *sc) { mxge_cmd_t cmd; struct mxge_slice_state *ss; size_t bytes; int err, i, max_intr_slots; err = mxge_send_cmd(sc, MXGEFW_CMD_GET_RX_RING_SIZE, &cmd); if (err != 0) { device_printf(sc->dev, "Cannot determine rx ring size\n"); return err; } sc->rx_ring_size = cmd.data0; max_intr_slots = 2 * (sc->rx_ring_size / sizeof (mcp_dma_addr_t)); bytes = sizeof (*sc->ss) * sc->num_slices; sc->ss = malloc(bytes, M_DEVBUF, M_NOWAIT | M_ZERO); if (sc->ss == NULL) return (ENOMEM); for (i = 0; i < sc->num_slices; i++) { ss = &sc->ss[i]; ss->sc = sc; /* allocate per-slice rx interrupt queues */ bytes = max_intr_slots * sizeof (*ss->rx_done.entry); err = mxge_dma_alloc(sc, &ss->rx_done.dma, bytes, 4096); if (err != 0) goto abort; ss->rx_done.entry = ss->rx_done.dma.addr; bzero(ss->rx_done.entry, bytes); /* * allocate the per-slice firmware stats; stats * (including tx) are used used only on the first * slice for now */ #ifndef IFNET_BUF_RING if (i > 0) continue; #endif bytes = sizeof (*ss->fw_stats); err = mxge_dma_alloc(sc, &ss->fw_stats_dma, sizeof (*ss->fw_stats), 64); if (err != 0) goto abort; ss->fw_stats = (mcp_irq_data_t *)ss->fw_stats_dma.addr; snprintf(ss->tx.mtx_name, sizeof(ss->tx.mtx_name), "%s:tx(%d)", device_get_nameunit(sc->dev), i); mtx_init(&ss->tx.mtx, ss->tx.mtx_name, NULL, MTX_DEF); #ifdef IFNET_BUF_RING ss->tx.br = buf_ring_alloc(2048, M_DEVBUF, M_WAITOK, &ss->tx.mtx); #endif } return (0); abort: mxge_free_slices(sc); return (ENOMEM); } static void mxge_slice_probe(mxge_softc_t *sc) { mxge_cmd_t cmd; char *old_fw; int msix_cnt, status, max_intr_slots; sc->num_slices = 1; /* * don't enable multiple slices if they are not enabled, * or if this is not an SMP system */ if (mxge_max_slices == 0 || mxge_max_slices == 1 || mp_ncpus < 2) return; /* see how many MSI-X interrupts are available */ msix_cnt = pci_msix_count(sc->dev); if (msix_cnt < 2) return; /* now load the slice aware firmware see what it supports */ old_fw = sc->fw_name; if (old_fw == mxge_fw_aligned) sc->fw_name = mxge_fw_rss_aligned; else sc->fw_name = mxge_fw_rss_unaligned; status = mxge_load_firmware(sc, 0); if (status != 0) { device_printf(sc->dev, "Falling back to a single slice\n"); return; } /* try to send a reset command to the card to see if it is alive */ memset(&cmd, 0, sizeof (cmd)); status = mxge_send_cmd(sc, MXGEFW_CMD_RESET, &cmd); if (status != 0) { device_printf(sc->dev, "failed reset\n"); goto abort_with_fw; } /* get rx ring size */ status = mxge_send_cmd(sc, MXGEFW_CMD_GET_RX_RING_SIZE, &cmd); if (status != 0) { device_printf(sc->dev, "Cannot determine rx ring size\n"); goto abort_with_fw; } max_intr_slots = 2 * (cmd.data0 / sizeof (mcp_dma_addr_t)); /* tell it the size of the interrupt queues */ cmd.data0 = max_intr_slots * sizeof (struct mcp_slot); status = mxge_send_cmd(sc, MXGEFW_CMD_SET_INTRQ_SIZE, &cmd); if (status != 0) { device_printf(sc->dev, "failed MXGEFW_CMD_SET_INTRQ_SIZE\n"); goto abort_with_fw; } /* ask the maximum number of slices it supports */ status = mxge_send_cmd(sc, MXGEFW_CMD_GET_MAX_RSS_QUEUES, &cmd); if (status != 0) { device_printf(sc->dev, "failed MXGEFW_CMD_GET_MAX_RSS_QUEUES\n"); goto abort_with_fw; } sc->num_slices = cmd.data0; if (sc->num_slices > msix_cnt) sc->num_slices = msix_cnt; if (mxge_max_slices == -1) { /* cap to number of CPUs in system */ if (sc->num_slices > mp_ncpus) sc->num_slices = mp_ncpus; } else { if (sc->num_slices > mxge_max_slices) sc->num_slices = mxge_max_slices; } /* make sure it is a power of two */ while (sc->num_slices & (sc->num_slices - 1)) sc->num_slices--; if (mxge_verbose) device_printf(sc->dev, "using %d slices\n", sc->num_slices); return; abort_with_fw: sc->fw_name = old_fw; (void) mxge_load_firmware(sc, 0); } static int mxge_add_msix_irqs(mxge_softc_t *sc) { size_t bytes; int count, err, i, rid; rid = PCIR_BAR(2); sc->msix_table_res = bus_alloc_resource_any(sc->dev, SYS_RES_MEMORY, &rid, RF_ACTIVE); if (sc->msix_table_res == NULL) { device_printf(sc->dev, "couldn't alloc MSIX table res\n"); return ENXIO; } count = sc->num_slices; err = pci_alloc_msix(sc->dev, &count); if (err != 0) { device_printf(sc->dev, "pci_alloc_msix: failed, wanted %d" "err = %d \n", sc->num_slices, err); goto abort_with_msix_table; } if (count < sc->num_slices) { device_printf(sc->dev, "pci_alloc_msix: need %d, got %d\n", count, sc->num_slices); device_printf(sc->dev, "Try setting hw.mxge.max_slices to %d\n", count); err = ENOSPC; goto abort_with_msix; } bytes = sizeof (*sc->msix_irq_res) * sc->num_slices; sc->msix_irq_res = malloc(bytes, M_DEVBUF, M_NOWAIT|M_ZERO); if (sc->msix_irq_res == NULL) { err = ENOMEM; goto abort_with_msix; } for (i = 0; i < sc->num_slices; i++) { rid = i + 1; sc->msix_irq_res[i] = bus_alloc_resource_any(sc->dev, SYS_RES_IRQ, &rid, RF_ACTIVE); if (sc->msix_irq_res[i] == NULL) { device_printf(sc->dev, "couldn't allocate IRQ res" " for message %d\n", i); err = ENXIO; goto abort_with_res; } } bytes = sizeof (*sc->msix_ih) * sc->num_slices; sc->msix_ih = malloc(bytes, M_DEVBUF, M_NOWAIT|M_ZERO); for (i = 0; i < sc->num_slices; i++) { err = bus_setup_intr(sc->dev, sc->msix_irq_res[i], INTR_TYPE_NET | INTR_MPSAFE, #if __FreeBSD_version > 700030 NULL, #endif mxge_intr, &sc->ss[i], &sc->msix_ih[i]); if (err != 0) { device_printf(sc->dev, "couldn't setup intr for " "message %d\n", i); goto abort_with_intr; } } if (mxge_verbose) { device_printf(sc->dev, "using %d msix IRQs:", sc->num_slices); for (i = 0; i < sc->num_slices; i++) printf(" %ld", rman_get_start(sc->msix_irq_res[i])); printf("\n"); } return (0); abort_with_intr: for (i = 0; i < sc->num_slices; i++) { if (sc->msix_ih[i] != NULL) { bus_teardown_intr(sc->dev, sc->msix_irq_res[i], sc->msix_ih[i]); sc->msix_ih[i] = NULL; } } free(sc->msix_ih, M_DEVBUF); abort_with_res: for (i = 0; i < sc->num_slices; i++) { rid = i + 1; if (sc->msix_irq_res[i] != NULL) bus_release_resource(sc->dev, SYS_RES_IRQ, rid, sc->msix_irq_res[i]); sc->msix_irq_res[i] = NULL; } free(sc->msix_irq_res, M_DEVBUF); abort_with_msix: pci_release_msi(sc->dev); abort_with_msix_table: bus_release_resource(sc->dev, SYS_RES_MEMORY, PCIR_BAR(2), sc->msix_table_res); return err; } static int mxge_add_single_irq(mxge_softc_t *sc) { int count, err, rid; count = pci_msi_count(sc->dev); if (count == 1 && pci_alloc_msi(sc->dev, &count) == 0) { rid = 1; } else { rid = 0; sc->legacy_irq = 1; } sc->irq_res = bus_alloc_resource(sc->dev, SYS_RES_IRQ, &rid, 0, ~0, 1, RF_SHAREABLE | RF_ACTIVE); if (sc->irq_res == NULL) { device_printf(sc->dev, "could not alloc interrupt\n"); return ENXIO; } if (mxge_verbose) device_printf(sc->dev, "using %s irq %ld\n", sc->legacy_irq ? "INTx" : "MSI", rman_get_start(sc->irq_res)); err = bus_setup_intr(sc->dev, sc->irq_res, INTR_TYPE_NET | INTR_MPSAFE, #if __FreeBSD_version > 700030 NULL, #endif mxge_intr, &sc->ss[0], &sc->ih); if (err != 0) { bus_release_resource(sc->dev, SYS_RES_IRQ, sc->legacy_irq ? 0 : 1, sc->irq_res); if (!sc->legacy_irq) pci_release_msi(sc->dev); } return err; } static void mxge_rem_msix_irqs(mxge_softc_t *sc) { int i, rid; for (i = 0; i < sc->num_slices; i++) { if (sc->msix_ih[i] != NULL) { bus_teardown_intr(sc->dev, sc->msix_irq_res[i], sc->msix_ih[i]); sc->msix_ih[i] = NULL; } } free(sc->msix_ih, M_DEVBUF); for (i = 0; i < sc->num_slices; i++) { rid = i + 1; if (sc->msix_irq_res[i] != NULL) bus_release_resource(sc->dev, SYS_RES_IRQ, rid, sc->msix_irq_res[i]); sc->msix_irq_res[i] = NULL; } free(sc->msix_irq_res, M_DEVBUF); bus_release_resource(sc->dev, SYS_RES_MEMORY, PCIR_BAR(2), sc->msix_table_res); pci_release_msi(sc->dev); return; } static void mxge_rem_single_irq(mxge_softc_t *sc) { bus_teardown_intr(sc->dev, sc->irq_res, sc->ih); bus_release_resource(sc->dev, SYS_RES_IRQ, sc->legacy_irq ? 0 : 1, sc->irq_res); if (!sc->legacy_irq) pci_release_msi(sc->dev); } static void mxge_rem_irq(mxge_softc_t *sc) { if (sc->num_slices > 1) mxge_rem_msix_irqs(sc); else mxge_rem_single_irq(sc); } static int mxge_add_irq(mxge_softc_t *sc) { int err; if (sc->num_slices > 1) err = mxge_add_msix_irqs(sc); else err = mxge_add_single_irq(sc); if (0 && err == 0 && sc->num_slices > 1) { mxge_rem_msix_irqs(sc); err = mxge_add_msix_irqs(sc); } return err; } static int mxge_attach(device_t dev) { mxge_softc_t *sc = device_get_softc(dev); struct ifnet *ifp; int err, rid; sc->dev = dev; mxge_fetch_tunables(sc); TASK_INIT(&sc->watchdog_task, 1, mxge_watchdog_task, sc); sc->tq = taskqueue_create_fast("mxge_taskq", M_WAITOK, taskqueue_thread_enqueue, &sc->tq); if (sc->tq == NULL) { err = ENOMEM; goto abort_with_nothing; } err = bus_dma_tag_create(NULL, /* parent */ 1, /* alignment */ 0, /* boundary */ BUS_SPACE_MAXADDR, /* low */ BUS_SPACE_MAXADDR, /* high */ NULL, NULL, /* filter */ 65536 + 256, /* maxsize */ MXGE_MAX_SEND_DESC, /* num segs */ 65536, /* maxsegsize */ 0, /* flags */ NULL, NULL, /* lock */ &sc->parent_dmat); /* tag */ if (err != 0) { device_printf(sc->dev, "Err %d allocating parent dmat\n", err); goto abort_with_tq; } ifp = sc->ifp = if_alloc(IFT_ETHER); if (ifp == NULL) { device_printf(dev, "can not if_alloc()\n"); err = ENOSPC; goto abort_with_parent_dmat; } if_initname(ifp, device_get_name(dev), device_get_unit(dev)); snprintf(sc->cmd_mtx_name, sizeof(sc->cmd_mtx_name), "%s:cmd", device_get_nameunit(dev)); mtx_init(&sc->cmd_mtx, sc->cmd_mtx_name, NULL, MTX_DEF); snprintf(sc->driver_mtx_name, sizeof(sc->driver_mtx_name), "%s:drv", device_get_nameunit(dev)); mtx_init(&sc->driver_mtx, sc->driver_mtx_name, MTX_NETWORK_LOCK, MTX_DEF); callout_init_mtx(&sc->co_hdl, &sc->driver_mtx, 0); mxge_setup_cfg_space(sc); /* Map the board into the kernel */ rid = PCIR_BARS; sc->mem_res = bus_alloc_resource(dev, SYS_RES_MEMORY, &rid, 0, ~0, 1, RF_ACTIVE); if (sc->mem_res == NULL) { device_printf(dev, "could not map memory\n"); err = ENXIO; goto abort_with_lock; } sc->sram = rman_get_virtual(sc->mem_res); sc->sram_size = 2*1024*1024 - (2*(48*1024)+(32*1024)) - 0x100; if (sc->sram_size > rman_get_size(sc->mem_res)) { device_printf(dev, "impossible memory region size %ld\n", rman_get_size(sc->mem_res)); err = ENXIO; goto abort_with_mem_res; } /* make NULL terminated copy of the EEPROM strings section of lanai SRAM */ bzero(sc->eeprom_strings, MXGE_EEPROM_STRINGS_SIZE); bus_space_read_region_1(rman_get_bustag(sc->mem_res), rman_get_bushandle(sc->mem_res), sc->sram_size - MXGE_EEPROM_STRINGS_SIZE, sc->eeprom_strings, MXGE_EEPROM_STRINGS_SIZE - 2); err = mxge_parse_strings(sc); if (err != 0) goto abort_with_mem_res; /* Enable write combining for efficient use of PCIe bus */ mxge_enable_wc(sc); /* Allocate the out of band dma memory */ err = mxge_dma_alloc(sc, &sc->cmd_dma, sizeof (mxge_cmd_t), 64); if (err != 0) goto abort_with_mem_res; sc->cmd = (mcp_cmd_response_t *) sc->cmd_dma.addr; err = mxge_dma_alloc(sc, &sc->zeropad_dma, 64, 64); if (err != 0) goto abort_with_cmd_dma; err = mxge_dma_alloc(sc, &sc->dmabench_dma, 4096, 4096); if (err != 0) goto abort_with_zeropad_dma; /* select & load the firmware */ err = mxge_select_firmware(sc); if (err != 0) goto abort_with_dmabench; sc->intr_coal_delay = mxge_intr_coal_delay; mxge_slice_probe(sc); err = mxge_alloc_slices(sc); if (err != 0) goto abort_with_dmabench; err = mxge_reset(sc, 0); if (err != 0) goto abort_with_slices; err = mxge_alloc_rings(sc); if (err != 0) { device_printf(sc->dev, "failed to allocate rings\n"); goto abort_with_slices; } err = mxge_add_irq(sc); if (err != 0) { device_printf(sc->dev, "failed to add irq\n"); goto abort_with_rings; } ifp->if_baudrate = IF_Gbps(10UL); ifp->if_capabilities = IFCAP_RXCSUM | IFCAP_TXCSUM | IFCAP_TSO4 | - IFCAP_VLAN_MTU; + IFCAP_VLAN_MTU | IFCAP_LINKSTATE; #ifdef INET ifp->if_capabilities |= IFCAP_LRO; #endif #ifdef MXGE_NEW_VLAN_API ifp->if_capabilities |= IFCAP_VLAN_HWTAGGING | IFCAP_VLAN_HWCSUM; /* Only FW 1.4.32 and newer can do TSO over vlans */ if (sc->fw_ver_major == 1 && sc->fw_ver_minor == 4 && sc->fw_ver_tiny >= 32) ifp->if_capabilities |= IFCAP_VLAN_HWTSO; #endif sc->max_mtu = mxge_max_mtu(sc); if (sc->max_mtu >= 9000) ifp->if_capabilities |= IFCAP_JUMBO_MTU; else device_printf(dev, "MTU limited to %d. Install " "latest firmware for 9000 byte jumbo support\n", sc->max_mtu - ETHER_HDR_LEN); ifp->if_hwassist = CSUM_TCP | CSUM_UDP | CSUM_TSO; ifp->if_capenable = ifp->if_capabilities; if (sc->lro_cnt == 0) ifp->if_capenable &= ~IFCAP_LRO; sc->csum_flag = 1; ifp->if_init = mxge_init; ifp->if_softc = sc; ifp->if_flags = IFF_BROADCAST | IFF_SIMPLEX | IFF_MULTICAST; ifp->if_ioctl = mxge_ioctl; ifp->if_start = mxge_start; /* Initialise the ifmedia structure */ ifmedia_init(&sc->media, 0, mxge_media_change, mxge_media_status); - mxge_set_media(sc, IFM_ETHER | IFM_AUTO); + mxge_media_init(sc); mxge_media_probe(sc); sc->dying = 0; ether_ifattach(ifp, sc->mac_addr); /* ether_ifattach sets mtu to ETHERMTU */ if (mxge_initial_mtu != ETHERMTU) mxge_change_mtu(sc, mxge_initial_mtu); mxge_add_sysctls(sc); #ifdef IFNET_BUF_RING ifp->if_transmit = mxge_transmit; ifp->if_qflush = mxge_qflush; #endif taskqueue_start_threads(&sc->tq, 1, PI_NET, "%s taskq", device_get_nameunit(sc->dev)); callout_reset(&sc->co_hdl, mxge_ticks, mxge_tick, sc); return 0; abort_with_rings: mxge_free_rings(sc); abort_with_slices: mxge_free_slices(sc); abort_with_dmabench: mxge_dma_free(&sc->dmabench_dma); abort_with_zeropad_dma: mxge_dma_free(&sc->zeropad_dma); abort_with_cmd_dma: mxge_dma_free(&sc->cmd_dma); abort_with_mem_res: bus_release_resource(dev, SYS_RES_MEMORY, PCIR_BARS, sc->mem_res); abort_with_lock: pci_disable_busmaster(dev); mtx_destroy(&sc->cmd_mtx); mtx_destroy(&sc->driver_mtx); if_free(ifp); abort_with_parent_dmat: bus_dma_tag_destroy(sc->parent_dmat); abort_with_tq: if (sc->tq != NULL) { taskqueue_drain(sc->tq, &sc->watchdog_task); taskqueue_free(sc->tq); sc->tq = NULL; } abort_with_nothing: return err; } static int mxge_detach(device_t dev) { mxge_softc_t *sc = device_get_softc(dev); if (mxge_vlans_active(sc)) { device_printf(sc->dev, "Detach vlans before removing module\n"); return EBUSY; } mtx_lock(&sc->driver_mtx); sc->dying = 1; if (sc->ifp->if_drv_flags & IFF_DRV_RUNNING) mxge_close(sc, 0); mtx_unlock(&sc->driver_mtx); ether_ifdetach(sc->ifp); if (sc->tq != NULL) { taskqueue_drain(sc->tq, &sc->watchdog_task); taskqueue_free(sc->tq); sc->tq = NULL; } callout_drain(&sc->co_hdl); ifmedia_removeall(&sc->media); mxge_dummy_rdma(sc, 0); mxge_rem_sysctls(sc); mxge_rem_irq(sc); mxge_free_rings(sc); mxge_free_slices(sc); mxge_dma_free(&sc->dmabench_dma); mxge_dma_free(&sc->zeropad_dma); mxge_dma_free(&sc->cmd_dma); bus_release_resource(dev, SYS_RES_MEMORY, PCIR_BARS, sc->mem_res); pci_disable_busmaster(dev); mtx_destroy(&sc->cmd_mtx); mtx_destroy(&sc->driver_mtx); if_free(sc->ifp); bus_dma_tag_destroy(sc->parent_dmat); return 0; } static int mxge_shutdown(device_t dev) { return 0; } /* This file uses Myri10GE driver indentation. Local Variables: c-file-style:"linux" tab-width:8 End: */ Index: user/imp/tbemd/sys/dev/mxge/if_mxge_var.h =================================================================== --- user/imp/tbemd/sys/dev/mxge/if_mxge_var.h (revision 206684) +++ user/imp/tbemd/sys/dev/mxge/if_mxge_var.h (revision 206685) @@ -1,358 +1,366 @@ /******************************************************************************* Copyright (c) 2006-2009, Myricom Inc. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Neither the name of the Myricom Inc, nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. $FreeBSD$ ***************************************************************************/ #define MXGE_ETH_STOPPED 0 #define MXGE_ETH_STOPPING 1 #define MXGE_ETH_STARTING 2 #define MXGE_ETH_RUNNING 3 #define MXGE_ETH_OPEN_FAILED 4 #define MXGE_FW_OFFSET 1024*1024 #define MXGE_EEPROM_STRINGS_SIZE 256 #define MXGE_MAX_SEND_DESC 128 #if ((__FreeBSD_version > 800000 && __FreeBSD_version < 800005) \ || __FreeBSD_version < 700111) #define MXGE_VIRT_JUMBOS 1 #else #define MXGE_VIRT_JUMBOS 0 #endif #if (__FreeBSD_version > 800082) #define IFNET_BUF_RING 1 #endif #ifndef VLAN_CAPABILITIES #define VLAN_CAPABILITIES(ifp) #define mxge_vlans_active(sc) (sc)->ifp->if_nvlans #else #define mxge_vlans_active(sc) (sc)->ifp->if_vlantrunk #endif #ifndef VLAN_TAG_VALUE #define MXGE_NEW_VLAN_API #endif #ifndef IFCAP_LRO #define IFCAP_LRO 0 #endif #ifndef IFCAP_TSO #define IFCAP_TSO 0 #endif #ifndef IFCAP_TSO4 #define IFCAP_TSO4 0 #endif #ifndef CSUM_TSO #define CSUM_TSO 0 #endif typedef struct { void *addr; bus_addr_t bus_addr; bus_dma_tag_t dmat; bus_dmamap_t map; } mxge_dma_t; typedef struct { mcp_slot_t *entry; mxge_dma_t dma; int cnt; int idx; int mask; } mxge_rx_done_t; typedef struct { uint32_t data0; uint32_t data1; uint32_t data2; } mxge_cmd_t; struct mxge_rx_buffer_state { struct mbuf *m; bus_dmamap_t map; }; struct mxge_tx_buffer_state { struct mbuf *m; bus_dmamap_t map; int flag; }; typedef struct { volatile mcp_kreq_ether_recv_t *lanai; /* lanai ptr for recv ring */ mcp_kreq_ether_recv_t *shadow; /* host shadow of recv ring */ struct mxge_rx_buffer_state *info; bus_dma_tag_t dmat; bus_dmamap_t extra_map; int cnt; int nbufs; int cl_size; int alloc_fail; int mask; /* number of rx slots -1 */ int mlen; } mxge_rx_ring_t; typedef struct { struct mtx mtx; #ifdef IFNET_BUF_RING struct buf_ring *br; #endif volatile mcp_kreq_ether_send_t *lanai; /* lanai ptr for sendq */ volatile uint32_t *send_go; /* doorbell for sendq */ volatile uint32_t *send_stop; /* doorbell for sendq */ mcp_kreq_ether_send_t *req_list; /* host shadow of sendq */ char *req_bytes; bus_dma_segment_t *seg_list; struct mxge_tx_buffer_state *info; bus_dma_tag_t dmat; int req; /* transmits submitted */ int mask; /* number of transmit slots -1 */ int done; /* transmits completed */ int pkt_done; /* packets completed */ int max_desc; /* max descriptors per xmit */ int queue_active; /* fw currently polling this queue*/ int activate; int deactivate; int stall; /* #times hw queue exhausted */ int wake; /* #times irq re-enabled xmit */ int watchdog_req; /* cache of req */ int watchdog_done; /* cache of done */ int watchdog_rx_pause; /* cache of pause rq recvd */ int defrag; char mtx_name[16]; } mxge_tx_ring_t; struct lro_entry; struct lro_entry { SLIST_ENTRY(lro_entry) next; struct mbuf *m_head; struct mbuf *m_tail; int timestamp; struct ip *ip; uint32_t tsval; uint32_t tsecr; uint32_t source_ip; uint32_t dest_ip; uint32_t next_seq; uint32_t ack_seq; uint32_t len; uint32_t data_csum; uint16_t window; uint16_t source_port; uint16_t dest_port; uint16_t append_cnt; uint16_t mss; }; SLIST_HEAD(lro_head, lro_entry); struct mxge_softc; typedef struct mxge_softc mxge_softc_t; struct mxge_slice_state { mxge_softc_t *sc; mxge_tx_ring_t tx; /* transmit ring */ mxge_rx_ring_t rx_small; mxge_rx_ring_t rx_big; mxge_rx_done_t rx_done; mcp_irq_data_t *fw_stats; volatile uint32_t *irq_claim; u_long ipackets; u_long opackets; u_long obytes; u_long omcasts; u_long oerrors; int if_drv_flags; struct lro_head lro_active; struct lro_head lro_free; int lro_queued; int lro_flushed; int lro_bad_csum; mxge_dma_t fw_stats_dma; struct sysctl_oid *sysctl_tree; struct sysctl_ctx_list sysctl_ctx; char scratch[256]; }; struct mxge_softc { struct ifnet* ifp; struct mxge_slice_state *ss; int csum_flag; /* rx_csums? */ int tx_boundary; /* boundary transmits cannot cross*/ int lro_cnt; bus_dma_tag_t parent_dmat; volatile uint8_t *sram; int sram_size; volatile uint32_t *irq_deassert; mcp_cmd_response_t *cmd; mxge_dma_t cmd_dma; mxge_dma_t zeropad_dma; struct pci_dev *pdev; int legacy_irq; int link_state; unsigned int rdma_tags_available; int intr_coal_delay; volatile uint32_t *intr_coal_delay_ptr; int wc; struct mtx cmd_mtx; struct mtx driver_mtx; int wake_queue; int stop_queue; int down_cnt; int watchdog_resets; int watchdog_countdown; int pause; struct resource *mem_res; struct resource *irq_res; struct resource **msix_irq_res; struct resource *msix_table_res; struct resource *msix_pba_res; void *ih; void **msix_ih; char *fw_name; char eeprom_strings[MXGE_EEPROM_STRINGS_SIZE]; char fw_version[128]; int fw_ver_major; int fw_ver_minor; int fw_ver_tiny; int adopted_rx_filter_bug; device_t dev; struct ifmedia media; int read_dma; int write_dma; int read_write_dma; int fw_multicast_support; int link_width; int max_mtu; int throttle; int tx_defrag; int media_flags; int need_media_probe; int num_slices; int rx_ring_size; int dying; + int connector; + int current_media; mxge_dma_t dmabench_dma; struct callout co_hdl; struct taskqueue *tq; struct task watchdog_task; struct sysctl_oid *slice_sysctl_tree; struct sysctl_ctx_list slice_sysctl_ctx; char *mac_addr_string; uint8_t mac_addr[6]; /* eeprom mac address */ uint16_t pectl; /* save PCIe CTL state */ char product_code_string[64]; char serial_number_string[64]; char cmd_mtx_name[16]; char driver_mtx_name[16]; }; #define MXGE_PCI_VENDOR_MYRICOM 0x14c1 #define MXGE_PCI_DEVICE_Z8E 0x0008 #define MXGE_PCI_DEVICE_Z8E_9 0x0009 #define MXGE_PCI_REV_Z8E 0 #define MXGE_PCI_REV_Z8ES 1 #define MXGE_XFP_COMPLIANCE_BYTE 131 #define MXGE_SFP_COMPLIANCE_BYTE 3 #define MXGE_MIN_THROTTLE 416 #define MXGE_MAX_THROTTLE 4096 + +/* Types of connectors on NICs supported by this driver */ +#define MXGE_CX4 0 +#define MXGE_XFP 1 +#define MXGE_SFP 2 +#define MXGE_QRF 3 #define MXGE_HIGHPART_TO_U32(X) \ (sizeof (X) == 8) ? ((uint32_t)((uint64_t)(X) >> 32)) : (0) #define MXGE_LOWPART_TO_U32(X) ((uint32_t)(X)) struct mxge_media_type { int flag; uint8_t bitmask; char *name; }; /* implement our own memory barriers, since bus_space_barrier cannot handle write-combining regions */ #if __FreeBSD_version < 800053 #if defined (__GNUC__) #if #cpu(i386) || defined __i386 || defined i386 || defined __i386__ || #cpu(x86_64) || defined __x86_64__ #define wmb() __asm__ __volatile__ ("sfence;": : :"memory") #elif #cpu(sparc64) || defined sparc64 || defined __sparcv9 #define wmb() __asm__ __volatile__ ("membar #MemIssue": : :"memory") #elif #cpu(sparc) || defined sparc || defined __sparc__ #define wmb() __asm__ __volatile__ ("stbar;": : :"memory") #else #define wmb() /* XXX just to make this compile */ #endif #else #error "unknown compiler" #endif #endif static inline void mxge_pio_copy(volatile void *to_v, void *from_v, size_t size) { register volatile uintptr_t *to; volatile uintptr_t *from; size_t i; to = (volatile uintptr_t *) to_v; from = from_v; for (i = (size / sizeof (uintptr_t)); i; i--) { *to = *from; to++; from++; } } void mxge_lro_flush(struct mxge_slice_state *ss, struct lro_entry *lro); int mxge_lro_rx(struct mxge_slice_state *ss, struct mbuf *m_head, uint32_t csum); /* This file uses Myri10GE driver indentation. Local Variables: c-file-style:"linux" tab-width:8 End: */ Index: user/imp/tbemd/sys/dev/sge/if_sge.c =================================================================== --- user/imp/tbemd/sys/dev/sge/if_sge.c (revision 206684) +++ user/imp/tbemd/sys/dev/sge/if_sge.c (revision 206685) @@ -1,1745 +1,1745 @@ /*- * Copyright (c) 2008-2010 Nikolay Denev * Copyright (c) 2007-2008 Alexander Pohoyda * Copyright (c) 1997, 1998, 1999 * Bill Paul . All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. All advertising materials mentioning features or use of this software * must display the following acknowledgement: * This product includes software developed by Bill Paul. * 4. Neither the name of the author nor the names of any co-contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY Bill Paul AND CONTRIBUTORS ``AS IS'' * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A * PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL AUTHORS OR * THE VOICES IN THEIR HEADS BE LIABLE FOR ANY DIRECT, INDIRECT, * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR * SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED * OF THE POSSIBILITY OF SUCH DAMAGE. */ #include __FBSDID("$FreeBSD$"); /* * SiS 190/191 PCI Ethernet NIC driver. * * Adapted to SiS 190 NIC by Alexander Pohoyda based on the original * SiS 900 driver by Bill Paul, using SiS 190/191 Solaris driver by * Masayuki Murayama and SiS 190/191 GNU/Linux driver by K.M. Liu * . Thanks to Pyun YongHyeon for * review and very useful comments. * * Adapted to SiS 191 NIC by Nikolay Denev with further ideas from the * Linux and Solaris drivers. */ #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include -#include "if_sgereg.h" +#include MODULE_DEPEND(sge, pci, 1, 1, 1); MODULE_DEPEND(sge, ether, 1, 1, 1); MODULE_DEPEND(sge, miibus, 1, 1, 1); /* "device miibus0" required. See GENERIC if you get errors here. */ #include "miibus_if.h" /* * Various supported device vendors/types and their names. */ static struct sge_type sge_devs[] = { { SIS_VENDORID, SIS_DEVICEID_190, "SiS190 Fast Ethernet" }, { SIS_VENDORID, SIS_DEVICEID_191, "SiS191 Fast/Gigabit Ethernet" }, { 0, 0, NULL } }; static int sge_probe(device_t); static int sge_attach(device_t); static int sge_detach(device_t); static int sge_shutdown(device_t); static int sge_suspend(device_t); static int sge_resume(device_t); static int sge_miibus_readreg(device_t, int, int); static int sge_miibus_writereg(device_t, int, int, int); static void sge_miibus_statchg(device_t); static int sge_newbuf(struct sge_softc *, int); static int sge_encap(struct sge_softc *, struct mbuf **); #ifndef __NO_STRICT_ALIGNMENT static __inline void sge_fixup_rx(struct mbuf *); #endif static __inline void sge_discard_rxbuf(struct sge_softc *, int); static void sge_rxeof(struct sge_softc *); static void sge_txeof(struct sge_softc *); static void sge_intr(void *); static void sge_tick(void *); static void sge_start(struct ifnet *); static void sge_start_locked(struct ifnet *); static int sge_ioctl(struct ifnet *, u_long, caddr_t); static void sge_init(void *); static void sge_init_locked(struct sge_softc *); static void sge_stop(struct sge_softc *); static void sge_watchdog(struct sge_softc *); static int sge_ifmedia_upd(struct ifnet *); static void sge_ifmedia_sts(struct ifnet *, struct ifmediareq *); static int sge_get_mac_addr_apc(struct sge_softc *, uint8_t *); static int sge_get_mac_addr_eeprom(struct sge_softc *, uint8_t *); static uint16_t sge_read_eeprom(struct sge_softc *, int); static void sge_rxfilter(struct sge_softc *); static void sge_reset(struct sge_softc *); static int sge_list_rx_init(struct sge_softc *); static int sge_list_rx_free(struct sge_softc *); static int sge_list_tx_init(struct sge_softc *); static int sge_list_tx_free(struct sge_softc *); static int sge_dma_alloc(struct sge_softc *); static void sge_dma_free(struct sge_softc *); static void sge_dma_map_addr(void *, bus_dma_segment_t *, int, int); static device_method_t sge_methods[] = { /* Device interface */ DEVMETHOD(device_probe, sge_probe), DEVMETHOD(device_attach, sge_attach), DEVMETHOD(device_detach, sge_detach), DEVMETHOD(device_suspend, sge_suspend), DEVMETHOD(device_resume, sge_resume), DEVMETHOD(device_shutdown, sge_shutdown), /* Bus interface */ DEVMETHOD(bus_print_child, bus_generic_print_child), DEVMETHOD(bus_driver_added, bus_generic_driver_added), /* MII interface */ DEVMETHOD(miibus_readreg, sge_miibus_readreg), DEVMETHOD(miibus_writereg, sge_miibus_writereg), DEVMETHOD(miibus_statchg, sge_miibus_statchg), KOBJMETHOD_END }; static driver_t sge_driver = { "sge", sge_methods, sizeof(struct sge_softc) }; static devclass_t sge_devclass; DRIVER_MODULE(sge, pci, sge_driver, sge_devclass, 0, 0); DRIVER_MODULE(miibus, sge, miibus_driver, miibus_devclass, 0, 0); /* * Register space access macros. */ #define CSR_WRITE_4(sc, reg, val) bus_write_4(sc->sge_res, reg, val) #define CSR_WRITE_2(sc, reg, val) bus_write_2(sc->sge_res, reg, val) #define CSR_WRITE_1(cs, reg, val) bus_write_1(sc->sge_res, reg, val) #define CSR_READ_4(sc, reg) bus_read_4(sc->sge_res, reg) #define CSR_READ_2(sc, reg) bus_read_2(sc->sge_res, reg) #define CSR_READ_1(sc, reg) bus_read_1(sc->sge_res, reg) /* Define to show Tx/Rx error status. */ #undef SGE_SHOW_ERRORS #define SGE_CSUM_FEATURES (CSUM_IP | CSUM_TCP | CSUM_UDP) static void sge_dma_map_addr(void *arg, bus_dma_segment_t *segs, int nseg, int error) { bus_addr_t *p; if (error != 0) return; KASSERT(nseg == 1, ("too many DMA segments, %d should be 1", nseg)); p = arg; *p = segs->ds_addr; } /* * Read a sequence of words from the EEPROM. */ static uint16_t sge_read_eeprom(struct sge_softc *sc, int offset) { uint32_t val; int i; KASSERT(offset <= EI_OFFSET, ("EEPROM offset too big")); CSR_WRITE_4(sc, ROMInterface, EI_REQ | EI_OP_RD | (offset << EI_OFFSET_SHIFT)); DELAY(500); for (i = 0; i < SGE_TIMEOUT; i++) { val = CSR_READ_4(sc, ROMInterface); if ((val & EI_REQ) == 0) break; DELAY(100); } if (i == SGE_TIMEOUT) { device_printf(sc->sge_dev, "EEPROM read timeout : 0x%08x\n", val); return (0xffff); } return ((val & EI_DATA) >> EI_DATA_SHIFT); } static int sge_get_mac_addr_eeprom(struct sge_softc *sc, uint8_t *dest) { uint16_t val; int i; val = sge_read_eeprom(sc, EEPROMSignature); if (val == 0xffff || val == 0) { device_printf(sc->sge_dev, "invalid EEPROM signature : 0x%04x\n", val); return (EINVAL); } for (i = 0; i < ETHER_ADDR_LEN; i += 2) { val = sge_read_eeprom(sc, EEPROMMACAddr + i / 2); dest[i + 0] = (uint8_t)val; dest[i + 1] = (uint8_t)(val >> 8); } if ((sge_read_eeprom(sc, EEPROMInfo) & 0x80) != 0) sc->sge_flags |= SGE_FLAG_RGMII; return (0); } /* * For SiS96x, APC CMOS RAM is used to store ethernet address. * APC CMOS RAM is accessed through ISA bridge. */ static int sge_get_mac_addr_apc(struct sge_softc *sc, uint8_t *dest) { #if defined(__amd64__) || defined(__i386__) devclass_t pci; device_t bus, dev = NULL; device_t *kids; struct apc_tbl { uint16_t vid; uint16_t did; } *tp, apc_tbls[] = { { SIS_VENDORID, 0x0965 }, { SIS_VENDORID, 0x0966 }, { SIS_VENDORID, 0x0968 } }; uint8_t reg; int busnum, cnt, i, j, numkids; cnt = sizeof(apc_tbls) / sizeof(apc_tbls[0]); pci = devclass_find("pci"); for (busnum = 0; busnum < devclass_get_maxunit(pci); busnum++) { bus = devclass_get_device(pci, busnum); if (!bus) continue; if (device_get_children(bus, &kids, &numkids) != 0) continue; for (i = 0; i < numkids; i++) { dev = kids[i]; if (pci_get_class(dev) == PCIC_BRIDGE && pci_get_subclass(dev) == PCIS_BRIDGE_ISA) { tp = apc_tbls; for (j = 0; j < cnt; j++) { if (pci_get_vendor(dev) == tp->vid && pci_get_device(dev) == tp->did) { free(kids, M_TEMP); goto apc_found; } tp++; } } } free(kids, M_TEMP); } device_printf(sc->sge_dev, "couldn't find PCI-ISA bridge\n"); return (EINVAL); apc_found: /* Enable port 0x78 and 0x79 to access APC registers. */ reg = pci_read_config(dev, 0x48, 1); pci_write_config(dev, 0x48, reg & ~0x02, 1); DELAY(50); pci_read_config(dev, 0x48, 1); /* Read stored ethernet address. */ for (i = 0; i < ETHER_ADDR_LEN; i++) { outb(0x78, 0x09 + i); dest[i] = inb(0x79); } outb(0x78, 0x12); if ((inb(0x79) & 0x80) != 0) sc->sge_flags |= SGE_FLAG_RGMII; /* Restore access to APC registers. */ pci_write_config(dev, 0x48, reg, 1); return (0); #else return (EINVAL); #endif } static int sge_miibus_readreg(device_t dev, int phy, int reg) { struct sge_softc *sc; uint32_t val; int i; sc = device_get_softc(dev); CSR_WRITE_4(sc, GMIIControl, (phy << GMI_PHY_SHIFT) | (reg << GMI_REG_SHIFT) | GMI_OP_RD | GMI_REQ); DELAY(10); for (i = 0; i < SGE_TIMEOUT; i++) { val = CSR_READ_4(sc, GMIIControl); if ((val & GMI_REQ) == 0) break; DELAY(10); } if (i == SGE_TIMEOUT) { device_printf(sc->sge_dev, "PHY read timeout : %d\n", reg); return (0); } return ((val & GMI_DATA) >> GMI_DATA_SHIFT); } static int sge_miibus_writereg(device_t dev, int phy, int reg, int data) { struct sge_softc *sc; uint32_t val; int i; sc = device_get_softc(dev); CSR_WRITE_4(sc, GMIIControl, (phy << GMI_PHY_SHIFT) | (reg << GMI_REG_SHIFT) | (data << GMI_DATA_SHIFT) | GMI_OP_WR | GMI_REQ); DELAY(10); for (i = 0; i < SGE_TIMEOUT; i++) { val = CSR_READ_4(sc, GMIIControl); if ((val & GMI_REQ) == 0) break; DELAY(10); } if (i == SGE_TIMEOUT) device_printf(sc->sge_dev, "PHY write timeout : %d\n", reg); return (0); } static void sge_miibus_statchg(device_t dev) { struct sge_softc *sc; struct mii_data *mii; struct ifnet *ifp; uint32_t ctl, speed; sc = device_get_softc(dev); mii = device_get_softc(sc->sge_miibus); ifp = sc->sge_ifp; if (mii == NULL || ifp == NULL || (ifp->if_drv_flags & IFF_DRV_RUNNING) == 0) return; speed = 0; sc->sge_flags &= ~SGE_FLAG_LINK; if ((mii->mii_media_status & (IFM_ACTIVE | IFM_AVALID)) == (IFM_ACTIVE | IFM_AVALID)) { switch (IFM_SUBTYPE(mii->mii_media_active)) { case IFM_10_T: sc->sge_flags |= SGE_FLAG_LINK; speed = SC_SPEED_10; break; case IFM_100_TX: sc->sge_flags |= SGE_FLAG_LINK; speed = SC_SPEED_100; break; case IFM_1000_T: if ((sc->sge_flags & SGE_FLAG_FASTETHER) == 0) { sc->sge_flags |= SGE_FLAG_LINK; speed = SC_SPEED_1000; } break; default: break; } } if ((sc->sge_flags & SGE_FLAG_LINK) == 0) return; /* Reprogram MAC to resolved speed/duplex/flow-control parameters. */ ctl = CSR_READ_4(sc, StationControl); ctl &= ~(0x0f000000 | SC_FDX | SC_SPEED_MASK); if (speed == SC_SPEED_1000) { ctl |= 0x07000000; sc->sge_flags |= SGE_FLAG_SPEED_1000; } else { ctl |= 0x04000000; sc->sge_flags &= ~SGE_FLAG_SPEED_1000; } #ifdef notyet if ((sc->sge_flags & SGE_FLAG_GMII) != 0) ctl |= 0x03000000; #endif ctl |= speed; if ((IFM_OPTIONS(mii->mii_media_active) & IFM_FDX) != 0) { ctl |= SC_FDX; sc->sge_flags |= SGE_FLAG_FDX; } else sc->sge_flags &= ~SGE_FLAG_FDX; CSR_WRITE_4(sc, StationControl, ctl); if ((sc->sge_flags & SGE_FLAG_RGMII) != 0) { CSR_WRITE_4(sc, RGMIIDelay, 0x0441); CSR_WRITE_4(sc, RGMIIDelay, 0x0440); } } static void sge_rxfilter(struct sge_softc *sc) { struct ifnet *ifp; struct ifmultiaddr *ifma; uint32_t crc, hashes[2]; uint16_t rxfilt; SGE_LOCK_ASSERT(sc); ifp = sc->sge_ifp; hashes[0] = hashes[1] = 0; rxfilt = AcceptMyPhys; if ((ifp->if_flags & IFF_BROADCAST) != 0) rxfilt |= AcceptBroadcast; if ((ifp->if_flags & (IFF_PROMISC | IFF_ALLMULTI)) != 0) { if ((ifp->if_flags & IFF_PROMISC) != 0) rxfilt |= AcceptAllPhys; rxfilt |= AcceptMulticast; hashes[0] = 0xFFFFFFFF; hashes[1] = 0xFFFFFFFF; goto done; } rxfilt |= AcceptMulticast; /* Now program new ones. */ if_maddr_rlock(ifp); TAILQ_FOREACH(ifma, &ifp->if_multiaddrs, ifma_link) { if (ifma->ifma_addr->sa_family != AF_LINK) continue; crc = ether_crc32_be(LLADDR((struct sockaddr_dl *) ifma->ifma_addr), ETHER_ADDR_LEN); hashes[crc >> 31] |= 1 << ((crc >> 26) & 0x1f); } if_maddr_runlock(ifp); done: CSR_WRITE_2(sc, RxMacControl, rxfilt | 0x02); CSR_WRITE_4(sc, RxHashTable, hashes[0]); CSR_WRITE_4(sc, RxHashTable2, hashes[1]); } static void sge_reset(struct sge_softc *sc) { CSR_WRITE_4(sc, IntrMask, 0); CSR_WRITE_4(sc, IntrStatus, 0xffffffff); /* Soft reset. */ CSR_WRITE_4(sc, IntrControl, 0x8000); CSR_READ_4(sc, IntrControl); DELAY(100); CSR_WRITE_4(sc, IntrControl, 0); /* Stop MAC. */ CSR_WRITE_4(sc, TX_CTL, 0x1a00); CSR_WRITE_4(sc, RX_CTL, 0x1a00); CSR_WRITE_4(sc, IntrMask, 0); CSR_WRITE_4(sc, IntrStatus, 0xffffffff); CSR_WRITE_4(sc, GMIIControl, 0); } /* * Probe for an SiS chip. Check the PCI vendor and device * IDs against our list and return a device name if we find a match. */ static int sge_probe(device_t dev) { struct sge_type *t; t = sge_devs; while (t->sge_name != NULL) { if ((pci_get_vendor(dev) == t->sge_vid) && (pci_get_device(dev) == t->sge_did)) { device_set_desc(dev, t->sge_name); return (BUS_PROBE_DEFAULT); } t++; } return (ENXIO); } /* * Attach the interface. Allocate softc structures, do ifmedia * setup and ethernet/BPF attach. */ static int sge_attach(device_t dev) { struct sge_softc *sc; struct ifnet *ifp; uint8_t eaddr[ETHER_ADDR_LEN]; int error = 0, rid; sc = device_get_softc(dev); sc->sge_dev = dev; mtx_init(&sc->sge_mtx, device_get_nameunit(dev), MTX_NETWORK_LOCK, MTX_DEF); callout_init_mtx(&sc->sge_stat_ch, &sc->sge_mtx, 0); /* * Map control/status registers. */ pci_enable_busmaster(dev); /* Allocate resources. */ sc->sge_res_id = PCIR_BAR(0); sc->sge_res_type = SYS_RES_MEMORY; sc->sge_res = bus_alloc_resource_any(dev, sc->sge_res_type, &sc->sge_res_id, RF_ACTIVE); if (sc->sge_res == NULL) { device_printf(dev, "couldn't allocate resource\n"); error = ENXIO; goto fail; } rid = 0; sc->sge_irq = bus_alloc_resource_any(dev, SYS_RES_IRQ, &rid, RF_SHAREABLE | RF_ACTIVE); if (sc->sge_irq == NULL) { device_printf(dev, "couldn't allocate IRQ resources\n"); error = ENXIO; goto fail; } sc->sge_rev = pci_get_revid(dev); if (pci_get_device(dev) == SIS_DEVICEID_190) sc->sge_flags |= SGE_FLAG_FASTETHER; /* Reset the adapter. */ sge_reset(sc); /* Get MAC address from the EEPROM. */ if ((pci_read_config(dev, 0x73, 1) & 0x01) != 0) sge_get_mac_addr_apc(sc, eaddr); else sge_get_mac_addr_eeprom(sc, eaddr); if ((error = sge_dma_alloc(sc)) != 0) goto fail; ifp = sc->sge_ifp = if_alloc(IFT_ETHER); if (ifp == NULL) { device_printf(dev, "cannot allocate ifnet structure.\n"); error = ENOSPC; goto fail; } ifp->if_softc = sc; if_initname(ifp, device_get_name(dev), device_get_unit(dev)); ifp->if_flags = IFF_BROADCAST | IFF_SIMPLEX | IFF_MULTICAST; ifp->if_ioctl = sge_ioctl; ifp->if_start = sge_start; ifp->if_init = sge_init; ifp->if_snd.ifq_drv_maxlen = SGE_TX_RING_CNT - 1; IFQ_SET_MAXLEN(&ifp->if_snd, ifp->if_snd.ifq_drv_maxlen); IFQ_SET_READY(&ifp->if_snd); ifp->if_capabilities = IFCAP_TXCSUM | IFCAP_RXCSUM; ifp->if_hwassist = SGE_CSUM_FEATURES; ifp->if_capenable = ifp->if_capabilities; /* * Do MII setup. */ if (mii_phy_probe(dev, &sc->sge_miibus, sge_ifmedia_upd, sge_ifmedia_sts)) { device_printf(dev, "no PHY found!\n"); error = ENXIO; goto fail; } /* * Call MI attach routine. */ ether_ifattach(ifp, eaddr); /* VLAN setup. */ ifp->if_capabilities |= IFCAP_VLAN_MTU; ifp->if_capenable = ifp->if_capabilities; /* Tell the upper layer(s) we support long frames. */ ifp->if_data.ifi_hdrlen = sizeof(struct ether_vlan_header); /* Hook interrupt last to avoid having to lock softc */ error = bus_setup_intr(dev, sc->sge_irq, INTR_TYPE_NET | INTR_MPSAFE, NULL, sge_intr, sc, &sc->sge_intrhand); if (error) { device_printf(dev, "couldn't set up irq\n"); ether_ifdetach(ifp); goto fail; } fail: if (error) sge_detach(dev); return (error); } /* * Shutdown hardware and free up resources. This can be called any * time after the mutex has been initialized. It is called in both * the error case in attach and the normal detach case so it needs * to be careful about only freeing resources that have actually been * allocated. */ static int sge_detach(device_t dev) { struct sge_softc *sc; struct ifnet *ifp; sc = device_get_softc(dev); ifp = sc->sge_ifp; /* These should only be active if attach succeeded. */ if (device_is_attached(dev)) { ether_ifdetach(ifp); SGE_LOCK(sc); sge_stop(sc); SGE_UNLOCK(sc); callout_drain(&sc->sge_stat_ch); } if (sc->sge_miibus) device_delete_child(dev, sc->sge_miibus); bus_generic_detach(dev); if (sc->sge_intrhand) bus_teardown_intr(dev, sc->sge_irq, sc->sge_intrhand); if (sc->sge_irq) bus_release_resource(dev, SYS_RES_IRQ, 0, sc->sge_irq); if (sc->sge_res) bus_release_resource(dev, sc->sge_res_type, sc->sge_res_id, sc->sge_res); if (ifp) if_free(ifp); sge_dma_free(sc); mtx_destroy(&sc->sge_mtx); return (0); } /* * Stop all chip I/O so that the kernel's probe routines don't * get confused by errant DMAs when rebooting. */ static int sge_shutdown(device_t dev) { struct sge_softc *sc; sc = device_get_softc(dev); SGE_LOCK(sc); sge_stop(sc); SGE_UNLOCK(sc); return (0); } static int sge_suspend(device_t dev) { struct sge_softc *sc; struct ifnet *ifp; sc = device_get_softc(dev); SGE_LOCK(sc); ifp = sc->sge_ifp; if ((ifp->if_drv_flags & IFF_DRV_RUNNING) != 0) sge_stop(sc); SGE_UNLOCK(sc); return (0); } static int sge_resume(device_t dev) { struct sge_softc *sc; struct ifnet *ifp; sc = device_get_softc(dev); SGE_LOCK(sc); ifp = sc->sge_ifp; if ((ifp->if_flags & IFF_UP) != 0) sge_init_locked(sc); SGE_UNLOCK(sc); return (0); } static int sge_dma_alloc(struct sge_softc *sc) { struct sge_chain_data *cd; struct sge_list_data *ld; int error, i; cd = &sc->sge_cdata; ld = &sc->sge_ldata; error = bus_dma_tag_create(bus_get_dma_tag(sc->sge_dev), 1, 0, /* alignment, boundary */ BUS_SPACE_MAXADDR_32BIT, /* lowaddr */ BUS_SPACE_MAXADDR, /* highaddr */ NULL, NULL, /* filter, filterarg */ BUS_SPACE_MAXSIZE_32BIT, /* maxsize */ 1, /* nsegments */ BUS_SPACE_MAXSIZE_32BIT, /* maxsegsize */ 0, /* flags */ NULL, /* lockfunc */ NULL, /* lockarg */ &cd->sge_tag); if (error != 0) { device_printf(sc->sge_dev, "could not create parent DMA tag.\n"); goto fail; } /* RX descriptor ring */ error = bus_dma_tag_create(cd->sge_tag, SGE_DESC_ALIGN, 0, /* alignment, boundary */ BUS_SPACE_MAXADDR, /* lowaddr */ BUS_SPACE_MAXADDR, /* highaddr */ NULL, NULL, /* filter, filterarg */ SGE_RX_RING_SZ, 1, /* maxsize,nsegments */ SGE_RX_RING_SZ, /* maxsegsize */ 0, /* flags */ NULL, /* lockfunc */ NULL, /* lockarg */ &cd->sge_rx_tag); if (error != 0) { device_printf(sc->sge_dev, "could not create Rx ring DMA tag.\n"); goto fail; } /* Allocate DMA'able memory and load DMA map for RX ring. */ error = bus_dmamem_alloc(cd->sge_rx_tag, (void **)&ld->sge_rx_ring, BUS_DMA_NOWAIT | BUS_DMA_ZERO | BUS_DMA_COHERENT, &cd->sge_rx_dmamap); if (error != 0) { device_printf(sc->sge_dev, "could not allocate DMA'able memory for Rx ring.\n"); goto fail; } error = bus_dmamap_load(cd->sge_rx_tag, cd->sge_rx_dmamap, ld->sge_rx_ring, SGE_RX_RING_SZ, sge_dma_map_addr, &ld->sge_rx_paddr, BUS_DMA_NOWAIT); if (error != 0) { device_printf(sc->sge_dev, "could not load DMA'able memory for Rx ring.\n"); } /* TX descriptor ring */ error = bus_dma_tag_create(cd->sge_tag, SGE_DESC_ALIGN, 0, /* alignment, boundary */ BUS_SPACE_MAXADDR, /* lowaddr */ BUS_SPACE_MAXADDR, /* highaddr */ NULL, NULL, /* filter, filterarg */ SGE_TX_RING_SZ, 1, /* maxsize,nsegments */ SGE_TX_RING_SZ, /* maxsegsize */ 0, /* flags */ NULL, /* lockfunc */ NULL, /* lockarg */ &cd->sge_tx_tag); if (error != 0) { device_printf(sc->sge_dev, "could not create Rx ring DMA tag.\n"); goto fail; } /* Allocate DMA'able memory and load DMA map for TX ring. */ error = bus_dmamem_alloc(cd->sge_tx_tag, (void **)&ld->sge_tx_ring, BUS_DMA_NOWAIT | BUS_DMA_ZERO | BUS_DMA_COHERENT, &cd->sge_tx_dmamap); if (error != 0) { device_printf(sc->sge_dev, "could not allocate DMA'able memory for Tx ring.\n"); goto fail; } error = bus_dmamap_load(cd->sge_tx_tag, cd->sge_tx_dmamap, ld->sge_tx_ring, SGE_TX_RING_SZ, sge_dma_map_addr, &ld->sge_tx_paddr, BUS_DMA_NOWAIT); if (error != 0) { device_printf(sc->sge_dev, "could not load DMA'able memory for Rx ring.\n"); goto fail; } /* Create DMA tag for Tx buffers. */ error = bus_dma_tag_create(cd->sge_tag, 1, 0, BUS_SPACE_MAXADDR, BUS_SPACE_MAXADDR, NULL, NULL, MCLBYTES * SGE_MAXTXSEGS, SGE_MAXTXSEGS, MCLBYTES, 0, NULL, NULL, &cd->sge_txmbuf_tag); if (error != 0) { device_printf(sc->sge_dev, "could not create Tx mbuf DMA tag.\n"); goto fail; } /* Create DMA tag for Rx buffers. */ error = bus_dma_tag_create(cd->sge_tag, SGE_RX_BUF_ALIGN, 0, BUS_SPACE_MAXADDR, BUS_SPACE_MAXADDR, NULL, NULL, MCLBYTES, 1, MCLBYTES, 0, NULL, NULL, &cd->sge_rxmbuf_tag); if (error != 0) { device_printf(sc->sge_dev, "could not create Rx mbuf DMA tag.\n"); goto fail; } /* Create DMA maps for Tx buffers. */ for (i = 0; i < SGE_TX_RING_CNT; i++) { error = bus_dmamap_create(cd->sge_txmbuf_tag, 0, &cd->sge_tx_map[i]); if (error != 0) { device_printf(sc->sge_dev, "could not create Tx DMA map.\n"); goto fail; } } /* Create spare DMA map for Rx buffer. */ error = bus_dmamap_create(cd->sge_rxmbuf_tag, 0, &cd->sge_rx_spare_map); if (error != 0) { device_printf(sc->sge_dev, "could not create spare Rx DMA map.\n"); goto fail; } /* Create DMA maps for Rx buffers. */ for (i = 0; i < SGE_RX_RING_CNT; i++) { error = bus_dmamap_create(cd->sge_rxmbuf_tag, 0, &cd->sge_rx_map[i]); if (error) { device_printf(sc->sge_dev, "could not create Rx DMA map.\n"); goto fail; } } fail: return (error); } static void sge_dma_free(struct sge_softc *sc) { struct sge_chain_data *cd; struct sge_list_data *ld; int i; cd = &sc->sge_cdata; ld = &sc->sge_ldata; /* Rx ring. */ if (cd->sge_rx_tag != NULL) { if (cd->sge_rx_dmamap != NULL) bus_dmamap_unload(cd->sge_rx_tag, cd->sge_rx_dmamap); if (cd->sge_rx_dmamap != NULL && ld->sge_rx_ring != NULL) bus_dmamem_free(cd->sge_rx_tag, ld->sge_rx_ring, cd->sge_rx_dmamap); ld->sge_rx_ring = NULL; cd->sge_rx_dmamap = NULL; bus_dma_tag_destroy(cd->sge_rx_tag); cd->sge_rx_tag = NULL; } /* Tx ring. */ if (cd->sge_tx_tag != NULL) { if (cd->sge_tx_dmamap != NULL) bus_dmamap_unload(cd->sge_tx_tag, cd->sge_tx_dmamap); if (cd->sge_tx_dmamap != NULL && ld->sge_tx_ring != NULL) bus_dmamem_free(cd->sge_tx_tag, ld->sge_tx_ring, cd->sge_tx_dmamap); ld->sge_tx_ring = NULL; cd->sge_tx_dmamap = NULL; bus_dma_tag_destroy(cd->sge_tx_tag); cd->sge_tx_tag = NULL; } /* Rx buffers. */ if (cd->sge_rxmbuf_tag != NULL) { for (i = 0; i < SGE_RX_RING_CNT; i++) { if (cd->sge_rx_map[i] != NULL) { bus_dmamap_destroy(cd->sge_rxmbuf_tag, cd->sge_rx_map[i]); cd->sge_rx_map[i] = NULL; } } if (cd->sge_rx_spare_map != NULL) { bus_dmamap_destroy(cd->sge_rxmbuf_tag, cd->sge_rx_spare_map); cd->sge_rx_spare_map = NULL; } bus_dma_tag_destroy(cd->sge_rxmbuf_tag); cd->sge_rxmbuf_tag = NULL; } /* Tx buffers. */ if (cd->sge_txmbuf_tag != NULL) { for (i = 0; i < SGE_TX_RING_CNT; i++) { if (cd->sge_tx_map[i] != NULL) { bus_dmamap_destroy(cd->sge_txmbuf_tag, cd->sge_tx_map[i]); cd->sge_tx_map[i] = NULL; } } bus_dma_tag_destroy(cd->sge_txmbuf_tag); cd->sge_txmbuf_tag = NULL; } if (cd->sge_tag != NULL) bus_dma_tag_destroy(cd->sge_tag); cd->sge_tag = NULL; } /* * Initialize the TX descriptors. */ static int sge_list_tx_init(struct sge_softc *sc) { struct sge_list_data *ld; struct sge_chain_data *cd; SGE_LOCK_ASSERT(sc); ld = &sc->sge_ldata; cd = &sc->sge_cdata; bzero(ld->sge_tx_ring, SGE_TX_RING_SZ); ld->sge_tx_ring[SGE_TX_RING_CNT - 1].sge_flags = htole32(RING_END); bus_dmamap_sync(cd->sge_tx_tag, cd->sge_tx_dmamap, BUS_DMASYNC_PREREAD | BUS_DMASYNC_PREWRITE); cd->sge_tx_prod = 0; cd->sge_tx_cons = 0; cd->sge_tx_cnt = 0; return (0); } static int sge_list_tx_free(struct sge_softc *sc) { struct sge_chain_data *cd; int i; SGE_LOCK_ASSERT(sc); cd = &sc->sge_cdata; for (i = 0; i < SGE_TX_RING_CNT; i++) { if (cd->sge_tx_mbuf[i] != NULL) { bus_dmamap_sync(cd->sge_txmbuf_tag, cd->sge_tx_map[i], BUS_DMASYNC_POSTWRITE); bus_dmamap_unload(cd->sge_txmbuf_tag, cd->sge_tx_map[i]); m_free(cd->sge_tx_mbuf[i]); cd->sge_tx_mbuf[i] = NULL; } } return (0); } /* * Initialize the RX descriptors and allocate mbufs for them. Note that * we arrange the descriptors in a closed ring, so that the last descriptor * has RING_END flag set. */ static int sge_list_rx_init(struct sge_softc *sc) { struct sge_chain_data *cd; int i; SGE_LOCK_ASSERT(sc); cd = &sc->sge_cdata; cd->sge_rx_cons = 0; bzero(sc->sge_ldata.sge_rx_ring, SGE_RX_RING_SZ); for (i = 0; i < SGE_RX_RING_CNT; i++) { if (sge_newbuf(sc, i) != 0) return (ENOBUFS); } bus_dmamap_sync(cd->sge_rx_tag, cd->sge_rx_dmamap, BUS_DMASYNC_PREREAD | BUS_DMASYNC_PREWRITE); return (0); } static int sge_list_rx_free(struct sge_softc *sc) { struct sge_chain_data *cd; int i; SGE_LOCK_ASSERT(sc); cd = &sc->sge_cdata; for (i = 0; i < SGE_RX_RING_CNT; i++) { if (cd->sge_rx_mbuf[i] != NULL) { bus_dmamap_sync(cd->sge_rxmbuf_tag, cd->sge_rx_map[i], BUS_DMASYNC_POSTREAD); bus_dmamap_unload(cd->sge_rxmbuf_tag, cd->sge_rx_map[i]); m_free(cd->sge_rx_mbuf[i]); cd->sge_rx_mbuf[i] = NULL; } } return (0); } /* * Initialize an RX descriptor and attach an MBUF cluster. */ static int sge_newbuf(struct sge_softc *sc, int prod) { struct mbuf *m; struct sge_desc *desc; struct sge_chain_data *cd; bus_dma_segment_t segs[1]; bus_dmamap_t map; int error, nsegs; SGE_LOCK_ASSERT(sc); cd = &sc->sge_cdata; m = m_getcl(M_DONTWAIT, MT_DATA, M_PKTHDR); if (m == NULL) return (ENOBUFS); m->m_len = m->m_pkthdr.len = MCLBYTES; m_adj(m, SGE_RX_BUF_ALIGN); error = bus_dmamap_load_mbuf_sg(cd->sge_rxmbuf_tag, cd->sge_rx_spare_map, m, segs, &nsegs, 0); if (error != 0) { m_freem(m); return (error); } KASSERT(nsegs == 1, ("%s: %d segments returned!", __func__, nsegs)); if (cd->sge_rx_mbuf[prod] != NULL) { bus_dmamap_sync(cd->sge_rxmbuf_tag, cd->sge_rx_map[prod], BUS_DMASYNC_POSTREAD); bus_dmamap_unload(cd->sge_rxmbuf_tag, cd->sge_rx_map[prod]); } map = cd->sge_rx_map[prod]; cd->sge_rx_map[prod] = cd->sge_rx_spare_map; cd->sge_rx_spare_map = map; bus_dmamap_sync(cd->sge_rxmbuf_tag, cd->sge_rx_map[prod], BUS_DMASYNC_PREREAD); cd->sge_rx_mbuf[prod] = m; desc = &sc->sge_ldata.sge_rx_ring[prod]; desc->sge_sts_size = 0; desc->sge_ptr = htole32(SGE_ADDR_LO(segs[0].ds_addr)); desc->sge_flags = htole32(segs[0].ds_len); if (prod == SGE_RX_RING_CNT - 1) desc->sge_flags |= htole32(RING_END); desc->sge_cmdsts = htole32(RDC_OWN | RDC_INTR | RDC_IP_CSUM | RDC_TCP_CSUM | RDC_UDP_CSUM); return (0); } #ifndef __NO_STRICT_ALIGNMENT static __inline void sge_fixup_rx(struct mbuf *m) { int i; uint16_t *src, *dst; src = mtod(m, uint16_t *); dst = src - 3; for (i = 0; i < (m->m_len / sizeof(uint16_t) + 1); i++) *dst++ = *src++; m->m_data -= (SGE_RX_BUF_ALIGN - ETHER_ALIGN); } #endif static __inline void sge_discard_rxbuf(struct sge_softc *sc, int index) { struct sge_desc *desc; desc = &sc->sge_ldata.sge_rx_ring[index]; desc->sge_sts_size = 0; desc->sge_flags = htole32(MCLBYTES - SGE_RX_BUF_ALIGN); if (index == SGE_RX_RING_CNT - 1) desc->sge_flags |= htole32(RING_END); desc->sge_cmdsts = htole32(RDC_OWN | RDC_INTR | RDC_IP_CSUM | RDC_TCP_CSUM | RDC_UDP_CSUM); } /* * A frame has been uploaded: pass the resulting mbuf chain up to * the higher level protocols. */ static void sge_rxeof(struct sge_softc *sc) { struct ifnet *ifp; struct mbuf *m; struct sge_chain_data *cd; struct sge_desc *cur_rx; uint32_t rxinfo, rxstat; int cons, prog; SGE_LOCK_ASSERT(sc); ifp = sc->sge_ifp; cd = &sc->sge_cdata; bus_dmamap_sync(cd->sge_rx_tag, cd->sge_rx_dmamap, BUS_DMASYNC_POSTREAD | BUS_DMASYNC_POSTWRITE); cons = cd->sge_rx_cons; for (prog = 0; prog < SGE_RX_RING_CNT; prog++, SGE_INC(cons, SGE_RX_RING_CNT)) { if ((ifp->if_drv_flags & IFF_DRV_RUNNING) == 0) break; cur_rx = &sc->sge_ldata.sge_rx_ring[cons]; rxinfo = le32toh(cur_rx->sge_cmdsts); if ((rxinfo & RDC_OWN) != 0) break; rxstat = le32toh(cur_rx->sge_sts_size); if (SGE_RX_ERROR(rxstat) != 0 || SGE_RX_NSEGS(rxstat) != 1) { /* XXX We don't support multi-segment frames yet. */ #ifdef SGE_SHOW_ERRORS device_printf(sc->sge_dev, "Rx error : 0x%b\n", rxstat, RX_ERR_BITS); #endif sge_discard_rxbuf(sc, cons); ifp->if_ierrors++; continue; } m = cd->sge_rx_mbuf[cons]; if (sge_newbuf(sc, cons) != 0) { sge_discard_rxbuf(sc, cons); ifp->if_iqdrops++; continue; } if ((ifp->if_capenable & IFCAP_RXCSUM) != 0) { if ((rxinfo & RDC_IP_CSUM) != 0 && (rxinfo & RDC_IP_CSUM_OK) != 0) m->m_pkthdr.csum_flags |= CSUM_IP_CHECKED | CSUM_IP_VALID; if (((rxinfo & RDC_TCP_CSUM) != 0 && (rxinfo & RDC_TCP_CSUM_OK) != 0) || ((rxinfo & RDC_UDP_CSUM) != 0 && (rxinfo & RDC_UDP_CSUM_OK) != 0)) { m->m_pkthdr.csum_flags |= CSUM_DATA_VALID | CSUM_PSEUDO_HDR; m->m_pkthdr.csum_data = 0xffff; } } /* * TODO : VLAN hardware tag stripping. */ m->m_pkthdr.len = m->m_len = SGE_RX_BYTES(rxstat) - ETHER_CRC_LEN; #ifndef __NO_STRICT_ALIGNMENT sge_fixup_rx(m); #endif m->m_pkthdr.rcvif = ifp; ifp->if_ipackets++; SGE_UNLOCK(sc); (*ifp->if_input)(ifp, m); SGE_LOCK(sc); } if (prog > 0) { bus_dmamap_sync(cd->sge_rx_tag, cd->sge_rx_dmamap, BUS_DMASYNC_PREREAD | BUS_DMASYNC_PREWRITE); cd->sge_rx_cons = cons; } } /* * A frame was downloaded to the chip. It's safe for us to clean up * the list buffers. */ static void sge_txeof(struct sge_softc *sc) { struct ifnet *ifp; struct sge_list_data *ld; struct sge_chain_data *cd; uint32_t txstat; int cons, prod; SGE_LOCK_ASSERT(sc); ifp = sc->sge_ifp; ld = &sc->sge_ldata; cd = &sc->sge_cdata; if (cd->sge_tx_cnt == 0) return; bus_dmamap_sync(cd->sge_tx_tag, cd->sge_tx_dmamap, BUS_DMASYNC_POSTREAD | BUS_DMASYNC_POSTWRITE); cons = cd->sge_tx_cons; prod = cd->sge_tx_prod; for (; cons != prod; SGE_INC(cons, SGE_TX_RING_CNT)) { txstat = le32toh(ld->sge_tx_ring[cons].sge_cmdsts); if ((txstat & TDC_OWN) != 0) break; cd->sge_tx_cnt--; ifp->if_drv_flags &= ~IFF_DRV_OACTIVE; if (cd->sge_tx_mbuf[cons] != NULL) { bus_dmamap_sync(cd->sge_txmbuf_tag, cd->sge_tx_map[cons], BUS_DMASYNC_POSTWRITE); bus_dmamap_unload(cd->sge_txmbuf_tag, cd->sge_tx_map[cons]); m_freem(cd->sge_tx_mbuf[cons]); cd->sge_tx_mbuf[cons] = NULL; if (SGE_TX_ERROR(txstat) != 0) { #ifdef SGE_SHOW_ERRORS device_printf(sc->sge_dev, "Tx error : 0x%b\n", txstat, TX_ERR_BITS); #endif ifp->if_oerrors++; } else { #ifdef notyet ifp->if_collisions += (txstat & 0xFFFF) - 1; #endif ifp->if_opackets++; } } } cd->sge_tx_cons = cons; if (cd->sge_tx_cnt == 0) sc->sge_timer = 0; } static void sge_tick(void *arg) { struct sge_softc *sc; struct mii_data *mii; struct ifnet *ifp; sc = arg; SGE_LOCK_ASSERT(sc); ifp = sc->sge_ifp; mii = device_get_softc(sc->sge_miibus); mii_tick(mii); if ((sc->sge_flags & SGE_FLAG_LINK) == 0) { sge_miibus_statchg(sc->sge_dev); if ((sc->sge_flags & SGE_FLAG_LINK) != 0 && !IFQ_DRV_IS_EMPTY(&ifp->if_snd)) sge_start_locked(ifp); } /* * Reclaim transmitted frames here as we do not request * Tx completion interrupt for every queued frames to * reduce excessive interrupts. */ sge_txeof(sc); sge_watchdog(sc); callout_reset(&sc->sge_stat_ch, hz, sge_tick, sc); } static void sge_intr(void *arg) { struct sge_softc *sc; struct ifnet *ifp; uint32_t status; sc = arg; SGE_LOCK(sc); ifp = sc->sge_ifp; status = CSR_READ_4(sc, IntrStatus); if (status == 0xFFFFFFFF || (status & SGE_INTRS) == 0) { /* Not ours. */ SGE_UNLOCK(sc); return; } /* Acknowledge interrupts. */ CSR_WRITE_4(sc, IntrStatus, status); /* Disable further interrupts. */ CSR_WRITE_4(sc, IntrMask, 0); /* * It seems the controller supports some kind of interrupt * moderation mechanism but we still don't know how to * enable that. To reduce number of generated interrupts * under load we check pending interrupts in a loop. This * will increase number of register access and is not correct * way to handle interrupt moderation but there seems to be * no other way at this time. */ for (;;) { if ((ifp->if_drv_flags & IFF_DRV_RUNNING) == 0) break; if ((status & (INTR_RX_DONE | INTR_RX_IDLE)) != 0) { sge_rxeof(sc); /* Wakeup Rx MAC. */ if ((status & INTR_RX_IDLE) != 0) CSR_WRITE_4(sc, RX_CTL, 0x1a00 | 0x000c | RX_CTL_POLL | RX_CTL_ENB); } if ((status & (INTR_TX_DONE | INTR_TX_IDLE)) != 0) sge_txeof(sc); status = CSR_READ_4(sc, IntrStatus); if ((status & SGE_INTRS) == 0) break; /* Acknowledge interrupts. */ CSR_WRITE_4(sc, IntrStatus, status); } if ((ifp->if_drv_flags & IFF_DRV_RUNNING) != 0) { /* Re-enable interrupts */ CSR_WRITE_4(sc, IntrMask, SGE_INTRS); if (!IFQ_DRV_IS_EMPTY(&ifp->if_snd)) sge_start_locked(ifp); } SGE_UNLOCK(sc); } /* * Encapsulate an mbuf chain in a descriptor by coupling the mbuf data * pointers to the fragment pointers. */ static int sge_encap(struct sge_softc *sc, struct mbuf **m_head) { struct mbuf *m; struct sge_desc *desc; bus_dma_segment_t txsegs[SGE_MAXTXSEGS]; bus_dmamap_t map; uint32_t cflags; int error, nsegs, prod; SGE_LOCK_ASSERT(sc); prod = sc->sge_cdata.sge_tx_prod; map = sc->sge_cdata.sge_tx_map[prod]; /* * Reading Windows inf file indicates SiS controller supports * TSO, VLAN hardware tag insertion/stripping, interrupt * moderation and Tx/Rx checksum offloading. Unfortunately * vendor didn't release these information so we're guessing * descriptor usage with trial and errors. * * Controller seems to support multi-fragmented buffers but * don't know how to enable that feature so limit number of * fragmented Tx buffers to single buffer until we understand * the controller internals. * I assume the controller can pad zero bytes if frame length * is less than 60 bytes and I also think the controller has * no Tx buffer alignment limitation. - Need testing! */ if ((*m_head)->m_next != NULL) { m = m_defrag(*m_head, M_DONTWAIT); if (m == NULL) { m_freem(*m_head); *m_head = NULL; return (ENOBUFS); } *m_head = m; } error = bus_dmamap_load_mbuf_sg(sc->sge_cdata.sge_tx_tag, map, *m_head, txsegs, &nsegs, 0); if (error != 0) { m_freem(*m_head); *m_head = NULL; return (error); } /* Check descriptor overrun. */ if (sc->sge_cdata.sge_tx_cnt + nsegs >= SGE_TX_RING_CNT) { bus_dmamap_unload(sc->sge_cdata.sge_tx_tag, map); return (ENOBUFS); } bus_dmamap_sync(sc->sge_cdata.sge_tx_tag, map, BUS_DMASYNC_PREWRITE); cflags = 0; if ((*m_head)->m_pkthdr.csum_flags & CSUM_IP) cflags |= TDC_IP_CSUM; if ((*m_head)->m_pkthdr.csum_flags & CSUM_TCP) cflags |= TDC_TCP_CSUM; if ((*m_head)->m_pkthdr.csum_flags & CSUM_UDP) cflags |= TDC_UDP_CSUM; desc = &sc->sge_ldata.sge_tx_ring[prod]; desc->sge_sts_size = htole32((*m_head)->m_pkthdr.len); desc->sge_ptr = htole32(SGE_ADDR_LO(txsegs[0].ds_addr)); desc->sge_flags = htole32(txsegs[0].ds_len); if (prod == SGE_TX_RING_CNT - 1) desc->sge_flags |= htole32(RING_END); desc->sge_cmdsts = htole32(TDC_DEF | TDC_CRC | TDC_PAD | cflags); #if 1 if ((sc->sge_flags & SGE_FLAG_SPEED_1000) != 0) desc->sge_cmdsts |= htole32(TDC_BST); #else if ((sc->sge_flags & SGE_FLAG_FDX) == 0) { desc->sge_cmdsts |= htole32(TDC_COL | TDC_CRS | TDC_BKF); if ((sc->sge_flags & SGE_FLAG_SPEED_1000) != 0) desc->sge_cmdsts |= htole32(TDC_EXT | TDC_BST); } #endif /* Request interrupt and give ownership to controller. */ if ((prod % SGE_TX_INTR_FRAMES) == 0) desc->sge_cmdsts |= htole32(TDC_OWN | TDC_INTR); else desc->sge_cmdsts |= htole32(TDC_OWN); sc->sge_cdata.sge_tx_mbuf[prod] = *m_head; sc->sge_cdata.sge_tx_cnt++; SGE_INC(sc->sge_cdata.sge_tx_prod, SGE_TX_RING_CNT); return (0); } static void sge_start(struct ifnet *ifp) { struct sge_softc *sc; sc = ifp->if_softc; SGE_LOCK(sc); sge_start_locked(ifp); SGE_UNLOCK(sc); } static void sge_start_locked(struct ifnet *ifp) { struct sge_softc *sc; struct mbuf *m_head; int queued = 0; sc = ifp->if_softc; SGE_LOCK_ASSERT(sc); if ((sc->sge_flags & SGE_FLAG_LINK) == 0 || (ifp->if_drv_flags & (IFF_DRV_RUNNING | IFF_DRV_OACTIVE)) != IFF_DRV_RUNNING) return; for (queued = 0; !IFQ_DRV_IS_EMPTY(&ifp->if_snd); ) { if (sc->sge_cdata.sge_tx_cnt == SGE_TX_RING_CNT - 1) { ifp->if_drv_flags |= IFF_DRV_OACTIVE; break; } IFQ_DRV_DEQUEUE(&ifp->if_snd, m_head); if (m_head == NULL) break; if (sge_encap(sc, &m_head)) { IFQ_DRV_PREPEND(&ifp->if_snd, m_head); ifp->if_drv_flags |= IFF_DRV_OACTIVE; break; } queued++; /* * If there's a BPF listener, bounce a copy of this frame * to him. */ BPF_MTAP(ifp, m_head); } if (queued > 0) { bus_dmamap_sync(sc->sge_cdata.sge_tx_tag, sc->sge_cdata.sge_tx_dmamap, BUS_DMASYNC_PREREAD | BUS_DMASYNC_PREWRITE); CSR_WRITE_4(sc, TX_CTL, 0x1a00 | TX_CTL_ENB | TX_CTL_POLL); sc->sge_timer = 5; } } static void sge_init(void *arg) { struct sge_softc *sc; sc = arg; SGE_LOCK(sc); sge_init_locked(sc); SGE_UNLOCK(sc); } static void sge_init_locked(struct sge_softc *sc) { struct ifnet *ifp; struct mii_data *mii; int i; SGE_LOCK_ASSERT(sc); ifp = sc->sge_ifp; mii = device_get_softc(sc->sge_miibus); if ((ifp->if_drv_flags & IFF_DRV_RUNNING) != 0) return; /* * Cancel pending I/O and free all RX/TX buffers. */ sge_stop(sc); sge_reset(sc); /* Init circular RX list. */ if (sge_list_rx_init(sc) == ENOBUFS) { device_printf(sc->sge_dev, "no memory for Rx buffers\n"); sge_stop(sc); return; } /* Init TX descriptors. */ sge_list_tx_init(sc); /* * Load the address of the RX and TX lists. */ CSR_WRITE_4(sc, TX_DESC, SGE_ADDR_LO(sc->sge_ldata.sge_tx_paddr)); CSR_WRITE_4(sc, RX_DESC, SGE_ADDR_LO(sc->sge_ldata.sge_rx_paddr)); CSR_WRITE_4(sc, TxMacControl, 0x60); CSR_WRITE_4(sc, 0x6c, 0); CSR_WRITE_4(sc, RxWakeOnLan, 0); CSR_WRITE_4(sc, RxWakeOnLanData, 0); /* Allow receiving VLAN frames. */ CSR_WRITE_2(sc, RxMPSControl, ETHER_MAX_LEN + ETHER_VLAN_ENCAP_LEN); for (i = 0; i < ETHER_ADDR_LEN; i++) CSR_WRITE_1(sc, RxMacAddr + i, IF_LLADDR(ifp)[i]); sge_rxfilter(sc); /* Initialize default speed/duplex information. */ if ((sc->sge_flags & SGE_FLAG_FASTETHER) == 0) sc->sge_flags |= SGE_FLAG_SPEED_1000; sc->sge_flags |= SGE_FLAG_FDX; if ((sc->sge_flags & SGE_FLAG_RGMII) != 0) CSR_WRITE_4(sc, StationControl, 0x04008001); else CSR_WRITE_4(sc, StationControl, 0x04000001); /* * XXX Try to mitigate interrupts. */ if (sc->sge_intrcontrol != 0) CSR_WRITE_4(sc, IntrControl, sc->sge_intrcontrol); if (sc->sge_intrtimer != 0) CSR_WRITE_4(sc, IntrTimer, sc->sge_intrtimer); /* * Clear and enable interrupts. */ CSR_WRITE_4(sc, IntrStatus, 0xFFFFFFFF); CSR_WRITE_4(sc, IntrMask, SGE_INTRS); /* Enable receiver and transmitter. */ CSR_WRITE_4(sc, TX_CTL, 0x1a00 | TX_CTL_ENB); CSR_WRITE_4(sc, RX_CTL, 0x1a00 | 0x000c | RX_CTL_POLL | RX_CTL_ENB); ifp->if_drv_flags |= IFF_DRV_RUNNING; ifp->if_drv_flags &= ~IFF_DRV_OACTIVE; sc->sge_flags &= ~SGE_FLAG_LINK; mii_mediachg(mii); callout_reset(&sc->sge_stat_ch, hz, sge_tick, sc); } /* * Set media options. */ static int sge_ifmedia_upd(struct ifnet *ifp) { struct sge_softc *sc; struct mii_data *mii; int error; sc = ifp->if_softc; SGE_LOCK(sc); mii = device_get_softc(sc->sge_miibus); sc->sge_flags &= ~SGE_FLAG_LINK; if (mii->mii_instance) { struct mii_softc *miisc; LIST_FOREACH(miisc, &mii->mii_phys, mii_list) mii_phy_reset(miisc); } error = mii_mediachg(mii); SGE_UNLOCK(sc); return (error); } /* * Report current media status. */ static void sge_ifmedia_sts(struct ifnet *ifp, struct ifmediareq *ifmr) { struct sge_softc *sc; struct mii_data *mii; sc = ifp->if_softc; SGE_LOCK(sc); mii = device_get_softc(sc->sge_miibus); if ((ifp->if_flags & IFF_UP) == 0) { SGE_UNLOCK(sc); return; } mii_pollstat(mii); SGE_UNLOCK(sc); ifmr->ifm_active = mii->mii_media_active; ifmr->ifm_status = mii->mii_media_status; } static int sge_ioctl(struct ifnet *ifp, u_long command, caddr_t data) { struct sge_softc *sc; struct ifreq *ifr; struct mii_data *mii; int error = 0, mask; sc = ifp->if_softc; ifr = (struct ifreq *)data; switch(command) { case SIOCSIFFLAGS: SGE_LOCK(sc); if ((ifp->if_flags & IFF_UP) != 0) { if ((ifp->if_drv_flags & IFF_DRV_RUNNING) != 0 && ((ifp->if_flags ^ sc->sge_if_flags) & (IFF_PROMISC | IFF_ALLMULTI)) != 0) sge_rxfilter(sc); else sge_init_locked(sc); } else if ((ifp->if_drv_flags & IFF_DRV_RUNNING) != 0) sge_stop(sc); sc->sge_if_flags = ifp->if_flags; SGE_UNLOCK(sc); break; case SIOCSIFCAP: SGE_LOCK(sc); mask = ifr->ifr_reqcap ^ ifp->if_capenable; if ((mask & IFCAP_TXCSUM) != 0 && (ifp->if_capabilities & IFCAP_TXCSUM) != 0) { ifp->if_capenable ^= IFCAP_TXCSUM; if ((ifp->if_capenable & IFCAP_TXCSUM) != 0) ifp->if_hwassist |= SGE_CSUM_FEATURES; else ifp->if_hwassist &= ~SGE_CSUM_FEATURES; } if ((mask & IFCAP_RXCSUM) != 0 && (ifp->if_capabilities & IFCAP_RXCSUM) != 0) ifp->if_capenable ^= IFCAP_RXCSUM; SGE_UNLOCK(sc); break; case SIOCADDMULTI: case SIOCDELMULTI: SGE_LOCK(sc); if ((ifp->if_drv_flags & IFF_DRV_RUNNING) != 0) sge_rxfilter(sc); SGE_UNLOCK(sc); break; case SIOCGIFMEDIA: case SIOCSIFMEDIA: mii = device_get_softc(sc->sge_miibus); error = ifmedia_ioctl(ifp, ifr, &mii->mii_media, command); break; default: error = ether_ioctl(ifp, command, data); break; } return (error); } static void sge_watchdog(struct sge_softc *sc) { struct ifnet *ifp; SGE_LOCK_ASSERT(sc); if (sc->sge_timer == 0 || --sc->sge_timer > 0) return; ifp = sc->sge_ifp; if ((sc->sge_flags & SGE_FLAG_LINK) == 0) { if (1 || bootverbose) device_printf(sc->sge_dev, "watchdog timeout (lost link)\n"); ifp->if_oerrors++; ifp->if_drv_flags &= ~IFF_DRV_RUNNING; sge_init_locked(sc); return; } device_printf(sc->sge_dev, "watchdog timeout\n"); ifp->if_oerrors++; ifp->if_drv_flags &= ~IFF_DRV_RUNNING; sge_init_locked(sc); if (!IFQ_DRV_IS_EMPTY(&sc->sge_ifp->if_snd)) sge_start_locked(ifp); } /* * Stop the adapter and free any mbufs allocated to the * RX and TX lists. */ static void sge_stop(struct sge_softc *sc) { struct ifnet *ifp; ifp = sc->sge_ifp; SGE_LOCK_ASSERT(sc); sc->sge_timer = 0; callout_stop(&sc->sge_stat_ch); ifp->if_drv_flags &= ~(IFF_DRV_RUNNING | IFF_DRV_OACTIVE); CSR_WRITE_4(sc, IntrMask, 0); CSR_READ_4(sc, IntrMask); CSR_WRITE_4(sc, IntrStatus, 0xffffffff); /* Stop TX/RX MAC. */ CSR_WRITE_4(sc, TX_CTL, 0x1a00); CSR_WRITE_4(sc, RX_CTL, 0x1a00); /* XXX Can we assume active DMA cycles gone? */ DELAY(2000); CSR_WRITE_4(sc, IntrMask, 0); CSR_WRITE_4(sc, IntrStatus, 0xffffffff); sc->sge_flags &= ~SGE_FLAG_LINK; sge_list_rx_free(sc); sge_list_tx_free(sc); } Index: user/imp/tbemd/sys/dev/siis/siis.c =================================================================== --- user/imp/tbemd/sys/dev/siis/siis.c (revision 206684) +++ user/imp/tbemd/sys/dev/siis/siis.c (revision 206685) @@ -1,1808 +1,1809 @@ /*- * Copyright (c) 2009 Alexander Motin * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer, * without modification, immediately at the beginning of the file. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. * IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. */ #include __FBSDID("$FreeBSD$"); #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include "siis.h" #include #include #include #include #include /* local prototypes */ static int siis_setup_interrupt(device_t dev); static void siis_intr(void *data); static int siis_suspend(device_t dev); static int siis_resume(device_t dev); static int siis_ch_suspend(device_t dev); static int siis_ch_resume(device_t dev); static void siis_ch_intr_locked(void *data); static void siis_ch_intr(void *data); static void siis_begin_transaction(device_t dev, union ccb *ccb); static void siis_dmasetprd(void *arg, bus_dma_segment_t *segs, int nsegs, int error); static void siis_execute_transaction(struct siis_slot *slot); static void siis_timeout(struct siis_slot *slot); static void siis_end_transaction(struct siis_slot *slot, enum siis_err_type et); static int siis_setup_fis(device_t dev, struct siis_cmd *ctp, union ccb *ccb, int tag); static void siis_dmainit(device_t dev); static void siis_dmasetupc_cb(void *xsc, bus_dma_segment_t *segs, int nsegs, int error); static void siis_dmafini(device_t dev); static void siis_slotsalloc(device_t dev); static void siis_slotsfree(device_t dev); static void siis_reset(device_t dev); static void siis_portinit(device_t dev); static int siis_wait_ready(device_t dev, int t); static int siis_sata_connect(struct siis_channel *ch); static void siis_issue_read_log(device_t dev); static void siis_process_read_log(device_t dev, union ccb *ccb); static void siisaction(struct cam_sim *sim, union ccb *ccb); static void siispoll(struct cam_sim *sim); MALLOC_DEFINE(M_SIIS, "SIIS driver", "SIIS driver data buffers"); static struct { uint32_t id; const char *name; int ports; int quirks; #define SIIS_Q_SNTF 1 #define SIIS_Q_NOMSI 2 } siis_ids[] = { {0x31241095, "SiI3124", 4, 0}, {0x31248086, "SiI3124", 4, 0}, {0x31321095, "SiI3132", 2, SIIS_Q_SNTF|SIIS_Q_NOMSI}, {0x02421095, "SiI3132", 2, SIIS_Q_SNTF|SIIS_Q_NOMSI}, {0x02441095, "SiI3132", 2, SIIS_Q_SNTF|SIIS_Q_NOMSI}, {0x31311095, "SiI3131", 1, SIIS_Q_SNTF|SIIS_Q_NOMSI}, {0x35311095, "SiI3531", 1, SIIS_Q_SNTF|SIIS_Q_NOMSI}, {0, NULL, 0, 0} }; static int siis_probe(device_t dev) { char buf[64]; int i; uint32_t devid = pci_get_devid(dev); for (i = 0; siis_ids[i].id != 0; i++) { if (siis_ids[i].id == devid) { snprintf(buf, sizeof(buf), "%s SATA controller", siis_ids[i].name); device_set_desc_copy(dev, buf); return (BUS_PROBE_VENDOR); } } return (ENXIO); } static int siis_attach(device_t dev) { struct siis_controller *ctlr = device_get_softc(dev); uint32_t devid = pci_get_devid(dev); device_t child; int error, i, unit; ctlr->dev = dev; for (i = 0; siis_ids[i].id != 0; i++) { if (siis_ids[i].id == devid) break; } ctlr->quirks = siis_ids[i].quirks; /* Global memory */ ctlr->r_grid = PCIR_BAR(0); if (!(ctlr->r_gmem = bus_alloc_resource_any(dev, SYS_RES_MEMORY, &ctlr->r_grid, RF_ACTIVE))) return (ENXIO); ctlr->gctl = ATA_INL(ctlr->r_gmem, SIIS_GCTL); /* Channels memory */ ctlr->r_rid = PCIR_BAR(2); if (!(ctlr->r_mem = bus_alloc_resource_any(dev, SYS_RES_MEMORY, &ctlr->r_rid, RF_ACTIVE))) return (ENXIO); /* Setup our own memory management for channels. */ ctlr->sc_iomem.rm_type = RMAN_ARRAY; ctlr->sc_iomem.rm_descr = "I/O memory addresses"; if ((error = rman_init(&ctlr->sc_iomem)) != 0) { bus_release_resource(dev, SYS_RES_MEMORY, ctlr->r_rid, ctlr->r_mem); bus_release_resource(dev, SYS_RES_MEMORY, ctlr->r_grid, ctlr->r_gmem); return (error); } if ((error = rman_manage_region(&ctlr->sc_iomem, rman_get_start(ctlr->r_mem), rman_get_end(ctlr->r_mem))) != 0) { bus_release_resource(dev, SYS_RES_MEMORY, ctlr->r_rid, ctlr->r_mem); bus_release_resource(dev, SYS_RES_MEMORY, ctlr->r_grid, ctlr->r_gmem); rman_fini(&ctlr->sc_iomem); return (error); } + pci_enable_busmaster(dev); /* Reset controller */ siis_resume(dev); /* Number of HW channels */ ctlr->channels = siis_ids[i].ports; /* Setup interrupts. */ if (siis_setup_interrupt(dev)) { bus_release_resource(dev, SYS_RES_MEMORY, ctlr->r_rid, ctlr->r_mem); bus_release_resource(dev, SYS_RES_MEMORY, ctlr->r_grid, ctlr->r_gmem); rman_fini(&ctlr->sc_iomem); return ENXIO; } /* Attach all channels on this controller */ for (unit = 0; unit < ctlr->channels; unit++) { child = device_add_child(dev, "siisch", -1); if (child == NULL) device_printf(dev, "failed to add channel device\n"); else device_set_ivars(child, (void *)(intptr_t)unit); } bus_generic_attach(dev); return 0; } static int siis_detach(device_t dev) { struct siis_controller *ctlr = device_get_softc(dev); device_t *children; int nchildren, i; /* Detach & delete all children */ if (!device_get_children(dev, &children, &nchildren)) { for (i = 0; i < nchildren; i++) device_delete_child(dev, children[i]); free(children, M_TEMP); } /* Free interrupts. */ if (ctlr->irq.r_irq) { bus_teardown_intr(dev, ctlr->irq.r_irq, ctlr->irq.handle); bus_release_resource(dev, SYS_RES_IRQ, ctlr->irq.r_irq_rid, ctlr->irq.r_irq); } pci_release_msi(dev); /* Free memory. */ rman_fini(&ctlr->sc_iomem); bus_release_resource(dev, SYS_RES_MEMORY, ctlr->r_rid, ctlr->r_mem); bus_release_resource(dev, SYS_RES_MEMORY, ctlr->r_grid, ctlr->r_gmem); return (0); } static int siis_suspend(device_t dev) { struct siis_controller *ctlr = device_get_softc(dev); bus_generic_suspend(dev); /* Put controller into reset state. */ ctlr->gctl |= SIIS_GCTL_GRESET; ATA_OUTL(ctlr->r_gmem, SIIS_GCTL, ctlr->gctl); return 0; } static int siis_resume(device_t dev) { struct siis_controller *ctlr = device_get_softc(dev); /* Set PCIe max read request size to at least 1024 bytes */ if (pci_get_max_read_req(dev) < 1024) pci_set_max_read_req(dev, 1024); /* Put controller into reset state. */ ctlr->gctl |= SIIS_GCTL_GRESET; ATA_OUTL(ctlr->r_gmem, SIIS_GCTL, ctlr->gctl); DELAY(10000); /* Get controller out of reset state and enable port interrupts. */ ctlr->gctl &= ~(SIIS_GCTL_GRESET | SIIS_GCTL_I2C_IE); ctlr->gctl |= 0x0000000f; ATA_OUTL(ctlr->r_gmem, SIIS_GCTL, ctlr->gctl); return (bus_generic_resume(dev)); } static int siis_setup_interrupt(device_t dev) { struct siis_controller *ctlr = device_get_softc(dev); int msi = ctlr->quirks & SIIS_Q_NOMSI ? 0 : 1; /* Process hints. */ resource_int_value(device_get_name(dev), device_get_unit(dev), "msi", &msi); if (msi < 0) msi = 0; else if (msi > 0) msi = min(1, pci_msi_count(dev)); /* Allocate MSI if needed/present. */ if (msi && pci_alloc_msi(dev, &msi) != 0) msi = 0; /* Allocate all IRQs. */ ctlr->irq.r_irq_rid = msi ? 1 : 0; if (!(ctlr->irq.r_irq = bus_alloc_resource_any(dev, SYS_RES_IRQ, &ctlr->irq.r_irq_rid, RF_SHAREABLE | RF_ACTIVE))) { device_printf(dev, "unable to map interrupt\n"); return ENXIO; } if ((bus_setup_intr(dev, ctlr->irq.r_irq, ATA_INTR_FLAGS, NULL, siis_intr, ctlr, &ctlr->irq.handle))) { /* SOS XXX release r_irq */ device_printf(dev, "unable to setup interrupt\n"); return ENXIO; } return (0); } /* * Common case interrupt handler. */ static void siis_intr(void *data) { struct siis_controller *ctlr = (struct siis_controller *)data; u_int32_t is; void *arg; int unit; is = ATA_INL(ctlr->r_gmem, SIIS_IS); for (unit = 0; unit < ctlr->channels; unit++) { if ((is & SIIS_IS_PORT(unit)) != 0 && (arg = ctlr->interrupt[unit].argument)) { ctlr->interrupt[unit].function(arg); } } /* Acknowledge interrupt, if MSI enabled. */ if (ctlr->irq.r_irq_rid) { ATA_OUTL(ctlr->r_gmem, SIIS_GCTL, ctlr->gctl | SIIS_GCTL_MSIACK); } } static struct resource * siis_alloc_resource(device_t dev, device_t child, int type, int *rid, u_long start, u_long end, u_long count, u_int flags) { struct siis_controller *ctlr = device_get_softc(dev); int unit = ((struct siis_channel *)device_get_softc(child))->unit; struct resource *res = NULL; int offset = unit << 13; long st; switch (type) { case SYS_RES_MEMORY: st = rman_get_start(ctlr->r_mem); res = rman_reserve_resource(&ctlr->sc_iomem, st + offset, st + offset + 0x2000, 0x2000, RF_ACTIVE, child); if (res) { bus_space_handle_t bsh; bus_space_tag_t bst; bsh = rman_get_bushandle(ctlr->r_mem); bst = rman_get_bustag(ctlr->r_mem); bus_space_subregion(bst, bsh, offset, 0x2000, &bsh); rman_set_bushandle(res, bsh); rman_set_bustag(res, bst); } break; case SYS_RES_IRQ: if (*rid == ATA_IRQ_RID) res = ctlr->irq.r_irq; break; } return (res); } static int siis_release_resource(device_t dev, device_t child, int type, int rid, struct resource *r) { switch (type) { case SYS_RES_MEMORY: rman_release_resource(r); return (0); case SYS_RES_IRQ: if (rid != ATA_IRQ_RID) return ENOENT; return (0); } return (EINVAL); } static int siis_setup_intr(device_t dev, device_t child, struct resource *irq, int flags, driver_filter_t *filter, driver_intr_t *function, void *argument, void **cookiep) { struct siis_controller *ctlr = device_get_softc(dev); int unit = (intptr_t)device_get_ivars(child); if (filter != NULL) { printf("siis.c: we cannot use a filter here\n"); return (EINVAL); } ctlr->interrupt[unit].function = function; ctlr->interrupt[unit].argument = argument; return (0); } static int siis_teardown_intr(device_t dev, device_t child, struct resource *irq, void *cookie) { struct siis_controller *ctlr = device_get_softc(dev); int unit = (intptr_t)device_get_ivars(child); ctlr->interrupt[unit].function = NULL; ctlr->interrupt[unit].argument = NULL; return (0); } static int siis_print_child(device_t dev, device_t child) { int retval; retval = bus_print_child_header(dev, child); retval += printf(" at channel %d", (int)(intptr_t)device_get_ivars(child)); retval += bus_print_child_footer(dev, child); return (retval); } devclass_t siis_devclass; static device_method_t siis_methods[] = { DEVMETHOD(device_probe, siis_probe), DEVMETHOD(device_attach, siis_attach), DEVMETHOD(device_detach, siis_detach), DEVMETHOD(device_suspend, siis_suspend), DEVMETHOD(device_resume, siis_resume), DEVMETHOD(bus_print_child, siis_print_child), DEVMETHOD(bus_alloc_resource, siis_alloc_resource), DEVMETHOD(bus_release_resource, siis_release_resource), DEVMETHOD(bus_setup_intr, siis_setup_intr), DEVMETHOD(bus_teardown_intr,siis_teardown_intr), { 0, 0 } }; static driver_t siis_driver = { "siis", siis_methods, sizeof(struct siis_controller) }; DRIVER_MODULE(siis, pci, siis_driver, siis_devclass, 0, 0); MODULE_VERSION(siis, 1); MODULE_DEPEND(siis, cam, 1, 1, 1); static int siis_ch_probe(device_t dev) { device_set_desc_copy(dev, "SIIS channel"); return (0); } static int siis_ch_attach(device_t dev) { struct siis_controller *ctlr = device_get_softc(device_get_parent(dev)); struct siis_channel *ch = device_get_softc(dev); struct cam_devq *devq; int rid, error, i, sata_rev = 0; ch->dev = dev; ch->unit = (intptr_t)device_get_ivars(dev); ch->quirks = ctlr->quirks; resource_int_value(device_get_name(dev), device_get_unit(dev), "pm_level", &ch->pm_level); resource_int_value(device_get_name(dev), device_get_unit(dev), "sata_rev", &sata_rev); for (i = 0; i < 16; i++) { ch->user[i].revision = sata_rev; ch->user[i].mode = 0; ch->user[i].bytecount = 8192; ch->user[i].tags = SIIS_MAX_SLOTS; ch->curr[i] = ch->user[i]; } mtx_init(&ch->mtx, "SIIS channel lock", NULL, MTX_DEF); rid = ch->unit; if (!(ch->r_mem = bus_alloc_resource_any(dev, SYS_RES_MEMORY, &rid, RF_ACTIVE))) return (ENXIO); siis_dmainit(dev); siis_slotsalloc(dev); siis_ch_resume(dev); mtx_lock(&ch->mtx); rid = ATA_IRQ_RID; if (!(ch->r_irq = bus_alloc_resource_any(dev, SYS_RES_IRQ, &rid, RF_SHAREABLE | RF_ACTIVE))) { bus_release_resource(dev, SYS_RES_MEMORY, ch->unit, ch->r_mem); device_printf(dev, "Unable to map interrupt\n"); return (ENXIO); } if ((bus_setup_intr(dev, ch->r_irq, ATA_INTR_FLAGS, NULL, siis_ch_intr_locked, dev, &ch->ih))) { device_printf(dev, "Unable to setup interrupt\n"); error = ENXIO; goto err1; } /* Create the device queue for our SIM. */ devq = cam_simq_alloc(SIIS_MAX_SLOTS); if (devq == NULL) { device_printf(dev, "Unable to allocate simq\n"); error = ENOMEM; goto err1; } /* Construct SIM entry */ ch->sim = cam_sim_alloc(siisaction, siispoll, "siisch", ch, device_get_unit(dev), &ch->mtx, 2, SIIS_MAX_SLOTS, devq); if (ch->sim == NULL) { device_printf(dev, "unable to allocate sim\n"); error = ENOMEM; goto err2; } if (xpt_bus_register(ch->sim, dev, 0) != CAM_SUCCESS) { device_printf(dev, "unable to register xpt bus\n"); error = ENXIO; goto err2; } if (xpt_create_path(&ch->path, /*periph*/NULL, cam_sim_path(ch->sim), CAM_TARGET_WILDCARD, CAM_LUN_WILDCARD) != CAM_REQ_CMP) { device_printf(dev, "unable to create path\n"); error = ENXIO; goto err3; } mtx_unlock(&ch->mtx); return (0); err3: xpt_bus_deregister(cam_sim_path(ch->sim)); err2: cam_sim_free(ch->sim, /*free_devq*/TRUE); err1: bus_release_resource(dev, SYS_RES_IRQ, ATA_IRQ_RID, ch->r_irq); bus_release_resource(dev, SYS_RES_MEMORY, ch->unit, ch->r_mem); mtx_unlock(&ch->mtx); return (error); } static int siis_ch_detach(device_t dev) { struct siis_channel *ch = device_get_softc(dev); mtx_lock(&ch->mtx); xpt_async(AC_LOST_DEVICE, ch->path, NULL); xpt_free_path(ch->path); xpt_bus_deregister(cam_sim_path(ch->sim)); cam_sim_free(ch->sim, /*free_devq*/TRUE); mtx_unlock(&ch->mtx); bus_teardown_intr(dev, ch->r_irq, ch->ih); bus_release_resource(dev, SYS_RES_IRQ, ATA_IRQ_RID, ch->r_irq); siis_ch_suspend(dev); siis_slotsfree(dev); siis_dmafini(dev); bus_release_resource(dev, SYS_RES_MEMORY, ch->unit, ch->r_mem); mtx_destroy(&ch->mtx); return (0); } static int siis_ch_suspend(device_t dev) { struct siis_channel *ch = device_get_softc(dev); /* Put port into reset state. */ ATA_OUTL(ch->r_mem, SIIS_P_CTLSET, SIIS_P_CTL_PORT_RESET); return (0); } static int siis_ch_resume(device_t dev) { struct siis_channel *ch = device_get_softc(dev); /* Get port out of reset state. */ ATA_OUTL(ch->r_mem, SIIS_P_CTLCLR, SIIS_P_CTL_PORT_RESET); ATA_OUTL(ch->r_mem, SIIS_P_CTLCLR, SIIS_P_CTL_32BIT); if (ch->pm_present) ATA_OUTL(ch->r_mem, SIIS_P_CTLSET, SIIS_P_CTL_PME); else ATA_OUTL(ch->r_mem, SIIS_P_CTLCLR, SIIS_P_CTL_PME); /* Enable port interrupts */ ATA_OUTL(ch->r_mem, SIIS_P_IESET, SIIS_P_IX_ENABLED); return (0); } devclass_t siisch_devclass; static device_method_t siisch_methods[] = { DEVMETHOD(device_probe, siis_ch_probe), DEVMETHOD(device_attach, siis_ch_attach), DEVMETHOD(device_detach, siis_ch_detach), DEVMETHOD(device_suspend, siis_ch_suspend), DEVMETHOD(device_resume, siis_ch_resume), { 0, 0 } }; static driver_t siisch_driver = { "siisch", siisch_methods, sizeof(struct siis_channel) }; DRIVER_MODULE(siisch, siis, siisch_driver, siis_devclass, 0, 0); struct siis_dc_cb_args { bus_addr_t maddr; int error; }; static void siis_dmainit(device_t dev) { struct siis_channel *ch = device_get_softc(dev); struct siis_dc_cb_args dcba; /* Command area. */ if (bus_dma_tag_create(bus_get_dma_tag(dev), 1024, 0, BUS_SPACE_MAXADDR, BUS_SPACE_MAXADDR, NULL, NULL, SIIS_WORK_SIZE, 1, SIIS_WORK_SIZE, 0, NULL, NULL, &ch->dma.work_tag)) goto error; if (bus_dmamem_alloc(ch->dma.work_tag, (void **)&ch->dma.work, 0, &ch->dma.work_map)) goto error; if (bus_dmamap_load(ch->dma.work_tag, ch->dma.work_map, ch->dma.work, SIIS_WORK_SIZE, siis_dmasetupc_cb, &dcba, 0) || dcba.error) { bus_dmamem_free(ch->dma.work_tag, ch->dma.work, ch->dma.work_map); goto error; } ch->dma.work_bus = dcba.maddr; /* Data area. */ if (bus_dma_tag_create(bus_get_dma_tag(dev), 1, 0, BUS_SPACE_MAXADDR, BUS_SPACE_MAXADDR, NULL, NULL, SIIS_SG_ENTRIES * PAGE_SIZE * SIIS_MAX_SLOTS, SIIS_SG_ENTRIES, 0xFFFFFFFF, 0, busdma_lock_mutex, &ch->mtx, &ch->dma.data_tag)) { goto error; } return; error: device_printf(dev, "WARNING - DMA initialization failed\n"); siis_dmafini(dev); } static void siis_dmasetupc_cb(void *xsc, bus_dma_segment_t *segs, int nsegs, int error) { struct siis_dc_cb_args *dcba = (struct siis_dc_cb_args *)xsc; if (!(dcba->error = error)) dcba->maddr = segs[0].ds_addr; } static void siis_dmafini(device_t dev) { struct siis_channel *ch = device_get_softc(dev); if (ch->dma.data_tag) { bus_dma_tag_destroy(ch->dma.data_tag); ch->dma.data_tag = NULL; } if (ch->dma.work_bus) { bus_dmamap_unload(ch->dma.work_tag, ch->dma.work_map); bus_dmamem_free(ch->dma.work_tag, ch->dma.work, ch->dma.work_map); ch->dma.work_bus = 0; ch->dma.work_map = NULL; ch->dma.work = NULL; } if (ch->dma.work_tag) { bus_dma_tag_destroy(ch->dma.work_tag); ch->dma.work_tag = NULL; } } static void siis_slotsalloc(device_t dev) { struct siis_channel *ch = device_get_softc(dev); int i; /* Alloc and setup command/dma slots */ bzero(ch->slot, sizeof(ch->slot)); for (i = 0; i < SIIS_MAX_SLOTS; i++) { struct siis_slot *slot = &ch->slot[i]; slot->dev = dev; slot->slot = i; slot->state = SIIS_SLOT_EMPTY; slot->ccb = NULL; callout_init_mtx(&slot->timeout, &ch->mtx, 0); if (bus_dmamap_create(ch->dma.data_tag, 0, &slot->dma.data_map)) device_printf(ch->dev, "FAILURE - create data_map\n"); } } static void siis_slotsfree(device_t dev) { struct siis_channel *ch = device_get_softc(dev); int i; /* Free all dma slots */ for (i = 0; i < SIIS_MAX_SLOTS; i++) { struct siis_slot *slot = &ch->slot[i]; callout_drain(&slot->timeout); if (slot->dma.data_map) { bus_dmamap_destroy(ch->dma.data_tag, slot->dma.data_map); slot->dma.data_map = NULL; } } } static void siis_notify_events(device_t dev) { struct siis_channel *ch = device_get_softc(dev); struct cam_path *dpath; u_int32_t status; int i; if (ch->quirks & SIIS_Q_SNTF) { status = ATA_INL(ch->r_mem, SIIS_P_SNTF); ATA_OUTL(ch->r_mem, SIIS_P_SNTF, status); } else { /* * Without SNTF we have no idea which device sent notification. * If PMP is connected, assume it, else - device. */ status = (ch->pm_present) ? 0x8000 : 0x0001; } if (bootverbose) device_printf(dev, "SNTF 0x%04x\n", status); for (i = 0; i < 16; i++) { if ((status & (1 << i)) == 0) continue; if (xpt_create_path(&dpath, NULL, xpt_path_path_id(ch->path), i, 0) == CAM_REQ_CMP) { xpt_async(AC_SCSI_AEN, dpath, NULL); xpt_free_path(dpath); } } } static void siis_phy_check_events(device_t dev) { struct siis_channel *ch = device_get_softc(dev); /* If we have a connection event, deal with it */ if (ch->pm_level == 0) { u_int32_t status = ATA_INL(ch->r_mem, SIIS_P_SSTS); union ccb *ccb; if (bootverbose) { if (((status & ATA_SS_DET_MASK) == ATA_SS_DET_PHY_ONLINE) && ((status & ATA_SS_SPD_MASK) != ATA_SS_SPD_NO_SPEED) && ((status & ATA_SS_IPM_MASK) == ATA_SS_IPM_ACTIVE)) { device_printf(dev, "CONNECT requested\n"); } else device_printf(dev, "DISCONNECT requested\n"); } siis_reset(dev); if ((ccb = xpt_alloc_ccb_nowait()) == NULL) return; if (xpt_create_path(&ccb->ccb_h.path, NULL, cam_sim_path(ch->sim), CAM_TARGET_WILDCARD, CAM_LUN_WILDCARD) != CAM_REQ_CMP) { xpt_free_ccb(ccb); return; } xpt_rescan(ccb); } } static void siis_ch_intr_locked(void *data) { device_t dev = (device_t)data; struct siis_channel *ch = device_get_softc(dev); mtx_lock(&ch->mtx); siis_ch_intr(data); mtx_unlock(&ch->mtx); } static void siis_ch_intr(void *data) { device_t dev = (device_t)data; struct siis_channel *ch = device_get_softc(dev); uint32_t istatus, sstatus, ctx, estatus, ok, err = 0; enum siis_err_type et; int i, ccs, port, tslots; mtx_assert(&ch->mtx, MA_OWNED); /* Read command statuses. */ sstatus = ATA_INL(ch->r_mem, SIIS_P_SS); ok = ch->rslots & ~sstatus; /* Complete all successfull commands. */ for (i = 0; i < SIIS_MAX_SLOTS; i++) { if ((ok >> i) & 1) siis_end_transaction(&ch->slot[i], SIIS_ERR_NONE); } /* Do we have any other events? */ if ((sstatus & SIIS_P_SS_ATTN) == 0) return; /* Read and clear interrupt statuses. */ istatus = ATA_INL(ch->r_mem, SIIS_P_IS) & (0xFFFF & ~SIIS_P_IX_COMMCOMP); ATA_OUTL(ch->r_mem, SIIS_P_IS, istatus); /* Process PHY events */ if (istatus & SIIS_P_IX_PHYRDYCHG) siis_phy_check_events(dev); /* Process NOTIFY events */ if (istatus & SIIS_P_IX_SDBN) siis_notify_events(dev); /* Process command errors */ if (istatus & SIIS_P_IX_COMMERR) { estatus = ATA_INL(ch->r_mem, SIIS_P_CMDERR); ctx = ATA_INL(ch->r_mem, SIIS_P_CTX); ccs = (ctx & SIIS_P_CTX_SLOT) >> SIIS_P_CTX_SLOT_SHIFT; port = (ctx & SIIS_P_CTX_PMP) >> SIIS_P_CTX_PMP_SHIFT; err = ch->rslots & sstatus; //device_printf(dev, "%s ERROR ss %08x is %08x rs %08x es %d act %d port %d serr %08x\n", // __func__, sstatus, istatus, ch->rslots, estatus, ccs, port, // ATA_INL(ch->r_mem, SIIS_P_SERR)); if (!ch->readlog && !ch->recovery) { xpt_freeze_simq(ch->sim, ch->numrslots); ch->recovery = 1; } if (ch->frozen) { union ccb *fccb = ch->frozen; ch->frozen = NULL; fccb->ccb_h.status &= ~CAM_STATUS_MASK; fccb->ccb_h.status |= CAM_REQUEUE_REQ | CAM_RELEASE_SIMQ; if (!(fccb->ccb_h.status & CAM_DEV_QFRZN)) { xpt_freeze_devq(fccb->ccb_h.path, 1); fccb->ccb_h.status |= CAM_DEV_QFRZN; } xpt_done(fccb); } if (estatus == SIIS_P_CMDERR_DEV || estatus == SIIS_P_CMDERR_SDB || estatus == SIIS_P_CMDERR_DATAFIS) { tslots = ch->numtslots[port]; for (i = 0; i < SIIS_MAX_SLOTS; i++) { /* XXX: requests in loading state. */ if (((ch->rslots >> i) & 1) == 0) continue; if (ch->slot[i].ccb->ccb_h.target_id != port) continue; if (tslots == 0) { /* Untagged operation. */ if (i == ccs) et = SIIS_ERR_TFE; else et = SIIS_ERR_INNOCENT; } else { /* Tagged operation. */ et = SIIS_ERR_NCQ; } siis_end_transaction(&ch->slot[i], et); } /* * We can't reinit port if there are some other * commands active, use resume to complete them. */ if (ch->rslots != 0) ATA_OUTL(ch->r_mem, SIIS_P_CTLSET, SIIS_P_CTL_RESUME); } else { if (estatus == SIIS_P_CMDERR_SENDFIS || estatus == SIIS_P_CMDERR_INCSTATE || estatus == SIIS_P_CMDERR_PPE || estatus == SIIS_P_CMDERR_SERVICE) { et = SIIS_ERR_SATA; } else et = SIIS_ERR_INVALID; for (i = 0; i < SIIS_MAX_SLOTS; i++) { /* XXX: requests in loading state. */ if (((ch->rslots >> i) & 1) == 0) continue; siis_end_transaction(&ch->slot[i], et); } } } } /* Must be called with channel locked. */ static int siis_check_collision(device_t dev, union ccb *ccb) { struct siis_channel *ch = device_get_softc(dev); mtx_assert(&ch->mtx, MA_OWNED); if ((ccb->ccb_h.func_code == XPT_ATA_IO) && (ccb->ataio.cmd.flags & CAM_ATAIO_FPDMA)) { /* Tagged command while we have no supported tag free. */ if (((~ch->oslots) & (0x7fffffff >> (31 - ch->curr[ccb->ccb_h.target_id].tags))) == 0) return (1); } if ((ccb->ccb_h.func_code == XPT_ATA_IO) && (ccb->ataio.cmd.flags & (CAM_ATAIO_CONTROL | CAM_ATAIO_NEEDRESULT))) { /* Atomic command while anything active. */ if (ch->numrslots != 0) return (1); } /* We have some atomic command running. */ if (ch->aslots != 0) return (1); return (0); } /* Must be called with channel locked. */ static void siis_begin_transaction(device_t dev, union ccb *ccb) { struct siis_channel *ch = device_get_softc(dev); struct siis_slot *slot; int tag, tags; mtx_assert(&ch->mtx, MA_OWNED); /* Choose empty slot. */ tags = SIIS_MAX_SLOTS; if ((ccb->ccb_h.func_code == XPT_ATA_IO) && (ccb->ataio.cmd.flags & CAM_ATAIO_FPDMA)) tags = ch->curr[ccb->ccb_h.target_id].tags; tag = fls((~ch->oslots) & (0x7fffffff >> (31 - tags))) - 1; /* Occupy chosen slot. */ slot = &ch->slot[tag]; slot->ccb = ccb; /* Update channel stats. */ ch->oslots |= (1 << slot->slot); ch->numrslots++; if ((ccb->ccb_h.func_code == XPT_ATA_IO) && (ccb->ataio.cmd.flags & CAM_ATAIO_FPDMA)) { ch->numtslots[ccb->ccb_h.target_id]++; } if ((ccb->ccb_h.func_code == XPT_ATA_IO) && (ccb->ataio.cmd.flags & (CAM_ATAIO_CONTROL | CAM_ATAIO_NEEDRESULT))) ch->aslots |= (1 << slot->slot); slot->dma.nsegs = 0; /* If request moves data, setup and load SG list */ if ((ccb->ccb_h.flags & CAM_DIR_MASK) != CAM_DIR_NONE) { void *buf; bus_size_t size; slot->state = SIIS_SLOT_LOADING; if (ccb->ccb_h.func_code == XPT_ATA_IO) { buf = ccb->ataio.data_ptr; size = ccb->ataio.dxfer_len; } else { buf = ccb->csio.data_ptr; size = ccb->csio.dxfer_len; } bus_dmamap_load(ch->dma.data_tag, slot->dma.data_map, buf, size, siis_dmasetprd, slot, 0); } else siis_execute_transaction(slot); } /* Locked by busdma engine. */ static void siis_dmasetprd(void *arg, bus_dma_segment_t *segs, int nsegs, int error) { struct siis_slot *slot = arg; struct siis_channel *ch = device_get_softc(slot->dev); struct siis_cmd *ctp; struct siis_dma_prd *prd; int i; mtx_assert(&ch->mtx, MA_OWNED); if (error) { device_printf(slot->dev, "DMA load error\n"); if (!ch->readlog) xpt_freeze_simq(ch->sim, 1); siis_end_transaction(slot, SIIS_ERR_INVALID); return; } KASSERT(nsegs <= SIIS_SG_ENTRIES, ("too many DMA segment entries\n")); /* Get a piece of the workspace for this request */ ctp = (struct siis_cmd *) (ch->dma.work + SIIS_CT_OFFSET + (SIIS_CT_SIZE * slot->slot)); /* Fill S/G table */ if (slot->ccb->ccb_h.func_code == XPT_ATA_IO) prd = &ctp->u.ata.prd[0]; else prd = &ctp->u.atapi.prd[0]; for (i = 0; i < nsegs; i++) { prd[i].dba = htole64(segs[i].ds_addr); prd[i].dbc = htole32(segs[i].ds_len); prd[i].control = 0; } prd[nsegs - 1].control = htole32(SIIS_PRD_TRM); slot->dma.nsegs = nsegs; bus_dmamap_sync(ch->dma.data_tag, slot->dma.data_map, ((slot->ccb->ccb_h.flags & CAM_DIR_IN) ? BUS_DMASYNC_PREREAD : BUS_DMASYNC_PREWRITE)); siis_execute_transaction(slot); } /* Must be called with channel locked. */ static void siis_execute_transaction(struct siis_slot *slot) { device_t dev = slot->dev; struct siis_channel *ch = device_get_softc(dev); struct siis_cmd *ctp; union ccb *ccb = slot->ccb; u_int64_t prb_bus; mtx_assert(&ch->mtx, MA_OWNED); /* Get a piece of the workspace for this request */ ctp = (struct siis_cmd *) (ch->dma.work + SIIS_CT_OFFSET + (SIIS_CT_SIZE * slot->slot)); ctp->control = 0; ctp->protocol_override = 0; ctp->transfer_count = 0; /* Special handling for Soft Reset command. */ if (ccb->ccb_h.func_code == XPT_ATA_IO) { if (ccb->ataio.cmd.flags & CAM_ATAIO_CONTROL) { ctp->control |= htole16(SIIS_PRB_SOFT_RESET); } else { ctp->control |= htole16(SIIS_PRB_PROTOCOL_OVERRIDE); if (ccb->ataio.cmd.flags & CAM_ATAIO_FPDMA) { ctp->protocol_override |= htole16(SIIS_PRB_PROTO_NCQ); } if ((ccb->ccb_h.flags & CAM_DIR_MASK) == CAM_DIR_IN) { ctp->protocol_override |= htole16(SIIS_PRB_PROTO_READ); } else if ((ccb->ccb_h.flags & CAM_DIR_MASK) == CAM_DIR_OUT) { ctp->protocol_override |= htole16(SIIS_PRB_PROTO_WRITE); } } } else if (ccb->ccb_h.func_code == XPT_SCSI_IO) { if ((ccb->ccb_h.flags & CAM_DIR_MASK) == CAM_DIR_IN) ctp->control |= htole16(SIIS_PRB_PACKET_READ); else if ((ccb->ccb_h.flags & CAM_DIR_MASK) == CAM_DIR_OUT) ctp->control |= htole16(SIIS_PRB_PACKET_WRITE); } /* Special handling for Soft Reset command. */ if ((ccb->ccb_h.func_code == XPT_ATA_IO) && (ccb->ataio.cmd.flags & CAM_ATAIO_CONTROL) && (ccb->ataio.cmd.control & ATA_A_RESET)) { /* Kick controller into sane state */ siis_portinit(dev); } /* Setup the FIS for this request */ if (!siis_setup_fis(dev, ctp, ccb, slot->slot)) { device_printf(ch->dev, "Setting up SATA FIS failed\n"); if (!ch->readlog) xpt_freeze_simq(ch->sim, 1); siis_end_transaction(slot, SIIS_ERR_INVALID); return; } bus_dmamap_sync(ch->dma.work_tag, ch->dma.work_map, BUS_DMASYNC_PREWRITE); /* Issue command to the controller. */ slot->state = SIIS_SLOT_RUNNING; ch->rslots |= (1 << slot->slot); prb_bus = ch->dma.work_bus + SIIS_CT_OFFSET + (SIIS_CT_SIZE * slot->slot); ATA_OUTL(ch->r_mem, SIIS_P_CACTL(slot->slot), prb_bus); ATA_OUTL(ch->r_mem, SIIS_P_CACTH(slot->slot), prb_bus >> 32); /* Start command execution timeout */ callout_reset(&slot->timeout, (int)ccb->ccb_h.timeout * hz / 1000, (timeout_t*)siis_timeout, slot); return; } /* Must be called with channel locked. */ static void siis_process_timeout(device_t dev) { struct siis_channel *ch = device_get_softc(dev); int i; mtx_assert(&ch->mtx, MA_OWNED); if (!ch->readlog && !ch->recovery) { xpt_freeze_simq(ch->sim, ch->numrslots); ch->recovery = 1; } /* Handle the rest of commands. */ for (i = 0; i < SIIS_MAX_SLOTS; i++) { /* Do we have a running request on slot? */ if (ch->slot[i].state < SIIS_SLOT_RUNNING) continue; siis_end_transaction(&ch->slot[i], SIIS_ERR_TIMEOUT); } } /* Must be called with channel locked. */ static void siis_rearm_timeout(device_t dev) { struct siis_channel *ch = device_get_softc(dev); int i; mtx_assert(&ch->mtx, MA_OWNED); for (i = 0; i < SIIS_MAX_SLOTS; i++) { struct siis_slot *slot = &ch->slot[i]; /* Do we have a running request on slot? */ if (slot->state < SIIS_SLOT_RUNNING) continue; if ((ch->toslots & (1 << i)) == 0) continue; callout_reset(&slot->timeout, (int)slot->ccb->ccb_h.timeout * hz / 1000, (timeout_t*)siis_timeout, slot); } } /* Locked by callout mechanism. */ static void siis_timeout(struct siis_slot *slot) { device_t dev = slot->dev; struct siis_channel *ch = device_get_softc(dev); mtx_assert(&ch->mtx, MA_OWNED); /* Check for stale timeout. */ if (slot->state < SIIS_SLOT_RUNNING) return; device_printf(dev, "Timeout on slot %d\n", slot->slot); device_printf(dev, "%s is %08x ss %08x rs %08x es %08x sts %08x serr %08x\n", __func__, ATA_INL(ch->r_mem, SIIS_P_IS), ATA_INL(ch->r_mem, SIIS_P_SS), ch->rslots, ATA_INL(ch->r_mem, SIIS_P_CMDERR), ATA_INL(ch->r_mem, SIIS_P_STS), ATA_INL(ch->r_mem, SIIS_P_SERR)); if (ch->toslots == 0) xpt_freeze_simq(ch->sim, 1); ch->toslots |= (1 << slot->slot); if ((ch->rslots & ~ch->toslots) == 0) siis_process_timeout(dev); else device_printf(dev, " ... waiting for slots %08x\n", ch->rslots & ~ch->toslots); } /* Must be called with channel locked. */ static void siis_end_transaction(struct siis_slot *slot, enum siis_err_type et) { device_t dev = slot->dev; struct siis_channel *ch = device_get_softc(dev); union ccb *ccb = slot->ccb; mtx_assert(&ch->mtx, MA_OWNED); bus_dmamap_sync(ch->dma.work_tag, ch->dma.work_map, BUS_DMASYNC_POSTWRITE); /* Read result registers to the result struct * May be incorrect if several commands finished same time, * so read only when sure or have to. */ if (ccb->ccb_h.func_code == XPT_ATA_IO) { struct ata_res *res = &ccb->ataio.res; if ((et == SIIS_ERR_TFE) || (ccb->ataio.cmd.flags & CAM_ATAIO_NEEDRESULT)) { int offs = SIIS_P_LRAM_SLOT(slot->slot) + 8; res->status = ATA_INB(ch->r_mem, offs + 2); res->error = ATA_INB(ch->r_mem, offs + 3); res->lba_low = ATA_INB(ch->r_mem, offs + 4); res->lba_mid = ATA_INB(ch->r_mem, offs + 5); res->lba_high = ATA_INB(ch->r_mem, offs + 6); res->device = ATA_INB(ch->r_mem, offs + 7); res->lba_low_exp = ATA_INB(ch->r_mem, offs + 8); res->lba_mid_exp = ATA_INB(ch->r_mem, offs + 9); res->lba_high_exp = ATA_INB(ch->r_mem, offs + 10); res->sector_count = ATA_INB(ch->r_mem, offs + 12); res->sector_count_exp = ATA_INB(ch->r_mem, offs + 13); } else bzero(res, sizeof(*res)); } if ((ccb->ccb_h.flags & CAM_DIR_MASK) != CAM_DIR_NONE) { bus_dmamap_sync(ch->dma.data_tag, slot->dma.data_map, (ccb->ccb_h.flags & CAM_DIR_IN) ? BUS_DMASYNC_POSTREAD : BUS_DMASYNC_POSTWRITE); bus_dmamap_unload(ch->dma.data_tag, slot->dma.data_map); } /* Set proper result status. */ if (et != SIIS_ERR_NONE || ch->recovery) { ch->eslots |= (1 << slot->slot); ccb->ccb_h.status |= CAM_RELEASE_SIMQ; } /* In case of error, freeze device for proper recovery. */ if (et != SIIS_ERR_NONE && !(ccb->ccb_h.status & CAM_DEV_QFRZN)) { xpt_freeze_devq(ccb->ccb_h.path, 1); ccb->ccb_h.status |= CAM_DEV_QFRZN; } ccb->ccb_h.status &= ~CAM_STATUS_MASK; switch (et) { case SIIS_ERR_NONE: ccb->ccb_h.status |= CAM_REQ_CMP; if (ccb->ccb_h.func_code == XPT_SCSI_IO) ccb->csio.scsi_status = SCSI_STATUS_OK; break; case SIIS_ERR_INVALID: ch->fatalerr = 1; ccb->ccb_h.status |= CAM_REQ_INVALID; break; case SIIS_ERR_INNOCENT: ccb->ccb_h.status |= CAM_REQUEUE_REQ; break; case SIIS_ERR_TFE: case SIIS_ERR_NCQ: if (ccb->ccb_h.func_code == XPT_SCSI_IO) { ccb->ccb_h.status |= CAM_SCSI_STATUS_ERROR; ccb->csio.scsi_status = SCSI_STATUS_CHECK_COND; } else { ccb->ccb_h.status |= CAM_ATA_STATUS_ERROR; } break; case SIIS_ERR_SATA: ch->fatalerr = 1; ccb->ccb_h.status |= CAM_UNCOR_PARITY; break; case SIIS_ERR_TIMEOUT: ch->fatalerr = 1; ccb->ccb_h.status |= CAM_CMD_TIMEOUT; break; default: ccb->ccb_h.status |= CAM_REQ_CMP_ERR; } /* Free slot. */ ch->oslots &= ~(1 << slot->slot); ch->rslots &= ~(1 << slot->slot); ch->aslots &= ~(1 << slot->slot); if (et != SIIS_ERR_TIMEOUT) { if (ch->toslots == (1 << slot->slot)) xpt_release_simq(ch->sim, TRUE); ch->toslots &= ~(1 << slot->slot); } slot->state = SIIS_SLOT_EMPTY; slot->ccb = NULL; /* Update channel stats. */ ch->numrslots--; if ((ccb->ccb_h.func_code == XPT_ATA_IO) && (ccb->ataio.cmd.flags & CAM_ATAIO_FPDMA)) { ch->numtslots[ccb->ccb_h.target_id]--; } /* If it was our READ LOG command - process it. */ if (ch->readlog) { siis_process_read_log(dev, ccb); /* If it was NCQ command error, put result on hold. */ } else if (et == SIIS_ERR_NCQ) { ch->hold[slot->slot] = ccb; ch->numhslots++; } else xpt_done(ccb); /* Unfreeze frozen command. */ if (ch->frozen && !siis_check_collision(dev, ch->frozen)) { union ccb *fccb = ch->frozen; ch->frozen = NULL; siis_begin_transaction(dev, fccb); xpt_release_simq(ch->sim, TRUE); } /* If we have no other active commands, ... */ if (ch->rslots == 0) { /* if there were timeouts or fatal error - reset port. */ if (ch->toslots != 0 || ch->fatalerr) { siis_reset(dev); } else { /* if we have slots in error, we can reinit port. */ if (ch->eslots != 0) siis_portinit(dev); /* if there commands on hold, we can do READ LOG. */ if (!ch->readlog && ch->numhslots) siis_issue_read_log(dev); } /* If all the reset of commands are in timeout - abort them. */ } else if ((ch->rslots & ~ch->toslots) == 0 && et != SIIS_ERR_TIMEOUT) siis_rearm_timeout(dev); } static void siis_issue_read_log(device_t dev) { struct siis_channel *ch = device_get_softc(dev); union ccb *ccb; struct ccb_ataio *ataio; int i; /* Find some holden command. */ for (i = 0; i < SIIS_MAX_SLOTS; i++) { if (ch->hold[i]) break; } if (i == SIIS_MAX_SLOTS) return; ch->readlog = 1; ccb = xpt_alloc_ccb_nowait(); if (ccb == NULL) { device_printf(dev, "Unable allocate READ LOG command"); return; /* XXX */ } ccb->ccb_h = ch->hold[i]->ccb_h; /* Reuse old header. */ ccb->ccb_h.func_code = XPT_ATA_IO; ccb->ccb_h.flags = CAM_DIR_IN; ccb->ccb_h.timeout = 1000; /* 1s should be enough. */ ataio = &ccb->ataio; ataio->data_ptr = malloc(512, M_SIIS, M_NOWAIT); if (ataio->data_ptr == NULL) { device_printf(dev, "Unable allocate memory for READ LOG command"); return; /* XXX */ } ataio->dxfer_len = 512; bzero(&ataio->cmd, sizeof(ataio->cmd)); ataio->cmd.flags = CAM_ATAIO_48BIT; ataio->cmd.command = 0x2F; /* READ LOG EXT */ ataio->cmd.sector_count = 1; ataio->cmd.sector_count_exp = 0; ataio->cmd.lba_low = 0x10; ataio->cmd.lba_mid = 0; ataio->cmd.lba_mid_exp = 0; siis_begin_transaction(dev, ccb); } static void siis_process_read_log(device_t dev, union ccb *ccb) { struct siis_channel *ch = device_get_softc(dev); uint8_t *data; struct ata_res *res; int i; ch->readlog = 0; data = ccb->ataio.data_ptr; if ((ccb->ccb_h.status & CAM_STATUS_MASK) == CAM_REQ_CMP && (data[0] & 0x80) == 0) { for (i = 0; i < SIIS_MAX_SLOTS; i++) { if (!ch->hold[i]) continue; if (ch->hold[i]->ccb_h.target_id != ccb->ccb_h.target_id) continue; if ((data[0] & 0x1F) == i) { res = &ch->hold[i]->ataio.res; res->status = data[2]; res->error = data[3]; res->lba_low = data[4]; res->lba_mid = data[5]; res->lba_high = data[6]; res->device = data[7]; res->lba_low_exp = data[8]; res->lba_mid_exp = data[9]; res->lba_high_exp = data[10]; res->sector_count = data[12]; res->sector_count_exp = data[13]; } else { ch->hold[i]->ccb_h.status &= ~CAM_STATUS_MASK; ch->hold[i]->ccb_h.status |= CAM_REQUEUE_REQ; } xpt_done(ch->hold[i]); ch->hold[i] = NULL; ch->numhslots--; } } else { if ((ccb->ccb_h.status & CAM_STATUS_MASK) != CAM_REQ_CMP) device_printf(dev, "Error while READ LOG EXT\n"); else if ((data[0] & 0x80) == 0) { device_printf(dev, "Non-queued command error in READ LOG EXT\n"); } for (i = 0; i < SIIS_MAX_SLOTS; i++) { if (!ch->hold[i]) continue; if (ch->hold[i]->ccb_h.target_id != ccb->ccb_h.target_id) continue; xpt_done(ch->hold[i]); ch->hold[i] = NULL; ch->numhslots--; } } free(ccb->ataio.data_ptr, M_SIIS); xpt_free_ccb(ccb); } static void siis_portinit(device_t dev) { struct siis_channel *ch = device_get_softc(dev); int i; ch->eslots = 0; ch->recovery = 0; ATA_OUTL(ch->r_mem, SIIS_P_CTLCLR, SIIS_P_CTL_RESUME); for (i = 0; i < 16; i++) { ATA_OUTL(ch->r_mem, SIIS_P_PMPSTS(i), 0), ATA_OUTL(ch->r_mem, SIIS_P_PMPQACT(i), 0); } ATA_OUTL(ch->r_mem, SIIS_P_CTLSET, SIIS_P_CTL_PORT_INIT); siis_wait_ready(dev, 1000); } static int siis_devreset(device_t dev) { struct siis_channel *ch = device_get_softc(dev); int timeout = 0; uint32_t val; ATA_OUTL(ch->r_mem, SIIS_P_CTLSET, SIIS_P_CTL_DEV_RESET); while (((val = ATA_INL(ch->r_mem, SIIS_P_STS)) & SIIS_P_CTL_DEV_RESET) != 0) { DELAY(1000); if (timeout++ > 100) { device_printf(dev, "device reset stuck (timeout %dms) " "status = %08x\n", timeout, val); return (EBUSY); } } return (0); } static int siis_wait_ready(device_t dev, int t) { struct siis_channel *ch = device_get_softc(dev); int timeout = 0; uint32_t val; while (((val = ATA_INL(ch->r_mem, SIIS_P_STS)) & SIIS_P_CTL_READY) == 0) { DELAY(1000); if (timeout++ > t) { device_printf(dev, "port is not ready (timeout %dms) " "status = %08x\n", t, val); return (EBUSY); } } return (0); } static void siis_reset(device_t dev) { struct siis_channel *ch = device_get_softc(dev); int i, retry = 0, sata_rev; uint32_t val; xpt_freeze_simq(ch->sim, 1); if (bootverbose) device_printf(dev, "SIIS reset...\n"); if (!ch->readlog && !ch->recovery) xpt_freeze_simq(ch->sim, ch->numrslots); /* Requeue frozen command. */ if (ch->frozen) { union ccb *fccb = ch->frozen; ch->frozen = NULL; fccb->ccb_h.status &= ~CAM_STATUS_MASK; fccb->ccb_h.status |= CAM_REQUEUE_REQ | CAM_RELEASE_SIMQ; if (!(fccb->ccb_h.status & CAM_DEV_QFRZN)) { xpt_freeze_devq(fccb->ccb_h.path, 1); fccb->ccb_h.status |= CAM_DEV_QFRZN; } xpt_done(fccb); } /* Requeue all running commands. */ for (i = 0; i < SIIS_MAX_SLOTS; i++) { /* Do we have a running request on slot? */ if (ch->slot[i].state < SIIS_SLOT_RUNNING) continue; /* XXX; Commands in loading state. */ siis_end_transaction(&ch->slot[i], SIIS_ERR_INNOCENT); } /* Finish all holden commands as-is. */ for (i = 0; i < SIIS_MAX_SLOTS; i++) { if (!ch->hold[i]) continue; xpt_done(ch->hold[i]); ch->hold[i] = NULL; ch->numhslots--; } if (ch->toslots != 0) xpt_release_simq(ch->sim, TRUE); ch->eslots = 0; ch->recovery = 0; ch->toslots = 0; ch->fatalerr = 0; /* Disable port interrupts */ ATA_OUTL(ch->r_mem, SIIS_P_IECLR, 0x0000FFFF); /* Set speed limit. */ sata_rev = ch->user[ch->pm_present ? 15 : 0].revision; if (sata_rev == 1) val = ATA_SC_SPD_SPEED_GEN1; else if (sata_rev == 2) val = ATA_SC_SPD_SPEED_GEN2; else if (sata_rev == 3) val = ATA_SC_SPD_SPEED_GEN3; else val = 0; ATA_OUTL(ch->r_mem, SIIS_P_SCTL, ATA_SC_DET_IDLE | val | ((ch->pm_level > 0) ? 0 : (ATA_SC_IPM_DIS_PARTIAL | ATA_SC_IPM_DIS_SLUMBER))); retry: siis_devreset(dev); /* Reset and reconnect PHY, */ if (!siis_sata_connect(ch)) { ch->devices = 0; /* Enable port interrupts */ ATA_OUTL(ch->r_mem, SIIS_P_IESET, SIIS_P_IX_ENABLED); if (bootverbose) device_printf(dev, "SIIS reset done: phy reset found no device\n"); /* Tell the XPT about the event */ xpt_async(AC_BUS_RESET, ch->path, NULL); xpt_release_simq(ch->sim, TRUE); return; } /* Wait for clearing busy status. */ if (siis_wait_ready(dev, 10000)) { device_printf(dev, "device ready timeout\n"); if (!retry) { device_printf(dev, "trying full port reset ...\n"); /* Get port to the reset state. */ ATA_OUTL(ch->r_mem, SIIS_P_CTLSET, SIIS_P_CTL_PORT_RESET); DELAY(10000); /* Get port out of reset state. */ ATA_OUTL(ch->r_mem, SIIS_P_CTLCLR, SIIS_P_CTL_PORT_RESET); ATA_OUTL(ch->r_mem, SIIS_P_CTLCLR, SIIS_P_CTL_32BIT); if (ch->pm_present) ATA_OUTL(ch->r_mem, SIIS_P_CTLSET, SIIS_P_CTL_PME); else ATA_OUTL(ch->r_mem, SIIS_P_CTLCLR, SIIS_P_CTL_PME); siis_wait_ready(dev, 5000); retry = 1; goto retry; } } ch->devices = 1; /* Enable port interrupts */ ATA_OUTL(ch->r_mem, SIIS_P_IS, 0xFFFFFFFF); ATA_OUTL(ch->r_mem, SIIS_P_IESET, SIIS_P_IX_ENABLED); if (bootverbose) device_printf(dev, "SIIS reset done: devices=%08x\n", ch->devices); /* Tell the XPT about the event */ xpt_async(AC_BUS_RESET, ch->path, NULL); xpt_release_simq(ch->sim, TRUE); } static int siis_setup_fis(device_t dev, struct siis_cmd *ctp, union ccb *ccb, int tag) { struct siis_channel *ch = device_get_softc(dev); u_int8_t *fis = &ctp->fis[0]; bzero(fis, 24); fis[0] = 0x27; /* host to device */ fis[1] = (ccb->ccb_h.target_id & 0x0f); if (ccb->ccb_h.func_code == XPT_SCSI_IO) { fis[1] |= 0x80; fis[2] = ATA_PACKET_CMD; if ((ccb->ccb_h.flags & CAM_DIR_MASK) != CAM_DIR_NONE && ch->curr[ccb->ccb_h.target_id].mode >= ATA_DMA) fis[3] = ATA_F_DMA; else { fis[5] = ccb->csio.dxfer_len; fis[6] = ccb->csio.dxfer_len >> 8; } fis[7] = ATA_D_LBA; fis[15] = ATA_A_4BIT; bzero(ctp->u.atapi.ccb, 16); bcopy((ccb->ccb_h.flags & CAM_CDB_POINTER) ? ccb->csio.cdb_io.cdb_ptr : ccb->csio.cdb_io.cdb_bytes, ctp->u.atapi.ccb, ccb->csio.cdb_len); } else if ((ccb->ataio.cmd.flags & CAM_ATAIO_CONTROL) == 0) { fis[1] |= 0x80; fis[2] = ccb->ataio.cmd.command; fis[3] = ccb->ataio.cmd.features; fis[4] = ccb->ataio.cmd.lba_low; fis[5] = ccb->ataio.cmd.lba_mid; fis[6] = ccb->ataio.cmd.lba_high; fis[7] = ccb->ataio.cmd.device; fis[8] = ccb->ataio.cmd.lba_low_exp; fis[9] = ccb->ataio.cmd.lba_mid_exp; fis[10] = ccb->ataio.cmd.lba_high_exp; fis[11] = ccb->ataio.cmd.features_exp; if (ccb->ataio.cmd.flags & CAM_ATAIO_FPDMA) { fis[12] = tag << 3; fis[13] = 0; } else { fis[12] = ccb->ataio.cmd.sector_count; fis[13] = ccb->ataio.cmd.sector_count_exp; } fis[15] = ATA_A_4BIT; } else { /* Soft reset. */ } return (20); } static int siis_sata_connect(struct siis_channel *ch) { u_int32_t status; int timeout; /* Wait up to 100ms for "connect well" */ for (timeout = 0; timeout < 100 ; timeout++) { status = ATA_INL(ch->r_mem, SIIS_P_SSTS); if (((status & ATA_SS_DET_MASK) == ATA_SS_DET_PHY_ONLINE) && ((status & ATA_SS_SPD_MASK) != ATA_SS_SPD_NO_SPEED) && ((status & ATA_SS_IPM_MASK) == ATA_SS_IPM_ACTIVE)) break; DELAY(1000); } if (timeout >= 100) { if (bootverbose) { device_printf(ch->dev, "SATA connect timeout status=%08x\n", status); } return (0); } if (bootverbose) { device_printf(ch->dev, "SATA connect time=%dms status=%08x\n", timeout, status); } /* Clear SATA error register */ ATA_OUTL(ch->r_mem, SIIS_P_SERR, 0xffffffff); return (1); } static void siisaction(struct cam_sim *sim, union ccb *ccb) { device_t dev; struct siis_channel *ch; CAM_DEBUG(ccb->ccb_h.path, CAM_DEBUG_TRACE, ("siisaction func_code=%x\n", ccb->ccb_h.func_code)); ch = (struct siis_channel *)cam_sim_softc(sim); dev = ch->dev; mtx_assert(&ch->mtx, MA_OWNED); switch (ccb->ccb_h.func_code) { /* Common cases first */ case XPT_ATA_IO: /* Execute the requested I/O operation */ case XPT_SCSI_IO: if (ch->devices == 0) { ccb->ccb_h.status = CAM_SEL_TIMEOUT; xpt_done(ccb); break; } /* Check for command collision. */ if (siis_check_collision(dev, ccb)) { /* Freeze command. */ ch->frozen = ccb; /* We have only one frozen slot, so freeze simq also. */ xpt_freeze_simq(ch->sim, 1); return; } siis_begin_transaction(dev, ccb); break; case XPT_EN_LUN: /* Enable LUN as a target */ case XPT_TARGET_IO: /* Execute target I/O request */ case XPT_ACCEPT_TARGET_IO: /* Accept Host Target Mode CDB */ case XPT_CONT_TARGET_IO: /* Continue Host Target I/O Connection*/ case XPT_ABORT: /* Abort the specified CCB */ /* XXX Implement */ ccb->ccb_h.status = CAM_REQ_INVALID; xpt_done(ccb); break; case XPT_SET_TRAN_SETTINGS: { struct ccb_trans_settings *cts = &ccb->cts; struct siis_device *d; if (cts->type == CTS_TYPE_CURRENT_SETTINGS) d = &ch->curr[ccb->ccb_h.target_id]; else d = &ch->user[ccb->ccb_h.target_id]; if (cts->xport_specific.sata.valid & CTS_SATA_VALID_REVISION) d->revision = cts->xport_specific.sata.revision; if (cts->xport_specific.sata.valid & CTS_SATA_VALID_MODE) d->mode = cts->xport_specific.sata.mode; if (cts->xport_specific.sata.valid & CTS_SATA_VALID_BYTECOUNT) d->bytecount = min(8192, cts->xport_specific.sata.bytecount); if (cts->xport_specific.sata.valid & CTS_SATA_VALID_TAGS) d->tags = min(SIIS_MAX_SLOTS, cts->xport_specific.sata.tags); if (cts->xport_specific.sata.valid & CTS_SATA_VALID_PM) { ch->pm_present = cts->xport_specific.sata.pm_present; if (ch->pm_present) ATA_OUTL(ch->r_mem, SIIS_P_CTLSET, SIIS_P_CTL_PME); else ATA_OUTL(ch->r_mem, SIIS_P_CTLCLR, SIIS_P_CTL_PME); } if (cts->xport_specific.sata.valid & CTS_SATA_VALID_TAGS) d->atapi = cts->xport_specific.sata.atapi; ccb->ccb_h.status = CAM_REQ_CMP; xpt_done(ccb); break; } case XPT_GET_TRAN_SETTINGS: /* Get default/user set transfer settings for the target */ { struct ccb_trans_settings *cts = &ccb->cts; struct siis_device *d; uint32_t status; if (cts->type == CTS_TYPE_CURRENT_SETTINGS) d = &ch->curr[ccb->ccb_h.target_id]; else d = &ch->user[ccb->ccb_h.target_id]; cts->protocol = PROTO_ATA; cts->protocol_version = PROTO_VERSION_UNSPECIFIED; cts->transport = XPORT_SATA; cts->transport_version = XPORT_VERSION_UNSPECIFIED; cts->proto_specific.valid = 0; cts->xport_specific.sata.valid = 0; if (cts->type == CTS_TYPE_CURRENT_SETTINGS && (ccb->ccb_h.target_id == 15 || (ccb->ccb_h.target_id == 0 && !ch->pm_present))) { status = ATA_INL(ch->r_mem, SIIS_P_SSTS) & ATA_SS_SPD_MASK; if (status & 0x0f0) { cts->xport_specific.sata.revision = (status & 0x0f0) >> 4; cts->xport_specific.sata.valid |= CTS_SATA_VALID_REVISION; } } else { cts->xport_specific.sata.revision = d->revision; cts->xport_specific.sata.valid |= CTS_SATA_VALID_REVISION; } cts->xport_specific.sata.mode = d->mode; cts->xport_specific.sata.valid |= CTS_SATA_VALID_MODE; cts->xport_specific.sata.bytecount = d->bytecount; cts->xport_specific.sata.valid |= CTS_SATA_VALID_BYTECOUNT; cts->xport_specific.sata.pm_present = ch->pm_present; cts->xport_specific.sata.valid |= CTS_SATA_VALID_PM; cts->xport_specific.sata.tags = d->tags; cts->xport_specific.sata.valid |= CTS_SATA_VALID_TAGS; cts->xport_specific.sata.atapi = d->atapi; cts->xport_specific.sata.valid |= CTS_SATA_VALID_ATAPI; ccb->ccb_h.status = CAM_REQ_CMP; xpt_done(ccb); break; } #if 0 case XPT_CALC_GEOMETRY: { struct ccb_calc_geometry *ccg; uint32_t size_mb; uint32_t secs_per_cylinder; ccg = &ccb->ccg; size_mb = ccg->volume_size / ((1024L * 1024L) / ccg->block_size); if (size_mb >= 1024 && (aha->extended_trans != 0)) { if (size_mb >= 2048) { ccg->heads = 255; ccg->secs_per_track = 63; } else { ccg->heads = 128; ccg->secs_per_track = 32; } } else { ccg->heads = 64; ccg->secs_per_track = 32; } secs_per_cylinder = ccg->heads * ccg->secs_per_track; ccg->cylinders = ccg->volume_size / secs_per_cylinder; ccb->ccb_h.status = CAM_REQ_CMP; xpt_done(ccb); break; } #endif case XPT_RESET_BUS: /* Reset the specified SCSI bus */ case XPT_RESET_DEV: /* Bus Device Reset the specified SCSI device */ siis_reset(dev); ccb->ccb_h.status = CAM_REQ_CMP; xpt_done(ccb); break; case XPT_TERM_IO: /* Terminate the I/O process */ /* XXX Implement */ ccb->ccb_h.status = CAM_REQ_INVALID; xpt_done(ccb); break; case XPT_PATH_INQ: /* Path routing inquiry */ { struct ccb_pathinq *cpi = &ccb->cpi; cpi->version_num = 1; /* XXX??? */ cpi->hba_inquiry = PI_SDTR_ABLE | PI_TAG_ABLE; cpi->hba_inquiry |= PI_SATAPM; cpi->target_sprt = 0; cpi->hba_misc = PIM_SEQSCAN; cpi->hba_eng_cnt = 0; cpi->max_target = 15; cpi->max_lun = 0; cpi->initiator_id = 0; cpi->bus_id = cam_sim_bus(sim); cpi->base_transfer_speed = 150000; strncpy(cpi->sim_vid, "FreeBSD", SIM_IDLEN); strncpy(cpi->hba_vid, "SIIS", HBA_IDLEN); strncpy(cpi->dev_name, cam_sim_name(sim), DEV_IDLEN); cpi->unit_number = cam_sim_unit(sim); cpi->transport = XPORT_SATA; cpi->transport_version = XPORT_VERSION_UNSPECIFIED; cpi->protocol = PROTO_ATA; cpi->protocol_version = PROTO_VERSION_UNSPECIFIED; cpi->ccb_h.status = CAM_REQ_CMP; cpi->maxio = MAXPHYS; xpt_done(ccb); break; } default: ccb->ccb_h.status = CAM_REQ_INVALID; xpt_done(ccb); break; } } static void siispoll(struct cam_sim *sim) { struct siis_channel *ch = (struct siis_channel *)cam_sim_softc(sim); siis_ch_intr(ch->dev); } Index: user/imp/tbemd/sys/dev/usb/input/ukbd.c =================================================================== --- user/imp/tbemd/sys/dev/usb/input/ukbd.c (revision 206684) +++ user/imp/tbemd/sys/dev/usb/input/ukbd.c (revision 206685) @@ -1,1891 +1,1889 @@ #include __FBSDID("$FreeBSD$"); /*- * Copyright (c) 1998 The NetBSD Foundation, Inc. * All rights reserved. * * This code is derived from software contributed to The NetBSD Foundation * by Lennart Augustsson (lennart@augustsson.net) at * Carlstedt Research & Technology. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. * */ /* * HID spec: http://www.usb.org/developers/devclass_docs/HID1_11.pdf */ #include "opt_compat.h" #include "opt_kbd.h" #include "opt_ukbd.h" #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #define USB_DEBUG_VAR ukbd_debug #include #include #include #include #include #include #include /* the initial key map, accent map and fkey strings */ #if defined(UKBD_DFLT_KEYMAP) && !defined(KLD_MODULE) #define KBD_DFLT_KEYMAP #include "ukbdmap.h" #endif /* the following file must be included after "ukbdmap.h" */ #include #if USB_DEBUG static int ukbd_debug = 0; static int ukbd_no_leds = 0; SYSCTL_NODE(_hw_usb, OID_AUTO, ukbd, CTLFLAG_RW, 0, "USB ukbd"); SYSCTL_INT(_hw_usb_ukbd, OID_AUTO, debug, CTLFLAG_RW, &ukbd_debug, 0, "Debug level"); SYSCTL_INT(_hw_usb_ukbd, OID_AUTO, no_leds, CTLFLAG_RW, &ukbd_no_leds, 0, "Disables setting of keyboard leds"); TUNABLE_INT("hw.usb.ukbd.debug", &ukbd_debug); TUNABLE_INT("hw.usb.ukbd.no_leds", &ukbd_no_leds); #endif -#define UPROTO_BOOT_KEYBOARD 1 - #define UKBD_EMULATE_ATSCANCODE 1 #define UKBD_DRIVER_NAME "ukbd" #define UKBD_NMOD 8 /* units */ #define UKBD_NKEYCODE 6 /* units */ #define UKBD_IN_BUF_SIZE (2*(UKBD_NMOD + (2*UKBD_NKEYCODE))) /* bytes */ #define UKBD_IN_BUF_FULL (UKBD_IN_BUF_SIZE / 2) /* bytes */ #define UKBD_NFKEY (sizeof(fkey_tab)/sizeof(fkey_tab[0])) /* units */ struct ukbd_data { uint8_t modifiers; #define MOD_CONTROL_L 0x01 #define MOD_CONTROL_R 0x10 #define MOD_SHIFT_L 0x02 #define MOD_SHIFT_R 0x20 #define MOD_ALT_L 0x04 #define MOD_ALT_R 0x40 #define MOD_WIN_L 0x08 #define MOD_WIN_R 0x80 uint8_t reserved; uint8_t keycode[UKBD_NKEYCODE]; uint8_t exten[8]; }; enum { UKBD_INTR_DT, UKBD_CTRL_LED, UKBD_N_TRANSFER, }; struct ukbd_softc { keyboard_t sc_kbd; keymap_t sc_keymap; accentmap_t sc_accmap; fkeytab_t sc_fkeymap[UKBD_NFKEY]; struct hid_location sc_loc_apple_eject; struct hid_location sc_loc_apple_fn; struct usb_callout sc_callout; struct ukbd_data sc_ndata; struct ukbd_data sc_odata; struct thread *sc_poll_thread; struct usb_device *sc_udev; struct usb_interface *sc_iface; struct usb_xfer *sc_xfer[UKBD_N_TRANSFER]; uint32_t sc_ntime[UKBD_NKEYCODE]; uint32_t sc_otime[UKBD_NKEYCODE]; uint32_t sc_input[UKBD_IN_BUF_SIZE]; /* input buffer */ uint32_t sc_time_ms; uint32_t sc_composed_char; /* composed char code, if non-zero */ #ifdef UKBD_EMULATE_ATSCANCODE uint32_t sc_buffered_char[2]; #endif uint32_t sc_flags; /* flags */ #define UKBD_FLAG_COMPOSE 0x0001 #define UKBD_FLAG_POLLING 0x0002 #define UKBD_FLAG_SET_LEDS 0x0004 #define UKBD_FLAG_ATTACHED 0x0010 #define UKBD_FLAG_GONE 0x0020 #define UKBD_FLAG_APPLE_EJECT 0x0040 #define UKBD_FLAG_APPLE_FN 0x0080 #define UKBD_FLAG_APPLE_SWAP 0x0100 #define UKBD_FLAG_TIMER_RUNNING 0x0200 int sc_mode; /* input mode (K_XLATE,K_RAW,K_CODE) */ int sc_state; /* shift/lock key state */ int sc_accents; /* accent key index (> 0) */ int sc_poll_tick_last; uint16_t sc_inputs; uint16_t sc_inputhead; uint16_t sc_inputtail; uint8_t sc_leds; /* store for async led requests */ uint8_t sc_iface_index; uint8_t sc_iface_no; uint8_t sc_kbd_id; uint8_t sc_led_id; uint8_t sc_poll_detected; }; #define KEY_ERROR 0x01 #define KEY_PRESS 0 #define KEY_RELEASE 0x400 #define KEY_INDEX(c) ((c) & 0xFF) #define SCAN_PRESS 0 #define SCAN_RELEASE 0x80 #define SCAN_PREFIX_E0 0x100 #define SCAN_PREFIX_E1 0x200 #define SCAN_PREFIX_CTL 0x400 #define SCAN_PREFIX_SHIFT 0x800 #define SCAN_PREFIX (SCAN_PREFIX_E0 | SCAN_PREFIX_E1 | \ SCAN_PREFIX_CTL | SCAN_PREFIX_SHIFT) #define SCAN_CHAR(c) ((c) & 0x7f) struct ukbd_mods { uint32_t mask, key; }; static const struct ukbd_mods ukbd_mods[UKBD_NMOD] = { {MOD_CONTROL_L, 0xe0}, {MOD_CONTROL_R, 0xe4}, {MOD_SHIFT_L, 0xe1}, {MOD_SHIFT_R, 0xe5}, {MOD_ALT_L, 0xe2}, {MOD_ALT_R, 0xe6}, {MOD_WIN_L, 0xe3}, {MOD_WIN_R, 0xe7}, }; #define NN 0 /* no translation */ /* * Translate USB keycodes to AT keyboard scancodes. */ /* * FIXME: Mac USB keyboard generates: * 0x53: keypad NumLock/Clear * 0x66: Power * 0x67: keypad = * 0x68: F13 * 0x69: F14 * 0x6a: F15 */ static const uint8_t ukbd_trtab[256] = { 0, 0, 0, 0, 30, 48, 46, 32, /* 00 - 07 */ 18, 33, 34, 35, 23, 36, 37, 38, /* 08 - 0F */ 50, 49, 24, 25, 16, 19, 31, 20, /* 10 - 17 */ 22, 47, 17, 45, 21, 44, 2, 3, /* 18 - 1F */ 4, 5, 6, 7, 8, 9, 10, 11, /* 20 - 27 */ 28, 1, 14, 15, 57, 12, 13, 26, /* 28 - 2F */ 27, 43, 43, 39, 40, 41, 51, 52, /* 30 - 37 */ 53, 58, 59, 60, 61, 62, 63, 64, /* 38 - 3F */ 65, 66, 67, 68, 87, 88, 92, 70, /* 40 - 47 */ 104, 102, 94, 96, 103, 99, 101, 98, /* 48 - 4F */ 97, 100, 95, 69, 91, 55, 74, 78,/* 50 - 57 */ 89, 79, 80, 81, 75, 76, 77, 71, /* 58 - 5F */ 72, 73, 82, 83, 86, 107, 122, NN, /* 60 - 67 */ NN, NN, NN, NN, NN, NN, NN, NN, /* 68 - 6F */ NN, NN, NN, NN, 115, 108, 111, 113, /* 70 - 77 */ 109, 110, 112, 118, 114, 116, 117, 119, /* 78 - 7F */ 121, 120, NN, NN, NN, NN, NN, 123, /* 80 - 87 */ 124, 125, 126, 127, 128, NN, NN, NN, /* 88 - 8F */ NN, NN, NN, NN, NN, NN, NN, NN, /* 90 - 97 */ NN, NN, NN, NN, NN, NN, NN, NN, /* 98 - 9F */ NN, NN, NN, NN, NN, NN, NN, NN, /* A0 - A7 */ NN, NN, NN, NN, NN, NN, NN, NN, /* A8 - AF */ NN, NN, NN, NN, NN, NN, NN, NN, /* B0 - B7 */ NN, NN, NN, NN, NN, NN, NN, NN, /* B8 - BF */ NN, NN, NN, NN, NN, NN, NN, NN, /* C0 - C7 */ NN, NN, NN, NN, NN, NN, NN, NN, /* C8 - CF */ NN, NN, NN, NN, NN, NN, NN, NN, /* D0 - D7 */ NN, NN, NN, NN, NN, NN, NN, NN, /* D8 - DF */ 29, 42, 56, 105, 90, 54, 93, 106, /* E0 - E7 */ NN, NN, NN, NN, NN, NN, NN, NN, /* E8 - EF */ NN, NN, NN, NN, NN, NN, NN, NN, /* F0 - F7 */ NN, NN, NN, NN, NN, NN, NN, NN, /* F8 - FF */ }; /* prototypes */ static void ukbd_timeout(void *); static void ukbd_set_leds(struct ukbd_softc *, uint8_t); static int ukbd_set_typematic(keyboard_t *, int); #ifdef UKBD_EMULATE_ATSCANCODE static int ukbd_key2scan(struct ukbd_softc *, int, int, int); #endif static uint32_t ukbd_read_char(keyboard_t *, int); static void ukbd_clear_state(keyboard_t *); static int ukbd_ioctl(keyboard_t *, u_long, caddr_t); static int ukbd_enable(keyboard_t *); static int ukbd_disable(keyboard_t *); static void ukbd_interrupt(struct ukbd_softc *); static int ukbd_is_polling(struct ukbd_softc *); static int ukbd_polls_other_thread(struct ukbd_softc *); static void ukbd_event_keyinput(struct ukbd_softc *); static device_probe_t ukbd_probe; static device_attach_t ukbd_attach; static device_detach_t ukbd_detach; static device_resume_t ukbd_resume; static uint8_t ukbd_any_key_pressed(struct ukbd_softc *sc) { uint8_t i; uint8_t j; for (j = i = 0; i < UKBD_NKEYCODE; i++) j |= sc->sc_odata.keycode[i]; return (j ? 1 : 0); } static void ukbd_start_timer(struct ukbd_softc *sc) { sc->sc_flags |= UKBD_FLAG_TIMER_RUNNING; usb_callout_reset(&sc->sc_callout, hz / 40, &ukbd_timeout, sc); } static void ukbd_put_key(struct ukbd_softc *sc, uint32_t key) { mtx_assert(&Giant, MA_OWNED); DPRINTF("0x%02x (%d) %s\n", key, key, (key & KEY_RELEASE) ? "released" : "pressed"); if (sc->sc_inputs < UKBD_IN_BUF_SIZE) { sc->sc_input[sc->sc_inputtail] = key; ++(sc->sc_inputs); ++(sc->sc_inputtail); if (sc->sc_inputtail >= UKBD_IN_BUF_SIZE) { sc->sc_inputtail = 0; } } else { DPRINTF("input buffer is full\n"); } } static void ukbd_do_poll(struct ukbd_softc *sc, uint8_t wait) { DPRINTFN(2, "polling\n"); /* update stats about last polling event */ sc->sc_poll_tick_last = ticks; sc->sc_poll_detected = 1; if (kdb_active == 0) { while (sc->sc_inputs == 0) { /* make sure the USB code gets a chance to run */ pause("UKBD", 1); /* check if we should wait */ if (!wait) break; } return; /* Only poll if KDB is active */ } while (sc->sc_inputs == 0) { usbd_transfer_poll(sc->sc_xfer, UKBD_N_TRANSFER); /* Delay-optimised support for repetition of keys */ if (ukbd_any_key_pressed(sc)) { /* a key is pressed - need timekeeping */ DELAY(1000); /* 1 millisecond has passed */ sc->sc_time_ms += 1; } ukbd_interrupt(sc); if (!wait) break; } } static int32_t ukbd_get_key(struct ukbd_softc *sc, uint8_t wait) { int32_t c; mtx_assert(&Giant, MA_OWNED); if (sc->sc_inputs == 0) { /* start transfer, if not already started */ usbd_transfer_start(sc->sc_xfer[UKBD_INTR_DT]); } if (ukbd_polls_other_thread(sc)) return (-1); if (sc->sc_flags & UKBD_FLAG_POLLING) ukbd_do_poll(sc, wait); if (sc->sc_inputs == 0) { c = -1; } else { c = sc->sc_input[sc->sc_inputhead]; --(sc->sc_inputs); ++(sc->sc_inputhead); if (sc->sc_inputhead >= UKBD_IN_BUF_SIZE) { sc->sc_inputhead = 0; } } return (c); } static void ukbd_interrupt(struct ukbd_softc *sc) { uint32_t n_mod; uint32_t o_mod; uint32_t now = sc->sc_time_ms; uint32_t dtime; uint8_t key; uint8_t i; uint8_t j; if (sc->sc_ndata.keycode[0] == KEY_ERROR) return; n_mod = sc->sc_ndata.modifiers; o_mod = sc->sc_odata.modifiers; if (n_mod != o_mod) { for (i = 0; i < UKBD_NMOD; i++) { if ((n_mod & ukbd_mods[i].mask) != (o_mod & ukbd_mods[i].mask)) { ukbd_put_key(sc, ukbd_mods[i].key | ((n_mod & ukbd_mods[i].mask) ? KEY_PRESS : KEY_RELEASE)); } } } /* Check for released keys. */ for (i = 0; i < UKBD_NKEYCODE; i++) { key = sc->sc_odata.keycode[i]; if (key == 0) { continue; } for (j = 0; j < UKBD_NKEYCODE; j++) { if (sc->sc_ndata.keycode[j] == 0) { continue; } if (key == sc->sc_ndata.keycode[j]) { goto rfound; } } ukbd_put_key(sc, key | KEY_RELEASE); rfound: ; } /* Check for pressed keys. */ for (i = 0; i < UKBD_NKEYCODE; i++) { key = sc->sc_ndata.keycode[i]; if (key == 0) { continue; } sc->sc_ntime[i] = now + sc->sc_kbd.kb_delay1; for (j = 0; j < UKBD_NKEYCODE; j++) { if (sc->sc_odata.keycode[j] == 0) { continue; } if (key == sc->sc_odata.keycode[j]) { /* key is still pressed */ sc->sc_ntime[i] = sc->sc_otime[j]; dtime = (sc->sc_otime[j] - now); if (!(dtime & 0x80000000)) { /* time has not elapsed */ goto pfound; } sc->sc_ntime[i] = now + sc->sc_kbd.kb_delay2; break; } } ukbd_put_key(sc, key | KEY_PRESS); /* * If any other key is presently down, force its repeat to be * well in the future (100s). This makes the last key to be * pressed do the autorepeat. */ for (j = 0; j != UKBD_NKEYCODE; j++) { if (j != i) sc->sc_ntime[j] = now + (100 * 1000); } pfound: ; } sc->sc_odata = sc->sc_ndata; memcpy(sc->sc_otime, sc->sc_ntime, sizeof(sc->sc_otime)); ukbd_event_keyinput(sc); } static void ukbd_event_keyinput(struct ukbd_softc *sc) { int c; if (ukbd_is_polling(sc)) return; if (sc->sc_inputs == 0) return; if (KBD_IS_ACTIVE(&sc->sc_kbd) && KBD_IS_BUSY(&sc->sc_kbd)) { /* let the callback function process the input */ (sc->sc_kbd.kb_callback.kc_func) (&sc->sc_kbd, KBDIO_KEYINPUT, sc->sc_kbd.kb_callback.kc_arg); } else { /* read and discard the input, no one is waiting for it */ do { c = ukbd_read_char(&sc->sc_kbd, 0); } while (c != NOKEY); } } static void ukbd_timeout(void *arg) { struct ukbd_softc *sc = arg; mtx_assert(&Giant, MA_OWNED); sc->sc_time_ms += 25; /* milliseconds */ ukbd_interrupt(sc); /* Make sure any leftover key events gets read out */ ukbd_event_keyinput(sc); if (ukbd_any_key_pressed(sc) || (sc->sc_inputs != 0)) { ukbd_start_timer(sc); } else { sc->sc_flags &= ~UKBD_FLAG_TIMER_RUNNING; } } static uint8_t ukbd_apple_fn(uint8_t keycode) { switch (keycode) { case 0x28: return 0x49; /* RETURN -> INSERT */ case 0x2a: return 0x4c; /* BACKSPACE -> DEL */ case 0x50: return 0x4a; /* LEFT ARROW -> HOME */ case 0x4f: return 0x4d; /* RIGHT ARROW -> END */ case 0x52: return 0x4b; /* UP ARROW -> PGUP */ case 0x51: return 0x4e; /* DOWN ARROW -> PGDN */ default: return keycode; } } static uint8_t ukbd_apple_swap(uint8_t keycode) { switch (keycode) { case 0x35: return 0x64; case 0x64: return 0x35; default: return keycode; } } static void ukbd_intr_callback(struct usb_xfer *xfer, usb_error_t error) { struct ukbd_softc *sc = usbd_xfer_softc(xfer); struct usb_page_cache *pc; uint8_t i; uint8_t offset; uint8_t id; uint8_t apple_fn; uint8_t apple_eject; int len; usbd_xfer_status(xfer, &len, NULL, NULL, NULL); pc = usbd_xfer_get_frame(xfer, 0); switch (USB_GET_STATE(xfer)) { case USB_ST_TRANSFERRED: DPRINTF("actlen=%d bytes\n", len); if (len == 0) { DPRINTF("zero length data\n"); goto tr_setup; } if (sc->sc_kbd_id != 0) { /* check and remove HID ID byte */ usbd_copy_out(pc, 0, &id, 1); if (id != sc->sc_kbd_id) { DPRINTF("wrong HID ID\n"); goto tr_setup; } offset = 1; len--; } else { offset = 0; } if (len > sizeof(sc->sc_ndata)) { len = sizeof(sc->sc_ndata); } if (len) { memset(&sc->sc_ndata, 0, sizeof(sc->sc_ndata)); usbd_copy_out(pc, offset, &sc->sc_ndata, len); if ((sc->sc_flags & UKBD_FLAG_APPLE_EJECT) && hid_get_data((uint8_t *)&sc->sc_ndata, len, &sc->sc_loc_apple_eject)) apple_eject = 1; else apple_eject = 0; if ((sc->sc_flags & UKBD_FLAG_APPLE_FN) && hid_get_data((uint8_t *)&sc->sc_ndata, len, &sc->sc_loc_apple_fn)) apple_fn = 1; else apple_fn = 0; #if USB_DEBUG DPRINTF("apple_eject=%u apple_fn=%u\n", apple_eject, apple_fn); if (sc->sc_ndata.modifiers) { DPRINTF("mod: 0x%04x\n", sc->sc_ndata.modifiers); } for (i = 0; i < UKBD_NKEYCODE; i++) { if (sc->sc_ndata.keycode[i]) { DPRINTF("[%d] = %d\n", i, sc->sc_ndata.keycode[i]); } } #endif /* USB_DEBUG */ if (apple_fn) { for (i = 0; i < UKBD_NKEYCODE; i++) { sc->sc_ndata.keycode[i] = ukbd_apple_fn(sc->sc_ndata.keycode[i]); } } if (sc->sc_flags & UKBD_FLAG_APPLE_SWAP) { for (i = 0; i < UKBD_NKEYCODE; i++) { sc->sc_ndata.keycode[i] = ukbd_apple_swap(sc->sc_ndata.keycode[i]); } } ukbd_interrupt(sc); if (!(sc->sc_flags & UKBD_FLAG_TIMER_RUNNING)) { if (ukbd_any_key_pressed(sc)) { ukbd_start_timer(sc); } } } case USB_ST_SETUP: tr_setup: if (sc->sc_inputs < UKBD_IN_BUF_FULL) { usbd_xfer_set_frame_len(xfer, 0, usbd_xfer_max_len(xfer)); usbd_transfer_submit(xfer); } else { DPRINTF("input queue is full!\n"); } break; default: /* Error */ DPRINTF("error=%s\n", usbd_errstr(error)); if (error != USB_ERR_CANCELLED) { /* try to clear stall first */ usbd_xfer_set_stall(xfer); goto tr_setup; } break; } } static void ukbd_set_leds_callback(struct usb_xfer *xfer, usb_error_t error) { struct usb_device_request req; struct usb_page_cache *pc; uint8_t buf[2]; struct ukbd_softc *sc = usbd_xfer_softc(xfer); #if USB_DEBUG if (ukbd_no_leds) return; #endif switch (USB_GET_STATE(xfer)) { case USB_ST_TRANSFERRED: case USB_ST_SETUP: if (sc->sc_flags & UKBD_FLAG_SET_LEDS) { sc->sc_flags &= ~UKBD_FLAG_SET_LEDS; req.bmRequestType = UT_WRITE_CLASS_INTERFACE; req.bRequest = UR_SET_REPORT; USETW2(req.wValue, UHID_OUTPUT_REPORT, 0); req.wIndex[0] = sc->sc_iface_no; req.wIndex[1] = 0; req.wLength[1] = 0; /* check if we need to prefix an ID byte */ if (sc->sc_led_id != 0) { req.wLength[0] = 2; buf[0] = sc->sc_led_id; buf[1] = sc->sc_leds; } else { req.wLength[0] = 1; buf[0] = sc->sc_leds; buf[1] = 0; } pc = usbd_xfer_get_frame(xfer, 0); usbd_copy_in(pc, 0, &req, sizeof(req)); pc = usbd_xfer_get_frame(xfer, 1); usbd_copy_in(pc, 0, buf, sizeof(buf)); usbd_xfer_set_frame_len(xfer, 0, sizeof(req)); usbd_xfer_set_frame_len(xfer, 1, req.wLength[0]); usbd_xfer_set_frames(xfer, 2); usbd_transfer_submit(xfer); } break; default: /* Error */ DPRINTFN(0, "error=%s\n", usbd_errstr(error)); break; } } static const struct usb_config ukbd_config[UKBD_N_TRANSFER] = { [UKBD_INTR_DT] = { .type = UE_INTERRUPT, .endpoint = UE_ADDR_ANY, .direction = UE_DIR_IN, .flags = {.pipe_bof = 1,.short_xfer_ok = 1,}, .bufsize = 0, /* use wMaxPacketSize */ .callback = &ukbd_intr_callback, }, [UKBD_CTRL_LED] = { .type = UE_CONTROL, .endpoint = 0x00, /* Control pipe */ .direction = UE_DIR_ANY, .bufsize = sizeof(struct usb_device_request) + 8, .callback = &ukbd_set_leds_callback, .timeout = 1000, /* 1 second */ }, }; static int ukbd_probe(device_t dev) { keyboard_switch_t *sw = kbd_get_switch(UKBD_DRIVER_NAME); struct usb_attach_arg *uaa = device_get_ivars(dev); void *d_ptr; int error; uint16_t d_len; DPRINTFN(11, "\n"); if (sw == NULL) { return (ENXIO); } if (uaa->usb_mode != USB_MODE_HOST) { return (ENXIO); } if (uaa->info.bInterfaceClass != UICLASS_HID) return (ENXIO); if ((uaa->info.bInterfaceSubClass == UISUBCLASS_BOOT) && - (uaa->info.bInterfaceProtocol == UPROTO_BOOT_KEYBOARD)) { + (uaa->info.bInterfaceProtocol == UIPROTO_BOOT_KEYBOARD)) { if (usb_test_quirk(uaa, UQ_KBD_IGNORE)) return (ENXIO); else return (BUS_PROBE_GENERIC); } error = usbd_req_get_hid_desc(uaa->device, NULL, &d_ptr, &d_len, M_TEMP, uaa->info.bIfaceIndex); if (error) return (ENXIO); /* * NOTE: we currently don't support USB mouse and USB keyboard * on the same USB endpoint. */ if (hid_is_collection(d_ptr, d_len, HID_USAGE2(HUP_GENERIC_DESKTOP, HUG_MOUSE))) { /* most likely a mouse */ error = ENXIO; } else if (hid_is_collection(d_ptr, d_len, HID_USAGE2(HUP_GENERIC_DESKTOP, HUG_KEYBOARD))) { if (usb_test_quirk(uaa, UQ_KBD_IGNORE)) error = ENXIO; else error = BUS_PROBE_GENERIC; } else error = ENXIO; free(d_ptr, M_TEMP); return (error); } static int ukbd_attach(device_t dev) { struct ukbd_softc *sc = device_get_softc(dev); struct usb_attach_arg *uaa = device_get_ivars(dev); int32_t unit = device_get_unit(dev); keyboard_t *kbd = &sc->sc_kbd; void *hid_ptr = NULL; usb_error_t err; uint32_t flags; uint16_t n; uint16_t hid_len; kbd_init_struct(kbd, UKBD_DRIVER_NAME, KB_OTHER, unit, 0, 0, 0); kbd->kb_data = (void *)sc; device_set_usb_desc(dev); sc->sc_udev = uaa->device; sc->sc_iface = uaa->iface; sc->sc_iface_index = uaa->info.bIfaceIndex; sc->sc_iface_no = uaa->info.bIfaceNum; sc->sc_mode = K_XLATE; usb_callout_init_mtx(&sc->sc_callout, &Giant, 0); err = usbd_transfer_setup(uaa->device, &uaa->info.bIfaceIndex, sc->sc_xfer, ukbd_config, UKBD_N_TRANSFER, sc, &Giant); if (err) { DPRINTF("error=%s\n", usbd_errstr(err)); goto detach; } /* setup default keyboard maps */ sc->sc_keymap = key_map; sc->sc_accmap = accent_map; for (n = 0; n < UKBD_NFKEY; n++) { sc->sc_fkeymap[n] = fkey_tab[n]; } kbd_set_maps(kbd, &sc->sc_keymap, &sc->sc_accmap, sc->sc_fkeymap, UKBD_NFKEY); KBD_FOUND_DEVICE(kbd); ukbd_clear_state(kbd); /* * FIXME: set the initial value for lock keys in "sc_state" * according to the BIOS data? */ KBD_PROBE_DONE(kbd); /* * Set boot protocol if we need the quirk. */ if (usb_test_quirk(uaa, UQ_KBD_BOOTPROTO)) { err = usbd_req_set_protocol(sc->sc_udev, NULL, sc->sc_iface_index, 0); if (err != USB_ERR_NORMAL_COMPLETION) { DPRINTF("set protocol error=%s\n", usbd_errstr(err)); goto detach; } } /* figure out if there is an ID byte in the data */ err = usbd_req_get_hid_desc(uaa->device, NULL, &hid_ptr, &hid_len, M_TEMP, uaa->info.bIfaceIndex); if (err == 0) { uint8_t apple_keys = 0; uint8_t temp_id; /* investigate if this is an Apple Keyboard */ if (hid_locate(hid_ptr, hid_len, HID_USAGE2(HUP_CONSUMER, HUG_APPLE_EJECT), hid_input, 0, &sc->sc_loc_apple_eject, &flags, &temp_id)) { if (flags & HIO_VARIABLE) sc->sc_flags |= UKBD_FLAG_APPLE_EJECT | UKBD_FLAG_APPLE_SWAP; DPRINTFN(1, "Found Apple eject-key\n"); apple_keys = 1; sc->sc_kbd_id = temp_id; } if (hid_locate(hid_ptr, hid_len, HID_USAGE2(0xFFFF, 0x0003), hid_input, 0, &sc->sc_loc_apple_fn, &flags, &temp_id)) { if (flags & HIO_VARIABLE) sc->sc_flags |= UKBD_FLAG_APPLE_FN; DPRINTFN(1, "Found Apple FN-key\n"); apple_keys = 1; sc->sc_kbd_id = temp_id; } if (apple_keys == 0) { /* * Assume the first HID ID contains the * keyboard data */ hid_report_size(hid_ptr, hid_len, hid_input, &sc->sc_kbd_id); } /* investigate if we need an ID-byte for the leds */ hid_report_size(hid_ptr, hid_len, hid_output, &sc->sc_led_id); free(hid_ptr, M_TEMP); } /* ignore if SETIDLE fails, hence it is not crucial */ err = usbd_req_set_idle(sc->sc_udev, NULL, sc->sc_iface_index, 0, 0); mtx_lock(&Giant); ukbd_ioctl(kbd, KDSETLED, (caddr_t)&sc->sc_state); KBD_INIT_DONE(kbd); mtx_unlock(&Giant); if (kbd_register(kbd) < 0) { goto detach; } KBD_CONFIG_DONE(kbd); ukbd_enable(kbd); #ifdef KBD_INSTALL_CDEV if (kbd_attach(kbd)) { goto detach; } #endif sc->sc_flags |= UKBD_FLAG_ATTACHED; if (bootverbose) { genkbd_diag(kbd, bootverbose); } /* lock keyboard mutex */ mtx_lock(&Giant); /* start the keyboard */ usbd_transfer_start(sc->sc_xfer[UKBD_INTR_DT]); mtx_unlock(&Giant); return (0); /* success */ detach: ukbd_detach(dev); return (ENXIO); /* error */ } static int ukbd_detach(device_t dev) { struct ukbd_softc *sc = device_get_softc(dev); int error; DPRINTF("\n"); mtx_lock(&Giant); sc->sc_flags |= UKBD_FLAG_GONE; usb_callout_stop(&sc->sc_callout); ukbd_disable(&sc->sc_kbd); #ifdef KBD_INSTALL_CDEV if (sc->sc_flags & UKBD_FLAG_ATTACHED) { error = kbd_detach(&sc->sc_kbd); if (error) { /* usb attach cannot return an error */ device_printf(dev, "WARNING: kbd_detach() " "returned non-zero! (ignored)\n"); } } #endif if (KBD_IS_CONFIGURED(&sc->sc_kbd)) { error = kbd_unregister(&sc->sc_kbd); if (error) { /* usb attach cannot return an error */ device_printf(dev, "WARNING: kbd_unregister() " "returned non-zero! (ignored)\n"); } } sc->sc_kbd.kb_flags = 0; mtx_unlock(&Giant); usbd_transfer_unsetup(sc->sc_xfer, UKBD_N_TRANSFER); usb_callout_drain(&sc->sc_callout); DPRINTF("%s: disconnected\n", device_get_nameunit(dev)); return (0); } static int ukbd_resume(device_t dev) { struct ukbd_softc *sc = device_get_softc(dev); mtx_lock(&Giant); ukbd_clear_state(&sc->sc_kbd); mtx_unlock(&Giant); return (0); } /* early keyboard probe, not supported */ static int ukbd_configure(int flags) { return (0); } /* detect a keyboard, not used */ static int ukbd__probe(int unit, void *arg, int flags) { return (ENXIO); } /* reset and initialize the device, not used */ static int ukbd_init(int unit, keyboard_t **kbdp, void *arg, int flags) { return (ENXIO); } /* test the interface to the device, not used */ static int ukbd_test_if(keyboard_t *kbd) { return (0); } /* finish using this keyboard, not used */ static int ukbd_term(keyboard_t *kbd) { return (ENXIO); } /* keyboard interrupt routine, not used */ static int ukbd_intr(keyboard_t *kbd, void *arg) { return (0); } /* lock the access to the keyboard, not used */ static int ukbd_lock(keyboard_t *kbd, int lock) { return (1); } /* * Enable the access to the device; until this function is called, * the client cannot read from the keyboard. */ static int ukbd_enable(keyboard_t *kbd) { if (!mtx_owned(&Giant)) { /* XXX cludge */ int retval; mtx_lock(&Giant); retval = ukbd_enable(kbd); mtx_unlock(&Giant); return (retval); } KBD_ACTIVATE(kbd); return (0); } /* disallow the access to the device */ static int ukbd_disable(keyboard_t *kbd) { if (!mtx_owned(&Giant)) { /* XXX cludge */ int retval; mtx_lock(&Giant); retval = ukbd_disable(kbd); mtx_unlock(&Giant); return (retval); } KBD_DEACTIVATE(kbd); return (0); } /* check if data is waiting */ static int ukbd_check(keyboard_t *kbd) { struct ukbd_softc *sc = kbd->kb_data; if (!KBD_IS_ACTIVE(kbd)) return (0); if (sc->sc_flags & UKBD_FLAG_POLLING) { if (!mtx_owned(&Giant)) { /* XXX cludge */ int retval; mtx_lock(&Giant); retval = ukbd_check(kbd); mtx_unlock(&Giant); return (retval); } } else { /* XXX the keyboard layer requires Giant */ if (!mtx_owned(&Giant)) return (0); } /* check if key belongs to this thread */ if (ukbd_polls_other_thread(sc)) return (0); if (sc->sc_flags & UKBD_FLAG_POLLING) ukbd_do_poll(sc, 0); #ifdef UKBD_EMULATE_ATSCANCODE if (sc->sc_buffered_char[0]) { return (1); } #endif if (sc->sc_inputs > 0) { return (1); } return (0); } /* check if char is waiting */ static int ukbd_check_char(keyboard_t *kbd) { struct ukbd_softc *sc = kbd->kb_data; if (!KBD_IS_ACTIVE(kbd)) return (0); if (sc->sc_flags & UKBD_FLAG_POLLING) { if (!mtx_owned(&Giant)) { /* XXX cludge */ int retval; mtx_lock(&Giant); retval = ukbd_check_char(kbd); mtx_unlock(&Giant); return (retval); } } else { /* XXX the keyboard layer requires Giant */ if (!mtx_owned(&Giant)) return (0); } /* check if key belongs to this thread */ if (ukbd_polls_other_thread(sc)) return (0); if ((sc->sc_composed_char > 0) && (!(sc->sc_flags & UKBD_FLAG_COMPOSE))) { return (1); } return (ukbd_check(kbd)); } /* read one byte from the keyboard if it's allowed */ static int ukbd_read(keyboard_t *kbd, int wait) { struct ukbd_softc *sc = kbd->kb_data; int32_t usbcode; #ifdef UKBD_EMULATE_ATSCANCODE uint32_t keycode; uint32_t scancode; #endif if (!KBD_IS_ACTIVE(kbd)) return (-1); if (sc->sc_flags & UKBD_FLAG_POLLING) { if (!mtx_owned(&Giant)) { /* XXX cludge */ int retval; mtx_lock(&Giant); retval = ukbd_read(kbd, wait); mtx_unlock(&Giant); return (retval); } } else { /* XXX the keyboard layer requires Giant */ if (!mtx_owned(&Giant)) return (-1); } /* check if key belongs to this thread */ if (ukbd_polls_other_thread(sc)) return (-1); #ifdef UKBD_EMULATE_ATSCANCODE if (sc->sc_buffered_char[0]) { scancode = sc->sc_buffered_char[0]; if (scancode & SCAN_PREFIX) { sc->sc_buffered_char[0] &= ~SCAN_PREFIX; return ((scancode & SCAN_PREFIX_E0) ? 0xe0 : 0xe1); } sc->sc_buffered_char[0] = sc->sc_buffered_char[1]; sc->sc_buffered_char[1] = 0; return (scancode); } #endif /* UKBD_EMULATE_ATSCANCODE */ /* XXX */ usbcode = ukbd_get_key(sc, (wait == FALSE) ? 0 : 1); if (!KBD_IS_ACTIVE(kbd) || (usbcode == -1)) return (-1); ++(kbd->kb_count); #ifdef UKBD_EMULATE_ATSCANCODE keycode = ukbd_trtab[KEY_INDEX(usbcode)]; if (keycode == NN) { return -1; } return (ukbd_key2scan(sc, keycode, sc->sc_ndata.modifiers, (usbcode & KEY_RELEASE))); #else /* !UKBD_EMULATE_ATSCANCODE */ return (usbcode); #endif /* UKBD_EMULATE_ATSCANCODE */ } /* read char from the keyboard */ static uint32_t ukbd_read_char(keyboard_t *kbd, int wait) { struct ukbd_softc *sc = kbd->kb_data; uint32_t action; uint32_t keycode; int32_t usbcode; #ifdef UKBD_EMULATE_ATSCANCODE uint32_t scancode; #endif if (!KBD_IS_ACTIVE(kbd)) return (NOKEY); if (sc->sc_flags & UKBD_FLAG_POLLING) { if (!mtx_owned(&Giant)) { /* XXX cludge */ int retval; mtx_lock(&Giant); retval = ukbd_read_char(kbd, wait); mtx_unlock(&Giant); return (retval); } } else { /* XXX the keyboard layer requires Giant */ if (!mtx_owned(&Giant)) return (NOKEY); } /* check if key belongs to this thread */ if (ukbd_polls_other_thread(sc)) return (NOKEY); next_code: /* do we have a composed char to return ? */ if ((sc->sc_composed_char > 0) && (!(sc->sc_flags & UKBD_FLAG_COMPOSE))) { action = sc->sc_composed_char; sc->sc_composed_char = 0; if (action > 0xFF) { goto errkey; } goto done; } #ifdef UKBD_EMULATE_ATSCANCODE /* do we have a pending raw scan code? */ if (sc->sc_mode == K_RAW) { scancode = sc->sc_buffered_char[0]; if (scancode) { if (scancode & SCAN_PREFIX) { sc->sc_buffered_char[0] = (scancode & ~SCAN_PREFIX); return ((scancode & SCAN_PREFIX_E0) ? 0xe0 : 0xe1); } sc->sc_buffered_char[0] = sc->sc_buffered_char[1]; sc->sc_buffered_char[1] = 0; return (scancode); } } #endif /* UKBD_EMULATE_ATSCANCODE */ /* see if there is something in the keyboard port */ /* XXX */ usbcode = ukbd_get_key(sc, (wait == FALSE) ? 0 : 1); if (usbcode == -1) { return (NOKEY); } ++kbd->kb_count; #ifdef UKBD_EMULATE_ATSCANCODE /* USB key index -> key code -> AT scan code */ keycode = ukbd_trtab[KEY_INDEX(usbcode)]; if (keycode == NN) { return (NOKEY); } /* return an AT scan code for the K_RAW mode */ if (sc->sc_mode == K_RAW) { return (ukbd_key2scan(sc, keycode, sc->sc_ndata.modifiers, (usbcode & KEY_RELEASE))); } #else /* !UKBD_EMULATE_ATSCANCODE */ /* return the byte as is for the K_RAW mode */ if (sc->sc_mode == K_RAW) { return (usbcode); } /* USB key index -> key code */ keycode = ukbd_trtab[KEY_INDEX(usbcode)]; if (keycode == NN) { return (NOKEY); } #endif /* UKBD_EMULATE_ATSCANCODE */ switch (keycode) { case 0x38: /* left alt (compose key) */ if (usbcode & KEY_RELEASE) { if (sc->sc_flags & UKBD_FLAG_COMPOSE) { sc->sc_flags &= ~UKBD_FLAG_COMPOSE; if (sc->sc_composed_char > 0xFF) { sc->sc_composed_char = 0; } } } else { if (!(sc->sc_flags & UKBD_FLAG_COMPOSE)) { sc->sc_flags |= UKBD_FLAG_COMPOSE; sc->sc_composed_char = 0; } } break; /* XXX: I don't like these... */ case 0x5c: /* print screen */ if (sc->sc_flags & ALTS) { keycode = 0x54; /* sysrq */ } break; case 0x68: /* pause/break */ if (sc->sc_flags & CTLS) { keycode = 0x6c; /* break */ } break; } /* return the key code in the K_CODE mode */ if (usbcode & KEY_RELEASE) { keycode |= SCAN_RELEASE; } if (sc->sc_mode == K_CODE) { return (keycode); } /* compose a character code */ if (sc->sc_flags & UKBD_FLAG_COMPOSE) { switch (keycode) { /* key pressed, process it */ case 0x47: case 0x48: case 0x49: /* keypad 7,8,9 */ sc->sc_composed_char *= 10; sc->sc_composed_char += keycode - 0x40; goto check_composed; case 0x4B: case 0x4C: case 0x4D: /* keypad 4,5,6 */ sc->sc_composed_char *= 10; sc->sc_composed_char += keycode - 0x47; goto check_composed; case 0x4F: case 0x50: case 0x51: /* keypad 1,2,3 */ sc->sc_composed_char *= 10; sc->sc_composed_char += keycode - 0x4E; goto check_composed; case 0x52: /* keypad 0 */ sc->sc_composed_char *= 10; goto check_composed; /* key released, no interest here */ case SCAN_RELEASE | 0x47: case SCAN_RELEASE | 0x48: case SCAN_RELEASE | 0x49: /* keypad 7,8,9 */ case SCAN_RELEASE | 0x4B: case SCAN_RELEASE | 0x4C: case SCAN_RELEASE | 0x4D: /* keypad 4,5,6 */ case SCAN_RELEASE | 0x4F: case SCAN_RELEASE | 0x50: case SCAN_RELEASE | 0x51: /* keypad 1,2,3 */ case SCAN_RELEASE | 0x52: /* keypad 0 */ goto next_code; case 0x38: /* left alt key */ break; default: if (sc->sc_composed_char > 0) { sc->sc_flags &= ~UKBD_FLAG_COMPOSE; sc->sc_composed_char = 0; goto errkey; } break; } } /* keycode to key action */ action = genkbd_keyaction(kbd, SCAN_CHAR(keycode), (keycode & SCAN_RELEASE), &sc->sc_state, &sc->sc_accents); if (action == NOKEY) { goto next_code; } done: return (action); check_composed: if (sc->sc_composed_char <= 0xFF) { goto next_code; } errkey: return (ERRKEY); } /* some useful control functions */ static int ukbd_ioctl(keyboard_t *kbd, u_long cmd, caddr_t arg) { /* translate LED_XXX bits into the device specific bits */ static const uint8_t ledmap[8] = { 0, 2, 1, 3, 4, 6, 5, 7, }; struct ukbd_softc *sc = kbd->kb_data; int i; #if defined(COMPAT_FREEBSD6) || defined(COMPAT_FREEBSD5) || \ defined(COMPAT_FREEBSD4) || defined(COMPAT_43) int ival; #endif if (!mtx_owned(&Giant)) { /* * XXX big problem: If scroll lock is pressed and "printf()" * is called, the CPU will get here, to un-scroll lock the * keyboard. But if "printf()" acquires the "Giant" lock, * there will be a locking order reversal problem, so the * keyboard system must get out of "Giant" first, before the * CPU can proceed here ... */ switch (cmd) { case KDGKBMODE: case KDSKBMODE: /* workaround for Geli */ mtx_lock(&Giant); i = ukbd_ioctl(kbd, cmd, arg); mtx_unlock(&Giant); return (i); default: return (EINVAL); } } switch (cmd) { case KDGKBMODE: /* get keyboard mode */ *(int *)arg = sc->sc_mode; break; #if defined(COMPAT_FREEBSD6) || defined(COMPAT_FREEBSD5) || \ defined(COMPAT_FREEBSD4) || defined(COMPAT_43) case _IO('K', 7): ival = IOCPARM_IVAL(arg); arg = (caddr_t)&ival; /* FALLTHROUGH */ #endif case KDSKBMODE: /* set keyboard mode */ switch (*(int *)arg) { case K_XLATE: if (sc->sc_mode != K_XLATE) { /* make lock key state and LED state match */ sc->sc_state &= ~LOCK_MASK; sc->sc_state |= KBD_LED_VAL(kbd); } /* FALLTHROUGH */ case K_RAW: case K_CODE: if (sc->sc_mode != *(int *)arg) { if (ukbd_is_polling(sc) == 0) ukbd_clear_state(kbd); sc->sc_mode = *(int *)arg; } break; default: return (EINVAL); } break; case KDGETLED: /* get keyboard LED */ *(int *)arg = KBD_LED_VAL(kbd); break; #if defined(COMPAT_FREEBSD6) || defined(COMPAT_FREEBSD5) || \ defined(COMPAT_FREEBSD4) || defined(COMPAT_43) case _IO('K', 66): ival = IOCPARM_IVAL(arg); arg = (caddr_t)&ival; /* FALLTHROUGH */ #endif case KDSETLED: /* set keyboard LED */ /* NOTE: lock key state in "sc_state" won't be changed */ if (*(int *)arg & ~LOCK_MASK) { return (EINVAL); } i = *(int *)arg; /* replace CAPS LED with ALTGR LED for ALTGR keyboards */ if (sc->sc_mode == K_XLATE && kbd->kb_keymap->n_keys > ALTGR_OFFSET) { if (i & ALKED) i |= CLKED; else i &= ~CLKED; } if (KBD_HAS_DEVICE(kbd)) { ukbd_set_leds(sc, ledmap[i & LED_MASK]); } KBD_LED_VAL(kbd) = *(int *)arg; break; case KDGKBSTATE: /* get lock key state */ *(int *)arg = sc->sc_state & LOCK_MASK; break; #if defined(COMPAT_FREEBSD6) || defined(COMPAT_FREEBSD5) || \ defined(COMPAT_FREEBSD4) || defined(COMPAT_43) case _IO('K', 20): ival = IOCPARM_IVAL(arg); arg = (caddr_t)&ival; /* FALLTHROUGH */ #endif case KDSKBSTATE: /* set lock key state */ if (*(int *)arg & ~LOCK_MASK) { return (EINVAL); } sc->sc_state &= ~LOCK_MASK; sc->sc_state |= *(int *)arg; /* set LEDs and quit */ return (ukbd_ioctl(kbd, KDSETLED, arg)); case KDSETREPEAT: /* set keyboard repeat rate (new * interface) */ if (!KBD_HAS_DEVICE(kbd)) { return (0); } if (((int *)arg)[1] < 0) { return (EINVAL); } if (((int *)arg)[0] < 0) { return (EINVAL); } if (((int *)arg)[0] < 200) /* fastest possible value */ kbd->kb_delay1 = 200; else kbd->kb_delay1 = ((int *)arg)[0]; kbd->kb_delay2 = ((int *)arg)[1]; return (0); #if defined(COMPAT_FREEBSD6) || defined(COMPAT_FREEBSD5) || \ defined(COMPAT_FREEBSD4) || defined(COMPAT_43) case _IO('K', 67): ival = IOCPARM_IVAL(arg); arg = (caddr_t)&ival; /* FALLTHROUGH */ #endif case KDSETRAD: /* set keyboard repeat rate (old * interface) */ return (ukbd_set_typematic(kbd, *(int *)arg)); case PIO_KEYMAP: /* set keyboard translation table */ case PIO_KEYMAPENT: /* set keyboard translation table * entry */ case PIO_DEADKEYMAP: /* set accent key translation table */ sc->sc_accents = 0; /* FALLTHROUGH */ default: return (genkbd_commonioctl(kbd, cmd, arg)); } return (0); } /* clear the internal state of the keyboard */ static void ukbd_clear_state(keyboard_t *kbd) { struct ukbd_softc *sc = kbd->kb_data; if (!mtx_owned(&Giant)) { /* XXX cludge */ mtx_lock(&Giant); ukbd_clear_state(kbd); mtx_unlock(&Giant); return; } sc->sc_flags &= ~(UKBD_FLAG_COMPOSE | UKBD_FLAG_POLLING); sc->sc_state &= LOCK_MASK; /* preserve locking key state */ sc->sc_accents = 0; sc->sc_composed_char = 0; #ifdef UKBD_EMULATE_ATSCANCODE sc->sc_buffered_char[0] = 0; sc->sc_buffered_char[1] = 0; #endif memset(&sc->sc_ndata, 0, sizeof(sc->sc_ndata)); memset(&sc->sc_odata, 0, sizeof(sc->sc_odata)); memset(&sc->sc_ntime, 0, sizeof(sc->sc_ntime)); memset(&sc->sc_otime, 0, sizeof(sc->sc_otime)); } /* save the internal state, not used */ static int ukbd_get_state(keyboard_t *kbd, void *buf, size_t len) { return (len == 0) ? 1 : -1; } /* set the internal state, not used */ static int ukbd_set_state(keyboard_t *kbd, void *buf, size_t len) { return (EINVAL); } static int ukbd_is_polling(struct ukbd_softc *sc) { int delta; if (sc->sc_flags & UKBD_FLAG_POLLING) return (1); /* polling */ delta = ticks - sc->sc_poll_tick_last; if ((delta < 0) || (delta >= hz)) { sc->sc_poll_detected = 0; return (0); /* not polling */ } return (sc->sc_poll_detected); } static int ukbd_polls_other_thread(struct ukbd_softc *sc) { return (ukbd_is_polling(sc) && (sc->sc_poll_thread != curthread)); } static int ukbd_poll(keyboard_t *kbd, int on) { struct ukbd_softc *sc = kbd->kb_data; if (!mtx_owned(&Giant)) { /* XXX cludge */ int retval; mtx_lock(&Giant); retval = ukbd_poll(kbd, on); mtx_unlock(&Giant); return (retval); } if (on) { sc->sc_flags |= UKBD_FLAG_POLLING; sc->sc_poll_thread = curthread; } else { sc->sc_flags &= ~UKBD_FLAG_POLLING; ukbd_start_timer(sc); /* start timer */ } return (0); } /* local functions */ static void ukbd_set_leds(struct ukbd_softc *sc, uint8_t leds) { DPRINTF("leds=0x%02x\n", leds); sc->sc_leds = leds; sc->sc_flags |= UKBD_FLAG_SET_LEDS; /* start transfer, if not already started */ usbd_transfer_start(sc->sc_xfer[UKBD_CTRL_LED]); } static int ukbd_set_typematic(keyboard_t *kbd, int code) { static const int delays[] = {250, 500, 750, 1000}; static const int rates[] = {34, 38, 42, 46, 50, 55, 59, 63, 68, 76, 84, 92, 100, 110, 118, 126, 136, 152, 168, 184, 200, 220, 236, 252, 272, 304, 336, 368, 400, 440, 472, 504}; if (code & ~0x7f) { return (EINVAL); } kbd->kb_delay1 = delays[(code >> 5) & 3]; kbd->kb_delay2 = rates[code & 0x1f]; return (0); } #ifdef UKBD_EMULATE_ATSCANCODE static int ukbd_key2scan(struct ukbd_softc *sc, int code, int shift, int up) { static const int scan[] = { /* 89 */ 0x11c, /* Enter */ /* 90-99 */ 0x11d, /* Ctrl-R */ 0x135, /* Divide */ 0x137 | SCAN_PREFIX_SHIFT, /* PrintScreen */ 0x138, /* Alt-R */ 0x147, /* Home */ 0x148, /* Up */ 0x149, /* PageUp */ 0x14b, /* Left */ 0x14d, /* Right */ 0x14f, /* End */ /* 100-109 */ 0x150, /* Down */ 0x151, /* PageDown */ 0x152, /* Insert */ 0x153, /* Delete */ 0x146, /* XXX Pause/Break */ 0x15b, /* Win_L(Super_L) */ 0x15c, /* Win_R(Super_R) */ 0x15d, /* Application(Menu) */ /* SUN TYPE 6 USB KEYBOARD */ 0x168, /* Sun Type 6 Help */ 0x15e, /* Sun Type 6 Stop */ /* 110 - 119 */ 0x15f, /* Sun Type 6 Again */ 0x160, /* Sun Type 6 Props */ 0x161, /* Sun Type 6 Undo */ 0x162, /* Sun Type 6 Front */ 0x163, /* Sun Type 6 Copy */ 0x164, /* Sun Type 6 Open */ 0x165, /* Sun Type 6 Paste */ 0x166, /* Sun Type 6 Find */ 0x167, /* Sun Type 6 Cut */ 0x125, /* Sun Type 6 Mute */ /* 120 - 128 */ 0x11f, /* Sun Type 6 VolumeDown */ 0x11e, /* Sun Type 6 VolumeUp */ 0x120, /* Sun Type 6 PowerDown */ /* Japanese 106/109 keyboard */ 0x73, /* Keyboard Intl' 1 (backslash / underscore) */ 0x70, /* Keyboard Intl' 2 (Katakana / Hiragana) */ 0x7d, /* Keyboard Intl' 3 (Yen sign) (Not using in jp106/109) */ 0x79, /* Keyboard Intl' 4 (Henkan) */ 0x7b, /* Keyboard Intl' 5 (Muhenkan) */ 0x5c, /* Keyboard Intl' 6 (Keypad ,) (For PC-9821 layout) */ }; if ((code >= 89) && (code < (89 + (sizeof(scan) / sizeof(scan[0]))))) { code = scan[code - 89]; } /* Pause/Break */ if ((code == 104) && (!(shift & (MOD_CONTROL_L | MOD_CONTROL_R)))) { code = (0x45 | SCAN_PREFIX_E1 | SCAN_PREFIX_CTL); } if (shift & (MOD_SHIFT_L | MOD_SHIFT_R)) { code &= ~SCAN_PREFIX_SHIFT; } code |= (up ? SCAN_RELEASE : SCAN_PRESS); if (code & SCAN_PREFIX) { if (code & SCAN_PREFIX_CTL) { /* Ctrl */ sc->sc_buffered_char[0] = (0x1d | (code & SCAN_RELEASE)); sc->sc_buffered_char[1] = (code & ~SCAN_PREFIX); } else if (code & SCAN_PREFIX_SHIFT) { /* Shift */ sc->sc_buffered_char[0] = (0x2a | (code & SCAN_RELEASE)); sc->sc_buffered_char[1] = (code & ~SCAN_PREFIX_SHIFT); } else { sc->sc_buffered_char[0] = (code & ~SCAN_PREFIX); sc->sc_buffered_char[1] = 0; } return ((code & SCAN_PREFIX_E0) ? 0xe0 : 0xe1); } return (code); } #endif /* UKBD_EMULATE_ATSCANCODE */ static keyboard_switch_t ukbdsw = { .probe = &ukbd__probe, .init = &ukbd_init, .term = &ukbd_term, .intr = &ukbd_intr, .test_if = &ukbd_test_if, .enable = &ukbd_enable, .disable = &ukbd_disable, .read = &ukbd_read, .check = &ukbd_check, .read_char = &ukbd_read_char, .check_char = &ukbd_check_char, .ioctl = &ukbd_ioctl, .lock = &ukbd_lock, .clear_state = &ukbd_clear_state, .get_state = &ukbd_get_state, .set_state = &ukbd_set_state, .get_fkeystr = &genkbd_get_fkeystr, .poll = &ukbd_poll, .diag = &genkbd_diag, }; KEYBOARD_DRIVER(ukbd, ukbdsw, ukbd_configure); static int ukbd_driver_load(module_t mod, int what, void *arg) { switch (what) { case MOD_LOAD: kbd_add_driver(&ukbd_kbd_driver); break; case MOD_UNLOAD: kbd_delete_driver(&ukbd_kbd_driver); break; } return (0); } static devclass_t ukbd_devclass; static device_method_t ukbd_methods[] = { DEVMETHOD(device_probe, ukbd_probe), DEVMETHOD(device_attach, ukbd_attach), DEVMETHOD(device_detach, ukbd_detach), DEVMETHOD(device_resume, ukbd_resume), {0, 0} }; static driver_t ukbd_driver = { .name = "ukbd", .methods = ukbd_methods, .size = sizeof(struct ukbd_softc), }; DRIVER_MODULE(ukbd, uhub, ukbd_driver, ukbd_devclass, ukbd_driver_load, 0); MODULE_DEPEND(ukbd, usb, 1, 1, 1); Index: user/imp/tbemd/sys/geom/eli/g_eli.c =================================================================== --- user/imp/tbemd/sys/geom/eli/g_eli.c (revision 206684) +++ user/imp/tbemd/sys/geom/eli/g_eli.c (revision 206685) @@ -1,1133 +1,1132 @@ /*- * Copyright (c) 2005-2006 Pawel Jakub Dawidek * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHORS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #include __FBSDID("$FreeBSD$"); #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include MALLOC_DEFINE(M_ELI, "eli data", "GEOM_ELI Data"); SYSCTL_DECL(_kern_geom); SYSCTL_NODE(_kern_geom, OID_AUTO, eli, CTLFLAG_RW, 0, "GEOM_ELI stuff"); u_int g_eli_debug = 0; TUNABLE_INT("kern.geom.eli.debug", &g_eli_debug); SYSCTL_UINT(_kern_geom_eli, OID_AUTO, debug, CTLFLAG_RW, &g_eli_debug, 0, "Debug level"); static u_int g_eli_tries = 3; TUNABLE_INT("kern.geom.eli.tries", &g_eli_tries); SYSCTL_UINT(_kern_geom_eli, OID_AUTO, tries, CTLFLAG_RW, &g_eli_tries, 0, "Number of tries for entering the passphrase"); static u_int g_eli_visible_passphrase = 0; TUNABLE_INT("kern.geom.eli.visible_passphrase", &g_eli_visible_passphrase); SYSCTL_UINT(_kern_geom_eli, OID_AUTO, visible_passphrase, CTLFLAG_RW, &g_eli_visible_passphrase, 0, "Turn on echo when entering the passphrase (for debug purposes only!!)"); u_int g_eli_overwrites = 5; TUNABLE_INT("kern.geom.eli.overwrites", &g_eli_overwrites); SYSCTL_UINT(_kern_geom_eli, OID_AUTO, overwrites, CTLFLAG_RW, &g_eli_overwrites, 0, "Number of times on-disk keys should be overwritten when destroying them"); static u_int g_eli_threads = 0; TUNABLE_INT("kern.geom.eli.threads", &g_eli_threads); SYSCTL_UINT(_kern_geom_eli, OID_AUTO, threads, CTLFLAG_RW, &g_eli_threads, 0, "Number of threads doing crypto work"); u_int g_eli_batch = 0; TUNABLE_INT("kern.geom.eli.batch", &g_eli_batch); SYSCTL_UINT(_kern_geom_eli, OID_AUTO, batch, CTLFLAG_RW, &g_eli_batch, 0, "Use crypto operations batching"); static eventhandler_tag g_eli_pre_sync = NULL; static int g_eli_destroy_geom(struct gctl_req *req, struct g_class *mp, struct g_geom *gp); static void g_eli_init(struct g_class *mp); static void g_eli_fini(struct g_class *mp); static g_taste_t g_eli_taste; static g_dumpconf_t g_eli_dumpconf; struct g_class g_eli_class = { .name = G_ELI_CLASS_NAME, .version = G_VERSION, .ctlreq = g_eli_config, .taste = g_eli_taste, .destroy_geom = g_eli_destroy_geom, .init = g_eli_init, .fini = g_eli_fini }; /* * Code paths: * BIO_READ: * g_eli_start -> g_io_request -> g_eli_read_done -> g_eli_crypto_run -> g_eli_crypto_read_done -> g_io_deliver * BIO_WRITE: * g_eli_start -> g_eli_crypto_run -> g_eli_crypto_write_done -> g_io_request -> g_eli_write_done -> g_io_deliver */ /* * EAGAIN from crypto(9) means, that we were probably balanced to another crypto * accelerator or something like this. * The function updates the SID and rerun the operation. */ int g_eli_crypto_rerun(struct cryptop *crp) { struct g_eli_softc *sc; struct g_eli_worker *wr; struct bio *bp; int error; bp = (struct bio *)crp->crp_opaque; sc = bp->bio_to->geom->softc; LIST_FOREACH(wr, &sc->sc_workers, w_next) { if (wr->w_number == bp->bio_pflags) break; } KASSERT(wr != NULL, ("Invalid worker (%u).", bp->bio_pflags)); G_ELI_DEBUG(1, "Rerunning crypto %s request (sid: %ju -> %ju).", bp->bio_cmd == BIO_READ ? "READ" : "WRITE", (uintmax_t)wr->w_sid, (uintmax_t)crp->crp_sid); wr->w_sid = crp->crp_sid; crp->crp_etype = 0; error = crypto_dispatch(crp); if (error == 0) return (0); G_ELI_DEBUG(1, "%s: crypto_dispatch() returned %d.", __func__, error); crp->crp_etype = error; return (error); } /* * The function is called afer reading encrypted data from the provider. * * g_eli_start -> g_io_request -> G_ELI_READ_DONE -> g_eli_crypto_run -> g_eli_crypto_read_done -> g_io_deliver */ void g_eli_read_done(struct bio *bp) { struct g_eli_softc *sc; struct bio *pbp; G_ELI_LOGREQ(2, bp, "Request done."); pbp = bp->bio_parent; if (pbp->bio_error == 0) pbp->bio_error = bp->bio_error; /* * Do we have all sectors already? */ pbp->bio_inbed++; if (pbp->bio_inbed < pbp->bio_children) return; g_destroy_bio(bp); if (pbp->bio_error != 0) { G_ELI_LOGREQ(0, pbp, "%s() failed", __func__); pbp->bio_completed = 0; if (pbp->bio_driver2 != NULL) { free(pbp->bio_driver2, M_ELI); pbp->bio_driver2 = NULL; } g_io_deliver(pbp, pbp->bio_error); return; } sc = pbp->bio_to->geom->softc; mtx_lock(&sc->sc_queue_mtx); bioq_insert_tail(&sc->sc_queue, pbp); mtx_unlock(&sc->sc_queue_mtx); wakeup(sc); } /* * The function is called after we encrypt and write data. * * g_eli_start -> g_eli_crypto_run -> g_eli_crypto_write_done -> g_io_request -> G_ELI_WRITE_DONE -> g_io_deliver */ void g_eli_write_done(struct bio *bp) { struct bio *pbp; G_ELI_LOGREQ(2, bp, "Request done."); pbp = bp->bio_parent; if (pbp->bio_error == 0) { if (bp->bio_error != 0) pbp->bio_error = bp->bio_error; } /* * Do we have all sectors already? */ pbp->bio_inbed++; if (pbp->bio_inbed < pbp->bio_children) return; free(pbp->bio_driver2, M_ELI); pbp->bio_driver2 = NULL; if (pbp->bio_error != 0) { G_ELI_LOGREQ(0, pbp, "Crypto WRITE request failed (error=%d).", pbp->bio_error); pbp->bio_completed = 0; } g_destroy_bio(bp); /* * Write is finished, send it up. */ pbp->bio_completed = pbp->bio_length; g_io_deliver(pbp, pbp->bio_error); } /* * This function should never be called, but GEOM made as it set ->orphan() * method for every geom. */ static void g_eli_orphan_spoil_assert(struct g_consumer *cp) { panic("Function %s() called for %s.", __func__, cp->geom->name); } static void g_eli_orphan(struct g_consumer *cp) { struct g_eli_softc *sc; g_topology_assert(); sc = cp->geom->softc; if (sc == NULL) return; g_eli_destroy(sc, 1); } /* * BIO_READ : G_ELI_START -> g_io_request -> g_eli_read_done -> g_eli_crypto_run -> g_eli_crypto_read_done -> g_io_deliver * BIO_WRITE: G_ELI_START -> g_eli_crypto_run -> g_eli_crypto_write_done -> g_io_request -> g_eli_write_done -> g_io_deliver */ static void g_eli_start(struct bio *bp) { struct g_eli_softc *sc; struct g_consumer *cp; struct bio *cbp; sc = bp->bio_to->geom->softc; KASSERT(sc != NULL, ("Provider's error should be set (error=%d)(device=%s).", bp->bio_to->error, bp->bio_to->name)); G_ELI_LOGREQ(2, bp, "Request received."); switch (bp->bio_cmd) { case BIO_READ: case BIO_WRITE: case BIO_GETATTR: case BIO_FLUSH: break; case BIO_DELETE: /* * We could eventually support BIO_DELETE request. * It could be done by overwritting requested sector with * random data g_eli_overwrites number of times. */ default: g_io_deliver(bp, EOPNOTSUPP); return; } cbp = g_clone_bio(bp); if (cbp == NULL) { g_io_deliver(bp, ENOMEM); return; } switch (bp->bio_cmd) { case BIO_READ: if (!(sc->sc_flags & G_ELI_FLAG_AUTH)) { bp->bio_driver2 = NULL; cbp->bio_done = g_eli_read_done; cp = LIST_FIRST(&sc->sc_geom->consumer); cbp->bio_to = cp->provider; G_ELI_LOGREQ(2, cbp, "Sending request."); /* * Read encrypted data from provider. */ g_io_request(cbp, cp); break; } bp->bio_pflags = 255; /* FALLTHROUGH */ case BIO_WRITE: bp->bio_driver1 = cbp; mtx_lock(&sc->sc_queue_mtx); bioq_insert_tail(&sc->sc_queue, bp); mtx_unlock(&sc->sc_queue_mtx); wakeup(sc); break; case BIO_GETATTR: case BIO_FLUSH: cbp->bio_done = g_std_done; cp = LIST_FIRST(&sc->sc_geom->consumer); cbp->bio_to = cp->provider; G_ELI_LOGREQ(2, cbp, "Sending request."); g_io_request(cbp, cp); break; } } /* * This is the main function for kernel worker thread when we don't have * hardware acceleration and we have to do cryptography in software. * Dedicated thread is needed, so we don't slow down g_up/g_down GEOM * threads with crypto work. */ static void g_eli_worker(void *arg) { struct g_eli_softc *sc; struct g_eli_worker *wr; struct bio *bp; wr = arg; sc = wr->w_softc; #ifdef SMP /* Before sched_bind() to a CPU, wait for all CPUs to go on-line. */ if (mp_ncpus > 1 && sc->sc_crypto == G_ELI_CRYPTO_SW && g_eli_threads == 0) { while (!smp_started) tsleep(wr, 0, "geli:smp", hz / 4); } #endif thread_lock(curthread); - sched_prio(curthread, PRIBIO); + sched_prio(curthread, PUSER); if (sc->sc_crypto == G_ELI_CRYPTO_SW && g_eli_threads == 0) sched_bind(curthread, wr->w_number); thread_unlock(curthread); G_ELI_DEBUG(1, "Thread %s started.", curthread->td_proc->p_comm); for (;;) { mtx_lock(&sc->sc_queue_mtx); bp = bioq_takefirst(&sc->sc_queue); if (bp == NULL) { if (sc->sc_flags & G_ELI_FLAG_DESTROY) { LIST_REMOVE(wr, w_next); crypto_freesession(wr->w_sid); free(wr, M_ELI); G_ELI_DEBUG(1, "Thread %s exiting.", curthread->td_proc->p_comm); wakeup(&sc->sc_workers); mtx_unlock(&sc->sc_queue_mtx); kproc_exit(0); } - msleep(sc, &sc->sc_queue_mtx, PRIBIO | PDROP, - "geli:w", 0); + msleep(sc, &sc->sc_queue_mtx, PDROP, "geli:w", 0); continue; } mtx_unlock(&sc->sc_queue_mtx); if (bp->bio_cmd == BIO_READ && bp->bio_pflags == 255) g_eli_auth_read(sc, bp); else if (sc->sc_flags & G_ELI_FLAG_AUTH) g_eli_auth_run(wr, bp); else g_eli_crypto_run(wr, bp); } } /* * Here we generate IV. It is unique for every sector. */ void g_eli_crypto_ivgen(struct g_eli_softc *sc, off_t offset, u_char *iv, size_t size) { u_char off[8], hash[SHA256_DIGEST_LENGTH]; SHA256_CTX ctx; if (!(sc->sc_flags & G_ELI_FLAG_NATIVE_BYTE_ORDER)) le64enc(off, (uint64_t)offset); /* Copy precalculated SHA256 context for IV-Key. */ bcopy(&sc->sc_ivctx, &ctx, sizeof(ctx)); SHA256_Update(&ctx, (uint8_t *)&offset, sizeof(offset)); SHA256_Final(hash, &ctx); bcopy(hash, iv, size); } int g_eli_read_metadata(struct g_class *mp, struct g_provider *pp, struct g_eli_metadata *md) { struct g_geom *gp; struct g_consumer *cp; u_char *buf = NULL; int error; g_topology_assert(); gp = g_new_geomf(mp, "eli:taste"); gp->start = g_eli_start; gp->access = g_std_access; /* * g_eli_read_metadata() is always called from the event thread. * Our geom is created and destroyed in the same event, so there * could be no orphan nor spoil event in the meantime. */ gp->orphan = g_eli_orphan_spoil_assert; gp->spoiled = g_eli_orphan_spoil_assert; cp = g_new_consumer(gp); error = g_attach(cp, pp); if (error != 0) goto end; error = g_access(cp, 1, 0, 0); if (error != 0) goto end; g_topology_unlock(); buf = g_read_data(cp, pp->mediasize - pp->sectorsize, pp->sectorsize, &error); g_topology_lock(); if (buf == NULL) goto end; eli_metadata_decode(buf, md); end: if (buf != NULL) g_free(buf); if (cp->provider != NULL) { if (cp->acr == 1) g_access(cp, -1, 0, 0); g_detach(cp); } g_destroy_consumer(cp); g_destroy_geom(gp); return (error); } /* * The function is called when we had last close on provider and user requested * to close it when this situation occur. */ static void g_eli_last_close(struct g_eli_softc *sc) { struct g_geom *gp; struct g_provider *pp; char ppname[64]; int error; g_topology_assert(); gp = sc->sc_geom; pp = LIST_FIRST(&gp->provider); strlcpy(ppname, pp->name, sizeof(ppname)); error = g_eli_destroy(sc, 1); KASSERT(error == 0, ("Cannot detach %s on last close (error=%d).", ppname, error)); G_ELI_DEBUG(0, "Detached %s on last close.", ppname); } int g_eli_access(struct g_provider *pp, int dr, int dw, int de) { struct g_eli_softc *sc; struct g_geom *gp; gp = pp->geom; sc = gp->softc; if (dw > 0) { if (sc->sc_flags & G_ELI_FLAG_RO) { /* Deny write attempts. */ return (EROFS); } /* Someone is opening us for write, we need to remember that. */ sc->sc_flags |= G_ELI_FLAG_WOPEN; return (0); } /* Is this the last close? */ if (pp->acr + dr > 0 || pp->acw + dw > 0 || pp->ace + de > 0) return (0); /* * Automatically detach on last close if requested. */ if ((sc->sc_flags & G_ELI_FLAG_RW_DETACH) || (sc->sc_flags & G_ELI_FLAG_WOPEN)) { g_eli_last_close(sc); } return (0); } static int g_eli_cpu_is_disabled(int cpu) { #ifdef SMP return ((hlt_cpus_mask & (1 << cpu)) != 0); #else return (0); #endif } struct g_geom * g_eli_create(struct gctl_req *req, struct g_class *mp, struct g_provider *bpp, const struct g_eli_metadata *md, const u_char *mkey, int nkey) { struct g_eli_softc *sc; struct g_eli_worker *wr; struct g_geom *gp; struct g_provider *pp; struct g_consumer *cp; struct cryptoini crie, cria; u_int i, threads; int error; G_ELI_DEBUG(1, "Creating device %s%s.", bpp->name, G_ELI_SUFFIX); gp = g_new_geomf(mp, "%s%s", bpp->name, G_ELI_SUFFIX); gp->softc = NULL; /* for a moment */ sc = malloc(sizeof(*sc), M_ELI, M_WAITOK | M_ZERO); gp->start = g_eli_start; /* * Spoiling cannot happen actually, because we keep provider open for * writing all the time or provider is read-only. */ gp->spoiled = g_eli_orphan_spoil_assert; gp->orphan = g_eli_orphan; gp->dumpconf = g_eli_dumpconf; /* * If detach-on-last-close feature is not enabled and we don't operate * on read-only provider, we can simply use g_std_access(). */ if (md->md_flags & (G_ELI_FLAG_WO_DETACH | G_ELI_FLAG_RO)) gp->access = g_eli_access; else gp->access = g_std_access; sc->sc_crypto = G_ELI_CRYPTO_SW; sc->sc_flags = md->md_flags; /* Backward compatibility. */ if (md->md_version < 2) sc->sc_flags |= G_ELI_FLAG_NATIVE_BYTE_ORDER; sc->sc_ealgo = md->md_ealgo; sc->sc_nkey = nkey; /* * Remember the keys in our softc structure. */ g_eli_mkey_propagate(sc, mkey); sc->sc_ekeylen = md->md_keylen; if (sc->sc_flags & G_ELI_FLAG_AUTH) { sc->sc_akeylen = sizeof(sc->sc_akey) * 8; sc->sc_aalgo = md->md_aalgo; sc->sc_alen = g_eli_hashlen(sc->sc_aalgo); sc->sc_data_per_sector = bpp->sectorsize - sc->sc_alen; /* * Some hash functions (like SHA1 and RIPEMD160) generates hash * which length is not multiple of 128 bits, but we want data * length to be multiple of 128, so we can encrypt without * padding. The line below rounds down data length to multiple * of 128 bits. */ sc->sc_data_per_sector -= sc->sc_data_per_sector % 16; sc->sc_bytes_per_sector = (md->md_sectorsize - 1) / sc->sc_data_per_sector + 1; sc->sc_bytes_per_sector *= bpp->sectorsize; /* * Precalculate SHA256 for HMAC key generation. * This is expensive operation and we can do it only once now or * for every access to sector, so now will be much better. */ SHA256_Init(&sc->sc_akeyctx); SHA256_Update(&sc->sc_akeyctx, sc->sc_akey, sizeof(sc->sc_akey)); } /* * Precalculate SHA256 for IV generation. * This is expensive operation and we can do it only once now or for * every access to sector, so now will be much better. */ SHA256_Init(&sc->sc_ivctx); SHA256_Update(&sc->sc_ivctx, sc->sc_ivkey, sizeof(sc->sc_ivkey)); gp->softc = sc; sc->sc_geom = gp; bioq_init(&sc->sc_queue); mtx_init(&sc->sc_queue_mtx, "geli:queue", NULL, MTX_DEF); pp = NULL; cp = g_new_consumer(gp); error = g_attach(cp, bpp); if (error != 0) { if (req != NULL) { gctl_error(req, "Cannot attach to %s (error=%d).", bpp->name, error); } else { G_ELI_DEBUG(1, "Cannot attach to %s (error=%d).", bpp->name, error); } goto failed; } /* * Keep provider open all the time, so we can run critical tasks, * like Master Keys deletion, without wondering if we can open * provider or not. * We don't open provider for writing only when user requested read-only * access. */ if (sc->sc_flags & G_ELI_FLAG_RO) error = g_access(cp, 1, 0, 1); else error = g_access(cp, 1, 1, 1); if (error != 0) { if (req != NULL) { gctl_error(req, "Cannot access %s (error=%d).", bpp->name, error); } else { G_ELI_DEBUG(1, "Cannot access %s (error=%d).", bpp->name, error); } goto failed; } LIST_INIT(&sc->sc_workers); bzero(&crie, sizeof(crie)); crie.cri_alg = sc->sc_ealgo; crie.cri_klen = sc->sc_ekeylen; crie.cri_key = sc->sc_ekey; if (sc->sc_flags & G_ELI_FLAG_AUTH) { bzero(&cria, sizeof(cria)); cria.cri_alg = sc->sc_aalgo; cria.cri_klen = sc->sc_akeylen; cria.cri_key = sc->sc_akey; crie.cri_next = &cria; } threads = g_eli_threads; if (threads == 0) threads = mp_ncpus; else if (threads > mp_ncpus) { /* There is really no need for too many worker threads. */ threads = mp_ncpus; G_ELI_DEBUG(0, "Reducing number of threads to %u.", threads); } for (i = 0; i < threads; i++) { if (g_eli_cpu_is_disabled(i)) { G_ELI_DEBUG(1, "%s: CPU %u disabled, skipping.", bpp->name, i); continue; } wr = malloc(sizeof(*wr), M_ELI, M_WAITOK | M_ZERO); wr->w_softc = sc; wr->w_number = i; /* * If this is the first pass, try to get hardware support. * Use software cryptography, if we cannot get it. */ if (LIST_EMPTY(&sc->sc_workers)) { error = crypto_newsession(&wr->w_sid, &crie, CRYPTOCAP_F_HARDWARE); if (error == 0) sc->sc_crypto = G_ELI_CRYPTO_HW; } if (sc->sc_crypto == G_ELI_CRYPTO_SW) { error = crypto_newsession(&wr->w_sid, &crie, CRYPTOCAP_F_SOFTWARE); } if (error != 0) { free(wr, M_ELI); if (req != NULL) { gctl_error(req, "Cannot set up crypto session " "for %s (error=%d).", bpp->name, error); } else { G_ELI_DEBUG(1, "Cannot set up crypto session " "for %s (error=%d).", bpp->name, error); } goto failed; } error = kproc_create(g_eli_worker, wr, &wr->w_proc, 0, 0, "g_eli[%u] %s", i, bpp->name); if (error != 0) { crypto_freesession(wr->w_sid); free(wr, M_ELI); if (req != NULL) { gctl_error(req, "Cannot create kernel thread " "for %s (error=%d).", bpp->name, error); } else { G_ELI_DEBUG(1, "Cannot create kernel thread " "for %s (error=%d).", bpp->name, error); } goto failed; } LIST_INSERT_HEAD(&sc->sc_workers, wr, w_next); /* If we have hardware support, one thread is enough. */ if (sc->sc_crypto == G_ELI_CRYPTO_HW) break; } /* * Create decrypted provider. */ pp = g_new_providerf(gp, "%s%s", bpp->name, G_ELI_SUFFIX); pp->sectorsize = md->md_sectorsize; pp->mediasize = bpp->mediasize; if (!(sc->sc_flags & G_ELI_FLAG_ONETIME)) pp->mediasize -= bpp->sectorsize; if (!(sc->sc_flags & G_ELI_FLAG_AUTH)) pp->mediasize -= (pp->mediasize % pp->sectorsize); else { pp->mediasize /= sc->sc_bytes_per_sector; pp->mediasize *= pp->sectorsize; } g_error_provider(pp, 0); G_ELI_DEBUG(0, "Device %s created.", pp->name); G_ELI_DEBUG(0, "Encryption: %s %u", g_eli_algo2str(sc->sc_ealgo), sc->sc_ekeylen); if (sc->sc_flags & G_ELI_FLAG_AUTH) G_ELI_DEBUG(0, " Integrity: %s", g_eli_algo2str(sc->sc_aalgo)); G_ELI_DEBUG(0, " Crypto: %s", sc->sc_crypto == G_ELI_CRYPTO_SW ? "software" : "hardware"); return (gp); failed: mtx_lock(&sc->sc_queue_mtx); sc->sc_flags |= G_ELI_FLAG_DESTROY; wakeup(sc); /* * Wait for kernel threads self destruction. */ while (!LIST_EMPTY(&sc->sc_workers)) { msleep(&sc->sc_workers, &sc->sc_queue_mtx, PRIBIO, "geli:destroy", 0); } mtx_destroy(&sc->sc_queue_mtx); if (cp->provider != NULL) { if (cp->acr == 1) g_access(cp, -1, -1, -1); g_detach(cp); } g_destroy_consumer(cp); g_destroy_geom(gp); bzero(sc, sizeof(*sc)); free(sc, M_ELI); return (NULL); } int g_eli_destroy(struct g_eli_softc *sc, boolean_t force) { struct g_geom *gp; struct g_provider *pp; g_topology_assert(); if (sc == NULL) return (ENXIO); gp = sc->sc_geom; pp = LIST_FIRST(&gp->provider); if (pp != NULL && (pp->acr != 0 || pp->acw != 0 || pp->ace != 0)) { if (force) { G_ELI_DEBUG(1, "Device %s is still open, so it " "cannot be definitely removed.", pp->name); } else { G_ELI_DEBUG(1, "Device %s is still open (r%dw%de%d).", pp->name, pp->acr, pp->acw, pp->ace); return (EBUSY); } } mtx_lock(&sc->sc_queue_mtx); sc->sc_flags |= G_ELI_FLAG_DESTROY; wakeup(sc); while (!LIST_EMPTY(&sc->sc_workers)) { msleep(&sc->sc_workers, &sc->sc_queue_mtx, PRIBIO, "geli:destroy", 0); } mtx_destroy(&sc->sc_queue_mtx); gp->softc = NULL; bzero(sc, sizeof(*sc)); free(sc, M_ELI); if (pp == NULL || (pp->acr == 0 && pp->acw == 0 && pp->ace == 0)) G_ELI_DEBUG(0, "Device %s destroyed.", gp->name); g_wither_geom_close(gp, ENXIO); return (0); } static int g_eli_destroy_geom(struct gctl_req *req __unused, struct g_class *mp __unused, struct g_geom *gp) { struct g_eli_softc *sc; sc = gp->softc; return (g_eli_destroy(sc, 0)); } static int g_eli_keyfiles_load(struct hmac_ctx *ctx, const char *provider) { u_char *keyfile, *data, *size; char *file, name[64]; int i; for (i = 0; ; i++) { snprintf(name, sizeof(name), "%s:geli_keyfile%d", provider, i); keyfile = preload_search_by_type(name); if (keyfile == NULL) return (i); /* Return number of loaded keyfiles. */ data = preload_search_info(keyfile, MODINFO_ADDR); if (data == NULL) { G_ELI_DEBUG(0, "Cannot find key file data for %s.", name); return (0); } data = *(void **)data; size = preload_search_info(keyfile, MODINFO_SIZE); if (size == NULL) { G_ELI_DEBUG(0, "Cannot find key file size for %s.", name); return (0); } file = preload_search_info(keyfile, MODINFO_NAME); if (file == NULL) { G_ELI_DEBUG(0, "Cannot find key file name for %s.", name); return (0); } G_ELI_DEBUG(1, "Loaded keyfile %s for %s (type: %s).", file, provider, name); g_eli_crypto_hmac_update(ctx, data, *(size_t *)size); } } static void g_eli_keyfiles_clear(const char *provider) { u_char *keyfile, *data, *size; char name[64]; int i; for (i = 0; ; i++) { snprintf(name, sizeof(name), "%s:geli_keyfile%d", provider, i); keyfile = preload_search_by_type(name); if (keyfile == NULL) return; data = preload_search_info(keyfile, MODINFO_ADDR); size = preload_search_info(keyfile, MODINFO_SIZE); if (data == NULL || size == NULL) continue; data = *(void **)data; bzero(data, *(size_t *)size); } } /* * Tasting is only made on boot. * We detect providers which should be attached before root is mounted. */ static struct g_geom * g_eli_taste(struct g_class *mp, struct g_provider *pp, int flags __unused) { struct g_eli_metadata md; struct g_geom *gp; struct hmac_ctx ctx; char passphrase[256]; u_char key[G_ELI_USERKEYLEN], mkey[G_ELI_DATAIVKEYLEN]; u_int i, nkey, nkeyfiles, tries; int error; g_trace(G_T_TOPOLOGY, "%s(%s, %s)", __func__, mp->name, pp->name); g_topology_assert(); if (root_mounted() || g_eli_tries == 0) return (NULL); G_ELI_DEBUG(3, "Tasting %s.", pp->name); error = g_eli_read_metadata(mp, pp, &md); if (error != 0) return (NULL); gp = NULL; if (strcmp(md.md_magic, G_ELI_MAGIC) != 0) return (NULL); if (md.md_version > G_ELI_VERSION) { printf("geom_eli.ko module is too old to handle %s.\n", pp->name); return (NULL); } if (md.md_provsize != pp->mediasize) return (NULL); /* Should we attach it on boot? */ if (!(md.md_flags & G_ELI_FLAG_BOOT)) return (NULL); if (md.md_keys == 0x00) { G_ELI_DEBUG(0, "No valid keys on %s.", pp->name); return (NULL); } if (md.md_iterations == -1) { /* If there is no passphrase, we try only once. */ tries = 1; } else { /* Ask for the passphrase no more than g_eli_tries times. */ tries = g_eli_tries; } for (i = 0; i < tries; i++) { g_eli_crypto_hmac_init(&ctx, NULL, 0); /* * Load all key files. */ nkeyfiles = g_eli_keyfiles_load(&ctx, pp->name); if (nkeyfiles == 0 && md.md_iterations == -1) { /* * No key files and no passphrase, something is * definitely wrong here. * geli(8) doesn't allow for such situation, so assume * that there was really no passphrase and in that case * key files are no properly defined in loader.conf. */ G_ELI_DEBUG(0, "Found no key files in loader.conf for %s.", pp->name); return (NULL); } /* Ask for the passphrase if defined. */ if (md.md_iterations >= 0) { printf("Enter passphrase for %s: ", pp->name); gets(passphrase, sizeof(passphrase), g_eli_visible_passphrase); } /* * Prepare Derived-Key from the user passphrase. */ if (md.md_iterations == 0) { g_eli_crypto_hmac_update(&ctx, md.md_salt, sizeof(md.md_salt)); g_eli_crypto_hmac_update(&ctx, passphrase, strlen(passphrase)); bzero(passphrase, sizeof(passphrase)); } else if (md.md_iterations > 0) { u_char dkey[G_ELI_USERKEYLEN]; pkcs5v2_genkey(dkey, sizeof(dkey), md.md_salt, sizeof(md.md_salt), passphrase, md.md_iterations); bzero(passphrase, sizeof(passphrase)); g_eli_crypto_hmac_update(&ctx, dkey, sizeof(dkey)); bzero(dkey, sizeof(dkey)); } g_eli_crypto_hmac_final(&ctx, key, 0); /* * Decrypt Master-Key. */ error = g_eli_mkey_decrypt(&md, key, mkey, &nkey); bzero(key, sizeof(key)); if (error == -1) { if (i == tries - 1) { G_ELI_DEBUG(0, "Wrong key for %s. No tries left.", pp->name); g_eli_keyfiles_clear(pp->name); return (NULL); } G_ELI_DEBUG(0, "Wrong key for %s. Tries left: %u.", pp->name, tries - i - 1); /* Try again. */ continue; } else if (error > 0) { G_ELI_DEBUG(0, "Cannot decrypt Master Key for %s (error=%d).", pp->name, error); g_eli_keyfiles_clear(pp->name); return (NULL); } G_ELI_DEBUG(1, "Using Master Key %u for %s.", nkey, pp->name); break; } /* * We have correct key, let's attach provider. */ gp = g_eli_create(NULL, mp, pp, &md, mkey, nkey); bzero(mkey, sizeof(mkey)); bzero(&md, sizeof(md)); if (gp == NULL) { G_ELI_DEBUG(0, "Cannot create device %s%s.", pp->name, G_ELI_SUFFIX); return (NULL); } return (gp); } static void g_eli_dumpconf(struct sbuf *sb, const char *indent, struct g_geom *gp, struct g_consumer *cp, struct g_provider *pp) { struct g_eli_softc *sc; g_topology_assert(); sc = gp->softc; if (sc == NULL) return; if (pp != NULL || cp != NULL) return; /* Nothing here. */ sbuf_printf(sb, "%s", indent); if (sc->sc_flags == 0) sbuf_printf(sb, "NONE"); else { int first = 1; #define ADD_FLAG(flag, name) do { \ if (sc->sc_flags & (flag)) { \ if (!first) \ sbuf_printf(sb, ", "); \ else \ first = 0; \ sbuf_printf(sb, name); \ } \ } while (0) ADD_FLAG(G_ELI_FLAG_NATIVE_BYTE_ORDER, "NATIVE-BYTE-ORDER"); ADD_FLAG(G_ELI_FLAG_ONETIME, "ONETIME"); ADD_FLAG(G_ELI_FLAG_BOOT, "BOOT"); ADD_FLAG(G_ELI_FLAG_WO_DETACH, "W-DETACH"); ADD_FLAG(G_ELI_FLAG_RW_DETACH, "RW-DETACH"); ADD_FLAG(G_ELI_FLAG_AUTH, "AUTH"); ADD_FLAG(G_ELI_FLAG_WOPEN, "W-OPEN"); ADD_FLAG(G_ELI_FLAG_DESTROY, "DESTROY"); ADD_FLAG(G_ELI_FLAG_RO, "READ-ONLY"); #undef ADD_FLAG } sbuf_printf(sb, "\n"); if (!(sc->sc_flags & G_ELI_FLAG_ONETIME)) { sbuf_printf(sb, "%s%u\n", indent, sc->sc_nkey); } sbuf_printf(sb, "%s", indent); switch (sc->sc_crypto) { case G_ELI_CRYPTO_HW: sbuf_printf(sb, "hardware"); break; case G_ELI_CRYPTO_SW: sbuf_printf(sb, "software"); break; default: sbuf_printf(sb, "UNKNOWN"); break; } sbuf_printf(sb, "\n"); if (sc->sc_flags & G_ELI_FLAG_AUTH) { sbuf_printf(sb, "%s%s\n", indent, g_eli_algo2str(sc->sc_aalgo)); } sbuf_printf(sb, "%s%u\n", indent, sc->sc_ekeylen); sbuf_printf(sb, "%s%s\n", indent, g_eli_algo2str(sc->sc_ealgo)); } static void g_eli_shutdown_pre_sync(void *arg, int howto) { struct g_class *mp; struct g_geom *gp, *gp2; struct g_provider *pp; struct g_eli_softc *sc; int error; mp = arg; DROP_GIANT(); g_topology_lock(); LIST_FOREACH_SAFE(gp, &mp->geom, geom, gp2) { sc = gp->softc; if (sc == NULL) continue; pp = LIST_FIRST(&gp->provider); KASSERT(pp != NULL, ("No provider? gp=%p (%s)", gp, gp->name)); if (pp->acr + pp->acw + pp->ace == 0) error = g_eli_destroy(sc, 1); else { sc->sc_flags |= G_ELI_FLAG_RW_DETACH; gp->access = g_eli_access; } } g_topology_unlock(); PICKUP_GIANT(); } static void g_eli_init(struct g_class *mp) { g_eli_pre_sync = EVENTHANDLER_REGISTER(shutdown_pre_sync, g_eli_shutdown_pre_sync, mp, SHUTDOWN_PRI_FIRST); if (g_eli_pre_sync == NULL) G_ELI_DEBUG(0, "Warning! Cannot register shutdown event."); } static void g_eli_fini(struct g_class *mp) { if (g_eli_pre_sync != NULL) EVENTHANDLER_DEREGISTER(shutdown_pre_sync, g_eli_pre_sync); } DECLARE_GEOM_CLASS(g_eli_class, g_eli); MODULE_DEPEND(g_eli, crypto, 1, 1, 1); Index: user/imp/tbemd/sys/geom/geom_io.c =================================================================== --- user/imp/tbemd/sys/geom/geom_io.c (revision 206684) +++ user/imp/tbemd/sys/geom/geom_io.c (revision 206685) @@ -1,797 +1,797 @@ /*- * Copyright (c) 2002 Poul-Henning Kamp * Copyright (c) 2002 Networks Associates Technology, Inc. * All rights reserved. * * This software was developed for the FreeBSD Project by Poul-Henning Kamp * and NAI Labs, the Security Research Division of Network Associates, Inc. * under DARPA/SPAWAR contract N66001-01-C-8035 ("CBOSS"), as part of the * DARPA CHATS research program. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. The names of the authors may not be used to endorse or promote * products derived from this software without specific prior written * permission. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #include __FBSDID("$FreeBSD$"); #include #include #include #include #include #include #include #include #include #include #include #include #include static struct g_bioq g_bio_run_down; static struct g_bioq g_bio_run_up; static struct g_bioq g_bio_run_task; static u_int pace; static uma_zone_t biozone; /* * The head of the list of classifiers used in g_io_request. * Use g_register_classifier() and g_unregister_classifier() * to add/remove entries to the list. * Classifiers are invoked in registration order. */ static TAILQ_HEAD(g_classifier_tailq, g_classifier_hook) g_classifier_tailq = TAILQ_HEAD_INITIALIZER(g_classifier_tailq); #include static void g_bioq_lock(struct g_bioq *bq) { mtx_lock(&bq->bio_queue_lock); } static void g_bioq_unlock(struct g_bioq *bq) { mtx_unlock(&bq->bio_queue_lock); } #if 0 static void g_bioq_destroy(struct g_bioq *bq) { mtx_destroy(&bq->bio_queue_lock); } #endif static void g_bioq_init(struct g_bioq *bq) { TAILQ_INIT(&bq->bio_queue); mtx_init(&bq->bio_queue_lock, "bio queue", NULL, MTX_DEF); } static struct bio * g_bioq_first(struct g_bioq *bq) { struct bio *bp; bp = TAILQ_FIRST(&bq->bio_queue); if (bp != NULL) { KASSERT((bp->bio_flags & BIO_ONQUEUE), ("Bio not on queue bp=%p target %p", bp, bq)); bp->bio_flags &= ~BIO_ONQUEUE; TAILQ_REMOVE(&bq->bio_queue, bp, bio_queue); bq->bio_queue_length--; } return (bp); } struct bio * g_new_bio(void) { struct bio *bp; bp = uma_zalloc(biozone, M_NOWAIT | M_ZERO); #ifdef KTR if ((KTR_COMPILE & KTR_GEOM) && (ktr_mask & KTR_GEOM)) { struct stack st; CTR1(KTR_GEOM, "g_new_bio(): %p", bp); stack_save(&st); CTRSTACK(KTR_GEOM, &st, 3, 0); } #endif return (bp); } struct bio * g_alloc_bio(void) { struct bio *bp; bp = uma_zalloc(biozone, M_WAITOK | M_ZERO); #ifdef KTR if ((KTR_COMPILE & KTR_GEOM) && (ktr_mask & KTR_GEOM)) { struct stack st; CTR1(KTR_GEOM, "g_alloc_bio(): %p", bp); stack_save(&st); CTRSTACK(KTR_GEOM, &st, 3, 0); } #endif return (bp); } void g_destroy_bio(struct bio *bp) { #ifdef KTR if ((KTR_COMPILE & KTR_GEOM) && (ktr_mask & KTR_GEOM)) { struct stack st; CTR1(KTR_GEOM, "g_destroy_bio(): %p", bp); stack_save(&st); CTRSTACK(KTR_GEOM, &st, 3, 0); } #endif uma_zfree(biozone, bp); } struct bio * g_clone_bio(struct bio *bp) { struct bio *bp2; bp2 = uma_zalloc(biozone, M_NOWAIT | M_ZERO); if (bp2 != NULL) { bp2->bio_parent = bp; bp2->bio_cmd = bp->bio_cmd; bp2->bio_length = bp->bio_length; bp2->bio_offset = bp->bio_offset; bp2->bio_data = bp->bio_data; bp2->bio_attribute = bp->bio_attribute; /* Inherit classification info from the parent */ bp2->bio_classifier1 = bp->bio_classifier1; bp2->bio_classifier2 = bp->bio_classifier2; bp->bio_children++; } #ifdef KTR if ((KTR_COMPILE & KTR_GEOM) && (ktr_mask & KTR_GEOM)) { struct stack st; CTR2(KTR_GEOM, "g_clone_bio(%p): %p", bp, bp2); stack_save(&st); CTRSTACK(KTR_GEOM, &st, 3, 0); } #endif return(bp2); } struct bio * g_duplicate_bio(struct bio *bp) { struct bio *bp2; bp2 = uma_zalloc(biozone, M_WAITOK | M_ZERO); bp2->bio_parent = bp; bp2->bio_cmd = bp->bio_cmd; bp2->bio_length = bp->bio_length; bp2->bio_offset = bp->bio_offset; bp2->bio_data = bp->bio_data; bp2->bio_attribute = bp->bio_attribute; bp->bio_children++; #ifdef KTR if ((KTR_COMPILE & KTR_GEOM) && (ktr_mask & KTR_GEOM)) { struct stack st; CTR2(KTR_GEOM, "g_duplicate_bio(%p): %p", bp, bp2); stack_save(&st); CTRSTACK(KTR_GEOM, &st, 3, 0); } #endif return(bp2); } void g_io_init() { g_bioq_init(&g_bio_run_down); g_bioq_init(&g_bio_run_up); g_bioq_init(&g_bio_run_task); biozone = uma_zcreate("g_bio", sizeof (struct bio), NULL, NULL, NULL, NULL, 0, 0); } int g_io_getattr(const char *attr, struct g_consumer *cp, int *len, void *ptr) { struct bio *bp; int error; g_trace(G_T_BIO, "bio_getattr(%s)", attr); bp = g_alloc_bio(); bp->bio_cmd = BIO_GETATTR; bp->bio_done = NULL; bp->bio_attribute = attr; bp->bio_length = *len; bp->bio_data = ptr; g_io_request(bp, cp); error = biowait(bp, "ggetattr"); *len = bp->bio_completed; g_destroy_bio(bp); return (error); } int g_io_flush(struct g_consumer *cp) { struct bio *bp; int error; g_trace(G_T_BIO, "bio_flush(%s)", cp->provider->name); bp = g_alloc_bio(); bp->bio_cmd = BIO_FLUSH; bp->bio_done = NULL; bp->bio_attribute = NULL; bp->bio_offset = cp->provider->mediasize; bp->bio_length = 0; bp->bio_data = NULL; g_io_request(bp, cp); error = biowait(bp, "gflush"); g_destroy_bio(bp); return (error); } static int g_io_check(struct bio *bp) { struct g_consumer *cp; struct g_provider *pp; cp = bp->bio_from; pp = bp->bio_to; /* Fail if access counters dont allow the operation */ switch(bp->bio_cmd) { case BIO_READ: case BIO_GETATTR: if (cp->acr == 0) return (EPERM); break; case BIO_WRITE: case BIO_DELETE: case BIO_FLUSH: if (cp->acw == 0) return (EPERM); break; default: return (EPERM); } /* if provider is marked for error, don't disturb. */ if (pp->error) return (pp->error); switch(bp->bio_cmd) { case BIO_READ: case BIO_WRITE: case BIO_DELETE: - /* Zero sectorsize is a probably lack of media */ - if (pp->sectorsize == 0) + /* Zero sectorsize or mediasize is probably a lack of media. */ + if (pp->sectorsize == 0 || pp->mediasize == 0) return (ENXIO); /* Reject I/O not on sector boundary */ if (bp->bio_offset % pp->sectorsize) return (EINVAL); /* Reject I/O not integral sector long */ if (bp->bio_length % pp->sectorsize) return (EINVAL); /* Reject requests before or past the end of media. */ if (bp->bio_offset < 0) return (EIO); if (bp->bio_offset > pp->mediasize) return (EIO); break; default: break; } return (0); } /* * bio classification support. * * g_register_classifier() and g_unregister_classifier() * are used to add/remove a classifier from the list. * The list is protected using the g_bio_run_down lock, * because the classifiers are called in this path. * * g_io_request() passes bio's that are not already classified * (i.e. those with bio_classifier1 == NULL) to g_run_classifiers(). * Classifiers can store their result in the two fields * bio_classifier1 and bio_classifier2. * A classifier that updates one of the fields should * return a non-zero value. * If no classifier updates the field, g_run_classifiers() sets * bio_classifier1 = BIO_NOTCLASSIFIED to avoid further calls. */ int g_register_classifier(struct g_classifier_hook *hook) { g_bioq_lock(&g_bio_run_down); TAILQ_INSERT_TAIL(&g_classifier_tailq, hook, link); g_bioq_unlock(&g_bio_run_down); return (0); } void g_unregister_classifier(struct g_classifier_hook *hook) { struct g_classifier_hook *entry; g_bioq_lock(&g_bio_run_down); TAILQ_FOREACH(entry, &g_classifier_tailq, link) { if (entry == hook) { TAILQ_REMOVE(&g_classifier_tailq, hook, link); break; } } g_bioq_unlock(&g_bio_run_down); } static void g_run_classifiers(struct bio *bp) { struct g_classifier_hook *hook; int classified = 0; TAILQ_FOREACH(hook, &g_classifier_tailq, link) classified |= hook->func(hook->arg, bp); if (!classified) bp->bio_classifier1 = BIO_NOTCLASSIFIED; } void g_io_request(struct bio *bp, struct g_consumer *cp) { struct g_provider *pp; int first; KASSERT(cp != NULL, ("NULL cp in g_io_request")); KASSERT(bp != NULL, ("NULL bp in g_io_request")); pp = cp->provider; KASSERT(pp != NULL, ("consumer not attached in g_io_request")); #ifdef DIAGNOSTIC KASSERT(bp->bio_driver1 == NULL, ("bio_driver1 used by the consumer (geom %s)", cp->geom->name)); KASSERT(bp->bio_driver2 == NULL, ("bio_driver2 used by the consumer (geom %s)", cp->geom->name)); KASSERT(bp->bio_pflags == 0, ("bio_pflags used by the consumer (geom %s)", cp->geom->name)); /* * Remember consumer's private fields, so we can detect if they were * modified by the provider. */ bp->_bio_caller1 = bp->bio_caller1; bp->_bio_caller2 = bp->bio_caller2; bp->_bio_cflags = bp->bio_cflags; #endif if (bp->bio_cmd & (BIO_READ|BIO_WRITE|BIO_GETATTR)) { KASSERT(bp->bio_data != NULL, ("NULL bp->data in g_io_request(cmd=%hhu)", bp->bio_cmd)); } if (bp->bio_cmd & (BIO_DELETE|BIO_FLUSH)) { KASSERT(bp->bio_data == NULL, ("non-NULL bp->data in g_io_request(cmd=%hhu)", bp->bio_cmd)); } if (bp->bio_cmd & (BIO_READ|BIO_WRITE|BIO_DELETE)) { KASSERT(bp->bio_offset % cp->provider->sectorsize == 0, ("wrong offset %jd for sectorsize %u", bp->bio_offset, cp->provider->sectorsize)); KASSERT(bp->bio_length % cp->provider->sectorsize == 0, ("wrong length %jd for sectorsize %u", bp->bio_length, cp->provider->sectorsize)); } g_trace(G_T_BIO, "bio_request(%p) from %p(%s) to %p(%s) cmd %d", bp, cp, cp->geom->name, pp, pp->name, bp->bio_cmd); bp->bio_from = cp; bp->bio_to = pp; bp->bio_error = 0; bp->bio_completed = 0; KASSERT(!(bp->bio_flags & BIO_ONQUEUE), ("Bio already on queue bp=%p", bp)); bp->bio_flags |= BIO_ONQUEUE; if (g_collectstats) binuptime(&bp->bio_t0); else getbinuptime(&bp->bio_t0); /* * The statistics collection is lockless, as such, but we * can not update one instance of the statistics from more * than one thread at a time, so grab the lock first. * * We also use the lock to protect the list of classifiers. */ g_bioq_lock(&g_bio_run_down); if (!TAILQ_EMPTY(&g_classifier_tailq) && !bp->bio_classifier1) g_run_classifiers(bp); if (g_collectstats & 1) devstat_start_transaction(pp->stat, &bp->bio_t0); if (g_collectstats & 2) devstat_start_transaction(cp->stat, &bp->bio_t0); pp->nstart++; cp->nstart++; first = TAILQ_EMPTY(&g_bio_run_down.bio_queue); TAILQ_INSERT_TAIL(&g_bio_run_down.bio_queue, bp, bio_queue); g_bio_run_down.bio_queue_length++; g_bioq_unlock(&g_bio_run_down); /* Pass it on down. */ if (first) wakeup(&g_wait_down); } void g_io_deliver(struct bio *bp, int error) { struct g_consumer *cp; struct g_provider *pp; int first; KASSERT(bp != NULL, ("NULL bp in g_io_deliver")); pp = bp->bio_to; KASSERT(pp != NULL, ("NULL bio_to in g_io_deliver")); cp = bp->bio_from; if (cp == NULL) { bp->bio_error = error; bp->bio_done(bp); return; } KASSERT(cp != NULL, ("NULL bio_from in g_io_deliver")); KASSERT(cp->geom != NULL, ("NULL bio_from->geom in g_io_deliver")); #ifdef DIAGNOSTIC /* * Some classes - GJournal in particular - can modify bio's * private fields while the bio is in transit; G_GEOM_VOLATILE_BIO * flag means it's an expected behaviour for that particular geom. */ if ((cp->geom->flags & G_GEOM_VOLATILE_BIO) == 0) { KASSERT(bp->bio_caller1 == bp->_bio_caller1, ("bio_caller1 used by the provider %s", pp->name)); KASSERT(bp->bio_caller2 == bp->_bio_caller2, ("bio_caller2 used by the provider %s", pp->name)); KASSERT(bp->bio_cflags == bp->_bio_cflags, ("bio_cflags used by the provider %s", pp->name)); } #endif KASSERT(bp->bio_completed >= 0, ("bio_completed can't be less than 0")); KASSERT(bp->bio_completed <= bp->bio_length, ("bio_completed can't be greater than bio_length")); g_trace(G_T_BIO, "g_io_deliver(%p) from %p(%s) to %p(%s) cmd %d error %d off %jd len %jd", bp, cp, cp->geom->name, pp, pp->name, bp->bio_cmd, error, (intmax_t)bp->bio_offset, (intmax_t)bp->bio_length); KASSERT(!(bp->bio_flags & BIO_ONQUEUE), ("Bio already on queue bp=%p", bp)); /* * XXX: next two doesn't belong here */ bp->bio_bcount = bp->bio_length; bp->bio_resid = bp->bio_bcount - bp->bio_completed; /* * The statistics collection is lockless, as such, but we * can not update one instance of the statistics from more * than one thread at a time, so grab the lock first. */ g_bioq_lock(&g_bio_run_up); if (g_collectstats & 1) devstat_end_transaction_bio(pp->stat, bp); if (g_collectstats & 2) devstat_end_transaction_bio(cp->stat, bp); cp->nend++; pp->nend++; if (error != ENOMEM) { bp->bio_error = error; first = TAILQ_EMPTY(&g_bio_run_up.bio_queue); TAILQ_INSERT_TAIL(&g_bio_run_up.bio_queue, bp, bio_queue); bp->bio_flags |= BIO_ONQUEUE; g_bio_run_up.bio_queue_length++; g_bioq_unlock(&g_bio_run_up); if (first) wakeup(&g_wait_up); return; } g_bioq_unlock(&g_bio_run_up); if (bootverbose) printf("ENOMEM %p on %p(%s)\n", bp, pp, pp->name); bp->bio_children = 0; bp->bio_inbed = 0; g_io_request(bp, cp); pace++; return; } void g_io_schedule_down(struct thread *tp __unused) { struct bio *bp; off_t excess; int error; for(;;) { g_bioq_lock(&g_bio_run_down); bp = g_bioq_first(&g_bio_run_down); if (bp == NULL) { CTR0(KTR_GEOM, "g_down going to sleep"); msleep(&g_wait_down, &g_bio_run_down.bio_queue_lock, PRIBIO | PDROP, "-", 0); continue; } CTR0(KTR_GEOM, "g_down has work to do"); g_bioq_unlock(&g_bio_run_down); if (pace > 0) { CTR1(KTR_GEOM, "g_down pacing self (pace %d)", pace); pause("g_down", hz/10); pace--; } error = g_io_check(bp); if (error) { CTR3(KTR_GEOM, "g_down g_io_check on bp %p provider " "%s returned %d", bp, bp->bio_to->name, error); g_io_deliver(bp, error); continue; } CTR2(KTR_GEOM, "g_down processing bp %p provider %s", bp, bp->bio_to->name); switch (bp->bio_cmd) { case BIO_READ: case BIO_WRITE: case BIO_DELETE: /* Truncate requests to the end of providers media. */ /* * XXX: What if we truncate because of offset being * bad, not length? */ excess = bp->bio_offset + bp->bio_length; if (excess > bp->bio_to->mediasize) { excess -= bp->bio_to->mediasize; bp->bio_length -= excess; if (excess > 0) CTR3(KTR_GEOM, "g_down truncated bio " "%p provider %s by %d", bp, bp->bio_to->name, excess); } /* Deliver zero length transfers right here. */ if (bp->bio_length == 0) { g_io_deliver(bp, 0); CTR2(KTR_GEOM, "g_down terminated 0-length " "bp %p provider %s", bp, bp->bio_to->name); continue; } break; default: break; } THREAD_NO_SLEEPING(); CTR4(KTR_GEOM, "g_down starting bp %p provider %s off %ld " "len %ld", bp, bp->bio_to->name, bp->bio_offset, bp->bio_length); bp->bio_to->geom->start(bp); THREAD_SLEEPING_OK(); } } void bio_taskqueue(struct bio *bp, bio_task_t *func, void *arg) { bp->bio_task = func; bp->bio_task_arg = arg; /* * The taskqueue is actually just a second queue off the "up" * queue, so we use the same lock. */ g_bioq_lock(&g_bio_run_up); KASSERT(!(bp->bio_flags & BIO_ONQUEUE), ("Bio already on queue bp=%p target taskq", bp)); bp->bio_flags |= BIO_ONQUEUE; TAILQ_INSERT_TAIL(&g_bio_run_task.bio_queue, bp, bio_queue); g_bio_run_task.bio_queue_length++; wakeup(&g_wait_up); g_bioq_unlock(&g_bio_run_up); } void g_io_schedule_up(struct thread *tp __unused) { struct bio *bp; for(;;) { g_bioq_lock(&g_bio_run_up); bp = g_bioq_first(&g_bio_run_task); if (bp != NULL) { g_bioq_unlock(&g_bio_run_up); THREAD_NO_SLEEPING(); CTR1(KTR_GEOM, "g_up processing task bp %p", bp); bp->bio_task(bp->bio_task_arg); THREAD_SLEEPING_OK(); continue; } bp = g_bioq_first(&g_bio_run_up); if (bp != NULL) { g_bioq_unlock(&g_bio_run_up); THREAD_NO_SLEEPING(); CTR4(KTR_GEOM, "g_up biodone bp %p provider %s off " "%jd len %ld", bp, bp->bio_to->name, bp->bio_offset, bp->bio_length); biodone(bp); THREAD_SLEEPING_OK(); continue; } CTR0(KTR_GEOM, "g_up going to sleep"); msleep(&g_wait_up, &g_bio_run_up.bio_queue_lock, PRIBIO | PDROP, "-", 0); } } void * g_read_data(struct g_consumer *cp, off_t offset, off_t length, int *error) { struct bio *bp; void *ptr; int errorc; KASSERT(length > 0 && length >= cp->provider->sectorsize && length <= MAXPHYS, ("g_read_data(): invalid length %jd", (intmax_t)length)); bp = g_alloc_bio(); bp->bio_cmd = BIO_READ; bp->bio_done = NULL; bp->bio_offset = offset; bp->bio_length = length; ptr = g_malloc(length, M_WAITOK); bp->bio_data = ptr; g_io_request(bp, cp); errorc = biowait(bp, "gread"); if (error != NULL) *error = errorc; g_destroy_bio(bp); if (errorc) { g_free(ptr); ptr = NULL; } return (ptr); } int g_write_data(struct g_consumer *cp, off_t offset, void *ptr, off_t length) { struct bio *bp; int error; KASSERT(length > 0 && length >= cp->provider->sectorsize && length <= MAXPHYS, ("g_write_data(): invalid length %jd", (intmax_t)length)); bp = g_alloc_bio(); bp->bio_cmd = BIO_WRITE; bp->bio_done = NULL; bp->bio_offset = offset; bp->bio_length = length; bp->bio_data = ptr; g_io_request(bp, cp); error = biowait(bp, "gwrite"); g_destroy_bio(bp); return (error); } int g_delete_data(struct g_consumer *cp, off_t offset, off_t length) { struct bio *bp; int error; KASSERT(length > 0 && length >= cp->provider->sectorsize, ("g_delete_data(): invalid length %jd", (intmax_t)length)); bp = g_alloc_bio(); bp->bio_cmd = BIO_DELETE; bp->bio_done = NULL; bp->bio_offset = offset; bp->bio_length = length; bp->bio_data = NULL; g_io_request(bp, cp); error = biowait(bp, "gdelete"); g_destroy_bio(bp); return (error); } void g_print_bio(struct bio *bp) { const char *pname, *cmd = NULL; if (bp->bio_to != NULL) pname = bp->bio_to->name; else pname = "[unknown]"; switch (bp->bio_cmd) { case BIO_GETATTR: cmd = "GETATTR"; printf("%s[%s(attr=%s)]", pname, cmd, bp->bio_attribute); return; case BIO_FLUSH: cmd = "FLUSH"; printf("%s[%s]", pname, cmd); return; case BIO_READ: cmd = "READ"; case BIO_WRITE: if (cmd == NULL) cmd = "WRITE"; case BIO_DELETE: if (cmd == NULL) cmd = "DELETE"; printf("%s[%s(offset=%jd, length=%jd)]", pname, cmd, (intmax_t)bp->bio_offset, (intmax_t)bp->bio_length); return; default: cmd = "UNKNOWN"; printf("%s[%s()]", pname, cmd); return; } /* NOTREACHED */ } Index: user/imp/tbemd/sys/kern/kern_ktr.c =================================================================== --- user/imp/tbemd/sys/kern/kern_ktr.c (revision 206684) +++ user/imp/tbemd/sys/kern/kern_ktr.c (revision 206685) @@ -1,351 +1,357 @@ /*- * Copyright (c) 2000 John Baldwin * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of the author nor the names of any co-contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ /* * This module holds the global variables used by KTR and the ktr_tracepoint() * function that does the actual tracing. */ #include __FBSDID("$FreeBSD$"); #include "opt_ddb.h" #include "opt_ktr.h" #include "opt_alq.h" #include #include #include #include #include #include #include #include #include #include #include #ifdef __sparc64__ #include #endif #ifdef DDB #include #include #endif #ifndef KTR_ENTRIES #define KTR_ENTRIES 1024 #endif #ifndef KTR_MASK #define KTR_MASK (KTR_GEN) #endif #ifndef KTR_CPUMASK #define KTR_CPUMASK (~0) #endif #ifndef KTR_TIME #define KTR_TIME get_cyclecount() #endif #ifndef KTR_CPU #define KTR_CPU PCPU_GET(cpuid) #endif SYSCTL_NODE(_debug, OID_AUTO, ktr, CTLFLAG_RD, 0, "KTR options"); int ktr_cpumask = KTR_CPUMASK; TUNABLE_INT("debug.ktr.cpumask", &ktr_cpumask); SYSCTL_INT(_debug_ktr, OID_AUTO, cpumask, CTLFLAG_RW, &ktr_cpumask, 0, "Bitmask of CPUs on which KTR logging is enabled"); int ktr_mask = KTR_MASK; TUNABLE_INT("debug.ktr.mask", &ktr_mask); SYSCTL_INT(_debug_ktr, OID_AUTO, mask, CTLFLAG_RW, &ktr_mask, 0, "Bitmask of KTR event classes for which logging is enabled"); int ktr_compile = KTR_COMPILE; SYSCTL_INT(_debug_ktr, OID_AUTO, compile, CTLFLAG_RD, &ktr_compile, 0, "Bitmask of KTR event classes compiled into the kernel"); int ktr_entries = KTR_ENTRIES; SYSCTL_INT(_debug_ktr, OID_AUTO, entries, CTLFLAG_RD, &ktr_entries, 0, "Number of entries in the KTR buffer"); int ktr_version = KTR_VERSION; SYSCTL_INT(_debug_ktr, OID_AUTO, version, CTLFLAG_RD, &ktr_version, 0, "Version of the KTR interface"); volatile int ktr_idx = 0; struct ktr_entry ktr_buf[KTR_ENTRIES]; static int sysctl_debug_ktr_clear(SYSCTL_HANDLER_ARGS) { int clear, error; clear = 0; error = sysctl_handle_int(oidp, &clear, 0, req); if (error || !req->newptr) return (error); if (clear) { bzero(ktr_buf, sizeof(ktr_buf)); ktr_idx = 0; } return (error); } SYSCTL_PROC(_debug_ktr, OID_AUTO, clear, CTLTYPE_INT|CTLFLAG_RW, 0, 0, sysctl_debug_ktr_clear, "I", "Clear KTR Buffer"); #ifdef KTR_VERBOSE int ktr_verbose = KTR_VERBOSE; TUNABLE_INT("debug.ktr.verbose", &ktr_verbose); SYSCTL_INT(_debug_ktr, OID_AUTO, verbose, CTLFLAG_RW, &ktr_verbose, 0, ""); #endif #ifdef KTR_ALQ struct alq *ktr_alq; char ktr_alq_file[MAXPATHLEN] = "/tmp/ktr.out"; int ktr_alq_cnt = 0; int ktr_alq_depth = KTR_ENTRIES; int ktr_alq_enabled = 0; int ktr_alq_failed = 0; int ktr_alq_max = 0; SYSCTL_INT(_debug_ktr, OID_AUTO, alq_max, CTLFLAG_RW, &ktr_alq_max, 0, "Maximum number of entries to write"); SYSCTL_INT(_debug_ktr, OID_AUTO, alq_cnt, CTLFLAG_RD, &ktr_alq_cnt, 0, "Current number of written entries"); SYSCTL_INT(_debug_ktr, OID_AUTO, alq_failed, CTLFLAG_RD, &ktr_alq_failed, 0, "Number of times we overran the buffer"); SYSCTL_INT(_debug_ktr, OID_AUTO, alq_depth, CTLFLAG_RW, &ktr_alq_depth, 0, "Number of items in the write buffer"); SYSCTL_STRING(_debug_ktr, OID_AUTO, alq_file, CTLFLAG_RW, ktr_alq_file, sizeof(ktr_alq_file), "KTR logging file"); static int sysctl_debug_ktr_alq_enable(SYSCTL_HANDLER_ARGS) { int error; int enable; enable = ktr_alq_enabled; error = sysctl_handle_int(oidp, &enable, 0, req); if (error || !req->newptr) return (error); if (enable) { if (ktr_alq_enabled) return (0); error = alq_open(&ktr_alq, (const char *)ktr_alq_file, req->td->td_ucred, ALQ_DEFAULT_CMODE, sizeof(struct ktr_entry), ktr_alq_depth); if (error == 0) { ktr_alq_cnt = 0; ktr_alq_failed = 0; ktr_alq_enabled = 1; } } else { if (ktr_alq_enabled == 0) return (0); ktr_alq_enabled = 0; alq_close(ktr_alq); ktr_alq = NULL; } return (error); } SYSCTL_PROC(_debug_ktr, OID_AUTO, alq_enable, CTLTYPE_INT|CTLFLAG_RW, 0, 0, sysctl_debug_ktr_alq_enable, "I", "Enable KTR logging"); #endif void ktr_tracepoint(u_int mask, const char *file, int line, const char *format, u_long arg1, u_long arg2, u_long arg3, u_long arg4, u_long arg5, u_long arg6) { struct ktr_entry *entry; #ifdef KTR_ALQ struct ale *ale = NULL; -#else - int newindex, saveindex; #endif + int newindex, saveindex; #if defined(KTR_VERBOSE) || defined(KTR_ALQ) struct thread *td; #endif int cpu; if (panicstr) return; if ((ktr_mask & mask) == 0) return; cpu = KTR_CPU; if (((1 << cpu) & ktr_cpumask) == 0) return; #if defined(KTR_VERBOSE) || defined(KTR_ALQ) td = curthread; if (td->td_pflags & TDP_INKTR) return; td->td_pflags |= TDP_INKTR; #endif #ifdef KTR_ALQ - if (ktr_alq_enabled && - td->td_critnest == 0 && - (td->td_flags & TDF_IDLETD) == 0 && - td != ald_thread) { - if (ktr_alq_max && ktr_alq_cnt > ktr_alq_max) + if (ktr_alq_enabled) { + if (td->td_critnest == 0 && + (td->td_flags & TDF_IDLETD) == 0 && + td != ald_thread) { + if (ktr_alq_max && ktr_alq_cnt > ktr_alq_max) + goto done; + if ((ale = alq_get(ktr_alq, ALQ_NOWAIT)) == NULL) { + ktr_alq_failed++; + goto done; + } + ktr_alq_cnt++; + entry = (struct ktr_entry *)ale->ae_data; + } else { goto done; - if ((ale = alq_get(ktr_alq, ALQ_NOWAIT)) == NULL) { - ktr_alq_failed++; - goto done; } - ktr_alq_cnt++; - entry = (struct ktr_entry *)ale->ae_data; } else - goto done; -#else - do { - saveindex = ktr_idx; - newindex = (saveindex + 1) & (KTR_ENTRIES - 1); - } while (atomic_cmpset_rel_int(&ktr_idx, saveindex, newindex) == 0); - entry = &ktr_buf[saveindex]; #endif + { + do { + saveindex = ktr_idx; + newindex = (saveindex + 1) & (KTR_ENTRIES - 1); + } while (atomic_cmpset_rel_int(&ktr_idx, saveindex, newindex) == 0); + entry = &ktr_buf[saveindex]; + } entry->ktr_timestamp = KTR_TIME; entry->ktr_cpu = cpu; entry->ktr_thread = curthread; if (file != NULL) while (strncmp(file, "../", 3) == 0) file += 3; entry->ktr_file = file; entry->ktr_line = line; #ifdef KTR_VERBOSE if (ktr_verbose) { #ifdef SMP printf("cpu%d ", cpu); #endif if (ktr_verbose > 1) { printf("%s.%d\t", entry->ktr_file, entry->ktr_line); } printf(format, arg1, arg2, arg3, arg4, arg5, arg6); printf("\n"); } #endif entry->ktr_desc = format; entry->ktr_parms[0] = arg1; entry->ktr_parms[1] = arg2; entry->ktr_parms[2] = arg3; entry->ktr_parms[3] = arg4; entry->ktr_parms[4] = arg5; entry->ktr_parms[5] = arg6; #ifdef KTR_ALQ - if (ale) + if (ktr_alq_enabled && ale) alq_post(ktr_alq, ale); done: #endif #if defined(KTR_VERBOSE) || defined(KTR_ALQ) td->td_pflags &= ~TDP_INKTR; #endif } #ifdef DDB struct tstate { int cur; int first; }; static struct tstate tstate; static int db_ktr_verbose; static int db_mach_vtrace(void); DB_SHOW_COMMAND(ktr, db_ktr_all) { tstate.cur = (ktr_idx - 1) & (KTR_ENTRIES - 1); tstate.first = -1; - db_ktr_verbose = index(modif, 'v') != NULL; + db_ktr_verbose = 0; + db_ktr_verbose |= (index(modif, 'v') != NULL) ? 2 : 0; + db_ktr_verbose |= (index(modif, 'V') != NULL) ? 1 : 0; /* just timestap please */ if (index(modif, 'a') != NULL) { db_disable_pager(); while (cncheckc() != -1) if (db_mach_vtrace() == 0) break; } else { while (!db_pager_quit) if (db_mach_vtrace() == 0) break; } } static int db_mach_vtrace(void) { struct ktr_entry *kp; if (tstate.cur == tstate.first) { db_printf("--- End of trace buffer ---\n"); return (0); } kp = &ktr_buf[tstate.cur]; /* Skip over unused entries. */ if (kp->ktr_desc == NULL) { db_printf("--- End of trace buffer ---\n"); return (0); } db_printf("%d (%p", tstate.cur, kp->ktr_thread); #ifdef SMP db_printf(":cpu%d", kp->ktr_cpu); #endif db_printf(")"); - if (db_ktr_verbose) { - db_printf(" %10.10lld %s.%d", (long long)kp->ktr_timestamp, - kp->ktr_file, kp->ktr_line); + if (db_ktr_verbose >= 1) { + db_printf(" %10.10lld", (long long)kp->ktr_timestamp); + } + if (db_ktr_verbose >= 2) { + db_printf(" %s.%d", kp->ktr_file, kp->ktr_line); } db_printf(": "); db_printf(kp->ktr_desc, kp->ktr_parms[0], kp->ktr_parms[1], kp->ktr_parms[2], kp->ktr_parms[3], kp->ktr_parms[4], kp->ktr_parms[5]); db_printf("\n"); if (tstate.first == -1) tstate.first = tstate.cur; if (--tstate.cur < 0) tstate.cur = KTR_ENTRIES - 1; return (1); } #endif /* DDB */ Index: user/imp/tbemd/sys/kern/vfs_cache.c =================================================================== --- user/imp/tbemd/sys/kern/vfs_cache.c (revision 206684) +++ user/imp/tbemd/sys/kern/vfs_cache.c (revision 206685) @@ -1,1226 +1,1226 @@ /*- * Copyright (c) 1989, 1993, 1995 * The Regents of the University of California. All rights reserved. * * This code is derived from software contributed to Berkeley by * Poul-Henning Kamp of the FreeBSD Project. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 4. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * @(#)vfs_cache.c 8.5 (Berkeley) 3/22/95 */ #include __FBSDID("$FreeBSD$"); #include "opt_kdtrace.h" #include "opt_ktrace.h" #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #ifdef KTRACE #include #endif #include SDT_PROVIDER_DECLARE(vfs); SDT_PROBE_DEFINE3(vfs, namecache, enter, done, "struct vnode *", "char *", "struct vnode *"); SDT_PROBE_DEFINE2(vfs, namecache, enter_negative, done, "struct vnode *", "char *"); SDT_PROBE_DEFINE1(vfs, namecache, fullpath, entry, "struct vnode *"); SDT_PROBE_DEFINE3(vfs, namecache, fullpath, hit, "struct vnode *", "struct char *", "struct vnode *"); SDT_PROBE_DEFINE1(vfs, namecache, fullpath, miss, "struct vnode *"); SDT_PROBE_DEFINE3(vfs, namecache, fullpath, return, "int", "struct vnode *", "struct char *"); SDT_PROBE_DEFINE3(vfs, namecache, lookup, hit, "struct vnode *", "char *", "struct vnode *"); SDT_PROBE_DEFINE2(vfs, namecache, lookup, hit_negative, "struct vnode *", "char *"); SDT_PROBE_DEFINE2(vfs, namecache, lookup, miss, "struct vnode *", "char *"); SDT_PROBE_DEFINE1(vfs, namecache, purge, done, "struct vnode *"); SDT_PROBE_DEFINE1(vfs, namecache, purge_negative, done, "struct vnode *"); SDT_PROBE_DEFINE1(vfs, namecache, purgevfs, done, "struct mount *"); SDT_PROBE_DEFINE3(vfs, namecache, zap, done, "struct vnode *", "char *", "struct vnode *"); SDT_PROBE_DEFINE2(vfs, namecache, zap_negative, done, "struct vnode *", "char *"); /* * This structure describes the elements in the cache of recent * names looked up by namei. */ struct namecache { LIST_ENTRY(namecache) nc_hash; /* hash chain */ LIST_ENTRY(namecache) nc_src; /* source vnode list */ TAILQ_ENTRY(namecache) nc_dst; /* destination vnode list */ struct vnode *nc_dvp; /* vnode of parent of name */ struct vnode *nc_vp; /* vnode the name refers to */ u_char nc_flag; /* flag bits */ u_char nc_nlen; /* length of name */ char nc_name[0]; /* segment name + nul */ }; /* * Name caching works as follows: * * Names found by directory scans are retained in a cache * for future reference. It is managed LRU, so frequently * used names will hang around. Cache is indexed by hash value * obtained from (vp, name) where vp refers to the directory * containing name. * * If it is a "negative" entry, (i.e. for a name that is known NOT to * exist) the vnode pointer will be NULL. * * Upon reaching the last segment of a path, if the reference * is for DELETE, or NOCACHE is set (rewrite), and the * name is located in the cache, it will be dropped. */ /* * Structures associated with name cacheing. */ #define NCHHASH(hash) \ (&nchashtbl[(hash) & nchash]) static LIST_HEAD(nchashhead, namecache) *nchashtbl; /* Hash Table */ static TAILQ_HEAD(, namecache) ncneg; /* Hash Table */ static u_long nchash; /* size of hash table */ SYSCTL_ULONG(_debug, OID_AUTO, nchash, CTLFLAG_RD, &nchash, 0, ""); static u_long ncnegfactor = 16; /* ratio of negative entries */ SYSCTL_ULONG(_debug, OID_AUTO, ncnegfactor, CTLFLAG_RW, &ncnegfactor, 0, ""); static u_long numneg; /* number of cache entries allocated */ SYSCTL_ULONG(_debug, OID_AUTO, numneg, CTLFLAG_RD, &numneg, 0, ""); static u_long numcache; /* number of cache entries allocated */ SYSCTL_ULONG(_debug, OID_AUTO, numcache, CTLFLAG_RD, &numcache, 0, ""); static u_long numcachehv; /* number of cache entries with vnodes held */ SYSCTL_ULONG(_debug, OID_AUTO, numcachehv, CTLFLAG_RD, &numcachehv, 0, ""); #if 0 static u_long numcachepl; /* number of cache purge for leaf entries */ SYSCTL_ULONG(_debug, OID_AUTO, numcachepl, CTLFLAG_RD, &numcachepl, 0, ""); #endif struct nchstats nchstats; /* cache effectiveness statistics */ static struct rwlock cache_lock; RW_SYSINIT(vfscache, &cache_lock, "Name Cache"); #define CACHE_UPGRADE_LOCK() rw_try_upgrade(&cache_lock) #define CACHE_RLOCK() rw_rlock(&cache_lock) #define CACHE_RUNLOCK() rw_runlock(&cache_lock) #define CACHE_WLOCK() rw_wlock(&cache_lock) #define CACHE_WUNLOCK() rw_wunlock(&cache_lock) /* * UMA zones for the VFS cache. * * The small cache is used for entries with short names, which are the * most common. The large cache is used for entries which are too big to * fit in the small cache. */ static uma_zone_t cache_zone_small; static uma_zone_t cache_zone_large; #define CACHE_PATH_CUTOFF 35 #define CACHE_ZONE_SMALL (sizeof(struct namecache) + CACHE_PATH_CUTOFF \ + 1) #define CACHE_ZONE_LARGE (sizeof(struct namecache) + NAME_MAX + 1) #define cache_alloc(len) uma_zalloc(((len) <= CACHE_PATH_CUTOFF) ? \ cache_zone_small : cache_zone_large, M_WAITOK) #define cache_free(ncp) do { \ if (ncp != NULL) \ uma_zfree(((ncp)->nc_nlen <= CACHE_PATH_CUTOFF) ? \ cache_zone_small : cache_zone_large, (ncp)); \ } while (0) static int doingcache = 1; /* 1 => enable the cache */ SYSCTL_INT(_debug, OID_AUTO, vfscache, CTLFLAG_RW, &doingcache, 0, ""); /* Export size information to userland */ SYSCTL_INT(_debug_sizeof, OID_AUTO, namecache, CTLFLAG_RD, 0, sizeof(struct namecache), ""); /* * The new name cache statistics */ static SYSCTL_NODE(_vfs, OID_AUTO, cache, CTLFLAG_RW, 0, "Name cache statistics"); #define STATNODE(mode, name, var) \ SYSCTL_ULONG(_vfs_cache, OID_AUTO, name, mode, var, 0, ""); STATNODE(CTLFLAG_RD, numneg, &numneg); STATNODE(CTLFLAG_RD, numcache, &numcache); static u_long numcalls; STATNODE(CTLFLAG_RD, numcalls, &numcalls); static u_long dothits; STATNODE(CTLFLAG_RD, dothits, &dothits); static u_long dotdothits; STATNODE(CTLFLAG_RD, dotdothits, &dotdothits); static u_long numchecks; STATNODE(CTLFLAG_RD, numchecks, &numchecks); static u_long nummiss; STATNODE(CTLFLAG_RD, nummiss, &nummiss); static u_long nummisszap; STATNODE(CTLFLAG_RD, nummisszap, &nummisszap); static u_long numposzaps; STATNODE(CTLFLAG_RD, numposzaps, &numposzaps); static u_long numposhits; STATNODE(CTLFLAG_RD, numposhits, &numposhits); static u_long numnegzaps; STATNODE(CTLFLAG_RD, numnegzaps, &numnegzaps); static u_long numneghits; STATNODE(CTLFLAG_RD, numneghits, &numneghits); static u_long numupgrades; STATNODE(CTLFLAG_RD, numupgrades, &numupgrades); SYSCTL_OPAQUE(_vfs_cache, OID_AUTO, nchstats, CTLFLAG_RD | CTLFLAG_MPSAFE, &nchstats, sizeof(nchstats), "LU", "VFS cache effectiveness statistics"); static void cache_zap(struct namecache *ncp); static int vn_vptocnp_locked(struct vnode **vp, struct ucred *cred, char *buf, u_int *buflen); static int vn_fullpath1(struct thread *td, struct vnode *vp, struct vnode *rdir, char *buf, char **retbuf, u_int buflen); static MALLOC_DEFINE(M_VFSCACHE, "vfscache", "VFS name cache entries"); /* * Flags in namecache.nc_flag */ #define NCF_WHITE 0x01 #define NCF_ISDOTDOT 0x02 #ifdef DIAGNOSTIC /* * Grab an atomic snapshot of the name cache hash chain lengths */ SYSCTL_NODE(_debug, OID_AUTO, hashstat, CTLFLAG_RW, NULL, "hash table stats"); static int sysctl_debug_hashstat_rawnchash(SYSCTL_HANDLER_ARGS) { int error; struct nchashhead *ncpp; struct namecache *ncp; int n_nchash; int count; n_nchash = nchash + 1; /* nchash is max index, not count */ if (!req->oldptr) return SYSCTL_OUT(req, 0, n_nchash * sizeof(int)); /* Scan hash tables for applicable entries */ for (ncpp = nchashtbl; n_nchash > 0; n_nchash--, ncpp++) { CACHE_RLOCK(); count = 0; LIST_FOREACH(ncp, ncpp, nc_hash) { count++; } CACHE_RUNLOCK(); error = SYSCTL_OUT(req, &count, sizeof(count)); if (error) return (error); } return (0); } SYSCTL_PROC(_debug_hashstat, OID_AUTO, rawnchash, CTLTYPE_INT|CTLFLAG_RD| CTLFLAG_MPSAFE, 0, 0, sysctl_debug_hashstat_rawnchash, "S,int", "nchash chain lengths"); static int sysctl_debug_hashstat_nchash(SYSCTL_HANDLER_ARGS) { int error; struct nchashhead *ncpp; struct namecache *ncp; int n_nchash; int count, maxlength, used, pct; if (!req->oldptr) return SYSCTL_OUT(req, 0, 4 * sizeof(int)); n_nchash = nchash + 1; /* nchash is max index, not count */ used = 0; maxlength = 0; /* Scan hash tables for applicable entries */ for (ncpp = nchashtbl; n_nchash > 0; n_nchash--, ncpp++) { count = 0; CACHE_RLOCK(); LIST_FOREACH(ncp, ncpp, nc_hash) { count++; } CACHE_RUNLOCK(); if (count) used++; if (maxlength < count) maxlength = count; } n_nchash = nchash + 1; pct = (used * 100 * 100) / n_nchash; error = SYSCTL_OUT(req, &n_nchash, sizeof(n_nchash)); if (error) return (error); error = SYSCTL_OUT(req, &used, sizeof(used)); if (error) return (error); error = SYSCTL_OUT(req, &maxlength, sizeof(maxlength)); if (error) return (error); error = SYSCTL_OUT(req, &pct, sizeof(pct)); if (error) return (error); return (0); } SYSCTL_PROC(_debug_hashstat, OID_AUTO, nchash, CTLTYPE_INT|CTLFLAG_RD| CTLFLAG_MPSAFE, 0, 0, sysctl_debug_hashstat_nchash, "I", "nchash chain lengths"); #endif /* * cache_zap(): * * Removes a namecache entry from cache, whether it contains an actual * pointer to a vnode or if it is just a negative cache entry. */ static void cache_zap(ncp) struct namecache *ncp; { struct vnode *vp; rw_assert(&cache_lock, RA_WLOCKED); CTR2(KTR_VFS, "cache_zap(%p) vp %p", ncp, ncp->nc_vp); #ifdef KDTRACE_HOOKS if (ncp->nc_vp != NULL) { SDT_PROBE(vfs, namecache, zap, done, ncp->nc_dvp, ncp->nc_name, ncp->nc_vp, 0, 0); } else { SDT_PROBE(vfs, namecache, zap_negative, done, ncp->nc_dvp, ncp->nc_name, 0, 0, 0); } #endif vp = NULL; LIST_REMOVE(ncp, nc_hash); if (ncp->nc_flag & NCF_ISDOTDOT) { if (ncp == ncp->nc_dvp->v_cache_dd) ncp->nc_dvp->v_cache_dd = NULL; } else { LIST_REMOVE(ncp, nc_src); if (LIST_EMPTY(&ncp->nc_dvp->v_cache_src)) { vp = ncp->nc_dvp; numcachehv--; } } if (ncp->nc_vp) { TAILQ_REMOVE(&ncp->nc_vp->v_cache_dst, ncp, nc_dst); if (ncp == ncp->nc_vp->v_cache_dd) ncp->nc_vp->v_cache_dd = NULL; } else { TAILQ_REMOVE(&ncneg, ncp, nc_dst); numneg--; } numcache--; cache_free(ncp); if (vp) vdrop(vp); } /* * Lookup an entry in the cache * * Lookup is called with dvp pointing to the directory to search, * cnp pointing to the name of the entry being sought. If the lookup * succeeds, the vnode is returned in *vpp, and a status of -1 is * returned. If the lookup determines that the name does not exist * (negative cacheing), a status of ENOENT is returned. If the lookup * fails, a status of zero is returned. If the directory vnode is * recycled out from under us due to a forced unmount, a status of * ENOENT is returned. * * vpp is locked and ref'd on return. If we're looking up DOTDOT, dvp is * unlocked. If we're looking up . an extra ref is taken, but the lock is * not recursively acquired. */ int cache_lookup(dvp, vpp, cnp) struct vnode *dvp; struct vnode **vpp; struct componentname *cnp; { struct namecache *ncp; u_int32_t hash; int error, ltype, wlocked; if (!doingcache) { cnp->cn_flags &= ~MAKEENTRY; return (0); } retry: CACHE_RLOCK(); wlocked = 0; numcalls++; error = 0; retry_wlocked: if (cnp->cn_nameptr[0] == '.') { if (cnp->cn_namelen == 1) { *vpp = dvp; CTR2(KTR_VFS, "cache_lookup(%p, %s) found via .", dvp, cnp->cn_nameptr); dothits++; SDT_PROBE(vfs, namecache, lookup, hit, dvp, ".", *vpp, 0, 0); goto success; } if (cnp->cn_namelen == 2 && cnp->cn_nameptr[1] == '.') { dotdothits++; if (dvp->v_cache_dd == NULL) { SDT_PROBE(vfs, namecache, lookup, miss, dvp, "..", NULL, 0, 0); goto unlock; } if ((cnp->cn_flags & MAKEENTRY) == 0) { if (!wlocked && !CACHE_UPGRADE_LOCK()) goto wlock; if (dvp->v_cache_dd->nc_flag & NCF_ISDOTDOT) cache_zap(dvp->v_cache_dd); dvp->v_cache_dd = NULL; CACHE_WUNLOCK(); return (0); } if (dvp->v_cache_dd->nc_flag & NCF_ISDOTDOT) *vpp = dvp->v_cache_dd->nc_vp; else *vpp = dvp->v_cache_dd->nc_dvp; /* Return failure if negative entry was found. */ if (*vpp == NULL) { ncp = dvp->v_cache_dd; goto negative_success; } CTR3(KTR_VFS, "cache_lookup(%p, %s) found %p via ..", dvp, cnp->cn_nameptr, *vpp); SDT_PROBE(vfs, namecache, lookup, hit, dvp, "..", *vpp, 0, 0); goto success; } } hash = fnv_32_buf(cnp->cn_nameptr, cnp->cn_namelen, FNV1_32_INIT); hash = fnv_32_buf(&dvp, sizeof(dvp), hash); LIST_FOREACH(ncp, (NCHHASH(hash)), nc_hash) { numchecks++; if (ncp->nc_dvp == dvp && ncp->nc_nlen == cnp->cn_namelen && !bcmp(ncp->nc_name, cnp->cn_nameptr, ncp->nc_nlen)) break; } /* We failed to find an entry */ if (ncp == NULL) { SDT_PROBE(vfs, namecache, lookup, miss, dvp, cnp->cn_nameptr, NULL, 0, 0); if ((cnp->cn_flags & MAKEENTRY) == 0) { nummisszap++; } else { nummiss++; } nchstats.ncs_miss++; goto unlock; } /* We don't want to have an entry, so dump it */ if ((cnp->cn_flags & MAKEENTRY) == 0) { numposzaps++; nchstats.ncs_badhits++; if (!wlocked && !CACHE_UPGRADE_LOCK()) goto wlock; cache_zap(ncp); CACHE_WUNLOCK(); return (0); } /* We found a "positive" match, return the vnode */ if (ncp->nc_vp) { numposhits++; nchstats.ncs_goodhits++; *vpp = ncp->nc_vp; CTR4(KTR_VFS, "cache_lookup(%p, %s) found %p via ncp %p", dvp, cnp->cn_nameptr, *vpp, ncp); SDT_PROBE(vfs, namecache, lookup, hit, dvp, ncp->nc_name, *vpp, 0, 0); goto success; } negative_success: /* We found a negative match, and want to create it, so purge */ if (cnp->cn_nameiop == CREATE) { numnegzaps++; nchstats.ncs_badhits++; if (!wlocked && !CACHE_UPGRADE_LOCK()) goto wlock; cache_zap(ncp); CACHE_WUNLOCK(); return (0); } if (!wlocked && !CACHE_UPGRADE_LOCK()) goto wlock; numneghits++; /* * We found a "negative" match, so we shift it to the end of * the "negative" cache entries queue to satisfy LRU. Also, * check to see if the entry is a whiteout; indicate this to * the componentname, if so. */ TAILQ_REMOVE(&ncneg, ncp, nc_dst); TAILQ_INSERT_TAIL(&ncneg, ncp, nc_dst); nchstats.ncs_neghits++; if (ncp->nc_flag & NCF_WHITE) cnp->cn_flags |= ISWHITEOUT; SDT_PROBE(vfs, namecache, lookup, hit_negative, dvp, ncp->nc_name, 0, 0, 0); CACHE_WUNLOCK(); return (ENOENT); wlock: /* * We need to update the cache after our lookup, so upgrade to * a write lock and retry the operation. */ CACHE_RUNLOCK(); CACHE_WLOCK(); numupgrades++; wlocked = 1; goto retry_wlocked; success: /* * On success we return a locked and ref'd vnode as per the lookup * protocol. */ if (dvp == *vpp) { /* lookup on "." */ VREF(*vpp); if (wlocked) CACHE_WUNLOCK(); else CACHE_RUNLOCK(); /* * When we lookup "." we still can be asked to lock it * differently... */ ltype = cnp->cn_lkflags & LK_TYPE_MASK; if (ltype != VOP_ISLOCKED(*vpp)) { if (ltype == LK_EXCLUSIVE) { vn_lock(*vpp, LK_UPGRADE | LK_RETRY); if ((*vpp)->v_iflag & VI_DOOMED) { /* forced unmount */ vrele(*vpp); *vpp = NULL; return (ENOENT); } } else vn_lock(*vpp, LK_DOWNGRADE | LK_RETRY); } return (-1); } ltype = 0; /* silence gcc warning */ if (cnp->cn_flags & ISDOTDOT) { ltype = VOP_ISLOCKED(dvp); VOP_UNLOCK(dvp, 0); } VI_LOCK(*vpp); if (wlocked) CACHE_WUNLOCK(); else CACHE_RUNLOCK(); error = vget(*vpp, cnp->cn_lkflags | LK_INTERLOCK, cnp->cn_thread); if (cnp->cn_flags & ISDOTDOT) { vn_lock(dvp, ltype | LK_RETRY); if (dvp->v_iflag & VI_DOOMED) { if (error == 0) vput(*vpp); *vpp = NULL; return (ENOENT); } } if (error) { *vpp = NULL; goto retry; } if ((cnp->cn_flags & ISLASTCN) && (cnp->cn_lkflags & LK_TYPE_MASK) == LK_EXCLUSIVE) { ASSERT_VOP_ELOCKED(*vpp, "cache_lookup"); } return (-1); unlock: if (wlocked) CACHE_WUNLOCK(); else CACHE_RUNLOCK(); return (0); } /* * Add an entry to the cache. */ void cache_enter(dvp, vp, cnp) struct vnode *dvp; struct vnode *vp; struct componentname *cnp; { struct namecache *ncp, *n2; struct nchashhead *ncpp; u_int32_t hash; int flag; int hold; int zap; int len; CTR3(KTR_VFS, "cache_enter(%p, %p, %s)", dvp, vp, cnp->cn_nameptr); VNASSERT(vp == NULL || (vp->v_iflag & VI_DOOMED) == 0, vp, - ("cahe_enter: Adding a doomed vnode")); + ("cache_enter: Adding a doomed vnode")); if (!doingcache) return; /* * Avoid blowout in namecache entries. */ if (numcache >= desiredvnodes * 2) return; flag = 0; if (cnp->cn_nameptr[0] == '.') { if (cnp->cn_namelen == 1) return; if (cnp->cn_namelen == 2 && cnp->cn_nameptr[1] == '.') { CACHE_WLOCK(); /* * If dotdot entry already exists, just retarget it * to new parent vnode, otherwise continue with new * namecache entry allocation. */ if ((ncp = dvp->v_cache_dd) != NULL && ncp->nc_flag & NCF_ISDOTDOT) { KASSERT(ncp->nc_dvp == dvp, ("wrong isdotdot parent")); if (ncp->nc_vp != NULL) TAILQ_REMOVE(&ncp->nc_vp->v_cache_dst, ncp, nc_dst); else TAILQ_REMOVE(&ncneg, ncp, nc_dst); if (vp != NULL) TAILQ_INSERT_HEAD(&vp->v_cache_dst, ncp, nc_dst); else TAILQ_INSERT_TAIL(&ncneg, ncp, nc_dst); ncp->nc_vp = vp; CACHE_WUNLOCK(); return; } dvp->v_cache_dd = NULL; SDT_PROBE(vfs, namecache, enter, done, dvp, "..", vp, 0, 0); CACHE_WUNLOCK(); flag = NCF_ISDOTDOT; } } hold = 0; zap = 0; /* * Calculate the hash key and setup as much of the new * namecache entry as possible before acquiring the lock. */ ncp = cache_alloc(cnp->cn_namelen); ncp->nc_vp = vp; ncp->nc_dvp = dvp; ncp->nc_flag = flag; len = ncp->nc_nlen = cnp->cn_namelen; hash = fnv_32_buf(cnp->cn_nameptr, len, FNV1_32_INIT); strlcpy(ncp->nc_name, cnp->cn_nameptr, len + 1); hash = fnv_32_buf(&dvp, sizeof(dvp), hash); CACHE_WLOCK(); /* * See if this vnode or negative entry is already in the cache * with this name. This can happen with concurrent lookups of * the same path name. */ ncpp = NCHHASH(hash); LIST_FOREACH(n2, ncpp, nc_hash) { if (n2->nc_dvp == dvp && n2->nc_nlen == cnp->cn_namelen && !bcmp(n2->nc_name, cnp->cn_nameptr, n2->nc_nlen)) { CACHE_WUNLOCK(); cache_free(ncp); return; } } if (flag == NCF_ISDOTDOT) { /* * See if we are trying to add .. entry, but some other lookup * has populated v_cache_dd pointer already. */ if (dvp->v_cache_dd != NULL) { CACHE_WUNLOCK(); cache_free(ncp); return; } KASSERT(vp == NULL || vp->v_type == VDIR, ("wrong vnode type %p", vp)); dvp->v_cache_dd = ncp; } numcache++; if (!vp) { numneg++; if (cnp->cn_flags & ISWHITEOUT) ncp->nc_flag |= NCF_WHITE; } else if (vp->v_type == VDIR) { if (flag != NCF_ISDOTDOT) { if ((n2 = vp->v_cache_dd) != NULL && (n2->nc_flag & NCF_ISDOTDOT) != 0) cache_zap(n2); vp->v_cache_dd = ncp; } } else { vp->v_cache_dd = NULL; } /* * Insert the new namecache entry into the appropriate chain * within the cache entries table. */ LIST_INSERT_HEAD(ncpp, ncp, nc_hash); if (flag != NCF_ISDOTDOT) { if (LIST_EMPTY(&dvp->v_cache_src)) { hold = 1; numcachehv++; } LIST_INSERT_HEAD(&dvp->v_cache_src, ncp, nc_src); } /* * If the entry is "negative", we place it into the * "negative" cache queue, otherwise, we place it into the * destination vnode's cache entries queue. */ if (vp) { TAILQ_INSERT_HEAD(&vp->v_cache_dst, ncp, nc_dst); SDT_PROBE(vfs, namecache, enter, done, dvp, ncp->nc_name, vp, 0, 0); } else { TAILQ_INSERT_TAIL(&ncneg, ncp, nc_dst); SDT_PROBE(vfs, namecache, enter_negative, done, dvp, ncp->nc_name, 0, 0, 0); } if (numneg * ncnegfactor > numcache) { ncp = TAILQ_FIRST(&ncneg); zap = 1; } if (hold) vhold(dvp); if (zap) cache_zap(ncp); CACHE_WUNLOCK(); } /* * Name cache initialization, from vfs_init() when we are booting */ static void nchinit(void *dummy __unused) { TAILQ_INIT(&ncneg); cache_zone_small = uma_zcreate("S VFS Cache", CACHE_ZONE_SMALL, NULL, NULL, NULL, NULL, UMA_ALIGN_PTR, UMA_ZONE_ZINIT); cache_zone_large = uma_zcreate("L VFS Cache", CACHE_ZONE_LARGE, NULL, NULL, NULL, NULL, UMA_ALIGN_PTR, UMA_ZONE_ZINIT); nchashtbl = hashinit(desiredvnodes * 2, M_VFSCACHE, &nchash); } SYSINIT(vfs, SI_SUB_VFS, SI_ORDER_SECOND, nchinit, NULL); /* * Invalidate all entries to a particular vnode. */ void cache_purge(vp) struct vnode *vp; { CTR1(KTR_VFS, "cache_purge(%p)", vp); SDT_PROBE(vfs, namecache, purge, done, vp, 0, 0, 0, 0); CACHE_WLOCK(); while (!LIST_EMPTY(&vp->v_cache_src)) cache_zap(LIST_FIRST(&vp->v_cache_src)); while (!TAILQ_EMPTY(&vp->v_cache_dst)) cache_zap(TAILQ_FIRST(&vp->v_cache_dst)); if (vp->v_cache_dd != NULL) { KASSERT(vp->v_cache_dd->nc_flag & NCF_ISDOTDOT, ("lost dotdot link")); cache_zap(vp->v_cache_dd); } KASSERT(vp->v_cache_dd == NULL, ("incomplete purge")); CACHE_WUNLOCK(); } /* * Invalidate all negative entries for a particular directory vnode. */ void cache_purge_negative(vp) struct vnode *vp; { struct namecache *cp, *ncp; CTR1(KTR_VFS, "cache_purge_negative(%p)", vp); SDT_PROBE(vfs, namecache, purge_negative, done, vp, 0, 0, 0, 0); CACHE_WLOCK(); LIST_FOREACH_SAFE(cp, &vp->v_cache_src, nc_src, ncp) { if (cp->nc_vp == NULL) cache_zap(cp); } CACHE_WUNLOCK(); } /* * Flush all entries referencing a particular filesystem. */ void cache_purgevfs(mp) struct mount *mp; { struct nchashhead *ncpp; struct namecache *ncp, *nnp; /* Scan hash tables for applicable entries */ SDT_PROBE(vfs, namecache, purgevfs, done, mp, 0, 0, 0, 0); CACHE_WLOCK(); for (ncpp = &nchashtbl[nchash]; ncpp >= nchashtbl; ncpp--) { LIST_FOREACH_SAFE(ncp, ncpp, nc_hash, nnp) { if (ncp->nc_dvp->v_mount == mp) cache_zap(ncp); } } CACHE_WUNLOCK(); } /* * Perform canonical checks and cache lookup and pass on to filesystem * through the vop_cachedlookup only if needed. */ int vfs_cache_lookup(ap) struct vop_lookup_args /* { struct vnode *a_dvp; struct vnode **a_vpp; struct componentname *a_cnp; } */ *ap; { struct vnode *dvp; int error; struct vnode **vpp = ap->a_vpp; struct componentname *cnp = ap->a_cnp; struct ucred *cred = cnp->cn_cred; int flags = cnp->cn_flags; struct thread *td = cnp->cn_thread; *vpp = NULL; dvp = ap->a_dvp; if (dvp->v_type != VDIR) return (ENOTDIR); if ((flags & ISLASTCN) && (dvp->v_mount->mnt_flag & MNT_RDONLY) && (cnp->cn_nameiop == DELETE || cnp->cn_nameiop == RENAME)) return (EROFS); error = VOP_ACCESS(dvp, VEXEC, cred, td); if (error) return (error); error = cache_lookup(dvp, vpp, cnp); if (error == 0) return (VOP_CACHEDLOOKUP(dvp, vpp, cnp)); if (error == -1) return (0); return (error); } #ifndef _SYS_SYSPROTO_H_ struct __getcwd_args { u_char *buf; u_int buflen; }; #endif /* * XXX All of these sysctls would probably be more productive dead. */ static int disablecwd; SYSCTL_INT(_debug, OID_AUTO, disablecwd, CTLFLAG_RW, &disablecwd, 0, "Disable the getcwd syscall"); /* Implementation of the getcwd syscall. */ int __getcwd(td, uap) struct thread *td; struct __getcwd_args *uap; { return (kern___getcwd(td, uap->buf, UIO_USERSPACE, uap->buflen)); } int kern___getcwd(struct thread *td, u_char *buf, enum uio_seg bufseg, u_int buflen) { char *bp, *tmpbuf; struct filedesc *fdp; struct vnode *cdir, *rdir; int error, vfslocked; if (disablecwd) return (ENODEV); if (buflen < 2) return (EINVAL); if (buflen > MAXPATHLEN) buflen = MAXPATHLEN; tmpbuf = malloc(buflen, M_TEMP, M_WAITOK); fdp = td->td_proc->p_fd; FILEDESC_SLOCK(fdp); cdir = fdp->fd_cdir; VREF(cdir); rdir = fdp->fd_rdir; VREF(rdir); FILEDESC_SUNLOCK(fdp); error = vn_fullpath1(td, cdir, rdir, tmpbuf, &bp, buflen); vfslocked = VFS_LOCK_GIANT(rdir->v_mount); vrele(rdir); VFS_UNLOCK_GIANT(vfslocked); vfslocked = VFS_LOCK_GIANT(cdir->v_mount); vrele(cdir); VFS_UNLOCK_GIANT(vfslocked); if (!error) { if (bufseg == UIO_SYSSPACE) bcopy(bp, buf, strlen(bp) + 1); else error = copyout(bp, buf, strlen(bp) + 1); #ifdef KTRACE if (KTRPOINT(curthread, KTR_NAMEI)) ktrnamei(bp); #endif } free(tmpbuf, M_TEMP); return (error); } /* * Thus begins the fullpath magic. */ #undef STATNODE #define STATNODE(name) \ static u_int name; \ SYSCTL_UINT(_vfs_cache, OID_AUTO, name, CTLFLAG_RD, &name, 0, "") static int disablefullpath; SYSCTL_INT(_debug, OID_AUTO, disablefullpath, CTLFLAG_RW, &disablefullpath, 0, "Disable the vn_fullpath function"); /* These count for kern___getcwd(), too. */ STATNODE(numfullpathcalls); STATNODE(numfullpathfail1); STATNODE(numfullpathfail2); STATNODE(numfullpathfail4); STATNODE(numfullpathfound); /* * Retrieve the full filesystem path that correspond to a vnode from the name * cache (if available) */ int vn_fullpath(struct thread *td, struct vnode *vn, char **retbuf, char **freebuf) { char *buf; struct filedesc *fdp; struct vnode *rdir; int error, vfslocked; if (disablefullpath) return (ENODEV); if (vn == NULL) return (EINVAL); buf = malloc(MAXPATHLEN, M_TEMP, M_WAITOK); fdp = td->td_proc->p_fd; FILEDESC_SLOCK(fdp); rdir = fdp->fd_rdir; VREF(rdir); FILEDESC_SUNLOCK(fdp); error = vn_fullpath1(td, vn, rdir, buf, retbuf, MAXPATHLEN); vfslocked = VFS_LOCK_GIANT(rdir->v_mount); vrele(rdir); VFS_UNLOCK_GIANT(vfslocked); if (!error) *freebuf = buf; else free(buf, M_TEMP); return (error); } /* * This function is similar to vn_fullpath, but it attempts to lookup the * pathname relative to the global root mount point. This is required for the * auditing sub-system, as audited pathnames must be absolute, relative to the * global root mount point. */ int vn_fullpath_global(struct thread *td, struct vnode *vn, char **retbuf, char **freebuf) { char *buf; int error; if (disablefullpath) return (ENODEV); if (vn == NULL) return (EINVAL); buf = malloc(MAXPATHLEN, M_TEMP, M_WAITOK); error = vn_fullpath1(td, vn, rootvnode, buf, retbuf, MAXPATHLEN); if (!error) *freebuf = buf; else free(buf, M_TEMP); return (error); } int vn_vptocnp(struct vnode **vp, struct ucred *cred, char *buf, u_int *buflen) { int error; CACHE_RLOCK(); error = vn_vptocnp_locked(vp, cred, buf, buflen); if (error == 0) { /* * vn_vptocnp_locked() dropped hold acquired by * VOP_VPTOCNP immediately after locking the * cache. Since we are going to drop the cache rlock, * re-hold the result. */ vhold(*vp); CACHE_RUNLOCK(); } return (error); } static int vn_vptocnp_locked(struct vnode **vp, struct ucred *cred, char *buf, u_int *buflen) { struct vnode *dvp; struct namecache *ncp; int error, vfslocked; TAILQ_FOREACH(ncp, &((*vp)->v_cache_dst), nc_dst) { if ((ncp->nc_flag & NCF_ISDOTDOT) == 0) break; } if (ncp != NULL) { if (*buflen < ncp->nc_nlen) { CACHE_RUNLOCK(); numfullpathfail4++; error = ENOMEM; SDT_PROBE(vfs, namecache, fullpath, return, error, vp, NULL, 0, 0); return (error); } *buflen -= ncp->nc_nlen; memcpy(buf + *buflen, ncp->nc_name, ncp->nc_nlen); SDT_PROBE(vfs, namecache, fullpath, hit, ncp->nc_dvp, ncp->nc_name, vp, 0, 0); *vp = ncp->nc_dvp; return (0); } SDT_PROBE(vfs, namecache, fullpath, miss, vp, 0, 0, 0, 0); vhold(*vp); CACHE_RUNLOCK(); vfslocked = VFS_LOCK_GIANT((*vp)->v_mount); vn_lock(*vp, LK_SHARED | LK_RETRY); error = VOP_VPTOCNP(*vp, &dvp, cred, buf, buflen); VOP_UNLOCK(*vp, 0); vdrop(*vp); VFS_UNLOCK_GIANT(vfslocked); if (error) { numfullpathfail2++; SDT_PROBE(vfs, namecache, fullpath, return, error, vp, NULL, 0, 0); return (error); } *vp = dvp; CACHE_RLOCK(); if ((*vp)->v_iflag & VI_DOOMED) { /* forced unmount */ CACHE_RUNLOCK(); vdrop(*vp); error = ENOENT; SDT_PROBE(vfs, namecache, fullpath, return, error, vp, NULL, 0, 0); return (error); } vdrop(*vp); return (0); } /* * The magic behind kern___getcwd() and vn_fullpath(). */ static int vn_fullpath1(struct thread *td, struct vnode *vp, struct vnode *rdir, char *buf, char **retbuf, u_int buflen) { int error, slash_prefixed; #ifdef KDTRACE_HOOKS struct vnode *startvp = vp; #endif buflen--; buf[buflen] = '\0'; error = 0; slash_prefixed = 0; SDT_PROBE(vfs, namecache, fullpath, entry, vp, 0, 0, 0, 0); numfullpathcalls++; CACHE_RLOCK(); if (vp->v_type != VDIR) { error = vn_vptocnp_locked(&vp, td->td_ucred, buf, &buflen); if (error) return (error); if (buflen == 0) { CACHE_RUNLOCK(); return (ENOMEM); } buf[--buflen] = '/'; slash_prefixed = 1; } while (vp != rdir && vp != rootvnode) { if (vp->v_vflag & VV_ROOT) { if (vp->v_iflag & VI_DOOMED) { /* forced unmount */ CACHE_RUNLOCK(); error = ENOENT; SDT_PROBE(vfs, namecache, fullpath, return, error, vp, NULL, 0, 0); break; } vp = vp->v_mount->mnt_vnodecovered; continue; } if (vp->v_type != VDIR) { CACHE_RUNLOCK(); numfullpathfail1++; error = ENOTDIR; SDT_PROBE(vfs, namecache, fullpath, return, error, vp, NULL, 0, 0); break; } error = vn_vptocnp_locked(&vp, td->td_ucred, buf, &buflen); if (error) break; if (buflen == 0) { CACHE_RUNLOCK(); error = ENOMEM; SDT_PROBE(vfs, namecache, fullpath, return, error, startvp, NULL, 0, 0); break; } buf[--buflen] = '/'; slash_prefixed = 1; } if (error) return (error); if (!slash_prefixed) { if (buflen == 0) { CACHE_RUNLOCK(); numfullpathfail4++; SDT_PROBE(vfs, namecache, fullpath, return, ENOMEM, startvp, NULL, 0, 0); return (ENOMEM); } buf[--buflen] = '/'; } numfullpathfound++; CACHE_RUNLOCK(); SDT_PROBE(vfs, namecache, fullpath, return, 0, startvp, buf + buflen, 0, 0); *retbuf = buf + buflen; return (0); } int vn_commname(struct vnode *vp, char *buf, u_int buflen) { struct namecache *ncp; int l; CACHE_RLOCK(); TAILQ_FOREACH(ncp, &vp->v_cache_dst, nc_dst) if ((ncp->nc_flag & NCF_ISDOTDOT) == 0) break; if (ncp == NULL) { CACHE_RUNLOCK(); return (ENOENT); } l = min(ncp->nc_nlen, buflen - 1); memcpy(buf, ncp->nc_name, l); CACHE_RUNLOCK(); buf[l] = '\0'; return (0); } Index: user/imp/tbemd/sys/net/if.c =================================================================== --- user/imp/tbemd/sys/net/if.c (revision 206684) +++ user/imp/tbemd/sys/net/if.c (revision 206685) @@ -1,3420 +1,3419 @@ /*- * Copyright (c) 1980, 1986, 1993 * The Regents of the University of California. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 4. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * @(#)if.c 8.5 (Berkeley) 1/9/95 * $FreeBSD$ */ #include "opt_compat.h" #include "opt_inet6.h" #include "opt_inet.h" #include "opt_carp.h" #include "opt_ddb.h" #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #ifdef DDB #include #endif #include #include #include #include #include #include #include #include #include #if defined(INET) || defined(INET6) /*XXX*/ #include #include #ifdef INET6 #include #include #endif #endif #ifdef INET #include #endif #if defined(INET) || defined(INET6) #ifdef DEV_CARP #include #endif #endif #include struct ifindex_entry { struct ifnet *ife_ifnet; }; SYSCTL_NODE(_net, PF_LINK, link, CTLFLAG_RW, 0, "Link layers"); SYSCTL_NODE(_net_link, 0, generic, CTLFLAG_RW, 0, "Generic link-management"); /* Log link state change events */ static int log_link_state_change = 1; SYSCTL_INT(_net_link, OID_AUTO, log_link_state_change, CTLFLAG_RW, &log_link_state_change, 0, "log interface link state change events"); /* Interface description */ static unsigned int ifdescr_maxlen = 1024; SYSCTL_UINT(_net, OID_AUTO, ifdescr_maxlen, CTLFLAG_RW, &ifdescr_maxlen, 0, "administrative maximum length for interface description"); MALLOC_DEFINE(M_IFDESCR, "ifdescr", "ifnet descriptions"); /* global sx for non-critical path ifdescr */ static struct sx ifdescr_sx; SX_SYSINIT(ifdescr_sx, &ifdescr_sx, "ifnet descr"); void (*bstp_linkstate_p)(struct ifnet *ifp, int state); void (*ng_ether_link_state_p)(struct ifnet *ifp, int state); void (*lagg_linkstate_p)(struct ifnet *ifp, int state); struct mbuf *(*tbr_dequeue_ptr)(struct ifaltq *, int) = NULL; /* * XXX: Style; these should be sorted alphabetically, and unprototyped * static functions should be prototyped. Currently they are sorted by * declaration order. */ static void if_attachdomain(void *); static void if_attachdomain1(struct ifnet *); static int ifconf(u_long, caddr_t); static void if_freemulti(struct ifmultiaddr *); static void if_init(void *); static void if_grow(void); static void if_route(struct ifnet *, int flag, int fam); static int if_setflag(struct ifnet *, int, int, int *, int); static int if_transmit(struct ifnet *ifp, struct mbuf *m); static void if_unroute(struct ifnet *, int flag, int fam); static void link_rtrequest(int, struct rtentry *, struct rt_addrinfo *); static int if_rtdel(struct radix_node *, void *); static int ifhwioctl(u_long, struct ifnet *, caddr_t, struct thread *); static int if_delmulti_locked(struct ifnet *, struct ifmultiaddr *, int); static void do_link_state_change(void *, int); static int if_getgroup(struct ifgroupreq *, struct ifnet *); static int if_getgroupmembers(struct ifgroupreq *); static void if_delgroups(struct ifnet *); static void if_attach_internal(struct ifnet *, int); static void if_detach_internal(struct ifnet *, int); #ifdef INET6 /* * XXX: declare here to avoid to include many inet6 related files.. * should be more generalized? */ extern void nd6_setmtu(struct ifnet *); #endif VNET_DEFINE(struct ifnethead, ifnet); /* depend on static init XXX */ VNET_DEFINE(struct ifgrouphead, ifg_head); VNET_DEFINE(int, if_index); static VNET_DEFINE(int, if_indexlim) = 8; /* Table of ifnet by index. */ static VNET_DEFINE(struct ifindex_entry *, ifindex_table); #define V_if_indexlim VNET(if_indexlim) #define V_ifindex_table VNET(ifindex_table) int ifqmaxlen = IFQ_MAXLEN; /* * The global network interface list (V_ifnet) and related state (such as * if_index, if_indexlim, and ifindex_table) are protected by an sxlock and * an rwlock. Either may be acquired shared to stablize the list, but both * must be acquired writable to modify the list. This model allows us to * both stablize the interface list during interrupt thread processing, but * also to stablize it over long-running ioctls, without introducing priority * inversions and deadlocks. */ struct rwlock ifnet_rwlock; struct sx ifnet_sxlock; /* * The allocation of network interfaces is a rather non-atomic affair; we * need to select an index before we are ready to expose the interface for * use, so will use this pointer value to indicate reservation. */ #define IFNET_HOLD (void *)(uintptr_t)(-1) static if_com_alloc_t *if_com_alloc[256]; static if_com_free_t *if_com_free[256]; MALLOC_DEFINE(M_IFNET, "ifnet", "interface internals"); MALLOC_DEFINE(M_IFADDR, "ifaddr", "interface address"); MALLOC_DEFINE(M_IFMADDR, "ether_multi", "link-level multicast address"); struct ifnet * ifnet_byindex_locked(u_short idx) { if (idx > V_if_index) return (NULL); if (V_ifindex_table[idx].ife_ifnet == IFNET_HOLD) return (NULL); return (V_ifindex_table[idx].ife_ifnet); } struct ifnet * ifnet_byindex(u_short idx) { struct ifnet *ifp; IFNET_RLOCK_NOSLEEP(); ifp = ifnet_byindex_locked(idx); IFNET_RUNLOCK_NOSLEEP(); return (ifp); } struct ifnet * ifnet_byindex_ref(u_short idx) { struct ifnet *ifp; IFNET_RLOCK_NOSLEEP(); ifp = ifnet_byindex_locked(idx); if (ifp == NULL || (ifp->if_flags & IFF_DYING)) { IFNET_RUNLOCK_NOSLEEP(); return (NULL); } if_ref(ifp); IFNET_RUNLOCK_NOSLEEP(); return (ifp); } /* * Allocate an ifindex array entry; return 0 on success or an error on * failure. */ static int ifindex_alloc_locked(u_short *idxp) { u_short idx; IFNET_WLOCK_ASSERT(); /* * Try to find an empty slot below V_if_index. If we fail, take the * next slot. */ for (idx = 1; idx <= V_if_index; idx++) { if (V_ifindex_table[idx].ife_ifnet == NULL) break; } /* Catch if_index overflow. */ if (idx < 1) return (ENOSPC); if (idx > V_if_index) V_if_index = idx; if (V_if_index >= V_if_indexlim) if_grow(); *idxp = idx; return (0); } static void ifindex_free_locked(u_short idx) { IFNET_WLOCK_ASSERT(); V_ifindex_table[idx].ife_ifnet = NULL; while (V_if_index > 0 && V_ifindex_table[V_if_index].ife_ifnet == NULL) V_if_index--; } static void ifindex_free(u_short idx) { IFNET_WLOCK(); ifindex_free_locked(idx); IFNET_WUNLOCK(); } static void ifnet_setbyindex_locked(u_short idx, struct ifnet *ifp) { IFNET_WLOCK_ASSERT(); V_ifindex_table[idx].ife_ifnet = ifp; } static void ifnet_setbyindex(u_short idx, struct ifnet *ifp) { IFNET_WLOCK(); ifnet_setbyindex_locked(idx, ifp); IFNET_WUNLOCK(); } struct ifaddr * ifaddr_byindex(u_short idx) { struct ifaddr *ifa; IFNET_RLOCK_NOSLEEP(); ifa = ifnet_byindex_locked(idx)->if_addr; if (ifa != NULL) ifa_ref(ifa); IFNET_RUNLOCK_NOSLEEP(); return (ifa); } /* * Network interface utility routines. * * Routines with ifa_ifwith* names take sockaddr *'s as * parameters. */ static void vnet_if_init(const void *unused __unused) { TAILQ_INIT(&V_ifnet); TAILQ_INIT(&V_ifg_head); if_grow(); /* create initial table */ vnet_if_clone_init(); } VNET_SYSINIT(vnet_if_init, SI_SUB_INIT_IF, SI_ORDER_FIRST, vnet_if_init, NULL); /* ARGSUSED*/ static void if_init(void *dummy __unused) { IFNET_LOCK_INIT(); if_clone_init(); } SYSINIT(interfaces, SI_SUB_INIT_IF, SI_ORDER_SECOND, if_init, NULL); #ifdef VIMAGE static void vnet_if_uninit(const void *unused __unused) { VNET_ASSERT(TAILQ_EMPTY(&V_ifnet)); VNET_ASSERT(TAILQ_EMPTY(&V_ifg_head)); free((caddr_t)V_ifindex_table, M_IFNET); } VNET_SYSUNINIT(vnet_if_uninit, SI_SUB_INIT_IF, SI_ORDER_FIRST, vnet_if_uninit, NULL); #endif static void if_grow(void) { u_int n; struct ifindex_entry *e; V_if_indexlim <<= 1; n = V_if_indexlim * sizeof(*e); e = malloc(n, M_IFNET, M_WAITOK | M_ZERO); if (V_ifindex_table != NULL) { memcpy((caddr_t)e, (caddr_t)V_ifindex_table, n/2); free((caddr_t)V_ifindex_table, M_IFNET); } V_ifindex_table = e; } /* * Allocate a struct ifnet and an index for an interface. A layer 2 * common structure will also be allocated if an allocation routine is * registered for the passed type. */ struct ifnet * if_alloc(u_char type) { struct ifnet *ifp; u_short idx; ifp = malloc(sizeof(struct ifnet), M_IFNET, M_WAITOK|M_ZERO); IFNET_WLOCK(); if (ifindex_alloc_locked(&idx) != 0) { IFNET_WUNLOCK(); free(ifp, M_IFNET); return (NULL); } ifnet_setbyindex_locked(idx, IFNET_HOLD); IFNET_WUNLOCK(); ifp->if_index = idx; ifp->if_type = type; ifp->if_alloctype = type; if (if_com_alloc[type] != NULL) { ifp->if_l2com = if_com_alloc[type](type, ifp); if (ifp->if_l2com == NULL) { free(ifp, M_IFNET); ifindex_free(idx); return (NULL); } } IF_ADDR_LOCK_INIT(ifp); TASK_INIT(&ifp->if_linktask, 0, do_link_state_change, ifp); ifp->if_afdata_initialized = 0; IF_AFDATA_LOCK_INIT(ifp); TAILQ_INIT(&ifp->if_addrhead); TAILQ_INIT(&ifp->if_prefixhead); TAILQ_INIT(&ifp->if_multiaddrs); TAILQ_INIT(&ifp->if_groups); #ifdef MAC mac_ifnet_init(ifp); #endif ifq_init(&ifp->if_snd, ifp); refcount_init(&ifp->if_refcount, 1); /* Index reference. */ ifnet_setbyindex(ifp->if_index, ifp); return (ifp); } /* * Do the actual work of freeing a struct ifnet, associated index, and layer * 2 common structure. This call is made when the last reference to an * interface is released. */ static void if_free_internal(struct ifnet *ifp) { KASSERT((ifp->if_flags & IFF_DYING), ("if_free_internal: interface not dying")); IFNET_WLOCK(); KASSERT(ifp == ifnet_byindex_locked(ifp->if_index), ("%s: freeing unallocated ifnet", ifp->if_xname)); ifindex_free_locked(ifp->if_index); IFNET_WUNLOCK(); if (if_com_free[ifp->if_alloctype] != NULL) if_com_free[ifp->if_alloctype](ifp->if_l2com, ifp->if_alloctype); #ifdef MAC mac_ifnet_destroy(ifp); #endif /* MAC */ if (ifp->if_description != NULL) free(ifp->if_description, M_IFDESCR); IF_AFDATA_DESTROY(ifp); IF_ADDR_LOCK_DESTROY(ifp); ifq_delete(&ifp->if_snd); free(ifp, M_IFNET); } /* * This version should only be called by intefaces that switch their type * after calling if_alloc(). if_free_type() will go away again now that we * have if_alloctype to cache the original allocation type. For now, assert * that they match, since we require that in practice. */ void if_free_type(struct ifnet *ifp, u_char type) { KASSERT(ifp->if_alloctype == type, ("if_free_type: type (%d) != alloctype (%d)", type, ifp->if_alloctype)); ifp->if_flags |= IFF_DYING; /* XXX: Locking */ if (!refcount_release(&ifp->if_refcount)) return; if_free_internal(ifp); } /* * This is the normal version of if_free(), used by device drivers to free a * detached network interface. The contents of if_free_type() will move into * here when if_free_type() goes away. */ void if_free(struct ifnet *ifp) { if_free_type(ifp, ifp->if_alloctype); } /* * Interfaces to keep an ifnet type-stable despite the possibility of the * driver calling if_free(). If there are additional references, we defer * freeing the underlying data structure. */ void if_ref(struct ifnet *ifp) { /* We don't assert the ifnet list lock here, but arguably should. */ refcount_acquire(&ifp->if_refcount); } void if_rele(struct ifnet *ifp) { if (!refcount_release(&ifp->if_refcount)) return; if_free_internal(ifp); } void ifq_init(struct ifaltq *ifq, struct ifnet *ifp) { mtx_init(&ifq->ifq_mtx, ifp->if_xname, "if send queue", MTX_DEF); if (ifq->ifq_maxlen == 0) ifq->ifq_maxlen = ifqmaxlen; ifq->altq_type = 0; ifq->altq_disc = NULL; ifq->altq_flags &= ALTQF_CANTCHANGE; ifq->altq_tbr = NULL; ifq->altq_ifp = ifp; } void ifq_delete(struct ifaltq *ifq) { mtx_destroy(&ifq->ifq_mtx); } /* * Perform generic interface initalization tasks and attach the interface * to the list of "active" interfaces. If vmove flag is set on entry * to if_attach_internal(), perform only a limited subset of initialization * tasks, given that we are moving from one vnet to another an ifnet which * has already been fully initialized. * * XXX: * - The decision to return void and thus require this function to * succeed is questionable. * - We should probably do more sanity checking. For instance we don't * do anything to insure if_xname is unique or non-empty. */ void if_attach(struct ifnet *ifp) { if_attach_internal(ifp, 0); } static void if_attach_internal(struct ifnet *ifp, int vmove) { unsigned socksize, ifasize; int namelen, masklen; struct sockaddr_dl *sdl; struct ifaddr *ifa; if (ifp->if_index == 0 || ifp != ifnet_byindex(ifp->if_index)) panic ("%s: BUG: if_attach called without if_alloc'd input()\n", ifp->if_xname); #ifdef VIMAGE ifp->if_vnet = curvnet; if (ifp->if_home_vnet == NULL) ifp->if_home_vnet = curvnet; #endif if_addgroup(ifp, IFG_ALL); getmicrotime(&ifp->if_lastchange); ifp->if_data.ifi_epoch = time_uptime; ifp->if_data.ifi_datalen = sizeof(struct if_data); KASSERT((ifp->if_transmit == NULL && ifp->if_qflush == NULL) || (ifp->if_transmit != NULL && ifp->if_qflush != NULL), ("transmit and qflush must both either be set or both be NULL")); if (ifp->if_transmit == NULL) { ifp->if_transmit = if_transmit; ifp->if_qflush = if_qflush; } if (!vmove) { #ifdef MAC mac_ifnet_create(ifp); #endif /* * Create a Link Level name for this device. */ namelen = strlen(ifp->if_xname); /* * Always save enough space for any possiable name so we * can do a rename in place later. */ masklen = offsetof(struct sockaddr_dl, sdl_data[0]) + IFNAMSIZ; socksize = masklen + ifp->if_addrlen; if (socksize < sizeof(*sdl)) socksize = sizeof(*sdl); socksize = roundup2(socksize, sizeof(long)); ifasize = sizeof(*ifa) + 2 * socksize; ifa = malloc(ifasize, M_IFADDR, M_WAITOK | M_ZERO); ifa_init(ifa); sdl = (struct sockaddr_dl *)(ifa + 1); sdl->sdl_len = socksize; sdl->sdl_family = AF_LINK; bcopy(ifp->if_xname, sdl->sdl_data, namelen); sdl->sdl_nlen = namelen; sdl->sdl_index = ifp->if_index; sdl->sdl_type = ifp->if_type; ifp->if_addr = ifa; ifa->ifa_ifp = ifp; ifa->ifa_rtrequest = link_rtrequest; ifa->ifa_addr = (struct sockaddr *)sdl; sdl = (struct sockaddr_dl *)(socksize + (caddr_t)sdl); ifa->ifa_netmask = (struct sockaddr *)sdl; sdl->sdl_len = masklen; while (namelen != 0) sdl->sdl_data[--namelen] = 0xff; TAILQ_INSERT_HEAD(&ifp->if_addrhead, ifa, ifa_link); /* Reliably crash if used uninitialized. */ ifp->if_broadcastaddr = NULL; } #ifdef VIMAGE else { /* * Update the interface index in the link layer address * of the interface. */ for (ifa = ifp->if_addr; ifa != NULL; ifa = TAILQ_NEXT(ifa, ifa_link)) { if (ifa->ifa_addr->sa_family == AF_LINK) { sdl = (struct sockaddr_dl *)ifa->ifa_addr; sdl->sdl_index = ifp->if_index; } } } #endif IFNET_WLOCK(); TAILQ_INSERT_TAIL(&V_ifnet, ifp, if_link); #ifdef VIMAGE curvnet->vnet_ifcnt++; #endif IFNET_WUNLOCK(); if (domain_init_status >= 2) if_attachdomain1(ifp); EVENTHANDLER_INVOKE(ifnet_arrival_event, ifp); if (IS_DEFAULT_VNET(curvnet)) devctl_notify("IFNET", ifp->if_xname, "ATTACH", NULL); /* Announce the interface. */ rt_ifannouncemsg(ifp, IFAN_ARRIVAL); } static void if_attachdomain(void *dummy) { struct ifnet *ifp; int s; s = splnet(); TAILQ_FOREACH(ifp, &V_ifnet, if_link) if_attachdomain1(ifp); splx(s); } SYSINIT(domainifattach, SI_SUB_PROTO_IFATTACHDOMAIN, SI_ORDER_SECOND, if_attachdomain, NULL); static void if_attachdomain1(struct ifnet *ifp) { struct domain *dp; int s; s = splnet(); /* * Since dp->dom_ifattach calls malloc() with M_WAITOK, we * cannot lock ifp->if_afdata initialization, entirely. */ if (IF_AFDATA_TRYLOCK(ifp) == 0) { splx(s); return; } if (ifp->if_afdata_initialized >= domain_init_status) { IF_AFDATA_UNLOCK(ifp); splx(s); printf("if_attachdomain called more than once on %s\n", ifp->if_xname); return; } ifp->if_afdata_initialized = domain_init_status; IF_AFDATA_UNLOCK(ifp); /* address family dependent data region */ bzero(ifp->if_afdata, sizeof(ifp->if_afdata)); for (dp = domains; dp; dp = dp->dom_next) { if (dp->dom_ifattach) ifp->if_afdata[dp->dom_family] = (*dp->dom_ifattach)(ifp); } splx(s); } /* * Remove any unicast or broadcast network addresses from an interface. */ void if_purgeaddrs(struct ifnet *ifp) { struct ifaddr *ifa, *next; TAILQ_FOREACH_SAFE(ifa, &ifp->if_addrhead, ifa_link, next) { if (ifa->ifa_addr->sa_family == AF_LINK) continue; #ifdef INET /* XXX: Ugly!! ad hoc just for INET */ if (ifa->ifa_addr->sa_family == AF_INET) { struct ifaliasreq ifr; bzero(&ifr, sizeof(ifr)); ifr.ifra_addr = *ifa->ifa_addr; if (ifa->ifa_dstaddr) ifr.ifra_broadaddr = *ifa->ifa_dstaddr; if (in_control(NULL, SIOCDIFADDR, (caddr_t)&ifr, ifp, NULL) == 0) continue; } #endif /* INET */ #ifdef INET6 if (ifa->ifa_addr->sa_family == AF_INET6) { in6_purgeaddr(ifa); /* ifp_addrhead is already updated */ continue; } #endif /* INET6 */ TAILQ_REMOVE(&ifp->if_addrhead, ifa, ifa_link); ifa_free(ifa); } } /* * Remove any multicast network addresses from an interface when an ifnet * is going away. */ static void if_purgemaddrs(struct ifnet *ifp) { struct ifmultiaddr *ifma; struct ifmultiaddr *next; IF_ADDR_LOCK(ifp); TAILQ_FOREACH_SAFE(ifma, &ifp->if_multiaddrs, ifma_link, next) if_delmulti_locked(ifp, ifma, 1); IF_ADDR_UNLOCK(ifp); } /* * Detach an interface, removing it from the list of "active" interfaces. * If vmove flag is set on entry to if_detach_internal(), perform only a * limited subset of cleanup tasks, given that we are moving an ifnet from * one vnet to another, where it must be fully operational. * * XXXRW: There are some significant questions about event ordering, and * how to prevent things from starting to use the interface during detach. */ void if_detach(struct ifnet *ifp) { if_detach_internal(ifp, 0); } static void if_detach_internal(struct ifnet *ifp, int vmove) { struct ifaddr *ifa; struct radix_node_head *rnh; int i, j; struct domain *dp; struct ifnet *iter; int found = 0; IFNET_WLOCK(); TAILQ_FOREACH(iter, &V_ifnet, if_link) if (iter == ifp) { TAILQ_REMOVE(&V_ifnet, ifp, if_link); found = 1; break; } #ifdef VIMAGE if (found) curvnet->vnet_ifcnt--; #endif IFNET_WUNLOCK(); if (!found) { if (vmove) panic("%s: ifp=%p not on the ifnet tailq %p", __func__, ifp, &V_ifnet); else return; /* XXX this should panic as well? */ } /* * Remove/wait for pending events. */ taskqueue_drain(taskqueue_swi, &ifp->if_linktask); /* * Remove routes and flush queues. */ if_down(ifp); #ifdef ALTQ if (ALTQ_IS_ENABLED(&ifp->if_snd)) altq_disable(&ifp->if_snd); if (ALTQ_IS_ATTACHED(&ifp->if_snd)) altq_detach(&ifp->if_snd); #endif if_purgeaddrs(ifp); #ifdef INET in_ifdetach(ifp); #endif #ifdef INET6 /* * Remove all IPv6 kernel structs related to ifp. This should be done * before removing routing entries below, since IPv6 interface direct * routes are expected to be removed by the IPv6-specific kernel API. * Otherwise, the kernel will detect some inconsistency and bark it. */ in6_ifdetach(ifp); #endif if_purgemaddrs(ifp); if (!vmove) { /* * Prevent further calls into the device driver via ifnet. */ if_dead(ifp); /* * Remove link ifaddr pointer and maybe decrement if_index. * Clean up all addresses. */ ifp->if_addr = NULL; /* We can now free link ifaddr. */ if (!TAILQ_EMPTY(&ifp->if_addrhead)) { ifa = TAILQ_FIRST(&ifp->if_addrhead); TAILQ_REMOVE(&ifp->if_addrhead, ifa, ifa_link); ifa_free(ifa); } } /* * Delete all remaining routes using this interface * Unfortuneatly the only way to do this is to slog through * the entire routing table looking for routes which point * to this interface...oh well... */ for (i = 1; i <= AF_MAX; i++) { for (j = 0; j < rt_numfibs; j++) { rnh = rt_tables_get_rnh(j, i); if (rnh == NULL) continue; RADIX_NODE_HEAD_LOCK(rnh); (void) rnh->rnh_walktree(rnh, if_rtdel, ifp); RADIX_NODE_HEAD_UNLOCK(rnh); } } /* Announce that the interface is gone. */ rt_ifannouncemsg(ifp, IFAN_DEPARTURE); EVENTHANDLER_INVOKE(ifnet_departure_event, ifp); if (IS_DEFAULT_VNET(curvnet)) devctl_notify("IFNET", ifp->if_xname, "DETACH", NULL); if_delgroups(ifp); /* * We cannot hold the lock over dom_ifdetach calls as they might * sleep, for example trying to drain a callout, thus open up the * theoretical race with re-attaching. */ IF_AFDATA_LOCK(ifp); i = ifp->if_afdata_initialized; ifp->if_afdata_initialized = 0; IF_AFDATA_UNLOCK(ifp); for (dp = domains; i > 0 && dp; dp = dp->dom_next) { if (dp->dom_ifdetach && ifp->if_afdata[dp->dom_family]) (*dp->dom_ifdetach)(ifp, ifp->if_afdata[dp->dom_family]); } } #ifdef VIMAGE /* * if_vmove() performs a limited version of if_detach() in current * vnet and if_attach()es the ifnet to the vnet specified as 2nd arg. * An attempt is made to shrink if_index in current vnet, find an * unused if_index in target vnet and calls if_grow() if necessary, * and finally find an unused if_xname for the target vnet. */ void if_vmove(struct ifnet *ifp, struct vnet *new_vnet) { u_short idx; /* * Detach from current vnet, but preserve LLADDR info, do not * mark as dead etc. so that the ifnet can be reattached later. */ if_detach_internal(ifp, 1); /* * Unlink the ifnet from ifindex_table[] in current vnet, and shrink * the if_index for that vnet if possible. * * NOTE: IFNET_WLOCK/IFNET_WUNLOCK() are assumed to be unvirtualized, * or we'd lock on one vnet and unlock on another. */ IFNET_WLOCK(); ifindex_free_locked(ifp->if_index); /* * Switch to the context of the target vnet. */ CURVNET_SET_QUIET(new_vnet); if (ifindex_alloc_locked(&idx) != 0) { IFNET_WUNLOCK(); panic("if_index overflow"); } ifp->if_index = idx; ifnet_setbyindex_locked(ifp->if_index, ifp); IFNET_WUNLOCK(); if_attach_internal(ifp, 1); CURVNET_RESTORE(); } /* * Move an ifnet to or from another child prison/vnet, specified by the jail id. */ static int if_vmove_loan(struct thread *td, struct ifnet *ifp, char *ifname, int jid) { struct prison *pr; struct ifnet *difp; /* Try to find the prison within our visibility. */ sx_slock(&allprison_lock); pr = prison_find_child(td->td_ucred->cr_prison, jid); sx_sunlock(&allprison_lock); if (pr == NULL) return (ENXIO); prison_hold_locked(pr); mtx_unlock(&pr->pr_mtx); /* Do not try to move the iface from and to the same prison. */ if (pr->pr_vnet == ifp->if_vnet) { prison_free(pr); return (EEXIST); } /* Make sure the named iface does not exists in the dst. prison/vnet. */ /* XXX Lock interfaces to avoid races. */ CURVNET_SET_QUIET(pr->pr_vnet); difp = ifunit(ifname); CURVNET_RESTORE(); if (difp != NULL) { prison_free(pr); return (EEXIST); } /* Move the interface into the child jail/vnet. */ if_vmove(ifp, pr->pr_vnet); /* Report the new if_xname back to the userland. */ sprintf(ifname, "%s", ifp->if_xname); prison_free(pr); return (0); } static int if_vmove_reclaim(struct thread *td, char *ifname, int jid) { struct prison *pr; struct vnet *vnet_dst; struct ifnet *ifp; /* Try to find the prison within our visibility. */ sx_slock(&allprison_lock); pr = prison_find_child(td->td_ucred->cr_prison, jid); sx_sunlock(&allprison_lock); if (pr == NULL) return (ENXIO); prison_hold_locked(pr); mtx_unlock(&pr->pr_mtx); /* Make sure the named iface exists in the source prison/vnet. */ CURVNET_SET(pr->pr_vnet); ifp = ifunit(ifname); /* XXX Lock to avoid races. */ if (ifp == NULL) { CURVNET_RESTORE(); prison_free(pr); return (ENXIO); } /* Do not try to move the iface from and to the same prison. */ vnet_dst = TD_TO_VNET(td); if (vnet_dst == ifp->if_vnet) { CURVNET_RESTORE(); prison_free(pr); return (EEXIST); } /* Get interface back from child jail/vnet. */ if_vmove(ifp, vnet_dst); CURVNET_RESTORE(); /* Report the new if_xname back to the userland. */ sprintf(ifname, "%s", ifp->if_xname); prison_free(pr); return (0); } #endif /* VIMAGE */ /* * Add a group to an interface */ int if_addgroup(struct ifnet *ifp, const char *groupname) { struct ifg_list *ifgl; struct ifg_group *ifg = NULL; struct ifg_member *ifgm; if (groupname[0] && groupname[strlen(groupname) - 1] >= '0' && groupname[strlen(groupname) - 1] <= '9') return (EINVAL); IFNET_WLOCK(); TAILQ_FOREACH(ifgl, &ifp->if_groups, ifgl_next) if (!strcmp(ifgl->ifgl_group->ifg_group, groupname)) { IFNET_WUNLOCK(); return (EEXIST); } if ((ifgl = (struct ifg_list *)malloc(sizeof(struct ifg_list), M_TEMP, M_NOWAIT)) == NULL) { IFNET_WUNLOCK(); return (ENOMEM); } if ((ifgm = (struct ifg_member *)malloc(sizeof(struct ifg_member), M_TEMP, M_NOWAIT)) == NULL) { free(ifgl, M_TEMP); IFNET_WUNLOCK(); return (ENOMEM); } TAILQ_FOREACH(ifg, &V_ifg_head, ifg_next) if (!strcmp(ifg->ifg_group, groupname)) break; if (ifg == NULL) { if ((ifg = (struct ifg_group *)malloc(sizeof(struct ifg_group), M_TEMP, M_NOWAIT)) == NULL) { free(ifgl, M_TEMP); free(ifgm, M_TEMP); IFNET_WUNLOCK(); return (ENOMEM); } strlcpy(ifg->ifg_group, groupname, sizeof(ifg->ifg_group)); ifg->ifg_refcnt = 0; TAILQ_INIT(&ifg->ifg_members); EVENTHANDLER_INVOKE(group_attach_event, ifg); TAILQ_INSERT_TAIL(&V_ifg_head, ifg, ifg_next); } ifg->ifg_refcnt++; ifgl->ifgl_group = ifg; ifgm->ifgm_ifp = ifp; IF_ADDR_LOCK(ifp); TAILQ_INSERT_TAIL(&ifg->ifg_members, ifgm, ifgm_next); TAILQ_INSERT_TAIL(&ifp->if_groups, ifgl, ifgl_next); IF_ADDR_UNLOCK(ifp); IFNET_WUNLOCK(); EVENTHANDLER_INVOKE(group_change_event, groupname); return (0); } /* * Remove a group from an interface */ int if_delgroup(struct ifnet *ifp, const char *groupname) { struct ifg_list *ifgl; struct ifg_member *ifgm; IFNET_WLOCK(); TAILQ_FOREACH(ifgl, &ifp->if_groups, ifgl_next) if (!strcmp(ifgl->ifgl_group->ifg_group, groupname)) break; if (ifgl == NULL) { IFNET_WUNLOCK(); return (ENOENT); } IF_ADDR_LOCK(ifp); TAILQ_REMOVE(&ifp->if_groups, ifgl, ifgl_next); IF_ADDR_UNLOCK(ifp); TAILQ_FOREACH(ifgm, &ifgl->ifgl_group->ifg_members, ifgm_next) if (ifgm->ifgm_ifp == ifp) break; if (ifgm != NULL) { TAILQ_REMOVE(&ifgl->ifgl_group->ifg_members, ifgm, ifgm_next); free(ifgm, M_TEMP); } if (--ifgl->ifgl_group->ifg_refcnt == 0) { TAILQ_REMOVE(&V_ifg_head, ifgl->ifgl_group, ifg_next); EVENTHANDLER_INVOKE(group_detach_event, ifgl->ifgl_group); free(ifgl->ifgl_group, M_TEMP); } IFNET_WUNLOCK(); free(ifgl, M_TEMP); EVENTHANDLER_INVOKE(group_change_event, groupname); return (0); } /* * Remove an interface from all groups */ static void if_delgroups(struct ifnet *ifp) { struct ifg_list *ifgl; struct ifg_member *ifgm; char groupname[IFNAMSIZ]; IFNET_WLOCK(); while (!TAILQ_EMPTY(&ifp->if_groups)) { ifgl = TAILQ_FIRST(&ifp->if_groups); strlcpy(groupname, ifgl->ifgl_group->ifg_group, IFNAMSIZ); IF_ADDR_LOCK(ifp); TAILQ_REMOVE(&ifp->if_groups, ifgl, ifgl_next); IF_ADDR_UNLOCK(ifp); TAILQ_FOREACH(ifgm, &ifgl->ifgl_group->ifg_members, ifgm_next) if (ifgm->ifgm_ifp == ifp) break; if (ifgm != NULL) { TAILQ_REMOVE(&ifgl->ifgl_group->ifg_members, ifgm, ifgm_next); free(ifgm, M_TEMP); } if (--ifgl->ifgl_group->ifg_refcnt == 0) { TAILQ_REMOVE(&V_ifg_head, ifgl->ifgl_group, ifg_next); EVENTHANDLER_INVOKE(group_detach_event, ifgl->ifgl_group); free(ifgl->ifgl_group, M_TEMP); } IFNET_WUNLOCK(); free(ifgl, M_TEMP); EVENTHANDLER_INVOKE(group_change_event, groupname); IFNET_WLOCK(); } IFNET_WUNLOCK(); } /* * Stores all groups from an interface in memory pointed * to by data */ static int if_getgroup(struct ifgroupreq *data, struct ifnet *ifp) { int len, error; struct ifg_list *ifgl; struct ifg_req ifgrq, *ifgp; struct ifgroupreq *ifgr = data; if (ifgr->ifgr_len == 0) { IF_ADDR_LOCK(ifp); TAILQ_FOREACH(ifgl, &ifp->if_groups, ifgl_next) ifgr->ifgr_len += sizeof(struct ifg_req); IF_ADDR_UNLOCK(ifp); return (0); } len = ifgr->ifgr_len; ifgp = ifgr->ifgr_groups; /* XXX: wire */ IF_ADDR_LOCK(ifp); TAILQ_FOREACH(ifgl, &ifp->if_groups, ifgl_next) { if (len < sizeof(ifgrq)) { IF_ADDR_UNLOCK(ifp); return (EINVAL); } bzero(&ifgrq, sizeof ifgrq); strlcpy(ifgrq.ifgrq_group, ifgl->ifgl_group->ifg_group, sizeof(ifgrq.ifgrq_group)); if ((error = copyout(&ifgrq, ifgp, sizeof(struct ifg_req)))) { IF_ADDR_UNLOCK(ifp); return (error); } len -= sizeof(ifgrq); ifgp++; } IF_ADDR_UNLOCK(ifp); return (0); } /* * Stores all members of a group in memory pointed to by data */ static int if_getgroupmembers(struct ifgroupreq *data) { struct ifgroupreq *ifgr = data; struct ifg_group *ifg; struct ifg_member *ifgm; struct ifg_req ifgrq, *ifgp; int len, error; IFNET_RLOCK(); TAILQ_FOREACH(ifg, &V_ifg_head, ifg_next) if (!strcmp(ifg->ifg_group, ifgr->ifgr_name)) break; if (ifg == NULL) { IFNET_RUNLOCK(); return (ENOENT); } if (ifgr->ifgr_len == 0) { TAILQ_FOREACH(ifgm, &ifg->ifg_members, ifgm_next) ifgr->ifgr_len += sizeof(ifgrq); IFNET_RUNLOCK(); return (0); } len = ifgr->ifgr_len; ifgp = ifgr->ifgr_groups; TAILQ_FOREACH(ifgm, &ifg->ifg_members, ifgm_next) { if (len < sizeof(ifgrq)) { IFNET_RUNLOCK(); return (EINVAL); } bzero(&ifgrq, sizeof ifgrq); strlcpy(ifgrq.ifgrq_member, ifgm->ifgm_ifp->if_xname, sizeof(ifgrq.ifgrq_member)); if ((error = copyout(&ifgrq, ifgp, sizeof(struct ifg_req)))) { IFNET_RUNLOCK(); return (error); } len -= sizeof(ifgrq); ifgp++; } IFNET_RUNLOCK(); return (0); } /* * Delete Routes for a Network Interface * * Called for each routing entry via the rnh->rnh_walktree() call above * to delete all route entries referencing a detaching network interface. * * Arguments: * rn pointer to node in the routing table * arg argument passed to rnh->rnh_walktree() - detaching interface * * Returns: * 0 successful * errno failed - reason indicated * */ static int if_rtdel(struct radix_node *rn, void *arg) { struct rtentry *rt = (struct rtentry *)rn; struct ifnet *ifp = arg; int err; if (rt->rt_ifp == ifp) { /* * Protect (sorta) against walktree recursion problems * with cloned routes */ if ((rt->rt_flags & RTF_UP) == 0) return (0); err = rtrequest_fib(RTM_DELETE, rt_key(rt), rt->rt_gateway, rt_mask(rt), rt->rt_flags|RTF_RNH_LOCKED, (struct rtentry **) NULL, rt->rt_fibnum); if (err) { log(LOG_WARNING, "if_rtdel: error %d\n", err); } } return (0); } /* * Wrapper functions for struct ifnet address list locking macros. These are * used by kernel modules to avoid encoding programming interface or binary * interface assumptions that may be violated when kernel-internal locking * approaches change. */ void if_addr_rlock(struct ifnet *ifp) { IF_ADDR_LOCK(ifp); } void if_addr_runlock(struct ifnet *ifp) { IF_ADDR_UNLOCK(ifp); } void if_maddr_rlock(struct ifnet *ifp) { IF_ADDR_LOCK(ifp); } void if_maddr_runlock(struct ifnet *ifp) { IF_ADDR_UNLOCK(ifp); } /* * Reference count functions for ifaddrs. */ void ifa_init(struct ifaddr *ifa) { mtx_init(&ifa->ifa_mtx, "ifaddr", NULL, MTX_DEF); refcount_init(&ifa->ifa_refcnt, 1); } void ifa_ref(struct ifaddr *ifa) { refcount_acquire(&ifa->ifa_refcnt); } void ifa_free(struct ifaddr *ifa) { if (refcount_release(&ifa->ifa_refcnt)) { mtx_destroy(&ifa->ifa_mtx); free(ifa, M_IFADDR); } } int ifa_add_loopback_route(struct ifaddr *ifa, struct sockaddr *ia) { int error = 0; struct rtentry *rt = NULL; struct rt_addrinfo info; static struct sockaddr_dl null_sdl = {sizeof(null_sdl), AF_LINK}; bzero(&info, sizeof(info)); info.rti_ifp = V_loif; info.rti_flags = ifa->ifa_flags | RTF_HOST | RTF_STATIC; info.rti_info[RTAX_DST] = ia; info.rti_info[RTAX_GATEWAY] = (struct sockaddr *)&null_sdl; error = rtrequest1_fib(RTM_ADD, &info, &rt, 0); if (error == 0 && rt != NULL) { RT_LOCK(rt); ((struct sockaddr_dl *)rt->rt_gateway)->sdl_type = ifa->ifa_ifp->if_type; ((struct sockaddr_dl *)rt->rt_gateway)->sdl_index = ifa->ifa_ifp->if_index; RT_REMREF(rt); RT_UNLOCK(rt); } else if (error != 0) log(LOG_INFO, "ifa_add_loopback_route: insertion failed\n"); return (error); } int ifa_del_loopback_route(struct ifaddr *ifa, struct sockaddr *ia) { int error = 0; struct rt_addrinfo info; struct sockaddr_dl null_sdl; bzero(&null_sdl, sizeof(null_sdl)); null_sdl.sdl_len = sizeof(null_sdl); null_sdl.sdl_family = AF_LINK; null_sdl.sdl_type = ifa->ifa_ifp->if_type; null_sdl.sdl_index = ifa->ifa_ifp->if_index; bzero(&info, sizeof(info)); info.rti_flags = ifa->ifa_flags | RTF_HOST | RTF_STATIC; info.rti_info[RTAX_DST] = ia; info.rti_info[RTAX_GATEWAY] = (struct sockaddr *)&null_sdl; error = rtrequest1_fib(RTM_DELETE, &info, NULL, 0); if (error != 0) log(LOG_INFO, "ifa_del_loopback_route: deletion failed\n"); return (error); } /* * XXX: Because sockaddr_dl has deeper structure than the sockaddr * structs used to represent other address families, it is necessary * to perform a different comparison. */ #define sa_equal(a1, a2) \ (bcmp((a1), (a2), ((a1))->sa_len) == 0) #define sa_dl_equal(a1, a2) \ ((((struct sockaddr_dl *)(a1))->sdl_len == \ ((struct sockaddr_dl *)(a2))->sdl_len) && \ (bcmp(LLADDR((struct sockaddr_dl *)(a1)), \ LLADDR((struct sockaddr_dl *)(a2)), \ ((struct sockaddr_dl *)(a1))->sdl_alen) == 0)) /* * Locate an interface based on a complete address. */ /*ARGSUSED*/ static struct ifaddr * ifa_ifwithaddr_internal(struct sockaddr *addr, int getref) { struct ifnet *ifp; struct ifaddr *ifa; IFNET_RLOCK_NOSLEEP(); TAILQ_FOREACH(ifp, &V_ifnet, if_link) { IF_ADDR_LOCK(ifp); TAILQ_FOREACH(ifa, &ifp->if_addrhead, ifa_link) { if (ifa->ifa_addr->sa_family != addr->sa_family) continue; if (sa_equal(addr, ifa->ifa_addr)) { if (getref) ifa_ref(ifa); IF_ADDR_UNLOCK(ifp); goto done; } /* IP6 doesn't have broadcast */ if ((ifp->if_flags & IFF_BROADCAST) && ifa->ifa_broadaddr && ifa->ifa_broadaddr->sa_len != 0 && sa_equal(ifa->ifa_broadaddr, addr)) { if (getref) ifa_ref(ifa); IF_ADDR_UNLOCK(ifp); goto done; } } IF_ADDR_UNLOCK(ifp); } ifa = NULL; done: IFNET_RUNLOCK_NOSLEEP(); return (ifa); } struct ifaddr * ifa_ifwithaddr(struct sockaddr *addr) { return (ifa_ifwithaddr_internal(addr, 1)); } int ifa_ifwithaddr_check(struct sockaddr *addr) { return (ifa_ifwithaddr_internal(addr, 0) != NULL); } /* * Locate an interface based on the broadcast address. */ /* ARGSUSED */ struct ifaddr * ifa_ifwithbroadaddr(struct sockaddr *addr) { struct ifnet *ifp; struct ifaddr *ifa; IFNET_RLOCK_NOSLEEP(); TAILQ_FOREACH(ifp, &V_ifnet, if_link) { IF_ADDR_LOCK(ifp); TAILQ_FOREACH(ifa, &ifp->if_addrhead, ifa_link) { if (ifa->ifa_addr->sa_family != addr->sa_family) continue; if ((ifp->if_flags & IFF_BROADCAST) && ifa->ifa_broadaddr && ifa->ifa_broadaddr->sa_len != 0 && sa_equal(ifa->ifa_broadaddr, addr)) { ifa_ref(ifa); IF_ADDR_UNLOCK(ifp); goto done; } } IF_ADDR_UNLOCK(ifp); } ifa = NULL; done: IFNET_RUNLOCK_NOSLEEP(); return (ifa); } /* * Locate the point to point interface with a given destination address. */ /*ARGSUSED*/ struct ifaddr * ifa_ifwithdstaddr(struct sockaddr *addr) { struct ifnet *ifp; struct ifaddr *ifa; IFNET_RLOCK_NOSLEEP(); TAILQ_FOREACH(ifp, &V_ifnet, if_link) { if ((ifp->if_flags & IFF_POINTOPOINT) == 0) continue; IF_ADDR_LOCK(ifp); TAILQ_FOREACH(ifa, &ifp->if_addrhead, ifa_link) { if (ifa->ifa_addr->sa_family != addr->sa_family) continue; if (ifa->ifa_dstaddr != NULL && sa_equal(addr, ifa->ifa_dstaddr)) { ifa_ref(ifa); IF_ADDR_UNLOCK(ifp); goto done; } } IF_ADDR_UNLOCK(ifp); } ifa = NULL; done: IFNET_RUNLOCK_NOSLEEP(); return (ifa); } /* * Find an interface on a specific network. If many, choice * is most specific found. */ struct ifaddr * ifa_ifwithnet(struct sockaddr *addr) { struct ifnet *ifp; struct ifaddr *ifa; struct ifaddr *ifa_maybe = NULL; u_int af = addr->sa_family; char *addr_data = addr->sa_data, *cplim; /* * AF_LINK addresses can be looked up directly by their index number, * so do that if we can. */ if (af == AF_LINK) { struct sockaddr_dl *sdl = (struct sockaddr_dl *)addr; if (sdl->sdl_index && sdl->sdl_index <= V_if_index) return (ifaddr_byindex(sdl->sdl_index)); } /* * Scan though each interface, looking for ones that have addresses * in this address family. Maintain a reference on ifa_maybe once * we find one, as we release the IF_ADDR_LOCK() that kept it stable * when we move onto the next interface. */ IFNET_RLOCK_NOSLEEP(); TAILQ_FOREACH(ifp, &V_ifnet, if_link) { IF_ADDR_LOCK(ifp); TAILQ_FOREACH(ifa, &ifp->if_addrhead, ifa_link) { char *cp, *cp2, *cp3; if (ifa->ifa_addr->sa_family != af) next: continue; if (af == AF_INET && ifp->if_flags & IFF_POINTOPOINT) { /* * This is a bit broken as it doesn't * take into account that the remote end may * be a single node in the network we are * looking for. * The trouble is that we don't know the * netmask for the remote end. */ if (ifa->ifa_dstaddr != NULL && sa_equal(addr, ifa->ifa_dstaddr)) { ifa_ref(ifa); IF_ADDR_UNLOCK(ifp); goto done; } } else { /* * if we have a special address handler, * then use it instead of the generic one. */ if (ifa->ifa_claim_addr) { if ((*ifa->ifa_claim_addr)(ifa, addr)) { ifa_ref(ifa); IF_ADDR_UNLOCK(ifp); goto done; } continue; } /* * Scan all the bits in the ifa's address. * If a bit dissagrees with what we are * looking for, mask it with the netmask * to see if it really matters. * (A byte at a time) */ if (ifa->ifa_netmask == 0) continue; cp = addr_data; cp2 = ifa->ifa_addr->sa_data; cp3 = ifa->ifa_netmask->sa_data; cplim = ifa->ifa_netmask->sa_len + (char *)ifa->ifa_netmask; while (cp3 < cplim) if ((*cp++ ^ *cp2++) & *cp3++) goto next; /* next address! */ /* * If the netmask of what we just found * is more specific than what we had before * (if we had one) then remember the new one * before continuing to search * for an even better one. */ if (ifa_maybe == NULL || rn_refines((caddr_t)ifa->ifa_netmask, (caddr_t)ifa_maybe->ifa_netmask)) { if (ifa_maybe != NULL) ifa_free(ifa_maybe); ifa_maybe = ifa; ifa_ref(ifa_maybe); } } } IF_ADDR_UNLOCK(ifp); } ifa = ifa_maybe; ifa_maybe = NULL; done: IFNET_RUNLOCK_NOSLEEP(); if (ifa_maybe != NULL) ifa_free(ifa_maybe); return (ifa); } /* * Find an interface address specific to an interface best matching * a given address. */ struct ifaddr * ifaof_ifpforaddr(struct sockaddr *addr, struct ifnet *ifp) { struct ifaddr *ifa; char *cp, *cp2, *cp3; char *cplim; struct ifaddr *ifa_maybe = NULL; u_int af = addr->sa_family; if (af >= AF_MAX) return (0); IF_ADDR_LOCK(ifp); TAILQ_FOREACH(ifa, &ifp->if_addrhead, ifa_link) { if (ifa->ifa_addr->sa_family != af) continue; if (ifa_maybe == NULL) ifa_maybe = ifa; if (ifa->ifa_netmask == 0) { if (sa_equal(addr, ifa->ifa_addr) || (ifa->ifa_dstaddr && sa_equal(addr, ifa->ifa_dstaddr))) goto done; continue; } if (ifp->if_flags & IFF_POINTOPOINT) { if (sa_equal(addr, ifa->ifa_dstaddr)) goto done; } else { cp = addr->sa_data; cp2 = ifa->ifa_addr->sa_data; cp3 = ifa->ifa_netmask->sa_data; cplim = ifa->ifa_netmask->sa_len + (char *)ifa->ifa_netmask; for (; cp3 < cplim; cp3++) if ((*cp++ ^ *cp2++) & *cp3) break; if (cp3 == cplim) goto done; } } ifa = ifa_maybe; done: if (ifa != NULL) ifa_ref(ifa); IF_ADDR_UNLOCK(ifp); return (ifa); } #include /* * Default action when installing a route with a Link Level gateway. * Lookup an appropriate real ifa to point to. * This should be moved to /sys/net/link.c eventually. */ static void link_rtrequest(int cmd, struct rtentry *rt, struct rt_addrinfo *info) { struct ifaddr *ifa, *oifa; struct sockaddr *dst; struct ifnet *ifp; RT_LOCK_ASSERT(rt); if (cmd != RTM_ADD || ((ifa = rt->rt_ifa) == 0) || ((ifp = ifa->ifa_ifp) == 0) || ((dst = rt_key(rt)) == 0)) return; ifa = ifaof_ifpforaddr(dst, ifp); if (ifa) { oifa = rt->rt_ifa; rt->rt_ifa = ifa; ifa_free(oifa); if (ifa->ifa_rtrequest && ifa->ifa_rtrequest != link_rtrequest) ifa->ifa_rtrequest(cmd, rt, info); } } /* * Mark an interface down and notify protocols of * the transition. * NOTE: must be called at splnet or eqivalent. */ static void if_unroute(struct ifnet *ifp, int flag, int fam) { struct ifaddr *ifa; KASSERT(flag == IFF_UP, ("if_unroute: flag != IFF_UP")); ifp->if_flags &= ~flag; getmicrotime(&ifp->if_lastchange); TAILQ_FOREACH(ifa, &ifp->if_addrhead, ifa_link) if (fam == PF_UNSPEC || (fam == ifa->ifa_addr->sa_family)) pfctlinput(PRC_IFDOWN, ifa->ifa_addr); ifp->if_qflush(ifp); #if defined(INET) || defined(INET6) #ifdef DEV_CARP if (ifp->if_carp) carp_carpdev_state(ifp->if_carp); #endif #endif rt_ifmsg(ifp); } /* * Mark an interface up and notify protocols of * the transition. * NOTE: must be called at splnet or eqivalent. */ static void if_route(struct ifnet *ifp, int flag, int fam) { struct ifaddr *ifa; KASSERT(flag == IFF_UP, ("if_route: flag != IFF_UP")); ifp->if_flags |= flag; getmicrotime(&ifp->if_lastchange); TAILQ_FOREACH(ifa, &ifp->if_addrhead, ifa_link) if (fam == PF_UNSPEC || (fam == ifa->ifa_addr->sa_family)) pfctlinput(PRC_IFUP, ifa->ifa_addr); #if defined(INET) || defined(INET6) #ifdef DEV_CARP if (ifp->if_carp) carp_carpdev_state(ifp->if_carp); #endif #endif rt_ifmsg(ifp); #ifdef INET6 in6_if_up(ifp); #endif } void (*vlan_link_state_p)(struct ifnet *); /* XXX: private from if_vlan */ void (*vlan_trunk_cap_p)(struct ifnet *); /* XXX: private from if_vlan */ /* * Handle a change in the interface link state. To avoid LORs * between driver lock and upper layer locks, as well as possible * recursions, we post event to taskqueue, and all job * is done in static do_link_state_change(). */ void if_link_state_change(struct ifnet *ifp, int link_state) { /* Return if state hasn't changed. */ if (ifp->if_link_state == link_state) return; ifp->if_link_state = link_state; taskqueue_enqueue(taskqueue_swi, &ifp->if_linktask); } static void do_link_state_change(void *arg, int pending) { struct ifnet *ifp = (struct ifnet *)arg; int link_state = ifp->if_link_state; CURVNET_SET(ifp->if_vnet); /* Notify that the link state has changed. */ rt_ifmsg(ifp); if (ifp->if_vlantrunk != NULL) (*vlan_link_state_p)(ifp); if ((ifp->if_type == IFT_ETHER || ifp->if_type == IFT_L2VLAN) && IFP2AC(ifp)->ac_netgraph != NULL) (*ng_ether_link_state_p)(ifp, link_state); #if defined(INET) || defined(INET6) #ifdef DEV_CARP if (ifp->if_carp) carp_carpdev_state(ifp->if_carp); #endif #endif if (ifp->if_bridge) { KASSERT(bstp_linkstate_p != NULL,("if_bridge bstp not loaded!")); (*bstp_linkstate_p)(ifp, link_state); } if (ifp->if_lagg) { KASSERT(lagg_linkstate_p != NULL,("if_lagg not loaded!")); (*lagg_linkstate_p)(ifp, link_state); } if (IS_DEFAULT_VNET(curvnet)) devctl_notify("IFNET", ifp->if_xname, (link_state == LINK_STATE_UP) ? "LINK_UP" : "LINK_DOWN", NULL); if (pending > 1) if_printf(ifp, "%d link states coalesced\n", pending); if (log_link_state_change) log(LOG_NOTICE, "%s: link state changed to %s\n", ifp->if_xname, (link_state == LINK_STATE_UP) ? "UP" : "DOWN" ); CURVNET_RESTORE(); } /* * Mark an interface down and notify protocols of * the transition. * NOTE: must be called at splnet or eqivalent. */ void if_down(struct ifnet *ifp) { if_unroute(ifp, IFF_UP, AF_UNSPEC); } /* * Mark an interface up and notify protocols of * the transition. * NOTE: must be called at splnet or eqivalent. */ void if_up(struct ifnet *ifp) { if_route(ifp, IFF_UP, AF_UNSPEC); } /* * Flush an interface queue. */ void if_qflush(struct ifnet *ifp) { struct mbuf *m, *n; struct ifaltq *ifq; ifq = &ifp->if_snd; IFQ_LOCK(ifq); #ifdef ALTQ if (ALTQ_IS_ENABLED(ifq)) ALTQ_PURGE(ifq); #endif n = ifq->ifq_head; while ((m = n) != 0) { n = m->m_act; m_freem(m); } ifq->ifq_head = 0; ifq->ifq_tail = 0; ifq->ifq_len = 0; IFQ_UNLOCK(ifq); } /* * Map interface name to interface structure pointer, with or without * returning a reference. */ struct ifnet * ifunit_ref(const char *name) { struct ifnet *ifp; IFNET_RLOCK_NOSLEEP(); TAILQ_FOREACH(ifp, &V_ifnet, if_link) { if (strncmp(name, ifp->if_xname, IFNAMSIZ) == 0 && !(ifp->if_flags & IFF_DYING)) break; } if (ifp != NULL) if_ref(ifp); IFNET_RUNLOCK_NOSLEEP(); return (ifp); } struct ifnet * ifunit(const char *name) { struct ifnet *ifp; IFNET_RLOCK_NOSLEEP(); TAILQ_FOREACH(ifp, &V_ifnet, if_link) { if (strncmp(name, ifp->if_xname, IFNAMSIZ) == 0) break; } IFNET_RUNLOCK_NOSLEEP(); return (ifp); } /* * Hardware specific interface ioctls. */ static int ifhwioctl(u_long cmd, struct ifnet *ifp, caddr_t data, struct thread *td) { struct ifreq *ifr; struct ifstat *ifs; int error = 0; int new_flags, temp_flags; size_t namelen, onamelen; size_t descrlen; char *descrbuf, *odescrbuf; char new_name[IFNAMSIZ]; struct ifaddr *ifa; struct sockaddr_dl *sdl; ifr = (struct ifreq *)data; switch (cmd) { case SIOCGIFINDEX: ifr->ifr_index = ifp->if_index; break; case SIOCGIFFLAGS: temp_flags = ifp->if_flags | ifp->if_drv_flags; ifr->ifr_flags = temp_flags & 0xffff; ifr->ifr_flagshigh = temp_flags >> 16; break; case SIOCGIFCAP: ifr->ifr_reqcap = ifp->if_capabilities; ifr->ifr_curcap = ifp->if_capenable; break; #ifdef MAC case SIOCGIFMAC: error = mac_ifnet_ioctl_get(td->td_ucred, ifr, ifp); break; #endif case SIOCGIFMETRIC: ifr->ifr_metric = ifp->if_metric; break; case SIOCGIFMTU: ifr->ifr_mtu = ifp->if_mtu; break; case SIOCGIFPHYS: ifr->ifr_phys = ifp->if_physical; break; case SIOCGIFDESCR: error = 0; sx_slock(&ifdescr_sx); - if (ifp->if_description == NULL) { - ifr->ifr_buffer.length = 0; + if (ifp->if_description == NULL) error = ENOMSG; - } else { + else { /* space for terminating nul */ descrlen = strlen(ifp->if_description) + 1; if (ifr->ifr_buffer.length < descrlen) - error = ENAMETOOLONG; + ifr->ifr_buffer.buffer = NULL; else error = copyout(ifp->if_description, ifr->ifr_buffer.buffer, descrlen); ifr->ifr_buffer.length = descrlen; } sx_sunlock(&ifdescr_sx); break; case SIOCSIFDESCR: error = priv_check(td, PRIV_NET_SETIFDESCR); if (error) return (error); /* * Copy only (length-1) bytes to make sure that * if_description is always nul terminated. The * length parameter is supposed to count the * terminating nul in. */ if (ifr->ifr_buffer.length > ifdescr_maxlen) return (ENAMETOOLONG); else if (ifr->ifr_buffer.length == 0) descrbuf = NULL; else { descrbuf = malloc(ifr->ifr_buffer.length, M_IFDESCR, M_WAITOK | M_ZERO); error = copyin(ifr->ifr_buffer.buffer, descrbuf, ifr->ifr_buffer.length - 1); if (error) { free(descrbuf, M_IFDESCR); break; } } sx_xlock(&ifdescr_sx); odescrbuf = ifp->if_description; ifp->if_description = descrbuf; sx_xunlock(&ifdescr_sx); getmicrotime(&ifp->if_lastchange); free(odescrbuf, M_IFDESCR); break; case SIOCSIFFLAGS: error = priv_check(td, PRIV_NET_SETIFFLAGS); if (error) return (error); /* * Currently, no driver owned flags pass the IFF_CANTCHANGE * check, so we don't need special handling here yet. */ new_flags = (ifr->ifr_flags & 0xffff) | (ifr->ifr_flagshigh << 16); if (ifp->if_flags & IFF_SMART) { /* Smart drivers twiddle their own routes */ } else if (ifp->if_flags & IFF_UP && (new_flags & IFF_UP) == 0) { int s = splimp(); if_down(ifp); splx(s); } else if (new_flags & IFF_UP && (ifp->if_flags & IFF_UP) == 0) { int s = splimp(); if_up(ifp); splx(s); } /* See if permanently promiscuous mode bit is about to flip */ if ((ifp->if_flags ^ new_flags) & IFF_PPROMISC) { if (new_flags & IFF_PPROMISC) ifp->if_flags |= IFF_PROMISC; else if (ifp->if_pcount == 0) ifp->if_flags &= ~IFF_PROMISC; log(LOG_INFO, "%s: permanently promiscuous mode %s\n", ifp->if_xname, (new_flags & IFF_PPROMISC) ? "enabled" : "disabled"); } ifp->if_flags = (ifp->if_flags & IFF_CANTCHANGE) | (new_flags &~ IFF_CANTCHANGE); if (ifp->if_ioctl) { (void) (*ifp->if_ioctl)(ifp, cmd, data); } getmicrotime(&ifp->if_lastchange); break; case SIOCSIFCAP: error = priv_check(td, PRIV_NET_SETIFCAP); if (error) return (error); if (ifp->if_ioctl == NULL) return (EOPNOTSUPP); if (ifr->ifr_reqcap & ~ifp->if_capabilities) return (EINVAL); error = (*ifp->if_ioctl)(ifp, cmd, data); if (error == 0) getmicrotime(&ifp->if_lastchange); break; #ifdef MAC case SIOCSIFMAC: error = mac_ifnet_ioctl_set(td->td_ucred, ifr, ifp); break; #endif case SIOCSIFNAME: error = priv_check(td, PRIV_NET_SETIFNAME); if (error) return (error); error = copyinstr(ifr->ifr_data, new_name, IFNAMSIZ, NULL); if (error != 0) return (error); if (new_name[0] == '\0') return (EINVAL); if (ifunit(new_name) != NULL) return (EEXIST); /* * XXX: Locking. Nothing else seems to lock if_flags, * and there are numerous other races with the * ifunit() checks not being atomic with namespace * changes (renames, vmoves, if_attach, etc). */ ifp->if_flags |= IFF_RENAMING; /* Announce the departure of the interface. */ rt_ifannouncemsg(ifp, IFAN_DEPARTURE); EVENTHANDLER_INVOKE(ifnet_departure_event, ifp); log(LOG_INFO, "%s: changing name to '%s'\n", ifp->if_xname, new_name); strlcpy(ifp->if_xname, new_name, sizeof(ifp->if_xname)); ifa = ifp->if_addr; IFA_LOCK(ifa); sdl = (struct sockaddr_dl *)ifa->ifa_addr; namelen = strlen(new_name); onamelen = sdl->sdl_nlen; /* * Move the address if needed. This is safe because we * allocate space for a name of length IFNAMSIZ when we * create this in if_attach(). */ if (namelen != onamelen) { bcopy(sdl->sdl_data + onamelen, sdl->sdl_data + namelen, sdl->sdl_alen); } bcopy(new_name, sdl->sdl_data, namelen); sdl->sdl_nlen = namelen; sdl = (struct sockaddr_dl *)ifa->ifa_netmask; bzero(sdl->sdl_data, onamelen); while (namelen != 0) sdl->sdl_data[--namelen] = 0xff; IFA_UNLOCK(ifa); EVENTHANDLER_INVOKE(ifnet_arrival_event, ifp); /* Announce the return of the interface. */ rt_ifannouncemsg(ifp, IFAN_ARRIVAL); ifp->if_flags &= ~IFF_RENAMING; break; #ifdef VIMAGE case SIOCSIFVNET: error = priv_check(td, PRIV_NET_SETIFVNET); if (error) return (error); error = if_vmove_loan(td, ifp, ifr->ifr_name, ifr->ifr_jid); break; #endif case SIOCSIFMETRIC: error = priv_check(td, PRIV_NET_SETIFMETRIC); if (error) return (error); ifp->if_metric = ifr->ifr_metric; getmicrotime(&ifp->if_lastchange); break; case SIOCSIFPHYS: error = priv_check(td, PRIV_NET_SETIFPHYS); if (error) return (error); if (ifp->if_ioctl == NULL) return (EOPNOTSUPP); error = (*ifp->if_ioctl)(ifp, cmd, data); if (error == 0) getmicrotime(&ifp->if_lastchange); break; case SIOCSIFMTU: { u_long oldmtu = ifp->if_mtu; error = priv_check(td, PRIV_NET_SETIFMTU); if (error) return (error); if (ifr->ifr_mtu < IF_MINMTU || ifr->ifr_mtu > IF_MAXMTU) return (EINVAL); if (ifp->if_ioctl == NULL) return (EOPNOTSUPP); error = (*ifp->if_ioctl)(ifp, cmd, data); if (error == 0) { getmicrotime(&ifp->if_lastchange); rt_ifmsg(ifp); } /* * If the link MTU changed, do network layer specific procedure. */ if (ifp->if_mtu != oldmtu) { #ifdef INET6 nd6_setmtu(ifp); #endif } break; } case SIOCADDMULTI: case SIOCDELMULTI: if (cmd == SIOCADDMULTI) error = priv_check(td, PRIV_NET_ADDMULTI); else error = priv_check(td, PRIV_NET_DELMULTI); if (error) return (error); /* Don't allow group membership on non-multicast interfaces. */ if ((ifp->if_flags & IFF_MULTICAST) == 0) return (EOPNOTSUPP); /* Don't let users screw up protocols' entries. */ if (ifr->ifr_addr.sa_family != AF_LINK) return (EINVAL); if (cmd == SIOCADDMULTI) { struct ifmultiaddr *ifma; /* * Userland is only permitted to join groups once * via the if_addmulti() KPI, because it cannot hold * struct ifmultiaddr * between calls. It may also * lose a race while we check if the membership * already exists. */ IF_ADDR_LOCK(ifp); ifma = if_findmulti(ifp, &ifr->ifr_addr); IF_ADDR_UNLOCK(ifp); if (ifma != NULL) error = EADDRINUSE; else error = if_addmulti(ifp, &ifr->ifr_addr, &ifma); } else { error = if_delmulti(ifp, &ifr->ifr_addr); } if (error == 0) getmicrotime(&ifp->if_lastchange); break; case SIOCSIFPHYADDR: case SIOCDIFPHYADDR: #ifdef INET6 case SIOCSIFPHYADDR_IN6: #endif case SIOCSLIFPHYADDR: case SIOCSIFMEDIA: case SIOCSIFGENERIC: error = priv_check(td, PRIV_NET_HWIOCTL); if (error) return (error); if (ifp->if_ioctl == NULL) return (EOPNOTSUPP); error = (*ifp->if_ioctl)(ifp, cmd, data); if (error == 0) getmicrotime(&ifp->if_lastchange); break; case SIOCGIFSTATUS: ifs = (struct ifstat *)data; ifs->ascii[0] = '\0'; case SIOCGIFPSRCADDR: case SIOCGIFPDSTADDR: case SIOCGLIFPHYADDR: case SIOCGIFMEDIA: case SIOCGIFGENERIC: if (ifp->if_ioctl == NULL) return (EOPNOTSUPP); error = (*ifp->if_ioctl)(ifp, cmd, data); break; case SIOCSIFLLADDR: error = priv_check(td, PRIV_NET_SETLLADDR); if (error) return (error); error = if_setlladdr(ifp, ifr->ifr_addr.sa_data, ifr->ifr_addr.sa_len); EVENTHANDLER_INVOKE(iflladdr_event, ifp); break; case SIOCAIFGROUP: { struct ifgroupreq *ifgr = (struct ifgroupreq *)ifr; error = priv_check(td, PRIV_NET_ADDIFGROUP); if (error) return (error); if ((error = if_addgroup(ifp, ifgr->ifgr_group))) return (error); break; } case SIOCGIFGROUP: if ((error = if_getgroup((struct ifgroupreq *)ifr, ifp))) return (error); break; case SIOCDIFGROUP: { struct ifgroupreq *ifgr = (struct ifgroupreq *)ifr; error = priv_check(td, PRIV_NET_DELIFGROUP); if (error) return (error); if ((error = if_delgroup(ifp, ifgr->ifgr_group))) return (error); break; } default: error = ENOIOCTL; break; } return (error); } /* * Interface ioctls. */ int ifioctl(struct socket *so, u_long cmd, caddr_t data, struct thread *td) { struct ifnet *ifp; struct ifreq *ifr; int error; int oif_flags; switch (cmd) { case SIOCGIFCONF: case OSIOCGIFCONF: #ifdef __amd64__ case SIOCGIFCONF32: #endif return (ifconf(cmd, data)); } ifr = (struct ifreq *)data; switch (cmd) { #ifdef VIMAGE case SIOCSIFRVNET: error = priv_check(td, PRIV_NET_SETIFVNET); if (error) return (error); return (if_vmove_reclaim(td, ifr->ifr_name, ifr->ifr_jid)); #endif case SIOCIFCREATE: case SIOCIFCREATE2: error = priv_check(td, PRIV_NET_IFCREATE); if (error) return (error); return (if_clone_create(ifr->ifr_name, sizeof(ifr->ifr_name), cmd == SIOCIFCREATE2 ? ifr->ifr_data : NULL)); case SIOCIFDESTROY: error = priv_check(td, PRIV_NET_IFDESTROY); if (error) return (error); return if_clone_destroy(ifr->ifr_name); case SIOCIFGCLONERS: return (if_clone_list((struct if_clonereq *)data)); case SIOCGIFGMEMB: return (if_getgroupmembers((struct ifgroupreq *)data)); } ifp = ifunit_ref(ifr->ifr_name); if (ifp == NULL) return (ENXIO); error = ifhwioctl(cmd, ifp, data, td); if (error != ENOIOCTL) { if_rele(ifp); return (error); } oif_flags = ifp->if_flags; if (so->so_proto == NULL) { if_rele(ifp); return (EOPNOTSUPP); } #ifndef COMPAT_43 error = ((*so->so_proto->pr_usrreqs->pru_control)(so, cmd, data, ifp, td)); if (error == EOPNOTSUPP && ifp != NULL && ifp->if_ioctl != NULL) error = (*ifp->if_ioctl)(ifp, cmd, data); #else { u_long ocmd = cmd; switch (cmd) { case SIOCSIFDSTADDR: case SIOCSIFADDR: case SIOCSIFBRDADDR: case SIOCSIFNETMASK: #if BYTE_ORDER != BIG_ENDIAN if (ifr->ifr_addr.sa_family == 0 && ifr->ifr_addr.sa_len < 16) { ifr->ifr_addr.sa_family = ifr->ifr_addr.sa_len; ifr->ifr_addr.sa_len = 16; } #else if (ifr->ifr_addr.sa_len == 0) ifr->ifr_addr.sa_len = 16; #endif break; case OSIOCGIFADDR: cmd = SIOCGIFADDR; break; case OSIOCGIFDSTADDR: cmd = SIOCGIFDSTADDR; break; case OSIOCGIFBRDADDR: cmd = SIOCGIFBRDADDR; break; case OSIOCGIFNETMASK: cmd = SIOCGIFNETMASK; } error = ((*so->so_proto->pr_usrreqs->pru_control)(so, cmd, data, ifp, td)); if (error == EOPNOTSUPP && ifp != NULL && ifp->if_ioctl != NULL) error = (*ifp->if_ioctl)(ifp, cmd, data); switch (ocmd) { case OSIOCGIFADDR: case OSIOCGIFDSTADDR: case OSIOCGIFBRDADDR: case OSIOCGIFNETMASK: *(u_short *)&ifr->ifr_addr = ifr->ifr_addr.sa_family; } } #endif /* COMPAT_43 */ if ((oif_flags ^ ifp->if_flags) & IFF_UP) { #ifdef INET6 if (ifp->if_flags & IFF_UP) { int s = splimp(); in6_if_up(ifp); splx(s); } #endif } if_rele(ifp); return (error); } /* * The code common to handling reference counted flags, * e.g., in ifpromisc() and if_allmulti(). * The "pflag" argument can specify a permanent mode flag to check, * such as IFF_PPROMISC for promiscuous mode; should be 0 if none. * * Only to be used on stack-owned flags, not driver-owned flags. */ static int if_setflag(struct ifnet *ifp, int flag, int pflag, int *refcount, int onswitch) { struct ifreq ifr; int error; int oldflags, oldcount; /* Sanity checks to catch programming errors */ KASSERT((flag & (IFF_DRV_OACTIVE|IFF_DRV_RUNNING)) == 0, ("%s: setting driver-owned flag %d", __func__, flag)); if (onswitch) KASSERT(*refcount >= 0, ("%s: increment negative refcount %d for flag %d", __func__, *refcount, flag)); else KASSERT(*refcount > 0, ("%s: decrement non-positive refcount %d for flag %d", __func__, *refcount, flag)); /* In case this mode is permanent, just touch refcount */ if (ifp->if_flags & pflag) { *refcount += onswitch ? 1 : -1; return (0); } /* Save ifnet parameters for if_ioctl() may fail */ oldcount = *refcount; oldflags = ifp->if_flags; /* * See if we aren't the only and touching refcount is enough. * Actually toggle interface flag if we are the first or last. */ if (onswitch) { if ((*refcount)++) return (0); ifp->if_flags |= flag; } else { if (--(*refcount)) return (0); ifp->if_flags &= ~flag; } /* Call down the driver since we've changed interface flags */ if (ifp->if_ioctl == NULL) { error = EOPNOTSUPP; goto recover; } ifr.ifr_flags = ifp->if_flags & 0xffff; ifr.ifr_flagshigh = ifp->if_flags >> 16; error = (*ifp->if_ioctl)(ifp, SIOCSIFFLAGS, (caddr_t)&ifr); if (error) goto recover; /* Notify userland that interface flags have changed */ rt_ifmsg(ifp); return (0); recover: /* Recover after driver error */ *refcount = oldcount; ifp->if_flags = oldflags; return (error); } /* * Set/clear promiscuous mode on interface ifp based on the truth value * of pswitch. The calls are reference counted so that only the first * "on" request actually has an effect, as does the final "off" request. * Results are undefined if the "off" and "on" requests are not matched. */ int ifpromisc(struct ifnet *ifp, int pswitch) { int error; int oldflags = ifp->if_flags; error = if_setflag(ifp, IFF_PROMISC, IFF_PPROMISC, &ifp->if_pcount, pswitch); /* If promiscuous mode status has changed, log a message */ if (error == 0 && ((ifp->if_flags ^ oldflags) & IFF_PROMISC)) log(LOG_INFO, "%s: promiscuous mode %s\n", ifp->if_xname, (ifp->if_flags & IFF_PROMISC) ? "enabled" : "disabled"); return (error); } /* * Return interface configuration * of system. List may be used * in later ioctl's (above) to get * other information. */ /*ARGSUSED*/ static int ifconf(u_long cmd, caddr_t data) { struct ifconf *ifc = (struct ifconf *)data; #ifdef __amd64__ struct ifconf32 *ifc32 = (struct ifconf32 *)data; struct ifconf ifc_swab; #endif struct ifnet *ifp; struct ifaddr *ifa; struct ifreq ifr; struct sbuf *sb; int error, full = 0, valid_len, max_len; #ifdef __amd64__ if (cmd == SIOCGIFCONF32) { ifc_swab.ifc_len = ifc32->ifc_len; ifc_swab.ifc_buf = (caddr_t)(uintptr_t)ifc32->ifc_buf; ifc = &ifc_swab; } #endif /* Limit initial buffer size to MAXPHYS to avoid DoS from userspace. */ max_len = MAXPHYS - 1; /* Prevent hostile input from being able to crash the system */ if (ifc->ifc_len <= 0) return (EINVAL); again: if (ifc->ifc_len <= max_len) { max_len = ifc->ifc_len; full = 1; } sb = sbuf_new(NULL, NULL, max_len + 1, SBUF_FIXEDLEN); max_len = 0; valid_len = 0; IFNET_RLOCK(); TAILQ_FOREACH(ifp, &V_ifnet, if_link) { int addrs; /* * Zero the ifr_name buffer to make sure we don't * disclose the contents of the stack. */ memset(ifr.ifr_name, 0, sizeof(ifr.ifr_name)); if (strlcpy(ifr.ifr_name, ifp->if_xname, sizeof(ifr.ifr_name)) >= sizeof(ifr.ifr_name)) { sbuf_delete(sb); IFNET_RUNLOCK(); return (ENAMETOOLONG); } addrs = 0; IF_ADDR_LOCK(ifp); TAILQ_FOREACH(ifa, &ifp->if_addrhead, ifa_link) { struct sockaddr *sa = ifa->ifa_addr; if (prison_if(curthread->td_ucred, sa) != 0) continue; addrs++; #ifdef COMPAT_43 if (cmd == OSIOCGIFCONF) { struct osockaddr *osa = (struct osockaddr *)&ifr.ifr_addr; ifr.ifr_addr = *sa; osa->sa_family = sa->sa_family; sbuf_bcat(sb, &ifr, sizeof(ifr)); max_len += sizeof(ifr); } else #endif if (sa->sa_len <= sizeof(*sa)) { ifr.ifr_addr = *sa; sbuf_bcat(sb, &ifr, sizeof(ifr)); max_len += sizeof(ifr); } else { sbuf_bcat(sb, &ifr, offsetof(struct ifreq, ifr_addr)); max_len += offsetof(struct ifreq, ifr_addr); sbuf_bcat(sb, sa, sa->sa_len); max_len += sa->sa_len; } if (!sbuf_overflowed(sb)) valid_len = sbuf_len(sb); } IF_ADDR_UNLOCK(ifp); if (addrs == 0) { bzero((caddr_t)&ifr.ifr_addr, sizeof(ifr.ifr_addr)); sbuf_bcat(sb, &ifr, sizeof(ifr)); max_len += sizeof(ifr); if (!sbuf_overflowed(sb)) valid_len = sbuf_len(sb); } } IFNET_RUNLOCK(); /* * If we didn't allocate enough space (uncommon), try again. If * we have already allocated as much space as we are allowed, * return what we've got. */ if (valid_len != max_len && !full) { sbuf_delete(sb); goto again; } ifc->ifc_len = valid_len; #ifdef __amd64__ if (cmd == SIOCGIFCONF32) ifc32->ifc_len = valid_len; #endif sbuf_finish(sb); error = copyout(sbuf_data(sb), ifc->ifc_req, ifc->ifc_len); sbuf_delete(sb); return (error); } /* * Just like ifpromisc(), but for all-multicast-reception mode. */ int if_allmulti(struct ifnet *ifp, int onswitch) { return (if_setflag(ifp, IFF_ALLMULTI, 0, &ifp->if_amcount, onswitch)); } struct ifmultiaddr * if_findmulti(struct ifnet *ifp, struct sockaddr *sa) { struct ifmultiaddr *ifma; IF_ADDR_LOCK_ASSERT(ifp); TAILQ_FOREACH(ifma, &ifp->if_multiaddrs, ifma_link) { if (sa->sa_family == AF_LINK) { if (sa_dl_equal(ifma->ifma_addr, sa)) break; } else { if (sa_equal(ifma->ifma_addr, sa)) break; } } return ifma; } /* * Allocate a new ifmultiaddr and initialize based on passed arguments. We * make copies of passed sockaddrs. The ifmultiaddr will not be added to * the ifnet multicast address list here, so the caller must do that and * other setup work (such as notifying the device driver). The reference * count is initialized to 1. */ static struct ifmultiaddr * if_allocmulti(struct ifnet *ifp, struct sockaddr *sa, struct sockaddr *llsa, int mflags) { struct ifmultiaddr *ifma; struct sockaddr *dupsa; ifma = malloc(sizeof *ifma, M_IFMADDR, mflags | M_ZERO); if (ifma == NULL) return (NULL); dupsa = malloc(sa->sa_len, M_IFMADDR, mflags); if (dupsa == NULL) { free(ifma, M_IFMADDR); return (NULL); } bcopy(sa, dupsa, sa->sa_len); ifma->ifma_addr = dupsa; ifma->ifma_ifp = ifp; ifma->ifma_refcount = 1; ifma->ifma_protospec = NULL; if (llsa == NULL) { ifma->ifma_lladdr = NULL; return (ifma); } dupsa = malloc(llsa->sa_len, M_IFMADDR, mflags); if (dupsa == NULL) { free(ifma->ifma_addr, M_IFMADDR); free(ifma, M_IFMADDR); return (NULL); } bcopy(llsa, dupsa, llsa->sa_len); ifma->ifma_lladdr = dupsa; return (ifma); } /* * if_freemulti: free ifmultiaddr structure and possibly attached related * addresses. The caller is responsible for implementing reference * counting, notifying the driver, handling routing messages, and releasing * any dependent link layer state. */ static void if_freemulti(struct ifmultiaddr *ifma) { KASSERT(ifma->ifma_refcount == 0, ("if_freemulti: refcount %d", ifma->ifma_refcount)); KASSERT(ifma->ifma_protospec == NULL, ("if_freemulti: protospec not NULL")); if (ifma->ifma_lladdr != NULL) free(ifma->ifma_lladdr, M_IFMADDR); free(ifma->ifma_addr, M_IFMADDR); free(ifma, M_IFMADDR); } /* * Register an additional multicast address with a network interface. * * - If the address is already present, bump the reference count on the * address and return. * - If the address is not link-layer, look up a link layer address. * - Allocate address structures for one or both addresses, and attach to the * multicast address list on the interface. If automatically adding a link * layer address, the protocol address will own a reference to the link * layer address, to be freed when it is freed. * - Notify the network device driver of an addition to the multicast address * list. * * 'sa' points to caller-owned memory with the desired multicast address. * * 'retifma' will be used to return a pointer to the resulting multicast * address reference, if desired. */ int if_addmulti(struct ifnet *ifp, struct sockaddr *sa, struct ifmultiaddr **retifma) { struct ifmultiaddr *ifma, *ll_ifma; struct sockaddr *llsa; int error; /* * If the address is already present, return a new reference to it; * otherwise, allocate storage and set up a new address. */ IF_ADDR_LOCK(ifp); ifma = if_findmulti(ifp, sa); if (ifma != NULL) { ifma->ifma_refcount++; if (retifma != NULL) *retifma = ifma; IF_ADDR_UNLOCK(ifp); return (0); } /* * The address isn't already present; resolve the protocol address * into a link layer address, and then look that up, bump its * refcount or allocate an ifma for that also. If 'llsa' was * returned, we will need to free it later. */ llsa = NULL; ll_ifma = NULL; if (ifp->if_resolvemulti != NULL) { error = ifp->if_resolvemulti(ifp, &llsa, sa); if (error) goto unlock_out; } /* * Allocate the new address. Don't hook it up yet, as we may also * need to allocate a link layer multicast address. */ ifma = if_allocmulti(ifp, sa, llsa, M_NOWAIT); if (ifma == NULL) { error = ENOMEM; goto free_llsa_out; } /* * If a link layer address is found, we'll need to see if it's * already present in the address list, or allocate is as well. * When this block finishes, the link layer address will be on the * list. */ if (llsa != NULL) { ll_ifma = if_findmulti(ifp, llsa); if (ll_ifma == NULL) { ll_ifma = if_allocmulti(ifp, llsa, NULL, M_NOWAIT); if (ll_ifma == NULL) { --ifma->ifma_refcount; if_freemulti(ifma); error = ENOMEM; goto free_llsa_out; } TAILQ_INSERT_HEAD(&ifp->if_multiaddrs, ll_ifma, ifma_link); } else ll_ifma->ifma_refcount++; ifma->ifma_llifma = ll_ifma; } /* * We now have a new multicast address, ifma, and possibly a new or * referenced link layer address. Add the primary address to the * ifnet address list. */ TAILQ_INSERT_HEAD(&ifp->if_multiaddrs, ifma, ifma_link); if (retifma != NULL) *retifma = ifma; /* * Must generate the message while holding the lock so that 'ifma' * pointer is still valid. */ rt_newmaddrmsg(RTM_NEWMADDR, ifma); IF_ADDR_UNLOCK(ifp); /* * We are certain we have added something, so call down to the * interface to let them know about it. */ if (ifp->if_ioctl != NULL) { (void) (*ifp->if_ioctl)(ifp, SIOCADDMULTI, 0); } if (llsa != NULL) free(llsa, M_IFMADDR); return (0); free_llsa_out: if (llsa != NULL) free(llsa, M_IFMADDR); unlock_out: IF_ADDR_UNLOCK(ifp); return (error); } /* * Delete a multicast group membership by network-layer group address. * * Returns ENOENT if the entry could not be found. If ifp no longer * exists, results are undefined. This entry point should only be used * from subsystems which do appropriate locking to hold ifp for the * duration of the call. * Network-layer protocol domains must use if_delmulti_ifma(). */ int if_delmulti(struct ifnet *ifp, struct sockaddr *sa) { struct ifmultiaddr *ifma; int lastref; #ifdef INVARIANTS struct ifnet *oifp; IFNET_RLOCK_NOSLEEP(); TAILQ_FOREACH(oifp, &V_ifnet, if_link) if (ifp == oifp) break; if (ifp != oifp) ifp = NULL; IFNET_RUNLOCK_NOSLEEP(); KASSERT(ifp != NULL, ("%s: ifnet went away", __func__)); #endif if (ifp == NULL) return (ENOENT); IF_ADDR_LOCK(ifp); lastref = 0; ifma = if_findmulti(ifp, sa); if (ifma != NULL) lastref = if_delmulti_locked(ifp, ifma, 0); IF_ADDR_UNLOCK(ifp); if (ifma == NULL) return (ENOENT); if (lastref && ifp->if_ioctl != NULL) { (void)(*ifp->if_ioctl)(ifp, SIOCDELMULTI, 0); } return (0); } /* * Delete all multicast group membership for an interface. * Should be used to quickly flush all multicast filters. */ void if_delallmulti(struct ifnet *ifp) { struct ifmultiaddr *ifma; struct ifmultiaddr *next; IF_ADDR_LOCK(ifp); TAILQ_FOREACH_SAFE(ifma, &ifp->if_multiaddrs, ifma_link, next) if_delmulti_locked(ifp, ifma, 0); IF_ADDR_UNLOCK(ifp); } /* * Delete a multicast group membership by group membership pointer. * Network-layer protocol domains must use this routine. * * It is safe to call this routine if the ifp disappeared. */ void if_delmulti_ifma(struct ifmultiaddr *ifma) { struct ifnet *ifp; int lastref; ifp = ifma->ifma_ifp; #ifdef DIAGNOSTIC if (ifp == NULL) { printf("%s: ifma_ifp seems to be detached\n", __func__); } else { struct ifnet *oifp; IFNET_RLOCK_NOSLEEP(); TAILQ_FOREACH(oifp, &V_ifnet, if_link) if (ifp == oifp) break; if (ifp != oifp) { printf("%s: ifnet %p disappeared\n", __func__, ifp); ifp = NULL; } IFNET_RUNLOCK_NOSLEEP(); } #endif /* * If and only if the ifnet instance exists: Acquire the address lock. */ if (ifp != NULL) IF_ADDR_LOCK(ifp); lastref = if_delmulti_locked(ifp, ifma, 0); if (ifp != NULL) { /* * If and only if the ifnet instance exists: * Release the address lock. * If the group was left: update the hardware hash filter. */ IF_ADDR_UNLOCK(ifp); if (lastref && ifp->if_ioctl != NULL) { (void)(*ifp->if_ioctl)(ifp, SIOCDELMULTI, 0); } } } /* * Perform deletion of network-layer and/or link-layer multicast address. * * Return 0 if the reference count was decremented. * Return 1 if the final reference was released, indicating that the * hardware hash filter should be reprogrammed. */ static int if_delmulti_locked(struct ifnet *ifp, struct ifmultiaddr *ifma, int detaching) { struct ifmultiaddr *ll_ifma; if (ifp != NULL && ifma->ifma_ifp != NULL) { KASSERT(ifma->ifma_ifp == ifp, ("%s: inconsistent ifp %p", __func__, ifp)); IF_ADDR_LOCK_ASSERT(ifp); } ifp = ifma->ifma_ifp; /* * If the ifnet is detaching, null out references to ifnet, * so that upper protocol layers will notice, and not attempt * to obtain locks for an ifnet which no longer exists. The * routing socket announcement must happen before the ifnet * instance is detached from the system. */ if (detaching) { #ifdef DIAGNOSTIC printf("%s: detaching ifnet instance %p\n", __func__, ifp); #endif /* * ifp may already be nulled out if we are being reentered * to delete the ll_ifma. */ if (ifp != NULL) { rt_newmaddrmsg(RTM_DELMADDR, ifma); ifma->ifma_ifp = NULL; } } if (--ifma->ifma_refcount > 0) return 0; /* * If this ifma is a network-layer ifma, a link-layer ifma may * have been associated with it. Release it first if so. */ ll_ifma = ifma->ifma_llifma; if (ll_ifma != NULL) { KASSERT(ifma->ifma_lladdr != NULL, ("%s: llifma w/o lladdr", __func__)); if (detaching) ll_ifma->ifma_ifp = NULL; /* XXX */ if (--ll_ifma->ifma_refcount == 0) { if (ifp != NULL) { TAILQ_REMOVE(&ifp->if_multiaddrs, ll_ifma, ifma_link); } if_freemulti(ll_ifma); } } if (ifp != NULL) TAILQ_REMOVE(&ifp->if_multiaddrs, ifma, ifma_link); if_freemulti(ifma); /* * The last reference to this instance of struct ifmultiaddr * was released; the hardware should be notified of this change. */ return 1; } /* * Set the link layer address on an interface. * * At this time we only support certain types of interfaces, * and we don't allow the length of the address to change. */ int if_setlladdr(struct ifnet *ifp, const u_char *lladdr, int len) { struct sockaddr_dl *sdl; struct ifaddr *ifa; struct ifreq ifr; IF_ADDR_LOCK(ifp); ifa = ifp->if_addr; if (ifa == NULL) { IF_ADDR_UNLOCK(ifp); return (EINVAL); } ifa_ref(ifa); IF_ADDR_UNLOCK(ifp); sdl = (struct sockaddr_dl *)ifa->ifa_addr; if (sdl == NULL) { ifa_free(ifa); return (EINVAL); } if (len != sdl->sdl_alen) { /* don't allow length to change */ ifa_free(ifa); return (EINVAL); } switch (ifp->if_type) { case IFT_ETHER: case IFT_FDDI: case IFT_XETHER: case IFT_ISO88025: case IFT_L2VLAN: case IFT_BRIDGE: case IFT_ARCNET: case IFT_IEEE8023ADLAG: case IFT_IEEE80211: bcopy(lladdr, LLADDR(sdl), len); ifa_free(ifa); break; default: ifa_free(ifa); return (ENODEV); } /* * If the interface is already up, we need * to re-init it in order to reprogram its * address filter. */ if ((ifp->if_flags & IFF_UP) != 0) { if (ifp->if_ioctl) { ifp->if_flags &= ~IFF_UP; ifr.ifr_flags = ifp->if_flags & 0xffff; ifr.ifr_flagshigh = ifp->if_flags >> 16; (*ifp->if_ioctl)(ifp, SIOCSIFFLAGS, (caddr_t)&ifr); ifp->if_flags |= IFF_UP; ifr.ifr_flags = ifp->if_flags & 0xffff; ifr.ifr_flagshigh = ifp->if_flags >> 16; (*ifp->if_ioctl)(ifp, SIOCSIFFLAGS, (caddr_t)&ifr); } #ifdef INET /* * Also send gratuitous ARPs to notify other nodes about * the address change. */ TAILQ_FOREACH(ifa, &ifp->if_addrhead, ifa_link) { if (ifa->ifa_addr->sa_family == AF_INET) arp_ifinit(ifp, ifa); } #endif } return (0); } /* * The name argument must be a pointer to storage which will last as * long as the interface does. For physical devices, the result of * device_get_name(dev) is a good choice and for pseudo-devices a * static string works well. */ void if_initname(struct ifnet *ifp, const char *name, int unit) { ifp->if_dname = name; ifp->if_dunit = unit; if (unit != IF_DUNIT_NONE) snprintf(ifp->if_xname, IFNAMSIZ, "%s%d", name, unit); else strlcpy(ifp->if_xname, name, IFNAMSIZ); } int if_printf(struct ifnet *ifp, const char * fmt, ...) { va_list ap; int retval; retval = printf("%s: ", ifp->if_xname); va_start(ap, fmt); retval += vprintf(fmt, ap); va_end(ap); return (retval); } void if_start(struct ifnet *ifp) { (*(ifp)->if_start)(ifp); } /* * Backwards compatibility interface for drivers * that have not implemented it */ static int if_transmit(struct ifnet *ifp, struct mbuf *m) { int error; IFQ_HANDOFF(ifp, m, error); return (error); } int if_handoff(struct ifqueue *ifq, struct mbuf *m, struct ifnet *ifp, int adjust) { int active = 0; IF_LOCK(ifq); if (_IF_QFULL(ifq)) { _IF_DROP(ifq); IF_UNLOCK(ifq); m_freem(m); return (0); } if (ifp != NULL) { ifp->if_obytes += m->m_pkthdr.len + adjust; if (m->m_flags & (M_BCAST|M_MCAST)) ifp->if_omcasts++; active = ifp->if_drv_flags & IFF_DRV_OACTIVE; } _IF_ENQUEUE(ifq, m); IF_UNLOCK(ifq); if (ifp != NULL && !active) (*(ifp)->if_start)(ifp); return (1); } void if_register_com_alloc(u_char type, if_com_alloc_t *a, if_com_free_t *f) { KASSERT(if_com_alloc[type] == NULL, ("if_register_com_alloc: %d already registered", type)); KASSERT(if_com_free[type] == NULL, ("if_register_com_alloc: %d free already registered", type)); if_com_alloc[type] = a; if_com_free[type] = f; } void if_deregister_com_alloc(u_char type) { KASSERT(if_com_alloc[type] != NULL, ("if_deregister_com_alloc: %d not registered", type)); KASSERT(if_com_free[type] != NULL, ("if_deregister_com_alloc: %d free not registered", type)); if_com_alloc[type] = NULL; if_com_free[type] = NULL; } #ifdef DDB static void if_show_ifnet(struct ifnet *ifp) { if (ifp == NULL) return; db_printf("%s:\n", ifp->if_xname); #define IF_DB_PRINTF(f, e) db_printf(" %s = " f "\n", #e, ifp->e); IF_DB_PRINTF("%s", if_dname); IF_DB_PRINTF("%d", if_dunit); IF_DB_PRINTF("%s", if_description); IF_DB_PRINTF("%u", if_index); IF_DB_PRINTF("%u", if_refcount); IF_DB_PRINTF("%d", if_index_reserved); IF_DB_PRINTF("%p", if_softc); IF_DB_PRINTF("%p", if_l2com); IF_DB_PRINTF("%p", if_vnet); IF_DB_PRINTF("%p", if_home_vnet); IF_DB_PRINTF("%p", if_addr); IF_DB_PRINTF("%p", if_llsoftc); IF_DB_PRINTF("%p", if_label); IF_DB_PRINTF("%u", if_pcount); IF_DB_PRINTF("0x%08x", if_flags); IF_DB_PRINTF("0x%08x", if_drv_flags); IF_DB_PRINTF("0x%08x", if_capabilities); IF_DB_PRINTF("0x%08x", if_capenable); IF_DB_PRINTF("%p", if_snd.ifq_head); IF_DB_PRINTF("%p", if_snd.ifq_tail); IF_DB_PRINTF("%d", if_snd.ifq_len); IF_DB_PRINTF("%d", if_snd.ifq_maxlen); IF_DB_PRINTF("%d", if_snd.ifq_drops); IF_DB_PRINTF("%p", if_snd.ifq_drv_head); IF_DB_PRINTF("%p", if_snd.ifq_drv_tail); IF_DB_PRINTF("%d", if_snd.ifq_drv_len); IF_DB_PRINTF("%d", if_snd.ifq_drv_maxlen); IF_DB_PRINTF("%d", if_snd.altq_type); IF_DB_PRINTF("%x", if_snd.altq_flags); #undef IF_DB_PRINTF } DB_SHOW_COMMAND(ifnet, db_show_ifnet) { if (!have_addr) { db_printf("usage: show ifnet \n"); return; } if_show_ifnet((struct ifnet *)addr); } DB_SHOW_ALL_COMMAND(ifnets, db_show_all_ifnets) { VNET_ITERATOR_DECL(vnet_iter); struct ifnet *ifp; u_short idx; VNET_FOREACH(vnet_iter) { CURVNET_SET_QUIET(vnet_iter); #ifdef VIMAGE db_printf("vnet=%p\n", curvnet); #endif for (idx = 1; idx <= V_if_index; idx++) { ifp = V_ifindex_table[idx].ife_ifnet; if (ifp == NULL) continue; db_printf( "%20s ifp=%p\n", ifp->if_xname, ifp); if (db_pager_quit) break; } CURVNET_RESTORE(); } } #endif Index: user/imp/tbemd/sys/net/vnet.c =================================================================== --- user/imp/tbemd/sys/net/vnet.c (revision 206684) +++ user/imp/tbemd/sys/net/vnet.c (revision 206685) @@ -1,827 +1,818 @@ /*- * Copyright (c) 2004-2009 University of Zagreb * Copyright (c) 2006-2009 FreeBSD Foundation * All rights reserved. * * This software was developed by the University of Zagreb and the * FreeBSD Foundation under sponsorship by the Stichting NLnet and the * FreeBSD Foundation. * * Copyright (c) 2009 Jeffrey Roberson * Copyright (c) 2009 Robert N. M. Watson * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #include __FBSDID("$FreeBSD$"); #include "opt_ddb.h" #include "opt_kdb.h" #include "opt_kdtrace.h" #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #ifdef DDB #include #include #endif #include #include #include /*- * This file implements core functions for virtual network stacks: * * - Virtual network stack management functions. * * - Virtual network stack memory allocator, which virtualizes global * variables in the network stack * * - Virtualized SYSINIT's/SYSUNINIT's, which allow network stack subsystems * to register startup/shutdown events to be run for each virtual network * stack instance. */ MALLOC_DEFINE(M_VNET, "vnet", "network stack control block"); /* * The virtual network stack list has two read-write locks, one sleepable and * the other not, so that the list can be stablized and walked in a variety * of network stack contexts. Both must be acquired exclusively to modify * the list, but a read lock of either lock is sufficient to walk the list. */ struct rwlock vnet_rwlock; struct sx vnet_sxlock; #define VNET_LIST_WLOCK() do { \ sx_xlock(&vnet_sxlock); \ rw_wlock(&vnet_rwlock); \ } while (0) #define VNET_LIST_WUNLOCK() do { \ rw_wunlock(&vnet_rwlock); \ sx_xunlock(&vnet_sxlock); \ } while (0) struct vnet_list_head vnet_head; struct vnet *vnet0; /* * The virtual network stack allocator provides storage for virtualized * global variables. These variables are defined/declared using the * VNET_DEFINE()/VNET_DECLARE() macros, which place them in the 'set_vnet' * linker set. The details of the implementation are somewhat subtle, but * allow the majority of most network subsystems to maintain * virtualization-agnostic. * * The virtual network stack allocator handles variables in the base kernel * vs. modules in similar but different ways. In both cases, virtualized * global variables are marked as such by being declared to be part of the * vnet linker set. These "master" copies of global variables serve two * functions: * * (1) They contain static initialization or "default" values for global * variables which will be propagated to each virtual network stack * instance when created. As with normal global variables, they default * to zero-filled. * * (2) They act as unique global names by which the variable can be referred * to, regardless of network stack instance. The single global symbol * will be used to calculate the location of a per-virtual instance * variable at run-time. * * Each virtual network stack instance has a complete copy of each * virtualized global variable, stored in a malloc'd block of memory * referred to by vnet->vnet_data_mem. Critical to the design is that each * per-instance memory block is laid out identically to the master block so * that the offset of each global variable is the same across all blocks. To * optimize run-time access, a precalculated 'base' address, * vnet->vnet_data_base, is stored in each vnet, and is the amount that can * be added to the address of a 'master' instance of a variable to get to the * per-vnet instance. * * Virtualized global variables are handled in a similar manner, but as each * module has its own 'set_vnet' linker set, and we want to keep all * virtualized globals togther, we reserve space in the kernel's linker set * for potential module variables using a per-vnet character array, * 'modspace'. The virtual network stack allocator maintains a free list to * track what space in the array is free (all, initially) and as modules are * linked, allocates portions of the space to specific globals. The kernel * module linker queries the virtual network stack allocator and will * bind references of the global to the location during linking. It also * calls into the virtual network stack allocator, once the memory is * initialized, in order to propagate the new static initializations to all * existing virtual network stack instances so that the soon-to-be executing * module will find every network stack instance with proper default values. */ /* - * Location of the kernel's 'set_vnet' linker set. - */ -extern uintptr_t *__start_set_vnet; -extern uintptr_t *__stop_set_vnet; - -#define VNET_START (uintptr_t)&__start_set_vnet -#define VNET_STOP (uintptr_t)&__stop_set_vnet - -/* * Number of bytes of data in the 'set_vnet' linker set, and hence the total * size of all kernel virtualized global variables, and the malloc(9) type * that will be used to allocate it. */ #define VNET_BYTES (VNET_STOP - VNET_START) MALLOC_DEFINE(M_VNET_DATA, "vnet_data", "VNET data"); /* * VNET_MODMIN is the minimum number of bytes we will reserve for the sum of * global variables across all loaded modules. As this actually sizes an * array declared as a virtualized global variable in the kernel itself, and * we want the virtualized global variable space to be page-sized, we may * have more space than that in practice. */ #define VNET_MODMIN 8192 #define VNET_SIZE roundup2(VNET_BYTES, PAGE_SIZE) #define VNET_MODSIZE (VNET_SIZE - (VNET_BYTES - VNET_MODMIN)) /* * Space to store virtualized global variables from loadable kernel modules, * and the free list to manage it. */ static VNET_DEFINE(char, modspace[VNET_MODMIN]); /* * Global lists of subsystem constructor and destructors for vnets. They are * registered via VNET_SYSINIT() and VNET_SYSUNINIT(). Both lists are * protected by the vnet_sysinit_sxlock global lock. */ static TAILQ_HEAD(vnet_sysinit_head, vnet_sysinit) vnet_constructors = TAILQ_HEAD_INITIALIZER(vnet_constructors); static TAILQ_HEAD(vnet_sysuninit_head, vnet_sysinit) vnet_destructors = TAILQ_HEAD_INITIALIZER(vnet_destructors); struct sx vnet_sysinit_sxlock; #define VNET_SYSINIT_WLOCK() sx_xlock(&vnet_sysinit_sxlock); #define VNET_SYSINIT_WUNLOCK() sx_xunlock(&vnet_sysinit_sxlock); #define VNET_SYSINIT_RLOCK() sx_slock(&vnet_sysinit_sxlock); #define VNET_SYSINIT_RUNLOCK() sx_sunlock(&vnet_sysinit_sxlock); struct vnet_data_free { uintptr_t vnd_start; int vnd_len; TAILQ_ENTRY(vnet_data_free) vnd_link; }; MALLOC_DEFINE(M_VNET_DATA_FREE, "vnet_data_free", "VNET resource accounting"); static TAILQ_HEAD(, vnet_data_free) vnet_data_free_head = TAILQ_HEAD_INITIALIZER(vnet_data_free_head); static struct sx vnet_data_free_lock; SDT_PROVIDER_DEFINE(vnet); SDT_PROBE_DEFINE1(vnet, functions, vnet_alloc, entry, "int"); SDT_PROBE_DEFINE2(vnet, functions, vnet_alloc, alloc, "int", "struct vnet *"); SDT_PROBE_DEFINE2(vnet, functions, vnet_alloc, return, "int", "struct vnet *"); SDT_PROBE_DEFINE2(vnet, functions, vnet_destroy, entry, "int", "struct vnet *"); SDT_PROBE_DEFINE1(vnet, functions, vnet_destroy, return, "int"); #ifdef DDB static void db_show_vnet_print_vs(struct vnet_sysinit *, int); #endif /* * Allocate a virtual network stack. */ struct vnet * vnet_alloc(void) { struct vnet *vnet; SDT_PROBE1(vnet, functions, vnet_alloc, entry, __LINE__); vnet = malloc(sizeof(struct vnet), M_VNET, M_WAITOK | M_ZERO); vnet->vnet_magic_n = VNET_MAGIC_N; SDT_PROBE2(vnet, functions, vnet_alloc, alloc, __LINE__, vnet); /* * Allocate storage for virtualized global variables and copy in * initial values form our 'master' copy. */ vnet->vnet_data_mem = malloc(VNET_SIZE, M_VNET_DATA, M_WAITOK); memcpy(vnet->vnet_data_mem, (void *)VNET_START, VNET_BYTES); /* * All use of vnet-specific data will immediately subtract VNET_START * from the base memory pointer, so pre-calculate that now to avoid * it on each use. */ vnet->vnet_data_base = (uintptr_t)vnet->vnet_data_mem - VNET_START; /* Initialize / attach vnet module instances. */ CURVNET_SET_QUIET(vnet); vnet_sysinit(); CURVNET_RESTORE(); VNET_LIST_WLOCK(); LIST_INSERT_HEAD(&vnet_head, vnet, vnet_le); VNET_LIST_WUNLOCK(); SDT_PROBE2(vnet, functions, vnet_alloc, return, __LINE__, vnet); return (vnet); } /* * Destroy a virtual network stack. */ void vnet_destroy(struct vnet *vnet) { struct ifnet *ifp, *nifp; SDT_PROBE2(vnet, functions, vnet_destroy, entry, __LINE__, vnet); KASSERT(vnet->vnet_sockcnt == 0, ("%s: vnet still has sockets", __func__)); VNET_LIST_WLOCK(); LIST_REMOVE(vnet, vnet_le); VNET_LIST_WUNLOCK(); CURVNET_SET_QUIET(vnet); /* Return all inherited interfaces to their parent vnets. */ TAILQ_FOREACH_SAFE(ifp, &V_ifnet, if_link, nifp) { if (ifp->if_home_vnet != ifp->if_vnet) if_vmove(ifp, ifp->if_home_vnet); } vnet_sysuninit(); CURVNET_RESTORE(); /* * Release storage for the virtual network stack instance. */ free(vnet->vnet_data_mem, M_VNET_DATA); vnet->vnet_data_mem = NULL; vnet->vnet_data_base = 0; vnet->vnet_magic_n = 0xdeadbeef; free(vnet, M_VNET); SDT_PROBE1(vnet, functions, vnet_destroy, return, __LINE__); } /* * Boot time initialization and allocation of virtual network stacks. */ static void vnet_init_prelink(void *arg) { rw_init(&vnet_rwlock, "vnet_rwlock"); sx_init(&vnet_sxlock, "vnet_sxlock"); sx_init(&vnet_sysinit_sxlock, "vnet_sysinit_sxlock"); LIST_INIT(&vnet_head); } SYSINIT(vnet_init_prelink, SI_SUB_VNET_PRELINK, SI_ORDER_FIRST, vnet_init_prelink, NULL); static void vnet0_init(void *arg) { /* Warn people before take off - in case we crash early. */ printf("WARNING: VIMAGE (virtualized network stack) is a highly " "experimental feature.\n"); /* * We MUST clear curvnet in vi_init_done() before going SMP, * otherwise CURVNET_SET() macros would scream about unnecessary * curvnet recursions. */ curvnet = prison0.pr_vnet = vnet0 = vnet_alloc(); } SYSINIT(vnet0_init, SI_SUB_VNET, SI_ORDER_FIRST, vnet0_init, NULL); static void vnet_init_done(void *unused) { curvnet = NULL; } SYSINIT(vnet_init_done, SI_SUB_VNET_DONE, SI_ORDER_FIRST, vnet_init_done, NULL); /* * Once on boot, initialize the modspace freelist to entirely cover modspace. */ static void vnet_data_startup(void *dummy __unused) { struct vnet_data_free *df; df = malloc(sizeof(*df), M_VNET_DATA_FREE, M_WAITOK | M_ZERO); df->vnd_start = (uintptr_t)&VNET_NAME(modspace); df->vnd_len = VNET_MODSIZE; TAILQ_INSERT_HEAD(&vnet_data_free_head, df, vnd_link); sx_init(&vnet_data_free_lock, "vnet_data alloc lock"); } SYSINIT(vnet_data, SI_SUB_KLD, SI_ORDER_FIRST, vnet_data_startup, 0); /* * When a module is loaded and requires storage for a virtualized global * variable, allocate space from the modspace free list. This interface * should be used only by the kernel linker. */ void * vnet_data_alloc(int size) { struct vnet_data_free *df; void *s; s = NULL; size = roundup2(size, sizeof(void *)); sx_xlock(&vnet_data_free_lock); TAILQ_FOREACH(df, &vnet_data_free_head, vnd_link) { if (df->vnd_len < size) continue; if (df->vnd_len == size) { s = (void *)df->vnd_start; TAILQ_REMOVE(&vnet_data_free_head, df, vnd_link); free(df, M_VNET_DATA_FREE); break; } s = (void *)df->vnd_start; df->vnd_len -= size; df->vnd_start = df->vnd_start + size; break; } sx_xunlock(&vnet_data_free_lock); return (s); } /* * Free space for a virtualized global variable on module unload. */ void vnet_data_free(void *start_arg, int size) { struct vnet_data_free *df; struct vnet_data_free *dn; uintptr_t start; uintptr_t end; size = roundup2(size, sizeof(void *)); start = (uintptr_t)start_arg; end = start + size; /* * Free a region of space and merge it with as many neighbors as * possible. Keeping the list sorted simplifies this operation. */ sx_xlock(&vnet_data_free_lock); TAILQ_FOREACH(df, &vnet_data_free_head, vnd_link) { if (df->vnd_start > end) break; /* * If we expand at the end of an entry we may have to merge * it with the one following it as well. */ if (df->vnd_start + df->vnd_len == start) { df->vnd_len += size; dn = TAILQ_NEXT(df, vnd_link); if (df->vnd_start + df->vnd_len == dn->vnd_start) { df->vnd_len += dn->vnd_len; TAILQ_REMOVE(&vnet_data_free_head, dn, vnd_link); free(dn, M_VNET_DATA_FREE); } sx_xunlock(&vnet_data_free_lock); return; } if (df->vnd_start == end) { df->vnd_start = start; df->vnd_len += size; sx_xunlock(&vnet_data_free_lock); return; } } dn = malloc(sizeof(*df), M_VNET_DATA_FREE, M_WAITOK | M_ZERO); dn->vnd_start = start; dn->vnd_len = size; if (df) TAILQ_INSERT_BEFORE(df, dn, vnd_link); else TAILQ_INSERT_TAIL(&vnet_data_free_head, dn, vnd_link); sx_xunlock(&vnet_data_free_lock); } /* * When a new virtualized global variable has been allocated, propagate its * initial value to each already-allocated virtual network stack instance. */ void vnet_data_copy(void *start, int size) { struct vnet *vnet; VNET_LIST_RLOCK(); LIST_FOREACH(vnet, &vnet_head, vnet_le) memcpy((void *)((uintptr_t)vnet->vnet_data_base + (uintptr_t)start), start, size); VNET_LIST_RUNLOCK(); } /* * Variants on sysctl_handle_foo that know how to handle virtualized global * variables: if 'arg1' is a pointer, then we transform it to the local vnet * offset. */ int vnet_sysctl_handle_int(SYSCTL_HANDLER_ARGS) { if (arg1 != NULL) arg1 = (void *)(curvnet->vnet_data_base + (uintptr_t)arg1); return (sysctl_handle_int(oidp, arg1, arg2, req)); } int vnet_sysctl_handle_opaque(SYSCTL_HANDLER_ARGS) { if (arg1 != NULL) arg1 = (void *)(curvnet->vnet_data_base + (uintptr_t)arg1); return (sysctl_handle_opaque(oidp, arg1, arg2, req)); } int vnet_sysctl_handle_string(SYSCTL_HANDLER_ARGS) { if (arg1 != NULL) arg1 = (void *)(curvnet->vnet_data_base + (uintptr_t)arg1); return (sysctl_handle_string(oidp, arg1, arg2, req)); } int vnet_sysctl_handle_uint(SYSCTL_HANDLER_ARGS) { if (arg1 != NULL) arg1 = (void *)(curvnet->vnet_data_base + (uintptr_t)arg1); return (sysctl_handle_int(oidp, arg1, arg2, req)); } /* * Support for special SYSINIT handlers registered via VNET_SYSINIT() * and VNET_SYSUNINIT(). */ void vnet_register_sysinit(void *arg) { struct vnet_sysinit *vs, *vs2; struct vnet *vnet; vs = arg; KASSERT(vs->subsystem > SI_SUB_VNET, ("vnet sysinit too early")); /* Add the constructor to the global list of vnet constructors. */ VNET_SYSINIT_WLOCK(); TAILQ_FOREACH(vs2, &vnet_constructors, link) { if (vs2->subsystem > vs->subsystem) break; if (vs2->subsystem == vs->subsystem && vs2->order > vs->order) break; } if (vs2 != NULL) TAILQ_INSERT_BEFORE(vs2, vs, link); else TAILQ_INSERT_TAIL(&vnet_constructors, vs, link); /* * Invoke the constructor on all the existing vnets when it is * registered. */ VNET_FOREACH(vnet) { CURVNET_SET_QUIET(vnet); vs->func(vs->arg); CURVNET_RESTORE(); } VNET_SYSINIT_WUNLOCK(); } void vnet_deregister_sysinit(void *arg) { struct vnet_sysinit *vs; vs = arg; /* Remove the constructor from the global list of vnet constructors. */ VNET_SYSINIT_WLOCK(); TAILQ_REMOVE(&vnet_constructors, vs, link); VNET_SYSINIT_WUNLOCK(); } void vnet_register_sysuninit(void *arg) { struct vnet_sysinit *vs, *vs2; vs = arg; /* Add the destructor to the global list of vnet destructors. */ VNET_SYSINIT_WLOCK(); TAILQ_FOREACH(vs2, &vnet_destructors, link) { if (vs2->subsystem > vs->subsystem) break; if (vs2->subsystem == vs->subsystem && vs2->order > vs->order) break; } if (vs2 != NULL) TAILQ_INSERT_BEFORE(vs2, vs, link); else TAILQ_INSERT_TAIL(&vnet_destructors, vs, link); VNET_SYSINIT_WUNLOCK(); } void vnet_deregister_sysuninit(void *arg) { struct vnet_sysinit *vs; struct vnet *vnet; vs = arg; /* * Invoke the destructor on all the existing vnets when it is * deregistered. */ VNET_SYSINIT_WLOCK(); VNET_FOREACH(vnet) { CURVNET_SET_QUIET(vnet); vs->func(vs->arg); CURVNET_RESTORE(); } /* Remove the destructor from the global list of vnet destructors. */ TAILQ_REMOVE(&vnet_destructors, vs, link); VNET_SYSINIT_WUNLOCK(); } /* * Invoke all registered vnet constructors on the current vnet. Used during * vnet construction. The caller is responsible for ensuring the new vnet is * the current vnet and that the vnet_sysinit_sxlock lock is locked. */ void vnet_sysinit(void) { struct vnet_sysinit *vs; VNET_SYSINIT_RLOCK(); TAILQ_FOREACH(vs, &vnet_constructors, link) { vs->func(vs->arg); } VNET_SYSINIT_RUNLOCK(); } /* * Invoke all registered vnet destructors on the current vnet. Used during * vnet destruction. The caller is responsible for ensuring the dying vnet * the current vnet and that the vnet_sysinit_sxlock lock is locked. */ void vnet_sysuninit(void) { struct vnet_sysinit *vs; VNET_SYSINIT_RLOCK(); TAILQ_FOREACH_REVERSE(vs, &vnet_destructors, vnet_sysuninit_head, link) { vs->func(vs->arg); } VNET_SYSINIT_RUNLOCK(); } /* * EVENTHANDLER(9) extensions. */ /* * Invoke the eventhandler function originally registered with the possibly * registered argument for all virtual network stack instances. * * This iterator can only be used for eventhandlers that do not take any * additional arguments, as we do ignore the variadic arguments from the * EVENTHANDLER_INVOKE() call. */ void vnet_global_eventhandler_iterator_func(void *arg, ...) { VNET_ITERATOR_DECL(vnet_iter); struct eventhandler_entry_vimage *v_ee; /* * There is a bug here in that we should actually cast things to * (struct eventhandler_entry_ ## name *) but that's not easily * possible in here so just re-using the variadic version we * defined for the generic vimage case. */ v_ee = arg; VNET_LIST_RLOCK(); VNET_FOREACH(vnet_iter) { CURVNET_SET(vnet_iter); ((vimage_iterator_func_t)v_ee->func)(v_ee->ee_arg); CURVNET_RESTORE(); } VNET_LIST_RUNLOCK(); } #ifdef VNET_DEBUG struct vnet_recursion { SLIST_ENTRY(vnet_recursion) vnr_le; const char *prev_fn; const char *where_fn; int where_line; struct vnet *old_vnet; struct vnet *new_vnet; }; static SLIST_HEAD(, vnet_recursion) vnet_recursions = SLIST_HEAD_INITIALIZER(vnet_recursions); static void vnet_print_recursion(struct vnet_recursion *vnr, int brief) { if (!brief) printf("CURVNET_SET() recursion in "); printf("%s() line %d, prev in %s()", vnr->where_fn, vnr->where_line, vnr->prev_fn); if (brief) printf(", "); else printf("\n "); printf("%p -> %p\n", vnr->old_vnet, vnr->new_vnet); } void vnet_log_recursion(struct vnet *old_vnet, const char *old_fn, int line) { struct vnet_recursion *vnr; /* Skip already logged recursion events. */ SLIST_FOREACH(vnr, &vnet_recursions, vnr_le) if (vnr->prev_fn == old_fn && vnr->where_fn == curthread->td_vnet_lpush && vnr->where_line == line && (vnr->old_vnet == vnr->new_vnet) == (curvnet == old_vnet)) return; vnr = malloc(sizeof(*vnr), M_VNET, M_NOWAIT | M_ZERO); if (vnr == NULL) panic("%s: malloc failed", __func__); vnr->prev_fn = old_fn; vnr->where_fn = curthread->td_vnet_lpush; vnr->where_line = line; vnr->old_vnet = old_vnet; vnr->new_vnet = curvnet; SLIST_INSERT_HEAD(&vnet_recursions, vnr, vnr_le); vnet_print_recursion(vnr, 0); #ifdef KDB kdb_backtrace(); #endif } #endif /* VNET_DEBUG */ /* * DDB(4). */ #ifdef DDB DB_SHOW_COMMAND(vnets, db_show_vnets) { VNET_ITERATOR_DECL(vnet_iter); VNET_FOREACH(vnet_iter) { db_printf("vnet = %p\n", vnet_iter); db_printf(" vnet_magic_n = 0x%x (%s, orig 0x%x)\n", vnet_iter->vnet_magic_n, (vnet_iter->vnet_magic_n == VNET_MAGIC_N) ? "ok" : "mismatch", VNET_MAGIC_N); db_printf(" vnet_ifcnt = %u\n", vnet_iter->vnet_ifcnt); db_printf(" vnet_sockcnt = %u\n", vnet_iter->vnet_sockcnt); db_printf(" vnet_data_mem = %p\n", vnet_iter->vnet_data_mem); db_printf(" vnet_data_base = 0x%jx\n", (uintmax_t)vnet_iter->vnet_data_base); db_printf("\n"); if (db_pager_quit) break; } } static void db_show_vnet_print_vs(struct vnet_sysinit *vs, int ddb) { const char *vsname, *funcname; c_db_sym_t sym; db_expr_t offset; #define xprint(...) \ if (ddb) \ db_printf(__VA_ARGS__); \ else \ printf(__VA_ARGS__) if (vs == NULL) { xprint("%s: no vnet_sysinit * given\n", __func__); return; } sym = db_search_symbol((vm_offset_t)vs, DB_STGY_ANY, &offset); db_symbol_values(sym, &vsname, NULL); sym = db_search_symbol((vm_offset_t)vs->func, DB_STGY_PROC, &offset); db_symbol_values(sym, &funcname, NULL); xprint("%s(%p)\n", (vsname != NULL) ? vsname : "", vs); xprint(" 0x%08x 0x%08x\n", vs->subsystem, vs->order); xprint(" %p(%s)(%p)\n", vs->func, (funcname != NULL) ? funcname : "", vs->arg); #undef xprint } DB_SHOW_COMMAND(vnet_sysinit, db_show_vnet_sysinit) { struct vnet_sysinit *vs; db_printf("VNET_SYSINIT vs Name(Ptr)\n"); db_printf(" Subsystem Order\n"); db_printf(" Function(Name)(Arg)\n"); TAILQ_FOREACH(vs, &vnet_constructors, link) { db_show_vnet_print_vs(vs, 1); if (db_pager_quit) break; } } DB_SHOW_COMMAND(vnet_sysuninit, db_show_vnet_sysuninit) { struct vnet_sysinit *vs; db_printf("VNET_SYSUNINIT vs Name(Ptr)\n"); db_printf(" Subsystem Order\n"); db_printf(" Function(Name)(Arg)\n"); TAILQ_FOREACH_REVERSE(vs, &vnet_destructors, vnet_sysuninit_head, link) { db_show_vnet_print_vs(vs, 1); if (db_pager_quit) break; } } #ifdef VNET_DEBUG DB_SHOW_COMMAND(vnetrcrs, db_show_vnetrcrs) { struct vnet_recursion *vnr; SLIST_FOREACH(vnr, &vnet_recursions, vnr_le) vnet_print_recursion(vnr, 1); } #endif #endif /* DDB */ Index: user/imp/tbemd/sys/net/vnet.h =================================================================== --- user/imp/tbemd/sys/net/vnet.h (revision 206684) +++ user/imp/tbemd/sys/net/vnet.h (revision 206685) @@ -1,420 +1,429 @@ /*- * Copyright (c) 2006-2009 University of Zagreb * Copyright (c) 2006-2009 FreeBSD Foundation * All rights reserved. * * This software was developed by the University of Zagreb and the * FreeBSD Foundation under sponsorship by the Stichting NLnet and the * FreeBSD Foundation. * * Copyright (c) 2009 Jeffrey Roberson * Copyright (c) 2009 Robert N. M. Watson * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * $FreeBSD$ */ /*- * This header file defines several sets of interfaces supporting virtualized * network stacks: * * - Definition of 'struct vnet' and functions and macros to allocate/free/ * manipulate it. * * - A virtual network stack memory allocator, which provides support for * virtualized global variables via a special linker set, set_vnet. * * - Virtualized sysinits/sysuninits, which allow constructors and * destructors to be run for each network stack subsystem as virtual * instances are created and destroyed. * * If VIMAGE isn't compiled into the kernel, virtualized global variables * compile to normal global variables, and virtualized sysinits to regular * sysinits. */ #ifndef _NET_VNET_H_ #define _NET_VNET_H_ /* * struct vnet describes a virtualized network stack, and is primarily a * pointer to storage for virtualized global variables. Expose to userspace * as required for libkvm. */ #if defined(_KERNEL) || defined(_WANT_VNET) #include struct vnet { LIST_ENTRY(vnet) vnet_le; /* all vnets list */ u_int vnet_magic_n; u_int vnet_ifcnt; u_int vnet_sockcnt; void *vnet_data_mem; uintptr_t vnet_data_base; }; #define VNET_MAGIC_N 0x3e0d8f29 /* * These two virtual network stack allocator definitions are also required * for libkvm so that it can evaluate virtualized global variables. */ #define VNET_SETNAME "set_vnet" #define VNET_SYMPREFIX "vnet_entry_" #endif #ifdef _KERNEL #ifdef VIMAGE #include #include /* for struct thread */ #include #include /* + * Location of the kernel's 'set_vnet' linker set. + */ +extern uintptr_t *__start_set_vnet; +extern uintptr_t *__stop_set_vnet; + +#define VNET_START (uintptr_t)&__start_set_vnet +#define VNET_STOP (uintptr_t)&__stop_set_vnet + +/* * Functions to allocate and destroy virtual network stacks. */ struct vnet *vnet_alloc(void); void vnet_destroy(struct vnet *vnet); /* * The current virtual network stack -- we may wish to move this to struct * pcpu in the future. */ #define curvnet curthread->td_vnet /* * Various macros -- get and set the current network stack, but also * assertions. */ #ifdef VNET_DEBUG void vnet_log_recursion(struct vnet *, const char *, int); #define VNET_ASSERT(condition) \ if (!(condition)) { \ printf("VNET_ASSERT @ %s:%d %s():\n", \ __FILE__, __LINE__, __FUNCTION__); \ panic(#condition); \ } #define CURVNET_SET_QUIET(arg) \ VNET_ASSERT((arg)->vnet_magic_n == VNET_MAGIC_N); \ struct vnet *saved_vnet = curvnet; \ const char *saved_vnet_lpush = curthread->td_vnet_lpush; \ curvnet = arg; \ curthread->td_vnet_lpush = __FUNCTION__; #define CURVNET_SET_VERBOSE(arg) \ CURVNET_SET_QUIET(arg) \ if (saved_vnet) \ vnet_log_recursion(saved_vnet, saved_vnet_lpush, __LINE__); #define CURVNET_SET(arg) CURVNET_SET_VERBOSE(arg) #define CURVNET_RESTORE() \ VNET_ASSERT(saved_vnet == NULL || \ saved_vnet->vnet_magic_n == VNET_MAGIC_N); \ curvnet = saved_vnet; \ curthread->td_vnet_lpush = saved_vnet_lpush; #else /* !VNET_DEBUG */ #define VNET_ASSERT(condition) #define CURVNET_SET(arg) \ struct vnet *saved_vnet = curvnet; \ curvnet = arg; #define CURVNET_SET_VERBOSE(arg) CURVNET_SET(arg) #define CURVNET_SET_QUIET(arg) CURVNET_SET(arg) #define CURVNET_RESTORE() \ curvnet = saved_vnet; #endif /* VNET_DEBUG */ extern struct vnet *vnet0; #define IS_DEFAULT_VNET(arg) ((arg) == vnet0) #define CRED_TO_VNET(cr) (cr)->cr_prison->pr_vnet #define TD_TO_VNET(td) CRED_TO_VNET((td)->td_ucred) #define P_TO_VNET(p) CRED_TO_VNET((p)->p_ucred) /* * Global linked list of all virtual network stacks, along with read locks to * access it. If a caller may sleep while accessing the list, it must use * the sleepable lock macros. */ LIST_HEAD(vnet_list_head, vnet); extern struct vnet_list_head vnet_head; extern struct rwlock vnet_rwlock; extern struct sx vnet_sxlock; #define VNET_LIST_RLOCK() sx_slock(&vnet_sxlock) #define VNET_LIST_RLOCK_NOSLEEP() rw_rlock(&vnet_rwlock) #define VNET_LIST_RUNLOCK() sx_sunlock(&vnet_sxlock) #define VNET_LIST_RUNLOCK_NOSLEEP() rw_runlock(&vnet_rwlock) /* * Iteration macros to walk the global list of virtual network stacks. */ #define VNET_ITERATOR_DECL(arg) struct vnet *arg #define VNET_FOREACH(arg) LIST_FOREACH((arg), &vnet_head, vnet_le) /* * Virtual network stack memory allocator, which allows global variables to * be automatically instantiated for each network stack instance. */ __asm__( #if defined(__arm__) ".section " VNET_SETNAME ", \"aw\", %progbits\n" #else ".section " VNET_SETNAME ", \"aw\", @progbits\n" #endif "\t.p2align " __XSTRING(CACHE_LINE_SHIFT) "\n" "\t.previous"); #define VNET_NAME(n) vnet_entry_##n #define VNET_DECLARE(t, n) extern t VNET_NAME(n) #define VNET_DEFINE(t, n) t VNET_NAME(n) __section(VNET_SETNAME) __used #define _VNET_PTR(b, n) (__typeof(VNET_NAME(n))*) \ ((b) + (uintptr_t)&VNET_NAME(n)) #define _VNET(b, n) (*_VNET_PTR(b, n)) /* * Virtualized global variable accessor macros. */ #define VNET_VNET_PTR(vnet, n) _VNET_PTR((vnet)->vnet_data_base, n) #define VNET_VNET(vnet, n) (*VNET_VNET_PTR((vnet), n)) #define VNET_PTR(n) VNET_VNET_PTR(curvnet, n) #define VNET(n) VNET_VNET(curvnet, n) /* * Virtual network stack allocator interfaces from the kernel linker. */ void *vnet_data_alloc(int size); void vnet_data_copy(void *start, int size); void vnet_data_free(void *start_arg, int size); /* * Sysctl variants for vnet-virtualized global variables. Include * to expose these definitions. * * Note: SYSCTL_PROC() handler functions will need to resolve pointer * arguments themselves, if required. */ #ifdef SYSCTL_OID int vnet_sysctl_handle_int(SYSCTL_HANDLER_ARGS); int vnet_sysctl_handle_opaque(SYSCTL_HANDLER_ARGS); int vnet_sysctl_handle_string(SYSCTL_HANDLER_ARGS); int vnet_sysctl_handle_uint(SYSCTL_HANDLER_ARGS); #define SYSCTL_VNET_INT(parent, nbr, name, access, ptr, val, descr) \ SYSCTL_OID(parent, nbr, name, \ CTLTYPE_INT|CTLFLAG_MPSAFE|CTLFLAG_VNET|(access), \ ptr, val, vnet_sysctl_handle_int, "I", descr) #define SYSCTL_VNET_PROC(parent, nbr, name, access, ptr, arg, handler, \ fmt, descr) \ SYSCTL_OID(parent, nbr, name, CTLFLAG_VNET|(access), ptr, arg, \ handler, fmt, descr) #define SYSCTL_VNET_STRING(parent, nbr, name, access, arg, len, descr) \ SYSCTL_OID(parent, nbr, name, \ CTLTYPE_STRING|CTLFLAG_VNET|(access), \ arg, len, vnet_sysctl_handle_string, "A", descr) #define SYSCTL_VNET_STRUCT(parent, nbr, name, access, ptr, type, descr) \ SYSCTL_OID(parent, nbr, name, \ CTLTYPE_OPAQUE|CTLFLAG_VNET|(access), ptr, \ sizeof(struct type), vnet_sysctl_handle_opaque, "S," #type, \ descr) #define SYSCTL_VNET_UINT(parent, nbr, name, access, ptr, val, descr) \ SYSCTL_OID(parent, nbr, name, \ CTLTYPE_UINT|CTLFLAG_MPSAFE|CTLFLAG_VNET|(access), \ ptr, val, vnet_sysctl_handle_uint, "IU", descr) #define VNET_SYSCTL_ARG(req, arg1) do { \ if (arg1 != NULL) \ arg1 = (void *)(TD_TO_VNET((req)->td)->vnet_data_base + \ (uintptr_t)(arg1)); \ } while (0) #endif /* SYSCTL_OID */ /* * Virtual sysinit mechanism, allowing network stack components to declare * startup and shutdown methods to be run when virtual network stack * instances are created and destroyed. */ #include /* * SYSINIT/SYSUNINIT variants that provide per-vnet constructors and * destructors. */ struct vnet_sysinit { enum sysinit_sub_id subsystem; enum sysinit_elem_order order; sysinit_cfunc_t func; const void *arg; TAILQ_ENTRY(vnet_sysinit) link; }; #define VNET_SYSINIT(ident, subsystem, order, func, arg) \ static struct vnet_sysinit ident ## _vnet_init = { \ subsystem, \ order, \ (sysinit_cfunc_t)(sysinit_nfunc_t)func, \ (arg) \ }; \ SYSINIT(vnet_init_ ## ident, subsystem, order, \ vnet_register_sysinit, &ident ## _vnet_init); \ SYSUNINIT(vnet_init_ ## ident, subsystem, order, \ vnet_deregister_sysinit, &ident ## _vnet_init) #define VNET_SYSUNINIT(ident, subsystem, order, func, arg) \ static struct vnet_sysinit ident ## _vnet_uninit = { \ subsystem, \ order, \ (sysinit_cfunc_t)(sysinit_nfunc_t)func, \ (arg) \ }; \ SYSINIT(vnet_uninit_ ## ident, subsystem, order, \ vnet_register_sysuninit, &ident ## _vnet_uninit); \ SYSUNINIT(vnet_uninit_ ## ident, subsystem, order, \ vnet_deregister_sysuninit, &ident ## _vnet_uninit) /* * Run per-vnet sysinits or sysuninits during vnet creation/destruction. */ void vnet_sysinit(void); void vnet_sysuninit(void); /* * Interfaces for managing per-vnet constructors and destructors. */ void vnet_register_sysinit(void *arg); void vnet_register_sysuninit(void *arg); void vnet_deregister_sysinit(void *arg); void vnet_deregister_sysuninit(void *arg); /* * EVENTHANDLER(9) extensions. */ #include void vnet_global_eventhandler_iterator_func(void *, ...); #define VNET_GLOBAL_EVENTHANDLER_REGISTER_TAG(tag, name, func, arg, priority) \ do { \ if (IS_DEFAULT_VNET(curvnet)) { \ (tag) = vimage_eventhandler_register(NULL, #name, func, \ arg, priority, \ vnet_global_eventhandler_iterator_func); \ } \ } while(0) #define VNET_GLOBAL_EVENTHANDLER_REGISTER(name, func, arg, priority) \ do { \ if (IS_DEFAULT_VNET(curvnet)) { \ vimage_eventhandler_register(NULL, #name, func, \ arg, priority, \ vnet_global_eventhandler_iterator_func); \ } \ } while(0) #else /* !VIMAGE */ /* * Various virtual network stack macros compile to no-ops without VIMAGE. */ #define curvnet NULL #define VNET_ASSERT(condition) #define CURVNET_SET(arg) #define CURVNET_SET_QUIET(arg) #define CURVNET_RESTORE() #define VNET_LIST_RLOCK() #define VNET_LIST_RLOCK_NOSLEEP() #define VNET_LIST_RUNLOCK() #define VNET_LIST_RUNLOCK_NOSLEEP() #define VNET_ITERATOR_DECL(arg) #define VNET_FOREACH(arg) #define IS_DEFAULT_VNET(arg) 1 #define CRED_TO_VNET(cr) NULL #define TD_TO_VNET(td) NULL #define P_TO_VNET(p) NULL /* * Versions of the VNET macros that compile to normal global variables and * standard sysctl definitions. */ #define VNET_NAME(n) n #define VNET_DECLARE(t, n) extern t n #define VNET_DEFINE(t, n) t n #define _VNET_PTR(b, n) &VNET_NAME(n) /* * Virtualized global variable accessor macros. */ #define VNET_VNET_PTR(vnet, n) (&(n)) #define VNET_VNET(vnet, n) (n) #define VNET_PTR(n) (&(n)) #define VNET(n) (n) /* * When VIMAGE isn't compiled into the kernel, virtaulized SYSCTLs simply * become normal SYSCTLs. */ #ifdef SYSCTL_OID #define SYSCTL_VNET_INT(parent, nbr, name, access, ptr, val, descr) \ SYSCTL_INT(parent, nbr, name, access, ptr, val, descr) #define SYSCTL_VNET_PROC(parent, nbr, name, access, ptr, arg, handler, \ fmt, descr) \ SYSCTL_PROC(parent, nbr, name, access, ptr, arg, handler, fmt, \ descr) #define SYSCTL_VNET_STRING(parent, nbr, name, access, arg, len, descr) \ SYSCTL_STRING(parent, nbr, name, access, arg, len, descr) #define SYSCTL_VNET_STRUCT(parent, nbr, name, access, ptr, type, descr) \ SYSCTL_STRUCT(parent, nbr, name, access, ptr, type, descr) #define SYSCTL_VNET_UINT(parent, nbr, name, access, ptr, val, descr) \ SYSCTL_UINT(parent, nbr, name, access, ptr, val, descr) #define VNET_SYSCTL_ARG(req, arg1) #endif /* SYSCTL_OID */ /* * When VIMAGE isn't compiled into the kernel, VNET_SYSINIT/VNET_SYSUNINIT * map into normal sysinits, which have the same ordering properties. */ #define VNET_SYSINIT(ident, subsystem, order, func, arg) \ SYSINIT(ident, subsystem, order, func, arg) #define VNET_SYSUNINIT(ident, subsystem, order, func, arg) \ SYSUNINIT(ident, subsystem, order, func, arg) /* * Without VIMAGE revert to the default implementation. */ #define VNET_GLOBAL_EVENTHANDLER_REGISTER_TAG(tag, name, func, arg, priority) \ (tag) = eventhandler_register(NULL, #name, func, arg, priority) #define VNET_GLOBAL_EVENTHANDLER_REGISTER(name, func, arg, priority) \ eventhandler_register(NULL, #name, func, arg, priority) #endif /* VIMAGE */ #endif /* _KERNEL */ #endif /* !_NET_VNET_H_ */ Index: user/imp/tbemd/sys/netipsec/key.c =================================================================== --- user/imp/tbemd/sys/netipsec/key.c (revision 206684) +++ user/imp/tbemd/sys/netipsec/key.c (revision 206685) @@ -1,8069 +1,8075 @@ /* $FreeBSD$ */ /* $KAME: key.c,v 1.191 2001/06/27 10:46:49 sakane Exp $ */ /*- * Copyright (C) 1995, 1996, 1997, and 1998 WIDE Project. * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of the project nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE PROJECT AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE PROJECT OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ /* * This code is referd to RFC 2367 */ #include "opt_inet.h" #include "opt_inet6.h" #include "opt_ipsec.h" #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #ifdef INET6 #include #include #include #endif /* INET6 */ #ifdef INET #include #endif #ifdef INET6 #include #endif /* INET6 */ #include #include #include #include #include #include #ifdef INET6 #include #endif #include #include /* randomness */ #include #define FULLMASK 0xff #define _BITS(bytes) ((bytes) << 3) /* * Note on SA reference counting: * - SAs that are not in DEAD state will have (total external reference + 1) * following value in reference count field. they cannot be freed and are * referenced from SA header. * - SAs that are in DEAD state will have (total external reference) * in reference count field. they are ready to be freed. reference from * SA header will be removed in key_delsav(), when the reference count * field hits 0 (= no external reference other than from SA header. */ VNET_DEFINE(u_int32_t, key_debug_level) = 0; static VNET_DEFINE(u_int, key_spi_trycnt) = 1000; #define V_key_spi_trycnt VNET(key_spi_trycnt) static VNET_DEFINE(u_int32_t, key_spi_minval) = 0x100; #define V_key_spi_minval VNET(key_spi_minval) static VNET_DEFINE(u_int32_t, key_spi_maxval) = 0x0fffffff; /* XXX */ #define V_key_spi_maxval VNET(key_spi_maxval) static VNET_DEFINE(u_int32_t, policy_id) = 0; #define V_policy_id VNET(policy_id) /*interval to initialize randseed,1(m)*/ static VNET_DEFINE(u_int, key_int_random) = 60; #define V_key_int_random VNET(key_int_random) /* interval to expire acquiring, 30(s)*/ static VNET_DEFINE(u_int, key_larval_lifetime) = 30; #define V_key_larval_lifetime VNET(key_larval_lifetime) /* counter for blocking SADB_ACQUIRE.*/ static VNET_DEFINE(int, key_blockacq_count) = 10; #define V_key_blockacq_count VNET(key_blockacq_count) /* lifetime for blocking SADB_ACQUIRE.*/ static VNET_DEFINE(int, key_blockacq_lifetime) = 20; #define V_key_blockacq_lifetime VNET(key_blockacq_lifetime) /* preferred old sa rather than new sa.*/ static VNET_DEFINE(int, key_preferred_oldsa) = 1; #define V_key_preferred_oldsa VNET(key_preferred_oldsa) static VNET_DEFINE(u_int32_t, acq_seq) = 0; #define V_acq_seq VNET(acq_seq) /* SPD */ static VNET_DEFINE(LIST_HEAD(_sptree, secpolicy), sptree[IPSEC_DIR_MAX]); #define V_sptree VNET(sptree) static struct mtx sptree_lock; #define SPTREE_LOCK_INIT() \ mtx_init(&sptree_lock, "sptree", \ "fast ipsec security policy database", MTX_DEF) #define SPTREE_LOCK_DESTROY() mtx_destroy(&sptree_lock) #define SPTREE_LOCK() mtx_lock(&sptree_lock) #define SPTREE_UNLOCK() mtx_unlock(&sptree_lock) #define SPTREE_LOCK_ASSERT() mtx_assert(&sptree_lock, MA_OWNED) static VNET_DEFINE(LIST_HEAD(_sahtree, secashead), sahtree); /* SAD */ #define V_sahtree VNET(sahtree) static struct mtx sahtree_lock; #define SAHTREE_LOCK_INIT() \ mtx_init(&sahtree_lock, "sahtree", \ "fast ipsec security association database", MTX_DEF) #define SAHTREE_LOCK_DESTROY() mtx_destroy(&sahtree_lock) #define SAHTREE_LOCK() mtx_lock(&sahtree_lock) #define SAHTREE_UNLOCK() mtx_unlock(&sahtree_lock) #define SAHTREE_LOCK_ASSERT() mtx_assert(&sahtree_lock, MA_OWNED) /* registed list */ static VNET_DEFINE(LIST_HEAD(_regtree, secreg), regtree[SADB_SATYPE_MAX + 1]); #define V_regtree VNET(regtree) static struct mtx regtree_lock; #define REGTREE_LOCK_INIT() \ mtx_init(®tree_lock, "regtree", "fast ipsec regtree", MTX_DEF) #define REGTREE_LOCK_DESTROY() mtx_destroy(®tree_lock) #define REGTREE_LOCK() mtx_lock(®tree_lock) #define REGTREE_UNLOCK() mtx_unlock(®tree_lock) #define REGTREE_LOCK_ASSERT() mtx_assert(®tree_lock, MA_OWNED) static VNET_DEFINE(LIST_HEAD(_acqtree, secacq), acqtree); /* acquiring list */ #define V_acqtree VNET(acqtree) static struct mtx acq_lock; #define ACQ_LOCK_INIT() \ mtx_init(&acq_lock, "acqtree", "fast ipsec acquire list", MTX_DEF) #define ACQ_LOCK_DESTROY() mtx_destroy(&acq_lock) #define ACQ_LOCK() mtx_lock(&acq_lock) #define ACQ_UNLOCK() mtx_unlock(&acq_lock) #define ACQ_LOCK_ASSERT() mtx_assert(&acq_lock, MA_OWNED) /* SP acquiring list */ static VNET_DEFINE(LIST_HEAD(_spacqtree, secspacq), spacqtree); #define V_spacqtree VNET(spacqtree) static struct mtx spacq_lock; #define SPACQ_LOCK_INIT() \ mtx_init(&spacq_lock, "spacqtree", \ "fast ipsec security policy acquire list", MTX_DEF) #define SPACQ_LOCK_DESTROY() mtx_destroy(&spacq_lock) #define SPACQ_LOCK() mtx_lock(&spacq_lock) #define SPACQ_UNLOCK() mtx_unlock(&spacq_lock) #define SPACQ_LOCK_ASSERT() mtx_assert(&spacq_lock, MA_OWNED) /* search order for SAs */ static const u_int saorder_state_valid_prefer_old[] = { SADB_SASTATE_DYING, SADB_SASTATE_MATURE, }; static const u_int saorder_state_valid_prefer_new[] = { SADB_SASTATE_MATURE, SADB_SASTATE_DYING, }; static const u_int saorder_state_alive[] = { /* except DEAD */ SADB_SASTATE_MATURE, SADB_SASTATE_DYING, SADB_SASTATE_LARVAL }; static const u_int saorder_state_any[] = { SADB_SASTATE_MATURE, SADB_SASTATE_DYING, SADB_SASTATE_LARVAL, SADB_SASTATE_DEAD }; static const int minsize[] = { sizeof(struct sadb_msg), /* SADB_EXT_RESERVED */ sizeof(struct sadb_sa), /* SADB_EXT_SA */ sizeof(struct sadb_lifetime), /* SADB_EXT_LIFETIME_CURRENT */ sizeof(struct sadb_lifetime), /* SADB_EXT_LIFETIME_HARD */ sizeof(struct sadb_lifetime), /* SADB_EXT_LIFETIME_SOFT */ sizeof(struct sadb_address), /* SADB_EXT_ADDRESS_SRC */ sizeof(struct sadb_address), /* SADB_EXT_ADDRESS_DST */ sizeof(struct sadb_address), /* SADB_EXT_ADDRESS_PROXY */ sizeof(struct sadb_key), /* SADB_EXT_KEY_AUTH */ sizeof(struct sadb_key), /* SADB_EXT_KEY_ENCRYPT */ sizeof(struct sadb_ident), /* SADB_EXT_IDENTITY_SRC */ sizeof(struct sadb_ident), /* SADB_EXT_IDENTITY_DST */ sizeof(struct sadb_sens), /* SADB_EXT_SENSITIVITY */ sizeof(struct sadb_prop), /* SADB_EXT_PROPOSAL */ sizeof(struct sadb_supported), /* SADB_EXT_SUPPORTED_AUTH */ sizeof(struct sadb_supported), /* SADB_EXT_SUPPORTED_ENCRYPT */ sizeof(struct sadb_spirange), /* SADB_EXT_SPIRANGE */ 0, /* SADB_X_EXT_KMPRIVATE */ sizeof(struct sadb_x_policy), /* SADB_X_EXT_POLICY */ sizeof(struct sadb_x_sa2), /* SADB_X_SA2 */ sizeof(struct sadb_x_nat_t_type),/* SADB_X_EXT_NAT_T_TYPE */ sizeof(struct sadb_x_nat_t_port),/* SADB_X_EXT_NAT_T_SPORT */ sizeof(struct sadb_x_nat_t_port),/* SADB_X_EXT_NAT_T_DPORT */ sizeof(struct sadb_address), /* SADB_X_EXT_NAT_T_OAI */ sizeof(struct sadb_address), /* SADB_X_EXT_NAT_T_OAR */ sizeof(struct sadb_x_nat_t_frag),/* SADB_X_EXT_NAT_T_FRAG */ }; static const int maxsize[] = { sizeof(struct sadb_msg), /* SADB_EXT_RESERVED */ sizeof(struct sadb_sa), /* SADB_EXT_SA */ sizeof(struct sadb_lifetime), /* SADB_EXT_LIFETIME_CURRENT */ sizeof(struct sadb_lifetime), /* SADB_EXT_LIFETIME_HARD */ sizeof(struct sadb_lifetime), /* SADB_EXT_LIFETIME_SOFT */ 0, /* SADB_EXT_ADDRESS_SRC */ 0, /* SADB_EXT_ADDRESS_DST */ 0, /* SADB_EXT_ADDRESS_PROXY */ 0, /* SADB_EXT_KEY_AUTH */ 0, /* SADB_EXT_KEY_ENCRYPT */ 0, /* SADB_EXT_IDENTITY_SRC */ 0, /* SADB_EXT_IDENTITY_DST */ 0, /* SADB_EXT_SENSITIVITY */ 0, /* SADB_EXT_PROPOSAL */ 0, /* SADB_EXT_SUPPORTED_AUTH */ 0, /* SADB_EXT_SUPPORTED_ENCRYPT */ sizeof(struct sadb_spirange), /* SADB_EXT_SPIRANGE */ 0, /* SADB_X_EXT_KMPRIVATE */ 0, /* SADB_X_EXT_POLICY */ sizeof(struct sadb_x_sa2), /* SADB_X_SA2 */ sizeof(struct sadb_x_nat_t_type),/* SADB_X_EXT_NAT_T_TYPE */ sizeof(struct sadb_x_nat_t_port),/* SADB_X_EXT_NAT_T_SPORT */ sizeof(struct sadb_x_nat_t_port),/* SADB_X_EXT_NAT_T_DPORT */ 0, /* SADB_X_EXT_NAT_T_OAI */ 0, /* SADB_X_EXT_NAT_T_OAR */ sizeof(struct sadb_x_nat_t_frag),/* SADB_X_EXT_NAT_T_FRAG */ }; static VNET_DEFINE(int, ipsec_esp_keymin) = 256; #define V_ipsec_esp_keymin VNET(ipsec_esp_keymin) static VNET_DEFINE(int, ipsec_esp_auth) = 0; #define V_ipsec_esp_auth VNET(ipsec_esp_auth) static VNET_DEFINE(int, ipsec_ah_keymin) = 128; #define V_ipsec_ah_keymin VNET(ipsec_ah_keymin) #ifdef SYSCTL_DECL SYSCTL_DECL(_net_key); #endif SYSCTL_VNET_INT(_net_key, KEYCTL_DEBUG_LEVEL, debug, CTLFLAG_RW, &VNET_NAME(key_debug_level), 0, ""); /* max count of trial for the decision of spi value */ SYSCTL_VNET_INT(_net_key, KEYCTL_SPI_TRY, spi_trycnt, CTLFLAG_RW, &VNET_NAME(key_spi_trycnt), 0, ""); /* minimum spi value to allocate automatically. */ SYSCTL_VNET_INT(_net_key, KEYCTL_SPI_MIN_VALUE, spi_minval, CTLFLAG_RW, &VNET_NAME(key_spi_minval), 0, ""); /* maximun spi value to allocate automatically. */ SYSCTL_VNET_INT(_net_key, KEYCTL_SPI_MAX_VALUE, spi_maxval, CTLFLAG_RW, &VNET_NAME(key_spi_maxval), 0, ""); /* interval to initialize randseed */ SYSCTL_VNET_INT(_net_key, KEYCTL_RANDOM_INT, int_random, CTLFLAG_RW, &VNET_NAME(key_int_random), 0, ""); /* lifetime for larval SA */ SYSCTL_VNET_INT(_net_key, KEYCTL_LARVAL_LIFETIME, larval_lifetime, CTLFLAG_RW, &VNET_NAME(key_larval_lifetime), 0, ""); /* counter for blocking to send SADB_ACQUIRE to IKEd */ SYSCTL_VNET_INT(_net_key, KEYCTL_BLOCKACQ_COUNT, blockacq_count, CTLFLAG_RW, &VNET_NAME(key_blockacq_count), 0, ""); /* lifetime for blocking to send SADB_ACQUIRE to IKEd */ SYSCTL_VNET_INT(_net_key, KEYCTL_BLOCKACQ_LIFETIME, blockacq_lifetime, CTLFLAG_RW, &VNET_NAME(key_blockacq_lifetime), 0, ""); /* ESP auth */ SYSCTL_VNET_INT(_net_key, KEYCTL_ESP_AUTH, esp_auth, CTLFLAG_RW, &VNET_NAME(ipsec_esp_auth), 0, ""); /* minimum ESP key length */ SYSCTL_VNET_INT(_net_key, KEYCTL_ESP_KEYMIN, esp_keymin, CTLFLAG_RW, &VNET_NAME(ipsec_esp_keymin), 0, ""); /* minimum AH key length */ SYSCTL_VNET_INT(_net_key, KEYCTL_AH_KEYMIN, ah_keymin, CTLFLAG_RW, &VNET_NAME(ipsec_ah_keymin), 0, ""); /* perfered old SA rather than new SA */ SYSCTL_VNET_INT(_net_key, KEYCTL_PREFERED_OLDSA, preferred_oldsa, CTLFLAG_RW, &VNET_NAME(key_preferred_oldsa), 0, ""); #define __LIST_CHAINED(elm) \ (!((elm)->chain.le_next == NULL && (elm)->chain.le_prev == NULL)) #define LIST_INSERT_TAIL(head, elm, type, field) \ do {\ struct type *curelm = LIST_FIRST(head); \ if (curelm == NULL) {\ LIST_INSERT_HEAD(head, elm, field); \ } else { \ while (LIST_NEXT(curelm, field)) \ curelm = LIST_NEXT(curelm, field);\ LIST_INSERT_AFTER(curelm, elm, field);\ }\ } while (0) #define KEY_CHKSASTATE(head, sav, name) \ do { \ if ((head) != (sav)) { \ ipseclog((LOG_DEBUG, "%s: state mismatched (TREE=%d SA=%d)\n", \ (name), (head), (sav))); \ continue; \ } \ } while (0) #define KEY_CHKSPDIR(head, sp, name) \ do { \ if ((head) != (sp)) { \ ipseclog((LOG_DEBUG, "%s: direction mismatched (TREE=%d SP=%d), " \ "anyway continue.\n", \ (name), (head), (sp))); \ } \ } while (0) MALLOC_DEFINE(M_IPSEC_SA, "secasvar", "ipsec security association"); MALLOC_DEFINE(M_IPSEC_SAH, "sahead", "ipsec sa head"); MALLOC_DEFINE(M_IPSEC_SP, "ipsecpolicy", "ipsec security policy"); MALLOC_DEFINE(M_IPSEC_SR, "ipsecrequest", "ipsec security request"); MALLOC_DEFINE(M_IPSEC_MISC, "ipsec-misc", "ipsec miscellaneous"); MALLOC_DEFINE(M_IPSEC_SAQ, "ipsec-saq", "ipsec sa acquire"); MALLOC_DEFINE(M_IPSEC_SAR, "ipsec-reg", "ipsec sa acquire"); /* * set parameters into secpolicyindex buffer. * Must allocate secpolicyindex buffer passed to this function. */ #define KEY_SETSECSPIDX(_dir, s, d, ps, pd, ulp, idx) \ do { \ bzero((idx), sizeof(struct secpolicyindex)); \ (idx)->dir = (_dir); \ (idx)->prefs = (ps); \ (idx)->prefd = (pd); \ (idx)->ul_proto = (ulp); \ bcopy((s), &(idx)->src, ((const struct sockaddr *)(s))->sa_len); \ bcopy((d), &(idx)->dst, ((const struct sockaddr *)(d))->sa_len); \ } while (0) /* * set parameters into secasindex buffer. * Must allocate secasindex buffer before calling this function. */ #define KEY_SETSECASIDX(p, m, r, s, d, idx) \ do { \ bzero((idx), sizeof(struct secasindex)); \ (idx)->proto = (p); \ (idx)->mode = (m); \ (idx)->reqid = (r); \ bcopy((s), &(idx)->src, ((const struct sockaddr *)(s))->sa_len); \ bcopy((d), &(idx)->dst, ((const struct sockaddr *)(d))->sa_len); \ } while (0) /* key statistics */ struct _keystat { u_long getspi_count; /* the avarage of count to try to get new SPI */ } keystat; struct sadb_msghdr { struct sadb_msg *msg; struct sadb_ext *ext[SADB_EXT_MAX + 1]; int extoff[SADB_EXT_MAX + 1]; int extlen[SADB_EXT_MAX + 1]; }; static struct secasvar *key_allocsa_policy __P((const struct secasindex *)); static void key_freesp_so __P((struct secpolicy **)); static struct secasvar *key_do_allocsa_policy __P((struct secashead *, u_int)); static void key_delsp __P((struct secpolicy *)); static struct secpolicy *key_getsp __P((struct secpolicyindex *)); static void _key_delsp(struct secpolicy *sp); static struct secpolicy *key_getspbyid __P((u_int32_t)); static u_int32_t key_newreqid __P((void)); static struct mbuf *key_gather_mbuf __P((struct mbuf *, const struct sadb_msghdr *, int, int, ...)); static int key_spdadd __P((struct socket *, struct mbuf *, const struct sadb_msghdr *)); static u_int32_t key_getnewspid __P((void)); static int key_spddelete __P((struct socket *, struct mbuf *, const struct sadb_msghdr *)); static int key_spddelete2 __P((struct socket *, struct mbuf *, const struct sadb_msghdr *)); static int key_spdget __P((struct socket *, struct mbuf *, const struct sadb_msghdr *)); static int key_spdflush __P((struct socket *, struct mbuf *, const struct sadb_msghdr *)); static int key_spddump __P((struct socket *, struct mbuf *, const struct sadb_msghdr *)); static struct mbuf *key_setdumpsp __P((struct secpolicy *, u_int8_t, u_int32_t, u_int32_t)); static u_int key_getspreqmsglen __P((struct secpolicy *)); static int key_spdexpire __P((struct secpolicy *)); static struct secashead *key_newsah __P((struct secasindex *)); static void key_delsah __P((struct secashead *)); static struct secasvar *key_newsav __P((struct mbuf *, const struct sadb_msghdr *, struct secashead *, int *, const char*, int)); #define KEY_NEWSAV(m, sadb, sah, e) \ key_newsav(m, sadb, sah, e, __FILE__, __LINE__) static void key_delsav __P((struct secasvar *)); static struct secashead *key_getsah __P((struct secasindex *)); static struct secasvar *key_checkspidup __P((struct secasindex *, u_int32_t)); static struct secasvar *key_getsavbyspi __P((struct secashead *, u_int32_t)); static int key_setsaval __P((struct secasvar *, struct mbuf *, const struct sadb_msghdr *)); static int key_mature __P((struct secasvar *)); static struct mbuf *key_setdumpsa __P((struct secasvar *, u_int8_t, u_int8_t, u_int32_t, u_int32_t)); static struct mbuf *key_setsadbmsg __P((u_int8_t, u_int16_t, u_int8_t, u_int32_t, pid_t, u_int16_t)); static struct mbuf *key_setsadbsa __P((struct secasvar *)); static struct mbuf *key_setsadbaddr __P((u_int16_t, const struct sockaddr *, u_int8_t, u_int16_t)); #ifdef IPSEC_NAT_T static struct mbuf *key_setsadbxport(u_int16_t, u_int16_t); static struct mbuf *key_setsadbxtype(u_int16_t); #endif static void key_porttosaddr(struct sockaddr *, u_int16_t); #define KEY_PORTTOSADDR(saddr, port) \ key_porttosaddr((struct sockaddr *)(saddr), (port)) static struct mbuf *key_setsadbxsa2 __P((u_int8_t, u_int32_t, u_int32_t)); static struct mbuf *key_setsadbxpolicy __P((u_int16_t, u_int8_t, u_int32_t)); static struct seckey *key_dup_keymsg(const struct sadb_key *, u_int, struct malloc_type *); static struct seclifetime *key_dup_lifemsg(const struct sadb_lifetime *src, struct malloc_type *type); #ifdef INET6 static int key_ismyaddr6 __P((struct sockaddr_in6 *)); #endif /* flags for key_cmpsaidx() */ #define CMP_HEAD 1 /* protocol, addresses. */ #define CMP_MODE_REQID 2 /* additionally HEAD, reqid, mode. */ #define CMP_REQID 3 /* additionally HEAD, reaid. */ #define CMP_EXACTLY 4 /* all elements. */ static int key_cmpsaidx __P((const struct secasindex *, const struct secasindex *, int)); static int key_cmpspidx_exactly __P((struct secpolicyindex *, struct secpolicyindex *)); static int key_cmpspidx_withmask __P((struct secpolicyindex *, struct secpolicyindex *)); static int key_sockaddrcmp __P((const struct sockaddr *, const struct sockaddr *, int)); static int key_bbcmp __P((const void *, const void *, u_int)); static u_int16_t key_satype2proto __P((u_int8_t)); static u_int8_t key_proto2satype __P((u_int16_t)); static int key_getspi __P((struct socket *, struct mbuf *, const struct sadb_msghdr *)); static u_int32_t key_do_getnewspi __P((struct sadb_spirange *, struct secasindex *)); static int key_update __P((struct socket *, struct mbuf *, const struct sadb_msghdr *)); #ifdef IPSEC_DOSEQCHECK static struct secasvar *key_getsavbyseq __P((struct secashead *, u_int32_t)); #endif static int key_add __P((struct socket *, struct mbuf *, const struct sadb_msghdr *)); static int key_setident __P((struct secashead *, struct mbuf *, const struct sadb_msghdr *)); static struct mbuf *key_getmsgbuf_x1 __P((struct mbuf *, const struct sadb_msghdr *)); static int key_delete __P((struct socket *, struct mbuf *, const struct sadb_msghdr *)); static int key_get __P((struct socket *, struct mbuf *, const struct sadb_msghdr *)); static void key_getcomb_setlifetime __P((struct sadb_comb *)); static struct mbuf *key_getcomb_esp __P((void)); static struct mbuf *key_getcomb_ah __P((void)); static struct mbuf *key_getcomb_ipcomp __P((void)); static struct mbuf *key_getprop __P((const struct secasindex *)); static int key_acquire __P((const struct secasindex *, struct secpolicy *)); static struct secacq *key_newacq __P((const struct secasindex *)); static struct secacq *key_getacq __P((const struct secasindex *)); static struct secacq *key_getacqbyseq __P((u_int32_t)); static struct secspacq *key_newspacq __P((struct secpolicyindex *)); static struct secspacq *key_getspacq __P((struct secpolicyindex *)); static int key_acquire2 __P((struct socket *, struct mbuf *, const struct sadb_msghdr *)); static int key_register __P((struct socket *, struct mbuf *, const struct sadb_msghdr *)); static int key_expire __P((struct secasvar *)); static int key_flush __P((struct socket *, struct mbuf *, const struct sadb_msghdr *)); static int key_dump __P((struct socket *, struct mbuf *, const struct sadb_msghdr *)); static int key_promisc __P((struct socket *, struct mbuf *, const struct sadb_msghdr *)); static int key_senderror __P((struct socket *, struct mbuf *, int)); static int key_validate_ext __P((const struct sadb_ext *, int)); static int key_align __P((struct mbuf *, struct sadb_msghdr *)); static struct mbuf *key_setlifetime(struct seclifetime *src, u_int16_t exttype); static struct mbuf *key_setkey(struct seckey *src, u_int16_t exttype); #if 0 static const char *key_getfqdn __P((void)); static const char *key_getuserfqdn __P((void)); #endif static void key_sa_chgstate __P((struct secasvar *, u_int8_t)); static struct mbuf *key_alloc_mbuf __P((int)); static __inline void sa_initref(struct secasvar *sav) { refcount_init(&sav->refcnt, 1); } static __inline void sa_addref(struct secasvar *sav) { refcount_acquire(&sav->refcnt); IPSEC_ASSERT(sav->refcnt != 0, ("SA refcnt overflow")); } static __inline int sa_delref(struct secasvar *sav) { IPSEC_ASSERT(sav->refcnt > 0, ("SA refcnt underflow")); return (refcount_release(&sav->refcnt)); } #define SP_ADDREF(p) do { \ (p)->refcnt++; \ IPSEC_ASSERT((p)->refcnt != 0, ("SP refcnt overflow")); \ } while (0) #define SP_DELREF(p) do { \ IPSEC_ASSERT((p)->refcnt > 0, ("SP refcnt underflow")); \ (p)->refcnt--; \ } while (0) /* * Update the refcnt while holding the SPTREE lock. */ void key_addref(struct secpolicy *sp) { SPTREE_LOCK(); SP_ADDREF(sp); SPTREE_UNLOCK(); } /* * Return 0 when there are known to be no SP's for the specified * direction. Otherwise return 1. This is used by IPsec code * to optimize performance. */ int key_havesp(u_int dir) { return (dir == IPSEC_DIR_INBOUND || dir == IPSEC_DIR_OUTBOUND ? LIST_FIRST(&V_sptree[dir]) != NULL : 1); } /* %%% IPsec policy management */ /* * allocating a SP for OUTBOUND or INBOUND packet. * Must call key_freesp() later. * OUT: NULL: not found * others: found and return the pointer. */ struct secpolicy * key_allocsp(struct secpolicyindex *spidx, u_int dir, const char* where, int tag) { struct secpolicy *sp; IPSEC_ASSERT(spidx != NULL, ("null spidx")); IPSEC_ASSERT(dir == IPSEC_DIR_INBOUND || dir == IPSEC_DIR_OUTBOUND, ("invalid direction %u", dir)); KEYDEBUG(KEYDEBUG_IPSEC_STAMP, printf("DP %s from %s:%u\n", __func__, where, tag)); /* get a SP entry */ KEYDEBUG(KEYDEBUG_IPSEC_DATA, printf("*** objects\n"); kdebug_secpolicyindex(spidx)); SPTREE_LOCK(); LIST_FOREACH(sp, &V_sptree[dir], chain) { KEYDEBUG(KEYDEBUG_IPSEC_DATA, printf("*** in SPD\n"); kdebug_secpolicyindex(&sp->spidx)); if (sp->state == IPSEC_SPSTATE_DEAD) continue; if (key_cmpspidx_withmask(&sp->spidx, spidx)) goto found; } sp = NULL; found: if (sp) { /* sanity check */ KEY_CHKSPDIR(sp->spidx.dir, dir, __func__); /* found a SPD entry */ sp->lastused = time_second; SP_ADDREF(sp); } SPTREE_UNLOCK(); KEYDEBUG(KEYDEBUG_IPSEC_STAMP, printf("DP %s return SP:%p (ID=%u) refcnt %u\n", __func__, sp, sp ? sp->id : 0, sp ? sp->refcnt : 0)); return sp; } /* * allocating a SP for OUTBOUND or INBOUND packet. * Must call key_freesp() later. * OUT: NULL: not found * others: found and return the pointer. */ struct secpolicy * key_allocsp2(u_int32_t spi, union sockaddr_union *dst, u_int8_t proto, u_int dir, const char* where, int tag) { struct secpolicy *sp; IPSEC_ASSERT(dst != NULL, ("null dst")); IPSEC_ASSERT(dir == IPSEC_DIR_INBOUND || dir == IPSEC_DIR_OUTBOUND, ("invalid direction %u", dir)); KEYDEBUG(KEYDEBUG_IPSEC_STAMP, printf("DP %s from %s:%u\n", __func__, where, tag)); /* get a SP entry */ KEYDEBUG(KEYDEBUG_IPSEC_DATA, printf("*** objects\n"); printf("spi %u proto %u dir %u\n", spi, proto, dir); kdebug_sockaddr(&dst->sa)); SPTREE_LOCK(); LIST_FOREACH(sp, &V_sptree[dir], chain) { KEYDEBUG(KEYDEBUG_IPSEC_DATA, printf("*** in SPD\n"); kdebug_secpolicyindex(&sp->spidx)); if (sp->state == IPSEC_SPSTATE_DEAD) continue; /* compare simple values, then dst address */ if (sp->spidx.ul_proto != proto) continue; /* NB: spi's must exist and match */ if (!sp->req || !sp->req->sav || sp->req->sav->spi != spi) continue; if (key_sockaddrcmp(&sp->spidx.dst.sa, &dst->sa, 1) == 0) goto found; } sp = NULL; found: if (sp) { /* sanity check */ KEY_CHKSPDIR(sp->spidx.dir, dir, __func__); /* found a SPD entry */ sp->lastused = time_second; SP_ADDREF(sp); } SPTREE_UNLOCK(); KEYDEBUG(KEYDEBUG_IPSEC_STAMP, printf("DP %s return SP:%p (ID=%u) refcnt %u\n", __func__, sp, sp ? sp->id : 0, sp ? sp->refcnt : 0)); return sp; } #if 0 /* * return a policy that matches this particular inbound packet. * XXX slow */ struct secpolicy * key_gettunnel(const struct sockaddr *osrc, const struct sockaddr *odst, const struct sockaddr *isrc, const struct sockaddr *idst, const char* where, int tag) { struct secpolicy *sp; const int dir = IPSEC_DIR_INBOUND; struct ipsecrequest *r1, *r2, *p; struct secpolicyindex spidx; KEYDEBUG(KEYDEBUG_IPSEC_STAMP, printf("DP %s from %s:%u\n", __func__, where, tag)); if (isrc->sa_family != idst->sa_family) { ipseclog((LOG_ERR, "%s: protocol family mismatched %d != %d\n.", __func__, isrc->sa_family, idst->sa_family)); sp = NULL; goto done; } SPTREE_LOCK(); LIST_FOREACH(sp, &V_sptree[dir], chain) { if (sp->state == IPSEC_SPSTATE_DEAD) continue; r1 = r2 = NULL; for (p = sp->req; p; p = p->next) { if (p->saidx.mode != IPSEC_MODE_TUNNEL) continue; r1 = r2; r2 = p; if (!r1) { /* here we look at address matches only */ spidx = sp->spidx; if (isrc->sa_len > sizeof(spidx.src) || idst->sa_len > sizeof(spidx.dst)) continue; bcopy(isrc, &spidx.src, isrc->sa_len); bcopy(idst, &spidx.dst, idst->sa_len); if (!key_cmpspidx_withmask(&sp->spidx, &spidx)) continue; } else { if (key_sockaddrcmp(&r1->saidx.src.sa, isrc, 0) || key_sockaddrcmp(&r1->saidx.dst.sa, idst, 0)) continue; } if (key_sockaddrcmp(&r2->saidx.src.sa, osrc, 0) || key_sockaddrcmp(&r2->saidx.dst.sa, odst, 0)) continue; goto found; } } sp = NULL; found: if (sp) { sp->lastused = time_second; SP_ADDREF(sp); } SPTREE_UNLOCK(); done: KEYDEBUG(KEYDEBUG_IPSEC_STAMP, printf("DP %s return SP:%p (ID=%u) refcnt %u\n", __func__, sp, sp ? sp->id : 0, sp ? sp->refcnt : 0)); return sp; } #endif /* * allocating an SA entry for an *OUTBOUND* packet. * checking each request entries in SP, and acquire an SA if need. * OUT: 0: there are valid requests. * ENOENT: policy may be valid, but SA with REQUIRE is on acquiring. */ int key_checkrequest(struct ipsecrequest *isr, const struct secasindex *saidx) { u_int level; int error; IPSEC_ASSERT(isr != NULL, ("null isr")); IPSEC_ASSERT(saidx != NULL, ("null saidx")); IPSEC_ASSERT(saidx->mode == IPSEC_MODE_TRANSPORT || saidx->mode == IPSEC_MODE_TUNNEL, ("unexpected policy %u", saidx->mode)); /* * XXX guard against protocol callbacks from the crypto * thread as they reference ipsecrequest.sav which we * temporarily null out below. Need to rethink how we * handle bundled SA's in the callback thread. */ IPSECREQUEST_LOCK_ASSERT(isr); /* get current level */ level = ipsec_get_reqlevel(isr); #if 0 /* * We do allocate new SA only if the state of SA in the holder is * SADB_SASTATE_DEAD. The SA for outbound must be the oldest. */ if (isr->sav != NULL) { if (isr->sav->sah == NULL) panic("%s: sah is null.\n", __func__); if (isr->sav == (struct secasvar *)LIST_FIRST( &isr->sav->sah->savtree[SADB_SASTATE_DEAD])) { KEY_FREESAV(&isr->sav); isr->sav = NULL; } } #else /* * we free any SA stashed in the IPsec request because a different * SA may be involved each time this request is checked, either * because new SAs are being configured, or this request is * associated with an unconnected datagram socket, or this request * is associated with a system default policy. * * The operation may have negative impact to performance. We may * want to check cached SA carefully, rather than picking new SA * every time. */ if (isr->sav != NULL) { KEY_FREESAV(&isr->sav); isr->sav = NULL; } #endif /* * new SA allocation if no SA found. * key_allocsa_policy should allocate the oldest SA available. * See key_do_allocsa_policy(), and draft-jenkins-ipsec-rekeying-03.txt. */ if (isr->sav == NULL) isr->sav = key_allocsa_policy(saidx); /* When there is SA. */ if (isr->sav != NULL) { if (isr->sav->state != SADB_SASTATE_MATURE && isr->sav->state != SADB_SASTATE_DYING) return EINVAL; return 0; } /* there is no SA */ error = key_acquire(saidx, isr->sp); if (error != 0) { /* XXX What should I do ? */ ipseclog((LOG_DEBUG, "%s: error %d returned from key_acquire\n", __func__, error)); return error; } if (level != IPSEC_LEVEL_REQUIRE) { /* XXX sigh, the interface to this routine is botched */ IPSEC_ASSERT(isr->sav == NULL, ("unexpected SA")); return 0; } else { return ENOENT; } } /* * allocating a SA for policy entry from SAD. * NOTE: searching SAD of aliving state. * OUT: NULL: not found. * others: found and return the pointer. */ static struct secasvar * key_allocsa_policy(const struct secasindex *saidx) { #define N(a) _ARRAYLEN(a) struct secashead *sah; struct secasvar *sav; u_int stateidx, arraysize; const u_int *state_valid; state_valid = NULL; /* silence gcc */ arraysize = 0; /* silence gcc */ SAHTREE_LOCK(); LIST_FOREACH(sah, &V_sahtree, chain) { if (sah->state == SADB_SASTATE_DEAD) continue; if (key_cmpsaidx(&sah->saidx, saidx, CMP_MODE_REQID)) { if (V_key_preferred_oldsa) { state_valid = saorder_state_valid_prefer_old; arraysize = N(saorder_state_valid_prefer_old); } else { state_valid = saorder_state_valid_prefer_new; arraysize = N(saorder_state_valid_prefer_new); } break; } } SAHTREE_UNLOCK(); if (sah == NULL) return NULL; /* search valid state */ for (stateidx = 0; stateidx < arraysize; stateidx++) { sav = key_do_allocsa_policy(sah, state_valid[stateidx]); if (sav != NULL) return sav; } return NULL; #undef N } /* * searching SAD with direction, protocol, mode and state. * called by key_allocsa_policy(). * OUT: * NULL : not found * others : found, pointer to a SA. */ static struct secasvar * key_do_allocsa_policy(struct secashead *sah, u_int state) { struct secasvar *sav, *nextsav, *candidate, *d; /* initilize */ candidate = NULL; SAHTREE_LOCK(); for (sav = LIST_FIRST(&sah->savtree[state]); sav != NULL; sav = nextsav) { nextsav = LIST_NEXT(sav, chain); /* sanity check */ KEY_CHKSASTATE(sav->state, state, __func__); /* initialize */ if (candidate == NULL) { candidate = sav; continue; } /* Which SA is the better ? */ IPSEC_ASSERT(candidate->lft_c != NULL, ("null candidate lifetime")); IPSEC_ASSERT(sav->lft_c != NULL, ("null sav lifetime")); /* What the best method is to compare ? */ if (V_key_preferred_oldsa) { if (candidate->lft_c->addtime > sav->lft_c->addtime) { candidate = sav; } continue; /*NOTREACHED*/ } /* preferred new sa rather than old sa */ if (candidate->lft_c->addtime < sav->lft_c->addtime) { d = candidate; candidate = sav; } else d = sav; /* * prepared to delete the SA when there is more * suitable candidate and the lifetime of the SA is not * permanent. */ if (d->lft_h->addtime != 0) { struct mbuf *m, *result; u_int8_t satype; key_sa_chgstate(d, SADB_SASTATE_DEAD); IPSEC_ASSERT(d->refcnt > 0, ("bogus ref count")); satype = key_proto2satype(d->sah->saidx.proto); if (satype == 0) goto msgfail; m = key_setsadbmsg(SADB_DELETE, 0, satype, 0, 0, d->refcnt - 1); if (!m) goto msgfail; result = m; /* set sadb_address for saidx's. */ m = key_setsadbaddr(SADB_EXT_ADDRESS_SRC, &d->sah->saidx.src.sa, d->sah->saidx.src.sa.sa_len << 3, IPSEC_ULPROTO_ANY); if (!m) goto msgfail; m_cat(result, m); /* set sadb_address for saidx's. */ m = key_setsadbaddr(SADB_EXT_ADDRESS_DST, &d->sah->saidx.dst.sa, d->sah->saidx.dst.sa.sa_len << 3, IPSEC_ULPROTO_ANY); if (!m) goto msgfail; m_cat(result, m); /* create SA extension */ m = key_setsadbsa(d); if (!m) goto msgfail; m_cat(result, m); if (result->m_len < sizeof(struct sadb_msg)) { result = m_pullup(result, sizeof(struct sadb_msg)); if (result == NULL) goto msgfail; } result->m_pkthdr.len = 0; for (m = result; m; m = m->m_next) result->m_pkthdr.len += m->m_len; mtod(result, struct sadb_msg *)->sadb_msg_len = PFKEY_UNIT64(result->m_pkthdr.len); if (key_sendup_mbuf(NULL, result, KEY_SENDUP_REGISTERED)) goto msgfail; msgfail: KEY_FREESAV(&d); } } if (candidate) { sa_addref(candidate); KEYDEBUG(KEYDEBUG_IPSEC_STAMP, printf("DP %s cause refcnt++:%d SA:%p\n", __func__, candidate->refcnt, candidate)); } SAHTREE_UNLOCK(); return candidate; } /* * allocating a usable SA entry for a *INBOUND* packet. * Must call key_freesav() later. * OUT: positive: pointer to a usable sav (i.e. MATURE or DYING state). * NULL: not found, or error occured. * * In the comparison, no source address is used--for RFC2401 conformance. * To quote, from section 4.1: * A security association is uniquely identified by a triple consisting * of a Security Parameter Index (SPI), an IP Destination Address, and a * security protocol (AH or ESP) identifier. * Note that, however, we do need to keep source address in IPsec SA. * IKE specification and PF_KEY specification do assume that we * keep source address in IPsec SA. We see a tricky situation here. */ struct secasvar * key_allocsa( union sockaddr_union *dst, u_int proto, u_int32_t spi, const char* where, int tag) { struct secashead *sah; struct secasvar *sav; u_int stateidx, arraysize, state; const u_int *saorder_state_valid; int chkport; IPSEC_ASSERT(dst != NULL, ("null dst address")); KEYDEBUG(KEYDEBUG_IPSEC_STAMP, printf("DP %s from %s:%u\n", __func__, where, tag)); #ifdef IPSEC_NAT_T chkport = (dst->sa.sa_family == AF_INET && dst->sa.sa_len == sizeof(struct sockaddr_in) && dst->sin.sin_port != 0); #else chkport = 0; #endif /* * searching SAD. * XXX: to be checked internal IP header somewhere. Also when * IPsec tunnel packet is received. But ESP tunnel mode is * encrypted so we can't check internal IP header. */ SAHTREE_LOCK(); if (V_key_preferred_oldsa) { saorder_state_valid = saorder_state_valid_prefer_old; arraysize = _ARRAYLEN(saorder_state_valid_prefer_old); } else { saorder_state_valid = saorder_state_valid_prefer_new; arraysize = _ARRAYLEN(saorder_state_valid_prefer_new); } LIST_FOREACH(sah, &V_sahtree, chain) { /* search valid state */ for (stateidx = 0; stateidx < arraysize; stateidx++) { state = saorder_state_valid[stateidx]; LIST_FOREACH(sav, &sah->savtree[state], chain) { /* sanity check */ KEY_CHKSASTATE(sav->state, state, __func__); /* do not return entries w/ unusable state */ if (sav->state != SADB_SASTATE_MATURE && sav->state != SADB_SASTATE_DYING) continue; if (proto != sav->sah->saidx.proto) continue; if (spi != sav->spi) continue; #if 0 /* don't check src */ /* check src address */ if (key_sockaddrcmp(&src->sa, &sav->sah->saidx.src.sa, chkport) != 0) continue; #endif /* check dst address */ if (key_sockaddrcmp(&dst->sa, &sav->sah->saidx.dst.sa, chkport) != 0) continue; sa_addref(sav); goto done; } } } sav = NULL; done: SAHTREE_UNLOCK(); KEYDEBUG(KEYDEBUG_IPSEC_STAMP, printf("DP %s return SA:%p; refcnt %u\n", __func__, sav, sav ? sav->refcnt : 0)); return sav; } /* * Must be called after calling key_allocsp(). * For both the packet without socket and key_freeso(). */ void _key_freesp(struct secpolicy **spp, const char* where, int tag) { struct secpolicy *sp = *spp; IPSEC_ASSERT(sp != NULL, ("null sp")); SPTREE_LOCK(); SP_DELREF(sp); KEYDEBUG(KEYDEBUG_IPSEC_STAMP, printf("DP %s SP:%p (ID=%u) from %s:%u; refcnt now %u\n", __func__, sp, sp->id, where, tag, sp->refcnt)); if (sp->refcnt == 0) { *spp = NULL; key_delsp(sp); } SPTREE_UNLOCK(); } /* * Must be called after calling key_allocsp(). * For the packet with socket. */ void key_freeso(struct socket *so) { IPSEC_ASSERT(so != NULL, ("null so")); switch (so->so_proto->pr_domain->dom_family) { #if defined(INET) || defined(INET6) #ifdef INET case PF_INET: #endif #ifdef INET6 case PF_INET6: #endif { struct inpcb *pcb = sotoinpcb(so); /* Does it have a PCB ? */ if (pcb == NULL) return; key_freesp_so(&pcb->inp_sp->sp_in); key_freesp_so(&pcb->inp_sp->sp_out); } break; #endif /* INET || INET6 */ default: ipseclog((LOG_DEBUG, "%s: unknown address family=%d.\n", __func__, so->so_proto->pr_domain->dom_family)); return; } } static void key_freesp_so(struct secpolicy **sp) { IPSEC_ASSERT(sp != NULL && *sp != NULL, ("null sp")); if ((*sp)->policy == IPSEC_POLICY_ENTRUST || (*sp)->policy == IPSEC_POLICY_BYPASS) return; IPSEC_ASSERT((*sp)->policy == IPSEC_POLICY_IPSEC, ("invalid policy %u", (*sp)->policy)); KEY_FREESP(sp); } /* * Must be called after calling key_allocsa(). * This function is called by key_freesp() to free some SA allocated * for a policy. */ void key_freesav(struct secasvar **psav, const char* where, int tag) { struct secasvar *sav = *psav; IPSEC_ASSERT(sav != NULL, ("null sav")); if (sa_delref(sav)) { KEYDEBUG(KEYDEBUG_IPSEC_STAMP, printf("DP %s SA:%p (SPI %u) from %s:%u; refcnt now %u\n", __func__, sav, ntohl(sav->spi), where, tag, sav->refcnt)); *psav = NULL; key_delsav(sav); } else { KEYDEBUG(KEYDEBUG_IPSEC_STAMP, printf("DP %s SA:%p (SPI %u) from %s:%u; refcnt now %u\n", __func__, sav, ntohl(sav->spi), where, tag, sav->refcnt)); } } /* %%% SPD management */ /* * free security policy entry. */ static void key_delsp(struct secpolicy *sp) { struct ipsecrequest *isr, *nextisr; IPSEC_ASSERT(sp != NULL, ("null sp")); SPTREE_LOCK_ASSERT(); sp->state = IPSEC_SPSTATE_DEAD; IPSEC_ASSERT(sp->refcnt == 0, ("SP with references deleted (refcnt %u)", sp->refcnt)); /* remove from SP index */ if (__LIST_CHAINED(sp)) LIST_REMOVE(sp, chain); for (isr = sp->req; isr != NULL; isr = nextisr) { if (isr->sav != NULL) { KEY_FREESAV(&isr->sav); isr->sav = NULL; } nextisr = isr->next; ipsec_delisr(isr); } _key_delsp(sp); } /* * search SPD * OUT: NULL : not found * others : found, pointer to a SP. */ static struct secpolicy * key_getsp(struct secpolicyindex *spidx) { struct secpolicy *sp; IPSEC_ASSERT(spidx != NULL, ("null spidx")); SPTREE_LOCK(); LIST_FOREACH(sp, &V_sptree[spidx->dir], chain) { if (sp->state == IPSEC_SPSTATE_DEAD) continue; if (key_cmpspidx_exactly(spidx, &sp->spidx)) { SP_ADDREF(sp); break; } } SPTREE_UNLOCK(); return sp; } /* * get SP by index. * OUT: NULL : not found * others : found, pointer to a SP. */ static struct secpolicy * key_getspbyid(u_int32_t id) { struct secpolicy *sp; SPTREE_LOCK(); LIST_FOREACH(sp, &V_sptree[IPSEC_DIR_INBOUND], chain) { if (sp->state == IPSEC_SPSTATE_DEAD) continue; if (sp->id == id) { SP_ADDREF(sp); goto done; } } LIST_FOREACH(sp, &V_sptree[IPSEC_DIR_OUTBOUND], chain) { if (sp->state == IPSEC_SPSTATE_DEAD) continue; if (sp->id == id) { SP_ADDREF(sp); goto done; } } done: SPTREE_UNLOCK(); return sp; } struct secpolicy * key_newsp(const char* where, int tag) { struct secpolicy *newsp = NULL; newsp = (struct secpolicy *) malloc(sizeof(struct secpolicy), M_IPSEC_SP, M_NOWAIT|M_ZERO); if (newsp) { SECPOLICY_LOCK_INIT(newsp); newsp->refcnt = 1; newsp->req = NULL; } KEYDEBUG(KEYDEBUG_IPSEC_STAMP, printf("DP %s from %s:%u return SP:%p\n", __func__, where, tag, newsp)); return newsp; } static void _key_delsp(struct secpolicy *sp) { SECPOLICY_LOCK_DESTROY(sp); free(sp, M_IPSEC_SP); } /* * create secpolicy structure from sadb_x_policy structure. * NOTE: `state', `secpolicyindex' in secpolicy structure are not set, * so must be set properly later. */ struct secpolicy * key_msg2sp(xpl0, len, error) struct sadb_x_policy *xpl0; size_t len; int *error; { struct secpolicy *newsp; IPSEC_ASSERT(xpl0 != NULL, ("null xpl0")); IPSEC_ASSERT(len >= sizeof(*xpl0), ("policy too short: %zu", len)); if (len != PFKEY_EXTLEN(xpl0)) { ipseclog((LOG_DEBUG, "%s: Invalid msg length.\n", __func__)); *error = EINVAL; return NULL; } if ((newsp = KEY_NEWSP()) == NULL) { *error = ENOBUFS; return NULL; } newsp->spidx.dir = xpl0->sadb_x_policy_dir; newsp->policy = xpl0->sadb_x_policy_type; /* check policy */ switch (xpl0->sadb_x_policy_type) { case IPSEC_POLICY_DISCARD: case IPSEC_POLICY_NONE: case IPSEC_POLICY_ENTRUST: case IPSEC_POLICY_BYPASS: newsp->req = NULL; break; case IPSEC_POLICY_IPSEC: { int tlen; struct sadb_x_ipsecrequest *xisr; struct ipsecrequest **p_isr = &newsp->req; /* validity check */ if (PFKEY_EXTLEN(xpl0) < sizeof(*xpl0)) { ipseclog((LOG_DEBUG, "%s: Invalid msg length.\n", __func__)); KEY_FREESP(&newsp); *error = EINVAL; return NULL; } tlen = PFKEY_EXTLEN(xpl0) - sizeof(*xpl0); xisr = (struct sadb_x_ipsecrequest *)(xpl0 + 1); while (tlen > 0) { /* length check */ if (xisr->sadb_x_ipsecrequest_len < sizeof(*xisr)) { ipseclog((LOG_DEBUG, "%s: invalid ipsecrequest " "length.\n", __func__)); KEY_FREESP(&newsp); *error = EINVAL; return NULL; } /* allocate request buffer */ /* NB: data structure is zero'd */ *p_isr = ipsec_newisr(); if ((*p_isr) == NULL) { ipseclog((LOG_DEBUG, "%s: No more memory.\n", __func__)); KEY_FREESP(&newsp); *error = ENOBUFS; return NULL; } /* set values */ switch (xisr->sadb_x_ipsecrequest_proto) { case IPPROTO_ESP: case IPPROTO_AH: case IPPROTO_IPCOMP: break; default: ipseclog((LOG_DEBUG, "%s: invalid proto type=%u\n", __func__, xisr->sadb_x_ipsecrequest_proto)); KEY_FREESP(&newsp); *error = EPROTONOSUPPORT; return NULL; } (*p_isr)->saidx.proto = xisr->sadb_x_ipsecrequest_proto; switch (xisr->sadb_x_ipsecrequest_mode) { case IPSEC_MODE_TRANSPORT: case IPSEC_MODE_TUNNEL: break; case IPSEC_MODE_ANY: default: ipseclog((LOG_DEBUG, "%s: invalid mode=%u\n", __func__, xisr->sadb_x_ipsecrequest_mode)); KEY_FREESP(&newsp); *error = EINVAL; return NULL; } (*p_isr)->saidx.mode = xisr->sadb_x_ipsecrequest_mode; switch (xisr->sadb_x_ipsecrequest_level) { case IPSEC_LEVEL_DEFAULT: case IPSEC_LEVEL_USE: case IPSEC_LEVEL_REQUIRE: break; case IPSEC_LEVEL_UNIQUE: /* validity check */ /* * If range violation of reqid, kernel will * update it, don't refuse it. */ if (xisr->sadb_x_ipsecrequest_reqid > IPSEC_MANUAL_REQID_MAX) { ipseclog((LOG_DEBUG, "%s: reqid=%d range " "violation, updated by kernel.\n", __func__, xisr->sadb_x_ipsecrequest_reqid)); xisr->sadb_x_ipsecrequest_reqid = 0; } /* allocate new reqid id if reqid is zero. */ if (xisr->sadb_x_ipsecrequest_reqid == 0) { u_int32_t reqid; if ((reqid = key_newreqid()) == 0) { KEY_FREESP(&newsp); *error = ENOBUFS; return NULL; } (*p_isr)->saidx.reqid = reqid; xisr->sadb_x_ipsecrequest_reqid = reqid; } else { /* set it for manual keying. */ (*p_isr)->saidx.reqid = xisr->sadb_x_ipsecrequest_reqid; } break; default: ipseclog((LOG_DEBUG, "%s: invalid level=%u\n", __func__, xisr->sadb_x_ipsecrequest_level)); KEY_FREESP(&newsp); *error = EINVAL; return NULL; } (*p_isr)->level = xisr->sadb_x_ipsecrequest_level; /* set IP addresses if there */ if (xisr->sadb_x_ipsecrequest_len > sizeof(*xisr)) { struct sockaddr *paddr; paddr = (struct sockaddr *)(xisr + 1); /* validity check */ if (paddr->sa_len > sizeof((*p_isr)->saidx.src)) { ipseclog((LOG_DEBUG, "%s: invalid " "request address length.\n", __func__)); KEY_FREESP(&newsp); *error = EINVAL; return NULL; } bcopy(paddr, &(*p_isr)->saidx.src, paddr->sa_len); paddr = (struct sockaddr *)((caddr_t)paddr + paddr->sa_len); /* validity check */ if (paddr->sa_len > sizeof((*p_isr)->saidx.dst)) { ipseclog((LOG_DEBUG, "%s: invalid " "request address length.\n", __func__)); KEY_FREESP(&newsp); *error = EINVAL; return NULL; } bcopy(paddr, &(*p_isr)->saidx.dst, paddr->sa_len); } (*p_isr)->sp = newsp; /* initialization for the next. */ p_isr = &(*p_isr)->next; tlen -= xisr->sadb_x_ipsecrequest_len; /* validity check */ if (tlen < 0) { ipseclog((LOG_DEBUG, "%s: becoming tlen < 0.\n", __func__)); KEY_FREESP(&newsp); *error = EINVAL; return NULL; } xisr = (struct sadb_x_ipsecrequest *)((caddr_t)xisr + xisr->sadb_x_ipsecrequest_len); } } break; default: ipseclog((LOG_DEBUG, "%s: invalid policy type.\n", __func__)); KEY_FREESP(&newsp); *error = EINVAL; return NULL; } *error = 0; return newsp; } static u_int32_t key_newreqid() { static u_int32_t auto_reqid = IPSEC_MANUAL_REQID_MAX + 1; auto_reqid = (auto_reqid == ~0 ? IPSEC_MANUAL_REQID_MAX + 1 : auto_reqid + 1); /* XXX should be unique check */ return auto_reqid; } /* * copy secpolicy struct to sadb_x_policy structure indicated. */ struct mbuf * key_sp2msg(sp) struct secpolicy *sp; { struct sadb_x_policy *xpl; int tlen; caddr_t p; struct mbuf *m; IPSEC_ASSERT(sp != NULL, ("null policy")); tlen = key_getspreqmsglen(sp); m = key_alloc_mbuf(tlen); if (!m || m->m_next) { /*XXX*/ if (m) m_freem(m); return NULL; } m->m_len = tlen; m->m_next = NULL; xpl = mtod(m, struct sadb_x_policy *); bzero(xpl, tlen); xpl->sadb_x_policy_len = PFKEY_UNIT64(tlen); xpl->sadb_x_policy_exttype = SADB_X_EXT_POLICY; xpl->sadb_x_policy_type = sp->policy; xpl->sadb_x_policy_dir = sp->spidx.dir; xpl->sadb_x_policy_id = sp->id; p = (caddr_t)xpl + sizeof(*xpl); /* if is the policy for ipsec ? */ if (sp->policy == IPSEC_POLICY_IPSEC) { struct sadb_x_ipsecrequest *xisr; struct ipsecrequest *isr; for (isr = sp->req; isr != NULL; isr = isr->next) { xisr = (struct sadb_x_ipsecrequest *)p; xisr->sadb_x_ipsecrequest_proto = isr->saidx.proto; xisr->sadb_x_ipsecrequest_mode = isr->saidx.mode; xisr->sadb_x_ipsecrequest_level = isr->level; xisr->sadb_x_ipsecrequest_reqid = isr->saidx.reqid; p += sizeof(*xisr); bcopy(&isr->saidx.src, p, isr->saidx.src.sa.sa_len); p += isr->saidx.src.sa.sa_len; bcopy(&isr->saidx.dst, p, isr->saidx.dst.sa.sa_len); p += isr->saidx.src.sa.sa_len; xisr->sadb_x_ipsecrequest_len = PFKEY_ALIGN8(sizeof(*xisr) + isr->saidx.src.sa.sa_len + isr->saidx.dst.sa.sa_len); } } return m; } /* m will not be freed nor modified */ static struct mbuf * #ifdef __STDC__ key_gather_mbuf(struct mbuf *m, const struct sadb_msghdr *mhp, int ndeep, int nitem, ...) #else key_gather_mbuf(m, mhp, ndeep, nitem, va_alist) struct mbuf *m; const struct sadb_msghdr *mhp; int ndeep; int nitem; va_dcl #endif { va_list ap; int idx; int i; struct mbuf *result = NULL, *n; int len; IPSEC_ASSERT(m != NULL, ("null mbuf")); IPSEC_ASSERT(mhp != NULL, ("null msghdr")); va_start(ap, nitem); for (i = 0; i < nitem; i++) { idx = va_arg(ap, int); if (idx < 0 || idx > SADB_EXT_MAX) goto fail; /* don't attempt to pull empty extension */ if (idx == SADB_EXT_RESERVED && mhp->msg == NULL) continue; if (idx != SADB_EXT_RESERVED && (mhp->ext[idx] == NULL || mhp->extlen[idx] == 0)) continue; if (idx == SADB_EXT_RESERVED) { len = PFKEY_ALIGN8(sizeof(struct sadb_msg)); IPSEC_ASSERT(len <= MHLEN, ("header too big %u", len)); MGETHDR(n, M_DONTWAIT, MT_DATA); if (!n) goto fail; n->m_len = len; n->m_next = NULL; m_copydata(m, 0, sizeof(struct sadb_msg), mtod(n, caddr_t)); } else if (i < ndeep) { len = mhp->extlen[idx]; n = key_alloc_mbuf(len); if (!n || n->m_next) { /*XXX*/ if (n) m_freem(n); goto fail; } m_copydata(m, mhp->extoff[idx], mhp->extlen[idx], mtod(n, caddr_t)); } else { n = m_copym(m, mhp->extoff[idx], mhp->extlen[idx], M_DONTWAIT); } if (n == NULL) goto fail; if (result) m_cat(result, n); else result = n; } va_end(ap); if ((result->m_flags & M_PKTHDR) != 0) { result->m_pkthdr.len = 0; for (n = result; n; n = n->m_next) result->m_pkthdr.len += n->m_len; } return result; fail: m_freem(result); return NULL; } /* * SADB_X_SPDADD, SADB_X_SPDSETIDX or SADB_X_SPDUPDATE processing * add an entry to SP database, when received * * from the user(?). * Adding to SP database, * and send * * to the socket which was send. * * SPDADD set a unique policy entry. * SPDSETIDX like SPDADD without a part of policy requests. * SPDUPDATE replace a unique policy entry. * * m will always be freed. */ static int key_spdadd(so, m, mhp) struct socket *so; struct mbuf *m; const struct sadb_msghdr *mhp; { struct sadb_address *src0, *dst0; struct sadb_x_policy *xpl0, *xpl; struct sadb_lifetime *lft = NULL; struct secpolicyindex spidx; struct secpolicy *newsp; int error; IPSEC_ASSERT(so != NULL, ("null socket")); IPSEC_ASSERT(m != NULL, ("null mbuf")); IPSEC_ASSERT(mhp != NULL, ("null msghdr")); IPSEC_ASSERT(mhp->msg != NULL, ("null msg")); if (mhp->ext[SADB_EXT_ADDRESS_SRC] == NULL || mhp->ext[SADB_EXT_ADDRESS_DST] == NULL || mhp->ext[SADB_X_EXT_POLICY] == NULL) { ipseclog((LOG_DEBUG, "key_spdadd: invalid message is passed.\n")); return key_senderror(so, m, EINVAL); } if (mhp->extlen[SADB_EXT_ADDRESS_SRC] < sizeof(struct sadb_address) || mhp->extlen[SADB_EXT_ADDRESS_DST] < sizeof(struct sadb_address) || mhp->extlen[SADB_X_EXT_POLICY] < sizeof(struct sadb_x_policy)) { ipseclog((LOG_DEBUG, "%s: invalid message is passed.\n", __func__)); return key_senderror(so, m, EINVAL); } if (mhp->ext[SADB_EXT_LIFETIME_HARD] != NULL) { if (mhp->extlen[SADB_EXT_LIFETIME_HARD] < sizeof(struct sadb_lifetime)) { ipseclog((LOG_DEBUG, "%s: invalid message is passed.\n", __func__)); return key_senderror(so, m, EINVAL); } lft = (struct sadb_lifetime *)mhp->ext[SADB_EXT_LIFETIME_HARD]; } src0 = (struct sadb_address *)mhp->ext[SADB_EXT_ADDRESS_SRC]; dst0 = (struct sadb_address *)mhp->ext[SADB_EXT_ADDRESS_DST]; xpl0 = (struct sadb_x_policy *)mhp->ext[SADB_X_EXT_POLICY]; /* * Note: do not parse SADB_X_EXT_NAT_T_* here: * we are processing traffic endpoints. */ /* make secindex */ /* XXX boundary check against sa_len */ KEY_SETSECSPIDX(xpl0->sadb_x_policy_dir, src0 + 1, dst0 + 1, src0->sadb_address_prefixlen, dst0->sadb_address_prefixlen, src0->sadb_address_proto, &spidx); /* checking the direciton. */ switch (xpl0->sadb_x_policy_dir) { case IPSEC_DIR_INBOUND: case IPSEC_DIR_OUTBOUND: break; default: ipseclog((LOG_DEBUG, "%s: Invalid SP direction.\n", __func__)); mhp->msg->sadb_msg_errno = EINVAL; return 0; } /* check policy */ /* key_spdadd() accepts DISCARD, NONE and IPSEC. */ if (xpl0->sadb_x_policy_type == IPSEC_POLICY_ENTRUST || xpl0->sadb_x_policy_type == IPSEC_POLICY_BYPASS) { ipseclog((LOG_DEBUG, "%s: Invalid policy type.\n", __func__)); return key_senderror(so, m, EINVAL); } /* policy requests are mandatory when action is ipsec. */ if (mhp->msg->sadb_msg_type != SADB_X_SPDSETIDX && xpl0->sadb_x_policy_type == IPSEC_POLICY_IPSEC && mhp->extlen[SADB_X_EXT_POLICY] <= sizeof(*xpl0)) { ipseclog((LOG_DEBUG, "%s: some policy requests part required\n", __func__)); return key_senderror(so, m, EINVAL); } /* * checking there is SP already or not. * SPDUPDATE doesn't depend on whether there is a SP or not. * If the type is either SPDADD or SPDSETIDX AND a SP is found, * then error. */ newsp = key_getsp(&spidx); if (mhp->msg->sadb_msg_type == SADB_X_SPDUPDATE) { if (newsp) { + SPTREE_LOCK(); newsp->state = IPSEC_SPSTATE_DEAD; + SPTREE_UNLOCK(); KEY_FREESP(&newsp); } } else { if (newsp != NULL) { KEY_FREESP(&newsp); ipseclog((LOG_DEBUG, "%s: a SP entry exists already.\n", __func__)); return key_senderror(so, m, EEXIST); } } /* allocation new SP entry */ if ((newsp = key_msg2sp(xpl0, PFKEY_EXTLEN(xpl0), &error)) == NULL) { return key_senderror(so, m, error); } if ((newsp->id = key_getnewspid()) == 0) { _key_delsp(newsp); return key_senderror(so, m, ENOBUFS); } /* XXX boundary check against sa_len */ KEY_SETSECSPIDX(xpl0->sadb_x_policy_dir, src0 + 1, dst0 + 1, src0->sadb_address_prefixlen, dst0->sadb_address_prefixlen, src0->sadb_address_proto, &newsp->spidx); /* sanity check on addr pair */ if (((struct sockaddr *)(src0 + 1))->sa_family != ((struct sockaddr *)(dst0+ 1))->sa_family) { _key_delsp(newsp); return key_senderror(so, m, EINVAL); } if (((struct sockaddr *)(src0 + 1))->sa_len != ((struct sockaddr *)(dst0+ 1))->sa_len) { _key_delsp(newsp); return key_senderror(so, m, EINVAL); } #if 1 if (newsp->req && newsp->req->saidx.src.sa.sa_family && newsp->req->saidx.dst.sa.sa_family) { if (newsp->req->saidx.src.sa.sa_family != newsp->req->saidx.dst.sa.sa_family) { _key_delsp(newsp); return key_senderror(so, m, EINVAL); } } #endif newsp->created = time_second; newsp->lastused = newsp->created; newsp->lifetime = lft ? lft->sadb_lifetime_addtime : 0; newsp->validtime = lft ? lft->sadb_lifetime_usetime : 0; newsp->refcnt = 1; /* do not reclaim until I say I do */ newsp->state = IPSEC_SPSTATE_ALIVE; LIST_INSERT_TAIL(&V_sptree[newsp->spidx.dir], newsp, secpolicy, chain); /* delete the entry in spacqtree */ if (mhp->msg->sadb_msg_type == SADB_X_SPDUPDATE) { struct secspacq *spacq = key_getspacq(&spidx); if (spacq != NULL) { /* reset counter in order to deletion by timehandler. */ spacq->created = time_second; spacq->count = 0; SPACQ_UNLOCK(); } } { struct mbuf *n, *mpolicy; struct sadb_msg *newmsg; int off; /* * Note: do not send SADB_X_EXT_NAT_T_* here: * we are sending traffic endpoints. */ /* create new sadb_msg to reply. */ if (lft) { n = key_gather_mbuf(m, mhp, 2, 5, SADB_EXT_RESERVED, SADB_X_EXT_POLICY, SADB_EXT_LIFETIME_HARD, SADB_EXT_ADDRESS_SRC, SADB_EXT_ADDRESS_DST); } else { n = key_gather_mbuf(m, mhp, 2, 4, SADB_EXT_RESERVED, SADB_X_EXT_POLICY, SADB_EXT_ADDRESS_SRC, SADB_EXT_ADDRESS_DST); } if (!n) return key_senderror(so, m, ENOBUFS); if (n->m_len < sizeof(*newmsg)) { n = m_pullup(n, sizeof(*newmsg)); if (!n) return key_senderror(so, m, ENOBUFS); } newmsg = mtod(n, struct sadb_msg *); newmsg->sadb_msg_errno = 0; newmsg->sadb_msg_len = PFKEY_UNIT64(n->m_pkthdr.len); off = 0; mpolicy = m_pulldown(n, PFKEY_ALIGN8(sizeof(struct sadb_msg)), sizeof(*xpl), &off); if (mpolicy == NULL) { /* n is already freed */ return key_senderror(so, m, ENOBUFS); } xpl = (struct sadb_x_policy *)(mtod(mpolicy, caddr_t) + off); if (xpl->sadb_x_policy_exttype != SADB_X_EXT_POLICY) { m_freem(n); return key_senderror(so, m, EINVAL); } xpl->sadb_x_policy_id = newsp->id; m_freem(m); return key_sendup_mbuf(so, n, KEY_SENDUP_ALL); } } /* * get new policy id. * OUT: * 0: failure. * others: success. */ static u_int32_t key_getnewspid() { u_int32_t newid = 0; int count = V_key_spi_trycnt; /* XXX */ struct secpolicy *sp; /* when requesting to allocate spi ranged */ while (count--) { newid = (V_policy_id = (V_policy_id == ~0 ? 1 : V_policy_id + 1)); if ((sp = key_getspbyid(newid)) == NULL) break; KEY_FREESP(&sp); } if (count == 0 || newid == 0) { ipseclog((LOG_DEBUG, "%s: to allocate policy id is failed.\n", __func__)); return 0; } return newid; } /* * SADB_SPDDELETE processing * receive * * from the user(?), and set SADB_SASTATE_DEAD, * and send, * * to the ikmpd. * policy(*) including direction of policy. * * m will always be freed. */ static int key_spddelete(so, m, mhp) struct socket *so; struct mbuf *m; const struct sadb_msghdr *mhp; { struct sadb_address *src0, *dst0; struct sadb_x_policy *xpl0; struct secpolicyindex spidx; struct secpolicy *sp; IPSEC_ASSERT(so != NULL, ("null so")); IPSEC_ASSERT(m != NULL, ("null mbuf")); IPSEC_ASSERT(mhp != NULL, ("null msghdr")); IPSEC_ASSERT(mhp->msg != NULL, ("null msg")); if (mhp->ext[SADB_EXT_ADDRESS_SRC] == NULL || mhp->ext[SADB_EXT_ADDRESS_DST] == NULL || mhp->ext[SADB_X_EXT_POLICY] == NULL) { ipseclog((LOG_DEBUG, "%s: invalid message is passed.\n", __func__)); return key_senderror(so, m, EINVAL); } if (mhp->extlen[SADB_EXT_ADDRESS_SRC] < sizeof(struct sadb_address) || mhp->extlen[SADB_EXT_ADDRESS_DST] < sizeof(struct sadb_address) || mhp->extlen[SADB_X_EXT_POLICY] < sizeof(struct sadb_x_policy)) { ipseclog((LOG_DEBUG, "%s: invalid message is passed.\n", __func__)); return key_senderror(so, m, EINVAL); } src0 = (struct sadb_address *)mhp->ext[SADB_EXT_ADDRESS_SRC]; dst0 = (struct sadb_address *)mhp->ext[SADB_EXT_ADDRESS_DST]; xpl0 = (struct sadb_x_policy *)mhp->ext[SADB_X_EXT_POLICY]; /* * Note: do not parse SADB_X_EXT_NAT_T_* here: * we are processing traffic endpoints. */ /* make secindex */ /* XXX boundary check against sa_len */ KEY_SETSECSPIDX(xpl0->sadb_x_policy_dir, src0 + 1, dst0 + 1, src0->sadb_address_prefixlen, dst0->sadb_address_prefixlen, src0->sadb_address_proto, &spidx); /* checking the direciton. */ switch (xpl0->sadb_x_policy_dir) { case IPSEC_DIR_INBOUND: case IPSEC_DIR_OUTBOUND: break; default: ipseclog((LOG_DEBUG, "%s: Invalid SP direction.\n", __func__)); return key_senderror(so, m, EINVAL); } /* Is there SP in SPD ? */ if ((sp = key_getsp(&spidx)) == NULL) { ipseclog((LOG_DEBUG, "%s: no SP found.\n", __func__)); return key_senderror(so, m, EINVAL); } /* save policy id to buffer to be returned. */ xpl0->sadb_x_policy_id = sp->id; + SPTREE_LOCK(); sp->state = IPSEC_SPSTATE_DEAD; + SPTREE_UNLOCK(); KEY_FREESP(&sp); { struct mbuf *n; struct sadb_msg *newmsg; /* * Note: do not send SADB_X_EXT_NAT_T_* here: * we are sending traffic endpoints. */ /* create new sadb_msg to reply. */ n = key_gather_mbuf(m, mhp, 1, 4, SADB_EXT_RESERVED, SADB_X_EXT_POLICY, SADB_EXT_ADDRESS_SRC, SADB_EXT_ADDRESS_DST); if (!n) return key_senderror(so, m, ENOBUFS); newmsg = mtod(n, struct sadb_msg *); newmsg->sadb_msg_errno = 0; newmsg->sadb_msg_len = PFKEY_UNIT64(n->m_pkthdr.len); m_freem(m); return key_sendup_mbuf(so, n, KEY_SENDUP_ALL); } } /* * SADB_SPDDELETE2 processing * receive * * from the user(?), and set SADB_SASTATE_DEAD, * and send, * * to the ikmpd. * policy(*) including direction of policy. * * m will always be freed. */ static int key_spddelete2(so, m, mhp) struct socket *so; struct mbuf *m; const struct sadb_msghdr *mhp; { u_int32_t id; struct secpolicy *sp; IPSEC_ASSERT(so != NULL, ("null socket")); IPSEC_ASSERT(m != NULL, ("null mbuf")); IPSEC_ASSERT(mhp != NULL, ("null msghdr")); IPSEC_ASSERT(mhp->msg != NULL, ("null msg")); if (mhp->ext[SADB_X_EXT_POLICY] == NULL || mhp->extlen[SADB_X_EXT_POLICY] < sizeof(struct sadb_x_policy)) { ipseclog((LOG_DEBUG, "%s: invalid message is passed.\n", __func__)); return key_senderror(so, m, EINVAL); } id = ((struct sadb_x_policy *)mhp->ext[SADB_X_EXT_POLICY])->sadb_x_policy_id; /* Is there SP in SPD ? */ if ((sp = key_getspbyid(id)) == NULL) { ipseclog((LOG_DEBUG, "%s: no SP found id:%u.\n", __func__, id)); return key_senderror(so, m, EINVAL); } + SPTREE_LOCK(); sp->state = IPSEC_SPSTATE_DEAD; + SPTREE_UNLOCK(); KEY_FREESP(&sp); { struct mbuf *n, *nn; struct sadb_msg *newmsg; int off, len; /* create new sadb_msg to reply. */ len = PFKEY_ALIGN8(sizeof(struct sadb_msg)); MGETHDR(n, M_DONTWAIT, MT_DATA); if (n && len > MHLEN) { MCLGET(n, M_DONTWAIT); if ((n->m_flags & M_EXT) == 0) { m_freem(n); n = NULL; } } if (!n) return key_senderror(so, m, ENOBUFS); n->m_len = len; n->m_next = NULL; off = 0; m_copydata(m, 0, sizeof(struct sadb_msg), mtod(n, caddr_t) + off); off += PFKEY_ALIGN8(sizeof(struct sadb_msg)); IPSEC_ASSERT(off == len, ("length inconsistency (off %u len %u)", off, len)); n->m_next = m_copym(m, mhp->extoff[SADB_X_EXT_POLICY], mhp->extlen[SADB_X_EXT_POLICY], M_DONTWAIT); if (!n->m_next) { m_freem(n); return key_senderror(so, m, ENOBUFS); } n->m_pkthdr.len = 0; for (nn = n; nn; nn = nn->m_next) n->m_pkthdr.len += nn->m_len; newmsg = mtod(n, struct sadb_msg *); newmsg->sadb_msg_errno = 0; newmsg->sadb_msg_len = PFKEY_UNIT64(n->m_pkthdr.len); m_freem(m); return key_sendup_mbuf(so, n, KEY_SENDUP_ALL); } } /* * SADB_X_GET processing * receive * * from the user(?), * and send, * * to the ikmpd. * policy(*) including direction of policy. * * m will always be freed. */ static int key_spdget(so, m, mhp) struct socket *so; struct mbuf *m; const struct sadb_msghdr *mhp; { u_int32_t id; struct secpolicy *sp; struct mbuf *n; IPSEC_ASSERT(so != NULL, ("null socket")); IPSEC_ASSERT(m != NULL, ("null mbuf")); IPSEC_ASSERT(mhp != NULL, ("null msghdr")); IPSEC_ASSERT(mhp->msg != NULL, ("null msg")); if (mhp->ext[SADB_X_EXT_POLICY] == NULL || mhp->extlen[SADB_X_EXT_POLICY] < sizeof(struct sadb_x_policy)) { ipseclog((LOG_DEBUG, "%s: invalid message is passed.\n", __func__)); return key_senderror(so, m, EINVAL); } id = ((struct sadb_x_policy *)mhp->ext[SADB_X_EXT_POLICY])->sadb_x_policy_id; /* Is there SP in SPD ? */ if ((sp = key_getspbyid(id)) == NULL) { ipseclog((LOG_DEBUG, "%s: no SP found id:%u.\n", __func__, id)); return key_senderror(so, m, ENOENT); } n = key_setdumpsp(sp, SADB_X_SPDGET, 0, mhp->msg->sadb_msg_pid); if (n != NULL) { m_freem(m); return key_sendup_mbuf(so, n, KEY_SENDUP_ONE); } else return key_senderror(so, m, ENOBUFS); } /* * SADB_X_SPDACQUIRE processing. * Acquire policy and SA(s) for a *OUTBOUND* packet. * send * * to KMD, and expect to receive * with SADB_X_SPDACQUIRE if error occured, * or * * with SADB_X_SPDUPDATE from KMD by PF_KEY. * policy(*) is without policy requests. * * 0 : succeed * others: error number */ int key_spdacquire(sp) struct secpolicy *sp; { struct mbuf *result = NULL, *m; struct secspacq *newspacq; IPSEC_ASSERT(sp != NULL, ("null secpolicy")); IPSEC_ASSERT(sp->req == NULL, ("policy exists")); IPSEC_ASSERT(sp->policy == IPSEC_POLICY_IPSEC, ("policy not IPSEC %u", sp->policy)); /* Get an entry to check whether sent message or not. */ newspacq = key_getspacq(&sp->spidx); if (newspacq != NULL) { if (V_key_blockacq_count < newspacq->count) { /* reset counter and do send message. */ newspacq->count = 0; } else { /* increment counter and do nothing. */ newspacq->count++; return 0; } SPACQ_UNLOCK(); } else { /* make new entry for blocking to send SADB_ACQUIRE. */ newspacq = key_newspacq(&sp->spidx); if (newspacq == NULL) return ENOBUFS; } /* create new sadb_msg to reply. */ m = key_setsadbmsg(SADB_X_SPDACQUIRE, 0, 0, 0, 0, 0); if (!m) return ENOBUFS; result = m; result->m_pkthdr.len = 0; for (m = result; m; m = m->m_next) result->m_pkthdr.len += m->m_len; mtod(result, struct sadb_msg *)->sadb_msg_len = PFKEY_UNIT64(result->m_pkthdr.len); return key_sendup_mbuf(NULL, m, KEY_SENDUP_REGISTERED); } /* * SADB_SPDFLUSH processing * receive * * from the user, and free all entries in secpctree. * and send, * * to the user. * NOTE: what to do is only marking SADB_SASTATE_DEAD. * * m will always be freed. */ static int key_spdflush(so, m, mhp) struct socket *so; struct mbuf *m; const struct sadb_msghdr *mhp; { struct sadb_msg *newmsg; struct secpolicy *sp; u_int dir; IPSEC_ASSERT(so != NULL, ("null socket")); IPSEC_ASSERT(m != NULL, ("null mbuf")); IPSEC_ASSERT(mhp != NULL, ("null msghdr")); IPSEC_ASSERT(mhp->msg != NULL, ("null msg")); if (m->m_len != PFKEY_ALIGN8(sizeof(struct sadb_msg))) return key_senderror(so, m, EINVAL); for (dir = 0; dir < IPSEC_DIR_MAX; dir++) { SPTREE_LOCK(); LIST_FOREACH(sp, &V_sptree[dir], chain) sp->state = IPSEC_SPSTATE_DEAD; SPTREE_UNLOCK(); } if (sizeof(struct sadb_msg) > m->m_len + M_TRAILINGSPACE(m)) { ipseclog((LOG_DEBUG, "%s: No more memory.\n", __func__)); return key_senderror(so, m, ENOBUFS); } if (m->m_next) m_freem(m->m_next); m->m_next = NULL; m->m_pkthdr.len = m->m_len = PFKEY_ALIGN8(sizeof(struct sadb_msg)); newmsg = mtod(m, struct sadb_msg *); newmsg->sadb_msg_errno = 0; newmsg->sadb_msg_len = PFKEY_UNIT64(m->m_pkthdr.len); return key_sendup_mbuf(so, m, KEY_SENDUP_ALL); } /* * SADB_SPDDUMP processing * receive * * from the user, and dump all SP leaves * and send, * ..... * to the ikmpd. * * m will always be freed. */ static int key_spddump(so, m, mhp) struct socket *so; struct mbuf *m; const struct sadb_msghdr *mhp; { struct secpolicy *sp; int cnt; u_int dir; struct mbuf *n; IPSEC_ASSERT(so != NULL, ("null socket")); IPSEC_ASSERT(m != NULL, ("null mbuf")); IPSEC_ASSERT(mhp != NULL, ("null msghdr")); IPSEC_ASSERT(mhp->msg != NULL, ("null msg")); /* search SPD entry and get buffer size. */ cnt = 0; SPTREE_LOCK(); for (dir = 0; dir < IPSEC_DIR_MAX; dir++) { LIST_FOREACH(sp, &V_sptree[dir], chain) { cnt++; } } if (cnt == 0) { SPTREE_UNLOCK(); return key_senderror(so, m, ENOENT); } for (dir = 0; dir < IPSEC_DIR_MAX; dir++) { LIST_FOREACH(sp, &V_sptree[dir], chain) { --cnt; n = key_setdumpsp(sp, SADB_X_SPDDUMP, cnt, mhp->msg->sadb_msg_pid); if (n) key_sendup_mbuf(so, n, KEY_SENDUP_ONE); } } SPTREE_UNLOCK(); m_freem(m); return 0; } static struct mbuf * key_setdumpsp(struct secpolicy *sp, u_int8_t type, u_int32_t seq, u_int32_t pid) { struct mbuf *result = NULL, *m; struct seclifetime lt; m = key_setsadbmsg(type, 0, SADB_SATYPE_UNSPEC, seq, pid, sp->refcnt); if (!m) goto fail; result = m; /* * Note: do not send SADB_X_EXT_NAT_T_* here: * we are sending traffic endpoints. */ m = key_setsadbaddr(SADB_EXT_ADDRESS_SRC, &sp->spidx.src.sa, sp->spidx.prefs, sp->spidx.ul_proto); if (!m) goto fail; m_cat(result, m); m = key_setsadbaddr(SADB_EXT_ADDRESS_DST, &sp->spidx.dst.sa, sp->spidx.prefd, sp->spidx.ul_proto); if (!m) goto fail; m_cat(result, m); m = key_sp2msg(sp); if (!m) goto fail; m_cat(result, m); if(sp->lifetime){ lt.addtime=sp->created; lt.usetime= sp->lastused; m = key_setlifetime(<, SADB_EXT_LIFETIME_CURRENT); if (!m) goto fail; m_cat(result, m); lt.addtime=sp->lifetime; lt.usetime= sp->validtime; m = key_setlifetime(<, SADB_EXT_LIFETIME_HARD); if (!m) goto fail; m_cat(result, m); } if ((result->m_flags & M_PKTHDR) == 0) goto fail; if (result->m_len < sizeof(struct sadb_msg)) { result = m_pullup(result, sizeof(struct sadb_msg)); if (result == NULL) goto fail; } result->m_pkthdr.len = 0; for (m = result; m; m = m->m_next) result->m_pkthdr.len += m->m_len; mtod(result, struct sadb_msg *)->sadb_msg_len = PFKEY_UNIT64(result->m_pkthdr.len); return result; fail: m_freem(result); return NULL; } /* * get PFKEY message length for security policy and request. */ static u_int key_getspreqmsglen(sp) struct secpolicy *sp; { u_int tlen; tlen = sizeof(struct sadb_x_policy); /* if is the policy for ipsec ? */ if (sp->policy != IPSEC_POLICY_IPSEC) return tlen; /* get length of ipsec requests */ { struct ipsecrequest *isr; int len; for (isr = sp->req; isr != NULL; isr = isr->next) { len = sizeof(struct sadb_x_ipsecrequest) + isr->saidx.src.sa.sa_len + isr->saidx.dst.sa.sa_len; tlen += PFKEY_ALIGN8(len); } } return tlen; } /* * SADB_SPDEXPIRE processing * send * * to KMD by PF_KEY. * * OUT: 0 : succeed * others : error number */ static int key_spdexpire(sp) struct secpolicy *sp; { struct mbuf *result = NULL, *m; int len; int error = -1; struct sadb_lifetime *lt; /* XXX: Why do we lock ? */ IPSEC_ASSERT(sp != NULL, ("null secpolicy")); /* set msg header */ m = key_setsadbmsg(SADB_X_SPDEXPIRE, 0, 0, 0, 0, 0); if (!m) { error = ENOBUFS; goto fail; } result = m; /* create lifetime extension (current and hard) */ len = PFKEY_ALIGN8(sizeof(*lt)) * 2; m = key_alloc_mbuf(len); if (!m || m->m_next) { /*XXX*/ if (m) m_freem(m); error = ENOBUFS; goto fail; } bzero(mtod(m, caddr_t), len); lt = mtod(m, struct sadb_lifetime *); lt->sadb_lifetime_len = PFKEY_UNIT64(sizeof(struct sadb_lifetime)); lt->sadb_lifetime_exttype = SADB_EXT_LIFETIME_CURRENT; lt->sadb_lifetime_allocations = 0; lt->sadb_lifetime_bytes = 0; lt->sadb_lifetime_addtime = sp->created; lt->sadb_lifetime_usetime = sp->lastused; lt = (struct sadb_lifetime *)(mtod(m, caddr_t) + len / 2); lt->sadb_lifetime_len = PFKEY_UNIT64(sizeof(struct sadb_lifetime)); lt->sadb_lifetime_exttype = SADB_EXT_LIFETIME_HARD; lt->sadb_lifetime_allocations = 0; lt->sadb_lifetime_bytes = 0; lt->sadb_lifetime_addtime = sp->lifetime; lt->sadb_lifetime_usetime = sp->validtime; m_cat(result, m); /* * Note: do not send SADB_X_EXT_NAT_T_* here: * we are sending traffic endpoints. */ /* set sadb_address for source */ m = key_setsadbaddr(SADB_EXT_ADDRESS_SRC, &sp->spidx.src.sa, sp->spidx.prefs, sp->spidx.ul_proto); if (!m) { error = ENOBUFS; goto fail; } m_cat(result, m); /* set sadb_address for destination */ m = key_setsadbaddr(SADB_EXT_ADDRESS_DST, &sp->spidx.dst.sa, sp->spidx.prefd, sp->spidx.ul_proto); if (!m) { error = ENOBUFS; goto fail; } m_cat(result, m); /* set secpolicy */ m = key_sp2msg(sp); if (!m) { error = ENOBUFS; goto fail; } m_cat(result, m); if ((result->m_flags & M_PKTHDR) == 0) { error = EINVAL; goto fail; } if (result->m_len < sizeof(struct sadb_msg)) { result = m_pullup(result, sizeof(struct sadb_msg)); if (result == NULL) { error = ENOBUFS; goto fail; } } result->m_pkthdr.len = 0; for (m = result; m; m = m->m_next) result->m_pkthdr.len += m->m_len; mtod(result, struct sadb_msg *)->sadb_msg_len = PFKEY_UNIT64(result->m_pkthdr.len); return key_sendup_mbuf(NULL, result, KEY_SENDUP_REGISTERED); fail: if (result) m_freem(result); return error; } /* %%% SAD management */ /* * allocating a memory for new SA head, and copy from the values of mhp. * OUT: NULL : failure due to the lack of memory. * others : pointer to new SA head. */ static struct secashead * key_newsah(saidx) struct secasindex *saidx; { struct secashead *newsah; IPSEC_ASSERT(saidx != NULL, ("null saidx")); newsah = malloc(sizeof(struct secashead), M_IPSEC_SAH, M_NOWAIT|M_ZERO); if (newsah != NULL) { int i; for (i = 0; i < sizeof(newsah->savtree)/sizeof(newsah->savtree[0]); i++) LIST_INIT(&newsah->savtree[i]); newsah->saidx = *saidx; /* add to saidxtree */ newsah->state = SADB_SASTATE_MATURE; SAHTREE_LOCK(); LIST_INSERT_HEAD(&V_sahtree, newsah, chain); SAHTREE_UNLOCK(); } return(newsah); } /* * delete SA index and all SA registerd. */ static void key_delsah(sah) struct secashead *sah; { struct secasvar *sav, *nextsav; u_int stateidx; int zombie = 0; IPSEC_ASSERT(sah != NULL, ("NULL sah")); SAHTREE_LOCK_ASSERT(); /* searching all SA registerd in the secindex. */ for (stateidx = 0; stateidx < _ARRAYLEN(saorder_state_any); stateidx++) { u_int state = saorder_state_any[stateidx]; LIST_FOREACH_SAFE(sav, &sah->savtree[state], chain, nextsav) { if (sav->refcnt == 0) { /* sanity check */ KEY_CHKSASTATE(state, sav->state, __func__); /* * do NOT call KEY_FREESAV here: * it will only delete the sav if refcnt == 1, * where we already know that refcnt == 0 */ key_delsav(sav); } else { /* give up to delete this sa */ zombie++; } } } if (!zombie) { /* delete only if there are savs */ /* remove from tree of SA index */ if (__LIST_CHAINED(sah)) LIST_REMOVE(sah, chain); if (sah->sa_route.ro_rt) { RTFREE(sah->sa_route.ro_rt); sah->sa_route.ro_rt = (struct rtentry *)NULL; } free(sah, M_IPSEC_SAH); } } /* * allocating a new SA with LARVAL state. key_add() and key_getspi() call, * and copy the values of mhp into new buffer. * When SAD message type is GETSPI: * to set sequence number from acq_seq++, * to set zero to SPI. * not to call key_setsava(). * OUT: NULL : fail * others : pointer to new secasvar. * * does not modify mbuf. does not free mbuf on error. */ static struct secasvar * key_newsav(m, mhp, sah, errp, where, tag) struct mbuf *m; const struct sadb_msghdr *mhp; struct secashead *sah; int *errp; const char* where; int tag; { struct secasvar *newsav; const struct sadb_sa *xsa; IPSEC_ASSERT(m != NULL, ("null mbuf")); IPSEC_ASSERT(mhp != NULL, ("null msghdr")); IPSEC_ASSERT(mhp->msg != NULL, ("null msg")); IPSEC_ASSERT(sah != NULL, ("null secashead")); newsav = malloc(sizeof(struct secasvar), M_IPSEC_SA, M_NOWAIT|M_ZERO); if (newsav == NULL) { ipseclog((LOG_DEBUG, "%s: No more memory.\n", __func__)); *errp = ENOBUFS; goto done; } switch (mhp->msg->sadb_msg_type) { case SADB_GETSPI: newsav->spi = 0; #ifdef IPSEC_DOSEQCHECK /* sync sequence number */ if (mhp->msg->sadb_msg_seq == 0) newsav->seq = (V_acq_seq = (V_acq_seq == ~0 ? 1 : ++V_acq_seq)); else #endif newsav->seq = mhp->msg->sadb_msg_seq; break; case SADB_ADD: /* sanity check */ if (mhp->ext[SADB_EXT_SA] == NULL) { free(newsav, M_IPSEC_SA); newsav = NULL; ipseclog((LOG_DEBUG, "%s: invalid message is passed.\n", __func__)); *errp = EINVAL; goto done; } xsa = (const struct sadb_sa *)mhp->ext[SADB_EXT_SA]; newsav->spi = xsa->sadb_sa_spi; newsav->seq = mhp->msg->sadb_msg_seq; break; default: free(newsav, M_IPSEC_SA); newsav = NULL; *errp = EINVAL; goto done; } /* copy sav values */ if (mhp->msg->sadb_msg_type != SADB_GETSPI) { *errp = key_setsaval(newsav, m, mhp); if (*errp) { free(newsav, M_IPSEC_SA); newsav = NULL; goto done; } } SECASVAR_LOCK_INIT(newsav); /* reset created */ newsav->created = time_second; newsav->pid = mhp->msg->sadb_msg_pid; /* add to satree */ newsav->sah = sah; sa_initref(newsav); newsav->state = SADB_SASTATE_LARVAL; SAHTREE_LOCK(); LIST_INSERT_TAIL(&sah->savtree[SADB_SASTATE_LARVAL], newsav, secasvar, chain); SAHTREE_UNLOCK(); done: KEYDEBUG(KEYDEBUG_IPSEC_STAMP, printf("DP %s from %s:%u return SP:%p\n", __func__, where, tag, newsav)); return newsav; } /* * free() SA variable entry. */ static void key_cleansav(struct secasvar *sav) { /* * Cleanup xform state. Note that zeroize'ing causes the * keys to be cleared; otherwise we must do it ourself. */ if (sav->tdb_xform != NULL) { sav->tdb_xform->xf_zeroize(sav); sav->tdb_xform = NULL; } else { KASSERT(sav->iv == NULL, ("iv but no xform")); if (sav->key_auth != NULL) bzero(sav->key_auth->key_data, _KEYLEN(sav->key_auth)); if (sav->key_enc != NULL) bzero(sav->key_enc->key_data, _KEYLEN(sav->key_enc)); } if (sav->key_auth != NULL) { if (sav->key_auth->key_data != NULL) free(sav->key_auth->key_data, M_IPSEC_MISC); free(sav->key_auth, M_IPSEC_MISC); sav->key_auth = NULL; } if (sav->key_enc != NULL) { if (sav->key_enc->key_data != NULL) free(sav->key_enc->key_data, M_IPSEC_MISC); free(sav->key_enc, M_IPSEC_MISC); sav->key_enc = NULL; } if (sav->sched) { bzero(sav->sched, sav->schedlen); free(sav->sched, M_IPSEC_MISC); sav->sched = NULL; } if (sav->replay != NULL) { free(sav->replay, M_IPSEC_MISC); sav->replay = NULL; } if (sav->lft_c != NULL) { free(sav->lft_c, M_IPSEC_MISC); sav->lft_c = NULL; } if (sav->lft_h != NULL) { free(sav->lft_h, M_IPSEC_MISC); sav->lft_h = NULL; } if (sav->lft_s != NULL) { free(sav->lft_s, M_IPSEC_MISC); sav->lft_s = NULL; } } /* * free() SA variable entry. */ static void key_delsav(sav) struct secasvar *sav; { IPSEC_ASSERT(sav != NULL, ("null sav")); IPSEC_ASSERT(sav->refcnt == 0, ("reference count %u > 0", sav->refcnt)); /* remove from SA header */ if (__LIST_CHAINED(sav)) LIST_REMOVE(sav, chain); key_cleansav(sav); SECASVAR_LOCK_DESTROY(sav); free(sav, M_IPSEC_SA); } /* * search SAD. * OUT: * NULL : not found * others : found, pointer to a SA. */ static struct secashead * key_getsah(saidx) struct secasindex *saidx; { struct secashead *sah; SAHTREE_LOCK(); LIST_FOREACH(sah, &V_sahtree, chain) { if (sah->state == SADB_SASTATE_DEAD) continue; if (key_cmpsaidx(&sah->saidx, saidx, CMP_REQID)) break; } SAHTREE_UNLOCK(); return sah; } /* * check not to be duplicated SPI. * NOTE: this function is too slow due to searching all SAD. * OUT: * NULL : not found * others : found, pointer to a SA. */ static struct secasvar * key_checkspidup(saidx, spi) struct secasindex *saidx; u_int32_t spi; { struct secashead *sah; struct secasvar *sav; /* check address family */ if (saidx->src.sa.sa_family != saidx->dst.sa.sa_family) { ipseclog((LOG_DEBUG, "%s: address family mismatched.\n", __func__)); return NULL; } sav = NULL; /* check all SAD */ SAHTREE_LOCK(); LIST_FOREACH(sah, &V_sahtree, chain) { if (!key_ismyaddr((struct sockaddr *)&sah->saidx.dst)) continue; sav = key_getsavbyspi(sah, spi); if (sav != NULL) break; } SAHTREE_UNLOCK(); return sav; } /* * search SAD litmited alive SA, protocol, SPI. * OUT: * NULL : not found * others : found, pointer to a SA. */ static struct secasvar * key_getsavbyspi(sah, spi) struct secashead *sah; u_int32_t spi; { struct secasvar *sav; u_int stateidx, state; sav = NULL; SAHTREE_LOCK_ASSERT(); /* search all status */ for (stateidx = 0; stateidx < _ARRAYLEN(saorder_state_alive); stateidx++) { state = saorder_state_alive[stateidx]; LIST_FOREACH(sav, &sah->savtree[state], chain) { /* sanity check */ if (sav->state != state) { ipseclog((LOG_DEBUG, "%s: " "invalid sav->state (queue: %d SA: %d)\n", __func__, state, sav->state)); continue; } if (sav->spi == spi) return sav; } } return NULL; } /* * copy SA values from PF_KEY message except *SPI, SEQ, PID, STATE and TYPE*. * You must update these if need. * OUT: 0: success. * !0: failure. * * does not modify mbuf. does not free mbuf on error. */ static int key_setsaval(sav, m, mhp) struct secasvar *sav; struct mbuf *m; const struct sadb_msghdr *mhp; { int error = 0; IPSEC_ASSERT(m != NULL, ("null mbuf")); IPSEC_ASSERT(mhp != NULL, ("null msghdr")); IPSEC_ASSERT(mhp->msg != NULL, ("null msg")); /* initialization */ sav->replay = NULL; sav->key_auth = NULL; sav->key_enc = NULL; sav->sched = NULL; sav->schedlen = 0; sav->iv = NULL; sav->lft_c = NULL; sav->lft_h = NULL; sav->lft_s = NULL; sav->tdb_xform = NULL; /* transform */ sav->tdb_encalgxform = NULL; /* encoding algorithm */ sav->tdb_authalgxform = NULL; /* authentication algorithm */ sav->tdb_compalgxform = NULL; /* compression algorithm */ /* Initialize even if NAT-T not compiled in: */ sav->natt_type = 0; sav->natt_esp_frag_len = 0; /* SA */ if (mhp->ext[SADB_EXT_SA] != NULL) { const struct sadb_sa *sa0; sa0 = (const struct sadb_sa *)mhp->ext[SADB_EXT_SA]; if (mhp->extlen[SADB_EXT_SA] < sizeof(*sa0)) { error = EINVAL; goto fail; } sav->alg_auth = sa0->sadb_sa_auth; sav->alg_enc = sa0->sadb_sa_encrypt; sav->flags = sa0->sadb_sa_flags; /* replay window */ if ((sa0->sadb_sa_flags & SADB_X_EXT_OLD) == 0) { sav->replay = (struct secreplay *) malloc(sizeof(struct secreplay)+sa0->sadb_sa_replay, M_IPSEC_MISC, M_NOWAIT|M_ZERO); if (sav->replay == NULL) { ipseclog((LOG_DEBUG, "%s: No more memory.\n", __func__)); error = ENOBUFS; goto fail; } if (sa0->sadb_sa_replay != 0) sav->replay->bitmap = (caddr_t)(sav->replay+1); sav->replay->wsize = sa0->sadb_sa_replay; } } /* Authentication keys */ if (mhp->ext[SADB_EXT_KEY_AUTH] != NULL) { const struct sadb_key *key0; int len; key0 = (const struct sadb_key *)mhp->ext[SADB_EXT_KEY_AUTH]; len = mhp->extlen[SADB_EXT_KEY_AUTH]; error = 0; if (len < sizeof(*key0)) { error = EINVAL; goto fail; } switch (mhp->msg->sadb_msg_satype) { case SADB_SATYPE_AH: case SADB_SATYPE_ESP: case SADB_X_SATYPE_TCPSIGNATURE: if (len == PFKEY_ALIGN8(sizeof(struct sadb_key)) && sav->alg_auth != SADB_X_AALG_NULL) error = EINVAL; break; case SADB_X_SATYPE_IPCOMP: default: error = EINVAL; break; } if (error) { ipseclog((LOG_DEBUG, "%s: invalid key_auth values.\n", __func__)); goto fail; } sav->key_auth = (struct seckey *)key_dup_keymsg(key0, len, M_IPSEC_MISC); if (sav->key_auth == NULL ) { ipseclog((LOG_DEBUG, "%s: No more memory.\n", __func__)); error = ENOBUFS; goto fail; } } /* Encryption key */ if (mhp->ext[SADB_EXT_KEY_ENCRYPT] != NULL) { const struct sadb_key *key0; int len; key0 = (const struct sadb_key *)mhp->ext[SADB_EXT_KEY_ENCRYPT]; len = mhp->extlen[SADB_EXT_KEY_ENCRYPT]; error = 0; if (len < sizeof(*key0)) { error = EINVAL; goto fail; } switch (mhp->msg->sadb_msg_satype) { case SADB_SATYPE_ESP: if (len == PFKEY_ALIGN8(sizeof(struct sadb_key)) && sav->alg_enc != SADB_EALG_NULL) { error = EINVAL; break; } sav->key_enc = (struct seckey *)key_dup_keymsg(key0, len, M_IPSEC_MISC); if (sav->key_enc == NULL) { ipseclog((LOG_DEBUG, "%s: No more memory.\n", __func__)); error = ENOBUFS; goto fail; } break; case SADB_X_SATYPE_IPCOMP: if (len != PFKEY_ALIGN8(sizeof(struct sadb_key))) error = EINVAL; sav->key_enc = NULL; /*just in case*/ break; case SADB_SATYPE_AH: case SADB_X_SATYPE_TCPSIGNATURE: default: error = EINVAL; break; } if (error) { ipseclog((LOG_DEBUG, "%s: invalid key_enc value.\n", __func__)); goto fail; } } /* set iv */ sav->ivlen = 0; switch (mhp->msg->sadb_msg_satype) { case SADB_SATYPE_AH: error = xform_init(sav, XF_AH); break; case SADB_SATYPE_ESP: error = xform_init(sav, XF_ESP); break; case SADB_X_SATYPE_IPCOMP: error = xform_init(sav, XF_IPCOMP); break; case SADB_X_SATYPE_TCPSIGNATURE: error = xform_init(sav, XF_TCPSIGNATURE); break; } if (error) { ipseclog((LOG_DEBUG, "%s: unable to initialize SA type %u.\n", __func__, mhp->msg->sadb_msg_satype)); goto fail; } /* reset created */ sav->created = time_second; /* make lifetime for CURRENT */ sav->lft_c = malloc(sizeof(struct seclifetime), M_IPSEC_MISC, M_NOWAIT); if (sav->lft_c == NULL) { ipseclog((LOG_DEBUG, "%s: No more memory.\n", __func__)); error = ENOBUFS; goto fail; } sav->lft_c->allocations = 0; sav->lft_c->bytes = 0; sav->lft_c->addtime = time_second; sav->lft_c->usetime = 0; /* lifetimes for HARD and SOFT */ { const struct sadb_lifetime *lft0; lft0 = (struct sadb_lifetime *)mhp->ext[SADB_EXT_LIFETIME_HARD]; if (lft0 != NULL) { if (mhp->extlen[SADB_EXT_LIFETIME_HARD] < sizeof(*lft0)) { error = EINVAL; goto fail; } sav->lft_h = key_dup_lifemsg(lft0, M_IPSEC_MISC); if (sav->lft_h == NULL) { ipseclog((LOG_DEBUG, "%s: No more memory.\n",__func__)); error = ENOBUFS; goto fail; } /* to be initialize ? */ } lft0 = (struct sadb_lifetime *)mhp->ext[SADB_EXT_LIFETIME_SOFT]; if (lft0 != NULL) { if (mhp->extlen[SADB_EXT_LIFETIME_SOFT] < sizeof(*lft0)) { error = EINVAL; goto fail; } sav->lft_s = key_dup_lifemsg(lft0, M_IPSEC_MISC); if (sav->lft_s == NULL) { ipseclog((LOG_DEBUG, "%s: No more memory.\n",__func__)); error = ENOBUFS; goto fail; } /* to be initialize ? */ } } return 0; fail: /* initialization */ key_cleansav(sav); return error; } /* * validation with a secasvar entry, and set SADB_SATYPE_MATURE. * OUT: 0: valid * other: errno */ static int key_mature(struct secasvar *sav) { int error; /* check SPI value */ switch (sav->sah->saidx.proto) { case IPPROTO_ESP: case IPPROTO_AH: /* * RFC 4302, 2.4. Security Parameters Index (SPI), SPI values * 1-255 reserved by IANA for future use, * 0 for implementation specific, local use. */ if (ntohl(sav->spi) <= 255) { ipseclog((LOG_DEBUG, "%s: illegal range of SPI %u.\n", __func__, (u_int32_t)ntohl(sav->spi))); return EINVAL; } break; } /* check satype */ switch (sav->sah->saidx.proto) { case IPPROTO_ESP: /* check flags */ if ((sav->flags & (SADB_X_EXT_OLD|SADB_X_EXT_DERIV)) == (SADB_X_EXT_OLD|SADB_X_EXT_DERIV)) { ipseclog((LOG_DEBUG, "%s: invalid flag (derived) " "given to old-esp.\n", __func__)); return EINVAL; } error = xform_init(sav, XF_ESP); break; case IPPROTO_AH: /* check flags */ if (sav->flags & SADB_X_EXT_DERIV) { ipseclog((LOG_DEBUG, "%s: invalid flag (derived) " "given to AH SA.\n", __func__)); return EINVAL; } if (sav->alg_enc != SADB_EALG_NONE) { ipseclog((LOG_DEBUG, "%s: protocol and algorithm " "mismated.\n", __func__)); return(EINVAL); } error = xform_init(sav, XF_AH); break; case IPPROTO_IPCOMP: if (sav->alg_auth != SADB_AALG_NONE) { ipseclog((LOG_DEBUG, "%s: protocol and algorithm " "mismated.\n", __func__)); return(EINVAL); } if ((sav->flags & SADB_X_EXT_RAWCPI) == 0 && ntohl(sav->spi) >= 0x10000) { ipseclog((LOG_DEBUG, "%s: invalid cpi for IPComp.\n", __func__)); return(EINVAL); } error = xform_init(sav, XF_IPCOMP); break; case IPPROTO_TCP: if (sav->alg_enc != SADB_EALG_NONE) { ipseclog((LOG_DEBUG, "%s: protocol and algorithm " "mismated.\n", __func__)); return(EINVAL); } error = xform_init(sav, XF_TCPSIGNATURE); break; default: ipseclog((LOG_DEBUG, "%s: Invalid satype.\n", __func__)); error = EPROTONOSUPPORT; break; } if (error == 0) { SAHTREE_LOCK(); key_sa_chgstate(sav, SADB_SASTATE_MATURE); SAHTREE_UNLOCK(); } return (error); } /* * subroutine for SADB_GET and SADB_DUMP. */ static struct mbuf * key_setdumpsa(struct secasvar *sav, u_int8_t type, u_int8_t satype, u_int32_t seq, u_int32_t pid) { struct mbuf *result = NULL, *tres = NULL, *m; int i; int dumporder[] = { SADB_EXT_SA, SADB_X_EXT_SA2, SADB_EXT_LIFETIME_HARD, SADB_EXT_LIFETIME_SOFT, SADB_EXT_LIFETIME_CURRENT, SADB_EXT_ADDRESS_SRC, SADB_EXT_ADDRESS_DST, SADB_EXT_ADDRESS_PROXY, SADB_EXT_KEY_AUTH, SADB_EXT_KEY_ENCRYPT, SADB_EXT_IDENTITY_SRC, SADB_EXT_IDENTITY_DST, SADB_EXT_SENSITIVITY, #ifdef IPSEC_NAT_T SADB_X_EXT_NAT_T_TYPE, SADB_X_EXT_NAT_T_SPORT, SADB_X_EXT_NAT_T_DPORT, SADB_X_EXT_NAT_T_OAI, SADB_X_EXT_NAT_T_OAR, SADB_X_EXT_NAT_T_FRAG, #endif }; m = key_setsadbmsg(type, 0, satype, seq, pid, sav->refcnt); if (m == NULL) goto fail; result = m; for (i = sizeof(dumporder)/sizeof(dumporder[0]) - 1; i >= 0; i--) { m = NULL; switch (dumporder[i]) { case SADB_EXT_SA: m = key_setsadbsa(sav); if (!m) goto fail; break; case SADB_X_EXT_SA2: m = key_setsadbxsa2(sav->sah->saidx.mode, sav->replay ? sav->replay->count : 0, sav->sah->saidx.reqid); if (!m) goto fail; break; case SADB_EXT_ADDRESS_SRC: m = key_setsadbaddr(SADB_EXT_ADDRESS_SRC, &sav->sah->saidx.src.sa, FULLMASK, IPSEC_ULPROTO_ANY); if (!m) goto fail; break; case SADB_EXT_ADDRESS_DST: m = key_setsadbaddr(SADB_EXT_ADDRESS_DST, &sav->sah->saidx.dst.sa, FULLMASK, IPSEC_ULPROTO_ANY); if (!m) goto fail; break; case SADB_EXT_KEY_AUTH: if (!sav->key_auth) continue; m = key_setkey(sav->key_auth, SADB_EXT_KEY_AUTH); if (!m) goto fail; break; case SADB_EXT_KEY_ENCRYPT: if (!sav->key_enc) continue; m = key_setkey(sav->key_enc, SADB_EXT_KEY_ENCRYPT); if (!m) goto fail; break; case SADB_EXT_LIFETIME_CURRENT: if (!sav->lft_c) continue; m = key_setlifetime(sav->lft_c, SADB_EXT_LIFETIME_CURRENT); if (!m) goto fail; break; case SADB_EXT_LIFETIME_HARD: if (!sav->lft_h) continue; m = key_setlifetime(sav->lft_h, SADB_EXT_LIFETIME_HARD); if (!m) goto fail; break; case SADB_EXT_LIFETIME_SOFT: if (!sav->lft_s) continue; m = key_setlifetime(sav->lft_s, SADB_EXT_LIFETIME_SOFT); if (!m) goto fail; break; #ifdef IPSEC_NAT_T case SADB_X_EXT_NAT_T_TYPE: m = key_setsadbxtype(sav->natt_type); if (!m) goto fail; break; case SADB_X_EXT_NAT_T_DPORT: m = key_setsadbxport( KEY_PORTFROMSADDR(&sav->sah->saidx.dst), SADB_X_EXT_NAT_T_DPORT); if (!m) goto fail; break; case SADB_X_EXT_NAT_T_SPORT: m = key_setsadbxport( KEY_PORTFROMSADDR(&sav->sah->saidx.src), SADB_X_EXT_NAT_T_SPORT); if (!m) goto fail; break; case SADB_X_EXT_NAT_T_OAI: case SADB_X_EXT_NAT_T_OAR: case SADB_X_EXT_NAT_T_FRAG: /* We do not (yet) support those. */ continue; #endif case SADB_EXT_ADDRESS_PROXY: case SADB_EXT_IDENTITY_SRC: case SADB_EXT_IDENTITY_DST: /* XXX: should we brought from SPD ? */ case SADB_EXT_SENSITIVITY: default: continue; } if (!m) goto fail; if (tres) m_cat(m, tres); tres = m; } m_cat(result, tres); if (result->m_len < sizeof(struct sadb_msg)) { result = m_pullup(result, sizeof(struct sadb_msg)); if (result == NULL) goto fail; } result->m_pkthdr.len = 0; for (m = result; m; m = m->m_next) result->m_pkthdr.len += m->m_len; mtod(result, struct sadb_msg *)->sadb_msg_len = PFKEY_UNIT64(result->m_pkthdr.len); return result; fail: m_freem(result); m_freem(tres); return NULL; } /* * set data into sadb_msg. */ static struct mbuf * key_setsadbmsg(u_int8_t type, u_int16_t tlen, u_int8_t satype, u_int32_t seq, pid_t pid, u_int16_t reserved) { struct mbuf *m; struct sadb_msg *p; int len; len = PFKEY_ALIGN8(sizeof(struct sadb_msg)); if (len > MCLBYTES) return NULL; MGETHDR(m, M_DONTWAIT, MT_DATA); if (m && len > MHLEN) { MCLGET(m, M_DONTWAIT); if ((m->m_flags & M_EXT) == 0) { m_freem(m); m = NULL; } } if (!m) return NULL; m->m_pkthdr.len = m->m_len = len; m->m_next = NULL; p = mtod(m, struct sadb_msg *); bzero(p, len); p->sadb_msg_version = PF_KEY_V2; p->sadb_msg_type = type; p->sadb_msg_errno = 0; p->sadb_msg_satype = satype; p->sadb_msg_len = PFKEY_UNIT64(tlen); p->sadb_msg_reserved = reserved; p->sadb_msg_seq = seq; p->sadb_msg_pid = (u_int32_t)pid; return m; } /* * copy secasvar data into sadb_address. */ static struct mbuf * key_setsadbsa(sav) struct secasvar *sav; { struct mbuf *m; struct sadb_sa *p; int len; len = PFKEY_ALIGN8(sizeof(struct sadb_sa)); m = key_alloc_mbuf(len); if (!m || m->m_next) { /*XXX*/ if (m) m_freem(m); return NULL; } p = mtod(m, struct sadb_sa *); bzero(p, len); p->sadb_sa_len = PFKEY_UNIT64(len); p->sadb_sa_exttype = SADB_EXT_SA; p->sadb_sa_spi = sav->spi; p->sadb_sa_replay = (sav->replay != NULL ? sav->replay->wsize : 0); p->sadb_sa_state = sav->state; p->sadb_sa_auth = sav->alg_auth; p->sadb_sa_encrypt = sav->alg_enc; p->sadb_sa_flags = sav->flags; return m; } /* * set data into sadb_address. */ static struct mbuf * key_setsadbaddr(u_int16_t exttype, const struct sockaddr *saddr, u_int8_t prefixlen, u_int16_t ul_proto) { struct mbuf *m; struct sadb_address *p; size_t len; len = PFKEY_ALIGN8(sizeof(struct sadb_address)) + PFKEY_ALIGN8(saddr->sa_len); m = key_alloc_mbuf(len); if (!m || m->m_next) { /*XXX*/ if (m) m_freem(m); return NULL; } p = mtod(m, struct sadb_address *); bzero(p, len); p->sadb_address_len = PFKEY_UNIT64(len); p->sadb_address_exttype = exttype; p->sadb_address_proto = ul_proto; if (prefixlen == FULLMASK) { switch (saddr->sa_family) { case AF_INET: prefixlen = sizeof(struct in_addr) << 3; break; case AF_INET6: prefixlen = sizeof(struct in6_addr) << 3; break; default: ; /*XXX*/ } } p->sadb_address_prefixlen = prefixlen; p->sadb_address_reserved = 0; bcopy(saddr, mtod(m, caddr_t) + PFKEY_ALIGN8(sizeof(struct sadb_address)), saddr->sa_len); return m; } /* * set data into sadb_x_sa2. */ static struct mbuf * key_setsadbxsa2(u_int8_t mode, u_int32_t seq, u_int32_t reqid) { struct mbuf *m; struct sadb_x_sa2 *p; size_t len; len = PFKEY_ALIGN8(sizeof(struct sadb_x_sa2)); m = key_alloc_mbuf(len); if (!m || m->m_next) { /*XXX*/ if (m) m_freem(m); return NULL; } p = mtod(m, struct sadb_x_sa2 *); bzero(p, len); p->sadb_x_sa2_len = PFKEY_UNIT64(len); p->sadb_x_sa2_exttype = SADB_X_EXT_SA2; p->sadb_x_sa2_mode = mode; p->sadb_x_sa2_reserved1 = 0; p->sadb_x_sa2_reserved2 = 0; p->sadb_x_sa2_sequence = seq; p->sadb_x_sa2_reqid = reqid; return m; } #ifdef IPSEC_NAT_T /* * Set a type in sadb_x_nat_t_type. */ static struct mbuf * key_setsadbxtype(u_int16_t type) { struct mbuf *m; size_t len; struct sadb_x_nat_t_type *p; len = PFKEY_ALIGN8(sizeof(struct sadb_x_nat_t_type)); m = key_alloc_mbuf(len); if (!m || m->m_next) { /*XXX*/ if (m) m_freem(m); return (NULL); } p = mtod(m, struct sadb_x_nat_t_type *); bzero(p, len); p->sadb_x_nat_t_type_len = PFKEY_UNIT64(len); p->sadb_x_nat_t_type_exttype = SADB_X_EXT_NAT_T_TYPE; p->sadb_x_nat_t_type_type = type; return (m); } /* * Set a port in sadb_x_nat_t_port. * In contrast to default RFC 2367 behaviour, port is in network byte order. */ static struct mbuf * key_setsadbxport(u_int16_t port, u_int16_t type) { struct mbuf *m; size_t len; struct sadb_x_nat_t_port *p; len = PFKEY_ALIGN8(sizeof(struct sadb_x_nat_t_port)); m = key_alloc_mbuf(len); if (!m || m->m_next) { /*XXX*/ if (m) m_freem(m); return (NULL); } p = mtod(m, struct sadb_x_nat_t_port *); bzero(p, len); p->sadb_x_nat_t_port_len = PFKEY_UNIT64(len); p->sadb_x_nat_t_port_exttype = type; p->sadb_x_nat_t_port_port = port; return (m); } /* * Get port from sockaddr. Port is in network byte order. */ u_int16_t key_portfromsaddr(struct sockaddr *sa) { switch (sa->sa_family) { #ifdef INET case AF_INET: return ((struct sockaddr_in *)sa)->sin_port; #endif #ifdef INET6 case AF_INET6: return ((struct sockaddr_in6 *)sa)->sin6_port; #endif } KEYDEBUG(KEYDEBUG_IPSEC_STAMP, printf("DP %s unexpected address family %d\n", __func__, sa->sa_family)); return (0); } #endif /* IPSEC_NAT_T */ /* * Set port in struct sockaddr. Port is in network byte order. */ static void key_porttosaddr(struct sockaddr *sa, u_int16_t port) { switch (sa->sa_family) { #ifdef INET case AF_INET: ((struct sockaddr_in *)sa)->sin_port = port; break; #endif #ifdef INET6 case AF_INET6: ((struct sockaddr_in6 *)sa)->sin6_port = port; break; #endif default: ipseclog((LOG_DEBUG, "%s: unexpected address family %d.\n", __func__, sa->sa_family)); break; } } /* * set data into sadb_x_policy */ static struct mbuf * key_setsadbxpolicy(u_int16_t type, u_int8_t dir, u_int32_t id) { struct mbuf *m; struct sadb_x_policy *p; size_t len; len = PFKEY_ALIGN8(sizeof(struct sadb_x_policy)); m = key_alloc_mbuf(len); if (!m || m->m_next) { /*XXX*/ if (m) m_freem(m); return NULL; } p = mtod(m, struct sadb_x_policy *); bzero(p, len); p->sadb_x_policy_len = PFKEY_UNIT64(len); p->sadb_x_policy_exttype = SADB_X_EXT_POLICY; p->sadb_x_policy_type = type; p->sadb_x_policy_dir = dir; p->sadb_x_policy_id = id; return m; } /* %%% utilities */ /* Take a key message (sadb_key) from the socket and turn it into one * of the kernel's key structures (seckey). * * IN: pointer to the src * OUT: NULL no more memory */ struct seckey * key_dup_keymsg(const struct sadb_key *src, u_int len, struct malloc_type *type) { struct seckey *dst; dst = (struct seckey *)malloc(sizeof(struct seckey), type, M_NOWAIT); if (dst != NULL) { dst->bits = src->sadb_key_bits; dst->key_data = (char *)malloc(len, type, M_NOWAIT); if (dst->key_data != NULL) { bcopy((const char *)src + sizeof(struct sadb_key), dst->key_data, len); } else { ipseclog((LOG_DEBUG, "%s: No more memory.\n", __func__)); free(dst, type); dst = NULL; } } else { ipseclog((LOG_DEBUG, "%s: No more memory.\n", __func__)); } return dst; } /* Take a lifetime message (sadb_lifetime) passed in on a socket and * turn it into one of the kernel's lifetime structures (seclifetime). * * IN: pointer to the destination, source and malloc type * OUT: NULL, no more memory */ static struct seclifetime * key_dup_lifemsg(const struct sadb_lifetime *src, struct malloc_type *type) { struct seclifetime *dst = NULL; dst = (struct seclifetime *)malloc(sizeof(struct seclifetime), type, M_NOWAIT); if (dst == NULL) { /* XXX counter */ ipseclog((LOG_DEBUG, "%s: No more memory.\n", __func__)); } else { dst->allocations = src->sadb_lifetime_allocations; dst->bytes = src->sadb_lifetime_bytes; dst->addtime = src->sadb_lifetime_addtime; dst->usetime = src->sadb_lifetime_usetime; } return dst; } /* compare my own address * OUT: 1: true, i.e. my address. * 0: false */ int key_ismyaddr(sa) struct sockaddr *sa; { #ifdef INET struct sockaddr_in *sin; struct in_ifaddr *ia; #endif IPSEC_ASSERT(sa != NULL, ("null sockaddr")); switch (sa->sa_family) { #ifdef INET case AF_INET: sin = (struct sockaddr_in *)sa; IN_IFADDR_RLOCK(); for (ia = V_in_ifaddrhead.tqh_first; ia; ia = ia->ia_link.tqe_next) { if (sin->sin_family == ia->ia_addr.sin_family && sin->sin_len == ia->ia_addr.sin_len && sin->sin_addr.s_addr == ia->ia_addr.sin_addr.s_addr) { IN_IFADDR_RUNLOCK(); return 1; } } IN_IFADDR_RUNLOCK(); break; #endif #ifdef INET6 case AF_INET6: return key_ismyaddr6((struct sockaddr_in6 *)sa); #endif } return 0; } #ifdef INET6 /* * compare my own address for IPv6. * 1: ours * 0: other * NOTE: derived ip6_input() in KAME. This is necessary to modify more. */ #include static int key_ismyaddr6(sin6) struct sockaddr_in6 *sin6; { struct in6_ifaddr *ia; #if 0 struct in6_multi *in6m; #endif IN6_IFADDR_RLOCK(); TAILQ_FOREACH(ia, &V_in6_ifaddrhead, ia_link) { if (key_sockaddrcmp((struct sockaddr *)&sin6, (struct sockaddr *)&ia->ia_addr, 0) == 0) { IN6_IFADDR_RUNLOCK(); return 1; } #if 0 /* * XXX Multicast * XXX why do we care about multlicast here while we don't care * about IPv4 multicast?? * XXX scope */ in6m = NULL; IN6_LOOKUP_MULTI(sin6->sin6_addr, ia->ia_ifp, in6m); if (in6m) { IN6_IFADDR_RUNLOCK(); return 1; } #endif } IN6_IFADDR_RUNLOCK(); /* loopback, just for safety */ if (IN6_IS_ADDR_LOOPBACK(&sin6->sin6_addr)) return 1; return 0; } #endif /*INET6*/ /* * compare two secasindex structure. * flag can specify to compare 2 saidxes. * compare two secasindex structure without both mode and reqid. * don't compare port. * IN: * saidx0: source, it can be in SAD. * saidx1: object. * OUT: * 1 : equal * 0 : not equal */ static int key_cmpsaidx( const struct secasindex *saidx0, const struct secasindex *saidx1, int flag) { int chkport = 0; /* sanity */ if (saidx0 == NULL && saidx1 == NULL) return 1; if (saidx0 == NULL || saidx1 == NULL) return 0; if (saidx0->proto != saidx1->proto) return 0; if (flag == CMP_EXACTLY) { if (saidx0->mode != saidx1->mode) return 0; if (saidx0->reqid != saidx1->reqid) return 0; if (bcmp(&saidx0->src, &saidx1->src, saidx0->src.sa.sa_len) != 0 || bcmp(&saidx0->dst, &saidx1->dst, saidx0->dst.sa.sa_len) != 0) return 0; } else { /* CMP_MODE_REQID, CMP_REQID, CMP_HEAD */ if (flag == CMP_MODE_REQID ||flag == CMP_REQID) { /* * If reqid of SPD is non-zero, unique SA is required. * The result must be of same reqid in this case. */ if (saidx1->reqid != 0 && saidx0->reqid != saidx1->reqid) return 0; } if (flag == CMP_MODE_REQID) { if (saidx0->mode != IPSEC_MODE_ANY && saidx0->mode != saidx1->mode) return 0; } #ifdef IPSEC_NAT_T /* * If NAT-T is enabled, check ports for tunnel mode. * Do not check ports if they are set to zero in the SPD. * Also do not do it for transport mode, as there is no * port information available in the SP. */ if (saidx1->mode == IPSEC_MODE_TUNNEL && saidx1->src.sa.sa_family == AF_INET && saidx1->dst.sa.sa_family == AF_INET && ((const struct sockaddr_in *)(&saidx1->src))->sin_port && ((const struct sockaddr_in *)(&saidx1->dst))->sin_port) chkport = 1; #endif /* IPSEC_NAT_T */ if (key_sockaddrcmp(&saidx0->src.sa, &saidx1->src.sa, chkport) != 0) { return 0; } if (key_sockaddrcmp(&saidx0->dst.sa, &saidx1->dst.sa, chkport) != 0) { return 0; } } return 1; } /* * compare two secindex structure exactly. * IN: * spidx0: source, it is often in SPD. * spidx1: object, it is often from PFKEY message. * OUT: * 1 : equal * 0 : not equal */ static int key_cmpspidx_exactly( struct secpolicyindex *spidx0, struct secpolicyindex *spidx1) { /* sanity */ if (spidx0 == NULL && spidx1 == NULL) return 1; if (spidx0 == NULL || spidx1 == NULL) return 0; if (spidx0->prefs != spidx1->prefs || spidx0->prefd != spidx1->prefd || spidx0->ul_proto != spidx1->ul_proto) return 0; return key_sockaddrcmp(&spidx0->src.sa, &spidx1->src.sa, 1) == 0 && key_sockaddrcmp(&spidx0->dst.sa, &spidx1->dst.sa, 1) == 0; } /* * compare two secindex structure with mask. * IN: * spidx0: source, it is often in SPD. * spidx1: object, it is often from IP header. * OUT: * 1 : equal * 0 : not equal */ static int key_cmpspidx_withmask( struct secpolicyindex *spidx0, struct secpolicyindex *spidx1) { /* sanity */ if (spidx0 == NULL && spidx1 == NULL) return 1; if (spidx0 == NULL || spidx1 == NULL) return 0; if (spidx0->src.sa.sa_family != spidx1->src.sa.sa_family || spidx0->dst.sa.sa_family != spidx1->dst.sa.sa_family || spidx0->src.sa.sa_len != spidx1->src.sa.sa_len || spidx0->dst.sa.sa_len != spidx1->dst.sa.sa_len) return 0; /* if spidx.ul_proto == IPSEC_ULPROTO_ANY, ignore. */ if (spidx0->ul_proto != (u_int16_t)IPSEC_ULPROTO_ANY && spidx0->ul_proto != spidx1->ul_proto) return 0; switch (spidx0->src.sa.sa_family) { case AF_INET: if (spidx0->src.sin.sin_port != IPSEC_PORT_ANY && spidx0->src.sin.sin_port != spidx1->src.sin.sin_port) return 0; if (!key_bbcmp(&spidx0->src.sin.sin_addr, &spidx1->src.sin.sin_addr, spidx0->prefs)) return 0; break; case AF_INET6: if (spidx0->src.sin6.sin6_port != IPSEC_PORT_ANY && spidx0->src.sin6.sin6_port != spidx1->src.sin6.sin6_port) return 0; /* * scope_id check. if sin6_scope_id is 0, we regard it * as a wildcard scope, which matches any scope zone ID. */ if (spidx0->src.sin6.sin6_scope_id && spidx1->src.sin6.sin6_scope_id && spidx0->src.sin6.sin6_scope_id != spidx1->src.sin6.sin6_scope_id) return 0; if (!key_bbcmp(&spidx0->src.sin6.sin6_addr, &spidx1->src.sin6.sin6_addr, spidx0->prefs)) return 0; break; default: /* XXX */ if (bcmp(&spidx0->src, &spidx1->src, spidx0->src.sa.sa_len) != 0) return 0; break; } switch (spidx0->dst.sa.sa_family) { case AF_INET: if (spidx0->dst.sin.sin_port != IPSEC_PORT_ANY && spidx0->dst.sin.sin_port != spidx1->dst.sin.sin_port) return 0; if (!key_bbcmp(&spidx0->dst.sin.sin_addr, &spidx1->dst.sin.sin_addr, spidx0->prefd)) return 0; break; case AF_INET6: if (spidx0->dst.sin6.sin6_port != IPSEC_PORT_ANY && spidx0->dst.sin6.sin6_port != spidx1->dst.sin6.sin6_port) return 0; /* * scope_id check. if sin6_scope_id is 0, we regard it * as a wildcard scope, which matches any scope zone ID. */ if (spidx0->dst.sin6.sin6_scope_id && spidx1->dst.sin6.sin6_scope_id && spidx0->dst.sin6.sin6_scope_id != spidx1->dst.sin6.sin6_scope_id) return 0; if (!key_bbcmp(&spidx0->dst.sin6.sin6_addr, &spidx1->dst.sin6.sin6_addr, spidx0->prefd)) return 0; break; default: /* XXX */ if (bcmp(&spidx0->dst, &spidx1->dst, spidx0->dst.sa.sa_len) != 0) return 0; break; } /* XXX Do we check other field ? e.g. flowinfo */ return 1; } /* returns 0 on match */ static int key_sockaddrcmp( const struct sockaddr *sa1, const struct sockaddr *sa2, int port) { #ifdef satosin #undef satosin #endif #define satosin(s) ((const struct sockaddr_in *)s) #ifdef satosin6 #undef satosin6 #endif #define satosin6(s) ((const struct sockaddr_in6 *)s) if (sa1->sa_family != sa2->sa_family || sa1->sa_len != sa2->sa_len) return 1; switch (sa1->sa_family) { case AF_INET: if (sa1->sa_len != sizeof(struct sockaddr_in)) return 1; if (satosin(sa1)->sin_addr.s_addr != satosin(sa2)->sin_addr.s_addr) { return 1; } if (port && satosin(sa1)->sin_port != satosin(sa2)->sin_port) return 1; break; case AF_INET6: if (sa1->sa_len != sizeof(struct sockaddr_in6)) return 1; /*EINVAL*/ if (satosin6(sa1)->sin6_scope_id != satosin6(sa2)->sin6_scope_id) { return 1; } if (!IN6_ARE_ADDR_EQUAL(&satosin6(sa1)->sin6_addr, &satosin6(sa2)->sin6_addr)) { return 1; } if (port && satosin6(sa1)->sin6_port != satosin6(sa2)->sin6_port) { return 1; } break; default: if (bcmp(sa1, sa2, sa1->sa_len) != 0) return 1; break; } return 0; #undef satosin #undef satosin6 } /* * compare two buffers with mask. * IN: * addr1: source * addr2: object * bits: Number of bits to compare * OUT: * 1 : equal * 0 : not equal */ static int key_bbcmp(const void *a1, const void *a2, u_int bits) { const unsigned char *p1 = a1; const unsigned char *p2 = a2; /* XXX: This could be considerably faster if we compare a word * at a time, but it is complicated on LSB Endian machines */ /* Handle null pointers */ if (p1 == NULL || p2 == NULL) return (p1 == p2); while (bits >= 8) { if (*p1++ != *p2++) return 0; bits -= 8; } if (bits > 0) { u_int8_t mask = ~((1<<(8-bits))-1); if ((*p1 & mask) != (*p2 & mask)) return 0; } return 1; /* Match! */ } static void key_flush_spd(time_t now) { static u_int16_t sptree_scangen = 0; u_int16_t gen = sptree_scangen++; struct secpolicy *sp; u_int dir; /* SPD */ for (dir = 0; dir < IPSEC_DIR_MAX; dir++) { restart: SPTREE_LOCK(); LIST_FOREACH(sp, &V_sptree[dir], chain) { if (sp->scangen == gen) /* previously handled */ continue; sp->scangen = gen; if (sp->state == IPSEC_SPSTATE_DEAD && sp->refcnt == 1) { /* * Ensure that we only decrease refcnt once, * when we're the last consumer. * Directly call SP_DELREF/key_delsp instead * of KEY_FREESP to avoid unlocking/relocking * SPTREE_LOCK before key_delsp: may refcnt * be increased again during that time ? * NB: also clean entries created by * key_spdflush */ SP_DELREF(sp); key_delsp(sp); SPTREE_UNLOCK(); goto restart; } if (sp->lifetime == 0 && sp->validtime == 0) continue; if ((sp->lifetime && now - sp->created > sp->lifetime) || (sp->validtime && now - sp->lastused > sp->validtime)) { sp->state = IPSEC_SPSTATE_DEAD; SPTREE_UNLOCK(); key_spdexpire(sp); goto restart; } } SPTREE_UNLOCK(); } } static void key_flush_sad(time_t now) { struct secashead *sah, *nextsah; struct secasvar *sav, *nextsav; /* SAD */ SAHTREE_LOCK(); LIST_FOREACH_SAFE(sah, &V_sahtree, chain, nextsah) { /* if sah has been dead, then delete it and process next sah. */ if (sah->state == SADB_SASTATE_DEAD) { key_delsah(sah); continue; } /* if LARVAL entry doesn't become MATURE, delete it. */ LIST_FOREACH_SAFE(sav, &sah->savtree[SADB_SASTATE_LARVAL], chain, nextsav) { /* Need to also check refcnt for a larval SA ??? */ if (now - sav->created > V_key_larval_lifetime) KEY_FREESAV(&sav); } /* * check MATURE entry to start to send expire message * whether or not. */ LIST_FOREACH_SAFE(sav, &sah->savtree[SADB_SASTATE_MATURE], chain, nextsav) { /* we don't need to check. */ if (sav->lft_s == NULL) continue; /* sanity check */ if (sav->lft_c == NULL) { ipseclog((LOG_DEBUG,"%s: there is no CURRENT " "time, why?\n", __func__)); continue; } /* check SOFT lifetime */ if (sav->lft_s->addtime != 0 && now - sav->created > sav->lft_s->addtime) { key_sa_chgstate(sav, SADB_SASTATE_DYING); /* * Actually, only send expire message if * SA has been used, as it was done before, * but should we always send such message, * and let IKE daemon decide if it should be * renegotiated or not ? * XXX expire message will actually NOT be * sent if SA is only used after soft * lifetime has been reached, see below * (DYING state) */ if (sav->lft_c->usetime != 0) key_expire(sav); } /* check SOFT lifetime by bytes */ /* * XXX I don't know the way to delete this SA * when new SA is installed. Caution when it's * installed too big lifetime by time. */ else if (sav->lft_s->bytes != 0 && sav->lft_s->bytes < sav->lft_c->bytes) { key_sa_chgstate(sav, SADB_SASTATE_DYING); /* * XXX If we keep to send expire * message in the status of * DYING. Do remove below code. */ key_expire(sav); } } /* check DYING entry to change status to DEAD. */ LIST_FOREACH_SAFE(sav, &sah->savtree[SADB_SASTATE_DYING], chain, nextsav) { /* we don't need to check. */ if (sav->lft_h == NULL) continue; /* sanity check */ if (sav->lft_c == NULL) { ipseclog((LOG_DEBUG, "%s: there is no CURRENT " "time, why?\n", __func__)); continue; } if (sav->lft_h->addtime != 0 && now - sav->created > sav->lft_h->addtime) { key_sa_chgstate(sav, SADB_SASTATE_DEAD); KEY_FREESAV(&sav); } #if 0 /* XXX Should we keep to send expire message until HARD lifetime ? */ else if (sav->lft_s != NULL && sav->lft_s->addtime != 0 && now - sav->created > sav->lft_s->addtime) { /* * XXX: should be checked to be * installed the valid SA. */ /* * If there is no SA then sending * expire message. */ key_expire(sav); } #endif /* check HARD lifetime by bytes */ else if (sav->lft_h->bytes != 0 && sav->lft_h->bytes < sav->lft_c->bytes) { key_sa_chgstate(sav, SADB_SASTATE_DEAD); KEY_FREESAV(&sav); } } /* delete entry in DEAD */ LIST_FOREACH_SAFE(sav, &sah->savtree[SADB_SASTATE_DEAD], chain, nextsav) { /* sanity check */ if (sav->state != SADB_SASTATE_DEAD) { ipseclog((LOG_DEBUG, "%s: invalid sav->state " "(queue: %d SA: %d): kill it anyway\n", __func__, SADB_SASTATE_DEAD, sav->state)); } /* * do not call key_freesav() here. * sav should already be freed, and sav->refcnt * shows other references to sav * (such as from SPD). */ } } SAHTREE_UNLOCK(); } static void key_flush_acq(time_t now) { struct secacq *acq, *nextacq; /* ACQ tree */ ACQ_LOCK(); for (acq = LIST_FIRST(&V_acqtree); acq != NULL; acq = nextacq) { nextacq = LIST_NEXT(acq, chain); if (now - acq->created > V_key_blockacq_lifetime && __LIST_CHAINED(acq)) { LIST_REMOVE(acq, chain); free(acq, M_IPSEC_SAQ); } } ACQ_UNLOCK(); } static void key_flush_spacq(time_t now) { struct secspacq *acq, *nextacq; /* SP ACQ tree */ SPACQ_LOCK(); for (acq = LIST_FIRST(&V_spacqtree); acq != NULL; acq = nextacq) { nextacq = LIST_NEXT(acq, chain); if (now - acq->created > V_key_blockacq_lifetime && __LIST_CHAINED(acq)) { LIST_REMOVE(acq, chain); free(acq, M_IPSEC_SAQ); } } SPACQ_UNLOCK(); } /* * time handler. * scanning SPD and SAD to check status for each entries, * and do to remove or to expire. * XXX: year 2038 problem may remain. */ void key_timehandler(void) { VNET_ITERATOR_DECL(vnet_iter); time_t now = time_second; VNET_LIST_RLOCK_NOSLEEP(); VNET_FOREACH(vnet_iter) { CURVNET_SET(vnet_iter); key_flush_spd(now); key_flush_sad(now); key_flush_acq(now); key_flush_spacq(now); CURVNET_RESTORE(); } VNET_LIST_RUNLOCK_NOSLEEP(); #ifndef IPSEC_DEBUG2 /* do exchange to tick time !! */ (void)timeout((void *)key_timehandler, (void *)0, hz); #endif /* IPSEC_DEBUG2 */ } u_long key_random() { u_long value; key_randomfill(&value, sizeof(value)); return value; } void key_randomfill(p, l) void *p; size_t l; { size_t n; u_long v; static int warn = 1; n = 0; n = (size_t)read_random(p, (u_int)l); /* last resort */ while (n < l) { v = random(); bcopy(&v, (u_int8_t *)p + n, l - n < sizeof(v) ? l - n : sizeof(v)); n += sizeof(v); if (warn) { printf("WARNING: pseudo-random number generator " "used for IPsec processing\n"); warn = 0; } } } /* * map SADB_SATYPE_* to IPPROTO_*. * if satype == SADB_SATYPE then satype is mapped to ~0. * OUT: * 0: invalid satype. */ static u_int16_t key_satype2proto(u_int8_t satype) { switch (satype) { case SADB_SATYPE_UNSPEC: return IPSEC_PROTO_ANY; case SADB_SATYPE_AH: return IPPROTO_AH; case SADB_SATYPE_ESP: return IPPROTO_ESP; case SADB_X_SATYPE_IPCOMP: return IPPROTO_IPCOMP; case SADB_X_SATYPE_TCPSIGNATURE: return IPPROTO_TCP; default: return 0; } /* NOTREACHED */ } /* * map IPPROTO_* to SADB_SATYPE_* * OUT: * 0: invalid protocol type. */ static u_int8_t key_proto2satype(u_int16_t proto) { switch (proto) { case IPPROTO_AH: return SADB_SATYPE_AH; case IPPROTO_ESP: return SADB_SATYPE_ESP; case IPPROTO_IPCOMP: return SADB_X_SATYPE_IPCOMP; case IPPROTO_TCP: return SADB_X_SATYPE_TCPSIGNATURE; default: return 0; } /* NOTREACHED */ } /* %%% PF_KEY */ /* * SADB_GETSPI processing is to receive * * from the IKMPd, to assign a unique spi value, to hang on the INBOUND * tree with the status of LARVAL, and send * * to the IKMPd. * * IN: mhp: pointer to the pointer to each header. * OUT: NULL if fail. * other if success, return pointer to the message to send. */ static int key_getspi(so, m, mhp) struct socket *so; struct mbuf *m; const struct sadb_msghdr *mhp; { struct sadb_address *src0, *dst0; struct secasindex saidx; struct secashead *newsah; struct secasvar *newsav; u_int8_t proto; u_int32_t spi; u_int8_t mode; u_int32_t reqid; int error; IPSEC_ASSERT(so != NULL, ("null socket")); IPSEC_ASSERT(m != NULL, ("null mbuf")); IPSEC_ASSERT(mhp != NULL, ("null msghdr")); IPSEC_ASSERT(mhp->msg != NULL, ("null msg")); if (mhp->ext[SADB_EXT_ADDRESS_SRC] == NULL || mhp->ext[SADB_EXT_ADDRESS_DST] == NULL) { ipseclog((LOG_DEBUG, "%s: invalid message is passed.\n", __func__)); return key_senderror(so, m, EINVAL); } if (mhp->extlen[SADB_EXT_ADDRESS_SRC] < sizeof(struct sadb_address) || mhp->extlen[SADB_EXT_ADDRESS_DST] < sizeof(struct sadb_address)) { ipseclog((LOG_DEBUG, "%s: invalid message is passed.\n", __func__)); return key_senderror(so, m, EINVAL); } if (mhp->ext[SADB_X_EXT_SA2] != NULL) { mode = ((struct sadb_x_sa2 *)mhp->ext[SADB_X_EXT_SA2])->sadb_x_sa2_mode; reqid = ((struct sadb_x_sa2 *)mhp->ext[SADB_X_EXT_SA2])->sadb_x_sa2_reqid; } else { mode = IPSEC_MODE_ANY; reqid = 0; } src0 = (struct sadb_address *)(mhp->ext[SADB_EXT_ADDRESS_SRC]); dst0 = (struct sadb_address *)(mhp->ext[SADB_EXT_ADDRESS_DST]); /* map satype to proto */ if ((proto = key_satype2proto(mhp->msg->sadb_msg_satype)) == 0) { ipseclog((LOG_DEBUG, "%s: invalid satype is passed.\n", __func__)); return key_senderror(so, m, EINVAL); } /* * Make sure the port numbers are zero. * In case of NAT-T we will update them later if needed. */ switch (((struct sockaddr *)(src0 + 1))->sa_family) { case AF_INET: if (((struct sockaddr *)(src0 + 1))->sa_len != sizeof(struct sockaddr_in)) return key_senderror(so, m, EINVAL); ((struct sockaddr_in *)(src0 + 1))->sin_port = 0; break; case AF_INET6: if (((struct sockaddr *)(src0 + 1))->sa_len != sizeof(struct sockaddr_in6)) return key_senderror(so, m, EINVAL); ((struct sockaddr_in6 *)(src0 + 1))->sin6_port = 0; break; default: ; /*???*/ } switch (((struct sockaddr *)(dst0 + 1))->sa_family) { case AF_INET: if (((struct sockaddr *)(dst0 + 1))->sa_len != sizeof(struct sockaddr_in)) return key_senderror(so, m, EINVAL); ((struct sockaddr_in *)(dst0 + 1))->sin_port = 0; break; case AF_INET6: if (((struct sockaddr *)(dst0 + 1))->sa_len != sizeof(struct sockaddr_in6)) return key_senderror(so, m, EINVAL); ((struct sockaddr_in6 *)(dst0 + 1))->sin6_port = 0; break; default: ; /*???*/ } /* XXX boundary check against sa_len */ KEY_SETSECASIDX(proto, mode, reqid, src0 + 1, dst0 + 1, &saidx); #ifdef IPSEC_NAT_T /* * Handle NAT-T info if present. * We made sure the port numbers are zero above, so we do * not have to worry in case we do not update them. */ if (mhp->ext[SADB_X_EXT_NAT_T_OAI] != NULL) ipseclog((LOG_DEBUG, "%s: NAT-T OAi present\n", __func__)); if (mhp->ext[SADB_X_EXT_NAT_T_OAR] != NULL) ipseclog((LOG_DEBUG, "%s: NAT-T OAr present\n", __func__)); if (mhp->ext[SADB_X_EXT_NAT_T_TYPE] != NULL && mhp->ext[SADB_X_EXT_NAT_T_SPORT] != NULL && mhp->ext[SADB_X_EXT_NAT_T_DPORT] != NULL) { struct sadb_x_nat_t_type *type; struct sadb_x_nat_t_port *sport, *dport; if (mhp->extlen[SADB_X_EXT_NAT_T_TYPE] < sizeof(*type) || mhp->extlen[SADB_X_EXT_NAT_T_SPORT] < sizeof(*sport) || mhp->extlen[SADB_X_EXT_NAT_T_DPORT] < sizeof(*dport)) { ipseclog((LOG_DEBUG, "%s: invalid nat-t message " "passed.\n", __func__)); return key_senderror(so, m, EINVAL); } sport = (struct sadb_x_nat_t_port *) mhp->ext[SADB_X_EXT_NAT_T_SPORT]; dport = (struct sadb_x_nat_t_port *) mhp->ext[SADB_X_EXT_NAT_T_DPORT]; if (sport) KEY_PORTTOSADDR(&saidx.src, sport->sadb_x_nat_t_port_port); if (dport) KEY_PORTTOSADDR(&saidx.dst, dport->sadb_x_nat_t_port_port); } #endif /* SPI allocation */ spi = key_do_getnewspi((struct sadb_spirange *)mhp->ext[SADB_EXT_SPIRANGE], &saidx); if (spi == 0) return key_senderror(so, m, EINVAL); /* get a SA index */ if ((newsah = key_getsah(&saidx)) == NULL) { /* create a new SA index */ if ((newsah = key_newsah(&saidx)) == NULL) { ipseclog((LOG_DEBUG, "%s: No more memory.\n",__func__)); return key_senderror(so, m, ENOBUFS); } } /* get a new SA */ /* XXX rewrite */ newsav = KEY_NEWSAV(m, mhp, newsah, &error); if (newsav == NULL) { /* XXX don't free new SA index allocated in above. */ return key_senderror(so, m, error); } /* set spi */ newsav->spi = htonl(spi); /* delete the entry in acqtree */ if (mhp->msg->sadb_msg_seq != 0) { struct secacq *acq; if ((acq = key_getacqbyseq(mhp->msg->sadb_msg_seq)) != NULL) { /* reset counter in order to deletion by timehandler. */ acq->created = time_second; acq->count = 0; } } { struct mbuf *n, *nn; struct sadb_sa *m_sa; struct sadb_msg *newmsg; int off, len; /* create new sadb_msg to reply. */ len = PFKEY_ALIGN8(sizeof(struct sadb_msg)) + PFKEY_ALIGN8(sizeof(struct sadb_sa)); MGETHDR(n, M_DONTWAIT, MT_DATA); if (len > MHLEN) { MCLGET(n, M_DONTWAIT); if ((n->m_flags & M_EXT) == 0) { m_freem(n); n = NULL; } } if (!n) return key_senderror(so, m, ENOBUFS); n->m_len = len; n->m_next = NULL; off = 0; m_copydata(m, 0, sizeof(struct sadb_msg), mtod(n, caddr_t) + off); off += PFKEY_ALIGN8(sizeof(struct sadb_msg)); m_sa = (struct sadb_sa *)(mtod(n, caddr_t) + off); m_sa->sadb_sa_len = PFKEY_UNIT64(sizeof(struct sadb_sa)); m_sa->sadb_sa_exttype = SADB_EXT_SA; m_sa->sadb_sa_spi = htonl(spi); off += PFKEY_ALIGN8(sizeof(struct sadb_sa)); IPSEC_ASSERT(off == len, ("length inconsistency (off %u len %u)", off, len)); n->m_next = key_gather_mbuf(m, mhp, 0, 2, SADB_EXT_ADDRESS_SRC, SADB_EXT_ADDRESS_DST); if (!n->m_next) { m_freem(n); return key_senderror(so, m, ENOBUFS); } if (n->m_len < sizeof(struct sadb_msg)) { n = m_pullup(n, sizeof(struct sadb_msg)); if (n == NULL) return key_sendup_mbuf(so, m, KEY_SENDUP_ONE); } n->m_pkthdr.len = 0; for (nn = n; nn; nn = nn->m_next) n->m_pkthdr.len += nn->m_len; newmsg = mtod(n, struct sadb_msg *); newmsg->sadb_msg_seq = newsav->seq; newmsg->sadb_msg_errno = 0; newmsg->sadb_msg_len = PFKEY_UNIT64(n->m_pkthdr.len); m_freem(m); return key_sendup_mbuf(so, n, KEY_SENDUP_ONE); } } /* * allocating new SPI * called by key_getspi(). * OUT: * 0: failure. * others: success. */ static u_int32_t key_do_getnewspi(spirange, saidx) struct sadb_spirange *spirange; struct secasindex *saidx; { u_int32_t newspi; u_int32_t min, max; int count = V_key_spi_trycnt; /* set spi range to allocate */ if (spirange != NULL) { min = spirange->sadb_spirange_min; max = spirange->sadb_spirange_max; } else { min = V_key_spi_minval; max = V_key_spi_maxval; } /* IPCOMP needs 2-byte SPI */ if (saidx->proto == IPPROTO_IPCOMP) { u_int32_t t; if (min >= 0x10000) min = 0xffff; if (max >= 0x10000) max = 0xffff; if (min > max) { t = min; min = max; max = t; } } if (min == max) { if (key_checkspidup(saidx, min) != NULL) { ipseclog((LOG_DEBUG, "%s: SPI %u exists already.\n", __func__, min)); return 0; } count--; /* taking one cost. */ newspi = min; } else { /* init SPI */ newspi = 0; /* when requesting to allocate spi ranged */ while (count--) { /* generate pseudo-random SPI value ranged. */ newspi = min + (key_random() % (max - min + 1)); if (key_checkspidup(saidx, newspi) == NULL) break; } if (count == 0 || newspi == 0) { ipseclog((LOG_DEBUG, "%s: to allocate spi is failed.\n", __func__)); return 0; } } /* statistics */ keystat.getspi_count = (keystat.getspi_count + V_key_spi_trycnt - count) / 2; return newspi; } /* * SADB_UPDATE processing * receive * * from the ikmpd, and update a secasvar entry whose status is SADB_SASTATE_LARVAL. * and send * * to the ikmpd. * * m will always be freed. */ static int key_update(so, m, mhp) struct socket *so; struct mbuf *m; const struct sadb_msghdr *mhp; { struct sadb_sa *sa0; struct sadb_address *src0, *dst0; #ifdef IPSEC_NAT_T struct sadb_x_nat_t_type *type; struct sadb_x_nat_t_port *sport, *dport; struct sadb_address *iaddr, *raddr; struct sadb_x_nat_t_frag *frag; #endif struct secasindex saidx; struct secashead *sah; struct secasvar *sav; u_int16_t proto; u_int8_t mode; u_int32_t reqid; int error; IPSEC_ASSERT(so != NULL, ("null socket")); IPSEC_ASSERT(m != NULL, ("null mbuf")); IPSEC_ASSERT(mhp != NULL, ("null msghdr")); IPSEC_ASSERT(mhp->msg != NULL, ("null msg")); /* map satype to proto */ if ((proto = key_satype2proto(mhp->msg->sadb_msg_satype)) == 0) { ipseclog((LOG_DEBUG, "%s: invalid satype is passed.\n", __func__)); return key_senderror(so, m, EINVAL); } if (mhp->ext[SADB_EXT_SA] == NULL || mhp->ext[SADB_EXT_ADDRESS_SRC] == NULL || mhp->ext[SADB_EXT_ADDRESS_DST] == NULL || (mhp->msg->sadb_msg_satype == SADB_SATYPE_ESP && mhp->ext[SADB_EXT_KEY_ENCRYPT] == NULL) || (mhp->msg->sadb_msg_satype == SADB_SATYPE_AH && mhp->ext[SADB_EXT_KEY_AUTH] == NULL) || (mhp->ext[SADB_EXT_LIFETIME_HARD] != NULL && mhp->ext[SADB_EXT_LIFETIME_SOFT] == NULL) || (mhp->ext[SADB_EXT_LIFETIME_HARD] == NULL && mhp->ext[SADB_EXT_LIFETIME_SOFT] != NULL)) { ipseclog((LOG_DEBUG, "%s: invalid message is passed.\n", __func__)); return key_senderror(so, m, EINVAL); } if (mhp->extlen[SADB_EXT_SA] < sizeof(struct sadb_sa) || mhp->extlen[SADB_EXT_ADDRESS_SRC] < sizeof(struct sadb_address) || mhp->extlen[SADB_EXT_ADDRESS_DST] < sizeof(struct sadb_address)) { ipseclog((LOG_DEBUG, "%s: invalid message is passed.\n", __func__)); return key_senderror(so, m, EINVAL); } if (mhp->ext[SADB_X_EXT_SA2] != NULL) { mode = ((struct sadb_x_sa2 *)mhp->ext[SADB_X_EXT_SA2])->sadb_x_sa2_mode; reqid = ((struct sadb_x_sa2 *)mhp->ext[SADB_X_EXT_SA2])->sadb_x_sa2_reqid; } else { mode = IPSEC_MODE_ANY; reqid = 0; } /* XXX boundary checking for other extensions */ sa0 = (struct sadb_sa *)mhp->ext[SADB_EXT_SA]; src0 = (struct sadb_address *)(mhp->ext[SADB_EXT_ADDRESS_SRC]); dst0 = (struct sadb_address *)(mhp->ext[SADB_EXT_ADDRESS_DST]); /* XXX boundary check against sa_len */ KEY_SETSECASIDX(proto, mode, reqid, src0 + 1, dst0 + 1, &saidx); /* * Make sure the port numbers are zero. * In case of NAT-T we will update them later if needed. */ KEY_PORTTOSADDR(&saidx.src, 0); KEY_PORTTOSADDR(&saidx.dst, 0); #ifdef IPSEC_NAT_T /* * Handle NAT-T info if present. */ if (mhp->ext[SADB_X_EXT_NAT_T_TYPE] != NULL && mhp->ext[SADB_X_EXT_NAT_T_SPORT] != NULL && mhp->ext[SADB_X_EXT_NAT_T_DPORT] != NULL) { if (mhp->extlen[SADB_X_EXT_NAT_T_TYPE] < sizeof(*type) || mhp->extlen[SADB_X_EXT_NAT_T_SPORT] < sizeof(*sport) || mhp->extlen[SADB_X_EXT_NAT_T_DPORT] < sizeof(*dport)) { ipseclog((LOG_DEBUG, "%s: invalid message.\n", __func__)); return key_senderror(so, m, EINVAL); } type = (struct sadb_x_nat_t_type *) mhp->ext[SADB_X_EXT_NAT_T_TYPE]; sport = (struct sadb_x_nat_t_port *) mhp->ext[SADB_X_EXT_NAT_T_SPORT]; dport = (struct sadb_x_nat_t_port *) mhp->ext[SADB_X_EXT_NAT_T_DPORT]; } else { type = 0; sport = dport = 0; } if (mhp->ext[SADB_X_EXT_NAT_T_OAI] != NULL && mhp->ext[SADB_X_EXT_NAT_T_OAR] != NULL) { if (mhp->extlen[SADB_X_EXT_NAT_T_OAI] < sizeof(*iaddr) || mhp->extlen[SADB_X_EXT_NAT_T_OAR] < sizeof(*raddr)) { ipseclog((LOG_DEBUG, "%s: invalid message\n", __func__)); return key_senderror(so, m, EINVAL); } iaddr = (struct sadb_address *)mhp->ext[SADB_X_EXT_NAT_T_OAI]; raddr = (struct sadb_address *)mhp->ext[SADB_X_EXT_NAT_T_OAR]; ipseclog((LOG_DEBUG, "%s: NAT-T OAi/r present\n", __func__)); } else { iaddr = raddr = NULL; } if (mhp->ext[SADB_X_EXT_NAT_T_FRAG] != NULL) { if (mhp->extlen[SADB_X_EXT_NAT_T_FRAG] < sizeof(*frag)) { ipseclog((LOG_DEBUG, "%s: invalid message\n", __func__)); return key_senderror(so, m, EINVAL); } frag = (struct sadb_x_nat_t_frag *) mhp->ext[SADB_X_EXT_NAT_T_FRAG]; } else { frag = 0; } #endif /* get a SA header */ if ((sah = key_getsah(&saidx)) == NULL) { ipseclog((LOG_DEBUG, "%s: no SA index found.\n", __func__)); return key_senderror(so, m, ENOENT); } /* set spidx if there */ /* XXX rewrite */ error = key_setident(sah, m, mhp); if (error) return key_senderror(so, m, error); /* find a SA with sequence number. */ #ifdef IPSEC_DOSEQCHECK if (mhp->msg->sadb_msg_seq != 0 && (sav = key_getsavbyseq(sah, mhp->msg->sadb_msg_seq)) == NULL) { ipseclog((LOG_DEBUG, "%s: no larval SA with sequence %u " "exists.\n", __func__, mhp->msg->sadb_msg_seq)); return key_senderror(so, m, ENOENT); } #else SAHTREE_LOCK(); sav = key_getsavbyspi(sah, sa0->sadb_sa_spi); SAHTREE_UNLOCK(); if (sav == NULL) { ipseclog((LOG_DEBUG, "%s: no such a SA found (spi:%u)\n", __func__, (u_int32_t)ntohl(sa0->sadb_sa_spi))); return key_senderror(so, m, EINVAL); } #endif /* validity check */ if (sav->sah->saidx.proto != proto) { ipseclog((LOG_DEBUG, "%s: protocol mismatched " "(DB=%u param=%u)\n", __func__, sav->sah->saidx.proto, proto)); return key_senderror(so, m, EINVAL); } #ifdef IPSEC_DOSEQCHECK if (sav->spi != sa0->sadb_sa_spi) { ipseclog((LOG_DEBUG, "%s: SPI mismatched (DB:%u param:%u)\n", __func__, (u_int32_t)ntohl(sav->spi), (u_int32_t)ntohl(sa0->sadb_sa_spi))); return key_senderror(so, m, EINVAL); } #endif if (sav->pid != mhp->msg->sadb_msg_pid) { ipseclog((LOG_DEBUG, "%s: pid mismatched (DB:%u param:%u)\n", __func__, sav->pid, mhp->msg->sadb_msg_pid)); return key_senderror(so, m, EINVAL); } /* copy sav values */ error = key_setsaval(sav, m, mhp); if (error) { KEY_FREESAV(&sav); return key_senderror(so, m, error); } /* check SA values to be mature. */ if ((mhp->msg->sadb_msg_errno = key_mature(sav)) != 0) { KEY_FREESAV(&sav); return key_senderror(so, m, 0); } #ifdef IPSEC_NAT_T /* * Handle more NAT-T info if present, * now that we have a sav to fill. */ if (type) sav->natt_type = type->sadb_x_nat_t_type_type; if (sport) KEY_PORTTOSADDR(&sav->sah->saidx.src, sport->sadb_x_nat_t_port_port); if (dport) KEY_PORTTOSADDR(&sav->sah->saidx.dst, dport->sadb_x_nat_t_port_port); #if 0 /* * In case SADB_X_EXT_NAT_T_FRAG was not given, leave it at 0. * We should actually check for a minimum MTU here, if we * want to support it in ip_output. */ if (frag) sav->natt_esp_frag_len = frag->sadb_x_nat_t_frag_fraglen; #endif #endif { struct mbuf *n; /* set msg buf from mhp */ n = key_getmsgbuf_x1(m, mhp); if (n == NULL) { ipseclog((LOG_DEBUG, "%s: No more memory.\n", __func__)); return key_senderror(so, m, ENOBUFS); } m_freem(m); return key_sendup_mbuf(so, n, KEY_SENDUP_ALL); } } /* * search SAD with sequence for a SA which state is SADB_SASTATE_LARVAL. * only called by key_update(). * OUT: * NULL : not found * others : found, pointer to a SA. */ #ifdef IPSEC_DOSEQCHECK static struct secasvar * key_getsavbyseq(sah, seq) struct secashead *sah; u_int32_t seq; { struct secasvar *sav; u_int state; state = SADB_SASTATE_LARVAL; /* search SAD with sequence number ? */ LIST_FOREACH(sav, &sah->savtree[state], chain) { KEY_CHKSASTATE(state, sav->state, __func__); if (sav->seq == seq) { sa_addref(sav); KEYDEBUG(KEYDEBUG_IPSEC_STAMP, printf("DP %s cause refcnt++:%d SA:%p\n", __func__, sav->refcnt, sav)); return sav; } } return NULL; } #endif /* * SADB_ADD processing * add an entry to SA database, when received * * from the ikmpd, * and send * * to the ikmpd. * * IGNORE identity and sensitivity messages. * * m will always be freed. */ static int key_add(so, m, mhp) struct socket *so; struct mbuf *m; const struct sadb_msghdr *mhp; { struct sadb_sa *sa0; struct sadb_address *src0, *dst0; #ifdef IPSEC_NAT_T struct sadb_x_nat_t_type *type; struct sadb_address *iaddr, *raddr; struct sadb_x_nat_t_frag *frag; #endif struct secasindex saidx; struct secashead *newsah; struct secasvar *newsav; u_int16_t proto; u_int8_t mode; u_int32_t reqid; int error; IPSEC_ASSERT(so != NULL, ("null socket")); IPSEC_ASSERT(m != NULL, ("null mbuf")); IPSEC_ASSERT(mhp != NULL, ("null msghdr")); IPSEC_ASSERT(mhp->msg != NULL, ("null msg")); /* map satype to proto */ if ((proto = key_satype2proto(mhp->msg->sadb_msg_satype)) == 0) { ipseclog((LOG_DEBUG, "%s: invalid satype is passed.\n", __func__)); return key_senderror(so, m, EINVAL); } if (mhp->ext[SADB_EXT_SA] == NULL || mhp->ext[SADB_EXT_ADDRESS_SRC] == NULL || mhp->ext[SADB_EXT_ADDRESS_DST] == NULL || (mhp->msg->sadb_msg_satype == SADB_SATYPE_ESP && mhp->ext[SADB_EXT_KEY_ENCRYPT] == NULL) || (mhp->msg->sadb_msg_satype == SADB_SATYPE_AH && mhp->ext[SADB_EXT_KEY_AUTH] == NULL) || (mhp->ext[SADB_EXT_LIFETIME_HARD] != NULL && mhp->ext[SADB_EXT_LIFETIME_SOFT] == NULL) || (mhp->ext[SADB_EXT_LIFETIME_HARD] == NULL && mhp->ext[SADB_EXT_LIFETIME_SOFT] != NULL)) { ipseclog((LOG_DEBUG, "%s: invalid message is passed.\n", __func__)); return key_senderror(so, m, EINVAL); } if (mhp->extlen[SADB_EXT_SA] < sizeof(struct sadb_sa) || mhp->extlen[SADB_EXT_ADDRESS_SRC] < sizeof(struct sadb_address) || mhp->extlen[SADB_EXT_ADDRESS_DST] < sizeof(struct sadb_address)) { /* XXX need more */ ipseclog((LOG_DEBUG, "%s: invalid message is passed.\n", __func__)); return key_senderror(so, m, EINVAL); } if (mhp->ext[SADB_X_EXT_SA2] != NULL) { mode = ((struct sadb_x_sa2 *)mhp->ext[SADB_X_EXT_SA2])->sadb_x_sa2_mode; reqid = ((struct sadb_x_sa2 *)mhp->ext[SADB_X_EXT_SA2])->sadb_x_sa2_reqid; } else { mode = IPSEC_MODE_ANY; reqid = 0; } sa0 = (struct sadb_sa *)mhp->ext[SADB_EXT_SA]; src0 = (struct sadb_address *)mhp->ext[SADB_EXT_ADDRESS_SRC]; dst0 = (struct sadb_address *)mhp->ext[SADB_EXT_ADDRESS_DST]; /* XXX boundary check against sa_len */ KEY_SETSECASIDX(proto, mode, reqid, src0 + 1, dst0 + 1, &saidx); /* * Make sure the port numbers are zero. * In case of NAT-T we will update them later if needed. */ KEY_PORTTOSADDR(&saidx.src, 0); KEY_PORTTOSADDR(&saidx.dst, 0); #ifdef IPSEC_NAT_T /* * Handle NAT-T info if present. */ if (mhp->ext[SADB_X_EXT_NAT_T_TYPE] != NULL && mhp->ext[SADB_X_EXT_NAT_T_SPORT] != NULL && mhp->ext[SADB_X_EXT_NAT_T_DPORT] != NULL) { struct sadb_x_nat_t_port *sport, *dport; if (mhp->extlen[SADB_X_EXT_NAT_T_TYPE] < sizeof(*type) || mhp->extlen[SADB_X_EXT_NAT_T_SPORT] < sizeof(*sport) || mhp->extlen[SADB_X_EXT_NAT_T_DPORT] < sizeof(*dport)) { ipseclog((LOG_DEBUG, "%s: invalid message.\n", __func__)); return key_senderror(so, m, EINVAL); } type = (struct sadb_x_nat_t_type *) mhp->ext[SADB_X_EXT_NAT_T_TYPE]; sport = (struct sadb_x_nat_t_port *) mhp->ext[SADB_X_EXT_NAT_T_SPORT]; dport = (struct sadb_x_nat_t_port *) mhp->ext[SADB_X_EXT_NAT_T_DPORT]; if (sport) KEY_PORTTOSADDR(&saidx.src, sport->sadb_x_nat_t_port_port); if (dport) KEY_PORTTOSADDR(&saidx.dst, dport->sadb_x_nat_t_port_port); } else { type = 0; } if (mhp->ext[SADB_X_EXT_NAT_T_OAI] != NULL && mhp->ext[SADB_X_EXT_NAT_T_OAR] != NULL) { if (mhp->extlen[SADB_X_EXT_NAT_T_OAI] < sizeof(*iaddr) || mhp->extlen[SADB_X_EXT_NAT_T_OAR] < sizeof(*raddr)) { ipseclog((LOG_DEBUG, "%s: invalid message\n", __func__)); return key_senderror(so, m, EINVAL); } iaddr = (struct sadb_address *)mhp->ext[SADB_X_EXT_NAT_T_OAI]; raddr = (struct sadb_address *)mhp->ext[SADB_X_EXT_NAT_T_OAR]; ipseclog((LOG_DEBUG, "%s: NAT-T OAi/r present\n", __func__)); } else { iaddr = raddr = NULL; } if (mhp->ext[SADB_X_EXT_NAT_T_FRAG] != NULL) { if (mhp->extlen[SADB_X_EXT_NAT_T_FRAG] < sizeof(*frag)) { ipseclog((LOG_DEBUG, "%s: invalid message\n", __func__)); return key_senderror(so, m, EINVAL); } frag = (struct sadb_x_nat_t_frag *) mhp->ext[SADB_X_EXT_NAT_T_FRAG]; } else { frag = 0; } #endif /* get a SA header */ if ((newsah = key_getsah(&saidx)) == NULL) { /* create a new SA header */ if ((newsah = key_newsah(&saidx)) == NULL) { ipseclog((LOG_DEBUG, "%s: No more memory.\n",__func__)); return key_senderror(so, m, ENOBUFS); } } /* set spidx if there */ /* XXX rewrite */ error = key_setident(newsah, m, mhp); if (error) { return key_senderror(so, m, error); } /* create new SA entry. */ /* We can create new SA only if SPI is differenct. */ SAHTREE_LOCK(); newsav = key_getsavbyspi(newsah, sa0->sadb_sa_spi); SAHTREE_UNLOCK(); if (newsav != NULL) { ipseclog((LOG_DEBUG, "%s: SA already exists.\n", __func__)); return key_senderror(so, m, EEXIST); } newsav = KEY_NEWSAV(m, mhp, newsah, &error); if (newsav == NULL) { return key_senderror(so, m, error); } /* check SA values to be mature. */ if ((error = key_mature(newsav)) != 0) { KEY_FREESAV(&newsav); return key_senderror(so, m, error); } #ifdef IPSEC_NAT_T /* * Handle more NAT-T info if present, * now that we have a sav to fill. */ if (type) newsav->natt_type = type->sadb_x_nat_t_type_type; #if 0 /* * In case SADB_X_EXT_NAT_T_FRAG was not given, leave it at 0. * We should actually check for a minimum MTU here, if we * want to support it in ip_output. */ if (frag) newsav->natt_esp_frag_len = frag->sadb_x_nat_t_frag_fraglen; #endif #endif /* * don't call key_freesav() here, as we would like to keep the SA * in the database on success. */ { struct mbuf *n; /* set msg buf from mhp */ n = key_getmsgbuf_x1(m, mhp); if (n == NULL) { ipseclog((LOG_DEBUG, "%s: No more memory.\n", __func__)); return key_senderror(so, m, ENOBUFS); } m_freem(m); return key_sendup_mbuf(so, n, KEY_SENDUP_ALL); } } /* m is retained */ static int key_setident(sah, m, mhp) struct secashead *sah; struct mbuf *m; const struct sadb_msghdr *mhp; { const struct sadb_ident *idsrc, *iddst; int idsrclen, iddstlen; IPSEC_ASSERT(sah != NULL, ("null secashead")); IPSEC_ASSERT(m != NULL, ("null mbuf")); IPSEC_ASSERT(mhp != NULL, ("null msghdr")); IPSEC_ASSERT(mhp->msg != NULL, ("null msg")); /* don't make buffer if not there */ if (mhp->ext[SADB_EXT_IDENTITY_SRC] == NULL && mhp->ext[SADB_EXT_IDENTITY_DST] == NULL) { sah->idents = NULL; sah->identd = NULL; return 0; } if (mhp->ext[SADB_EXT_IDENTITY_SRC] == NULL || mhp->ext[SADB_EXT_IDENTITY_DST] == NULL) { ipseclog((LOG_DEBUG, "%s: invalid identity.\n", __func__)); return EINVAL; } idsrc = (const struct sadb_ident *)mhp->ext[SADB_EXT_IDENTITY_SRC]; iddst = (const struct sadb_ident *)mhp->ext[SADB_EXT_IDENTITY_DST]; idsrclen = mhp->extlen[SADB_EXT_IDENTITY_SRC]; iddstlen = mhp->extlen[SADB_EXT_IDENTITY_DST]; /* validity check */ if (idsrc->sadb_ident_type != iddst->sadb_ident_type) { ipseclog((LOG_DEBUG, "%s: ident type mismatch.\n", __func__)); return EINVAL; } switch (idsrc->sadb_ident_type) { case SADB_IDENTTYPE_PREFIX: case SADB_IDENTTYPE_FQDN: case SADB_IDENTTYPE_USERFQDN: default: /* XXX do nothing */ sah->idents = NULL; sah->identd = NULL; return 0; } /* make structure */ sah->idents = malloc(sizeof(struct secident), M_IPSEC_MISC, M_NOWAIT); if (sah->idents == NULL) { ipseclog((LOG_DEBUG, "%s: No more memory.\n", __func__)); return ENOBUFS; } sah->identd = malloc(sizeof(struct secident), M_IPSEC_MISC, M_NOWAIT); if (sah->identd == NULL) { free(sah->idents, M_IPSEC_MISC); sah->idents = NULL; ipseclog((LOG_DEBUG, "%s: No more memory.\n", __func__)); return ENOBUFS; } sah->idents->type = idsrc->sadb_ident_type; sah->idents->id = idsrc->sadb_ident_id; sah->identd->type = iddst->sadb_ident_type; sah->identd->id = iddst->sadb_ident_id; return 0; } /* * m will not be freed on return. * it is caller's responsibility to free the result. */ static struct mbuf * key_getmsgbuf_x1(m, mhp) struct mbuf *m; const struct sadb_msghdr *mhp; { struct mbuf *n; IPSEC_ASSERT(m != NULL, ("null mbuf")); IPSEC_ASSERT(mhp != NULL, ("null msghdr")); IPSEC_ASSERT(mhp->msg != NULL, ("null msg")); /* create new sadb_msg to reply. */ n = key_gather_mbuf(m, mhp, 1, 9, SADB_EXT_RESERVED, SADB_EXT_SA, SADB_X_EXT_SA2, SADB_EXT_ADDRESS_SRC, SADB_EXT_ADDRESS_DST, SADB_EXT_LIFETIME_HARD, SADB_EXT_LIFETIME_SOFT, SADB_EXT_IDENTITY_SRC, SADB_EXT_IDENTITY_DST); if (!n) return NULL; if (n->m_len < sizeof(struct sadb_msg)) { n = m_pullup(n, sizeof(struct sadb_msg)); if (n == NULL) return NULL; } mtod(n, struct sadb_msg *)->sadb_msg_errno = 0; mtod(n, struct sadb_msg *)->sadb_msg_len = PFKEY_UNIT64(n->m_pkthdr.len); return n; } static int key_delete_all __P((struct socket *, struct mbuf *, const struct sadb_msghdr *, u_int16_t)); /* * SADB_DELETE processing * receive * * from the ikmpd, and set SADB_SASTATE_DEAD, * and send, * * to the ikmpd. * * m will always be freed. */ static int key_delete(so, m, mhp) struct socket *so; struct mbuf *m; const struct sadb_msghdr *mhp; { struct sadb_sa *sa0; struct sadb_address *src0, *dst0; struct secasindex saidx; struct secashead *sah; struct secasvar *sav = NULL; u_int16_t proto; IPSEC_ASSERT(so != NULL, ("null socket")); IPSEC_ASSERT(m != NULL, ("null mbuf")); IPSEC_ASSERT(mhp != NULL, ("null msghdr")); IPSEC_ASSERT(mhp->msg != NULL, ("null msg")); /* map satype to proto */ if ((proto = key_satype2proto(mhp->msg->sadb_msg_satype)) == 0) { ipseclog((LOG_DEBUG, "%s: invalid satype is passed.\n", __func__)); return key_senderror(so, m, EINVAL); } if (mhp->ext[SADB_EXT_ADDRESS_SRC] == NULL || mhp->ext[SADB_EXT_ADDRESS_DST] == NULL) { ipseclog((LOG_DEBUG, "%s: invalid message is passed.\n", __func__)); return key_senderror(so, m, EINVAL); } if (mhp->extlen[SADB_EXT_ADDRESS_SRC] < sizeof(struct sadb_address) || mhp->extlen[SADB_EXT_ADDRESS_DST] < sizeof(struct sadb_address)) { ipseclog((LOG_DEBUG, "%s: invalid message is passed.\n", __func__)); return key_senderror(so, m, EINVAL); } if (mhp->ext[SADB_EXT_SA] == NULL) { /* * Caller wants us to delete all non-LARVAL SAs * that match the src/dst. This is used during * IKE INITIAL-CONTACT. */ ipseclog((LOG_DEBUG, "%s: doing delete all.\n", __func__)); return key_delete_all(so, m, mhp, proto); } else if (mhp->extlen[SADB_EXT_SA] < sizeof(struct sadb_sa)) { ipseclog((LOG_DEBUG, "%s: invalid message is passed.\n", __func__)); return key_senderror(so, m, EINVAL); } sa0 = (struct sadb_sa *)mhp->ext[SADB_EXT_SA]; src0 = (struct sadb_address *)(mhp->ext[SADB_EXT_ADDRESS_SRC]); dst0 = (struct sadb_address *)(mhp->ext[SADB_EXT_ADDRESS_DST]); /* XXX boundary check against sa_len */ KEY_SETSECASIDX(proto, IPSEC_MODE_ANY, 0, src0 + 1, dst0 + 1, &saidx); /* * Make sure the port numbers are zero. * In case of NAT-T we will update them later if needed. */ KEY_PORTTOSADDR(&saidx.src, 0); KEY_PORTTOSADDR(&saidx.dst, 0); #ifdef IPSEC_NAT_T /* * Handle NAT-T info if present. */ if (mhp->ext[SADB_X_EXT_NAT_T_SPORT] != NULL && mhp->ext[SADB_X_EXT_NAT_T_DPORT] != NULL) { struct sadb_x_nat_t_port *sport, *dport; if (mhp->extlen[SADB_X_EXT_NAT_T_SPORT] < sizeof(*sport) || mhp->extlen[SADB_X_EXT_NAT_T_DPORT] < sizeof(*dport)) { ipseclog((LOG_DEBUG, "%s: invalid message.\n", __func__)); return key_senderror(so, m, EINVAL); } sport = (struct sadb_x_nat_t_port *) mhp->ext[SADB_X_EXT_NAT_T_SPORT]; dport = (struct sadb_x_nat_t_port *) mhp->ext[SADB_X_EXT_NAT_T_DPORT]; if (sport) KEY_PORTTOSADDR(&saidx.src, sport->sadb_x_nat_t_port_port); if (dport) KEY_PORTTOSADDR(&saidx.dst, dport->sadb_x_nat_t_port_port); } #endif /* get a SA header */ SAHTREE_LOCK(); LIST_FOREACH(sah, &V_sahtree, chain) { if (sah->state == SADB_SASTATE_DEAD) continue; if (key_cmpsaidx(&sah->saidx, &saidx, CMP_HEAD) == 0) continue; /* get a SA with SPI. */ sav = key_getsavbyspi(sah, sa0->sadb_sa_spi); if (sav) break; } if (sah == NULL) { SAHTREE_UNLOCK(); ipseclog((LOG_DEBUG, "%s: no SA found.\n", __func__)); return key_senderror(so, m, ENOENT); } key_sa_chgstate(sav, SADB_SASTATE_DEAD); KEY_FREESAV(&sav); SAHTREE_UNLOCK(); { struct mbuf *n; struct sadb_msg *newmsg; /* create new sadb_msg to reply. */ /* XXX-BZ NAT-T extensions? */ n = key_gather_mbuf(m, mhp, 1, 4, SADB_EXT_RESERVED, SADB_EXT_SA, SADB_EXT_ADDRESS_SRC, SADB_EXT_ADDRESS_DST); if (!n) return key_senderror(so, m, ENOBUFS); if (n->m_len < sizeof(struct sadb_msg)) { n = m_pullup(n, sizeof(struct sadb_msg)); if (n == NULL) return key_senderror(so, m, ENOBUFS); } newmsg = mtod(n, struct sadb_msg *); newmsg->sadb_msg_errno = 0; newmsg->sadb_msg_len = PFKEY_UNIT64(n->m_pkthdr.len); m_freem(m); return key_sendup_mbuf(so, n, KEY_SENDUP_ALL); } } /* * delete all SAs for src/dst. Called from key_delete(). */ static int key_delete_all(struct socket *so, struct mbuf *m, const struct sadb_msghdr *mhp, u_int16_t proto) { struct sadb_address *src0, *dst0; struct secasindex saidx; struct secashead *sah; struct secasvar *sav, *nextsav; u_int stateidx, state; src0 = (struct sadb_address *)(mhp->ext[SADB_EXT_ADDRESS_SRC]); dst0 = (struct sadb_address *)(mhp->ext[SADB_EXT_ADDRESS_DST]); /* XXX boundary check against sa_len */ KEY_SETSECASIDX(proto, IPSEC_MODE_ANY, 0, src0 + 1, dst0 + 1, &saidx); /* * Make sure the port numbers are zero. * In case of NAT-T we will update them later if needed. */ KEY_PORTTOSADDR(&saidx.src, 0); KEY_PORTTOSADDR(&saidx.dst, 0); #ifdef IPSEC_NAT_T /* * Handle NAT-T info if present. */ if (mhp->ext[SADB_X_EXT_NAT_T_SPORT] != NULL && mhp->ext[SADB_X_EXT_NAT_T_DPORT] != NULL) { struct sadb_x_nat_t_port *sport, *dport; if (mhp->extlen[SADB_X_EXT_NAT_T_SPORT] < sizeof(*sport) || mhp->extlen[SADB_X_EXT_NAT_T_DPORT] < sizeof(*dport)) { ipseclog((LOG_DEBUG, "%s: invalid message.\n", __func__)); return key_senderror(so, m, EINVAL); } sport = (struct sadb_x_nat_t_port *) mhp->ext[SADB_X_EXT_NAT_T_SPORT]; dport = (struct sadb_x_nat_t_port *) mhp->ext[SADB_X_EXT_NAT_T_DPORT]; if (sport) KEY_PORTTOSADDR(&saidx.src, sport->sadb_x_nat_t_port_port); if (dport) KEY_PORTTOSADDR(&saidx.dst, dport->sadb_x_nat_t_port_port); } #endif SAHTREE_LOCK(); LIST_FOREACH(sah, &V_sahtree, chain) { if (sah->state == SADB_SASTATE_DEAD) continue; if (key_cmpsaidx(&sah->saidx, &saidx, CMP_HEAD) == 0) continue; /* Delete all non-LARVAL SAs. */ for (stateidx = 0; stateidx < _ARRAYLEN(saorder_state_alive); stateidx++) { state = saorder_state_alive[stateidx]; if (state == SADB_SASTATE_LARVAL) continue; for (sav = LIST_FIRST(&sah->savtree[state]); sav != NULL; sav = nextsav) { nextsav = LIST_NEXT(sav, chain); /* sanity check */ if (sav->state != state) { ipseclog((LOG_DEBUG, "%s: invalid " "sav->state (queue %d SA %d)\n", __func__, state, sav->state)); continue; } key_sa_chgstate(sav, SADB_SASTATE_DEAD); KEY_FREESAV(&sav); } } } SAHTREE_UNLOCK(); { struct mbuf *n; struct sadb_msg *newmsg; /* create new sadb_msg to reply. */ /* XXX-BZ NAT-T extensions? */ n = key_gather_mbuf(m, mhp, 1, 3, SADB_EXT_RESERVED, SADB_EXT_ADDRESS_SRC, SADB_EXT_ADDRESS_DST); if (!n) return key_senderror(so, m, ENOBUFS); if (n->m_len < sizeof(struct sadb_msg)) { n = m_pullup(n, sizeof(struct sadb_msg)); if (n == NULL) return key_senderror(so, m, ENOBUFS); } newmsg = mtod(n, struct sadb_msg *); newmsg->sadb_msg_errno = 0; newmsg->sadb_msg_len = PFKEY_UNIT64(n->m_pkthdr.len); m_freem(m); return key_sendup_mbuf(so, n, KEY_SENDUP_ALL); } } /* * SADB_GET processing * receive * * from the ikmpd, and get a SP and a SA to respond, * and send, * * to the ikmpd. * * m will always be freed. */ static int key_get(so, m, mhp) struct socket *so; struct mbuf *m; const struct sadb_msghdr *mhp; { struct sadb_sa *sa0; struct sadb_address *src0, *dst0; struct secasindex saidx; struct secashead *sah; struct secasvar *sav = NULL; u_int16_t proto; IPSEC_ASSERT(so != NULL, ("null socket")); IPSEC_ASSERT(m != NULL, ("null mbuf")); IPSEC_ASSERT(mhp != NULL, ("null msghdr")); IPSEC_ASSERT(mhp->msg != NULL, ("null msg")); /* map satype to proto */ if ((proto = key_satype2proto(mhp->msg->sadb_msg_satype)) == 0) { ipseclog((LOG_DEBUG, "%s: invalid satype is passed.\n", __func__)); return key_senderror(so, m, EINVAL); } if (mhp->ext[SADB_EXT_SA] == NULL || mhp->ext[SADB_EXT_ADDRESS_SRC] == NULL || mhp->ext[SADB_EXT_ADDRESS_DST] == NULL) { ipseclog((LOG_DEBUG, "%s: invalid message is passed.\n", __func__)); return key_senderror(so, m, EINVAL); } if (mhp->extlen[SADB_EXT_SA] < sizeof(struct sadb_sa) || mhp->extlen[SADB_EXT_ADDRESS_SRC] < sizeof(struct sadb_address) || mhp->extlen[SADB_EXT_ADDRESS_DST] < sizeof(struct sadb_address)) { ipseclog((LOG_DEBUG, "%s: invalid message is passed.\n", __func__)); return key_senderror(so, m, EINVAL); } sa0 = (struct sadb_sa *)mhp->ext[SADB_EXT_SA]; src0 = (struct sadb_address *)mhp->ext[SADB_EXT_ADDRESS_SRC]; dst0 = (struct sadb_address *)mhp->ext[SADB_EXT_ADDRESS_DST]; /* XXX boundary check against sa_len */ KEY_SETSECASIDX(proto, IPSEC_MODE_ANY, 0, src0 + 1, dst0 + 1, &saidx); /* * Make sure the port numbers are zero. * In case of NAT-T we will update them later if needed. */ KEY_PORTTOSADDR(&saidx.src, 0); KEY_PORTTOSADDR(&saidx.dst, 0); #ifdef IPSEC_NAT_T /* * Handle NAT-T info if present. */ if (mhp->ext[SADB_X_EXT_NAT_T_SPORT] != NULL && mhp->ext[SADB_X_EXT_NAT_T_DPORT] != NULL) { struct sadb_x_nat_t_port *sport, *dport; if (mhp->extlen[SADB_X_EXT_NAT_T_SPORT] < sizeof(*sport) || mhp->extlen[SADB_X_EXT_NAT_T_DPORT] < sizeof(*dport)) { ipseclog((LOG_DEBUG, "%s: invalid message.\n", __func__)); return key_senderror(so, m, EINVAL); } sport = (struct sadb_x_nat_t_port *) mhp->ext[SADB_X_EXT_NAT_T_SPORT]; dport = (struct sadb_x_nat_t_port *) mhp->ext[SADB_X_EXT_NAT_T_DPORT]; if (sport) KEY_PORTTOSADDR(&saidx.src, sport->sadb_x_nat_t_port_port); if (dport) KEY_PORTTOSADDR(&saidx.dst, dport->sadb_x_nat_t_port_port); } #endif /* get a SA header */ SAHTREE_LOCK(); LIST_FOREACH(sah, &V_sahtree, chain) { if (sah->state == SADB_SASTATE_DEAD) continue; if (key_cmpsaidx(&sah->saidx, &saidx, CMP_HEAD) == 0) continue; /* get a SA with SPI. */ sav = key_getsavbyspi(sah, sa0->sadb_sa_spi); if (sav) break; } SAHTREE_UNLOCK(); if (sah == NULL) { ipseclog((LOG_DEBUG, "%s: no SA found.\n", __func__)); return key_senderror(so, m, ENOENT); } { struct mbuf *n; u_int8_t satype; /* map proto to satype */ if ((satype = key_proto2satype(sah->saidx.proto)) == 0) { ipseclog((LOG_DEBUG, "%s: there was invalid proto in SAD.\n", __func__)); return key_senderror(so, m, EINVAL); } /* create new sadb_msg to reply. */ n = key_setdumpsa(sav, SADB_GET, satype, mhp->msg->sadb_msg_seq, mhp->msg->sadb_msg_pid); if (!n) return key_senderror(so, m, ENOBUFS); m_freem(m); return key_sendup_mbuf(so, n, KEY_SENDUP_ONE); } } /* XXX make it sysctl-configurable? */ static void key_getcomb_setlifetime(comb) struct sadb_comb *comb; { comb->sadb_comb_soft_allocations = 1; comb->sadb_comb_hard_allocations = 1; comb->sadb_comb_soft_bytes = 0; comb->sadb_comb_hard_bytes = 0; comb->sadb_comb_hard_addtime = 86400; /* 1 day */ comb->sadb_comb_soft_addtime = comb->sadb_comb_soft_addtime * 80 / 100; comb->sadb_comb_soft_usetime = 28800; /* 8 hours */ comb->sadb_comb_hard_usetime = comb->sadb_comb_hard_usetime * 80 / 100; } /* * XXX reorder combinations by preference * XXX no idea if the user wants ESP authentication or not */ static struct mbuf * key_getcomb_esp() { struct sadb_comb *comb; struct enc_xform *algo; struct mbuf *result = NULL, *m, *n; int encmin; int i, off, o; int totlen; const int l = PFKEY_ALIGN8(sizeof(struct sadb_comb)); m = NULL; for (i = 1; i <= SADB_EALG_MAX; i++) { algo = esp_algorithm_lookup(i); if (algo == NULL) continue; /* discard algorithms with key size smaller than system min */ if (_BITS(algo->maxkey) < V_ipsec_esp_keymin) continue; if (_BITS(algo->minkey) < V_ipsec_esp_keymin) encmin = V_ipsec_esp_keymin; else encmin = _BITS(algo->minkey); if (V_ipsec_esp_auth) m = key_getcomb_ah(); else { IPSEC_ASSERT(l <= MLEN, ("l=%u > MLEN=%lu", l, (u_long) MLEN)); MGET(m, M_DONTWAIT, MT_DATA); if (m) { M_ALIGN(m, l); m->m_len = l; m->m_next = NULL; bzero(mtod(m, caddr_t), m->m_len); } } if (!m) goto fail; totlen = 0; for (n = m; n; n = n->m_next) totlen += n->m_len; IPSEC_ASSERT((totlen % l) == 0, ("totlen=%u, l=%u", totlen, l)); for (off = 0; off < totlen; off += l) { n = m_pulldown(m, off, l, &o); if (!n) { /* m is already freed */ goto fail; } comb = (struct sadb_comb *)(mtod(n, caddr_t) + o); bzero(comb, sizeof(*comb)); key_getcomb_setlifetime(comb); comb->sadb_comb_encrypt = i; comb->sadb_comb_encrypt_minbits = encmin; comb->sadb_comb_encrypt_maxbits = _BITS(algo->maxkey); } if (!result) result = m; else m_cat(result, m); } return result; fail: if (result) m_freem(result); return NULL; } static void key_getsizes_ah( const struct auth_hash *ah, int alg, u_int16_t* min, u_int16_t* max) { *min = *max = ah->keysize; if (ah->keysize == 0) { /* * Transform takes arbitrary key size but algorithm * key size is restricted. Enforce this here. */ switch (alg) { case SADB_X_AALG_MD5: *min = *max = 16; break; case SADB_X_AALG_SHA: *min = *max = 20; break; case SADB_X_AALG_NULL: *min = 1; *max = 256; break; default: DPRINTF(("%s: unknown AH algorithm %u\n", __func__, alg)); break; } } } /* * XXX reorder combinations by preference */ static struct mbuf * key_getcomb_ah() { struct sadb_comb *comb; struct auth_hash *algo; struct mbuf *m; u_int16_t minkeysize, maxkeysize; int i; const int l = PFKEY_ALIGN8(sizeof(struct sadb_comb)); m = NULL; for (i = 1; i <= SADB_AALG_MAX; i++) { #if 1 /* we prefer HMAC algorithms, not old algorithms */ if (i != SADB_AALG_SHA1HMAC && i != SADB_AALG_MD5HMAC) continue; #endif algo = ah_algorithm_lookup(i); if (!algo) continue; key_getsizes_ah(algo, i, &minkeysize, &maxkeysize); /* discard algorithms with key size smaller than system min */ if (_BITS(minkeysize) < V_ipsec_ah_keymin) continue; if (!m) { IPSEC_ASSERT(l <= MLEN, ("l=%u > MLEN=%lu", l, (u_long) MLEN)); MGET(m, M_DONTWAIT, MT_DATA); if (m) { M_ALIGN(m, l); m->m_len = l; m->m_next = NULL; } } else M_PREPEND(m, l, M_DONTWAIT); if (!m) return NULL; comb = mtod(m, struct sadb_comb *); bzero(comb, sizeof(*comb)); key_getcomb_setlifetime(comb); comb->sadb_comb_auth = i; comb->sadb_comb_auth_minbits = _BITS(minkeysize); comb->sadb_comb_auth_maxbits = _BITS(maxkeysize); } return m; } /* * not really an official behavior. discussed in pf_key@inner.net in Sep2000. * XXX reorder combinations by preference */ static struct mbuf * key_getcomb_ipcomp() { struct sadb_comb *comb; struct comp_algo *algo; struct mbuf *m; int i; const int l = PFKEY_ALIGN8(sizeof(struct sadb_comb)); m = NULL; for (i = 1; i <= SADB_X_CALG_MAX; i++) { algo = ipcomp_algorithm_lookup(i); if (!algo) continue; if (!m) { IPSEC_ASSERT(l <= MLEN, ("l=%u > MLEN=%lu", l, (u_long) MLEN)); MGET(m, M_DONTWAIT, MT_DATA); if (m) { M_ALIGN(m, l); m->m_len = l; m->m_next = NULL; } } else M_PREPEND(m, l, M_DONTWAIT); if (!m) return NULL; comb = mtod(m, struct sadb_comb *); bzero(comb, sizeof(*comb)); key_getcomb_setlifetime(comb); comb->sadb_comb_encrypt = i; /* what should we set into sadb_comb_*_{min,max}bits? */ } return m; } /* * XXX no way to pass mode (transport/tunnel) to userland * XXX replay checking? * XXX sysctl interface to ipsec_{ah,esp}_keymin */ static struct mbuf * key_getprop(saidx) const struct secasindex *saidx; { struct sadb_prop *prop; struct mbuf *m, *n; const int l = PFKEY_ALIGN8(sizeof(struct sadb_prop)); int totlen; switch (saidx->proto) { case IPPROTO_ESP: m = key_getcomb_esp(); break; case IPPROTO_AH: m = key_getcomb_ah(); break; case IPPROTO_IPCOMP: m = key_getcomb_ipcomp(); break; default: return NULL; } if (!m) return NULL; M_PREPEND(m, l, M_DONTWAIT); if (!m) return NULL; totlen = 0; for (n = m; n; n = n->m_next) totlen += n->m_len; prop = mtod(m, struct sadb_prop *); bzero(prop, sizeof(*prop)); prop->sadb_prop_len = PFKEY_UNIT64(totlen); prop->sadb_prop_exttype = SADB_EXT_PROPOSAL; prop->sadb_prop_replay = 32; /* XXX */ return m; } /* * SADB_ACQUIRE processing called by key_checkrequest() and key_acquire2(). * send * * to KMD, and expect to receive * with SADB_ACQUIRE if error occured, * or * with SADB_GETSPI * from KMD by PF_KEY. * * XXX x_policy is outside of RFC2367 (KAME extension). * XXX sensitivity is not supported. * XXX for ipcomp, RFC2367 does not define how to fill in proposal. * see comment for key_getcomb_ipcomp(). * * OUT: * 0 : succeed * others: error number */ static int key_acquire(const struct secasindex *saidx, struct secpolicy *sp) { struct mbuf *result = NULL, *m; struct secacq *newacq; u_int8_t satype; int error = -1; u_int32_t seq; IPSEC_ASSERT(saidx != NULL, ("null saidx")); satype = key_proto2satype(saidx->proto); IPSEC_ASSERT(satype != 0, ("null satype, protocol %u", saidx->proto)); /* * We never do anything about acquirng SA. There is anather * solution that kernel blocks to send SADB_ACQUIRE message until * getting something message from IKEd. In later case, to be * managed with ACQUIRING list. */ /* Get an entry to check whether sending message or not. */ if ((newacq = key_getacq(saidx)) != NULL) { if (V_key_blockacq_count < newacq->count) { /* reset counter and do send message. */ newacq->count = 0; } else { /* increment counter and do nothing. */ newacq->count++; return 0; } } else { /* make new entry for blocking to send SADB_ACQUIRE. */ if ((newacq = key_newacq(saidx)) == NULL) return ENOBUFS; } seq = newacq->seq; m = key_setsadbmsg(SADB_ACQUIRE, 0, satype, seq, 0, 0); if (!m) { error = ENOBUFS; goto fail; } result = m; /* * No SADB_X_EXT_NAT_T_* here: we do not know * anything related to NAT-T at this time. */ /* set sadb_address for saidx's. */ m = key_setsadbaddr(SADB_EXT_ADDRESS_SRC, &saidx->src.sa, FULLMASK, IPSEC_ULPROTO_ANY); if (!m) { error = ENOBUFS; goto fail; } m_cat(result, m); m = key_setsadbaddr(SADB_EXT_ADDRESS_DST, &saidx->dst.sa, FULLMASK, IPSEC_ULPROTO_ANY); if (!m) { error = ENOBUFS; goto fail; } m_cat(result, m); /* XXX proxy address (optional) */ /* set sadb_x_policy */ if (sp) { m = key_setsadbxpolicy(sp->policy, sp->spidx.dir, sp->id); if (!m) { error = ENOBUFS; goto fail; } m_cat(result, m); } /* XXX identity (optional) */ #if 0 if (idexttype && fqdn) { /* create identity extension (FQDN) */ struct sadb_ident *id; int fqdnlen; fqdnlen = strlen(fqdn) + 1; /* +1 for terminating-NUL */ id = (struct sadb_ident *)p; bzero(id, sizeof(*id) + PFKEY_ALIGN8(fqdnlen)); id->sadb_ident_len = PFKEY_UNIT64(sizeof(*id) + PFKEY_ALIGN8(fqdnlen)); id->sadb_ident_exttype = idexttype; id->sadb_ident_type = SADB_IDENTTYPE_FQDN; bcopy(fqdn, id + 1, fqdnlen); p += sizeof(struct sadb_ident) + PFKEY_ALIGN8(fqdnlen); } if (idexttype) { /* create identity extension (USERFQDN) */ struct sadb_ident *id; int userfqdnlen; if (userfqdn) { /* +1 for terminating-NUL */ userfqdnlen = strlen(userfqdn) + 1; } else userfqdnlen = 0; id = (struct sadb_ident *)p; bzero(id, sizeof(*id) + PFKEY_ALIGN8(userfqdnlen)); id->sadb_ident_len = PFKEY_UNIT64(sizeof(*id) + PFKEY_ALIGN8(userfqdnlen)); id->sadb_ident_exttype = idexttype; id->sadb_ident_type = SADB_IDENTTYPE_USERFQDN; /* XXX is it correct? */ if (curproc && curproc->p_cred) id->sadb_ident_id = curproc->p_cred->p_ruid; if (userfqdn && userfqdnlen) bcopy(userfqdn, id + 1, userfqdnlen); p += sizeof(struct sadb_ident) + PFKEY_ALIGN8(userfqdnlen); } #endif /* XXX sensitivity (optional) */ /* create proposal/combination extension */ m = key_getprop(saidx); #if 0 /* * spec conformant: always attach proposal/combination extension, * the problem is that we have no way to attach it for ipcomp, * due to the way sadb_comb is declared in RFC2367. */ if (!m) { error = ENOBUFS; goto fail; } m_cat(result, m); #else /* * outside of spec; make proposal/combination extension optional. */ if (m) m_cat(result, m); #endif if ((result->m_flags & M_PKTHDR) == 0) { error = EINVAL; goto fail; } if (result->m_len < sizeof(struct sadb_msg)) { result = m_pullup(result, sizeof(struct sadb_msg)); if (result == NULL) { error = ENOBUFS; goto fail; } } result->m_pkthdr.len = 0; for (m = result; m; m = m->m_next) result->m_pkthdr.len += m->m_len; mtod(result, struct sadb_msg *)->sadb_msg_len = PFKEY_UNIT64(result->m_pkthdr.len); return key_sendup_mbuf(NULL, result, KEY_SENDUP_REGISTERED); fail: if (result) m_freem(result); return error; } static struct secacq * key_newacq(const struct secasindex *saidx) { struct secacq *newacq; /* get new entry */ newacq = malloc(sizeof(struct secacq), M_IPSEC_SAQ, M_NOWAIT|M_ZERO); if (newacq == NULL) { ipseclog((LOG_DEBUG, "%s: No more memory.\n", __func__)); return NULL; } /* copy secindex */ bcopy(saidx, &newacq->saidx, sizeof(newacq->saidx)); newacq->seq = (V_acq_seq == ~0 ? 1 : ++V_acq_seq); newacq->created = time_second; newacq->count = 0; /* add to acqtree */ ACQ_LOCK(); LIST_INSERT_HEAD(&V_acqtree, newacq, chain); ACQ_UNLOCK(); return newacq; } static struct secacq * key_getacq(const struct secasindex *saidx) { struct secacq *acq; ACQ_LOCK(); LIST_FOREACH(acq, &V_acqtree, chain) { if (key_cmpsaidx(saidx, &acq->saidx, CMP_EXACTLY)) break; } ACQ_UNLOCK(); return acq; } static struct secacq * key_getacqbyseq(seq) u_int32_t seq; { struct secacq *acq; ACQ_LOCK(); LIST_FOREACH(acq, &V_acqtree, chain) { if (acq->seq == seq) break; } ACQ_UNLOCK(); return acq; } static struct secspacq * key_newspacq(spidx) struct secpolicyindex *spidx; { struct secspacq *acq; /* get new entry */ acq = malloc(sizeof(struct secspacq), M_IPSEC_SAQ, M_NOWAIT|M_ZERO); if (acq == NULL) { ipseclog((LOG_DEBUG, "%s: No more memory.\n", __func__)); return NULL; } /* copy secindex */ bcopy(spidx, &acq->spidx, sizeof(acq->spidx)); acq->created = time_second; acq->count = 0; /* add to spacqtree */ SPACQ_LOCK(); LIST_INSERT_HEAD(&V_spacqtree, acq, chain); SPACQ_UNLOCK(); return acq; } static struct secspacq * key_getspacq(spidx) struct secpolicyindex *spidx; { struct secspacq *acq; SPACQ_LOCK(); LIST_FOREACH(acq, &V_spacqtree, chain) { if (key_cmpspidx_exactly(spidx, &acq->spidx)) { /* NB: return holding spacq_lock */ return acq; } } SPACQ_UNLOCK(); return NULL; } /* * SADB_ACQUIRE processing, * in first situation, is receiving * * from the ikmpd, and clear sequence of its secasvar entry. * * In second situation, is receiving * * from a user land process, and return * * to the socket. * * m will always be freed. */ static int key_acquire2(so, m, mhp) struct socket *so; struct mbuf *m; const struct sadb_msghdr *mhp; { const struct sadb_address *src0, *dst0; struct secasindex saidx; struct secashead *sah; u_int16_t proto; int error; IPSEC_ASSERT(so != NULL, ("null socket")); IPSEC_ASSERT(m != NULL, ("null mbuf")); IPSEC_ASSERT(mhp != NULL, ("null msghdr")); IPSEC_ASSERT(mhp->msg != NULL, ("null msg")); /* * Error message from KMd. * We assume that if error was occured in IKEd, the length of PFKEY * message is equal to the size of sadb_msg structure. * We do not raise error even if error occured in this function. */ if (mhp->msg->sadb_msg_len == PFKEY_UNIT64(sizeof(struct sadb_msg))) { struct secacq *acq; /* check sequence number */ if (mhp->msg->sadb_msg_seq == 0) { ipseclog((LOG_DEBUG, "%s: must specify sequence " "number.\n", __func__)); m_freem(m); return 0; } if ((acq = key_getacqbyseq(mhp->msg->sadb_msg_seq)) == NULL) { /* * the specified larval SA is already gone, or we got * a bogus sequence number. we can silently ignore it. */ m_freem(m); return 0; } /* reset acq counter in order to deletion by timehander. */ acq->created = time_second; acq->count = 0; m_freem(m); return 0; } /* * This message is from user land. */ /* map satype to proto */ if ((proto = key_satype2proto(mhp->msg->sadb_msg_satype)) == 0) { ipseclog((LOG_DEBUG, "%s: invalid satype is passed.\n", __func__)); return key_senderror(so, m, EINVAL); } if (mhp->ext[SADB_EXT_ADDRESS_SRC] == NULL || mhp->ext[SADB_EXT_ADDRESS_DST] == NULL || mhp->ext[SADB_EXT_PROPOSAL] == NULL) { /* error */ ipseclog((LOG_DEBUG, "%s: invalid message is passed.\n", __func__)); return key_senderror(so, m, EINVAL); } if (mhp->extlen[SADB_EXT_ADDRESS_SRC] < sizeof(struct sadb_address) || mhp->extlen[SADB_EXT_ADDRESS_DST] < sizeof(struct sadb_address) || mhp->extlen[SADB_EXT_PROPOSAL] < sizeof(struct sadb_prop)) { /* error */ ipseclog((LOG_DEBUG, "%s: invalid message is passed.\n", __func__)); return key_senderror(so, m, EINVAL); } src0 = (struct sadb_address *)mhp->ext[SADB_EXT_ADDRESS_SRC]; dst0 = (struct sadb_address *)mhp->ext[SADB_EXT_ADDRESS_DST]; /* XXX boundary check against sa_len */ KEY_SETSECASIDX(proto, IPSEC_MODE_ANY, 0, src0 + 1, dst0 + 1, &saidx); /* * Make sure the port numbers are zero. * In case of NAT-T we will update them later if needed. */ KEY_PORTTOSADDR(&saidx.src, 0); KEY_PORTTOSADDR(&saidx.dst, 0); #ifndef IPSEC_NAT_T /* * Handle NAT-T info if present. */ if (mhp->ext[SADB_X_EXT_NAT_T_SPORT] != NULL && mhp->ext[SADB_X_EXT_NAT_T_DPORT] != NULL) { struct sadb_x_nat_t_port *sport, *dport; if (mhp->extlen[SADB_X_EXT_NAT_T_SPORT] < sizeof(*sport) || mhp->extlen[SADB_X_EXT_NAT_T_DPORT] < sizeof(*dport)) { ipseclog((LOG_DEBUG, "%s: invalid message.\n", __func__)); return key_senderror(so, m, EINVAL); } sport = (struct sadb_x_nat_t_port *) mhp->ext[SADB_X_EXT_NAT_T_SPORT]; dport = (struct sadb_x_nat_t_port *) mhp->ext[SADB_X_EXT_NAT_T_DPORT]; if (sport) KEY_PORTTOSADDR(&saidx.src, sport->sadb_x_nat_t_port_port); if (dport) KEY_PORTTOSADDR(&saidx.dst, dport->sadb_x_nat_t_port_port); } #endif /* get a SA index */ SAHTREE_LOCK(); LIST_FOREACH(sah, &V_sahtree, chain) { if (sah->state == SADB_SASTATE_DEAD) continue; if (key_cmpsaidx(&sah->saidx, &saidx, CMP_MODE_REQID)) break; } SAHTREE_UNLOCK(); if (sah != NULL) { ipseclog((LOG_DEBUG, "%s: a SA exists already.\n", __func__)); return key_senderror(so, m, EEXIST); } error = key_acquire(&saidx, NULL); if (error != 0) { ipseclog((LOG_DEBUG, "%s: error %d returned from key_acquire\n", __func__, mhp->msg->sadb_msg_errno)); return key_senderror(so, m, error); } return key_sendup_mbuf(so, m, KEY_SENDUP_REGISTERED); } /* * SADB_REGISTER processing. * If SATYPE_UNSPEC has been passed as satype, only return sabd_supported. * receive * * from the ikmpd, and register a socket to send PF_KEY messages, * and send * * to KMD by PF_KEY. * If socket is detached, must free from regnode. * * m will always be freed. */ static int key_register(so, m, mhp) struct socket *so; struct mbuf *m; const struct sadb_msghdr *mhp; { struct secreg *reg, *newreg = 0; IPSEC_ASSERT(so != NULL, ("null socket")); IPSEC_ASSERT(m != NULL, ("null mbuf")); IPSEC_ASSERT(mhp != NULL, ("null msghdr")); IPSEC_ASSERT(mhp->msg != NULL, ("null msg")); /* check for invalid register message */ if (mhp->msg->sadb_msg_satype >= sizeof(V_regtree)/sizeof(V_regtree[0])) return key_senderror(so, m, EINVAL); /* When SATYPE_UNSPEC is specified, only return sabd_supported. */ if (mhp->msg->sadb_msg_satype == SADB_SATYPE_UNSPEC) goto setmsg; /* check whether existing or not */ REGTREE_LOCK(); LIST_FOREACH(reg, &V_regtree[mhp->msg->sadb_msg_satype], chain) { if (reg->so == so) { REGTREE_UNLOCK(); ipseclog((LOG_DEBUG, "%s: socket exists already.\n", __func__)); return key_senderror(so, m, EEXIST); } } /* create regnode */ newreg = malloc(sizeof(struct secreg), M_IPSEC_SAR, M_NOWAIT|M_ZERO); if (newreg == NULL) { REGTREE_UNLOCK(); ipseclog((LOG_DEBUG, "%s: No more memory.\n", __func__)); return key_senderror(so, m, ENOBUFS); } newreg->so = so; ((struct keycb *)sotorawcb(so))->kp_registered++; /* add regnode to regtree. */ LIST_INSERT_HEAD(&V_regtree[mhp->msg->sadb_msg_satype], newreg, chain); REGTREE_UNLOCK(); setmsg: { struct mbuf *n; struct sadb_msg *newmsg; struct sadb_supported *sup; u_int len, alen, elen; int off; int i; struct sadb_alg *alg; /* create new sadb_msg to reply. */ alen = 0; for (i = 1; i <= SADB_AALG_MAX; i++) { if (ah_algorithm_lookup(i)) alen += sizeof(struct sadb_alg); } if (alen) alen += sizeof(struct sadb_supported); elen = 0; for (i = 1; i <= SADB_EALG_MAX; i++) { if (esp_algorithm_lookup(i)) elen += sizeof(struct sadb_alg); } if (elen) elen += sizeof(struct sadb_supported); len = sizeof(struct sadb_msg) + alen + elen; if (len > MCLBYTES) return key_senderror(so, m, ENOBUFS); MGETHDR(n, M_DONTWAIT, MT_DATA); if (len > MHLEN) { MCLGET(n, M_DONTWAIT); if ((n->m_flags & M_EXT) == 0) { m_freem(n); n = NULL; } } if (!n) return key_senderror(so, m, ENOBUFS); n->m_pkthdr.len = n->m_len = len; n->m_next = NULL; off = 0; m_copydata(m, 0, sizeof(struct sadb_msg), mtod(n, caddr_t) + off); newmsg = mtod(n, struct sadb_msg *); newmsg->sadb_msg_errno = 0; newmsg->sadb_msg_len = PFKEY_UNIT64(len); off += PFKEY_ALIGN8(sizeof(struct sadb_msg)); /* for authentication algorithm */ if (alen) { sup = (struct sadb_supported *)(mtod(n, caddr_t) + off); sup->sadb_supported_len = PFKEY_UNIT64(alen); sup->sadb_supported_exttype = SADB_EXT_SUPPORTED_AUTH; off += PFKEY_ALIGN8(sizeof(*sup)); for (i = 1; i <= SADB_AALG_MAX; i++) { struct auth_hash *aalgo; u_int16_t minkeysize, maxkeysize; aalgo = ah_algorithm_lookup(i); if (!aalgo) continue; alg = (struct sadb_alg *)(mtod(n, caddr_t) + off); alg->sadb_alg_id = i; alg->sadb_alg_ivlen = 0; key_getsizes_ah(aalgo, i, &minkeysize, &maxkeysize); alg->sadb_alg_minbits = _BITS(minkeysize); alg->sadb_alg_maxbits = _BITS(maxkeysize); off += PFKEY_ALIGN8(sizeof(*alg)); } } /* for encryption algorithm */ if (elen) { sup = (struct sadb_supported *)(mtod(n, caddr_t) + off); sup->sadb_supported_len = PFKEY_UNIT64(elen); sup->sadb_supported_exttype = SADB_EXT_SUPPORTED_ENCRYPT; off += PFKEY_ALIGN8(sizeof(*sup)); for (i = 1; i <= SADB_EALG_MAX; i++) { struct enc_xform *ealgo; ealgo = esp_algorithm_lookup(i); if (!ealgo) continue; alg = (struct sadb_alg *)(mtod(n, caddr_t) + off); alg->sadb_alg_id = i; alg->sadb_alg_ivlen = ealgo->blocksize; alg->sadb_alg_minbits = _BITS(ealgo->minkey); alg->sadb_alg_maxbits = _BITS(ealgo->maxkey); off += PFKEY_ALIGN8(sizeof(struct sadb_alg)); } } IPSEC_ASSERT(off == len, ("length assumption failed (off %u len %u)", off, len)); m_freem(m); return key_sendup_mbuf(so, n, KEY_SENDUP_REGISTERED); } } /* * free secreg entry registered. * XXX: I want to do free a socket marked done SADB_RESIGER to socket. */ void key_freereg(struct socket *so) { struct secreg *reg; int i; IPSEC_ASSERT(so != NULL, ("NULL so")); /* * check whether existing or not. * check all type of SA, because there is a potential that * one socket is registered to multiple type of SA. */ REGTREE_LOCK(); for (i = 0; i <= SADB_SATYPE_MAX; i++) { LIST_FOREACH(reg, &V_regtree[i], chain) { if (reg->so == so && __LIST_CHAINED(reg)) { LIST_REMOVE(reg, chain); free(reg, M_IPSEC_SAR); break; } } } REGTREE_UNLOCK(); } /* * SADB_EXPIRE processing * send * * to KMD by PF_KEY. * NOTE: We send only soft lifetime extension. * * OUT: 0 : succeed * others : error number */ static int key_expire(struct secasvar *sav) { int s; int satype; struct mbuf *result = NULL, *m; int len; int error = -1; struct sadb_lifetime *lt; /* XXX: Why do we lock ? */ s = splnet(); /*called from softclock()*/ IPSEC_ASSERT (sav != NULL, ("null sav")); IPSEC_ASSERT (sav->sah != NULL, ("null sa header")); /* set msg header */ satype = key_proto2satype(sav->sah->saidx.proto); IPSEC_ASSERT(satype != 0, ("invalid proto, satype %u", satype)); m = key_setsadbmsg(SADB_EXPIRE, 0, satype, sav->seq, 0, sav->refcnt); if (!m) { error = ENOBUFS; goto fail; } result = m; /* create SA extension */ m = key_setsadbsa(sav); if (!m) { error = ENOBUFS; goto fail; } m_cat(result, m); /* create SA extension */ m = key_setsadbxsa2(sav->sah->saidx.mode, sav->replay ? sav->replay->count : 0, sav->sah->saidx.reqid); if (!m) { error = ENOBUFS; goto fail; } m_cat(result, m); /* create lifetime extension (current and soft) */ len = PFKEY_ALIGN8(sizeof(*lt)) * 2; m = key_alloc_mbuf(len); if (!m || m->m_next) { /*XXX*/ if (m) m_freem(m); error = ENOBUFS; goto fail; } bzero(mtod(m, caddr_t), len); lt = mtod(m, struct sadb_lifetime *); lt->sadb_lifetime_len = PFKEY_UNIT64(sizeof(struct sadb_lifetime)); lt->sadb_lifetime_exttype = SADB_EXT_LIFETIME_CURRENT; lt->sadb_lifetime_allocations = sav->lft_c->allocations; lt->sadb_lifetime_bytes = sav->lft_c->bytes; lt->sadb_lifetime_addtime = sav->lft_c->addtime; lt->sadb_lifetime_usetime = sav->lft_c->usetime; lt = (struct sadb_lifetime *)(mtod(m, caddr_t) + len / 2); lt->sadb_lifetime_len = PFKEY_UNIT64(sizeof(struct sadb_lifetime)); lt->sadb_lifetime_exttype = SADB_EXT_LIFETIME_SOFT; lt->sadb_lifetime_allocations = sav->lft_s->allocations; lt->sadb_lifetime_bytes = sav->lft_s->bytes; lt->sadb_lifetime_addtime = sav->lft_s->addtime; lt->sadb_lifetime_usetime = sav->lft_s->usetime; m_cat(result, m); /* set sadb_address for source */ m = key_setsadbaddr(SADB_EXT_ADDRESS_SRC, &sav->sah->saidx.src.sa, FULLMASK, IPSEC_ULPROTO_ANY); if (!m) { error = ENOBUFS; goto fail; } m_cat(result, m); /* set sadb_address for destination */ m = key_setsadbaddr(SADB_EXT_ADDRESS_DST, &sav->sah->saidx.dst.sa, FULLMASK, IPSEC_ULPROTO_ANY); if (!m) { error = ENOBUFS; goto fail; } m_cat(result, m); /* * XXX-BZ Handle NAT-T extensions here. */ if ((result->m_flags & M_PKTHDR) == 0) { error = EINVAL; goto fail; } if (result->m_len < sizeof(struct sadb_msg)) { result = m_pullup(result, sizeof(struct sadb_msg)); if (result == NULL) { error = ENOBUFS; goto fail; } } result->m_pkthdr.len = 0; for (m = result; m; m = m->m_next) result->m_pkthdr.len += m->m_len; mtod(result, struct sadb_msg *)->sadb_msg_len = PFKEY_UNIT64(result->m_pkthdr.len); splx(s); return key_sendup_mbuf(NULL, result, KEY_SENDUP_REGISTERED); fail: if (result) m_freem(result); splx(s); return error; } /* * SADB_FLUSH processing * receive * * from the ikmpd, and free all entries in secastree. * and send, * * to the ikmpd. * NOTE: to do is only marking SADB_SASTATE_DEAD. * * m will always be freed. */ static int key_flush(so, m, mhp) struct socket *so; struct mbuf *m; const struct sadb_msghdr *mhp; { struct sadb_msg *newmsg; struct secashead *sah, *nextsah; struct secasvar *sav, *nextsav; u_int16_t proto; u_int8_t state; u_int stateidx; IPSEC_ASSERT(so != NULL, ("null socket")); IPSEC_ASSERT(mhp != NULL, ("null msghdr")); IPSEC_ASSERT(mhp->msg != NULL, ("null msg")); /* map satype to proto */ if ((proto = key_satype2proto(mhp->msg->sadb_msg_satype)) == 0) { ipseclog((LOG_DEBUG, "%s: invalid satype is passed.\n", __func__)); return key_senderror(so, m, EINVAL); } /* no SATYPE specified, i.e. flushing all SA. */ SAHTREE_LOCK(); for (sah = LIST_FIRST(&V_sahtree); sah != NULL; sah = nextsah) { nextsah = LIST_NEXT(sah, chain); if (mhp->msg->sadb_msg_satype != SADB_SATYPE_UNSPEC && proto != sah->saidx.proto) continue; for (stateidx = 0; stateidx < _ARRAYLEN(saorder_state_alive); stateidx++) { state = saorder_state_any[stateidx]; for (sav = LIST_FIRST(&sah->savtree[state]); sav != NULL; sav = nextsav) { nextsav = LIST_NEXT(sav, chain); key_sa_chgstate(sav, SADB_SASTATE_DEAD); KEY_FREESAV(&sav); } } sah->state = SADB_SASTATE_DEAD; } SAHTREE_UNLOCK(); if (m->m_len < sizeof(struct sadb_msg) || sizeof(struct sadb_msg) > m->m_len + M_TRAILINGSPACE(m)) { ipseclog((LOG_DEBUG, "%s: No more memory.\n", __func__)); return key_senderror(so, m, ENOBUFS); } if (m->m_next) m_freem(m->m_next); m->m_next = NULL; m->m_pkthdr.len = m->m_len = sizeof(struct sadb_msg); newmsg = mtod(m, struct sadb_msg *); newmsg->sadb_msg_errno = 0; newmsg->sadb_msg_len = PFKEY_UNIT64(m->m_pkthdr.len); return key_sendup_mbuf(so, m, KEY_SENDUP_ALL); } /* * SADB_DUMP processing * dump all entries including status of DEAD in SAD. * receive * * from the ikmpd, and dump all secasvar leaves * and send, * ..... * to the ikmpd. * * m will always be freed. */ static int key_dump(so, m, mhp) struct socket *so; struct mbuf *m; const struct sadb_msghdr *mhp; { struct secashead *sah; struct secasvar *sav; u_int16_t proto; u_int stateidx; u_int8_t satype; u_int8_t state; int cnt; struct sadb_msg *newmsg; struct mbuf *n; IPSEC_ASSERT(so != NULL, ("null socket")); IPSEC_ASSERT(m != NULL, ("null mbuf")); IPSEC_ASSERT(mhp != NULL, ("null msghdr")); IPSEC_ASSERT(mhp->msg != NULL, ("null msg")); /* map satype to proto */ if ((proto = key_satype2proto(mhp->msg->sadb_msg_satype)) == 0) { ipseclog((LOG_DEBUG, "%s: invalid satype is passed.\n", __func__)); return key_senderror(so, m, EINVAL); } /* count sav entries to be sent to the userland. */ cnt = 0; SAHTREE_LOCK(); LIST_FOREACH(sah, &V_sahtree, chain) { if (mhp->msg->sadb_msg_satype != SADB_SATYPE_UNSPEC && proto != sah->saidx.proto) continue; for (stateidx = 0; stateidx < _ARRAYLEN(saorder_state_any); stateidx++) { state = saorder_state_any[stateidx]; LIST_FOREACH(sav, &sah->savtree[state], chain) { cnt++; } } } if (cnt == 0) { SAHTREE_UNLOCK(); return key_senderror(so, m, ENOENT); } /* send this to the userland, one at a time. */ newmsg = NULL; LIST_FOREACH(sah, &V_sahtree, chain) { if (mhp->msg->sadb_msg_satype != SADB_SATYPE_UNSPEC && proto != sah->saidx.proto) continue; /* map proto to satype */ if ((satype = key_proto2satype(sah->saidx.proto)) == 0) { SAHTREE_UNLOCK(); ipseclog((LOG_DEBUG, "%s: there was invalid proto in " "SAD.\n", __func__)); return key_senderror(so, m, EINVAL); } for (stateidx = 0; stateidx < _ARRAYLEN(saorder_state_any); stateidx++) { state = saorder_state_any[stateidx]; LIST_FOREACH(sav, &sah->savtree[state], chain) { n = key_setdumpsa(sav, SADB_DUMP, satype, --cnt, mhp->msg->sadb_msg_pid); if (!n) { SAHTREE_UNLOCK(); return key_senderror(so, m, ENOBUFS); } key_sendup_mbuf(so, n, KEY_SENDUP_ONE); } } } SAHTREE_UNLOCK(); m_freem(m); return 0; } /* * SADB_X_PROMISC processing * * m will always be freed. */ static int key_promisc(so, m, mhp) struct socket *so; struct mbuf *m; const struct sadb_msghdr *mhp; { int olen; IPSEC_ASSERT(so != NULL, ("null socket")); IPSEC_ASSERT(m != NULL, ("null mbuf")); IPSEC_ASSERT(mhp != NULL, ("null msghdr")); IPSEC_ASSERT(mhp->msg != NULL, ("null msg")); olen = PFKEY_UNUNIT64(mhp->msg->sadb_msg_len); if (olen < sizeof(struct sadb_msg)) { #if 1 return key_senderror(so, m, EINVAL); #else m_freem(m); return 0; #endif } else if (olen == sizeof(struct sadb_msg)) { /* enable/disable promisc mode */ struct keycb *kp; if ((kp = (struct keycb *)sotorawcb(so)) == NULL) return key_senderror(so, m, EINVAL); mhp->msg->sadb_msg_errno = 0; switch (mhp->msg->sadb_msg_satype) { case 0: case 1: kp->kp_promisc = mhp->msg->sadb_msg_satype; break; default: return key_senderror(so, m, EINVAL); } /* send the original message back to everyone */ mhp->msg->sadb_msg_errno = 0; return key_sendup_mbuf(so, m, KEY_SENDUP_ALL); } else { /* send packet as is */ m_adj(m, PFKEY_ALIGN8(sizeof(struct sadb_msg))); /* TODO: if sadb_msg_seq is specified, send to specific pid */ return key_sendup_mbuf(so, m, KEY_SENDUP_ALL); } } static int (*key_typesw[]) __P((struct socket *, struct mbuf *, const struct sadb_msghdr *)) = { NULL, /* SADB_RESERVED */ key_getspi, /* SADB_GETSPI */ key_update, /* SADB_UPDATE */ key_add, /* SADB_ADD */ key_delete, /* SADB_DELETE */ key_get, /* SADB_GET */ key_acquire2, /* SADB_ACQUIRE */ key_register, /* SADB_REGISTER */ NULL, /* SADB_EXPIRE */ key_flush, /* SADB_FLUSH */ key_dump, /* SADB_DUMP */ key_promisc, /* SADB_X_PROMISC */ NULL, /* SADB_X_PCHANGE */ key_spdadd, /* SADB_X_SPDUPDATE */ key_spdadd, /* SADB_X_SPDADD */ key_spddelete, /* SADB_X_SPDDELETE */ key_spdget, /* SADB_X_SPDGET */ NULL, /* SADB_X_SPDACQUIRE */ key_spddump, /* SADB_X_SPDDUMP */ key_spdflush, /* SADB_X_SPDFLUSH */ key_spdadd, /* SADB_X_SPDSETIDX */ NULL, /* SADB_X_SPDEXPIRE */ key_spddelete2, /* SADB_X_SPDDELETE2 */ }; /* * parse sadb_msg buffer to process PFKEYv2, * and create a data to response if needed. * I think to be dealed with mbuf directly. * IN: * msgp : pointer to pointer to a received buffer pulluped. * This is rewrited to response. * so : pointer to socket. * OUT: * length for buffer to send to user process. */ int key_parse(m, so) struct mbuf *m; struct socket *so; { struct sadb_msg *msg; struct sadb_msghdr mh; u_int orglen; int error; int target; IPSEC_ASSERT(so != NULL, ("null socket")); IPSEC_ASSERT(m != NULL, ("null mbuf")); #if 0 /*kdebug_sadb assumes msg in linear buffer*/ KEYDEBUG(KEYDEBUG_KEY_DUMP, ipseclog((LOG_DEBUG, "%s: passed sadb_msg\n", __func__)); kdebug_sadb(msg)); #endif if (m->m_len < sizeof(struct sadb_msg)) { m = m_pullup(m, sizeof(struct sadb_msg)); if (!m) return ENOBUFS; } msg = mtod(m, struct sadb_msg *); orglen = PFKEY_UNUNIT64(msg->sadb_msg_len); target = KEY_SENDUP_ONE; if ((m->m_flags & M_PKTHDR) == 0 || m->m_pkthdr.len != m->m_pkthdr.len) { ipseclog((LOG_DEBUG, "%s: invalid message length.\n",__func__)); V_pfkeystat.out_invlen++; error = EINVAL; goto senderror; } if (msg->sadb_msg_version != PF_KEY_V2) { ipseclog((LOG_DEBUG, "%s: PF_KEY version %u is mismatched.\n", __func__, msg->sadb_msg_version)); V_pfkeystat.out_invver++; error = EINVAL; goto senderror; } if (msg->sadb_msg_type > SADB_MAX) { ipseclog((LOG_DEBUG, "%s: invalid type %u is passed.\n", __func__, msg->sadb_msg_type)); V_pfkeystat.out_invmsgtype++; error = EINVAL; goto senderror; } /* for old-fashioned code - should be nuked */ if (m->m_pkthdr.len > MCLBYTES) { m_freem(m); return ENOBUFS; } if (m->m_next) { struct mbuf *n; MGETHDR(n, M_DONTWAIT, MT_DATA); if (n && m->m_pkthdr.len > MHLEN) { MCLGET(n, M_DONTWAIT); if ((n->m_flags & M_EXT) == 0) { m_free(n); n = NULL; } } if (!n) { m_freem(m); return ENOBUFS; } m_copydata(m, 0, m->m_pkthdr.len, mtod(n, caddr_t)); n->m_pkthdr.len = n->m_len = m->m_pkthdr.len; n->m_next = NULL; m_freem(m); m = n; } /* align the mbuf chain so that extensions are in contiguous region. */ error = key_align(m, &mh); if (error) return error; msg = mh.msg; /* check SA type */ switch (msg->sadb_msg_satype) { case SADB_SATYPE_UNSPEC: switch (msg->sadb_msg_type) { case SADB_GETSPI: case SADB_UPDATE: case SADB_ADD: case SADB_DELETE: case SADB_GET: case SADB_ACQUIRE: case SADB_EXPIRE: ipseclog((LOG_DEBUG, "%s: must specify satype " "when msg type=%u.\n", __func__, msg->sadb_msg_type)); V_pfkeystat.out_invsatype++; error = EINVAL; goto senderror; } break; case SADB_SATYPE_AH: case SADB_SATYPE_ESP: case SADB_X_SATYPE_IPCOMP: case SADB_X_SATYPE_TCPSIGNATURE: switch (msg->sadb_msg_type) { case SADB_X_SPDADD: case SADB_X_SPDDELETE: case SADB_X_SPDGET: case SADB_X_SPDDUMP: case SADB_X_SPDFLUSH: case SADB_X_SPDSETIDX: case SADB_X_SPDUPDATE: case SADB_X_SPDDELETE2: ipseclog((LOG_DEBUG, "%s: illegal satype=%u\n", __func__, msg->sadb_msg_type)); V_pfkeystat.out_invsatype++; error = EINVAL; goto senderror; } break; case SADB_SATYPE_RSVP: case SADB_SATYPE_OSPFV2: case SADB_SATYPE_RIPV2: case SADB_SATYPE_MIP: ipseclog((LOG_DEBUG, "%s: type %u isn't supported.\n", __func__, msg->sadb_msg_satype)); V_pfkeystat.out_invsatype++; error = EOPNOTSUPP; goto senderror; case 1: /* XXX: What does it do? */ if (msg->sadb_msg_type == SADB_X_PROMISC) break; /*FALLTHROUGH*/ default: ipseclog((LOG_DEBUG, "%s: invalid type %u is passed.\n", __func__, msg->sadb_msg_satype)); V_pfkeystat.out_invsatype++; error = EINVAL; goto senderror; } /* check field of upper layer protocol and address family */ if (mh.ext[SADB_EXT_ADDRESS_SRC] != NULL && mh.ext[SADB_EXT_ADDRESS_DST] != NULL) { struct sadb_address *src0, *dst0; u_int plen; src0 = (struct sadb_address *)(mh.ext[SADB_EXT_ADDRESS_SRC]); dst0 = (struct sadb_address *)(mh.ext[SADB_EXT_ADDRESS_DST]); /* check upper layer protocol */ if (src0->sadb_address_proto != dst0->sadb_address_proto) { ipseclog((LOG_DEBUG, "%s: upper layer protocol " "mismatched.\n", __func__)); V_pfkeystat.out_invaddr++; error = EINVAL; goto senderror; } /* check family */ if (PFKEY_ADDR_SADDR(src0)->sa_family != PFKEY_ADDR_SADDR(dst0)->sa_family) { ipseclog((LOG_DEBUG, "%s: address family mismatched.\n", __func__)); V_pfkeystat.out_invaddr++; error = EINVAL; goto senderror; } if (PFKEY_ADDR_SADDR(src0)->sa_len != PFKEY_ADDR_SADDR(dst0)->sa_len) { ipseclog((LOG_DEBUG, "%s: address struct size " "mismatched.\n", __func__)); V_pfkeystat.out_invaddr++; error = EINVAL; goto senderror; } switch (PFKEY_ADDR_SADDR(src0)->sa_family) { case AF_INET: if (PFKEY_ADDR_SADDR(src0)->sa_len != sizeof(struct sockaddr_in)) { V_pfkeystat.out_invaddr++; error = EINVAL; goto senderror; } break; case AF_INET6: if (PFKEY_ADDR_SADDR(src0)->sa_len != sizeof(struct sockaddr_in6)) { V_pfkeystat.out_invaddr++; error = EINVAL; goto senderror; } break; default: ipseclog((LOG_DEBUG, "%s: unsupported address family\n", __func__)); V_pfkeystat.out_invaddr++; error = EAFNOSUPPORT; goto senderror; } switch (PFKEY_ADDR_SADDR(src0)->sa_family) { case AF_INET: plen = sizeof(struct in_addr) << 3; break; case AF_INET6: plen = sizeof(struct in6_addr) << 3; break; default: plen = 0; /*fool gcc*/ break; } /* check max prefix length */ if (src0->sadb_address_prefixlen > plen || dst0->sadb_address_prefixlen > plen) { ipseclog((LOG_DEBUG, "%s: illegal prefixlen.\n", __func__)); V_pfkeystat.out_invaddr++; error = EINVAL; goto senderror; } /* * prefixlen == 0 is valid because there can be a case when * all addresses are matched. */ } if (msg->sadb_msg_type >= sizeof(key_typesw)/sizeof(key_typesw[0]) || key_typesw[msg->sadb_msg_type] == NULL) { V_pfkeystat.out_invmsgtype++; error = EINVAL; goto senderror; } return (*key_typesw[msg->sadb_msg_type])(so, m, &mh); senderror: msg->sadb_msg_errno = error; return key_sendup_mbuf(so, m, target); } static int key_senderror(so, m, code) struct socket *so; struct mbuf *m; int code; { struct sadb_msg *msg; IPSEC_ASSERT(m->m_len >= sizeof(struct sadb_msg), ("mbuf too small, len %u", m->m_len)); msg = mtod(m, struct sadb_msg *); msg->sadb_msg_errno = code; return key_sendup_mbuf(so, m, KEY_SENDUP_ONE); } /* * set the pointer to each header into message buffer. * m will be freed on error. * XXX larger-than-MCLBYTES extension? */ static int key_align(m, mhp) struct mbuf *m; struct sadb_msghdr *mhp; { struct mbuf *n; struct sadb_ext *ext; size_t off, end; int extlen; int toff; IPSEC_ASSERT(m != NULL, ("null mbuf")); IPSEC_ASSERT(mhp != NULL, ("null msghdr")); IPSEC_ASSERT(m->m_len >= sizeof(struct sadb_msg), ("mbuf too small, len %u", m->m_len)); /* initialize */ bzero(mhp, sizeof(*mhp)); mhp->msg = mtod(m, struct sadb_msg *); mhp->ext[0] = (struct sadb_ext *)mhp->msg; /*XXX backward compat */ end = PFKEY_UNUNIT64(mhp->msg->sadb_msg_len); extlen = end; /*just in case extlen is not updated*/ for (off = sizeof(struct sadb_msg); off < end; off += extlen) { n = m_pulldown(m, off, sizeof(struct sadb_ext), &toff); if (!n) { /* m is already freed */ return ENOBUFS; } ext = (struct sadb_ext *)(mtod(n, caddr_t) + toff); /* set pointer */ switch (ext->sadb_ext_type) { case SADB_EXT_SA: case SADB_EXT_ADDRESS_SRC: case SADB_EXT_ADDRESS_DST: case SADB_EXT_ADDRESS_PROXY: case SADB_EXT_LIFETIME_CURRENT: case SADB_EXT_LIFETIME_HARD: case SADB_EXT_LIFETIME_SOFT: case SADB_EXT_KEY_AUTH: case SADB_EXT_KEY_ENCRYPT: case SADB_EXT_IDENTITY_SRC: case SADB_EXT_IDENTITY_DST: case SADB_EXT_SENSITIVITY: case SADB_EXT_PROPOSAL: case SADB_EXT_SUPPORTED_AUTH: case SADB_EXT_SUPPORTED_ENCRYPT: case SADB_EXT_SPIRANGE: case SADB_X_EXT_POLICY: case SADB_X_EXT_SA2: #ifdef IPSEC_NAT_T case SADB_X_EXT_NAT_T_TYPE: case SADB_X_EXT_NAT_T_SPORT: case SADB_X_EXT_NAT_T_DPORT: case SADB_X_EXT_NAT_T_OAI: case SADB_X_EXT_NAT_T_OAR: case SADB_X_EXT_NAT_T_FRAG: #endif /* duplicate check */ /* * XXX Are there duplication payloads of either * KEY_AUTH or KEY_ENCRYPT ? */ if (mhp->ext[ext->sadb_ext_type] != NULL) { ipseclog((LOG_DEBUG, "%s: duplicate ext_type " "%u\n", __func__, ext->sadb_ext_type)); m_freem(m); V_pfkeystat.out_dupext++; return EINVAL; } break; default: ipseclog((LOG_DEBUG, "%s: invalid ext_type %u\n", __func__, ext->sadb_ext_type)); m_freem(m); V_pfkeystat.out_invexttype++; return EINVAL; } extlen = PFKEY_UNUNIT64(ext->sadb_ext_len); if (key_validate_ext(ext, extlen)) { m_freem(m); V_pfkeystat.out_invlen++; return EINVAL; } n = m_pulldown(m, off, extlen, &toff); if (!n) { /* m is already freed */ return ENOBUFS; } ext = (struct sadb_ext *)(mtod(n, caddr_t) + toff); mhp->ext[ext->sadb_ext_type] = ext; mhp->extoff[ext->sadb_ext_type] = off; mhp->extlen[ext->sadb_ext_type] = extlen; } if (off != end) { m_freem(m); V_pfkeystat.out_invlen++; return EINVAL; } return 0; } static int key_validate_ext(ext, len) const struct sadb_ext *ext; int len; { const struct sockaddr *sa; enum { NONE, ADDR } checktype = NONE; int baselen = 0; const int sal = offsetof(struct sockaddr, sa_len) + sizeof(sa->sa_len); if (len != PFKEY_UNUNIT64(ext->sadb_ext_len)) return EINVAL; /* if it does not match minimum/maximum length, bail */ if (ext->sadb_ext_type >= sizeof(minsize) / sizeof(minsize[0]) || ext->sadb_ext_type >= sizeof(maxsize) / sizeof(maxsize[0])) return EINVAL; if (!minsize[ext->sadb_ext_type] || len < minsize[ext->sadb_ext_type]) return EINVAL; if (maxsize[ext->sadb_ext_type] && len > maxsize[ext->sadb_ext_type]) return EINVAL; /* more checks based on sadb_ext_type XXX need more */ switch (ext->sadb_ext_type) { case SADB_EXT_ADDRESS_SRC: case SADB_EXT_ADDRESS_DST: case SADB_EXT_ADDRESS_PROXY: baselen = PFKEY_ALIGN8(sizeof(struct sadb_address)); checktype = ADDR; break; case SADB_EXT_IDENTITY_SRC: case SADB_EXT_IDENTITY_DST: if (((const struct sadb_ident *)ext)->sadb_ident_type == SADB_X_IDENTTYPE_ADDR) { baselen = PFKEY_ALIGN8(sizeof(struct sadb_ident)); checktype = ADDR; } else checktype = NONE; break; default: checktype = NONE; break; } switch (checktype) { case NONE: break; case ADDR: sa = (const struct sockaddr *)(((const u_int8_t*)ext)+baselen); if (len < baselen + sal) return EINVAL; if (baselen + PFKEY_ALIGN8(sa->sa_len) != len) return EINVAL; break; } return 0; } void key_init(void) { int i; for (i = 0; i < IPSEC_DIR_MAX; i++) LIST_INIT(&V_sptree[i]); LIST_INIT(&V_sahtree); for (i = 0; i <= SADB_SATYPE_MAX; i++) LIST_INIT(&V_regtree[i]); LIST_INIT(&V_acqtree); LIST_INIT(&V_spacqtree); /* system default */ V_ip4_def_policy.policy = IPSEC_POLICY_NONE; V_ip4_def_policy.refcnt++; /*never reclaim this*/ if (!IS_DEFAULT_VNET(curvnet)) return; SPTREE_LOCK_INIT(); REGTREE_LOCK_INIT(); SAHTREE_LOCK_INIT(); ACQ_LOCK_INIT(); SPACQ_LOCK_INIT(); #ifndef IPSEC_DEBUG2 timeout((void *)key_timehandler, (void *)0, hz); #endif /*IPSEC_DEBUG2*/ /* initialize key statistics */ keystat.getspi_count = 1; printf("IPsec: Initialized Security Association Processing.\n"); } #ifdef VIMAGE void key_destroy(void) { struct secpolicy *sp, *nextsp; struct secacq *acq, *nextacq; struct secspacq *spacq, *nextspacq; struct secashead *sah, *nextsah; struct secreg *reg; int i; SPTREE_LOCK(); for (i = 0; i < IPSEC_DIR_MAX; i++) { for (sp = LIST_FIRST(&V_sptree[i]); sp != NULL; sp = nextsp) { nextsp = LIST_NEXT(sp, chain); if (__LIST_CHAINED(sp)) { LIST_REMOVE(sp, chain); free(sp, M_IPSEC_SP); } } } SPTREE_UNLOCK(); SAHTREE_LOCK(); for (sah = LIST_FIRST(&V_sahtree); sah != NULL; sah = nextsah) { nextsah = LIST_NEXT(sah, chain); if (__LIST_CHAINED(sah)) { LIST_REMOVE(sah, chain); free(sah, M_IPSEC_SAH); } } SAHTREE_UNLOCK(); REGTREE_LOCK(); for (i = 0; i <= SADB_SATYPE_MAX; i++) { LIST_FOREACH(reg, &V_regtree[i], chain) { if (__LIST_CHAINED(reg)) { LIST_REMOVE(reg, chain); free(reg, M_IPSEC_SAR); break; } } } REGTREE_UNLOCK(); ACQ_LOCK(); for (acq = LIST_FIRST(&V_acqtree); acq != NULL; acq = nextacq) { nextacq = LIST_NEXT(acq, chain); if (__LIST_CHAINED(acq)) { LIST_REMOVE(acq, chain); free(acq, M_IPSEC_SAQ); } } ACQ_UNLOCK(); SPACQ_LOCK(); for (spacq = LIST_FIRST(&V_spacqtree); spacq != NULL; spacq = nextspacq) { nextspacq = LIST_NEXT(spacq, chain); if (__LIST_CHAINED(spacq)) { LIST_REMOVE(spacq, chain); free(spacq, M_IPSEC_SAQ); } } SPACQ_UNLOCK(); } #endif /* * XXX: maybe This function is called after INBOUND IPsec processing. * * Special check for tunnel-mode packets. * We must make some checks for consistency between inner and outer IP header. * * xxx more checks to be provided */ int key_checktunnelsanity(sav, family, src, dst) struct secasvar *sav; u_int family; caddr_t src; caddr_t dst; { IPSEC_ASSERT(sav->sah != NULL, ("null SA header")); /* XXX: check inner IP header */ return 1; } /* record data transfer on SA, and update timestamps */ void key_sa_recordxfer(sav, m) struct secasvar *sav; struct mbuf *m; { IPSEC_ASSERT(sav != NULL, ("Null secasvar")); IPSEC_ASSERT(m != NULL, ("Null mbuf")); if (!sav->lft_c) return; /* * XXX Currently, there is a difference of bytes size * between inbound and outbound processing. */ sav->lft_c->bytes += m->m_pkthdr.len; /* to check bytes lifetime is done in key_timehandler(). */ /* * We use the number of packets as the unit of * allocations. We increment the variable * whenever {esp,ah}_{in,out}put is called. */ sav->lft_c->allocations++; /* XXX check for expires? */ /* * NOTE: We record CURRENT usetime by using wall clock, * in seconds. HARD and SOFT lifetime are measured by the time * difference (again in seconds) from usetime. * * usetime * v expire expire * -----+-----+--------+---> t * <--------------> HARD * <-----> SOFT */ sav->lft_c->usetime = time_second; /* XXX check for expires? */ return; } /* dumb version */ void key_sa_routechange(dst) struct sockaddr *dst; { struct secashead *sah; struct route *ro; SAHTREE_LOCK(); LIST_FOREACH(sah, &V_sahtree, chain) { ro = &sah->sa_route; if (ro->ro_rt && dst->sa_len == ro->ro_dst.sa_len && bcmp(dst, &ro->ro_dst, dst->sa_len) == 0) { RTFREE(ro->ro_rt); ro->ro_rt = (struct rtentry *)NULL; } } SAHTREE_UNLOCK(); } static void key_sa_chgstate(struct secasvar *sav, u_int8_t state) { IPSEC_ASSERT(sav != NULL, ("NULL sav")); SAHTREE_LOCK_ASSERT(); if (sav->state != state) { if (__LIST_CHAINED(sav)) LIST_REMOVE(sav, chain); sav->state = state; LIST_INSERT_HEAD(&sav->sah->savtree[state], sav, chain); } } void key_sa_stir_iv(sav) struct secasvar *sav; { IPSEC_ASSERT(sav->iv != NULL, ("null IV")); key_randomfill(sav->iv, sav->ivlen); } /* XXX too much? */ static struct mbuf * key_alloc_mbuf(l) int l; { struct mbuf *m = NULL, *n; int len, t; len = l; while (len > 0) { MGET(n, M_DONTWAIT, MT_DATA); if (n && len > MLEN) MCLGET(n, M_DONTWAIT); if (!n) { m_freem(m); return NULL; } n->m_next = NULL; n->m_len = 0; n->m_len = M_TRAILINGSPACE(n); /* use the bottom of mbuf, hoping we can prepend afterwards */ if (n->m_len > len) { t = (n->m_len - len) & ~(sizeof(long) - 1); n->m_data += t; n->m_len = len; } len -= n->m_len; if (m) m_cat(m, n); else m = n; } return m; } /* * Take one of the kernel's security keys and convert it into a PF_KEY * structure within an mbuf, suitable for sending up to a waiting * application in user land. * * IN: * src: A pointer to a kernel security key. * exttype: Which type of key this is. Refer to the PF_KEY data structures. * OUT: * a valid mbuf or NULL indicating an error * */ static struct mbuf * key_setkey(struct seckey *src, u_int16_t exttype) { struct mbuf *m; struct sadb_key *p; int len; if (src == NULL) return NULL; len = PFKEY_ALIGN8(sizeof(struct sadb_key) + _KEYLEN(src)); m = key_alloc_mbuf(len); if (m == NULL) return NULL; p = mtod(m, struct sadb_key *); bzero(p, len); p->sadb_key_len = PFKEY_UNIT64(len); p->sadb_key_exttype = exttype; p->sadb_key_bits = src->bits; bcopy(src->key_data, _KEYBUF(p), _KEYLEN(src)); return m; } /* * Take one of the kernel's lifetime data structures and convert it * into a PF_KEY structure within an mbuf, suitable for sending up to * a waiting application in user land. * * IN: * src: A pointer to a kernel lifetime structure. * exttype: Which type of lifetime this is. Refer to the PF_KEY * data structures for more information. * OUT: * a valid mbuf or NULL indicating an error * */ static struct mbuf * key_setlifetime(struct seclifetime *src, u_int16_t exttype) { struct mbuf *m = NULL; struct sadb_lifetime *p; int len = PFKEY_ALIGN8(sizeof(struct sadb_lifetime)); if (src == NULL) return NULL; m = key_alloc_mbuf(len); if (m == NULL) return m; p = mtod(m, struct sadb_lifetime *); bzero(p, len); p->sadb_lifetime_len = PFKEY_UNIT64(len); p->sadb_lifetime_exttype = exttype; p->sadb_lifetime_allocations = src->allocations; p->sadb_lifetime_bytes = src->bytes; p->sadb_lifetime_addtime = src->addtime; p->sadb_lifetime_usetime = src->usetime; return m; } Index: user/imp/tbemd/usr.sbin/config/config.h =================================================================== --- user/imp/tbemd/usr.sbin/config/config.h (revision 206684) +++ user/imp/tbemd/usr.sbin/config/config.h (revision 206685) @@ -1,201 +1,203 @@ /* * Copyright (c) 1980, 1993 * The Regents of the University of California. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 4. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * @(#)config.h 8.1 (Berkeley) 6/6/93 * $FreeBSD$ */ /* * Config. */ #include #include #include #include struct cfgfile { STAILQ_ENTRY(cfgfile) cfg_next; char *cfg_path; }; STAILQ_HEAD(, cfgfile) cfgfiles; struct file_list { STAILQ_ENTRY(file_list) f_next; char *f_fn; /* the name */ int f_type; /* type */ u_char f_flags; /* see below */ char *f_compilewith; /* special make rule if present */ char *f_depends; /* additional dependancies */ char *f_clean; /* File list to add to clean rule */ char *f_warn; /* warning message */ }; struct files_name { char *f_name; STAILQ_ENTRY(files_name) f_next; }; /* * Types. */ #define NORMAL 1 #define PROFILING 3 #define NODEPEND 4 #define LOCAL 5 #define DEVDONE 0x80000000 #define TYPEMASK 0x7fffffff /* * Attributes (flags). */ #define NO_IMPLCT_RULE 1 #define NO_OBJ 2 #define BEFORE_DEPEND 4 #define NOWERROR 16 struct device { int d_done; /* processed */ char *d_name; /* name of device (e.g. rk11) */ #define UNKNOWN -2 /* -2 means not set yet */ STAILQ_ENTRY(device) d_next; /* Next one in list */ }; struct config { char *s_sysname; }; /* * Config has a global notion of which machine type is * being used. It uses the name of the machine in choosing * files and directories. Thus if the name of the machine is ``i386'', * it will build from ``Makefile.i386'' and use ``../i386/inline'' * in the makerules, etc. machinearch is the global notion of the * MACHINE_ARCH for this MACHINE. */ char *machinename; char *machinearch; /* * For each machine, a set of CPU's may be specified as supported. * These and the options (below) are put in the C flags in the makefile. */ struct cputype { char *cpu_name; SLIST_ENTRY(cputype) cpu_next; }; SLIST_HEAD(, cputype) cputype; /* * A set of options may also be specified which are like CPU types, * but which may also specify values for the options. * A separate set of options may be defined for make-style options. */ struct opt { char *op_name; char *op_value; int op_ownfile; /* true = own file, false = makefile */ SLIST_ENTRY(opt) op_next; SLIST_ENTRY(opt) op_append; }; SLIST_HEAD(opt_head, opt) opt, mkopt, rmopts; struct opt_list { char *o_name; char *o_file; + int o_flags; +#define OL_ALIAS 1 SLIST_ENTRY(opt_list) o_next; }; SLIST_HEAD(, opt_list) otab; struct hint { char *hint_name; STAILQ_ENTRY(hint) hint_next; }; STAILQ_HEAD(hint_head, hint) hints; /* * Tag present in the kernelconf.tmlp template file. It's mandatory for those * two strings to be the same. Otherwise you'll get into trouble. */ #define KERNCONFTAG "%%KERNCONFFILE%%" /* * Faked option to note, that the configuration file has been taken from the * kernel file and inclusion of DEFAULTS etc.. isn't nessesery, because we * already have a list of all required devices. */ #define OPT_AUTOGEN "CONFIG_AUTOGENERATED" extern char *ident; extern char *env; extern char kernconfstr[]; extern int do_trace; extern int envmode; extern int hintmode; extern int incignore; char *get_word(FILE *); char *get_quoted_word(FILE *); char *path(const char *); char *raisestr(char *); void remember(const char *); void moveifchanged(const char *, const char *); int yyparse(void); int yylex(void); void options(void); void makefile(void); void makeenv(void); void makehints(void); void headers(void); void cfgfile_add(const char *); void cfgfile_removeall(void); extern STAILQ_HEAD(device_head, device) dtab; extern char errbuf[80]; extern int yyline; extern const char *yyfile; extern STAILQ_HEAD(file_list_head, file_list) ftab; extern STAILQ_HEAD(files_name_head, files_name) fntab; extern int profiling; extern int debugging; extern int found_defaults; extern int maxusers; extern char *PREFIX; /* Config file name - for error messages */ extern char srcdir[]; /* root of the kernel source tree */ #define eq(a,b) (!strcmp(a,b)) #define ns(s) strdup(s) Index: user/imp/tbemd/usr.sbin/config/mkoptions.c =================================================================== --- user/imp/tbemd/usr.sbin/config/mkoptions.c (revision 206684) +++ user/imp/tbemd/usr.sbin/config/mkoptions.c (revision 206685) @@ -1,367 +1,392 @@ /* * Copyright (c) 1995 Peter Wemm * Copyright (c) 1980, 1993 * The Regents of the University of California. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 4. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #ifndef lint #if 0 static char sccsid[] = "@(#)mkheaders.c 8.1 (Berkeley) 6/6/93"; #endif static const char rcsid[] = "$FreeBSD$"; #endif /* not lint */ /* * Make all the .h files for the optional entries */ #include #include #include #include #include #include "config.h" #include "y.tab.h" static struct users { int u_default; int u_min; int u_max; } users = { 8, 2, 512 }; static char *lower(char *); static void read_options(void); static void do_option(char *); static char *tooption(char *); void options(void) { char buf[40]; struct cputype *cp; struct opt_list *ol; struct opt *op; /* Fake the cpu types as options. */ SLIST_FOREACH(cp, &cputype, cpu_next) { op = (struct opt *)calloc(1, sizeof(*op)); if (op == NULL) err(EXIT_FAILURE, "calloc"); op->op_name = ns(cp->cpu_name); SLIST_INSERT_HEAD(&opt, op, op_next); } if (maxusers == 0) { /* printf("maxusers not specified; will auto-size\n"); */ } else if (maxusers < users.u_min) { printf("minimum of %d maxusers assumed\n", users.u_min); maxusers = users.u_min; } else if (maxusers > users.u_max) printf("warning: maxusers > %d (%d)\n", users.u_max, maxusers); /* Fake MAXUSERS as an option. */ op = (struct opt *)calloc(1, sizeof(*op)); if (op == NULL) err(EXIT_FAILURE, "calloc"); op->op_name = ns("MAXUSERS"); snprintf(buf, sizeof(buf), "%d", maxusers); op->op_value = ns(buf); SLIST_INSERT_HEAD(&opt, op, op_next); read_options(); + SLIST_FOREACH(op, &opt, op_next) { + SLIST_FOREACH(ol, &otab, o_next) { + if (eq(op->op_name, ol->o_name) && + (ol->o_flags & OL_ALIAS)) { + printf("Mapping option %s to %s.\n", + op->op_name, ol->o_file); + op->op_name = ol->o_file; + break; + } + } + } SLIST_FOREACH(ol, &otab, o_next) do_option(ol->o_name); SLIST_FOREACH(op, &opt, op_next) { if (!op->op_ownfile && strncmp(op->op_name, "DEV_", 4)) { printf("%s: unknown option \"%s\"\n", PREFIX, op->op_name); exit(1); } } } /* * Generate an .h file */ static void do_option(char *name) { char *file, *inw; const char *basefile; struct opt_list *ol; struct opt *op; struct opt_head op_head; FILE *inf, *outf; char *value; char *oldvalue; int seen; int tidy; file = tooption(name); - /* * Check to see if the option was specified.. */ value = NULL; SLIST_FOREACH(op, &opt, op_next) { if (eq(name, op->op_name)) { oldvalue = value; value = op->op_value; if (value == NULL) value = ns("1"); if (oldvalue != NULL && !eq(value, oldvalue)) printf( "%s: option \"%s\" redefined from %s to %s\n", PREFIX, op->op_name, oldvalue, value); op->op_ownfile++; } } remember(file); inf = fopen(file, "r"); if (inf == 0) { outf = fopen(file, "w"); if (outf == 0) err(1, "%s", file); /* was the option in the config file? */ if (value) { fprintf(outf, "#define %s %s\n", name, value); } /* else empty file */ (void) fclose(outf); return; } basefile = ""; SLIST_FOREACH(ol, &otab, o_next) if (eq(name, ol->o_name)) { basefile = ol->o_file; break; } oldvalue = NULL; SLIST_INIT(&op_head); seen = 0; tidy = 0; for (;;) { char *cp; char *invalue; /* get the #define */ if ((inw = get_word(inf)) == 0 || inw == (char *)EOF) break; /* get the option name */ if ((inw = get_word(inf)) == 0 || inw == (char *)EOF) break; inw = ns(inw); /* get the option value */ if ((cp = get_word(inf)) == 0 || cp == (char *)EOF) break; /* option value */ invalue = ns(cp); /* malloced */ if (eq(inw, name)) { oldvalue = invalue; invalue = value; seen++; } SLIST_FOREACH(ol, &otab, o_next) if (eq(inw, ol->o_name)) break; if (!eq(inw, name) && !ol) { printf("WARNING: unknown option `%s' removed from %s\n", inw, file); tidy++; } else if (ol != NULL && !eq(basefile, ol->o_file)) { printf("WARNING: option `%s' moved from %s to %s\n", inw, basefile, ol->o_file); tidy++; } else { op = (struct opt *) calloc(1, sizeof *op); if (op == NULL) err(EXIT_FAILURE, "calloc"); op->op_name = inw; op->op_value = invalue; SLIST_INSERT_HEAD(&op_head, op, op_next); } /* EOL? */ cp = get_word(inf); if (cp == (char *)EOF) break; } (void) fclose(inf); if (!tidy && ((value == NULL && oldvalue == NULL) || (value && oldvalue && eq(value, oldvalue)))) { while (!SLIST_EMPTY(&op_head)) { op = SLIST_FIRST(&op_head); SLIST_REMOVE_HEAD(&op_head, op_next); free(op->op_name); free(op->op_value); free(op); } return; } if (value && !seen) { /* New option appears */ op = (struct opt *) calloc(1, sizeof *op); if (op == NULL) err(EXIT_FAILURE, "calloc"); op->op_name = ns(name); op->op_value = value ? ns(value) : NULL; SLIST_INSERT_HEAD(&op_head, op, op_next); } outf = fopen(file, "w"); if (outf == 0) err(1, "%s", file); while (!SLIST_EMPTY(&op_head)) { op = SLIST_FIRST(&op_head); /* was the option in the config file? */ if (op->op_value) { fprintf(outf, "#define %s %s\n", op->op_name, op->op_value); } SLIST_REMOVE_HEAD(&op_head, op_next); free(op->op_name); free(op->op_value); free(op); } (void) fclose(outf); } /* * Find the filename to store the option spec into. */ static char * tooption(char *name) { static char hbuf[MAXPATHLEN]; char nbuf[MAXPATHLEN]; struct opt_list *po; /* "cannot happen"? the otab list should be complete.. */ (void) strlcpy(nbuf, "options.h", sizeof(nbuf)); SLIST_FOREACH(po, &otab, o_next) { if (eq(po->o_name, name)) { strlcpy(nbuf, po->o_file, sizeof(nbuf)); break; } } (void) strlcpy(hbuf, path(nbuf), sizeof(hbuf)); return (hbuf); } /* * read the options and options. files */ static void read_options(void) { FILE *fp; char fname[MAXPATHLEN]; char *wd, *this, *val; struct opt_list *po; int first = 1; char genopt[MAXPATHLEN]; + int flags = 0; SLIST_INIT(&otab); (void) snprintf(fname, sizeof(fname), "../../conf/options"); openit: fp = fopen(fname, "r"); if (fp == 0) { return; } next: + flags = 0; wd = get_word(fp); if (wd == (char *)EOF) { (void) fclose(fp); if (first == 1) { first++; (void) snprintf(fname, sizeof fname, "../../conf/options.%s", machinename); fp = fopen(fname, "r"); if (fp != 0) goto next; (void) snprintf(fname, sizeof fname, "options.%s", machinename); goto openit; } return; } if (wd == 0) goto next; if (wd[0] == '#') { while (((wd = get_word(fp)) != (char *)EOF) && wd) ; goto next; } this = ns(wd); val = get_word(fp); if (val == (char *)EOF) return; if (val == 0) { char *s = ns(this); (void) snprintf(genopt, sizeof(genopt), "opt_%s.h", lower(s)); val = genopt; free(s); + } else if (eq(val, "=")) { + val = get_word(fp); + if (val == (char *)EOF) { + printf("%s: unexpected end of file\n", fname); + exit(1); + } + if (val == 0) { + printf("%s: Expected a right hand side at %s\n", fname, + this); + exit(1); + } + flags |= OL_ALIAS; } val = ns(val); SLIST_FOREACH(po, &otab, o_next) { if (eq(po->o_name, this)) { printf("%s: Duplicate option %s.\n", fname, this); exit(1); } } po = (struct opt_list *) calloc(1, sizeof *po); if (po == NULL) err(EXIT_FAILURE, "calloc"); po->o_name = this; po->o_file = val; + po->o_flags = flags; SLIST_INSERT_HEAD(&otab, po, o_next); goto next; } static char * lower(char *str) { char *cp = str; while (*str) { if (isupper(*str)) *str = tolower(*str); str++; } return (cp); } Index: user/imp/tbemd/usr.sbin/pmcstat/pmcpl_callgraph.c =================================================================== --- user/imp/tbemd/usr.sbin/pmcstat/pmcpl_callgraph.c (revision 206684) +++ user/imp/tbemd/usr.sbin/pmcstat/pmcpl_callgraph.c (revision 206685) @@ -1,683 +1,685 @@ /*- * Copyright (c) 2005-2007, Joseph Koshy * Copyright (c) 2007 The FreeBSD Foundation * All rights reserved. * * Portions of this software were developed by A. Joseph Koshy under * sponsorship from the FreeBSD Foundation and Google, Inc. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ /* * Transform a hwpmc(4) log into human readable form, and into * gprof(1) compatible profiles. */ #include __FBSDID("$FreeBSD$"); #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include "pmcstat.h" #include "pmcstat_log.h" #include "pmcstat_top.h" #include "pmcpl_callgraph.h" /* Get the sample value in percent related to nsamples. */ #define PMCPL_CG_COUNTP(a) \ ((a)->pcg_count * 100.0 / nsamples) /* * The toplevel CG nodes (i.e., with rank == 0) are placed in a hash table. */ struct pmcstat_cgnode_hash_list pmcstat_cgnode_hash[PMCSTAT_NHASH]; int pmcstat_cgnode_hash_count; static pmcstat_interned_string pmcstat_previous_filename_printed; static struct pmcstat_cgnode * pmcstat_cgnode_allocate(struct pmcstat_image *image, uintfptr_t pc) { struct pmcstat_cgnode *cg; if ((cg = malloc(sizeof(*cg))) == NULL) err(EX_OSERR, "ERROR: Cannot allocate callgraph node"); cg->pcg_image = image; cg->pcg_func = pc; cg->pcg_count = 0; cg->pcg_nchildren = 0; LIST_INIT(&cg->pcg_children); return (cg); } /* * Free a node and its children. */ static void pmcstat_cgnode_free(struct pmcstat_cgnode *cg) { struct pmcstat_cgnode *cgc, *cgtmp; LIST_FOREACH_SAFE(cgc, &cg->pcg_children, pcg_sibling, cgtmp) pmcstat_cgnode_free(cgc); free(cg); } /* * Look for a callgraph node associated with pmc `pmcid' in the global * hash table that corresponds to the given `pc' value in the process * `pp'. */ static struct pmcstat_cgnode * pmcstat_cgnode_hash_lookup_pc(struct pmcstat_process *pp, pmc_id_t pmcid, uintfptr_t pc, int usermode) { struct pmcstat_pcmap *ppm; struct pmcstat_symbol *sym; struct pmcstat_image *image; struct pmcstat_cgnode *cg; struct pmcstat_cgnode_hash *h; uintfptr_t loadaddress; unsigned int i, hash; ppm = pmcstat_process_find_map(usermode ? pp : pmcstat_kernproc, pc); if (ppm == NULL) return (NULL); image = ppm->ppm_image; loadaddress = ppm->ppm_lowpc + image->pi_vaddr - image->pi_start; pc -= loadaddress; /* Convert to an offset in the image. */ /* * Try determine the function at this offset. If we can't * find a function round leave the `pc' value alone. */ if ((sym = pmcstat_symbol_search(image, pc)) != NULL) pc = sym->ps_start; for (hash = i = 0; i < sizeof(uintfptr_t); i++) hash += (pc >> i) & 0xFF; hash &= PMCSTAT_HASH_MASK; cg = NULL; LIST_FOREACH(h, &pmcstat_cgnode_hash[hash], pch_next) { if (h->pch_pmcid != pmcid) continue; cg = h->pch_cgnode; assert(cg != NULL); if (cg->pcg_image == image && cg->pcg_func == pc) return (cg); } /* * We haven't seen this (pmcid, pc) tuple yet, so allocate a * new callgraph node and a new hash table entry for it. */ cg = pmcstat_cgnode_allocate(image, pc); if ((h = malloc(sizeof(*h))) == NULL) err(EX_OSERR, "ERROR: Could not allocate callgraph node"); h->pch_pmcid = pmcid; h->pch_cgnode = cg; LIST_INSERT_HEAD(&pmcstat_cgnode_hash[hash], h, pch_next); pmcstat_cgnode_hash_count++; return (cg); } /* * Compare two callgraph nodes for sorting. */ static int pmcstat_cgnode_compare(const void *a, const void *b) { const struct pmcstat_cgnode *const *pcg1, *const *pcg2, *cg1, *cg2; pcg1 = (const struct pmcstat_cgnode *const *) a; cg1 = *pcg1; pcg2 = (const struct pmcstat_cgnode *const *) b; cg2 = *pcg2; /* Sort in reverse order */ if (cg1->pcg_count < cg2->pcg_count) return (1); if (cg1->pcg_count > cg2->pcg_count) return (-1); return (0); } /* * Find (allocating if a needed) a callgraph node in the given * parent with the same (image, pcoffset) pair. */ static struct pmcstat_cgnode * pmcstat_cgnode_find(struct pmcstat_cgnode *parent, struct pmcstat_image *image, uintfptr_t pcoffset) { struct pmcstat_cgnode *child; LIST_FOREACH(child, &parent->pcg_children, pcg_sibling) { if (child->pcg_image == image && child->pcg_func == pcoffset) return (child); } /* * Allocate a new structure. */ child = pmcstat_cgnode_allocate(image, pcoffset); /* * Link it into the parent. */ LIST_INSERT_HEAD(&parent->pcg_children, child, pcg_sibling); parent->pcg_nchildren++; return (child); } /* * Print one callgraph node. The output format is: * * indentation %(parent's samples) #nsamples function@object */ static void pmcstat_cgnode_print(struct pmcstat_cgnode *cg, int depth, uint32_t total) { uint32_t n; const char *space; struct pmcstat_symbol *sym; struct pmcstat_cgnode **sortbuffer, **cgn, *pcg; space = " "; if (depth > 0) (void) fprintf(args.pa_graphfile, "%*s", depth, space); if (cg->pcg_count == total) (void) fprintf(args.pa_graphfile, "100.0%% "); else (void) fprintf(args.pa_graphfile, "%05.2f%% ", 100.0 * cg->pcg_count / total); n = fprintf(args.pa_graphfile, " [%u] ", cg->pcg_count); /* #samples is a 12 character wide field. */ if (n < 12) (void) fprintf(args.pa_graphfile, "%*s", 12 - n, space); if (depth > 0) (void) fprintf(args.pa_graphfile, "%*s", depth, space); sym = pmcstat_symbol_search(cg->pcg_image, cg->pcg_func); if (sym) (void) fprintf(args.pa_graphfile, "%s", pmcstat_string_unintern(sym->ps_name)); else (void) fprintf(args.pa_graphfile, "%p", (void *) (cg->pcg_image->pi_vaddr + cg->pcg_func)); if (pmcstat_previous_filename_printed != cg->pcg_image->pi_fullpath) { pmcstat_previous_filename_printed = cg->pcg_image->pi_fullpath; (void) fprintf(args.pa_graphfile, " @ %s\n", pmcstat_string_unintern( pmcstat_previous_filename_printed)); } else (void) fprintf(args.pa_graphfile, "\n"); if (cg->pcg_nchildren == 0) return; if ((sortbuffer = (struct pmcstat_cgnode **) malloc(sizeof(struct pmcstat_cgnode *) * cg->pcg_nchildren)) == NULL) err(EX_OSERR, "ERROR: Cannot print callgraph"); cgn = sortbuffer; LIST_FOREACH(pcg, &cg->pcg_children, pcg_sibling) *cgn++ = pcg; assert(cgn - sortbuffer == (int) cg->pcg_nchildren); qsort(sortbuffer, cg->pcg_nchildren, sizeof(struct pmcstat_cgnode *), pmcstat_cgnode_compare); for (cgn = sortbuffer, n = 0; n < cg->pcg_nchildren; n++, cgn++) pmcstat_cgnode_print(*cgn, depth+1, cg->pcg_count); free(sortbuffer); } /* * Record a callchain. */ void pmcpl_cg_process(struct pmcstat_process *pp, struct pmcstat_pmcrecord *pmcr, uint32_t nsamples, uintfptr_t *cc, int usermode, uint32_t cpu) { uintfptr_t pc, loadaddress; uint32_t n; struct pmcstat_image *image; struct pmcstat_pcmap *ppm; struct pmcstat_symbol *sym; struct pmcstat_cgnode *parent, *child; struct pmcstat_process *km; pmc_id_t pmcid; (void) cpu; /* * Find the callgraph node recorded in the global hash table * for this (pmcid, pc). */ pc = cc[0]; pmcid = pmcr->pr_pmcid; parent = pmcstat_cgnode_hash_lookup_pc(pp, pmcid, pc, usermode); if (parent == NULL) { pmcstat_stats.ps_callchain_dubious_frames++; pmcr->pr_dubious_frames++; return; } parent->pcg_count++; /* * For each return address in the call chain record, subject * to the maximum depth desired. * - Find the image associated with the sample. Stop if there * there is no valid image at that address. * - Find the function that overlaps the return address. * - If found: use the start address of the function. * If not found (say an object's symbol table is not present or * is incomplete), round down to th gprof bucket granularity. * - Convert return virtual address to an offset in the image. * - Look for a child with the same {offset,image} tuple, * inserting one if needed. * - Increment the count of occurrences of the child. */ km = pmcstat_kernproc; for (n = 1; n < (uint32_t) args.pa_graphdepth && n < nsamples; n++, parent = child) { pc = cc[n]; ppm = pmcstat_process_find_map(usermode ? pp : km, pc); if (ppm == NULL) { /* Detect full frame capture (kernel + user). */ if (!usermode) { ppm = pmcstat_process_find_map(pp, pc); if (ppm != NULL) km = pp; } } if (ppm == NULL) return; image = ppm->ppm_image; loadaddress = ppm->ppm_lowpc + image->pi_vaddr - image->pi_start; pc -= loadaddress; if ((sym = pmcstat_symbol_search(image, pc)) != NULL) pc = sym->ps_start; child = pmcstat_cgnode_find(parent, image, pc); child->pcg_count++; } } /* * Printing a callgraph for a PMC. */ static void pmcstat_callgraph_print_for_pmcid(struct pmcstat_pmcrecord *pmcr) { int n, nentries; uint32_t nsamples; pmc_id_t pmcid; struct pmcstat_cgnode **sortbuffer, **cgn; struct pmcstat_cgnode_hash *pch; /* * We pull out all callgraph nodes in the top-level hash table * with a matching PMC id. We then sort these based on the * frequency of occurrence. Each callgraph node is then * printed. */ nsamples = 0; pmcid = pmcr->pr_pmcid; if ((sortbuffer = (struct pmcstat_cgnode **) malloc(sizeof(struct pmcstat_cgnode *) * pmcstat_cgnode_hash_count)) == NULL) err(EX_OSERR, "ERROR: Cannot sort callgraph"); cgn = sortbuffer; for (n = 0; n < PMCSTAT_NHASH; n++) LIST_FOREACH(pch, &pmcstat_cgnode_hash[n], pch_next) if (pch->pch_pmcid == pmcid) { nsamples += pch->pch_cgnode->pcg_count; *cgn++ = pch->pch_cgnode; } nentries = cgn - sortbuffer; assert(nentries <= pmcstat_cgnode_hash_count); if (nentries == 0) { free(sortbuffer); return; } qsort(sortbuffer, nentries, sizeof(struct pmcstat_cgnode *), pmcstat_cgnode_compare); (void) fprintf(args.pa_graphfile, "@ %s [%u samples]\n\n", pmcstat_string_unintern(pmcr->pr_pmcname), nsamples); for (cgn = sortbuffer, n = 0; n < nentries; n++, cgn++) { pmcstat_previous_filename_printed = NULL; pmcstat_cgnode_print(*cgn, 0, nsamples); (void) fprintf(args.pa_graphfile, "\n"); } free(sortbuffer); } /* * Print out callgraphs. */ static void pmcstat_callgraph_print(void) { struct pmcstat_pmcrecord *pmcr; LIST_FOREACH(pmcr, &pmcstat_pmcs, pr_next) pmcstat_callgraph_print_for_pmcid(pmcr); } static void pmcstat_cgnode_topprint(struct pmcstat_cgnode *cg, int depth, uint32_t nsamples) { int v_attrs, vs_len, ns_len, width, len, n, nchildren; float v; char ns[30], vs[10]; struct pmcstat_symbol *sym; struct pmcstat_cgnode **sortbuffer, **cgn, *pcg; (void) depth; /* Format value. */ v = PMCPL_CG_COUNTP(cg); snprintf(vs, sizeof(vs), "%.1f", v); v_attrs = PMCSTAT_ATTRPERCENT(v); /* Format name. */ sym = pmcstat_symbol_search(cg->pcg_image, cg->pcg_func); if (sym != NULL) { snprintf(ns, sizeof(ns), "%s", pmcstat_string_unintern(sym->ps_name)); } else snprintf(ns, sizeof(ns), "%p", (void *)cg->pcg_func); PMCSTAT_ATTRON(v_attrs); PMCSTAT_PRINTW("%5.5s", vs); PMCSTAT_ATTROFF(v_attrs); PMCSTAT_PRINTW(" %-10.10s %-20.20s", pmcstat_string_unintern(cg->pcg_image->pi_name), ns); nchildren = cg->pcg_nchildren; if (nchildren == 0) { PMCSTAT_PRINTW("\n"); return; } width = pmcstat_displaywidth - 40; if ((sortbuffer = (struct pmcstat_cgnode **) malloc(sizeof(struct pmcstat_cgnode *) * nchildren)) == NULL) err(EX_OSERR, "ERROR: Cannot print callgraph"); cgn = sortbuffer; LIST_FOREACH(pcg, &cg->pcg_children, pcg_sibling) *cgn++ = pcg; assert(cgn - sortbuffer == (int)nchildren); qsort(sortbuffer, nchildren, sizeof(struct pmcstat_cgnode *), pmcstat_cgnode_compare); /* Count how many callers. */ for (cgn = sortbuffer, n = 0; n < nchildren; n++, cgn++) { pcg = *cgn; v = PMCPL_CG_COUNTP(pcg); if (v < pmcstat_threshold) break; } nchildren = n; for (cgn = sortbuffer, n = 0; n < nchildren; n++, cgn++) { pcg = *cgn; /* Format value. */ if (nchildren > 1) { v = PMCPL_CG_COUNTP(pcg); vs_len = snprintf(vs, sizeof(vs), ":%.1f", v); v_attrs = PMCSTAT_ATTRPERCENT(v); } else vs_len = 0; /* Format name. */ sym = pmcstat_symbol_search(pcg->pcg_image, pcg->pcg_func); if (sym != NULL) { ns_len = snprintf(ns, sizeof(ns), "%s", pmcstat_string_unintern(sym->ps_name)); } else ns_len = snprintf(ns, sizeof(ns), "%p", (void *)pcg->pcg_func); len = ns_len + vs_len + 1; if (width - len < 0) { PMCSTAT_PRINTW(" ..."); break; } width -= len; PMCSTAT_PRINTW(" %s", ns); if (nchildren > 1) { PMCSTAT_ATTRON(v_attrs); PMCSTAT_PRINTW("%s", vs); PMCSTAT_ATTROFF(v_attrs); } } PMCSTAT_PRINTW("\n"); free(sortbuffer); } /* * Top mode display. */ void pmcpl_cg_topdisplay(void) { int n, nentries; uint32_t nsamples; struct pmcstat_cgnode **sortbuffer, **cgn; struct pmcstat_cgnode_hash *pch; struct pmcstat_pmcrecord *pmcr; pmcr = pmcstat_pmcindex_to_pmcr(pmcstat_pmcinfilter); + if (!pmcr) + err(EX_SOFTWARE, "ERROR: invalid pmcindex"); /* * We pull out all callgraph nodes in the top-level hash table * with a matching PMC index. We then sort these based on the * frequency of occurrence. Each callgraph node is then * printed. */ nsamples = 0; if ((sortbuffer = (struct pmcstat_cgnode **) malloc(sizeof(struct pmcstat_cgnode *) * pmcstat_cgnode_hash_count)) == NULL) err(EX_OSERR, "ERROR: Cannot sort callgraph"); cgn = sortbuffer; for (n = 0; n < PMCSTAT_NHASH; n++) LIST_FOREACH(pch, &pmcstat_cgnode_hash[n], pch_next) if (pmcr == NULL || pch->pch_pmcid == pmcr->pr_pmcid) { nsamples += pch->pch_cgnode->pcg_count; *cgn++ = pch->pch_cgnode; } nentries = cgn - sortbuffer; assert(nentries <= pmcstat_cgnode_hash_count); if (nentries == 0) { free(sortbuffer); return; } qsort(sortbuffer, nentries, sizeof(struct pmcstat_cgnode *), pmcstat_cgnode_compare); PMCSTAT_PRINTW("%5.5s %-10.10s %-20.20s %s\n", "%SAMP", "IMAGE", "FUNCTION", "CALLERS"); nentries = min(pmcstat_displayheight - 2, nentries); for (cgn = sortbuffer, n = 0; n < nentries; n++, cgn++) { if (PMCPL_CG_COUNTP(*cgn) < pmcstat_threshold) break; pmcstat_cgnode_topprint(*cgn, 0, nsamples); } free(sortbuffer); } /* * Handle top mode keypress. */ int pmcpl_cg_topkeypress(int c, WINDOW *w) { (void) c; (void) w; return 0; } int pmcpl_cg_init(void) { int i; pmcstat_cgnode_hash_count = 0; pmcstat_previous_filename_printed = NULL; for (i = 0; i < PMCSTAT_NHASH; i++) { LIST_INIT(&pmcstat_cgnode_hash[i]); } return (0); } void pmcpl_cg_shutdown(FILE *mf) { int i; struct pmcstat_cgnode_hash *pch, *pchtmp; (void) mf; if (args.pa_flags & FLAG_DO_CALLGRAPHS) pmcstat_callgraph_print(); /* * Free memory. */ for (i = 0; i < PMCSTAT_NHASH; i++) { LIST_FOREACH_SAFE(pch, &pmcstat_cgnode_hash[i], pch_next, pchtmp) { pmcstat_cgnode_free(pch->pch_cgnode); LIST_REMOVE(pch, pch_next); free(pch); } } } Index: user/imp/tbemd/usr.sbin/pmcstat/pmcpl_gprof.c =================================================================== --- user/imp/tbemd/usr.sbin/pmcstat/pmcpl_gprof.c (revision 206684) +++ user/imp/tbemd/usr.sbin/pmcstat/pmcpl_gprof.c (revision 206685) @@ -1,533 +1,535 @@ /*- * Copyright (c) 2005-2007, Joseph Koshy * Copyright (c) 2007 The FreeBSD Foundation * Copyright (c) 2009, Fabien Thomas * All rights reserved. * * Portions of this software were developed by A. Joseph Koshy under * sponsorship from the FreeBSD Foundation and Google, Inc. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ /* * Transform a hwpmc(4) log into human readable form, and into * gprof(1) compatible profiles. */ #include __FBSDID("$FreeBSD$"); #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include "pmcstat.h" #include "pmcstat_log.h" #include "pmcpl_callgraph.h" #include "pmcpl_gprof.h" /* * struct pmcstat_gmonfile tracks a given 'gmon.out' file. These * files are mmap()'ed in as needed. */ struct pmcstat_gmonfile { LIST_ENTRY(pmcstat_gmonfile) pgf_next; /* list of entries */ int pgf_overflow; /* whether a count overflowed */ pmc_id_t pgf_pmcid; /* id of the associated pmc */ size_t pgf_nbuckets; /* #buckets in this gmon.out */ unsigned int pgf_nsamples; /* #samples in this gmon.out */ pmcstat_interned_string pgf_name; /* pathname of gmon.out file */ size_t pgf_ndatabytes; /* number of bytes mapped */ void *pgf_gmondata; /* pointer to mmap'ed data */ FILE *pgf_file; /* used when writing gmon arcs */ }; /* * Prototypes */ static void pmcstat_gmon_create_file(struct pmcstat_gmonfile *_pgf, struct pmcstat_image *_image); static pmcstat_interned_string pmcstat_gmon_create_name(const char *_sd, struct pmcstat_image *_img, pmc_id_t _pmcid); static void pmcstat_gmon_map_file(struct pmcstat_gmonfile *_pgf); static void pmcstat_gmon_unmap_file(struct pmcstat_gmonfile *_pgf); static struct pmcstat_gmonfile *pmcstat_image_find_gmonfile(struct pmcstat_image *_i, pmc_id_t _id); /* * Create a gmon.out file and size it. */ static void pmcstat_gmon_create_file(struct pmcstat_gmonfile *pgf, struct pmcstat_image *image) { int fd; size_t count; struct gmonhdr gm; const char *pathname; char buffer[DEFAULT_BUFFER_SIZE]; pathname = pmcstat_string_unintern(pgf->pgf_name); if ((fd = open(pathname, O_RDWR|O_NOFOLLOW|O_CREAT, S_IRUSR|S_IWUSR|S_IRGRP|S_IROTH)) < 0) err(EX_OSERR, "ERROR: Cannot open \"%s\"", pathname); gm.lpc = image->pi_start; gm.hpc = image->pi_end; gm.ncnt = (pgf->pgf_nbuckets * sizeof(HISTCOUNTER)) + sizeof(struct gmonhdr); gm.version = GMONVERSION; gm.profrate = 0; /* use ticks */ gm.histcounter_type = 0; /* compatibility with moncontrol() */ gm.spare[0] = gm.spare[1] = 0; /* Write out the gmon header */ if (write(fd, &gm, sizeof(gm)) < 0) goto error; /* Zero fill the samples[] array */ (void) memset(buffer, 0, sizeof(buffer)); count = pgf->pgf_ndatabytes - sizeof(struct gmonhdr); while (count > sizeof(buffer)) { if (write(fd, &buffer, sizeof(buffer)) < 0) goto error; count -= sizeof(buffer); } if (write(fd, &buffer, count) < 0) goto error; (void) close(fd); return; error: err(EX_OSERR, "ERROR: Cannot write \"%s\"", pathname); } /* * Determine the full pathname of a gmon.out file for a given * (image,pmcid) combination. Return the interned string. */ pmcstat_interned_string pmcstat_gmon_create_name(const char *samplesdir, struct pmcstat_image *image, pmc_id_t pmcid) { const char *pmcname; char fullpath[PATH_MAX]; pmcname = pmcstat_pmcid_to_name(pmcid); + if (!pmcname) + err(EX_SOFTWARE, "ERROR: cannot find pmcid"); (void) snprintf(fullpath, sizeof(fullpath), "%s/%s/%s", samplesdir, pmcname, pmcstat_string_unintern(image->pi_samplename)); return (pmcstat_string_intern(fullpath)); } /* * Mmap in a gmon.out file for processing. */ static void pmcstat_gmon_map_file(struct pmcstat_gmonfile *pgf) { int fd; const char *pathname; pathname = pmcstat_string_unintern(pgf->pgf_name); /* the gmon.out file must already exist */ if ((fd = open(pathname, O_RDWR | O_NOFOLLOW, 0)) < 0) err(EX_OSERR, "ERROR: cannot open \"%s\"", pathname); pgf->pgf_gmondata = mmap(NULL, pgf->pgf_ndatabytes, PROT_READ|PROT_WRITE, MAP_NOSYNC|MAP_SHARED, fd, 0); if (pgf->pgf_gmondata == MAP_FAILED) err(EX_OSERR, "ERROR: cannot map \"%s\"", pathname); (void) close(fd); } /* * Unmap a gmon.out file after sync'ing its data to disk. */ static void pmcstat_gmon_unmap_file(struct pmcstat_gmonfile *pgf) { (void) msync(pgf->pgf_gmondata, pgf->pgf_ndatabytes, MS_SYNC); (void) munmap(pgf->pgf_gmondata, pgf->pgf_ndatabytes); pgf->pgf_gmondata = NULL; } static void pmcstat_gmon_append_arc(struct pmcstat_image *image, pmc_id_t pmcid, uintptr_t rawfrom, uintptr_t rawto, uint32_t count) { struct rawarc arc; /* from */ const char *pathname; struct pmcstat_gmonfile *pgf; if ((pgf = pmcstat_image_find_gmonfile(image, pmcid)) == NULL) return; if (pgf->pgf_file == NULL) { pathname = pmcstat_string_unintern(pgf->pgf_name); if ((pgf->pgf_file = fopen(pathname, "a")) == NULL) return; } arc.raw_frompc = rawfrom + image->pi_vaddr; arc.raw_selfpc = rawto + image->pi_vaddr; arc.raw_count = count; (void) fwrite(&arc, sizeof(arc), 1, pgf->pgf_file); } static struct pmcstat_gmonfile * pmcstat_image_find_gmonfile(struct pmcstat_image *image, pmc_id_t pmcid) { struct pmcstat_gmonfile *pgf; LIST_FOREACH(pgf, &image->pi_gmlist, pgf_next) if (pgf->pgf_pmcid == pmcid) return (pgf); return (NULL); } static void pmcstat_cgnode_do_gmon_arcs(struct pmcstat_cgnode *cg, pmc_id_t pmcid) { struct pmcstat_cgnode *cgc; /* * Look for child nodes that belong to the same image. */ LIST_FOREACH(cgc, &cg->pcg_children, pcg_sibling) { if (cgc->pcg_image == cg->pcg_image) pmcstat_gmon_append_arc(cg->pcg_image, pmcid, cgc->pcg_func, cg->pcg_func, cgc->pcg_count); if (cgc->pcg_nchildren > 0) pmcstat_cgnode_do_gmon_arcs(cgc, pmcid); } } static void pmcstat_callgraph_do_gmon_arcs_for_pmcid(pmc_id_t pmcid) { int n; struct pmcstat_cgnode_hash *pch; for (n = 0; n < PMCSTAT_NHASH; n++) LIST_FOREACH(pch, &pmcstat_cgnode_hash[n], pch_next) if (pch->pch_pmcid == pmcid && pch->pch_cgnode->pcg_nchildren > 1) pmcstat_cgnode_do_gmon_arcs(pch->pch_cgnode, pmcid); } static void pmcstat_callgraph_do_gmon_arcs(void) { struct pmcstat_pmcrecord *pmcr; LIST_FOREACH(pmcr, &pmcstat_pmcs, pr_next) pmcstat_callgraph_do_gmon_arcs_for_pmcid(pmcr->pr_pmcid); } void pmcpl_gmon_initimage(struct pmcstat_image *pi) { int count, nlen; char *sn; char name[NAME_MAX]; /* * Look for a suitable name for the sample files associated * with this image: if `basename(path)`+".gmon" is available, * we use that, otherwise we try iterating through * `basename(path)`+ "~" + NNN + ".gmon" till we get a free * entry. */ if ((sn = basename(pmcstat_string_unintern(pi->pi_execpath))) == NULL) err(EX_OSERR, "ERROR: Cannot process \"%s\"", pmcstat_string_unintern(pi->pi_execpath)); nlen = strlen(sn); nlen = min(nlen, (int) (sizeof(name) - sizeof(".gmon"))); snprintf(name, sizeof(name), "%.*s.gmon", nlen, sn); /* try use the unabridged name first */ if (pmcstat_string_lookup(name) == NULL) pi->pi_samplename = pmcstat_string_intern(name); else { /* * Otherwise use a prefix from the original name and * upto 3 digits. */ nlen = strlen(sn); nlen = min(nlen, (int) (sizeof(name)-sizeof("~NNN.gmon"))); count = 0; do { if (++count > 999) errx(EX_CANTCREAT, "ERROR: cannot create a " "gmon file for \"%s\"", name); snprintf(name, sizeof(name), "%.*s~%3.3d.gmon", nlen, sn, count); if (pmcstat_string_lookup(name) == NULL) { pi->pi_samplename = pmcstat_string_intern(name); count = 0; } } while (count > 0); } LIST_INIT(&pi->pi_gmlist); } void pmcpl_gmon_shutdownimage(struct pmcstat_image *pi) { struct pmcstat_gmonfile *pgf, *pgftmp; LIST_FOREACH_SAFE(pgf, &pi->pi_gmlist, pgf_next, pgftmp) { if (pgf->pgf_file) (void) fclose(pgf->pgf_file); LIST_REMOVE(pgf, pgf_next); free(pgf); } } void pmcpl_gmon_newpmc(pmcstat_interned_string ps, struct pmcstat_pmcrecord *pr) { struct stat st; char fullpath[PATH_MAX]; (void) pr; /* * Create the appropriate directory to hold gmon.out files. */ (void) snprintf(fullpath, sizeof(fullpath), "%s/%s", args.pa_samplesdir, pmcstat_string_unintern(ps)); /* If the path name exists, it should be a directory */ if (stat(fullpath, &st) == 0 && S_ISDIR(st.st_mode)) return; if (mkdir(fullpath, S_IRWXU|S_IRGRP|S_IXGRP|S_IROTH|S_IXOTH) < 0) err(EX_OSERR, "ERROR: Cannot create directory \"%s\"", fullpath); } /* * Increment the bucket in the gmon.out file corresponding to 'pmcid' * and 'pc'. */ void pmcpl_gmon_process(struct pmcstat_process *pp, struct pmcstat_pmcrecord *pmcr, uint32_t nsamples, uintfptr_t *cc, int usermode, uint32_t cpu) { struct pmcstat_pcmap *map; struct pmcstat_image *image; struct pmcstat_gmonfile *pgf; uintfptr_t bucket; HISTCOUNTER *hc; pmc_id_t pmcid; (void) nsamples; (void) usermode; (void) cpu; map = pmcstat_process_find_map(usermode ? pp : pmcstat_kernproc, cc[0]); if (map == NULL) { /* Unknown offset. */ pmcstat_stats.ps_samples_unknown_offset++; return; } assert(cc[0] >= map->ppm_lowpc && cc[0] < map->ppm_highpc); image = map->ppm_image; pmcid = pmcr->pr_pmcid; /* * If this is the first time we are seeing a sample for * this executable image, try determine its parameters. */ if (image->pi_type == PMCSTAT_IMAGE_UNKNOWN) pmcstat_image_determine_type(image); assert(image->pi_type != PMCSTAT_IMAGE_UNKNOWN); /* Ignore samples in images that we know nothing about. */ if (image->pi_type == PMCSTAT_IMAGE_INDETERMINABLE) { pmcstat_stats.ps_samples_indeterminable++; return; } /* * Find the gmon file corresponding to 'pmcid', creating it if * needed. */ pgf = pmcstat_image_find_gmonfile(image, pmcid); if (pgf == NULL) { if ((pgf = calloc(1, sizeof(*pgf))) == NULL) err(EX_OSERR, "ERROR:"); pgf->pgf_gmondata = NULL; /* mark as unmapped */ pgf->pgf_name = pmcstat_gmon_create_name(args.pa_samplesdir, image, pmcid); pgf->pgf_pmcid = pmcid; assert(image->pi_end > image->pi_start); pgf->pgf_nbuckets = (image->pi_end - image->pi_start) / FUNCTION_ALIGNMENT; /* see */ pgf->pgf_ndatabytes = sizeof(struct gmonhdr) + pgf->pgf_nbuckets * sizeof(HISTCOUNTER); pgf->pgf_nsamples = 0; pgf->pgf_file = NULL; pmcstat_gmon_create_file(pgf, image); LIST_INSERT_HEAD(&image->pi_gmlist, pgf, pgf_next); } /* * Map the gmon file in if needed. It may have been mapped * out under memory pressure. */ if (pgf->pgf_gmondata == NULL) pmcstat_gmon_map_file(pgf); assert(pgf->pgf_gmondata != NULL); /* * */ bucket = (cc[0] - map->ppm_lowpc) / FUNCTION_ALIGNMENT; assert(bucket < pgf->pgf_nbuckets); hc = (HISTCOUNTER *) ((uintptr_t) pgf->pgf_gmondata + sizeof(struct gmonhdr)); /* saturating add */ if (hc[bucket] < 0xFFFFU) /* XXX tie this to sizeof(HISTCOUNTER) */ hc[bucket]++; else /* mark that an overflow occurred */ pgf->pgf_overflow = 1; pgf->pgf_nsamples++; } /* * Shutdown module. */ void pmcpl_gmon_shutdown(FILE *mf) { int i; struct pmcstat_gmonfile *pgf; struct pmcstat_image *pi; /* * Sync back all gprof flat profile data. */ for (i = 0; i < PMCSTAT_NHASH; i++) { LIST_FOREACH(pi, &pmcstat_image_hash[i], pi_next) { if (mf) (void) fprintf(mf, " \"%s\" => \"%s\"", pmcstat_string_unintern(pi->pi_execpath), pmcstat_string_unintern( pi->pi_samplename)); /* flush gmon.out data to disk */ LIST_FOREACH(pgf, &pi->pi_gmlist, pgf_next) { pmcstat_gmon_unmap_file(pgf); if (mf) (void) fprintf(mf, " %s/%d", pmcstat_pmcid_to_name( pgf->pgf_pmcid), pgf->pgf_nsamples); if (pgf->pgf_overflow && args.pa_verbosity >= 1) warnx("WARNING: profile \"%s\" " "overflowed.", pmcstat_string_unintern( pgf->pgf_name)); } if (mf) (void) fprintf(mf, "\n"); } } /* * Compute arcs and add these to the gprof files. */ if (args.pa_flags & FLAG_DO_GPROF && args.pa_graphdepth > 1) pmcstat_callgraph_do_gmon_arcs(); } Index: user/imp/tbemd/usr.sbin/pmcstat/pmcstat_log.c =================================================================== --- user/imp/tbemd/usr.sbin/pmcstat/pmcstat_log.c (revision 206684) +++ user/imp/tbemd/usr.sbin/pmcstat/pmcstat_log.c (revision 206685) @@ -1,2145 +1,2143 @@ /*- * Copyright (c) 2005-2007, Joseph Koshy * Copyright (c) 2007 The FreeBSD Foundation * All rights reserved. * * Portions of this software were developed by A. Joseph Koshy under * sponsorship from the FreeBSD Foundation and Google, Inc. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ /* * Transform a hwpmc(4) log into human readable form, and into * gprof(1) compatible profiles. */ #include __FBSDID("$FreeBSD$"); #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include "pmcstat.h" #include "pmcstat_log.h" #include "pmcstat_top.h" #define PMCSTAT_ALLOCATE 1 /* * PUBLIC INTERFACES * * pmcstat_initialize_logging() initialize this module, called first * pmcstat_shutdown_logging() orderly shutdown, called last * pmcstat_open_log() open an eventlog for processing * pmcstat_process_log() print/convert an event log * pmcstat_display_log() top mode display for the log * pmcstat_close_log() finish processing an event log * * IMPLEMENTATION NOTES * * We correlate each 'callchain' or 'sample' entry seen in the event * log back to an executable object in the system. Executable objects * include: * - program executables, * - shared libraries loaded by the runtime loader, * - dlopen()'ed objects loaded by the program, * - the runtime loader itself, * - the kernel and kernel modules. * * Each process that we know about is treated as a set of regions that * map to executable objects. Processes are described by * 'pmcstat_process' structures. Executable objects are tracked by * 'pmcstat_image' structures. The kernel and kernel modules are * common to all processes (they reside at the same virtual addresses * for all processes). Individual processes can have their text * segments and shared libraries loaded at process-specific locations. * * A given executable object can be in use by multiple processes * (e.g., libc.so) and loaded at a different address in each. * pmcstat_pcmap structures track per-image mappings. * * The sample log could have samples from multiple PMCs; we * generate one 'gmon.out' profile per PMC. * * IMPLEMENTATION OF GMON OUTPUT * * Each executable object gets one 'gmon.out' profile, per PMC in * use. Creation of 'gmon.out' profiles is done lazily. The * 'gmon.out' profiles generated for a given sampling PMC are * aggregates of all the samples for that particular executable * object. * * IMPLEMENTATION OF SYSTEM-WIDE CALLGRAPH OUTPUT * * Each active pmcid has its own callgraph structure, described by a * 'struct pmcstat_callgraph'. Given a process id and a list of pc * values, we map each pc value to a tuple (image, symbol), where * 'image' denotes an executable object and 'symbol' is the closest * symbol that precedes the pc value. Each pc value in the list is * also given a 'rank' that reflects its depth in the call stack. */ struct pmcstat_pmcs pmcstat_pmcs = LIST_HEAD_INITIALIZER(pmcstat_pmcs); /* * All image descriptors are kept in a hash table. */ struct pmcstat_image_hash_list pmcstat_image_hash[PMCSTAT_NHASH]; /* * All process descriptors are kept in a hash table. */ struct pmcstat_process_hash_list pmcstat_process_hash[PMCSTAT_NHASH]; struct pmcstat_stats pmcstat_stats; /* statistics */ struct pmcstat_process *pmcstat_kernproc; /* kernel 'process' */ #include "pmcpl_gprof.h" #include "pmcpl_callgraph.h" #include "pmcpl_annotate.h" #include "pmcpl_calltree.h" struct pmc_plugins { const char *pl_name; /* name */ /* configure */ int (*pl_configure)(char *opt); /* init and shutdown */ int (*pl_init)(void); void (*pl_shutdown)(FILE *mf); /* sample processing */ void (*pl_process)(struct pmcstat_process *pp, struct pmcstat_pmcrecord *pmcr, uint32_t nsamples, uintfptr_t *cc, int usermode, uint32_t cpu); /* image */ void (*pl_initimage)(struct pmcstat_image *pi); void (*pl_shutdownimage)(struct pmcstat_image *pi); /* pmc */ void (*pl_newpmc)(pmcstat_interned_string ps, struct pmcstat_pmcrecord *pr); /* top display */ void (*pl_topdisplay)(void); /* top keypress */ int (*pl_topkeypress)(int c, WINDOW *w); } plugins[] = { { .pl_name = "none", }, { .pl_name = "callgraph", .pl_init = pmcpl_cg_init, .pl_shutdown = pmcpl_cg_shutdown, .pl_process = pmcpl_cg_process, .pl_topkeypress = pmcpl_cg_topkeypress, .pl_topdisplay = pmcpl_cg_topdisplay }, { .pl_name = "gprof", .pl_shutdown = pmcpl_gmon_shutdown, .pl_process = pmcpl_gmon_process, .pl_initimage = pmcpl_gmon_initimage, .pl_shutdownimage = pmcpl_gmon_shutdownimage, .pl_newpmc = pmcpl_gmon_newpmc }, { .pl_name = "annotate", .pl_process = pmcpl_annotate_process }, { .pl_name = "calltree", .pl_configure = pmcpl_ct_configure, .pl_init = pmcpl_ct_init, .pl_shutdown = pmcpl_ct_shutdown, .pl_process = pmcpl_ct_process, .pl_topkeypress = pmcpl_ct_topkeypress, .pl_topdisplay = pmcpl_ct_topdisplay }, { .pl_name = NULL } }; int pmcstat_mergepmc; int pmcstat_pmcinfilter = 0; /* PMC filter for top mode. */ float pmcstat_threshold = 0.5; /* Cost filter for top mode. */ /* * Prototypes */ static struct pmcstat_image *pmcstat_image_from_path(pmcstat_interned_string _path, int _iskernelmodule); static void pmcstat_image_get_aout_params(struct pmcstat_image *_image); static void pmcstat_image_get_elf_params(struct pmcstat_image *_image); static void pmcstat_image_link(struct pmcstat_process *_pp, struct pmcstat_image *_i, uintfptr_t _lpc); static void pmcstat_pmcid_add(pmc_id_t _pmcid, pmcstat_interned_string _name); static void pmcstat_process_aout_exec(struct pmcstat_process *_pp, struct pmcstat_image *_image, uintfptr_t _entryaddr); static void pmcstat_process_elf_exec(struct pmcstat_process *_pp, struct pmcstat_image *_image, uintfptr_t _entryaddr); static void pmcstat_process_exec(struct pmcstat_process *_pp, pmcstat_interned_string _path, uintfptr_t _entryaddr); static struct pmcstat_process *pmcstat_process_lookup(pid_t _pid, int _allocate); static int pmcstat_string_compute_hash(const char *_string); static void pmcstat_string_initialize(void); static int pmcstat_string_lookup_hash(pmcstat_interned_string _is); static void pmcstat_string_shutdown(void); static void pmcstat_stats_reset(void); /* * A simple implementation of interned strings. Each interned string * is assigned a unique address, so that subsequent string compares * can be done by a simple pointer comparision instead of using * strcmp(). This speeds up hash table lookups and saves memory if * duplicate strings are the norm. */ struct pmcstat_string { LIST_ENTRY(pmcstat_string) ps_next; /* hash link */ int ps_len; int ps_hash; char *ps_string; }; static LIST_HEAD(,pmcstat_string) pmcstat_string_hash[PMCSTAT_NHASH]; /* * PMC count. */ int pmcstat_npmcs; /* * PMC Top mode pause state. */ int pmcstat_pause; static void pmcstat_stats_reset(void) { struct pmcstat_pmcrecord *pr; /* Flush PMCs stats. */ LIST_FOREACH(pr, &pmcstat_pmcs, pr_next) { pr->pr_samples = 0; pr->pr_dubious_frames = 0; } /* Flush global stats. */ bzero(&pmcstat_stats, sizeof(struct pmcstat_stats)); } /* * Compute a 'hash' value for a string. */ static int pmcstat_string_compute_hash(const char *s) { int hash; for (hash = 0; *s; s++) hash ^= *s; return (hash & PMCSTAT_HASH_MASK); } /* * Intern a copy of string 's', and return a pointer to the * interned structure. */ pmcstat_interned_string pmcstat_string_intern(const char *s) { struct pmcstat_string *ps; const struct pmcstat_string *cps; int hash, len; if ((cps = pmcstat_string_lookup(s)) != NULL) return (cps); hash = pmcstat_string_compute_hash(s); len = strlen(s); if ((ps = malloc(sizeof(*ps))) == NULL) err(EX_OSERR, "ERROR: Could not intern string"); ps->ps_len = len; ps->ps_hash = hash; ps->ps_string = strdup(s); LIST_INSERT_HEAD(&pmcstat_string_hash[hash], ps, ps_next); return ((pmcstat_interned_string) ps); } const char * pmcstat_string_unintern(pmcstat_interned_string str) { const char *s; s = ((const struct pmcstat_string *) str)->ps_string; return (s); } pmcstat_interned_string pmcstat_string_lookup(const char *s) { struct pmcstat_string *ps; int hash, len; hash = pmcstat_string_compute_hash(s); len = strlen(s); LIST_FOREACH(ps, &pmcstat_string_hash[hash], ps_next) if (ps->ps_len == len && ps->ps_hash == hash && strcmp(ps->ps_string, s) == 0) return (ps); return (NULL); } static int pmcstat_string_lookup_hash(pmcstat_interned_string s) { const struct pmcstat_string *ps; ps = (const struct pmcstat_string *) s; return (ps->ps_hash); } /* * Initialize the string interning facility. */ static void pmcstat_string_initialize(void) { int i; for (i = 0; i < PMCSTAT_NHASH; i++) LIST_INIT(&pmcstat_string_hash[i]); } /* * Destroy the string table, free'ing up space. */ static void pmcstat_string_shutdown(void) { int i; struct pmcstat_string *ps, *pstmp; for (i = 0; i < PMCSTAT_NHASH; i++) LIST_FOREACH_SAFE(ps, &pmcstat_string_hash[i], ps_next, pstmp) { LIST_REMOVE(ps, ps_next); free(ps->ps_string); free(ps); } } /* * Determine whether a given executable image is an A.OUT object, and * if so, fill in its parameters from the text file. * Sets image->pi_type. */ static void pmcstat_image_get_aout_params(struct pmcstat_image *image) { int fd; ssize_t nbytes; struct exec ex; const char *path; char buffer[PATH_MAX]; path = pmcstat_string_unintern(image->pi_execpath); assert(path != NULL); if (image->pi_iskernelmodule) errx(EX_SOFTWARE, "ERROR: a.out kernel modules are " "unsupported \"%s\"", path); (void) snprintf(buffer, sizeof(buffer), "%s%s", args.pa_fsroot, path); if ((fd = open(buffer, O_RDONLY, 0)) < 0 || (nbytes = read(fd, &ex, sizeof(ex))) < 0) { warn("WARNING: Cannot determine type of \"%s\"", path); image->pi_type = PMCSTAT_IMAGE_INDETERMINABLE; if (fd != -1) (void) close(fd); return; } (void) close(fd); if ((unsigned) nbytes != sizeof(ex) || N_BADMAG(ex)) return; image->pi_type = PMCSTAT_IMAGE_AOUT; /* TODO: the rest of a.out processing */ return; } /* * Helper function. */ static int pmcstat_symbol_compare(const void *a, const void *b) { const struct pmcstat_symbol *sym1, *sym2; sym1 = (const struct pmcstat_symbol *) a; sym2 = (const struct pmcstat_symbol *) b; if (sym1->ps_end <= sym2->ps_start) return (-1); if (sym1->ps_start >= sym2->ps_end) return (1); return (0); } /* * Map an address to a symbol in an image. */ struct pmcstat_symbol * pmcstat_symbol_search(struct pmcstat_image *image, uintfptr_t addr) { struct pmcstat_symbol sym; if (image->pi_symbols == NULL) return (NULL); sym.ps_name = NULL; sym.ps_start = addr; sym.ps_end = addr + 1; return (bsearch((void *) &sym, image->pi_symbols, image->pi_symcount, sizeof(struct pmcstat_symbol), pmcstat_symbol_compare)); } /* * Add the list of symbols in the given section to the list associated * with the object. */ static void pmcstat_image_add_symbols(struct pmcstat_image *image, Elf *e, Elf_Scn *scn, GElf_Shdr *sh) { int firsttime; size_t n, newsyms, nshsyms, nfuncsyms; struct pmcstat_symbol *symptr; char *fnname; GElf_Sym sym; Elf_Data *data; if ((data = elf_getdata(scn, NULL)) == NULL) return; /* * Determine the number of functions named in this * section. */ nshsyms = sh->sh_size / sh->sh_entsize; for (n = nfuncsyms = 0; n < nshsyms; n++) { if (gelf_getsym(data, (int) n, &sym) != &sym) return; if (GELF_ST_TYPE(sym.st_info) == STT_FUNC) nfuncsyms++; } if (nfuncsyms == 0) return; /* * Allocate space for the new entries. */ firsttime = image->pi_symbols == NULL; symptr = realloc(image->pi_symbols, sizeof(*symptr) * (image->pi_symcount + nfuncsyms)); if (symptr == image->pi_symbols) /* realloc() failed. */ return; image->pi_symbols = symptr; /* * Append new symbols to the end of the current table. */ symptr += image->pi_symcount; for (n = newsyms = 0; n < nshsyms; n++) { if (gelf_getsym(data, (int) n, &sym) != &sym) return; if (GELF_ST_TYPE(sym.st_info) != STT_FUNC) continue; if (!firsttime && pmcstat_symbol_search(image, sym.st_value)) continue; /* We've seen this symbol already. */ if ((fnname = elf_strptr(e, sh->sh_link, sym.st_name)) == NULL) continue; symptr->ps_name = pmcstat_string_intern(fnname); symptr->ps_start = sym.st_value - image->pi_vaddr; symptr->ps_end = symptr->ps_start + sym.st_size; symptr++; newsyms++; } image->pi_symcount += newsyms; assert(newsyms <= nfuncsyms); /* * Return space to the system if there were duplicates. */ if (newsyms < nfuncsyms) image->pi_symbols = realloc(image->pi_symbols, sizeof(*symptr) * image->pi_symcount); /* * Keep the list of symbols sorted. */ qsort(image->pi_symbols, image->pi_symcount, sizeof(*symptr), pmcstat_symbol_compare); /* * Deal with function symbols that have a size of 'zero' by * making them extend to the next higher address. These * symbols are usually defined in assembly code. */ for (symptr = image->pi_symbols; symptr < image->pi_symbols + (image->pi_symcount - 1); symptr++) if (symptr->ps_start == symptr->ps_end) symptr->ps_end = (symptr+1)->ps_start; } /* * Examine an ELF file to determine the size of its text segment. * Sets image->pi_type if anything conclusive can be determined about * this image. */ static void pmcstat_image_get_elf_params(struct pmcstat_image *image) { int fd; size_t i, nph, nsh; const char *path, *elfbase; char *p, *endp; uintfptr_t minva, maxva; Elf *e; Elf_Scn *scn; GElf_Ehdr eh; GElf_Phdr ph; GElf_Shdr sh; enum pmcstat_image_type image_type; char buffer[PATH_MAX]; assert(image->pi_type == PMCSTAT_IMAGE_UNKNOWN); image->pi_start = minva = ~(uintfptr_t) 0; image->pi_end = maxva = (uintfptr_t) 0; image->pi_type = image_type = PMCSTAT_IMAGE_INDETERMINABLE; image->pi_isdynamic = 0; image->pi_dynlinkerpath = NULL; image->pi_vaddr = 0; path = pmcstat_string_unintern(image->pi_execpath); assert(path != NULL); /* * Look for kernel modules under FSROOT/KERNELPATH/NAME, * and user mode executable objects under FSROOT/PATHNAME. */ if (image->pi_iskernelmodule) (void) snprintf(buffer, sizeof(buffer), "%s%s/%s", args.pa_fsroot, args.pa_kernel, path); else (void) snprintf(buffer, sizeof(buffer), "%s%s", args.pa_fsroot, path); e = NULL; if ((fd = open(buffer, O_RDONLY, 0)) < 0 || (e = elf_begin(fd, ELF_C_READ, NULL)) == NULL || (elf_kind(e) != ELF_K_ELF)) { warnx("WARNING: Cannot determine the type of \"%s\".", buffer); goto done; } if (gelf_getehdr(e, &eh) != &eh) { warnx("WARNING: Cannot retrieve the ELF Header for " "\"%s\": %s.", buffer, elf_errmsg(-1)); goto done; } if (eh.e_type != ET_EXEC && eh.e_type != ET_DYN && !(image->pi_iskernelmodule && eh.e_type == ET_REL)) { warnx("WARNING: \"%s\" is of an unsupported ELF type.", buffer); goto done; } image_type = eh.e_ident[EI_CLASS] == ELFCLASS32 ? PMCSTAT_IMAGE_ELF32 : PMCSTAT_IMAGE_ELF64; /* * Determine the virtual address where an executable would be * loaded. Additionally, for dynamically linked executables, * save the pathname to the runtime linker. */ if (eh.e_type == ET_EXEC) { if (elf_getphnum(e, &nph) == 0) { warnx("WARNING: Could not determine the number of " "program headers in \"%s\": %s.", buffer, elf_errmsg(-1)); goto done; } for (i = 0; i < eh.e_phnum; i++) { if (gelf_getphdr(e, i, &ph) != &ph) { warnx("WARNING: Retrieval of PHDR entry #%ju " "in \"%s\" failed: %s.", (uintmax_t) i, buffer, elf_errmsg(-1)); goto done; } switch (ph.p_type) { case PT_DYNAMIC: image->pi_isdynamic = 1; break; case PT_INTERP: if ((elfbase = elf_rawfile(e, NULL)) == NULL) { warnx("WARNING: Cannot retrieve the " "interpreter for \"%s\": %s.", buffer, elf_errmsg(-1)); goto done; } image->pi_dynlinkerpath = pmcstat_string_intern(elfbase + ph.p_offset); break; case PT_LOAD: if (ph.p_offset == 0) image->pi_vaddr = ph.p_vaddr; break; } } } /* * Get the min and max VA associated with this ELF object. */ if (elf_getshnum(e, &nsh) == 0) { warnx("WARNING: Could not determine the number of sections " "for \"%s\": %s.", buffer, elf_errmsg(-1)); goto done; } for (i = 0; i < nsh; i++) { if ((scn = elf_getscn(e, i)) == NULL || gelf_getshdr(scn, &sh) != &sh) { warnx("WARNING: Could not retrieve section header " "#%ju in \"%s\": %s.", (uintmax_t) i, buffer, elf_errmsg(-1)); goto done; } if (sh.sh_flags & SHF_EXECINSTR) { minva = min(minva, sh.sh_addr); maxva = max(maxva, sh.sh_addr + sh.sh_size); } if (sh.sh_type == SHT_SYMTAB || sh.sh_type == SHT_DYNSYM) pmcstat_image_add_symbols(image, e, scn, &sh); } image->pi_start = minva; image->pi_end = maxva; image->pi_type = image_type; image->pi_fullpath = pmcstat_string_intern(buffer); /* Build display name */ endp = buffer; for (p = buffer; *p; p++) if (*p == '/') endp = p+1; image->pi_name = pmcstat_string_intern(endp); done: (void) elf_end(e); if (fd >= 0) (void) close(fd); return; } /* * Given an image descriptor, determine whether it is an ELF, or AOUT. * If no handler claims the image, set its type to 'INDETERMINABLE'. */ void pmcstat_image_determine_type(struct pmcstat_image *image) { assert(image->pi_type == PMCSTAT_IMAGE_UNKNOWN); /* Try each kind of handler in turn */ if (image->pi_type == PMCSTAT_IMAGE_UNKNOWN) pmcstat_image_get_elf_params(image); if (image->pi_type == PMCSTAT_IMAGE_UNKNOWN) pmcstat_image_get_aout_params(image); /* * Otherwise, remember that we tried to determine * the object's type and had failed. */ if (image->pi_type == PMCSTAT_IMAGE_UNKNOWN) image->pi_type = PMCSTAT_IMAGE_INDETERMINABLE; } /* * Locate an image descriptor given an interned path, adding a fresh * descriptor to the cache if necessary. This function also finds a * suitable name for this image's sample file. * * We defer filling in the file format specific parts of the image * structure till the time we actually see a sample that would fall * into this image. */ static struct pmcstat_image * pmcstat_image_from_path(pmcstat_interned_string internedpath, int iskernelmodule) { int hash; struct pmcstat_image *pi; hash = pmcstat_string_lookup_hash(internedpath); /* First, look for an existing entry. */ LIST_FOREACH(pi, &pmcstat_image_hash[hash], pi_next) if (pi->pi_execpath == internedpath && pi->pi_iskernelmodule == iskernelmodule) return (pi); /* * Allocate a new entry and place it at the head of the hash * and LRU lists. */ pi = malloc(sizeof(*pi)); if (pi == NULL) return (NULL); pi->pi_type = PMCSTAT_IMAGE_UNKNOWN; pi->pi_execpath = internedpath; pi->pi_start = ~0; pi->pi_end = 0; pi->pi_entry = 0; pi->pi_vaddr = 0; pi->pi_isdynamic = 0; pi->pi_iskernelmodule = iskernelmodule; pi->pi_dynlinkerpath = NULL; pi->pi_symbols = NULL; pi->pi_symcount = 0; pi->pi_addr2line = NULL; if (plugins[args.pa_pplugin].pl_initimage != NULL) plugins[args.pa_pplugin].pl_initimage(pi); if (plugins[args.pa_plugin].pl_initimage != NULL) plugins[args.pa_plugin].pl_initimage(pi); LIST_INSERT_HEAD(&pmcstat_image_hash[hash], pi, pi_next); return (pi); } /* * Record the fact that PC values from 'start' to 'end' come from * image 'image'. */ static void pmcstat_image_link(struct pmcstat_process *pp, struct pmcstat_image *image, uintfptr_t start) { struct pmcstat_pcmap *pcm, *pcmnew; uintfptr_t offset; assert(image->pi_type != PMCSTAT_IMAGE_UNKNOWN && image->pi_type != PMCSTAT_IMAGE_INDETERMINABLE); if ((pcmnew = malloc(sizeof(*pcmnew))) == NULL) err(EX_OSERR, "ERROR: Cannot create a map entry"); /* * Adjust the map entry to only cover the text portion * of the object. */ offset = start - image->pi_vaddr; pcmnew->ppm_lowpc = image->pi_start + offset; pcmnew->ppm_highpc = image->pi_end + offset; pcmnew->ppm_image = image; assert(pcmnew->ppm_lowpc < pcmnew->ppm_highpc); /* Overlapped mmap()'s are assumed to never occur. */ TAILQ_FOREACH(pcm, &pp->pp_map, ppm_next) if (pcm->ppm_lowpc >= pcmnew->ppm_highpc) break; if (pcm == NULL) TAILQ_INSERT_TAIL(&pp->pp_map, pcmnew, ppm_next); else TAILQ_INSERT_BEFORE(pcm, pcmnew, ppm_next); } /* * Unmap images in the range [start..end) associated with process * 'pp'. */ static void pmcstat_image_unmap(struct pmcstat_process *pp, uintfptr_t start, uintfptr_t end) { struct pmcstat_pcmap *pcm, *pcmtmp, *pcmnew; assert(pp != NULL); assert(start < end); /* * Cases: * - we could have the range completely in the middle of an * existing pcmap; in this case we have to split the pcmap * structure into two (i.e., generate a 'hole'). * - we could have the range covering multiple pcmaps; these * will have to be removed. * - we could have either 'start' or 'end' falling in the * middle of a pcmap; in this case shorten the entry. */ TAILQ_FOREACH_SAFE(pcm, &pp->pp_map, ppm_next, pcmtmp) { assert(pcm->ppm_lowpc < pcm->ppm_highpc); if (pcm->ppm_highpc <= start) continue; if (pcm->ppm_lowpc >= end) return; if (pcm->ppm_lowpc >= start && pcm->ppm_highpc <= end) { /* * The current pcmap is completely inside the * unmapped range: remove it entirely. */ TAILQ_REMOVE(&pp->pp_map, pcm, ppm_next); free(pcm); } else if (pcm->ppm_lowpc < start && pcm->ppm_highpc > end) { /* * Split this pcmap into two; curtail the * current map to end at [start-1], and start * the new one at [end]. */ if ((pcmnew = malloc(sizeof(*pcmnew))) == NULL) err(EX_OSERR, "ERROR: Cannot split a map " "entry"); pcmnew->ppm_image = pcm->ppm_image; pcmnew->ppm_lowpc = end; pcmnew->ppm_highpc = pcm->ppm_highpc; pcm->ppm_highpc = start; TAILQ_INSERT_AFTER(&pp->pp_map, pcm, pcmnew, ppm_next); return; } else if (pcm->ppm_lowpc < start && pcm->ppm_highpc <= end) pcm->ppm_highpc = start; else if (pcm->ppm_lowpc >= start && pcm->ppm_highpc > end) pcm->ppm_lowpc = end; else assert(0); } } /* * Resolve file name and line number for the given address. */ int pmcstat_image_addr2line(struct pmcstat_image *image, uintfptr_t addr, char *sourcefile, size_t sourcefile_len, unsigned *sourceline, char *funcname, size_t funcname_len) { static int addr2line_warn = 0; char *sep, cmdline[PATH_MAX], imagepath[PATH_MAX]; int fd; if (image->pi_addr2line == NULL) { snprintf(imagepath, sizeof(imagepath), "%s.symbols", pmcstat_string_unintern(image->pi_fullpath)); fd = open(imagepath, O_RDONLY); if (fd < 0) { snprintf(imagepath, sizeof(imagepath), "%s", pmcstat_string_unintern(image->pi_fullpath)); } else close(fd); snprintf(cmdline, sizeof(cmdline), "addr2line -Cfe \"%s\"", imagepath); image->pi_addr2line = popen(cmdline, "r+"); if (image->pi_addr2line == NULL) { if (!addr2line_warn) { addr2line_warn = 1; warnx("WARNING: addr2line is needed" "for source code information."); } return (0); } } if (feof(image->pi_addr2line) || ferror(image->pi_addr2line)) { warnx("WARNING: addr2line pipe error"); pclose(image->pi_addr2line); image->pi_addr2line = NULL; return (0); } fprintf(image->pi_addr2line, "%p\n", (void *)addr); if (fgets(funcname, funcname_len, image->pi_addr2line) == NULL) { warnx("WARNING: addr2line function name read error"); return (0); } sep = strchr(funcname, '\n'); if (sep != NULL) *sep = '\0'; if (fgets(sourcefile, sourcefile_len, image->pi_addr2line) == NULL) { warnx("WARNING: addr2line source file read error"); return (0); } sep = strchr(sourcefile, ':'); if (sep == NULL) { warnx("WARNING: addr2line source line separator missing"); return (0); } *sep = '\0'; *sourceline = atoi(sep+1); if (*sourceline == 0) return (0); return (1); } /* * Add a {pmcid,name} mapping. */ static void pmcstat_pmcid_add(pmc_id_t pmcid, pmcstat_interned_string ps) { struct pmcstat_pmcrecord *pr, *prm; /* Replace an existing name for the PMC. */ prm = NULL; LIST_FOREACH(pr, &pmcstat_pmcs, pr_next) if (pr->pr_pmcid == pmcid) { pr->pr_pmcname = ps; return; } else if (pr->pr_pmcname == ps) prm = pr; /* * Otherwise, allocate a new descriptor and call the * plugins hook. */ if ((pr = malloc(sizeof(*pr))) == NULL) err(EX_OSERR, "ERROR: Cannot allocate pmc record"); pr->pr_pmcid = pmcid; pr->pr_pmcname = ps; pr->pr_pmcin = pmcstat_npmcs++; pr->pr_samples = 0; pr->pr_dubious_frames = 0; pr->pr_merge = prm == NULL ? pr : prm; LIST_INSERT_HEAD(&pmcstat_pmcs, pr, pr_next); if (plugins[args.pa_pplugin].pl_newpmc != NULL) plugins[args.pa_pplugin].pl_newpmc(ps, pr); if (plugins[args.pa_plugin].pl_newpmc != NULL) plugins[args.pa_plugin].pl_newpmc(ps, pr); } /* * Given a pmcid in use, find its human-readable name. */ const char * pmcstat_pmcid_to_name(pmc_id_t pmcid) { struct pmcstat_pmcrecord *pr; LIST_FOREACH(pr, &pmcstat_pmcs, pr_next) if (pr->pr_pmcid == pmcid) return (pmcstat_string_unintern(pr->pr_pmcname)); - err(EX_SOFTWARE, "ERROR: cannot find pmcid"); return NULL; } /* * Convert PMC index to name. */ const char * pmcstat_pmcindex_to_name(int pmcin) { struct pmcstat_pmcrecord *pr; LIST_FOREACH(pr, &pmcstat_pmcs, pr_next) if (pr->pr_pmcin == pmcin) return pmcstat_string_unintern(pr->pr_pmcname); return NULL; } /* * Return PMC record with given index. */ struct pmcstat_pmcrecord * pmcstat_pmcindex_to_pmcr(int pmcin) { struct pmcstat_pmcrecord *pr; LIST_FOREACH(pr, &pmcstat_pmcs, pr_next) if (pr->pr_pmcin == pmcin) return pr; - err(EX_SOFTWARE, "ERROR: invalid pmcindex"); return NULL; } /* * Get PMC record by id, apply merge policy. */ static struct pmcstat_pmcrecord * pmcstat_lookup_pmcid(pmc_id_t pmcid) { struct pmcstat_pmcrecord *pr; LIST_FOREACH(pr, &pmcstat_pmcs, pr_next) { if (pr->pr_pmcid == pmcid) { if (pmcstat_mergepmc) return pr->pr_merge; return pr; } } return NULL; } /* * Associate an AOUT image with a process. */ static void pmcstat_process_aout_exec(struct pmcstat_process *pp, struct pmcstat_image *image, uintfptr_t entryaddr) { (void) pp; (void) image; (void) entryaddr; /* TODO Implement a.out handling */ } /* * Associate an ELF image with a process. */ static void pmcstat_process_elf_exec(struct pmcstat_process *pp, struct pmcstat_image *image, uintfptr_t entryaddr) { uintmax_t libstart; struct pmcstat_image *rtldimage; assert(image->pi_type == PMCSTAT_IMAGE_ELF32 || image->pi_type == PMCSTAT_IMAGE_ELF64); /* Create a map entry for the base executable. */ pmcstat_image_link(pp, image, image->pi_vaddr); /* * For dynamically linked executables we need to determine * where the dynamic linker was mapped to for this process, * Subsequent executable objects that are mapped in by the * dynamic linker will be tracked by log events of type * PMCLOG_TYPE_MAP_IN. */ if (image->pi_isdynamic) { /* * The runtime loader gets loaded just after the maximum * possible heap address. Like so: * * [ TEXT DATA BSS HEAP -->*RTLD SHLIBS <--STACK] * ^ ^ * 0 VM_MAXUSER_ADDRESS * * The exact address where the loader gets mapped in * will vary according to the size of the executable * and the limits on the size of the process'es data * segment at the time of exec(). The entry address * recorded at process exec time corresponds to the * 'start' address inside the dynamic linker. From * this we can figure out the address where the * runtime loader's file object had been mapped to. */ rtldimage = pmcstat_image_from_path(image->pi_dynlinkerpath, 0); if (rtldimage == NULL) { warnx("WARNING: Cannot find image for \"%s\".", pmcstat_string_unintern(image->pi_dynlinkerpath)); pmcstat_stats.ps_exec_errors++; return; } if (rtldimage->pi_type == PMCSTAT_IMAGE_UNKNOWN) pmcstat_image_get_elf_params(rtldimage); if (rtldimage->pi_type != PMCSTAT_IMAGE_ELF32 && rtldimage->pi_type != PMCSTAT_IMAGE_ELF64) { warnx("WARNING: rtld not an ELF object \"%s\".", pmcstat_string_unintern(image->pi_dynlinkerpath)); return; } libstart = entryaddr - rtldimage->pi_entry; pmcstat_image_link(pp, rtldimage, libstart); } } /* * Find the process descriptor corresponding to a PID. If 'allocate' * is zero, we return a NULL if a pid descriptor could not be found or * a process descriptor process. If 'allocate' is non-zero, then we * will attempt to allocate a fresh process descriptor. Zombie * process descriptors are only removed if a fresh allocation for the * same PID is requested. */ static struct pmcstat_process * pmcstat_process_lookup(pid_t pid, int allocate) { uint32_t hash; struct pmcstat_pcmap *ppm, *ppmtmp; struct pmcstat_process *pp, *pptmp; hash = (uint32_t) pid & PMCSTAT_HASH_MASK; /* simplicity wins */ LIST_FOREACH_SAFE(pp, &pmcstat_process_hash[hash], pp_next, pptmp) if (pp->pp_pid == pid) { /* Found a descriptor, check and process zombies */ if (allocate && pp->pp_isactive == 0) { /* remove maps */ TAILQ_FOREACH_SAFE(ppm, &pp->pp_map, ppm_next, ppmtmp) { TAILQ_REMOVE(&pp->pp_map, ppm, ppm_next); free(ppm); } /* remove process entry */ LIST_REMOVE(pp, pp_next); free(pp); break; } return (pp); } if (!allocate) return (NULL); if ((pp = malloc(sizeof(*pp))) == NULL) err(EX_OSERR, "ERROR: Cannot allocate pid descriptor"); pp->pp_pid = pid; pp->pp_isactive = 1; TAILQ_INIT(&pp->pp_map); LIST_INSERT_HEAD(&pmcstat_process_hash[hash], pp, pp_next); return (pp); } /* * Associate an image and a process. */ static void pmcstat_process_exec(struct pmcstat_process *pp, pmcstat_interned_string path, uintfptr_t entryaddr) { struct pmcstat_image *image; if ((image = pmcstat_image_from_path(path, 0)) == NULL) { pmcstat_stats.ps_exec_errors++; return; } if (image->pi_type == PMCSTAT_IMAGE_UNKNOWN) pmcstat_image_determine_type(image); assert(image->pi_type != PMCSTAT_IMAGE_UNKNOWN); switch (image->pi_type) { case PMCSTAT_IMAGE_ELF32: case PMCSTAT_IMAGE_ELF64: pmcstat_stats.ps_exec_elf++; pmcstat_process_elf_exec(pp, image, entryaddr); break; case PMCSTAT_IMAGE_AOUT: pmcstat_stats.ps_exec_aout++; pmcstat_process_aout_exec(pp, image, entryaddr); break; case PMCSTAT_IMAGE_INDETERMINABLE: pmcstat_stats.ps_exec_indeterminable++; break; default: err(EX_SOFTWARE, "ERROR: Unsupported executable type for " "\"%s\"", pmcstat_string_unintern(path)); } } /* * Find the map entry associated with process 'p' at PC value 'pc'. */ struct pmcstat_pcmap * pmcstat_process_find_map(struct pmcstat_process *p, uintfptr_t pc) { struct pmcstat_pcmap *ppm; TAILQ_FOREACH(ppm, &p->pp_map, ppm_next) { if (pc >= ppm->ppm_lowpc && pc < ppm->ppm_highpc) return (ppm); if (pc < ppm->ppm_lowpc) return (NULL); } return (NULL); } /* * Convert a hwpmc(4) log to profile information. A system-wide * callgraph is generated if FLAG_DO_CALLGRAPHS is set. gmon.out * files usable by gprof(1) are created if FLAG_DO_GPROF is set. */ static int pmcstat_analyze_log(void) { uint32_t cpu, cpuflags; uintfptr_t pc; pid_t pid; struct pmcstat_image *image; struct pmcstat_process *pp, *ppnew; struct pmcstat_pcmap *ppm, *ppmtmp; struct pmclog_ev ev; struct pmcstat_pmcrecord *pmcr; pmcstat_interned_string image_path; assert(args.pa_flags & FLAG_DO_ANALYSIS); if (elf_version(EV_CURRENT) == EV_NONE) err(EX_UNAVAILABLE, "Elf library intialization failed"); while (pmclog_read(args.pa_logparser, &ev) == 0) { assert(ev.pl_state == PMCLOG_OK); switch (ev.pl_type) { case PMCLOG_TYPE_INITIALIZE: if ((ev.pl_u.pl_i.pl_version & 0xFF000000) != PMC_VERSION_MAJOR << 24 && args.pa_verbosity > 0) warnx("WARNING: Log version 0x%x does not " "match compiled version 0x%x.", ev.pl_u.pl_i.pl_version, PMC_VERSION_MAJOR); break; case PMCLOG_TYPE_MAP_IN: /* * Introduce an address range mapping for a * userland process or the kernel (pid == -1). * * We always allocate a process descriptor so * that subsequent samples seen for this * address range are mapped to the current * object being mapped in. */ pid = ev.pl_u.pl_mi.pl_pid; if (pid == -1) pp = pmcstat_kernproc; else pp = pmcstat_process_lookup(pid, PMCSTAT_ALLOCATE); assert(pp != NULL); image_path = pmcstat_string_intern(ev.pl_u.pl_mi. pl_pathname); image = pmcstat_image_from_path(image_path, pid == -1); if (image->pi_type == PMCSTAT_IMAGE_UNKNOWN) pmcstat_image_determine_type(image); if (image->pi_type != PMCSTAT_IMAGE_INDETERMINABLE) pmcstat_image_link(pp, image, ev.pl_u.pl_mi.pl_start); break; case PMCLOG_TYPE_MAP_OUT: /* * Remove an address map. */ pid = ev.pl_u.pl_mo.pl_pid; if (pid == -1) pp = pmcstat_kernproc; else pp = pmcstat_process_lookup(pid, 0); if (pp == NULL) /* unknown process */ break; pmcstat_image_unmap(pp, ev.pl_u.pl_mo.pl_start, ev.pl_u.pl_mo.pl_end); break; case PMCLOG_TYPE_PCSAMPLE: /* * Note: the `PCSAMPLE' log entry is not * generated by hpwmc(4) after version 2. */ /* * We bring in the gmon file for the image * currently associated with the PMC & pid * pair and increment the appropriate entry * bin inside this. */ pmcstat_stats.ps_samples_total++; pc = ev.pl_u.pl_s.pl_pc; pp = pmcstat_process_lookup(ev.pl_u.pl_s.pl_pid, PMCSTAT_ALLOCATE); /* Get PMC record. */ pmcr = pmcstat_lookup_pmcid(ev.pl_u.pl_s.pl_pmcid); assert(pmcr != NULL); pmcr->pr_samples++; /* * Call the plugins processing * TODO: move pmcstat_process_find_map inside plugins */ if (plugins[args.pa_pplugin].pl_process != NULL) plugins[args.pa_pplugin].pl_process( pp, pmcr, 1, &pc, pmcstat_process_find_map(pp, pc) != NULL, 0); plugins[args.pa_plugin].pl_process( pp, pmcr, 1, &pc, pmcstat_process_find_map(pp, pc) != NULL, 0); break; case PMCLOG_TYPE_CALLCHAIN: pmcstat_stats.ps_samples_total++; cpuflags = ev.pl_u.pl_cc.pl_cpuflags; cpu = PMC_CALLCHAIN_CPUFLAGS_TO_CPU(cpuflags); /* Filter on the CPU id. */ if ((args.pa_cpumask & (1 << cpu)) == 0) { pmcstat_stats.ps_samples_skipped++; break; } pp = pmcstat_process_lookup(ev.pl_u.pl_cc.pl_pid, PMCSTAT_ALLOCATE); /* Get PMC record. */ pmcr = pmcstat_lookup_pmcid(ev.pl_u.pl_cc.pl_pmcid); assert(pmcr != NULL); pmcr->pr_samples++; /* * Call the plugins processing */ if (plugins[args.pa_pplugin].pl_process != NULL) plugins[args.pa_pplugin].pl_process( pp, pmcr, ev.pl_u.pl_cc.pl_npc, ev.pl_u.pl_cc.pl_pc, PMC_CALLCHAIN_CPUFLAGS_TO_USERMODE(cpuflags), cpu); plugins[args.pa_plugin].pl_process( pp, pmcr, ev.pl_u.pl_cc.pl_npc, ev.pl_u.pl_cc.pl_pc, PMC_CALLCHAIN_CPUFLAGS_TO_USERMODE(cpuflags), cpu); break; case PMCLOG_TYPE_PMCALLOCATE: /* * Record the association pmc id between this * PMC and its name. */ pmcstat_pmcid_add(ev.pl_u.pl_a.pl_pmcid, pmcstat_string_intern(ev.pl_u.pl_a.pl_evname)); break; case PMCLOG_TYPE_PROCEXEC: /* * Change the executable image associated with * a process. */ pp = pmcstat_process_lookup(ev.pl_u.pl_x.pl_pid, PMCSTAT_ALLOCATE); /* delete the current process map */ TAILQ_FOREACH_SAFE(ppm, &pp->pp_map, ppm_next, ppmtmp) { TAILQ_REMOVE(&pp->pp_map, ppm, ppm_next); free(ppm); } /* associate this process image */ image_path = pmcstat_string_intern( ev.pl_u.pl_x.pl_pathname); assert(image_path != NULL); pmcstat_process_exec(pp, image_path, ev.pl_u.pl_x.pl_entryaddr); break; case PMCLOG_TYPE_PROCEXIT: /* * Due to the way the log is generated, the * last few samples corresponding to a process * may appear in the log after the process * exit event is recorded. Thus we keep the * process' descriptor and associated data * structures around, but mark the process as * having exited. */ pp = pmcstat_process_lookup(ev.pl_u.pl_e.pl_pid, 0); if (pp == NULL) break; pp->pp_isactive = 0; /* mark as a zombie */ break; case PMCLOG_TYPE_SYSEXIT: pp = pmcstat_process_lookup(ev.pl_u.pl_se.pl_pid, 0); if (pp == NULL) break; pp->pp_isactive = 0; /* make a zombie */ break; case PMCLOG_TYPE_PROCFORK: /* * Allocate a process descriptor for the new * (child) process. */ ppnew = pmcstat_process_lookup(ev.pl_u.pl_f.pl_newpid, PMCSTAT_ALLOCATE); /* * If we had been tracking the parent, clone * its address maps. */ pp = pmcstat_process_lookup(ev.pl_u.pl_f.pl_oldpid, 0); if (pp == NULL) break; TAILQ_FOREACH(ppm, &pp->pp_map, ppm_next) pmcstat_image_link(ppnew, ppm->ppm_image, ppm->ppm_lowpc); break; default: /* other types of entries are not relevant */ break; } } if (ev.pl_state == PMCLOG_EOF) return (PMCSTAT_FINISHED); else if (ev.pl_state == PMCLOG_REQUIRE_DATA) return (PMCSTAT_RUNNING); err(EX_DATAERR, "ERROR: event parsing failed (record %jd, " "offset 0x%jx)", (uintmax_t) ev.pl_count + 1, ev.pl_offset); } /* * Print log entries as text. */ static int pmcstat_print_log(void) { struct pmclog_ev ev; uint32_t npc; while (pmclog_read(args.pa_logparser, &ev) == 0) { assert(ev.pl_state == PMCLOG_OK); switch (ev.pl_type) { case PMCLOG_TYPE_CALLCHAIN: PMCSTAT_PRINT_ENTRY("callchain", "%d 0x%x %d %d %c", ev.pl_u.pl_cc.pl_pid, ev.pl_u.pl_cc.pl_pmcid, PMC_CALLCHAIN_CPUFLAGS_TO_CPU(ev.pl_u.pl_cc. \ pl_cpuflags), ev.pl_u.pl_cc.pl_npc, PMC_CALLCHAIN_CPUFLAGS_TO_USERMODE(ev.pl_u.pl_cc.\ pl_cpuflags) ? 'u' : 's'); for (npc = 0; npc < ev.pl_u.pl_cc.pl_npc; npc++) PMCSTAT_PRINT_ENTRY("...", "%p", (void *) ev.pl_u.pl_cc.pl_pc[npc]); break; case PMCLOG_TYPE_CLOSELOG: PMCSTAT_PRINT_ENTRY("closelog",); break; case PMCLOG_TYPE_DROPNOTIFY: PMCSTAT_PRINT_ENTRY("drop",); break; case PMCLOG_TYPE_INITIALIZE: PMCSTAT_PRINT_ENTRY("initlog","0x%x \"%s\"", ev.pl_u.pl_i.pl_version, pmc_name_of_cputype(ev.pl_u.pl_i.pl_arch)); if ((ev.pl_u.pl_i.pl_version & 0xFF000000) != PMC_VERSION_MAJOR << 24 && args.pa_verbosity > 0) warnx("WARNING: Log version 0x%x != expected " "version 0x%x.", ev.pl_u.pl_i.pl_version, PMC_VERSION); break; case PMCLOG_TYPE_MAP_IN: PMCSTAT_PRINT_ENTRY("map-in","%d %p \"%s\"", ev.pl_u.pl_mi.pl_pid, (void *) ev.pl_u.pl_mi.pl_start, ev.pl_u.pl_mi.pl_pathname); break; case PMCLOG_TYPE_MAP_OUT: PMCSTAT_PRINT_ENTRY("map-out","%d %p %p", ev.pl_u.pl_mo.pl_pid, (void *) ev.pl_u.pl_mo.pl_start, (void *) ev.pl_u.pl_mo.pl_end); break; case PMCLOG_TYPE_PCSAMPLE: PMCSTAT_PRINT_ENTRY("sample","0x%x %d %p %c", ev.pl_u.pl_s.pl_pmcid, ev.pl_u.pl_s.pl_pid, (void *) ev.pl_u.pl_s.pl_pc, ev.pl_u.pl_s.pl_usermode ? 'u' : 's'); break; case PMCLOG_TYPE_PMCALLOCATE: PMCSTAT_PRINT_ENTRY("allocate","0x%x \"%s\" 0x%x", ev.pl_u.pl_a.pl_pmcid, ev.pl_u.pl_a.pl_evname, ev.pl_u.pl_a.pl_flags); break; case PMCLOG_TYPE_PMCATTACH: PMCSTAT_PRINT_ENTRY("attach","0x%x %d \"%s\"", ev.pl_u.pl_t.pl_pmcid, ev.pl_u.pl_t.pl_pid, ev.pl_u.pl_t.pl_pathname); break; case PMCLOG_TYPE_PMCDETACH: PMCSTAT_PRINT_ENTRY("detach","0x%x %d", ev.pl_u.pl_d.pl_pmcid, ev.pl_u.pl_d.pl_pid); break; case PMCLOG_TYPE_PROCCSW: PMCSTAT_PRINT_ENTRY("cswval","0x%x %d %jd", ev.pl_u.pl_c.pl_pmcid, ev.pl_u.pl_c.pl_pid, ev.pl_u.pl_c.pl_value); break; case PMCLOG_TYPE_PROCEXEC: PMCSTAT_PRINT_ENTRY("exec","0x%x %d %p \"%s\"", ev.pl_u.pl_x.pl_pmcid, ev.pl_u.pl_x.pl_pid, (void *) ev.pl_u.pl_x.pl_entryaddr, ev.pl_u.pl_x.pl_pathname); break; case PMCLOG_TYPE_PROCEXIT: PMCSTAT_PRINT_ENTRY("exitval","0x%x %d %jd", ev.pl_u.pl_e.pl_pmcid, ev.pl_u.pl_e.pl_pid, ev.pl_u.pl_e.pl_value); break; case PMCLOG_TYPE_PROCFORK: PMCSTAT_PRINT_ENTRY("fork","%d %d", ev.pl_u.pl_f.pl_oldpid, ev.pl_u.pl_f.pl_newpid); break; case PMCLOG_TYPE_USERDATA: PMCSTAT_PRINT_ENTRY("userdata","0x%x", ev.pl_u.pl_u.pl_userdata); break; case PMCLOG_TYPE_SYSEXIT: PMCSTAT_PRINT_ENTRY("exit","%d", ev.pl_u.pl_se.pl_pid); break; default: fprintf(args.pa_printfile, "unknown event (type %d).\n", ev.pl_type); } } if (ev.pl_state == PMCLOG_EOF) return (PMCSTAT_FINISHED); else if (ev.pl_state == PMCLOG_REQUIRE_DATA) return (PMCSTAT_RUNNING); errx(EX_DATAERR, "ERROR: event parsing failed " "(record %jd, offset 0x%jx).", (uintmax_t) ev.pl_count + 1, ev.pl_offset); /*NOTREACHED*/ } /* * Public Interfaces. */ /* * Close a logfile, after first flushing all in-module queued data. */ int pmcstat_close_log(void) { if (pmc_flush_logfile() < 0) err(EX_OSERR, "ERROR: logging failed"); return (args.pa_flags & FLAG_HAS_PIPE ? PMCSTAT_EXITING : PMCSTAT_FINISHED); } /* * Open a log file, for reading or writing. * * The function returns the fd of a successfully opened log or -1 in * case of failure. */ int pmcstat_open_log(const char *path, int mode) { int error, fd; size_t hlen; const char *p, *errstr; struct addrinfo hints, *res, *res0; char hostname[MAXHOSTNAMELEN]; errstr = NULL; fd = -1; /* * If 'path' is "-" then open one of stdin or stdout depending * on the value of 'mode'. * * If 'path' contains a ':' and does not start with a '/' or '.', * and is being opened for writing, treat it as a "host:port" * specification and open a network socket. * * Otherwise, treat 'path' as a file name and open that. */ if (path[0] == '-' && path[1] == '\0') fd = (mode == PMCSTAT_OPEN_FOR_READ) ? 0 : 1; else if (mode == PMCSTAT_OPEN_FOR_WRITE && path[0] != '/' && path[0] != '.' && strchr(path, ':') != NULL) { p = strrchr(path, ':'); hlen = p - path; if (p == path || hlen >= sizeof(hostname)) { errstr = strerror(EINVAL); goto done; } assert(hlen < sizeof(hostname)); (void) strncpy(hostname, path, hlen); hostname[hlen] = '\0'; (void) memset(&hints, 0, sizeof(hints)); hints.ai_family = AF_UNSPEC; hints.ai_socktype = SOCK_STREAM; if ((error = getaddrinfo(hostname, p+1, &hints, &res0)) != 0) { errstr = gai_strerror(error); goto done; } fd = -1; for (res = res0; res; res = res->ai_next) { if ((fd = socket(res->ai_family, res->ai_socktype, res->ai_protocol)) < 0) { errstr = strerror(errno); continue; } if (connect(fd, res->ai_addr, res->ai_addrlen) < 0) { errstr = strerror(errno); (void) close(fd); fd = -1; continue; } errstr = NULL; break; } freeaddrinfo(res0); } else if ((fd = open(path, mode == PMCSTAT_OPEN_FOR_READ ? O_RDONLY : (O_WRONLY|O_CREAT|O_TRUNC), S_IRUSR|S_IWUSR|S_IRGRP|S_IROTH)) < 0) errstr = strerror(errno); done: if (errstr) errx(EX_OSERR, "ERROR: Cannot open \"%s\" for %s: %s.", path, (mode == PMCSTAT_OPEN_FOR_READ ? "reading" : "writing"), errstr); return (fd); } /* * Process a log file in offline analysis mode. */ int pmcstat_process_log(void) { /* * If analysis has not been asked for, just print the log to * the current output file. */ if (args.pa_flags & FLAG_DO_PRINT) return (pmcstat_print_log()); else return (pmcstat_analyze_log()); } /* * Refresh top display. */ static void pmcstat_refresh_top(void) { int v_attrs; float v; char pmcname[40]; struct pmcstat_pmcrecord *pmcpr; /* If in pause mode do not refresh display. */ if (pmcstat_pause) return; /* Wait until PMC pop in the log. */ pmcpr = pmcstat_pmcindex_to_pmcr(pmcstat_pmcinfilter); if (pmcpr == NULL) return; /* Format PMC name. */ if (pmcstat_mergepmc) snprintf(pmcname, sizeof(pmcname), "[%s]", pmcstat_string_unintern(pmcpr->pr_pmcname)); else snprintf(pmcname, sizeof(pmcname), "%s.%d", pmcstat_string_unintern(pmcpr->pr_pmcname), pmcstat_pmcinfilter); /* Format samples count. */ if (pmcstat_stats.ps_samples_total > 0) v = (pmcpr->pr_samples * 100.0) / pmcstat_stats.ps_samples_total; else v = 0.; v_attrs = PMCSTAT_ATTRPERCENT(v); PMCSTAT_PRINTBEGIN(); PMCSTAT_PRINTW("PMC: %s Samples: %u ", pmcname, pmcpr->pr_samples); PMCSTAT_ATTRON(v_attrs); PMCSTAT_PRINTW("(%.1f%%) ", v); PMCSTAT_ATTROFF(v_attrs); PMCSTAT_PRINTW(", %u unresolved\n\n", pmcpr->pr_dubious_frames); if (plugins[args.pa_plugin].pl_topdisplay != NULL) plugins[args.pa_plugin].pl_topdisplay(); PMCSTAT_PRINTEND(); } /* * Find the next pmc index to display. */ static void pmcstat_changefilter(void) { int pmcin; struct pmcstat_pmcrecord *pmcr; /* * Find the next merge target. */ if (pmcstat_mergepmc) { pmcin = pmcstat_pmcinfilter; do { pmcr = pmcstat_pmcindex_to_pmcr(pmcstat_pmcinfilter); if (pmcr == pmcr->pr_merge) break; pmcstat_pmcinfilter++; if (pmcstat_pmcinfilter >= pmcstat_npmcs) pmcstat_pmcinfilter = 0; } while (pmcstat_pmcinfilter != pmcin); } } /* * Top mode keypress. */ int pmcstat_keypress_log(void) { int c, ret = 0; WINDOW *w; w = newwin(1, 0, 1, 0); c = wgetch(w); wprintw(w, "Key: %c => ", c); switch (c) { case 'c': wprintw(w, "enter mode 'd' or 'a' => "); c = wgetch(w); if (c == 'd') { args.pa_topmode = PMCSTAT_TOP_DELTA; wprintw(w, "switching to delta mode"); } else { args.pa_topmode = PMCSTAT_TOP_ACCUM; wprintw(w, "switching to accumulation mode"); } break; case 'm': pmcstat_mergepmc = !pmcstat_mergepmc; /* * Changing merge state require data reset. */ if (plugins[args.pa_plugin].pl_shutdown != NULL) plugins[args.pa_plugin].pl_shutdown(NULL); pmcstat_stats_reset(); if (plugins[args.pa_plugin].pl_init != NULL) plugins[args.pa_plugin].pl_init(); /* Update filter to be on a merge target. */ pmcstat_changefilter(); wprintw(w, "merge PMC %s", pmcstat_mergepmc ? "on" : "off"); break; case 'n': /* Close current plugin. */ if (plugins[args.pa_plugin].pl_shutdown != NULL) plugins[args.pa_plugin].pl_shutdown(NULL); /* Find next top display available. */ do { args.pa_plugin++; if (plugins[args.pa_plugin].pl_name == NULL) args.pa_plugin = 0; } while (plugins[args.pa_plugin].pl_topdisplay == NULL); /* Open new plugin. */ pmcstat_stats_reset(); if (plugins[args.pa_plugin].pl_init != NULL) plugins[args.pa_plugin].pl_init(); wprintw(w, "switching to plugin %s", plugins[args.pa_plugin].pl_name); break; case 'p': pmcstat_pmcinfilter++; if (pmcstat_pmcinfilter >= pmcstat_npmcs) pmcstat_pmcinfilter = 0; pmcstat_changefilter(); wprintw(w, "switching to PMC %s.%d", pmcstat_pmcindex_to_name(pmcstat_pmcinfilter), pmcstat_pmcinfilter); break; case ' ': pmcstat_pause = !pmcstat_pause; if (pmcstat_pause) wprintw(w, "pause => press space again to continue"); break; case 'q': wprintw(w, "exiting..."); ret = 1; default: if (plugins[args.pa_plugin].pl_topkeypress != NULL) if (plugins[args.pa_plugin].pl_topkeypress(c, w)) ret = 1; } wrefresh(w); delwin(w); return ret; } /* * Top mode display. */ void pmcstat_display_log(void) { pmcstat_refresh_top(); /* Reset everythings if delta mode. */ if (args.pa_topmode == PMCSTAT_TOP_DELTA) { if (plugins[args.pa_plugin].pl_shutdown != NULL) plugins[args.pa_plugin].pl_shutdown(NULL); pmcstat_stats_reset(); if (plugins[args.pa_plugin].pl_init != NULL) plugins[args.pa_plugin].pl_init(); } } /* * Configure a plugins. */ void pmcstat_pluginconfigure_log(char *opt) { if (strncmp(opt, "threshold=", 10) == 0) { pmcstat_threshold = atof(opt+10); } else { if (plugins[args.pa_plugin].pl_configure != NULL) { if (!plugins[args.pa_plugin].pl_configure(opt)) err(EX_USAGE, "ERROR: unknown option <%s>.", opt); } } } /* * Initialize module. */ void pmcstat_initialize_logging(void) { int i; /* use a convenient format for 'ldd' output */ if (setenv("LD_TRACE_LOADED_OBJECTS_FMT1","%o \"%p\" %x\n",1) != 0) err(EX_OSERR, "ERROR: Cannot setenv"); /* Initialize hash tables */ pmcstat_string_initialize(); for (i = 0; i < PMCSTAT_NHASH; i++) { LIST_INIT(&pmcstat_image_hash[i]); LIST_INIT(&pmcstat_process_hash[i]); } /* * Create a fake 'process' entry for the kernel with pid -1. * hwpmc(4) will subsequently inform us about where the kernel * and any loaded kernel modules are mapped. */ if ((pmcstat_kernproc = pmcstat_process_lookup((pid_t) -1, PMCSTAT_ALLOCATE)) == NULL) err(EX_OSERR, "ERROR: Cannot initialize logging"); /* PMC count. */ pmcstat_npmcs = 0; /* Merge PMC with same name. */ pmcstat_mergepmc = args.pa_mergepmc; /* * Initialize plugins */ if (plugins[args.pa_pplugin].pl_init != NULL) plugins[args.pa_pplugin].pl_init(); if (plugins[args.pa_plugin].pl_init != NULL) plugins[args.pa_plugin].pl_init(); } /* * Shutdown module. */ void pmcstat_shutdown_logging(void) { int i; FILE *mf; struct pmcstat_image *pi, *pitmp; struct pmcstat_process *pp, *pptmp; struct pmcstat_pcmap *ppm, *ppmtmp; /* determine where to send the map file */ mf = NULL; if (args.pa_mapfilename != NULL) mf = (strcmp(args.pa_mapfilename, "-") == 0) ? args.pa_printfile : fopen(args.pa_mapfilename, "w"); if (mf == NULL && args.pa_flags & FLAG_DO_GPROF && args.pa_verbosity >= 2) mf = args.pa_printfile; if (mf) (void) fprintf(mf, "MAP:\n"); /* * Shutdown the plugins */ if (plugins[args.pa_plugin].pl_shutdown != NULL) plugins[args.pa_plugin].pl_shutdown(mf); if (plugins[args.pa_pplugin].pl_shutdown != NULL) plugins[args.pa_pplugin].pl_shutdown(mf); for (i = 0; i < PMCSTAT_NHASH; i++) { LIST_FOREACH_SAFE(pi, &pmcstat_image_hash[i], pi_next, pitmp) { if (plugins[args.pa_plugin].pl_shutdownimage != NULL) plugins[args.pa_plugin].pl_shutdownimage(pi); if (plugins[args.pa_pplugin].pl_shutdownimage != NULL) plugins[args.pa_pplugin].pl_shutdownimage(pi); free(pi->pi_symbols); if (pi->pi_addr2line != NULL) pclose(pi->pi_addr2line); LIST_REMOVE(pi, pi_next); free(pi); } LIST_FOREACH_SAFE(pp, &pmcstat_process_hash[i], pp_next, pptmp) { TAILQ_FOREACH_SAFE(ppm, &pp->pp_map, ppm_next, ppmtmp) { TAILQ_REMOVE(&pp->pp_map, ppm, ppm_next); free(ppm); } LIST_REMOVE(pp, pp_next); free(pp); } } pmcstat_string_shutdown(); /* * Print errors unless -q was specified. Print all statistics * if verbosity > 1. */ #define PRINT(N,V) do { \ if (pmcstat_stats.ps_##V || args.pa_verbosity >= 2) \ (void) fprintf(args.pa_printfile, " %-40s %d\n",\ N, pmcstat_stats.ps_##V); \ } while (0) if (args.pa_verbosity >= 1 && (args.pa_flags & FLAG_DO_ANALYSIS) && (args.pa_flags & FLAG_DO_TOP) == 0) { (void) fprintf(args.pa_printfile, "CONVERSION STATISTICS:\n"); PRINT("#exec/a.out", exec_aout); PRINT("#exec/elf", exec_elf); PRINT("#exec/unknown", exec_indeterminable); PRINT("#exec handling errors", exec_errors); PRINT("#samples/total", samples_total); PRINT("#samples/unclaimed", samples_unknown_offset); PRINT("#samples/unknown-object", samples_indeterminable); PRINT("#callchain/dubious-frames", callchain_dubious_frames); } if (mf) (void) fclose(mf); } Index: user/imp/tbemd/usr.sbin/sysinstall/devices.c =================================================================== --- user/imp/tbemd/usr.sbin/sysinstall/devices.c (revision 206684) +++ user/imp/tbemd/usr.sbin/sysinstall/devices.c (revision 206685) @@ -1,615 +1,616 @@ /* * The new sysinstall program. * * This is probably the last program in the `sysinstall' line - the next * generation being essentially a complete rewrite. * * $FreeBSD$ * * Copyright (c) 1995 * Jordan Hubbard. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer, * verbatim and that no modifications are made prior to this * point in the file. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY JORDAN HUBBARD ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL JORDAN HUBBARD OR HIS PETS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, LIFE OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * */ #include "sysinstall.h" #include #include #include #include #include #include #include #include #include #include #include #include #include #include static Device *Devices[DEV_MAX]; static int numDevs; #define DEVICE_ENTRY(type, name, descr, max) { type, name, descr, max } #define CDROM(name, descr, max) \ DEVICE_ENTRY(DEVICE_TYPE_CDROM, name, descr, max) #define DISK(name, descr, max) \ DEVICE_ENTRY(DEVICE_TYPE_DISK, name, descr, max) #define FLOPPY(name, descr, max) \ DEVICE_ENTRY(DEVICE_TYPE_FLOPPY, name, descr, max) #define NETWORK(name, descr) \ DEVICE_ENTRY(DEVICE_TYPE_NETWORK, name, descr, 0) #define SERIAL(name, descr, max) \ DEVICE_ENTRY(DEVICE_TYPE_NETWORK, name, descr, max) #define USB(name, descr, max) \ DEVICE_ENTRY(DEVICE_TYPE_USB, name, descr, max) static struct _devname { DeviceType type; char *name; char *description; int max; } device_names[] = { CDROM("cd%d", "SCSI CDROM drive", 4), CDROM("mcd%d", "Mitsumi (old model) CDROM drive", 4), CDROM("scd%d", "Sony CDROM drive - CDU31/33A type", 4), CDROM("acd%d", "ATAPI/IDE CDROM", 4), DISK("da%d", "SCSI disk device", 16), DISK("ad%d", "ATA/IDE disk device", 16), DISK("ada%d", "SATA disk device", 16), DISK("ar%d", "ATA/IDE RAID device", 16), DISK("afd%d", "ATAPI/IDE floppy device", 4), DISK("mlxd%d", "Mylex RAID disk", 4), DISK("amrd%d", "AMI MegaRAID drive", 4), DISK("idad%d", "Compaq RAID array", 4), DISK("twed%d", "3ware ATA RAID array", 4), DISK("aacd%d", "Adaptec FSA RAID array", 4), DISK("ipsd%d", "IBM ServeRAID RAID array", 4), DISK("mfid%d", "LSI MegaRAID SAS array", 4), FLOPPY("fd%d", "floppy drive unit A", 4), SERIAL("cuad%d", "%s on device %s (COM%d)", 16), USB("da%da", "USB Mass Storage Device", 16), NETWORK("ae", "Attansic/Atheros L2 Fast Ethernet"), NETWORK("age", "Attansic/Atheros L1 Gigabit Ethernet"), NETWORK("alc", "Atheros AR8131/AR8132 PCIe Ethernet"), NETWORK("ale", "Atheros AR8121/AR8113/AR8114 PCIe Ethernet"), NETWORK("an", "Aironet 4500/4800 802.11 wireless adapter"), NETWORK("ath", "Atheros IEEE 802.11 wireless adapter"), NETWORK("aue", "ADMtek USB Ethernet adapter"), NETWORK("axe", "ASIX Electronics USB Ethernet adapter"), NETWORK("bce", "Broadcom NetXtreme II Gigabit Ethernet card"), NETWORK("bfe", "Broadcom BCM440x PCI Ethernet card"), NETWORK("bge", "Broadcom BCM570x PCI Gigabit Ethernet card"), NETWORK("bm", "Apple BMAC Built-in Ethernet"), NETWORK("bwn", "Broadcom BCM43xx IEEE 802.11 wireless adapter"), NETWORK("cas", "Sun Cassini/Cassini+ or NS DP83065 Saturn Ethernet"), NETWORK("cue", "CATC USB Ethernet adapter"), NETWORK("cxgb", "Chelsio T3 10Gb Ethernet card"), NETWORK("fpa", "DEC DEFPA PCI FDDI card"), NETWORK("sr", "SDL T1/E1 sync serial PCI card"), NETWORK("cc3i", "SDL HSSI sync serial PCI card"), NETWORK("en", "Efficient Networks ATM PCI card"), NETWORK("dc", "DEC/Intel 21143 (and clones) PCI Fast Ethernet card"), NETWORK("de", "DEC DE435 PCI NIC or other DC21040-AA based card"), NETWORK("fxp", "Intel EtherExpress Pro/100B PCI Fast Ethernet card"), NETWORK("ed", "Novell NE1000/2000; 3C503; NE2000-compatible PCMCIA"), NETWORK("ep", "3Com 3C509 Ethernet card/3C589 PCMCIA"), NETWORK("em", "Intel(R) PRO/1000 Ethernet card"), NETWORK("et", "Agere ET1310 based PCI Express Gigabit Ethernet card"), NETWORK("ex", "Intel EtherExpress Pro/10 Ethernet card"), NETWORK("fe", "Fujitsu MB86960A/MB86965A Ethernet card"), NETWORK("gem", "Apple GMAC or Sun ERI/GEM Ethernet adapter"), NETWORK("hme", "Sun HME (Happy Meal Ethernet) Ethernet adapter"), NETWORK("ie", "AT&T StarLAN 10 and EN100; 3Com 3C507; NI5210"), NETWORK("igb", "Intel(R) PRO/1000 PCI Express Gigabit Ethernet card"), NETWORK("ipw", "Intel PRO/Wireless 2100 IEEE 802.11 adapter"), NETWORK("iwi", "Intel PRO/Wireless 2200BG/2225BG/2915ABG adapter"), NETWORK("iwn", "Intel Wireless WiFi Link 4965AGN IEEE 802.11n adapter"), NETWORK("ixgb", "Intel(R) PRO/10Gb Ethernet card"), NETWORK("ixgbe", "Intel(R) PRO/10Gb Ethernet card"), NETWORK("jme", "JMicron JMC250 Gigabit/JMC260 Fast Ethernet"), NETWORK("kue", "Kawasaki LSI USB Ethernet adapter"), NETWORK("le", "AMD Am7900 LANCE or Am79C9xx PCnet Ethernet adapter"), NETWORK("lge", "Level 1 LXT1001 Gigabit Ethernet card"), NETWORK("malo", "Marvell Libertas 88W8335 802.11 wireless adapter"), NETWORK("msk", "Marvell/SysKonnect Yukon II Gigabit Ethernet"), NETWORK("mxge", "Myricom Myri10GE 10Gb Ethernet card"), NETWORK("nfe", "NVIDIA nForce MCP Ethernet"), NETWORK("nge", "NatSemi PCI Gigabit Ethernet card"), NETWORK("nve", "NVIDIA nForce MCP Ethernet"), NETWORK("nxge", "Neterion Xframe 10GbE Server/Storage adapter"), NETWORK("pcn", "AMD Am79c79x PCI Ethernet card"), NETWORK("ral", "Ralink Technology IEEE 802.11 wireless adapter"), NETWORK("ray", "Raytheon Raylink 802.11 wireless adapter"), NETWORK("re", "RealTek 8139C+/8169/8169S/8110S PCI Ethernet card"), NETWORK("rl", "RealTek 8129/8139 PCI Ethernet card"), NETWORK("rue", "RealTek USB Ethernet card"), NETWORK("rum", "Ralink Technology USB IEEE 802.11 wireless adapter"), NETWORK("sf", "Adaptec AIC-6915 PCI Ethernet card"), + NETWORK("sge", "Silicon Integrated Systems SiS190/191 Ethernet"), NETWORK("sis", "SiS 900/SiS 7016 PCI Ethernet card"), #ifdef PC98 NETWORK("snc", "SONIC Ethernet card"), #endif NETWORK("sn", "SMC/Megahertz Ethernet card"), NETWORK("ste", "Sundance ST201 PCI Ethernet card"), NETWORK("stge", "Sundance/Tamarack TC9021 Gigabit Ethernet"), NETWORK("sk", "SysKonnect PCI Gigabit Ethernet card"), NETWORK("tx", "SMC 9432TX Ethernet card"), NETWORK("txp", "3Com 3cR990 Ethernet card"), NETWORK("ti", "Alteon Networks PCI Gigabit Ethernet card"), NETWORK("tl", "Texas Instruments ThunderLAN PCI Ethernet card"), NETWORK("uath", "Atheros AR5005UG and AR5005UX USB wireless adapter"), NETWORK("upgt", "Conexant/Intersil PrismGT USB wireless adapter"), NETWORK("ural", "Ralink Technology RT2500USB 802.11 wireless adapter"), NETWORK("urtw", "Realtek 8187L USB wireless adapter"), NETWORK("vge", "VIA VT612x PCI Gigabit Ethernet card"), NETWORK("vr", "VIA VT3043/VT86C100A Rhine PCI Ethernet card"), NETWORK("vlan", "IEEE 802.1Q VLAN network interface"), NETWORK("vx", "3COM 3c590 / 3c595 Ethernet card"), NETWORK("wb", "Winbond W89C840F PCI Ethernet card"), NETWORK("wi", "Lucent WaveLAN/IEEE 802.11 wireless adapter"), NETWORK("wpi", "Intel 3945ABG IEEE 802.11 wireless adapter"), NETWORK("xe", "Xircom/Intel EtherExpress Pro100/16 Ethernet card"), NETWORK("xl", "3COM 3c90x / 3c90xB PCI Ethernet card"), NETWORK("zyd", "ZyDAS ZD1211/ZD1211B USB 802.11 wireless adapter"), NETWORK("fwe", "FireWire Ethernet emulation"), NETWORK("fwip", "IP over FireWire"), NETWORK("plip", "Parallel Port IP (PLIP) peer connection"), NETWORK("lo", "Loop-back (local) network interface"), NETWORK("disc", "Software discard network interface"), { 0, NULL, NULL, 0 } }; Device * new_device(char *name) { Device *dev; dev = safe_malloc(sizeof(Device)); bzero(dev, sizeof(Device)); if (name) SAFE_STRCPY(dev->name, name); return dev; } /* Stubs for unimplemented strategy routines */ Boolean dummyInit(Device *dev) { return TRUE; } FILE * dummyGet(Device *dev, char *dist, Boolean probe) { return NULL; } void dummyShutdown(Device *dev) { return; } static int deviceTry(struct _devname dev, char *try, int i) { int fd; char unit[80]; snprintf(unit, sizeof unit, dev.name, i); snprintf(try, FILENAME_MAX, "/dev/%s", unit); if (isDebug()) msgDebug("deviceTry: attempting to open %s\n", try); fd = open(try, O_RDONLY); if (fd >= 0) { if (isDebug()) msgDebug("deviceTry: open of %s succeeded on first try.\n", try); } else { if (isDebug()) msgDebug("deviceTry: open of %s failed.\n", try); } return fd; } /* Register a new device in the devices array */ Device * deviceRegister(char *name, char *desc, char *devname, DeviceType type, Boolean enabled, Boolean (*init)(Device *), FILE * (*get)(Device *, char *, Boolean), void (*shutdown)(Device *), void *private) { Device *newdev = NULL; if (numDevs == DEV_MAX) msgFatal("Too many devices found!"); else { newdev = new_device(name); newdev->description = desc; newdev->devname = devname; newdev->type = type; newdev->enabled = enabled; newdev->init = init ? init : dummyInit; newdev->get = get ? get : dummyGet; newdev->shutdown = shutdown ? shutdown : dummyShutdown; newdev->private = private; Devices[numDevs] = newdev; Devices[++numDevs] = NULL; } return newdev; } /* Reset the registered device chain */ void deviceReset(void) { int i; for (i = 0; i < numDevs; i++) { DEVICE_SHUTDOWN(Devices[i]); /* XXX this potentially leaks Devices[i]->private if it's being * used to point to something dynamic, but you're not supposed * to call this routine at such times that some open instance * has its private ptr pointing somewhere anyway. XXX */ free(Devices[i]); } Devices[numDevs = 0] = NULL; } /* Get all device information for devices we have attached */ void deviceGetAll(void) { int i, j, fd, s; struct ifconf ifc; struct ifreq *ifptr, *end; int ifflags; char buffer[INTERFACE_MAX * sizeof(struct ifreq)]; char **names; msgNotify("Probing devices, please wait (this can take a while)..."); /* First go for the network interfaces. Stolen shamelessly from ifconfig! */ ifc.ifc_len = sizeof(buffer); ifc.ifc_buf = buffer; s = socket(AF_INET, SOCK_DGRAM, 0); if (s < 0) goto skipif; /* Jump over network iface probing */ if (ioctl(s, SIOCGIFCONF, (char *) &ifc) < 0) goto skipif; /* Jump over network iface probing */ close(s); ifflags = ifc.ifc_req->ifr_flags; end = (struct ifreq *) (ifc.ifc_buf + ifc.ifc_len); for (ifptr = ifc.ifc_req; ifptr < end; ifptr++) { char *descr; /* If it's not a link entry, forget it */ if (ifptr->ifr_ifru.ifru_addr.sa_family != AF_LINK) goto loopend; /* Eliminate network devices that don't make sense */ if (!strncmp(ifptr->ifr_name, "lo", 2)) goto loopend; /* If we have a slip device, don't register it */ if (!strncmp(ifptr->ifr_name, "sl", 2)) { goto loopend; } /* And the same for ppp */ if (!strncmp(ifptr->ifr_name, "tun", 3) || !strncmp(ifptr->ifr_name, "ppp", 3)) { goto loopend; } /* Try and find its description */ for (i = 0, descr = NULL; device_names[i].name; i++) { int len = strlen(device_names[i].name); if (!ifptr->ifr_name || !ifptr->ifr_name[0]) continue; else if (!strncmp(ifptr->ifr_name, device_names[i].name, len)) { descr = device_names[i].description; break; } } if (!descr) descr = ""; deviceRegister(ifptr->ifr_name, descr, strdup(ifptr->ifr_name), DEVICE_TYPE_NETWORK, TRUE, mediaInitNetwork, NULL, mediaShutdownNetwork, NULL); if (isDebug()) msgDebug("Found a network device named %s\n", ifptr->ifr_name); close(s); if ((s = socket(AF_INET, SOCK_DGRAM, 0)) < 0) continue; loopend: if (ifptr->ifr_addr.sa_len) /* I'm not sure why this is here - it's inherited */ ifptr = (struct ifreq *)((caddr_t)ifptr + ifptr->ifr_addr.sa_len - sizeof(struct sockaddr)); close(s); } skipif: /* Next, try to find all the types of devices one might need * during the second stage of the installation. */ for (i = 0; device_names[i].name; i++) { for (j = 0; j < device_names[i].max; j++) { char try[FILENAME_MAX]; switch(device_names[i].type) { case DEVICE_TYPE_CDROM: fd = deviceTry(device_names[i], try, j); if (fd >= 0 || errno == EBUSY) { /* EBUSY if already mounted */ char n[BUFSIZ]; if (fd >= 0) close(fd); snprintf(n, sizeof n, device_names[i].name, j); deviceRegister(strdup(n), device_names[i].description, strdup(try), DEVICE_TYPE_CDROM, TRUE, mediaInitCDROM, mediaGetCDROM, mediaShutdownCDROM, NULL); if (isDebug()) msgDebug("Found a CDROM device for %s\n", try); } break; case DEVICE_TYPE_DISK: /* nothing to do */ break; case DEVICE_TYPE_FLOPPY: fd = deviceTry(device_names[i], try, j); if (fd >= 0) { char n[BUFSIZ]; close(fd); snprintf(n, sizeof n, device_names[i].name, j); deviceRegister(strdup(n), device_names[i].description, strdup(try), DEVICE_TYPE_FLOPPY, TRUE, mediaInitFloppy, mediaGetFloppy, mediaShutdownFloppy, NULL); if (isDebug()) msgDebug("Found a floppy device for %s\n", try); } break; case DEVICE_TYPE_USB: fd = deviceTry(device_names[i], try, j); if (fd >= 0) { char n[BUFSIZ]; close(fd); snprintf(n, sizeof(n), device_names[i].name, j); deviceRegister(strdup(n), device_names[i].description, strdup(try), DEVICE_TYPE_USB, TRUE, mediaInitUSB, mediaGetUSB, mediaShutdownUSB, NULL); if (isDebug()) msgDebug("Found a USB disk for %s\n", try); } break; case DEVICE_TYPE_NETWORK: fd = deviceTry(device_names[i], try, j); /* The only network devices that you can open this way are serial ones */ if (fd >= 0) { char *newdesc, *cp; close(fd); cp = device_names[i].description; /* Serial devices get a slip and ppp device each, if supported */ newdesc = safe_malloc(strlen(cp) + 40); sprintf(newdesc, cp, "SLIP interface", try, j + 1); deviceRegister("sl0", newdesc, strdup(try), DEVICE_TYPE_NETWORK, TRUE, mediaInitNetwork, NULL, mediaShutdownNetwork, NULL); msgDebug("Add mapping for %s to sl0\n", try); newdesc = safe_malloc(strlen(cp) + 50); sprintf(newdesc, cp, "PPP interface", try, j + 1); deviceRegister("ppp0", newdesc, strdup(try), DEVICE_TYPE_NETWORK, TRUE, mediaInitNetwork, NULL, mediaShutdownNetwork, NULL); if (isDebug()) msgDebug("Add mapping for %s to ppp0\n", try); } break; default: break; } } } /* Finally, go get the disks and look for partitions to register */ if ((names = Disk_Names()) != NULL) { int i; for (i = 0; names[i]; i++) { Chunk *c1; Disk *d; /* Ignore memory disks */ if (!strncmp(names[i], "md", 2)) continue; /* * XXX * Due to unknown reasons, Disk_Names() returns SCSI CDROM as a * valid disk. This is main reason why sysinstall presents SCSI * CDROM to available disks in Fdisk/Label menu. In addition, * adding a blank SCSI CDROM to the menu generates floating point * exception in sparc64. Disk_Names() just extracts sysctl * "kern.disks". Why GEOM treats SCSI CDROM as a disk is beyond * me and that should be investigated. * For temporary workaround, ignore SCSI CDROM device. */ if (!strncmp(names[i], "cd", 2)) continue; d = Open_Disk(names[i]); if (!d) { msgDebug("Unable to open disk %s\n", names[i]); continue; } deviceRegister(names[i], names[i], d->name, DEVICE_TYPE_DISK, FALSE, dummyInit, dummyGet, dummyShutdown, d); if (isDebug()) msgDebug("Found a disk device named %s\n", names[i]); /* Look for existing DOS partitions to register as "DOS media devices" * XXX: libdisks handling of extended partitions is too * simplistic - it does not handle them containing (for * example) UFS partitions */ for (c1 = d->chunks->part; c1; c1 = c1->next) { if (c1->type == fat || c1->type == efi || c1->type == extended) { Device *dev; char devname[80]; /* Got one! */ snprintf(devname, sizeof devname, "/dev/%s", c1->name); dev = deviceRegister(c1->name, c1->name, strdup(devname), DEVICE_TYPE_DOS, TRUE, mediaInitDOS, mediaGetDOS, mediaShutdownDOS, NULL); dev->private = c1; if (isDebug()) msgDebug("Found a DOS partition %s\n", c1->name); } else if (c1->type == freebsd) { Device *dev; char devname[80]; Chunk *c2; for (c2 = c1->part; c2; c2 = c2->next) { if (c2->type != part || c2->subtype != 7) continue; /* Got one! */ snprintf(devname, sizeof devname, "/dev/%s", c1->name); dev = deviceRegister(c2->name, c2->name, strdup(devname), DEVICE_TYPE_UFS, TRUE, mediaInitUFS, mediaGetUFS, mediaShutdownUFS, NULL); dev->private = c2; if (isDebug()) msgDebug("Found a UFS sub-partition %s\n", c2->name); } } } } free(names); } dialog_clear_norefresh(); } /* Rescan all devices, after closing previous set - convenience function */ void deviceRescan(void) { deviceReset(); deviceGetAll(); } /* * Find all devices that match the criteria, allowing "wildcarding" as well * by allowing NULL or ANY values to match all. The array returned is static * and may be used until the next invocation of deviceFind(). */ Device ** deviceFind(char *name, DeviceType class) { static Device *found[DEV_MAX]; int i, j; j = 0; for (i = 0; i < numDevs; i++) { if ((!name || !strcmp(Devices[i]->name, name)) && (class == DEVICE_TYPE_ANY || class == Devices[i]->type)) found[j++] = Devices[i]; } found[j] = NULL; return j ? found : NULL; } Device ** deviceFindDescr(char *name, char *desc, DeviceType class) { static Device *found[DEV_MAX]; int i, j; j = 0; for (i = 0; i < numDevs; i++) { if ((!name || !strcmp(Devices[i]->name, name)) && (!desc || !strcmp(Devices[i]->description, desc)) && (class == DEVICE_TYPE_ANY || class == Devices[i]->type)) found[j++] = Devices[i]; } found[j] = NULL; return j ? found : NULL; } int deviceCount(Device **devs) { int i; if (!devs) return 0; for (i = 0; devs[i]; i++); return i; } /* * Create a menu listing all the devices of a certain type in the system. * The passed-in menu is expected to be a "prototype" from which the new * menu is cloned. */ DMenu * deviceCreateMenu(DMenu *menu, DeviceType type, int (*hook)(dialogMenuItem *d), int (*check)(dialogMenuItem *d)) { Device **devs; int numdevs; DMenu *tmp = NULL; int i, j; devs = deviceFind(NULL, type); numdevs = deviceCount(devs); if (!numdevs) return NULL; tmp = (DMenu *)safe_malloc(sizeof(DMenu) + (sizeof(dialogMenuItem) * (numdevs + 1))); bcopy(menu, tmp, sizeof(DMenu)); for (i = 0; devs[i]; i++) { tmp->items[i].prompt = devs[i]->name; for (j = 0; j < numDevs; j++) { if (devs[i] == Devices[j]) { tmp->items[i].title = Devices[j]->description; break; } } if (j == numDevs) tmp->items[i].title = ""; tmp->items[i].fire = hook; tmp->items[i].checked = check; } tmp->items[i].title = NULL; return tmp; } Index: user/imp/tbemd =================================================================== --- user/imp/tbemd (revision 206684) +++ user/imp/tbemd (revision 206685) Property changes on: user/imp/tbemd ___________________________________________________________________ Modified: svn:mergeinfo ## -0,0 +0,1 ## Merged /head:r206628-206684