Index: head/usr.bin/mkuzip/mkuzip.8 =================================================================== --- head/usr.bin/mkuzip/mkuzip.8 (revision 296625) +++ head/usr.bin/mkuzip/mkuzip.8 (revision 296626) @@ -1,172 +1,175 @@ .\"- .\" Copyright (c) 2004-2016 Maxim Sobolev .\" All rights reserved. .\" .\" Redistribution and use in source and binary forms, with or without .\" modification, are permitted provided that the following conditions .\" are met: .\" 1. Redistributions of source code must retain the above copyright .\" notice, this list of conditions and the following disclaimer. .\" 2. Redistributions in binary form must reproduce the above copyright .\" notice, this list of conditions and the following disclaimer in the .\" documentation and/or other materials provided with the distribution. .\" .\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND .\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE .\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE .\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE .\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL .\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS .\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) .\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT .\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY .\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF .\" SUCH DAMAGE. .\" .\" $FreeBSD$ .\" .Dd March 17, 2006 .Dt MKUZIP 8 .Os .Sh NAME .Nm mkuzip .Nd compress disk image for use with .Xr geom_uzip 4 class .Sh SYNOPSIS .Nm .Op Fl v .Op Fl o Ar outfile .Op Fl s Ar cluster_size .Ar infile .Sh DESCRIPTION The .Nm utility compresses a disk image file so that the .Xr geom_uzip 4 class will be able to decompress the resulting image at run-time. This allows for a significant reduction of size of disk image at the expense of some CPU time required to decompress the data each time it is read. The .Nm utility works in two phases: .Bl -enum .It An .Ar infile image is split into clusters; each cluster is compressed using .Xr zlib 3 or .Xr lzma 3 . .It The resulting set of compressed clusters along with headers that allow locating each individual cluster is written to the output file. .El .Pp The options are: .Bl -tag -width indent .It Fl o Ar outfile Name of the output file .Ar outfile . The default is to use the input name with the suffix .Pa .uzip for the .Xr zlib 3 compression or .Pa .ulzma for the .Xr lzma 3 . .It Fl L Use .Xr lzma 3 compression algorithm instead of the default .Xr zlib 3 . The .Xr lzma 3 provides noticeable better compression levels on the same data set at the expense of much slower compression speed (10-20x) and somewhat slower decompression (2-3x). .It Fl s Ar cluster_size Split the image into clusters of .Ar cluster_size bytes, 16384 bytes by default. The .Ar cluster_size should be a multiple of 512 bytes. .It Fl v Display verbose messages. .It Fl Z Disable zero-blocks detection and elimination. When this option is set, the .Nm would compress empty blocks (i.e. clusters that consist of only zero bytes) just as it would any other block. When the option is not set, the .Nm detects such blocks and skips them from the output. Setting .Fl Z results is slight increase of compressed image size, typically less than 0.1% of a final size of the compressed image. .It Fl d Enable de-duplication. When the option is enabled the .Nm detects identical blocks in the input and replaces each subsequent occurence of such block with pointer to the very first one in the output. Setting this option results is moderate decrease of compressed image size, typically around 3-5% of a final size of the compressed image. +.It Fl S +Print summary about the compression ratio as well as output +file size after file has been processed. .El .Sh NOTES The compression ratio largely depends on the cluster size used. .\" The following two sentences are unclear: how can gzip(1) be .\" used in a comparable fashion, and wouldn't a gzip-compressed .\" image suffer from larger cluster sizes as well? For large cluster sizes (16K and higher), typical compression ratios are only 1-2% less than those achieved with .Xr gzip 1 . However, it should be kept in mind that larger cluster sizes lead to higher overhead in the .Xr geom_uzip 4 class, as the class has to decompress the whole cluster even if only a few bytes from that cluster have to be read. .Pp The .Nm utility inserts a short shell script at the beginning of the generated image, which makes it possible to .Dq run the image just like any other shell script. The script tries to load the .Xr geom_uzip 4 class if it is not loaded, configure the image as an .Xr md 4 disk device using .Xr mdconfig 8 , and automatically mount it using .Xr mount_cd9660 8 on the mount point provided as the first argument to the script. .Pp The de-duplication is a .Fx specific feature and while it does not require any changes to on-disk compressed image format, however it did require some matching changes to the .Xr geom_uzip 4 to handle resulting images correctly. .Sh EXIT STATUS .Ex -std .Sh SEE ALSO .Xr gzip 1 , .Xr xz 1 , .Xr zlib 3 , .Xr lzma 3 , .Xr geom 4 , .Xr geom_uzip 4 , .Xr md 4 , .Xr mdconfig 8 , .Xr mount_cd9660 8 .Sh AUTHORS .An Maxim Sobolev Aq Mt sobomax@FreeBSD.org Index: head/usr.bin/mkuzip/mkuzip.c =================================================================== --- head/usr.bin/mkuzip/mkuzip.c (revision 296625) +++ head/usr.bin/mkuzip/mkuzip.c (revision 296626) @@ -1,373 +1,379 @@ /* * Copyright (c) 2004-2016 Maxim Sobolev * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * */ #include __FBSDID("$FreeBSD$"); #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include "mkuzip.h" #include "mkuz_cloop.h" #include "mkuz_blockcache.h" #include "mkuz_zlib.h" #include "mkuz_lzma.h" #define DEFINE_RAW_METHOD(func, rval, args...) typedef rval (*func##_t)(args) #define DEFAULT_CLSTSIZE 16384 DEFINE_RAW_METHOD(f_init, void *, uint32_t); DEFINE_RAW_METHOD(f_compress, void, const char *, uint32_t *); struct mkuz_format { const char *magic; const char *default_sufx; f_init_t f_init; f_compress_t f_compress; }; static struct mkuz_format uzip_fmt = { .magic = CLOOP_MAGIC_ZLIB, .default_sufx = DEFAULT_SUFX_ZLIB, .f_init = &mkuz_zlib_init, .f_compress = &mkuz_zlib_compress }; static struct mkuz_format ulzma_fmt = { .magic = CLOOP_MAGIC_LZMA, .default_sufx = DEFAULT_SUFX_LZMA, .f_init = &mkuz_lzma_init, .f_compress = &mkuz_lzma_compress }; static char *readblock(int, char *, u_int32_t); static void usage(void); static void cleanup(void); static int memvcmp(const void *, unsigned char, size_t); static char *cleanfile = NULL; int main(int argc, char **argv) { char *iname, *oname, *obuf, *ibuf; uint64_t *toc; int fdr, fdw, i, opt, verbose, no_zcomp, tmp, en_dedup; + int summary; struct iovec iov[2]; struct stat sb; uint32_t destlen; uint64_t offset, last_offset; struct cloop_header hdr; struct mkuz_blkcache_hit *chit; const struct mkuz_format *handler; memset(&hdr, 0, sizeof(hdr)); hdr.blksz = DEFAULT_CLSTSIZE; oname = NULL; verbose = 0; no_zcomp = 0; en_dedup = 0; + summary = 0; handler = &uzip_fmt; - while((opt = getopt(argc, argv, "o:s:vZdL")) != -1) { + while((opt = getopt(argc, argv, "o:s:vZdLS")) != -1) { switch(opt) { case 'o': oname = optarg; break; case 's': tmp = atoi(optarg); if (tmp <= 0) { errx(1, "invalid cluster size specified: %s", optarg); /* Not reached */ } hdr.blksz = tmp; break; case 'v': verbose = 1; break; case 'Z': no_zcomp = 1; break; case 'd': en_dedup = 1; break; case 'L': handler = &ulzma_fmt; break; + case 'S': + summary = 1; + break; + default: usage(); /* Not reached */ } } argc -= optind; argv += optind; if (argc != 1) { usage(); /* Not reached */ } strcpy(hdr.magic, handler->magic); if (en_dedup != 0) { hdr.magic[CLOOP_OFS_VERSN] = CLOOP_MAJVER_3; hdr.magic[CLOOP_OFS_COMPR] = tolower(hdr.magic[CLOOP_OFS_COMPR]); } obuf = handler->f_init(hdr.blksz); iname = argv[0]; if (oname == NULL) { asprintf(&oname, "%s%s", iname, handler->default_sufx); if (oname == NULL) { err(1, "can't allocate memory"); /* Not reached */ } } ibuf = mkuz_safe_malloc(hdr.blksz); signal(SIGHUP, exit); signal(SIGINT, exit); signal(SIGTERM, exit); signal(SIGXCPU, exit); signal(SIGXFSZ, exit); atexit(cleanup); fdr = open(iname, O_RDONLY); if (fdr < 0) { err(1, "open(%s)", iname); /* Not reached */ } if (fstat(fdr, &sb) != 0) { err(1, "fstat(%s)", iname); /* Not reached */ } if (S_ISCHR(sb.st_mode)) { off_t ms; if (ioctl(fdr, DIOCGMEDIASIZE, &ms) < 0) { err(1, "ioctl(DIOCGMEDIASIZE)"); /* Not reached */ } sb.st_size = ms; } else if (!S_ISREG(sb.st_mode)) { fprintf(stderr, "%s: not a character device or regular file\n", iname); exit(1); } hdr.nblocks = sb.st_size / hdr.blksz; if ((sb.st_size % hdr.blksz) != 0) { if (verbose != 0) fprintf(stderr, "file size is not multiple " "of %d, padding data\n", hdr.blksz); hdr.nblocks++; } toc = mkuz_safe_malloc((hdr.nblocks + 1) * sizeof(*toc)); fdw = open(oname, O_WRONLY | O_TRUNC | O_CREAT, S_IRWXU | S_IRGRP | S_IXGRP | S_IROTH | S_IXOTH); if (fdw < 0) { err(1, "open(%s)", oname); /* Not reached */ } cleanfile = oname; /* Prepare header that we will write later when we have index ready. */ iov[0].iov_base = (char *)&hdr; iov[0].iov_len = sizeof(hdr); iov[1].iov_base = (char *)toc; iov[1].iov_len = (hdr.nblocks + 1) * sizeof(*toc); offset = iov[0].iov_len + iov[1].iov_len; /* Reserve space for header */ lseek(fdw, offset, SEEK_SET); if (verbose != 0) fprintf(stderr, "data size %ju bytes, number of clusters " "%u, index length %zu bytes\n", sb.st_size, hdr.nblocks, iov[1].iov_len); last_offset = 0; for(i = 0; i == 0 || ibuf != NULL; i++) { ibuf = readblock(fdr, ibuf, hdr.blksz); if (ibuf != NULL) { if (no_zcomp == 0 && \ memvcmp(ibuf, '\0', hdr.blksz) != 0) { /* All zeroes block */ destlen = 0; } else { handler->f_compress(ibuf, &destlen); } } else { destlen = DEV_BSIZE - (offset % DEV_BSIZE); memset(obuf, 0, destlen); if (verbose != 0) fprintf(stderr, "padding data with %lu bytes " "so that file size is multiple of %d\n", (u_long)destlen, DEV_BSIZE); } if (destlen > 0 && en_dedup != 0) { chit = mkuz_blkcache_regblock(fdw, i, offset, destlen, obuf); /* * There should be at least one non-empty block * between us and the backref'ed offset, otherwise * we won't be able to parse that sequence correctly * as it would be indistinguishible from another * empty block. */ if (chit != NULL && chit->offset == last_offset) { chit = NULL; } } else { chit = NULL; } if (chit != NULL) { toc[i] = htobe64(chit->offset); } else { if (destlen > 0 && write(fdw, obuf, destlen) < 0) { err(1, "write(%s)", oname); /* Not reached */ } toc[i] = htobe64(offset); last_offset = offset; offset += destlen; } if (ibuf != NULL && verbose != 0) { fprintf(stderr, "cluster #%d, in %u bytes, " "out len=%lu offset=%lu", i, hdr.blksz, chit == NULL ? (u_long)destlen : 0, (u_long)be64toh(toc[i])); if (chit != NULL) { fprintf(stderr, " (backref'ed to #%d)", chit->blkno); } fprintf(stderr, "\n"); } } close(fdr); - if (verbose != 0) + if (verbose != 0 || summary != 0) fprintf(stderr, "compressed data to %ju bytes, saved %lld " "bytes, %.2f%% decrease.\n", offset, (long long)(sb.st_size - offset), 100.0 * (long long)(sb.st_size - offset) / (float)sb.st_size); /* Convert to big endian */ hdr.blksz = htonl(hdr.blksz); hdr.nblocks = htonl(hdr.nblocks); /* Write headers into pre-allocated space */ lseek(fdw, 0, SEEK_SET); if (writev(fdw, iov, 2) < 0) { err(1, "writev(%s)", oname); /* Not reached */ } cleanfile = NULL; close(fdw); exit(0); } static char * readblock(int fd, char *ibuf, u_int32_t clstsize) { int numread; bzero(ibuf, clstsize); numread = read(fd, ibuf, clstsize); if (numread < 0) { err(1, "read() failed"); /* Not reached */ } if (numread == 0) { return NULL; } return ibuf; } static void usage(void) { - fprintf(stderr, "usage: mkuzip [-vZdL] [-o outfile] [-s cluster_size] " + fprintf(stderr, "usage: mkuzip [-vZdLS] [-o outfile] [-s cluster_size] " "infile\n"); exit(1); } void * mkuz_safe_malloc(size_t size) { void *retval; retval = malloc(size); if (retval == NULL) { err(1, "can't allocate memory"); /* Not reached */ } return retval; } static void cleanup(void) { if (cleanfile != NULL) unlink(cleanfile); } static int memvcmp(const void *memory, unsigned char val, size_t size) { const u_char *mm; mm = (const u_char *)memory; return (*mm == val) && memcmp(mm, mm + 1, size - 1) == 0; }