Page MenuHomeFreeBSD
Differential D15175

multimedia/ffmpeg: update to 4.0
ClosedPublic
Actions

Authored by jbeich on Apr 23 2018, 8:16 PM.
Tags
None
Referenced Files
Unknown Object (File)
Fri, Mar 21, 5:43 AM
Unknown Object (File)
Tue, Mar 18, 2:39 PM
Unknown Object (File)
Thu, Mar 13, 12:11 AM
Unknown Object (File)
Tue, Mar 11, 7:09 AM
Unknown Object (File)
Sat, Mar 1, 2:48 PM
Unknown Object (File)
Sun, Feb 23, 10:16 AM
Unknown Object (File)
Feb 22 2025, 10:01 PM
Unknown Object (File)
Feb 20 2025, 6:17 AM
View All Files
Subscribers
val_packett.cool

Details

Reviewers
None
Group Reviewers
multimedia
Commits
rP468844: multimedia/ffmpeg: update to 4.0
Summary
  • Add more options (some are hidden due to missing dependencies)
  • Make LibreSSL use to libtls
  • Drop FFSERVER after 6b35a83214f1
  • Drop -mstackrealign after ea2f72a2c14c
Test Plan
  • poudriere bulk -t multimedia/ffmpeg is green on 10.3 i386/amd64, 10.4 i386/amd64, 11.1 i386/amd64/armv6/aarch64, 12.0 i386/amd64/armv6/armv7/aarch64
  • consumers will be tested via bug 227726

Diff Detail

Lint
No Lint Coverage
Unit
No Test Coverage
Build Status
Buildable 16275
Build 16222: arc lint + arc unit

Event Timeline

jbeich created this revision.Apr 23 2018, 8:16 PM
Harbormaster completed remote builds in B16273: Diff 41779.Apr 23 2018, 8:16 PM
jbeich updated this revision to Diff 41781.Apr 23 2018, 8:34 PM
jbeich edited the summary of this revision. (Show Details)
jbeich edited the test plan for this revision. (Show Details)
  • Connect LV2 to the build
Harbormaster completed remote builds in B16275: Diff 41781.Apr 23 2018, 8:34 PM
jbeich updated this revision to Diff 41782.Apr 23 2018, 8:49 PM
  • Unbreak LV2=on build
Harbormaster completed remote builds in B16276: Diff 41782.Apr 23 2018, 8:49 PM
jbeich updated this revision to Diff 41788.Apr 24 2018, 1:24 AM
  • Add BEIGNET option as it has a dependency
Harbormaster completed remote builds in B16278: Diff 41788.Apr 24 2018, 1:24 AM
val_packett.cool added a subscriber: val_packett.cool.Apr 24 2018, 10:28 AM
jbeich updated this revision to Diff 41964.Apr 29 2018, 1:02 AM

Rebased after rP468598 to skip gstreamer1-libav bustage

Harbormaster completed remote builds in B16333: Diff 41964.Apr 29 2018, 1:02 AM
This revision was not accepted when it landed; it landed in state Needs Review.May 2 2018, 3:07 PM
Closed by commit rP468844: multimedia/ffmpeg: update to 4.0 (authored by jbeich). · Explain Why
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

PathSize
multimedia/
ffmpeg/
63 lines
6 lines
files/
14 lines
49 lines

Diff 41781

View Options

multimedia/ffmpeg/Makefile

  • This file was added.
/*-
* SPDX-License-Identifier: BSD-2-Clause-FreeBSD
*
* Copyright (c) 2022 The FreeBSD Foundation
*
* This software was developed by Mark Johnston under sponsorship from
* the FreeBSD Foundation.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are
* met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in
* the documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*/
#include <sys/param.h>
#include <sys/errno.h>
#include <sys/queue.h>
#include <assert.h>
#include <fcntl.h>
#include <stdbool.h>
#include <stddef.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <util.h>
#include "makefs.h"
#include "zfs.h"
#define VDEV_LABEL_SPACE \
((off_t)(VDEV_LABEL_START_SIZE + VDEV_LABEL_END_SIZE))
_Static_assert(VDEV_LABEL_SPACE <= MINDEVSIZE, "");
#define MINMSSIZE ((off_t)1 << 24) /* 16MB */
#define DFLTMSSIZE ((off_t)1 << 29) /* 512MB */
#define MAXMSSIZE ((off_t)1 << 34) /* 16GB */
#define INDIR_LEVELS 6
/* Indirect blocks are always 128KB. */
#define BLKPTR_PER_INDIR (MAXBLOCKSIZE / sizeof(blkptr_t))
struct dnode_cursor {
char inddir[INDIR_LEVELS][MAXBLOCKSIZE];
off_t indloc;
off_t indspace;
dnode_phys_t *dnode;
off_t dataoff;
off_t datablksz;
};
void
zfs_prep_opts(fsinfo_t *fsopts)
{
zfs_opt_t *zfs = ecalloc(1, sizeof(*zfs));
const option_t zfs_options[] = {
{ '\0', "bootfs", &zfs->bootfs, OPT_STRPTR,
0, 0, "Bootable dataset" },
{ '\0', "mssize", &zfs->mssize, OPT_INT64,
MINMSSIZE, MAXMSSIZE, "Metaslab size" },
{ '\0', "poolname", &zfs->poolname, OPT_STRPTR,
0, 0, "ZFS pool name" },
{ '\0', "rootpath", &zfs->rootpath, OPT_STRPTR,
0, 0, "Prefix for all dataset mount points" },
{ '\0', "ashift", &zfs->ashift, OPT_INT32,
MINBLOCKSHIFT, MAXBLOCKSHIFT, "ZFS pool ashift" },
{ '\0', "nowarn", &zfs->nowarn, OPT_BOOL,
0, 0, "Suppress warning about experimental ZFS support" },
{ .name = NULL }
};
STAILQ_INIT(&zfs->datasetdescs);
fsopts->fs_specific = zfs;
fsopts->fs_options = copy_opts(zfs_options);
}
int
zfs_parse_opts(const char *option, fsinfo_t *fsopts)
{
zfs_opt_t *zfs;
struct dataset_desc *dsdesc;
char buf[BUFSIZ], *opt, *val;
int rv;
zfs = fsopts->fs_specific;
opt = val = estrdup(option);
opt = strsep(&val, "=");
if (strcmp(opt, "fs") == 0) {
if (val == NULL)
errx(1, "invalid filesystem parameters `%s'", option);
/*
* Dataset descriptions will be parsed later, in dsl_init().
* Just stash them away for now.
*/
dsdesc = ecalloc(1, sizeof(*dsdesc));
dsdesc->params = estrdup(val);
free(opt);
STAILQ_INSERT_TAIL(&zfs->datasetdescs, dsdesc, next);
return (1);
}
free(opt);
rv = set_option(fsopts->fs_options, option, buf, sizeof(buf));
return (rv == -1 ? 0 : 1);
}
static void
zfs_size_vdev(fsinfo_t *fsopts)
{
zfs_opt_t *zfs;
off_t asize, mssize, vdevsize, vdevsize1;
zfs = fsopts->fs_specific;
assert(fsopts->maxsize != 0);
assert(zfs->ashift != 0);
/*
* Figure out how big the vdev should be.
*/
vdevsize = rounddown2(fsopts->maxsize, 1 << zfs->ashift);
if (vdevsize < MINDEVSIZE)
errx(1, "maximum image size is too small");
if (vdevsize < fsopts->minsize || vdevsize > fsopts->maxsize) {
errx(1, "image size bounds must be multiples of %d",
1 << zfs->ashift);
}
asize = vdevsize - VDEV_LABEL_SPACE;
/*
* Size metaslabs according to the following heuristic:
* - provide at least 8 metaslabs,
* - without using a metaslab size larger than 512MB.
* This approximates what OpenZFS does without being complicated. In
* practice we expect pools to be expanded upon first use, and OpenZFS
* does not resize metaslabs in that case, so there is no right answer
* here. In general we want to provide large metaslabs even if the
* image size is small, and 512MB is a reasonable size for pools up to
* several hundred gigabytes.
*
* The user may override this heuristic using the "-o mssize" option.
*/
mssize = zfs->mssize;
if (mssize == 0) {
mssize = MAX(MIN(asize / 8, DFLTMSSIZE), MINMSSIZE);
if (!powerof2(mssize))
mssize = 1l << (flsll(mssize) - 1);
}
if (!powerof2(mssize))
errx(1, "metaslab size must be a power of 2");
/*
* If we have some slop left over, try to cover it by resizing the vdev,
* subject to the maxsize and minsize parameters.
*/
if (asize % mssize != 0) {
vdevsize1 = rounddown2(asize, mssize) + VDEV_LABEL_SPACE;
if (vdevsize1 < fsopts->minsize)
vdevsize1 = roundup2(asize, mssize) + VDEV_LABEL_SPACE;
if (vdevsize1 <= fsopts->maxsize)
vdevsize = vdevsize1;
}
asize = vdevsize - VDEV_LABEL_SPACE;
zfs->asize = asize;
zfs->vdevsize = vdevsize;
zfs->mssize = mssize;
zfs->msshift = flsll(mssize) - 1;
zfs->mscount = asize / mssize;
}
/*
* Validate options and set some default values.
*/
static void
zfs_check_opts(fsinfo_t *fsopts)
{
zfs_opt_t *zfs;
zfs = fsopts->fs_specific;
if (fsopts->offset != 0)
errx(1, "unhandled offset option");
if (fsopts->maxsize == 0)
errx(1, "an image size must be specified");
if (zfs->poolname == NULL)
errx(1, "a pool name must be specified");
if (zfs->rootpath == NULL)
easprintf(&zfs->rootpath, "/%s", zfs->poolname);
if (zfs->rootpath[0] != '/')
errx(1, "mountpoint `%s' must be absolute", zfs->rootpath);
if (zfs->ashift == 0)
zfs->ashift = 12;
zfs_size_vdev(fsopts);
}
void
zfs_cleanup_opts(fsinfo_t *fsopts)
{
struct dataset_desc *d, *tmp;
zfs_opt_t *zfs;
zfs = fsopts->fs_specific;
free(zfs->rootpath);
free(zfs->bootfs);
free(__DECONST(void *, zfs->poolname));
STAILQ_FOREACH_SAFE(d, &zfs->datasetdescs, next, tmp) {
free(d->params);
free(d);
}
free(zfs);
free(fsopts->fs_options);
}
static size_t
nvlist_size(const nvlist_t *nvl)
{
return (sizeof(nvl->nv_header) + nvl->nv_size);
}
static void
nvlist_copy(const nvlist_t *nvl, char *buf, size_t sz)
{
assert(sz >= nvlist_size(nvl));
memcpy(buf, &nvl->nv_header, sizeof(nvl->nv_header));
memcpy(buf + sizeof(nvl->nv_header), nvl->nv_data, nvl->nv_size);
}
static nvlist_t *
pool_config_nvcreate(zfs_opt_t *zfs)
{
nvlist_t *featuresnv, *poolnv;
poolnv = nvlist_create(NV_UNIQUE_NAME);
nvlist_add_uint64(poolnv, ZPOOL_CONFIG_POOL_TXG, TXG);
nvlist_add_uint64(poolnv, ZPOOL_CONFIG_VERSION, SPA_VERSION);
nvlist_add_uint64(poolnv, ZPOOL_CONFIG_POOL_STATE, POOL_STATE_EXPORTED);
nvlist_add_string(poolnv, ZPOOL_CONFIG_POOL_NAME, zfs->poolname);
nvlist_add_uint64(poolnv, ZPOOL_CONFIG_POOL_GUID, zfs->poolguid);
nvlist_add_uint64(poolnv, ZPOOL_CONFIG_TOP_GUID, zfs->vdevguid);
nvlist_add_uint64(poolnv, ZPOOL_CONFIG_GUID, zfs->vdevguid);
nvlist_add_uint64(poolnv, ZPOOL_CONFIG_VDEV_CHILDREN, 1);
featuresnv = nvlist_create(NV_UNIQUE_NAME);
nvlist_add_nvlist(poolnv, ZPOOL_CONFIG_FEATURES_FOR_READ, featuresnv);
nvlist_destroy(featuresnv);
return (poolnv);
}
static nvlist_t *
pool_disk_vdev_config_nvcreate(zfs_opt_t *zfs)
{
nvlist_t *diskvdevnv;
assert(zfs->objarrid != 0);
diskvdevnv = nvlist_create(NV_UNIQUE_NAME);
nvlist_add_string(diskvdevnv, ZPOOL_CONFIG_TYPE, VDEV_TYPE_DISK);
nvlist_add_uint64(diskvdevnv, ZPOOL_CONFIG_ASHIFT, zfs->ashift);
nvlist_add_uint64(diskvdevnv, ZPOOL_CONFIG_ASIZE, zfs->asize);
nvlist_add_uint64(diskvdevnv, ZPOOL_CONFIG_GUID, zfs->vdevguid);
nvlist_add_uint64(diskvdevnv, ZPOOL_CONFIG_ID, 0);
nvlist_add_string(diskvdevnv, ZPOOL_CONFIG_PATH, "/dev/null");
nvlist_add_uint64(diskvdevnv, ZPOOL_CONFIG_WHOLE_DISK, 1);
nvlist_add_uint64(diskvdevnv, ZPOOL_CONFIG_CREATE_TXG, TXG);
nvlist_add_uint64(diskvdevnv, ZPOOL_CONFIG_METASLAB_ARRAY,
zfs->objarrid);
nvlist_add_uint64(diskvdevnv, ZPOOL_CONFIG_METASLAB_SHIFT,
zfs->msshift);
return (diskvdevnv);
}
static nvlist_t *
pool_root_vdev_config_nvcreate(zfs_opt_t *zfs)
{
nvlist_t *diskvdevnv, *rootvdevnv;
diskvdevnv = pool_disk_vdev_config_nvcreate(zfs);
rootvdevnv = nvlist_create(NV_UNIQUE_NAME);
nvlist_add_uint64(rootvdevnv, ZPOOL_CONFIG_ID, 0);
nvlist_add_uint64(rootvdevnv, ZPOOL_CONFIG_GUID, zfs->poolguid);
nvlist_add_string(rootvdevnv, ZPOOL_CONFIG_TYPE, VDEV_TYPE_ROOT);
nvlist_add_uint64(rootvdevnv, ZPOOL_CONFIG_CREATE_TXG, TXG);
nvlist_add_nvlist_array(rootvdevnv, ZPOOL_CONFIG_CHILDREN, &diskvdevnv,
1);
nvlist_destroy(diskvdevnv);
return (rootvdevnv);
}
/*
* Create the pool's "config" object, which contains an nvlist describing pool
* parameters and the vdev topology. It is similar but not identical to the
* nvlist stored in vdev labels. The main difference is that vdev labels do not
* describe the full vdev tree and in particular do not contain the "root"
* meta-vdev.
*/
static void
pool_init_objdir_config(zfs_opt_t *zfs, zfs_zap_t *objdir)
{
dnode_phys_t *dnode;
nvlist_t *poolconfig, *vdevconfig;
void *configbuf;
uint64_t dnid;
off_t configloc, configblksz;
int error;
dnode = objset_dnode_bonus_alloc(zfs->mos, DMU_OT_PACKED_NVLIST,
DMU_OT_PACKED_NVLIST_SIZE, sizeof(uint64_t), &dnid);
poolconfig = pool_config_nvcreate(zfs);
vdevconfig = pool_root_vdev_config_nvcreate(zfs);
nvlist_add_nvlist(poolconfig, ZPOOL_CONFIG_VDEV_TREE, vdevconfig);
nvlist_destroy(vdevconfig);
error = nvlist_export(poolconfig);
if (error != 0)
errc(1, error, "nvlist_export");
configblksz = nvlist_size(poolconfig);
configloc = objset_space_alloc(zfs, zfs->mos, &configblksz);
configbuf = ecalloc(1, configblksz);
nvlist_copy(poolconfig, configbuf, configblksz);
vdev_pwrite_dnode_data(zfs, dnode, configbuf, configblksz, configloc);
dnode->dn_datablkszsec = configblksz >> MINBLOCKSHIFT;
dnode->dn_flags = DNODE_FLAG_USED_BYTES;
*(uint64_t *)DN_BONUS(dnode) = nvlist_size(poolconfig);
zap_add_uint64(objdir, DMU_POOL_CONFIG, dnid);
nvlist_destroy(poolconfig);
free(configbuf);
}
/*
* Add objects block pointer list objects, used for deferred frees. We don't do
* anything with them, but they need to be present or OpenZFS will refuse to
* import the pool.
*/
static void
pool_init_objdir_bplists(zfs_opt_t *zfs __unused, zfs_zap_t *objdir)
{
uint64_t dnid;
(void)objset_dnode_bonus_alloc(zfs->mos, DMU_OT_BPOBJ, DMU_OT_BPOBJ_HDR,
BPOBJ_SIZE_V2, &dnid);
zap_add_uint64(objdir, DMU_POOL_FREE_BPOBJ, dnid);
(void)objset_dnode_bonus_alloc(zfs->mos, DMU_OT_BPOBJ, DMU_OT_BPOBJ_HDR,
BPOBJ_SIZE_V2, &dnid);
zap_add_uint64(objdir, DMU_POOL_SYNC_BPLIST, dnid);
}
/*
* Add required feature metadata objects. We don't know anything about ZFS
* features, so the objects are just empty ZAPs.
*/
static void
pool_init_objdir_feature_maps(zfs_opt_t *zfs, zfs_zap_t *objdir)
{
dnode_phys_t *dnode;
uint64_t dnid;
dnode = objset_dnode_alloc(zfs->mos, DMU_OTN_ZAP_METADATA, &dnid);
zap_add_uint64(objdir, DMU_POOL_FEATURES_FOR_READ, dnid);
zap_write(zfs, zap_alloc(zfs->mos, dnode));
dnode = objset_dnode_alloc(zfs->mos, DMU_OTN_ZAP_METADATA, &dnid);
zap_add_uint64(objdir, DMU_POOL_FEATURES_FOR_WRITE, dnid);
zap_write(zfs, zap_alloc(zfs->mos, dnode));
dnode = objset_dnode_alloc(zfs->mos, DMU_OTN_ZAP_METADATA, &dnid);
zap_add_uint64(objdir, DMU_POOL_FEATURE_DESCRIPTIONS, dnid);
zap_write(zfs, zap_alloc(zfs->mos, dnode));
}
static void
pool_init_objdir_dsl(zfs_opt_t *zfs, zfs_zap_t *objdir)
{
zap_add_uint64(objdir, DMU_POOL_ROOT_DATASET,
dsl_dir_id(zfs->rootdsldir));
}
static void
pool_init_objdir_poolprops(zfs_opt_t *zfs, zfs_zap_t *objdir)
{
dnode_phys_t *dnode;
uint64_t id;
dnode = objset_dnode_alloc(zfs->mos, DMU_OT_POOL_PROPS, &id);
zap_add_uint64(objdir, DMU_POOL_PROPS, id);
zfs->poolprops = zap_alloc(zfs->mos, dnode);
}
/*
* Initialize the MOS object directory, the root of virtually all of the pool's
* data and metadata.
*/
static void
pool_init_objdir(zfs_opt_t *zfs)
{
zfs_zap_t *zap;
dnode_phys_t *objdir;
objdir = objset_dnode_lookup(zfs->mos, DMU_POOL_DIRECTORY_OBJECT);
zap = zap_alloc(zfs->mos, objdir);
pool_init_objdir_config(zfs, zap);
pool_init_objdir_bplists(zfs, zap);
pool_init_objdir_feature_maps(zfs, zap);
pool_init_objdir_dsl(zfs, zap);
pool_init_objdir_poolprops(zfs, zap);
zap_write(zfs, zap);
}
/*
* Initialize the meta-object set (MOS) and immediately write out several
* special objects whose contents are already finalized, including the object
* directory.
*
* Once the MOS is finalized, it'll look roughly like this:
*
* object directory (ZAP)
* |-> vdev config object (nvlist)
* |-> features for read
* |-> features for write
* |-> feature descriptions
* |-> sync bplist
* |-> free bplist
* |-> pool properties
* L-> root DSL directory
* |-> DSL child directory (ZAP)
* | |-> $MOS (DSL dir)
* | | |-> child map
* | | L-> props (ZAP)
* | |-> $FREE (DSL dir)
* | | |-> child map
* | | L-> props (ZAP)
* | |-> $ORIGIN (DSL dir)
* | | |-> child map
* | | |-> dataset
* | | | L-> deadlist
* | | |-> snapshot
* | | | |-> deadlist
* | | | L-> snapshot names
* | | |-> props (ZAP)
* | | L-> clones (ZAP)
* | |-> dataset 1 (DSL dir)
* | | |-> DSL dataset
* | | | |-> snapshot names
* | | | L-> deadlist
* | | |-> child map
* | | | L-> ...
* | | L-> props
* | |-> dataset 2
* | | L-> ...
* | |-> ...
* | L-> dataset n
* |-> DSL root dataset
* | |-> snapshot names
* | L-> deadlist
* L-> props (ZAP)
* space map object array
* |-> space map 1
* |-> space map 2
* |-> ...
* L-> space map n (zfs->mscount)
*
* The space map object array is pointed to by the "msarray" property in the
* pool configuration.
*/
static void
pool_init(zfs_opt_t *zfs)
{
uint64_t dnid;
zfs->poolguid = ((uint64_t)random() << 32) | random();
zfs->vdevguid = ((uint64_t)random() << 32) | random();
zfs->mos = objset_alloc(zfs, DMU_OST_META);
(void)objset_dnode_alloc(zfs->mos, DMU_OT_OBJECT_DIRECTORY, &dnid);
assert(dnid == DMU_POOL_DIRECTORY_OBJECT);
(void)objset_dnode_alloc(zfs->mos, DMU_OT_OBJECT_ARRAY, &zfs->objarrid);
dsl_init(zfs);
pool_init_objdir(zfs);
}
static void
pool_labels_write(zfs_opt_t *zfs)
{
uberblock_t *ub;
vdev_label_t *label;
nvlist_t *poolconfig, *vdevconfig;
int error;
label = ecalloc(1, sizeof(*label));
/*
* Assemble the vdev configuration and store it in the label.
*/
poolconfig = pool_config_nvcreate(zfs);
vdevconfig = pool_disk_vdev_config_nvcreate(zfs);
nvlist_add_nvlist(poolconfig, ZPOOL_CONFIG_VDEV_TREE, vdevconfig);
nvlist_destroy(vdevconfig);
error = nvlist_export(poolconfig);
if (error != 0)
errc(1, error, "nvlist_export");
nvlist_copy(poolconfig, label->vl_vdev_phys.vp_nvlist,
sizeof(label->vl_vdev_phys.vp_nvlist));
nvlist_destroy(poolconfig);
/*
* Fill out the uberblock. Just make each one the same. The embedded
* checksum is calculated in vdev_label_write().
*/
for (size_t uoff = 0; uoff < sizeof(label->vl_uberblock);
uoff += (1 << zfs->ashift)) {
ub = (uberblock_t *)(&label->vl_uberblock[0] + uoff);
ub->ub_magic = UBERBLOCK_MAGIC;
ub->ub_version = SPA_VERSION;
ub->ub_txg = TXG;
ub->ub_guid_sum = zfs->poolguid + zfs->vdevguid;
ub->ub_timestamp = 0;
ub->ub_software_version = SPA_VERSION;
ub->ub_mmp_magic = MMP_MAGIC;
ub->ub_mmp_delay = 0;
ub->ub_mmp_config = 0;
ub->ub_checkpoint_txg = 0;
objset_root_blkptr_copy(zfs->mos, &ub->ub_rootbp);
}
/*
* Write out four copies of the label: two at the beginning of the vdev
* and two at the end.
*/
for (int i = 0; i < VDEV_LABELS; i++)
vdev_label_write(zfs, i, label);
free(label);
}
static void
pool_fini(zfs_opt_t *zfs)
{
zap_write(zfs, zfs->poolprops);
dsl_write(zfs);
objset_write(zfs, zfs->mos);
pool_labels_write(zfs);
}
struct dnode_cursor *
dnode_cursor_init(zfs_opt_t *zfs, zfs_objset_t *os, dnode_phys_t *dnode,
off_t size, off_t blksz)
{
struct dnode_cursor *c;
uint64_t nbppindir, indlevel, ndatablks, nindblks;
assert(dnode->dn_nblkptr == 1);
assert(blksz <= MAXBLOCKSIZE);
if (blksz == 0) {
/* Must be between 1<<ashift and 128KB. */
blksz = MIN(MAXBLOCKSIZE, MAX(1 << zfs->ashift,
powerof2(size) ? size : (1ul << flsll(size))));
}
assert(powerof2(blksz));
/*
* Do we need indirect blocks? Figure out how many levels are needed
* (indlevel == 1 means no indirect blocks) and how much space is needed
* (it has to be allocated up-front to break the dependency cycle
* described in objset_write()).
*/
ndatablks = size == 0 ? 0 : howmany(size, blksz);
nindblks = 0;
for (indlevel = 1, nbppindir = 1; ndatablks > nbppindir; indlevel++) {
nbppindir *= BLKPTR_PER_INDIR;
nindblks += howmany(ndatablks, indlevel * nbppindir);
}
assert(indlevel < INDIR_LEVELS);
dnode->dn_nlevels = (uint8_t)indlevel;
dnode->dn_maxblkid = ndatablks > 0 ? ndatablks - 1 : 0;
dnode->dn_datablkszsec = blksz >> MINBLOCKSHIFT;
c = ecalloc(1, sizeof(*c));
if (nindblks > 0) {
c->indspace = nindblks * MAXBLOCKSIZE;
c->indloc = objset_space_alloc(zfs, os, &c->indspace);
}
c->dnode = dnode;
c->dataoff = 0;
c->datablksz = blksz;
return (c);
}
static void
_dnode_cursor_flush(zfs_opt_t *zfs, struct dnode_cursor *c, int levels)
{
blkptr_t *bp, *pbp;
void *buf;
uint64_t fill;
off_t blkid, blksz, loc;
assert(levels > 0);
assert(levels <= c->dnode->dn_nlevels - 1);
blksz = MAXBLOCKSIZE;
blkid = (c->dataoff / c->datablksz) / BLKPTR_PER_INDIR;
for (int level = 1; level <= levels; level++) {
buf = c->inddir[level - 1];
if (level == c->dnode->dn_nlevels - 1) {
pbp = &c->dnode->dn_blkptr[0];
} else {
uint64_t iblkid;
iblkid = blkid & (BLKPTR_PER_INDIR - 1);
pbp = (blkptr_t *)
&c->inddir[level][iblkid * sizeof(blkptr_t)];
}
/*
* Space for indirect blocks is allocated up-front; see the
* comment in objset_write().
*/
loc = c->indloc;
c->indloc += blksz;
assert(c->indspace >= blksz);
c->indspace -= blksz;
bp = buf;
fill = 0;
for (size_t i = 0; i < BLKPTR_PER_INDIR; i++)
fill += BP_GET_FILL(&bp[i]);
vdev_pwrite_dnode_indir(zfs, c->dnode, level, fill, buf, blksz,
loc, pbp);
memset(buf, 0, MAXBLOCKSIZE);
blkid /= BLKPTR_PER_INDIR;
}
}
blkptr_t *
dnode_cursor_next(zfs_opt_t *zfs, struct dnode_cursor *c, off_t off)
{
off_t blkid, l1id;
int levels;
if (c->dnode->dn_nlevels == 1) {
assert(off < MAXBLOCKSIZE);
return (&c->dnode->dn_blkptr[0]);
}
assert(off % c->datablksz == 0);
/* Do we need to flush any full indirect blocks? */
if (off > 0) {
blkid = off / c->datablksz;
for (levels = 0; levels < c->dnode->dn_nlevels - 1; levels++) {
if (blkid % BLKPTR_PER_INDIR != 0)
break;
blkid /= BLKPTR_PER_INDIR;
}
if (levels > 0)
_dnode_cursor_flush(zfs, c, levels);
}
c->dataoff = off;
l1id = (off / c->datablksz) & (BLKPTR_PER_INDIR - 1);
return ((blkptr_t *)&c->inddir[0][l1id * sizeof(blkptr_t)]);
}
void
dnode_cursor_finish(zfs_opt_t *zfs, struct dnode_cursor *c)
{
int levels;
levels = c->dnode->dn_nlevels - 1;
if (levels > 0)
_dnode_cursor_flush(zfs, c, levels);
assert(c->indspace == 0);
free(c);
}
void
zfs_makefs(const char *image, const char *dir, fsnode *root, fsinfo_t *fsopts)
{
zfs_opt_t *zfs;
int dirfd;
zfs = fsopts->fs_specific;
/*
* Use a fixed seed to provide reproducible pseudo-random numbers for
* on-disk structures when needed (e.g., GUIDs, ZAP hash salts).
*/
srandom(1729);
zfs_check_opts(fsopts);
if (!zfs->nowarn) {
fprintf(stderr,
"ZFS support is currently considered experimental. "
"Do not use it for anything critical.\n");
}
allanjudeUnsubmitted
Not Done
Inline Actions

when ZFS grows the pool, it will add more metaslabs of this same size.

So we want to keep the number of slabs small, so that growing the pool to 100x the size doesn't result in 16,000 metaslabs.

The only downside is, any slack that doesn't fit into a metaslab will be unused, so if we just set a metaslab size of 4G, then we'd only have 8G usable from a 10G image.

Although without knowing how large they are going to grow it, it can be difficult to guess what number is right.

allanjude: when ZFS grows the pool, it will add more metaslabs of this same size. So we want to keep the…
markjAuthorUnsubmitted
Done
Inline Actions

Ahh, I thought that ZFS would potentially resize the metaslabs themselves. Apparently not. I'm not sure how best to handle this, given that OpenZFS itself doesn't seem to bother (at least based on the end of the comment in vdev_metaslab_set_size()).

It's not really possible to know which metaslab size is the "right" one, though. The tradeoffs involved also depend on whether certain optional features are enabled, like the log spacemap and spacemap histograms, I think. Right now, makefs' choice is too closely tied to the initial image size and will give excessively small metaslabs for most purposes. If our chosen metaslab size is large, we can also adjust the output image size to compensate based on the -m and -M parameters, so unused space isn't a major problem.

I'm inclined to keep it simple and do the following:

  • let the metaslab size be 10% of the image size, up to 512MB
  • add a -o msshift option to let the user override the heuristic

That'll give basically the same size as OpenZFS for images expanded to up to 100GB. Beyond that, the answer might just be to create a separate pool if you really need a larger metaslab size, or somehow teach OpenZFS to resize metaslabs if it can determine that it'd be profitable to do so.

markj: Ahh, I thought that ZFS would potentially resize the metaslabs themselves. Apparently not. I'm…
dirfd = open(dir, O_DIRECTORY | O_RDONLY);
if (dirfd < 0)
err(1, "open(%s)", dir);
vdev_init(zfs, image);
pool_init(zfs);
fs_build(zfs, dirfd, root);
pool_fini(zfs);
vdev_fini(zfs);
}
emasteUnsubmitted
Not Done
Inline Actions

(sigh)

emaste: (sigh)
View Options

multimedia/ffmpeg/distinfo

  • This file was added.
/*-
* SPDX-License-Identifier: BSD-2-Clause-FreeBSD
*
* Copyright (c) 2022 The FreeBSD Foundation
*
* This software was developed by Mark Johnston under sponsorship from
* the FreeBSD Foundation.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are
* met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in
* the documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*/
#include <sys/param.h>
#include <sys/errno.h>
#include <sys/queue.h>
#include <assert.h>
#include <fcntl.h>
#include <stdbool.h>
#include <stddef.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <util.h>
#include "makefs.h"
#include "zfs.h"
#define VDEV_LABEL_SPACE \
((off_t)(VDEV_LABEL_START_SIZE + VDEV_LABEL_END_SIZE))
_Static_assert(VDEV_LABEL_SPACE <= MINDEVSIZE, "");
#define MINMSSIZE ((off_t)1 << 24) /* 16MB */
#define DFLTMSSIZE ((off_t)1 << 29) /* 512MB */
#define MAXMSSIZE ((off_t)1 << 34) /* 16GB */
#define INDIR_LEVELS 6
/* Indirect blocks are always 128KB. */
#define BLKPTR_PER_INDIR (MAXBLOCKSIZE / sizeof(blkptr_t))
struct dnode_cursor {
char inddir[INDIR_LEVELS][MAXBLOCKSIZE];
off_t indloc;
off_t indspace;
dnode_phys_t *dnode;
off_t dataoff;
off_t datablksz;
};
void
zfs_prep_opts(fsinfo_t *fsopts)
{
zfs_opt_t *zfs = ecalloc(1, sizeof(*zfs));
const option_t zfs_options[] = {
{ '\0', "bootfs", &zfs->bootfs, OPT_STRPTR,
0, 0, "Bootable dataset" },
{ '\0', "mssize", &zfs->mssize, OPT_INT64,
MINMSSIZE, MAXMSSIZE, "Metaslab size" },
{ '\0', "poolname", &zfs->poolname, OPT_STRPTR,
0, 0, "ZFS pool name" },
{ '\0', "rootpath", &zfs->rootpath, OPT_STRPTR,
0, 0, "Prefix for all dataset mount points" },
{ '\0', "ashift", &zfs->ashift, OPT_INT32,
MINBLOCKSHIFT, MAXBLOCKSHIFT, "ZFS pool ashift" },
{ '\0', "nowarn", &zfs->nowarn, OPT_BOOL,
0, 0, "Suppress warning about experimental ZFS support" },
{ .name = NULL }
};
STAILQ_INIT(&zfs->datasetdescs);
fsopts->fs_specific = zfs;
fsopts->fs_options = copy_opts(zfs_options);
}
int
zfs_parse_opts(const char *option, fsinfo_t *fsopts)
{
zfs_opt_t *zfs;
struct dataset_desc *dsdesc;
char buf[BUFSIZ], *opt, *val;
int rv;
zfs = fsopts->fs_specific;
opt = val = estrdup(option);
opt = strsep(&val, "=");
if (strcmp(opt, "fs") == 0) {
if (val == NULL)
errx(1, "invalid filesystem parameters `%s'", option);
/*
* Dataset descriptions will be parsed later, in dsl_init().
* Just stash them away for now.
*/
dsdesc = ecalloc(1, sizeof(*dsdesc));
dsdesc->params = estrdup(val);
free(opt);
STAILQ_INSERT_TAIL(&zfs->datasetdescs, dsdesc, next);
return (1);
}
free(opt);
rv = set_option(fsopts->fs_options, option, buf, sizeof(buf));
return (rv == -1 ? 0 : 1);
}
static void
zfs_size_vdev(fsinfo_t *fsopts)
{
zfs_opt_t *zfs;
off_t asize, mssize, vdevsize, vdevsize1;
zfs = fsopts->fs_specific;
assert(fsopts->maxsize != 0);
assert(zfs->ashift != 0);
/*
* Figure out how big the vdev should be.
*/
vdevsize = rounddown2(fsopts->maxsize, 1 << zfs->ashift);
if (vdevsize < MINDEVSIZE)
errx(1, "maximum image size is too small");
if (vdevsize < fsopts->minsize || vdevsize > fsopts->maxsize) {
errx(1, "image size bounds must be multiples of %d",
1 << zfs->ashift);
}
asize = vdevsize - VDEV_LABEL_SPACE;
/*
* Size metaslabs according to the following heuristic:
* - provide at least 8 metaslabs,
* - without using a metaslab size larger than 512MB.
* This approximates what OpenZFS does without being complicated. In
* practice we expect pools to be expanded upon first use, and OpenZFS
* does not resize metaslabs in that case, so there is no right answer
* here. In general we want to provide large metaslabs even if the
* image size is small, and 512MB is a reasonable size for pools up to
* several hundred gigabytes.
*
* The user may override this heuristic using the "-o mssize" option.
*/
mssize = zfs->mssize;
if (mssize == 0) {
mssize = MAX(MIN(asize / 8, DFLTMSSIZE), MINMSSIZE);
if (!powerof2(mssize))
mssize = 1l << (flsll(mssize) - 1);
}
if (!powerof2(mssize))
errx(1, "metaslab size must be a power of 2");
/*
* If we have some slop left over, try to cover it by resizing the vdev,
* subject to the maxsize and minsize parameters.
*/
if (asize % mssize != 0) {
vdevsize1 = rounddown2(asize, mssize) + VDEV_LABEL_SPACE;
if (vdevsize1 < fsopts->minsize)
vdevsize1 = roundup2(asize, mssize) + VDEV_LABEL_SPACE;
if (vdevsize1 <= fsopts->maxsize)
vdevsize = vdevsize1;
}
asize = vdevsize - VDEV_LABEL_SPACE;
zfs->asize = asize;
zfs->vdevsize = vdevsize;
zfs->mssize = mssize;
zfs->msshift = flsll(mssize) - 1;
zfs->mscount = asize / mssize;
}
/*
* Validate options and set some default values.
*/
static void
zfs_check_opts(fsinfo_t *fsopts)
{
zfs_opt_t *zfs;
zfs = fsopts->fs_specific;
if (fsopts->offset != 0)
errx(1, "unhandled offset option");
if (fsopts->maxsize == 0)
errx(1, "an image size must be specified");
if (zfs->poolname == NULL)
errx(1, "a pool name must be specified");
if (zfs->rootpath == NULL)
easprintf(&zfs->rootpath, "/%s", zfs->poolname);
if (zfs->rootpath[0] != '/')
errx(1, "mountpoint `%s' must be absolute", zfs->rootpath);
if (zfs->ashift == 0)
zfs->ashift = 12;
zfs_size_vdev(fsopts);
}
void
zfs_cleanup_opts(fsinfo_t *fsopts)
{
struct dataset_desc *d, *tmp;
zfs_opt_t *zfs;
zfs = fsopts->fs_specific;
free(zfs->rootpath);
free(zfs->bootfs);
free(__DECONST(void *, zfs->poolname));
STAILQ_FOREACH_SAFE(d, &zfs->datasetdescs, next, tmp) {
free(d->params);
free(d);
}
free(zfs);
free(fsopts->fs_options);
}
static size_t
nvlist_size(const nvlist_t *nvl)
{
return (sizeof(nvl->nv_header) + nvl->nv_size);
}
static void
nvlist_copy(const nvlist_t *nvl, char *buf, size_t sz)
{
assert(sz >= nvlist_size(nvl));
memcpy(buf, &nvl->nv_header, sizeof(nvl->nv_header));
memcpy(buf + sizeof(nvl->nv_header), nvl->nv_data, nvl->nv_size);
}
static nvlist_t *
pool_config_nvcreate(zfs_opt_t *zfs)
{
nvlist_t *featuresnv, *poolnv;
poolnv = nvlist_create(NV_UNIQUE_NAME);
nvlist_add_uint64(poolnv, ZPOOL_CONFIG_POOL_TXG, TXG);
nvlist_add_uint64(poolnv, ZPOOL_CONFIG_VERSION, SPA_VERSION);
nvlist_add_uint64(poolnv, ZPOOL_CONFIG_POOL_STATE, POOL_STATE_EXPORTED);
nvlist_add_string(poolnv, ZPOOL_CONFIG_POOL_NAME, zfs->poolname);
nvlist_add_uint64(poolnv, ZPOOL_CONFIG_POOL_GUID, zfs->poolguid);
nvlist_add_uint64(poolnv, ZPOOL_CONFIG_TOP_GUID, zfs->vdevguid);
nvlist_add_uint64(poolnv, ZPOOL_CONFIG_GUID, zfs->vdevguid);
nvlist_add_uint64(poolnv, ZPOOL_CONFIG_VDEV_CHILDREN, 1);
featuresnv = nvlist_create(NV_UNIQUE_NAME);
nvlist_add_nvlist(poolnv, ZPOOL_CONFIG_FEATURES_FOR_READ, featuresnv);
nvlist_destroy(featuresnv);
return (poolnv);
}
static nvlist_t *
pool_disk_vdev_config_nvcreate(zfs_opt_t *zfs)
{
nvlist_t *diskvdevnv;
assert(zfs->objarrid != 0);
diskvdevnv = nvlist_create(NV_UNIQUE_NAME);
nvlist_add_string(diskvdevnv, ZPOOL_CONFIG_TYPE, VDEV_TYPE_DISK);
nvlist_add_uint64(diskvdevnv, ZPOOL_CONFIG_ASHIFT, zfs->ashift);
nvlist_add_uint64(diskvdevnv, ZPOOL_CONFIG_ASIZE, zfs->asize);
nvlist_add_uint64(diskvdevnv, ZPOOL_CONFIG_GUID, zfs->vdevguid);
nvlist_add_uint64(diskvdevnv, ZPOOL_CONFIG_ID, 0);
nvlist_add_string(diskvdevnv, ZPOOL_CONFIG_PATH, "/dev/null");
nvlist_add_uint64(diskvdevnv, ZPOOL_CONFIG_WHOLE_DISK, 1);
nvlist_add_uint64(diskvdevnv, ZPOOL_CONFIG_CREATE_TXG, TXG);
nvlist_add_uint64(diskvdevnv, ZPOOL_CONFIG_METASLAB_ARRAY,
zfs->objarrid);
nvlist_add_uint64(diskvdevnv, ZPOOL_CONFIG_METASLAB_SHIFT,
zfs->msshift);
return (diskvdevnv);
}
static nvlist_t *
pool_root_vdev_config_nvcreate(zfs_opt_t *zfs)
{
nvlist_t *diskvdevnv, *rootvdevnv;
diskvdevnv = pool_disk_vdev_config_nvcreate(zfs);
rootvdevnv = nvlist_create(NV_UNIQUE_NAME);
nvlist_add_uint64(rootvdevnv, ZPOOL_CONFIG_ID, 0);
nvlist_add_uint64(rootvdevnv, ZPOOL_CONFIG_GUID, zfs->poolguid);
nvlist_add_string(rootvdevnv, ZPOOL_CONFIG_TYPE, VDEV_TYPE_ROOT);
nvlist_add_uint64(rootvdevnv, ZPOOL_CONFIG_CREATE_TXG, TXG);
nvlist_add_nvlist_array(rootvdevnv, ZPOOL_CONFIG_CHILDREN, &diskvdevnv,
1);
nvlist_destroy(diskvdevnv);
return (rootvdevnv);
}
/*
* Create the pool's "config" object, which contains an nvlist describing pool
* parameters and the vdev topology. It is similar but not identical to the
* nvlist stored in vdev labels. The main difference is that vdev labels do not
* describe the full vdev tree and in particular do not contain the "root"
* meta-vdev.
*/
static void
pool_init_objdir_config(zfs_opt_t *zfs, zfs_zap_t *objdir)
{
dnode_phys_t *dnode;
nvlist_t *poolconfig, *vdevconfig;
void *configbuf;
uint64_t dnid;
off_t configloc, configblksz;
int error;
dnode = objset_dnode_bonus_alloc(zfs->mos, DMU_OT_PACKED_NVLIST,
DMU_OT_PACKED_NVLIST_SIZE, sizeof(uint64_t), &dnid);
poolconfig = pool_config_nvcreate(zfs);
vdevconfig = pool_root_vdev_config_nvcreate(zfs);
nvlist_add_nvlist(poolconfig, ZPOOL_CONFIG_VDEV_TREE, vdevconfig);
nvlist_destroy(vdevconfig);
error = nvlist_export(poolconfig);
if (error != 0)
errc(1, error, "nvlist_export");
configblksz = nvlist_size(poolconfig);
configloc = objset_space_alloc(zfs, zfs->mos, &configblksz);
configbuf = ecalloc(1, configblksz);
nvlist_copy(poolconfig, configbuf, configblksz);
vdev_pwrite_dnode_data(zfs, dnode, configbuf, configblksz, configloc);
dnode->dn_datablkszsec = configblksz >> MINBLOCKSHIFT;
dnode->dn_flags = DNODE_FLAG_USED_BYTES;
*(uint64_t *)DN_BONUS(dnode) = nvlist_size(poolconfig);
zap_add_uint64(objdir, DMU_POOL_CONFIG, dnid);
nvlist_destroy(poolconfig);
free(configbuf);
}
/*
* Add objects block pointer list objects, used for deferred frees. We don't do
* anything with them, but they need to be present or OpenZFS will refuse to
* import the pool.
*/
static void
pool_init_objdir_bplists(zfs_opt_t *zfs __unused, zfs_zap_t *objdir)
{
uint64_t dnid;
(void)objset_dnode_bonus_alloc(zfs->mos, DMU_OT_BPOBJ, DMU_OT_BPOBJ_HDR,
BPOBJ_SIZE_V2, &dnid);
zap_add_uint64(objdir, DMU_POOL_FREE_BPOBJ, dnid);
(void)objset_dnode_bonus_alloc(zfs->mos, DMU_OT_BPOBJ, DMU_OT_BPOBJ_HDR,
BPOBJ_SIZE_V2, &dnid);
zap_add_uint64(objdir, DMU_POOL_SYNC_BPLIST, dnid);
}
/*
* Add required feature metadata objects. We don't know anything about ZFS
* features, so the objects are just empty ZAPs.
*/
static void
pool_init_objdir_feature_maps(zfs_opt_t *zfs, zfs_zap_t *objdir)
{
dnode_phys_t *dnode;
uint64_t dnid;
dnode = objset_dnode_alloc(zfs->mos, DMU_OTN_ZAP_METADATA, &dnid);
zap_add_uint64(objdir, DMU_POOL_FEATURES_FOR_READ, dnid);
zap_write(zfs, zap_alloc(zfs->mos, dnode));
dnode = objset_dnode_alloc(zfs->mos, DMU_OTN_ZAP_METADATA, &dnid);
zap_add_uint64(objdir, DMU_POOL_FEATURES_FOR_WRITE, dnid);
zap_write(zfs, zap_alloc(zfs->mos, dnode));
dnode = objset_dnode_alloc(zfs->mos, DMU_OTN_ZAP_METADATA, &dnid);
zap_add_uint64(objdir, DMU_POOL_FEATURE_DESCRIPTIONS, dnid);
zap_write(zfs, zap_alloc(zfs->mos, dnode));
}
static void
pool_init_objdir_dsl(zfs_opt_t *zfs, zfs_zap_t *objdir)
{
zap_add_uint64(objdir, DMU_POOL_ROOT_DATASET,
dsl_dir_id(zfs->rootdsldir));
}
static void
pool_init_objdir_poolprops(zfs_opt_t *zfs, zfs_zap_t *objdir)
{
dnode_phys_t *dnode;
uint64_t id;
dnode = objset_dnode_alloc(zfs->mos, DMU_OT_POOL_PROPS, &id);
zap_add_uint64(objdir, DMU_POOL_PROPS, id);
zfs->poolprops = zap_alloc(zfs->mos, dnode);
}
/*
* Initialize the MOS object directory, the root of virtually all of the pool's
* data and metadata.
*/
static void
pool_init_objdir(zfs_opt_t *zfs)
{
zfs_zap_t *zap;
dnode_phys_t *objdir;
objdir = objset_dnode_lookup(zfs->mos, DMU_POOL_DIRECTORY_OBJECT);
zap = zap_alloc(zfs->mos, objdir);
pool_init_objdir_config(zfs, zap);
pool_init_objdir_bplists(zfs, zap);
pool_init_objdir_feature_maps(zfs, zap);
pool_init_objdir_dsl(zfs, zap);
pool_init_objdir_poolprops(zfs, zap);
zap_write(zfs, zap);
}
/*
* Initialize the meta-object set (MOS) and immediately write out several
* special objects whose contents are already finalized, including the object
* directory.
*
* Once the MOS is finalized, it'll look roughly like this:
*
* object directory (ZAP)
* |-> vdev config object (nvlist)
* |-> features for read
* |-> features for write
* |-> feature descriptions
* |-> sync bplist
* |-> free bplist
* |-> pool properties
* L-> root DSL directory
* |-> DSL child directory (ZAP)
* | |-> $MOS (DSL dir)
* | | |-> child map
* | | L-> props (ZAP)
* | |-> $FREE (DSL dir)
* | | |-> child map
* | | L-> props (ZAP)
* | |-> $ORIGIN (DSL dir)
* | | |-> child map
* | | |-> dataset
* | | | L-> deadlist
* | | |-> snapshot
* | | | |-> deadlist
* | | | L-> snapshot names
* | | |-> props (ZAP)
* | | L-> clones (ZAP)
* | |-> dataset 1 (DSL dir)
* | | |-> DSL dataset
* | | | |-> snapshot names
* | | | L-> deadlist
* | | |-> child map
* | | | L-> ...
* | | L-> props
* | |-> dataset 2
* | | L-> ...
* | |-> ...
* | L-> dataset n
* |-> DSL root dataset
* | |-> snapshot names
* | L-> deadlist
* L-> props (ZAP)
* space map object array
* |-> space map 1
* |-> space map 2
* |-> ...
* L-> space map n (zfs->mscount)
*
* The space map object array is pointed to by the "msarray" property in the
* pool configuration.
*/
static void
pool_init(zfs_opt_t *zfs)
{
uint64_t dnid;
zfs->poolguid = ((uint64_t)random() << 32) | random();
zfs->vdevguid = ((uint64_t)random() << 32) | random();
zfs->mos = objset_alloc(zfs, DMU_OST_META);
(void)objset_dnode_alloc(zfs->mos, DMU_OT_OBJECT_DIRECTORY, &dnid);
assert(dnid == DMU_POOL_DIRECTORY_OBJECT);
(void)objset_dnode_alloc(zfs->mos, DMU_OT_OBJECT_ARRAY, &zfs->objarrid);
dsl_init(zfs);
pool_init_objdir(zfs);
}
static void
pool_labels_write(zfs_opt_t *zfs)
{
uberblock_t *ub;
vdev_label_t *label;
nvlist_t *poolconfig, *vdevconfig;
int error;
label = ecalloc(1, sizeof(*label));
/*
* Assemble the vdev configuration and store it in the label.
*/
poolconfig = pool_config_nvcreate(zfs);
vdevconfig = pool_disk_vdev_config_nvcreate(zfs);
nvlist_add_nvlist(poolconfig, ZPOOL_CONFIG_VDEV_TREE, vdevconfig);
nvlist_destroy(vdevconfig);
error = nvlist_export(poolconfig);
if (error != 0)
errc(1, error, "nvlist_export");
nvlist_copy(poolconfig, label->vl_vdev_phys.vp_nvlist,
sizeof(label->vl_vdev_phys.vp_nvlist));
nvlist_destroy(poolconfig);
/*
* Fill out the uberblock. Just make each one the same. The embedded
* checksum is calculated in vdev_label_write().
*/
for (size_t uoff = 0; uoff < sizeof(label->vl_uberblock);
uoff += (1 << zfs->ashift)) {
ub = (uberblock_t *)(&label->vl_uberblock[0] + uoff);
ub->ub_magic = UBERBLOCK_MAGIC;
ub->ub_version = SPA_VERSION;
ub->ub_txg = TXG;
ub->ub_guid_sum = zfs->poolguid + zfs->vdevguid;
ub->ub_timestamp = 0;
ub->ub_software_version = SPA_VERSION;
ub->ub_mmp_magic = MMP_MAGIC;
ub->ub_mmp_delay = 0;
ub->ub_mmp_config = 0;
ub->ub_checkpoint_txg = 0;
objset_root_blkptr_copy(zfs->mos, &ub->ub_rootbp);
}
/*
* Write out four copies of the label: two at the beginning of the vdev
* and two at the end.
*/
for (int i = 0; i < VDEV_LABELS; i++)
vdev_label_write(zfs, i, label);
free(label);
}
static void
pool_fini(zfs_opt_t *zfs)
{
zap_write(zfs, zfs->poolprops);
dsl_write(zfs);
objset_write(zfs, zfs->mos);
pool_labels_write(zfs);
}
struct dnode_cursor *
dnode_cursor_init(zfs_opt_t *zfs, zfs_objset_t *os, dnode_phys_t *dnode,
off_t size, off_t blksz)
{
struct dnode_cursor *c;
uint64_t nbppindir, indlevel, ndatablks, nindblks;
assert(dnode->dn_nblkptr == 1);
assert(blksz <= MAXBLOCKSIZE);
if (blksz == 0) {
/* Must be between 1<<ashift and 128KB. */
blksz = MIN(MAXBLOCKSIZE, MAX(1 << zfs->ashift,
powerof2(size) ? size : (1ul << flsll(size))));
}
assert(powerof2(blksz));
/*
* Do we need indirect blocks? Figure out how many levels are needed
* (indlevel == 1 means no indirect blocks) and how much space is needed
* (it has to be allocated up-front to break the dependency cycle
* described in objset_write()).
*/
ndatablks = size == 0 ? 0 : howmany(size, blksz);
nindblks = 0;
for (indlevel = 1, nbppindir = 1; ndatablks > nbppindir; indlevel++) {
nbppindir *= BLKPTR_PER_INDIR;
nindblks += howmany(ndatablks, indlevel * nbppindir);
}
assert(indlevel < INDIR_LEVELS);
dnode->dn_nlevels = (uint8_t)indlevel;
dnode->dn_maxblkid = ndatablks > 0 ? ndatablks - 1 : 0;
dnode->dn_datablkszsec = blksz >> MINBLOCKSHIFT;
c = ecalloc(1, sizeof(*c));
if (nindblks > 0) {
c->indspace = nindblks * MAXBLOCKSIZE;
c->indloc = objset_space_alloc(zfs, os, &c->indspace);
}
c->dnode = dnode;
c->dataoff = 0;
c->datablksz = blksz;
return (c);
}
static void
_dnode_cursor_flush(zfs_opt_t *zfs, struct dnode_cursor *c, int levels)
{
blkptr_t *bp, *pbp;
void *buf;
uint64_t fill;
off_t blkid, blksz, loc;
assert(levels > 0);
assert(levels <= c->dnode->dn_nlevels - 1);
blksz = MAXBLOCKSIZE;
blkid = (c->dataoff / c->datablksz) / BLKPTR_PER_INDIR;
for (int level = 1; level <= levels; level++) {
buf = c->inddir[level - 1];
if (level == c->dnode->dn_nlevels - 1) {
pbp = &c->dnode->dn_blkptr[0];
} else {
uint64_t iblkid;
iblkid = blkid & (BLKPTR_PER_INDIR - 1);
pbp = (blkptr_t *)
&c->inddir[level][iblkid * sizeof(blkptr_t)];
}
/*
* Space for indirect blocks is allocated up-front; see the
* comment in objset_write().
*/
loc = c->indloc;
c->indloc += blksz;
assert(c->indspace >= blksz);
c->indspace -= blksz;
bp = buf;
fill = 0;
for (size_t i = 0; i < BLKPTR_PER_INDIR; i++)
fill += BP_GET_FILL(&bp[i]);
vdev_pwrite_dnode_indir(zfs, c->dnode, level, fill, buf, blksz,
loc, pbp);
memset(buf, 0, MAXBLOCKSIZE);
blkid /= BLKPTR_PER_INDIR;
}
}
blkptr_t *
dnode_cursor_next(zfs_opt_t *zfs, struct dnode_cursor *c, off_t off)
{
off_t blkid, l1id;
int levels;
if (c->dnode->dn_nlevels == 1) {
assert(off < MAXBLOCKSIZE);
return (&c->dnode->dn_blkptr[0]);
}
assert(off % c->datablksz == 0);
/* Do we need to flush any full indirect blocks? */
if (off > 0) {
blkid = off / c->datablksz;
for (levels = 0; levels < c->dnode->dn_nlevels - 1; levels++) {
if (blkid % BLKPTR_PER_INDIR != 0)
break;
blkid /= BLKPTR_PER_INDIR;
}
if (levels > 0)
_dnode_cursor_flush(zfs, c, levels);
}
c->dataoff = off;
l1id = (off / c->datablksz) & (BLKPTR_PER_INDIR - 1);
return ((blkptr_t *)&c->inddir[0][l1id * sizeof(blkptr_t)]);
}
void
dnode_cursor_finish(zfs_opt_t *zfs, struct dnode_cursor *c)
{
int levels;
levels = c->dnode->dn_nlevels - 1;
if (levels > 0)
_dnode_cursor_flush(zfs, c, levels);
assert(c->indspace == 0);
free(c);
}
void
zfs_makefs(const char *image, const char *dir, fsnode *root, fsinfo_t *fsopts)
{
zfs_opt_t *zfs;
int dirfd;
zfs = fsopts->fs_specific;
/*
* Use a fixed seed to provide reproducible pseudo-random numbers for
* on-disk structures when needed (e.g., GUIDs, ZAP hash salts).
*/
srandom(1729);
zfs_check_opts(fsopts);
if (!zfs->nowarn) {
fprintf(stderr,
"ZFS support is currently considered experimental. "
"Do not use it for anything critical.\n");
}
allanjudeUnsubmitted
Not Done
Inline Actions

when ZFS grows the pool, it will add more metaslabs of this same size.

So we want to keep the number of slabs small, so that growing the pool to 100x the size doesn't result in 16,000 metaslabs.

The only downside is, any slack that doesn't fit into a metaslab will be unused, so if we just set a metaslab size of 4G, then we'd only have 8G usable from a 10G image.

Although without knowing how large they are going to grow it, it can be difficult to guess what number is right.

allanjude: when ZFS grows the pool, it will add more metaslabs of this same size. So we want to keep the…
markjAuthorUnsubmitted
Done
Inline Actions

Ahh, I thought that ZFS would potentially resize the metaslabs themselves. Apparently not. I'm not sure how best to handle this, given that OpenZFS itself doesn't seem to bother (at least based on the end of the comment in vdev_metaslab_set_size()).

It's not really possible to know which metaslab size is the "right" one, though. The tradeoffs involved also depend on whether certain optional features are enabled, like the log spacemap and spacemap histograms, I think. Right now, makefs' choice is too closely tied to the initial image size and will give excessively small metaslabs for most purposes. If our chosen metaslab size is large, we can also adjust the output image size to compensate based on the -m and -M parameters, so unused space isn't a major problem.

I'm inclined to keep it simple and do the following:

  • let the metaslab size be 10% of the image size, up to 512MB
  • add a -o msshift option to let the user override the heuristic

That'll give basically the same size as OpenZFS for images expanded to up to 100GB. Beyond that, the answer might just be to create a separate pool if you really need a larger metaslab size, or somehow teach OpenZFS to resize metaslabs if it can determine that it'd be profitable to do so.

markj: Ahh, I thought that ZFS would potentially resize the metaslabs themselves. Apparently not. I'm…
dirfd = open(dir, O_DIRECTORY | O_RDONLY);
if (dirfd < 0)
err(1, "open(%s)", dir);
vdev_init(zfs, image);
pool_init(zfs);
fs_build(zfs, dirfd, root);
pool_fini(zfs);
vdev_fini(zfs);
}
emasteUnsubmitted
Not Done
Inline Actions

(sigh)

emaste: (sigh)
View Options

multimedia/ffmpeg/files/patch-configure

  • This file was added.
/*-
* SPDX-License-Identifier: BSD-2-Clause-FreeBSD
*
* Copyright (c) 2022 The FreeBSD Foundation
*
* This software was developed by Mark Johnston under sponsorship from
* the FreeBSD Foundation.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are
* met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in
* the documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*/
#include <sys/param.h>
#include <sys/errno.h>
#include <sys/queue.h>
#include <assert.h>
#include <fcntl.h>
#include <stdbool.h>
#include <stddef.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <util.h>
#include "makefs.h"
#include "zfs.h"
#define VDEV_LABEL_SPACE \
((off_t)(VDEV_LABEL_START_SIZE + VDEV_LABEL_END_SIZE))
_Static_assert(VDEV_LABEL_SPACE <= MINDEVSIZE, "");
#define MINMSSIZE ((off_t)1 << 24) /* 16MB */
#define DFLTMSSIZE ((off_t)1 << 29) /* 512MB */
#define MAXMSSIZE ((off_t)1 << 34) /* 16GB */
#define INDIR_LEVELS 6
/* Indirect blocks are always 128KB. */
#define BLKPTR_PER_INDIR (MAXBLOCKSIZE / sizeof(blkptr_t))
struct dnode_cursor {
char inddir[INDIR_LEVELS][MAXBLOCKSIZE];
off_t indloc;
off_t indspace;
dnode_phys_t *dnode;
off_t dataoff;
off_t datablksz;
};
void
zfs_prep_opts(fsinfo_t *fsopts)
{
zfs_opt_t *zfs = ecalloc(1, sizeof(*zfs));
const option_t zfs_options[] = {
{ '\0', "bootfs", &zfs->bootfs, OPT_STRPTR,
0, 0, "Bootable dataset" },
{ '\0', "mssize", &zfs->mssize, OPT_INT64,
MINMSSIZE, MAXMSSIZE, "Metaslab size" },
{ '\0', "poolname", &zfs->poolname, OPT_STRPTR,
0, 0, "ZFS pool name" },
{ '\0', "rootpath", &zfs->rootpath, OPT_STRPTR,
0, 0, "Prefix for all dataset mount points" },
{ '\0', "ashift", &zfs->ashift, OPT_INT32,
MINBLOCKSHIFT, MAXBLOCKSHIFT, "ZFS pool ashift" },
{ '\0', "nowarn", &zfs->nowarn, OPT_BOOL,
0, 0, "Suppress warning about experimental ZFS support" },
{ .name = NULL }
};
STAILQ_INIT(&zfs->datasetdescs);
fsopts->fs_specific = zfs;
fsopts->fs_options = copy_opts(zfs_options);
}
int
zfs_parse_opts(const char *option, fsinfo_t *fsopts)
{
zfs_opt_t *zfs;
struct dataset_desc *dsdesc;
char buf[BUFSIZ], *opt, *val;
int rv;
zfs = fsopts->fs_specific;
opt = val = estrdup(option);
opt = strsep(&val, "=");
if (strcmp(opt, "fs") == 0) {
if (val == NULL)
errx(1, "invalid filesystem parameters `%s'", option);
/*
* Dataset descriptions will be parsed later, in dsl_init().
* Just stash them away for now.
*/
dsdesc = ecalloc(1, sizeof(*dsdesc));
dsdesc->params = estrdup(val);
free(opt);
STAILQ_INSERT_TAIL(&zfs->datasetdescs, dsdesc, next);
return (1);
}
free(opt);
rv = set_option(fsopts->fs_options, option, buf, sizeof(buf));
return (rv == -1 ? 0 : 1);
}
static void
zfs_size_vdev(fsinfo_t *fsopts)
{
zfs_opt_t *zfs;
off_t asize, mssize, vdevsize, vdevsize1;
zfs = fsopts->fs_specific;
assert(fsopts->maxsize != 0);
assert(zfs->ashift != 0);
/*
* Figure out how big the vdev should be.
*/
vdevsize = rounddown2(fsopts->maxsize, 1 << zfs->ashift);
if (vdevsize < MINDEVSIZE)
errx(1, "maximum image size is too small");
if (vdevsize < fsopts->minsize || vdevsize > fsopts->maxsize) {
errx(1, "image size bounds must be multiples of %d",
1 << zfs->ashift);
}
asize = vdevsize - VDEV_LABEL_SPACE;
/*
* Size metaslabs according to the following heuristic:
* - provide at least 8 metaslabs,
* - without using a metaslab size larger than 512MB.
* This approximates what OpenZFS does without being complicated. In
* practice we expect pools to be expanded upon first use, and OpenZFS
* does not resize metaslabs in that case, so there is no right answer
* here. In general we want to provide large metaslabs even if the
* image size is small, and 512MB is a reasonable size for pools up to
* several hundred gigabytes.
*
* The user may override this heuristic using the "-o mssize" option.
*/
mssize = zfs->mssize;
if (mssize == 0) {
mssize = MAX(MIN(asize / 8, DFLTMSSIZE), MINMSSIZE);
if (!powerof2(mssize))
mssize = 1l << (flsll(mssize) - 1);
}
if (!powerof2(mssize))
errx(1, "metaslab size must be a power of 2");
/*
* If we have some slop left over, try to cover it by resizing the vdev,
* subject to the maxsize and minsize parameters.
*/
if (asize % mssize != 0) {
vdevsize1 = rounddown2(asize, mssize) + VDEV_LABEL_SPACE;
if (vdevsize1 < fsopts->minsize)
vdevsize1 = roundup2(asize, mssize) + VDEV_LABEL_SPACE;
if (vdevsize1 <= fsopts->maxsize)
vdevsize = vdevsize1;
}
asize = vdevsize - VDEV_LABEL_SPACE;
zfs->asize = asize;
zfs->vdevsize = vdevsize;
zfs->mssize = mssize;
zfs->msshift = flsll(mssize) - 1;
zfs->mscount = asize / mssize;
}
/*
* Validate options and set some default values.
*/
static void
zfs_check_opts(fsinfo_t *fsopts)
{
zfs_opt_t *zfs;
zfs = fsopts->fs_specific;
if (fsopts->offset != 0)
errx(1, "unhandled offset option");
if (fsopts->maxsize == 0)
errx(1, "an image size must be specified");
if (zfs->poolname == NULL)
errx(1, "a pool name must be specified");
if (zfs->rootpath == NULL)
easprintf(&zfs->rootpath, "/%s", zfs->poolname);
if (zfs->rootpath[0] != '/')
errx(1, "mountpoint `%s' must be absolute", zfs->rootpath);
if (zfs->ashift == 0)
zfs->ashift = 12;
zfs_size_vdev(fsopts);
}
void
zfs_cleanup_opts(fsinfo_t *fsopts)
{
struct dataset_desc *d, *tmp;
zfs_opt_t *zfs;
zfs = fsopts->fs_specific;
free(zfs->rootpath);
free(zfs->bootfs);
free(__DECONST(void *, zfs->poolname));
STAILQ_FOREACH_SAFE(d, &zfs->datasetdescs, next, tmp) {
free(d->params);
free(d);
}
free(zfs);
free(fsopts->fs_options);
}
static size_t
nvlist_size(const nvlist_t *nvl)
{
return (sizeof(nvl->nv_header) + nvl->nv_size);
}
static void
nvlist_copy(const nvlist_t *nvl, char *buf, size_t sz)
{
assert(sz >= nvlist_size(nvl));
memcpy(buf, &nvl->nv_header, sizeof(nvl->nv_header));
memcpy(buf + sizeof(nvl->nv_header), nvl->nv_data, nvl->nv_size);
}
static nvlist_t *
pool_config_nvcreate(zfs_opt_t *zfs)
{
nvlist_t *featuresnv, *poolnv;
poolnv = nvlist_create(NV_UNIQUE_NAME);
nvlist_add_uint64(poolnv, ZPOOL_CONFIG_POOL_TXG, TXG);
nvlist_add_uint64(poolnv, ZPOOL_CONFIG_VERSION, SPA_VERSION);
nvlist_add_uint64(poolnv, ZPOOL_CONFIG_POOL_STATE, POOL_STATE_EXPORTED);
nvlist_add_string(poolnv, ZPOOL_CONFIG_POOL_NAME, zfs->poolname);
nvlist_add_uint64(poolnv, ZPOOL_CONFIG_POOL_GUID, zfs->poolguid);
nvlist_add_uint64(poolnv, ZPOOL_CONFIG_TOP_GUID, zfs->vdevguid);
nvlist_add_uint64(poolnv, ZPOOL_CONFIG_GUID, zfs->vdevguid);
nvlist_add_uint64(poolnv, ZPOOL_CONFIG_VDEV_CHILDREN, 1);
featuresnv = nvlist_create(NV_UNIQUE_NAME);
nvlist_add_nvlist(poolnv, ZPOOL_CONFIG_FEATURES_FOR_READ, featuresnv);
nvlist_destroy(featuresnv);
return (poolnv);
}
static nvlist_t *
pool_disk_vdev_config_nvcreate(zfs_opt_t *zfs)
{
nvlist_t *diskvdevnv;
assert(zfs->objarrid != 0);
diskvdevnv = nvlist_create(NV_UNIQUE_NAME);
nvlist_add_string(diskvdevnv, ZPOOL_CONFIG_TYPE, VDEV_TYPE_DISK);
nvlist_add_uint64(diskvdevnv, ZPOOL_CONFIG_ASHIFT, zfs->ashift);
nvlist_add_uint64(diskvdevnv, ZPOOL_CONFIG_ASIZE, zfs->asize);
nvlist_add_uint64(diskvdevnv, ZPOOL_CONFIG_GUID, zfs->vdevguid);
nvlist_add_uint64(diskvdevnv, ZPOOL_CONFIG_ID, 0);
nvlist_add_string(diskvdevnv, ZPOOL_CONFIG_PATH, "/dev/null");
nvlist_add_uint64(diskvdevnv, ZPOOL_CONFIG_WHOLE_DISK, 1);
nvlist_add_uint64(diskvdevnv, ZPOOL_CONFIG_CREATE_TXG, TXG);
nvlist_add_uint64(diskvdevnv, ZPOOL_CONFIG_METASLAB_ARRAY,
zfs->objarrid);
nvlist_add_uint64(diskvdevnv, ZPOOL_CONFIG_METASLAB_SHIFT,
zfs->msshift);
return (diskvdevnv);
}
static nvlist_t *
pool_root_vdev_config_nvcreate(zfs_opt_t *zfs)
{
nvlist_t *diskvdevnv, *rootvdevnv;
diskvdevnv = pool_disk_vdev_config_nvcreate(zfs);
rootvdevnv = nvlist_create(NV_UNIQUE_NAME);
nvlist_add_uint64(rootvdevnv, ZPOOL_CONFIG_ID, 0);
nvlist_add_uint64(rootvdevnv, ZPOOL_CONFIG_GUID, zfs->poolguid);
nvlist_add_string(rootvdevnv, ZPOOL_CONFIG_TYPE, VDEV_TYPE_ROOT);
nvlist_add_uint64(rootvdevnv, ZPOOL_CONFIG_CREATE_TXG, TXG);
nvlist_add_nvlist_array(rootvdevnv, ZPOOL_CONFIG_CHILDREN, &diskvdevnv,
1);
nvlist_destroy(diskvdevnv);
return (rootvdevnv);
}
/*
* Create the pool's "config" object, which contains an nvlist describing pool
* parameters and the vdev topology. It is similar but not identical to the
* nvlist stored in vdev labels. The main difference is that vdev labels do not
* describe the full vdev tree and in particular do not contain the "root"
* meta-vdev.
*/
static void
pool_init_objdir_config(zfs_opt_t *zfs, zfs_zap_t *objdir)
{
dnode_phys_t *dnode;
nvlist_t *poolconfig, *vdevconfig;
void *configbuf;
uint64_t dnid;
off_t configloc, configblksz;
int error;
dnode = objset_dnode_bonus_alloc(zfs->mos, DMU_OT_PACKED_NVLIST,
DMU_OT_PACKED_NVLIST_SIZE, sizeof(uint64_t), &dnid);
poolconfig = pool_config_nvcreate(zfs);
vdevconfig = pool_root_vdev_config_nvcreate(zfs);
nvlist_add_nvlist(poolconfig, ZPOOL_CONFIG_VDEV_TREE, vdevconfig);
nvlist_destroy(vdevconfig);
error = nvlist_export(poolconfig);
if (error != 0)
errc(1, error, "nvlist_export");
configblksz = nvlist_size(poolconfig);
configloc = objset_space_alloc(zfs, zfs->mos, &configblksz);
configbuf = ecalloc(1, configblksz);
nvlist_copy(poolconfig, configbuf, configblksz);
vdev_pwrite_dnode_data(zfs, dnode, configbuf, configblksz, configloc);
dnode->dn_datablkszsec = configblksz >> MINBLOCKSHIFT;
dnode->dn_flags = DNODE_FLAG_USED_BYTES;
*(uint64_t *)DN_BONUS(dnode) = nvlist_size(poolconfig);
zap_add_uint64(objdir, DMU_POOL_CONFIG, dnid);
nvlist_destroy(poolconfig);
free(configbuf);
}
/*
* Add objects block pointer list objects, used for deferred frees. We don't do
* anything with them, but they need to be present or OpenZFS will refuse to
* import the pool.
*/
static void
pool_init_objdir_bplists(zfs_opt_t *zfs __unused, zfs_zap_t *objdir)
{
uint64_t dnid;
(void)objset_dnode_bonus_alloc(zfs->mos, DMU_OT_BPOBJ, DMU_OT_BPOBJ_HDR,
BPOBJ_SIZE_V2, &dnid);
zap_add_uint64(objdir, DMU_POOL_FREE_BPOBJ, dnid);
(void)objset_dnode_bonus_alloc(zfs->mos, DMU_OT_BPOBJ, DMU_OT_BPOBJ_HDR,
BPOBJ_SIZE_V2, &dnid);
zap_add_uint64(objdir, DMU_POOL_SYNC_BPLIST, dnid);
}
/*
* Add required feature metadata objects. We don't know anything about ZFS
* features, so the objects are just empty ZAPs.
*/
static void
pool_init_objdir_feature_maps(zfs_opt_t *zfs, zfs_zap_t *objdir)
{
dnode_phys_t *dnode;
uint64_t dnid;
dnode = objset_dnode_alloc(zfs->mos, DMU_OTN_ZAP_METADATA, &dnid);
zap_add_uint64(objdir, DMU_POOL_FEATURES_FOR_READ, dnid);
zap_write(zfs, zap_alloc(zfs->mos, dnode));
dnode = objset_dnode_alloc(zfs->mos, DMU_OTN_ZAP_METADATA, &dnid);
zap_add_uint64(objdir, DMU_POOL_FEATURES_FOR_WRITE, dnid);
zap_write(zfs, zap_alloc(zfs->mos, dnode));
dnode = objset_dnode_alloc(zfs->mos, DMU_OTN_ZAP_METADATA, &dnid);
zap_add_uint64(objdir, DMU_POOL_FEATURE_DESCRIPTIONS, dnid);
zap_write(zfs, zap_alloc(zfs->mos, dnode));
}
static void
pool_init_objdir_dsl(zfs_opt_t *zfs, zfs_zap_t *objdir)
{
zap_add_uint64(objdir, DMU_POOL_ROOT_DATASET,
dsl_dir_id(zfs->rootdsldir));
}
static void
pool_init_objdir_poolprops(zfs_opt_t *zfs, zfs_zap_t *objdir)
{
dnode_phys_t *dnode;
uint64_t id;
dnode = objset_dnode_alloc(zfs->mos, DMU_OT_POOL_PROPS, &id);
zap_add_uint64(objdir, DMU_POOL_PROPS, id);
zfs->poolprops = zap_alloc(zfs->mos, dnode);
}
/*
* Initialize the MOS object directory, the root of virtually all of the pool's
* data and metadata.
*/
static void
pool_init_objdir(zfs_opt_t *zfs)
{
zfs_zap_t *zap;
dnode_phys_t *objdir;
objdir = objset_dnode_lookup(zfs->mos, DMU_POOL_DIRECTORY_OBJECT);
zap = zap_alloc(zfs->mos, objdir);
pool_init_objdir_config(zfs, zap);
pool_init_objdir_bplists(zfs, zap);
pool_init_objdir_feature_maps(zfs, zap);
pool_init_objdir_dsl(zfs, zap);
pool_init_objdir_poolprops(zfs, zap);
zap_write(zfs, zap);
}
/*
* Initialize the meta-object set (MOS) and immediately write out several
* special objects whose contents are already finalized, including the object
* directory.
*
* Once the MOS is finalized, it'll look roughly like this:
*
* object directory (ZAP)
* |-> vdev config object (nvlist)
* |-> features for read
* |-> features for write
* |-> feature descriptions
* |-> sync bplist
* |-> free bplist
* |-> pool properties
* L-> root DSL directory
* |-> DSL child directory (ZAP)
* | |-> $MOS (DSL dir)
* | | |-> child map
* | | L-> props (ZAP)
* | |-> $FREE (DSL dir)
* | | |-> child map
* | | L-> props (ZAP)
* | |-> $ORIGIN (DSL dir)
* | | |-> child map
* | | |-> dataset
* | | | L-> deadlist
* | | |-> snapshot
* | | | |-> deadlist
* | | | L-> snapshot names
* | | |-> props (ZAP)
* | | L-> clones (ZAP)
* | |-> dataset 1 (DSL dir)
* | | |-> DSL dataset
* | | | |-> snapshot names
* | | | L-> deadlist
* | | |-> child map
* | | | L-> ...
* | | L-> props
* | |-> dataset 2
* | | L-> ...
* | |-> ...
* | L-> dataset n
* |-> DSL root dataset
* | |-> snapshot names
* | L-> deadlist
* L-> props (ZAP)
* space map object array
* |-> space map 1
* |-> space map 2
* |-> ...
* L-> space map n (zfs->mscount)
*
* The space map object array is pointed to by the "msarray" property in the
* pool configuration.
*/
static void
pool_init(zfs_opt_t *zfs)
{
uint64_t dnid;
zfs->poolguid = ((uint64_t)random() << 32) | random();
zfs->vdevguid = ((uint64_t)random() << 32) | random();
zfs->mos = objset_alloc(zfs, DMU_OST_META);
(void)objset_dnode_alloc(zfs->mos, DMU_OT_OBJECT_DIRECTORY, &dnid);
assert(dnid == DMU_POOL_DIRECTORY_OBJECT);
(void)objset_dnode_alloc(zfs->mos, DMU_OT_OBJECT_ARRAY, &zfs->objarrid);
dsl_init(zfs);
pool_init_objdir(zfs);
}
static void
pool_labels_write(zfs_opt_t *zfs)
{
uberblock_t *ub;
vdev_label_t *label;
nvlist_t *poolconfig, *vdevconfig;
int error;
label = ecalloc(1, sizeof(*label));
/*
* Assemble the vdev configuration and store it in the label.
*/
poolconfig = pool_config_nvcreate(zfs);
vdevconfig = pool_disk_vdev_config_nvcreate(zfs);
nvlist_add_nvlist(poolconfig, ZPOOL_CONFIG_VDEV_TREE, vdevconfig);
nvlist_destroy(vdevconfig);
error = nvlist_export(poolconfig);
if (error != 0)
errc(1, error, "nvlist_export");
nvlist_copy(poolconfig, label->vl_vdev_phys.vp_nvlist,
sizeof(label->vl_vdev_phys.vp_nvlist));
nvlist_destroy(poolconfig);
/*
* Fill out the uberblock. Just make each one the same. The embedded
* checksum is calculated in vdev_label_write().
*/
for (size_t uoff = 0; uoff < sizeof(label->vl_uberblock);
uoff += (1 << zfs->ashift)) {
ub = (uberblock_t *)(&label->vl_uberblock[0] + uoff);
ub->ub_magic = UBERBLOCK_MAGIC;
ub->ub_version = SPA_VERSION;
ub->ub_txg = TXG;
ub->ub_guid_sum = zfs->poolguid + zfs->vdevguid;
ub->ub_timestamp = 0;
ub->ub_software_version = SPA_VERSION;
ub->ub_mmp_magic = MMP_MAGIC;
ub->ub_mmp_delay = 0;
ub->ub_mmp_config = 0;
ub->ub_checkpoint_txg = 0;
objset_root_blkptr_copy(zfs->mos, &ub->ub_rootbp);
}
/*
* Write out four copies of the label: two at the beginning of the vdev
* and two at the end.
*/
for (int i = 0; i < VDEV_LABELS; i++)
vdev_label_write(zfs, i, label);
free(label);
}
static void
pool_fini(zfs_opt_t *zfs)
{
zap_write(zfs, zfs->poolprops);
dsl_write(zfs);
objset_write(zfs, zfs->mos);
pool_labels_write(zfs);
}
struct dnode_cursor *
dnode_cursor_init(zfs_opt_t *zfs, zfs_objset_t *os, dnode_phys_t *dnode,
off_t size, off_t blksz)
{
struct dnode_cursor *c;
uint64_t nbppindir, indlevel, ndatablks, nindblks;
assert(dnode->dn_nblkptr == 1);
assert(blksz <= MAXBLOCKSIZE);
if (blksz == 0) {
/* Must be between 1<<ashift and 128KB. */
blksz = MIN(MAXBLOCKSIZE, MAX(1 << zfs->ashift,
powerof2(size) ? size : (1ul << flsll(size))));
}
assert(powerof2(blksz));
/*
* Do we need indirect blocks? Figure out how many levels are needed
* (indlevel == 1 means no indirect blocks) and how much space is needed
* (it has to be allocated up-front to break the dependency cycle
* described in objset_write()).
*/
ndatablks = size == 0 ? 0 : howmany(size, blksz);
nindblks = 0;
for (indlevel = 1, nbppindir = 1; ndatablks > nbppindir; indlevel++) {
nbppindir *= BLKPTR_PER_INDIR;
nindblks += howmany(ndatablks, indlevel * nbppindir);
}
assert(indlevel < INDIR_LEVELS);
dnode->dn_nlevels = (uint8_t)indlevel;
dnode->dn_maxblkid = ndatablks > 0 ? ndatablks - 1 : 0;
dnode->dn_datablkszsec = blksz >> MINBLOCKSHIFT;
c = ecalloc(1, sizeof(*c));
if (nindblks > 0) {
c->indspace = nindblks * MAXBLOCKSIZE;
c->indloc = objset_space_alloc(zfs, os, &c->indspace);
}
c->dnode = dnode;
c->dataoff = 0;
c->datablksz = blksz;
return (c);
}
static void
_dnode_cursor_flush(zfs_opt_t *zfs, struct dnode_cursor *c, int levels)
{
blkptr_t *bp, *pbp;
void *buf;
uint64_t fill;
off_t blkid, blksz, loc;
assert(levels > 0);
assert(levels <= c->dnode->dn_nlevels - 1);
blksz = MAXBLOCKSIZE;
blkid = (c->dataoff / c->datablksz) / BLKPTR_PER_INDIR;
for (int level = 1; level <= levels; level++) {
buf = c->inddir[level - 1];
if (level == c->dnode->dn_nlevels - 1) {
pbp = &c->dnode->dn_blkptr[0];
} else {
uint64_t iblkid;
iblkid = blkid & (BLKPTR_PER_INDIR - 1);
pbp = (blkptr_t *)
&c->inddir[level][iblkid * sizeof(blkptr_t)];
}
/*
* Space for indirect blocks is allocated up-front; see the
* comment in objset_write().
*/
loc = c->indloc;
c->indloc += blksz;
assert(c->indspace >= blksz);
c->indspace -= blksz;
bp = buf;
fill = 0;
for (size_t i = 0; i < BLKPTR_PER_INDIR; i++)
fill += BP_GET_FILL(&bp[i]);
vdev_pwrite_dnode_indir(zfs, c->dnode, level, fill, buf, blksz,
loc, pbp);
memset(buf, 0, MAXBLOCKSIZE);
blkid /= BLKPTR_PER_INDIR;
}
}
blkptr_t *
dnode_cursor_next(zfs_opt_t *zfs, struct dnode_cursor *c, off_t off)
{
off_t blkid, l1id;
int levels;
if (c->dnode->dn_nlevels == 1) {
assert(off < MAXBLOCKSIZE);
return (&c->dnode->dn_blkptr[0]);
}
assert(off % c->datablksz == 0);
/* Do we need to flush any full indirect blocks? */
if (off > 0) {
blkid = off / c->datablksz;
for (levels = 0; levels < c->dnode->dn_nlevels - 1; levels++) {
if (blkid % BLKPTR_PER_INDIR != 0)
break;
blkid /= BLKPTR_PER_INDIR;
}
if (levels > 0)
_dnode_cursor_flush(zfs, c, levels);
}
c->dataoff = off;
l1id = (off / c->datablksz) & (BLKPTR_PER_INDIR - 1);
return ((blkptr_t *)&c->inddir[0][l1id * sizeof(blkptr_t)]);
}
void
dnode_cursor_finish(zfs_opt_t *zfs, struct dnode_cursor *c)
{
int levels;
levels = c->dnode->dn_nlevels - 1;
if (levels > 0)
_dnode_cursor_flush(zfs, c, levels);
assert(c->indspace == 0);
free(c);
}
void
zfs_makefs(const char *image, const char *dir, fsnode *root, fsinfo_t *fsopts)
{
zfs_opt_t *zfs;
int dirfd;
zfs = fsopts->fs_specific;
/*
* Use a fixed seed to provide reproducible pseudo-random numbers for
* on-disk structures when needed (e.g., GUIDs, ZAP hash salts).
*/
srandom(1729);
zfs_check_opts(fsopts);
if (!zfs->nowarn) {
fprintf(stderr,
"ZFS support is currently considered experimental. "
"Do not use it for anything critical.\n");
}
allanjudeUnsubmitted
Not Done
Inline Actions

when ZFS grows the pool, it will add more metaslabs of this same size.

So we want to keep the number of slabs small, so that growing the pool to 100x the size doesn't result in 16,000 metaslabs.

The only downside is, any slack that doesn't fit into a metaslab will be unused, so if we just set a metaslab size of 4G, then we'd only have 8G usable from a 10G image.

Although without knowing how large they are going to grow it, it can be difficult to guess what number is right.

allanjude: when ZFS grows the pool, it will add more metaslabs of this same size. So we want to keep the…
markjAuthorUnsubmitted
Done
Inline Actions

Ahh, I thought that ZFS would potentially resize the metaslabs themselves. Apparently not. I'm not sure how best to handle this, given that OpenZFS itself doesn't seem to bother (at least based on the end of the comment in vdev_metaslab_set_size()).

It's not really possible to know which metaslab size is the "right" one, though. The tradeoffs involved also depend on whether certain optional features are enabled, like the log spacemap and spacemap histograms, I think. Right now, makefs' choice is too closely tied to the initial image size and will give excessively small metaslabs for most purposes. If our chosen metaslab size is large, we can also adjust the output image size to compensate based on the -m and -M parameters, so unused space isn't a major problem.

I'm inclined to keep it simple and do the following:

  • let the metaslab size be 10% of the image size, up to 512MB
  • add a -o msshift option to let the user override the heuristic

That'll give basically the same size as OpenZFS for images expanded to up to 100GB. Beyond that, the answer might just be to create a separate pool if you really need a larger metaslab size, or somehow teach OpenZFS to resize metaslabs if it can determine that it'd be profitable to do so.

markj: Ahh, I thought that ZFS would potentially resize the metaslabs themselves. Apparently not. I'm…
dirfd = open(dir, O_DIRECTORY | O_RDONLY);
if (dirfd < 0)
err(1, "open(%s)", dir);
vdev_init(zfs, image);
pool_init(zfs);
fs_build(zfs, dirfd, root);
pool_fini(zfs);
vdev_fini(zfs);
}
emasteUnsubmitted
Not Done
Inline Actions

(sigh)

emaste: (sigh)
View Options

multimedia/ffmpeg/files/patch-libavformat_tls__openssl.c

  • This file was added.
/*-
* SPDX-License-Identifier: BSD-2-Clause-FreeBSD
*
* Copyright (c) 2022 The FreeBSD Foundation
*
* This software was developed by Mark Johnston under sponsorship from
* the FreeBSD Foundation.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are
* met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in
* the documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*/
#include <sys/param.h>
#include <sys/errno.h>
#include <sys/queue.h>
#include <assert.h>
#include <fcntl.h>
#include <stdbool.h>
#include <stddef.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <util.h>
#include "makefs.h"
#include "zfs.h"
#define VDEV_LABEL_SPACE \
((off_t)(VDEV_LABEL_START_SIZE + VDEV_LABEL_END_SIZE))
_Static_assert(VDEV_LABEL_SPACE <= MINDEVSIZE, "");
#define MINMSSIZE ((off_t)1 << 24) /* 16MB */
#define DFLTMSSIZE ((off_t)1 << 29) /* 512MB */
#define MAXMSSIZE ((off_t)1 << 34) /* 16GB */
#define INDIR_LEVELS 6
/* Indirect blocks are always 128KB. */
#define BLKPTR_PER_INDIR (MAXBLOCKSIZE / sizeof(blkptr_t))
struct dnode_cursor {
char inddir[INDIR_LEVELS][MAXBLOCKSIZE];
off_t indloc;
off_t indspace;
dnode_phys_t *dnode;
off_t dataoff;
off_t datablksz;
};
void
zfs_prep_opts(fsinfo_t *fsopts)
{
zfs_opt_t *zfs = ecalloc(1, sizeof(*zfs));
const option_t zfs_options[] = {
{ '\0', "bootfs", &zfs->bootfs, OPT_STRPTR,
0, 0, "Bootable dataset" },
{ '\0', "mssize", &zfs->mssize, OPT_INT64,
MINMSSIZE, MAXMSSIZE, "Metaslab size" },
{ '\0', "poolname", &zfs->poolname, OPT_STRPTR,
0, 0, "ZFS pool name" },
{ '\0', "rootpath", &zfs->rootpath, OPT_STRPTR,
0, 0, "Prefix for all dataset mount points" },
{ '\0', "ashift", &zfs->ashift, OPT_INT32,
MINBLOCKSHIFT, MAXBLOCKSHIFT, "ZFS pool ashift" },
{ '\0', "nowarn", &zfs->nowarn, OPT_BOOL,
0, 0, "Suppress warning about experimental ZFS support" },
{ .name = NULL }
};
STAILQ_INIT(&zfs->datasetdescs);
fsopts->fs_specific = zfs;
fsopts->fs_options = copy_opts(zfs_options);
}
int
zfs_parse_opts(const char *option, fsinfo_t *fsopts)
{
zfs_opt_t *zfs;
struct dataset_desc *dsdesc;
char buf[BUFSIZ], *opt, *val;
int rv;
zfs = fsopts->fs_specific;
opt = val = estrdup(option);
opt = strsep(&val, "=");
if (strcmp(opt, "fs") == 0) {
if (val == NULL)
errx(1, "invalid filesystem parameters `%s'", option);
/*
* Dataset descriptions will be parsed later, in dsl_init().
* Just stash them away for now.
*/
dsdesc = ecalloc(1, sizeof(*dsdesc));
dsdesc->params = estrdup(val);
free(opt);
STAILQ_INSERT_TAIL(&zfs->datasetdescs, dsdesc, next);
return (1);
}
free(opt);
rv = set_option(fsopts->fs_options, option, buf, sizeof(buf));
return (rv == -1 ? 0 : 1);
}
static void
zfs_size_vdev(fsinfo_t *fsopts)
{
zfs_opt_t *zfs;
off_t asize, mssize, vdevsize, vdevsize1;
zfs = fsopts->fs_specific;
assert(fsopts->maxsize != 0);
assert(zfs->ashift != 0);
/*
* Figure out how big the vdev should be.
*/
vdevsize = rounddown2(fsopts->maxsize, 1 << zfs->ashift);
if (vdevsize < MINDEVSIZE)
errx(1, "maximum image size is too small");
if (vdevsize < fsopts->minsize || vdevsize > fsopts->maxsize) {
errx(1, "image size bounds must be multiples of %d",
1 << zfs->ashift);
}
asize = vdevsize - VDEV_LABEL_SPACE;
/*
* Size metaslabs according to the following heuristic:
* - provide at least 8 metaslabs,
* - without using a metaslab size larger than 512MB.
* This approximates what OpenZFS does without being complicated. In
* practice we expect pools to be expanded upon first use, and OpenZFS
* does not resize metaslabs in that case, so there is no right answer
* here. In general we want to provide large metaslabs even if the
* image size is small, and 512MB is a reasonable size for pools up to
* several hundred gigabytes.
*
* The user may override this heuristic using the "-o mssize" option.
*/
mssize = zfs->mssize;
if (mssize == 0) {
mssize = MAX(MIN(asize / 8, DFLTMSSIZE), MINMSSIZE);
if (!powerof2(mssize))
mssize = 1l << (flsll(mssize) - 1);
}
if (!powerof2(mssize))
errx(1, "metaslab size must be a power of 2");
/*
* If we have some slop left over, try to cover it by resizing the vdev,
* subject to the maxsize and minsize parameters.
*/
if (asize % mssize != 0) {
vdevsize1 = rounddown2(asize, mssize) + VDEV_LABEL_SPACE;
if (vdevsize1 < fsopts->minsize)
vdevsize1 = roundup2(asize, mssize) + VDEV_LABEL_SPACE;
if (vdevsize1 <= fsopts->maxsize)
vdevsize = vdevsize1;
}
asize = vdevsize - VDEV_LABEL_SPACE;
zfs->asize = asize;
zfs->vdevsize = vdevsize;
zfs->mssize = mssize;
zfs->msshift = flsll(mssize) - 1;
zfs->mscount = asize / mssize;
}
/*
* Validate options and set some default values.
*/
static void
zfs_check_opts(fsinfo_t *fsopts)
{
zfs_opt_t *zfs;
zfs = fsopts->fs_specific;
if (fsopts->offset != 0)
errx(1, "unhandled offset option");
if (fsopts->maxsize == 0)
errx(1, "an image size must be specified");
if (zfs->poolname == NULL)
errx(1, "a pool name must be specified");
if (zfs->rootpath == NULL)
easprintf(&zfs->rootpath, "/%s", zfs->poolname);
if (zfs->rootpath[0] != '/')
errx(1, "mountpoint `%s' must be absolute", zfs->rootpath);
if (zfs->ashift == 0)
zfs->ashift = 12;
zfs_size_vdev(fsopts);
}
void
zfs_cleanup_opts(fsinfo_t *fsopts)
{
struct dataset_desc *d, *tmp;
zfs_opt_t *zfs;
zfs = fsopts->fs_specific;
free(zfs->rootpath);
free(zfs->bootfs);
free(__DECONST(void *, zfs->poolname));
STAILQ_FOREACH_SAFE(d, &zfs->datasetdescs, next, tmp) {
free(d->params);
free(d);
}
free(zfs);
free(fsopts->fs_options);
}
static size_t
nvlist_size(const nvlist_t *nvl)
{
return (sizeof(nvl->nv_header) + nvl->nv_size);
}
static void
nvlist_copy(const nvlist_t *nvl, char *buf, size_t sz)
{
assert(sz >= nvlist_size(nvl));
memcpy(buf, &nvl->nv_header, sizeof(nvl->nv_header));
memcpy(buf + sizeof(nvl->nv_header), nvl->nv_data, nvl->nv_size);
}
static nvlist_t *
pool_config_nvcreate(zfs_opt_t *zfs)
{
nvlist_t *featuresnv, *poolnv;
poolnv = nvlist_create(NV_UNIQUE_NAME);
nvlist_add_uint64(poolnv, ZPOOL_CONFIG_POOL_TXG, TXG);
nvlist_add_uint64(poolnv, ZPOOL_CONFIG_VERSION, SPA_VERSION);
nvlist_add_uint64(poolnv, ZPOOL_CONFIG_POOL_STATE, POOL_STATE_EXPORTED);
nvlist_add_string(poolnv, ZPOOL_CONFIG_POOL_NAME, zfs->poolname);
nvlist_add_uint64(poolnv, ZPOOL_CONFIG_POOL_GUID, zfs->poolguid);
nvlist_add_uint64(poolnv, ZPOOL_CONFIG_TOP_GUID, zfs->vdevguid);
nvlist_add_uint64(poolnv, ZPOOL_CONFIG_GUID, zfs->vdevguid);
nvlist_add_uint64(poolnv, ZPOOL_CONFIG_VDEV_CHILDREN, 1);
featuresnv = nvlist_create(NV_UNIQUE_NAME);
nvlist_add_nvlist(poolnv, ZPOOL_CONFIG_FEATURES_FOR_READ, featuresnv);
nvlist_destroy(featuresnv);
return (poolnv);
}
static nvlist_t *
pool_disk_vdev_config_nvcreate(zfs_opt_t *zfs)
{
nvlist_t *diskvdevnv;
assert(zfs->objarrid != 0);
diskvdevnv = nvlist_create(NV_UNIQUE_NAME);
nvlist_add_string(diskvdevnv, ZPOOL_CONFIG_TYPE, VDEV_TYPE_DISK);
nvlist_add_uint64(diskvdevnv, ZPOOL_CONFIG_ASHIFT, zfs->ashift);
nvlist_add_uint64(diskvdevnv, ZPOOL_CONFIG_ASIZE, zfs->asize);
nvlist_add_uint64(diskvdevnv, ZPOOL_CONFIG_GUID, zfs->vdevguid);
nvlist_add_uint64(diskvdevnv, ZPOOL_CONFIG_ID, 0);
nvlist_add_string(diskvdevnv, ZPOOL_CONFIG_PATH, "/dev/null");
nvlist_add_uint64(diskvdevnv, ZPOOL_CONFIG_WHOLE_DISK, 1);
nvlist_add_uint64(diskvdevnv, ZPOOL_CONFIG_CREATE_TXG, TXG);
nvlist_add_uint64(diskvdevnv, ZPOOL_CONFIG_METASLAB_ARRAY,
zfs->objarrid);
nvlist_add_uint64(diskvdevnv, ZPOOL_CONFIG_METASLAB_SHIFT,
zfs->msshift);
return (diskvdevnv);
}
static nvlist_t *
pool_root_vdev_config_nvcreate(zfs_opt_t *zfs)
{
nvlist_t *diskvdevnv, *rootvdevnv;
diskvdevnv = pool_disk_vdev_config_nvcreate(zfs);
rootvdevnv = nvlist_create(NV_UNIQUE_NAME);
nvlist_add_uint64(rootvdevnv, ZPOOL_CONFIG_ID, 0);
nvlist_add_uint64(rootvdevnv, ZPOOL_CONFIG_GUID, zfs->poolguid);
nvlist_add_string(rootvdevnv, ZPOOL_CONFIG_TYPE, VDEV_TYPE_ROOT);
nvlist_add_uint64(rootvdevnv, ZPOOL_CONFIG_CREATE_TXG, TXG);
nvlist_add_nvlist_array(rootvdevnv, ZPOOL_CONFIG_CHILDREN, &diskvdevnv,
1);
nvlist_destroy(diskvdevnv);
return (rootvdevnv);
}
/*
* Create the pool's "config" object, which contains an nvlist describing pool
* parameters and the vdev topology. It is similar but not identical to the
* nvlist stored in vdev labels. The main difference is that vdev labels do not
* describe the full vdev tree and in particular do not contain the "root"
* meta-vdev.
*/
static void
pool_init_objdir_config(zfs_opt_t *zfs, zfs_zap_t *objdir)
{
dnode_phys_t *dnode;
nvlist_t *poolconfig, *vdevconfig;
void *configbuf;
uint64_t dnid;
off_t configloc, configblksz;
int error;
dnode = objset_dnode_bonus_alloc(zfs->mos, DMU_OT_PACKED_NVLIST,
DMU_OT_PACKED_NVLIST_SIZE, sizeof(uint64_t), &dnid);
poolconfig = pool_config_nvcreate(zfs);
vdevconfig = pool_root_vdev_config_nvcreate(zfs);
nvlist_add_nvlist(poolconfig, ZPOOL_CONFIG_VDEV_TREE, vdevconfig);
nvlist_destroy(vdevconfig);
error = nvlist_export(poolconfig);
if (error != 0)
errc(1, error, "nvlist_export");
configblksz = nvlist_size(poolconfig);
configloc = objset_space_alloc(zfs, zfs->mos, &configblksz);
configbuf = ecalloc(1, configblksz);
nvlist_copy(poolconfig, configbuf, configblksz);
vdev_pwrite_dnode_data(zfs, dnode, configbuf, configblksz, configloc);
dnode->dn_datablkszsec = configblksz >> MINBLOCKSHIFT;
dnode->dn_flags = DNODE_FLAG_USED_BYTES;
*(uint64_t *)DN_BONUS(dnode) = nvlist_size(poolconfig);
zap_add_uint64(objdir, DMU_POOL_CONFIG, dnid);
nvlist_destroy(poolconfig);
free(configbuf);
}
/*
* Add objects block pointer list objects, used for deferred frees. We don't do
* anything with them, but they need to be present or OpenZFS will refuse to
* import the pool.
*/
static void
pool_init_objdir_bplists(zfs_opt_t *zfs __unused, zfs_zap_t *objdir)
{
uint64_t dnid;
(void)objset_dnode_bonus_alloc(zfs->mos, DMU_OT_BPOBJ, DMU_OT_BPOBJ_HDR,
BPOBJ_SIZE_V2, &dnid);
zap_add_uint64(objdir, DMU_POOL_FREE_BPOBJ, dnid);
(void)objset_dnode_bonus_alloc(zfs->mos, DMU_OT_BPOBJ, DMU_OT_BPOBJ_HDR,
BPOBJ_SIZE_V2, &dnid);
zap_add_uint64(objdir, DMU_POOL_SYNC_BPLIST, dnid);
}
/*
* Add required feature metadata objects. We don't know anything about ZFS
* features, so the objects are just empty ZAPs.
*/
static void
pool_init_objdir_feature_maps(zfs_opt_t *zfs, zfs_zap_t *objdir)
{
dnode_phys_t *dnode;
uint64_t dnid;
dnode = objset_dnode_alloc(zfs->mos, DMU_OTN_ZAP_METADATA, &dnid);
zap_add_uint64(objdir, DMU_POOL_FEATURES_FOR_READ, dnid);
zap_write(zfs, zap_alloc(zfs->mos, dnode));
dnode = objset_dnode_alloc(zfs->mos, DMU_OTN_ZAP_METADATA, &dnid);
zap_add_uint64(objdir, DMU_POOL_FEATURES_FOR_WRITE, dnid);
zap_write(zfs, zap_alloc(zfs->mos, dnode));
dnode = objset_dnode_alloc(zfs->mos, DMU_OTN_ZAP_METADATA, &dnid);
zap_add_uint64(objdir, DMU_POOL_FEATURE_DESCRIPTIONS, dnid);
zap_write(zfs, zap_alloc(zfs->mos, dnode));
}
static void
pool_init_objdir_dsl(zfs_opt_t *zfs, zfs_zap_t *objdir)
{
zap_add_uint64(objdir, DMU_POOL_ROOT_DATASET,
dsl_dir_id(zfs->rootdsldir));
}
static void
pool_init_objdir_poolprops(zfs_opt_t *zfs, zfs_zap_t *objdir)
{
dnode_phys_t *dnode;
uint64_t id;
dnode = objset_dnode_alloc(zfs->mos, DMU_OT_POOL_PROPS, &id);
zap_add_uint64(objdir, DMU_POOL_PROPS, id);
zfs->poolprops = zap_alloc(zfs->mos, dnode);
}
/*
* Initialize the MOS object directory, the root of virtually all of the pool's
* data and metadata.
*/
static void
pool_init_objdir(zfs_opt_t *zfs)
{
zfs_zap_t *zap;
dnode_phys_t *objdir;
objdir = objset_dnode_lookup(zfs->mos, DMU_POOL_DIRECTORY_OBJECT);
zap = zap_alloc(zfs->mos, objdir);
pool_init_objdir_config(zfs, zap);
pool_init_objdir_bplists(zfs, zap);
pool_init_objdir_feature_maps(zfs, zap);
pool_init_objdir_dsl(zfs, zap);
pool_init_objdir_poolprops(zfs, zap);
zap_write(zfs, zap);
}
/*
* Initialize the meta-object set (MOS) and immediately write out several
* special objects whose contents are already finalized, including the object
* directory.
*
* Once the MOS is finalized, it'll look roughly like this:
*
* object directory (ZAP)
* |-> vdev config object (nvlist)
* |-> features for read
* |-> features for write
* |-> feature descriptions
* |-> sync bplist
* |-> free bplist
* |-> pool properties
* L-> root DSL directory
* |-> DSL child directory (ZAP)
* | |-> $MOS (DSL dir)
* | | |-> child map
* | | L-> props (ZAP)
* | |-> $FREE (DSL dir)
* | | |-> child map
* | | L-> props (ZAP)
* | |-> $ORIGIN (DSL dir)
* | | |-> child map
* | | |-> dataset
* | | | L-> deadlist
* | | |-> snapshot
* | | | |-> deadlist
* | | | L-> snapshot names
* | | |-> props (ZAP)
* | | L-> clones (ZAP)
* | |-> dataset 1 (DSL dir)
* | | |-> DSL dataset
* | | | |-> snapshot names
* | | | L-> deadlist
* | | |-> child map
* | | | L-> ...
* | | L-> props
* | |-> dataset 2
* | | L-> ...
* | |-> ...
* | L-> dataset n
* |-> DSL root dataset
* | |-> snapshot names
* | L-> deadlist
* L-> props (ZAP)
* space map object array
* |-> space map 1
* |-> space map 2
* |-> ...
* L-> space map n (zfs->mscount)
*
* The space map object array is pointed to by the "msarray" property in the
* pool configuration.
*/
static void
pool_init(zfs_opt_t *zfs)
{
uint64_t dnid;
zfs->poolguid = ((uint64_t)random() << 32) | random();
zfs->vdevguid = ((uint64_t)random() << 32) | random();
zfs->mos = objset_alloc(zfs, DMU_OST_META);
(void)objset_dnode_alloc(zfs->mos, DMU_OT_OBJECT_DIRECTORY, &dnid);
assert(dnid == DMU_POOL_DIRECTORY_OBJECT);
(void)objset_dnode_alloc(zfs->mos, DMU_OT_OBJECT_ARRAY, &zfs->objarrid);
dsl_init(zfs);
pool_init_objdir(zfs);
}
static void
pool_labels_write(zfs_opt_t *zfs)
{
uberblock_t *ub;
vdev_label_t *label;
nvlist_t *poolconfig, *vdevconfig;
int error;
label = ecalloc(1, sizeof(*label));
/*
* Assemble the vdev configuration and store it in the label.
*/
poolconfig = pool_config_nvcreate(zfs);
vdevconfig = pool_disk_vdev_config_nvcreate(zfs);
nvlist_add_nvlist(poolconfig, ZPOOL_CONFIG_VDEV_TREE, vdevconfig);
nvlist_destroy(vdevconfig);
error = nvlist_export(poolconfig);
if (error != 0)
errc(1, error, "nvlist_export");
nvlist_copy(poolconfig, label->vl_vdev_phys.vp_nvlist,
sizeof(label->vl_vdev_phys.vp_nvlist));
nvlist_destroy(poolconfig);
/*
* Fill out the uberblock. Just make each one the same. The embedded
* checksum is calculated in vdev_label_write().
*/
for (size_t uoff = 0; uoff < sizeof(label->vl_uberblock);
uoff += (1 << zfs->ashift)) {
ub = (uberblock_t *)(&label->vl_uberblock[0] + uoff);
ub->ub_magic = UBERBLOCK_MAGIC;
ub->ub_version = SPA_VERSION;
ub->ub_txg = TXG;
ub->ub_guid_sum = zfs->poolguid + zfs->vdevguid;
ub->ub_timestamp = 0;
ub->ub_software_version = SPA_VERSION;
ub->ub_mmp_magic = MMP_MAGIC;
ub->ub_mmp_delay = 0;
ub->ub_mmp_config = 0;
ub->ub_checkpoint_txg = 0;
objset_root_blkptr_copy(zfs->mos, &ub->ub_rootbp);
}
/*
* Write out four copies of the label: two at the beginning of the vdev
* and two at the end.
*/
for (int i = 0; i < VDEV_LABELS; i++)
vdev_label_write(zfs, i, label);
free(label);
}
static void
pool_fini(zfs_opt_t *zfs)
{
zap_write(zfs, zfs->poolprops);
dsl_write(zfs);
objset_write(zfs, zfs->mos);
pool_labels_write(zfs);
}
struct dnode_cursor *
dnode_cursor_init(zfs_opt_t *zfs, zfs_objset_t *os, dnode_phys_t *dnode,
off_t size, off_t blksz)
{
struct dnode_cursor *c;
uint64_t nbppindir, indlevel, ndatablks, nindblks;
assert(dnode->dn_nblkptr == 1);
assert(blksz <= MAXBLOCKSIZE);
if (blksz == 0) {
/* Must be between 1<<ashift and 128KB. */
blksz = MIN(MAXBLOCKSIZE, MAX(1 << zfs->ashift,
powerof2(size) ? size : (1ul << flsll(size))));
}
assert(powerof2(blksz));
/*
* Do we need indirect blocks? Figure out how many levels are needed
* (indlevel == 1 means no indirect blocks) and how much space is needed
* (it has to be allocated up-front to break the dependency cycle
* described in objset_write()).
*/
ndatablks = size == 0 ? 0 : howmany(size, blksz);
nindblks = 0;
for (indlevel = 1, nbppindir = 1; ndatablks > nbppindir; indlevel++) {
nbppindir *= BLKPTR_PER_INDIR;
nindblks += howmany(ndatablks, indlevel * nbppindir);
}
assert(indlevel < INDIR_LEVELS);
dnode->dn_nlevels = (uint8_t)indlevel;
dnode->dn_maxblkid = ndatablks > 0 ? ndatablks - 1 : 0;
dnode->dn_datablkszsec = blksz >> MINBLOCKSHIFT;
c = ecalloc(1, sizeof(*c));
if (nindblks > 0) {
c->indspace = nindblks * MAXBLOCKSIZE;
c->indloc = objset_space_alloc(zfs, os, &c->indspace);
}
c->dnode = dnode;
c->dataoff = 0;
c->datablksz = blksz;
return (c);
}
static void
_dnode_cursor_flush(zfs_opt_t *zfs, struct dnode_cursor *c, int levels)
{
blkptr_t *bp, *pbp;
void *buf;
uint64_t fill;
off_t blkid, blksz, loc;
assert(levels > 0);
assert(levels <= c->dnode->dn_nlevels - 1);
blksz = MAXBLOCKSIZE;
blkid = (c->dataoff / c->datablksz) / BLKPTR_PER_INDIR;
for (int level = 1; level <= levels; level++) {
buf = c->inddir[level - 1];
if (level == c->dnode->dn_nlevels - 1) {
pbp = &c->dnode->dn_blkptr[0];
} else {
uint64_t iblkid;
iblkid = blkid & (BLKPTR_PER_INDIR - 1);
pbp = (blkptr_t *)
&c->inddir[level][iblkid * sizeof(blkptr_t)];
}
/*
* Space for indirect blocks is allocated up-front; see the
* comment in objset_write().
*/
loc = c->indloc;
c->indloc += blksz;
assert(c->indspace >= blksz);
c->indspace -= blksz;
bp = buf;
fill = 0;
for (size_t i = 0; i < BLKPTR_PER_INDIR; i++)
fill += BP_GET_FILL(&bp[i]);
vdev_pwrite_dnode_indir(zfs, c->dnode, level, fill, buf, blksz,
loc, pbp);
memset(buf, 0, MAXBLOCKSIZE);
blkid /= BLKPTR_PER_INDIR;
}
}
blkptr_t *
dnode_cursor_next(zfs_opt_t *zfs, struct dnode_cursor *c, off_t off)
{
off_t blkid, l1id;
int levels;
if (c->dnode->dn_nlevels == 1) {
assert(off < MAXBLOCKSIZE);
return (&c->dnode->dn_blkptr[0]);
}
assert(off % c->datablksz == 0);
/* Do we need to flush any full indirect blocks? */
if (off > 0) {
blkid = off / c->datablksz;
for (levels = 0; levels < c->dnode->dn_nlevels - 1; levels++) {
if (blkid % BLKPTR_PER_INDIR != 0)
break;
blkid /= BLKPTR_PER_INDIR;
}
if (levels > 0)
_dnode_cursor_flush(zfs, c, levels);
}
c->dataoff = off;
l1id = (off / c->datablksz) & (BLKPTR_PER_INDIR - 1);
return ((blkptr_t *)&c->inddir[0][l1id * sizeof(blkptr_t)]);
}
void
dnode_cursor_finish(zfs_opt_t *zfs, struct dnode_cursor *c)
{
int levels;
levels = c->dnode->dn_nlevels - 1;
if (levels > 0)
_dnode_cursor_flush(zfs, c, levels);
assert(c->indspace == 0);
free(c);
}
void
zfs_makefs(const char *image, const char *dir, fsnode *root, fsinfo_t *fsopts)
{
zfs_opt_t *zfs;
int dirfd;
zfs = fsopts->fs_specific;
/*
* Use a fixed seed to provide reproducible pseudo-random numbers for
* on-disk structures when needed (e.g., GUIDs, ZAP hash salts).
*/
srandom(1729);
zfs_check_opts(fsopts);
if (!zfs->nowarn) {
fprintf(stderr,
"ZFS support is currently considered experimental. "
"Do not use it for anything critical.\n");
}
allanjudeUnsubmitted
Not Done
Inline Actions

when ZFS grows the pool, it will add more metaslabs of this same size.

So we want to keep the number of slabs small, so that growing the pool to 100x the size doesn't result in 16,000 metaslabs.

The only downside is, any slack that doesn't fit into a metaslab will be unused, so if we just set a metaslab size of 4G, then we'd only have 8G usable from a 10G image.

Although without knowing how large they are going to grow it, it can be difficult to guess what number is right.

allanjude: when ZFS grows the pool, it will add more metaslabs of this same size. So we want to keep the…
markjAuthorUnsubmitted
Done
Inline Actions

Ahh, I thought that ZFS would potentially resize the metaslabs themselves. Apparently not. I'm not sure how best to handle this, given that OpenZFS itself doesn't seem to bother (at least based on the end of the comment in vdev_metaslab_set_size()).

It's not really possible to know which metaslab size is the "right" one, though. The tradeoffs involved also depend on whether certain optional features are enabled, like the log spacemap and spacemap histograms, I think. Right now, makefs' choice is too closely tied to the initial image size and will give excessively small metaslabs for most purposes. If our chosen metaslab size is large, we can also adjust the output image size to compensate based on the -m and -M parameters, so unused space isn't a major problem.

I'm inclined to keep it simple and do the following:

  • let the metaslab size be 10% of the image size, up to 512MB
  • add a -o msshift option to let the user override the heuristic

That'll give basically the same size as OpenZFS for images expanded to up to 100GB. Beyond that, the answer might just be to create a separate pool if you really need a larger metaslab size, or somehow teach OpenZFS to resize metaslabs if it can determine that it'd be profitable to do so.

markj: Ahh, I thought that ZFS would potentially resize the metaslabs themselves. Apparently not. I'm…
dirfd = open(dir, O_DIRECTORY | O_RDONLY);
if (dirfd < 0)
err(1, "open(%s)", dir);
vdev_init(zfs, image);
pool_init(zfs);
fs_build(zfs, dirfd, root);
pool_fini(zfs);
vdev_fini(zfs);
}
emasteUnsubmitted
Not Done
Inline Actions

(sigh)

emaste: (sigh)
View Options

multimedia/ffmpeg/pkg-plist

Loading...