Changeset View
Changeset View
Standalone View
Standalone View
testing.txt
- This file was added.
To: freebsd-current, freebsd-fs, freebsd-ports | |||||
ngie: Remove file? | |||||
emasteUnsubmitted Not Done Inline ActionsYes this one is also not for inclusion in the commit emaste: Yes this one is also not for inclusion in the commit | |||||
Subject: 64-bit inodes (ino64) Status Update and Call for Testing | |||||
Inodes are data structures corresponding to objects in a file system, | |||||
such as files and directories. FreeBSD has historically used 32-bit | |||||
values to identify inodes, which limits file systems to somewhat under | |||||
2^32 objects. Many modern file systems internally use 64-bit identifiers | |||||
and FreeBSD needs to follow suit to properly and fully support these | |||||
file systems. | |||||
The 64-bit inode project, also known as ino64, started life in 2009 or so | |||||
in a project branch by Google Summer of Code student Gleb Kurtsou. Since | |||||
then many people have had a hand in updating it and addressing | |||||
regressions, and as part of the path to being committed to head it is now | |||||
ready for wider testing. | |||||
This note explains how to perform useful testing of the ino64 branch, | |||||
beyond typical smoke tests. | |||||
1. Overview. | |||||
The ino64 branch extends the basic system types ino_t and dev_t from | |||||
32-bit to 64-bit, and nlink_t from 16-bit to 64-bit. The struct dirent | |||||
layout is modified due to the larger size of ino_t, and also gains a | |||||
d_off (directory offset) member. As ino64 implies an ABI change anyway | |||||
the struct statfs f_mntfromname[] and f_mntonname[] array length | |||||
MNAMELEN is increased from 88 to 1024, to allow for longer mount path | |||||
names. | |||||
ABI breakage is mitigated by providing compatibility using versioned | |||||
symbols, ingenious use of the existing padding in structures, and by | |||||
employing other tricks. Unfortunately, not everything can be fixed, | |||||
especially outside the base system. For instance, third-party APIs | |||||
which pass struct stat around are broken in backward and forward- | |||||
incompatible way. | |||||
2. Motivation. | |||||
The main risk of the ino64 change is the uncontrolled ABI breakage. | |||||
Due to expansion of the basic types dev_t, ino_t and struct dirent, | |||||
the impact is not limited to one part of the system, but affects: | |||||
- kernel/userspace interface (syscalls ABI, mostly stat(2), kinfo | |||||
and more) | |||||
- libc interface (mostly related to the readdir(3), FTS(3)) | |||||
- collateral damage in other libraries that happens to use changed types | |||||
in the interfaces. See, for instance, libprocstat, for which compat | |||||
was provided using symbol versioning, and libutil, which shlib version | |||||
was bumped. | |||||
3. Quirks. | |||||
We handled kinfo sysctl MIBs, but other MIBs which report structures | |||||
depended on the changed type, are not handled in general. It was | |||||
wblockUnsubmitted Not Done Inline Actionss/depended/dependant/ We handled kinfo sysctl MIBs, but, in general, other MIBs which report structures depending on the changed type are not handled. wblock: s/depended/dependant/
or possibly "depending". The "in general" at the end is kind of… | |||||
kibAuthorUnsubmitted Not Done Inline ActionsThe word meant to inform that we provide no general mechanism to handle sysctls which return structures with changed layout. Moving 'in general' to the place you proposed changes the meaning. kib: The word meant to inform that we provide no general mechanism to handle sysctls which return… | |||||
wblockUnsubmitted Done Inline ActionsAh, that helps. Well, see what you think: We handled kinfo sysctl MIBs, but there is no general mechanism to handle other sysctl MIBS which return structures where the layout has changed. wblock: Ah, that helps. Well, see what you think:
```We handled kinfo sysctl MIBs, but there is no… | |||||
wblockUnsubmitted Not Done Inline Actionss/depended/depending/ wblock: s/depended/depending/ | |||||
considered that the breakage is either in the management interfaces, | |||||
where we usually allow ABI slip, or is not important. | |||||
Struct xvnode changed layout, no compat shims are provided. | |||||
For struct xtty, dev_t tty device member was reduced to uint32_t. It | |||||
was decided that keeping ABI compat in this case is more useful than | |||||
reporting 64bit dev_t, for the sake of pstat. | |||||
4. Testing procedure. | |||||
The ino64 project can be tested by cloning the project branch from | |||||
GitHub or by applying the patch <from the Phabricator review | located | |||||
at URL | attached> to a working tree. The authorative source is the | |||||
GitHub, I do not promise to update the review for each update. | |||||
To clone from GitHub: | |||||
% git clone -b ino64 https://github.com/FreeBSDFoundation/freebsd.git | |||||
To fetch the patch from Phabricator: | |||||
- URL | |||||
- Click download raw diff from the right | |||||
Or | |||||
% arc patch D12345 | |||||
4.1 New kernel, old world. | |||||
Build and install pristine HEAD world, apply patch and only build and | |||||
install updated kernel. The system must work same as with the pristine | |||||
kernel. | |||||
4.2 New kernel, new world, old third-party applications. | |||||
Build and install patched kernel and world. Applications compiled on the | |||||
pristine HEAD (e.g. installed by pkg from the regular portmgr builds) must | |||||
work without a regression. | |||||
4.3 32bit compat. | |||||
Same as 4.1 and 4.2, but for 32bit (i386) binaries on the amd64 host. | |||||
Note that big-endian host, like powerpc, might expose additional | |||||
bugs in the 32bit compat with the patch, but the testing is too cumbersome | |||||
to arrange. | |||||
4.4 Targeted tests. | |||||
Useful programs to check items 4.1, 4.2 and 4.3 are versions of the | |||||
following programs, taken from the pristine system: | |||||
stat(8). Use it on regular file, file in /dev, socket, pipe and so | |||||
on. For both native and 32bit compat, stat(8) must print reasonable | |||||
information. | |||||
procstat(1). Use it with the -f option to examine processes files. | |||||
kinfo(9) data must be returned in the format acceptable for older | |||||
apps. | |||||
Use pristine find(1) binary with many arbitrary options on a system with | |||||
installed patched world, in particular, libc. Find examines FTS(3), | |||||
and compat shims in libc are non-trivial. | |||||
4.5 NFS server and client test. | |||||
Check that the NFS server in the patched kernel operates correctly and without | |||||
performance regressions. Same for client. | |||||
NFS should be checked for all four combination of patched/unpatched | |||||
kernel server/client, because the filehandle format includes inode number. | |||||
4.6 Other filesystems | |||||
Generally, filesystems should see no change in the system behaviour, | |||||
since patch goal is to provide space to grow in the ABI. In | |||||
particular, local filesystem layout must stay same. Of course, it is | |||||
possible that some reliance on the exact sizes of the changed types | |||||
was left unnoticed during the patch review, in which case e.g. on-disk | |||||
format would be broken. We do not expect this to slip in, but it is | |||||
possible and should be watched for. | |||||
4.7 Test accounting | |||||
The process accounting, as documented in acct(5), changed format of | |||||
the records due to dev_t increase. Verify that the programs like | |||||
sa(8) and accton(8) correctly work with both old and new accounting | |||||
records. | |||||
5. Ports Status with ino64 | |||||
A ports exp-run for ino64 is open in PR 218320. The failing ports each | |||||
responsible for more than 1 skipped port are: | |||||
lang/ghc 497 | |||||
multimedia/webcamd 62 | |||||
lang/gcc6-aux 54 | |||||
devel/libgtop 39 | |||||
sysutils/py-psutil 13 | |||||
devel/llvm38 6 | |||||
lang/rust 4 | |||||
sysutils/py-psutil121 3 | |||||
Patches are available for lang/llvm39, lang/llvm40, lang/ghc, and | |||||
lang/rust in the topic branch as ports.patch, and llvm38 can be fixed | |||||
in the same way as llvm39 and llvm40. Assistance with investigating | |||||
and fixing the port failures will be greatly appreciated. | |||||
Below is an overview of the problems and proposed solutions, probably | |||||
mostly relevant to the ports maintainers. | |||||
5.1. LLVM | |||||
LLVM includes a component called Address Sanitizer or ASAN, which tries | |||||
to intercept syscalls, and contains knowledge of the layout of many | |||||
system structures. Since stat and lstat syscalls were removed and | |||||
several types and structures changed, this has to be reflected in the | |||||
ASAN hacks. | |||||
5.2. lang/ghc | |||||
The ghc compiler and parts of the runtime are written in Haskell, | |||||
which means that to compile ghc, you need a working Haskell compiler | |||||
for bootstrap. By default for ghc, the runtime is provided in the form | |||||
of static libraries. Static libraries reference default versions of | |||||
libc symbols, which are assigned the ELF symbol version at the final | |||||
linking stage. As result, using such libraries results in using the | |||||
updated syscall, but internally the code still uses old system types. | |||||
The end result is the random memory corruption because both libc and | |||||
kernel assume new types. | |||||
This situation cannot be fixed by symbol versioning, because versioning | |||||
acts too late. Instead, we hacked the bootstrap compiler by providing | |||||
symbols for modified syscalls in the shipped static libraries, which | |||||
symbols direct execution to the compat variants of the syscalls. This | |||||
allows the bootstrap compiler to generate working code. After the | |||||
stage0, compiler operates on new structures and things stabilize. | |||||
The real solution is, of course, to re-package the bootstrap compiler, | |||||
but for some time we need to support pre-ino64 HEAD in ports. Also, | |||||
learning full scope of GHC maintainance duties, required for that, | |||||
is too much for the ino64 task. | |||||
5.3. lang/rust | |||||
Rustc has a similar structure to GHC, and same issue. The same solution | |||||
of patching the bootstrap was done. | |||||
Also rust libstd and liblibc provide rustified definitions of the | |||||
system structures, which were updated to reflect the updated layout. | |||||
I failed to understand why e.g. struct stat has to be defined in 3 | |||||
places at least, but all found locations were patched. | |||||
6. Next Steps | |||||
The tentative schedule for the ino64 project: | |||||
2017-04-20 Post wide call for testing | |||||
Investigate and address port failures with maintainer support | |||||
2017-05-05 Request second exp-run with initial patches applied | |||||
Investigate and address port failures with maintainer support | |||||
2017-05-19 Commit to HEAD | |||||
Address post-commit failures where feasible |
Remove file?