You note that ``If we then crashed before the dirent write, we would recover with a state where the file was still linked, but had been truncated to zero. The resulting state could be considered corruption.'' The filesystem is not corrupted in the sense that it needs fsck to be run to clean it up. We simply end up with an unexpected result.

It would be trivial to request high priority for synchronous writes in bwrite() and if desired synchronous reads in bread(). That would have effects for several filesystems.

Added a couple of inline comments.

In D26511#591360, @kib wrote:

The block being written with the barrier is a newly allocated block of inodes. The write is done asynchronously and the cylinder group is then updated to reflect that the additional inodes are available. The reason for the barrier is so that the cylinder group buffer with the newly expanded inode map cannot be written before the newly allocated set of inodes.

Since the incore cylinder group includes the newly created inodes, some other thread can come along and try to use one of those newly allocated inodes. but, it will block on the inode buffer until its write has completed.

The bug in this instance is that there is an assumption that the write cannot fail. Clearly this is a bad assumption. So, the correct fix is to not depend on the barrier write, but rather to create a callback that updates the cylinder group once the write of the new inodes has completed successfully.

In the specific case of kostik1316, doing that would deadlock the machine. Problem is that CoW write returned ENOSPC, which means that there is no way to correctly free space on the volume. I am not sure what do to there. Most likely any other write would also return ENOSPC, so in fact consistency of the volume is not too badly broken if we just fail there. If we start tracking the write as a dependency for cg write, then all that buffers add to the dirty space which eventually hang buffer subsystem.

I believe (based on Peter testing) that just erroring write allows us to unmount.

What we are trying to do where the barrier write is being used in UFS is expand the number of available inodes. It is OK to abandon the write of the zero'ed out inode block as the expansion can be put off to later. But, we must not update the cylinder group to say that these new inodes are available if we have not been able to zero them out. As it is currently written, the cylinder group is updated but the on-disk inodes are not zero'ed out. If the unmount succeeds in writing out the dirty buffers we will end up with a corrupted filesystem because fsck_ffs will try to check the non-zero'ed out inodes and raise numerous errors trying to correct the inconsistencies that arise from the random data in the uninitialzed inode block.

In D26511#590951, @kib wrote:

In D26511#590923, @mckusick wrote:

In D26511#590229, @kib wrote:

In D26511#590172, @mckusick wrote:

As long as the buffer remains locked until it is successfully written, this should be fine.

Sorry, I do not fully understand you note. Do you mean that it is fine to have B_BARRIER set as far as buffer is not unlocked ? If not, could you please clarify.

We are depending on this being written before it can be used. If it were unlocked, then some other thread could get it and make use of it. See comment above use of babarrierwrite in sys/ufs/ffs/ffs_alloc.c.

I still do not understand what do you mean by 'use'. In the kostik1316 dump, most probable scenario was that babarrierwrite() for the inode block failed with ENOSPC, and then bufdone() does brelse() on the buffer. So it is unlocked, but this is somewhat unrelated to the issue of leaking B_BARRIER.

In D26511#590229, @kib wrote:

In D26511#590172, @mckusick wrote:

As long as the buffer remains locked until it is successfully written, this should be fine.

Sorry, I do not fully understand you note. Do you mean that it is fine to have B_BARRIER set as far as buffer is not unlocked ? If not, could you please clarify.

We are depending on this being written before it can be used. If it were unlocked, then some other thread could get it and make use of it. See comment above use of babarrierwrite in sys/ufs/ffs/ffs_alloc.c.

As long as the buffer remains locked until it is successfully written, this should be fine.

This change looks reasonable to me as at least for amd64 every path out of the kernel appears to check TDF_ASTPENDING and calls ast() if set.

It was a horrible hack at the time and should have been tossed decades ago.

A new review? If so, I suggest that you close this one and give a pointer to the new one.

What is the status with this change?

The get_parent_vp() function seems to be handling two different problems.
In its first instance it is dealing with locking its parent. Here we already have the function vn_vget_ino() which presumably could be used to handle this situation.
In its second and third instances it is avoiding a LOR of acquiring an inode lock while holding a buffer lock. Here we need a function like the proposed get_parent_vp(). However, we will no longer need the first (vp) argument as we are acquiring a child inode so do not have to release the already held parent inode before acquiring the child inode.

This change will provide the interface that we will need to notify daemon processes for our forcible unmount and forcible downgrade to read-only as well as re-establishment of read-write after a successful background fsck.

I concur with the proposed change and also agree that cem's suggestion is a good one.

This should not be done as written as it can cause a denial of service (infinite loop) as described in my previous comment.

Done.

If data is being written to the filesystem faster than the disk can write it then a sync with MNT_WAIT will never finish. The only safe way to use sync with MNT_WAIT is to first suspend the filesystem to create a finite number of write operations that need to be done.

I cannot speak to the correctness of the whole patch, but looking over D25577, D25578, and this one, D25579, I see nothing that looks incorrect.

Per the above commentary, this change should be withdrawn.

Having read Chuck's argument, I agree with it. I was thinking that the g_label_ufs.c routines passed in M_UFSMNT as their malloc type. But they do not. They use M_GEOM malloc type. So there is no need for them to have an definition for M_UFSMNT.

Thanks for this cleanup. I found the previous transformation grating for the reasons that you listed.

This looks like a reasonable change to me.

I believe that the existing test already has this covered. The test (resid > uio->uio_resid) is checking that any data has been written. In the typical case this is at the end of the file and the size will have increased. But if the write was in the middle of a file filling in a hole then this test will catch that and cause the inode to be written.

Interesting that I never added myself to the committers-src.dot file.
For reference, the file that actually controls access to doing src commits is /usr/src/svnadmin/conf/access.
For interesting statistics on the history of commits, fetch these files:

fetch https://people.freebsd.org/~rodrigo/commits-base.txt
fetch https://people.freebsd.org/~rodrigo/commits-doc.txt
fetch https://people.freebsd.org/~rodrigo/commits-ports.txt

I agree that there should be documentation of MFC guidelines somewhere.
For a change of this sort an MFC delay in the range of 1-2 weeks is reasonable.

Your argument has pursuaded me. Looks good to go.

One other suggestion is to change "normal" to "timeshare" or perhaps "timeshr" as that name is more descriptive of what is being reported.

On further though, I agree with you that ffs_subr.c should only be shared functions.

This looks like a big improvement. My one suggestion is that you add the priority value for normal (as level - PRI_MIN_TIMESHARE).

Rather than moving all these functions from a file in which they logically belong to one in which they do not, how about we just put #ifdef FFS around the _KERNEL part. This produces minimal changes, leaves things where they logically belong, and reduces the size of the extra copy of ffs_subr.o.

This looks good to go.

Overall this looks like a correct change. As noted in my inline comment, I do not think that you should change the semantics of waitfor.

Finally, this looks good to go.

I believe that this change is correct as it stands.

Peter,
Can you run one more test run over this code just to be sure that it patches cleanly and that none of our final changes broke anything?

This looks good to me. I am fine with not supporting the old behavior.

This rather long block is a summary of the discussion that Kostik
and I had in email. Dropping it in here so that it is part of the
public record. Regular font are my comments; italics font are
Kostik's response to my comments.

As imp has noted, we have had readdir(3) and the getdirents(2) for nearly 40 years and many filesystems do not support directly reading directories. While I personally still use od(1) to inspect UFS directories, that is a pretty lame reason for keeping the functionality. And returning EISDIR will make us more compatible with Linux and remain in compliance with POSIX. So, all told I am in favor of this change. Adding a sysctl to revert to the old behavior seems reasonable so long as it defaults to the new (EISDIR) behavior.

I concur with this change.
It should be MFC'ed to stable/12 and stable/11.

Changes moved to D24210.

These changes will be done as part of D24210 so that all the updates to make the dump and restore utilities compile with -fno-common are in one place.

Rather than using EXTERN, place each global declaration into the file in which it is primarily used.

Taking over with Kyle Evans blessing.

Now looks good.

Rather than doing EXTERN, move the explicit declarations into file as this:
mapsize - main.c
usedinomap - main.c
dumpdirmap - main.c
dumpinomap - main.c
disk - main.c
tape - main.c
popenout - main.c
dumpdates - itime.c
temp - unused, remove from dump.h, main.c, and definition of _PATH_DTMP from pathnames.h
lastlevel - itime.c
level - main.c
uflag - main.c
diskfd - main.c
tapefd - tape.c
pipeout - main.c
curino - tape.c
newtape - tape.c
density - already declared in main.c
tapesize - main.c
tsize - main.c
asize - tape.c
etapes - main.c
nonodump - main.c
unlimited - main.c
cachesize - already declared in main.c
rsync_friendly - main.c
notify - already declared in main.c
blockswritten - already declared in main.c
tapeno - already declared in main.c
tstart_writing - main.c
tend_writing - main.c
passno - main.c
sblock - main.c
dev_bsize - already declared in main.c
dev_bshift - main.c
tp_bshift - main.c
dumpdates - leave in itime.c
nddates - leave in itime.c
If you would like me to take over this project, I am willing to do so.

I prefer just using extern and then declaring the actual variables in the function in which they are most prominent, but using EXTERN as you have done does work fine.

While u_spcl is used only in dump/tape.c, it is used in four different files in restore and is not meant to be a separate instance in each one. So, you will have to make it an extern in the header file and then declare it in dump/tape.c and restore/dump.c. I prefer that you do it that way as opposed to the much uglier EXTERN approach.

This looks complete. I had missed the VOP_* manual pages, good catch.
I am happy to have you clean up the duplicate EIOs.
Thanks for doing this.

While read(2) and write(2) are the most likely system calls to return EINTEGRITY, really any system call that accesses a filesystem can potentially return EINTEGRITY. Using the metric that if it can return EIO, then it can now return EINTEGRITY leads to this list of system calls needing EINTEGRITY added: access.2, acct.2, bind.2, chdir.2, chflags.2, chmod.2, chown.2, chroot.2, copy_file_range.2, execve.2, fhlink.2, fhreadlink.2, fsync.2, getdirentries.2, getfh.2, getfsstat.2, intro.2, ktrace.2, link.2, lio_listio.2, mkdir.2, mkfifo.2, mknod.2, mount.2, msync.2, open.2, pathconf.2, posix_fallocate.2, quotactl.2, readlink.2, rename.2, rmdir.2, sendfile.2, stat.2, statfs.2, swapon.2, symlink.2, truncate.2, undelete.2, unlink.2, utimensat.2, and utimes.2.

This is a good example of a place that where EINTEGRITY is intended to be used.
At the moment mount(2) is the only system call that lists EINTEGRITY as a possible error. If your change is going to cause EINTEGRITY to be returned to other system calls (like read(2) or wirite(2)) then you should add it to their list of possible error returns.

Looks good to me.

With the above suggested changes, I agree with this proposal.

I am happy with this revision.

The logic is easier for me to follow.