vt(4): Add support for console screen reader using sysctls.

[• hselasky]

Summary

This allows the contents of the current FreeBSD VT console to be accessed
by vtspeakd(1) for example. The API is defined by the following sysctls:

kern.vt.screen_reader.feed
kern.vt.screen_reader.text_utf8
kern.vt.screen_reader.col
kern.vt.screen_reader.row
kern.vt.screen_reader.lines
kern.vt.screen_reader.hangup_pid

MFC after: 1 week
Sponsored by: NVIDIA Networking

Test Plan

1. Patch, build and install new kernel.
2. Apply and install beep(1) patches, See https://reviews.freebsd.org/D35772
3. Apply and install vtspeakd(1) patches, See https://reviews.freebsd.org/D35776

Then reboot and run:
vtspeakd -c

For example.

[• hselasky]

Fixed some spelling

[• hselasky]

Some more fixes.

[imp]

I'm not at all sure this is the right place for this...

sys/dev/vt/vt_accessibility.c

91

please construct the messages using sbuf.

[• hselasky]

This is my first attempt implementing something like this. I needed to extend the term_char_t type to 64-bits to make room for a bit which keeps track of what characters have been printed or not.

When storing new characters this bit gets reset to zero.

sys/dev/vt/vt_accessibility.c

91

Sure, but devd has its limits on the message size including headers (1Kbyte I think)

[pauamma_gundo.com]

Niiiiiiiiiice. (Added Accessibility as a group reviewer.)

[pauamma_gundo.com]

Are you looking for feature testing at this time?

[• hselasky]

All feedback is welcome!

Some comments from a friend of mine:

Q: We translate all the special characters (such as '.' and '/') into "period" and "divide". This means those characters will always be spoken because espeak sees it as a word. A sysctl to disable that will be nice, but not required.
A: The problem is shell escaping - need to be careful.

Q: Is there any way to get the cursor position (<line number>:<column position>)?
A: Probably yes.

Q: NVDA and others use CAPS LOCK+{u,i,o} to move the cursor up one line, say current line or move cursor down.
A: Not sure if they have a patent on that.

[• hselasky]

Are you looking for feature testing at this time?

Yes, please test as much as you can.

[pauamma_gundo.com]

In D35754#811359, @hselasky wrote:

Are you looking for feature testing at this time?

Yes, please test as much as you can.

I don't use a screenreader myself, but I'm going to ask for testers on freebsd-accessibility@. Before I do, can you confirm my understanding of what's needed, per outline below?

• A fairly recent 14-current install (eg 20220708 or later from https://download.freebsd.org/snapshots/ISO-IMAGES/14.0/) using https://docs.freebsd.org/en/books/handbook/cutting-edge/#current-stable 24.5.1. Using FreeBSD-CURRENT
• audio/espeak installed and configured (for the default configuration) or an alternative (if willing to adapt devd/accessibility.conf to their preferred software)
• this patch applied, then buildworld+buildkernel+installkernel+installworld per https://docs.freebsd.org/en/books/handbook/cutting-edge/#makeworld
• sysctl kern.vt.accessibility.enable=1 as root.

Overall, this is fairly involved, and while I'm willing to help people through this, I may reach my limit fast. How much are you willing and able to help (me help) testers?

[pauamma_gundo.com]

In D35754#811358, @hselasky wrote:

All feedback is welcome!

Two things:

• I would make the devd conf filename more specific, perhaps screenreading.espeak.conf or something similar.
• Likewise, maybe kern.vt.screenreading.enable for the sysctl name.

[• hselasky]

Hi,

You just need to patch /usr/src and build a new kernel.

Then you run:

cp -i /usr/src/sbin/devd/accessibility.conf /etc/devd/

Then you reboot and run:

pkg install espeak

Then:

sysctl kern.vt.accessibility.enable=1

Beware that the inner APIs of this feature may change.

I've gotten some feedback and will probably export all text as a sysctl nodes, to avoid shell escaping, either as raw UTF-8 or spoken text (no raw characters).

--HPS

[imp]

I'd really rather you do all this not with devd.
It puts too many strings through it: devd wasn't really designed for a high-throughput introspection interface. You are already introducing weird interfaces to cope with flow control and I'd rather not have more-critical messages delayed waiting for all the console output to play out (I also worry a weird error starving devd as well).
There's also too much string code in the kernel with this..
Is there some reason you can't open use the snp device to snoop console output and pass that through to easyspeak w/o the need to "stand on your head" to get cope with devd's weird string quoting quirks. You can get the characters directly.
And best of all, you don't need a new kernel for it, but you may have to load snp.ko.

[• hselasky]

Hi Warner,

In D35754#811558, @imp wrote:

I'd really rather you do all this not with devd.

I agree. Consider this a proof of concept.
When I have some more time I will make a sysctl only API skipping the devd part.

It puts too many strings through it: devd wasn't really designed for a high-throughput introspection interface. You are already introducing weird interfaces to cope with flow control and I'd rather not have more-critical messages delayed waiting for all the console output to play out (I also worry a weird error starving devd as well).

I know exactly what you mean, understood.

There's also too much string code in the kernel with this..
Is there some reason you can't open use the snp device to snoop console output and pass that through to easyspeak w/o the need to "stand on your head" to get cope with devd's weird string quoting quirks. You can get the characters directly.
And best of all, you don't need a new kernel for it, but you may have to load snp.ko.

I didn't look that much into snp yet, but I guess it doesn't understand switching the console ALT+F12345 ... and it doesn't understand where you are on the screen or if you are scrolling the history.

Also if you simply copy the terminal stream there might be too much to read. For example you open some manual page and it starts reading, then you change your mind and close the manual page and clear the screen using CTRL+L . Then the espeak should stop after completing its last one or two spoken lines and not everything. How is SNP supposed to know that?

Adding a 32nd bit the the term_char_t was an easy solution on the other hand to keep track of dirty text areas.

[pauamma_gundo.com]

In D35754#811499, @hselasky wrote:

I've gotten some feedback and will probably export all text as a sysctl nodes, to avoid shell escaping, either as raw UTF-8 or spoken text (no raw characters).

I'm going to hold back the accessibility@ CFT until the design settles, so there's just one installation procedure.

[• hselasky]

Please also see D35772 .

[• hselasky]

@imp: No more devd dependency. Trying to keep this stuff as simple as possible.

[imp]

Better. Might consider having a char device to publish this, but given you are looking at different views maybe not...

I'd also still consider pushing the 'interpretation of the text' to a filter in userland vs having it in the kernel. While it makes the userland side simpler, the kernel is complicated with stuff that might not be required to be here.... Just a thought.

[• hselasky]

@imp: Moving the chatty stuff to userspace is a possibility. Then the kernel patch for vt(4) will be quite small and focused.

[• hselasky]

Moved chatty to userspace as suggested by imp@

Scan for first non-word separator when outputting line.

[• hselasky]

Fix multi-line support. Wrap some long lines.

[• hselasky]

Ping.

[• hselasky]

• Implement support for sending hangup signal to a speak program on bell.
• Remove support for using beep as output.
• Added manual page for vtspeakd(1).

[pauamma_gundo.com]

I should soonish be able to repurpose my current daily driver laptop to test -current stuff. Then I'll write up the procedure and post a CFT to accessibility@.

[• hselasky]

In D35754#838405, @pauamma wrote:

I should soonish be able to repurpose my current daily driver laptop to test -current stuff. Then I'll write up the procedure and post a CFT to accessibility@.

@pauamma : Looking forward to that!

[trueos_norwegianrockcat.com]

While I don't have much to add for the code, I'll offer a suggestion with my "researcher in universal design of ICT" hat on.

I would suggest that instead of having hw.vt.accessibility.* to hw.vt.screenreader (or consolereader?) My reasoning for this suggestion is that accessibility in general is a big area dealing many things (both input and output), e.g., making a braille display work in FreeBSD would probably require some sort of device driver beyond the VT. The code here is more to "read" the content off the screen and into sysctls for that can be used by a text-to-speech service (or something else like the braille display I mentioned).

So, while I agree that this is real accessibility work, I think calling the sysctls "screenreader" might fit better or at least avoid confusion. That is, people interested in accessibility typically understand what a screen reader is, and people who are not as interested in accessibility will have a better idea what the sysctls do.

I'll just finish by saying that this is a great feature to add! I'm wondering how things will work with libdialog-based UIs (or curses) or if that the library could communicate to the screenreader and let it know that this part of the screen is a button, but those sort of things can be explored after this lands.

[• hselasky]

Your input is much appreciated. I'll wait for others to complete testing before making final decisions.

[pauamma_gundo.com]

Can you go over the write-up below before I post it and tell me if I missed anything or got anything wrong?

Installing and testing the console screenreading patches:

• Note: for installing the OS and compiling the patches, you can if needed use whatever screenreading setup you're comfortable with.
• install a 14.0 snapshot (https://download.freebsd.org/snapshots/ISO-IMAGES/14.0/) with the src (system source) option. If you intend to help debug in addition to testing features, install the kernel-dbg option in addition to the default base-dbg. Create a non-root user.
• Login as root
• pkg install espeak-ng
• fetch -o vt-driver.diff 'https://reviews.freebsd.org/D35754?download=true' # Mind the single quotes here and in the next 2 commands.
• fetch -o beep.diff 'https://reviews.freebsd.org/D35772?download=true'
• fetch -o vtspeakd.diff 'https://reviews.freebsd.org/D35776?download=true'
• cd /usr/src
• patch -p 1 < /root/beep.diff
• patch -p 1 < /root/vtspeakd.diff
• patch -p 1 < /root/vt-driver.diff
• cd /usr/src/usr.bin/beep
• make && make install
• cd /usr/src/usr.bin/vtspeakd
• make && make install
• cd /usr/src
• make kernel INSTKERNNAME=kernel.speechsynth KERNCONF=GENERIC # Or the configuration you're using, but if that doesn't work, try GENERIC next.
• make installkernel.debug INSTKERNNAME=kernel.speechsynth KERNCONF=GENERIC
• nextboot -k kernel.speechsynth
• shutdown -r now Rebooting for one-shot testing console speech synth kernel
• Login as non-root user
• beep -t "Hi" # Mind the double quotes here and in the commands below that use -t.
• beep -t "ERROR" -D 50
• beep -F 220 -t "Hi, this is a test." -D 50
• beep -n 0
• beep -n 1
• beep -n 2Installing and testing the console screenreading patches:
• Note: for installing the OS and compiling the patches, you can if needed use whatever screenreading setup you're comfortable with.
• install a 14.0 snapshot (https://download.freebsd.org/snapshots/ISO-IMAGES/14.0/) with the src (system source) option. If you intend to help debug in addition to testing features, install the kernel-dbg option in addition to the default base-dbg. Create a non-root user.
• Login as root
• pkg install espeak-ng
• fetch -o vt-driver.diff 'https://reviews.freebsd.org/D35754?download=true' # Mind the single quotes here and in the next 2 commands.
• fetch -o beep.diff 'https://reviews.freebsd.org/D35772?download=true'
• fetch -o vtspeakd.diff 'https://reviews.freebsd.org/D35776?download=true'
• cd /usr/src
• patch -p 1 < /root/beep.diff
• patch -p 1 < /root/vtspeakd.diff
• patch -p 1 < /root/vt-driver.diff
• cd /usr/src/usr.bin/beep
• make && make install
• cd /usr/src/usr.bin/vtspeakd
• make && make install
• cd /usr/src
• make kernel INSTKERNNAME=kernel.speechsynth KERNCONF=GENERIC # Or the configuration you're using, but if that doesn't work, try GENERIC next.
• make installkernel.debug INSTKERNNAME=kernel.speechsynth KERNCONF=GENERIC
• nextboot -k kernel.speechsynth
• shutdown -r now Rebooting for one-shot testing console speech synth kernel
• Login as non-root user
• beep -t "Hi" # Mind the double quotes here and in the commands below that use -t.
• beep -t "ERROR" -D 50
• beep -F 220 -t "Hi, this is a test." -D 50
• beep -n 0
• beep -n 1
• beep -n 2
• beep -t "Ψωμί! Παιδειά! Και όχι επαιτειά!" # Or whichever non-ASCII text you can enter.
• Login as root
• vtspeakd -B # Listen to the initial screen contents
• Login as non-root user
• echo Hi from the screenreader.

[• hselasky]

@pauamma : Your commands for testing look good! I think you got everything right. I'm impressed!

The beep text command is just there. I'm not sure if it is useful, but the beep -n X for numbers is more useful I think. Then you can easily distinguish between different characters and binary codes. Typically when they are not that long.

Like you easily hear the difference between 5 and 6, if in doubt.

[• hselasky]

@pauamma : You should also test that CTRL+B stops the speaker output and that CTRL+L clears the screen.

[pauamma_gundo.com]

In D35754#842191, @hselasky wrote:

@pauamma : You should also test that CTRL+B stops the speaker output and that CTRL+L clears the screen.

Both work, if by "stops the speaker output" you mean it skips reading the current line and starts reading the next one.

[• hselasky]

• Renamed accessibility to screen reader.
• Fixed LOR.

[• hselasky]

@pauamma : Can you remove the 3-old patches and download fresh patches again and re-test?

[pauamma_gundo.com]

In D35754#851648, @hselasky wrote:

@pauamma : Can you remove the 3-old patches and download fresh patches again and re-test?

I can, and did, and I can't trigger the LOR so far. Control-B still seems to have no effect when displaying ttyv0 and vtspeakd was started on ttyv1. With that same vtspeakd VTY, control-B works on ttyv2.

[pauamma_gundo.com]

Tangentially: do you plan to write this up for the Q4 2022 status report?

[• hselasky]

@pauamma : Yes, I want to submit this soon. Can you repeat all outstanding issues with my patch-set which are not solved?

--HPS

[pauamma_gundo.com]

In D35754#857506, @hselasky wrote:

@pauamma : Yes, I want to submit this soon. Can you repeat all outstanding issues with my patch-set which are not solved?

Here are the ones I noticed while using it:

• Functionality: Control-B still seems to have no effect when displaying ttyv0 and vtspeakd was started on ttyv1. With that same vtspeakd VTY, control-B works on ttyv2. Control-B also has no effect when used on ttyv0 and vtspeakd was started on ttyv0.

• UX: I don't know whether it's espeak-ng or my hearing, but ffffffff sounds like eh-eh-eh-eh-eh-eh-eh-eh to me, which is disconcerting. (Trying other voices changed the sound output slightly but not in ways that help.)

Eventually, this (and specifically vtspeakd) may need to be extended or modified if it's to become (part of) a basic screenreader, but these reviews are probably not the place to discuss it.

[• hselasky]

Regarding:

Functionality: Control-B still seems to have no effect when displaying ttyv0 and vtspeakd was started on ttyv1. With that same vtspeakd VTY, control-B works on ttyv2. Control-B also has no effect when used on ttyv0 and vtspeakd was started on ttyv0.

Can you explain to me step by step how to reproduce. I don't fully get this. Are you using the serial console. I.E. my patches only support characters going to a real VT console, and that VT console must be selected by using typically ALT+F1 ... F2 ... F3 and so on.

[pauamma_gundo.com]

In D35754#857510, @hselasky wrote:

Regarding:

Functionality: Control-B still seems to have no effect when displaying ttyv0 and vtspeakd was started on ttyv1. With that same vtspeakd VTY, control-B works on ttyv2. Control-B also has no effect when used on ttyv0 and vtspeakd was started on ttyv0.

Can you explain to me step by step how to reproduce. I don't fully get this. Are you using the serial console. I.E. my patches only support characters going to a real VT console, and that VT console must be selected by using typically ALT+F1 ... F2 ... F3 and so on.

No, using the built-in VTY (but booting with EFI and using a USB keyboard, if that makes a difference; that laptop's builtin keyboard is broken and keeps spewing garbage characters, so I disabled it in devices.hint).

Steps to reproduce:
1- Boot and wait for the ttyv0 login prompt.
2- Using alt-F2, switch to ttyv1.
3- Login as root.
4- At the # prompt, enter "clear; vtspeakd".
5- Switch back to ttyv0 using alt-F1.
6- As vtspeakd starts reading each line, press control-B once per line, a second or two in.
7- When vtspeakd is done reading, switch to ttyv2 using alt-F3.
8- Login as root.
9- As vtspeakd starts reading each line, press control-B once per line, a second or two in.
Expected: at steps 6 and 9, reading each line is aborted on successive control-B presses and vtspeakd skips to the next line.
Observed: at step 6, each line is read out in full (incorrect); at step 9, vtspeak aborts and skips to the next line on every control-B press (correct).

[pauamma_gundo.com]

In D35754#857506, @hselasky wrote:

@pauamma : Yes, I want to submit this soon. Can you repeat all outstanding issues with my patch-set which are not solved?

Do you need more time to work on your status report submission?

[• hselasky]

In D35754#857657, @pauamma_gundo.com wrote:

In D35754#857510, @hselasky wrote:

Regarding:

Functionality: Control-B still seems to have no effect when displaying ttyv0 and vtspeakd was started on ttyv1. With that same vtspeakd VTY, control-B works on ttyv2. Control-B also has no effect when used on ttyv0 and vtspeakd was started on ttyv0.

Can you explain to me step by step how to reproduce. I don't fully get this. Are you using the serial console. I.E. my patches only support characters going to a real VT console, and that VT console must be selected by using typically ALT+F1 ... F2 ... F3 and so on.

No, using the built-in VTY (but booting with EFI and using a USB keyboard, if that makes a difference; that laptop's builtin keyboard is broken and keeps spewing garbage characters, so I disabled it in devices.hint).

Steps to reproduce:
1- Boot and wait for the ttyv0 login prompt.
2- Using alt-F2, switch to ttyv1.
3- Login as root.
4- At the # prompt, enter "clear; vtspeakd".
5- Switch back to ttyv0 using alt-F1.
6- As vtspeakd starts reading each line, press control-B once per line, a second or two in.
7- When vtspeakd is done reading, switch to ttyv2 using alt-F3.
8- Login as root.
9- As vtspeakd starts reading each line, press control-B once per line, a second or two in.
Expected: at steps 6 and 9, reading each line is aborted on successive control-B presses and vtspeakd skips to the next line.
Observed: at step 6, each line is read out in full (incorrect); at step 9, vtspeak aborts and skips to the next line on every control-B press (correct).

If the console is in input mode or not reading any text, then CTRL+B doesn't work, like reading a line. Maybe I should fix that?

--HPS

[pauamma_gundo.com]

In D35754#903091, @hselasky wrote:

In D35754#857657, @pauamma_gundo.com wrote:

Expected: at steps 6 and 9, reading each line is aborted on successive control-B presses and vtspeakd skips to the next line.
Observed: at step 6, each line is read out in full (incorrect); at step 9, vtspeak aborts and skips to the next line on every control-B press (correct).

If the console is in input mode or not reading any text, then CTRL+B doesn't work, like reading a line. Maybe I should fix that?

I think you should, at least when it's in input mode. (And your description of the cause feels more likely correct than mine.)

[markj]

sys/conf/files

3468

This should sort below dev/vt/vt_font.c.

sys/dev/vt/vt.h

247

This line is too long.

sys/dev/vt/vt_buf.c

749

Is part of the patch missing?

918

The lines here are too long. Maybe it's a sign that the loop body should be split into a separate function?

sys/dev/vt/vt_screen_reader.c

70 ↗

(On Diff #113389)

There is no need to set CTLFLAG_MPSAFE on UINT/U16 sysctls.

77 ↗

(On Diff #113389)

80 ↗

(On Diff #113389)

87 ↗

(On Diff #113389)

Lines here are too long.

156 ↗

(On Diff #113389)

Isn't this dangerous? What if the screen reader crashes and its PID is reused for something else? Or does something protect against that?

sys/sys/terminal.h

76

The comment above states that this encoding is chosen to save memory. What are the effects of doubling the width here? For example, vt_constext[] alone grows from 1MB to 2MB, which seems like a lot of overhead. Maybe some more clever encoding is possible?

[markj]

sys/dev/vt/vt_screen_reader.c

156 ↗

(On Diff #113389)

Looking at vtspeakd, I think a better mechanism would be to use a device file interface rather than sysctls, and have kernel state attached to a device file descriptor. Then that state can be released reliably when vtspeakd exits, even if it crashes.

[• hselasky]

sys/dev/vt/vt_screen_reader.c

156 ↗

(On Diff #113389)

Yes, I can do that.

Then it will be a header file with some IOCTLs inside.

Do you have any thoughts about the name or location of such a header file?

What about the name of the /dev/xxx node?

--HPS

[markj]

sys/dev/vt/vt_screen_reader.c

156 ↗

(On Diff #113389)

Can it be an ioctl on ttyvN, handled by vtterm_ioctl()? There are already some vt(4)-specific ioctls.

[asiciliano]

I am interested to take/continue this feature. Are others already working?

[markj]

In D35754#943181, @asiciliano wrote:

I am interested to take/continue this feature. Are others already working?

Not that I've seen. I think there are two major things which need to be addressed: the userspace interface should be converted to use file descriptors rather than sysctls, and we should find a way to avoid making term_char_t larger.

[asiciliano]

In D35754#943208, @markj wrote:

In D35754#943181, @asiciliano wrote:

I am interested to take/continue this feature. Are others already working?

Not that I've seen. I think there are two major things which need to be addressed: the userspace interface should be converted to use file descriptors rather than sysctls, and we should find a way to avoid making term_char_t larger.

@markj ok, probably you are referencing to your previous comment:

Can it be an ioctl on ttyvN, handled by vtterm_ioctl()? There are already some vt(4)-specific ioctls.

I need a while to complete a new utility to allow to blind people to run "make config" in /usr/ports. Then I'll start to study/work on this feature, of course I' ll open a review to find the better solution.