Page MenuHomeFreeBSD

vt: Add support for console accessibility using sysctls.
Needs ReviewPublic

Authored by hselasky on Jul 8 2022, 10:50 AM.

Details

Summary

This allows the contents of the current FreeBSD VT console to be accessed
by vtspeakd(1) for example. The following sysctls are provided:

kern.vt.accessibility.feed
kern.vt.accessibility.text_utf8
kern.vt.accessibility.col
kern.vt.accessibility.row
kern.vt.accessibility.lines

MFC after: 1 week
Sponsored by: NVIDIA Networking

Test Plan
  1. Patch, build and install new kernel.
  2. Apply and install beep(1) patches, See https://reviews.freebsd.org/D35772
  3. Apply and install vtspeakd(1) patches, See https://reviews.freebsd.org/D35776

Then reboot and run:
vtspeakd -sc

For example.

Diff Detail

Repository
rG FreeBSD src repository
Lint
Lint Skipped
Unit
Unit Tests Skipped

Event Timeline

I'm not at all sure this is the right place for this...

sys/dev/vt/vt_accessibility.c
90

please construct the messages using sbuf.

This is my first attempt implementing something like this. I needed to extend the term_char_t type to 64-bits to make room for a bit which keeps track of what characters have been printed or not.

When storing new characters this bit gets reset to zero.

sys/dev/vt/vt_accessibility.c
90

Sure, but devd has its limits on the message size including headers (1Kbyte I think)

pauamma added a subscriber: pauamma.

Niiiiiiiiiice. (Added Accessibility as a group reviewer.)

Are you looking for feature testing at this time?

All feedback is welcome!

Some comments from a friend of mine:

Q: We translate all the special characters (such as '.' and '/') into "period" and "divide". This means those characters will always be spoken because espeak sees it as a word. A sysctl to disable that will be nice, but not required.
A: The problem is shell escaping - need to be careful.

Q: Is there any way to get the cursor position (<line number>:<column position>)?
A: Probably yes.

Q: NVDA and others use CAPS LOCK+{u,i,o} to move the cursor up one line, say current line or move cursor down.
A: Not sure if they have a patent on that.

Are you looking for feature testing at this time?

Yes, please test as much as you can.

Are you looking for feature testing at this time?

Yes, please test as much as you can.

I don't use a screenreader myself, but I'm going to ask for testers on freebsd-accessibility@. Before I do, can you confirm my understanding of what's needed, per outline below?

Overall, this is fairly involved, and while I'm willing to help people through this, I may reach my limit fast. How much are you willing and able to help (me help) testers?

All feedback is welcome!

Two things:

  • I would make the devd conf filename more specific, perhaps screenreading.espeak.conf or something similar.
  • Likewise, maybe kern.vt.screenreading.enable for the sysctl name.

Hi,

You just need to patch /usr/src and build a new kernel.

Then you run:

cp -i /usr/src/sbin/devd/accessibility.conf /etc/devd/

Then you reboot and run:

pkg install espeak

Then:

sysctl kern.vt.accessibility.enable=1

Beware that the inner APIs of this feature may change.

I've gotten some feedback and will probably export all text as a sysctl nodes, to avoid shell escaping, either as raw UTF-8 or spoken text (no raw characters).

--HPS

I'd really rather you do all this not with devd.
It puts too many strings through it: devd wasn't really designed for a high-throughput introspection interface. You are already introducing weird interfaces to cope with flow control and I'd rather not have more-critical messages delayed waiting for all the console output to play out (I also worry a weird error starving devd as well).
There's also too much string code in the kernel with this..
Is there some reason you can't open use the snp device to snoop console output and pass that through to easyspeak w/o the need to "stand on your head" to get cope with devd's weird string quoting quirks. You can get the characters directly.
And best of all, you don't need a new kernel for it, but you may have to load snp.ko.

Hi Warner,

In D35754#811558, @imp wrote:

I'd really rather you do all this not with devd.

I agree. Consider this a proof of concept.
When I have some more time I will make a sysctl only API skipping the devd part.

It puts too many strings through it: devd wasn't really designed for a high-throughput introspection interface. You are already introducing weird interfaces to cope with flow control and I'd rather not have more-critical messages delayed waiting for all the console output to play out (I also worry a weird error starving devd as well).

I know exactly what you mean, understood.

There's also too much string code in the kernel with this..
Is there some reason you can't open use the snp device to snoop console output and pass that through to easyspeak w/o the need to "stand on your head" to get cope with devd's weird string quoting quirks. You can get the characters directly.
And best of all, you don't need a new kernel for it, but you may have to load snp.ko.

I didn't look that much into snp yet, but I guess it doesn't understand switching the console ALT+F12345 ... and it doesn't understand where you are on the screen or if you are scrolling the history.

Also if you simply copy the terminal stream there might be too much to read. For example you open some manual page and it starts reading, then you change your mind and close the manual page and clear the screen using CTRL+L . Then the espeak should stop after completing its last one or two spoken lines and not everything. How is SNP supposed to know that?

Adding a 32nd bit the the term_char_t was an easy solution on the other hand to keep track of dirty text areas.

I've gotten some feedback and will probably export all text as a sysctl nodes, to avoid shell escaping, either as raw UTF-8 or spoken text (no raw characters).

I'm going to hold back the accessibility@ CFT until the design settles, so there's just one installation procedure.

hselasky retitled this revision from vt: Add support for console accessibility using devd infrastructure. to vt: Add support for console accessibility using sysctls..
hselasky edited the summary of this revision. (Show Details)
hselasky edited the test plan for this revision. (Show Details)

@imp: No more devd dependency. Trying to keep this stuff as simple as possible.

Better. Might consider having a char device to publish this, but given you are looking at different views maybe not...

I'd also still consider pushing the 'interpretation of the text' to a filter in userland vs having it in the kernel. While it makes the userland side simpler, the kernel is complicated with stuff that might not be required to be here.... Just a thought.

@imp: Moving the chatty stuff to userspace is a possibility. Then the kernel patch for vt(4) will be quite small and focused.

hselasky edited the summary of this revision. (Show Details)

Moved chatty to userspace as suggested by imp@

Scan for first non-word separator when outputting line.

Fix multi-line support. Wrap some long lines.