Details

Reviewers

tabthorpe
hrs
rene
delphij
lwhsu
kwm
bcr

Group Reviewers

doceng

Commits

rD47306: Add PO translation support.

Summary

Give the doc toolchain a way to create PO files, and generate translated XML files from them.

Translators generate PO files with 'make po', then enter translated strings with poedit or the like. After enough strings have been translated to make a useful document, 'make tran' is used to generate the translated DocBook XML file from the PO file.

Translators commit their PO files, and the generated XML is only used temporarily for creating the final output format.

devel/gettext-tools and textproc/itstool are required. These will be added as dependencies to textproc/docproj.

Diff Detail

Lint

Lint Skipped

Unit

Tests Skipped

Event Timeline

wblock updated this revision to Diff 6559.Jun 29 2015, 9:10 PM

wblock retitled this revision from to Doc toolchain PO translation support.

wblock updated this object.

wblock edited the test plan for this revision. (Show Details)

wblock added reviewers: doceng, bcr, tabthorpe, rene, kwm, feld.

wblock set the repository for this revision to rD FreeBSD doc repository - subversion.

Herald added subscribers: emaste, bdrewery. · View Herald TranscriptJun 29 2015, 9:10 PM

These are quick comments and probably I will make more concrete suggestions later. I feel, however, what is the best depends on the expected translation workflow. Are po files created and then left without being committed? If possible, could you please summarize a typical procedure which a translator will follow into the SUMMARY section?

share/mk/doc.docbook.mk
275	Please do not use $PWD and use $DOC_PREFIX. Current doc.*.mk are also broken in this regard, but current directory should be carefully handled because make(1) uses OBJDIR.
283	Please consider to define a proper makefile target for a po file. Once we have such a target, make(1) can detect an old .po file compared to the xml file and kick a target for merge.

wblock updated this object.Jun 30 2015, 12:31 AM

Add back ${DOC}.po target. Actually had that before and removed it in a flash of misinspiration.

DTD modifications for <buildtarget> split off to https://reviews.freebsd.org/D2958.

wblock updated this object.Jun 30 2015, 2:37 PM

wblock added reviewers: lwhsu, delphij.Jul 2 2015, 12:16 AM

I think we need to consider a few cases when doing the msgmerge. I can think of the following:

String is not present in the translations's PO file. This could be due to the following reasons in the en_US reference:

a) the string was added (new sentence/paragraph). This is probably the default case and is handled well by msgmerge.
b) the string was changed (can msgmerge detect and map these in the translation PO file accordingly?). What happens when a string is split up into two (or more) sentences? Are new IDs being generated for both strings or is the original one preserved and an additional one for the second sentence created?
c) the string was deleted (and so should be in the PO file to be removed in the translation as well)

I remember Kris Moore had to write a script to delete all strings in their Pootle database each time they imported a new en_US version for translators to avoid keeping old strings around.

These cases might be a bit to much for this review to cover. I think the effort should be made in trying out gettext tools and when we do, we should determine how to best deal with these cases. I hope this can be automated to cover these cases and let translators focus on the actual translation process, not having to ask "Is this string actually relevant anymore?".

Vastly reworked. PWD is not used. The normalized file is generated in the non-English directory where the user is working. Both itstool or po4a are supported, with po4a the default. Entities are not expanded. xmlns:xlink attributes are filtered out (crudely, that needs work), so that translators do not see them.

wblock updated this object.Jul 8 2015, 11:22 PM

wblock updated this object.

PWD is not needed with the latest version.

Remove English parent doc dependencies that could not be satisfied. This was obscured in testing by previous copies of those files.

Also expand on notes on use of xmllint to remove redundant xmlns:xlink attributes. That is fragile and needs to be replaced with a better method.

Add options to keep entities or expand them. Add a simple filter to remove redundant attributes so translators do not have to deal with them.

feld resigned from this revision.Jul 12 2015, 10:23 PM

feld removed a reviewer: feld.

I tried po4a-gettextize for 8-bit encodings such as EUC-JP and GB2312, but it seems {article,book}.translate.xml contains localized entity references expanded in wrong encoding. ${XMLLINT} for localized directory includes XML_CATALOG_FILES for localized versions, not for en_US.ISO8859-1. So the rendering results will have problem with expansion of entity reference (i.e. localized entities are used instead of English versions). This has to be fixed.

This revision now requires changes to proceed.Jul 13 2015, 7:56 PM

Changes for doc.docbook.mk is not commit ready because most of rules are broken in terms of their dependency chain. I tried to fix them and put a diff to the following URL:

http://people.allbsd.org/~hrs/FreeBSD/doc.docbook.mk.20150714-1.diff

This still needs more improvement but should be much better than the current one.

hrs added inline comments.Jul 13 2015, 8:59 PM

share/mk/doc.docbook.mk
296	Please do not edit XML file by using sed or something which does not understand document structure. It can break DTD conformance and generally is not allowed in XML processing the doc tree. Eliminating redundant namespace attributes can be done --nsclean option in xmllint, for example.
307	Why can we assume UTF-8 here?

Remove the option to preserve entities. More than one source has said that is a mistake for translators.
Remove the option to use po4a, itstool works as long as there is no need to preserve entities.
Use ISO_LANG as the name of the two-character language code.

Rework patch based on hrs@'s suggestions.

Rework.

This new diff addresses the worst problems. The part I don't like is that I had to wrap the translation targets in a .if to keep them from being built when non-translation FORMATS like html are being built. There might be a better way to do that.

If this diff is not close to being acceptable, please let me know and I'll just rewrite these functions as scripts to go in the Tools directory. Trying to get this Makefile to do the job is not worth spending much more time. We need to be getting translation teams involved in testing the translation functions. Thanks!

Locale-style language codes (xx_YY) are used, and have been tested with the latest version of poedit (thanks to rodrigo@ for the updated port). The POSET_CMD script now correctly sets the language type in the header. Makefile changes have been reduced and simplified.

Closed by commit rD47306: Add PO translation support. (authored by wblock). · Explain WhyAug 24 2015, 3:54 PM

This revision was automatically updated to reflect the committed changes.

Diff 6795

View Options

share/mk/doc.commands.mk

Show All 26 Lines

.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT .\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT

.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY .\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY

.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF .\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF

.\" SUCH DAMAGE. .\" SUCH DAMAGE.

.\" .\"

.\" @(#)time.3 8.1 (Berkeley) 6/4/93 .\" @(#)time.3 8.1 (Berkeley) 6/4/93

.\" $FreeBSD$ .\" $FreeBSD$

.\" .\"

.Dd March 4, 2022 .Dd April 2, 2022

.Dt TIME 3 .Dt TIME 3

.Os .Os

.Sh NAME .Sh NAME

.Nm time .Nm time

.Nd get time of day .Nd get time of day

.Sh LIBRARY .Sh LIBRARY

.Lb libc .Lb libc

.Sh SYNOPSIS .Sh SYNOPSIS

.In time.h .In time.h

.Ft time_t .Ft time_t

.Fn time "time_t *tloc" .Fn time "time_t *tloc"

.Sh DESCRIPTION .Sh DESCRIPTION

The The

.Fn time .Fn time

function function

returns the value of time in seconds since 0 hours, 0 minutes, returns the value of time in seconds since 0 hours, 0 minutes,

0 seconds, January 1, 1970, Coordinated Universal Time. 0 seconds, January 1, 1970, Coordinated Universal Timei (UTC).

impUnsubmitted

Done

extra 'i' here.

imp: extra 'i' here.

If an error occurs, If an error occurs,

.Fn time .Fn time

returns the value returns the value

.Po Vt time_t Pc Ns \-1 . .Po Vt time_t Pc Ns \-1 .

.Pp .Pp

The return value is also stored in The return value is also stored in

.No \&* Ns Va tloc , .No \&* Ns Va tloc ,

provided that provided that

.Va tloc .Va tloc

is non-null. is non-null.

.Sh ERRORS .Sh ERRORS

The The

.Fn time .Fn time

function may fail for any of the reasons described in function may fail for any of the reasons described in

.Xr clock_gettime 2 . .Xr clock_gettime 2 .

.Sh SEE ALSO .Sh SEE ALSO

.Xr clock_gettime 2 , .Xr clock_gettime 2 ,

.Xr gettimeofday 2 , .Xr gettimeofday 2 ,

.Xr ctime 3 .Xr ctime 3

.Sh STANDARDS .Sh STANDARDS

The The

.Nm .Nm

function conforms to function conforms to

.St -p1003.1-2001 . .St -p1003.1-2008 .

.Sh HISTORY .Sh HISTORY

A A

.Fn time .Fn time

function appeared in system call first appeared in

.At v6 . .At v1 .

That version counted time in sixtieths of a second with a 32-bit return value,

pauamma_gundo.comUnsubmitted

Not Done

.At v1 .

- Through the

- .At v3 ,

+ Until and including.At v3 ,

pauamma_gundo.com:

ensuring an integer overflow crisis every 2.26 years.

.At v6

impUnsubmitted

Done

This change to seconds happened in the 4th edition, not the 6th edition.
The 3rd edition returned time in 1/HZ (1/60) since Jan 1, 1972.
The 1st and 2nd Edition return since Jan 1, 1971

imp: This change to seconds happened in the 4th edition, not the 6th edition. The 3rd edition…

impUnsubmitted

Done

The new epoch in the 3rd edition presents challenges to reconstructing dates from tapes...
And the headaches with it motivated the change in the 4th edition...

imp: The new epoch in the 3rd edition presents challenges to reconstructing dates from tapes... And…

the granularity of the return value was reduced to whole seconds,

delaying the aforementioned crisis until 2038.

.Bx 4.1c

impUnsubmitted

Done

This is close, but not quite right.

v7 introduced the ftime() system call which returned time at a millisecond level, though retained the gtime() system call (exposed as time() in userland). time() could have been implemented as a wrapper around ftime, but that wasn't done.

4.1c implemented a higher-precision time function (gettimeofday) to replace ftime() and reimplemented time() in terms of that.

imp: This is close, but not quite right. v7 introduced the ftime() system call which returned time…

jillesUnsubmitted

Done

And a more recent change: git commit 358ed16f750576f266ebf86ea1ee56204d71bb52 (in 9.0 and 8.1) changed the implementation of time() to use clock_gettime(CLOCK_SECOND) instead of gettimeofday(), for performance reasons.

The practical effect is that applications should use time() if a resolution of 1 second is good enough for them; time() is standard but CLOCK_SECOND is not.

jilles: And a more recent change: git commit 358ed16f750576f266ebf86ea1ee56204d71bb52 (in 9.0 and 8.1)…

the function was moved out of the kernel into the C standard library and

impUnsubmitted

Done

I'd make this briefer:
"The time system call first appeared in 1st edition. Through the 3rd edition, it returned 60 Hz ticks since an epoch that changed occasionally because it was a 32-bit value that overflowed in a little over 2 years."
and maybe leave out the tape date information. It's a bit more verbose than we normally do. And tape dates varied because they were both the 1971 and 1972 epoch, and you never knew which type of tape you had.

imp: I'd make this briefer: "The time system call first appeared in 1st edition. Through the 3rd…

reimplemented with

.Xr gettimeofday 2 .

.Sh BUGS .Sh BUGS

Neither Neither

.St -isoC-99 .St -isoC-99

nor nor

pauamma_gundo.comUnsubmitted

Not Done

system call, which returned time at a millisecond level,

- though retained the

+ though it retained the

.Fn gtime

pauamma_gundo.com:

.St -p1003.1-2001 .St -p1003.1-2001

requires requires

impUnsubmitted

Not Done

i386 is the only platform left with a 32-bit time_t, and the only one affected by the crisis. not entirely sure how to work that into this, however.

imp: i386 is the only platform left with a 32-bit time_t, and the only one affected by the crisis.

gbeAuthorUnsubmitted

Done

I left this out for the moment, since I am not sure what about arm6, or is i386 the only left 32bit architecture?

gbe: I left this out for the moment, since I am not sure what about arm6, or is i386 the only left…

pauamma_gundo.comUnsubmitted

Not Done

I left this out for the moment, since I am not sure what about arm6, or is i386 the only left 32bit architecture?

https://cgit.freebsd.org/src/tree/sys/arm/include/_types.h (which I think is arm[4-6]) has

typedef	__int64_t	__time_t;		/* time()... */

pauamma_gundo.com: > I left this out for the moment, since I am not sure what about arm6, or is i386 the only left…

.Fn time .Fn time

to set to set

.Va errno .Va errno

on failure; thus, it is impossible for an application to distinguish on failure; thus, it is impossible for an application to distinguish

the valid time value \-1 (representing the last UTC second of 1969) the valid time value \-1 (representing the last UTC second of 1969)

from the error return value. from the error return value.

.Pp .Pp

Systems conforming to earlier versions of the C and Systems conforming to earlier versions of the C and

.Tn POSIX .Tn POSIX

standards (including older versions of standards (including older versions of

.Fx ) .Fx )

did not set did not set

.No \&* Ns Va tloc .No \&* Ns Va tloc

in the error case. in the error case.

pauamma_gundo.comUnsubmitted

Not Done

.Fn time

- in terms of that.

+ using that.

.Pp

Since

pauamma_gundo.com:

View Options

share/mk/doc.docbook.mk

Show All 26 Lines

.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT .\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT

.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY .\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY

.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF .\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF

.\" SUCH DAMAGE. .\" SUCH DAMAGE.

.\" .\"

.\" @(#)time.3 8.1 (Berkeley) 6/4/93 .\" @(#)time.3 8.1 (Berkeley) 6/4/93

.\" $FreeBSD$ .\" $FreeBSD$

.\" .\"

.Dd March 4, 2022 .Dd April 2, 2022

.Dt TIME 3 .Dt TIME 3

.Os .Os

.Sh NAME .Sh NAME

.Nm time .Nm time

.Nd get time of day .Nd get time of day

.Sh LIBRARY .Sh LIBRARY

.Lb libc .Lb libc

.Sh SYNOPSIS .Sh SYNOPSIS

.In time.h .In time.h

.Ft time_t .Ft time_t

.Fn time "time_t *tloc" .Fn time "time_t *tloc"

.Sh DESCRIPTION .Sh DESCRIPTION

The The

.Fn time .Fn time

function function

returns the value of time in seconds since 0 hours, 0 minutes, returns the value of time in seconds since 0 hours, 0 minutes,

0 seconds, January 1, 1970, Coordinated Universal Time. 0 seconds, January 1, 1970, Coordinated Universal Timei (UTC).

impUnsubmitted

Done

extra 'i' here.

imp: extra 'i' here.

If an error occurs, If an error occurs,

.Fn time .Fn time

returns the value returns the value

.Po Vt time_t Pc Ns \-1 . .Po Vt time_t Pc Ns \-1 .

.Pp .Pp

The return value is also stored in The return value is also stored in

.No \&* Ns Va tloc , .No \&* Ns Va tloc ,

provided that provided that

.Va tloc .Va tloc

is non-null. is non-null.

.Sh ERRORS .Sh ERRORS

The The

.Fn time .Fn time

function may fail for any of the reasons described in function may fail for any of the reasons described in

.Xr clock_gettime 2 . .Xr clock_gettime 2 .

.Sh SEE ALSO .Sh SEE ALSO

.Xr clock_gettime 2 , .Xr clock_gettime 2 ,

.Xr gettimeofday 2 , .Xr gettimeofday 2 ,

.Xr ctime 3 .Xr ctime 3

.Sh STANDARDS .Sh STANDARDS

The The

.Nm .Nm

function conforms to function conforms to

.St -p1003.1-2001 . .St -p1003.1-2008 .

.Sh HISTORY .Sh HISTORY

A A

.Fn time .Fn time

function appeared in system call first appeared in

.At v6 . .At v1 .

That version counted time in sixtieths of a second with a 32-bit return value,

pauamma_gundo.comUnsubmitted

Not Done

.At v1 .

- Through the

- .At v3 ,

+ Until and including.At v3 ,

pauamma_gundo.com:

ensuring an integer overflow crisis every 2.26 years.

.At v6

impUnsubmitted

Done

This change to seconds happened in the 4th edition, not the 6th edition.
The 3rd edition returned time in 1/HZ (1/60) since Jan 1, 1972.
The 1st and 2nd Edition return since Jan 1, 1971

imp: This change to seconds happened in the 4th edition, not the 6th edition. The 3rd edition…

impUnsubmitted

Done

The new epoch in the 3rd edition presents challenges to reconstructing dates from tapes...
And the headaches with it motivated the change in the 4th edition...

imp: The new epoch in the 3rd edition presents challenges to reconstructing dates from tapes... And…

the granularity of the return value was reduced to whole seconds,

delaying the aforementioned crisis until 2038.

.Bx 4.1c

impUnsubmitted

Done

This is close, but not quite right.

4.1c implemented a higher-precision time function (gettimeofday) to replace ftime() and reimplemented time() in terms of that.

imp: This is close, but not quite right. v7 introduced the ftime() system call which returned time…

jillesUnsubmitted

Done

The practical effect is that applications should use time() if a resolution of 1 second is good enough for them; time() is standard but CLOCK_SECOND is not.

jilles: And a more recent change: git commit 358ed16f750576f266ebf86ea1ee56204d71bb52 (in 9.0 and 8.1)…

the function was moved out of the kernel into the C standard library and

impUnsubmitted

Done

imp: I'd make this briefer: "The time system call first appeared in 1st edition. Through the 3rd…

reimplemented with

.Xr gettimeofday 2 .

.Sh BUGS .Sh BUGS

Neither Neither

.St -isoC-99 .St -isoC-99

nor nor

pauamma_gundo.comUnsubmitted

Not Done

system call, which returned time at a millisecond level,

- though retained the

+ though it retained the

.Fn gtime

pauamma_gundo.com:

.St -p1003.1-2001 .St -p1003.1-2001

requires requires

impUnsubmitted

Not Done

i386 is the only platform left with a 32-bit time_t, and the only one affected by the crisis. not entirely sure how to work that into this, however.

imp: i386 is the only platform left with a 32-bit time_t, and the only one affected by the crisis.

gbeAuthorUnsubmitted

Done

I left this out for the moment, since I am not sure what about arm6, or is i386 the only left 32bit architecture?

gbe: I left this out for the moment, since I am not sure what about arm6, or is i386 the only left…

pauamma_gundo.comUnsubmitted

Not Done

I left this out for the moment, since I am not sure what about arm6, or is i386 the only left 32bit architecture?

https://cgit.freebsd.org/src/tree/sys/arm/include/_types.h (which I think is arm[4-6]) has

typedef	__int64_t	__time_t;		/* time()... */

pauamma_gundo.com: > I left this out for the moment, since I am not sure what about arm6, or is i386 the only left…

.Fn time .Fn time

to set to set

.Va errno .Va errno

on failure; thus, it is impossible for an application to distinguish on failure; thus, it is impossible for an application to distinguish

the valid time value \-1 (representing the last UTC second of 1969) the valid time value \-1 (representing the last UTC second of 1969)

from the error return value. from the error return value.

.Pp .Pp

Systems conforming to earlier versions of the C and Systems conforming to earlier versions of the C and

.Tn POSIX .Tn POSIX

standards (including older versions of standards (including older versions of

.Fx ) .Fx )

did not set did not set

.No \&* Ns Va tloc .No \&* Ns Va tloc

in the error case. in the error case.

pauamma_gundo.comUnsubmitted

Not Done

.Fn time

- in terms of that.

+ using that.

.Pp

Since

pauamma_gundo.com:

Doc toolchain PO translation support
ClosedPublic
Actions

Details

Diff Detail

Event Timeline

Revision Contents
Changeset List

Diff 6795

share/mk/doc.commands.mk

share/mk/doc.docbook.mk

Doc toolchain PO translation supportClosedPublicActions

Details

Diff Detail

Event Timeline

Revision ContentsChangeset List

Diff 6795

share/mk/doc.commands.mk

share/mk/doc.docbook.mk

Doc toolchain PO translation support
ClosedPublic
Actions

Revision Contents
Changeset List