Bug in TTXN (belongs to the AFDKO)

Arno Enslin's picture

It’s from Minion Pro, dumped with TTXN:

lookup frac_DFLT_dflt_2 {
sub [eight.numerator five.numerator four.numerator nine.numerator one.numerator
seven.numerator six.numerator three.numerator two.numerator
zero.numerator]
[space uni00A0]' ;
C sub space by space.frac;
C sub uni00A0 by uni00A0.frac;
} frac_DFLT_dflt_2;

"C sub"? (A working) TTXN would be cool, because when you have to fix a bug in the features of a font, you cannot really trust in FontLab, because it interprets the GSUB table likewise. After removing the GSUB tags, it should be possible to use the dumped table as feature file with makeotf (belongs to the AFDKO).

TTXN does not seem to be able to store the classes, but puts the character lists directly into the features. That’s not a bug, but should be fixed likewise.

And TTXN does sometimes not include, by which character/class something shall be replaced (End of code: "zero.numerator]' ;"):

lookup frac_DFLT_dflt_5 {
sub [cent.denominator comma.denominator dollar.denominator eight.denominator
five.denominator fiveeighths four.denominator fraction hyphen.denominator
nine.denominator one.denominator oneeighth onehalf onequarter onethird
parenleft.denominator parenright.denominator period.denominator
seven.denominator seveneighths six.denominator slash.frac
three.denominator threeeighths threequarters two.denominator twothirds
uni2215 zero.denominator]
[cent.numerator comma.numerator dollar.numerator eight.numerator
five.numerator four.numerator hyphen.numerator nine.numerator
one.numerator parenleft.numerator parenright.numerator period.numerator
seven.numerator six.numerator three.numerator two.numerator
zero.numerator]' ;

andreas's picture

Maybe you should add it to the bug tracker at SourceForge.

http://sourceforge.net/tracker/?group_id=29196&atid=395372

Arno Enslin's picture

@ Andreas

It’s not included in FontTools. TTXN is different from TTX. The options are the same, but it is a separate Python file. It seems to be developed by Adobe.

Arno Enslin's picture

I don’t know anything about Python, but I assume, that the problem with the "C sub …" is located at the following lines in TTXN.py:

ChainINDENT = INDENT + " C "

May it be, that not the character "C" was meant, but a control character?

And the other problem may be located here, although there are more lines like that:

rule = "sub %s %s' %s by " % (backTxt, inputTxt, lookAheadTxt)

%s seems to be a placeholder for the variables (?) in the parentheses.

In contradiction to other lines, in which anything is substituted there is no placeholder after the word by.

I don’t want to try to correct anything in code, that I don’t understand. But I assume, it is not so difficult to fix these bugs. But maybe there are more bugs, which I have not yet discovered.

-----------

I wish, more of you would use the AFDKO instead of FontLab for generating the features. I mean, I am amateur, but there seem to be many professionals on the forum, that use FontLab for the OT features. But the AFDKO is really cool and as more people find those bugs, as easier it is, to improve the AFDKO.

Michel Boyer's picture

You can also generate the feature file with FontForge. PC users can use fontforge-mingw, a self contained binary (cf http://www.typophile.com/node/71038). Mac users can install FontForge with Fink (binary) or Macports (requires compiling).

On the mac I dot it with a script; doing the same on the PC is more tricky and I won't have access to that machine before wednesday and I tried just once, to check. I'll thus just say how to do it with the graphical interface. After opening the font.

You click "Element > Font Info"

You then click "Lookups", right click on a feature and click on "Save Lookup..." to save the feature file for that lookup. You click on "Save Feature File..." to save the full feature file of that font.

Arno Enslin's picture

Thanks, Michel!

A few months ago, I already wanted to install FontForge. It is not yet installed. What is true for the AFDKO, is also true for FontForge. More people should try it. I will do.

Arno Enslin's picture

I think, I could make out, where the problem is, if I do the follwing:

When I replace this

rule = "sub %s %s' %s by " % (backTxt, inputTxt, lookAheadTxt)

by this

rule = "sub %s backTxt %s' inputTxt %s lookAheadTxt by " % (backTxt, inputTxt, lookAheadTxt)

I probably can make out, which rule in the Python code is responsible for which decompiled sub rule.

Michel Boyer's picture

Thanks, Michel!

Welcome. And I should try Volt. I would have if it had been available for OS X (or X Windows on Unix). For me, Microsoft Windows is the last resort.

Michel Boyer's picture

Arno,

As was noted in the Calibri kerning thread and in John Hudson's comment in the Checking Kerning Pairs in fonts thread, ttx does not output all a kern subtable when it overflows.

Here is a patch for the file _k_e_r_n.py.

34c34,35
< version, length = struct.unpack(">HH", data[:4])
---
> version, length, coverage, nPairs = struct.unpack(">HHHH", data[:8])
> if ((self.version == 0.0) & (length < 14+6*nPairs)): length = 14+6*nPairs # Undocumented hack

If I apply it, I get all kerning pairs of Calibri's kern table with ttx and ttxn.

I have no time to check if this is ok or not for all possible cases. If you want to have a look at that, that would be great.

Arno Enslin's picture

@ Michel

34c34, 35

Does this mean, that I shall replace line 34 by the other line and add a new line above the line, that is actually line 35?

This

version, length = struct.unpack(">HH", data[:4])
length = int(length)

shall be replaced by?:

version, length, coverage, nPairs = struct.unpack(">HHHH", data[:8])
if ((self.version == 0.0) & (length < 14+6*nPairs)): length = 14+6*nPairs # Undocumented hack
length = int(length)

I have no time to check if this is ok or not for all possible cases. If you want to have a look at that, that would be great.

I don’t know, how to check all possible cases. The Calibri kerning thread looks complicated. I feel, that I miss theory about these overfloating (?) tables and sub tables. I could compare dumps of some system fonts with the original and the unpached kern.py. Most of the new MS systems fonts are not installed here (except from Consolas), but I think, I have them somewhere.

Edited

I replaced it now in the way described in this post, but I get an error. Obviously I did replace the wrong code. I would try it again, if you explain a bit more precisely, what shall be replaced by what.

And I found the Windows 7 system fonts. Calibri 5.62 is the latest version, that I have here.

By the way, since this is another TTX bug, we maybe should ask a moderator to move the discussion.

Arno Enslin's picture

Okay, I tried it again and now it seems to work.

I replaced

version, length = struct.unpack(">HH", data[:4])
length = int(length)

by

version, length, coverage, nPairs = struct.unpack(">HHHH", data[:8])
length = int(length)
if ((self.version == 0.0) & (length < 14+6*nPairs)): length = 14+6*nPairs # Undocumented hack

and get 26706 pairs.

Did I replace it correctly this time?

calibri.ttx: 26706
cambriab.ttx: 31738
cambriai.ttx: 36452
cambriaz.ttx: 35714
constan.ttx: 36740
constanb.ttx: 36573
constani.ttx: 38688
constanz.ttx: 38417

But this was probably not the check, for which you have asked (all possible cases).

Edited

Something very odd has just happened here: I had decompiled the kern tables of the fonts above only. And then I have merged them back (likewise with TTX). The checksums are the same as the ones of the original fonts. And the digital signatures are valid! That’s very odd!

Michel Boyer's picture

I was indeed replacing line 34 with two lines, 34 and 35, with the same indentation as old line 34, i.e. the if of line 35 is aligned with version of line 34 (which remains where it was). Here is how it looked on the screen with the proper indentation.


I have an old version of Calibri with a different number of pairs. The one you get is the one John Hudson says there is. You thus got the right number. The funny thing is indeed that if you run ttx on the resulting file, it gives you back the big kerning table. You can see it with DTL OTMaster Light.

What worries me is the specs, that I don't want to read, and I did not check either what struct.unpack actually does.

Michel Boyer's picture

Sorry, it is not "overfloat" but "overflow". According to http://www.microsoft.com/typography/otspec/kern.htm, the kern table starts with two fields that are USHORT, the first containing a version number, the second the number of subtables. USHORT is a 16 bit unsigned short interger, and thus may be from 0 to 2**16-1, i.e. from 0 to 65535. Then follow the subtables, each starting with the fields version, length and coverage, which are also USHORT. The definition of the field "length" is "Length of the subtable, in bytes (including this header)." That means that the value of the length field may be from 0 to 65535, and thus the subtable may take at most 65535 bytes, else it "overflows". Of course, a subtable with 26706 kerning pairs, each taking 6 bytes, gives much more than the alloted 65535 bytes. It seems someone has done a "theological reinterpretation" of the specs. I'd like to see it in some "official" written text.

Added: if in the spec they replaced "Length of the subtable, in bytes (including this header)." with "Length of the subtable modulo 65536, in bytes ...", with some explanations, I would like it better.

Arno Enslin's picture

I was indeed replacing line 34 with two lines, 34 and 35, with the same indentation as old line 34, i.e. the if of line 35 is aligned with version of line 34 (which remains where it was). Here is how it looked on the screen with the proper indentation.

That’s odd, because it did not work here. I will check that again.

Arno Enslin's picture

@ Michel

Quoting myself: I will check that again.

I obviously made a mistake in my first attempt yesterday. The replacement, as it was explained by you, is working.

Arno Enslin's picture

Back to the bug in TTXN

There is a bug in the lines above line 1235. At least two of the variables backTxt, inputTxt, lookAheadTxt are not correctly defined. And maybe the word "by" is missing.

And the other bug is in in line 111 (ChainINDENT = INDENT + " C ").

I am not competent to fix that. And it is so obviously, that there is a problem with the part of TTXN, which is responsible for the features, that I assume, there are more bugs.

Read Roberts's picture

Hi Arno;

I am not about to fix this particular problem, because ttxn is not meant for making a compile-able font - it is meant only to help compare different versions of basically the same font, and has output features that work against what you want. (see below).

You have lots of company in the desire for a tool that will decompile GPOS and GSUB tables to something directly usable. In the long run, I do plan to add this ability to 'spot'. In the short term - the next several months - the only path to what you want with the FDK tools is use the "spot -t GPOS=7" or "spot -t GSUB=7" commands, and then expect to do a lot of manual editing.

What 'ttxn' is about is producing a text dump of a font, that makes it easy to compare two different versions of a font. The idea is that you use ttxn to do a text dump of each of the two versions, and you use a text comparison tool, like the "Search for Differences.." in BBedit.It is useful only for relatively small differences, such as you get with most bug fixes. If you have just added a few glyphs, or corrected a few paths, or tweaked some kern values, or fixed some GPOS and GSUB differences, then ttxn is great for confirming that what you think you fixed did in fact change, and - just as important! - that nothing else changed.

What makes ttxn more interesting than other tools which make text dump of fonts is that ttxn tries to remove all the things that make fonts look different inside, but have no functional consequences. For example, glyph order is normalized to alphabetic in all places where you see a list of glyphs. In the feature file, I tried to flatten all the different ways in which you can express a particular functional idea to the simplest common denominator. So, lookup order is important, but language system order or offset or whether coverage tables are shared aren't.

In the case that you report, the "by " is missing from the end of the contextual rule, and is instead replaced by a series of simple substitution rules, prefixed by "C", which comprise the set of simple substitutions which are applied to the contextual target string.The first 'C' rule is applied to the first glyph in the input string, the second to the second glyph in the input string, and so on. I did it this way just as a matter of expediency - it was easier, given the several different paths by which you end up with the secondary lookup tables - and I wasn't thinking about making a compile-able output.

All that said, feel free to re-purpose the tool as you want - just give the variant a different name. If you are in doubt about just what is going to happen as a result of a given edit, just try and see. You can always check that you are getting what you want out of your fonts.

Arno Enslin's picture

@ Read Roberts

Thank you!

All that said, feel free to re-purpose the tool as you want - just give the variant a different name.

This would require, that I understand Python. I am probably not going to try re-purposing ttxn.

What’s the best place for reporting bugs or proposing new functions of the AFDKO?

When will the next version of the AFDKO be released?

Arno Enslin's picture

@ Michel

I have exported the features of a font with Fontforge-mingw now, but I was searching for a tool, that rebuilds the feature file as close as possible to the original file with the syntax as it is required by the AFDKO and with classes outside the features – the same output, as you get it, if you export the features with FontLab, but without the FontLab bugs.

Michel Boyer's picture

I have never "massaged" feature files exported by FontForge and I don't use FontLab (all I have is a demo version). As for using FontForge's fea output in AFDKO, I think that is what John Hudson wanted to do after this post: http://typophile.com/node/69658. From a subsequent message, I understand John had a bug in some .fea file and a little afterwards was posted a message concerning a bug that had been corrected in makeotf http://typophile.com/node/70019. My guess is that the match is not perfect yet though these posts may not be related...

blokland's picture

Arno: [...] a tool, that rebuilds the feature file as close as possible to the original file with the syntax as it is required by the AFDKO [...]

Perhaps the Export... function in OTM (Light) is useful?

John Hudson's picture

Michel, to my my knowledge those two posts are not related.

The feature syntax extracted from the FontForge source turned out to have a number of bugs (identified for me by Karsten Leucke), which caused makeotf to fail. When these were fixed, makeotf worked fine. The question I was left with was whether FontForge would compile the feature code with these syntax errors, but since I didn't need an answer to that question to complete my project, I didn't pursue it. The feature code was part of an open source FontForge file, so unless there were a disconnect between that source and the shipping version of that font, perhaps FontForge is more forgiving of syntax errors than makeotf.

Arno Enslin's picture

@ Frank

OTMaster is useful. But for me it is easier to fix the bugs in the feature file exported by FontLab. FontLab rebuilds the classes, OTMaster doesn’t. I loose the overview in feature files, that were exported by OTMaster. The problem with FontLab is, that the (de)compiler is old. (Which version of the AFDKO decompiler is built in FontLab? It is the AFDKO decompiler, isn’t it?)

Khaled Hosny's picture

@ John Hudson

It is not clear what version of FontForge were you using, but there were indeed a number of bugs in feature file generation (and application) code that I reported and was fixed in the last release, not sure if those are the bugs you encountered, but if you can elaborate more, I can try reporting any remaining bugs.

twardoch's picture

Arno,

there is no "AFDKO decompiler". As Read Roberts (the principal developer of AFDKO) mentioned above, the ability to fully decompile binary OpenType Layout tables into feature definitions in AFDKO syntax isn't part of AFDKO.

Several products allow partial decompilation into syntax that is fully or mostly compatible with AFDKO:
— the Adobe FDK for OpenType itself (in the spot and ttxn tools, but the output is only partially compatible with the strict AFDKO syntax)
— FontLab Studio 5 (but this is based on the AFDKO version 1.6 syntax, not the new 2.5 syntax, since FLS5 includes the AFDKO version 1.6 compiler)
— FontForge (which uses syntax mostly compatible with AFDKO, but uses its own code to compile from the syntax into binary tables)
— DTL OTMaster (which does decompile into AFDKO 2.5 syntax, but in a way that I find somewhat difficult to read)

In addition, FontLab Studio 5 includes an AFDKO-to-VOLT syntax converter, but it’s not quite perfect. Also, the Font-TTF package written in Perl by Martin Hosken includes a ttf2volt decompiler (which also works on .otf fonts despite its name) and a volt2ttf compiler, which use the VOLT syntax, but also only have partial support.

So, there are two principal syntaxes for defining OpenType Layout features support: AFDKO and VOLT. There are two "full compilers", i.e. AFDKO and VOLT themselves. There are two "partial compilers" (FontForge for the AFDKO syntax, and volt2ttf for the VOLT syntax). And there are several partial decompilers, but none that would support the formats fully. There is even a partial AAT-to-AFDKO syntax decompiler (in FontLab Studio), but none that would do the reverse thing.

There is one partial AFDKO-to-VOLT converter (in FLS5), and there is no direct VOLT-to-AFDKO converter. Conversion can, however, be done using the binary tables as an intermediary (i.e. compile from one syntax, then decompile into another). But since all the decompilers are partial, the conversion is not hassle-free.

Of course, we at FontLab are working towards the goal of being able to complete these tasks better. I hope that Read Roberts can help us here by writing a fully-featured decompiler into the AFDKO syntax that would become part of the standard AFDKO distribution, and could be integrated into font development tools that incorporate the AFDKO libraries.

Best,
Adam Twardoch
Fontlab Ltd.

Arno Enslin's picture

Thanks, Adam!

Do you have a list of problems with FontLab, if it tries to decompile fonts, that were built with the AFDKO 2.5? A way, how the user can identify problems, that are caused by the changes of the syntax? A comparison of input Makeotf and output FontLab?

This is from NewFeatureSyntaxIn2.5.txt:

# Examples of new feature file syntax for FDK 2.5
# Oct 21 2008

# languagesystem DFLT dflt;
# stand-alone lookup: definded outside of a feature block
# Names for stylistic set features
# Multiple substitution
# Reverse chaining substitution
# Contextual chaining substitution with explicit lookup references
# Named anchor and named value record
# Cursive adjustment
# Mark adjustments
# Contextual chaining positioning
# GDEF table definition

Khaled Hosny's picture

@twardoch

There are two "partial compilers" (FontForge for the AFDKO syntax, and volt2ttf for the VOLT syntax).

To the best of my knowledge, FontForge even supports the documented-but-not-yet-implemented parts of adobe's feature file syntax. I'd like to know about what is not implemented in FontForge to qualify it for a "partial compiler".

twardoch's picture

Khaled,

I think my point of being "partial" is that George does not claim to support the AFDKO syntax per se, but is using a syntax that is "almost identical". As far as I remember reading the docs.

I'm not saying it's a bad thing, it's just not AFDKO.

Khaled Hosny's picture

OK, it was a genuine question, since I intend to use feature files as an exchange format, but I don't have regular access to non-linux machines to actually test with AFDKO, so I was interested in knowing any "known" incompatibilities.

Michel Boyer's picture

Here is a copy-paste from http://fontforge.sourceforge.net/featurefile.html. This is what George Williams says about it (or at least about the specs)

Major differences between FontForge's and Adobe's interpretation of feature files

Not really any, any more. Except that FontForge supports much of the syntax which adobe documents but does not support. Adobe continues to reserve the right to change the syntax of anything they do not currently support.

In November 2008 Adobe made radical changes to the feature file format. This meant that my extensions were no longer needed. It also as an incompatible change from the earlier format. FontForge should (still) be able to parse files written in the earlier format, as well as files written in the new format. It will only produce files in the new format now (actually there is a compile time flag which will revert output to the old format, but I haven't tested it).

Syndicate content Syndicate content