Fixing Korean input in 9front ktrans (2)
In the previous patch, I generated an 11,212-line hangul.map and loaded it into a hash table for lookup. But the Unicode code point of any Hangul syllable can be computed directly from its initial, medial, and final indices, so there is no need for a separate map file. I removed hangul.map to compute syllables directly in code, and fixed a few remaining bugs with uppercase input and buffer handling along the way. Apologies to the 9front members who got stuck carrying a giant hangul.map in the source tree because I sent that first patch without thinking it through.
Before the fix
Previously, ktrans read all 11,212 lines of /lib/ktrans/hangul.map from top to bottom on startup and registered every key and its prefixes in a hash table. On top of that, near the end of the file, ㅖ had been written with a lowercase p instead of P, so it got overwritten by p ㅔ on the next line. Syllables containing ㅖ such as dP (예) worked, but typing P (ㅖ) on its own left P as is.
Uppercase handling was also missing. In Dubeolsik, only the seven keys Q (ㅃ), W (ㅉ), E (ㄸ), R (ㄲ), T (ㅆ), O (ㅒ), and P (ㅖ) are distinguished by Shift; all other letters should produce the same Jamo in uppercase as in lowercase. Without this, releasing Shift slightly late when typing '까' (RK) produced ㄲK, and GKS (한) remained GKS (before video).
There was also an issue in dubeollkup with the code that clears the line buffer when composition of the previous syllable ends.
if(hmapget(hangul, line->b, &lkup) < 0){
if(hmapget(hangul, buf, &lkup) < 0){
resetstr(line, nil);
line->p = pushutf(line->p, strend(line), e1, utflen(e1));
if(hmapget(hangul, line->b, &lkup) < 0)
return;
}
e1 points to the last character just typed inside line->b. When there is only one character in line, e1 and line->b point to the same address, so calling resetstr(line, nil) first sets line->b[0] to \0 and wipes out e1 as well. On the other hand, when line has two or more characters and a non-Hangul character like a comma (,) is typed, the code clears line and pushes , into it before the lookup fails and returns, leaving , in line.
Syllable calculation
I removed /lib/ktrans/hangul.map and wrote Jamo arrays and hangullkup in main.c.
static char chotab[] = "rRseEfaqQtTdwWczxvg";
static char *jungtab[] = {
"k", "o", "i", "O", "j", "p", "u", "P", "h", "hk", "ho",
"hl", "y", "n", "nj", "np", "nl", "b", "m", "ml", "l",
};
static char *jongtab[] = {
"", "r", "R", "rt", "s", "sw", "sg", "e", "f", "fr", "fa", "fq",
"ft", "fx", "fv", "fg", "a", "q", "qt", "t", "T", "d", "w", "c",
"z", "x", "v", "g",
};
static char *jamotab[] = {
"r", "R", "", "s", "", "", "e", "E", "f", "",
"", "", "", "", "", "", "a", "q", "Q", "",
"t", "T", "d", "w", "W", "c", "z", "x", "v", "g",
};
static Rune
hangullkup(char *s)
{
char *p;
int i, j, l, cho;
if(s[0] == '\0')
return 0;
for(i = 0; i < nelem(jamotab); i++)
if(strcmp(s, jamotab[i]) == 0)
return 0x3131 + i;
for(i = 0; i < nelem(jungtab); i++)
if(strcmp(s, jungtab[i]) == 0)
return 0x314f + i;
if((p = strchr(chotab, s[0])) == nil)
return 0;
cho = p - chotab;
s++;
for(i = 0; i < nelem(jungtab); i++){
l = strlen(jungtab[i]);
if(strncmp(s, jungtab[i], l) != 0)
continue;
for(j = 0; j < nelem(jongtab); j++)
if(strcmp(s + l, jongtab[j]) == 0)
return 0xac00 + (cho*21 + i)*28 + j;
}
return 0;
}
A standalone Jamo returns a Unicode Hangul Compatibility Jamo value from 0x3131 to 0x3163. The consonant range from 0x3131 ('ㄱ') to 0x314E ('ㅎ') has 30 slots, with 11 compound finals such as ㄳ, ㄵ, and ㄶ mixed in among them. In Dubeolsik, typing consonants in a row without a vowel should produce two characters (ㄱㅅ for rt) rather than combining into a single ㄳ, so the compound final slots in jamotab are left as empty strings (""). The 21 vowels from 0x314F ('ㅏ') to 0x3163 ('ㅣ') are in the same order as syllable medials, so jungtab handles both standalone vowels and medials.
If the input is not a standalone Jamo, hangullkup computes a syllable starting from 0xAC00 ('가'). It looks up s[0] in chotab to get the initial consonant index cho, then looks through the rest of the string for a prefix jungtab[i] whose remainder matches jongtab[j], and returns 0xac00 + (cho*21 + i)*28 + j. Syllables without a final consonant match jongtab[0], which is "". For an input with a compound vowel like rhkr (곽), the leading h (ㅗ) passes strncmp first, but the remaining kr starts with the vowel key k and is not in jongtab, so the loop moves on and matches hk (ㅘ) and r (ㄱ).
Updating the lookup code
Since hangullkup returns a Rune instead of a string pointer, I added emitrune to convert it to UTF-8 and output it, and updated dubeolbksp and dubeollkup to match.
static void
emitrune(Rune r)
{
char s[UTFmax+1];
s[runetochar(s, &r)] = '\0';
emitutf(output, s, 1);
}
static void
dubeolbksp(Str *line)
{
Rune r;
popstr(line);
/* lookup and emit remaining Jamo to output */
if((r = hangullkup(line->b)) == 0)
return;
emitrune(r);
}
static void
dubeollkup(Str *line)
{
Rune r, r2;
char buf[UTFmax*2];
char *e2, *e1;
e1 = peekstr(line->p, line->b);
if((r = hangullkup(line->b)) == 0){
if(e1 != line->b){
e2 = peekstr(e1, line->b);
pushutf(buf, buf+sizeof buf, e2, utflen(e2));
}else
pushutf(buf, buf+sizeof buf, e1, utflen(e1));
if((r2 = hangullkup(buf)) == 0){
pushutf(buf, buf+sizeof buf, e1, 1);
resetstr(line, nil);
if((r = hangullkup(buf)) == 0)
return;
line->p = pushutf(line->p, strend(line), buf, 1);
}else{
/* treat Jongseong as Choseong when it matches with new Jamo */
popstr(line);
popstr(line);
r = hangullkup(line->b);
emitutf(output, backspace, 2);
emitrune(r);
emitrune(r2);
line->p = pushutf(line->b, strend(line), buf, utflen(buf));
return;
}
}
if(utflen(line->b) == 1)
emitutf(output, backspace, 1);
else
emitutf(output, backspace, 2);
emitrune(r);
}
I moved the code that copies the last two keys into buf inside the if block so it only runs when looking up the whole line->b fails. When buf also fails to match and composition of the previous syllable ends, it copies e1 into buf before resetstr(line, nil) clears line, and only pushes buf back into line if it is a valid Jamo. When a non-Hangul character like a comma is typed, line stays empty.
In keythread, I added code to convert uppercase letters other than QWERTOP to lowercase when in Korean mode.
case LangKO:
if(isupper(*p) && strchr("QWERTOP", *p) == nil)
*p = tolower(*p);
break;
Now RK produces '까', GKS produces '한', and rMF produces '글' (after video).
I added a test to test.c to verify standalone ㅖ, single consonants and compound finals, uppercase input, and backspace after punctuation.
P O hk rt rt\b rtk rkrt rkrtk rkfrt rkfrk RK GKS rMF rk,\b
ㅖ ㅒ ㅘ ㄱㅅ ㄱ ㄱ사 갃 각사 갉ㅅ 갈가 까 한 글 가
In rkfrt, '갉' is followed by t (ㅅ) rather than a vowel, so the final consonant does not carry over and the result is '갉ㅅ'. rk,\b checks that erasing the comma with backspace in '가,' leaves the preceding '가' intact.