
Constructing Languages
▴ Next: Why my apps will soon be gone from the Google Play Store
以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。
▴ Next: Why my apps will soon be gone from the Google Play Store
▾ Previous: Around The World, Part 18: Charting the seas
Fri, 9 Aug 2024 by Thomas ten Cate · Comments
Game Development, Around the World
This post is part of a series about Around The World, a game of exploration and discovery at sea. The game is currently in early development. If you're new to this project, you can read up on it here.
I’m having a bit of an “off” day today, so let’s do something fun, even if it’s rather low priority for the game: generating foreign languages.
The idea is that the player starts out in a region where they speak the local language, which will be represented as English (or some other real-world language, if the game ever gets translated). However, as the player ventures out and discovers new civilizations, they’ll run into a language barrier. Text like quest descriptions and gossip will be shown in incomprehensible gibberish instead, until the player’s character learns the local language, or hires someone to translate. At that point, the foreign text is partially or entirely replaced by English. Of course, I don’t want to use real-world languages in my fictional world, so: we need an algorithm to produce languages!
A more immediate reason to write a language generator is for naming cities, seas, coasts, islands and the like. Right now, city names are just drawn from a list of real-world ports, which is a bit weird at best.
A fairly well known generator for place names is due to Martin O’Leary. He has a great description on his website, and published the source code on GitHub.
I’ve experimented with O’Leary’s algorithm in the past, but I was never entirely happy with the results. One problem might have been that I was trying to do two things at once:
Repeatedly, I ran into the problem that as I added more diversity to words within a language, each language became less distinctive and recognizable, resulting in a uniform soup of words.
But a more fundamental issue was that generated words looked too artificial to me, which is probably due to the rigid phoneme structure that the algorithm uses. So instead, I’ll start with a more classic approach.
Unlike Martin O’Leary’s generator, a Markov chain needs to be created from input data, for example a large amount of text. If we feed it English text, it’ll generate words that look more or less like English (or, in many cases, are English). Since we don’t have any text in a language we haven’t yet generated, this sounds useless, but bear with me – I have some Ideas.
For our purposes, you can think of a Markov chain as a table that contains, for each combination of preceding letters of some fixed length, the probability distribution of the letter that follows. For example, one row in the table could look like this:
... ba | 50% d, 20% g, 10% r, 10% t, 10% y ...
This particular table is based on counting trigrams, i.e. combinations of three letters. In this case, the trigrams bad, bag, bar, bat and bay were encountered in the input text. The left column contains the leftmost two letters of the trigram, and the right column indicates how frequently each next letters was encountered. There are special indicators for start-of-word and end-of-word, which can be treated just like other letters.
To produce the Markov chain, we need some input data. I want a diverse set of languages, so after some searching, I settled on the UDHR corpus, which contains the Universal Declaration of Human Rights translated into 372 languages. This text is rather short: only 1783 words in English. For a normal Markov chain word generator, this would be pretty limiting. But for my Ideas, it shouldn’t matter.
Here’s the full table for the English version, if we count bigrams (combinations of two letters):
^ | a 270, t 256, o 168, i 101, r 96, s 96, e 93, h 92, f 89, p 82, b 69, w 58, c 57, d 43, n 41, m 32, l 27, u 21, g 17, j 7, v 4, k 3 a | n 174, l 128, r 93, t 92, s 68, c 30, $ 20, m 17, v 16, g 16, i 13, d 9, w 9, b 8, y 7, u 2, f 1, p 1, k 1 b | e 52, l 18, y 13, i 7, a 5, o 4, j 4, s 3, u 2, r 1 c | l 45, o 43, e 43, t 38, h 32, i 30, a 19, $ 12, u 10, r 9, c 5, k 5, y 1 d | $ 181, e 45, i 34, o 23, u 15, a 12, l 3, s 3, v 2, g 2, r 2, y 1, h 1 e | $ 363, r 144, n 106, d 84, s 66, c 51, v 40, l 39, e 38, a 32, m 23, t 21, q 16, i 13, f 12, p 7, x 7, o 6, g 3, y 3, k 2, b 1 f | $ 98, r 39, o 31, e 16, u 16, a 11, f 9, i 4 g | h 61, $ 33, e 22, a 12, n 10, r 8, i 8, u 4, s 3, o 3, t 1 h | e 174, a 75, i 60, t 56, $ 36, o 25, u 14, r 3, m 1, y 1, n 1 i | n 135, o 109, t 92, g 71, s 69, c 66, e 25, m 24, v 22, a 22, l 19, r 14, f 8, d 6, b 6, p 5, z 4, $ 1, h 1 j | u 6, o 5, e 4 k | $ 7, i 4, e 3, s 3, n 1 l | $ 120, e 74, l 55, i 47, a 29, y 23, o 15, d 12, t 8, u 7, f 4, v 2, s 2 m | e 49, a 30, i 23, $ 21, o 17, s 17, p 13, m 8, b 7, u 4 n | $ 171, d 142, t 83, a 59, e 58, g 41, i 34, c 34, s 32, o 24, y 19, j 4, h 2, n 2, u 2, l 2, m 2, k 1, v 1, f 1 o | n 189, $ 97, f 95, r 94, m 48, u 36, t 35, c 22, p 19, l 13, d 9, o 8, s 8, y 7, g 6, w 6, v 6, b 4, i 2, h 1, k 1 p | r 43, e 43, l 16, o 12, u 11, a 8, i 6, m 4, $ 4, p 3, t 1, s 1 q | u 16 r | e 128, i 100, $ 85, t 53, y 53, o 43, a 42, s 25, d 11, n 11, m 9, r 8, v 6, b 5, g 5, k 5, u 4, f 4, p 4, c 4, l 2, h 2 s | $ 214, e 47, t 46, h 33, o 32, s 24, u 16, i 15, a 11, p 11, c 9, l 3, d 2, r 1, y 1 t | h 199, i 161, $ 121, o 90, e 75, y 37, a 35, s 33, r 24, l 11, u 10, t 5, w 1, m 1 u | n 42, r 29, m 17, a 17, l 16, t 16, c 14, s 11, b 9, d 6, i 6, p 5, g 4, e 3, f 1 v | e 76, i 10, a 8, o 5 w | h 22, i 18, o 12, $ 10, e 6, a 5, n 2 x | i 2, p 2, e 2, $ 1 y | $ 127, o 31, m 3, r 1, w 1, l 1, s 1, i 1 z | a 3, e 1
I’m using ^ to indicate the start of the word and $ for the end of the word.
Now let’s generate a word using this table, picking the most likely letters each time:
Indeed, an is a rather common English word. If we had started with t instead, can you see where we are most likely to end up?
Now let’s write some code to automate this process, and also randomly select the next letter according to the weighting, instead of just picking the most common one. Here are 10 words produced by the table above:
no ofessinyoory tofen thal frtitopof as eraricolalees con berthigior derthirghtiredendacorouly
… Okay. Not exactly English. What if we use trigrams instead of bigrams? The table is too large to show in full, but here’s some output:
and the beitradefoull mation ance weedomine in artion torights a
Better. Four of those are actual English words, the rest look like parts of words glued together, but that’s fine. Let’s try it on some more languages. I’ve added capitalization and punctuation manually.
Looks quite reasonable to me. I don’t think I’ll use non-Latin-based alphabets, but it’s neat to see that the algorithm correctly inferred that Chinese words mostly consist of a single character.
(Fun fact: DeepL translates the generated mock Chinese text as “The United States of America is the only country in the world that has a universal mandate for the protection of human rights.” So my code, in all its simplicity, is already on par with modern LLMs when it comes to hallucinating! But the sentence is clearly ambiguous, because Google instead translates it as “He strives to make the best use of his talents and to fulfill his family responsibilities”… Chinese is as wonderful as it is mysterious to me.)
Now, how do we get from these Markov chains for real languages to Markov chains for generated languages, that don’t really exist? Here, my Idea is to not create a Markov chain of letters, but of letter classes: vowels and different types of consonants. When generating words, we substitute each class with a letter from that class. This should reduce the resemblance to the input text, and hopefully combine the strengths of Martin O’Leary’s algorithm (pronouncable, plausible words) with those of Markov chains (natural word structures).
I’ll start with these letter classes, grouped roughly using my very limited knowledge of phonetics:
Linguists will recoil in horror at this table: I’m assuming a 1:1 correspondence between spelling and pronunciation here, which is very wrong in general. But since the generated languages will be fictional, nobody will know how to pronounce them anyway.
Now before we create the Markov chain from our input text, we first replace each letter in the text by its class label. This yields a much shorter table, which hopefully captures the word structure (phonotactics) of the input language. For English:
Of course, the quality of generated words is pretty hard to judge now:
GAPSAMKA GAAT AMT TG PGA AVALTAM TGALAALA VAGA AGAGTA AVAL
To map these class labels back to actual letters, we use the occurrences of letters in the source language as weights. For example, in English, ‘e’ is the most common vowel, so it’ll have the largest probability of being chosen if class ‘A’ is encountered.
GAPSAMKA -> repsancu GAAT -> hiet AMT -> unt TG -> dr PGA -> pha AVALTAM -> evaltom TGALAALA -> thiloela VAGA -> vere AGAGTA -> ehirda AVAL -> owel
It certainly doesn’t look like English anymore, but it does look somewhat like a real language.
Can we still distinguish our input languages from each other?
We still get some the double vowels typical of Dutch and Finnish, and the rhythm of the Spanish words also seems familiar. English is all over the place, but that’s fair, considering it’s something of a bastard language.
However, I’m not thrilled about phrases like “pocrancoenicin proenpracon depdaruehosunil”. Those are altogether too many long words in a sequence.
Fortunately, I don’t want to generate entirely random sentences; I want to “translate” English into these generated languages. That lets us use the original English sentence structure as a basis, which doesn’t have such word sequences. So let’s start with a famous English sentence:
It is a truth universally acknowledged, that a single man in possession of a good fortune must be in want of a wife.
For each word, we’ll generate a word in our target language that has the same length. I’m just doing this brute-force by generating words until one of the right length comes up (allowing a little leeway if this takes too many attempts):
In real languages, of course, word lengths are not the same between languages. Some, like Greenlandic, are famous for their long words – the longest Greenlandic word in our input text is “suliffissaaruttoornaveersaartitaanissamut” at 41 letters. Other languages might consist mainly of shorter words. The distribution of word lengths is a significant part of each language’s characteristic look and feel.
To incorporate word length into our generator, we’ll start by tallying how often each word length occurs in the input text:
Now, rather than just drawing random samples from this distribution, I would prefer that short words in the input sentence remain short in the translation, and long words remain long. So we want to somehow “remap” word lengths from English to Greenlandic. We can do that by looking at quantiles. For example, say we want to know the appropriate word length for the translation of the 5-letter word “truth” into Greenlandic. First we calculate: what proportion of English words are shorter than 5 letters? Let’s say it’s 30%. We then generate a word such that (roughly) 30% of Greenlandic words are shorter than that word. Let’s see if it works:
It seems to do the trick. In particular, mock Finnish and mock Zulu have become considerably more representative of the
正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。