One source of information I have used for similar purposes is Open Office Dictionaries and other dictionaries for spelling checkers. For instance, if you click on the link for Canadian English (zip file) you get a folder containing a file with extension .dic with 62341 entries (including "derived" entries). Other dictionaries can be much larger. The .dic file is plain text. If you remove what follows the slash after each word, you get a file on which you can run programs to extract pairs, count them, etc. Of course, that gives no information on the frequency with which those pairs occur in actual texts but that gives information on possible pairs for the language you chose. Some dictionaries are utf-8 encoded, others are latin1 and so on. The encoding is given at the first line of a second file with extension .aff. Some programming ability is thus required.

Thanks Michael -- I'd tried using the Mac OS built-in dictionary for those languages I've installed, but it doesn't support wildcards (or if it does, the asterisk isn't used for this). Never thought, though, to try opening the actual file (a senior moment).

André

I always wish there would be some linguistics textbook that covers this stuff. Maybe Steve Peters will chime in here with some help.

If you have time to figure out the syntax to sift through text file wordlists it’s pretty easy to put this stuff together using Python or just Bash scripting (grep "*öö*" file.txt | wc -l). The OpenWall wordlists disk is worth it’s low price if you don’t need to analyze actual text. Ask around in the netsec world and I’m sure even more dictionaries exist. Project Gutenberg and similar resources probably have real texts covering many of the languages you need to analyze.

James wrote: I always wish there would be some linguistics textbook that covers this stuff.

Linguistics texts generally aren't that concerned with orthography, so this isn't a likely source. You'll find lots of information on the pairings of various sounds , but any statistics presented will likely involve IPA rather than orthographic representations.

André

Frequency analysis is what you really need.

Most obviously. To get frequencies (absolute or relative) of bigrams, all you need is a very basic script that can be run on some utf-8 encoded input. To get such a script (for alphabetic bigrams), you can just copy what is between the cut lines and paste it in a terminal window and you will get an executable file named bigrams in your current folder.

----
cat >bigrams <<'EOF'
#!/usr/bin/python

# M. Boyer 2009

import codecs, sys
infile=codecs.open(sys.argv[1],"r","utf-8")
text=infile.read(); infile.close()

tallies={};  nbdata=0;  prev=' '
def tallyq(c):
  return c.isalpha()

for char in text:
  if (tallyq(prev) and tallyq(char)):
    datum=prev+char # ; datum=datum.lower()
    nbdata=nbdata+1
    if datum in tallies:
      tallies[datum]=tallies[datum]+1
    else:
      tallies[datum]=1
  prev=char

for d in tallies:
  print('%s;%d;%.3f%%' %
     (d.encode('utf-8'), tallies[d], 100.0*tallies[d]/nbdata))
EOF
chmod 755 bigrams
----

Then you decide what you want to run it on. For instance, if you want to run it on Chekhov's text Дама с собачкой (The lady with the little dog), you can type (or copy and paste) the line

   lynx -dump http://lib.ru/LITRA/CHEHOW/d.txt > dama.txt

and then run (maybe after removing some html references at the bottom)

   ./bigrams dama.txt | sort

Here is a copy paste of part of the output

то;372;1.927%
тп;1;0.005%
тр;85;0.440%
тс;31;0.161%
ту;26;0.135%
тф;2;0.010%
тх;1;0.005%
тч;7;0.036%

There were 372 occurrences of то which reprensents 1.927% of all bigrams (after cleaning the text).

With the internet, there are now many sources of texts in all languages. There is also nothing to prevent you from running the script on a dictionary to know possible combinations; it seems you then don't need the frequencies but it may still be interesting to see what were the words containing bigrams with very low frequencies. A simple grep answers the question.

Michel

[added] I guess the mac does not come with lynx installed. I must have installed it myself. That example may be more for Linux than mac users. Sorry.

Ohai.

The LetterMeter from Peter Bilak and Just van Rossum can run a text for single letter and letter pair occurence. Then it is just a matter of feeding it with the texts you deem appropriate.

Says the website:
LetterMeter is a text analysis tool, used in the Type&Media classes (postgraduate course of type design) at the Royal Academy of Art in The Hague. LetterMeter is designed for comparing multilingual texts and measuring the frequency of particular glyphs.

Because it is Unicode based, it will work with the majority of languages. The current version will recognize Latin, Greek and Cyrillic glyphs, and sort them according to their formal attributes. LetterMeter's results include statistics for the incidences of round/square/open/diagonal left and right sides of glyphs, ratios of vowels/consonants, and counts of glyphs with accents, ascenders and descenders, in any given text(s).

LetterMeter was developed jointly by Peter Bilak and Just van Rossum, whom I would like to thank for the Python programming. Vera Evstafieva helped with the Cyrillic specifications, and Panos Haratzopoulos with the Greek.

LetterMeter is created using Python. and works only on Mac OS X. Although it is available for free, it is copyrighted, and you may not redistribute it. All rights reserved, © 2003, Peter Bilak, Just van Rossum.

For TEH DOWNLOADS at Typotheque

Here are some results I got for English. http://groups.google.ca/group/comp.lang.postscript/msg/34c2bb049b42f668?...

I used to use a C program to count the most common digrams, then augment it against punctuation, to generate kerning pair lists for URW Kernus.