Machine Translation with Inferred Stochastic Finite-State Transducers von Klaus Suttner HS: Endliche Automaten Dozentin: Karin Haenelt Seminar für Computerlinguistik Universität Heidelberg 29.06.09
Finite-state Transducer Definition Ein Finite-state Transducer ist ein Tupel T : Σ,, Q, q 0, F, δ Σ = Eingabealphabet = Ausgabealphabet Q = Menge von Zuständen q 0 = Startzustand F Q = Menge von Endzuständen δ Q Σ Q φ heißt eine Übersetzung von s, falls: φ = (φ 0, s 1, t 1, φ 1 )(φ 1, s 2, t 2, φ 2 )... (φ l 1, s l, t l, φ l ) mit s = l d(s, t) bezeichne alle Übersetzungen des Paars (s, t).
Stochastic Finite-state Transducer Definition Ein Stochastic Finite-state Transducer ist ein Tupel T : Σ,, Q, q 0, F, p, f mit der Erweiterung: p : Q Σ Q [0, 1] f : Q [0, 1] und den Eigenschaften: q Q : f (q) + p(q, a, ω, q ) = 1 (a,ω,q ) Σ Q P Tp (s, t) = 1 (s,t) Σ
Stochastic Finite-state Transducer Wahrscheinlichkeit einer Übersetzung (s, t): P Tp (s, t) = P Tp (φ) φ d(s,t) l P Tp (φ) = f (q l ) p(q i 1, s i, t i, q i ) i=0 Übersetzung einer Zeichenkette s: ˆt = arg max t P T p (s, t)
Viterbi-Approximation Approximation: P Tp V Tp (s, t) = max P Tp (φ) φ d(s,t) t = arg max V T t p (s, t) = arg max max t φ d(s,t) P Tp (φ) Berechnung: max V T t p (s, t) = max(v ( s, q)f (q)) q Q V (i, q) = max q Q,ω (V (i 1, q )p(q, s i, ω, q)), i 0, q q 0 V (0, q 0 ) = 1
Verfahren jeder Paar (s, t) Zeichenkette z Induktion einer regulären Grammatik Erzeugung des Transducers
Übersetzung IBM-Modell 1-5 e(i) = i-te Wort des Originalsatzes (hier: englisch) f(j) = j-te Wort des Zielsatzes (hier: französisch) a(i) = Alignierung für Wort i des Originalsatzes Fertility φ spurious words p 1 Position NULL = e 0 Distortion d(a(i) e i, s, t ) d(5 2, 4, 6) Translation t e i f a(i)
IBM-Modell 3 1 For each English word e i indexed by i = 1...l, choose fertility φ(i) with probability n(φ(e i ) e i ). 2 Choose the number φ(0) of "spurious" French words to be generated from e 0 = NULL using probability p 1 and n i=1 φ(i) from step 1. 3 Let m m = n i=0 φ(i) be the sum of fertilities for all words, including NULL. 4 For each i = 0...l, and each k = 1...φ(i), choose a French word τ ik with probability t(τ ik e i ). 5 For each i = 1...l, and each k = 1...φ(i), choose target French position π ik with probability d(π ik i, l, m). 6 For each k = 1...φ 0, choose a position π 0k from the φ(0) k + 1 remaining vacant positions in 1...m, for a total probability of 1 φ(0)!. 7 Output the French sentence with words τ ik in positions π ik (0 i l, 1 k φ(i)).
IBM-Modell 3 Beispiel input : Mary did not slap the green witch choose fertilities: Mary not slap slap slap the green witch choose number of spurious words: Mary not slap slap slap NULL the green witch choose translations: Mary no daba una bofetada a la verde bruja choose target positions: Mary no daba una bofetada a la bruja verde
EM-Algorithmus Beispiel Satz 1: Align. 1: Satz 2: Align. 2: X b y t(x b) = 1 2 t(y b) = 1 2 t(x c) = 1 2 t(y c) = 1 2
EM-Algorithmus Beispiel Satz 1: Align. 1: Satz 2: t(x b) = 1 2 t(y b) = 1 2 t(x c) = 1 2 t(y c) = 1 2 Align. 2: X b y P(a, f e) = 1 2 1 2 1 2 = 1 8 P(a, f e) = 1 2 1 2 1 2 = 1 8 P(a, f e) = 1 2
EM-Algorithmus Beispiel Satz 1: Align. 1: Satz 2: t(x b) = 1 2 t(y b) = 1 2 t(x c) = 1 2 t(y c) = 1 2 Align. 2: X b y P(a, f e) = 1 8 P(a, f e) = 1 8 P(a, f e) = 1 2
EM-Algorithmus Beispiel Satz 1: Align. 1: Satz 2: Align. 2: X b y P(a, f e) = 1 8 P(a f, e) = 1 8 2 8 P(a, f e) = 1 8 P(a f, e) = 1 8 2 8 P(a, f e) = 1 2 P(a f, e) = 1 2 1 2 = 1 2 = 1 2 = 1 t(x b) = 1 2 t(y b) = 1 2 t(x c) = 1 2 t(y c) = 1 2
EM-Algorithmus Beispiel Satz 1: Align. 1: Satz 2: Align. 2: X b y P(a, f e) = 1 8 P(a f, e) = 1 8 2 8 P(a, f e) = 1 8 P(a f, e) = 1 8 2 8 P(a, f e) = 1 2 P(a f, e) = 1 2 1 2 Zähle Treffer (counts): tc(x b) = 1 2 tc(y b) = 1 + 1 = 3 2 2 tc(x c) = 1 2 = 1 2 = 1 2 = 1 t(x b) = 1 2 tc(y c) = 1 2 t(y b) = 1 2 t(x c) = 1 2 t(y c) = 1 2
EM-Algorithmus Beispiel Satz 1: Align. 1: Satz 2: Align. 2: X b y P(a, f e) = 1 8 P(a f, e) = 1 8 2 8 P(a, f e) = 1 8 P(a f, e) = 1 8 2 8 P(a, f e) = 1 2 P(a f, e) = 1 2 1 2 Zähle Treffer (counts): tc(x b) = 1 2 tc(y b) = 3 2 tc(x c) = 1 2 = 1 2 = 1 2 = 1 t(x b) = 1 2 tc(y c) = 1 2 t(y b) = 1 2 t(x c) = 1 2 t(y c) = 1 2
EM-Algorithmus Beispiel Satz 1: Align. 1: Satz 2: t(x b) = 1 2 t(y b) = 1 2 t(x c) = 1 2 t(y c) = 1 2 Align. 2: X b y P(a, f e) = 1 8 P(a f, e) = 1 8 2 8 P(a, f e) = 1 8 P(a f, e) = 1 8 2 8 P(a, f e) = 1 2 P(a f, e) = 1 2 1 2 tc(x b) = 1 2 Normalisiere: = 1 2 = 1 2 = 1 t(x b) = t(x b) t( b) = 1 2 42 tc(y b) = 3 2 t(y b) = 3 2 42 = 3 4 tc(x c) = 1 2 t(x c) = 1 2 1 = 1 2 tc(y c) = 1 2 t(y c) = 1 2 1 = 1 2
EM-Algorithmus Beispiel Satz 1: Align. 1: Satz 2: t(x b) = 1 4 t(y b) = 3 4 t(x c) = 1 2 t(y c) = 1 2 Align. 2: X b y P(a, f e) = 1 8 P(a f, e) = 1 8 2 8 = 1 2 P(a, f e) = 1 8 P(a f, e) = 1 8 2 8 = 1 2 P(a, f e) = 1 2 P(a f, e) = 1 2 1 2 = 1 Normalisiere: tc(x b) = 1 2 t(x b) = 1 2 42 = 1 4 tc(y b) = 3 2 t(y b) = 3 2 42 = 3 4 tc(x c) = 1 2 t(x c) = 1 2 1 = 1 2 tc(y c) = 1 2 t(y c) = 1 2 1 = 1 2
EM-Algorithmus Beispiel Satz 1: Align. 1: Satz 2: t(x b) = 1 4 t(y b) = 3 4 t(x c) = 1 2 t(y c) = 1 2 Align. 2: X b y P(a, f e) = 1 4 1 2 1 2 = 1 16 P(a, f e) = 3 4 1 2 1 2 = 3 16 P(a, f e) = 3 4
EM-Algorithmus Beispiel Satz 1: Align. 1: Satz 2: t(x b) = 1 4 t(y b) = 3 4 t(x c) = 1 2 t(y c) = 1 2 Align. 2: X b y P(a, f e) = 1 16 P(a, f e) = 3 16 P(a, f e) = 3 4
EM-Algorithmus Beispiel Satz 1: Align. 1: Satz 2: t(x b) = 1 4 t(y b) = 3 4 t(x c) = 1 2 t(y c) = 1 2 Align. 2: X b y P(a, f e) = 1 16 P(a f, e) = 1 16 4 16 P(a, f e) = 3 16 P(a f, e) = 3 16 4 16 P(a, f e) = 3 4 P(a f, e) = 3 4 3 4 = 1 4 = 3 4 = 1
EM-Algorithmus Beispiel Satz 1: Align. 1: Satz 2: t(x b) = 1 4 t(y b) = 3 4 t(x c) = 1 2 t(y c) = 1 2 Align. 2: X b y P(a, f e) = 1 16 P(a f, e) = 1 4 P(a, f e) = 3 16 P(a f, e) = 3 4 P(a, f e) = 3 4 P(a f, e) = 1 Zähle Treffer (counts): tc(x b) = 1 4 tc(y b) = 3 4 + 1 = 7 4 tc(x c) = 3 4 tc(y c) = 1 4
EM-Algorithmus Beispiel Satz 1: Align. 1: Satz 2: t(x b) = 1 4 t(y b) = 3 4 t(x c) = 1 2 t(y c) = 1 2 Align. 2: X b y P(a, f e) = 1 16 P(a f, e) = 1 4 P(a, f e) = 3 16 P(a f, e) = 3 4 P(a, f e) = 3 4 P(a f, e) = 1 Zähle Treffer (counts): tc(x b) = 1 4 tc(y b) = 7 4 tc(x c) = 3 4 tc(y c) = 1 4
EM-Algorithmus Beispiel Satz 1: Align. 1: Satz 2: t(x b) = 1 4 t(y b) = 3 4 t(x c) = 1 2 t(y c) = 1 2 Align. 2: X b y P(a, f e) = 1 16 P(a f, e) = 1 4 P(a, f e) = 3 16 P(a f, e) = 3 4 P(a, f e) = 3 4 P(a f, e) = 1 Normalisiere: tc(x b) = 1 4 t(x b) = 1 4 84 = 1 8 tc(y b) = 7 4 t(y b) = 7 4 84 = 7 8 tc(x c) = 3 4 t(x c) = 3 4 1 = 3 4 tc(y c) = 1 4 t(y c) = 1 4 1 = 1 4
EM-Algorithmus Beispiel Satz 1: Align. 1: Satz 2: t(x b) = 1 8 t(y b) = 7 8 t(x c) = 3 4 t(y c) = 1 4 Align. 2: X b y P(a, f e) = 1 16 P(a f, e) = 1 4 P(a, f e) = 3 16 P(a f, e) = 3 4 P(a, f e) = 3 4 P(a f, e) = 1 Normalisiere: tc(x b) = 1 4 t(x b) = 1 4 84 = 1 8 tc(y b) = 7 4 t(y b) = 7 4 84 = 7 8 tc(x c) = 3 4 t(x c) = 3 4 1 = 3 4 tc(y c) = 1 4 t(y c) = 1 4 1 = 1 4
Beispiel
Segmentierung Beispiel Cuánto cuesta una habitación individual por semana? how(2) much(2) does(3) a(4) single(6) room(5) cost(3) per(7) week(8)?(9)
Segmentierung z = s, 1 i z : (s i, t j t j+1...t j+l ) falls j : a(j) = i und j < j : a(j ) > a(j) z i = und für j : j j j + l, a(j ) a(j) (s i, λ) sonst
Segmentierung z = s, 1 i z : (s i, t j t j+1...t j+l ) falls j : a(j) = i und j < j : a(j ) > a(j) z i = und für j : j j j + l, a(j ) a(j) (s i, λ) sonst Cuánto cuesta una habitación individual por semana? How much does a single room cost per week? (, λ), (Cuánto, how much) (cuesta, does) (una, a) (habitación, λ) (individual, single room cost) (por, per) (semana, week) (?,?)
Segmentierung z = s, 1 i z : (s i, t j t j+1...t j+l ) falls j : a(j) = i und j < j : a(j ) > a(j) z i = und für j : j j j + l, a(j ) a(j) (s i, λ) sonst una camera doppia a(1) double(3) room(2) una camera a(1) room(2) la camera singola the(1) single(3) room(2) la camera the(1) room(2) (una, a) (camera, λ) (doppia, double room) (una, a) (camera, room) (la, the) (camera, λ) (singola, single room) (la, the) (camera, room)
Segmentierung una camera doppia a(1) double(3) room(2) una camera a(1) room(2) la camera singola the(1) single(3) room(2) la camera the(1) room(2) (una, a) (camera, λ) (doppia, double room) (una, a) (camera, room) (la, the) (camera, λ) (singola, single room) (la, the) (camera, room)
Transducer n-gramme Der Übergang (q, a, b 1 b 2...b k, q ) ist im Transducer T enthalten, wenn die beiden Zustände q = (z i n+1...z i 1 ) und q = (z i n+2...z i ) in T existieren und z i = (a, b 1 b 2...b k ). Die Wahrscheinlichkeit ist p(z i z i n+1...z i 1 ). p(z i z i n+1...z i 1 ) = c(z i n+1...z i 1 z i ) c(z i n+1...z i 1 ) Anzahl der Zustände = Anzahl der k-gramme (k < n) Anzahl der Übergänge = Anzahl der k-gramme (k n) + Back-off Anzahl der k-gramme hängt von Segmentierung ab. Monotonie
Smoothing I drove a new car.
Evaluation Gesprächssituation: Hotel WER = Word Error Rate, SER = Sentence Error Rate, BLEU = Anzahl der korrekten n-gramme in % EU-Trans-0: Sätze: Wortschatz: insgesamt 490.000 spanisch 686 unterschiedliche 168.629 englisch 513
Evaluation Gesprächssituation: Hotel WER = Word Error Rate, SER = Sentence Error Rate, BLEU = Anzahl der korrekten n-gramme in % EU-Trans-1a: Sätze: Wortschatz: insgesamt 10.000 spanisch 6.622 unterschiedliche 6.636 deutsch 4.890
Evaluation Übersetzungsbeispiele
Literatur Casacuberta, Francisco, Vidal, Enrique. 2004. Machine translation with inferred stochastic finite-state transducers. Computational Linguistics 30(2):205 225. Brown, Peter F., Stephen A. Della Pietra, Vincent J. Della Pietra, and Robert L. Mercer. 1993. The mathematics of statistical machine translation. Computational Linguistics, 19(2):263 310