Neural Machine Translation

Neural Machine Translation

---

Translation

Source-language text → Target-language text

 

Machine Translation

Sentence x from source language → Sentence y from target language

Most Biggest Research Area: ACL 2021 Accepted Papers, EMNLP 2021 Accepted Papers

 

 

History of Machine Translation

 

---

Statisical Machine Translation

(~2015)

Probabilistic model from data

P(x|y): Translation Model

P(y): Language Model

Alignment: Word-level correspondence between source sentence x and target sentence y

→ Target sentence 必, But some words have no counterpart

→ SMT의 한계점

 

 

Difficulties of SMT

- Extremely complex

- Many subcomponents

- Feature engineering

- Human effort to maintain

→ 단어, 문법, 용례 등 주기적인 update 必

---

Neural Machine Translation

2014 Neurips: Sequence-to-sequence Learning with Neural Networks

2015 ICLR: Neural Machine Translation by Jointly Learning to Align and Translate

 

Evaluation of Machine Translation

Reference: Half of my heart is in Hanana ooh na na

Predicted: Half as my heart is in Obama ooh na

 

BLEU

(Bilingual Evaluation Understudy)

- N-gram overlap between machine translation output and reference sentence

- Precision for n-grams for 1~4

- Add brevity penalty for too short translations

 

 

Diverse N-grams to calculate precisions

Reference: Half of my heart is in Havana ooh na na 

Predicted (from model 1): Half as my heart is in Obama ooh na

Predicted (from model 2): Havana na in heart my is Half ooh of na

Metric	Model 1	Model 2
Precision (1-gram)	7/9	10/10
Precision (2-gram)	4/8	0/9
Precision (3-gram)	2/7	0/8
Precision (4-gram)	1/7	0/7
Brevity penalty	9/10	10/10
BLEU	0.9 x sqrt(0.07)^(1/4)	0

→ BLEU score의 한계점: 0 값으로 인한  BLEU score 값이 0으로 산출될 수 있음

 

Advanced/Other Evaluation Criteria

· SacreBLEU

0값을 0에 가까운 값으로 smoothing

· YiSi

· ESIM

Ground-truth와 얼마나 일치하는지가 아닌, 얼마나 의미가 유사한지를 측정

 

 

Supervised Neural Machine Translation

Open source parallel corpus: OPUS

→ Drawback) Many monolingual corpus, few parallel corpus

 

 

Unsupervised Neural Machine Translation

On monolingual dataset: Denoising (Encoder)

On bilingual dataset: Backtranslation (번역)

 

Denoising

Optimize the probability of encoding a nosied version of the sentence → Reconstructing with the decoder

 

Backtranslation

Retranslating sentence from the target language back to its source language

 

 

Difficulties of NMT

(in bilingual NMT)

1. Domain Mismatch

In-domain과 Out-domain 차이

Difference of In-domain and Out-domain

 

2. Amout of Traninig Data

Outperforms SMT under high-resource conditions

 

3. Rare Words

Solution: Sub-word Embedding

 

4. Long Sentences

Approach: Attention is all you need, Transformer-XL, Train short, test long

 

5. Word Alignment

Attention이 반드시 correspond하지는 않는다.

 

6. Beam Search

Too slow ..

→ Solution) Threshold 설정하여 Normalization

Increasing the beam size does not consistently improve translation quality

 

---

Multilingual Neural Machine Translation

Different lanugate pair들의 translate parameter들을 sharing

Zero-shot translation: Japanese↔English, Korean↔English의 두 translation에서 paramter 공유하여 Korean↔Japanese translation generate

 

---

References

Papers with Code - Machine Translation

statistical machine translation – a translator thinking outside the box

Incorporating BERT into Neural Machine Translation

Unsupervised Neural Machine Translation

Unsupervised Machine Translation Using Monolingual Corpora Only

Zero-Shot Translation with Google’s Multilingual Neural Machine Translation System