📋 Annotation Guidelines

Complete Reference for Translation Evaluation

🚀 Quick Start Guide

You will evaluate machine translations of social media text. For each item:

  1. Read the original tweet
  2. Evaluate each translation on two dimensions
  3. Provide your own translation if you have a better translation

📊 Evaluation Dimensions

1. MEANINGFULNESS (How understandable is the translation?)

Score Label Description When to Use
5 Perfect Flawless English, sounds natural, no grammatical errors Like a native speaker wrote it
4 Good Mostly natural, minor grammatical issues, fully understandable Good but with small mistakes
3 Acceptable Understandable but with significant grammatical issues You can understand but it's clearly non-native
2 Poor English Difficult to understand, many errors, but some meaning preserved Struggle to understand, lots of errors
1 Incomprehensible Very poor English, cannot understand the meaning Makes no sense
NT Not translated Empty, missing, or not actually translated Blank or shows error

2. FAITHFULNESS (How accurate is the translation to the original?)

Score Label Description When to Use
E Perfect Perfectly captures all meaning from the original All information preserved
D Mostly accurate Captures main meaning, misses minor details Main idea correct, small details missing
C Somewhat accurate Captures basic meaning but misses significant aspects Basic meaning there, important details wrong
B Mostly wrong Mostly incorrect but some connection to original Only a little bit correct
A Completely wrong No connection to original meaning Totally unrelated
NT Not translated Empty, missing, or not actually translated Blank or shows error

⚡ Quick Decision Guide

1
Read the original tweet carefully
2
Read each translation one by one
3
Ask: Is it understandable? → Choose Meaningfulness score
4
Ask: Is it accurate? → Choose Faithfulness score
5
Repeat for each translation (1, 2, 3...)
6
Final check: Are there perfect translation(s)?
✅ YES → Provide your own translation in the optional field

🆘 Troubleshooting

Issue Solution
Translation is empty/missing Mark as NT (Not translated) for both dimensions
Can't decide between scores Go with your gut feeling - first instinct is usually right
Original is confusing Do your best - sometimes tweets are ambiguous
Technical problems Refresh the page, your progress is auto-saved

⌨️ Keyboard Shortcuts

🎯 Remember

Your evaluations help improve translation systems for millions of users. Thank you for your careful work!

⭐ Consistency is more important than perfection!