結果提交及評估公式
結果提交
各系統的輸出結果存為一個結果檔 (run)，檔名格式為
RITEVAL-(Team Name)-(Lang)-(Subtask Name)-(Run Number).txt- Team Name = 報名時所使用的 Group ID
- Lang = CS, CT
- Subtask Name = FV, SVBC, SVMC
- Run Number = 01, 02, ... 05
例如：
RITEVAL-NTOUA-CT-SVBC-01.txt結果檔內容格式
結果檔中每一行對應到每道題目，格式如下：
t2_ID [SPACE] Label [SPACE] Confidence其中 Confidence 表示推測的信心度。
中文 FV 任務每道題目都應標出所屬三種標籤 (E, C, U 請參考任務定義) 之一。
中文 SV 的 BC 任務每道題目都應標出所屬兩種標籤 (Y, N 請參考任務定義) 之一。
中文 SV 的 MC 任務每道題目都應標出所屬四種標籤 (F, B, C, I 請參考任務定義) 之一。
FV 結果檔內容範例如下：
1 E 0.8522 U 0.9943 E 0.7894 C 1.000 :SV 結果檔內容與 FV 相同，僅標籤集合不同。
系統描述
亦請提供各系統之簡短描述 (各結果所對應之系統需各提交一個系統描述檔)，
包括策略、所使用之資訊或特徵、資源 (例如語言資源或 www)、以及所使用工具 (例 NLP 工具) 等等。
系統描述檔取名規則如下：
RITEVAL-(Team Name)-(Lang)-(Subtask Name)-(Run Number)-sysdesc.txt- Team Name = 報名時所使用的 Group ID
- Lang = CS, CT
- Subtask Name = FV, SVBC, SVMC
- Run Number = 01, 02, ... 05
例如：
RITEVAL-NTOUA-CT-SVBC-01-sysdesc.txt系統描述檔內容範例如下：
1. Approach [ ]rule: [x]statistics: SVM [ ]hybrid2. Feature/Information [ ]Overlapping [x]Alignment [ ]Transformation [x]Char/Word Overlapping [x]Syntactic Information [ ]Predicate-Argument Relationship [x]Named Entity [ ]Entity/Event [x]Temporal/Numeric Information [ ]Entailment [ ]Modality [ ]Polarity [x]Synonym/Antonym [x]Hypernym/Hyponym [ ]Meronym/Holonym [ ]Entity/Event Relationship [ ]Entailment Rule3. Resources: word segmentation, syntactic parser4. Tools: WordNet, Wikipedia提交方式
請將所有檔案壓縮成一個檔案，以 e-mail 寄至 rite-val-organizers@nii.ac.jp , 期限是 2014/8/7 23:59 (已延長)。
請在信件檔題標明所參加之任務以及語言為何。
收到結果檔後，我們會在 24 小時內回信確認。若您沒收到確認信，請與我們聯絡。
結果評估公式
Macro-F (所有類別的 F-measures 之 Macro-averaging 平均值)
Submission
Results from one system comprise a run. Name a run file in the following format:
RITEVAL-(Team Name)-(Lang)-(Subtask Name)-(Run Number).txt- Team Name = use the same short id used in NTCIR registration (Group ID).
- Lang = CS, CT
- Subtask Name = FV, SVBC, SVMC
- Run Number = 01, 02, ... 05
Example:
RITEVAL-NTOUA-CT-SVBC-01.txtRun Format
Each line contains the result for one t2 in the following format:
t2_ID [SPACE] Label [SPACE] Confidencewhere Confidence is the confidence score.
In Chinese FV Subtasks, each t2 should be tagged in one of the three labels (E, C, U cf. Task Definition).
In Chinese SV-BC Subtasks, each t2 should be tagged in one of the two labels (Y, N cf. Task Definition).
In Chinese SV-MC Subtasks, each t2 should be tagged in one of the four labels (F, B, C, I cf. Task Definition).
Examples of one FV run:
1 E 0.8522 U 0.9943 E 0.7894 C 1.000 :SV runs look like FV runs except that their label sets are different.
System Description
Please also create a text file (one file for each run) to provide brief system description,
including approaches, used information or features, resources (any language resource, web etc), and tools (NLP tools etc).
Name a run file in the following format:
RITEVAL-(Team Name)-(Lang)-(Subtask Name)-(Run Number)-sysdesc.txt- Team Name = use the same short id used in NTCIR registration (Group ID).
- Lang = CS, CT
- Subtask Name = FV, SVBC, SVMC
- Run Number = 01, 02, ... 05
Example:
RITEVAL-NTOUA-CT-SVBC-01-sysdesc.txtExample of a system description file is as follows:
1. Approach [ ]rule: [x]statistics: SVM [ ]hybrid2. Feature/Information [ ]Overlapping [x]Alignment [ ]Transformation [x]Char/Word Overlapping [x]Syntactic Information [ ]Predicate-Argument Relationship [x]Named Entity [ ]Entity/Event [x]Temporal/Numeric Information [ ]Entailment [ ]Modality [ ]Polarity [x]Synonym/Antonym [x]Hypernym/Hyponym [ ]Meronym/Holonym [ ]Entity/Event Relationship [ ]Entailment Rule3. Resources: word segmentation, syntactic parser4. Tools: WordNet, WikipediaRun Submission
Archive all files in zip, and send the result to rite-val-organizers@nii.ac.jp via email attachment by the end of the formal run period (2014/8/7 23:59, extended).
Specify which subtasks and languages you are participating in the mail title.
We'll email you a notification of acceptance as a reply within 24 hours of submission. Contact us if you did not receive the notification.
Evaluation Metrics
Macro-F (Macro-averaging F-measures over all labels)