Pre-requisites
Python programming
Explanation of activity
The students are requested to download, train, configure and use different ML/Ai models that claim to do fact-checking and benchmark their performance using the well-known FEVER dataset. The students are given a set of models to choose from (see references below) and implement and use these models as the see fit. Other models can be used but this needs to be properly justified.
Teacher materials
- James Thorne, Andreas Vlachos, Christos Christodoulopoulos, Arpit Mittal: FEVER: a Large-scale Dataset for Fact Extraction and VERification. NAACL-HLT 2018: 809-819
- Wanjun Zhong, Jingjing Xu, Duyu Tang, Zenan Xu, Nan Duan, Ming Zhou, Jiahai Wang, Jian Yin: Reasoning Over Semantic-Level Graph for Fact Checking. ACL 2020: 6170-6180
- Mohamed H. Gad-Elrab, Daria Stepanova, Jacopo Urbani, Gerhard Weikum: Tracy: Tracing Facts over Knowledge Graphs and Text. WWW 2019: 3516-3520
Student materials
- James Thorne, Andreas Vlachos, Christos Christodoulopoulos, Arpit Mittal: FEVER: a Large-scale Dataset for Fact Extraction and VERification. NAACL-HLT 2018: 809-819
- Wanjun Zhong, Jingjing Xu, Duyu Tang, Zenan Xu, Nan Duan, Ming Zhou, Jiahai Wang, Jian Yin: Reasoning Over Semantic-Level Graph for Fact Checking. ACL 2020: 6170-6180
- Mohamed H. Gad-Elrab, Daria Stepanova, Jacopo Urbani, Gerhard Weikum: Tracy: Tracing Facts over Knowledge Graphs and Text. WWW 2019: 3516-3520
Assessing and evaluation
The implementation, proper training and fine tuning and finally the performance are the important elements of the evaluation process. The students can choose to implement two or three of the proposed algorithms and then they need to compare the results over the same dataset.
The are also assessed based on the quality of the discussion around the comparison of the implemented methods, the metrics used, the proper capturing of the information, etc.
