The OpenFact project team took part in the CheckThat! organized as part of the international conference CLEF 2023 (Conference and Labs of the Evaluation Forum). The method proposed by our scientists took first place. This method detects English sentences that need to be reviewed because of potential misleading and therefore are worth fact-checking.
The main method of the OpenFact team achieved the highest score of F1 (the harmonic mean of precision and recall of the response) giving first place in the ranking, ahead of methods developed by 10 other teams, including the Fraunhofer Institute and Accenture. The second method of our scientists also turned out to be better than the competition (although more models could be submitted for evaluation, officially only one can be indicated in the classification). The winning method involved using a large GPT language model of 13 billion parameters pre-trained on a general text corpus of 800GB and then training it for a classification task on a specially selected training set. The second best solution was to train the local DeBERT model. Detailed information on the implementation of the winning method can be found in the paper “OpenFact at CheckThat! 2023: Head-to-Head GPT vs. BERT – A Comparative Study of Transformers Language Models for the Detection of Check-worthy Claims” .
Detecting sentences that require checking is extremely important in the fact-checking process, because it allows you to filter the message stream and direct the attention of a professional fact-checker or automatic verification algorithm. CLEF CheckThat! The Lab has been organized since 2018, and its goal is to develop methods for automatic identification and verification of statements appearing in politicians’ statements. The international conference was held in Bologna, Bucharest, Lugano, Avignon and the results of this year’s edition will be presented in September at the conference in Thessaloniki, Greece. CLEF CheckThat! Lab and FEVER are the most important global events devoted exclusively to the issue of automatic verification of fake news.
It is worth mentioning that in March 2023, the results of the OpenFact team regarding the detection of fake news using artificial intelligence for the Polish language were rated by the National Center for Research and Development as the best in Poland. Winning this prestigious competition confirms that the achievements of the OpenFact project team are globally relevant and that the methods developed by the OpenFact team are equally effective in other languages.
Members of the OpenFact project team: Prof. Witold Abramowicz, Prof. Krzysztof Wecel, Dr. Włodzimierz Lewoniewski, Dr. Piotr Stolarski, Dr. Milena Stróżyna, Ewelina Księżniak, Marcin Sawiński.
The OpenFact project is financed by the National Center for Research and Development under the INFOSTRATEG I program “Advanced information, telecommunications and mechatronic technologies”.