Automatic transparency evaluation for open knowledge extraction systems

Journal of Biomedical Semantics · 3 authors, 2 centres

AI SUMMARY

FIDELITY 100%

This summary was generated by AI from a single paper. It has not been reviewed by a clinician and is not clinical advice. Verify against the source before acting on it.

AGENT 01 · STRUCTURED SUMMARY

TRACE TO SOURCE

“

TL;DR

It benchmarked three OKE systems on the CORD-19 corpus, finding that Covid-on-the-Web had the highest mean transparency score.

Full summary

571 CHARS

The authors evaluated the transparency of three linked datasets generated by state-of-the-art OKE systems (CORD-19-on-FHIR, Covid-on-the-Web, COVID-KG) from the same CORD-19 corpus. The evaluation was automated using a combination of three FAIRness assessment tools and the Luzzu linked data quality framework, assessing six transparency dimensions including provenance, interpretability, and licensing. The work demonstrates that transparency differences between OKE systems can be automatically quantified, which has implications for trustworthy AI and data governance.

STUDY TYPE

other

SPECIALTY

research methods

SUMMARISED BY

AI pipeline

FIDELITY CHECK

100% · A

ORIGINAL PAPER

PMC10468861

Automatic transparency evaluation for open knowledge extraction systems

Journal of Biomedical Semantics

READ THE FULL PAPER

ABSTRACT

Abstract Background This paper proposes Cyrus, a new transparency evaluation framework, for Open Knowledge Extraction (OKE) systems. Cyrus is based on the state-of-the-art transparency models and linked data quality assessment dimensions. It brings together a comprehensive view of transparency dimensions for OKE systems. The Cyrus framework is used to evaluate the transparency of three linked datasets, which are built from the same corpus by three state-of-the-art OKE systems. The evaluation is automatically performed using a combination of three state-of-the-art FAIRness (Findability, Accessibility, Interoperability, Reusability) assessment tools and a linked data quality evaluation framework, called Luzzu. This evaluation includes six Cyrus data transparency dimensions for which existing assessment tools could be identified. OKE systems extract structured knowledge from unstructured or semi-structured text in the form of linked data. These systems are fundamental components of advanced knowledge services. However, due to the lack of a transparency framework for OKE, most OKE systems are not transparent. This means that their processes and outcomes are not understandable and interpretable. A comprehensive framework sheds light on different aspects of transparency, allows comparison between the transparency of different systems by supporting the development of transparency scores, gives insight into the transparency weaknesses of the system, and ways to improve them. Automatic transparency evaluation helps with scalability and facilitates transparency assessment. The transparency problem has been identified as critical by the European Union Trustworthy Artificial Intelligence (AI) guidelines. In this paper, Cyrus provides the first comprehensive view of transparency dimensions for OKE systems by merging the perspectives of the FAccT (Fairness, Accountability, and Transparency), FAIR, and linked data quality research communities. Results In Cyrus, data transparency includes ten dimensions which are grouped in two categories. In this paper, six of these dimensions, i.e., provenance, interpretability, understandability, licensing, availability, interlinking have been evaluated automatically for three state-of-the-art OKE systems, using the state-of-the-art metrics and tools. Covid-on-the-Web is identified to have the highest mean transparency. Conclusions This is the first research to study the transparency of OKE systems that provides a comprehensive set of transparency dimensions spanning ethics, trustworthy AI, and data quality approaches to transparency. It also demonstrates how to perform automated transparency evaluation that combines existing FAIRness and linked data quality assessment tools for the first time. We show that state-of-the-art OKE systems vary in the transparency of the linked data generated and that these differences can be automatically quantified leading to potential applications in trustworthy AI, compliance, data protection, data governance, and future OKE system design and testing. Keywords: Transparency framework, Automatic transparency evaluation, Open knowledge extraction, FAIRness assessment, Quality evaluation

CITATION EXPORT

Maryam Basereh, Annalina Caputo, Rob Brennan. Automatic transparency evaluation for open knowledge extraction systems. Journal of Biomedical Semantics. (2023). https://doi.org/10.1186/s13326-023-00293-9

AUTHORS

3 · 2 CENTRES

Maryam Basereh, Annalina Caputo, Rob Brennan

1School of Computing, Dublin City University, Dublin, Ireland
2ADAPT Centre, School of Computer Science, University College Dublin, Dublin, Ireland