Description
Since data quality matters this track opens a call for an open-ended question: how do we assess the quality of semantic table annotation (STA) datasets? Manual inspection could be a key solution to answer this question, but what about large-scale datasets? When each of which contains hundreds of thousands of tables? Random-based checks could be an alternative but would it be a good enough solution? What we think is a promising solution is an automated way that runs specific tests on a given dataset. It then yields insights.
Task
Please provide a system that can assess the quality level of an STI benchmark. Your system should be able to detect and display a list of potential data quality issues. We believe the most challenging part is determining what qualifies as a data issue.
Datasets
We use the same datasets in the Accuracy Track this year. The evaluation criteria will be announced later.