This track asks participants to match table metadata only, e.g., column names, to knowledge graphs without any access to table data and content. This is a challenging task due to the limited available context that could be used by annotation systems to perform the semantic linking. LLMs are a promising way to achieve such task that could be utilized in different ways.
Round 1 dataset consists of a select set of web table metadata that need to be mapped to DBpedia ontology.
Check out the README file for input/output format, sample input/output, and an evaluation script. Note that the output of the mapping is a JSONL file with each line containing a mapping of a column ID to an array of DBpedia property URIs and scores, which will be sorted in descending order by score for evaluation. Round 1 data has one mapping for each column, which is the most relevant property it maps to (e.g., if the column is about movie directors, the correct mapping should be https://dbpedia.org/ontology/director).
Round 2 dataset consists of a select set of open data table metadata that need to be mapped to a custom glossary (dictionary of term labels and descriptions).
Check out the README file for input/output format, sample input/output, and an evaluation script. Note that the output of the mapping is a JSONL file with each line containing a mapping of a column ID to an array of glossary items and scores, which will be sorted in descending order by score for evaluation. Similar to Round 1 data, Round 2 data has one mapping for each column, which is the most relevant glossary item it maps to. We acknowledge that there may be more than one relevant glossary item suitable for each column, which is why we use Hit@k scores for evaluation. We may use additional scores that are not included in the evaluate.py script for the final ranking of the submissions.
Submission: Submit up to 4 result sets for the test set using the following forms. You can participate in either round or both rounds: