ICDAR 2023 Competition and CLEF 2023 Lab on Document Information Localization and Extraction

DocILE is a large-scale research benchmark for cross-evaluation of machine learning methods for Key Information Localization and Extraction (KILE) and Line Item Recognition (LIR) from semi-structured business documents such as invoices, orders etc. Such large-scale benchmark was previously missing (Skalický et al., 2022), hindering comparative evaluation.

Quick Links:
Join DocILE announcements google group to receive updates and news about the challenge
Form to get access to the dataset
Github repository
Dataset paper (Šimsa et al., 2023) with Supplementary Material
Make submissions to the benchmark via Robust Reading Competition website

Competition and Prizes

Official Results

Congratulations to all participating teams. Links to the competition overview paper and participants working notes will appear here once the CEUR Workshop Proceedings are ready.

	Method	Team/Affiliation
KILE: 1st place	GraphDoc (RRC)	USTC-iFLYTEK
KILE: 2nd place	YOLOv8 (RRC)	UWB
KILE: 3rd place	Union-RoBERTa (RRC)	UIT@AICLUB_TAB
LIR: 1st place	GraphDoc (RRC)	USTC-iFLYTEK
LIR: 2nd place	-	-
LIR: 3rd place	YOLOv8 (RRC)	UWB
Best Paper	GraphDoc (RRC)	USTC-iFLYTEK

Note that the 2nd place for the LIR task was not awarded as only one method managed to beat the baselines in the RRC benchmark.

Competition Information

The DocILE'23 competition is running as a CLEF 2023 lab and an ICDAR 2023 competition with a single leaderboard. The deadline for submissions is on May 10 May 24, 2023. The competition comes with a prize pool of $9000:

• Top three eligible teams on the KILE leaderboard will receive $2000, $1000 and $500 respectively.
• Top three eligible teams on the LIR leaderboard will receive $2000, $1000 and $500 respectively.
• A $2000 best-paper award, selected by the lab organizers and the steering committee.

In order to participate in the competition (and be eligible for prizes), you need to follow the competition rules. Most notably:

• You need to register using the CLEF Labs Registration Form.
• It is prohibited to use external document datasets and models trained on these datasets.

The competition/benchmark submissions will be handled via the DocILE page at the Robust Reading Competition website. The benchmark will stay live after the competition ends.

Dataset and Benchmark Paper

The dataset, the benchmark tasks and the evaluation criteria are described in detail in the dataset paper (Šimsa et al., 2023) which was accepted to ICDAR 2023. The provided link is to arXiv version that includes Supplementary Material. To cite the dataset, please use the following BibTeX entry:

@misc{simsa2023docile,
    title={{DocILE} Benchmark for Document Information Localization and Extraction},
    author={{\v{S}}imsa, {\v{S}}t{\v{e}}p{\'a}n and {\v{S}}ulc, Milan and U{\v{r}}i{\v{c}}{\'a}{\v{r}}, Michal and Patel, Yash and Hamdi, Ahmed and Koci{\'a}n, Mat{\v{e}}j and Skalick{\`y}, Maty{\'a}{\v{s}} and Matas, Ji{\v{r}}{\'\i} and Doucet, Antoine and Coustaty, Micka{\"e}l and Karatzas, Dimosthenis},
    url = {https://arxiv.org/abs/2302.05658},
    journal={arXiv preprint arXiv:2302.05658},
    year={2023}
}