Evaluate Docling on various datasets.
Evaluate docling on various datasets. You can use the cli
docling-eval % poetry run evaluate --help
Usage: evaluate [OPTIONS]
╭─ Options ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ * --task -t [create|evaluate|visualize] Evaluation task [default: None] [required] │
│ * --modality -m [end-to-end|layout|table_structure|code_transcription|math Evaluation modality [default: None] [required] │
│ _transcription|reading_order|markdown_text|captioning|bbox │
│ es_text] │
│ * --benchmark -b [DPBench|OmniDocBench|WordScape|PubLayNet|DocLayNetV1|DocL Benchmark name [default: None] [required] │
│ ayNetV2|FUNSD|Pub1M|PubTabNet|FinTabNet|WikiTabNet] │
│ * --output-dir -o PATH Output directory [default: None] [required] │
│ --input-dir -i PATH Input directory [default: None] │
│ --converter_type -c [Docling|SmolDocling] Type of document converter [default: Docling] │
│ --split -s TEXT Dataset split [default: test] │
│ --artifacts-path -a PATH Load artifacts from local path [default: None] │
│ --max-items -n INTEGER How many items to load from the original dataset │
│ [default: 1000] │
│ --help Show this message and exit. │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯- DP-Bench benchmarks: Text, layout, reading order and table structure evaluation on the DP-Bench dataset.
- OmniDocBench benchmarks: Text, layout, reading order and table structure evaluation on the OmniDocBench dataset.
- DocLayNetV1 Benchmarks: Text and layout evaluation on the DocLayNet v1.2 dataset.
- FinTabnet Benchmarks: Table structure evaluation on the FinTabNet dataset.
- PubTabNet benchmarks: Table structure evaluation on the PubTabNet dataset.
- Pub1M benchmarks: Table structure evaluation on the Pub1M dataset.
Please read Contributing to Docling for details.
The Docling codebase is under MIT license. For individual model usage, please refer to the model licenses found in the original packages.
Docling-eval has been brought to you by IBM.