logo_test/README.md

Logo Detection Test Framework

A testing framework for evaluating logo detection accuracy using DETR (DEtection TRansformer) and CLIP (Contrastive Language-Image Pre-training) models.

Burnley Test: Averaged Embeddings with DINOv2

A targeted test using DetectLogosEmbeddings to detect two specific logos (barnfield and vertu) in 516 Burnley match images. Reference embeddings are averaged across all images in each reference directory, and matching uses margin-based comparison (margin=0.05).

Test command:

uv run python test_burnley_detection.py -e dinov2 -t 0.7 --margin 0.05 --output-file results_average_embeddings.txt

Results (DINOv2, threshold 0.70, margin 0.05):

Metric	Value
True Positives	28
False Positives	36
False Negatives	125
Total Expected	146
Precision	43.8%
Recall	19.2%
F1 Score	26.7%

Ground truth is derived from filename prefixes: vertu_ (vertu logo), barnfield_ (barnfield logo), barnfield+vertu_ (both logos). Images without these prefixes are treated as negatives.

Low recall suggests many logos go undetected by DETR or fall below the similarity threshold. The relatively low precision indicates DINOv2 averaged embeddings struggle to discriminate between the two logos in this domain. Further tuning of thresholds, margin, and embedding model (e.g. CLIP or SigLIP) may improve results.

---

Recommended Settings

Based on extensive testing with the LogoDet-3K dataset, these are the optimal settings:

Parameter	Recommended Value	Notes
Matching Method	multi-ref	Best balance of precision and recall
Similarity Aggregation	max (default)	Max outperforms mean aggregation
Embedding Model	openai/clip-vit-large-patch14	Significantly outperforms DINOv2
CLIP Threshold	0.70	Good precision/recall balance
DETR Threshold	0.50	Default detection confidence
Margin	0.05	Reduces false positives
Refs per Logo	7-10	More references = better accuracy
Preprocessing	default	Best precision; letterbox/stretch hurt precision

Example command with recommended settings:

uv run python test_logo_detection.py \
    --matching-method multi-ref \
    --refs-per-logo 10 \
    --threshold 0.70 \
    --margin 0.05 \
    --use-max-similarity

Performance Benchmarks

With recommended settings (multi-ref max, threshold 0.70, margin 0.05):

Refs/Logo	Precision	Recall	F1 Score
1	45.8%	65.9%	54.0%
3	40.5%	72.4%	51.9%
5	47.2%	72.6%	57.2%
7	51.0%	79.9%	62.3%
10	50.2%	81.6%	62.1%

Key findings:

• More reference images per logo consistently improves recall
• 7+ refs provides the best precision/recall balance
• Diminishing returns beyond 10 refs

Matching Method Comparison

Method	Precision	Recall	F1	Use Case
simple	1.3%	203%*	2.5%	Not recommended (too many FPs)
margin	69.8%	16.3%	26.4%	High precision, low recall
multi-ref (mean)	51.8%	63.1%	56.9%	Balanced
multi-ref (max)	51.8%	75.3%	61.4%	Best overall

*Simple method returns all matches above threshold, causing many duplicates.

Embedding Model Comparison

Model	Precision	Recall	F1	Recommendation
openai/clip-vit-large-patch14	49.1%	77.0%	59.9%	Recommended
facebook/dinov2-small	22.4%	42.8%	29.5%	Not recommended
facebook/dinov2-large	32.2%	28.5%	30.2%	Not recommended

CLIP significantly outperforms DINOv2 for logo matching tasks.

Preprocessing Mode Comparison

Mode	Precision	Recall	F1	Notes
default	50.2%	81.6%	62.1%	Recommended - best precision
letterbox	42.4%	119%*	62.6%	Higher recall but worse precision
stretch	34.5%	113%*	52.9%	Not recommended

*Recall >100% indicates multiple detections per expected logo.

Recommendation: Use default preprocessing. While letterbox shows marginally higher F1, it has significantly worse precision (more false positives).

---

Overview

This project provides tools to:

• Detect logos in images using a fine-tuned DETR model
• Match detected logos against reference images using CLIP embeddings
• Evaluate detection accuracy with precision, recall, and F1 metrics

Architecture

The system uses a two-stage pipeline:

1. DETR - Identifies potential logo regions (bounding boxes) in images
2. CLIP - Extracts feature embeddings for each detected region and compares against reference logos

Installation

Requires Python 3.12+. Uses uv for package management.

# Install dependencies
uv sync

# Or using pip
pip install -r requirements.txt

Usage

Prepare Test Data

The test framework requires the LogoDet-3K dataset. Download it and place it in the project directory:

logo_test/
├── LogoDet-3K/           # Dataset directory (required)
│   ├── Clothes/          # Category directories
│   │   ├── Adidas/       # Brand directories with images + XML annotations
│   │   ├── Nike/
│   │   └── ...
│   ├── Electronic/
│   ├── Food/
│   └── ...

The dataset should contain images with corresponding Pascal VOC format XML annotation files that define logo bounding boxes.

Then run the preparation script:

uv run python prepare_test_data.py

This script:

1. Scans LogoDet-3K/ for images and XML annotation files
2. Extracts cropped logo regions using bounding box data → saves to reference_logos/
3. Copies full images → saves to test_images/
4. Creates test_data_mapping.db SQLite database with ground truth mappings

Run Detection Tests

# Basic test with default settings (margin-based matching)
uv run python test_logo_detection.py

# Test with more logos and custom threshold
uv run python test_logo_detection.py -n 20 --threshold 0.75

# Use multi-ref matching method
uv run python test_logo_detection.py --matching-method multi-ref \
    --refs-per-logo 5 --min-matching-refs 2

# Reproducible test with seed
uv run python test_logo_detection.py -n 50 --seed 42

Key Parameters

Parameter	Default	Description
-n, --num-logos	10	Number of reference logos to sample
-t, --threshold	0.7	Similarity threshold for matching
-d, --detr-threshold	0.5	DETR detection confidence threshold
-e, --embedding-model	openai/clip-vit-large-patch14	Embedding model (CLIP or DINOv2)
--matching-method	margin	Matching method: simple, margin, or multi-ref
--margin	0.05	Margin over second-best match (margin/multi-ref)
--refs-per-logo	3	Reference images per logo
--min-matching-refs	1	Min refs that must match (multi-ref only)
--use-max-similarity	False	Use max instead of mean similarity (multi-ref only)
--positive-samples	5	Positive test images per logo
--negative-samples	20	Negative test images per logo
-s, --seed	None	Random seed for reproducibility
--output-file	None	Append results summary to file (clean output)
--clear-cache	False	Clear embedding cache before running

Matching Methods:

• simple - Returns all logos above threshold (not recommended - too many false positives)
• margin - Requires margin over second-best match (high precision, low recall)
• multi-ref - Recommended. Aggregates scores across multiple reference images per logo

See --help for all options.

Run Comparison Tests

# Compare all matching methods
./run_comparison_tests.sh

# Test various threshold/margin combinations
./run_threshold_tests.sh

# Compare embedding models (CLIP vs DINOv2)
./run_model_comparison.sh

# Test different refs-per-logo values
./run_refs_per_logo_test.sh

Script	Purpose	Output File
run_comparison_tests.sh	Compare matching methods	test_results/comparison_*.txt
run_threshold_tests.sh	Test threshold/margin combinations	test_results/threshold_*.txt
run_model_comparison.sh	Compare CLIP vs DINOv2 models	test_results/model_comparison_results.txt
run_refs_per_logo_test.sh	Test refs-per-logo values	test_results/refs_per_logo_analysis.txt
run_preprocess_test.sh	Compare preprocessing modes	test_results/preprocessing_comparison.txt

Project Structure

logo_test/
├── logo_detection_detr.py      # Core detection library (DetectLogosDETR class)
├── test_logo_detection.py      # Test script for accuracy evaluation
├── prepare_test_data.py        # Script to prepare test database
├── run_comparison_tests.sh     # Compare all matching methods
├── run_threshold_tests.sh      # Test threshold/margin combinations
├── run_model_comparison.sh     # Compare CLIP vs DINOv2 models
├── test_data_mapping.db        # SQLite database with ground truth
├── reference_logos/            # Reference logo images (not in git)
├── test_images/                # Test images (not in git)
├── LogoDet-3K/                 # Source dataset (not in git)
├── logo_detection_detr_usage.md        # API usage guide
├── logo_detection_test_methodology.md  # Test methodology documentation
└── test_results_analysis.md    # Analysis of test results

Accuracy Improvement Techniques

The framework implements several techniques to improve detection accuracy:

1. Non-Maximum Suppression (NMS) - Removes overlapping duplicate detections
2. Minimum Box Size Filtering - Filters out noise from tiny detections
3. Confidence Threshold Filtering - Removes low-confidence detections
4. Multiple Reference Images - Uses multiple refs per logo for robust matching
5. Margin-Based Matching - Requires confidence margin over second-best match
6. Multi-Ref Matching - Aggregates similarity scores across references
7. Embedding Caching - Caches embeddings to avoid recomputation

Models

Detection Model

• DETR: Pravallika6/detr-finetuned-logo-detection_v2

Embedding Models (selectable via -e/--embedding-model)

Model	Type	Description
openai/clip-vit-large-patch14	CLIP	Default. General-purpose vision-language model
openai/clip-vit-base-patch32	CLIP	Smaller, faster CLIP variant
facebook/dinov2-small	DINOv2	Self-supervised, good for visual similarity
facebook/dinov2-base	DINOv2	Larger DINOv2 variant
facebook/dinov2-large	DINOv2	Largest DINOv2 variant

Models are automatically downloaded from HuggingFace on first run and cached in ~/.cache/huggingface/.

Note: When switching between embedding models, use --clear-cache to ensure embeddings are recomputed with the new model.

Documentation

• API Usage Guide - How to use the DetectLogosDETR class
• Test Methodology - Detailed explanation of test framework and tuning

License

MIT