48d9145810
- Add -e/--embedding-model parameter to Key Parameters table - Add --clear-cache parameter - Document all 3 test scripts with output file table - Update project structure with new scripts and analysis doc - Expand Models section with embedding model options table - Add note about clearing cache when switching models - Add test_results_analysis.md for documenting test findings
144 lines
6.2 KiB
Markdown
144 lines
6.2 KiB
Markdown
# Logo Detection Test Results Analysis
|
|
|
|
This document provides analysis of logo detection test results across different matching methods and configurations.
|
|
|
|
---
|
|
|
|
## Test Run: CLIP Defaults with All Matching Methods
|
|
|
|
**Date**: 2025-12-31
|
|
**Embedding Model**: openai/clip-vit-large-patch14 (default)
|
|
|
|
### Test Configuration
|
|
|
|
| Parameter | Value |
|
|
|-----------|-------|
|
|
| Reference logos | 20 |
|
|
| Refs per logo | 10 |
|
|
| Total reference embeddings | 189 |
|
|
| Positive samples per logo | 20 |
|
|
| Negative samples per logo | 100 |
|
|
| Test images processed | ~2,350 |
|
|
| Similarity threshold | 0.70 |
|
|
| DETR threshold | 0.50 |
|
|
| Margin | 0.05 |
|
|
| Min matching refs | 3 |
|
|
| Random seed | 42 |
|
|
|
|
### Results Summary
|
|
|
|
| Method | TP | FP | FN | Precision | Recall | F1 |
|
|
|--------|---:|---:|---:|----------:|-------:|---:|
|
|
| Simple | 751 | 58,221 | 9 | 1.3% | 203.5%* | 2.5% |
|
|
| Margin | 60 | 26 | 310 | 69.8% | 16.3% | 26.4% |
|
|
| Multi-ref (mean) | 233 | 217 | 170 | 51.8% | 63.1% | 56.9% |
|
|
| Multi-ref (max) | 278 | 259 | 136 | 51.8% | 75.3% | 61.4% |
|
|
|
|
*Recall >100% indicates multiple true positive detections per expected logo (multiple detected regions matching the same logo).
|
|
|
|
### Analysis by Method
|
|
|
|
#### Simple Matching
|
|
|
|
The simple method returns ALL logos above the similarity threshold without any rejection logic. This serves as a baseline to understand the raw discriminative power of CLIP embeddings.
|
|
|
|
**Observations**:
|
|
- 58,221 false positives vs 751 true positives (~78:1 ratio)
|
|
- At threshold 0.70, CLIP embeddings are not discriminative enough to distinguish between different logos
|
|
- The extremely high false positive count indicates that unrelated logo regions frequently produce similarity scores above 0.70
|
|
- This method is unsuitable for production use but valuable for understanding the embedding space
|
|
|
|
#### Margin-Based Matching
|
|
|
|
The margin method requires the best match to exceed the second-best by a minimum margin (0.05), rejecting ambiguous matches.
|
|
|
|
**Observations**:
|
|
- Highest precision (69.8%) but very low recall (16.3%)
|
|
- Only 60 true positives out of 369 expected
|
|
- The margin requirement is too strict when using multiple references per logo
|
|
- With 10 refs per logo, references from the SAME logo compete with each other
|
|
- Example: If Logo A has refs scoring 0.85 and 0.84, the margin is only 0.01, causing rejection
|
|
- This explains why margin matching produces fewer matches than multi-ref methods
|
|
|
|
#### Multi-Ref Matching (Mean Similarity)
|
|
|
|
Uses the average similarity across all reference images for each logo.
|
|
|
|
**Observations**:
|
|
- Balanced precision (51.8%) and recall (63.1%)
|
|
- F1 score of 56.9%
|
|
- False positive ratio approximately 1:1 with true positives (217 FP vs 233 TP)
|
|
- Mean aggregation penalizes logos where some references don't match well
|
|
- More conservative than max aggregation
|
|
|
|
#### Multi-Ref Matching (Max Similarity)
|
|
|
|
Uses the highest similarity score from any single reference image.
|
|
|
|
**Observations**:
|
|
- Best F1 score (61.4%) and recall (75.3%)
|
|
- Same precision as mean method (51.8%)
|
|
- 278 true positives vs 259 false positives (still approximately 1:1)
|
|
- Max aggregation is more lenient, improving recall at no precision cost
|
|
- Better suited when reference images capture different logo variants
|
|
|
|
### Key Findings
|
|
|
|
#### 1. CLIP Embedding Similarity Distribution
|
|
|
|
The simple matching results reveal a fundamental issue: at threshold 0.70, the CLIP embedding space does not provide sufficient separation between different logos. The 78:1 false positive to true positive ratio indicates that:
|
|
|
|
- Many unrelated images produce high cosine similarity scores
|
|
- The threshold would need to be significantly higher (0.85+) to reduce false positives
|
|
- Even then, recall would likely suffer
|
|
|
|
#### 2. Margin Method Limitation with Multiple References
|
|
|
|
The margin-based matching method was designed assuming one reference per logo. When using multiple references (10 per logo in this test), references from the same logo compete against each other in the margin calculation. This causes legitimate matches to be rejected when two references from the same logo have similar scores.
|
|
|
|
#### 3. False Positive Rate Remains High
|
|
|
|
Even the best-performing method (multi-ref max) produces nearly as many false positives as true positives:
|
|
- 278 correct matches
|
|
- 259 incorrect matches
|
|
- This 1:1 ratio is problematic for production use cases
|
|
|
|
#### 4. Trade-off Between Precision and Recall
|
|
|
|
| Goal | Best Method | Trade-off |
|
|
|------|-------------|-----------|
|
|
| Maximize precision | Margin | Very low recall (16.3%) |
|
|
| Maximize recall | Multi-ref (max) | Lower precision (51.8%) |
|
|
| Balance both | Multi-ref (max) | Best F1 but still ~50% precision |
|
|
|
|
### Deficiencies of This Approach
|
|
|
|
#### CLIP Model Limitations
|
|
|
|
1. **General-Purpose Training**: CLIP was trained on text-image pairs for general visual understanding, not for fine-grained logo discrimination. Logo matching requires distinguishing between visually similar brand marks, which CLIP's training objective doesn't optimize for.
|
|
|
|
2. **Embedding Space Density**: The cosine similarity scores cluster in a narrow range (0.6-0.9 for most images), making threshold-based discrimination difficult. Small differences in embedding similarity don't reliably indicate visual differences.
|
|
|
|
3. **Scale and Context Sensitivity**: CLIP embeddings are affected by the context around detected regions. A logo on a busy background may produce different embeddings than the same logo on a clean background.
|
|
|
|
4. **No Logo-Specific Features**: CLIP doesn't learn features specific to logo recognition such as:
|
|
- Typography and font shapes
|
|
- Brand-specific color combinations
|
|
- Geometric patterns and symmetry
|
|
- Edge and contour characteristics
|
|
|
|
#### Detection Pipeline Issues
|
|
|
|
1. **DETR Detection Quality**: The pipeline assumes DETR correctly identifies logo regions. Detection errors (missed logos, partial detections, non-logo regions) propagate to the matching stage.
|
|
|
|
2. **Cropping Artifacts**: Detected regions are cropped and resized before embedding extraction. This may introduce artifacts that affect embedding quality.
|
|
|
|
3. **Threshold Sensitivity**: The entire system is highly sensitive to the similarity threshold parameter. A 0.05 change in threshold can dramatically alter precision/recall balance.
|
|
|
|
---
|
|
|
|
## Test Run: [Next Test Name]
|
|
|
|
*Results pending...*
|
|
|
|
--- |