Jersey Color Detection Accuracy — Round 2 Analysis

Summary Comparison

Key Findings

1. The constrained prompt is the best prompt for both models

The constrained vocabulary prompt delivered the strongest results across the board:

2. Exact match rates jumped significantly

The constrained prompt's biggest impact was converting similar matches into exact matches by forcing models to use the ground truth vocabulary:

This came partly from eliminating vocabulary mismatch (e.g., grey→gray, navy→navy blue) and partly from teaching models to use specific color terms like "maroon" and "light blue."

3. Targeted color improvements

The constrained prompt's explicit color guidance fixed the worst systematic errors:

• Maroon: Cut in half for both models. Previously the most-missed color for Qwen; now ranks 5th.
• Light blue: Near-elimination of the "light blue → blue" confusion for both models (7→1 for Qwen, 3→1 for Gemini).
• Gray/grey: The spelling normalization instruction eliminated the grey→gray similar-match penalty for Gemini entirely (10 confusions → 0). However, gray detection misses remain unchanged — these are images where gray jerseys aren't detected at all, not a naming issue.
• Teal and black remain stubbornly problematic regardless of prompt.

4. New overcorrection pattern with constrained prompt

• Qwen + Constrained reported "maroon" as an extra/wrong color 5 times (was 0 previously). It's now calling some brown and red jerseys "maroon" — the opposite of the original problem. Specific cases: 007 (brown→maroon), 031 (brown→maroon), 048 (red→maroon), 142 (orange→maroon).
• Gemini + Constrained reported "light blue" as an extra/wrong color 2 times (was 0 previously), including misidentifying navy blue as light blue (image 081).

This overcorrection is a smaller problem than the original misses it replaced, but worth noting.

5. The capstone prompt did not improve results

• Qwen: 78.2% recall (same), 87.3% precision (slight improvement)
• Gemini: 78.2% recall (down from 79.7%), 88.7% precision (down from 91.4%)

6. Gemini speed improvement from concurrency

The concurrent processing optimization (8 workers + session reuse + JPEG quality 85) delivered major speed gains:

That's roughly an 8x speedup for the first two prompts. The constrained prompt run was slightly slower (344s) due to its longer prompt text (2223 chars vs ~1500 chars).

Persistently Failed Images

These 10 images failed across all six runs, representing the hardest cases for current VLMs regardless of model or prompt:

Notable improvements: Images 029 (maroon), 087/141/161 (light blue), and 099 (maroon) were previously persistent failures but were fixed by the constrained prompt for at least one model.

Model Comparison

Recommendations

1. Use the constrained prompt (jersey_prompt_constrained.txt) — it is the clear winner for both models, improving recall and precision simultaneously.
2. Post-processing normalization could still recover additional matches: map grey → gray (catches any remaining Gemini outputs) and navy → navy blue (catches shorthand usage).
3. Consider a brown/maroon calibration — the constrained prompt overcorrected on Qwen, turning brown→maroon confusion into a new error source. Adding "Use 'brown' for warm, non-reddish dark colors" or similar guidance may help.
4. Gray and black detection remain unsolved at the prompt level — these are likely image quality or model perception limitations that no amount of prompt engineering will fix. These colors may benefit from a secondary computer vision pass (e.g., dominant color extraction from the jersey region).
5. Retire the capstone prompt — it offered no benefit over the original and performed worse than the constrained prompt in every metric.

Appendix: Color Similarity Families Used for Scoring

Appendix: Constrained Prompt (jersey_prompt_constrained.txt)

You are an expert at detecting sports jerseys in images. Carefully examine the provided image and identify all visible sports jerseys.

CRITICAL INSTRUCTIONS:
1. ONLY detect jerseys that are CLEARLY VISIBLE in the image
2. ONLY include jersey numbers that you can ACTUALLY READ in the image
3. If you CANNOT see any jerseys, you MUST return {"jerseys": []}
4. DO NOT make up, imagine, or guess jersey numbers that aren't visible
5. DO NOT include jerseys if you cannot clearly see the number

COLOR VOCABULARY:
For "jersey_color" and "number_color", you MUST choose from this list ONLY:
red, blue, dark blue, navy blue, light blue, green, yellow, gold, orange, purple, black, white, gray, brown, dark brown, maroon, teal, pink

Important color distinctions:
- Use "maroon" for dark brownish-red, NOT "red"
- Use "light blue" for pale or sky blue, NOT "blue"
- Use "navy blue" for very dark blue, NOT "blue" or "dark blue"
- Use "teal" for blue-green, NOT "green" or "blue"
- Use "gray" (not "grey") for silver or neutral tones
- Use "dark brown" for very dark brown, NOT "black"
- Use "gold" for metallic or deep yellow, NOT "yellow"

RESPONSE FORMAT:
Respond ONLY with a valid JSON object. No explanations, no markdown, no extra text.

Use DOUBLE QUOTES (") for all JSON keys and string values.

The JSON must have a single key "jerseys" with an array of dictionaries.

Each dictionary must have exactly these three keys:
- "jersey_number": The number on the jersey (as a string, only if clearly visible)
- "jersey_color": The primary color of the jersey (MUST be from the color list above)
- "number_color": The color of the number on the jersey (MUST be from the color list above)

Example response for an image WITH visible jerseys:
{
  "jerseys": [
    {
      "jersey_number": "10",
      "jersey_color": "maroon",
      "number_color": "gold"
    },
    {
      "jersey_number": "42",
      "jersey_color": "light blue",
      "number_color": "white"
    }
  ]
}

Example response for an image WITHOUT jerseys or with unclear numbers:
{"jerseys": []}

REMEMBER: Only include jerseys with numbers you can ACTUALLY SEE in the image. When in doubt, return empty array.

Now analyze the image and return the JSON object.