Pollinator Identification: building an AI challenge
How do you test a model fairly when neighboring video frames look almost the same?
Summary
We designed and launched a machine learning competition on Codabench for recognizing pollinator species in camera footage of flowers. The hardest part was not the model but the test set: splitting the data so that a good score means the model recognizes insects, not camera angles.
Six frames in a row from one camera. A random train/test split would put almost identical images on both sides.
Key results
- 0.23baseline Macro F1 on camera angles seen in training
- 0.07the same model on angles it has never seen
- 0.74Macro F1 of our EfficientNet in the second challenge, up from 0.36
The problem
Different pollinators matter for different crops, and spotting invasive species early protects them, so farmers and researchers need models that can tell insects apart in camera footage. INRAE filmed flowers with up to 31 cameras from different angles, and the frames were turned into 1,536-number feature vectors with a pretrained ResNet. Our job was to turn this into a fair competition: a task, a dataset split, a metric, a baseline and a scoring program that other students could submit to.
What we did
- Measured how similar the frames are. Two frames of the same species from the same camera have a cosine similarity of about 0.79 on average and up to 0.99, so a random split would leak and even a simple nearest-neighbor model would look nearly perfect.
- Cleaned the labels: removed “no insect” (over 99% of frames), the vague “other” class, and two species filmed from only one angle, leaving 7 species.
- Split by species and camera angle instead of by frame, so a species is never tested on an angle it was trained on. Phase 1 tests on angles the model saw for other species; Phase 2 only on angles it never saw at all.
- Chose Macro F1 as the metric, so rare species like the bumblebee count as much as common ones.
- Ran a grid search over logistic regression, random forest, KNN and SVM with different PCA sizes, with bootstrapped confidence intervals. Picked a random forest on 100 PCA components as a simple, reproducible baseline.
- Deployed everything on Codabench (data, scoring and ingestion programs, starting kit) and ran the competition in two phases.
Results
The split did its job and exposed how hard the task is: the baseline drops from 0.23 to 0.07 Macro F1 on unseen angles. The confusion matrices show the model mostly predicts the most common species (hoverfly), and species filmed from only one training angle do not generalize.
In the second half of the course we solved another team’s challenge: classifying 8 wheat grain varieties from images. A random forest on raw pixels reached 0.36 Macro F1, a small CNN 0.44, and an EfficientNet pretrained on ImageNet 0.74.
What I learned
- The split decides what the score means. Splitting by angle instead of by frame turned an almost perfect score into an honest one.
- Choose the metric for the question. Macro F1 stops a model from looking good by ignoring rare species.
- Transfer learning wins on small image datasets. A pretrained network doubled the score of a model trained from scratch.