Desert Lion Conservation
2023
Automatic species identification in the Namib Desert biome
Desert Lion Conservation is a non-profit organisation dedicated to the conservation of desert-adapted lions in the Northern Namib. Their main focus is to collect base-line ecological data on the lion population and to study their behaviour, biology and adaptation to survive in the harsh environment. They use this information to collaborate with other conservation bodies in the quest to find a solution to human-lion conflict, to elevate the tourism value of lions, and to contribute to the conservation of the species.
Addax Data Science, in partnership with Smart Parks, has been asked to develop a species recognition model to identify 30 species or higher level taxons present in the Skeleton Coast National Park. The model was trained on a set of more than 850,000 images, of which ± 20% were provided by the client (see fig. 2). The rest was sourced from comparable ecological projects.
As you can see in figure 2, the dataset is highly imbalanced. The largest class, ‘cattle’, consists of more than a third of the entire dataset. Nevertheless, through the application of a technique which combines up- and downsampling, the training process successfully achieved equal weighting for each class. The resulting classification model has an overall validation accuracy of 95.3%, a precision of 95.4%, and a recall of 95.3%.
Precision focuses on the reliability of positive predictions, while recall focuses on the probability of capturing all relevant positive instances. To illustrate, consider an image with 10 lions. If the model predicts 11 lions, the precision would be 91% (10/11, with one false prediction) and the recall would be 100% (10/10, correctly identifying all 10 lions). Now, envision the same image, but the model predicts 9 lions. This results in a precision of 100% (9/9, with all predictions being correct) and a recall of 90% (9/10, missing one lion).
In most projects, both precision and recall hold significance. This is why accuracy is often mentioned, as it is a balance between both. Refer to table 1 below for a detailed breakdown of the model’s class-specific metrics. A comprehensive discussion of the results follows afterwards. A large majority of the classes reach validation metrics above 90%.
| Class | Precision | Recall | Accuracy |
|---|---|---|---|
| aardwolf | 84.9% | 93.5% | 89.0% |
| african wild cat | 98.9% | 98.5% | 98.7% |
| baboon | 93.9% | 96.5% | 95.2% |
| bird | 92.1% | 92.9% | 92.5% |
| brown hyaena | 96.3% | 97.6% | 97.0% |
| caracal | 75.0% | 85.7% | 80.0% |
| cattle | 96.2% | 99.1% | 97.6% |
| cheetah | 88.8% | 95.9% | 92.2% |
| donkey | 93.0% | 67.7% | 78.4% |
| elephant | 97.3% | 95.5% | 96.4% |
| fox | 80.4% | 77.4% | 78.9% |
| gemsbok | 94.4% | 96.1% | 95.3% |
| genet | 87.5% | 50.0% | 63.6% |
| giraffe | 97.5% | 96.9% | 97.2% |
| hare | 98.6% | 97.5% | 98.0% |
| honey badger | 70.7% | 78.4% | 74.4% |
| hyrax | 94.1% | 100.0% | 97.0% |
| jackal | 83.5% | 90.5% | 86.9% |
| klipspringer | 77.8% | 93.3% | 84.8% |
| kudu | 90.8% | 98.8% | 94.6% |
| leopard | 93.4% | 81.4% | 87.0% |
| lion | 96.8% | 96.6% | 96.7% |
| mongoose | 76.5% | 89.5% | 82.5% |
| ostrich | 92.9% | 94.9% | 93.9% |
| porcupine | 98.0% | 96.5% | 97.2% |
| rhinoceros | 89.1% | 94.8% | 91.9% |
| spotted hyaena | 96.4% | 92.9% | 94.6% |
| springbok | 91.5% | 90.3% | 90.9% |
| steenbok | 85.7% | 95.0% | 90.1% |
| zebra | 98.1% | 93.5% | 95.7% |
Validation metrics calculated during training, however, may not necessarily serve as a reliable guide for future predictions. To ensure that the model is flexible and performs well in scenarios beyond the training dataset, conducting an out-of-sample validation is helpful. Out-of-sample images refer to those captured in contexts that were never encountered during the training phase. These may include images from new locations, different seasons, and so on. In this project, we gathered an out-of-sample dataset from the previously unseen year 2023. The metrics derived from this dataset are slightly lower, with an overall accuracy of 93.4%, a precision of 93.8%, and a recall of 93.2%.
Below, you can see a visual representation of the model’s confusion matrix (fig. 3). It tabulates the predicted species against the actual species in a format that allows for a quick assessment of how well the model is performing. The matrix provides insight into which species are often confused by the model. For example, the graph indicates that about 5% of the foxes in the dataset are misclassified as jackals. Even though combining these animals into one group called ‘canid’ would make the model perform better, we’ve chosen to keep as much species separate as possible. This way, we provide more detailed information for the end user, even if there’s some overlap in the model’s predictions.
Let’s delve further into the relatively low-performing classes. What causes the accuracy of certain classes to drop below 90%? Besides potential factors like the distinctiveness of the species, including morphology and behaviour, there is a clear pattern of limited availability of training data. Figure 4 illustrates that all classes with an accuracy below 90% (indicated by the vertical dashed line) also have fewer than 7000 images available (marked by the horizontal dashed line). However, this doesn’t necessarily mean that classes with less than 7000 images are destined to score below 90%. A third of the classes below the 7000 image threshold still achieve above 90% accuracy.
The availability of training images is not the sole factor affecting class accuracy, but a large dataset certainly contributes to a higher accurately. In this project, all classes with more than 7000 images performed above 90% accuracy.
The model described here is freely available through our open-source camera trap analysis platform, EcoAssist.
