Fish Identification with Deep Learning & AI

Gramener and Microsoft AI for earth help
nisqually river foundation augment
fish identification by 73% accuracy
through deep learning AI models

Download this case study

Challenge

Earlier, biologists would manually identify and record ﬁsh species

Approach

Now, Deep Learning AI models have been trained to detect the ﬁsh

Outcome

AI-driven Image Recognition. 80% potential cost saving. Time taken reduced by 5X

A 2-minute summary of the project

THE PROBLEM

The manual process to identify and name any living species through captured videos is resource intensive from a time, human, and cost perspective. So, when the Nisqually River Foundation, a Washington based nature conservation organization, encountered a similar challenge to measure and monitor the Salmon species ﬁsh identiﬁcation, they approached us for an automated technology-driven solution.

AI-DRIVEN SOLUTION FOR FISH IDENTIFICATION

First, the collected video feeds were processed to extract the relevant frames. Deep learning AI models were then trained to draw bounding boxes around each ﬁsh passing by the camera. The entire workﬂow encapsulated in a Web App automated the process of video feed input, detection and classiﬁcation. The automated AI solution leveraged the latest implemented deep learning algorithms using the Microsoft Azure and Cognitive Services platform stack. Given the nature of the problem and the format of the video ﬁles, processing power was a key requirement for the training and validation phases. A GPU machine was the natural choice to run the object detection models. Hence, we selected the NC6 GPU VM in the Azure portal.

THE APPROACH

PARTNERING WITH MICROSOFT AI FOR EARTH TEAM

The Microsoft AI for Earth team was a key enabler and inﬂuencer for project’s success, through timely access for technical support and resolution of AI platform queries.

Technology Used to Resolve the Issue

For a reliable cloud solution with machine learning capabilities, Microsoft Azure Data Science Virtual Machine (DSVM) was chosen.
For the purpose of extracting frames from the videos and tag them, Microsoft Visual Object Tagging Tool (VOTT) came in handy.
The ﬁnal object detection algorithm chosen was the YOLO V3 Video detection algorithm.

Our Contribution to Augment the Process

The ﬁrst challenge was to process the videos and tag the ﬁsh. The heavy manual work involved in this was automated by leveraging the MS VOTT tool.
The tagged frames were then used to train a model using CNTK and Faster RCNN. This model was then tested against more frames extracted from the videos. While this solution was good, it lacked speed and real-time video detection capabilities.
As an enhancement to the solution, we moved to video object detection using YOLO V3, which provides a faster solution with real-time capabilities.

QUANTIFIABLE RESULTS FROM THE INTERVENTION

This web-based AI solution would save the client, valuable hours of expert biologist time and infrastructure costs spent in manually reviewing the videos. As part of a planned upgrade, an enhanced version of the solution has been provided to the customer, which predicts to deliver substantial cost savings to the tune of 80%.