Gramener and Microsoft AI for Earth help Nisqually River Foundation augment ﬁsh identiﬁcation by 73% accuracy through Deep Learning AI models
A 2-minute Summary of the Project
To identify Salmon species in the Nisqually River, the Nisqually Indian Tribe has installed one video camera and infrared sensors in a ﬁsh ladder at a dam on the river. The camera is triggered to capture 30 seconds of video when any ﬁsh swims past the infrared sensors. Then the recorded videos are scrutinized manually to identify and name the ﬁsh species in it.
The Business Challenge
The manual process to identify and name any living species through captured videos is resource intensive from a time, human, and cost perspective. So, when the Nisqually River Foundation, a Washington based nature conservation organization, encountered a similar challenge to measure and monitor the Salmon species ﬁsh identiﬁcation, they approached us for an automated technology-driven solution.
AI-Driven Solution for Fish Identification
First, the collected video feeds were processed to extract the relevant frames. Deep learning AI models were then trained to draw bounding boxes around each ﬁsh passing by the camera. The entire workﬂow encapsulated in a Web App automated the process of video feed input, detection and classiﬁcation. The automated AI solution leveraged the latest implemented deep learning algorithms using the Microsoft Azure and Cognitive Services platform stack.
Given the nature of the problem and the format of the video ﬁles, processing power was a key requirement for the training and validation phases. A GPU machine was the natural choice to run the object detection models. Hence, we selected the NC6 GPU VM in the Azure portal.
Partnering with Microsoft AI for Earth Team
The Microsoft AI for Earth team was a key enabler and inﬂuencer for project’s success, through timely access for technical support and resolution of AI platform queries.
Technology Used to Resolve the Issue
- For a reliable cloud solution with machine learning capabilities, Microsoft Azure Data Science Virtual Machine (DSVM) was chosen.
- For the purpose of extracting frames from the videos and tag them, Microsoft Visual Object Tagging Tool (VOTT) came in handy.
- The ﬁnal object detection algorithm chosen was the YOLO V3 Video detection algorithm.
Our Contribution to Augment the Process
- The ﬁrst challenge was to process the videos and tag the ﬁsh. The heavy manual work involved in this was automated by leveraging the MS VOTT tool.
- The tagged frames were then used to train a model using CNTK and Faster RCNN. This model was then tested against more frames extracted from the videos. While this solution was good, it lacked speed and real-time video detection capabilities.
- As an enhancement to the solution, we moved to video object detection using YOLO V3, which provides a faster solution with real-time capabilities.
Quantiﬁable Results from the Intervention
This web-based AI solution would save the client, valuable hours of expert biologist time and infrastructure costs spent in manually reviewing the videos. As part of a planned upgrade, an enhanced version of the solution has been provided to the customer, which predicts to deliver substantial cost savings to the tune of 80%.
Our Work Mentioned in Science Journals
- Elsevier: MSR-YOLO: Method to Enhance Fish Detection and Tracking in Fish Farms
- Georgia State University Scholarworks: Technology Enabled Social Responsibility Projects and an Empirical Test of CSR's Impact on Firm Performance