qlyoung's wiki

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
bird_bar [2023/09/09 00:19] – [Thoughts] qlyoungbird_bar [2024/07/03 19:16] (current) – change ai tag to machine learning qlyoung
Line 1: Line 1:
-At the start of 2021 I received a window-mount bird feeder as a secret santa gift (thank you!). As someone who loves birds I was excited to put it up and get a close up view of some of the birds that inhabit the woods around where I live. It’s a great little feeder and within around 3 days I had birds showing up regularly.+====== bird bar ======
  
-{{:feeder-with-chickadee.jpg?200|Bird feeder with Carolina chickadee perched on it}}+{{ :feeder-with-chickadee.jpg?200|Bird feeder with Carolina chickadee perched on it}} 
 + 
 +At the start of 2021 I received a window-mount bird feeder as a secret santa gift (thank you!). As someone who loves birds I was excited to put it up and get a close up view of some of the birds that inhabit the woods around where I live. It’s a great little feeder and within around 3 days I had birds showing up regularly.
  
 Shortly after installing the feeder I had the idea to mount a camera pointing at it and stream it to Twitch, so that I could watch the birds while I was at my computer. While watching I found myself wondering about a few of the species I saw, and looking up pictures trying to identify them. Then it hit me - this is a textbook computer vision problem. I could build something that used realtime computer vision to identify birds as they appeared on camera. Shortly after installing the feeder I had the idea to mount a camera pointing at it and stream it to Twitch, so that I could watch the birds while I was at my computer. While watching I found myself wondering about a few of the species I saw, and looking up pictures trying to identify them. Then it hit me - this is a textbook computer vision problem. I could build something that used realtime computer vision to identify birds as they appeared on camera.
Line 17: Line 19:
 {{:feeder-with-camera.jpg?400|Bird feeder showing webcam pointed at it}} {{:feeder-with-camera-outside.jpg?200|Picture of webcam attached to the side of my apartment building}} {{:webcam-condom.jpg?200|Picture of the webcam completely wrapped in plastic wrap secured by orange duct tape sitting on my window sill}} {{:feeder-with-camera.jpg?400|Bird feeder showing webcam pointed at it}} {{:feeder-with-camera-outside.jpg?200|Picture of webcam attached to the side of my apartment building}} {{:webcam-condom.jpg?200|Picture of the webcam completely wrapped in plastic wrap secured by orange duct tape sitting on my window sill}}
  
-Additional weatherproofing measures included a plastic tupperware lid taped over the camera as a sort of primitive precipitation shield.+{{ :webcam-tupperware-lid.jpg?200|Picture of a tupperware lid taped over the camera as a sort of primitive rain shield}}
  
-{{:webcam-tupperware-lid.jpg?200|Picture of a tupperware lid taped over the camera as a sort of primitive rain shield}}+Additional weatherproofing measures included a plastic tupperware lid taped over the camera as a sort of primitive precipitation shield.
  
 Say what you will, but this setup survived a thunderstorm immediately followed by freezing temperatures and several hours of snow. All for $0. Say what you will, but this setup survived a thunderstorm immediately followed by freezing temperatures and several hours of snow. All for $0.
Line 27: Line 29:
 ===== Bird Identification ===== ===== Bird Identification =====
  
-I’d read about [[https://pjreddie.com/darknet/yolo/|YOLO] some years before and began to reacquaint myself. It’s come quite far and seems to be more or less the state of the art for realtime computer vision object detection and classification. I downloaded the latest version ([[https://github.com/ultralytics/yolov5|YOLOv5]] at time of writing) and ran the webcam demo. It ran well over 30fps with good accuracy on my RTX3080, correctly picking out myself as “person”, my phone as “cell phone”, and my light switch as “clock”.+{{ :me-with-phone-yolo-detection.png?200|Screen capture of webcam feed after applying YOLOv5's out-of-box 'small' model to a scene of me holding up my cell phone. Picture is heavily blurred}}
  
-{{:me-with-phone-yolo-detection.png?200|Screen capture of webcam feed after applying YOLOv5's out-of-box 'small' model to a scene of me holding up my cell phone. Picture is heavily blurred}}+I’d read about [[https://pjreddie.com/darknet/yolo/|YOLO]] some years before and began to reacquaint myself. It’s come quite far and seems to be more or less the state of the art for realtime computer vision object detection and classification. I downloaded the latest version ([[https://github.com/ultralytics/yolov5|YOLOv5]] at time of writing) and ran the webcam demo. It ran well over 30fps with good accuracy on my RTX3080, correctly picking out myself as “person”, my phone as “cell phone”, and my light switch as “clock”.
  
 Out of the box YOLOv5 is trained on COCO, which is a dataset of _co_mmon objects in _co_ntext. This dataset is able to identify a picture of a Carolina chickadee as “bird”. Tufted titmice are also identified as “bird”. All birds are “bird” to COCO (at least the ones I tried). Out of the box YOLOv5 is trained on COCO, which is a dataset of _co_mmon objects in _co_ntext. This dataset is able to identify a picture of a Carolina chickadee as “bird”. Tufted titmice are also identified as “bird”. All birds are “bird” to COCO (at least the ones I tried).
Line 39: Line 41:
 ===== Dataset ===== ===== Dataset =====
  
-A quick Google search for “north american birds dataset” yielded probably the most convenient dataset I could possibly have asked for. Behold, NABirds!+A quick Google search for “north american birds dataset” yielded probably the most convenient dataset I could possibly have asked for. Behold, [[https://dl.allaboutbirds.org/nabirds|NABirds]]!
  
 >> NABirds V1 is a collection of 48,000 annotated photographs of the 400 species of birds that are commonly observed in North America. More than 100 photographs are available for each species, including separate annotations for males, females and juveniles that comprise 700 visual categories. This dataset is to be used for fine-grained visual categorization experiments. >> NABirds V1 is a collection of 48,000 annotated photographs of the 400 species of birds that are commonly observed in North America. More than 100 photographs are available for each species, including separate annotations for males, females and juveniles that comprise 700 visual categories. This dataset is to be used for fine-grained visual categorization experiments.
Line 59: Line 61:
 YOLOv5 offers multiple network sizes, from n to x (n for nano, x for x). n, s and m sizes are recommended for mobile or workstation deployments, while l and x variants are geared towards cloud / datacenter deployments (i.e. expensive datacenter GPUs / tensor processors). The larger variants take longer to train and longer to evaluate. Since the model needed to be evaluated on each frame in a video feed, holding all else constant, for this project the choice of model size would ultimately dictate the achievable framerate. YOLOv5 offers multiple network sizes, from n to x (n for nano, x for x). n, s and m sizes are recommended for mobile or workstation deployments, while l and x variants are geared towards cloud / datacenter deployments (i.e. expensive datacenter GPUs / tensor processors). The larger variants take longer to train and longer to evaluate. Since the model needed to be evaluated on each frame in a video feed, holding all else constant, for this project the choice of model size would ultimately dictate the achievable framerate.
  
-{{:yolo-model-comparison-graph.png?200|Graph showing evaluation time versus average precision against the COCO dataset}}+{{ :yolo-model-comparison-graph.png?400|Graph showing evaluation time versus average precision against the COCO dataset}}
  
 Since the webcam demo with the s model ran at a good framerate on my GPU I chose that one to start. Since the webcam demo with the s model ran at a good framerate on my GPU I chose that one to start.
Line 75: Line 77:
 For the data oriented, here is the summary information for the training run of the model I ended up using: For the data oriented, here is the summary information for the training run of the model I ended up using:
  
-{{:final-training-results.png?200|Composite image with various graphs depicting various training metrics, including recall, precision, loss, and mean average precision on both training and validation sets}} {{:pr-curve.png?200|PR curve}}+{{:final-training-results.png?400|Composite image with various graphs depicting various training metrics, including recall, precision, loss, and mean average precision on both training and validation sets}} {{:pr-curve.png?400|PR curve}}
  
 These metrics are all good and show that the model trained very nicely on the dataset. These metrics are all good and show that the model trained very nicely on the dataset.
Line 83: Line 85:
 Trying it on an image with three species of chickadee, that to my eye look almost identical: Trying it on an image with three species of chickadee, that to my eye look almost identical:
  
-{{:three-chickadee.png?200|Composite picture of carolina, mountain and black-capped chickadees}}+{{:three-chickadee.png?400|Composite picture of carolina, mountain and black-capped chickadees}}
  
 I’m not sure if these were in the training set; I just searched for the first images of each species I found on Google Images. I’m not sure if these were in the training set; I just searched for the first images of each species I found on Google Images.
Line 93: Line 95:
 ===== Video flow ===== ===== Video flow =====
  
-Having demonstrated that it could identify birds with relative accuracy, it was time to get it working on a live video feed. Even though YOLO is amazingly fast relative to other methods, it still needs a GPU in order to evaluate the model fast enough to process video in real time. I set a goal of 30fps; on my 3080, my final model averages roughly 0.020s per frame, sufficient to pull around 40-50fps. This is a good tradeoff between model size/accuracy and evaluation speed. The average with the s model looks to be roughly 0.015s, technically sufficient to pull 60fps, but without much headroom. There are spikes up to the 0.02 range, suggesting that 60fps would likely be jittery. The webcam I used for this project didn’t really suffice anyway.+Having demonstrated that it could identify birds with relative accuracy, it was time to get it working on a live video feed.
  
-So while I had a GPU that can perform inference fast enough, the problem was that the video feed was in the next room, and I don’t much feel like moving my desktop into the kitchen. I considered several approaches to this problem, including:+Even though YOLO is amazingly fast relative to other methods, it still needs a GPU in order perform inference fast enough to produce inference results for each frame of a video feed. I set a goal of 30fps; on my 3080, my final model averages roughly 0.020s per frame, sufficient to pull around 40-50fps. This is a good tradeoff between model size/accuracy and evaluation speed. The average with the s model looks to be roughly 0.015s, technically sufficient to pull 60fps, but without much headroom. There are spikes up to the 0.02 range, suggesting that 60fps would likely be jittery. The webcam I was using at the time didn’t really suffice for 60fps anyway. 
 + 
 +So while I had a GPU that could perform inference fast enough, it was in the computer in my bedroom and I didn’t much feel like moving it into the kitchen. I considered several approaches to this problem, including:
  
   * A very long USB extender to the webcam   * A very long USB extender to the webcam
  
-I actually tried this, and it did work. The problem was that it required me to have my bedroom window open a little bit to route the cable through it, which isn’t ideal for several reasons. It also caused difficulties with detect.py, the program included with YOLOv5 that provides a convenient interface to performing inference on various media sources; when given a direct webcam feed, the output is 640x640. I wanted to use the full resolution of the camera (1920x1080). I briefly investigated patching detect.py, but the holidays were drawing to a close and I wanted a functioning stream sooner rather than later.+I actually tried this, and it did work. The problem was that it required me to have my bedroom window open a little bit to route the cable through it, which isn’t ideal for several reasons. Also, when using YOLO's video detection script on a webcam device it produces annotated frames of size 640x640. I wanted to stream the full resolution of the camera (1920x1080). I briefly investigated patching detect.py, but the holidays were drawing to a close and I wanted a functioning stream sooner rather than later, so for the time being I resolved to use a video feed since the script produced annotated at the native resolution when given a video.
  
   * eGPU enclosure attached to the webcam host   * eGPU enclosure attached to the webcam host
  
-By this point in the project I’d replaced the laptop with a NUC I happened to have lying around. If this was a recent NUC with Thunderbolt 3 support, an eGPU enclosure would have been the cleanest and easiest solution. While I do have a spare GPU I could have used, this NUC had no capability to support an eGPU and I didn’t have an enclosure. This route would have entailed purchasing a new NUC and an eGPU enclosure - neither of which are particularly inexpensive - and waiting for delivery. Nixed this option.+By this point in the project I’d replaced the laptop with a NUC I happened to have lying around. If this was a recent NUC with Thunderbolt 3 support, an eGPU enclosure would have been the cleanest and easiest solution. I wanted to avoid buying a new NUC and eGPU enclosure since together those cost quite a bit, and I didn't care to wait for delivery so I nixed this option.
  
   * Some kind of network streaming setup   * Some kind of network streaming setup
  
-No cables through windows, no expensive enclosures. I have a pretty good home network. This is what I went with. After several hours experimenting with RTMP servers, HTTP streaming tools, and the like, I ended up with this setup:+Advantages: no cables through windows, no new hardware. I have a pretty good home network and am pretty handy with this stuff so I went with that. After several hours experimenting with RTMP servers, HTTP streaming tools, and the like, I ended up with this setup:
  
-{{:video-routing.png?400|Video routing setup; shows video feed streaming from webcam host in the kitchen, to my desktop where inference is performed, then to Twitch}}+{{ :video-routing.png?400|Video routing setup; shows video feed streaming from webcam host in the kitchen, to my desktop where inference is performed, then to Twitch}}
  
 I tried a bunch of other things, including streaming RTMP to a local NGINX server, using VLC as an RTSP source on the webcam box, etc, but this was the setup that was the most stable, had the highest framerate, and lowest artifacts. Actually detect.py does support consuming RTSP feeds directly, but whatever implementation OpenCV uses under the hood introduces some significant artifacts into the output. Using VLC to consume the RTSP feed and rebroadcast it locally as an HTTP stream turned out better. The downside to this is that VLC seems to crash from time to time, but a quick batch script fixed that right up: I tried a bunch of other things, including streaming RTMP to a local NGINX server, using VLC as an RTSP source on the webcam box, etc, but this was the setup that was the most stable, had the highest framerate, and lowest artifacts. Actually detect.py does support consuming RTSP feeds directly, but whatever implementation OpenCV uses under the hood introduces some significant artifacts into the output. Using VLC to consume the RTSP feed and rebroadcast it locally as an HTTP stream turned out better. The downside to this is that VLC seems to crash from time to time, but a quick batch script fixed that right up:
Line 128: Line 132:
 Why yes, I do have Windows experience :’) Why yes, I do have Windows experience :’)
  
-Eventually the above setup proved unreliable. It also required running lots of software on both the camera host as well as my desktop, and since it used my desktop GPU for inference it limited what I could use my desktop computer for - not to mention the stream going down every time I rebooted my computer.+{{ :setup.jpg?400|}}
  
-After deciding that I wanted to maintain this as a long term project I purchased a NUC and an eGPU enclosure. I initially tried to use the enclosure with an RTX 3070, but I couldn’t get it working with that card so I used a 1070 that was lying around instead which worked flawlessly. The 1070 runs at about 25fps when inferencing with my bird model which is more than enough. The whole thing sits on my kitchen floor and is relatively unobtrusive.+I ran it that way for a couple months or so, but eventually the above setup proved too unreliable. It required running lots of software on both the camera host as well as my desktop, and since it used my desktop GPU for inference it limited what I could use my computer for (read: no gaming). Also, the stream went down every time I rebooted my computer.
  
-{{:setup.jpg?400|}}+After deciding that I wanted to maintain this as a long term installation I ponied up for a NUC and an eGPU enclosure. I initially tried to use the enclosure with an RTX 3070, but I couldn’t get it working with that card so I used a spare 1070 instead which worked flawlessly. The 1070 runs at about 25fps when inferencing with my bird model which is more than enough to look snappy overlaid on a video feed. The whole thing sits on my kitchen floor and is relatively unobtrusive.
  
-What I was doing before was simply streaming the window displayed by YOLO’s detect.py convenience script. However, at 25fps this isn’t a great viewing experience. I wanted to stream 30 or 60fps video and overlay the labels on top of it. Doing this turned out to be rather difficult because you cannot multiplex camera devices on Windows; only one program can have a handle on the camera and its video feed to the exclusion of all others. Fortunately there is some [[https://very-soft.com/product/webcamsplitter|software]] which works around this. I purchased that software and used it to split the camera feed in two. OBS directly consumes one feed and the other one goes into YOLO for inferencing. The resulting labeled frames are displayed in a live preview window. I then patched YOLO so that the preview window, which normally displays the source frame annotated with the inferencing results, only displayed the annotations on a black background without the source frame. That window is used as a layer in OBS with a luma filter applied to make the black parts transparent. With some additional tweaks to get the canvas sizing and aspect ratio correct this allowed me to composite the 25fps inferencing results on top of the high quality 60fps video coming from the camera.+==== 60fps ==== 
 + 
 +Up to this point I was streaming the window with annotated frames displayed by YOLO’s detect.py convenience script. However, this window updates only as often as an inferencing run completes, so around 25fps. It doesn't look good on a livestream. It would be better to stream video straight from the camera at native framerates (ideally 60fps) and overlay the labels on top of it. 
 + 
 +Doing this turned out to be rather difficult because you cannot multiplex camera devices on Windows; only one program can have a handle on the camera and its video feed to the exclusion of all others. Fortunately there is some [[https://very-soft.com/product/webcamsplitter|software]] which works around this. I purchased that software and used it to create two virtual camera feeds. OBS directly consumes one feed and the other one goes into YOLO for inferencing. The resulting labeled frames are displayed in a live preview window. I patched YOLO so that the preview window, which normally displays the source frame annotated with the inferencing results, only displayed the annotations on a black background without the source frame. That window is used as a layer in OBS with a luma filter applied to make the black parts transparent. With some additional tweaks to get the canvas sizing and aspect ratio correct this allowed me to composite the 25fps inferencing results on top of the high quality 60fps video coming from the camera. 
 + 
 +{{ :nuc-perf.jpg?400|}}
  
 For encoding I use Nvenc on the 1070. That keeps the stream at a solid 60fps, which the NUC CPU can’t accomplish. Between inferencing and video encode the card is getting put to great use. For encoding I use Nvenc on the 1070. That keeps the stream at a solid 60fps, which the NUC CPU can’t accomplish. Between inferencing and video encode the card is getting put to great use.
  
-{{:nuc-perf.jpg?400|}}+This was stable for over a year, until I decided to install Windows 11. What could go wrong? 
 + 
 +==== Camera ==== 
 + 
 +The original setup used an off-brand 720p webcam wrapped in a righteous amount of plastic wrap for weatherproofing. Surprisingly the weatherproofing worked well and there was never a major failure while using the first camera. However, the quality and color on that camera wasn’t good and an upgrade was due. I already had a Logitech Brio 4k webcam intended for remote work, but it ended up largely unused so it was repurposed for birdwatching. 
 + 
 +While the plastic wrap method never had any major failures it wasn’t ideal either. Heavy humidity created fogging inside the plastic that could take a few hours to wear off. It needed replacing anytime the camera was adjusted. Due to these problems and the higher cost of the Brio I decided to build a weatherproof enclosure. 
 + 
 +The feeder is constructed of acrylic. My initial plan was to use acrylic sheeting build out an extension to the feeder big enough to house the camera. I picked up some acrylic sheeting from Amazon and began researching appropriate adhesives. It turns out most adhesives don’t work very well on acrylic, at least not for my use case – the load bearing joints between the sheets were thin and I needed the construction to be rigid enough to support its own weight and the weight of the camera without sagging. Since the enclosure would be suspended over air relying on its inherent rigidity for structure the adhesive needed to be strong. 
 + 
 +The best way to adhere acrylic to itself is using acrylic cement. Acrylic cement dissolves the surfaces of the two pieces to be bonded, allowing them to mingle, and then evaporates away. This effectively fuses the two pieces together with a fairly strong bond (though not as strong as if the piece had been manufactured that way). 
 + 
 +{{:acrylic-cement.jpg?400 |}} 
 + 
 +{{:assembled-box.jpg?400 |}} 
 + 
 +Three sides were opaque to prevent sunlight reflections within the box. Joints were caulked and taped the joints to increase weather resistance. I played around with using magnets to secure the enclosure to the main feeder body but didn’t come up with anything I liked, so I glued it to the feeder with more acrylic cement, threw my camera in there and called it a day. 
 + 
 +{{ :full-setup-with-first-weather-shield.jpg?600 |}} 
 + 
 +This weatherproofing solution turned out great. It successfully protected the camera from all inclement weather until I retired that feeder, surviving rain, snow, and high winds over the course of the year. 
 + 
 +====== Switching to Linux ====== 
 + 
 +As it turns out, the webcam splitter software appears to rely on some undocumented / unofficial Windows 10 APIs and does not work on Windows 11. I decided to bite the bullet and just put Linux on the NUC. 
 + 
 +Similar to Windows, on Linux only one device can be reading from a camera device file at a time. Unlike Windows there is a very easy way to work around this called [[https://wiki.archlinux.org/title/V4l2loopback|v4l2loopback]]. This is a kernel module that allows you to create virtual video device files that can be fed video from a userspace application such as [[w>gstreamer]]. These virtual device files support multiple clients. 
 + 
 +tl;dr: 
 + 
 +<code bash> 
 +# load v4l2loopback module; this creates a few loopbacks 
 +sudo modprobe v4l2loopback 
 +# set desired parameters on loopback device /dev/video2 
 +sudo v4l2loopback-ctl set-fps 60 /dev/video2 
 +# pipe video from camera device /dev/video0 to loopback 
 +gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,width=1920,height=1080 ! videoconvert ! v4l2sink device=/dev/video2 
 +</code> 
 + 
 +After this, any number of clients can read from /dev/video2. I replicated my video + annotations compositing setup in OBS as described earlier without issue. 
 + 
 +On the off chance this helps someone, here's how you set video camera parameters on device 0 (/dev/video0) from the command line: 
 + 
 +<code> 
 +sudo v4l2-ctl -d 0 -c focus_automatic_continuous=0 
 +sudo v4l2-ctl -d 0 -c focus_absolute=75 
 +sudo v4l2-ctl -d 0 -c backlight_compensation=0 
 +sudo v4l2-ctl -d 0 -c auto_exposure=3 
 +</code>
  
-Note that the webcam splitter software appears to rely on some undocumented / unofficial windows API as it doesn't work on windows 11.+Around this time I also replaced the feeder since the last one was all scratched up and the acrylic had gotten all cloudy. I also replaced the cheap camera with a Logitech Streamcam duct taped to the feeder. I found that the Brio I was using didn't correctly handle gain adjustment so bright sunlight would blow it out. The Streamcam, which is much more affordable, doesn't have that issue.
  
 You can watch the [[https://twitch.tv/thebirdbar|livestream here]]. You can watch the [[https://twitch.tv/thebirdbar|livestream here]].
Line 190: Line 248:
  
  
-{{tag>from blog}}+{{tag>from_blog technology machine_learning nature}}
  
  
  
  
Panorama theme by desbest
bird_bar.1694218750.txt.gz · Last modified: 2023/09/09 00:19 by qlyoung
CC Attribution-Noncommercial-Share Alike 4.0 International Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Noncommercial-Share Alike 4.0 International