deepsense.aideepsense.ai logo
  • Clients’ stories
  • Industries
    • Retail
    • Manufacturing
    • Financial Services
    • IT Operations
    • TMT and Other
  • Train your team
  • R&D Hub
  • Blog
  • About us
    • Our story
    • Management
    • Advisory Board
    • Press center
    • Careers
    • Internship program
  • Contact
  • Menu Menu

Region of interest pooling in TensorFlow – example

April 25, 2017/in Data science, Deep learning, Machine learning, Neptune /by Krzysztof Dziedzic, Patryk Miziuła and Błażej Osiński

In the previous post we explained what region of interest pooling (RoI pooling for short) is. In this one, we present an example of applying RoI pooling in TensorFlow. We base it on our custom RoI pooling TensorFlow operation. We also use Neptune as a support in our experiment performance tracking.

Example overview

Our goal is to detect cars in the images. We’d like to construct a network that is able to automatically draw a box around every car.
In our example we deal with car images from the Pascal VOC 2007 dataset. For simplicity we choose only cars not marked as truncated.

Exemplary images from Pascal VOC 2007 dataset
Related:  Region of interest pooling explained

Neptune

We manage our experiment using Neptune. It’s a pretty handy tool:

  • We track the tuning in real time. Especially, we preview the currently estimated bounding boxes.
  • We can change model hyperparameters on the fly.
  • We can easily integrate Neptune with TensorFlow and get all the charts, graphs and summary objects from the TensorFlow graph.
  • We store the executed experiments in an aesthetic list.

Network architecture

In our example we use the Fast R-CNN architecture.
The network has two inputs:

  1. Batch of images
  2. Batch of potential bounding boxes – RoI proposals
    In the Fast R-CNN model RoI proposals are generated via an external algorithm, for example selective search. In our example, we take ground truth bounding boxes from the Pascal annotations and generate more negative bounding boxes ourselves.

The network has two outputs:

  1. Batch of RoI proposals not classified as background (with corrected coordinates)
  2. Probabilities that RoI proposals consist of objects of the consecutive categories

The network consists of three main parts:

  1. Deep convolutional neural network
    • Input: images
    • Output: feature map

    We use the popular VGG16 network pretrained on the ImageNet dataset.

  2. RoI pooling layer
    • Input: feature map, RoI proposals resized to a feature map
    • Output: max-pooled RoI proposals
  3. Fully connected layer with RoI features
    • Input: max-pooled RoI proposals
    • Output: corrected RoI proposals, probabilities
RoI pooling in TensorFlow scheme
Fast R-CNN architecture

We note that our detection task can be also solved with the Faster R-CNN architecture, which works significantly faster :). However, the implementation of Faster R-CNN requires much more code to write, so we chose the simpler Fast R-CNN.

Loss function

We tune the network to minimize the loss given by
\(loss = frac 1nsum_{i=1}^n frac 1{k_i} sum_{j=1}^{k_i} loss_{ij}\)
where:

  • \(n\) is a number of images in a batch,
  • \(k_i\) is a number of RoI proposals for the image \(i\),
  • \(loss_{ij}\) is a loss for the RoI proposal \(j\) for the image \(i\).

For a single RoI proposal, \(loss_{ij}\) is the sum of the classification and regression loss, where:

  • classification loss is the common cross entropy,
  • regression loss is a smooth L1 distance between the rescaled coordinates of a RoI proposal and the ground-truth box. The regression loss is computed if the ground-truth box is not categorized as background, otherwise it’s defined as 0.
Related:  Playing Atari with deep reinforcement learning - deepsense.ai’s approach

Implementation details

Prerequisites

To run the code we provide, you need the following software:

  • CUDA 8,
  • TensorFlow 1.0 with GPU support,
  • our custom RoI pooling TensorFlow operation,
  • OpenCV,
  • Neptune (version 1.5): apply for our Early Adopters Program or try it immediately with Neptune Go.

Repository

You can download our code from our GitHub repository. It consists of two folders with the following content:

File Purpose
code
main.py The script to execute.
fast_rcnn.py Builds the TensorFlow graph.
trainer.py Preprocesses data and trains the network.
neptune_handler.py Contains Neptune utilities.
config.yaml Neptune configuration file.
get_data.py Downloads images from Pascal VOC 2007 dataset
data
vgg16-20160129.tfmodel.torrent References to weights of the pretrained network.

Description

When we run main.py , the script trainer.py  first restores the VGG16 network with the pretrained weights. Then it adds the RoI pooling layer and the fully connected layer. Finally, it begins tuning the entire network with use of provided images and RoI proposals. It also sends information to Neptune, so we can track the tuning progress in real time.
After cloning the repository, please download the file vgg16-20160129.tfmodel  referred to by the torrent file vgg16-20160129.tfmodel.torrent  and save it in the data  directory. Also, please run the script get_data.py to download needed images:

python get_data.py

Let’s test our RoI pooling in TensorFlow!

We run the script main.py  from the code folder by typing:

neptune run --
            --im_folder $PWD/../data/images
            --roidb $PWD/../data/roidb
            --pretrained_path $PWD/../data/vgg16-20160129.tfmodel

If we want to also use a non-default learning rate value or the number of epochs, we can add:

--learning_rate 1e-03 --num_epochs 200

to the command at the end.
After a while, we can start observing the tuning progress in Neptune:

RoI pooling in TensorFlow - tuning
Tracking the network tuning in Neptune

Moreover, we can display the RoIs fitted to the cars by our network. We could just load all the processed images, but this procedure would take much of resources. That’s why we decided to activate this feature by a simple Neptune action.
To do that, we can go to the Actions tab and click ‘RUN’ to start sending the images.

RoI pooling in TensorFlow - turning on the image sending in Neptune
Turning on the image sending

After that, we can go to the Channels tab and expand the channels ‘region proposals for RoI pooling’ and ‘network detections’ by clicking ‘+’ signs.

Roi pooling in TensorFlow - expanding image channels in Neptune
Expanding image channels

Now we can see the RoIs in real time!

RoI pooling in TensorFlow - RoI preview
RoI proposals preview in Neptune

We can click on the pictures to zoom them. If we want Neptune to stop sending new images, we go to the Actions tab and click ‘RUN’ again.
An exemplary NeptuneGo execution of our script can be found here.

Related:  Logo detection and brand visibility analytics - example

Summary

We hope you enjoy our example of RoI pooling in TensorFlow and experiment managing features offered by Neptune. If you want to comment our work, don’t be hesitate to leave us feedback!

References

  • R. Girshick, Fast R-CNN, IEEE International Conference on Computer Vision (ICCV), 2015.
  • S. Ren, K. He, R. Girshick & J. Sun, Faster R-CNN: towards real-time object detection with Region Proposal Networks, Neural Information Processing Systems (NIPS), 2015.
  • deepsense.ai, Region of interest pooling explained, 2017.
Share this entry
  • Share on Facebook
  • Share on Twitter
  • Share on WhatsApp
  • Share on LinkedIn
  • Share on Reddit
  • Share by Mail
https://deepsense.ai/wp-content/uploads/2019/02/roi-pooling-in-tensorflow-example.jpg 337 1140 Krzysztof Dziedzic https://deepsense.ai/wp-content/uploads/2019/04/DS_logo_color.svg Krzysztof Dziedzic2017-04-25 11:46:062021-01-05 16:49:52Region of interest pooling in TensorFlow – example

Start your search here

NEWSLETTER SUBSCRIPTION

    You can modify your privacy settings and unsubscribe from our lists at any time (see our privacy policy).

    This site is protected by reCAPTCHA and the Google privacy policy and terms of service apply.

    THE NEWEST AI MONTHLY DIGEST

    • AI Monthly Digest 20 - TL;DRAI Monthly Digest 20 – TL;DRMay 12, 2020

    CATEGORIES

    • Artificial Intelligence
    • AIOps
    • Big data & Spark
    • Data science
    • Deep learning
    • Machine learning
    • Neptune
    • Reinforcement learning
    • Seahorse
    • Job offer
    • Popular posts
    • AI Monthly Digest
    • Press release

    POPULAR POSTS

    • AI trends for 2021AI trends for 2021January 7, 2021
    • A comprehensive guide to demand forecastingA comprehensive guide to demand forecastingMay 28, 2019
    • What is reinforcement learning? The complete guideWhat is reinforcement learning? The complete guideJuly 5, 2018

    Would you like
    to learn more?

    Contact us!
    • deepsense.ai logo white
    • Industries
    • Retail
    • Manufacturing
    • Financial Services
    • IT Operations
    • TMT and Other
    • Knowledge base
    • Blog
    • Press center
    • deepsense.ai
    • Our story
    • Management
    • Scientific Advisory Board
    • Careers
    • Support
    • Terms of service
    • Privacy policy
    • Contact us
    • Join our community
    • facebook logo linkedin logo twitter logo
    • © deepsense.ai 2014-
    Scroll to top

    This site uses cookies. By continuing to browse the site, you are agreeing to our use of cookies.

    OKLearn more

    Cookie and Privacy Settings



    How we use cookies

    We may request cookies to be set on your device. We use cookies to let us know when you visit our websites, how you interact with us, to enrich your user experience, and to customize your relationship with our website.

    Click on the different category headings to find out more. You can also change some of your preferences. Note that blocking some types of cookies may impact your experience on our websites and the services we are able to offer.

    Essential Website Cookies

    These cookies are strictly necessary to provide you with services available through our website and to use some of its features.

    Because these cookies are strictly necessary to deliver the website, refuseing them will have impact how our site functions. You always can block or delete cookies by changing your browser settings and force blocking all cookies on this website. But this will always prompt you to accept/refuse cookies when revisiting our site.

    We fully respect if you want to refuse cookies but to avoid asking you again and again kindly allow us to store a cookie for that. You are free to opt out any time or opt in for other cookies to get a better experience. If you refuse cookies we will remove all set cookies in our domain.

    We provide you with a list of stored cookies on your computer in our domain so you can check what we stored. Due to security reasons we are not able to show or modify cookies from other domains. You can check these in your browser security settings.

    Other external services

    We also use different external services like Google Webfonts, Google Maps, and external Video providers. Since these providers may collect personal data like your IP address we allow you to block them here. Please be aware that this might heavily reduce the functionality and appearance of our site. Changes will take effect once you reload the page.

    Google Webfont Settings:

    Google Map Settings:

    Google reCaptcha Settings:

    Vimeo and Youtube video embeds:

    Privacy Policy

    You can read about our cookies and privacy settings in detail on our Privacy Policy Page.

    Accept settingsHide notification only