Calculate distance of detected object

[angelocarbone]

Calculate the distance to the detection boxes using the depth map and overlay the distance value in the center of each detection box.

Then run this on a few YouTube videos with city driving, and record the processing output and post it on YouTube directly from the notebook if possible.

[ppirrip]

How do we get start on this? I am thinking to have the assumption that the camera plan and the object plane are align (no pitch and yaw misalignment) for now, and proceed with depth estimation used the bounding box calculated (xc,yc,height,width) from the detection algo, and our guess to the size of the object in the image. We can start with some cats/dogs or whatever household object picture from our cell camera I guess

[mslavescu]

Try to create a function getDistanceToClosestFrontObject() that computes the distance using the SfMLearner depth (averaged over the pixels in the detection box) and the width of the detection box from SSD object detection.

We can test it visually on any YouTube video with driving, including the ones from games, like the one from The Crew.

We can automatically evaluate the function on KITTI and CityScape datasets where we have both depth and detection boxes ground truth.

Use the getDistanceToClosestFrontObject() function to estimate when to trigger warnings for FCW.

[ppirrip]

I have been trying to put the SfM and SSD demo together for the depth output on the mask/bounding box, but there are misc issue in setting/configing/etc, those two networks together, not sure is one in Caffe and one in plain TF whatever other reason. In the future maybe we should stick with one framework

Edit: Seems like it is a common TF question: tensorflow/tensorflow#3270

Edit 2: Relatively certain is the name crash between the two models' parameter during loading. The two models were both trained within the tf.global.variable namespace.

[mslavescu]

Lets try the workaround here, to work with 2 or more Tensorflow models loaded in same GPU:

https://stackoverflow.com/questions/41607144/loading-two-models-from-saver-in-the-same-tensorflow-session

All 3 demos we currently have use Tensorflow.

[mslavescu]

See also this article:
http://bretahajek.com/2017/04/importing-multiple-tensorflow-models-graphs/

[ppirrip]

I found out about the tf graph last night too http://cv-tricks.com/tensorflow-tutorial/save-restore-tensorflow-models-quick-complete-tutorial/, and with the two additional ref seems like now I can load both ckpt in the same environment. Will put them together later in the day.

[mslavescu]

Super!
Would be interesting to reuse first set of layers if they do the same thing in both networks, like grayscale, blur, poolling etc, to speedup the overall networks processing.

[ppirrip]

Ar, merging of the two network, that's what I have in mind too. Make both use the same backbone would be nice too. Anyway, I still have issues after loading the weight, just not playing nice together yet. This graph/session in TF is interesting to say at least, but something has to deal with sooner or later.

[ppirrip]

Seems like I got the SSD and SfM NN working together in one notebook now. There is still some strange issue at the loop (image won't display until I interrupt the runtime etc.) The note book is uploaded to https://github.com/OSSDC/OSSDC-VisionBasedACC/blob/master/object_detection/ssd_sfm_demo_colaboratory.ipynb

[mslavescu]

Super!
I'll try it tonight.