Visual Localization: Template Script to Feed Camera Data to Vision Stack

[WithanagePerera]

After talking with Daniel, we realized the current implementation of the vision stack code doesn't have any test/template scripts to subscribe to the camera publisher to actually process data, so it'd be hard for new members to understand how to use the code.

Daniel did find an old test script from the original mil repository (using ROS1) that was used to test vision stack with camera data, so I'm going to transfer that over to mil2 and make adjustments to it to use the new front cam as well as the new syntax changes.

I'll also validate that new publishers are created for each layer in the vision stack so we can extract processed data from individual layers.

[WithanagePerera]

Working on this in https://github.com/uf-mil/vision_stack/tree/Praveen-VisionStackTests

[WithanagePerera]

Got a minimal publisher working earlier. Today I did some more in-depth testing to feed the data from the /front_cam/image_raw topic. Had to bring in cv_bridge to handle conversion from the ROS Image format to the OpenCV format.

Besides that, I kept running into issues with the VisionStack.py class like with shallow copies, matplot lib not being properly imported (it was part of an exception response for some reason), and also a weird issue with how the topic name was being constructed. I put some hard-coded fixes for these issues just to keep going with my minimal publisher, which I did get to work.

My next steps are to create a simple app to visualize the images I'm getting from the publisher to see if VisionStack layers are actually working. From there, I'll talk a bit with Daniel about the naming issue and the matplot lib import, then create a more robust solution and hopefully give this test code to the rest of my team.

[WithanagePerera]

During the meeting on Tuesday, Carlos/Adam showed me a method to use rqt_image to visualize topics publishing image data, and we validated that my Vision Stack test script works properly with the 2 layers that were being instantiated.

I'll create a version with all of the layers to validate that it scales properly, then release it to the team so that they can test the layer scripts that they're developing.

[WithanagePerera]

Fixed up the remaining issues with the layers (got the paths to weights working for ObjectDetectionLayer) and tested all of them out simultaneously. The only problematic layers were Sobel and RobustScaling because they return floating points which isn't supported for publisher encoding(?).

I also changed the logic for topic names so that they use the class name of the layer which now makes it super obvious which layer is which. Created a PR that's being reviewed by Daniel.

[WithanagePerera]

PR Link: uf-mil/vision_stack#41

[WithanagePerera]

Made some fixes to the comments Daniel left.

• Test Script for Vision Stack vision_stack#41 (comment)
  Removed front_cam from the naming, but it's still not a great solution because what if I apply these filters for both bottom cam and front cam? Then we would have topics with the same name. I was thinking of creating a new parameter to the VisionStack class to pass custom names, but I feel like that might introduce some issues.

• Test Script for Vision Stack vision_stack#41 (comment)
  When I was first working on the test script, I hadn't used the cv_bridge conversion function to convert the raw topic data to a numpy array, so the copy() function wasn't implemented for it, hence I brought in the copy library to get deep copies. I've seen then reverted it back since I brought in the cv_bridge conversion.

• Test Script for Vision Stack vision_stack#41 (comment)
  Removed the debug statement, honestly forgot about that.

• Test Script for Vision Stack vision_stack#41 (comment)
  Going to create a new PR for the changes I made to the layer scripts.