Seeking help for step 4: merge 2 results and visualization

[zjaxxon]

Hi, thank you so much for the outstanding work! I am trying to reproduce the result and currently struck at step 4. Could you please provide some instructions on how to use pcd_ensemble, or how to visualize the seg_pcd result? I am not sure if I understand the code correctly, but it seems only group ids are save in the .pth output file. I am trying to visualize the coordinates by their group.

[dreamingaa]

I have encountered the same problem, have you solved it?

[rayryeng]

Just providing some comments for those who are stuck here. You need to access the Segmentator application found in the original ScanNet repo: https://github.com/ScanNet/ScanNet/tree/master/Segmentator

Save this directory to your machine, and you will need a C++ compiler that supports C++11 to build it. You can use either the CMake build file, or use the Makefile to build the source. In other words, use either mkdir build && cd build && cmake .. && make or just do make. This will produce a segmentator application which you would use on your input PLY file. FYI, the PLY file needs to have face indices that will be used to compute the face normals, so it can't just be a point cloud itself. This application groups mesh faces together by looking at the orientation of the normals and faces within their proximity to group things together to produce labels. The authors claim to use the application with the default parameters, so just do:

./segmentator input.ply

where input.ply is the input mesh with the vertices and face indices. Once that is computed, it will output a JSON file that tells you which faces correspond to which group ID. The path to this JSON is the org_path parameter for pcd_ensemble. The new_path is the path to the output segmentation PyTorch dictionary from the sam3d.py file after it calls seg_pcd. The data_path is the PyTorch dictionary that stores the 3D scene representation or ultimately, this is the scene that we want to project our labels onto when running SAM - this was done in the preprocess_scannet.py file. If you are using a custom dataset, you will need to create a dictionary that has the "coords", "color" and "scene_id", as well as the "normal" keys that represent the vertex coordinates, the colour found at each coordinate and a unique identifier that tells you what the scene is. Because I used a custom dataset, I set this to be a dummy string. This dictionary needs to be saved using torch.save.

If you want to visualize the segmentation labels, you can use the visualize_partition function that is found in util.py, which is what the pcd_ensemble function does at the end when merging the results together. The final input parameter of pcd_ensemble is the vis_path where the 3D points of the scene will be saved as a file called pc.ply at this location.

I didn't use the sam3d.ply file exactly - I extracted some pieces of it so I could run this on my own custom data, but the pcd_ensemble function I managed to get running from start to finish by removing all of the bits that are reliant on the ScanNet dataset.

Hope this helps!