Manual segmentation action extension

CamiTK is a great tool that brings you the visualization power of VTK, image processing power of ITK and GUI handling capabilities of Qt, but simplifies it so that you can concentrate on your work. There are many ways to go about implementing what you need in CamiTK. In this tutorial, you will be presented with on such way: development in increments.

For this tutorial, let’s create a segmentation tool that lets you manually segment an image by drawing along the contours of the objects of interest similar to the ITK-SNAP tool¹ using the polygon mode.

First, let’s define the required functionality of this tool.

1. Draw contours in a given orthogonal viewer (axial, coronal or sagittal) of CamiTK-imp (as different objects can be easily seen in different views)
2. Copy an already drawn contour and paste it on another slice (as modifying a contour may be faster than drawing it from scratch)
3. Visually see the segmentation in all four default viewers (axial, coronal, sagittal and 3D) of CamiTK-imp
4. Save the segmentation as an image
5. Make the tool less error prone

And let’s do the implementation in increments.

Step 01: Use the wizard to create a CEP

We will use this tool to perform some operations on images. Hence, we only need to create a CamiTK Action that acts on CamiTK ImageComponents. But following the good practices of CamiTK let us create a CEP called Segmentor. The Segmentor CEP will have only one action also called Segmentor.

Let’s create this CEP using the CamiTK-Wizard. While creating the action let’s add two parameters to it.

1. Contour Color: the color of the contour
2. Contour Width: the line width of the contour

Build the CEP and add the Segmentor action to CamiTK-imp. Load an image to CamiTK-imp and you will find the Segmentor action under Tutorial action family. When you apply the action, it will create a copy of the current image by executing the automatically generated code.

Step 02: Remove the unnecessary code snippets and create the necessary GUI components

Usually, an action is executed when the user clicks the Apply button. This behavior is suitable when you perform only one operation on the data component. Since we are planning to use this tool to perform multiple operations on an image (draw contours, paste the last drawn contour, create segmented image, etc.), using only the Apply button is not adequate. Consequently, we should disable the default Apply button and provide our own GUI widgets to handle our needs.

We can disable the default Apply button in the following manner:

dynamic_cast<ActionWidget*>(Action::getWidget())->setButtonVisibility(false);

Let us determine the additional GUI widgets that we require for this tool.

• Radio buttons to specify on which viewer the contours are drawn
• Single slice related widgets
  • Push button to complete the current contour
  • Push button to reset the current contour
  • Push button to accept the current drawn contour as the final contour
  • Push button to paste the last drawn contour on the selected slice
• Whole image related widgets
  • Reset everything
  • Save the segmentation as an image

The above mentioned widgets will be created and added to a QFrame. The default widget created by the Action will also be added to this QFrame. Ultimately, this QFrame will be returned by the overloaded getWidget( ) method of the action.

Finally, let’s remove the content of the apply( ) method and delete the process( ) method given by default.

When you load the action and click on the provided buttons nothing executes as we have not specified what to do when the user interacts with those widgets.

Step 03: Draw contours in a given orthogonal viewer (axial, coronal or sagittal) of CamiTK-imp

As the viewer on which the contours are drawn is going to be used throughout the implementation, we will create an ENum for the viewers. We will update the currently selected viewer depending on the radio button selected in the GUI. We could listen to the Clicked( ) signal of the radio button and set the currently selected viewer accordingly. In order to do this, we need to access the GUI widgets. Hence, we should set the objects names of the three radio buttons using thesetObjectName( ) method. Since we are going to use the signals and slots mechanism, we have to include the Q_OBJECT macro in the class header file.

In order to draw contours on a viewer we can use the vtkContourWidget². We will create a member variable that will hold an instance of a vtkContourWidget. The contour should be drawn on the selected slice of the selected viewer. In order to calculate the plane where the contour should be drawn, we require the information of the ImageComponent on which the contours are drawn. We will use another member variable to hold the currently selected ImageComponent call it currentImageComp. We should set/check currentImageComp every time the QWidget of the action is requested. If the selected ImageComponent is different from the previously selected one, then we have to handle the changes. We will see how this is done at a later step. For the moment, let’s assume that the actions will be carried out on the same ImageComponent.

We should draw on only one slice at a time. Hence, every time we want to draw contours on a different viewer or a different slice of the same viewer we have to reset the contours. Let’s define the resetCurrentContourWidget( ) method that handles the above mentioned requirements. And let’s setup this method to be executed when the Reset button is clicked and also when the selected viewer is changed.

Once the action is loaded, you will be able to draw contours on any of the slices of the image.

Step 04: Update the color and width of the contour and implement automatic contour completion

Try changing the color or width of the contour. You will observe that nothing happens although we update the color and width of the contour inside the resetCurrentContourWidget( ) method. That is because the values of properties of an action are updated only when the Apply button is clicked. But we have disabled the Apply button of the action previously. In order to correct this behavior all we need to do is to set the setAutoUpdateProperty( ) to true of the ActionWidget when we generate our custom QWidget of the action.

The vtkContourWidget comes with a handy method to complete the currently drawn contour. In order to have a closed loop we have to make sure at least two points are already drawn when completing the contour. Let’s implement that in a method called completeCurrentContour( ) and set it up with the clicked( ) signal of the Complete button.

Now you will be able to complete the contour drawn.

Step 05: Build the segmentation image and show the region inside the contour in the orthogonal viewers

First we are going to create an image that will hold information about the development of the segmentation. Let’s define a member variable to hold this image called segmentedImage. The first time the currentImageComp is assigned, we create segmentedImage and set every voxel of it to black. Every time a new region is added this image will be updated to reflect the changes. In other words, segmentedImage will work as a mask to represent the segmentation.

Now that we have our segmentedImage, we can move onto showing the region inside the contour on the respective orthogonal viewers. It is done in two steps.

1. Find the voxels that correspond to the region inside the contour and set those voxels of segmentedImage as white (done by updateSegmentedImage( ) method)
2. Find the intersection between the white voxels of segmentedImage and the currently displayed slice of the currentImageComp and draw an overlay on each orthogonal viewer (done by drawOverlaysOn2DViewers( ) method)

First step will be implemented in the updateSegmentedImage( ) method. The heart of this step is the use of vtkPolyDataToImageStencil³ filter that converts the PolyData into an image stencil. The PolyData from the contourWidget will first be triangulated and then extruded using the vtkContourTriangulator and vtkLinearExtrusionFilter filters respectively. With contours, vtkPolyDataToImageStencil only works if the image planes are aligned with the axial planes (see ⁴). Consequently, if the contours are drawn on the sagittal or coronal viewers we have to transform the relevant plane to be along the axial plane. For the sagittal viewer the original x,y,z coordinates will become y,z,x and for the coronal viewer the original x,y,z coordinates will become z,x,y. Once the PolyData is transformed into an image stencil, it will be cut using another temporary while image in order to find the corresponding selected voxels from the original region inside the contour. Finally, these voxels will be marked as white in the segmentedImage. In order to achieve this, we do not have to check the whole segmentedImage but only the corresponding slice along the relevant direction.

In the second step we will implement the drawOverlaysOn2DViewers( ) method. We will have to interact with all 2D orthogonal viewers during this step. In order to do that let’s create an actor and a mapper for each orthogonal viewer and add that to each RendererWidget or the corresponding viewer. We do not have to do this every time the drawOverlaysOn2DViewers( ) method is called. Instead we will do this once when the QWidget of the action is created and make the actors and mappers available as members of the class. Since vtkImageData is a structured point dataset, we can use vtkImageDataGeometryFilter to extract a single slice from the segmentedImage. Then, we can use vtkThresholdPoints to retrieve only the points that are white. Finally, we can give these white points as the input for the mapper of the corresponding viewer which in turn will take care of drawing the overlay.

These steps will be executed with the clicked( ) signal of the Accept button. Let’s call the corresponding slot acceptCurrentContour( ).

Now when you click the Accept button the region inside the contour will be highlighted and the corresponding regions of other viewers will also be highlighted.

Step 06: Auto update the segmented regions with slice sliders

Draw contours on more than one slice and you will observe that the previously drawn contours become invisible when you move back to the previous slice.

That is normal as we have to update the 2D overlay when the slices are changed. We can do that by listening to the selectionChanged( ) signal of the 2D viewers. We only have to update the corresponding 2D viewer. Let’s name the corresponding slot as updateOverlayOnSingle2DViewer( ) method. Since repeating the same lines of code is not a good coding practice, we will modify the drawOverlaysOn2DViewers( ) method to handle this situation as well.

Now when you change the viewer sliders the segmented regions will be correctly updated.

Step 07: Show the segmented regions in the 3D viewer

We will create another member variable to hold the segmented mesh and call it segmentedMesh. As we already have an updated segmentedImage, it is very easy to get a surface mesh of the segmented regions using vtkMarchingCubes filter⁵. This behavior is implemented in the updateSegmentedMesh( ) method.

Now when you add a new region to the segmentation, it will immediately be shown in the 3D viewer.

Step 08: Correcting for any arbitrary positioning in the world coordinates

After opening an image in CamiTK-imp, use the Edit Frame action under Frame to give an arbitrary transform to the image in the world coordinates. And then start drawing contours on the image. You will observe that the 3D mesh drawn by our tool is not placed correctly.

In order to correct this mishap we need to transform all PolyData belonging to the segmentedMesh from the image to the world coordinates. The updateSegmentedMesh() method is modified accordingly.

Now even if you draw on an image which is subjected to an arbitrary transformation the segmentation mesh will be correctly shown in the 3D viewer.

Step 09: Paste the last contour on another slice

In order to paste the last contour on another slice, all we need to do is to calculate the translation of the last drawn contour along the corresponding axis and then transform the PolyData of the contour to the new location. We also need to clean the PolyData as otherwise too many points will be generated as the use of this function increases. Then, we have to reset the contourWidget at the new location but this time we should initialize it with the PolyData from the previous contour. This functionality is implemented in the pasteLastContour( ) method and it will be triggered when the clicked( ) signal is emitted from the Paste Last button.

Similar to step 06, we will refrain from copying the same lines of code to reset the contour. Instead, lets modify the already implemented resetCurrentContourWidget( ) method and bring the common commands to a common method.

Now you can use Paste Last button to paste the last drawn contour on another image.

Step 10: Implement Reset All and Save As Image methods

In order to reset everything all we need to do is reset the segmentedImage and then update 3D viewer and 2D viewers. Let’s implement this behavior in a method called resetEverything( ) and set this to be executed when clicked( ) signal of Reset button is emitted.

In order to save the segmented image, we create a new ImageComponent and save is using the action Save. This behavior is implemented in the saveSegmentedImage( ) method and is executed when clicked( ) signal of Save As Image button is emitted.

Now the Reset and Save As Image button are functional.

Step 11: Make the tool less error prone

All the desired functionalities of the tool are implemented. However, since we can click any button without any restrictions, certain click combinations will lead the tool to crash. (e.g. Clicking Paste Last when you haven’t drawn anything leads the tool to crash.)

If we could restrict the access to Accept and Paste Last buttons we can make the tool more robust. Accept button should be enabled when:

1. the contour is closed
2. when the last contour is pasted on another slice

Paste Last button should be enabled only when Accept button is clicked. All other times both buttons should be disabled.

One way of completing the contour is by placing the last point of the contour on the first point of it. We can capture this event if we listen to EndInteractionEvent of the vtkContourWidget. The easiest way to do this is by making the Segmentor class a subclass of vtkCommand class. When we do that, we have to overload the Execute( ) method of the vtkCommand class in order to instantiate Segmentor class. Let’s encapsulate the changing button availability in a method called changeButtonAvailability( ).

Certains buttons are enabled/disabled appropriately in order to make the tool more robust.

Step 12: Handle different ImageComponents

When the currentImageComp on which the Segmentor action is applied changes, we have to reset everything in order to make the tool robust. Deleting the segmentedImage and segmentedMesh get the job done. Hence, users are advised to finish segmenting one image before moving on to another.

Now the user can change the ImageComponent on which the contours are drawn.

Almost done!

It is our hope that you have gained some insight into making a CamiTK ready tool by increments. As with any piece of code, you may run into bugs with this tool. If that is the case, do not hesitate to let us know and we will try to fix them asap.

References