How can I apply CRF with image?

[sangrockEG]

Hello.

I tried to apply your CRF-layer to my semantic segmentation model.
But I found that the layer does not get an image as its input.
forward(self, x, spatial_spacings=None, verbose=False)

x is an output of my model, which functions as a unary potential, right?
At first, I thought the spatial_spacings is somewhat related to the data term, but it seems not...
(Because it has a shape of (batch_size, len(spatial)).)

Any help will be really appreciated.

[ArthurDevlieger]

Hi sangrockEG,
I am having the same issue, did you solve this issue or find another way to implement a CRF layer?

[sangrockEG]

Hi, ArthurDevlieger.
Sadly I failed to solve this.
I gave up training my model in an end-to-end manner.
And just employ typical denseCRF.

[ArthurDevlieger]

So than you just tuned the CRF parameters on a validation set or something? And which denseCRF implementation did you use?

[sangrockEG]

I used pydenseCRF with commonly used parameters.

d.addPairwiseGaussian(sxy=3/scale_factor, compat=3)
d.addPairwiseBilateral(sxy=80/scale_factor, srgb=13, rgbim=np.copy(img), compat=10)

I can't sure this setting is general for all cases, but it works pretty well in my case. (PASCAL VOC 2012)

[mishgon]

@sangrockEG , @ArthurDevlieger , in the current implementation, the pairwise potential kernel function includes only smoothness kernel term, which does not depend on the input image. Therefore, image is not fed as input to CRF.forward. Unfortunately, the appearance kernel term (which requires an image as input) is not supported for now.