The "commonz, S = self.TransformerEncoder(commonz)" is incorrect, and the code cannot be implemented as currently written.

[RichWang2022]

I recently attempted to reproduce your code and encountered several issues. It appears that the nn.TransformerEncoder only yields a single output. However, in line 74 of the file network.py, the code "commonz, S = self.TransformerEncoder(commonz)" seems to contain an error. I would appreciate if you could address this issue at your earliest convenience. Furthermore, I am interested in understanding how the global relationship matrix S is computed, as it is a critical aspect of your study. Please consider updating your code accordingly.

[marmotxcw]

#1

[guozhuaizhuai123]

I want to ask you some question, how compute the S

[forro1]

Hi, I also met the situation, and it seems like the error is caused by a version issue. Although the author gives a solution in FILE Obtain-S.docx, it feels not much clear. Here is how I solve it:
Due to the update of TransformerEncoderLayer, the shape of input features should be (batch, seq, feature), while the shape of commonz is (batchsize, dim). And the return of S also needs changing.
So first in FILE network.py, changing function GCFAgg(self, xs) to:

    def GCFAgg(self, xs):
        zs = []
        for v in range(self.view):
            x = xs[v]
            z = self.encoders[v](x)
            zs.append(z)
        
        commonz = torch.cat(zs, dim=-1)
        commonz = commonz.unsqueeze(1)
        commonz, S = self.TransformerEncoderLayer(commonz)
        commonz = commonz.squeeze(1)
        commonz = normalize(self.Common_view(commonz), dim=1)
        return commonz, S.squeeze(0)

Secondly, change the forward function of class TransformerEncoderLayer:

    def forward(self, src: Tensor, src_mask: Optional[Tensor] = None, src_key_padding_mask: Optional[Tensor] = None) -> Tensor:
        r"""Pass the input through the encoder layer.

        Args:
            src: the sequence to the encoder layer (required).
            src_mask: the mask for the src sequence (optional).
            src_key_padding_mask: the mask for the src keys per batch (optional).

        Shape:
            see the docs in Transformer class.
        """
        src2, S = self.self_attn(src, src, src, attn_mask=src_mask,
                              key_padding_mask=src_key_padding_mask, need_weights=True)
        src = src + self.dropout1(src2)
        src = self.norm1(src)
        src2 = self.linear2(self.dropout(self.activation(self.linear1(src))))
        src = src + self.dropout2(src2)
        src = self.norm2(src)
        return src, S

Now, the error should be solved as expected.
In addition, if the dataset is small (data size is lower than batch_size), try to reduce batch size to avoid errors caused by loss calculation. Or set drop_last=False when initializing dataloader and reset the batchsize same as the size of dataset.
Hope this can make help