Hello, I read your paper and was impressed by the solid work. At the same time, I have a question: under the GQA settings, why are the attention scores merged before the softmax, rather than after? Before the softmax, due to the influence of the Query, the mean and variance of the attention scores might vary significantly across the different heads.
Hello, I read your paper and was impressed by the solid work. At the same time, I have a question: under the GQA settings, why are the attention scores merged before the softmax, rather than after? Before the softmax, due to the influence of the Query, the mean and variance of the attention scores might vary significantly across the different heads.