Question about Training Time and Multi-GPU Scaling of R-SCoRe

[AbbeGo]

Thank you for your excellent work on R-SCoRe. I have a question regarding the training time and multi-GPU scalability.

When training the final model on the Aachen dataset, I observe that the training process takes approximately 5–6 hours on a single RTX 4090 or RTX 3090 GPU. The training time appears to be similar on both GPUs.

Additionally, when using multiple GPUs for training, I do not observe a noticeable speed improvement compared to single-GPU training.

I would like to confirm:

Is a 5–6 hour training time expected for the final model?

Is the current implementation designed to benefit from multi-GPU parallelization?

If multi-GPU training is supported, are there specific configurations required to achieve speedup?

I would appreciate any clarification or guidance on this matter.

[xjiangan]

Thank you for your interest in R-SCoRe and for the detailed observations.

Training time on Aachen:

On our side, training the final model takes about ~4 hours on 4× RTX 4090. We generally aim to keep the runtime within 4 hours so that it fits into our cluster’s short job queue. A 5–6 hour runtime is still within a reasonable range. The exact time can vary depending on factors such as CPU performance, storage speed, and the PyTorch/CUDA configuration. In our experience, the 4090 is usually faster than the 3090.

Multi-GPU training:

The current configuration uses a fixed batch size per GPU. As a result, adding more GPUs increases the total batch size rather than reducing the wall-clock time, which is why you do not see a speedup by default.

If your goal is to reduce training time, you can instead:

• Keep the total batch size fixed, and
• Reduce the per-GPU batch size proportionally to the number of GPUs.

From a localization accuracy perspective, for larger scenes a larger total batch size can be beneficial for training stability and performance, while for smaller scenes the benefit is usually marginal. We did not specifically optimize training efficiency to achieve certain accuracy for certain scene size, so there is room for improvement.

If you are interested in faster training, you may also experiment with:

• Fewer training steps / batch size with adjusted learning rates
• Alternative optimizers / schedules
• MoE network architecture

In some preliminary experiments, we were able to reduce training time by several times while maintaining similar localization accuracy.

Hope this clarifies things, and please let us know if you run into further issues.

[AbbeGo]

Thank you for your reply. I am very interested in the improvements you mentioned earlier, such as the introduction of Mixture-of-Experts (MoE). May I ask whether these modifications are planned to be released in the corresponding GitHub repository?

I am also highly interested in your latest work, ImLoc. Could you kindly share whether there are any plans for making the code publicly available?

In addition, I am curious about your shift from Scene Coordinate Regression (SCR) methods to Feature Matching (FM)-based approaches. Do you believe that SCR methods have reached a performance bottleneck in large-scale localization scenarios? In particular, do you think SCR struggles to further improve accuracy in large environments, or that it may lack robustness under scene variations such as illumination changes or dynamic elements (e.g., pedestrians)?

I would greatly appreciate your insights on these points.