As titled, when we do benchmarking, there are two typical scenarios: one is to control the batch size to ensure that the server receives a specific batch size; the other is to control the request rate to ensure that the client sends requests at a specific rate. And we hope to specify multiple values in the genai bench for both scenarios so that we do not need to repeat tests each time (this may require clearing cache after each test).
As titled, when we do benchmarking, there are two typical scenarios: one is to control the batch size to ensure that the server receives a specific batch size; the other is to control the request rate to ensure that the client sends requests at a specific rate. And we hope to specify multiple values in the genai bench for both scenarios so that we do not need to repeat tests each time (this may require clearing cache after each test).