RocketMQ is unable to handle request size of 10Kb

[lazyCoder10]

For an academic project, we are benchmarking RocketMQ with the OpenMessagingBenchmark Framework (https://openmessaging.cloud/docs/benchmarks/). When running the simulation for 10Kb message size with 1-topic-1partition configuration, the benchmarking tool is throwing the below error:

16:07:36.962 [local-worker-1-1] WARN MessageProducer - Write error on message
java.util.concurrent.CompletionException: org.apache.rocketmq.client.exception.MQClientException: executor rejected
For more information, please visit the url, http://rocketmq.apache.org/docs/faq/
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332) ~[?:?]
at java.util.concurrent.CompletableFuture.uniRunNow(CompletableFuture.java:813) ~[?:?]
at java.util.concurrent.CompletableFuture.uniRunStage(CompletableFuture.java:803) ~[?:?]
at java.util.concurrent.CompletableFuture.thenRun(CompletableFuture.java:2195) ~[?:?]
at io.openmessaging.benchmark.worker.MessageProducer.sendMessage(MessageProducer.java:47) ~[classes/:?]
at io.openmessaging.benchmark.worker.LocalWorker.lambda$submitProducersToExecutor$6(LocalWorker.java:206) ~[classes/:?]
at java.util.ArrayList.forEach(ArrayList.java:1511) ~[?:?]
at io.openmessaging.benchmark.worker.LocalWorker.lambda$submitProducersToExecutor$7(LocalWorker.java:204) ~[classes/:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.65.Final.jar:4.1.65.Final]
at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: org.apache.rocketmq.client.exception.MQClientException: executor rejected
For more information, please visit the url, http://rocketmq.apache.org/docs/faq/
at org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl.send(DefaultMQProducerImpl.java:519) ~[rocketmq-client-4.9.3.jar:4.9.3]
at org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl.send(DefaultMQProducerImpl.java:482) ~[rocketmq-client-4.9.3.jar:4.9.3]
at org.apache.rocketmq.client.producer.DefaultMQProducer.send(DefaultMQProducer.java:376) ~[rocketmq-client-4.9.3.jar:4.9.3]
at io.openmessaging.benchmark.driver.rocketmq.RocketMQBenchmarkProducer.sendAsync(RocketMQBenchmarkProducer.java:43) ~[classes/:?]
at io.openmessaging.benchmark.worker.MessageProducer.sendMessage(MessageProducer.java:46) ~[classes/:?]
... 9 more
Caused by: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2f39d323[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@22dc1731[Wrapped task = org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl$3@cf5f484]] rejected from java.util.concurrent.ThreadPoolExecutor@312a63fb[Running, pool size = 16, active threads = 16, queued tasks = 50000, completed tasks = 118613]
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2070) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:833) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1365) ~[?:?]
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:123) ~[?:?]
at org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl.send(DefaultMQProducerImpl.java:501) ~[rocketmq-client-4.9.3.jar:4.9.3]
at org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl.send(DefaultMQProducerImpl.java:482) ~[rocketmq-client-4.9.3.jar:4.9.3]
at org.apache.rocketmq.client.producer.DefaultMQProducer.send(DefaultMQProducer.java:376) ~[rocketmq-client-4.9.3.jar:4.9.3]
at io.openmessaging.benchmark.driver.rocketmq.RocketMQBenchmarkProducer.sendAsync(RocketMQBenchmarkProducer.java:43) ~[classes/:?]
at io.openmessaging.benchmark.worker.MessageProducer.sendMessage(MessageProducer.java:46) ~[classes/:?]
... 9 more

But when we reduced the message size to 1kb with the same topic and partition configuration, it works as expected. To resolve the issue I have followed the following steps:

• Increased the JVM memory to 24G for both broker and nameserver. Primarily my assumption was the error is due to memory starvation of JVM. But after increasing the memory the problem still remains. All the brokers have 32G of memory and 8 Core CPU.
• Changed the adminBrokerThreadPoolNums to 100000 in broker properties. We are using 3-master-no-slave configuration. But the problem still remains.

[GenerousMan]

Maybe 10KB will put a lot of pressure on the broker under the tps set in openmessaging, resulting in the rejection of some message sending requests.
Appropriately reducing the sending rate may solve this problem, or we need to turn on backpressure in such occasions.

[lazyCoder10]

Actually we are comparing several JVM based MQs to learn how the MQs behave in different scenario. If we rate limit for rocketMq, won't that be an unfair advantage?

Without limiting the tps to a lower level, what would be your suggestion? We need to run load test for 10Kb, 100Kb, 1000Kb message size.

[mymx2]

focus:

• Comparative Evaluation of Java Virtual Machine-Based Message Queue Services: A Study on Kafka, Artemis, Pulsar, and RocketMQ
• [ISSUE #9451]Optimize back pressure default value to 1024. #9452