[Improvement]: Improve worker message passing to reduce memory leak

[Improvement]: Improve worker message passing to reduce memory leak #1896