Multiprocessing

[Jiaxuan777]

Hello,
I'm glad to see your project. It's useful for what I am doing now. But I have a problem to inquire you. I have put this WSI Normalizer in my dataloader, and the transform part is in the function getitem(). When I set the 'num_workers': > 1 to accelerate. It will cause the error like this:

I spent two days on it. However, I cannot solve it except for setting the ‘num workers = 0’,but my data is so big, this way is too slow.
And I referred to this settings.Peter554/StainTools#43 But it still no use. Can you help me? Thank you!

[cwlkr]

Hello,

Yes, this is unfortunately not straight forward to solve. The problem is in how CUDA is used in accelerating tasks which does not lend to multiprocessing. CUDA, does multiprocessing by splitting a task into smaller simple step tasks that can be run in parallel on GPU kernels, while CPU multiprocessing takes a tasks and runs the same task for different inputs in parallel. Unfortunately, in how CUDA is constructed these are not easily inter-mixable. This is why ‘num workers = 0’, works while setting num_workers > 0 will crash.

My code works therefore best for larger input tiles (normalizing a WSI), but not in a parallel DataLoader.

I have found that in StainTools, it is actually faster to use SPAMS settings akin to the wonderful stain_mixup project. I have used the same in my related augmentation strategy stainlist_augmentation.

Best,
Cédric

[Jiaxuan777]

Thank you! I konw it.