多卡训练检测模型，启动训练时卡死，没有任何反应，一直卡在Retry to connect to 192.168.70.105:47715 while the server is not yet listening这个地方

[chiuchangfu]

请提供下述完整信息以便快速定位问题/Please provide the following information to quickly locate the problem

• 系统环境/System Environment：CentOS Linux release 7.8.2003 (Core)
• 版本号/Version：
  • Python：3.9.17
  • Paddle：2.5.1
  • PaddleOCR：2.7
  • CUDA版本：11.2
  • cuDNN版本：8.2.1
  • 显卡型号：NVIDIA GeForce RTX 2080 Ti
  • 显卡数量：4
• 问题相关组件/Related components：PP-OCRv4-server
• 运行指令/Command Code：python -m paddle.distributed.launch --gpus '0,1,2,3' tools/train.py -c configs/det/ch_PP-OCRv4/ch_PP-OCRv4_det_teacher.yml
• 完整报错/Complete Error Message：
  [2023/08/25 10:24:40] ppocr INFO: learning_rate : 0.001
  [2023/08/25 10:24:40] ppocr INFO: name : Cosine
  [2023/08/25 10:24:40] ppocr INFO: warmup_epoch : 2
  [2023/08/25 10:24:40] ppocr INFO: name : Adam
  [2023/08/25 10:24:40] ppocr INFO: regularizer :
  [2023/08/25 10:24:40] ppocr INFO: factor : 1e-6
  [2023/08/25 10:24:40] ppocr INFO: name : L2
  [2023/08/25 10:24:40] ppocr INFO: PostProcess :
  [2023/08/25 10:24:40] ppocr INFO: box_thresh : 0.6
  [2023/08/25 10:24:40] ppocr INFO: max_candidates : 1000
  [2023/08/25 10:24:40] ppocr INFO: name : DBPostProcess
  [2023/08/25 10:24:40] ppocr INFO: thresh : 0.3
  [2023/08/25 10:24:40] ppocr INFO: unclip_ratio : 1.5
  [2023/08/25 10:24:40] ppocr INFO: Train :
  [2023/08/25 10:24:40] ppocr INFO: dataset :
  [2023/08/25 10:24:40] ppocr INFO: data_dir : ./doc/datasets/changfu/train/
  [2023/08/25 10:24:40] ppocr INFO: label_file_list : ['./doc/datasets/changfu/train.txt']
  [2023/08/25 10:24:40] ppocr INFO: name : SimpleDataSet
  [2023/08/25 10:24:40] ppocr INFO: ratio_list : [1.0]
  [2023/08/25 10:24:40] ppocr INFO: transforms :
  [2023/08/25 10:24:40] ppocr INFO: DecodeImage :
  [2023/08/25 10:24:40] ppocr INFO: channel_first : False
  [2023/08/25 10:24:40] ppocr INFO: img_mode : BGR
  [2023/08/25 10:24:40] ppocr INFO: DetLabelEncode : None
  [2023/08/25 10:24:40] ppocr INFO: CopyPaste : None
  [2023/08/25 10:24:40] ppocr INFO: IaaAugment :
  [2023/08/25 10:24:40] ppocr INFO: augmenter_args :
  [2023/08/25 10:24:40] ppocr INFO: args :
  [2023/08/25 10:24:40] ppocr INFO: p : 0.5
  [2023/08/25 10:24:40] ppocr INFO: type : Fliplr
  [2023/08/25 10:24:40] ppocr INFO: args :
  [2023/08/25 10:24:40] ppocr INFO: rotate : [-10, 10]
  [2023/08/25 10:24:40] ppocr INFO: type : Affine
  [2023/08/25 10:24:40] ppocr INFO: args :
  [2023/08/25 10:24:40] ppocr INFO: size : [0.5, 3]
  [2023/08/25 10:24:40] ppocr INFO: type : Resize
  [2023/08/25 10:24:40] ppocr INFO: EastRandomCropData :
  [2023/08/25 10:24:40] ppocr INFO: keep_ratio : True
  [2023/08/25 10:24:40] ppocr INFO: max_tries : 50
  [2023/08/25 10:24:40] ppocr INFO: size : [640, 640]
  [2023/08/25 10:24:40] ppocr INFO: MakeBorderMap :
  [2023/08/25 10:24:40] ppocr INFO: shrink_ratio : 0.4
  [2023/08/25 10:24:40] ppocr INFO: thresh_max : 0.7
  [2023/08/25 10:24:40] ppocr INFO: thresh_min : 0.3
  [2023/08/25 10:24:40] ppocr INFO: total_epoch : 500
  [2023/08/25 10:24:40] ppocr INFO: MakeShrinkMap :
  [2023/08/25 10:24:40] ppocr INFO: min_text_size : 8
  [2023/08/25 10:24:40] ppocr INFO: shrink_ratio : 0.4
  [2023/08/25 10:24:40] ppocr INFO: total_epoch : 500
  [2023/08/25 10:24:40] ppocr INFO: NormalizeImage :
  [2023/08/25 10:24:40] ppocr INFO: mean : [0.485, 0.456, 0.406]
  [2023/08/25 10:24:40] ppocr INFO: order : hwc
  [2023/08/25 10:24:40] ppocr INFO: scale : 1./255.
  [2023/08/25 10:24:40] ppocr INFO: std : [0.229, 0.224, 0.225]
  [2023/08/25 10:24:40] ppocr INFO: ToCHWImage : None
  [2023/08/25 10:24:40] ppocr INFO: KeepKeys :
  [2023/08/25 10:24:40] ppocr INFO: keep_keys : ['image', 'threshold_map', 'threshold_mask', 'shrink_map', 'shrink_mask']
  [2023/08/25 10:24:40] ppocr INFO: loader :
  [2023/08/25 10:24:40] ppocr INFO: batch_size_per_card : 4
  [2023/08/25 10:24:40] ppocr INFO: drop_last : False
  [2023/08/25 10:24:40] ppocr INFO: num_workers : 4
  [2023/08/25 10:24:40] ppocr INFO: shuffle : True
  [2023/08/25 10:24:40] ppocr INFO: profiler_options : None
  [2023/08/25 10:24:40] ppocr INFO: train with paddle 2.5.1 and device Place(gpu:0)
  ======================= Modified FLAGS detected =======================
  FLAGS(name='FLAGS_selected_gpus', current_value='0', default_value='')
  =======================================================================
  I0825 10:24:40.016446 74745 tcp_utils.cc:181] The server starts to listen on IP_ANY:47715
  I0825 10:26:47.185032 74745 tcp_utils.cc:107] Retry to connect to 192.168.70.105:47715 while the server is not yet listening.
  I0825 10:28:57.489110 74745 tcp_utils.cc:107] Retry to connect to 192.168.70.105:47715 while the server is not yet listening.
  I0825 10:31:07.792940 74745 tcp_utils.cc:107] Retry to connect to 192.168.70.105:47715 while the server is not yet listening.

[SWHL]

如果问题仍然存在，请拉取仓库main分支最新代码，再次尝试

[SidneyRey]

训练表格识别时，碰到同样的问题

[SWHL]

paddlepaddle升级到最新版本试试