Questions about AP on scannet

[sunmaosheng755]

Hi, Thanks for your great work.
By training your model 3detr-m, I got the following results on scannet(epoch1080):
====================Final Eval Numbers.
mAP0. 25, mAP0. 50: 62.12, 43.58
AR0. 25, AR0. 50: 74.09, 54.71
====================Best Eval Numbers.
mAP0. 25, mAP0. 50: 63.78, 39.67
AR0. 25, AR0. 50: 78.57, 53.71
With your help, the current result is close to 65.0. Is this a reasonable result?
Can you help me see what caused the result not to reach 65.0 / 47.0?

[sunmaosheng755]

• _libgcc_mutex=0.1=main
• blas=1.0=mkl
• bzip2=1.0.8=h7b6447c_0
• ca-certificates=2021.7.5=h06a4308_1
• certifi=2021.5.30=py36h06a4308_0
• cudatoolkit=10.2.89=hfd86e86_1
• dataclasses=0.8=pyh4f3eec9_6

- ffmpeg=4.3=hf484d3e_0

• freetype=2.10.4=h5ab3b9f_0
• giflib=5.2.1=h7b6447c_0
• gmp=6.2.1=h2531618_2
• gnutls=3.6.15=he1e5248_0
• intel-openmp=2021.3.0=h06a4308_3350
• jpeg=9b=h024ee3a_2
• lame=3.100=h7b6447c_0
• lcms2=2.12=h3be6417_0
• ld_impl_linux-64=2.35.1=h7274673_9
• libffi=3.3=he6710b0_2
• libgcc-ng=9.1.0=hdf63c60_0
• libiconv=1.15=h63c8f33_5
• libidn2=2.3.2=h7f8727e_0
• libpng=1.6.37=hbc83047_0
• libstdcxx-ng=9.1.0=hdf63c60_0
• libtasn1=4.16.0=h27cfd23_0
• libtiff=4.2.0=h85742a9_0
• libunistring=0.9.10=h27cfd23_0
• libuv=1.40.0=h7b6447c_0
• libwebp=1.2.0=h89dd481_0
• libwebp-base=1.2.0=h27cfd23_0
• lz4-c=1.9.3=h295c915_1
• mkl=2020.2=256
• mkl-service=2.3.0=py36he8ac12f_0
• mkl_fft=1.3.0=py36h54f3939_0
• mkl_random=1.1.1=py36h0573a6f_0
• ncurses=6.2=he6710b0_1
• nettle=3.7.3=hbbd107a_1
• ninja=1.10.2=hff7bd54_1
• numpy=1.19.2=py36h54aff64_0
• numpy-base=1.19.2=py36hfa32c7d_0
• olefile=0.46=pyhd3eb1b0_0
• openh264=2.1.0=hd408876_0
• openssl=1.1.1l=h7f8727e_0
• pillow=8.3.1=py36h5aabda8_0
• pip=21.2.2=py36h06a4308_0
• python=3.6.13=h12debd9_1

- pytorch=1.8.2=py3.6_cuda10.2_cudnn7.6.5_0

• readline=8.1=h27cfd23_0
• setuptools=58.0.4=py36h06a4308_0
• six=1.16.0=pyhd3eb1b0_0
• sqlite=3.36.0=hc218d9a_0
• tk=8.6.10=hbc83047_0

- torchaudio=0.8.2=py36

- torchvision=0.9.2=py36_cu102

• typing_extensions=3.10.0.2=pyh06a4308_0
• wheel=0.37.0=pyhd3eb1b0_1
• xz=5.2.5=h7b6447c_0
• zlib=1.2.11=h7b6447c_3
• zstd=1.4.9=haebb681_0
• pip:
  • ffmpeg=4.3=hf484d3e_0
  • pytorch=1.8.2=py3.6_cuda10.2_cudnn7.6.5_0
  • torchvision=0.9.2=py36_cu102
  • torchaudio=0.8.2=py36
  • decorator==4.4.2
  • networkx==2.5.1
  • plyfile==0.7.4
  • pointnet2==0.0.0
  • protobuf==3.19.3
  • scipy==1.5.4
  • tensorboardx==2.4.1
  • trimesh==2.35.39

[sunmaosheng755]

[imisra]

Hi @sunmaosheng755

Apologies for the late response. I have responded to your detailed email.

[Arielqjl]

Hi, I've got the similar performance on vanilla encoder:
====================Final Eval Numbers.
mAP0.25, mAP0.50: 61.32, 39.90
AR0.25, AR0.50: 74.54, 52.25
and masked encoder:
====================Final Eval Numbers.
mAP0.25, mAP0.50: 62.30, 43.99
AR0.25, AR0.50: 74.97, 55.70
Would you please help me reach the results of 65.0/47.0?
Thank you!