Skip to content

Commit

Permalink
Results from GH action on NVIDIA_RTX4090x1
Browse files Browse the repository at this point in the history
  • Loading branch information
@arjunsuresh
arjunsuresh committed Dec 25, 2024
1 parent 1e3ced2 commit bb62ad8
Show file tree
Hide file tree
Showing 54 changed files with 20,572 additions and 20,572 deletions.
16 changes: 8 additions & 8 deletions closed/MLCommons/systems/RTX4090x1-nvidia_original-gpu-tensorrt-vdefault-default_config.json
View file
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
{
"accelerator_frequency": "2520000 MHz",
"accelerator_frequency": "2610000 MHz",
"accelerator_host_interconnect": "N/A",
"accelerator_interconnect": "N/A",
"accelerator_interconnect_topology": "",
"accelerator_memory_capacity": "23.64971923828125 GB",
"accelerator_memory_capacity": "23.54595947265625 GB",
"accelerator_memory_configuration": "N/A",
"accelerator_model_name": "NVIDIA GeForce RTX 4090",
"accelerator_on-chip_memories": "",
Expand All @@ -16,17 +16,17 @@
"host_network_card_count": "1",
"host_networking": "Gig Ethernet",
"host_networking_topology": "N/A",
"host_processor_caches": "L1d cache: 576 KiB, L1i cache: 384 KiB, L2 cache: 24 MiB, L3 cache: ",
"host_processor_core_count": "24",
"host_processor_frequency": "5800.0000",
"host_processor_caches": "L1d cache: 512 KiB, L1i cache: 512 KiB, L2 cache: 16 MiB, L3 cache: 64 MiB",
"host_processor_core_count": "16",
"host_processor_frequency": "5881.0000",
"host_processor_interconnect": "",
"host_processor_model_name": "13th Gen Intel(R) Core(TM) i9-13900K",
"host_processor_model_name": "AMD Ryzen 9 7950X 16-Core Processor",
"host_processors_per_node": "1",
"host_storage_capacity": "9.4T",
"host_storage_capacity": "6.8T",
"host_storage_type": "SSD",
"hw_notes": "",
"number_of_nodes": "1",
"operating_system": "Ubuntu 20.04 (linux-6.8.0-49-generic-glibc2.31)",
"operating_system": "Ubuntu 20.04 (linux-6.8.0-51-generic-glibc2.31)",
"other_software_stack": "Python: 3.8.10, GCC-9.4.0, Using Docker , CUDA 12.2",
"status": "available",
"submitter": "MLCommons",
Expand Down
8 changes: 4 additions & 4 deletions ...a_original-gpu-tensorrt-vdefault-default_config/retinanet/multistream/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ pip install -U cmind

cm rm cache -f

cm pull repo mlcommons@mlperf-automations --checkout=a9e8329cf5f036aea3c491f0a375cce2d89b5cd1
cm pull repo mlcommons@mlperf-automations --checkout=a90475d2de72bf0622cebe8d5ca8eb8c9d872fbd

cm run script \
--tags=app,mlperf,inference,generic,_nvidia,_retinanet,_tensorrt,_cuda,_valid,_r4.1-dev_default,_multistream \
Expand Down Expand Up @@ -71,7 +71,7 @@ cm run script \
--env.CM_DOCKER_REUSE_EXISTING_CONTAINER=yes \
--env.CM_DOCKER_DETACHED_MODE=yes \
--env.CM_MLPERF_INFERENCE_RESULTS_DIR_=/home/arjun/gh_action_results/valid_results \
--env.CM_DOCKER_CONTAINER_ID=95290226c7d6 \
--env.CM_DOCKER_CONTAINER_ID=a095b379b769 \
--env.CM_MLPERF_LOADGEN_COMPLIANCE_TEST=TEST01 \
--add_deps_recursive.compiler.tags=gcc \
--add_deps_recursive.coco2014-original.tags=_full \
Expand Down Expand Up @@ -129,7 +129,7 @@ Platform: RTX4090x1-nvidia_original-gpu-tensorrt-vdefault-default_config
Model Precision: int8

### Accuracy Results
`mAP`: `37.314`, Required accuracy for closed division `>= 37.1745`
`mAP`: `37.317`, Required accuracy for closed division `>= 37.1745`

### Performance Results
`Samples per query`: `11664411.0`
`Samples per query`: `11614264.0`
40 changes: 20 additions & 20 deletions ..._original-gpu-tensorrt-vdefault-default_config/retinanet/multistream/accuracy_console.out
View file
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
[2024-12-23 05:56:14,278 main.py:229 INFO] Detected system ID: KnownSystem.RTX4090x1
[2024-12-23 05:56:14,362 harness.py:249 INFO] The harness will load 2 plugins: ['build/plugins/NMSOptPlugin/libnmsoptplugin.so', 'build/plugins/retinanetConcatPlugin/libretinanetconcatplugin.so']
[2024-12-23 05:56:14,362 generate_conf_files.py:107 INFO] Generated measurements/ entries for RTX4090x1_TRT/retinanet/MultiStream
[2024-12-23 05:56:14,362 __init__.py:46 INFO] Running command: ./build/bin/harness_default --plugins="build/plugins/NMSOptPlugin/libnmsoptplugin.so,build/plugins/retinanetConcatPlugin/libretinanetconcatplugin.so" --logfile_outdir="/cm-mount/home/arjun/gh_action_results/valid_results/RTX4090x1-nvidia_original-gpu-tensorrt-vdefault-default_config/retinanet/multistream/accuracy" --logfile_prefix="mlperf_log_" --performance_sample_count=64 --test_mode="AccuracyOnly" --gpu_copy_streams=1 --gpu_inference_streams=1 --use_deque_limit=true --gpu_batch_size=2 --map_path="data_maps/open-images-v6-mlperf/val_map.txt" --mlperf_conf_path="/home/cmuser/CM/repos/local/cache/90c7069b92e34687/inference/mlperf.conf" --tensor_path="build/preprocessed_data/open-images-v6-mlperf/validation/Retinanet/int8_linear" --use_graphs=true --user_conf_path="/home/cmuser/CM/repos/mlcommons@mlperf-automations/script/generate-mlperf-inference-user-conf/tmp/58cef9af0ea540d997b8689f4e8b3c81.conf" --gpu_engines="./build/engines/RTX4090x1/retinanet/MultiStream/retinanet-MultiStream-gpu-b2-int8.lwis_k_99_MaxP.plan" --max_dlas=0 --scenario MultiStream --model retinanet --response_postprocess openimageeffnms
[2024-12-23 05:56:14,362 __init__.py:53 INFO] Overriding Environment
[2024-12-25 02:55:09,307 main.py:229 INFO] Detected system ID: KnownSystem.RTX4090x1
[2024-12-25 02:55:09,391 harness.py:249 INFO] The harness will load 2 plugins: ['build/plugins/NMSOptPlugin/libnmsoptplugin.so', 'build/plugins/retinanetConcatPlugin/libretinanetconcatplugin.so']
[2024-12-25 02:55:09,391 generate_conf_files.py:107 INFO] Generated measurements/ entries for RTX4090x1_TRT/retinanet/MultiStream
[2024-12-25 02:55:09,391 __init__.py:46 INFO] Running command: ./build/bin/harness_default --plugins="build/plugins/NMSOptPlugin/libnmsoptplugin.so,build/plugins/retinanetConcatPlugin/libretinanetconcatplugin.so" --logfile_outdir="/cm-mount/home/arjun/gh_action_results/valid_results/RTX4090x1-nvidia_original-gpu-tensorrt-vdefault-default_config/retinanet/multistream/accuracy" --logfile_prefix="mlperf_log_" --performance_sample_count=64 --test_mode="AccuracyOnly" --gpu_copy_streams=1 --gpu_inference_streams=1 --use_deque_limit=true --gpu_batch_size=2 --map_path="data_maps/open-images-v6-mlperf/val_map.txt" --mlperf_conf_path="/home/cmuser/CM/repos/local/cache/90c7069b92e34687/inference/mlperf.conf" --tensor_path="build/preprocessed_data/open-images-v6-mlperf/validation/Retinanet/int8_linear" --use_graphs=true --user_conf_path="/home/cmuser/CM/repos/mlcommons@mlperf-automations/script/generate-mlperf-inference-user-conf/tmp/6d1a55124ae14cd5803848d8020fb9c7.conf" --gpu_engines="./build/engines/RTX4090x1/retinanet/MultiStream/retinanet-MultiStream-gpu-b2-int8.lwis_k_99_MaxP.plan" --max_dlas=0 --scenario MultiStream --model retinanet --response_postprocess openimageeffnms
[2024-12-25 02:55:09,391 __init__.py:53 INFO] Overriding Environment
benchmark : Benchmark.Retinanet
buffer_manager_thread_count : 0
data_dir : /home/cmuser/CM/repos/local/cache/a8c152aef5494496/data
Expand All @@ -12,7 +12,7 @@ gpu_copy_streams : 1
gpu_inference_streams : 1
input_dtype : int8
input_format : linear
log_dir : /home/cmuser/CM/repos/local/cache/ba8d5f2a6bc546f9/repo/closed/NVIDIA/build/logs/2024.12.23-05.56.13
log_dir : /home/cmuser/CM/repos/local/cache/ba8d5f2a6bc546f9/repo/closed/NVIDIA/build/logs/2024.12.25-02.55.08
map_path : data_maps/open-images-v6-mlperf/val_map.txt
mlperf_conf_path : /home/cmuser/CM/repos/local/cache/90c7069b92e34687/inference/mlperf.conf
multi_stream_expected_latency_ns : 0
Expand All @@ -26,7 +26,7 @@ tensor_path : build/preprocessed_data/open-images-v6-mlperf/validation/Retinanet
test_mode : AccuracyOnly
use_deque_limit : True
use_graphs : True
user_conf_path : /home/cmuser/CM/repos/mlcommons@mlperf-automations/script/generate-mlperf-inference-user-conf/tmp/58cef9af0ea540d997b8689f4e8b3c81.conf
user_conf_path : /home/cmuser/CM/repos/mlcommons@mlperf-automations/script/generate-mlperf-inference-user-conf/tmp/6d1a55124ae14cd5803848d8020fb9c7.conf
system_id : RTX4090x1
config_name : RTX4090x1_retinanet_MultiStream
workload_setting : WorkloadSetting(HarnessType.LWIS, AccuracyTarget.k_99, PowerSetting.MaxP)
Expand All @@ -40,7 +40,7 @@ power_limit : None
cpu_freq : None
&&&& RUNNING Default_Harness # ./build/bin/harness_default
[I] mlperf.conf path: /home/cmuser/CM/repos/local/cache/90c7069b92e34687/inference/mlperf.conf
[I] user.conf path: /home/cmuser/CM/repos/mlcommons@mlperf-automations/script/generate-mlperf-inference-user-conf/tmp/58cef9af0ea540d997b8689f4e8b3c81.conf
[I] user.conf path: /home/cmuser/CM/repos/mlcommons@mlperf-automations/script/generate-mlperf-inference-user-conf/tmp/6d1a55124ae14cd5803848d8020fb9c7.conf
Creating QSL.
Finished Creating QSL.
Setting up SUT.
Expand All @@ -59,7 +59,7 @@ Setting up SUT.
[I] Creating batcher thread: 0 EnableBatcherThreadPerDevice: false
Finished setting up SUT.
Starting warmup. Running for a minimum of 5 seconds.
Finished warmup. Ran for 5.14369s.
Finished warmup. Ran for 5.14283s.
Starting running actual test.

No warnings encountered during test.
Expand All @@ -72,34 +72,34 @@ Device Device:0.GPU processed:
PerSampleCudaMemcpy Calls: 0
BatchedCudaMemcpy Calls: 12392
&&&& PASSED Default_Harness # ./build/bin/harness_default
[2024-12-23 05:57:06,174 run_harness.py:166 INFO] Result: Accuracy run detected.
[2024-12-23 05:57:06,174 __init__.py:46 INFO] Running command: python3 /home/cmuser/CM/repos/local/cache/ba8d5f2a6bc546f9/repo/closed/NVIDIA/build/inference/vision/classification_and_detection/tools/accuracy-openimages.py --mlperf-accuracy-file /cm-mount/home/arjun/gh_action_results/valid_results/RTX4090x1-nvidia_original-gpu-tensorrt-vdefault-default_config/retinanet/multistream/accuracy/mlperf_log_accuracy.json --openimages-dir /home/cmuser/CM/repos/local/cache/a8c152aef5494496/preprocessed_data/open-images-v6-mlperf --output-file build/retinanet-results.json
[2024-12-25 02:56:00,565 run_harness.py:166 INFO] Result: Accuracy run detected.
[2024-12-25 02:56:00,565 __init__.py:46 INFO] Running command: python3 /home/cmuser/CM/repos/local/cache/ba8d5f2a6bc546f9/repo/closed/NVIDIA/build/inference/vision/classification_and_detection/tools/accuracy-openimages.py --mlperf-accuracy-file /cm-mount/home/arjun/gh_action_results/valid_results/RTX4090x1-nvidia_original-gpu-tensorrt-vdefault-default_config/retinanet/multistream/accuracy/mlperf_log_accuracy.json --openimages-dir /home/cmuser/CM/repos/local/cache/a8c152aef5494496/preprocessed_data/open-images-v6-mlperf --output-file build/retinanet-results.json
loading annotations into memory...
Done (t=0.41s)
Done (t=0.42s)
creating index...
index created!
Loading and preparing results...
DONE (t=15.76s)
DONE (t=16.55s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=121.54s).
DONE (t=128.46s).
Accumulating evaluation results...
DONE (t=30.72s).
DONE (t=30.09s).
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.373
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.522
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.404
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.022
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.403
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.023
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.125
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.412
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.419
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.599
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.598
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.628
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.082
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.345
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.344
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.677
mAP=37.314%
mAP=37.317%

======================== Result summaries: ========================

Loading

0 comments on commit bb62ad8

Please sign in to comment.