Skip to content

Commit

Permalink
Results from GH action on NVIDIA_RTX4090x1
Browse files Browse the repository at this point in the history
  • Loading branch information
@arjunsuresh
arjunsuresh committed Dec 23, 2024
1 parent 13a16ab commit 0352c88
Show file tree
Hide file tree
Showing 51 changed files with 7,476 additions and 7,398 deletions.
16 changes: 8 additions & 8 deletions closed/MLCommons/systems/RTX4090x1-nvidia_original-gpu-tensorrt-vdefault-default_config.json
View file
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
{
"accelerator_frequency": "2520000 MHz",
"accelerator_frequency": "2610000 MHz",
"accelerator_host_interconnect": "N/A",
"accelerator_interconnect": "N/A",
"accelerator_interconnect_topology": "",
"accelerator_memory_capacity": "23.64971923828125 GB",
"accelerator_memory_capacity": "23.54595947265625 GB",
"accelerator_memory_configuration": "N/A",
"accelerator_model_name": "NVIDIA GeForce RTX 4090",
"accelerator_on-chip_memories": "",
Expand All @@ -16,17 +16,17 @@
"host_network_card_count": "1",
"host_networking": "Gig Ethernet",
"host_networking_topology": "N/A",
"host_processor_caches": "L1d cache: 576 KiB, L1i cache: 384 KiB, L2 cache: 24 MiB, L3 cache: ",
"host_processor_core_count": "24",
"host_processor_frequency": "5800.0000",
"host_processor_caches": "L1d cache: 512 KiB, L1i cache: 512 KiB, L2 cache: 16 MiB, L3 cache: 64 MiB",
"host_processor_core_count": "16",
"host_processor_frequency": "5881.0000",
"host_processor_interconnect": "",
"host_processor_model_name": "13th Gen Intel(R) Core(TM) i9-13900K",
"host_processor_model_name": "AMD Ryzen 9 7950X 16-Core Processor",
"host_processors_per_node": "1",
"host_storage_capacity": "9.4T",
"host_storage_capacity": "6.8T",
"host_storage_type": "SSD",
"hw_notes": "",
"number_of_nodes": "1",
"operating_system": "Ubuntu 20.04 (linux-6.8.0-49-generic-glibc2.31)",
"operating_system": "Ubuntu 20.04 (linux-6.8.0-51-generic-glibc2.31)",
"other_software_stack": "Python: 3.8.10, GCC-9.4.0, Using Docker , CUDA 12.2",
"status": "available",
"submitter": "MLCommons",
Expand Down
14 changes: 7 additions & 7 deletions ...ia_original-gpu-tensorrt-vdefault-default_config/resnet50/multistream/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ This experiment is generated using the [MLCommons Collective Mind automation fra

## Host platform

* OS version: Linux-6.8.0-49-generic-x86_64-with-glibc2.29
* OS version: Linux-6.8.0-51-generic-x86_64-with-glibc2.29
* CPU version: x86_64
* Python version: 3.8.10 (default, Nov 7 2024, 13:10:47)
[GCC 9.4.0]
Expand Down Expand Up @@ -56,8 +56,8 @@ cm run script \
--env.CM_MLPERF_INFERENCE_VERSION=4.1-dev \
--env.CM_RUN_MLPERF_INFERENCE_APP_DEFAULTS=r4.1-dev_default \
--env.CM_MLPERF_LOADGEN_ALL_MODES=yes \
--env.CM_MLPERF_INFERENCE_SOURCE_VERSION=4.1.23 \
--env.CM_MLPERF_LAST_RELEASE=v4.1 \
--env.CM_MLPERF_INFERENCE_SOURCE_VERSION=5.0.4 \
--env.CM_MLPERF_LAST_RELEASE=v5.0 \
--env.CM_TMP_PIP_VERSION_STRING= \
--env.CM_MODEL=resnet50 \
--env.CM_MLPERF_CLEAN_SUBMISSION_DIR=yes \
Expand All @@ -71,7 +71,7 @@ cm run script \
--env.CM_DOCKER_REUSE_EXISTING_CONTAINER=yes \
--env.CM_DOCKER_DETACHED_MODE=yes \
--env.CM_MLPERF_INFERENCE_RESULTS_DIR_=/home/arjun/gh_action_results/valid_results \
--env.CM_DOCKER_CONTAINER_ID=6956fa7efe5f \
--env.CM_DOCKER_CONTAINER_ID=74b53a08a206 \
--env.CM_MLPERF_LOADGEN_COMPLIANCE_TEST=TEST04 \
--add_deps_recursive.compiler.tags=gcc \
--add_deps_recursive.coco2014-original.tags=_full \
Expand Down Expand Up @@ -105,10 +105,10 @@ cm run script \
--print_env=False \
--print_deps=False \
--dump_version_info=True \
--env.CM_DATASET_IMAGENET_PATH=/home/cmuser/CM/repos/local/cache/ca6a4aaf13e34e0e/imagenet-2012-val \
--env.CM_DATASET_IMAGENET_PATH=/home/cmuser/CM/repos/local/cache/14a94e79cca24141/imagenet-2012-val \
--env.OUTPUT_BASE_DIR=/cm-mount/home/arjun/gh_action_results \
--env.CM_MLPERF_INFERENCE_SUBMISSION_DIR=/cm-mount/home/arjun/gh_action_submissions \
--env.MLPERF_SCRATCH_PATH=/home/cmuser/CM/repos/local/cache/5b2b0cc913a4453a
--env.MLPERF_SCRATCH_PATH=/home/cmuser/CM/repos/local/cache/a8c152aef5494496
```
*Note that if you want to use the [latest automation recipes](https://docs.mlcommons.org/inference) for MLPerf (CM scripts),
you should simply reload mlcommons@mlperf-automations without checkout and clean CM cache as follows:*
Expand All @@ -130,4 +130,4 @@ Model Precision: int8
`acc`: `76.064`, Required accuracy for closed division `>= 75.6954`

### Performance Results
`Samples per query`: `467192.0`
`Samples per query`: `454539.0`
38 changes: 19 additions & 19 deletions ...a_original-gpu-tensorrt-vdefault-default_config/resnet50/multistream/accuracy_console.out
View file
Original file line number Diff line number Diff line change
@@ -1,31 +1,31 @@
[2024-12-22 23:29:05,648 main.py:229 INFO] Detected system ID: KnownSystem.RTX4090x1
[2024-12-22 23:29:05,784 generate_conf_files.py:107 INFO] Generated measurements/ entries for RTX4090x1_TRT/resnet50/MultiStream
[2024-12-22 23:29:05,784 __init__.py:46 INFO] Running command: ./build/bin/harness_default --logfile_outdir="/cm-mount/home/arjun/gh_action_results/valid_results/RTX4090x1-nvidia_original-gpu-tensorrt-vdefault-default_config/resnet50/multistream/accuracy" --logfile_prefix="mlperf_log_" --performance_sample_count=2048 --test_mode="AccuracyOnly" --gpu_copy_streams=1 --gpu_inference_streams=1 --use_deque_limit=true --gpu_batch_size=8 --map_path="data_maps/imagenet/val_map.txt" --mlperf_conf_path="/home/cmuser/CM/repos/local/cache/c1d8c371d52d46a3/inference/mlperf.conf" --tensor_path="build/preprocessed_data/imagenet/ResNet50/int8_linear" --use_graphs=true --user_conf_path="/home/cmuser/CM/repos/mlcommons@mlperf-automations/script/generate-mlperf-inference-user-conf/tmp/fcc74ebe116e4c9e892bbad3769cf305.conf" --gpu_engines="./build/engines/RTX4090x1/resnet50/MultiStream/resnet50-MultiStream-gpu-b8-int8.lwis_k_99_MaxP.plan" --max_dlas=0 --scenario MultiStream --model resnet50
[2024-12-22 23:29:05,784 __init__.py:53 INFO] Overriding Environment
[2024-12-23 03:38:38,158 main.py:229 INFO] Detected system ID: KnownSystem.RTX4090x1
[2024-12-23 03:38:38,327 generate_conf_files.py:107 INFO] Generated measurements/ entries for RTX4090x1_TRT/resnet50/MultiStream
[2024-12-23 03:38:38,328 __init__.py:46 INFO] Running command: ./build/bin/harness_default --logfile_outdir="/cm-mount/home/arjun/gh_action_results/valid_results/RTX4090x1-nvidia_original-gpu-tensorrt-vdefault-default_config/resnet50/multistream/accuracy" --logfile_prefix="mlperf_log_" --performance_sample_count=2048 --test_mode="AccuracyOnly" --gpu_copy_streams=1 --gpu_inference_streams=1 --use_deque_limit=true --gpu_batch_size=8 --map_path="data_maps/imagenet/val_map.txt" --mlperf_conf_path="/home/cmuser/CM/repos/local/cache/90c7069b92e34687/inference/mlperf.conf" --tensor_path="build/preprocessed_data/imagenet/ResNet50/int8_linear" --use_graphs=true --user_conf_path="/home/cmuser/CM/repos/mlcommons@mlperf-automations/script/generate-mlperf-inference-user-conf/tmp/638bf59dc76546779c698e6cffb4cb34.conf" --gpu_engines="./build/engines/RTX4090x1/resnet50/MultiStream/resnet50-MultiStream-gpu-b8-int8.lwis_k_99_MaxP.plan" --max_dlas=0 --scenario MultiStream --model resnet50
[2024-12-23 03:38:38,328 __init__.py:53 INFO] Overriding Environment
benchmark : Benchmark.ResNet50
buffer_manager_thread_count : 0
data_dir : /home/cmuser/CM/repos/local/cache/5b2b0cc913a4453a/data
data_dir : /home/cmuser/CM/repos/local/cache/a8c152aef5494496/data
disable_beta1_smallk : True
gpu_batch_size : 8
gpu_copy_streams : 1
gpu_inference_streams : 1
input_dtype : int8
input_format : linear
log_dir : /home/cmuser/CM/repos/local/cache/dfbf240f980947f5/repo/closed/NVIDIA/build/logs/2024.12.22-23.29.04
log_dir : /home/cmuser/CM/repos/local/cache/ba8d5f2a6bc546f9/repo/closed/NVIDIA/build/logs/2024.12.23-03.38.37
map_path : data_maps/imagenet/val_map.txt
mlperf_conf_path : /home/cmuser/CM/repos/local/cache/c1d8c371d52d46a3/inference/mlperf.conf
mlperf_conf_path : /home/cmuser/CM/repos/local/cache/90c7069b92e34687/inference/mlperf.conf
multi_stream_expected_latency_ns : 0
multi_stream_samples_per_query : 8
multi_stream_target_latency_percentile : 99
precision : int8
preprocessed_data_dir : /home/cmuser/CM/repos/local/cache/5b2b0cc913a4453a/preprocessed_data
preprocessed_data_dir : /home/cmuser/CM/repos/local/cache/a8c152aef5494496/preprocessed_data
scenario : Scenario.MultiStream
system : SystemConfiguration(host_cpu_conf=CPUConfiguration(layout={CPU(name='13th Gen Intel(R) Core(TM) i9-13900K', architecture=<CPUArchitecture.x86_64: AliasedName(name='x86_64', aliases=(), patterns=())>, core_count=24, threads_per_core=1): 1}), host_mem_conf=MemoryConfiguration(host_memory_capacity=Memory(quantity=131.634476, byte_suffix=<ByteSuffix.GB: (1000, 3)>, _num_bytes=131634476000), comparison_tolerance=0.05), accelerator_conf=AcceleratorConfiguration(layout=defaultdict(<class 'int'>, {GPU(name='NVIDIA GeForce RTX 4090', accelerator_type=<AcceleratorType.Discrete: AliasedName(name='Discrete', aliases=(), patterns=())>, vram=Memory(quantity=23.98828125, byte_suffix=<ByteSuffix.GiB: (1024, 3)>, _num_bytes=25757220864), max_power_limit=450.0, pci_id='0x268410DE', compute_sm=89): 1})), numa_conf=None, system_id='RTX4090x1')
system : SystemConfiguration(host_cpu_conf=CPUConfiguration(layout={CPU(name='AMD Ryzen 9 7950X 16-Core Processor', architecture=<CPUArchitecture.x86_64: AliasedName(name='x86_64', aliases=(), patterns=())>, core_count=16, threads_per_core=2): 1}), host_mem_conf=MemoryConfiguration(host_memory_capacity=Memory(quantity=131.080068, byte_suffix=<ByteSuffix.GB: (1000, 3)>, _num_bytes=131080068000), comparison_tolerance=0.05), accelerator_conf=AcceleratorConfiguration(layout=defaultdict(<class 'int'>, {GPU(name='NVIDIA GeForce RTX 4090', accelerator_type=<AcceleratorType.Discrete: AliasedName(name='Discrete', aliases=(), patterns=())>, vram=Memory(quantity=23.98828125, byte_suffix=<ByteSuffix.GiB: (1024, 3)>, _num_bytes=25757220864), max_power_limit=450.0, pci_id='0x268410DE', compute_sm=89): 1})), numa_conf=None, system_id='RTX4090x1')
tensor_path : build/preprocessed_data/imagenet/ResNet50/int8_linear
test_mode : AccuracyOnly
use_deque_limit : True
use_graphs : True
user_conf_path : /home/cmuser/CM/repos/mlcommons@mlperf-automations/script/generate-mlperf-inference-user-conf/tmp/fcc74ebe116e4c9e892bbad3769cf305.conf
user_conf_path : /home/cmuser/CM/repos/mlcommons@mlperf-automations/script/generate-mlperf-inference-user-conf/tmp/638bf59dc76546779c698e6cffb4cb34.conf
system_id : RTX4090x1
config_name : RTX4090x1_resnet50_MultiStream
workload_setting : WorkloadSetting(HarnessType.LWIS, AccuracyTarget.k_99, PowerSetting.MaxP)
Expand All @@ -38,27 +38,27 @@ skip_file_checks : False
power_limit : None
cpu_freq : None
&&&& RUNNING Default_Harness # ./build/bin/harness_default
[I] mlperf.conf path: /home/cmuser/CM/repos/local/cache/c1d8c371d52d46a3/inference/mlperf.conf
[I] user.conf path: /home/cmuser/CM/repos/mlcommons@mlperf-automations/script/generate-mlperf-inference-user-conf/tmp/fcc74ebe116e4c9e892bbad3769cf305.conf
[I] mlperf.conf path: /home/cmuser/CM/repos/local/cache/90c7069b92e34687/inference/mlperf.conf
[I] user.conf path: /home/cmuser/CM/repos/mlcommons@mlperf-automations/script/generate-mlperf-inference-user-conf/tmp/638bf59dc76546779c698e6cffb4cb34.conf
Creating QSL.
Finished Creating QSL.
Setting up SUT.
[I] [TRT] Loaded engine size: 26 MiB
[I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +7, GPU +10, now: CPU 72, GPU 837 (MiB)
[I] [TRT] [MemUsageChange] Init cuDNN: CPU +1, GPU +10, now: CPU 73, GPU 847 (MiB)
[I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +7, GPU +10, now: CPU 76, GPU 844 (MiB)
[I] [TRT] [MemUsageChange] Init cuDNN: CPU +1, GPU +10, now: CPU 77, GPU 854 (MiB)
[I] [TRT] [MemUsageChange] TensorRT-managed allocation in engine deserialization: CPU +0, GPU +24, now: CPU 0, GPU 24 (MiB)
[I] Device:0.GPU: [0] ./build/engines/RTX4090x1/resnet50/MultiStream/resnet50-MultiStream-gpu-b8-int8.lwis_k_99_MaxP.plan has been successfully loaded.
[E] [TRT] 3: [runtime.cpp::~Runtime::401] Error Code 3: API Usage Error (Parameter check failed at: runtime/rt/runtime.cpp::~Runtime::401, condition: mEngineCounter.use_count() == 1 Destroying a runtime before destroying deserialized engines created by the runtime leads to undefined behavior.)
[I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +1, GPU +10, now: CPU 47, GPU 839 (MiB)
[I] [TRT] [MemUsageChange] Init cuDNN: CPU +0, GPU +8, now: CPU 47, GPU 847 (MiB)
[I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +10, now: CPU 51, GPU 846 (MiB)
[I] [TRT] [MemUsageChange] Init cuDNN: CPU +0, GPU +8, now: CPU 51, GPU 854 (MiB)
[I] [TRT] [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +17, now: CPU 0, GPU 41 (MiB)
[I] Start creating CUDA graphs
[I] Capture 8 CUDA graphs
[I] Finish creating CUDA graphs
[I] Creating batcher thread: 0 EnableBatcherThreadPerDevice: false
Finished setting up SUT.
Starting warmup. Running for a minimum of 5 seconds.
Finished warmup. Ran for 5.02615s.
Finished warmup. Ran for 5.02397s.
Starting running actual test.

No warnings encountered during test.
Expand All @@ -71,8 +71,8 @@ Device Device:0.GPU processed:
PerSampleCudaMemcpy Calls: 0
BatchedCudaMemcpy Calls: 6250
&&&& PASSED Default_Harness # ./build/bin/harness_default
[2024-12-22 23:29:18,470 run_harness.py:166 INFO] Result: Accuracy run detected.
[2024-12-22 23:29:18,470 __init__.py:46 INFO] Running command: python3 /home/cmuser/CM/repos/local/cache/dfbf240f980947f5/repo/closed/NVIDIA/build/inference/vision/classification_and_detection/tools/accuracy-imagenet.py --mlperf-accuracy-file /cm-mount/home/arjun/gh_action_results/valid_results/RTX4090x1-nvidia_original-gpu-tensorrt-vdefault-default_config/resnet50/multistream/accuracy/mlperf_log_accuracy.json --imagenet-val-file data_maps/imagenet/val_map.txt --dtype int32
[2024-12-23 03:38:51,723 run_harness.py:166 INFO] Result: Accuracy run detected.
[2024-12-23 03:38:51,723 __init__.py:46 INFO] Running command: python3 /home/cmuser/CM/repos/local/cache/ba8d5f2a6bc546f9/repo/closed/NVIDIA/build/inference/vision/classification_and_detection/tools/accuracy-imagenet.py --mlperf-accuracy-file /cm-mount/home/arjun/gh_action_results/valid_results/RTX4090x1-nvidia_original-gpu-tensorrt-vdefault-default_config/resnet50/multistream/accuracy/mlperf_log_accuracy.json --imagenet-val-file data_maps/imagenet/val_map.txt --dtype int32
accuracy=76.064%, good=38032, total=50000

======================== Result summaries: ========================
Expand Down
Loading

0 comments on commit 0352c88

Please sign in to comment.