TWiki> HEPIX Web>CpuBenchmark>HEP-Workloads>HEPWorkloadReproducibility (2019-08-05, ManfredAlefExternal) EditAttachPDF

HEP Workloads: Reproducibility of Benchmark Results

This Twiki page is aimed at internal discussions of the HEPiX Benchmarking Working Group about the spread in the results of repeated benchmark runs.

It does definitively not provide any final numbers.

Spread of 50 repeated benchmark runs (latest HEP software release)
Hardware model: c01-010-156.gridka.de (SL7) - Turbo Boost enabled in BIOS
Workload Version No. of Total scores Stdev (Max-Min) Success Av.
events copies Min Max Mean Stdev / Mean rate runtime
ALICE gen-sim v0.7   20 0.769 0.893 0.868 0.024 0.027 0.143 100% 170s
  40 0.884 0.939 0.901 0.012 0.014 0.061 70% 283s
v0.6   20 0.700 0.891 0.865 0.031 0.036 0.220 94% 305s
  40 0.884 1.322 0.912 0.051 0.056 0.524 81% 431s
ATLAS gen-bmk v0.7 5 20 493.4 510.9 502.6 4.649 0.009 0.035   294s
40 771.2 840.4 802.2 17.89 0.022 0.086 461s
v0.3 5 20 476.5 518.2 502.8 8.721 0.017 0.083 439s
40 713.0 837.9 795.1 23.97 0.030 0.157 581s
ATLAS sim-bmk (8 threads per copy)
  v0.6 10 5 0.127 0.131 0.128 0.001 0.005 0.027   3737s
ATLAS sim-bmk (4 threads per copy)
  v0.16 5 5 0.083 0.086 0.085 0.001 0.009 0.039   1603s
10 0.092 0.093 0.093 0.000 0.000 0.013 2726s
v0.9 5 0.082 0.085 0.084 0.001 0.009 0.039 1621s
10 0.093 0.094 0.093 0.000 0.003 0.014 2697s
v0.7 5 0.082 0.085 0.084 0.001 0.009 0.035 1740s
10 0.093 0.094 0.093 0.000 0.034 0.013 2831s
ATLAS reco-bmk
CPU_score (All)   5 20 42.82 46.00 45.26 0.662 0.009 0.070   1070s
(ESDtoAOD) 38.86 41.98 41.28 0.661 0.016 0.076
(HITtoRDO) 0.439 0.469 0.467 0.006 0.014 0.065
(RAWtoESD) 1.132 1.453 1.418 0.077 0.054 0.226
(RDOtoRDOTrigger) 2.050 2.114 2.087 0.019 0.009 0.031
CPU_score (All) 40 62.70 73.15 65.68 2.562 0.039 0.159   1783s
(ESDtoAOD) 57.73 68.14 60.61 2.534 0.042 0.172
(HITtoRDO) 0.625 0.630 0.628 0.001 0.002 0.008
(RAWtoESD) 1.747 1.773 1.756 0.007 0.004 0.015
(RDOtoRDOTrigger) 2.583 2.960 2.680 0.121 0.045 0.141
CMS gen-sim ttbar (4 threads per copy)
CPU_score v0.13 25 5 0.221 0.229 0.225 0.002 0.009 0.034   634s
througput_score 0.850 0.881 0.863 0.008 0.009 0.036
CPU_score 10 0.246 0.248 0.247 0.001 0.002 0.009   1093s
througput_score 0.954 0.965 0.960 0.003 0.003 0.011
CPU_score v0.5 25 10 0.244 0.250 0.247 0.001 0.004 0.026   1104s
througput_score 0.925 0.971 0.956 0.005 0.006 0.048
CPU_score v0.5 20 10 0.248 0.252 0.250 0.001 0.004 0.019   1012s
througput_score 0.951 0.973 0.963 0.004 0.004 0.023
CPU_score v0.5 100 10 0.247 0.252 0.250 0.001 0.004 0.021   4272s
througput_score 0.951 0.997 0.976 0.013 0.013 0.021
CPU_score v0.4 100 10 0.247 0.358 0.253 0.017 0.059 0.438   4264s
througput_score 0.977 1.075 0.989 0.016 0.016 0.099
CMS digi (4 threads per copy)
CPU_score v0.9 100 5 1.106 1.114 1.099 0.015 0.013 0.049   768s
througput_score 3.394 4.315 3.867 0.277 0.071 0.237
CPU_score 10 1.319 1.337 1.327 0.005 0.00.36 0.014   1140s
througput_score 4.440 5.233 5.027 0.293 0.058 0.158
CPU_score v0.5 100 10 4.349 4.437 4.393 0.027 0.006 0.021   410s
througput_score 4.114 4.239 4.190 0.027 0.006 0.020
CPU_score v0.4 50 10 2.176 2.225 2.193 0.013 0.006 0.022   399s
througput_score 4.136 4.253 4.184 0.026 0.006 0.028
CMS reco (4 threads per copy)
CPU_score v0.12 100 5 0.625 0.646 0.639 0.005 0.008 0.031   1126s
througput_score 2.469 2.548 2.521 0.021 0.008 0.031
CPU_score 10 0.743 0.757 0.750 0.003 0.004 0.014   1427s
througput_score 2.936 2.985 2.967 0.013 0.004 0.016
CPU_score v0.2-v0.3 100 10 0.745 0.757 0.752 0.002 0.003 0.016   1774s
througput_score 2.940 2.989 2.973 0.011 0.004 0.017
CPU_score v0.1 50 10 0.738 0.749 0.744 0.002 0.003 0.015   844s
througput_score 2.913 2.955 2.933 0.012 0.004 0.014
LHCb gen sim v0.1   20 104.9 109.6 108.4 1.042 0.009 0.043   1242s
  40 114.9 117.1 116.1 0.338 0.003 0.019   2119s
KV-bmk v1.0   20 29.10 30.38 30.62 0.336 0.011 0.049   223s
  40 35.44 36.75 35.93 0.227 0.006 0.037   339s
Hardware model: c01-010-155.gridka.de (SL6) - Turbo Boost enabled in BIOS
Workload Version No. of Total scores Stdev (Max-Min) Success Av.
events copies Min Max Mean Stdev / Mean rate runtime
ALICE gen-sim v0.7 5 20 0.838 0.864 0.850 0.008 0.009 0.030 90% 174s
40 0.919 0.981 0.938 0.018 0.019 0.073 80% 279s
v0.6   20 0.812 0.874 0.849 0.011 0.012 0.064 94% 176s
  40 0.799 0.981 0.932 0.030 0.032 0.196 84% 312s
ATLAS gen-bmk v0.7 5 20 478.8 506.5 479.6 6.537 0.013 0.056   307s
40 703.5 818.1 758.1 37.08 0.049 0.151   433s
v0.3 5 20 456.5 516.6 499.2 9.692 0.019 0.120   324s
40 653.3 834.5 752.4 43.35 0.058 0.241   443s
ATLAS sim-bmk (8 threads per copy)
  v0.7 10 5 0.129 0.137 0.133 0.012 0.009 0.059   3493s
ATLAS sim-bmk (4 threads per copy)
  v0.9 5 5 0.082 0.086 0.085 0.001 0.012 0.055   1610s
  v0.9 5 10 0.095 0.098 0.097 0.001 0.006 0.025   2644s
  v0.7 5 5 0.083 0.087 0.085 0.001 0.011 0.042   1583s
10 0.095 0.097 0.096 0.000 0.005 0.024   2668s
ATLAS reco-bmk
CPU_score (All)   5 20 45.37 46.81 46.32 0.359 0.008 0.031   1003s
(ESDtoAOD) 41.59 43.05 42.62 0.368 0.009 0.034
(HITtoRDO) 0.438 0.470 0.461 0.008 0.017 0.070
(RAWtoESD) 1.358 1.431 1.397 0.018 0.013 0.052
(RDOtoRDOTrigger) 1.684 1.976 1.844 0.082 0.044 0.158
CPU_score (All) 40 66.09 78.22 70.39 3.468 0.049 0.172   2040s
(ESDtoAOD) 61.24 73.33 65.49 3.454 0.053 0.185
(HITtoRDO) 0.629 0.647 0.634 0.004 0.007 0.029
(RAWtoESD) 1.728 1.773 1.752 0.013 0.007 0.026
(RDOtoRDOTrigger) 2.431 2.603 2.511 0.047 0.019 0.069
CMS gen-sim ttbar (4 threads per copy)
CPU_score v0.8-v0.9 100 10 0.258 0.290 0.264 0.006 0.025 0.119   4110s
througput_score 1.005 1.051 1.022 0.009 0.009 0.044
CPU_score v0.5 20 10 0.258 0.262 0.264 0.004 0.014 0.060   860s
througput_score 0.985 1.022 0.999 0.007 0.007 0.036
CPU_score v0.5 100 10 0.258 0.274 0.263 0.003 0.012 0.060   860s
througput_score 0.985 1.029 1.009 0.012 0.012 0.043
CPU_score v0.4 100 10 0.257 0.318 0.265 0.013 0.051 0.229   4028s
througput_score 1.004 1.074 1.020 0.013 0.012 0.069
CMS digi (4 threads per copy)
CPU_score v0.5 100 10 4.077 4.477 4.348 0.101 0.023 0.093   385s
througput_score 3.840 4.189 4.076 0.087 0.021 0.086
CPU_score v0.4 50 10 1.892 2.130 2.030 0.080 0.040 0.117   425s
througput_score 3.315 3.996 3.676 0.240 0.065 0.185
CMS reco (4 threads per copy)
CPU_score v0.2-v0.3 100 10 0.740 0.772 0.758 0.009 0.012 0.042   1786s
througput_score 2.724 2.979 2.874 0.083 0.029 0.089
CPU_score v0.1 50 10 0.717 0.750 0.731 0.009 0.013 0.044   928s
througput_score 2.558 2.825 2.635 0.073 0.028 0.101
LHCb gen sim v0.1   20 103.4 109.2 106.9 1.176 0.011 0.054   1156s
  40 117.0 122.6 119.7 0.960 0.008 0.047   1935s
KV-bmk v1.0   20 30.78 31.26 31.12 0.120 0.004 0.015   304s
  40 35.19 36.58 35.85 0.330 0.009 0.039   340s

Spread of 20 repeated benchmark runs (latest HEP software release)
Hardware model: c01-010-156.gridka.de (SL7) - Turbo Boost disabled in BIOS
Workload Version No. of Total scores Stdev (Max-Min) Success Av.
events copies Min Max Mean Stdev / Mean rate runtime
ALICE gen-sim v0.6   20 0.669 0.758 0.670 0.030 0.043 0.128 95% 223s
  40 0.694 0.811 0.738 0.029 0.040 0.159 85% 372s
ATLAS gen-bmk v0.3 5 20 417.395 430.162 424.785 3.303 0.008 0.030 100% 337s
5 40 629.448 707.317 672.776 17.461 0.026 0.116 100% 498s
ATLAS sim-bmk (8 threads per copy)
  v0.4 10 5 0.087 0.111 0.107 0.007 0.061 0.216 100% 4243s
CMS gen-sim ttbar (4 threads per copy)
CPU_score v0.4 20 10 0.213 0.217 0.215 0.001 0.005 0.020 100% 1030s
througput_score 0.821 0.838 0.831 0.001 0.005 0.020
LHCb gen sim v0.1   20 95.261 96.051 95.578 0.231 0.002 0.008 100% 1311s
  40 105.988 107.131 106.636 0.254 0.002 0.011 100% 2187s
HS06 (10 runs)     40 412.4 418.0 414.2 1.520 0.004 0.014    
Hardware model: c01-010-155.gridka.de (SL6) - Turbo Boost disabled in BIOS
Workload Version No. of Total scores Stdev (Max-Min) Success Av.
events copies Min Max Mean Stdev / Mean rate runtime
ALICE gen-sim v0.6   20 0.752 0.788 0.769 0.009 0.012 0.047 100% 199s
  40 0.763 0.922 0.867 0.045 0.052 0.183 85% 323s
ATLAS gen-bmk v0.3 5 20 413.165 446.711 439.067 8.343 0.019 0.076 100% 332s
5 40 575.117 740.849 663.477 46.827 0.071 0.250 100% 498s
ATLAS sim-bmk (8 threads per copy)
  v0.4 10 5 0.047 0.121 0.111 0.017 0.155 0.661 100% 3936s
CMS gen-sim ttbar (4 threads per copy)
CPU_score v0.4 20 10 0.231 0.241 0.237 0.002 0.010 0.041 100% 961s
througput_score 0.870 0.901 0.891 0.008 0.009 0.035
LHCb gen sim v0.1   20 95.360 98.875 97.249 0.890 0.009 0.036 100% 1250s
  40 109.450 115.184 113.674 1.166 0.010 0.050 100% 2044s
HS06 (10 runs)     40 414.6 421.2 417.8 2.133 0.005 0.016    

Spread of 20 repeated benchmark runs (latest HEP software release)
Hardware model: c01-010-156.gridka.de
Workload Date No. of Total scores Stdev/ (Max-Min)/ Success Av.
events copies Min Max Mean Stdev Mean Mean rate runtime
ALICE gen-sim v0.4 2019-02-27   1 0.053 0.056 0.055 0 0 0 033 100%  
  20 0.822 0.884 0.870 0.013 0.015 0.072 100%  
  32 0.794 0.921 0.876 0.039 0.045 0.137 90%  
  40 0.800 0.910 0.884 0.030 0.034 0.126 80%  
ATLAS gen-bmk 2019-03-02 5 1 28.571 29.672 29.394 0.319 0.011 0.041 100%  
5 20 477.579 516.113 501.354 8.729 0.017 0.077 100%  
5 32 581.384 699.518 635.287 32.660 0.051 0.186 100%  
5 40 739.762 839.345 797.658 21.559 0.027 0.125 100% 433s
2019-03-04 25 40 538.660 605.196 578.275 14.382 0.025 0.115 100% 449s
2019-03-05 100 40 654.203 744.648 707.456 25.138 0.036 0.128 100% 634s
2019-03-06 500 40 607.589 667.795 642.080 15.961 0.025 0.094 100% 1896s
2019-03-06 2500 40 643.047 680.619 658.053 9.299 0.014 0.057 100% 8539s
CMS gen-sim ttbar v0.2 (4 threads per copy)
CPU_score 2019-03-14 20 10 0.246 0.251 0.249 0 0 0.019 100% 884s
througput_score 10 0.943 0.963 0.959 0.007 0.007 0.027 100% 884s
LHCb gen sim 2019-03-01   1 5.589 6.417 6.221 0.243 0.039 0.133 100%  
  20 108.310 109.489 108.972 0.339 0.003 0.011 100%  
  32 112.797 114.795 113.612 0.554 0.005 0.018 100%  
  40 115.924 116.967 116.286 0.262 0.002 0.009 100%  

Docker versus Singularity (average of 20 runs)
Workload Copies Threads per copy Singularity Docker
ATLAS sim 5 8 0.118 0.118
LHCb gen-sim 20 1 91.79 91.67
40 1 104.3 104.7

Spread of 20 repeated benchmark runs
Workload Date No. of Hardware model Runtime
copies c01-010-156.gridka.de c01-025-131.gridka.de c01-028-182.gridka.de
ALICE gen-sim 2019-02-07 1 0.0329...0.0333 0.0312...0.0321 0.0260...1.6667 O(3')
16     0.0221...1.9999 O(4')
20 0.0273...2.9986 0.0414...1.9484   O(4')
24     0.0776...1.3623 O(6')
32 0.0179...2.0441 0.0312...2.5807 0.0473...1.4164 O(6')
40 0.0403...1.2904 0.0127...2.7695   O(8')
ATLAS gen-bmk 2019-02-08 1 28.0112...29.3255     O(4')
16     289.7805...329.9172 O(6')
20 490.4275...508.0936 367.2221...419.9154   O(5')
24     368.9814...439.7616 O(7')
32 598.5838...708.9543 494.9507...605.0028 436.1996...531.0295 O(6')
40 751.6043...842.3921 563.2879...680.9631   O(7')
LHCb gen-sim 2019-02-07 1 5.5044...6.2811 5.6604...6.3401 4.2387...4.3335 O(16')
16     59.2598...60.5848 O(25')
20 104.1588...106.9673 91.3763...97.5531   O(19')
24     63.6833...64.9746 O(35')
32 110.3778...112.3555 103.4095...106.3811 67.3353...68.6971 O(30')
40 113.1566...113.8484 109.2494...110.8703   O(35')
 
HS06 scores of the systems under test are provided for information only:
HS06   16     261  
20 374 333    
24     305  
32 447 390 327  
40 465 416    

Hardware models
Hostname Processor(s) Number of OS
cores logical processors
c01-010-155.gridka.de 2x Intel Xeon E5-2660v3 (2.6 GHz) - Haswell 20 40 SL6
c01-010-156.gridka.de 2x Intel Xeon E5-2660v3 (2.6 GHz) - Haswell 20 40 SL7
c01-025-131.gridka.de 2x Intel Xeon E5-2630v4 (2.2 GHz) - Broadwell 20 40 SL7
c01-028-182.gridka.de 2x Intel Xeon E5-2665 (2.4 GHz) - Sandy Bridge 16 32 SL7

-- ManfredAlefExternal - 2019-02-06

Edit | Attach | Watch | Print version | History: r46 < r45 < r44 < r43 < r42 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r46 - 2019-08-05 - ManfredAlefExternal
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    HEPIX All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright &© 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback