Implementation: add results from scaling benchmarks

2 years ago · fac000317d
parent 1de4bf6875
commit fac000317d
9 changed files with 200 additions and 17 deletions
--- a/.gitignore
+++ b/.gitignore
@ -11,4 +11,6 @@
 *.blg
 *.fls

-include/*.tex
+include/*.tex
+
+!plots/*.pdf
--- a/include/04-implementation.md
+++ b/include/04-implementation.md
@ -444,7 +444,7 @@ This ensures the benchmark runs are completely isolated and no transient side ef
 The setup described above is entirely scripted to minimize potential for variation.

 Each configuration run is started with an interval of 60 seconds.
-Since configuration runs take longer than 60 seconds (see Figure \ref{fig:config-run-variance}), the applications needs process multiple in parallel.
+Since configuration runs take longer than 60 seconds (see Figure \ref{fig:config-run-variance-static}), the applications needs to process multiple in parallel.
 Each configuration run contains the same amount of work.

 The application continues to run after the work has finished (time to completion) in order to demonstrate the cost-savings made possible through autoscaling.
@ -452,13 +452,13 @@ If the benchmark is stopped as soon as the workload is completed, only the perfo

 \begin{figure}[ht]
 \centering
-\includegraphics[width=\textwidth]{images/static-baseline-benchmark.pdf}
-\caption{\label{fig:performance-cost-benchmark} Performance benchmark results of static scaling. Number in square brackets indicate number of replicas.}
+\includegraphics[width=\textwidth]{plots/performance-cost-benchmark-static.pdf}
+\caption{\label{fig:performance-cost-benchmark-static} Results of scaling benchmark. Scaling setting indicates static number of replicas or autoscaling policy.}
 \end{figure}

 As explained at the beginning of this chapter, the number of executors in the target application can be adjusted.
 To get a baseline for the application behavior, we tested several static values for replicas.
-The graph in Figure \ref{fig:performance-cost-benchmark} shows the results with the cost (replica seconds) on the x-axis and the time to completion (time until the simulated user has all desired results) on the y-axis.
+The graph in Figure \ref{fig:performance-cost-benchmark-static} shows the results with the cost (replica seconds) on the x-axis and the time to completion (time until the simulated user has all desired results) on the y-axis.
 The lowest cost (4.000 replica seconds) is achieved when using 2 replicas, which also has by far the slowest time to completion (2.000 seconds).
 The highest cost (14.000 replica seconds) is recorded with 20 replicas, which also has the lowest time to completion (4.000 seconds).
 Due to the constant workload size, the benchmarks with 10 and 15 replicas had almost the same time to completion, while having significantly fewer replicas and therefore replica seconds.
@ -475,16 +475,16 @@ The same behavior is illustrated in Figure \ref{fig:20-vs-3-replicas}, but with
 With just 3 replicas, there are many tasks queued for a long period of time (yellow line), which causes the average queue time (green line) to continuously rise, up to a value of 6 minutes.
 With 20 replicas, the average queue time never rises above 30 seconds.

-Figure \ref{fig:config-run-variance} shows the execution time of individual configuration runs during the benchmark.
-It is logical that when the overall time to completion is lower (as shown in Figure \ref{fig:performance-cost-benchmark}), also the time of individual workloads must be smaller.
+Figure \ref{fig:config-run-variance-static} shows the execution time of individual configuration runs during the benchmark.
+It is logical that when the overall time to completion is lower (as shown in Figure \ref{fig:performance-cost-benchmark-static}), also the time of individual workloads must be smaller.
 The plot confirms this and also shows that a low number of replicas has a significant effect on the variance of the configuration run's execution times.
 While each configuration run has the same workload, with a low number of replicas a significant increase in variance (in addition to an increase of the average) can be observed.
 This is explained by the fact that with few replicas, the same executor needs to run multiple workloads sequentially, which slows down some of them drastically.

 \begin{figure}[ht]
 \centering
-\includegraphics[width=\textwidth]{images/config-run-variance.pdf}
-\caption{\label{fig:config-run-variance} Execution time of individual configuration runs (3 benchmarks, 10 configuration runs per benchmark). Numbers in blue indicate mean.}
+\includegraphics[width=\textwidth]{plots/config-run-variance-static.pdf}
+\caption{\label{fig:config-run-variance-static} Execution time of individual configuration runs (3 benchmarks, 10 configuration runs per benchmark). Numbers in blue indicate the mean value.}
 \end{figure}

 After having established the performance and cost characteristics of static configuration, we transition into testing a basic autoscaling setup.
@ -493,7 +493,7 @@ We will now evaluate the behavior of this policy.
 As Figure \ref{fig:hpa-scaling-v1} shows, the benchmark with this policy had a similar before a static configuration with 10 replicas: the application utilized around 18600 replica seconds during the benchmark and the total time to completion was approx. 700 seconds.

 This result highlights the strength and weaknesses of this autoscaling policy: it enabled the application to achieve the same performance as with a static configuration of 10 replicas.
-As Figure \ref{fig:performance-cost-benchmark} shows, this is the maximum amount of performance the application is able to deliver for this particular workload.
+As Figure \ref{fig:performance-cost-benchmark-static} shows, this is the maximum amount of performance the application is able to deliver for this particular workload.
 At the same time, the autoscaling policy was just as costly as a static configuration of 10 replicas, even though for significant periods the application was just running with 1 replica.
 This is explained by Figure \ref{fig:hpa-scaling-v1}: the policy overscaled the number of replicas (more than the necessary value established previously) and even reached the replica limit (`maxReplicas` in Listing \ref{src:hpa-scale-v1}).

@ -506,7 +506,7 @@ This section will address these issue by fine-tuning the scaling policy.

 * **Delayed scale down**: the number of replicas is not reduced soon enough after the workload has finished, as is evident by the number of queued tasks vs. the number of replicas in Figure \ref{fig:hpa-scaling-v1}.
 * **Potential for premature scale down**: if a task has a long execution time, the autoscaler might scale reduce the number of replicas too early because the target metric only considers queued tasks.
-* **Overscaling of replicas**: our previous experiments with static replica configurations have shown that provisioning 20 replicas (as shown in Figure \ref{fig:performance-cost-benchmark}) is ineffective, since it does not increase performance.
+* **Overscaling of replicas**: our previous experiments with static replica configurations have shown that provisioning 20 replicas (as shown in Figure \ref{fig:performance-cost-benchmark-static}) is ineffective, since it does not increase performance.

 The delayed scale down can be tackled by adjusting the *downscale stabilization window* of the scaling policies (shown in Listing \ref{src:downscale-behavior}).
 This setting specifies how soon HPA will start scaling down the deployment after it has detected that the scaling metric is below the target \cite{KubernetesDocumentationHorizontalPod_2021}.
@ -544,19 +544,31 @@ This is necessary because -- with the change explained previously -- the scaling

 ....

-Instead of using the scaling metric directly, the raw value is averaged: it is divided by the number of active pod replicas (Listing \ref{src:improved-scaling-logic}) \cite{KubernetesDocumentationHorizontalPod_2021}.
-In order to find an appropriate value for `averageValue`, we performed several benchmarks.
-They have been labeled as `hpav2`, `hpav3`, `hpav4` and `hpa5` for the values 2, 3, 4 and 5, respectively.
-These benchmarks also include the other optimizations outlined above.
-
 \begin{lstlisting}[caption=Improved scaling logic for HPA, label=src:improved-scaling-logic, language=yaml, numbers=left]
      target:
        type: AverageValue
        averageValue: 2
 \end{lstlisting}

+Instead of using the scaling metric directly, the raw value is averaged: it is divided by the number of active pod replicas (Listing \ref{src:improved-scaling-logic}) \cite{KubernetesDocumentationHorizontalPod_2021}.
+In order to find an appropriate value for `averageValue`, we performed several benchmarks.
+Figure \ref{fig:performance-cost-benchmark-autoscaling} and \ref{fig:config-run-variance-autoscaling} show the results of these benchmarks.
+The different `averageValues` have been labeled as `hpav2`, `hpav3`, `hpav4` and `hpa5` for the values 2, 3, 4 and 5, respectively.
+These benchmarks also include the other optimizations outlined above.
+
+\begin{figure}[ht]
+\centering
+\includegraphics[width=\textwidth]{plots/performance-cost-benchmark-autoscaling.pdf}
+\caption{\label{fig:performance-cost-benchmark-autoscaling} Results of autoscaling benchmark. Scaling setting indicates static number of replicas or autoscaling policy.}
+\end{figure}
+
+\begin{figure}[ht]
+\centering
+\includegraphics[width=\textwidth]{plots/config-run-variance-autoscaling.pdf}
+\caption{\label{fig:config-run-variance-autoscaling} Execution time of individual configuration runs. Numbers in blue indicate the mean value.}
+\end{figure}
+
 \todo{Discuss results of optimizations}
 also talk about average queue time

-
 ## Real-world test scenario
--- a/plots/config-run-analysis.py
+++ b/plots/config-run-analysis.py
@ -0,0 +1,77 @@
+import matplotlib.pyplot as plt
+import seaborn as sns
+import json
+import sys
+
+with open('scaling-benchmarks.json', 'r') as fd:
+    data = json.load(fd)
+
+
+#labels = list(data.keys())
+#labels = ['2', '3', '5', '10', '15', '20', 'hpav1']
+labels = ['5', '10', 'hpav1', 'hpav2', 'hpav3', 'hpav4', 'hpav5']
+expanded_labels = []
+config_run_times = []
+
+for label in labels:
+    expanded_labels.append(label)
+    exec_times = []
+    for run in data[label]:
+        exec_times += run['execution_times']
+
+    config_run_times.append(exec_times)
+
+fig, axl = plt.subplots(nrows=1, ncols=1, figsize=(7, 3))
+axl.yaxis.grid(True, zorder=0.4)
+axl.set_ylabel('Configuration Run Time (in Seconds)')
+axl.set_xlabel('Scaling Setting')
+
+# Do a regular boxplot:
+# bplot2 = axl.boxplot(config_run_times,
+#                      notch=False,  # notch shape
+#                      vert=True,  # vertical box alignment
+#                      patch_artist=True,  # fill with color
+#                      labels=["[" + l + "]" for l in expanded_labels], # will be used to label x-ticks
+#                      )
+# axl.set_xticks(range(1, len(labels) + 1))
+# axl.set_xticklabels(labels)
+
+# Do a violin plot
+plot = sns.violinplot(data=config_run_times,
+                      ax=axl,
+                      scale="count",
+               )
+
+axl.set_xticks(range(0, len(labels)))
+axl.set_xticklabels(labels)
+
+# Add median values at the top:
+def flatten(list_of_lists):
+    return [val for sublist in list_of_lists for val in sublist]
+
+def median(alist):
+    return "%.1f" % (sum(alist)/len(alist),)
+
+#axl.set_yticks([i*10 for i in range(0,200,25)])
+#plot_y_max = max(flatten(config_run_times))+600
+#plot_y_min = min(flatten(config_run_times))-200
+#offset = 200
+axl.set_yticks(range(0,500,100))
+plot_y_max = max(flatten(config_run_times))+150
+plot_y_min = min(flatten(config_run_times))-100
+offset = 50
+
+
+axl.set_ylim([plot_y_min,plot_y_max])
+
+for i, label in enumerate(labels):
+    axl.text(i, plot_y_max-offset, median(config_run_times[i]),
+             horizontalalignment='center',color='b')
+
+
+fig.tight_layout()
+fig.savefig('config-run-variance-autoscaling.pdf',
+            format='pdf',
+            # dpi=100,
+            # bbox_inches='tight',
+            )
--- a/plots/config-run-variance-autoscaling.pdf
+++ b/plots/config-run-variance-autoscaling.pdf
--- a/plots/config-run-variance-static.pdf
+++ b/plots/config-run-variance-static.pdf
--- a/plots/performance-analysis.py
+++ b/plots/performance-analysis.py
@ -0,0 +1,46 @@
+import matplotlib.pyplot as plt
+import seaborn as sns
+import json
+import sys
+
+with open('scaling-benchmarks.json', 'r') as fd:
+    data = json.load(fd)
+
+
+#labels = list(data.keys())
+labels = ['2', '3', '5', '10', '15', '20', 'hpav1']
+#labels = ['5', '10', 'hpav1', 'hpav2', 'hpav3', 'hpav4', 'hpav5']
+
+fig, axr = plt.subplots(nrows=1, ncols=1, figsize=(7, 3.5))
+axr.grid(True, zorder=1.0)
+
+for label in labels:
+    x_vals = [run['time_to_completion'] for run in data[label]]
+    y_vals = [run['replica_seconds'] for run in data[label]]
+    sns.scatterplot(
+        x=x_vals,
+        y=y_vals,
+        zorder=2.0, # make sure it's on top of the grid
+        s=70, # controls size of dots
+        label=label,
+    )
+
+    # text_x = max(x_vals) + 25
+    # text_y = max(y_vals) + 25
+    # if label == "2":
+    #     text_x -= 15
+    #     text_y += 0.1*text_y
+    # if label == "20":
+    #     text_y -= 0.01*text_y
+    # axr.text(text_x, text_y, "[" + label + "]")
+
+axr.legend(title='Scaling setting:')
+axr.set_xlabel('Performance: Time to Completion (in Seconds)')
+axr.set_ylabel('Cost: Replica Seconds')
+
+fig.tight_layout()
+
+# save the plot as a file
+fig.savefig('performance-cost-benchmark-static.pdf',
+            format='pdf',
+            )
--- a/plots/performance-cost-benchmark-autoscaling.pdf
+++ b/plots/performance-cost-benchmark-autoscaling.pdf
--- a/plots/performance-cost-benchmark-static.pdf
+++ b/plots/performance-cost-benchmark-static.pdf
--- a/plots/scaling-benchmarks.json
+++ b/plots/scaling-benchmarks.json
@ -0,0 +1,46 @@
+{
+"2": [
+{"start_time":"2021-04-22 18:08:36.685791454 +0300 EEST m=+537.269743285","end_time":"2021-04-22 18:41:21.887211436 +0300 EEST m=+2502.471163103","config_runs":10,"mock_settings":"python3 ci/tools/scripts/setup_mock_data.py \\\n    --no-threat --no-execution-run --no-compliance-run --no-configuration-run --no-failing \\\n    -c 20 --assets-per-group 5 --tasklists-per-group 2 --policies-per-group 2","execution_times":[400.51,603.226,830.808,1048.481,1156.825,1197.54,1219.67,1288.274,1302.755,1361.179],"time_to_completion":1965.201,"replica_seconds":3926},
+{"start_time":"2021-04-22 18:50:55.606836378 +0300 EEST m=+531.385739069","end_time":"2021-04-22 19:23:52.532418751 +0300 EEST m=+2508.311321502","config_runs":10,"mock_settings":"python3 ci/tools/scripts/setup_mock_data.py \\\n    --no-threat --no-execution-run --no-compliance-run --no-configuration-run --no-failing \\\n    -c 20 --assets-per-group 5 --tasklists-per-group 2 --policies-per-group 2","execution_times":[300.13,532.507,834.049,1056.589,1197.392,1221.934,1281.557,1306.407,1354.468,1376.254],"time_to_completion":1976.925,"replica_seconds":3952},
+{"start_time":"2021-04-22 19:33:34.882763738 +0300 EEST m=+539.420680587","end_time":"2021-04-22 20:06:26.28288876 +0300 EEST m=+2510.820805684","config_runs":10,"mock_settings":"python3 ci/tools/scripts/setup_mock_data.py \\\n    --no-threat --no-execution-run --no-compliance-run --no-configuration-run --no-failing \\\n    -c 20 --assets-per-group 5 --tasklists-per-group 2 --policies-per-group 2","execution_times":[287.434,535.278,860.482,1077.441,1210.97,1240.629,1276.407,1312.03,1346.463,1366.058],"time_to_completion":1971.4,"replica_seconds":3934}],
+"3": [
+{"start_time":"2021-04-22 20:20:32.365657798 +0300 EEST m=+532.927044493","end_time":"2021-04-22 20:43:01.194262941 +0300 EEST m=+1881.755649691","config_runs":10,"mock_settings":"python3 ci/tools/scripts/setup_mock_data.py \\\n    --no-threat --no-execution-run --no-compliance-run --no-configuration-run --no-failing \\\n    -c 20 --assets-per-group 5 --tasklists-per-group 2 --policies-per-group 2","execution_times":[200.722,333.114,396.449,534.512,653.773,719.494,713.805,747.036,725.212,745.521],"time_to_completion":1348.828,"replica_seconds":5391},
+{"start_time":"2021-04-22 22:11:13.405687529 +0300 EEST m=+535.272827199","end_time":"2021-04-22 22:34:05.096971978 +0300 EEST m=+1906.964111739","config_runs":10,"mock_settings":"python3 ci/tools/scripts/setup_mock_data.py \\\n    --no-threat --no-execution-run --no-compliance-run --no-configuration-run --no-failing \\\n    -c 20 --assets-per-group 5 --tasklists-per-group 2 --policies-per-group 2","execution_times":[218.686,355.733,402.242,535.956,655.638,726.335,752.837,737.502,743.486,763.386],"time_to_completion":1371.691,"replica_seconds":5391},
+{"start_time":"2021-04-23 07:11:29.105792142 +0300 EEST m=+531.953082610","end_time":"2021-04-23 07:33:25.96847478 +0300 EEST m=+1848.815765218","config_runs":10,"mock_settings":"python3 ci/tools/scripts/setup_mock_data.py \\\n    --no-threat --no-execution-run --no-compliance-run --no-configuration-run --no-failing \\\n    -c 20 --assets-per-group 5 --tasklists-per-group 2 --policies-per-group 2","execution_times":[180.279,281.405,382.798,513.461,584.144,698.717,721.278,709.806,708.444,709.878],"time_to_completion":1316.862,"replica_seconds":5400}],
+"5": [
+{"start_time":"2021-04-23 12:11:31.941766565 +0300 EEST m=+543.841987489","end_time":"2021-04-23 12:25:53.172703183 +0300 EEST m=+1405.072924170","config_runs":10,"mock_settings":"python3 ci/tools/scripts/setup_mock_data.py \\\n    --no-threat --no-execution-run --no-compliance-run --no-configuration-run --no-failing \\\n    -c 20 --assets-per-group 5 --tasklists-per-group 2 --policies-per-group 2","execution_times":[87.024,159.533,169.983,222.326,284.632,209.187,277.148,290.27,274.187,258.167],"time_to_completion":861.23,"replica_seconds":9000},
+{"start_time":"2021-04-23 12:51:02.477664269 +0300 EEST m=+527.447880450","end_time":"2021-04-23 13:05:23.399529557 +0300 EEST m=+1388.369745773","config_runs":10,"mock_settings":"python3 ci/tools/scripts/setup_mock_data.py \\\n    --no-threat --no-execution-run --no-compliance-run --no-configuration-run --no-failing \\\n    -c 20 --assets-per-group 5 --tasklists-per-group 2 --policies-per-group 2","execution_times":[85.443,149.153,163.443,174.074,208.991,225.368,278.62,292.538,270.287,252.372],"time_to_completion":860.921,"replica_seconds":8995},
+{"start_time":"2021-04-23 13:30:45.617707352 +0300 EEST m=+540.052938314","end_time":"2021-04-23 13:45:06.098998495 +0300 EEST m=+1400.534229436","config_runs":10,"mock_settings":"python3 ci/tools/scripts/setup_mock_data.py \\\n    --no-threat --no-execution-run --no-compliance-run --no-configuration-run --no-failing \\\n    -c 20 --assets-per-group 5 --tasklists-per-group 2 --policies-per-group 2","execution_times":[92.154,134.23,179.699,205.6,236.221,267.963,283.105,309.72,279.982,254.626],"time_to_completion":860.481,"replica_seconds":8990}],
+"10": [
+{"start_time":"2021-04-23 09:59:15.57365416 +0300 EEST m=+530.401156705","end_time":"2021-04-23 10:10:35.357025813 +0300 EEST m=+1210.184528359","config_runs":10,"mock_settings":"python3 ci/tools/scripts/setup_mock_data.py \\\n    --no-threat --no-execution-run --no-compliance-run --no-configuration-run --no-failing \\\n    -c 20 --assets-per-group 5 --tasklists-per-group 2 --policies-per-group 2","execution_times":[101.575,84.768,88.525,85.337,83.324,78.63,84.757,85.874,96.909,78.88],"time_to_completion":679.783,"replica_seconds":18000},
+{"start_time":"2021-04-23 19:59:03.813673707 +0300 EEST m=+536.826641630","end_time":"2021-04-23 20:10:44.358338782 +0300 EEST m=+1237.371306752","config_runs":10,"mock_settings":"python3 ci/tools/scripts/setup_mock_data.py \\\n    --no-threat --no-execution-run --no-compliance-run --no-configuration-run --no-failing \\\n    -c 20 --assets-per-group 5 --tasklists-per-group 2 --policies-per-group 2","execution_times":[85.74,92.581,86.013,85.719,85.462,95.963,86.211,89.934,89.806,92.04],"time_to_completion":700.544,"replica_seconds":17920},
+{"start_time":"2021-04-23 20:38:46.33766965 +0300 EEST m=+539.365044840","end_time":"2021-04-23 20:50:16.358693982 +0300 EEST m=+1229.386069155","config_runs":10,"mock_settings":"python3 ci/tools/scripts/setup_mock_data.py \\\n    --no-threat --no-execution-run --no-compliance-run --no-configuration-run --no-failing \\\n    -c 20 --assets-per-group 5 --tasklists-per-group 2 --policies-per-group 2","execution_times":[86.611,85.798,89.886,93.224,89.114,85.007,89.492,87.942,88.329,85.178],"time_to_completion":690.021,"replica_seconds":18000}],
+"15": [
+{"start_time":"2021-04-23 21:18:24.98170041 +0300 EEST m=+535.546435665","end_time":"2021-04-23 21:29:55.359391343 +0300 EEST m=+1225.924126517","config_runs":10,"mock_settings":"python3 ci/tools/scripts/setup_mock_data.py \\\n    --no-threat --no-execution-run --no-compliance-run --no-configuration-run --no-failing \\\n    -c 20 --assets-per-group 5 --tasklists-per-group 2 --policies-per-group 2","execution_times":[89.15,83.536,84.328,87.653,84.198,85.533,92.783,79.717,82.591,84.136],"time_to_completion":690.377,"replica_seconds":26985},
+{"start_time":"2021-04-23 22:01:51.925668107 +0300 EEST m=+290.700784811","end_time":"2021-04-23 22:13:26.342513274 +0300 EEST m=+985.117629946","config_runs":10,"mock_settings":"python3 ci/tools/scripts/setup_mock_data.py \\\n    --no-threat --no-execution-run --no-compliance-run --no-configuration-run --no-failing \\\n    -c 20 --assets-per-group 5 --tasklists-per-group 2 --policies-per-group 2","execution_times":[87.306,87.008,92.015,92.752,84.568,90.643,90.027,88.888,94.704,89.471],"time_to_completion":694.416,"replica_seconds":26820},
+{"start_time":"2021-04-23 22:41:32.653755679 +0300 EEST m=+537.432203527","end_time":"2021-04-23 22:53:03.080450835 +0300 EEST m=+1227.858898677","config_runs":10,"mock_settings":"python3 ci/tools/scripts/setup_mock_data.py \\\n    --no-threat --no-execution-run --no-compliance-run --no-configuration-run --no-failing \\\n    -c 20 --assets-per-group 5 --tasklists-per-group 2 --policies-per-group 2","execution_times":[90.456,92.746,89.752,88.687,97.381,87.903,85.754,88.129,98.395,82.799],"time_to_completion":690.426,"replica_seconds":27000}],
+"20": [
+{"start_time":"2021-04-23 23:21:20.421915288 +0300 EEST m=+544.616550887","end_time":"2021-04-23 23:33:12.007437811 +0300 EEST m=+1256.202073267","config_runs":10,"mock_settings":"python3 ci/tools/scripts/setup_mock_data.py \\\n    --no-threat --no-execution-run --no-compliance-run --no-configuration-run --no-failing \\\n    -c 20 --assets-per-group 5 --tasklists-per-group 2 --policies-per-group 2","execution_times":[85.699,101.27,83.715,84.722,88.523,83.798,92.11,86.952,80.345,100.908],"time_to_completion":711.585,"replica_seconds":36000},
+{"start_time":"2021-04-24 00:01:00.825707333 +0300 EEST m=+537.147528924","end_time":"2021-04-24 00:12:33.632496876 +0300 EEST m=+1229.954318438","config_runs":10,"mock_settings":"python3 ci/tools/scripts/setup_mock_data.py \\\n    --no-threat --no-execution-run --no-compliance-run --no-configuration-run --no-failing \\\n    -c 20 --assets-per-group 5 --tasklists-per-group 2 --policies-per-group 2","execution_times":[83.154,86.782,85.088,85.937,82.424,93.91,86.716,89.188,100.608,86.939],"time_to_completion":692.806,"replica_seconds":35940},
+{"start_time":"2021-04-24 00:41:12.561878198 +0300 EEST m=+534.532860267","end_time":"2021-04-24 00:52:53.91477247 +0300 EEST m=+1235.885754443","config_runs":10,"mock_settings":"python3 ci/tools/scripts/setup_mock_data.py \\\n    --no-threat --no-execution-run --no-compliance-run --no-configuration-run --no-failing \\\n    -c 20 --assets-per-group 5 --tasklists-per-group 2 --policies-per-group 2","execution_times":[91.072,91.947,88.866,86.754,94.106,89,95.568,94.889,86.082,99.899],"time_to_completion":701.352,"replica_seconds":36000}],
+"hpav1": [
+{"start_time":"2021-04-23 07:55:37.238114485 +0300 EEST m=+541.528714702","end_time":"2021-04-23 08:07:17.458089444 +0300 EEST m=+1241.748689714","config_runs":10,"mock_settings":"python3 ci/tools/scripts/setup_mock_data.py \\\n    --no-threat --no-execution-run --no-compliance-run --no-configuration-run --no-failing \\\n    -c 20 --assets-per-group 5 --tasklists-per-group 2 --policies-per-group 2","execution_times":[183.291,153.907,96.872,88.776,83.571,89.249,82.646,86.252,95.102,89.751],"time_to_completion":700.219,"replica_seconds":18241},
+{"start_time":"2021-04-23 08:35:19.38170654 +0300 EEST m=+538.972741231","end_time":"2021-04-23 08:46:49.668113742 +0300 EEST m=+1229.259148419","config_runs":10,"mock_settings":"python3 ci/tools/scripts/setup_mock_data.py \\\n    --no-threat --no-execution-run --no-compliance-run --no-configuration-run --no-failing \\\n    -c 20 --assets-per-group 5 --tasklists-per-group 2 --policies-per-group 2","execution_times":[240.81,175.048,82.58,91.847,83.354,79.875,95.163,93.172,85.041,87.228],"time_to_completion":690.286,"replica_seconds":18625},
+{"start_time":"2021-04-23 09:14:46.301782334 +0300 EEST m=+523.682013912","end_time":"2021-04-23 09:26:18.575651917 +0300 EEST m=+1215.955883341","config_runs":10,"mock_settings":"python3 ci/tools/scripts/setup_mock_data.py \\\n    --no-threat --no-execution-run --no-compliance-run --no-configuration-run --no-failing \\\n    -c 20 --assets-per-group 5 --tasklists-per-group 2 --policies-per-group 2","execution_times":[188.985,95.133,93.551,89.54,87.052,81.939,84.268,79.721,85.117,87.759],"time_to_completion":692.273,"replica_seconds":18697}],
+"hpav2": [
+{"start_time":"2021-04-23 16:46:44.042632418 +0300 EEST m=+534.114126241","end_time":"2021-04-23 16:58:14.00042728 +0300 EEST m=+1224.071921141","config_runs":10,"mock_settings":"python3 ci/tools/scripts/setup_mock_data.py \\\n    --no-threat --no-execution-run --no-compliance-run --no-configuration-run --no-failing \\\n    -c 20 --assets-per-group 5 --tasklists-per-group 2 --policies-per-group 2","execution_times":[196.176,133.164,90.589,86.871,128.023,86.987,92.587,85.138,88.256,86.376],"time_to_completion":689.957,"replica_seconds":11012},
+{"start_time":"2021-04-24 07:38:23.744834057 +0300 EEST m=+534.696966462","end_time":"2021-04-24 07:49:53.681162643 +0300 EEST m=+1224.633295101","config_runs":10,"mock_settings":"python3 ci/tools/scripts/setup_mock_data.py \\\n    --no-threat --no-execution-run --no-compliance-run --no-configuration-run --no-failing \\\n    -c 20 --assets-per-group 5 --tasklists-per-group 2 --policies-per-group 2","execution_times":[182.404,152.511,145.141,110.746,89.356,80.144,96.639,93.631,85.618,82.737],"time_to_completion":689.936,"replica_seconds":10567},
+{"start_time":"2021-04-24 08:18:04.049766968 +0300 EEST m=+537.235510193","end_time":"2021-04-24 08:29:33.805796739 +0300 EEST m=+1226.991539990","config_runs":10,"mock_settings":"python3 ci/tools/scripts/setup_mock_data.py \\\n    --no-threat --no-execution-run --no-compliance-run --no-configuration-run --no-failing \\\n    -c 20 --assets-per-group 5 --tasklists-per-group 2 --policies-per-group 2","execution_times":[183.024,118.492,96.179,84.889,81.672,93.271,89.933,84.725,92.915,87.721],"time_to_completion":689.756,"replica_seconds":10743}],
+"hpav3": [
+{"start_time":"2021-04-23 18:39:53.021749796 +0300 EEST m=+268.578466646","end_time":"2021-04-23 18:51:25.630831025 +0300 EEST m=+961.187547862","config_runs":10,"mock_settings":"python3 ci/tools/scripts/setup_mock_data.py \\\n    --no-threat --no-execution-run --no-compliance-run --no-configuration-run --no-failing \\\n    -c 20 --assets-per-group 5 --tasklists-per-group 2 --policies-per-group 2","execution_times":[190.188,135.591,98.83,87.461,86.326,91.335,82.933,92.991,89.044,86.378],"time_to_completion":692.609,"replica_seconds":9498},
+{"start_time":"2021-04-24 08:57:38.217776346 +0300 EEST m=+531.077181864","end_time":"2021-04-24 09:08:59.498112288 +0300 EEST m=+1212.357517858","config_runs":10,"mock_settings":"python3 ci/tools/scripts/setup_mock_data.py \\\n    --no-threat --no-execution-run --no-compliance-run --no-configuration-run --no-failing \\\n    -c 20 --assets-per-group 5 --tasklists-per-group 2 --policies-per-group 2","execution_times":[180.65,122.402,77.415,85.846,91.755,95.715,122.821,90.041,106.447,79.22],"time_to_completion":681.28,"replica_seconds":8451},
+{"start_time":"2021-04-24 09:37:25.145724546 +0300 EEST m=+543.827686903","end_time":"2021-04-24 09:49:08.822249496 +0300 EEST m=+1247.504211829","config_runs":10,"mock_settings":"python3 ci/tools/scripts/setup_mock_data.py \\\n    --no-threat --no-execution-run --no-compliance-run --no-configuration-run --no-failing \\\n    -c 20 --assets-per-group 5 --tasklists-per-group 2 --policies-per-group 2","execution_times":[192.955,169.764,154.476,84.2,114.964,86.121,93.016,126.221,111.792,95.622],"time_to_completion":703.676,"replica_seconds":8531}],
+"hpav4": [
+{"start_time":"2021-04-23 15:03:46.785710656 +0300 EEST m=+539.494768426","end_time":"2021-04-23 15:15:30.774566433 +0300 EEST m=+1243.483624180","config_runs":10,"mock_settings":"python3 ci/tools/scripts/setup_mock_data.py \\\n    --no-threat --no-execution-run --no-compliance-run --no-configuration-run --no-failing \\\n    -c 20 --assets-per-group 5 --tasklists-per-group 2 --policies-per-group 2","execution_times":[199.135,134.694,91.468,92.563,119.402,114.288,128.775,97.409,133.912,96.254],"time_to_completion":703.988,"replica_seconds":7625},
+{"start_time":"2021-04-24 10:17:42.917707044 +0300 EEST m=+527.895535414","end_time":"2021-04-24 10:29:23.889262334 +0300 EEST m=+1228.867090746","config_runs":10,"mock_settings":"python3 ci/tools/scripts/setup_mock_data.py \\\n    --no-threat --no-execution-run --no-compliance-run --no-configuration-run --no-failing \\\n    -c 20 --assets-per-group 5 --tasklists-per-group 2 --policies-per-group 2","execution_times":[186.515,184.394,139.319,81.976,88.858,121.758,103.404,108.905,117.381,90.656],"time_to_completion":700.971,"replica_seconds":7475},
+{"start_time":"2021-04-24 10:58:07.309716212 +0300 EEST m=+529.258708644","end_time":"2021-04-24 11:09:49.056091439 +0300 EEST m=+1231.005083795","config_runs":10,"mock_settings":"python3 ci/tools/scripts/setup_mock_data.py \\\n    --no-threat --no-execution-run --no-compliance-run --no-configuration-run --no-failing \\\n    -c 20 --assets-per-group 5 --tasklists-per-group 2 --policies-per-group 2","execution_times":[191.107,123.687,92.064,89.977,89.825,135.715,93.445,130.762,121.788,91.918],"time_to_completion":701.746,"replica_seconds":7632}],
+"hpav5": [
+{"start_time":"2021-04-23 19:19:23.933701222 +0300 EEST m=+527.603123457","end_time":"2021-04-23 19:31:37.871511594 +0300 EEST m=+1261.540933913","config_runs":10,"mock_settings":"python3 ci/tools/scripts/setup_mock_data.py \\\n    --no-threat --no-execution-run --no-compliance-run --no-configuration-run --no-failing \\\n    -c 20 --assets-per-group 5 --tasklists-per-group 2 --policies-per-group 2","execution_times":[245.727,215.634,83.668,146.78,121.145,146.502,105.206,134.187,142.139,123.075],"time_to_completion":733.937,"replica_seconds":6577},
+{"start_time":"2021-04-24 11:37:39.813729687 +0300 EEST m=+529.310820611","end_time":"2021-04-24 11:49:41.252677515 +0300 EEST m=+1250.749768499","config_runs":10,"mock_settings":"python3 ci/tools/scripts/setup_mock_data.py \\\n    --no-threat --no-execution-run --no-compliance-run --no-configuration-run --no-failing \\\n    -c 20 --assets-per-group 5 --tasklists-per-group 2 --policies-per-group 2","execution_times":[184.476,103.858,136.93,88.774,132.856,199.417,157.243,148.596,165.462,116.995],"time_to_completion":721.438,"replica_seconds":6796},
+{"start_time":"2021-04-24 14:00:37.365651384 +0300 EEST m=+537.136091772","end_time":"2021-04-24 14:12:39.554796933 +0300 EEST m=+1259.325237298","config_runs":10,"mock_settings":"python3 ci/tools/scripts/setup_mock_data.py \\\n    --no-threat --no-execution-run --no-compliance-run --no-configuration-run --no-failing \\\n    -c 20 --assets-per-group 5 --tasklists-per-group 2 --policies-per-group 2","execution_times":[193.955,90.22,109.771,86.624,142.376,144.37,140.194,145.807,148.949,113.994],"time_to_completion":722.189,"replica_seconds":6629}]
+}