Last week, we published a blog with recommendation on securing Couchbase data platform in response to industry-wide security vulnerabilities. We continued to analyze the potential performance impact caused by the patched OS binaries and this blog post captures the detailed evaluation.
As mentioned in previous blog, for these attacks to be feasible, attackers must be able to run malicious processes on the same host and processor as the victim processes. The risk of this attack happening on a controlled production environment is low since there are restrictions in place on the applications running.
If you can control access to a machine, you may not need (wherever applicable) to set the kernel parameters to enable the mitigations and hence not incur performance degradation.
Impact Analysis
When you patch your OS with the binaries currently available to mitigate these attacks you will likely see increased CPU utilization. For example, CPU usage increased by about 20% during one of our tests. (Check your Web Console monitoring page to access bucket statistics.)
To evaluate performance impact we ran the following tests
- Scenario ‘A’ – Couchbase Cluster is sufficiently sized with additional capacity for production deployment.
- Scenario ‘B’ – Couchbase Cluster is sized appropriately to handle existing load, but without further capacity available.
Workload
YCSB Workload A with mixed workload (50/50 read/write).
Scenario A
Ran YCSB Workload A on Couchbase Cluster – 4 nodes, 2 buckets, Intel Processor E5-2630 v4 with 20 hyper-threaded cores, 64GB RAM.
We compared the performance between un-patched ‘CentOS 7 kernel 3.10.0-514.2.2’ and patched ‘CentOS 7 kernel 3.10.0.693.11.6’ version.
Results – Below graph shows CPU utilization increased roughly by 30% on the patched machines. However, the performance impact to Throughput and Latency was under 5% since these machines had additional CPU capacity to grow.
|
|
Un-patched – CentOS 7 kernel 3.10.0-514.2.2 (in μs) | Patched – CentOS 7 kernel 3.10.0-693.11.6 (in μs) | |
| Read Latency | Average | 187 | 196 |
| 95th | 213 | 228 | |
| 99th | 1,497 | 1,525 | |
| Write Latency | Average | 209 | 219 |
| 95th | 236 | 254 | |
| 99th | 1,583 | 1,606 |
Scenario B
Run YCSB Workload A on Couchbase Cluster – 3 nodes, 1 bucket, Intel Processor E5-2680 v3 with 12 cores, 64GB RAM.
Results: Below graph shows Throughput and Latency on patched machine is affected since CPU is running close to maximum utilization on these machines.
|
|
Un-patched – CentOS 7 kernel 3.10.0-514.2.2 (in μs) | Patched – CentOS 7 kernel 3.10.0-693.11.6 (in μs) | |
| Read Latency | Average | 847 | 1,318 |
| 95th | 2,627 | 4,093 | |
| 99th | 7,207 | 9,167 | |
| Write Latency | Average | 879 | 1,352 |
| 95th | 2,703 | 4,163 | |
| 99th | 7,315 | 9,271 |
Conclusion
We recommend re-evaluating your sizing for Couchbase cluster to ensure it has additional CPU capacity before applying OS patch to have a minimal impact on Throughput and Latency.
