SAPS Olympics

As it’s clearly stated in the official site (http://sap.com/benchmark), the SAPS (SAP Application Performance Standard) is a hardware-independent unit of measurement that describes the performance of a system configuration in the SAP environment. Almost everyone in the computer system performance arena is aware of the SAPS, and indeed the SAPS has become a de facto standard measure of performance.

As I browse the SAPS list sometimes, I’ve decided that I’m putting the hat of objective journalist here and predigest the list for you. I don’t pretend to teach or comment anything else, at least by now. After reading this you may draw conclusions by yourself.

There are 774 official SAPS results posted at the public official site (SD 2-Tier) http://global.sap.com/solutions/benchmark/sd2tier.epx. The first one dates back to 1996, a Siemens machine. The last benchmark corresponds to a HP ProLiant DL380 Gen9, with certification number 2014037, published last October 17^th.

For this analysis to be more relevant to you I have exclusively considered systems that have been measured with the latest benchmark specification: SAP Enhancement Package 5 for ERP 6.0 (SAP has not updated the benchmark environment to any newer versions EHP6 or EHP7). There are 75 such systems (as of Oct 20^th 2014), and the first one is a HP ProLiant BL465c Gen8, with certification number 2012026, published Jun 4^th 2012. So we are analyzing a little bit more than the last two years.

 

SAP Technology Partners

Which SAP Technology Partners (a SAP concept) have actively been benchmarking SAPS? Here are the results: Dell: 14, HP: 14, IBM: 14, Cisco Systems: 13, Fujitsu: 8, NEC: 4, Hitachi: 3, Oracle: 3, Stratus Technologies: 1, and Unisys: 1.

 

By CPU family

Count by CPU family type:  Intel Xeon: 58, IBM POWER: 9, AMD Opteron: 3, Oracle SPARC: 3, and Fujitsu SPARC64: 2.

 

By Operating System

Count by Operating System:  Windows Server: 41, Red Hat Enterprise Linux: 17, SUSE Linux Enterprise Server: 5, IBM AIX: 7 and Solaris 11: 5.

 

By Relational Database Management System

Count by the Relational Database Management System? SQL Server: 36, SAP/Sybase ASE: 21, IBM DB2: 13, Oracle: 5.

 

Absolute Number Of SAPS (SAPS per System)

Gold: Fujitsu SPARC M10-4S 40 chips SPARC64 X @3000 MHz, (640 cores, 1280 threads): 844420 SAPS.

Silver: Oracle SPARC M6-32 32 chips SPARC M6 @3600 MHz (384 cores, 3072 threads): 793930 SAPS.

Bronze: Oracle SPARC Server M5-32 32 chips SPARCM5 @3600 MHz (192 cores, 1536 threads): 472600 SAPS.

The four first positions are held by Oracle and Fujitsu SPARC servers.

The last (which kind of medal does correspond to this?): Cisco UCS C240 M3 2 Intel Xeon Processor E5-2640 (12 cores, 24 threads) 24600 SAPS.

 

 

SAPS per core

Gold: IBM Power Enterprise System E870 with 8 chips POWER8 @4190 MHz (80 cores, 640 threads): 5451 SAPS/core.

Silver: IBM Power System S824 with 4 chips POWER8 @3520 MHz (24 cores, 192 threads): 4828 SAPS/core.

Bronze: IBM Flex System p260 Compute Node with 2 chips IBM POWER7+ @4100 MHz (16 cores, 64 threads): 3419 SAPS/core.

The nine first positions are held by IBM POWER servers. They are the absolute winners in this category!

Lowest: HP ProLiant BL465c Gen8 2  chips AMD Opteron 6278 @2400 MHz (32 cores, 32 threads): 864 SAPS/core.

 

SAPS per thread

Gold: Dell PowerEdge R730 with 2 Intel Xeon Processor E5-2699 v3 @2300 MHz (36 cores, 72 threads): 1252 SAPS/thread.

Silver: HP ProLiant DL380 Gen9 with 2 Intel Xeon Processor E5-2699 v3 @2300 MHz (36 cores, 72 threads): 1221 SAPS/thread.

Bronze: Dell PowerEdge R730 with 2 Intel Xeon Processor E5-2699 v3 @2300 MHz (36 cores, 72 threads): 1217 SAPS/thread.

The seven first positions are held by servers with Intel Xeon Processor E5-2699 v3.

Lowest: Oracle SPARC Server T5-8 8 chips SPARC T5 @3600 MHz (128 cores, 1024 threads) 278 SAPS/thread. 

The Effect of IO Improvement: Bandwidth

Recently I’ve had the opportunity to deal with database systems that claim to have a very smart storage subsystem that makes them superior every time, everywhere. I’m going to analyze this from the point of view of the performance: what is the actual effect of this smartness?

A visit to the doctor

Typically a visit to the doctor consists of two differentiated stages: the interaction with the nurse (when arriving check if you’re expected to come, some admin stuff...), and with the doctor himself.  Let’s say you spend 2 minutes with the nurse, and 10 minutes with the doctor. That means the nurse is capable of attending 30 patients per hour, and the doctor 10 patients per hour. Technically it can be said that the nurse bandwidth is 30 pph (patients per hour) and the doctor bandwidth is 6 pph. 

Those are the performance numbers when the stages are considered individually. But what is the overall bandwidth of the medical center (nurse + doctor) system? The one of the nurse? The one of the doctor? Somewhere in between? If you think a little bit the conclusion must be crystal clear: the doctor’s, and just because his bandwidth, his capability to service patients, is the lesser of the two. You may try to inject in the medical center more than 6 pph, but they will proceed through the system at a maximum 6 pph rate, the medical center bandwidth. The doctor is the bottleneck stage of the medical center.

If you were the managing director and decided it was necessary to increase the “performance” of the hospital, what would you do? Employ smarter or more nurses, or employ smarter or more doctors? 

If you change the current nurse and employ a new one that is twice as efficient, that is, reduces from 2 min to 1 min the time spent with each patient, you would have increased the bandwidth of the nurse stage from 30 pph to 60 pph. But the doctor will continue being the bottleneck. The system bandwidth will remain the same! Conclusion: the nurse stage improvement is not effective.

If you change the current doctor and employ a new one that is twice as efficient, that is, reduces from 10 min to 5 min the time spent with each patient, you would have increased the bandwidth of the doctor stage from 6 pph to 12 pph. The doctor will continue being the bottleneck, but at an increased rate of 12 pph. Perfect! This is the right way to increase the medical center bandwidth, which has gone up from previous 6 pph to new 12 pph. The doctor (bottleneck) improvement is fully effective. 

A very important “rule” you should keep in gold letters: the only effective way to improve the bandwidth of a composite (several stages) system is to improve the bandwidth of the bottleneck stage.

It is worth to point out that which of the stages is the bottleneck also depends on the workload: the doctor is the bottleneck for the above workload, a normal patient visit. But imagine a new workload that consists of heavy administrative work that must be processed by the nurse and the doctor only has to stamp his signature. What do you think would be the bottleneck then?

Note: why doctors and medical centers? No particular reason beyond I’ve been today at one of them and it has occurred to me they are a very simple scenario everyone would have experienced.

A visit to a database machine

For the purpose of this simple analysis the database machine is also a two stage (or subsystem) system: the database node and the storage node. One user request, or visit, to the system places demands for service on the database and on the storage.

For a particular workload the database and storage nodes have a certain processing capacity. The (maximum) processing capacity is called bandwidth, and is measured in user requests per unit time, typically requests per second (req/s) or per minute (req/min).

The stage with the lesser of the two bandwidths -for this particular workload- is the bottleneck, and the system bandwidth will be equal to the bottleneck bandwidth, no more, no less. 

Consider a certain heavy IO workload, with high degree of IO contents, for which the storage is the bottleneck. Any improvement on the IO demand, being achieved by a reduction of IOs caused by smart intelligence or by using more or faster storage nodes, will increase the bandwidth of the storage stage. And, consequently, the bandwidth of the overall system / database machine. 

Let’s consider these particular values for the heavy IO workload: the bandwidth of the database node (B_DB) is B_DB=10 req/s (or 600 req/min), and the bandwidth of the storage node (B_ST) is B_ST=5 req/s (or 300 req/min). The system bandwidth (B) is equal to the one of the storage (bottleneck), that is, B= B_ST=5 req/s. If the IO reduction technique halves the IO demand, the improved storage bandwidth will be B’_ST=10 req/s (or 600 req/min). The database machine bandwidth will increase from 5 req/s to 10 req/s. That is B’= B’_ST=10 req/s.

Let’s explore the other extreme: a light IO workload, for with the database is the bottleneck (and the storage is not). Any reduction of the demand placed on the storage subsystem will have null, or very limited, effect as it is not improving the bottleneck. In this case to have or not a smarter storage doesn’t matter at all. You better wouldn’t pay for it.

Let’s use this particular values for the light IO workload: B_DB=5 req/s (or 300 req/min) and B_ST=10 req/s (or 600 req/min). The system bandwidth (B) is equal to the one of the database (bottleneck), that is, B= B_DB=5 req/s. If the IO reduction technique halves the IO demand, the improved storage bandwidth will be B’_ST=20 req/s (or 1200 req/min). But the overall database machine bandwidth will remain the same: B’= B_DB=5 req/s.

This light IO scenario is exactly what happens in benchmarks that have a very low IO content, like the SAP SD benchmark that meters the SAPS of the systems. It doesn’t stress the IO subsystem and B is limited by B_DB, no matter what extraordinary capable or smart storage you use.

Let’s point out here that the bandwidth of the overall system will never be greater than the bandwidth of the database node. This particularly means that the SAPS of the database machine will be always equal or less than the SAPS of the database node.

In the real world

Back in the real world there are more tones that merely black or white. But to know the extremes should be of help to us.

Any IO improvement, for example with smarter storage, has its maximum effect when the storage is the bottleneck. It has no effect at all when the storage is not the bottleneck (and the database node is). Consequently from the bandwidth point of view there is no such an every time, everywhere absolute positive improvement for having a smarter storage. It will depend on where the bottleneck is placed. 

But what it seems clear is that this smartness will cost you a positive absolute number.

Demystifying Virtual Capacity Mini Papers

The IBM Technology Leadership Council from Brazil has translated to Portuguese and published two brief articles I have written on virtual capacity. Here are the public links:

https://www.ibm.com/developerworks/community/blogs/tlcbr/resource/mp/TLC-BR_Mini_Paper_Ano_9_N_213_VirtualCapacityPartI.pdf

https://www.ibm.com/developerworks/community/blogs/tlcbr/resource/mp/TLC-BR_Mini_Paper_Ano_9_N_214_VirtualCapacityPartII.pdf

As some of you –like me- might not be able to fully understand Portuguese, I expose below the English version for both mini papers.

Demystifying Virtual Capacity, Part I

In a virtualized system virtual machines share the underlying physical machine (PM). This physical machine has a well-defined processing capacity. Let us call virtual capacity to the actual capacity used by a virtual machine.

How much of the real physical capacity goes to a particular virtual machine (VM)? What are the parameters that determine the virtual capacity? In this paper we will briefly review these factors. Different hypervisors and virtualization technologies use different names for them, but in all the cases the underlying concepts are largely the same.

·         Physical Machine capacity. The virtualization layer distributes the PM capacity among its VMs. Of course, this PM capacity is the absolute limit of the virtual capacity: a hosted VM cannot be larger than the hosting PM. Virtualization overhead should also be taken into account. The capacity is typically measured in processor cores or aggregated CPU cycles (MHz).

·         Guaranteed capacity. This is the capacity the VM will have for sure when/if demanded. In example, consider a guaranteed capacity of 4 cores. If the VM workload demand is 2 cores, the VM will use 2 cores. But if the demand is 5 cores, it will use 4 for sure and the remaining 1 core may be used or not depending on additional factors. It is also called entitlement and reservation. It’s measured in capacity units.

·         Exclusive use attribute. Flag or binary value indicating if the guaranteed capacity is set aside for the exclusive use of the VM. If it is not, the unused guaranteed capacity is available for use by the rest of VMs sharing the PM. Other name for this parameter is dedicated use.

·         Limit / cap attribute. Flag or binary value that indicates if the guaranteed capacity can be exceeded or not. May the VM virtual capacity go beyond the guaranteed capacity in case of need? The response is yes if unlimited / uncapped, and no if limited / capped. Some hypervisors specify a limit capacity not tied to the guaranteed capacity.

·         Virtual cores. A fundamental, but sometimes tricky concept, and the link between the physical and the virtual world. The operative system inside the VM sees (virtual) cores and dispatches runnable processes to them. Then the hypervisor dispatches virtual cores to physical cores. The number of virtual cores may limit the virtual capacity: i.e. a VM with 2 virtual cores never can have a virtual capacity greater than 2 physical cores.

·         Relative priority. This parameter sets relative priorities between VMs that compete for capacity. This competence may happen when the sum of demands are greater than the PM capacity. You can find this concept under several names like uncapped weight and shares.

The actual virtual capacity depends on all the above factors. Let's go from theory to practice, and play with a simple scenario: two VMs, the red and the blue, sharing an 8 core PM.

Red and blue virtual machines are both defined in the same way:

·         Guaranteed capacity: 4 cores.

·         Exclusive use: NO.

·         Uncapped / Unlimited.

·         Virtual cores: 8.

·         Relative priority: 128.

What do you think it would happen when red customers place a demand on the red VM of 5 cores, and at the same time blue customers place a demand on the blue VM of 5 cores?

According to the above parameterization, it's possible for the red VM to use 5 physical cores because it's uncapped/unlimited and it has at least 5 virtual cores. But to be able to go beyond its guarantee (4 cores), there must be available free physical capacity. And this is not the case, because the other VM -the blue VM- is using its 4 cores of guaranteed capacity.

So the final capacity distribution in the above conditions is: the red VM is using 4 cores, and the blue VM is using 4 cores. Consequently the physical machine is 100% busy (all 8 cores used). You may conclude, from the sizer point of view, that the physical machine is undersized, not being able to meet all demands placed on it.

What would happen if the blue demand decreases from 5 to 1 core? In the forthcoming second part we will solve this and will play with more complex and subtle cases.

Demystifying Virtual Capacity, Part II

In the previous mini paper we defined the virtual capacity concept and identified the generic factors it depends on: physical machine (PM) capacity, guaranteed capacity, exclusive use attribute, limit / cap attribute and relative priority.

A very simple thought scenario was proposed: an 8-core PM with two virtual machines (VM), the red and the blue, parameterized with the following configuration:

VM	Guarant. Capacity	vCores	Exclusive Use	Limit / Cap	Priority
Red	4	8	No	No	128
Blue	4	8	No	No	128

If the red demand is 5 cores and the blue demand is 1 core, what would be the resulting capacity distribution?  

The blue will get 1 core, as this demand is well below its guaranteed 4 cores. The remaining 3 cores are not used and, as it has not the exclusive use attribute set, they are ceded back as free capacity.

The red usage increase to 5 cores, 4 of them coming from its guaranteed capacity and the additional 1 core coming from the available free capacity. Now the PM is 75% busy (6 of 8 cores) and there are no unsatisfied demands.

A general rule is that when all the VMs are uncapped and without exclusivity, and the sum of all demands are less than the PM capacity all VM demands can be satisfied.

Let’s consider a case with competence, that is, the PM capacity is not enough to satisfy the sum of VM demands (being all of them uncapped and with no exclusive use). How the scarce PM capacity is distributed?

Suppose the same 8-core PM with 3 VMs, the red, the blue and the green, with the following configuration:

VM	Guarant. Capacity	vCores	Exclusive Use	Limit / Cap	Priority
Red	3	8	No	No	128
Blue	3	8	No	No	128
Green	2	8	No	No	128

The demands are: red and blue 4 cores, and green 1 core. The sum of demands are 9 cores, more than the physical capacity (8 cores).

First group, VMs with demands less or equal than their guarantees. They are serviced and the rest of the guarantee is ceded and increase the free capacity. The green VM is in this group: it uses 1 core and cedes 1 core.

Second group, VMs with demands higher than their guarantees. They receive their guarantee plus the right proportion, according to their priorities, of the free capacity. The red and the blue are in this group, they both use 3 cores (the guaranteed) plus 0.5 cores, coming from the 1 core divided into two equal parts, as both VMs have the same priority and thus receive the same fraction. Is this situation, the PM is 100% busy and there are unsatisfied demands.

What would happen if the green VM is shutdown or its demand drops to zero? Or if the blue VM is limited/capped? Or if the red VM number of virtual cores is set to 2? Or if the green demand goes up to 4 cores?

Calculations to solve the generic case, if you know what to do and how to do them, are simple. I’ve created a spreadsheet implementing them. Simple, functional (but not fool proof): the virtual capacity demystifier. It comes with a companion presentation illustrating its usage. You should experiment with the tool to fully comprehend and understand virtual capacities.

A final word: perhaps I should have added ths following subtitle: “...in a perfect world”. In the real world there are second order effects -overheads, inefficiencies, cache misses,...- that diminish virtual capacities we have obtained. These effects belong to the realm of advanced practitioners and performance gurus, but you should be aware of them.