25 / 54
W H I T E P A P E R
www.persistent.com
© 2017 Persistent Systems Ltd. All rights reserved.
8 Appendix 1 – Performance and scalability
Performance
Performance is a complex, derived requirement which is a function of several query and data requirements (query
response times, query types, user scales and data volumes). It generally refers to sustained query response time
during a time interval, on a well-specified workload under control where type of query mix, user scales and data
volumes are precisely defined.
An example of a way to measure performance is the Composite Query per Hour, or QphH@size, as defined by the
TPC-H performance benchmark
[16] ,a well-defined analytic workload. The number of concurrent query streams,
a complexity mix of queries including sequential scans, aggregation, multi-table joins and sorting, the database
volume size and the query response times are all part of the computation of QphH@size, which boils down to the
number of completed queries per hour at a database scale factor (volume). Even though our performance scale
is applied informally throughout this document, we should define it precisely. On a 10 TB TPC-H database, using
the TPC-H analytic workload, our performance level definitions are:
•
Very High
performance refers to a QphH@size > 10 million. Only an MPP database is capable of this
result today on the TPC-H.
•
High
performance refers to a QphH@size between 1 and 10 million. This is one order of magnitude
below. This is within the reach of SMP databases: Microsoft SQL Server is capable of high performance
on this benchmark, with a QphH@size result above 1.1 million.
•
Medium
performance refers to a QphH@size between 100,000 and 1 million. This is yet an order of
magnitude below.
•
Average
performance and below will refer to a QphH@size below 100,000. This corresponds to
two orders of magnitude below what we call high performance, which corresponds roughly with the
performance associated with Hadoop systems today.
Scalability
Scalability is a critical requirement to overcome performance limits (response time and/or throughput) by adding
computing resources. Some benchmarks include the notion of resources needed to get performance: TPC-H,
for instance, defines a price-performance metric ($/QphH/@Size) where $ is the total system price
19 .This made
sense in on-premise environments hosting analytics solutions at the time TPC-H was defined. In the cloud
computing realm, cost, although relevant, no longer is the most important issue: design of a scalable design has
become most desirable and challenging system property, to ensure that the system capacity can be augmented
by adding additional infrastructure resources, to either support large end user scales and/or to respond to sudden
load fluctuations on a cloud-based service due to demand surges or troughs from the end users. Three notions
of scalability are commonly being talked about in this context:
scale-up, scale out and elasticity
.
Scaling up, or vertical scaling
, generally refers to adding resources to a single node in a system, typically involving
the addition of processors and/or memory to a single computer. Such vertical scaling of existing systems also
enables them to use virtualization technology more effectively, as it provides more resources for the hosted set
of operating system and application modules to share.
Scaling out
, or horizontal scaling, means adding and linking together other lower-performance machines to
collectively do the work of a much more advanced one. With these types of distributed setups, it’s easy to handle
a larger workload by running data through different system trajectories.
19
TPC-H specifies how to price resources and how to express the total system price.
25