Skip to main content
Version: 3.4 (unsupported)

ScalarDB Benchmarks

This repository contains benchmark programs for ScalarDB.

Available workloads

  • TPC-C

Prerequisites

  • Java (OpenJDK 8 or higher)
  • Gradle
  • Kelpie

The benchmark uses Kelpie, which is a simple yet general framework for performing end-to-end testing such as system benchmarking and verification. Get the latest version of Kelpie from here and unzip the archive.

Usage

Set up an environment

This benchmark requires the followings:

  • A client to execute this benchmark
  • A target database (See here for supported databases)

Build

./gradlew shadowJar

Create tables

Before loading initial data, the tables must be defined using ScalarDB Schema Loader. Get the latest schema loader here and execute it with the workload-specific schema file. For setting ScalarDB properties, see also the related documents here.

For example, execute the following command with tpcc-schema.json to create tables for TPC-C benchmark.

java -jar scalardb-schema-loader-<version>.jar --config /path/to/scalardb.properties -f tpcc-schema.json --coordinator

Load and run

  1. Prepare a configuration file
    • A configuration file requires at least the locations of workload modules to run and the database configuration. The following example shows the case for running TPC-C benchmark. The database configuration should be matched with the benchmark environment set up above. You can use the ScalarDB property file instead of specifying each configuration item. If the config_file is specified, all other configuration items will be ignored.
      [modules]
      [modules.preprocessor]
      name = "com.scalar.db.benchmarks.tpcc.TpccLoader"
      path = "./build/libs/scalardb-benchmarks-all.jar"
      [modules.processor]
      name = "com.scalar.db.benchmarks.tpcc.TpccBench"
      path = "./build/libs/scalardb-benchmarks-all.jar"
      [modules.postprocessor]
      name = "com.scalar.db.benchmarks.tpcc.TpccReporter"
      path = "./build/libs/scalardb-benchmarks-all.jar"

      [database_config]
      contact_points = "localhost"
      contact_port = 9042
      username = "cassandra"
      password = "cassandra"
      storage = "cassandra"
      #config_file = "/path/to/scalardb.properties"
    • You can define static parameters to pass to modules in the file. For details, see example configuration files such as tpcc-benchmark-config.toml and available parameters in the following section.
  2. Run a benchmark
    ${kelpie}/bin/kelpie --config your_config.toml
    • ${kelpie} is a Kelpie directory, which is extracted from the archive you downloaded above.
    • There are other options such as --only-pre (i.e., loading data) and --only-process (i.e., running benchmark), which run only the specified process. --except-pre and --except-process run a job without the specified process.

Common parameters

namedescriptiondefault
concurrencyNumber of threads for benchmarking.1
run_for_secDuration of benchmark (in seconds).60
ramp_for_secDuration of ramp up time before benchmark (in seconds).0

Workload-specific parameters

TPC-C

namedescriptiondefault
num_warehousesNumber of warehouses (scale factor) for benchmarking.1
load_concurrencyNumber of threads for loading.1
load_start_warehouseStart ID of loading warehouse. This option can be useful with --skip-item-load when loading large scale data with multiple clients or adding additional warehouses.1
load_end_warehouseEnd ID of loading warehouse. You can use either --num-warehouses or --end-warehouse to specify the number of loading warehouses.1
skip_item_loadWhether or not to skip loading item tablefalse
use_table_indexWhether or not to use a generic table-based secondary index instead of ScalarDB's secondary index.false
np_onlyRun benchmark with only new-order and payment transactions (50% each).false
rate_new_orderPercentage of new-order transaction. Specify all the rate parameters to use your own mix.N/A
rate_paymentPercentage of payment transaction. Specify all the rate parameters to use your own mix.N/A
rate_order_statusPercentage of order-status transaction. Specify all the rate parameters to use your own mix.N/A
rate_deliveryPercentage of delivery transaction. Specify all the rate parameters to use your own mix.N/A
rate_stock_levelPercentage of stock-level transaction. Specify all the rate parameters to use your own mix.N/A
backoffSleep time in milliseconds inserted after a transaction is aborted due to a conflict.0