YCSBストレステストMongoDB

Ycsb Stress Test Mongodb

動作環境：CentOS7.5 + YCSB0.16バージョン+ MongoDB 3.6.6

Note: Refer to the release version of YCSB 0.15.0 for instructions. Tested in previous releases, unchanged in this release version supported by MongoDB It is 3.6, but it is not supported before the 3.6 version. 1. Install the JDK: Use RPM directly for ease of use: #rpm -ivh jdk-8u172-linux-x64.rpm # java -version java version '1.8.0_172' Java(TM) SE Runtime Environment (build 1.8.0_172-b11) Java HotSpot(TM) 64-Bit Server VM (build 25.172-b11, mixed mode) 2.maven #wget http://mirrors.tuna.tsinghua.edu.cn/apache/maven/maven-3/3.5.4/binaries/apache-maven-3.5.4-bin.tar.gz #tar -xzf apache-maven-3.5.4-bin.tar.gz -C /usr/local/apache-maven-3.5.4-bin #mv /usr/local/apache-maven-3.5.4-bin /usr/local/maven #vim /etc/profile.d/mvn.sh #/bin/bash export M2_HOME=/usr/local/maven export MAVEN_OPTS='-Xms256m -Xmx512m' export PATH=$M2_HOME/bin:$PATH #source /etc/profile.d/mvn.sh --Verification: # mvn --version Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe 2018-06-18T02:33:14+08:00) Maven home: /usr/local/maven Java version: 1.8.0_172, vendor: Oracle Corporation, runtime: /usr/local/jdk/jre Default locale: en_US, platform encoding: UTF-8 OS name: 'linux', version: '3.10.0-327.el7.x86_64', arch: 'amd64', family: 'unix' 3.YCSB： #git clone https://github.com/brianfrankcooper/YCSB.git # cd YCSB --Create all: # mvn clean package -- Compile only for one package: mvn -pl com.yahoo.ycsb:mongodb-binding -am clean package

コンパイル情報：

[INFO] Scanning for projects... [INFO] ------------------------------------------------------------------------ [INFO] Reactor Build Order: [INFO] [INFO] YCSB Root [pom] [INFO] Core YCSB [jar] [INFO] Per Datastore Binding descriptor [jar] [INFO] YCSB Datastore Binding Parent [pom] [INFO] Accumulo 1.6 DB Binding [jar] [INFO] Accumulo 1.7 DB Binding [jar] [INFO] Accumulo 1.8 DB Binding [jar] [INFO] Aerospike DB Binding [jar] [INFO] ArangoDB Binding [jar] [INFO] AsyncHBase Client Binding for Apache HBase [jar] [INFO] Cassandra 2.1+ DB Binding [jar] [INFO] Cloud Spanner DB Binding [jar] [INFO] Couchbase Binding [jar] [INFO] Couchbase Java SDK 2.x Binding [jar] [INFO] Azure DocumentDB Binding [jar] [INFO] Azure table storage Binding [jar] [INFO] DynamoDB DB Binding [jar] [INFO] Elasticsearch Binding [jar] [INFO] Elasticsearch 5.x Binding [jar] [INFO] FoundationDB Binding [jar] [INFO] Geode DB Binding [jar] [INFO] Google Cloud Datastore Binding [jar] [INFO] Google Cloud Bigtable Binding [jar] [INFO] HBase 0.98.x DB Binding [jar] [INFO] HBase 1.0 DB Binding [jar] [INFO] HBase 1.2 DB Binding [jar] [INFO] HBase 1.4 DB Binding [jar] [INFO] HBase 2.0 DB Binding [jar] [INFO] Hypertable DB Binding [jar] [INFO] Apache Ignite [jar] [INFO] Infinispan DB Binding [jar] [INFO] JDBC DB Binding [jar] [INFO] Kudu DB Binding [jar] [INFO] memcached binding [jar] [INFO] MapRDB Binding [jar] [INFO] MapR JSON DB Binding [jar] [INFO] MongoDB Binding [jar] [INFO] Oracle NoSQL Database Binding [jar] [INFO] OrientDB Binding [jar] [INFO] rados of Ceph FS binding [jar] [INFO] Redis DB Binding [jar] [INFO] Rest Client Binding [jar] [INFO] Riak KV Binding [jar] [INFO] RocksDB Java Binding [jar] [INFO] S3 Storage Binding [jar] [INFO] Solr Binding [jar] [INFO] Solr 6 Binding [jar] [INFO] Tarantool DB Binding [jar] [INFO] YCSB Release Distribution Builder [pom] [INFO] [INFO] ------------------------------------------------- [INFO] Building YCSB Root 0.16.0-SNAPSHOT [1/49] [INFO] --------------------------------[ pom ]--------------------------------- [INFO] ...

単一のパッケージのみをコンパイルします。

# mvn -pl com.yahoo.ycsb:mongodb-binding -am clean package [INFO] Building tar: /data/soft/YCSB/mongodb/target/ycsb-mongodb-binding-0.16.0-SNAPSHOT.tar.gz [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] YCSB Root 0.16.0-SNAPSHOT .......................... SUCCESS [04:49 min] [INFO] Core YCSB .......................................... SUCCESS [03:05 min] [INFO] Per Datastore Binding descriptor ................... SUCCESS [ 0.192 s] [INFO] YCSB Datastore Binding Parent ...................... SUCCESS [01:21 min] [INFO] MongoDB Binding 0.16.0-SNAPSHOT .................... SUCCESS [03:17 min] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 12:34 min [INFO] Finished at: 2018-08-25T20:37:34+08:00 [INFO] ------------------------------------------------------------------------

YCSBが提供するテストモード：

File name Test scenario Read update insertion ratio Application example workloada Update heavy workload 50/50 Session store recording recent actions workloadb Read mostly workload 95/5 photo tagging add a tag is an update, but most operations are to read tags workloadc Read only 100/0 user profile cache, where profiles are constructed elsewhere (e.g., Hadoop) workloadd Read latest workload 95/0/5 user status updates people want to read the latest workloade Short ranges 95/5 threaded conversations, where each scan is for the posts in a given thread (assumed to be clustered by thread id) workloadf Read-modify-write workload 50/50 user database, where user records are read and modified by the user or to record user activity. Note: Other default parameters: Default data size: 1 KB records (10 fields, 100 bytes each, plus key) Request distribution: zipfian

YCSBが提供する構成パラメーターの説明：

Parameter Description Fieldcount single record field number, default 10 Fieldlength The size of each field, default 100B Readallfields whether to read all records, the default is true Readproportion read ratio (default 0.95) Updateproportion update ratio (default 0.05) Insertproportion insertion ratio (default 0) Scanproportion scan ratio (default 0) Readmodifywriteproportion read, modify, writeback ratio of the same record (default: 0) Requestdistribution record selection policy: uniform, zipfian, latest (default: uniform) Maxscanlength maximum number of scan records (default: 100) Scanlengthdistribution scan record selection policy (default: uniform) Insertorder record insertion strategy: ordered, hashed (default: hased) Operationcount operation execution number Maxexecutiontime maximum execution time, in seconds Table table name (default value usertable) Recordcount Initializes the number of records (default: 0)

MongDB構成パラメーター：

Parameter Description Mongodb.url MongoDB URI or connection string Mongodb.batchsize When a large number of insert operations, batch commit submission, improve throughput Mongodb.upsert defaults to false.true means update operation, false means insert operation. The default value of mongodb.writeConcern is acknowledged. Optional value: - `errors_ignored` - `unacknowledged` - `acknowledged` - `journaled` - `replica_acknowledged` - `majority` mongodb.readPreference default value is primary Optional value: - `primary` - `primary_preferred` - `secondary` - `secondary_preferred` - `nearest` The default value of mongodb.maxconnections is 100. The value of maxPoolSize in the mongodb.url selection. mongodb.threadsAllowedToBlockForConnectionMultiplier The default value is 5. The value of waitQueueMultiple in the mongodb.url selection.

YCSB実行可能ファイルのヘルプ情報：

# cd /data/soft/YCSB/mongodb/target/ycsb-mongodb-binding-0.16.0-SNAPSHOT # ./bin/ycsb --help usage: ./bin/ycsb command database [options] Commands: load Execute the load phase run Execute the transaction phase shell Interactive mode Databases: accumulo https://github.com/brianfrankcooper/YCSB/tree/master/accumulo accumulo1.6 https://github.com/brianfrankcooper/YCSB/tree/master/accumulo1.6 ... Options: -P file Specify workload file -cp path Additional Java classpath entries -jvm-args args Additional arguments to the JVM -p key=value Override workload property -s Print status to stderr -target n Target ops/sec (default: unthrottled) -threads n Number of client threads (default: 1) Workload Files: There are various predefined workloads under workloads/ directory. See https://github.com/brianfrankcooper/YCSB/wiki/Core-Properties for the list of workload properties. positional arguments: {load,run,shell} Command to run. {accumulo,accumulo1.6,accumulo1.7,accumulo1.8,aerospike,arangodb,arangodb3,asynchbase,azuredocumentdb,azuretablestorage,basic,basicts,cassandra-cql,cassandra2-cql,cloudspanner,couchbase,couchbase2,dynamodb,elasticsearch,elasticsearch5,foundationdb,geode,googlebigtable,googledatastore,hbase098,hbase10,hbase12,hbase14,hbase20,hypertable,ignite,ignite-sql,infinispan,infinispan-cs,jdbc,kudu,maprdb,maprjsondb,memcached,mongodb,mongodb-async,nosqldb,orientdb,rados,redis,rest,riak,rocksdb,s3,solr,solr6,tarantool} Database to test. optional arguments: -h, --help show this help message and exit -cp CLASSPATH Additional classpath entries, e.g. '-cp /tmp/hbase-1.0.1.1/conf'. Will be prepended to the YCSB classpath. -jvm-args JVM_ARGS Additional arguments to pass to 'java', e.g. '-Xmx4g'

YCSBワークロードの構成と使用：

方法1：圧力測定パラメータとmonogBDの構成情報を一緒に書き込みます。

Description of the scene: The number of records is 1000, the number of operations is 1000, and the read/write scene is read and updated. Each record is 2k (250bytes * 8 columns), the number of concurrent is 100, write concern is acknowledged # cat workloads/workloada | grep -v ^# recordcount=1000 operationcount=1000 workload=com.yahoo.ycsb.workloads.CoreWorkload readallfields=true readproportion=0.5 updateproportion=0.5 scanproportion=0 insertproportion=0 fieldlength=250 fieldcount=8 requestdistribution=zipfian maxexecutiontime=180 threadcount=100 mongodb.url=mongodb://192.168.0.151:27017/ycsb mongodb.database=ycsb mongodb.writeConcern=replica_acknowledged --Instructions: #./bin/ycsb --load mongodb -s -P workloads/workloada

方法2：圧力測定パラメーターをmongoDBの構成情報から分離します。 mongoDBの構成情報は、ycsbの後に書き込まれます。

# cat workloads/workloadb | grep -v ^# recordcount=1000 operationcount=1000 workload=com.yahoo.ycsb.workloads.CoreWorkload readallfields=true readproportion=0.95 updateproportion=0.05 scanproportion=0 insertproportion=0 --Instructions: #./bin/ycsb --load mongodb -s -P workloads/workloadb -p mongodb.url=mongodb://user:root@xxxxx:27017/ycsb

MongoDBの圧力測定モード：

Asynchronous mode: #./bin/ycsb.sh load mongodb-async -s -P workloads/workloada Sync mode: (default) #./bin/ycsb.sh load mongodb -s -P workloads/workloada

MongoDBのストレステストの手順：

--The steps of the stress test: 1. Start mongoDB, build a test database, the library name is ycsb, the user and password are both ycsb 2. Write a workload file: 3. Load data: load 3. Run the pressure test: run 4. Use mongostat and iostat together during pressure measurement mongostat -uycsb -pycsb --authenticationDatabase=ycsb -h192.168.0.151 iostat -xmt 2 It is recommended to use the monitoring of graphical interfaces such as pmm. 5. Under different stress tests. 6. Issue a test report.