文档HOT
中文
注册
我要评分
文档获取效率
文档正确性
内容完整性
文档易理解
提交
在线提单
论坛求助

创建工作负载

COSBench测试模型

对象存储网关RGW部署在1台存储节点上，每台存储节点启动4个不同端口号的RGW网关实例。客户端如安装测试工具所述，COSBench Controller节点分别向存储节点上的RGW网关发送读写负载测试命令。

COSBench测试流程

COSBench对象存储测试共分为3个阶段，分别为准备阶段、读写测试阶段、清理阶段。其中准备阶段会创建对象存储Bucket并向Bucket中初始化测试数据；读写测试阶段会对Bucket中的数据进行读取、写入和混合读写测试；清理阶段会清空Bucket中的对象并删除Bucket。

创建负载配置文件

COSBench的工作负载定义在XML文件中，文件的结构如图1所示。

图1 COSBench负载文件结构

下面以256KB为例展示各个测试流程的负载配置文件：

  • 256k_prepare.xml 
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
    		 
    <?xml version='1.0' encoding='utf-8'?> 
<workload config="" description="create s3 bucket" name="create-bucket"> 
    <auth config="" type="none" /> 
    <workflow config=""> 
        <workstage closuredelay="0" config="" name="createbucket"> 
            <auth config="" type="none" /> 
            <work afr="0" config="containers=r(1,24)" division="container" interval="5" name="rgw1-create" rampdown="0" rampup="0" runtime="0" totalBytes="0" totalOps="1" type="init" workers="5"> 
                <auth config="" type="none" /> 
                <storage config="accesskey=test1;secretkey=test1;endpoint=http://192.168.3.135:10001;path_style_access=true" type="s3" /> 
                <operation config="containers=r(1,6);objects=r(0,0);sizes=c(0)B" division="container" id="none" ratio="100" type="init" /> 
            </work> 
        <work afr="0" config="containers=r(1,24)" division="container" interval="5" name="rgw2-create" rampdown="0" rampup="0" runtime="0" totalBytes="0" totalOps="1" type="init" workers="5"> 
                <auth config="" type="none" /> 
                <storage config="accesskey=test1;secretkey=test1;endpoint=http://192.168.3.135:10002;path_style_access=true" type="s3" /> 
                <operation config="containers=r(7,12);objects=r(0,0);sizes=c(0)B" division="container" id="none" ratio="100" type="init" /> 
            </work> 
        <work afr="0" config="containers=r(1,24)" division="container" interval="5" name="rgw3-create" rampdown="0" rampup="0" runtime="0" totalBytes="0" totalOps="1" type="init" workers="5"> 
                <auth config="" type="none" /> 
                <storage config="accesskey=test1;secretkey=test1;endpoint=http://192.168.3.135:10003;path_style_access=true" type="s3" /> 
                <operation config="containers=r(13,18);objects=r(0,0);sizes=c(0)B" division="container" id="none" ratio="100" type="init" /> 
            </work> 
        <work afr="0" config="containers=r(1,24)" division="container" interval="5" name="rgw4-create" rampdown="0" rampup="0" runtime="0" totalBytes="0" totalOps="1" type="init" workers="5"> 
                <auth config="" type="none" /> 
                <storage config="accesskey=test1;secretkey=test1;endpoint=http://192.168.3.135:10004;path_style_access=true" type="s3" /> 
                <operation config="containers=r(19,24);objects=r(0,0);sizes=c(0)B" division="container" id="none" ratio="100" type="init" /> 
            </work> 
        </workstage> 
 
        <workstage closuredelay="0" config="" name="prepareobject"> 
            <auth config="" type="none" /> 
            <work afr="200000" config="containers=r(1,24);objects=r(1,2000);sizes=c(256)KB" division="container" interval="5" name="prepare1" runtime="120" type="prepare" workers="5"> 
                <auth config="" type="none" /> 
                <storage config="accesskey=test1;secretkey=test1;endpoint=http://192.168.3.135:10001;path_style_access=true" type="s3" /> 
                <operation config="containers=r(1,6);objects=r(1,2000);sizes=c(256)KB" division="container" id="none" ratio="100" type="prepare" /> 
            </work> 
        <work afr="200000" config="containers=r(1,24);objects=r(1,2000);sizes=c(256)KB" division="container" interval="5" name="prepare2" runtime="120" type="prepare" workers="5"> 
                <auth config="" type="none" /> 
                <storage config="accesskey=test1;secretkey=test1;endpoint=http://192.168.3.135:10002;path_style_access=true" type="s3" /> 
                <operation config="containers=r(7,12);objects=r(1,2000);sizes=c(256)KB" division="container" id="none" ratio="100" type="prepare" /> 
            </work> 
        <work afr="200000" config="containers=r(1,24);objects=r(1,2000);sizes=c(256)KB" division="container" interval="5" name="prepare3" runtime="120" type="prepare" workers="5"> 
                <auth config="" type="none" /> 
                <storage config="accesskey=test1;secretkey=test1;endpoint=http://192.168.3.135:10003;path_style_access=true" type="s3" /> 
                <operation config="containers=r(13,18);objects=r(1,2000);sizes=c(256)KB" division="container" id="none" ratio="100" type="prepare" /> 
            </work> 
        <work afr="200000" config="containers=r(1,24);objects=r(1,2000);sizes=c(256)KB" division="container" interval="5" name="prepare4" runtime="120" type="prepare" workers="5"> 
                <auth config="" type="none" /> 
                <storage config="accesskey=test1;secretkey=test1;endpoint=http://192.168.3.135:10004;path_style_access=true" type="s3" /> 
                <operation config="containers=r(19,24);objects=r(1,2000);sizes=c(256)KB" division="container" id="none" ratio="100" type="prepare" /> 
            </work> 
        </workstage> 
    </workflow> 
</workload>
  • 256k_put.xml 
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
    		 
    <?xml version='1.0' encoding='utf-8'?> 
<workload config="" description="create s3 bucket" name="create-bucket"> 
    <auth config="" type="none" /> 
    <workflow config=""> 
 
        <workstage closuredelay="0" config="" name="putobj"> 
            <auth config="" type="none" /> 
            <work afr="200000" config="containers=r(1,24);objects=r(1,2000)" division="container" interval="5" name="rgw1-put" runtime="300" type="write" workers="5"> 
                <auth config="" type="none" /> 
                <storage config="accesskey=test1;secretkey=test1;endpoint=http://192.168.3.135:10001;path_style_access=true" type="s3" /> 
                <operation config="containers=r(1,6);objects=r(1,2000);sizes=c(256)KB" division="container" id="none" ratio="100" type="write" /> 
            </work> 
        <work afr="200000" config="containers=r(1,24);objects=r(1,2000)" division="container" interval="5" name="rgw2-put" runtime="300" type="write" workers="5"> 
                <auth config="" type="none" /> 
                <storage config="accesskey=test1;secretkey=test1;endpoint=http://192.168.3.135:10002;path_style_access=true" type="s3" /> 
                <operation config="containers=r(7,12);objects=r(1,2000);sizes=c(256)KB" division="container" id="none" ratio="100" type="write" /> 
            </work> 
        <work afr="200000" config="containers=r(1,24);objects=r(1,2000)" division="container" interval="5" name="rgw3-put" runtime="300" type="write" workers="5"> 
                <auth config="" type="none" /> 
                <storage config="accesskey=test1;secretkey=test1;endpoint=http://192.168.3.135:10003;path_style_access=true" type="s3" /> 
                <operation config="containers=r(13,18);objects=r(1,2000);sizes=c(256)KB" division="container" id="none" ratio="100" type="write" /> 
            </work> 
        <work afr="200000" config="containers=r(1,24);objects=r(1,2000)" division="container" interval="5" name="rgw4-put" runtime="300" type="write" workers="5"> 
                <auth config="" type="none" /> 
                <storage config="accesskey=test1;secretkey=test1;endpoint=http://192.168.3.135:10004;path_style_access=true" type="s3" /> 
                <operation config="containers=r(19,24);objects=r(1,2000);sizes=c(256)KB" division="container" id="none" ratio="100" type="write" /> 
            </work> 
        </workstage> 
 
    </workflow> 
</workload>
  • 256k_get.xml 
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
    		 
    <?xml version='1.0' encoding='utf-8'?> 
<workload config="" description="create s3 bucket" name="create-bucket"> 
    <auth config="" type="none" /> 
    <workflow config=""> 
 
        <workstage closuredelay="0" config="" name="getobj"> 
            <auth config="" type="none" /> 
            <work afr="200000" config="containers=r(1,24);objects=r(1,2000)" division="container" interval="5" name="rgw1-get" runtime="60" type="read" workers="5"> 
                <auth config="" type="none" /> 
                <storage config="accesskey=test1;secretkey=test1;endpoint=http://192.168.3.135:10001;path_style_access=true" type="s3" /> 
                <operation config="containers=r(1,6);objects=r(1,2000)" division="container" id="none" ratio="100" type="read" /> 
            </work> 
        <work afr="200000" config="containers=r(1,24);objects=r(1,2000)" division="container" interval="5" name="rgw2-get" runtime="60" type="read" workers="5"> 
                <auth config="" type="none" /> 
                <storage config="accesskey=test1;secretkey=test1;endpoint=http://192.168.3.135:10002;path_style_access=true" type="s3" /> 
                <operation config="containers=r(7,12);objects=r(1,2000)" division="container" id="none" ratio="100" type="read" /> 
            </work> 
        <work afr="200000" config="containers=r(1,24);objects=r(1,2000)" division="container" interval="5" name="rgw3-get" runtime="60" type="read" workers="5"> 
                <auth config="" type="none" /> 
                <storage config="accesskey=test1;secretkey=test1;endpoint=http://192.168.3.135:10003;path_style_access=true" type="s3" /> 
                <operation config="containers=r(13,18);objects=r(1,2000)" division="container" id="none" ratio="100" type="read" /> 
            </work> 
        <work afr="200000" config="containers=r(1,24);objects=r(1,2000)" division="container" interval="5" name="rgw4-get" runtime="60" type="read" workers="5"> 
                <auth config="" type="none" /> 
                <storage config="accesskey=test1;secretkey=test1;endpoint=http://192.168.3.135:10004;path_style_access=true" type="s3" /> 
                <operation config="containers=r(19,24);objects=r(1,2000)" division="container" id="none" ratio="100" type="read" /> 
            </work> 
        </workstage> 
 
    </workflow> 
</workload>
  • 256k_putget.xml 
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
    		 
    <?xml version='1.0' encoding='utf-8'?> 
<workload config="" description="create s3 bucket" name="create-bucket"> 
    <auth config="" type="none" /> 
    <workflow config=""> 
 
        <workstage closuredelay="0" config="" name="rwobj"> 
            <auth config="" type="none" /> 
            <work afr="200000" config="containers=r(1,24);objects=r(1,2000)" division="container" interval="5" name="rgw1-rw" runtime="120" type="write" workers="5"> 
                <auth config="" type="none" /> 
                <storage config="accesskey=test1;secretkey=test1;endpoint=http://192.168.3.135:10001;path_style_access=true" type="s3" /> 
                <operation config="containers=r(1,6);objects=r(1,1000);sizes=c(256)KB" division="container" id="none" ratio="30" type="write" /> 
                <operation config="containers=r(1,6);objects=r(1001,2000)" division="container" id="none" ratio="70" type="read" /> 
            </work> 
        <work afr="200000" config="containers=r(1,24);objects=r(1,2000)" division="container" interval="5" name="rgw2-rw" runtime="120" type="write" workers="5"> 
                <auth config="" type="none" /> 
                <storage config="accesskey=test1;secretkey=test1;endpoint=http://192.168.3.135:10002;path_style_access=true" type="s3" /> 
                <operation config="containers=r(7,12);objects=r(1,1000);sizes=c(256)KB" division="container" id="none" ratio="30" type="write" /> 
                <operation config="containers=r(7,12);objects=r(1001,2000)" division="container" id="none" ratio="70" type="read" /> 
            </work> 
        <work afr="200000" config="containers=r(1,24);objects=r(1,2000)" division="container" interval="5" name="rgw3-rw" runtime="120" type="write" workers="5"> 
                <auth config="" type="none" /> 
                <storage config="accesskey=test1;secretkey=test1;endpoint=http://192.168.3.135:10003;path_style_access=true" type="s3" /> 
                <operation config="containers=r(13,18);objects=r(1,1000);sizes=c(256)KB" division="container" id="none" ratio="30" type="write" /> 
                <operation config="containers=r(13,18);objects=r(1001,2000)" division="container" id="none" ratio="70" type="read" /> 
            </work> 
        <work afr="200000" config="containers=r(1,24);objects=r(1,2000)" division="container" interval="5" name="rgw4-rw" runtime="120" type="write" workers="5"> 
                <auth config="" type="none" /> 
                <storage config="accesskey=test1;secretkey=test1;endpoint=http://192.168.3.135:10004;path_style_access=true" type="s3" /> 
                <operation config="containers=r(19,24);objects=r(1,1000);sizes=c(256)KB" division="container" id="none" ratio="30" type="write" /> 
                <operation config="containers=r(19,24);objects=r(1001,2000)" division="container" id="none" ratio="70" type="read" /> 
            </work> 
        </workstage> 
 
    </workflow> 
</workload>
  • 256k_clean.xml 
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
    		 
    <?xml version='1.0' encoding='utf-8'?> 
<workload config="" description="create s3 bucket" name="create-bucket"> 
    <auth config="" type="none" /> 
    <workflow config=""> 
 
                <workstage closuredelay="0" config="" name="cleanup"> 
            <auth config="" type="none" /> 
            <work afr="0" config="containers=r(1,24);objects=r(1,2000);" division="object" interval="5" name="rgw1-cleanup" runtime="0" type="cleanup" workers="5"> 
                <auth config="" type="none" /> 
                <storage config="accesskey=test1;secretkey=test1;endpoint=http://192.168.3.135:10001;path_style_access=true" type="s3" /> 
                <operation config="containers=r(1,6);objects=r(1,2000);deleteContainer=false;" division="object" id="none" ratio="100" type="cleanup" /> 
            </work> 
        <work afr="0" config="containers=r(1,24);objects=r(1,2000);" division="object" interval="5" name="rgw2-cleanup" runtime="0" type="cleanup" workers="5"> 
                <auth config="" type="none" /> 
                <storage config="accesskey=test1;secretkey=test1;endpoint=http://192.168.3.135:10002;path_style_access=true" type="s3" /> 
                <operation config="containers=r(7,12);objects=r(1,2000);deleteContainer=false;" division="object" id="none" ratio="100" type="cleanup" /> 
            </work> 
        <work afr="0" config="containers=r(1,24);objects=r(1,2000);" division="object" interval="5" name="rgw3-cleanup" runtime="0" type="cleanup" workers="5"> 
                <auth config="" type="none" /> 
                <storage config="accesskey=test1;secretkey=test1;endpoint=http://192.168.3.135:10003;path_style_access=true" type="s3" /> 
                <operation config="containers=r(13,18);objects=r(1,2000);deleteContainer=false;" division="object" id="none" ratio="100" type="cleanup" /> 
            </work> 
        <work afr="0" config="containers=r(1,24);objects=r(1,2000);" division="object" interval="5" name="rgw4-cleanup" runtime="0" type="cleanup" workers="5"> 
                <auth config="" type="none" /> 
                <storage config="accesskey=test1;secretkey=test1;endpoint=http://192.168.3.135:10004;path_style_access=true" type="s3" /> 
                <operation config="containers=r(19,24);objects=r(1,2000);deleteContainer=false;" division="object" id="none" ratio="100" type="cleanup" /> 
            </work> 
        </workstage> 
 
                <workstage closuredelay="0" config="" name="dispose"> 
            <auth config="" type="none" /> 
            <work afr="0" config="containers=r(1,24);" division="container" interval="5" name="rgw1-dispose" runtime="0" type="dispose" workers="5"> 
                <auth config="" type="none" /> 
                <storage config="accesskey=test1;secretkey=test1;endpoint=http://192.168.3.135:10004;path_style_access=true" type="s3" /> 
                <operation config="containers=r(1,6);objects=r(0,0);sizes=c(0)B;" division="container" id="none" ratio="100" type="dispose" /> 
            </work> 
        <work afr="0" config="containers=r(1,24);" division="container" interval="5" name="rgw2-dispose" runtime="0" type="dispose" workers="5"> 
                <auth config="" type="none" /> 
                <storage config="accesskey=test1;secretkey=test1;endpoint=http://192.168.3.135:10001;path_style_access=true" type="s3" /> 
                <operation config="containers=r(7,12);objects=r(0,0);sizes=c(0)B;" division="container" id="none" ratio="100" type="dispose" /> 
            </work> 
        <work afr="0" config="containers=r(1,24);" division="container" interval="5" name="rgw3-dispose" runtime="0" type="dispose" workers="5"> 
                <auth config="" type="none" /> 
                <storage config="accesskey=test1;secretkey=test1;endpoint=http://192.168.3.135:10002;path_style_access=true" type="s3" /> 
                <operation config="containers=r(13,18);objects=r(0,0);sizes=c(0)B;" division="container" id="none" ratio="100" type="dispose" /> 
            </work> 
        <work afr="0" config="containers=r(1,24);" division="container" interval="5" name="rgw4-dispose" runtime="0" type="dispose" workers="5"> 
                <auth config="" type="none" /> 
                <storage config="accesskey=test1;secretkey=test1;endpoint=http://192.168.3.135:10003;path_style_access=true" type="s3" /> 
                <operation config="containers=r(19,24);objects=r(0,0);sizes=c(0)B;" division="container" id="none" ratio="100" type="dispose" /> 
            </work> 
        </workstage> 
 
    </workflow> 
</workload>
  1. 上述配置文件中：
    • containers表示创建的Bucket编号。
    • objects表示每个Bucket中的对象编号。
    • accesskey和secretkey是创建对象存储S3接口用户时创建的，可用radosgw-admin user info --uid=<S3_user_ID>查询。
    • sizes表示每次读取/写入的数据块大小。
    • endpoint表示对象网关的IP地址及端口号，上述配置文件中以IP地址192.168.3.135，端口号10001~10004为例进行说明。
  2. 上述配置文件是对应1台存储节点上4个不同端口的对象网关RGW实例的负载配置。修改endpoint中的IP及端口号即可生成不同RGW网关的负载配置文件。此外，将sizes=c(256)KB分别改为sizes=c(1)MB和sizes=c(4)MB即可生成1MB和4MB数据块大小的负载配置文件。
父主题： 测试步骤