ZFS SAN/NAS - 備援機再造 - Infiniband 卡 + NVMe SSD (軟體RAID 比硬體 RAID 快喲)

最近又到特價季節, 所以又是更新伺服器的好時機了

我有一台古董的 SuperServer 6025W-URB, 在IDC機房內當備援使用, 最近為了架設 Openstack Liberty, 又讓它加入 Cinder Storage Nodes

未升級前的照片, 只有 16gb RAM, 以及還無加裝 Infiniband Card 以及


目前已經把記憶體 (二手, 總共 NT$9000) 以及 Infiniband Card (新品 NT$9000) 安裝進去了

CPU info

root@nas:~# lshw -class processor
*-cpu:0
description: CPU
product: Intel(R) Xeon(R) CPU E5472 @ 3.00GHz
vendor: Intel Corp.
physical id: 4
bus info: cpu@0
version: Intel(R) Xeon(R) CPU E5472 @ 3.00GHz
slot: LGA771/CPU1
size: 3020MHz
width: 64 bits
clock: 1600MHz
configuration: cores=4 enabledcores=4 threads=4
*-cpu:1
description: CPU
product: Intel(R) Xeon(R) CPU E5472 @ 3.00GHz
vendor: Intel Corp.
physical id: 5
bus info: cpu@1
version: Intel(R) Xeon(R) CPU E5472 @ 3.00GHz
slot: LGA771/CPU2
size: 3020MHz
width: 64 bits
clock: 1600MHz
configuration: cores=4 enabledcores=4 threads=4

記憶體 64GB

root@nas:~# free -m
total used free shared buffers cached
Mem: 64495 29598 34897 182 274 978
-/+ buffers/cache: 28345 36150
Swap: 16379 0 16379
root@nas:~# dmidecode --type 17
# dmidecode 2.12
SMBIOS 2.5 present.

Handle 0x0017, DMI type 17, 27 bytes
Memory Device
Array Handle: 0x0016
Error Information Handle: No Error
Total Width: 72 bits
Data Width: 64 bits
Size: 8192 MB
Form Factor: DIMM
Set: 1
Locator: DIMM1A
Bank Locator: Bank1
Type: DDR2 FB-DIMM
Type Detail: Synchronous
Speed: 667 MHz
Manufacturer: CE80
Serial Number: 45788D3F
Asset Tag: Not Specified
Part Number: M395T1K66AZ4-CE66

目前的 Network (已經增加了 Mellanox 56 gbps 的 Infiniband Card)

root@nas:~# lspci |grep Network
0b:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]
0c:00.0 Ethernet controller: Intel Corporation 82575EB Gigabit Network Connection (rev 02)
0c:00.1 Ethernet controller: Intel Corporation 82575EB Gigabit Network Connection (rev 02)

目前 ZFS Pool 的狀況, 無SSD 加速, WD 黑標 7200rpm 4TB SATA 6.0Gb/s 連接到 SATA 3.0Gb/s 的 Controller

root@nas:~# zpool status
pool: vmdisk
state: ONLINE
scan: scrub repaired 0 in 6h29m with 0 errors on Sun Nov 22 06:47:39 2015
config:

NAME STATE READ WRITE CKSUM
vmdisk ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
sdb ONLINE 0 0 0
sdc ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
sdd ONLINE 0 0 0
sde ONLINE 0 0 0
mirror-2 ONLINE 0 0 0
sdf ONLINE 0 0 0
sdg ONLINE 0 0 0

errors: No known data errors
root@nas:~# hdparm -I /dev/sdb

/dev/sdb:

ATA device, with non-removable media
Model Number: WDC WD4001FAEX-00MJRA0
Serial Number: WD-WCC1F0078****
Firmware Revision: 01.01L01
Transport: Serial, SATA 1.0a, SATA II Extensions, SATA Rev 2.5, SATA Rev 2.6, SATA Rev 3.0
Standards:
Supported: 8 7 6 5
Likely used: 8
Configuration:
Logical max current
cylinders 16383 16383
heads 16 16
sectors/track 63 63
--
CHS current addressable sectors: 16514064
LBA user addressable sectors: 268435455
LBA48 user addressable sectors: 7814037168
Logical/Physical Sector size: 512 bytes
device size with M = 1024*1024: 3815447 MBytes
device size with M = 1000*1000: 4000787 MBytes (4000 GB)
cache/buffer size = unknown
root@nas:~# lspci -vnnk -s 00:1f.2 ZFS 使用的 SATA 3.0 Gbps Controller
00:1f.2 SATA controller [0106]: Intel Corporation 631xESB/632xESB SATA AHCI Controller [8086:2681] (rev 09) (prog-if 01 [AHCI 1.0])
Subsystem: Super Micro Computer Inc Device [15d9:ac80]
Flags: bus master, 66MHz, medium devsel, latency 0, IRQ 17
I/O ports at 18b0 [size=8]
I/O ports at 18a4 [size=4]
I/O ports at 18a8 [size=8]
I/O ports at 18a0 [size=4]
I/O ports at 1880 [size=32]
Memory at d9004400 (32-bit, non-prefetchable) [size=1K]
Capabilities: [70] Power Management version 2
Capabilities: [a8] SATA HBA v1.0
Kernel driver in use: ahci
root@nas:~# lspci -vnnk -s 06:00.0 ROOT FS 的 SAS 3.0 Gbps Hardware RAID1 有 256MB DRAM
06:00.0 RAID bus controller [0104]: LSI Logic / Symbios Logic MegaRAID SAS 1078 [1000:0060] (rev 04)
Subsystem: Super Micro Computer Inc MegaRAID SAS PCI Express ROMB [15d9:c080]
Flags: bus master, fast devsel, latency 0, IRQ 33
Memory at d8c40000 (64-bit, non-prefetchable) [size=256K]
I/O ports at 2000 [size=256]
Memory at d8c00000 (64-bit, non-prefetchable) [size=256K]
[virtual] Expansion ROM at d8c80000 [disabled] [size=128K]
Capabilities: [b0] Express Endpoint, MSI 00
Capabilities: [c4] MSI: Enable- Count=1/4 Maskable- 64bit+
Capabilities: [d4] MSI-X: Enable+ Count=4 Masked-
Capabilities: [e0] Power Management version 2
Capabilities: [ec] Vital Product Data
Capabilities: [100] Power Budgeting <?>
Kernel driver in use: megaraid_sas

還未安裝的主角

OCZ RevoDrive 350 SSD 480GB PCI-E 2 x8 (RVD350-FHPX28-480G) (新品 NT$ 13000)


LSI2008/IBM M5015 SAS RAID 卡 (二手 NT$ 2000)


再2顆 WD 黑標 7200rpm 4TB (新品 專案價 $4000 可遇不可求)


由於我的記憶體有 64GB, 所以在還未升級上述的３個配備前的測試, 都要使用超過 64GB 的檔案來做測試. (ZFS 本身會建立 32GB 的 RAM Disk 當L1) vmdisk 是我的 zpool 下面

root@nas:/vmdisk# dd if=/dev/zero of=zerofile.000 bs=1M count=100000; sleep 30 ; dd if=zerofile.000 of=/dev/null bs=1M
100000+0 records in
100000+0 records out
104857600000 bytes (105 GB) copied, 274.539 s, 382 MB/s 寫入速度
100000+0 records in
100000+0 records out
104857600000 bytes (105 GB) copied, 222.822 s, 471 MB/s 讀取速度

相同的伺服器, 現在測試在 LSI 1078 SAS RAID 1 15000rpm 146GB x2 上面, 檔案大小 63G /root/ 是在 SAS RAID1 下面,

root@nas:~# dd if=/dev/zero of=zerofile.000 bs=1M count=60000; sleep 30 ; dd if=zerofile.000 of=/dev/null bs=1M
60000+0 records in
60000+0 records out
62914560000 bytes (63 GB) copied, 725.167 s, 86.8 MB/s
60000+0 records in
60000+0 records out
62914560000 bytes (63 GB) copied, 809.041 s, 77.8 MB/s

完全不令我訝異, ZFS 的軟體RAID 不出我所料, 比 LSI 的硬體 RAID 還快, 雙方都無 SSD 做 cache 的情況下, ZFS 無 RAM DISK協助, 但是 LSI 硬體RAID還有犯規多 256MB DRAM 在卡上加速. 上述的測試只是簡單測試就可以看出 ZFS 是多麼優秀, 希望最後不會是 "信者信之，不信者恒不信". IT 技術的東西不該是宗教, 應該讓數字說話.

Infiniband Card IB-IPOIB 的 eth 測試 (如果是RDMA 速度會快4倍到 40Gbps)

root@nas:~# iperf -c san (1 gbps eth)
------------------------------------------------------------
Client connecting to san, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.253.20 port 55386 connected with 192.168.253.3 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 1.10 GBytes 943 Mbits/sec
root@nas:~# iperf -c san-ib (10 gbps IB over IP)
------------------------------------------------------------
Client connecting to san-ib, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[ 3] local 10.0.3.20 port 57029 connected with 10.0.3.3 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 9.96 GBytes 8.56 Gbits/sec

IB 的速度沒有到 10 Gb/s, 看來我還有 Driver 的版本需要修正. 或者我插入的 pci-e 沒有提供足夠的頻寬給卡 pci-e 3.0 @ pci-e 2.0 的槽的問題.

3個禮拜後我會完成所有升級設備的安裝, 預計我的 zpool 會有 1200 MB/s ~ 1800 MB/s 的 寫入速度, 讀取速度預估 2200 MB/s ~ 3200 MB/s, 如果沒有達成上述的預測, 那麼我的升級就算是失敗 (續代)

接下來我不會測 hardware raid vs software raid. 因為我沒有 SAS 3.0 的卡以及搭配的 HD 在這台上面. (不過我有一台 X9 系列的伺服器要準備升級使用 Intel 750 NVMe 了喲)