Storage | How to Create Bcache in ESOS - BEONTEC

Storage | How to Create Bcache in ESOS

Create a software based caching device with SSD and HDD. This procedure is the same for NVMe (SSM) storage modules. In this case you need to create a Linux software raid over both modules first.

You should consider providing at least two hdd being mirrored and at least one SSD able to act as caching device (writethrough). In case you would like to use the caching device also to cache written data (writeback) you should provide at least two ssd.

In this scenario 'sda' will be the backing device and 'sdc' the caching device with the following specs:

sda=12 TB R10 array powered by 8x 3TB HDD WD Green-AV 5400rpm
sdc= 550 GB R10 array powered by 4x 275GB SSD Crucial MX300

CONSIDERATIONS

Before you begin you should make sure you meet the following requirements:

Only use SSD of the same type and model
Only use HDD of the same type and model
Connect the backing and caching devices to the same HBA or same RAID controller.
Do not mix between controller, eg. running the backing device on one HBA and the caching device on another one.
Make sure you only use ECC RAM being validated for your motherboard, eg. Intel certified RAM for Intel server mainboards.
Replace faulty RAM modules and validate by doing a health check, e.g. with Memtest 86+

HOW TO CREATE THE BCACHE

Before you start make sure you will format the correct devices in your system by checking the devices with fdisk:

# fdisk -l | less

Compare the device sizes with the volumes you had created in your HBA or RAID controller.

1. Format and register devices as backing and caching device

# make-bcache -C /dev/sdc
# echo "/dev/sdc" > /sys/fs/bcache/register
# make-bcache -B /dev/sda
# echo "/dev/sda" > /sys/fs/bcache/register

Make sure you have no further data on sda and sdc since the devices will be formatted during the make-bcache process!

2. Attach caching device to backing device

# bcache-super-show /dev/sdc
# echo "6d4ab278-0844-4a50-8e74-87aeda4fd353" > /sys/block/sda/bcache/attach

The UUID comes from 'cset.uuid' in the bcache-super-show command. The device '/dev/bcache0' should be created now!

3. Enable write cache mode [writeback]

Check which caching mode is enabled:

# cat /sys/block/bcache0/bcache/cache_mode

In case [writethrough] is selected you may want to switch to writeback mode:

# echo writeback > /sys/block/bcache0/bcache/cache_mode

4. Enable read only cache mode [writethrough]

When switching the caching mode from writeback to writethrough and dirty cache is existing on the caching device it will be flushed automatically to the backing device. After switching the caching mode make sure the cache is clean:

# echo writethrough > /sys/block/bcache0/bcache/cache_mode
$ watch cat /sys/block/bcache0/bcache/state

In case the cache stays being dirty you may want to force the flush by executing:

# echo 0 > /sys/block/bcache0/bcache/writeback_percent
$ watch cat /sys/block/bcache0/bcache/state

When becoming clean revert back the 'writeback_percent' value to 10:

# echo 10 > /sys/block/bcache0/bcache/writeback_percent

Check that everything has been correctly setup

# cat /sys/block/bcache0/bcache/state

5. Format the bcache0 with the file system xfs

# mkfs.xfs /dev/bcache0
# mkfs.xfs /dev/bcache0 -L "fs1-bc0"

With the '-L' parameter you can label the new file system already during the creation process. But you are good to go just when executing the first line.

6. Create directory for mounting the new file system

# cd /mnt/vdisks/
# mkdir fs1-bc0
# ls

7. Mount the new device bcache0 to the just created folder

# mount /dev/bcache0 /mnt/vdisks/fs1-bc0/

8. Update /etc/fstab file to let ESOS see the new disk being able to create virtual disks in the TUI later

# vi /etc/fstab

Add the entry:
LABEL=fs1-bc0 /mnt/vdisks/fs1-bc0 xfs defaults 1 1

9. ESOS TUI: Go to menu item 'Back-End Storage' and create a new virtual disk on the label fs1-bcache0

Check whether the file "vd" was created at /mnt/vdisks/fs1-bc0/

# cd /mnt/vdisks/fs1-bc0/
# ls

Is a "vd" file visible at "/mnt/vdisks/fs1-bc0/"?

HOW TO DELETE THE BCACHE

If you want to permanently destroy the bcache volume, you need to wipe the bcache superblock from the underlying device. This operation is not exposed through the sysfs interface. So:

1. Stop the bcache device as usual with

# echo 1 > /sys/block/<device>/bcache/stop

2. Wipe the superblocks with overwriting the head:

# head -c 1m /dev/zero > /dev/<caching device>
# head -c 1m /dev/zero > /dev/<backing device>

For some reason wipefs is not included in ESOS 0.1.9 being used during this tutorial.

"If you have a sufficiently new version of util-linux, you can use wipefs instead, which is more precise in wiping the bcache signature: wipefs -a /dev/<device>.) Obviously, you need to be careful to select the right device because this is a destructive operation that will wipe the header of the device. Take note that you will no longer have access to any data in the bcache volume!" Source: http://unix.stackexchange.com/questions/225017/how-to-remove-bcache0-volume

HOW TO REBOOT ESOS CORRECTLY | VERY IMPORTANT STEPS IN VMWARE ENVIRONMENT

1. Shutdown all VMs running on any storage hosted by ESOS

2. Make sure the ESXi log files are still being accessible by the hypervisor, eg. on a local storage or another shared storage.

So in case the ESXi logs are stored on an ESOS storage switch it over to a storage device which is still accessible when ESOS is being rebooted!

3. Check whether the cache status is dirty:

# cat /sys/block/bcache0/bcache/state

If the cache shows up as dirty you need to flush the dirty cache by:

# echo 0 > /sys/block/bcache0/bcache/writeback_percent

Observe the flushing process with:

# watch cat /sys/block/bcache0/bcache/dirty_data

As soon you see '0.0' you should also change the cache_mode to 'writethrough' by:

# echo writethrough > /sys/block/bcache0/bcache/cache_mode

Verify this change by:

# cat /sys/block/bcache0/bcache/cache_mode

As soon you see 'writethrough' in brackets you can reboot ESOS.

4. Reboot ESOS by executing

# reset

In case SCST is unable to find the mount point defined in the /etc/fstab file (e.g. when loading the entry LABEL=fs1-bc0 /mnt/vdisks/fs1-bc0 xfs defaults 1 1) this means that the directories of the mount points do not exists. (For some reason ESOS deletes these directories during shutdown process and is not creating thus not being able to mount the devices.) In this case proceed with step 5.

5. Login to ESOS > in TUI go to "Interface" > "Exit to Shell" and execute the following commands:

# mkdir /mnt/vdisks/fs1-bc0
# mkdir /mnt/vdisks/fs2-ssd
# mount /dev/bcache0 /mnt/vdisks/fs1-bc0
# mount /dev/sdb1 /mnt/vdisks/fs2-ssd
# /etc/rc.d/rc.scst start
# exit

Note: /etc/fstab gives you orientation which folders need to be created!

ESOS should be running now. :)

To automate the directory creation you can add these lines to the config file /etc/pre-scst_xtra_conf:
mkdir /mnt/vdisks/fs1-bc0
mkdir /mnt/vdisks/fs2-ssd
mount /dev/bcache0 /mnt/vdisks/fs1-bc0
mount /dev/sdb1 /mnt/vdisks/fs2-ssd

In case the file does not exist create /etc/pre-scst_xtra_conf and insert the four lines as shown above:

# touch /etc/pre-scst_xtra_conf
# vi /etc/pre-scst_xtra_conf