INT 21h

Hi, I am Vladimir Smagin, SysAdmin and

Kaptain. Telegram Email / GIT / RSS / GPG

Remove automatically created raid partition for /home in Hetzner

№ 11312 В разделе Sysadmin от August 14th, 2021,
В подшивках: Hetzner, Linux

For example, you got real server in Hetzner.

Automatical installer made own partitioning and you mad about this:

root@k8s ~ # df -h
Filesystem      Size  Used Avail Use% Mounted on
tmpfs            13G  1,9M   13G   1% /run
tmpfs            63G     0   63G   0% /dev/shm
tmpfs           5,0M     0  5,0M   0% /run/lock
tmpfs           4,0M     0  4,0M   0% /sys/fs/cgroup
/dev/md3        1,5T   77M  1,5T   1% /home
/dev/md2        2,0T  8,5G  1,9T   1% /
/dev/md1        487M  312M  150M  68% /boot
tmpfs            13G  4,0K   13G   1% /run/user/0

WTF?! 1.5Tb for /home?! Lets remove md3 and extend root partition.

Look at partition map:

root@k8s ~/databases # fdisk -l

Disk /dev/nvme1n1: 3,49 TiB, 3840755982336 bytes, 7501476528 sectors
Disk model: SAMSUNG MZQL23T8HCLS-00A07              
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 131072 bytes / 131072 bytes
Disklabel type: gpt
Disk identifier: 05341056-A6AC-4EA5-B7BA-E9ADEF16810E

Device              Start        End    Sectors  Size Type
/dev/nvme1n1p1       4096    8392703    8388608    4G Linux RAID
/dev/nvme1n1p2    8392704    9441279    1048576  512M Linux RAID
/dev/nvme1n1p3    9441280 4233105407 4223664128    2T Linux RAID
/dev/nvme1n1p4 4233105408 7501476494 3268371087  1,5T Linux RAID
/dev/nvme1n1p5       2048       4095       2048    1M BIOS boot


Disk /dev/nvme0n1: 3,49 TiB, 3840755982336 bytes, 7501476528 sectors
Disk model: SAMSUNG MZQL23T8HCLS-00A07              
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 131072 bytes / 131072 bytes
Disklabel type: gpt
Disk identifier: F0BE7B87-1AF8-4D5F-92BB-C69D880D9ACE

Device              Start        End    Sectors  Size Type
/dev/nvme0n1p1       4096    8392703    8388608    4G Linux RAID
/dev/nvme0n1p2    8392704    9441279    1048576  512M Linux RAID
/dev/nvme0n1p3    9441280 4233105407 4223664128    2T Linux RAID
/dev/nvme0n1p4 4233105408 7501476494 3268371087  1,5T Linux RAID
/dev/nvme0n1p5       2048       4095       2048    1M BIOS boot


Disk /dev/md0: 4 GiB, 4289724416 bytes, 8378368 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 131072 bytes / 131072 bytes


Disk /dev/md1: 511 MiB, 535822336 bytes, 1046528 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 131072 bytes / 131072 bytes


Disk /dev/md2: 1,97 TiB, 2162380767232 bytes, 4223399936 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 131072 bytes / 131072 bytes


Disk /dev/md3: 1,52 TiB, 1673270722560 bytes, 3268106880 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 131072 bytes / 131072 bytes

Looks good, we can remove partition 3 and 4 on both disks and create new 3.5Tb partition.

At first, umount /home.

Stop md3:

root@k8s / # mdadm --stop /dev/md3
mdadm: stopped /dev/md3

mdstat

cat /proc/mdstat 
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] 
md3 : inactive nvme1n1p4[1](S)
      1634053447 blocks super 1.2
       
md2 : active raid1 nvme0n1p3[0] nvme1n1p3[1]
      2111699968 blocks super 1.2 [2/2] [UU]
      bitmap: 5/16 pages [20KB], 65536KB chunk

md1 : active raid1 nvme0n1p2[0] nvme1n1p2[1]
      523264 blocks super 1.2 [2/2] [UU]
      
md0 : active raid1 nvme0n1p1[0] nvme1n1p1[1]
      4189184 blocks super 1.2 [2/2] [UU]

Good! md3 stopped.

Now remove partition nvme0n1p3 from md2, you cant extend it without this.

root@k8s /etc # mdadm /dev/md2 --fail /dev/nvme0n1p3 --remove  /dev/nvme0n1p3
mdadm: set /dev/nvme0n1p3 faulty in /dev/md2
mdadm: hot removed /dev/nvme0n1p3 from /dev/md2

root@k8s /etc # cat /proc/mdstat 
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] 
md3 : active (auto-read-only) raid1 nvme1n1p4[1]
      1634053440 blocks super 1.2 [2/1] [_U]
      bitmap: 0/13 pages [0KB], 65536KB chunk

md2 : active raid1 nvme1n1p3[1]
      2111699968 blocks super 1.2 [2/1] [_U]
      bitmap: 5/16 pages [20KB], 65536KB chunk

md1 : active raid1 nvme0n1p2[0] nvme1n1p2[1]
      523264 blocks super 1.2 [2/2] [UU]
      
md0 : active raid1 nvme0n1p1[0] nvme1n1p1[1]
      4189184 blocks super 1.2 [2/2] [UU]

mdstat looks good, partition removed from md2.

Run fdisk and remove partition 3 and 4 from nvme0n1 and create new on freed space with bigger size.

root@k8s /etc # fdisk /dev/nvme0n1
Command (m for help): p
Disk /dev/nvme0n1: 3,49 TiB, 3840755982336 bytes, 7501476528 sectors
Disk model: SAMSUNG MZQL23T8HCLS-00A07              
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 131072 bytes / 131072 bytes
Disklabel type: gpt
Disk identifier: F0BE7B87-1AF8-4D5F-92BB-C69D880D9ACE

Device              Start        End    Sectors  Size Type
/dev/nvme0n1p1       4096    8392703    8388608    4G Linux RAID
/dev/nvme0n1p2    8392704    9441279    1048576  512M Linux RAID
/dev/nvme0n1p3    9441280 4233105407 4223664128    2T Linux RAID
/dev/nvme0n1p4 4233105408 7501476494 3268371087  1,5T Linux RAID
/dev/nvme0n1p5       2048       4095       2048    1M BIOS boot

Command (m for help): d
Partition number (1-5, default 5): 4

Command (m for help): d
Partition number (1-3,5, default 5): 3

Command (m for help): p
Disk /dev/nvme0n1: 3,49 TiB, 3840755982336 bytes, 7501476528 sectors
Disk model: SAMSUNG MZQL23T8HCLS-00A07              
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 131072 bytes / 131072 bytes
Disklabel type: gpt
Disk identifier: F0BE7B87-1AF8-4D5F-92BB-C69D880D9ACE

Device           Start     End Sectors  Size Type
/dev/nvme0n1p1    4096 8392703 8388608    4G Linux RAID
/dev/nvme0n1p2 8392704 9441279 1048576  512M Linux RAID
/dev/nvme0n1p5    2048    4095    2048    1M BIOS boot

Command (m for help): n
Partition number (3,4,6-128, default 3): 3
First sector (9441280-7501476494, default 9441280): 
Last sector, +/-sectors or +/-size{K,M,G,T,P} (9441280-7501476494, default 7501476494): 

Created a new partition 3 of type 'Linux filesystem' and of size 3,5 TiB.
Partition #3 contains a linux_raid_member signature.

Do you want to remove the signature? [Y]es/[N]o: n

Command (m for help): p

Disk /dev/nvme0n1: 3,49 TiB, 3840755982336 bytes, 7501476528 sectors
Disk model: SAMSUNG MZQL23T8HCLS-00A07              
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 131072 bytes / 131072 bytes
Disklabel type: gpt
Disk identifier: F0BE7B87-1AF8-4D5F-92BB-C69D880D9ACE

Device           Start        End    Sectors  Size Type
/dev/nvme0n1p1    4096    8392703    8388608    4G Linux RAID
/dev/nvme0n1p2 8392704    9441279    1048576  512M Linux RAID
/dev/nvme0n1p3 9441280 7501476494 7492035215  3,5T Linux filesystem
/dev/nvme0n1p5    2048       4095       2048    1M BIOS boot

Command (m for help): w
The partition table has been altered.
Syncing disks.

GOOOOOOOD! Now add disk back to raid and wait resync complete.

root@k8s /etc # mdadm -a /dev/md2 /dev/nvme0n1p3
mdadm: re-added /dev/nvme0n1p3
root@k8s /etc # cat /proc/mdstat 
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] 
md3 : active (auto-read-only) raid1 nvme1n1p4[1]
      1634053440 blocks super 1.2 [2/1] [_U]
      bitmap: 0/13 pages [0KB], 65536KB chunk

md2 : active raid1 nvme0n1p3[0] nvme1n1p3[1]
      2111699968 blocks super 1.2 [2/1] [_U]
      [>....................]  recovery =  0.4% (10490496/2111699968) finish=176.6min speed=198186K/sec
      bitmap: 5/16 pages [20KB], 65536KB chunk

md1 : active raid1 nvme0n1p2[0] nvme1n1p2[1]
      523264 blocks super 1.2 [2/2] [UU]
      
md0 : active raid1 nvme0n1p1[0] nvme1n1p1[1]
      4189184 blocks super 1.2 [2/2] [UU]

After few snickers bars look at mdstat again. Complete!

root@k8s /etc # cat /proc/mdstat 
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] 
md3 : active (auto-read-only) raid1 nvme1n1p4[1]
      1634053440 blocks super 1.2 [2/1] [_U]
      bitmap: 0/13 pages [0KB], 65536KB chunk

md2 : active raid1 nvme0n1p3[0] nvme1n1p3[1]
      2111699968 blocks super 1.2 [2/2] [UU]
      bitmap: 4/16 pages [16KB], 65536KB chunk

md1 : active raid1 nvme0n1p2[0] nvme1n1p2[1]
      523264 blocks super 1.2 [2/2] [UU]
      
md0 : active raid1 nvme0n1p1[0] nvme1n1p1[1]
      4189184 blocks super 1.2 [2/2] [UU]

So now you ready to do absolutely same steps for nvme1n1 (remove from md2, delete and create partitions, return back to md2 and wait resync).

After this check info

root@k8s /etc # mdadm -D /dev/md2
/dev/md2:
           Version : 1.2
     Creation Time : Thu Aug  5 13:08:12 2021
        Raid Level : raid1
        Array Size : 2111699968 (2013.87 GiB 2162.38 GB)
     Used Dev Size : 2111699968 (2013.87 GiB 2162.38 GB)
      Raid Devices : 2
     Total Devices : 2
       Persistence : Superblock is persistent

     Intent Bitmap : Internal

       Update Time : Fri Aug 13 16:54:00 2021
             State : active 
    Active Devices : 2
   Working Devices : 2
    Failed Devices : 0
     Spare Devices : 0

Consistency Policy : bitmap

              Name : rescue:2
              UUID : 7ce3c768:02323e8c:2aea5c1b:4ec26c8b
            Events : 21941

    Number   Major   Minor   RaidDevice State
       0     259        9        0      active sync   /dev/nvme0n1p3
       1     259        3        1      active sync   /dev/nvme1n1p3

Looks like you did changes but kernel cant see them. Reboot your server

Ok, you did it. Now resize partition:

root@k8s ~ # mdadm --grow /dev/md2 -z max
mdadm: component size of /dev/md2 has been set to 3745885511K
root@k8s ~ # resize2fs /dev/md2
resize2fs 1.45.7 (28-Jan-2021)
Filesystem at /dev/md2 is mounted on /; on-line resizing required
old_desc_blocks = 252, new_desc_blocks = 447
The filesystem on /dev/md2 is now 936471377 (4k) blocks long.

root@k8s ~ # df -h|grep md
/dev/md2        3,5T   23G  3,3T   1% /
/dev/md1        487M  312M  150M  68% /boot

WOOOOAAAAH!