INT 21h
Another raid6 recovery story
№ 11223 В разделе "Sysadmin" от February 14th, 2021,
Ahh… Again… You thinking affected server is a backup of backup server but somehow this old as dinosaur shit server contains part of production without any copy in git or somewhere else.
60000 power-on hours of each hard drive. Yeeeaah.
Load rescue OS and check mdstat. Two disks already dead and failing third, FS already corrupted. Everything as we love.
>$ cat /proc/mdstat
md2 : active raid6 sda3[6] sdb3[5] sdd3[4] sdf3[3] sdg3[0]
1073085440 blocks super 1.2 level 6, 512k chunk, algorithm 2 [7/5] [U__UUUU]
bitmap: 2/2 pages [8KB], 65536KB chunk
You can try replace dead disks, reassemble raid in-place and fix FS, but who knows, result may be much worse than current state. So I decided to create partitions images and work with them.
Copy images over network to new server, if dd won’t work use ddrescue.
dd if=/dev/sda3 bs=1M | gzip | ssh root@new_server 'gzip -d | dd of=/sda3 bs=1M' dd if=/dev/sdb3 bs=1M | gzip | ssh root@new_server 'gzip -d | dd of=/sdb3 bs=1M' dd if=/dev/sdd3 bs=1M | gzip | ssh root@new_server 'gzip -d | dd of=/sdd3 bs=1M' dd if=/dev/sdf3 bs=1M | gzip | ssh root@new_server 'gzip -d | dd of=/sdf3 bs=1M' dd if=/dev/sdg3 bs=1M | gzip | ssh root@new_server 'gzip -d | dd of=/sdg3 bs=1M'
Now you have all images to reassemble partition, so use losetup to imaginate that image file is a block device, because mdadm not working with images directly.
losetup -P /dev/loop0 /sda3 losetup -P /dev/loop1 /sdb3 losetup -P /dev/loop2 /sdd3 losetup -P /dev/loop3 /sdf3 losetup -P /dev/loop4 /sdg3
Now run cat /proc/mdstat
md127 : active raid6 loop4[6] loop3[5] loop2[4] loop1[3] loop0[0]
1073085440 blocks super 1.2 level 6, 512k chunk, algorithm 2 [7/5] [U__UUUU]
bitmap: 2/2 pages [8KB], 65536KB chunk
unused devices:
If raid not assembled automaticaly run mdadm -A md127 /dev/loop0 /dev/loop1 /dev/loop2 /dev/loop3 /dev/loop4.
Yes, you assembled raid partition from images! Lets try to fix filesystem.
fsck -y /dev/md127
But this is not the right time to give up
Try to use another supeblock fsck -b 32768 -y /dev/md127
AAaaaand fsck can’t write changes to first superblock, lol. BUT! You can use alive superblock to mount FS!
mount -o sb=131072 /dev/md127 /mnt
Add Microsoft marketplace back to VSCodium
№ 11218 В разделе "Sysadmin" от February 10th, 2021,
Someone replaced microsoft’s marketplace to codium, so many extensions just absent now. This link will help you return back all your favorite extensions.
Replace in /usr/share/codium/resources/app/product.json:
"extensionsGallery": {
"serviceUrl": "https://marketplace.visualstudio.com/_apis/public/gallery",
"itemUrl": "https://marketplace.visualstudio.com/items"
}
https://github.com/VSCodium/vscodium/blob/master/DOCS.md#extensions-marketplace
Зарядное устройство Электрон-3М
№ 11202 В разделах: Бытовая электроника Мои авто users guide от January 10th, 2021,
Оч давно пользуюсь этим зарядным, но начали появляться внезапные отключения тока пока не выдернешь из розетки и заново не запустишь. Инструкции нет и пришлось тщательно погуглить. Выяснилось, что отключние заряда может произойти только при замыкании клемм или склеивании ласт аккумулятора 😀
Прилагаю все найденное к этому посту, если хотите отремонтировать плату или изготовить новую.
Инструкция
Virtualbox: mount physical disk
№ 11191 В разделе "Sysadmin"
от January 4th, 2021,
В подшивках: Linux, Virtualbox
Add your user to disk group and relogin\reboot.
Next create volume file:
VBoxManage internalcommands createrawvmdk -filename 16Gflash.vmdk -rawdisk /dev/sdf
Add disk and mount to virtual machine
Credentials and other secrets from Vault to your containers at startup
№ 11183 В разделах: Programming
Sysadmin
от January 2nd, 2021,
В подшивках: Docker, Go, Kubernetes, Security, Vault
What if you stored your database credentials in Vault and want to make ENV variables with them for your application at container startup? You can do it for Kubernetes deployments or plain Docker containers with my small program vault-envs.
Add to your Dockerfile additional steps:
- install my vault-envs programs that “converts” secret to ENV variables
- create\modify entrypoint script where or call vault-envs and other pre-startup actions
Add to your Dockerfile steps:
...
...
# add Ubuntu\Debian repo and install vault-envs with fresh certificates
RUN curl http://deb.blindage.org/gpg-key.asc | apt-key add - && \
echo "deb http://deb.blindage.org bionic main" | tee /etc/apt/sources.list.d/21h.list && \
apt update
RUN apt install -y ca-certificates vault-envs
# copy entrypoint script
COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]
Your entrypoint script will look like:
#!/bin/bash
...
...
export eval `vault-envs -token "$VAULT_TOKEN" \
-vault-url https://vault.blindage.org \
-vault-path /prod/crm/connection_postgres -envs-prefix "PG_"`
export eval `vault-envs -token "$VAULT_TOKEN" \
-vault-url https://vault.blindage.org \
-vault-path /prod/crm/connection_mysql -envs-prefix "MYSQL_"`
export eval `vault-envs -token "$VAULT_TOKEN" \
-vault-url https://vault.blindage.org \
-vault-path /prod/crm/connection_api`
...
...
exec "$@"
If some vars names is identical they will be overwritten at next vault-envs call, so I used prefix.
Now build image and run
docker run --rm -e VAULT_TOKEN=s.QQmLlqnHnRAEO9eUeoggeK1n crm printenv
and see results at container console:
... VAULT_RETRIEVER=vault-envs PG_DB_PASS=postgres PG_DB_PORT=5432 PG_DB_USER=postgres PG_DB_HOST=db-postgres PG_DB_NAME=crm MYSQL_DB_HOST=mysql.wordpress MYSQL_DB_PASS= MYSQL_DB_PORT=3306 MYSQL_DB_USER=root MYSQL_DB_NAME=wordpress API_HOST=http://crm/api API_TOKEN=giWroufpepfexHyentOnWebBydHojGhokEpAnyibnipNirryesaccasayls4 ...
Wooh! You did it.
Последние записи
Комментарии
Микроблог перейти
© Vladimir Smagin, 2005-2021. Копирование материалов без разрешения запрещено. GPG DA4CD0F5E222EA727D6A40C413BCE12E5618F071 *
Fortune cookie: Sex and mathematics have one thing in common. You can do each while thinking about the other.