INT 21h

Hi, I am Vladimir Smagin, SysAdmin and

Kaptain. Telegram Email / GIT / RSS / GPG

Another raid6 recovery story

№ 11223 В разделе "Sysadmin" от February 14th, 2021,
В подшивках: Linux, Security

Ahh… Again… You thinking affected server is a backup of backup server but somehow this old as dinosaur shit server contains part of production without any copy in git or somewhere else.

60000 power-on hours of each hard drive. Yeeeaah.

Load rescue OS and check mdstat. Two disks already dead and failing third, FS already corrupted. Everything as we love.

>$ cat /proc/mdstat 
md2 : active raid6 sda3[6] sdb3[5] sdd3[4] sdf3[3] sdg3[0]
      1073085440 blocks super 1.2 level 6, 512k chunk, algorithm 2 [7/5] [U__UUUU]
      bitmap: 2/2 pages [8KB], 65536KB chunk

You can try replace dead disks, reassemble raid in-place and fix FS, but who knows, result may be much worse than current state. So I decided to create partitions images and work with them.

Copy images over network to new server, if dd won’t work use ddrescue.

dd if=/dev/sda3 bs=1M | gzip | ssh root@new_server 'gzip -d | dd of=/sda3 bs=1M'
dd if=/dev/sdb3 bs=1M | gzip | ssh root@new_server 'gzip -d | dd of=/sdb3 bs=1M'
dd if=/dev/sdd3 bs=1M | gzip | ssh root@new_server 'gzip -d | dd of=/sdd3 bs=1M'
dd if=/dev/sdf3 bs=1M | gzip | ssh root@new_server 'gzip -d | dd of=/sdf3 bs=1M'
dd if=/dev/sdg3 bs=1M | gzip | ssh root@new_server 'gzip -d | dd of=/sdg3 bs=1M'

Now you have all images to reassemble partition, so use losetup to imaginate that image file is a block device, because mdadm not working with images directly.

losetup -P /dev/loop0 /sda3
losetup -P /dev/loop1 /sdb3
losetup -P /dev/loop2 /sdd3
losetup -P /dev/loop3 /sdf3
losetup -P /dev/loop4 /sdg3

Now run cat /proc/mdstat

md127 : active raid6 loop4[6] loop3[5] loop2[4] loop1[3] loop0[0]
      1073085440 blocks super 1.2 level 6, 512k chunk, algorithm 2 [7/5] [U__UUUU]
      bitmap: 2/2 pages [8KB], 65536KB chunk

unused devices:

If raid not assembled automaticaly run mdadm -A md127 /dev/loop0 /dev/loop1 /dev/loop2 /dev/loop3 /dev/loop4.

Yes, you assembled raid partition from images! Lets try to fix filesystem.

fsck -y /dev/md127

But this is not the right time to give up

Try to use another supeblock fsck -b 32768 -y /dev/md127

AAaaaand fsck can’t write changes to first superblock, lol. BUT! You can use alive superblock to mount FS!

mount -o sb=131072 /dev/md127 /mnt

Нет комментариев »

Credentials and other secrets from Vault to your containers at startup

№ 11183 В разделах: Programming Sysadmin от January 2nd, 2021,
В подшивках: Docker, Go, Kubernetes, Security, Vault

What if you stored your database credentials in Vault and want to make ENV variables with them for your application at container startup? You can do it for Kubernetes deployments or plain Docker containers with my small program vault-envs.

Add to your Dockerfile additional steps:

install my vault-envs programs that “converts” secret to ENV variables
create\modify entrypoint script where or call vault-envs and other pre-startup actions

Add to your Dockerfile steps:

...
...
# add Ubuntu\Debian repo and install vault-envs with fresh certificates
RUN curl http://deb.blindage.org/gpg-key.asc | apt-key add - && \
    echo "deb http://deb.blindage.org bionic main" | tee /etc/apt/sources.list.d/21h.list && \
    apt update
RUN apt install -y ca-certificates vault-envs

# copy entrypoint script
COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh

ENTRYPOINT ["/entrypoint.sh"]

Your entrypoint script will look like:

#!/bin/bash

...
...

export eval `vault-envs -token "$VAULT_TOKEN" \
        -vault-url https://vault.blindage.org \
        -vault-path /prod/crm/connection_postgres -envs-prefix "PG_"`

export eval `vault-envs -token "$VAULT_TOKEN" \
        -vault-url https://vault.blindage.org \
        -vault-path /prod/crm/connection_mysql -envs-prefix "MYSQL_"`

export eval `vault-envs -token "$VAULT_TOKEN" \
        -vault-url https://vault.blindage.org \
        -vault-path /prod/crm/connection_api`

...
...

exec "$@"

If some vars names is identical they will be overwritten at next vault-envs call, so I used prefix.

Now build image and run

docker run --rm -e VAULT_TOKEN=s.QQmLlqnHnRAEO9eUeoggeK1n crm printenv

and see results at container console:

...
VAULT_RETRIEVER=vault-envs
PG_DB_PASS=postgres
PG_DB_PORT=5432
PG_DB_USER=postgres
PG_DB_HOST=db-postgres
PG_DB_NAME=crm
MYSQL_DB_HOST=mysql.wordpress
MYSQL_DB_PASS=
MYSQL_DB_PORT=3306
MYSQL_DB_USER=root
MYSQL_DB_NAME=wordpress
API_HOST=http://crm/api
API_TOKEN=giWroufpepfexHyentOnWebBydHojGhokEpAnyibnipNirryesaccasayls4
...

Wooh! You did it.

Нет комментариев »

GNUK

№ 11163 В разделах: Electronics Sysadmin от December 31st, 2020,
В подшивках: GPG, Linux, Security

Замутил себе GNUK в качестве хранилища ключей для дешифровки писем, бэкапов и прочего хлама, а также ssh авторизации на серверах. Стоит оно всего 140 руб, что гораздо дешевле аналогов за 50 баксов 🙂 На отлично работает под линуксами с GnuPG.

https://wiki.debian.org/GNUK
https://blog.danman.eu/2-usb-crypto-token-for-use-with-gpg-and-ssh/
https://nx3d.org/gnuk-st-link-v2/

Заказал тут https://aliexpress.ru/item/32792925130.html Для прошивки нужен второй такой или любой другой прошивальщик STM32.

Нет комментариев »

Centralize fail2ban blacklisting with ip-blocker-db

№ 11031 В разделе "Sysadmin" от June 17th, 2020,
В подшивках: Linux, Security

My own IP storage for fail2ban. Written to blacklist fucking botnets bruteforcing my servers. It centalize information about blocks across all my servers in one single watch tower.

https://git.blindage.org/21h/ip-blocker-db

Нет комментариев »

Vault secret retrieve and save to JSON file

№ 10280 В разделах: Programming Sysadmin от September 1st, 2019,
В подшивках: Go, Security, Vault

I wrote small program to retrieve secrets from Vault and provide them to my PHP and Python apps. ENV variables with connection credentials is useful with Docker containers and even Kubernetes, list of secrets to retrieve can be stored inside Docker image.

Secret stored in Vault