INT 21h

Hi, I am Vladimir Smagin, SysAdmin and

Kaptain. Telegram Email / GIT / Thingiverse / RSS / GPG

Credentials and other secrets from Vault to your containers at startup

№ 11183 В разделах: Programming Sysadmin от January 2nd, 2021,
В подшивках: Docker, Go, Kubernetes, Security, Vault

What if you stored your database credentials in Vault and want to make ENV variables with them for your application at container startup? You can do it for Kubernetes deployments or plain Docker containers with my small program vault-envs.

Add to your Dockerfile additional steps:

install my vault-envs programs that “converts” secret to ENV variables
create\modify entrypoint script where or call vault-envs and other pre-startup actions

Add to your Dockerfile steps:

...
...
# add Ubuntu\Debian repo and install vault-envs with fresh certificates
RUN curl http://deb.blindage.org/gpg-key.asc | apt-key add - && \
    echo "deb http://deb.blindage.org bionic main" | tee /etc/apt/sources.list.d/21h.list && \
    apt update
RUN apt install -y ca-certificates vault-envs

# copy entrypoint script
COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh

ENTRYPOINT ["/entrypoint.sh"]

Your entrypoint script will look like:

#!/bin/bash

...
...

export eval `vault-envs -token "$VAULT_TOKEN" \
        -vault-url https://vault.blindage.org \
        -vault-path /prod/crm/connection_postgres -envs-prefix "PG_"`

export eval `vault-envs -token "$VAULT_TOKEN" \
        -vault-url https://vault.blindage.org \
        -vault-path /prod/crm/connection_mysql -envs-prefix "MYSQL_"`

export eval `vault-envs -token "$VAULT_TOKEN" \
        -vault-url https://vault.blindage.org \
        -vault-path /prod/crm/connection_api`

...
...

exec "$@"

If some vars names is identical they will be overwritten at next vault-envs call, so I used prefix.

Now build image and run

docker run --rm -e VAULT_TOKEN=s.QQmLlqnHnRAEO9eUeoggeK1n crm printenv

and see results at container console:

...
VAULT_RETRIEVER=vault-envs
PG_DB_PASS=postgres
PG_DB_PORT=5432
PG_DB_USER=postgres
PG_DB_HOST=db-postgres
PG_DB_NAME=crm
MYSQL_DB_HOST=mysql.wordpress
MYSQL_DB_PASS=
MYSQL_DB_PORT=3306
MYSQL_DB_USER=root
MYSQL_DB_NAME=wordpress
API_HOST=http://crm/api
API_TOKEN=giWroufpepfexHyentOnWebBydHojGhokEpAnyibnipNirryesaccasayls4
...

Wooh! You did it.

Нет комментариев »

Golang: marshal and unmarshal iota (yaml, json, toml)

№ 11025 В разделе "Programming" от June 16th, 2020,
В подшивках: Go

Tried to unmarshal enum value from app config file and failed?

DB:
  masterDB: "ips.sdb"
  clean: Full

Use strings instead of iota.

const (
	CleanRecreate  = "Recreate" // completely remove DB file and create again
	CleanFull      = "Full" // delete all and vacuum
	CleanLastDay   = "LastDay" // remove all before last day
	CleanLastWeek  = "LastWeek"
	CleanLastMonth = "LastMonth"
	CleanNever     = "Never" // do nothing
)

type CleanType string

type AppConfig struct {
	DB    struct {
		MasterDB string    `yaml:"masterDB"`        // master sqlite database
		Clean    CleanType `yaml:"clean,omitempty"` // cleanup at startup
	} `yaml:"DB"`
}

Now load config file:

func main() {
	flag.Parse()
	if *configFilename == "" {
		log.Fatalln("Set configuration filename")
	}

	// read settings from file
	log.Println("Loading config file", *configFilename)
	appConfig := AppConfig{}

	yamlFile, err := ioutil.ReadFile(*configFilename)
	if err != nil {
		log.Fatalf("Config read error: %v\n", err)
	}
	err = yaml.Unmarshal(yamlFile, &appConfig)
	if err != nil {
		log.Fatalf("Config format error: %v\n", err)
	}

	switch appConfig.DB.Clean {
	case CleanRecreate:
		log.Println("Recreate cleanup option set")
		os.Remove(appConfig.DB.MasterDB)
	case CleanFull:
		log.Println("Full cleanup option set")
	case CleanLastDay:
		log.Println("Save only last day cleanup option set")
	case CleanLastWeek:
		log.Println("Save only last week option set")
	case CleanLastMonth:
		log.Println("Save only last month option set")
	}
	dbHandler := dbLoadFile(appConfig.DB.MasterDB)
	defer dbHandler.Close()
}

Here is another solution https://gist.github.com/lummie/7f5c237a17853c031a57277371528e87#file-enum-go

Нет комментариев »

Warm Image operator for Kubernetes

№ 11010 В разделе "Sysadmin" от May 28th, 2020,
В подшивках: Go, Kubernetes, Kubernetes Operator, Operator SDK

For example, you have huge image with your software and running POD on node. When POD moving to another node your image downloads to new node minute or two. This operator forces nodes to download image before rescheduling, so POD starts faster.

It runs /bin/sh with infinite loop on specified image as DaemonSet with additional options like NodeSelector, Affinity or resource limits. You can specify custom command if your image not contains /bin/sh interpreter or you want to run own script.

Your first warmer:

apiVersion: blindage.org/v1alpha1
kind: WarmImage
metadata:
  name: mongo4
spec:
  image: mongo
  version: "4"
  nodeSelector:
    node-role.kubernetes.io/master: ""

Now you warmed mongo:4 on all master nodes.

Repository here https://git.blindage.org/21h/warm-image-operator

Нет комментариев »

CronJobs operator for Kubernetes

№ 10306 В разделах: Programming Sysadmin от September 17th, 2019,
В подшивках: Go, Kubernetes, Kubernetes Operator, Operator SDK

Helps to control multiple cronjobs with same image, but different commands.

Checkout code and documentation https://git.blindage.org/21h/cron-operator

Нет комментариев »

Vault secret retrieve and save to JSON file

№ 10280 В разделах: Programming Sysadmin от September 1st, 2019,
В подшивках: Go, Security, Vault

I wrote small program to retrieve secrets from Vault and provide them to my PHP and Python apps. ENV variables with connection credentials is useful with Docker containers and even Kubernetes, list of secrets to retrieve can be stored inside Docker image.

Secret stored in Vault