INT 21h

Hi, I am Vladimir Smagin, SysAdmin and

Kaptain. Telegram Email / GIT / RSS / GPG

Credentials and other secrets from Vault to your containers at startup

№ 11183 В разделах: Programming Sysadmin от January 2nd, 2021,
В подшивках: Docker, Go, Kubernetes, Security, Vault

What if you stored your database credentials in Vault and want to make ENV variables with them for your application at container startup? You can do it for Kubernetes deployments or plain Docker containers with my small program vault-envs.

Add to your Dockerfile additional steps:

install my vault-envs programs that “converts” secret to ENV variables
create\modify entrypoint script where or call vault-envs and other pre-startup actions

Add to your Dockerfile steps:

...
...
# add Ubuntu\Debian repo and install vault-envs with fresh certificates
RUN curl http://deb.blindage.org/gpg-key.asc | apt-key add - && \
    echo "deb http://deb.blindage.org bionic main" | tee /etc/apt/sources.list.d/21h.list && \
    apt update
RUN apt install -y ca-certificates vault-envs

# copy entrypoint script
COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh

ENTRYPOINT ["/entrypoint.sh"]

Your entrypoint script will look like:

#!/bin/bash

...
...

export eval `vault-envs -token "$VAULT_TOKEN" \
        -vault-url https://vault.blindage.org \
        -vault-path /prod/crm/connection_postgres -envs-prefix "PG_"`

export eval `vault-envs -token "$VAULT_TOKEN" \
        -vault-url https://vault.blindage.org \
        -vault-path /prod/crm/connection_mysql -envs-prefix "MYSQL_"`

export eval `vault-envs -token "$VAULT_TOKEN" \
        -vault-url https://vault.blindage.org \
        -vault-path /prod/crm/connection_api`

...
...

exec "$@"

If some vars names is identical they will be overwritten at next vault-envs call, so I used prefix.

Now build image and run

docker run --rm -e VAULT_TOKEN=s.QQmLlqnHnRAEO9eUeoggeK1n crm printenv

and see results at container console:

...
VAULT_RETRIEVER=vault-envs
PG_DB_PASS=postgres
PG_DB_PORT=5432
PG_DB_USER=postgres
PG_DB_HOST=db-postgres
PG_DB_NAME=crm
MYSQL_DB_HOST=mysql.wordpress
MYSQL_DB_PASS=
MYSQL_DB_PORT=3306
MYSQL_DB_USER=root
MYSQL_DB_NAME=wordpress
API_HOST=http://crm/api
API_TOKEN=giWroufpepfexHyentOnWebBydHojGhokEpAnyibnipNirryesaccasayls4
...

Wooh! You did it.

Нет комментариев »

Golang: marshal and unmarshal iota (yaml, json, toml)

№ 11025 В разделе "Programming" от June 16th, 2020,
В подшивках: Go

Tried to unmarshal enum value from app config file and failed?

DB:
  masterDB: "ips.sdb"
  clean: Full

Use strings instead of iota.

const (
	CleanRecreate  = "Recreate" // completely remove DB file and create again
	CleanFull      = "Full" // delete all and vacuum
	CleanLastDay   = "LastDay" // remove all before last day
	CleanLastWeek  = "LastWeek"
	CleanLastMonth = "LastMonth"
	CleanNever     = "Never" // do nothing
)

type CleanType string

type AppConfig struct {
	DB    struct {
		MasterDB string    `yaml:"masterDB"`        // master sqlite database
		Clean    CleanType `yaml:"clean,omitempty"` // cleanup at startup
	} `yaml:"DB"`
}

Now load config file:

func main() {
	flag.Parse()
	if *configFilename == "" {
		log.Fatalln("Set configuration filename")
	}

	// read settings from file
	log.Println("Loading config file", *configFilename)
	appConfig := AppConfig{}

	yamlFile, err := ioutil.ReadFile(*configFilename)
	if err != nil {
		log.Fatalf("Config read error: %v\n", err)
	}
	err = yaml.Unmarshal(yamlFile, &appConfig)
	if err != nil {
		log.Fatalf("Config format error: %v\n", err)
	}

	switch appConfig.DB.Clean {
	case CleanRecreate:
		log.Println("Recreate cleanup option set")
		os.Remove(appConfig.DB.MasterDB)
	case CleanFull:
		log.Println("Full cleanup option set")
	case CleanLastDay:
		log.Println("Save only last day cleanup option set")
	case CleanLastWeek:
		log.Println("Save only last week option set")
	case CleanLastMonth:
		log.Println("Save only last month option set")
	}
	dbHandler := dbLoadFile(appConfig.DB.MasterDB)
	defer dbHandler.Close()
}

Here is another solution https://gist.github.com/lummie/7f5c237a17853c031a57277371528e87#file-enum-go

Нет комментариев »

Warm Image operator for Kubernetes

№ 11010 В разделе "Sysadmin" от May 28th, 2020,
В подшивках: Go, Kubernetes, Kubernetes Operator, Operator SDK

For example, you have huge image with your software and running POD on node. When POD moving to another node your image downloads to new node minute or two. This operator forces nodes to download image before rescheduling, so POD starts faster.

It runs /bin/sh with infinite loop on specified image as DaemonSet with additional options like NodeSelector, Affinity or resource limits. You can specify custom command if your image not contains /bin/sh interpreter or you want to run own script.

Your first warmer:

apiVersion: blindage.org/v1alpha1
kind: WarmImage
metadata:
  name: mongo4
spec:
  image: mongo
  version: "4"
  nodeSelector:
    node-role.kubernetes.io/master: ""

Now you warmed mongo:4 on all master nodes.

Repository here https://git.blindage.org/21h/warm-image-operator

Нет комментариев »

Простой способ подготовки библиотеки на golang к тестированию

№ 11006 В разделе "Programming" от May 26th, 2020,
В подшивках: Go

Чтобы без проблем тестировать программы, написанные с использованием вашей библиотеки ее необходимо подготовить для этого. Делаем интерфейс, который будет использоваться в тестах, где ваши реальные функции будут заменены функциями с тестовыми данными.

package main

import "fmt"

// library 

type FooAdapter interface {
	Read() string
}

type Foo struct {
	mvar    string
}

func NewFoo(v string) FooAdapter {
	return &Foo{mvar:v}
}

func (a *Foo) Read() string {
	return "orig: " + a.mvar
}

// test 

func NewFooStub(v string) FooAdapter {
	return &FooStub{mvar: v}
}

type FooStub struct {
	mvar string
}

func (s *FooStub) Read() string {
	return "stub: " + s.mvar
}

func main() {
	z := NewFoo("o")
	fmt.Println("Read", z.Read())
	m := NewFooStub("s")
	fmt.Println("Read", m.Read())
}

https://play.golang.org/p/9utaWeDjNjo

Нет комментариев »

Hetzner DNS golang library

№ 10996 В разделе "Programming" от May 6th, 2020,
В подшивках: Go, Hetzner

I made this library to interact with Hetzner DNS API in most easy way. Hopefully in future it will be used for Hetzner external-dns provider. Check out example directory and API_help.md.

Get your own token on Hetzner DNS and place it to token variable and run code

token := "jcB2UywP9XtZGhvhSHpH5m"
zone := "vhSHpH5mjcB2UywP9XtZGh"

log.Println("Create new instance")
hdns := hclouddns.New(token)

log.Println("Get zone", zone)

allRecords, err := hdns.GetRecords(zone)
if err != nil {
	log.Fatalln(err)
}

log.Println(allRecords.Records)
log.Println(allRecords.Error)