Commit f1655fed authored by Justin Palpant's avatar Justin Palpant

Merge branch 'the-great-k8s-migration' into 'master'

IT IS FINISHED

See merge request !28
parents 2bebc1a9 35598ab5
[submodule "cluster-management/monitoring"]
path = cluster-management/monitoring
url = https://github.com/coreos/prometheus-operator.git
[submodule "external-storage-contrib"]
path = cluster-management/storage/external-storage
url = https://github.com/kubernetes-incubator/external-storage.git
# What Is It?
This project will hold the files I use to describe the infrastructure for hosting applications in my homelab.
[[_TOC_]]
# Copyright and License
Copyright 2017 Justin Palpant
......@@ -13,11 +15,10 @@ All rights reserved.
### Services
- Default Gateway
- DHCP: 192.168.0.0/24
- DHCP: 192.168.0.100-192.168.0.254
- Port forwarding:
- - 443/tcp: 192.168.0.33:443 (flowproxy service on Docker Swarm, pegged to node IP)
- - 80/tcp: 192.168.0.10:443 (open HTTP port on NAS for http-01 challenge)
- - 636/tcp: 192.168.0.33:636 (ldapproxy service on Docker Swarm, pegged to node IP)
- - 443/tcp: 192.168.0.33:443
- - 80/tcp: 192.168.0.33:443
- - 1195/udp: 192.168.0.10:1195 (OpenVPN service on NAS)
## NAS
......@@ -25,65 +26,104 @@ All rights reserved.
192.168.0.10/nas.sfo.palpant.us
### Services
- https://dsm.palpant.us
- https://video.palpant.us
- https://files.palpant.us
- LDAP server (not ldap.palpant.us though)
- dsm.palpant.us backend: http://nas.sfo.palpant.us:5010
- video.palpant.us backend: http://nas.sfo.palpant.us:7001
- files.palpant.us backend: http://nas.sfo.palpant.us:9007
- ldap.palpant.us backend: ldaps://nas.sfo.palpant.us:636
- RADIUS server
- NFS shares
- DNS server for *.sfo.palpant.us and *.palpant.us (internal)
- DNS server for *.sfo.palpant.us and *.palpant.us with forwarding zone for svc.cluster.local to 192.168.0.33:53
- iSCSI targets (3): 2 LUNs on Target 1 for legacy container-direct mounting, and then one LUN each under targets 2 and 21 to serve as backing blocks for Ceph
## Kubernetes Cluster
## Nodes
### 192.168.0.32/ubuntu-mac-01.sfo.palpant.us
Ubuntu 16.04.3 running on /dev/sda4 on Macbook
#### Ports
Kubernetes ports:
tcp 0 0 127.0.0.1:10248 0.0.0.0:* LISTEN 941/kubelet
tcp 0 0 127.0.0.1:10249 0.0.0.0:* LISTEN 2071/kube-proxy
tcp6 0 0 :::10250 :::* LISTEN 941/kubelet
tcp6 0 0 :::10255 :::* LISTEN 941/kubelet
tcp6 0 0 :::9100 :::* LISTEN 1984/node_exporter
Calico ports:
tcp 0 0 0.0.0.0:179 0.0.0.0:* LISTEN 3167/bird
tcp6 0 0 :::9099 :::* LISTEN 3168/calico-felix
Application ports:
tcp6 0 0 :::443 :::* LISTEN 2071/kube-proxy
tcp6 0 0 :::636 :::* LISTEN 2071/kube-proxy
tcp6 0 0 :::80 :::* LISTEN 2071/kube-proxy
## Docker Swarm
### Nodes
{node IP/node hostname}:{hardware/software}
System ports:
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1339/sshd
tcp6 0 0 :::22 :::* LISTEN 1339/sshd
- 192.168.0.31/ubuntu-dev-1.sfo.palpant.us:Virtualbox VM on Mac OSX/Ubuntu 16.04.3
- 192.168.0.32/ubuntu-dev-2.sfo.palpant.us:Virtualbox VM on Mac OSX/Ubuntu 16.04.3
- 193.168.0.33/ubuntu-udoo-01.sfo.palpant.us:Udoo X86 Advanced/Ubuntu 16.04.3
Unknown?
tcp6 0 0 :::23105 :::* LISTEN 2071/kube-proxy
tcp6 0 0 :::10256 :::* LISTEN 2071/kube-proxy
### Volumes
{volume_name}: {storage type+location} - {description}
- `flowproxy_syslog`: host local, global distribution - sharing the SYSLOG socket for the flowproxy HAProxy services
- `gitlab_git-data`: nas.sfo.palpant.us:/volume1/Docker Volumes/gitlab/git-data, NFS - actual git repositories in GitLab
- `gitlab_git-ssh`: nas.sfo.palpant.us:/volume1/Docker Volumes/gitlab/ssh, NFS - ssh data, fairly static, haven't changed this in ages. Possibly where GitLab puts SSH keys?
- `gitlab_gitlab-backups`: nas.sfo.palpant.us:/volume1/Docker Volumes/gitlab/backups, NFS - place where backups go when you do gitlab-rake gitlab:backup:create
- `gitlab_gitlab-ci-builds`: nas.sfo.palpant.us:/volume1/Docker Volumes/gitlab/gitlab-ci-builds, NFS - not used
- `gitlab_gitlab-postgres`: host local, ubuntu-udoo-01 - all postgres data for GitLab postgres service. Bound to host because pg doesn't like NFS mounts, ugh.
- `gitlab_gitlab-rails-shared`: nas.sfo.palpant.us:/volume1/Docker Volumes/gitlab/gitlab-rails-shared, NFS - shared files? Not sure
- `gitlab_gitlab-rails-uploads`: nas.sfo.palpant.us:/volume1/Docker Volumes/gitlab/gitlab-rails-uploads, NFS - uploaded files in Gitlab? Not sure
- `gitlab_gitlab-redis`: nas.sfo.palpant.us:/volume1/Docker Volumes/gitlab/redis, NFS - place for gitlab to create Redis dumps as needed. Not often used.
- `prom_grafana_data`: host local, ubuntu-udoo-01 - place for Grafana data, such as dashboard state, users, etc. Grafana also does not seem to like NFS, but I didn't try that hard.
- `prom_prometheus_data`: host local, ubuntu-udoo-01 - place for all scraped Prometheus data. Host local because Promtheus [strongly discourages NFS](https://github.com/prometheus/prometheus/issues/1600#issuecomment-215372732)
- `slackhooks_syslog`: host local, global - sharing the SYSLOG socket for the slackhooks HAProxy services
- `tim_static_web`: NFS mount, global - so Tim can make changes to his website without the Docker build process.
- `traefik-certs` host local, ubuntu-udoo-01 - ALL certs for Traefik. Rarely updated and a static file, but Traefik won't use a cert file with open permissions, so NFS doesn't work. Backed up to NFS though, at /Docker Volumes/traefik, manual process.
### 193.168.0.33/ubuntu-udoo-01.sfo.palpant.us
Ubuntu 16.04.3 running on Udoo X86 Advanced SBC
Kuberenetes Master Node
### Ports
{port number}: {service name:port} - description
#### Ports
Kubernetes ports:
tcp 0 0 127.0.0.1:10248 0.0.0.0:* LISTEN 849/kubelet
tcp 0 0 127.0.0.1:10249 0.0.0.0:* LISTEN 2114/kube-proxy
tcp 0 0 127.0.0.1:2379 0.0.0.0:* LISTEN 1574/etcd
tcp 0 0 127.0.0.1:2380 0.0.0.0:* LISTEN 1574/etcd
tcp6 0 0 :::10250 :::* LISTEN 849/kubelet
tcp6 0 0 :::6443 :::* LISTEN 11246/kube-apiserve
tcp6 0 0 :::10251 :::* LISTEN 11534/kube-schedule
tcp6 0 0 :::10252 :::* LISTEN 11423/kube-controll
tcp6 0 0 :::9100 :::* LISTEN 1954/node_exporter
tcp6 0 0 :::10255 :::* LISTEN 849/kubelet
- 443: `flowproxy_proxy:443` - the L4 reverse proxy (with SNI) at the border, receives all 443 traffic straight from the router
- 636: `ldapproxy_haproxy:636` - the L4 reverse proxy with SNI at the border, receives all 636 traffic straight from the router
- 16080: `tlsauto_traefik:80` - I think this is only opened so tlsauto_traefik has internet access. Or maybe for HTTP testing. Not really used
- 18080: `prom_cadvisor:8080` - only opened for testing prometheus metrics, not useful
- 19090: `prom_prometheus:9090` - opened for low-level prometheus management and debugging, which is often useful
- 19093: `prom_alertmanager:9093` - opened for low-level alertmanager management and debugging, which is great for silencing alerts
Calico ports:
tcp 0 0 0.0.0.0:179 0.0.0.0:* LISTEN 3269/bird
tcp6 0 0 :::6666 :::* LISTEN 1997/etcd
tcp6 0 0 :::9099 :::* LISTEN 3268/calico-felix
tcp6 0 0 :::6667 :::* LISTEN 1997/etcd
Application ports:
tcp6 0 0 :::443 :::* LISTEN 2114/kube-proxy
tcp6 0 0 :::636 :::* LISTEN 2114/kube-proxy
tcp6 0 0 :::80 :::* LISTEN 2114/kube-proxy
# Projects
All of this infrastructure is working together to support my personal projects, many of which get integrated as components of the infrastructure, others of which are run on the infrastructure.
System ports:
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 929/sshd
tcp6 0 0 :::22 :::* LISTEN 929/sshd
## L4 TCP Reverse Proxies with SNI
NFS
tcp 0 0 0.0.0.0:45092 0.0.0.0:* LISTEN 10020/rpc.statd
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 10034/rpcbind
tcp6 0 0 :::111 :::* LISTEN 10034/rpcbind
tcp6 0 0 :::49650 :::* LISTEN 10020/rpc.statd
## Docker Registry
Unknown?
tcp 0 0 0.0.0.0:38122 0.0.0.0:* LISTEN -
tcp6 0 0 :::23105 :::* LISTEN 2114/kube-proxy
tcp6 0 0 :::44716 :::* LISTEN -
tcp6 0 0 :::10256 :::* LISTEN 2114/kube-proxy
## Traefik with automatic certificate acquisition
## Gitlab
## Cluster-available services
## Prometheus + Grafana
### Ingress
### Dynamic storage provisioning
Coming soon!
### Monitoring
### Private Docker Registry
### Source control with locally hosted Gitlab
## Glances
# Versions
Here I'll keep track of the current production version of the base image of each application, container, and other important software, for quick reference
......@@ -92,33 +132,31 @@ Here I'll keep track of the current production version of the base image of each
### Docker nodes
- Docker Version - 17.09.0-ce
- Ubuntu Version - 16.04.3 LTS
- Kubernetes Version 1.8.4
- Calico Version 2.6
### NAS
- Synology Version - DSM 6.1.3-15152 Update 8
## Services
## Cluster Services
### Registry
- Docker Registry - 2, unknown, pulled from "latest"
- Docker Registry - 2.6.2
- Cesanta Docker Auth - 1.3
### Proxying
- Traefik - 1.4.0-rc4
- HAProxy - 1.7-alpine
- rsyslogd - unknown, jumanjiman/rsyslog:latest
### Ingress
- nginx-ingress-controller
- kube-lego
### Prometheus
- Prometheus - v1.7.2
- Grafana - 4.5.2
- Alertmanager - v0.9.1
- cadvisor - unknown, google/cadvisor:latest
- node-exporter - unknown, basi/node-exporter:latest
### Monitoring
- prometheus-operator
- prometheus
- alertmanager
- grafana
### Gitlab
- gitlab-ce - 10.0.3
- Postgresql - 9.6.2-alpine
- Redis - 4.0-alpine
### Static sites
## Applications
https://justin.palpant.us, https://tim.palpant.us, https://visualizer.palpant.us
All unknown, nginx:latest
......@@ -128,13 +166,166 @@ Diagrams are good and useful. I should make one.
# To-do list
In no particular order, it would be good to:
- Draw a complete system diagram
- Fix versions of unversioned containers
- Add project descriptions
- Set CPU and memory limits and allocations on other containers
- Improve alerting for CPU usage, memory usage, disk space, and temperature in alertmanager
- Add alerting for container failure/replacement in alertmanager
- Add scrapers for traefik, HAProxy, and gitlab to prometheus and connect to prometheus
- Add dashboarding for traefik, HAProxy, and gitlab to grafana
- Update GitLab to 10.1.0-ce.0, Traefik to v1.4.1
- Validate GitLab config again and disable extra gitlab components
Update: postponing many of these to-dos in order to fully move to Kubernetes first. Some of the work would be duplicated on both Swarm and k8s, which would be wasteful.
- See Kubernetes Migration section below for what is currently happening
- **POSTPONED** Draw a complete system diagram
- **POSTPONED** Fix versions of unversioned containers
- **ELIMINATED** Add project descriptions
- Need to re-think this
- **POSTPONED** Set CPU and memory limits and allocations on other containers
- **POSTPONED** Improve alerting for CPU usage, memory usage, disk space, and temperature in alertmanager
- **ELIMINATED** Add alerting for container failure/replacement in alertmanager
- kubernetes alerting is sufficient
- **ELIMINATED** Add scrapers for traefik, HAProxy, and gitlab to prometheus and connect to prometheus
- these services are no longer part of the infrastructure
- **ELIMINATED** Add dashboarding for traefik, HAProxy, and gitlab to grafana
- these services are no longer part of the infrastructure
- **ELIMINATED** Update GitLab to 10.1.0-ce.0, Traefik to v1.4.1
- **DONE** Validate GitLab config again and disable extra gitlab components
- extra components have been turned off in the migration process and the reduced config has been updated and validated.
# Kubernetes Migration
I am in the middle of migrating all cluster services to Kubernetes. Excluding the L4 reverse proxy for LDAP, all services have been cut over from the Docker Swarm system! Here's what should be done next:
- **DONE** Cut over LDAP reverse proxy
- used the nginx-ingress tcp services to do this
- **DONE** Eliminate Docker Swarm from the system entirely.
- **DONE** Debug prometheus-operator -> prometheus sync. Is it working already?
- appears to be working actually
## Project Management
- Update rest of README
## Application management
- Pin and update versions
- GitLab is now at least one minor version behind
- Monitoring
- Enable GitLab liveness and readiness check
- Liveliness and readiness checks for other applications?
## Monitoring
### Grafana
- Improve Grafana dashboards using tooling around configmaps
- Instant mode for all dials and counters
- e2e latency metrics incorrect
- add cpu and memory limits for containers and change fill to fill only current usage
- Add graph for temperature of node on Node page, using hwmon
### Alertmanager
- Fix alertmanager to talk to Slack again using tooling around alertmanager.yml secret
- Add alert for hwmon temperature with respect to critical temperature,
### Applications
- Write Prometheus exporter for WeMo Insight switch, scrape, and dashboard on one of the cluster pages
- Dashboard for nginx-vts stats
- Dashboard for prometheus itself!
## Storage
### Ceph
- Install ceph with helm
- Set up ceph management: [web UI](https://github.com/ceph/calamari), if possible, definitely try the CLI
- Set up ceph monitoring: [prometheus exporter](https://github.com/digitalocean/ceph_exporter)
- Set up ceph rgw for S3-like interface!!
### NFS
- Install [nfs-client provisioner](https://github.com/kubernetes-incubator/external-storage/tree/master/nfs-client) and test
- Move Registry to auto-provisioned volume
# Node Provisioning guide
## Ubuntu
Install ubuntu-server-16.04.3
Include OpenSSH
Choose a hostname
Do not create a swap partition or kubelet will be angry
Edit `/etc/network/interfaces` to assign a static IP and resolvers
Something like this
```
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
source /etc/network/interfaces.d/*
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
auto enp2s0f0
iface enp2s0f0 inet static
address 192.168.0.32
up ip route replace default via 192.168.0.1 dev enp2s0f0
dns-nameservers 192.168.0.10
dns-search sfo.palpant.us palpant.us svc.cluster.local
```
But watch out for the interface name~
`sudo apt update && sudo apt upgrade`
`sudo apt install nfs-common glances build-essential update-notifier`
## DNS
Set the DNS record for the new hostname in the sfo.palpant.us zone
## LDAP
[Instructions](https://help.ubuntu.com/community/LDAPClientAuthentication)
`sudo apt-get install ldap-auth-client nscd`
Go through the wizard
Possibly edit /etc/nsswitch.conf to re-order ldap and passwd login
`sudo pam-auth-update`
`sudo service nscd restart`
## Docker
[Instructions](https://docs.docker.com/engine/installation/linux/docker-ce/ubuntu/#install-using-the-repository)
```bash
sudo apt-get install \
apt-transport-https \
ca-certificates \
curl \
software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository \
"deb [arch=amd64] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) \
stable"
sudo apt-get update
sudo apt-get install docker-ce
```
## Kubeadm
[Instructions](https://kubernetes.io/docs/setup/independent/install-kubeadm/#installing-kubeadm-kubelet-and-kubectl)
```bash
apt-get update && apt-get install -y apt-transport-https
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial main
EOF
apt-get update
apt-get install -y kubelet kubeadm kubectl
```
[How to generate the join token CA cert hash](https://github.com/kubernetes/kubeadm/issues/519)
(On master)
`kubeadm token create`
`openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'`
(On node)
`kubeadm join --token $TOKEN $MASTER_IP:$MASTER_PORT --discovery-ca-cert-sha sha256:$SHA`
## NTP
`sudo timedatectl set-ntp on`
## iSCSI
[Setting up auto-connect to one or more targets](https://library.netapp.com/ecmdocs/ECMP1654943/html/GUID-8EC685B4-8CB6-40D8-A8D5-031A3899BCDC.html)
[General iSCSI initiator help](https://help.ubuntu.com/lts/serverguide/iscsi-initiator.html)
Edit /etc/iscsid/iscsi.conf to set up CHAP
Do NOT enable node.startup, but rather set up individual target logins
## For Mac OSX dual-booting Ubuntu:
Need to deal with fan control unfortunately, so install [mbpfan](https://github.com/dgraziotin/mbpfan), with [instructions](https://ineed.coffee/3838/a-beginners-tutorial-for-mbpfan-under-ubuntu/)
FROM cesanta/docker_auth:1.3
LABEL maintainer Justin Palpant <justin@palpant.us>
COPY start.sh /start.sh
RUN chmod +x /start.sh
RUN chmod 777 /start.sh
COPY config /config
ENTRYPOINT /start.sh
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: docker-auth
namespace: prod
spec:
replicas: 1
template:
metadata:
labels:
app: docker-auth
spec:
containers:
- name: docker-auth
image: cesanta/docker_auth:1.3
volumeMounts:
- name: docker-auth-config
mountPath: /config
readOnly: true
- name: docker-auth-certs
mountPath: /certs
readOnly: true
resources:
limits:
cpu: 0.1
memory: 50Mi
requests:
memory: 20Mi
ports:
- containerPort: 5001
command: ["/docker_auth/auth_server", "-v=2", "--alsologtostderr", "/config/auth_config.yml"]
volumes:
- name: docker-auth-config
configMap:
name: docker-auth
- name: docker-auth-certs
secret:
secretName: certs-docker-auth
......@@ -9,11 +9,11 @@
server:
addr: ":5001"
certificate: "/run/secrets/auth-cert"
key: "/run/secrets/auth-key"
certificate: "/certs/tls.crt"
key: "/certs/tls.key"
token:
issuer: "Acme auth server" # Must match issuer in the Registry config.
issuer: "PalpantLab Auth Server" # Must match issuer in the Registry config.
expiration: 900
# LDAP authentication.
......@@ -41,4 +41,4 @@ acl:
comment: "Logged in users can query the catalog."
- match: {account: "/.+/"}
actions: ["pull"]
comment: "Logged in users can query the catalog."
comment: "Logged in users can pull anything."
#! /bin/bash
kubectl create configmap docker-auth --from-file=config/auth_config.yml -n prod
version: '3.3'
services:
auth:
image: registry.palpant.us/justin/palpantlab-auth:1.3-1
# ports:
# - 5001:5001
# command: ["--v=2", "--alsologtostderr", "/config/auth_config.yml"]
networks:
- flowproxy
- haproxy-ldap
secrets:
- auth-cert
- auth-key
deploy:
mode: replicated
replicas: 2
update_config:
parallelism: 1
delay: 10s
monitor: 5s
failure_action: rollback
networks:
flowproxy:
external: true
haproxy-ldap:
external: true
secrets:
auth-cert:
external: true
auth-key:
external: true
---
apiVersion: v1
kind: Service
metadata:
name: docker-auth
namespace: prod
spec:
type: ClusterIP
ports:
- port: 5001
selector:
app: docker-auth
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
ingress.kubernetes.io/secure-backends: "true"
ingress.kubernetes.io/force-ssl-redirect: "true"
kubernetes.io/tls-acme: "true"
name: docker-auth
namespace: prod
spec:
rules:
- host: auth.palpant.us
http:
paths:
- backend:
serviceName: kube-lego-nginx
servicePort: 8080
path: /.well-known/acme-challenge
- backend:
serviceName: docker-auth
servicePort: 5001
path: /
# This section is only required if TLS is to be enabled for the Ingress
tls:
- hosts:
- auth.palpant.us
secretName: certs-docker-auth
#!/bin/sh
# Copy over certificates to correct place and update certificate storage
# find "/config/ldap_certificates" -type f -exec cp -fv {} /usr/local/share/ca-certificates/ \;
# update-ca-certificates
# # Replace newline and carriage returns in password file
# cat /config/ldap_password.txt | tr -d '\r\n' > /tmp/ldap_password.txt.clean
# # If we see a custom config file, we load that instead of the default one
CONF_PATH=/config/auth_config.yml
if [ -f $CONF_PATH.custom ]; then
CONF_PATH=$CONF_PATH.custom
fi
# Start the auth server
/docker_auth/auth_server -v=5 $CONF_PATH
## nginx configuration
## Ref: https://github.com/kubernetes/ingress/blob/master/controllers/nginx/configuration.md
##
controller:
name: controller
image:
repository: gcr.io/google_containers/nginx-ingress-controller
tag: "0.9.0-beta.15"
pullPolicy: IfNotPresent
config: {}
# Required for use with CNI based kubernetes installations (such as ones set up by kubeadm),
# since CNI and hostport don't mix yet. Can be deprecated once https://github.com/kubernetes/kubernetes/issues/23920
# is merged
hostNetwork: false
## Required only if defaultBackend.enabled = false
## Must be <namespace>/<service_name>
##
defaultBackendService: ""
## Optionally specify the secret name for default SSL certificate
## Must be <namespace>/<secret_name>
##
defaultSSLCertificate: "ingress-controller/cert-default"
## Election ID to use for status update
##
electionID: ingress-controller-leader
## Name of the ingress class to route through this controller
##
ingressClass: nginx
# labels to add to the pod container metadata
podLabels: {}
# key: value