One SSL, One Domain with Multiple Services: HAproxy on Kubernetes

Today’s systems depends on multiple services with multiple back-ends. Also, in micro-service architecture, there are multiple services that front-end have to be communicate. This problem can be solved with multiple domain and multiple SSL bind to them but there is a solution that require less cost and maintenance over domains and their SSLs.

When handling lots of clients and lots of services, we have to use more reliable and highly available tools in our infrastructure. Kubernetes and HAProxy one of the most reliable and highly available tools in industry. Following architecture help you to reduce managing these connections.

Architecture

This architecture provide, communication between your front-end Apps and multiple service on your company, this architecture reduce workload and cost of your system. HAProxy terminates SSL communication and provide communication with multiple services over one SSL.

Solutions is Reverse Proxy with SSL termination on HAproxy. Following solution deployed on bare metal Kubernetes. This means that if you deploy this solution on cloud, you can use Load Balancer instead of NodePort to get more dynamism.

HAProxy

HAproxy is one of the most widely used Highly Available Load Balancer software over TCP&HTTP. It is fast, reliable and open-source :)

http://www.haproxy.org/

For more information about HAProxy and it’s features please visit following link: HAProxy

HAProxy Configurations

HAProxy manages all of it configuration from a file called haproxy.cfg. Current stable version is HAProxy 2.1-stable. Documentation of latest stable version of haproxy about configuration can be reach from HAProxy Configurations.

First of all our SSL file must be ready for HAProxy. How can we make ready our SSL for HAProxy. There are multiple file type for SSL certificate such as p12, crt and key or jks for Java.

Following commands describe how to convert file to file.

• jks to p12

keytool -importkeystore -srckeystore filename.jks -destkeystore filename.p12 -srcstoretype jks -deststoretype pkcs12

• p12 to crt and key

openssl pkcs12 -in filename.p12 -nocerts -out filename.key
openssl pkcs12 -in filename.p12 -clcerts -nokeys -out filename.crt

• key to key without passphrase

openssl rsa -in filename.key -out filename.key

• key and crt to HAProxy ready pem

sudo cat filename.crt filename.key \
           | sudo tee filename.pem

Our pem file is ready to read by HAProxy.

• haproxy.cfg

#haproxy.cfg
#provide statistical monitoring of services routed on HAProxy
listen stats
    bind *:30090
    mode http
    timeout connect 10s
    timeout client 30s
    timeout server 30s
    option forwardfor
    option httpclose
    stats enable
    stats show-legends
    stats refresh 5s
    stats uri /stats
    stats realm Haproxy\ Statistics
    stats auth username:pass # Username and Password for stats page
    stats admin if TRUE
# provide global configuration to prevent configuration duplication
global
   maxconn 50000
   tune.ssl.default-dh-param 2048
#provide default configurations for defined frontend and backend configurations
defaults
    timeout connect 10s
    timeout client 100s
    timeout server 100s
    log global
# provide configuration to connection from clients
frontend https-in
   bind *:443 ssl crt /usr/local/etc/haproxy/filename.pem
   mode http
   redirect scheme https if !{ ssl_fc }
acl uri_service1 path_beg /objectdetection
use_backend databossobjectdetectionservice if uri_service1
acl uri_service2 path_beg /superresolution
use_backend databosssuperresolutionservice if uri_service2
acl uri_service3 path_beg /textsummary
use_backend databosstextsummarizationservice uri_service3
acl uri_service4 path_beg /videoanomaly
use_backend databossvideoanomalyservice if uri_service4
default_backend databosswebservice
#provide configuration to load balance servers
backend databossobjectdetectionservice
   mode http
   balance     roundrobin
   server      objectdetectionserviceapp1 IP:Port check
   server      objectdetectionserviceapp2 IP:Port check
   server      objectdetectionserviceapp3 IP:Port check
backend databosssuperresolutionservice
   mode http
   balance     roundrobin
   server      ssuperresolutionserviceapp1 IP:Port check
   server      ssuperresolutionserviceapp2 IP:Port check
   server      ssuperresolutionserviceapp3 IP:Port check
backend databosstextsummarizationservice
   mode http
   balance     roundrobin
   server      textsummarizationserviceapp1 IP:Port check
   server      textsummarizationserviceapp2 IP:Port check
   server      textsummarizationserviceapp3 IP:Port check
backend databossvideoanomalyservice
   mode http
   balance     roundrobin
   server      videoanomalyserviceapp1 IP:Port check
backend databosswebservice
   mode http
   balance     roundrobin
   server      webserviceapp1 IP:port check
   server      webserviceapp2 IP:port check

More information about essentials of haproxy configuration could be read from a Chad Lavoie’s blog.

Kubernetes

Deployment of HAProxy on Bare Metal Kubernetes will introduce as Deployment, NodePort, ConfigMap and Secret.

We apply ConfigMap will hold haproxy.cfg and Secret will hold api.pem files. When configurations persisted on Kubernetes environment, it provide flexibility that statefull apps could act as stateless that automatically deployed on any cluster node.

First we have to create our ConfigMap and Secret with following commands.

kubectl create cm haproxy-cfg --from-file=haproxy.cfg
kubectl create secret generic api-ssl --from-file=filename.pem

There will be two NodePort for stats page: *:30090 and for HTTPS endpoint: *:443.

• haproxy-svc.yaml

apiVersion: v1
kind: Service
metadata:
  name: haproxy-svc
spec:
  selector:
    app: databoss-haproxy
  type: NodePort
  ports:
  - name: https
    port: 443
    nodePort: 30443

• haproxy-stats-svc.yaml

apiVersion: v1
kind: Service
metadata:
  name: haproxy-stats-svc
spec:
  selector:
    app: databoss-haproxy
  type: NodePort
  ports:
  - name: http
    port: 30090
    nodePort: 31090

Now we apply services.

kubectl apply -f haproxy-svc.yaml
kubectl apply -f haproxy-stats-svc.yaml

After creating services on Kubernetes, we are ready to apply our Deployment .

Note: Following Deployment YAML can be used after Kubernetes v1.16. Because of the changes in Kubernetes API, we have to migrate our old Deployments to new Deployments. This added to apps/v1 and we have to writeselector in our deployment specs.

After Kubernetes v1.16, Deployment in the extensions/v1beta1, apps/v1beta1, and apps/v1beta2 API versions is no longer served

Please read the Vallery Lancey (Lyft)’s blog about API deprecation and converting old API objects to new API objects.

• haproxy.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: databoss-haproxy
spec:
  selector:
    matchLabels:
      app: databoss-haproxy
  replicas: 1
  template:
    metadata:
      labels:
        app: databoss-haproxy
    spec:
      volumes:
        - name: ha-volume
          configMap:
            name: haproxy-cfg
            items:
                - key: haproxy.cfg
                  path: haproxy.cfg
        - name: ha-ssl
          secret:
            secretName: api-ssl
            items:
                - key: filename.pem
                  path: filename.pem
      containers:
        - image: haproxy
          name: haproxy
          volumeMounts:
            - name:  ha-volume
              mountPath: /usr/local/etc/haproxy/haproxy.cfg
              subPath: haproxy.cfg
            - name: ha-ssl
              mountPath: /usr/local/etc/haproxy/filename.pem
              subPath: filename.pem
          imagePullPolicy: Always
          env:
          - name: TZ
            value: "Turkey"

Now we apply deployment.

kubectl apply -f haproxy.yaml

It’s done!!!

Conclusion

We deploy HAProxy on Kubernetes that handles multiple services over one domain and one SSL. It reduce our workload to manage&maintain multiple domain and SSL. Also, we have the power of highly available Load Balancer and highly available orchestration service to reduce downtime of our services.

In this article, we learn how we could provide communication between our Client and all our different kind of services on different environments with Only One Domain&SSL.

In Kubernetes HAProxy deployment, Kubernetes RBAC can be integrated. It could provide authorization layer on Kubernetes within your organization.