One SSL, One Domain with Multiple Services: HAproxy on Kubernetes

Bugra Ozturk
5 min readApr 4, 2020

--

Today’s systems depends on multiple services with multiple back-ends. Also, in micro-service architecture, there are multiple services that front-end have to be communicate. This problem can be solved with multiple domain and multiple SSL bind to them but there is a solution that require less cost and maintenance over domains and their SSLs.

When handling lots of clients and lots of services, we have to use more reliable and highly available tools in our infrastructure. Kubernetes and HAProxy one of the most reliable and highly available tools in industry. Following architecture help you to reduce managing these connections.

Architecture

This architecture provide, communication between your front-end Apps and multiple service on your company, this architecture reduce workload and cost of your system. HAProxy terminates SSL communication and provide communication with multiple services over one SSL.

Solutions is Reverse Proxy with SSL termination on HAproxy. Following solution deployed on bare metal Kubernetes. This means that if you deploy this solution on cloud, you can use Load Balancer instead of NodePort to get more dynamism.

HAProxy

HAproxy is one of the most widely used Highly Available Load Balancer software over TCP&HTTP. It is fast, reliable and open-source :)

http://www.haproxy.org/

For more information about HAProxy and it’s features please visit following link: HAProxy

HAProxy Configurations

HAProxy manages all of it configuration from a file called haproxy.cfg. Current stable version is HAProxy 2.1-stable. Documentation of latest stable version of haproxy about configuration can be reach from HAProxy Configurations.

First of all our SSL file must be ready for HAProxy. How can we make ready our SSL for HAProxy. There are multiple file type for SSL certificate such as p12, crt and key or jks for Java.

Following commands describe how to convert file to file.

  • jks to p12
keytool -importkeystore -srckeystore filename.jks -destkeystore filename.p12 -srcstoretype jks -deststoretype pkcs12
  • p12 to crt and key
openssl pkcs12 -in filename.p12 -nocerts -out filename.key
openssl pkcs12 -in filename.p12 -clcerts -nokeys -out filename.crt
  • key to key without passphrase
openssl rsa -in filename.key -out filename.key
  • key and crt to HAProxy ready pem
sudo cat filename.crt filename.key \
| sudo tee filename.pem

Our pem file is ready to read by HAProxy.

  • haproxy.cfg
#haproxy.cfg#provide statistical monitoring of services routed on HAProxy
listen stats
bind *:30090
mode http
timeout connect 10s
timeout client 30s
timeout server 30s
option forwardfor
option httpclose
stats enable
stats show-legends
stats refresh 5s
stats uri /stats
stats realm Haproxy\ Statistics
stats auth username:pass # Username and Password for stats page
stats admin if TRUE
# provide global configuration to prevent configuration duplication
global
maxconn 50000
tune.ssl.default-dh-param 2048
#provide default configurations for defined frontend and backend configurationsdefaults
timeout connect 10s
timeout client 100s
timeout server 100s
log global
# provide configuration to connection from clientsfrontend https-in
bind *:443 ssl crt /usr/local/etc/haproxy/filename.pem
mode http
redirect scheme https if !{ ssl_fc }
acl uri_service1 path_beg /objectdetection
use_backend databossobjectdetectionservice if uri_service1
acl uri_service2 path_beg /superresolution
use_backend databosssuperresolutionservice if uri_service2
acl uri_service3 path_beg /textsummary
use_backend databosstextsummarizationservice uri_service3
acl uri_service4 path_beg /videoanomaly
use_backend databossvideoanomalyservice if uri_service4
default_backend databosswebservice#provide configuration to load balance serversbackend databossobjectdetectionservice
mode http
balance roundrobin
server objectdetectionserviceapp1 IP:Port check
server objectdetectionserviceapp2 IP:Port check
server objectdetectionserviceapp3 IP:Port check
backend databosssuperresolutionservice
mode http
balance roundrobin
server ssuperresolutionserviceapp1 IP:Port check
server ssuperresolutionserviceapp2 IP:Port check
server ssuperresolutionserviceapp3 IP:Port check
backend databosstextsummarizationservice
mode http
balance roundrobin
server textsummarizationserviceapp1 IP:Port check
server textsummarizationserviceapp2 IP:Port check
server textsummarizationserviceapp3 IP:Port check
backend databossvideoanomalyservice
mode http
balance roundrobin
server videoanomalyserviceapp1 IP:Port check
backend databosswebservice
mode http
balance roundrobin
server webserviceapp1 IP:port check
server webserviceapp2 IP:port check

More information about essentials of haproxy configuration could be read from a Chad Lavoie’s blog.

Kubernetes

Deployment of HAProxy on Bare Metal Kubernetes will introduce as Deployment, NodePort, ConfigMap and Secret.

We apply ConfigMap will hold haproxy.cfg and Secret will hold api.pem files. When configurations persisted on Kubernetes environment, it provide flexibility that statefull apps could act as stateless that automatically deployed on any cluster node.

First we have to create our ConfigMap and Secret with following commands.

kubectl create cm haproxy-cfg --from-file=haproxy.cfg
kubectl create secret generic api-ssl --from-file=filename.pem

There will be two NodePort for stats page: *:30090 and for HTTPS endpoint: *:443.

  • haproxy-svc.yaml
apiVersion: v1
kind: Service
metadata:
name: haproxy-svc
spec:
selector:
app: databoss-haproxy
type: NodePort
ports:
- name: https
port: 443
nodePort: 30443
  • haproxy-stats-svc.yaml
apiVersion: v1
kind: Service
metadata:
name: haproxy-stats-svc
spec:
selector:
app: databoss-haproxy
type: NodePort
ports:
- name: http
port: 30090
nodePort: 31090

Now we apply services.

kubectl apply -f haproxy-svc.yaml
kubectl apply -f haproxy-stats-svc.yaml

After creating services on Kubernetes, we are ready to apply our Deployment .

Note: Following Deployment YAML can be used after Kubernetes v1.16. Because of the changes in Kubernetes API, we have to migrate our old Deployments to new Deployments. This added to apps/v1 and we have to writeselector in our deployment specs.

After Kubernetes v1.16, Deployment in the extensions/v1beta1, apps/v1beta1, and apps/v1beta2 API versions is no longer served

Please read the Vallery Lancey (Lyft)’s blog about API deprecation and converting old API objects to new API objects.

  • haproxy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: databoss-haproxy
spec:
selector:
matchLabels:
app: databoss-haproxy
replicas: 1
template:
metadata:
labels:
app: databoss-haproxy
spec:
volumes:
- name: ha-volume
configMap:
name: haproxy-cfg
items:
- key: haproxy.cfg
path: haproxy.cfg
- name: ha-ssl
secret:
secretName: api-ssl
items:
- key: filename.pem
path: filename.pem
containers:
- image: haproxy
name: haproxy
volumeMounts:
- name: ha-volume
mountPath: /usr/local/etc/haproxy/haproxy.cfg
subPath: haproxy.cfg
- name: ha-ssl
mountPath: /usr/local/etc/haproxy/filename.pem
subPath: filename.pem
imagePullPolicy: Always
env:
- name: TZ
value: "Turkey"

Now we apply deployment.

kubectl apply -f haproxy.yaml

It’s done!!!

Conclusion

We deploy HAProxy on Kubernetes that handles multiple services over one domain and one SSL. It reduce our workload to manage&maintain multiple domain and SSL. Also, we have the power of highly available Load Balancer and highly available orchestration service to reduce downtime of our services.

In this article, we learn how we could provide communication between our Client and all our different kind of services on different environments with Only One Domain&SSL.

In Kubernetes HAProxy deployment, Kubernetes RBAC can be integrated. It could provide authorization layer on Kubernetes within your organization.

--

--