Set up JBoss EAP 7 clustering in OpenShift using DNS_PING

Red Hat JBoss Enterprise Application Platform (JBoss EAP) 7 is an open source platform for highly transactional, web-scale Java applications deployed in war/jar/ear and abides by specifications such as Jakarta 8. But it also provides messaging, distributed caching, and clustering features, as explained in the Introduction to JBoss EAP Guide. 

In terms of clustering, when JBoss EAP 7 is deployed in Red Hat OpenShift Container Platform (RHOCP), by default JBoss EAP 7 image will have clustering capabilities enabled. To provide those clustering capabilities, JBoss EAP 7 relies on JGroups discovery protocols, such as DNS_PING or KUBE_PING, also known as ping protocols, to provide the ping capabilities for the clustering within the pods. 

In this two-part article series, I will provide more information about both of those mechanisms and how they differ. Both articles are based on solutions/articles such as EAP 7 image clustering in OCP 4, which provides extensive details on clustering. In this first part, I will describe the process when using DNS_PING as discovery protocol. 

It is important to highlight that although they are enabled by default, clustering capabilities are triggered/started on JBoss EAP 7 after the deployment of clustered applications. Therefore, clustering logs (such as the following one) are logged after the clustering capabilities start:

$ oc logs eap-example-0 | grep ISPN000094
00:01:01,023 INFO  [org.infinispan.CLUSTER] (ServerService Thread Pool -- 14) ISPN000094: Received new cluster view for channel ee: [eap-example-0] (1) [eap-example-0]

DNS_PING setting

The setting of DNS_PING as a discovery protocol is done using the environment variables JGROUPS_PING_PROTOCOL, OPENSHIFT_DNS_PING_SERVICE_NAME, and OPENSHIFT_DNS_PING_SERVICE_PORT. DNS_PING. The following table summarizes this.

When doing deployment via template in JBoss EAP 7 (i.e., Source-to-Image deployment is done) the deployment will use DNS_PING instead of KUBE_PING, which is the default usage when deploying via JBoss EAP Operator for example. Also, openshift.DNS_PING is the deprecated name, and the current name is: dns.DNS_PING. You can see other variables here.

DNS_PING operation

The JBoss EAP 7 configuration points to a service via the dns.DNS_PING properties. The following deployment via template:

$ cat /opt/eap/standalone/configuration/standalone-openshift.xml  | grep -A 3 -B 3 dns
            <stacks>
                <stack name="tcp">
                    <transport type="TCP" socket-binding="jgroups-tcp"/>
                    <protocol type="dns.DNS_PING">
                        <property name="dns_query">eap-app-ping</property>
                        <property name="async_discovery_use_separate_thread_per_request">true</property>
                    </protocol>
                    <protocol type="MERGE3"/>

So then the service eap-app-ping will be used to connect the pods—and to binding those pods a selector is used—this label must be on the pods. For instance, in the following example eap-app is the selector the service will bind, meaning any pod with that selector will be connected via this service.

$ oc get service eap-app-ping -o yaml
apiVersion: v1
kind: Service
metadata:
  annotations:
    description: The JGroups ping port for clustering.
...
spec:
  clusterIP: None
  clusterIPs:
  - None
...
  publishNotReadyAddresses: true
  selector:
    deploymentConfig: eap-app  <------------------------ here the deploymentConfig is set for the eap-app

Given JBoss EAP 7 is deployed via DeploymentConfig YAML, so then the DeploymentConfig and the pods will have the respective labels:

$ oc get dc/eap-app -o yaml | grep -B 5 -A 5 JGROUPS
        deploymentConfig: eap-app
      name: eap-app
    spec:
      containers:
      - env:
        - name: JGROUPS_PING_PROTOCOL
          value: dns.DNS_PING
        - name: OPENSHIFT_DNS_PING_SERVICE_NAME
          value: eap-app-ping
        - name: OPENSHIFT_DNS_PING_SERVICE_PORT
          value: "8888"

Pod labels:

  selector:
    deploymentConfig: eap-app  <------------------------ here the deploymentConfig is set for the eap-app

The pod's selector is crucial for the service (svc) to find it—otherwise, the service will not find the pods and therefore JBoss EAP 7 won't be able to cluster.

Service details

Clustering capabilities rely solely on eap-app-ping (as previously shown), so other services are not mandatory. The eap-app-ping is a headless service, meaning the service will point straight to the Pods' IPs. This can be shown as:

oc get ep,svc -o wide
NAME                                                     ENDPOINTS                                                          AGE
endpoints/eap-app-ping                                   1.1.1.127:8888,1.1.1.128:8888                                 75m

In the previous endpoints, 1.1.1.127:8888, 1.1.1.128:8888 (*example IPs) are two pods IPs. So, the requests go straight to those pods. This can be used to confirm if the headless service is functioning properly and connecting the pods. 

From the OpenShift Container Platform console, go to Network > Services, and the eap-app-ping should show the related pods (the ones it connects to) on the console, this will show more details on this service.

Troubleshooting DNS_PING

One can set DNS_PING for tracing or JGroups and have more details on the clustering operation:

### dns_ping as trace
$ oc set env dc/eap-app LOGGER_CATEGORIES=org.jgroups.protocols.DNS_PING:TRACE
### jgroups as trace
$ oc set env dc/eap-app LOGGER_CATEGORIES=org.jgroups:TRACE

The tracing log will be as follows:

•  dns.DNS_PING: set the discovery protocol.
• logging: Category org.jgroups.protocols.dns with level TRACE. This message will be added in case TRACE is set.
• Initializing DNS Context with factory: com.sun.jndi.dns.DnsContextFactory and URL.
• Resolving DNS query: eap-app-ping of a type: A. The query is sent to eap-app-ping service.
• Collect the entries: eap-app-3-gkcjb: entries collected from DNS (in 2 ms): [127.1.1.1:0, 127.1.1.2:0, 127.1.1.3:0, 127.1.1.4:0]. Collect these.
• Send discovery request: Sending discovery request to 127.1.1.2.:7600. Example request sent.
• Received: GET_MBRS_REQ and GET_MBRS_RSP for each member.
• The node will get GET_MBRS_RSP from other nodes, which will join the cluster: ISPN100000: Node eap-app-pod-2 joined the cluster.
• The reply will be name address and coordinator: sending the response eap-app-pod-1, name=eap-app-pod-1, addr=127.1.1.:7600, coordinator.
• Rebalance: ISPN100002: Starting rebalance with members [eap-app-pod-1, eap-app-pod-2], phase READ_OLD_WRITE_ALL, topology id 2.

A note about JBoss EAP 8

Regarding JBoss EAP 8 container deployment with clustering, the default simple build (such as the following) will not bring clustering capabilities:

### Value file below does not have clustering capabilities:
build:
  uri: https://github.com/jboss-developer/jboss-eap-quickstarts.git
  ref: 8.0.x
  contextDir: helloworld
  env:
  - name: MAVEN_OPTS_APPEND
    value: '-XX:MetaspaceSize=256m -XX:MaxMetaspaceSize=256m'
deploy:
  replicas: 3

To have clustering capabilities, deploy JBoss EAP 8 with a configuration/layer that provides JGroups subsystem, such as cloud-default-config, which is a layer that mimics what used to be present in the JBoss EAP 7.4.x images default installed server.

build:
  uri: https://github.com/jboss-developer/jboss-eap-quickstarts.git
  ref: 8.0.x
  contextDir: helloworld
  env:
  - name: GALLEON_PROVISION_FEATURE_PACKS 
    value: 'org.jboss.eap:wildfly-ee-galleon-pack, org.jboss.eap.cloud:eap-cloud-galleon-pack'
  - name: GALLEON_PROVISION_LAYERS 
    value: 'cloud-default-config'
  - name: GALLEON_PROVISION_CHANNELS 
    value: 'org.jboss.eap.channels:eap-8.0'
deploy:
  replicas: 3  

To verify clustering capabilities see the start of the JBoss EAP pods, see the following.

Disabled:

     INFO Clustering feature is not enabled, no jgroups subsystem present in server configuration.

Enabled:

    INFO Configuring JGroups cluster traffic encryption protocol to SYM_ENCRYPT. 
    WARN Detected missing JGroups encryption configuration, the communication within the cluster WILL NOT be encrypted. 
    WARN No password defined for JGroups cluster. AUTH protocol will be disabled. Please define JGROUPS_CLUSTER_PASSWORD. 
     INFO Configuring JGroups discovery protocol to dns.DNS_PING    

OpenShift Service Mesh Usage and Support

Red Hat OpenShift Service Mesh is not currently supported in JBoss EAP 7 nor in its side-car or ambient mesh configurations. However, evidently, in some circumstances, it can be useful for testing/verifying observability or security as well as access control. Red Hat Service Mesh Operator OpenShift Container Image uses Istio and allows the insertion of the envoy proxy as a sidecar (so two containers in the same pod: JBoss EAP and envoy proxy). The requests and connection exchanges will be done through the envoy. 

What's next

This article described the setting up and operation of JBoss EAP 7 clustering via DNS_PING. The solutions EAP 7 image clustering in OCP 4, EAP 7 clustering deployments boundaries in OCP cluster, and EAP 7 DNS_PING tracing logs in OCP 4 describe the JBoss EAP 7 clustering properties and setting, boundaries of the JBoss EAP 7 DNS_PING services, and DNS_PING tracing usage and output interpretation.

The next installment will address the usage and operation of KUBE_PING as discovery protocol, which uses DNS A or SRV record entries to perform discovery, and it was done specifically for OpenShift/Kubernetes (referenced here).

For any other specific inquiries, please open a case with Red Hat support. Our global team of experts can help you with any issues. Special thanks to Mr. Dennis Reed for his outstanding collaboration and Alexander Barbosa for his review of this article.