Skip to main content
Skip table of contents

Securing Elastic Search and Kibana

Feature

To enable secure login for Kibana and secure communication between the Elastic Search data store and Germain Services, the following steps can be followed. Please note that this document assumes Elastic Search is being run using Docker Compose.

CODE 
version: '3.3'
services:

    germain-es1:
        image: docker.elastic.co/elasticsearch/elasticsearch:7.17.6
        restart: always
        mem_limit: 58g
        ports:
            - 9200:9200
            - 9300:9300
        environment:
            node.name: "node01"
            cluster.name: "uhc-elastic-prod"
            discovery.seed_hosts: "node01:9300,node02:9300,node03:9300"
            cluster.initial_master_nodes: "node01,node02,node03"
            bootstrap.memory_lock: "true"
            http.max_content_length: "150MB"
            xpack.security.enabled: "false"
            network.publish_host: "node01"
            transport.publish_port: "9300"
            path.repo: "/usr/share/elasticsearch/backup/uhc_repo"
        ulimits:
            memlock:
                soft: -1
                hard: -1
        volumes:
            - /es1/data:/usr/share/elasticsearch/data
            - /es1/logs:/usr/share/elasticsearch/logs
            - /elasticsearch_backup:/usr/share/elasticsearch/backup
            - /opt/elastic/plugins/germainapm-es-plugin:/usr/share/elasticsearch/plugins/germainapm-es-plugin

    germain-kibana:
        image: docker.elastic.co/kibana/kibana:7.17.6
        restart: always
        ports:
            - 5601:5601
        environment:
            ELASTICSEARCH_HOSTS: "http://node01:9200"

Pre-Requisites

  • Get a PFX File for our certificate update. ( Referring to this file as germainkibana-prod.domain.com.pfx on this document )

  • Generate the key file and cert file using the below OpenSSL commands.

    CODE 
    openssl pkcs12 -in germainkibana-prod.uhc.com.pfx -nocerts -out key.pem -nodes
openssl pkcs12 -in germainkibana-prod.uhc.com.pfx -nokeys -out cert.pem

  • Import the certificate to the existing Java cacert file your current Germain services use.

    CODE 
    keytool -import -trustcacerts -storepass changeit -file /path/to/cert.pem -alias youralias -keystore /path/to/cacerts

    • Alternatively, you can mount the cacert file to the docker container

      CODE 
      volumes:
    # Mount cacerts
    - ${PWD}/path/to/cacerts:/etc/ssl/certs/java/cacerts

  • Generate Keystore to store the key password for SSL using Docker image. To generate this Keystore, we use a docker container and mount the config directory into a local directory so we can extract the elasticsearch.keystore file from the same directory for future use

    CODE 
    docker run -it --rm \
-v /path/to/local-dir/config:/usr/share/elasticsearch/config \
elasticsearch:7.17.6 \
bin/elasticsearch-keystore \
add xpack.security.transport.ssl.keystore.secure_password


docker run -it --rm \
-v /path/to/local-dir/config:/usr/share/elasticsearch/config \
elasticsearch:7.17.6 \
bin/elasticsearch-keystore \
add xpack.security.transport.ssl.truststore.secure_password


docker run -it --rm \
-v /path/to/local-dir/config:/usr/share/elasticsearch/config \
elasticsearch:7.17.6 \
bin/elasticsearch-keystore \
add xpack.security.http.ssl.keystore.secure_password


docker run -it --rm \
-v /path/to/local-dir/config:/usr/share/elasticsearch/config \
elasticsearch:7.17.6 \
bin/elasticsearch-keystore \
add xpack.security.http.ssl.truststore.secure_password

  • Change the elastic search bootstrap password.

    CODE 
    docker run -it --rm \
-v /path/to/local-dir/config:/usr/share/elasticsearch/config \
elasticsearch:7.17.6 \
bin/elasticsearch-keystore \
add bootstrap.password

  • Download a local installation of Kibana and get the kibana.yml file to be mounted to docker. Then change/add the properties as bellow

    CODE 
    elasticsearch.username: "elastic"
elasticsearch.password: "your-super-secret-password"

  • In the same file, uncomment and update properties for SSL

    CODE 
    server.ssl.enabled: true
server.ssl.certificate: /usr/share/kibana/config/cert.pem
server.ssl.key: /usr/share/kibana/config/key.pem
elasticsearch.ssl.certificateAuthorities: [ "/usr/share/kibana/config/cert.pem" ]

During Upgrade

  • Bring down all the Services, Engines and then Elastic Search as well.

  • Enable Elastic Search Security settings by adding the below environment variables

    CODE 
    #security
    http.cors.enabled: "true"
    http.cors.allow-origin: "*"
    xpack.security.enabled: "true"

  • Setup SSL Settings to enable true by adding the below environment variables

    CODE 
    # ssl
    xpack.security.http.ssl.enabled: "true"
    xpack.security.http.ssl.verification_mode: none
    xpack.security.http.ssl.client_authentication: none
    xpack.security.http.ssl.keystore.path: /usr/share/elasticsearch/config/germainkibana-prod.domain.com.pfx
    xpack.security.http.ssl.truststore.path: /usr/share/elasticsearch/config/germainkibana-prod.domain.com.pfx

  • Enable Transport Security Settings by adding the below environment variables

    CODE 
    # transport
      xpack.security.transport.ssl.enabled: "true"
      xpack.security.transport.ssl.verification_mode: none
      xpack.security.transport.ssl.client_authentication: none
      xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/germainkibana-prod.domain.com.pfx
      xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/germainkibana-prod.domain.com.pfx

  • Mount the certificate and keystore files by adding the below volumes

    CODE 
    volumes:
        - ${PWD}/path/to/germainkibana-prod.uhc.com.pfx:/usr/share/elasticsearch/config/germainkibana-prod.uhc.com.pfx
        - ${PWD}/path/to/elasticsearch.keystore:/usr/share/elasticsearch/config/elasticsearch.keystore

  • Update zookeeper config to use secure elastic search

    CODE 
    set /config/germain/application/germain.elastic.url https://node01:9200
set /config/germain/application/germain.elastic.username elastic
set /config/germain/application/germain.elastic.password your-super-secret-password
set /config/germain/application/germain.indexer.url https://node01:9200
set /config/germain/application/germain.indexer.username elastic
set /config/germain/application/germain.indexer.username your-super-secret-password

The final docker-compose file should look like this.

CODE 
version: '3.3'
services:

    germain-es1:
        image: docker.elastic.co/elasticsearch/elasticsearch:7.17.6
        restart: always
        mem_limit: 58g
        ports:
            - 9200:9200
            - 9300:9300
        environment:
            node.name: "node01.domain.com"
            cluster.name: "uhc-elastic-prod"
            discovery.seed_hosts: "node01.domain.com:9300,node02.domain.com:9300,node03.domain.com:9300"
            cluster.initial_master_nodes: "node01.domain.com,node02.domain.com,node03.domain.com"
            bootstrap.memory_lock: "true"
            http.max_content_length: "150MB"
            network.publish_host: "node01.domain.com"
            transport.publish_port: "9300"
            path.repo: "/usr/share/elasticsearch/backup/uhc_repo"
            #security
            http.cors.enabled: "true"
            http.cors.allow-origin: "*"
            xpack.security.enabled: "true"
            # ssl
            xpack.security.http.ssl.enabled: "true"
            xpack.security.http.ssl.verification_mode: none
            xpack.security.http.ssl.client_authentication: none
            xpack.security.http.ssl.keystore.path: /usr/share/elasticsearch/config/germainkibana-prod.domain.com.pfx
            xpack.security.http.ssl.truststore.path: /usr/share/elasticsearch/config/germainkibana-prod.domain.com.pfx
            # transport
            xpack.security.transport.ssl.enabled: "true"
            xpack.security.transport.ssl.verification_mode: none
            xpack.security.transport.ssl.client_authentication: none
            xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/germainkibana-prod.domain.com.pfx
            xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/germainkibana-prod.domain.com.pfx
        ulimits:
            memlock:
                soft: -1
                hard: -1
        volumes:
            - /es1/data:/usr/share/elasticsearch/data
            - /es1/logs:/usr/share/elasticsearch/logs
            - /elasticsearch_backup:/usr/share/elasticsearch/backup
            - /opt/elastic/plugins/germainapm-es-plugin:/usr/share/elasticsearch/plugins/germainapm-es-plugin
            - ${PWD}/path/to/germainkibana-prod.uhc.com.pfx:/usr/share/elasticsearch/config/germainkibana-prod.uhc.com.pfx
            - ${PWD}/path/to/elasticsearch.keystore:/usr/share/elasticsearch/config/elasticsearch.keystore

    germain-kibana:
        image: docker.elastic.co/kibana/kibana:7.17.6
        restart: always
        ports:
            - 5601:5601
        environment:
            SERVER_HOST: 0.0.0.0
            ELASTICSEARCH_HOSTS: "http://node01:9200"
        volumes:
            - /opt/elastic/ssl/kibana/kibana.yml:/usr/share/kibana/config/kibana.yml
            - /opt/elastic/ssl/kibana/cert.pem:/usr/share/kibana/config/cert.pem 
            - /opt/elastic/ssl/kibana/key.pem:/usr/share/kibana/config/key.pem

After Upgrade

  • Start Elastic Cluster and validate

    • SSL Enabled for Kibana

    • SSL Enabled for API

    • Cert is valid

  • Start ActiveMQ, and Zookeeper, then validate Zookeeper config.

  • Storage service and confirm the connection is established successfully.

  • Once confirmed, start all other services.

  • Validate Service status on logs and APM State Section

  • Start Engines

  • Validate data reception and Dashboards

RollBack Plan

  • Bring down all the Services, Engines and then Elastic Search as well.

  • Revert all the docker-compose files back to the original.

  • Start Elastic Search

  • Start Services

  • Validate Services

  • Start Engines

  • Validate data reception and Dashboards

JavaScript errors detected

Please note, these errors can depend on your browser setup.

If this problem persists, please contact our support.

Contact Support Close