Skip to main content
Skip table of contents

Securing Elastic Search and Kibana

Feature

To enable secure login for Kibana and secure communication between the Elastic Search data store and Germain Services, the following steps can be followed. Please note that this document assumes Elastic Search is being run using Docker Compose.

CODE
version: '3.3'
services:

    germain-es1:
        image: docker.elastic.co/elasticsearch/elasticsearch:7.17.6
        restart: always
        mem_limit: 58g
        ports:
            - 9200:9200
            - 9300:9300
        environment:
            node.name: "node01"
            cluster.name: "uhc-elastic-prod"
            discovery.seed_hosts: "node01:9300,node02:9300,node03:9300"
            cluster.initial_master_nodes: "node01,node02,node03"
            bootstrap.memory_lock: "true"
            http.max_content_length: "150MB"
            xpack.security.enabled: "false"
            network.publish_host: "node01"
            transport.publish_port: "9300"
            path.repo: "/usr/share/elasticsearch/backup/uhc_repo"
        ulimits:
            memlock:
                soft: -1
                hard: -1
        volumes:
            - /es1/data:/usr/share/elasticsearch/data
            - /es1/logs:/usr/share/elasticsearch/logs
            - /elasticsearch_backup:/usr/share/elasticsearch/backup
            - /opt/elastic/plugins/germainapm-es-plugin:/usr/share/elasticsearch/plugins/germainapm-es-plugin

    germain-kibana:
        image: docker.elastic.co/kibana/kibana:7.17.6
        restart: always
        ports:
            - 5601:5601
        environment:
            ELASTICSEARCH_HOSTS: "http://node01:9200"

Pre-Requisites

  • Get a PFX File for our certificate update. ( Referring to this file as germainkibana-prod.domain.com.pfx on this document )

  • Generate the key file and cert file using the below OpenSSL commands.

    CODE
    openssl pkcs12 -in germainkibana-prod.uhc.com.pfx -nocerts -out key.pem -nodes
    openssl pkcs12 -in germainkibana-prod.uhc.com.pfx -nokeys -out cert.pem
  • Import the certificate to the existing Java cacert file your current Germain services use.

    CODE
    keytool -import -trustcacerts -storepass changeit -file /path/to/cert.pem -alias youralias -keystore /path/to/cacerts
    • Alternatively, you can mount the cacert file to the docker container

      CODE
      volumes:
          # Mount cacerts
          - ${PWD}/path/to/cacerts:/etc/ssl/certs/java/cacerts
  • Generate Keystore to store the key password for SSL using Docker image. To generate this Keystore, we use a docker container and mount the config directory into a local directory so we can extract the elasticsearch.keystore file from the same directory for future use

    CODE
    docker run -it --rm \
    -v /path/to/local-dir/config:/usr/share/elasticsearch/config \
    elasticsearch:7.17.6 \
    bin/elasticsearch-keystore \
    add xpack.security.transport.ssl.keystore.secure_password
    
    
    docker run -it --rm \
    -v /path/to/local-dir/config:/usr/share/elasticsearch/config \
    elasticsearch:7.17.6 \
    bin/elasticsearch-keystore \
    add xpack.security.transport.ssl.truststore.secure_password
    
    
    docker run -it --rm \
    -v /path/to/local-dir/config:/usr/share/elasticsearch/config \
    elasticsearch:7.17.6 \
    bin/elasticsearch-keystore \
    add xpack.security.http.ssl.keystore.secure_password
    
    
    docker run -it --rm \
    -v /path/to/local-dir/config:/usr/share/elasticsearch/config \
    elasticsearch:7.17.6 \
    bin/elasticsearch-keystore \
    add xpack.security.http.ssl.truststore.secure_password
  • Change the elastic search bootstrap password.

    CODE
    docker run -it --rm \
    -v /path/to/local-dir/config:/usr/share/elasticsearch/config \
    elasticsearch:7.17.6 \
    bin/elasticsearch-keystore \
    add bootstrap.password
  • Download a local installation of Kibana and get the kibana.yml file to be mounted to docker. Then change/add the properties as bellow

    CODE
    elasticsearch.username: "elastic"
    elasticsearch.password: "your-super-secret-password"
  • In the same file, uncomment and update properties for SSL

    CODE
    server.ssl.enabled: true
    server.ssl.certificate: /usr/share/kibana/config/cert.pem
    server.ssl.key: /usr/share/kibana/config/key.pem
    elasticsearch.ssl.certificateAuthorities: [ "/usr/share/kibana/config/cert.pem" ]

During Upgrade

  • Bring down all the Services, Engines and then Elastic Search as well.

  • Enable Elastic Search Security settings by adding the below environment variables

    CODE
    #security
        http.cors.enabled: "true"
        http.cors.allow-origin: "*"
        xpack.security.enabled: "true"
  • Setup SSL Settings to enable true by adding the below environment variables

    CODE
    # ssl
        xpack.security.http.ssl.enabled: "true"
        xpack.security.http.ssl.verification_mode: none
        xpack.security.http.ssl.client_authentication: none
        xpack.security.http.ssl.keystore.path: /usr/share/elasticsearch/config/germainkibana-prod.domain.com.pfx
        xpack.security.http.ssl.truststore.path: /usr/share/elasticsearch/config/germainkibana-prod.domain.com.pfx
  • Enable Transport Security Settings by adding the below environment variables

    CODE
    # transport
          xpack.security.transport.ssl.enabled: "true"
          xpack.security.transport.ssl.verification_mode: none
          xpack.security.transport.ssl.client_authentication: none
          xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/germainkibana-prod.domain.com.pfx
          xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/germainkibana-prod.domain.com.pfx
  • Mount the certificate and keystore files by adding the below volumes

    CODE
    volumes:
            - ${PWD}/path/to/germainkibana-prod.uhc.com.pfx:/usr/share/elasticsearch/config/germainkibana-prod.uhc.com.pfx
            - ${PWD}/path/to/elasticsearch.keystore:/usr/share/elasticsearch/config/elasticsearch.keystore
  • Update zookeeper config to use secure elastic search

    CODE
    set /config/germain/application/germain.elastic.url https://node01:9200
    set /config/germain/application/germain.elastic.username elastic
    set /config/germain/application/germain.elastic.password your-super-secret-password
    set /config/germain/application/germain.indexer.url https://node01:9200
    set /config/germain/application/germain.indexer.username elastic
    set /config/germain/application/germain.indexer.username your-super-secret-password

The final docker-compose file should look like this.

CODE
version: '3.3'
services:

    germain-es1:
        image: docker.elastic.co/elasticsearch/elasticsearch:7.17.6
        restart: always
        mem_limit: 58g
        ports:
            - 9200:9200
            - 9300:9300
        environment:
            node.name: "node01.domain.com"
            cluster.name: "uhc-elastic-prod"
            discovery.seed_hosts: "node01.domain.com:9300,node02.domain.com:9300,node03.domain.com:9300"
            cluster.initial_master_nodes: "node01.domain.com,node02.domain.com,node03.domain.com"
            bootstrap.memory_lock: "true"
            http.max_content_length: "150MB"
            network.publish_host: "node01.domain.com"
            transport.publish_port: "9300"
            path.repo: "/usr/share/elasticsearch/backup/uhc_repo"
            #security
            http.cors.enabled: "true"
            http.cors.allow-origin: "*"
            xpack.security.enabled: "true"
            # ssl
            xpack.security.http.ssl.enabled: "true"
            xpack.security.http.ssl.verification_mode: none
            xpack.security.http.ssl.client_authentication: none
            xpack.security.http.ssl.keystore.path: /usr/share/elasticsearch/config/germainkibana-prod.domain.com.pfx
            xpack.security.http.ssl.truststore.path: /usr/share/elasticsearch/config/germainkibana-prod.domain.com.pfx
            # transport
            xpack.security.transport.ssl.enabled: "true"
            xpack.security.transport.ssl.verification_mode: none
            xpack.security.transport.ssl.client_authentication: none
            xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/germainkibana-prod.domain.com.pfx
            xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/germainkibana-prod.domain.com.pfx
        ulimits:
            memlock:
                soft: -1
                hard: -1
        volumes:
            - /es1/data:/usr/share/elasticsearch/data
            - /es1/logs:/usr/share/elasticsearch/logs
            - /elasticsearch_backup:/usr/share/elasticsearch/backup
            - /opt/elastic/plugins/germainapm-es-plugin:/usr/share/elasticsearch/plugins/germainapm-es-plugin
            - ${PWD}/path/to/germainkibana-prod.uhc.com.pfx:/usr/share/elasticsearch/config/germainkibana-prod.uhc.com.pfx
            - ${PWD}/path/to/elasticsearch.keystore:/usr/share/elasticsearch/config/elasticsearch.keystore

    germain-kibana:
        image: docker.elastic.co/kibana/kibana:7.17.6
        restart: always
        ports:
            - 5601:5601
        environment:
            SERVER_HOST: 0.0.0.0
            ELASTICSEARCH_HOSTS: "http://node01:9200"
        volumes:
            - /opt/elastic/ssl/kibana/kibana.yml:/usr/share/kibana/config/kibana.yml
            - /opt/elastic/ssl/kibana/cert.pem:/usr/share/kibana/config/cert.pem 
            - /opt/elastic/ssl/kibana/key.pem:/usr/share/kibana/config/key.pem

After Upgrade

  • Start Elastic Cluster and validate

    • SSL Enabled for Kibana

    • SSL Enabled for API

    • Cert is valid

  • Start ActiveMQ, and Zookeeper, then validate Zookeeper config.

  • Storage service and confirm the connection is established successfully.

  • Once confirmed, start all other services.

  • Validate Service status on logs and APM State Section

  • Start Engines

  • Validate data reception and Dashboards

RollBack Plan

  • Bring down all the Services, Engines and then Elastic Search as well.

  • Revert all the docker-compose files back to the original.

  • Start Elastic Search

  • Start Services

  • Validate Services

  • Start Engines

  • Validate data reception and Dashboards

JavaScript errors detected

Please note, these errors can depend on your browser setup.

If this problem persists, please contact our support.