Feature

Platform for real-time search, analytics, and logging. Germain prefers Elastic over any other datastore.

Percentiles or Std Deviation need row data

Percentiles or std deviation measures are not supported by Elastic’s roll-up mechanism; only min, max, avg, sum, count are supported. Percentiles or std deviation are supported by germain/Elastic as long as raw data is there.

Roll-ups

Germain’s "aggregation" equates to "roll-up" in Elastic; same basic idea, compress data over time by reducing detail level.

  • Built-in support for roll-up at configurable time, into variable time windows.

  • For now, germain rolls up into hourly index that can be kept past raw data window.

  • Single API call to query both raw and aggregated indexes, results are merged automatically.

Timeseries Indexes

Timeseries datastreams are generally append only / write once; in order to update data, we either use Elastic API or directly push update to underlying index.

For our fact datastreams, we use 1-day "hot" storage, afterwards read-only "cold" storage for as long as raw retention is configured.

Configuration

Basic high-level process to setup Elasticsearch as data store for Germain. Please contact us for script and more details.

Prepare

  1. Get host for ES and Germain

  2. Get DB for Germain config (SQL DB)

  3. Export config json from Stage Germain

Create a Temporary Germain Server

It will be used to setup the Elasticsearch database pre data migration.

  1. It will need a SQL database for the config

  2. In common.properties:

    1. Change the queue names to have new prefixes so that it can use the same ActiveMQ without interfering

    2. Change the germain.elastic.properties.indexPrefix to be more meaningful (Stage)

    3. Configure the connection to ES

  3. No Services will be needed

Setup Elasticsearch DB

  1. Make sure the Elastic DB host is the same timezone as the data source (Central Time)

    1. Import the search indexes from servicesDistro/install/indexer/es

  2. Prepare the DB

    1. Start Germain

      1. Verify it connects okay in DB

    2. Use special REST endpoint that applies the rest of the config for Germain to ES DB

Data Migration Script

  1. Fill in the details in the migration tools .yml file

  2. Run the script one per data type. Arguments as per examples (same folder as tool)

    1. Multiple can be run in parallel

    2. It might adapt a script/tool to automate

Query Validation for Elastic

Germain supports the validation of Elastic Search queries

  1. Go to Germain Workspace > Left Menu > System > Engine Settings > Component Types

  2. Click :plus: icon

  3. Select Elasticsearch Query Monitor Component

Create new component on Component Types page

Create new component on Component Types page

Elasticsearch Query Monitor Component

Elasticsearch Query Monitor Component Wizard