Elasticsearch Database (Preferred)
Feature
Platform for real-time search, analytics, and logging. Germain prefers Elastic over any other datastore.
Percentiles or Std Deviation need row data
Percentiles or std deviation measures are not supported by Elastic’s roll-up mechanism; only min, max, avg, sum, count are supported. Percentiles or std deviation are supported by germain/Elastic as long as raw data is there.
Roll-ups
Germain’s "aggregation" equates to "roll-up" in Elastic; same basic idea, compress data over time by reducing detail level.
Built-in support for roll-up at configurable time, into variable time windows.
For now, germain rolls up into hourly index that can be kept past raw data window.
Single API call to query both raw and aggregated indexes, results are merged automatically.
Timeseries Indexes
Timeseries datastreams are generally append only / write once; in order to update data, we either use Elastic API or directly push update to underlying index.
For our fact datastreams, we use 1-day "hot" storage, afterwards read-only "cold" storage for as long as raw retention is configured.
Configuration
Basic high-level process to setup Elasticsearch as data store for Germain. Please contact us for script and more details.
Prepare
Get host for ES and Germain
Get DB for Germain config (SQL DB)
Export config json from Stage Germain
Create a Temporary Germain Server
It will be used to setup the Elasticsearch database pre data migration.
It will need a SQL database for the config
In
common.properties
:Change the queue names to have new prefixes so that it can use the same ActiveMQ without interfering
Change the
germain.elastic.properties.indexPrefix
to be more meaningful (Stage)Configure the connection to ES
No Services will be needed
Setup Elasticsearch DB
Make sure the Elastic DB host is the same timezone as the data source (Central Time)
Import the search indexes from
servicesDistro/install/indexer/es
Prepare the DB
Start Germain
Verify it connects okay in DB
Use special REST endpoint that applies the rest of the config for Germain to ES DB
Data Migration Script
Fill in the details in the migration tools .yml file
Run the script one per data type. Arguments as per examples (same folder as tool)
Multiple can be run in parallel
It might adapt a script/tool to automate
Query Validation for Elastic
Germain supports the validation of Elastic Search queries
Go to Germain Workspace > Left Menu > System > Engine Settings > Component Types
Click :plus: icon
Select Elasticsearch Query Monitor Component

Create new component on Component Types page

Elasticsearch Query Monitor Component Wizard