Data Ingestion API

Feature

Germain UX exposes a REST API that enables users to insert data into the Germain Datamart. The Ingestion API serves as a means to seamlessly integrate external data sources with Germain UX, allowing users to import data from various systems, applications, or sources directly into the Germain Datamart for analysis and monitoring. By leveraging the Ingestion API, users can programmatically push data into Germain UX, ensuring that the monitoring system stays up to date with the latest information from different data sources. This API facilitates the ingestion of data in a structured manner, ensuring data integrity and consistency within the Germain Datamart.

Whether it's user activity logs, application performance metrics, server logs, or any other relevant data, the Ingestion API provides a convenient and standardized method for importing this data into Germain UX, enabling comprehensive monitoring and analysis of the collected information.

By utilizing the Ingestion API, organizations can leverage the power of Germain UX to gain valuable insights, detect anomalies, optimize performance, and make data-driven decisions based on a holistic view of their systems and applications.

For detailed information on how to use the Ingestion API and integrate external data sources with Germain UX, please refer to the documentation or developer resources provided by the Germain UX platform.

Usage

The primary REST interface of the ingestion API provides a way to submit a list of facts to Germain UX:

CODE

POST /ingestion/fact
[
    { fact },
    { fact },
    ...
]

A status code of 200 indicates that the request was received successfully. In case of an error, a specific error message will be returned by the service.

Data Model Fields

Every fact submitted this way must conform to one of the fact model types available in Germain UX. While each model will have a different set of properties, there are some that are shared across all models. The following example shows common properties shared across fact types:

JSON

[
    {
        "id": "b43bb20b234a93ba00ed16cb",
        "myClassName": "GenericEvent",
        "type": "Custom:Error",
        "timestamp": "2024-03-14T17:38:00Z",
   		"provenance": "S|2023.4|de7ae27c3e4e|ANALYTICS|6b4a1d76-51ef-4d2a-9f0c-67fae88469bd"
		"dataSource: "a92068c",
		"update: false,
		...
    },
    ...
]

Field Name	Required?	Default Value	Description
id	no	yes	Unique identifier for this data point. Stores up to 12 bytes or 24 hexadecimal characters. If not provided, Germain UX will generate a unique ID automatically.
myClassName	yes	no	The name of the mode type of this fact. See Data Model for a list of available types.
type	yes	no	A string value that matches the fact category of the KPI you want to store this data as.
timestamp	yes	no	A timestamp associated with this data point. Timestamp should be formatted as either a ISO-8601 compliant string, or a Unix epoch value with milliseconds.
provenance	no	yes	The provenance value identifies the source that generated this data point. Germain UX components use a hierarchical value that encodes the type, version, and ID of the source that generated the data. You are free to chose a custom value, but we recommend that you prefix it with “X\|” to identify it as such in that case.
dataSource	no	no	The ID of the data source configured in Germain UX that this data was collected for.
update	no	yes	If set to true, will mark this fact as an update to a previous fact with the same ID. Defaults to false.

Examples

Error Event

The following example records a generic error including the affected user and error message.

CODE

curl -X POST 'https://<your germain env URL>/ingestion/fact' \
--header 'Content-Type: application/json' \
--data '[{
    "myClassName": "GenericEvent",
    "type": "System:Error",
    "timestamp": "2024-01-22T14:21:23Z",
    "provenance": "X|integration",
    "name": "SYS-ERR-348",
    "system": {
      "hostname": "XDF1938"
    },
    "user": {
      "name": "admin"
    },
    "details": "Unable to process request. Remaining requests have been cancelled."
  }]'

SQL Statement Execution

The following example records the execution of a SQL query. Specifically, it records which database the query was executed by, the SQL text, and the query duration.

CODE

curl -X POST 'https://<your germain env URL>/ingestion/fact' \
--header 'Content-Type: application/json' \
--data '[{
    "myClassName": "GenericTransaction",
    "type": "SQL:Query",
    "timestamp": "2024-01-22T14:21:23Z",
    "provenance": "X|integration",
    "name": "gqrwquv0utrb7",
    "system": {
      "hostname": "XDF1938"
    },
    "database": {
      "name": "SPROD-1",
      "flavor": "ORACLE"
    },
    "user": {
      "name": "perftest"
    },
    "duration": 0.562,
    "details": "select * from COMMON_TABS where QNAME = ? order by LAST_MOD"
  }]'

Mulesoft API Metric

The following example creates a data point in GermainUX each time a “SR Update” occurs in Mulesoft.

CODE

curl -X POST 'https://<your germain env URL>/ingestion/fact' \
--header 'Content-Type: application/json' \
--data '[{
        "myClassName": "GenericMetric",
        "type": "Mulesoft:APIMetric",
        "name": "SR Update",
        "timestamp": "2022-05-16T20:54:40Z",
        "value": 500,
        "database":{
            "name":"1234-1234554-134514-145",
            "flavor":"ORACLE"
        },
        "pid":"320",
        "application": {
            "component": "4.3.2",
            "name": "CSMulesoft-Cases-api"
        },
        "system": {
            "name": "CSSendSMS",
            "hostname": "Platform"
        }
    }]'

In GermainUX:

A datapoint gets inserted into GermainUX’s data warehouse, and it looks like this on a GermainUX's drillthrough dashboard:

Authentication

No authentication is required to submit new facts. In order to secure the ingestion REST endpoints, we recommend setting up IP-range ACLs at the level of your load balancer or firewall. Please contact our support team to help you configure this.

Feature Availability: 2022.1 or later