Performance optimizations & a whole new world of analytics - GA
We are pleased to announce the general availability of Germain 8.5 and the new architecture that it entails, which makes Germain significantly faster and opens up a whole new world in terms of the types of data analysis, correlation and visualization that can be achieved with this release.
WITHOUT OPTIMIZATION (Germain 8.4 on Oracle DB this time)
Click on "Drill-through" (of Germain dashboard) on "Business Svc Txns" portlet takes timed out after 10 minutes
Volume for 1 day: 16 million rows
Showing 100 rows, sort by duration
Database: Oracle Enterprise Edition 12.1.0.2.v10
Database hardware: Amazon RDS db.r3.xlarge
WITH OPTIMIZATION (Germain 8.5)
Click on "Drill-through" (of Germain dashboard) on "Business Svc Txns" portlet takes 4.2 second
Volume for 1 day: 16 million rows
Showing 100 rows, sort by duration
Database: Oracle Enterprise Edition 12.1.0.2.v10
Database hardware: Amazon RDS db.r3.xlarge
What did we do in this release that significantly improved performance?
We used both Mondrian (Top-level/Aggregate and Pivot dashboards) and JPA (for Drill-through and RCA dashboards) to generate SQL to run against Germain's datamart; this new 8.5 architecture eliminates both of these systems and provides a unified query layer instead that can service all queries.
- For high-volume data, we are using dedicated collections (on traditional RDBMs that means tables, on NoSQL there are actual collections) instead of storing data together with similar data types. For example, before this release, we used to store all transaction data (User clicks, Workflows, Business service, SQL) in a single table – once that table grew, performance degraded. In this release, each of these types of data can be written to a dedicated collection, which will be smaller and perform better.
- Better aggregation – replaced the way data is aggregated, making it now easy and quick to add any data type for aggregation.
Any other optimization to come?
Optimization and improvement of what we do is an ongoing process and we prioritize our work based on our customers' needs. This 8.5 release is the start of a new architecture. A number of other performance optimizations are in our roadmap (short and mid-term; priority of implementing these additional optimizations will depend on whether our customers need them or not). Why have we not included them into this release? some of our customers wanted to get a significantly faster dashboard ASAP, this 8.5 release is delivering on that premise. Yet, as customers increase their use of Germain, they may potentially (in many 100 of millions of analytics per day later..) need these new optimizations. Because our customers are our top priority, we will deliver these as soon as we feel they need.
Outlook with this new architecture?
This new architecture opens up a whole new world of analytics capabilities in Germain. Most analytics that Germain couldn't deliver before are now supported (or in the process of being supported) with this new architecture. For instance, percentiles (99.5%, 99.9%,..) are now supported. Raw metrics can now be made available on Top-level dashboard. In addition, you are going to start seeing many new analytics, correlation and visualization capabilities in subsequent releases. NoSQL data-store support is now easier for us to provide as Germain is now big-data native (e.g. mongoDb is coming next, in addition to MemSQL in-memory datastore that is supported).
What to not forget?
- Databases (RDBMS and noSQL) need to be maintained, so customers are required to do their due diligence there and follow the database maintenance best-practices that are typically required. As always, we are available to provide any guidance, at no additional charge.
- Performance results (e.g. 4.2 sec response time mentioned above, etc) will vary based on your database hardware and software; the same SQL takes 0.05 millisecond on mySQL for instance; yet it can also be slower on any database, if you are running a slower database hardware/storage)