What's new from Elastic for DevOps and Big Data?

Elastic is popular for its open-source search experiences among users in the fields of databases and Big Data. It aims at deriving actionable insights from a large pool of data and has gained widespread popularity owing to its reliability, transparency, and ease of use. Elasticsearch is one of the most popular services offered by Elastic. Elasticsearch can be integrated with Hadoop using the ES-Hadoop connect, a two-way connector that will enhance the power of insights drawn from big data.

Elastic has come up with some amazing releases recently, considering the present market scenario and analyzing the requirements of software users in the technology landscape. All these upgrades are aimed toward enhanced performance. Here’s a look at the new releases with which you can gain productive insights that could help you propel your business to the next level.

On the 1st of April 2019, Elastic announced the new Elasticsearch JavaScript client (RC1). This JavaScript client was developed with the aim of overcoming versioning problems faced by the old JavaScript client. It will eliminate the need to re-publish the older versions and the time & effort spent on searching scoped packages. Elastic used package reorganization in which a new package having a new codebase and a correct version number is published. This causes the current un-scoped package to be dependent upon a scoped module which, in turn, points toward the latest version.

Elastic Infrastructure’s 7.0 edition was released on the 10th of April 2019, which tracks and monitors the KPIs of the entire application stack. It is also capable of finding and fixing flaws and deficiencies in the system. This new release is an integration of four new modules which enhance the system’s observability and visibility. It consists of the NATS module, which collects and stores performance metrics. It also includes the Open Source Message System used in Kubernetes and cloud deployments. The SQL Server Module (beta) incorporating log transition metrics and AWE EC2 has direct access to instances in the cloud, which it utilizes to collect utilization resources.

It also performs the important task of monitoring Prometheus time series metrics and alerts the user about undesired fluctuations occurring in it. In case of Kubernetes-based servers and other services which are dependent on such servers, root-cause analysis becomes extremely difficult due to the complicated nature of such systems.

The ECS (Elastic Common Schema) is designed to establish uniformity while transforming a conceptual data model into a logical one for it to be compatible with the database. This improves the efficiency of resource analysis and helps in obtaining meaningful insights from such data. It also simplifies and facilitates the wide variety of operations performed on databases, including search, compare, and correlate, among others.

On the same day, Elastic released Elasticsearch for Apache Hadoop 7.0.0 with an aim to eliminate the need for cascading and further slicing of partitions. Getting rid of cascading further avoids additional costs involved in testing new features that are added to the suite as the technology upgrades. It is important to note that this release will need Java 8 or a higher version to run efficiently.

Logstash 7.0.0, released on the same day, comes with a plethora of plugins, facilitating backward compatibility. It ensures ease of upgrading without the need to change configuration file settings. JRuby is the most preferred implementation of Ruby on Rails owing to its ability to cater to a wide range of Java libraries. Index lifecycle management is another important aspect that is taken care of in this release to grant them control over several management processes. In this version, the facility of auto-detect has been added with the intention of deploying supported ILMs automatically.

Its Kibana 7.0.0 release focuses on looks & appearances and uses a dark theme. It also comes with a more responsive dashboard, canvas enhancements, and structured improvements of saved objects. It loads easily, comes with an attractive font, along with a spacious appearance. You can no longer choose between dark and light modes. However, you can customize the theme by clicking on management and then on the advanced settings option. The font size can also be adjusted as per your preference.

The timepicker simplifies data queries that are difficult to understand. The interface is made compact, making it suitable to be accessed on smartphones. The drop-down menu for filters now describes the actions listed in it. It not only facilitates moving an object conveniently from one Kibana instance to another but also supports exporting dashboards. To make things easier for users, more keyboard shortcuts have now been added. You can access a detailed list of these shortcuts from the help tab. Please note that KQL (Kibana Query Language) is set as default. However, those who prefer Lucene Query Syntax need not worry; they can simply turn KQL off.

Last but not least, Elasticsearch 7.0.0, based on Lucene 8.0.0, is popular for high speeds, safety, and performance. Faster top k retrieval is a prominent feature of this release. It enables tribe nodes to be replaced for aligning and combining queries. It also reduces round trips in cross-cluster searches resulting in faster searches. In the earlier versions, search requests worked in a circular fashion that had a negative impact on search latency. In this release, each node keeps a check on other nodes and internally manages the frequency of search requests. This improves the overall search throughput.

It also provides cross-cluster search frameworks and faster results for default searches. This version comes with major modifications that have reduced OOM (Out of Memory) errors considerably. Moreover, the cluster coordination layer has been reconsidered and rebuilt. The master_nodes_setting in this version has been removed to ease cluster shrinking. It also enhances application safety and minimizes the possibility of misconfiguration. The all-new circuit breaker introduced in this release is a delight for Java wizards. It keeps track of the memory used by JVM and rejects requests if they exceed the set threshold.

Well, that was a small effort from my side to provide you with some overview of these new releases. You may expect more fun & exciting releases from Elastic in the near future.