There are many areas where we can use the Elastic Stack, such as logging where we mainly use Elastic Stack or for searching using Elasticsearch or for dashboarding for monitoring but these are just a few use case of the Elastic Stack which we primarily use, there are many other areas where we can use the power of Elastic Stack. We can use Elastic Stack for the following use cases:

• System Performance Monitoring
• Log Management
• Application Performance Monitoring
• Application Data Analysis
• Security Monitoring and Alerting
• Data Visualization

Let's discuss each of these in detail.

When we run any application in production, we need to make it stable by avoiding anything that can impact the application's performance; this can be anything, such as the system, database, or any third-party dependencies, since if anything fails it impacts the application. In this section, we'll see how system monitoring can help us to avoid situations where the system can cause the application to outage.

Let's discuss the factors that can hamper application's performance. There can be number of reasons, such as the system memory or CPU creating a bottleneck because of an increase in user hits. In this situation, we can do multiple things, such as optimizing the application if it's possible and increasing the memory or CPU. We can do it to mitigate the outrage of the application, but it's only possible if we're monitoring the system metrics of the servers on which the application has been deployed. Using the monitoring, we can configure the alert whenever the threshold value of any component increases. In this way, you can protect yourself from any application outage because of system performance.

Log Management is one of the key use cases of Elastic Stack, and it has been primarily used for this purpose for many years. There are many benefits of log management using Elastic Stack, and I'll explain Elastic Stack simplifies things when it comes to monitoring logs. Let's say you have a log file and you need to explore it to get the root cause of the application outage – how are you going to proceed? Where will you open the file and what are you going to search and filter? We just need to push the log data into Elasticsearch and configure Kibana to read this data. We can use Filebeat to read the log files, such as Apache access and error logs. Apart from system logs, we can also configure Filebeat to capture application logs. Instead of Filebeat, we can use Logstash to take file data as input and output it to Elasticsearch.

Elastic Stack APM monitors applications and helps developers and system administrators monitor software applications for performance and availability. It also helps them to identify any current issues, or ones that may occur in the near future. Using APM, we can find and fix any bug in the code, as it makes the problems in the code searchable. By integrating APM with our code, we can monitor our code and make it better and more efficient. Elastic APM provides us with custom preconfigured dashboards in Kibana. We can integrate application data using APM and server stats, network details, and log details using Beats. This makes it easy to monitor everything in a single place.

We can apply machine learning to APM data by using the APM UI to find any abnormal behavior in the data. Alerts can also be applied to get an email notification if anything goes wrong in the code. Currently, Elastic APM supports Node.js, Python, Java, Ruby, Go, and JavaScript. It's easy to configure APM with your application and it requires only a few lines of code.

With X-Pack, we can enable security, alerting, and monitoring with our Elastic setup. These features are very important and we need them to protect our Elastic Stack from external access and any possible issues. Now let's discuss each of them in detail.

Security is a very important feature of X-Pack; without it, anyone can open the URL and access everything in Kibana, including index patterns, data, visualizations, and dashboards. During X-Pack installation and setup, we create the default user credentials. For security, we have role management and user management, using which we can restrict user access and secure the Elastic Stack.

Monitoring provides us the insight on Elasticsearch, Logstash, and Kibana. Monitoring comes with X-Pack, which we can install after installing the basic Elastic Stack setup. Monitoring-related data is stored in Elasticsearch, which we can see from Kibana. We have built-in status warning in Kibana, custom alerts can be configured on data in the indices used for monitoring.

Elastic Stack uses alerting to keep an eye on any activity, such as whether CPU usage increases, memory consumption goes beyond some threshold, the response time of an application goes up, or 503 errors are increasing. By creating alerts, we can proactively monitor the system or application behavior and can apply a check before anything actually goes wrong.

Using alerts, we can notify every stakeholder without missing anything. We can apply alerts to detect specific issues, such as a user logged in from a different location, credit card numbers are showing in application logs, or the indexing rate of Elasticsearch increases. These are just some examples; we can apply alerts in so many cases.

There are different ways to notify the users, as there are lots of built-in integrations available for emails, slack, and so on. Apart from these built-in options, we can integrate alerts with any existing system by integrating the webhook output provided by Elastic Stack. Alerts also have simple template support, which we can use to customize the notification. I'll cover how we can configure the alerts in the coming chapters.

Data visualization is the main feature of Kibana and it's the best way to get information from the raw data. As we know, a picture tells a thousand words, so we can easily learn about a data trend by just seeing a simple chart. Kibana is popular because it has the ability to create dashboards for KPIs using data from different sources; we can even use Beats to get ready—made dashboards. We have different types of visualizations in Kibana, such as basic charts, data, time-series, and maps, which we'll cover in coming chapters. If we have data in Elasticsearch, we can create visualizations by creating index patterns in Kibana for those indexes in Elasticsearch.