Splunk on Kubernetes clusters using Cluster Profiles

Splunk integration in Spectro Cloud

Splunk helps search, analyze, and visualize machine-generated data from different sources (websites, applications, sensors, and other devices) and could be an insightful tool to help troubleshoot issues quickly. On Kubernetes platforms, even administrators without a tremendous amount of Kubernetes experience can use Splunk to help troubleshoot applications deployed on the platform.

In this blog, let's look into the following use cases:

• How Splunk can be integrated into Kubernetes clusters provisioned by Spectro Cloud, and
• How to use Splunk for troubleshooting.

Before we deep dive into the integration steps, a quick overview of Splunk plugin architecture:

Splunk Connect for Kubernetes plugin helps forward the following data from Kubernetes clusters to Splunk:

• Logs,
• Metrics, and
• Objects.

This plugin leverages the components:

• Fluentd,
• JQ plugin (For transforming data), and
• Splunk HTTP Event Collector.

Before you begin with the integration, the following prerequisites are required on the Splunk side:

• Use Splunk Enterprise 7.0 or later / Splunk Cloud,
• Setup HTTP Event Collector in Splunk, and
• Have a minimum of two Splunk indexes ready to collect the data.

Deploying Splunk plugin on Kubernetes Cluster

For deploying the Splunk plugin on a Kubernetes cluster provisioned by Spectro Cloud, follow the steps below:

1. Spectro Cloud provides a declarative model for Kubernetes infrastructure layers, called a cluster profile. When creating your cluster profile, select Logging layer > Splunk Connect for Kubernetes.
2. Choose the desired version of the Splunk chart to deploy on the cluster.
3. Update the chart values with your Splunk config details accordingly.
4. Finish the cluster profile.
5. For new clusters, choose the profile which has Splunk integration in the cluster provisioning wizard.
6. For existing clusters, once you add Splunk integration to the profile, clusters will show an update notification. Apply the notification for Splunk Connect for Kubernetes plugin to be deployed on the cluster.

Once deployed, you’ll see the following Splunk related deployments running in the Kubernetes cluster.

And within a few seconds, you’ll see all the logs and other information from your Kubernetes cluster in Splunk:

From here on, you can use Splunk’s native features to troubleshoot issues quickly. For example, to look up all the logs from a specific namespace pods in the last 15 mins, you could use the search feature to filter the information.

Likewise, to find out HTTP 404 (Not found) errors that occurred in the last hour, you could run a wildcard search as shown below. In this example, Splunk will match all the index data that includes 404 patterns.

These tools in Splunk will allow the platform administrators to quickly troubleshoot issues, even when they don’t have much knowledge on what each of the different applications deployed on the Kubernetes cluster does.

The search was just one feature in Splunk. You could also setup alerts to be triggered for specific conditions, generate reports for analytics, and do much more with Splunk integration. One more noteworthy aspect of Splunk is 3rd party app integrations. There are many apps that provide default search patterns and fancy visualizations to make applications troubleshooting a cakewalk.