SPLK-1003 Splunk Enterprise Certified Admin – Forwarder Management And User Management

1. Splunk Forwarder Management

The forwarder management mainly focuses on your universal forwarders. So in this we’ll be configuring forwarder management and also a deployment manager which acts as a forwarding forwarder management so that it manages your universal forwarder along with your other deployment clients like Indexes. Our chats avoided us these Splunk components which can also be managed by our deployment manager. So this will be our central management control, so that we’ll be using this server for configuring any configurations on the Splunk, so that we need not log into any of the Splunk instance anymore once the deployment manager has been set up.

And also your forwarder management can be used as a deployment management console where you can check the health of all other Splunk components like Indexer, AV, forwarder searches and including your forwarders and these forwarders and also your other Splunk components communicate to your deployment server using a terminology called Phoning home.

This is nothing, but each forwarder contacts your deployment server for checking for updates and also reports its health status every 30 seconds. That is by default this can be increased or decreased depending on the need. And in our next lab session we’ll be seeing how we can increase this phoning home interval and what are the benefits of this.

And also we’ll see how to create our own server groups and what are the criteria when we are defining our server groups. And we’ll also notice how we can check for the apps deployed for this server groups. Let’s say we have created one server group called Windows and another server group called Linux. We can check for the deployment errors using forwarder management. And also we can deploy Windows app to the Windows group of servers, Linux app to the Linux group of servers, all from single point.

2. Creating ServerClass.conf File

In this video we’ll be going through an example server class which is used for creating our deployment server and also for managing the deployment clients. So we’ll see what are the syntax that we need to follow for creating a successful server class. And also remember, Server Class is used for creating server groups based on technology or based on the application they are running or based on specific location they are located. It’s completely upon splunk, admin or architect who decides? These servers or these deployment clients should be grouped as one so that we can deploy the configuration using single group. We’ll see an example. So here I’ve created one server class conf entry of how it looks typically in a production environment. As you can see, I have commented out the file just to give a brief description about what configuration I’m doing. I’m creating a first server group that is named as Server Group Name. So this will be my group name. I’ll change it to application servers. As you can see, Server Class, this tanza represents the group of servers.

So here you can blacklist everything that is dot star refers to all other client except these three clients which are whitelisted. So these three group of servers now belongs to application servers group. Now any configuration we need to deploy to this application servers, we need to create an entry of another server class where it mentions the group name that is our application servers and the application name preceded by a syntax called App.

So this app and server clause is the syntax that we need to follow in order to specify the group names and the application name whitelist is used for grouping the devices based on any technology or similarity in their function or configuration that are deployed on these devices. And this application will be deployed only on these whitelisted application servers. So what happens once the application is deployed is defined under this stanza. That is, it will restart the Splunk demon process once the application has been deployed using deployment server. If we set this to false after application deployment, the server will not be restarted.

That means this is not the actual server, it is the service that the application restart once your configuration has been deployed through your deployment server. Now let us see how we can edit this configuration file. I’ll just copy this configuration file that we just gone through go to our Splunk deployment server. So our sum Splunk deployment server contains the configuration under System local server class. So this is the file where all the groups are defined as. You can see as part of configuring this instance as a deployment server, I believe in the third course of our discussion, we have edited this configuration. We have created a group called All Apps and this group consists of all the application. But at present we have not added any clients to these server class. So it is kind of a dummy configuration where it is presently not effective and it is not deploying any configuration.

3. ServerClass and DeploymentClient Configuration Files

So we’ll input our configuration that we just gone through into our server classcom. As you can see now we have created a new application servers with the client server. One, we can also mention IP address. We can also mention fully qualified domain name which represents your or the deployment clients. So let us save and restart our deployment server. Once restarted, we should be able to see our newly created group and also the name of the application. It has restarted. Let us go back and refresh our deployment server. So once you log in, click on Settings Forward a management. You should be able to see the newly created server group that is Application Server. As you can see, we have created new group that is Application server.

Now we can add application into this repository and we’ll see how we can pop up some values. Here it says the repository location is our deployments app. That is by default, we have not changed this location. Now let us put some apps here. So if you go to opt Splunk, this is your Splunk home directory under Etc deployment apps. This is where all your application has to be placed so that it can be deployed to your clients. As you can see in the README document, this directory is the default repository location for deployable apps in the deployment server. So this apps will be deployed into your searches indexers, Universal Forwarders wherever you need these apps to be deployed, you can control it via your deployment server.

In order to deploy applications or configurations using deployment server, we need to meet two specific requirements. That is, we need to know where we need to deploy our application that our clients and which applications are supposed to deploy it on the clients. So as of now, we don’t have any clients reporting to our deployment server. Let us go ahead and configure a couple of clients to our deployment server. As part of our configuration, we’ll make our index server as the first client to our deployment server. We have seen from our previous videos how to add deployment server by multiple ways. In this video, I’ll be using Splunk CLI to add deployment clients report to deployment server. So it is asking for my admin credentials of Splunk configuration as updated. So we will add one more client that will be our universal forwarder. So, we all know the universal Forwarder. We have installed it on our local machine, that is local laptop. All other machines with respect to Splunk, Indexes, Searched, Heavy Forwarder and Deployment server are all present on the Amazon cloud. Let me quickly check whether we have our universal forwarder up and running. Okay, it has stopped. Let us start it. So once our universal forwarding is up, we’ll upload the deployment client on this universal forwarder also so that we should be able to see two clients on our deployment server.

4. Apps on Deployment Server

It is up. Let me add deployment client using Splunk CLI. This is our deployment server IP and 80 89 is our management port. The syntax is IP colon port. Okay, we are successfully updated configuration on our universal forwarder and our Splunk indexer. Now let us see whether we are receiving any clients on deployment server. So once we refresh, we should be able to see any communications happening between our clients and the deployment server. As you can see, now we have one instance, that is our indexer which is hosted on Amazon AWS is able to report to our client without any issues. So in the meantime, we should be able to see our universal forwarder also reporting to our deployment server. Let us refresh it once more.

Now, it might take a few more minutes in order to see our universal forwarder under our client list. So usually it takes around a couple of minutes, two to five minutes in order to see your deployment client. Now we are able to see our universal forwarder which is installed on my laptop, able to successfully report to a deployment server. Now we got our first prerequisite ready, that is to deploy the configurations we got our client. Now we need an applications. So for this application, we’ll be using the Splunk. com which we created as part of our application creation tutorial. So this application will be deploying it on our index. In order to deploy that, we need this application to be under etc deployment apps of our deployment server. So let us go ahead and copy. This is our deployment server.

I’ve logged in. Let me log in to my application user. Let me check. Splunk is up. Same should be up. So I’ve already copied our application under Temp. Yes, this is our application. Now let us copy this application under our deployment apps. It is deployment apps. So now we have our first application to be deployed. Let us reload our deployment server. As you can see, we have our Splunk demo app has been deployed. So here we have a couple of actions. One, we can make sure what happens when this application happens. So this application after installation, it will be under Enable State.

If you don’t want this to enable, as soon as installed, you can choose disabling it, that is by unchecking the Enable app option. If you check this, as soon as the application is deployed, the configuration will take effect. And also we have another option whether to restart the Splunk demon that is your Splunk or application. Once our application has been deployed, that is whichever the configuration which is packaged as an application which requires Splunk to be restarted can be made this option as restart Splunk D. And also this option is used for making a Splunk demo that is our own app to be deployed under which server group. We don’t need it to be under application servers or all apps. We’ll be creating our own group called Index. We’ll be deploying it on that.

5. Deploying Apps using Deployment Server

Now we have our clients and application ready. Let us create a server group that is called as server class in Splunk and I’ll name this as Indexers. So for Indexes I’ll be deploying our own custom created app. As you can see we have successfully created server group group called Indexers. Now I’ll add an applications that is Splunk demo we have successfully added click on Save. As you can see this indexer group has this Splunk demo. Now there is an option add Clients. So this option we can choose which client we can pick and you can also whitelist based on Hostname IP address or DNS or even including your wild cards we’ll be adding based on Hostname.

This is our host name. Once you have added Hostname, click on Preview. You’ll be able to see this checkmark or choose on matched buttons so that you can see all the instances or the clients that are coming under your whitelist. You can also explicitly mention any blacklist although by default all other apps or all other clients which are not matching this criteria will be discarded. And also you can match by filter that is using Windows or Linux machine. As you can see we have one Windows client and one Linux client so this is OS based match filter which will be very useful while deploying add ons and input configurations. I’ll click on save. Now as soon as I click on Save you should be able to see we have one client in our server group and one app as part of this group. So we have 100% clients deployed apps successfully.

As we click on Save the app was copied into our indexes and it was deployed. Let us go back to our forwarder management console. We’ll be able to see that a few seconds ago that was when our indexer reported to our deployment server and picked up the configuration to deploy one app. Now let us log into our indexer and verify whether we have our new configuration as part of for configuration deployment using Deployment Server all the apps that are deployed using your deployment server will be under etc app of your client but the server deployment server holds this under etc deployment apps on the client. It will be deployed under etc apps as you can see now we have our new Splunk. com demo application deployed on our indexer.

Now let us quickly log in and validate how it is. Let us log into our indexer Splunk. com that is our demo application which was as part of our search and we have now successfully deployed without taking much of a SL copying apps and reloading your Splunk instance or deploying the configuration. We have all done this using deployment server with using just your forwarder management console. Always remember when you have selected an application that which requires restarting of Splunk D by default when you deploy this application, your Splunk process on any server will be restarted.

6. Creating Server Groups Using ServerClass.conf

Now we have understood a complete picture of how the back end syntax looks for a server class file on your deployment server. Now let us see how we can group your clients based on server groups whitelist and blacklisting based on IP address, domain name or your host name. We will see this using Splunk web. You can create your server class. com using Splunk web. In order to do that, go to your deployment server. So let us quickly log into our deployment server. Upon logging in, click on Settings. Select Forwarder Management. Under Forwarder Management you should be able to see all your clients reporting to this deployment server. As you can see there is a Windows machine and there is a Linux machine. So this is our indexer, this is our universal forwarder installed on laptop. So here we have three server classes. One is Indexer application server and all apps.

We will create one more server class and see how we can filter out based on IP address, hostname and also OS type. So let us create a server class for demonstration. You can give this any name you want. You can give it Windows servers, Linux servers, application servers, database servers, staging QA you can give it anything based on technology or any grouping that makes sense to deploy the configuration at once. We’ll make this as demonstrating Server class group. Click on say it will be taken into a different screen where you can whitelist or blacklist your clients.

So here we’ll be seeing under Ad Clients menu click on Add Clients. Under Ad Clients you’ll be able to see all your clients which are reporting as part of your deployment server. You can include an IP address or a domain name that is complete DNS name with a wild character. I can say Star Splunk. com so that all my PCs will become under Demonstration Server class group. Similarly, I can add IP with a wild card that is something like this where you can add with wildcards all your IPS belonging to a subnet can be part of our demonstration server group. And if you would like to do a whitelist, you can add n number of whitelisting and by default all the servers to any server group under blacklist that is by default blacklist will be dot star that refers to all the machines.

Similarly, you can manually blacklist some of the application. Let’s say I have a subnet defined ten dot zero dot zero dot zero slash 24 but I need two IPS out of this blacklist which doesn’t require to be as part of this group. I can blacklist them separated by comma. As you can see we are creating a group for ten subnet and here we are blacklisting two IPS out of these subnet so that these two IPS will not get any configuration under this group. So we have rest of the servers under this subnet will get the configuration as part of this group. Let’s say we don’t need whitelist, we don’t need blacklist. We are creating a group called Windows Demonstration Group. So here you can deploy Windows configuration by choosing the OS type. I have selected Windows 64. If I click on Save or if I click on Preview, I’ll be able to see all the OS which are matching Windows.

Also, if you would like to filter based on the OS, let’s say under all my subnet 00:24, I have my ten series subnet where I need all my Windows servers. So you can just select your Windows Filter type OS. No need to blacklist any specific IDs, only IPS matching your Windows machines will be able to match your criteria. So here for demonstration, I’ll use my PC where the universal forwarder has been installed and I’ll choose the filter type as Windows. If you click on Preview, as you can see there is a small checkmark and if you click on Match you’ll be able to see all the clients which are matching your present criteria. So this is some of the ways where you can create your own group of servers, where you can deploy specific set of config recent.

7. Creating Base Configurations

In this part of the discussion we’ll be seeing how to create a base config. The base config is nothing but a configuration which is common across all your Splunk instance. Let’s say I have 200 forwarders which needs to have minimum configuration in order to operate in my Splunk environment. That is nothing but inputs and outputs conf. It needs to know what it has to collect and where it has to send. That is defined in our inputs and outputs conf. So we’ll be seeing how we can create a base config and deploy it using our deployment server as part of our lab exercise. And we’ll also be seeing how we can use an application to change the default certificate that is used for Splunk TCP communication for exchange of the logs using SSL.

These are some of the most common practices where we will be using deployment server to chain the configuration across the organization where all your universal forwarder are installed, that is inputs, outputs and also SSL certificates across your universal forwarders. Now let us create a server group for our universal forwarders. So I’ll create Windows universal forwarders under Windows universal folders I’ll say dot, star, match everything with Windows x 64. So as you can see our Windows machine has been successfully picked up. I’ll click on save so that we have created our Windows universal forwarder server class.

Now we need to add an application that contains baseconfig. So for that we need to log into our deployment server. This is our index, this is searcher yes, this is our deployment server where we’ll be creating our base config application. I’ll create a directory base config. So here I’ll be creating one more directory that is local. Under local we’ll be creating our outputscom. So this outputcom contains your indexer, IP address and port number on which you are receiving the logs. Before that we’ll see what is the present configuration on our universal forwarder. Go to Splunk home. This is our Splunk home etc system local. This is our present outputs con.

As you all know by now, there shouldn’t be any files under system local because this will be overwriting configuration. We’ll make sure the outputs conference empty. We laid it the same information under our deployment server and we’ll change it to our indexer. This is our indexer IP save it once it we need some of the minimum directory structures of our Splunk application. For that we’ll copy some of the contents from our previous app that is Splunk. We’ll copy metadata information so that our application is enabled by default. We got our metadata and that should be more than sufficient.