¡Come! Another one of logs please! This time we will see how to exploit the Logs of our beloved VMware vSphere environment, from the vCSA or vCenter Server Appliance we will redirect the logs to Logstash to be processed and stored in Elasticsearch, finally we will be able to visualize them in a more graphic way with Grafana. This way we will understand in real time what is happening or we will be able to have reports with the data of the last 24 hours, week, month… Fully customizable, free…

Well, what to say about vCenter Server that you don't already know, we know it's the heart of our VMware virtual infrastructures, What is the access portal to the platform?, that things happen there, Sometimes weird, There are tasks, We have events… There is a lot of sauce that is cooked and what less than to get juice out of it. Why not send all vCSA logs to Logstash, there by means of filters we will separate those ugly text files like themselves, after separating them into different fields that allow us to consult in a more comfortable way, the time will be stored in Elasticsearch to visualize them with Kibana or Grafana. To taste, As you know, I'm more in favor of the latter, But you don't have to always 😉 listen to me Well, to the subject, from Grafana we will be able to visually see what has been collected, to see any alarms that have happened, Logon attempts to the platform, just like in vCenter you will be able to filter and know the Tasks or Events, make cute tables with a field to look for some text sometime, as a user and know what he did…

As usual, first we obviously have to have the Elastic Stack installed, What has been Logstash, Elasticsearch and Kibana; then we can go to our vCSA, to your management website, Remember HTTPS://DIRECCION_IP_VCSA:5480, There we can from the “Syslog” configure to pull logs to Logstash, so we indicate the IP address to which to send the Logs and the port that interests us.

Input {
        TCP {
                type => "VCSA"
                port => "1515"
                tags => ["VMware,VCSA"]
        }
}

filter {

        if [type] == "VCSA" {
                        Grok {
                            match => { "Message" => [
                              "<%{POSINT:syslog_pri}>%{TIMESTAMP_ISO8601:Date} %{HOSTNAME:Servidor_VCSA} %{PROG:Nombre_log} - - -\s+%{TIMESTAMP_ISO8601:Log_fecha} \[%{GREEDYDATA:Tipo_tarea} %{WORD:Log_nivel} %{GREEDYDATA:Title} opId=] %{GREEDYDATA:Message}",
                              "<%{POSINT:syslog_pri}>%{TIMESTAMP_ISO8601:Date} %{HOSTNAME:Servidor_VCSA} %{PROG:Nombre_log} %{INT:Proceso_id} - -  Event \[%{INT:Evento_id}\] \[1-1\] \[%{TIMESTAMP_ISO8601:Log_fecha}\] \[%{GREEDYDATA:Title}\] \[%{WORD:Log_nivel}\] \[%{GREEDYDATA:User}\] \[%{DATE:Place}\] \[%{INT:Evento_id}\] \[%{GREEDYDATA:Message}\]",
                              "<%{POSINT:syslog_pri}>%{TIMESTAMP_ISO8601:Date} %{HOSTNAME:Servidor_VCSA} %{PROG:Nombre_log} - - - %{TIMESTAMP_ISO8601:Log_fecha} %{WORD:Log_nivel} %{PROG:Log_proceso}\[%{INT:Evento_id}\] \[%{GREEDYDATA:Originator} sub=%{GREEDYDATA:Title}\] %{TIMESTAMP_ISO8601:HTTP_fecha} %{WORD:HTTP_log_accion} %{UNIXPATH:HTTP_ruta} %{PROG:HTTP_version} %{INT:HTTP_codigo} %{PROG:HTTP_codigo_respuesta} - %{INT:HTTP_bytes_recibidos} %{INT:HTTP_bytes_enviados} %{INT:HTTP_duracion} %{INT:HTTP_tiempo_respuesta} %{INT:HTTP_req_forwarder} %{IPV4:HTTP_ip_origen1}:%{INT:HTTP_puerto_origen1} %{IPV4:HTTP_ip_destino1}:%{INT:HTTP_puerto_destino1} %{IPV4:HTTP_ip_origen2}:%{INT:HTTP_puerto_origen2} %{IPV4:HTTP_ip_destino2}:%{INT:HTTP_puerto_destino2}",
                              "<%{POSINT:syslog_pri}>%{TIMESTAMP_ISO8601:Date} %{HOSTNAME:Servidor_VCSA} %{PROG:Nombre_log} - - - %{TIMESTAMP_ISO8601:Log_fecha} %{WORD:Log_nivel} %{PROG:Log_proceso}\[%{INT:Evento_id}\] \[%{GREEDYDATA:Originator} sub=%{GREEDYDATA:Title}\] %{TIMESTAMP_ISO8601:HTTP_fecha} %{WORD:HTTP_log_accion} %{UNIXPATH:HTTP_ruta} %{PROG:HTTP_version} %{INT:HTTP_codigo} %{PROG:HTTP_codigo_respuesta} - %{INT:HTTP_bytes_recibidos} %{INT:HTTP_bytes_enviados} %{INT:HTTP_duracion} %{INT:HTTP_tiempo_respuesta} %{INT:HTTP_req_forwarder} %{IPV4:HTTP_ip_origen1}:%{INT:HTTP_puerto_origen1} %{IPV4:HTTP_ip_destino1}:%{INT:HTTP_puerto_destino1} %{GREEDYDATA:Message}",
                              "<%{POSINT:syslog_pri}>%{TIMESTAMP_ISO8601:Date} %{HOSTNAME:Servidor_VCSA} %{PROG:Nombre_log} - - - %{TIMESTAMP_ISO8601:Log_fecha} %{WORD:Log_nivel} %{PROG:Log_proceso}\[%{INT:Evento_id}\] \[%{GREEDYDATA:Originator} sub=%{GREEDYDATA:Title} opId=%{PROG:Op_id}\] %{GREEDYDATA:Message}",
                              "<%{POSINT:syslog_pri}>%{TIMESTAMP_ISO8601:Date} %{HOSTNAME:Servidor_VCSA} %{PROG:Nombre_log} - - -\s+%{TIMESTAMP_ISO8601:Log_fecha} \[%{PROG:Log_proceso}\] %{WORD:Log_nivel}\s+opId=%{DATE:Op_id} - %{GREEDYDATA:Message}",
                              "<%{POSINT:syslog_pri}>%{TIMESTAMP_ISO8601:Date} %{HOSTNAME:Servidor_VCSA} %{PROG:Nombre_log} - - - %{TIMESTAMP_ISO8601:Log_fecha} %{WORD:Log_nivel} %{PROG:Log_proceso}\[%{INT:Evento_id}\] \[%{GREEDYDATA:Originator} sub=%{GREEDYDATA:Title}\] \[%{DATE:Tipo_unit}\] %{GREEDYDATA:Message}",
                              "<%{POSINT:syslog_pri}>%{TIMESTAMP_ISO8601:Date} %{HOSTNAME:Servidor_VCSA} %{PROG:Nombre_log} - - - %{TIMESTAMP_ISO8601:Log_fecha} %{WORD:Log_nivel} %{PROG:Log_proceso}\[%{INT:Evento_id}\] \[%{GREEDYDATA:Originator} sub=%{GREEDYDATA:Title}\] %{GREEDYDATA:Message}",
                              "<%{POSINT:syslog_pri}>%{TIMESTAMP_ISO8601:Date} %{HOSTNAME:Servidor_VCSA} %{PROG:Nombre_log} - - - \[%{HTTPDATE:Log_fecha}\] %{PROG:Log_proceso} \[%{WORD:HTTP_peticion}\] %{IPV4:HTTP_ip_origen1}:%{INT:HTTP_puerto_origen1} To %{HOSTNAME:HTTP_equipo_destino} %{INT:HTTP_puerto_destino1} - %{PROG:HTTP_version} %{WORD:HTTP_log_accion} %{UNIXPATH:HTTP_ruta}\s+[%{WORD:HTTP_respuesta}\] %{INT:HTTP_codigo} - %{INT:HTTP_bytes} bytess+[%{PROG:HTTP_log_proceso}\] process %{INT:Tiempo_total}ms / commit %{INT:Tiempo_conexion}ms / conn \[\+\]",
                              "<%{POSINT:syslog_pri}>%{TIMESTAMP_ISO8601:Date} %{HOSTNAME:Servidor_VCSA} %{PROG:Nombre_log} - - -\s+%{TIMESTAMP_ISO8601:Log_fecha} %{WORD:Log_nivel} %{PROG:Log_proceso} \[%{DATE:Log_proceso}\] \[opID=%{PROG:Op_id}\] %{GREEDYDATA:Message}",
                              "<%{POSINT:syslog_pri}>%{TIMESTAMP_ISO8601:Date} %{HOSTNAME:Servidor_VCSA} %{PROG:Nombre_log} - - -\s+%{TIMESTAMP_ISO8601:Log_fecha} %{WORD:Log_nivel} %{PROG:Log_proceso}\[%{DATE:Log_proceso}\] \[OpId=%{PROG:Op_id}\] %{GREEDYDATA:Message}",
                              "<%{POSINT:syslog_pri}>%{TIMESTAMP_ISO8601:Date} %{HOSTNAME:Servidor_VCSA} %{PROG:Nombre_log} - - -\s+%{TIMESTAMP_ISO8601:Log_fecha} \|\s+%{WORD:Log_nivel} \| %{PROG:Log_proceso} \| %{DATE:Log_tarea} \| %{INT:Pid} \| %{GREEDYDATA:Message}",
                              "<%{POSINT:syslog_pri}>%{TIMESTAMP_ISO8601:Date} %{HOSTNAME:Servidor_VCSA} %{PROG:Nombre_log} - - -\s+%{TIMESTAMP_ISO8601:Log_fecha} \|\s+%{WORD:Log_nivel} \| %{PROG:Log_proceso} \| %{DATE:Log_tarea} \| %{GREEDYDATA:Message}",
                              "<%{POSINT:syslog_pri}>%{TIMESTAMP_ISO8601:Date} %{HOSTNAME:Servidor_VCSA} %{PROG:Nombre_log} - - -\s+[%{TIMESTAMP_ISO8601:Log_fecha}\] \[%{WORD:Log_nivel} \] %{PROG:Log_proceso} %{PROG:Task} %{GREEDYDATA:Message}",
                              "<%{POSINT:syslog_pri}>%{TIMESTAMP_ISO8601:Date} %{HOSTNAME:Servidor_VCSA} %{PROG:Nombre_log} - - -\s+%{TIMESTAMP_ISO8601:Log_fecha} \[%{PROG:Log_proceso}\s+%{WORD:Log_nivel} %{GREEDYDATA:Title} opId=%{PROG:Op_id}\] %{GREEDYDATA:Message}",
                              "<%{POSINT:syslog_pri}>%{TIMESTAMP_ISO8601:Date} %{HOSTNAME:Servidor_VCSA} %{PROG:Nombre_log} - - -\s+%{TIMESTAMP_ISO8601:Log_fecha} \[%{PROG:Log_proceso}\s+%{WORD:Log_nivel} %{GREEDYDATA:Title} opId=%{PROG:Op_id} %{WORD:Nipu}\] %{GREEDYDATA:Message}",
                              "<%{POSINT:syslog_pri}>%{TIMESTAMP_ISO8601:Date} %{HOSTNAME:Servidor_VCSA} %{PROG:Nombre_log} - - -\s+[%{TIMESTAMP_ISO8601:Log_fecha}\] %{WORD:Log_nivel}\[%{PROG:Log_proceso}\] - %{GREEDYDATA:Message}",
                              "<%{POSINT:syslog_pri}>%{TIMESTAMP_ISO8601:Date} %{HOSTNAME:Servidor_VCSA} %{PROG:Nombre_log} - - -\s+%{TIMESTAMP_ISO8601:Log_fecha} %{INT:Log_bytes} %{INT:Pid} %{PROG:Log_proceso} %{INT:HTTP_codigo} \"%{GREEDYDATA:Title}\" %{WORD:HTTP_log_accion} %{UNIXPATH:HTTP_ruta} %{PROG:HTTP_version}",
                              "<%{POSINT:syslog_pri}>%{TIMESTAMP_ISO8601:Date} %{HOSTNAME:Servidor_VCSA} procstate - - - %{DATE:User}\s+%{PROG:Log_pid}\s+%{DATE:CPU}\s+%{DATE:Memory}\s+%{DATE:MemoriaVirtual}\s+%{DATE:MemoriaProceso}\s+%{DATE:TTY}\s+%{DATE:Stat}\s+%{DATE:Hora_inicio}\s+%{DATE:Tiempo}\s+%{GREEDYDATA:Message}",
                              "<%{POSINT:syslog_pri}>%{TIMESTAMP_ISO8601:Date} %{HOSTNAME:Servidor_VCSA} sca - - -\s+%{TIMESTAMP_ISO8601:Log_fecha} %{WORD:Log_nivel} %{PROG:Process} \[%{PROG:Pool}\] %{GREEDYDATA:Message}",
                              "<%{POSINT:syslog_pri}>%{TIMESTAMP_ISO8601:Date} %{HOSTNAME:Servidor_VCSA} applmgmt-audit - - - %{TIMESTAMP_ISO8601:Log_fecha}: %{GREEDYDATA:Message}",
                              "<%{POSINT:syslog_pri}>%{TIMESTAMP_ISO8601:Date} %{HOSTNAME:Servidor_VCSA} applmgmt - - - %{TIMESTAMP_ISO8601:Log_fecha} \[%{INT:Evento_id}\]%{GREEDYDATA:Message}",
                              "<%{POSINT:syslog_pri}>%{TIMESTAMP_ISO8601:Date} %{HOSTNAME:Servidor_VCSA} procstate - - - %{GREEDYDATA:Message}",
                              "<%{POSINT:syslog_pri}>%{TIMESTAMP_ISO8601:Date} %{HOSTNAME:Servidor_VCSA} sca-vmon.std - - - %{GREEDYDATA:Message}",
                              "<%{POSINT:syslog_pri}>%{TIMESTAMP_ISO8601:Date} %{HOSTNAME:Servidor_VCSA} vsan-health-main - - - %{GREEDYDATA:Message}",
                              "<%{POSINT:syslog_pri}>%{TIMESTAMP_ISO8601:Date} %{HOSTNAME:Servidor_VCSA} vum-log4cpp - - - %{GREEDYDATA:Message}",
                              "<%{POSINT:syslog_pri}>%{TIMESTAMP_ISO8601:Date} %{HOSTNAME:Servidor_VCSA} vmcad - - - %{GREEDYDATA:Message}",
                              "<%{POSINT:syslog_pri}>%{TIMESTAMP_ISO8601:Date} %{HOSTNAME:Servidor_VCSA} eam-main - - - %{GREEDYDATA:Message}",
                              "<%{POSINT:syslog_pri}>%{TIMESTAMP_ISO8601:Date} %{HOSTNAME:Servidor_VCSA} cis-license - - - %{GREEDYDATA:Message}",
                              "<%{POSINT:syslog_pri}>%{TIMESTAMP_ISO8601:Date} %{HOSTNAME:Servidor_VCSA} vmdird - - - %{GREEDYDATA:Message}",
                              "<%{POSINT:syslog_pri}>%{TIMESTAMP_ISO8601:Date} %{HOSTNAME:Servidor_VCSA} sps-gc - - - %{GREEDYDATA:Message}",
                              "<%{POSINT:syslog_pri}>%{TIMESTAMP_ISO8601:Date} %{HOSTNAME:Servidor_VCSA} vstats - - - %{GREEDYDATA:Message}",
                              "<%{POSINT:syslog_pri}>%{TIMESTAMP_ISO8601:Date} %{HOSTNAME:Servidor_VCSA} vapi-runtime - - - %{GREEDYDATA:Message}",
                              "<%{POSINT:syslog_pri}>%{TIMESTAMP_ISO8601:Date} %{HOSTNAME:Servidor_VCSA} vpxd-profiler - - - %{GREEDYDATA:Message}",
                              "<%{POSINT:syslog_pri}>%{TIMESTAMP_ISO8601:Date} %{HOSTNAME:Servidor_VCSA} rsyslogd - - - %{GREEDYDATA:Message}",
                              "<%{POSINT:syslog_pri}>%{TIMESTAMP_ISO8601:Date} %{HOSTNAME:Servidor_VCSA} trustmanagement-svcs - - - %{GREEDYDATA:Message}",
                              "<%{POSINT:syslog_pri}>%{TIMESTAMP_ISO8601:Date} %{HOSTNAME:Servidor_VCSA} %{PROG:Nombre_log} - - - %{SYSLOGTIMESTAMP:Date} %{PROG:Log_proceso}\[%{INT:Pid}\]: %{GREEDYDATA:Message}",
                              "<%{POSINT:syslog_pri}>%{TIMESTAMP_ISO8601:Date} %{HOSTNAME:Servidor_VCSA} %{PROG:Nombre_log} - - - %{TIMESTAMP_ISO8601:Log_fecha} %{TZ:Zona_horaria} %{PROG:Process} %{INT:Log_id}\s+LOG:\s+%{GREEDYDATA:Message}",
                              "<%{POSINT:syslog_pri}>%{TIMESTAMP_ISO8601:Date} %{HOSTNAME:Servidor_VCSA} %{PROG:Nombre_log} - - - %{TIMESTAMP_ISO8601:Log_fecha}\| %{PROG:Process}\| I005: %{GREEDYDATA:Message}",
                              "<%{POSINT:syslog_pri}>%{TIMESTAMP_ISO8601:Date} %{HOSTNAME:Servidor_VCSA} %{PROG:Nombre_log} - - - %{TIMESTAMP_ISO8601:Log_fecha} %{PROG:Process} %{WORD:Log_nivel} %{PROG:Task} %{GREEDYDATA:Message}",
                              "<%{POSINT:syslog_pri}>%{TIMESTAMP_ISO8601:Date} %{HOSTNAME:Servidor_VCSA} ui-access - - - %{IPV4:HTTP_ip_destino1} %{IPV4:HTTP_ip_origen1} - - \[%{HTTPDATE:Date}\] \"%{WORD:HTTP_log_accion} %{UNIXPATH:HTTP_ruta} %{PROG:HTTP_version}\" %{INT:HTTP_codigo} %{PROG:HTTP_codigo_respuesta} - %{DATE:HTTP_bytes_recibidos} %{GREEDYDATA:Message}",
                              "<%{POSINT:syslog_pri}>%{TIMESTAMP_ISO8601:Date} %{HOSTNAME:Servidor_VCSA} eam-access - - - %{IPV4:HTTP_ip_destino1} %{IPV4:HTTP_ip_origen1} - - \[%{HTTPDATE:Date}\] \[%{PROG:Log_proceso}\] \"%{WORD:HTTP_log_accion} %{UNIXPATH:HTTP_ruta} %{PROG:HTTP_version}\" %{INT:HTTP_codigo} %{PROG:HTTP_codigo_respuesta} \[Processing time %{INT:Tiempo} msec\] %{GREEDYDATA:Message}",
                              "<%{POSINT:syslog_pri}>%{TIMESTAMP_ISO8601:Date} %{HOSTNAME:Servidor_VCSA} %{PROG:Nombre_log} - - - %{TIMESTAMP_ISO8601:Log_fecha} \[%{PROG:Process}\]\[%{WORD:Log_nivel}\] %{PROG:Task}: %{GREEDYDATA:Message}",
                              "<%{POSINT:syslog_pri}>%{TIMESTAMP_ISO8601:Date} %{HOSTNAME:Servidor_VCSA} %{PROG:Nombre_log} - - - %{TIMESTAMP_ISO8601:Log_fecha} %{PROG:Process} %{WORD:Log_nivel} %{PROG:Task} %{GREEDYDATA:Message}",
                              "<%{POSINT:syslog_pri}>%{TIMESTAMP_ISO8601:Date} %{HOSTNAME:Servidor_VCSA} %{PROG:Nombre_log} - - - \[%{TIMESTAMP_ISO8601:Log_fecha}\] \[%{WORD:Log_nivel}\s+] %{GREEDYDATA:Message}",
                              "<%{POSINT:syslog_pri}>%{TIMESTAMP_ISO8601:Date} %{HOSTNAME:Servidor_VCSA} %{PROG:Nombre_log} - - - %{TIMESTAMP_ISO8601:Log_fecha} \| %{PROG:Log_proceso} \| %{GREEDYDATA:Message}",
                              "<%{POSINT:syslog_pri}>%{TIMESTAMP_ISO8601:Date} %{HOSTNAME:Servidor_VCSA} CROND %{INT:Pid} - - \(%{USERNAME:User}\) CMD \(%{GREEDYDATA:Message}\)",
                              "<%{POSINT:syslog_pri}>%{TIMESTAMP_ISO8601:Date} %{HOSTNAME:Servidor_VCSA} %{PROG:Nombre_log} - - -\s+%{TIMESTAMP_ISO8601:Log_fecha} \|\s+%{WORD:Log_nivel} \| %{PROG:Evento_id}\ \| %{PROG:Process} \| %{DATE:Task} \| %{GREEDYDATA:Message}",
                              "<%{POSINT:syslog_pri}>%{TIMESTAMP_ISO8601:Date} %{HOSTNAME:Servidor_VCSA} %{PROG:Nombre_log} - - -\s+%{TIMESTAMP_ISO8601:Log_fecha} %{PROG:Log_proceso} %{WORD:Log_nivel} %{PROG:Task} %{GREEDYDATA:Message}"
                                       ]
                            }
                        }
        }
}

output {

        if ([type]=="VCSA"){

                ElasticSearch {
                   index => "vmware_vcsa-%{+YYYY. MM.dd}"
                   hosts=> "DIRECCION_IP_ELASTICSEARCH:9200"
                }

        }
}

Ahora ya será el momento de crear el fichero de configuración en Logstash para recibir, tratar y sacar los Logs a Elasticsearch, We can call it '/etc/logstash/conf.d/vmware-vcsa.conf', where in the input we will have the listening port where it will receive the logs from the vCSA. Then come the filters that as always I apologize to the wise, but they are valid filters and can be greatly improved, but they will allow us to process the logs of a vCSA with version 7.x. In the end, we'll output Elasticsearch, we indicate the destination IP to store the data there, and followed by a few credentials, Don't be like me who doesn't indicate the parameters of 'username’ and password.

Once the configuration file has been created, we will remember to restart the Logstash service to reload the new configuration. After, we will go to Kibana and once the data is coming in we can go to "Management" > "Stack management" > "Kibana" > "Index Patterns" > "Create index pattern" to create the index pattern, As I said, as usual (in this case and without the quotation marks) 'vmware_vcsa-*' and we'll have the data already stored in Elasticsearch correctly. Now we could connect from "Discover" to our VMware vCSA index and visualize that it's collecting data, that we start having vCenter logs…

After creating the index in Kibana, now in Grafana we should create a "Data Source" that points against our Elasticsearch and the newly created VMware vCSA index. Then it's letting your imagination run wild, make a Dashboard with different Dashboards, with different data to visualize, view log types, Graphs with reception by type, in stalemate to view specific data for example login attempts, Correct, Incorrect, latest Alarms, DRS records, Events, Tasks… un World Map with incoming connections if you are a brave person who has it public on the Internet… un style panel Sankey to view source/destination IPs/users… Top up error logs and try to fix them…

So that, The usual thing, all this to try to help, inspire… if you need, to take ideas and improve them, so with this you can see what happens on your VMware vSphere virtual platform, everything that happens in vCenter, in real time, Putting an automatic refresh every 10 seconds is very cool, It will also help us to analyze the summary of the last day, or knowing when something strange happens... Thank you to all and more to those who move this type of content on social networks!!!