Configure ESXi Syslog to SEM

This article will help with configuring ESXi to syslog to Security Event Manager (formerly Log & Event Manager).

If anything here conflicts with VMware documentation, use VMware documentation as the definitive source.

To graphically configure the ESXi server to syslog, make the following changes.
 

1. Open a browser to Vsphere or the ESXi client access to ESXi.
2. In the vSphere Web Client inventory, select the host.
3. Click the Manage tab.
4. In the System panel, click Advanced System Settings.
5. Locate the Syslog section of the Advanced System Settings list.
6. To set up logging globally, select the setting to change, and then click the Edit icon.
7. Enter udp://LEM-appliance-ip-or-hostname:514.

   (Steps 8 & 9 define ESXi firewall rule to allow syslog output to SEM.)

8. Click the configuration tab, select the Security Profile menu, and then on the Firewall section, click Properties.
9. Scroll down to the syslog rule, select the check the box next to it, and then click OK.

ESXi command line to configure syslog.  

esxcli system syslog config set --loghost=<LEM-ip-address-or-hostname>:514
  (assume LEM ip is 10.2.3.4, replace <LEM-ip-address-or-hostname> with udp://10.2.3.4:514)
esxcli system syslog reload

Other commands: 

esxcli syswtem syslog config get  this will display how it is set)
esxcli system syslog config set --logdir=/path/to/vmfs/directory/ --loghost=RemoteHostname --logdir-unique=true|false --default-rotate=NNN --default-size=NNN       

(entire command if needed)

https://docs.vmware.com/en/VMware-vS...etup-guide.pdf 

Troubleshooting syslogs not being sent from the ESXi server:
 

1. Check the service:   ps -Cuv | grep vmsyslogd.

   - or -

   Look at the log on ESXi:   /var/log/.vmsyslogd.err/var/log/.vmsyslogd.err.

2. Start the service:  /usr/lib/vmware/vmsyslog/bin/vmsyslogd.
3. Load the syslog config:  esxcli system syslog reload.
   full command: 

(esxcli [--hostname --username] system syslog reload)

nc -z 10.11.12.13 514   (run on ESXi, to test if ESXi can reach LEM (10.11.12.13) udp port 514

When ESXi is syslogging to SEM, any of the following SEM logs could contain alerts:
 

• local4.log (typically the VMware ESXi Hostd connector)
• local6.log (typically the VMware ESXi Vmkernel connector)
• auth.log (typically the VMware ESXi messages connector) (typically the VMware ESXi messages connector
• cron.log (typically the VMware ESXi messages connector)
• daemon.log (typically the VMware ESXi messages connector)
• kern.log (typically the VMware ESXi messages connector)
• mail.log (typically the VMware ESXi messages connector)
• syslog.log (typically the VMware ESXi messages connector)
• user.log (typically the VMware ESXi messages connector)
• vmkwarning (typically the VMware ESXi Vmkwarning connector)

The above logs can be viewed from a vSphere/Hyper-V console (or putty session), the checklogs command under the Appliance menu, or with the help of SolarWinds Customer Support.

vSphere 6.0 - https://docs.vmware.com/en/VMware-vS...E0512DC21.html 

NOTE: vSphere allows changing the level of events syslogged.

Verify that the user you use to log in to the vCenter Server instance is a member of the SystemConfiguration.Administrators group in the vCenter Single Sign-On domain.

Procedure:

1. Log in as administrator@your_domain_name to the vCenter Server instance in the vCenter Server Appliance by using the vSphere Web Client.
2. On the vSphere Web Client Home page, click System Configuration.
3. Under System Configuration, click Nodes, and then select a node from the list.
4. Click the Related Objects tab.
5. From list of services running in the node you selected, Right-click VMware Syslog Service, select Settings, and then click Edit.
6. From the Common Log Level drop-down list select the log files to redirect:  info - Only informational log files are redirected to the remote machine.
           notice - Only notices are redirected to the remote machine., & Notice indicates normal but significant condition.
           warn - Only warnings are redirected to the remote machine.
           error - Only error messages are redirected to the remote machine.
           crit - Only critical log files are redirected to the remote machine.
           alert - Only alerts are redirected to the remote machine., & Alert indicates that action must be taken immediately.
           emerg - Only emergency log files are redirected to the remote machine. (Emergency indicates that the system stopped responding and cannot be used.)
7. In the Remote Syslog Host text box, enter the FQDN or IP address of the machine on which you want to export the log files.
8. In the Remote Syslog Port text box enter the port number to use for communication with the machine on which you want to export the log files.
9. From the Remote Syslog Protocol drop-down select the protocol to use. Select OK, and then from the Actions menu, click Restart to apply the configuration changes. 

       
For reference, here are some of the logs kept on ESXi:

/var/log/auth.log: ESXi Shell authentication success and failure.
/var/log/dhclient.log: DHCP client service, discovery, lease requests/renewals.
/var/log/esxupdate.log: ESXi patch and update installation logs.
/var/log/lacp.log: Link Aggregation Control Protocol logs.
/var/log/hostd.log: Host mgmt-service, vm/host tasks, VSphere/vCenter-comm, vpxa & SDK.
/var/log/hostd-probe.log: Host management service responsiveness checker.
/var/log/rhttpproxy.log: HTTP connections proxied on behalf of other ESXi host webservices.
/var/log/shell.log: ESXi Shell usage, enable/disable and commands entered.
/var/log/sysboot.log: Early VMkernel startup and module loading.
/var/log/boot.gz: boot log info, read using zcat /var/log/boot.gz|more.
/var/log/syslog.log: Mgmt service init, watchdogs, scheduled tasks and DCUI use.
/var/log/usb.log: USB arbitration, discovery & pass-through to VMs.
/var/log/vobd.log: VMkernel Observation events, similar to vob.component.event.
/var/log/vmkernel.log: Core VMkernel, device discovery, storage, networking, device/driver.
/var/log/vmkwarning.log: Warning/Alert logs from VMkernel logs.
/var/log/vmksummary.log: ESXi host startup/shutdown, heartbeat, VM's running, resources.
/var/log/Xorg.log: Video acceleration.

Vcenter
   
/var/log/vpxa.log: vCenter vpxa agent, vCenter & Host mgmt hostd agent.
/var/log/fdm.log: vSphere High Availability logs, produced by the fdm service