Configuring centralized logging from ESXi

VMware best practices dictate that ESXi virtualization hosts should have their logs stored remotely. ESXi supports sending log data to a remote log collector via the syslog protocol, which allows Papertrail to ingest it.

Prepare firewall

Connecting to Papertrail requires opening a new outgoing firewall rule, which can only be performed by SSHing to the ESXi host itself. Once the SSH service has been started via the “Security Policy” section of the VMware client, log in and run the following command to generate a new firewall rule:

$ cat <<EOF > /etc/vmware/firewall/papertrail.xml
<ConfigRoot>
  <service id='1000'>
    <id>Papertrail</id>
    <rule>
      <direction>outbound</direction>
      <protocol>tcp</protocol>
      <porttype>dst</porttype>
      <port>XXXXX</port>
    </rule>
    <enabled>true</enabled>
    <required>false</required>
  </service>
</ConfigRoot>
EOF

where XXXXX is the port number shown under log destinations.

Once that’s done, refresh the firewall configuration using:

$ esxcli network firewall refresh

Note that these new rules will not persist across a reboot unless they are applied via a vSphere Installation Bundle (VIB). A custom VIB can be created by following the instructions in this VMware Knowledge Base article.

Enable syslog

Once the firewall is ready, configure remote syslog using:

$ esxcli system syslog config set --loghost='ssl://logsN.papertrailapp.com:XXXXX'
$ esxcli system syslog reload

where logsN and XXXXX are the name and port number shown under log destinations.

Certificate verification

To secure the syslog traffic against man-in-the-middle attacks, enable certificate verification.

To install Papertrail’s root certificate bundle, download papertrail-bundle.pem, and append the contents to /etc/vmware/ssl/castore.pem.

If the ESXi host is managed by vCenter, trusted root certificates are controlled by the Platform Services Controller. After downloading papertrail-bundle.pem, follow these instructions instead:

Finally, after using either technique to install Papertrail’s root certificate bundle, enable certificate verification:

$ esxcli system syslog config set --check-ssl-certs=true
$ esxcli system syslog reload