Enabling Centralized Logging in VMware vCloud Director

Products

VMware Cloud Director

Issue/Introduction

In vCloud Director, there are two types of logs that a System Administrator may wish to preserve via Syslog:

Audit log messages - These are the Audit Messages which are stored in the vCloud Director Database and retained by default for 90 days.
Diagnostic log files - These are the log files present in the /opt/vmware/vcloud-director/logs directory of the vCloud Director Cells

This article describes how to configure vCloud Director to send either log output to a central syslog server.

Environment

VMware Cloud Director for Service Provider 8.x
VMware Cloud Director for Service Provider 9.x

Resolution

Prerequisites

Verify that your syslog host runs a version of syslog that supports listening for remote connections. Most modern versions of syslog support use of the -r option to enable listening for remote connections. In addition, the open source program syslog-ng has this capability.
Verify that the appropriate firewall ports are open on both the vCloud Director host for outbound UDP traffic as well as on the syslog host for inbound UDP access.

Limitations

SyslogAppender only supports UDP. Use of TCP or other more advanced syslog features requires modifications of the cell's runtime configuration, which is not supported.
The Jetty request log is configured by Jetty, so there is no way to centrally log the access and error information using Log4j.
When you upgrade the cell to a new release of vCloud Director, changes to log4j.properties revert to the default, so you must re-enter these customizations after the upgrade.

Configure Audit Message Handling

To configure Audit Message Handling in vCloud Director, follow the steps below:

1. Login to a vCloud Director Cell as root
2. Run the following command, substituting IP Address and Port for valid values.
/opt/vmware/vcloud-director/bin/cell-management-tool configure-audit-syslog -loghost <IP Address> -logport <Port>

Note: The command does not test the combination of host and port for network accessibility or the presence of a running syslog service.

3. Verify that entries are now placed in the /opt/vmware/vcloud-director/etc/global.properties and /opt/vmware/vcloud-director/etc/responses.properties like those below.
audit.syslog.host = <IP Address>
audit.syslog.port = <Port>
4. Repeat the above procedure on all additional vCloud Director Cells within the Server Group.
(Optional)
5. To unconfigure Audit Message Handling, simply run the following command:
/opt/vmware/vcloud-director/bin/cell-management-tool configure-audit-syslog -d

Note: This will delete the entries in the global.properties and responses.properties files.

Configure Diagnostic Log Handling

To configure Diagnostic Log Handling in vCloud Director, follow the steps below:

1. Backup the log4j.properties file on the cell
    cp /opt/vmware/vcloud-director/etc/log4j.properties /opt/vmware/vcloud-director/etc/log4j.properties.date.bak
2. Open the log4j.properties file using a Log Editor
3. Amend the Root Logger entry at then beginning of the file to add the syslog option

Original
    # Root logger
    log4j.rootLogger=ERROR, vcloud.system.debug, vcloud.system.info
Amended
    # Root logger
    log4j.rootLogger=ERROR, vcloud.system.debug, vcloud.system.info, vcloud.system.syslog

4. Add a new section at the end of the Log4J file for our Syslog Server settings, substituting for syslog.fqdn:514 valid values.
Please simply enter the IP or fqdn of your syslog server instead of 'syslog.fqdn' and change the port if necessary for your syslog server:
    #Entries for sending vCD Cell logs to syslog server:
    log4j.appender.vcloud.system.syslog=org.apache.log4j.net.SyslogAppender
    log4j.appender.vcloud.system.syslog.syslogHost=syslog.fqdn:514

5. Configure the log appenders and layout.
    log4j.appender.vcloud.system.syslog.facility=LOCAL1
    log4j.appender.vcloud.system.syslog.layout=com.vmware.vcloud.logging.layout.CustomPatternLayout
    log4j.appender.vcloud.system.syslog.layout.ConversionPattern=%d{ISO8601} | %-8.8p | %-25.50t | %-30.50c{1} | %m | %x%n
    log4j.appender.vcloud.system.syslog.threshold=INFO

Note: The example above is using INFO as a threshold value.
This will mean the syslog server will receive the same content that would be in the vCloud Director Cell's INFO log, /opt/vmware/vcloud-director/logs/vcloud-container-info.log.
If you would like to send full debug logs to the syslog server, /opt/vmware/vcloud-director/logs/vcloud-container-debug.log, then you can simply change the INFO below to DEBUG:

6. To ensure the changes take effect I would also recommend stopping and starting the vCloud Director Cell services on the Cells after making these changes.
For more information, please see Use the Cell Management Tool to Quiesce and Shut Down a Cell

7. Repeat this procedure for each Cell in your vCloud Director Server group.