/store/log partition full due to large stream.log-x.stderr files
search cancel

/store/log partition full due to large stream.log-x.stderr files

book

Article ID: 317702

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

Provide a workaround or a solution to this problem.

Symptoms:
/storage/log full due to stream.log-x.stderr filling out with authz logs.

stream.log-x.stderr logs with large size and displaying multiple entries like the one below:

/var/log/vmware/wcp/stdstream.log-x.stderr
 
2023/06/14 18:46:21 User name and domain: vsphere-webclient-81a910c3-9857-4e45-b203-7533fd4bf6980xc000e83850
time="2023-06-14T18:46:21Z" level=debug msg="Validating permissions; operation: com.vmware.cdc.change_log.get_current_sequence, requested: map[{isOperation:false id:com.vmware.cdc resourceType:}:[] {isOperation:true id:com.vmware.cdc.change_log.get_current_sequence resourceType:}:[System.Anonymous]]" opID=wcp-AuthzFilter
time="2023-06-14T18:46:21Z" level=debug msg="Requesting authz svc; user: vsphere.local\\vsphere-webclient-81a910c3-9857-4e45-b203-7533fd4bf698, groups: [[email protected] [email protected] [email protected] 


Cause

Some authz logs are being redirected to stderr causing storage issues on VCSA

Resolution

The fix for this issue has been published in vCenter Server 7.0.3 P08, which was released as vCenter Server 7.0 Update 3o

Workaround:
1. Make a backup copy of the wcpsvc-vmonconfig.json config file.

cp /etc/vmware/vmware-vmon/svcCfgfiles/wcpsvc-vmonconfig.json /tmp/wcpsvc-vmonconfig.json.bak

Note: Files in /tmp are removed once vCenter is rebooted. Therefore, you might want to save this in your computer.

2.Edit the wcpsvc-vmonconfig.json file 

vim /etc/vmware/vmware-vmon/svcCfgfiles/wcpsvc-vmonconfig.json

3. Remove the line beginning with StreamRedirectFile and save the file.
 
{
    "Name" : "wcp",
    "StartCommand" : "/usr/lib/vmware-wcp/wcpsvc.launcher",
    "StartCommandArgs": ["--port", "8920", "--logfile", "/var/log/vmware/wcp/wcpsvc.log",
                         "--configfile", "/etc/vmware/wcp/wcpsvc.yaml",
                         "--audit-logfile", "/var/log/vmware/wcp/wcp-audit.log",
                         "--incident-logfile", "/var/log/vmware/wcp/wcp-incident.log"],
    "PreStartCommand" : "%VMWARE_PYTHON_BIN%",
    "PreStartCommandArgs" : ["/usr/lib/vmware-wcp/wcpsvc-prestart.py"],
    "StreamRedirectFile" : "/var/log/vmware/wcp/stdstream.log",                                -----------------------> # line to be modified
    "RecoveryActionProfiles" :
    {
        "DEFAULT" :
        {
            "CRASH" : ["RESTART_SERVICE", "RESTART_SERVICE", "RESTART_SERVICE", "RESTART_SERVICE", "RESTART_SERVICE", "NO_ACTION"],
            "HEALTHFAIL" : ["NO_ACTION"]
        }
    },
    "WorkingDir" : "/usr/lib/vmware-wcp/",
    "DependsOn" : ["vpxd", "eam", "vmware-vpostgres", "lookupsvc", "cis-license", "vapi-endpoint", "sts", "trustmanagement"],
    "ApiHealthCommand" : "%VMWARE_PYTHON_BIN%",
    "ApiHealthCommandArgs" : ["/usr/lib/vmware-vmon/vmonApiHealthCmd.py", "-n", "wcp", "-u", "/wcp/health", "-t", "10"],
    "StartTimeout" : 120,
    "StopTimeout" : 30
}

4. Restart wcp service

service-control --stop wcp && service-control --start wcp

Additional Information

https://kb.vmware.com/s/article/76563

Impact/Risks:
High, vCenter might crash due to full partitions.