How to enable logging to troubleshoot NetBackup for Oracle

Manual:
NetBackup 5.0 for Oracle System Administrator’s Guide for Unix/Windows
NetBackup 5.1 for Oracle System Administrator’s Guide for Unix/Windows
NetBackup 6.0 for Oracle System Administrator’s Guide for Unix/Windows
NetBackup 6.5 for Oracle System Administrator’s Guide for Unix/Windows
NetBackup 7.0 for Oracle System Administrator’s Guide for Unix/Windows

Page:  N/A

Modification Type: Supplement

Modification:
The majority of cases at Symantec require the collection of log files in order to determine the nature of the backup or restore problem. Given the comprehensive nature of NetBackup ™, knowing which logs to collect and which logs are unnecessary can present a daunting task. This TechNote is designed to assist a user in determining what logs a Support Engineer needs to see in order to expedite the troubleshooting of an Oracle case. While there are cases where logs other than the ones documented are required for resolution, this TechNote should cover the requirements for nearly all cases.

By default, log files are placed in the <install_path>\veritas\netbackup\logs (Windows) and/usr/openv/netbackup/logs (UNIX/Linux) directories on a NetBackup server.  NetBackup logs can grow quite large, and it may be necessary to move the log location. To do this, please refer to TechNote 239186 (  http://support.veritas.com/docs/239186 ). This TechNote covers the way in which Symantec log files are relocated on a Windows server. For information on how to relocate logs on a UNIX master or media ser  r, please refer to TechNote 266619 ( http://support.veritas.com/docs/266619 ).

To enable logging on any of the NetBackup processes, a subdirectory must be created in the NetBackup logs directory on the NetBackup host.

Unix: /usr/openv/netbackup/logs

Windows: <install_path>\Veritas\NetBackup\logs

The following log folders should be created on the Oracle client server:

dbclient: This log contains debugging information and execution status for the NetBackup Oracle client processes.  Ensure that the Oracle user has permissions to write to this directory.

bpdbsbora: This log contains debugging information and execution status for the bpdbsbora utility which saves, retrieves and executes templates.

bphdb: This log contains debugging information for the process that executes the backup/restore script or template.  The obk_stdout and obk_stderr logs contain the stdout and stderr from the script if it does not redirect those outputs elsewhere.

bpubsora: This log contains debugging information and execution status for Template Wizard connections to the Oracle instance during database browse and template creation.  If logging into the Wizard as the Oracle user, than the directory must be writable by that user.

user_ops: This directory is created automatically and must remain readable and writeable by the Oracle user.  The comm and progress files therein are updated by the NetBackup server processes as the backup or restore progresses.  It is always present, and should be gathered for most cases.

bpcd: This log contains debugging information for the connection attempts to update the comm and progress files.

vnetd: This log contains debugging information for the connection attempts from the servers to bpcd and dbclient.

Create the following directories on the media server:

bpbrm: The bpbrm process handles backup and restore requests.

bptm: The bptm process handles interactions with the tape device. If backing up to a disk storage unit, the bpdm folder should be created.  At NetBackup 6.5 and above, bptm is used for both tape and disk storage units.

Create the following directory on the master server:

bprd: The bprd process handles the inbound backup, list, and restore requests from the client server.

bpcompatd: May be needed in rare instances if nbjm is unable to connect to the client to update comm files.

nbpem and nbjm may be needed occassionally if the jobs do not go Active in a timely manner for the expected policy.

If this is an RMAN PROXY backup with the Snapshot Client, then include these additional logs:

bpfis: From both the backup client and the off-host client.

vnetd and bpcd: From the off-host client.

To set the debugging level, go into the Backup, Archive and Restore GUI, and select File | NetBackup Client Properties, and click on the Troubleshooting tab. Increase the General,Verbose, and Database levels to 5. On Unix, the debug level can be changed by editing the bp.conf file and adding the following line:
VERBOSE = 5

If the problem is a performance issue, use VERBOSE = 6 to display delays between dbclient and Oracle.  (This applies only to 5.x and 6.5.4 and above).

In order to ensure the processes pick up the new verbosity settings, it is best to stop and restart all the NetBackup services on the hosts.  These processes are key.  The bprd process can pick up a verbosity change via bprdreq -rereadconfig but requires a restart if the logging directory was created.  Thebpbrm and bptm processes do not require a restart unless the job will join a long running MPX group.  On Windows, the Oracle processes must be stopped and restarted for dbclient to pickup up a verbosity change.

Once logging is enabled, run the backup or restore, and collect and send the following information for the Support Engineer to review:

1. Collect the logs from the above directories, making sure to rename the files to reflect the folder from which the log comes, and forward to the Support Engineer
2. In addition, on the Oracle client, send the script file (rename it to .txt so it isn’t stripped off by anti-virus software), the Recovery Manager (RMAN) output file, the initSID.ora file, and the zipped contents of theuser_ops directory.
3. In the bphdb directory, there should be two additional sets of files, oracle_stdout.<date> andoracle_stderr.<date>. Send these files from the date of the backup or restore attempt.
4. Note the names each of the machines