Problem

BUG REPORT: NetBackup SQL, Oracle database agents and user initiated backups are failing with exit status code 41 after upgrading to NetBackup 6.5 or 7.0.

Solution

Bug: 1987535 Symptoms: After an upgrade from NetBackup 6.0, some database and user-initiated backups may fail. The nbjm process fails to write any messages to the progress and comm files on the client host, and the data flow never starts. Child application jobs will show as active in the Activity Monitor and the job details will show that tapes will have been mounted and positioned, but there will be no further progress updates after the Begin Writing notification. Normal file system backups that are initiated from the master to the same client are successful because they do not rely upon progress or comm file updates to initiate the data transfer. Log Files: The following log sequences show the nbjm process encountering a failure, which then leads to a dbclient timeout while awaiting for the missing comm/progress log updates. The nbjm (OID 117) debug log shows a failure while attempting to perform the update to the comm file.
02/24/10 10:29:35.952 [Debug] NB 51216 nbjm 117 PID:6960 TID:6900 File ID:117 [No context] 1 [ProgLogComm::TranslateAndLog] (018378E8) xlatemsg (translated) failed with status=1, jobid=61317(../JMProgLogComm.cpp:547)

The dbclient log shows an error after CLIENT_READ_TIMEOUT expires without the expected update to the comm file.
10:40:41.435 [2040.4076] <2> logconnections: BPRD CONNECT FROM client.4627 TO master.13724 10:41:06.137 [2040.4076] <4> serverResponse: Not a candidate for alternate buffer method: shared memory is not in use. 11:41:15.706 [2040.4076] <16> readCommFile: ERR - timed out after 3600 seconds while reading from C:\Program Files\Veritas\NetBackup\Logs\user_ops\mssql\logs\0224110104037-2040-4272-000000-prg

Cause: There is an incorrect DLL reference in the nblog.conf file on the master.

The default entry for KeyName:Default.L10nLib in the nblog.conf file after the upgrade is:
Default.L10nLib=<install_path>\VERITAS\NetBackup\bin\vxextwincat.dll

NOTE: In most cases, <install_path> is C:\Program Files ie

Default.L10nLib=C:\Program Files\VERITAS\NetBackup\bin\vxextwincat.dll

Workarounds: For machines which have been upgraded to NetBackup 6.5, confirm that <install_path>\VERITAS\NetBackup\bin\vxextwincat_1.dll exists on the master server. If it does, follow this procedure: 1. Open a DOS command prompt 2. Change directory to <install_path>\Veritas\NetBackup 3. Execute the following command:
vxlogcfg -a -p 51216 -s Default.L10nLib="<install_path>\Veritas\NetBackup\bin\vxextwincat_1.dll"

NOTE: In most cases, <install_path> is C:\Program Files For NetBackup 7.0, confirm that <install_path>\VERITAS\NetBackup\bin\vxextwincat_3.dll exists on the master server. 1. Open a DOS command prompt 2. Change directory to <install_path>\Veritas\NetBackup 3. Execute the following command:
vxlogcfg -a -p 51216 -s Default.L10nLib="<install_path>\Veritas\NetBackup\bin\vxextwincat_3.dll"

If successful, the following message should be displayed:

The configuration settings were added successfully.

After running the above command, be sure to verify the correct path was added to the nblog.conf file by opening it in a text editor (such as Notepad) and finding the vxextwincat substring. Compare it to the appropriate KeyName listed above for the version of NetBackup being run. Note: It is not recommended to edit/modify the nblog.conf file using a text editor as this may corrupt the file. After viewing this using a text editor, make sure to not save the file before closing it. Note: It is necessary to stop and restart nbjm (or all of NetBackup) on the master server to pickup the change. Then, retry the failed backup - it should succeed.

ETA of Fix: Symantec has acknowledged that the above mentioned issue (Etrack 1987535) is present in the current version(s) of the product(s) mentioned at the end of this article. Symantec is committed to product quality and satisfied customers. This issue was scheduled to be addressed in the following release: NetBackup 7.0 Release Update 1 (7.0.1) When NetBackup 7.0.1 is released, please visit the following link for download and readme information: http://www.symantec.com/business/support/overview.jsp?pid=15143 There are currently no plans to fix this issue in a future NetBackup 6.5 Release Update. Please use the workaround listed above if this issue is encountered with any NetBackup 6.5.x release. Please note that Symantec Corporation reserves the right to remove any fix from the targeted release if it does not pass quality assurance tests or introduces new risks to overall code stability. Symantec's plans are subject to change and any action taken by you based on the above information or your reliance upon the above information is made at your own risk.

Problem
NetBackup for SQL Server backup fails with Status Code 2 or Status Code 41 for a specific database using full text index

Error
Status Code 41: network connection timed out

Solution
The NetBackup for SQL Server database agent hangs near the end of a backup on SQL Server 2005 SP1 if the database has two (2) full text index catalog files on different drives. A little about Full-text indexing: Full-text indexing is a function of the Microsoft Search Service SQL Server databases and tables can be enabled for full-text indexing The index data are stored in catalog files inside the database. Each database can contain one or more full-text catalogs. The existence of multiple full text catalog files (Example: pageData and cerosfulltextindex) can be checked in the properties of the SQL Server database if the database is still available on a

server. The NetBackup Client Progress Log shows: ***Note: "Trace Level Max" is the reason for the messages about percent processed. 14:45:20 INF - Beginning backup on server NBUMS of client xyz-beetle INF - ODBC return code <2>, SQL State <01000>, SQL Message <3211><[Microsoft][SQL Native Client][SQL Server]10 percent processed.>. INF - SQL Message <3211><[Microsoft][SQL Native Client][SQL Server]20 percent processed.> INF - SQL Message <3211><[Microsoft][SQL Native Client][SQL Server]30 percent processed.> INF - SQL Message <3211><[Microsoft][SQL Native Client][SQL Server]40 percent processed.> INF - SQL Message <3211><[Microsoft][SQL Native Client][SQL Server]50 percent processed.> INF - SQL Message <3211><[Microsoft][SQL Native Client][SQL Server]60 percent processed.> INF - SQL Message <3211><[Microsoft][SQL Native Client][SQL Server]70 percent processed.> INF - SQL Message <3211><[Microsoft][SQL Native Client][SQL Server]80 percent processed.> INF - SQL Message <3211><[Microsoft][SQL Native Client][SQL Server]90 percent processed.> 14:51:08 INF - Server status = 41 14:51:10 INF - Backup by SYSTEM on client xyz-beetle using policy SQL_Daily, sched Default-Policy-Disk:network connection timed out Conditions: This problem will NOT be encountered on backups if: A database only has a single full text catalog file. A database has multiple full text catalog files but they exist on the same disk drive as the database. Note: the index files can be in different paths on the same drive. Workaround: Move the catalog file(s) to the same disk drive as the database. The backup will complete successfully with Status 0. Solution: In the NetBackup for SQL Server GUI or NetBackup for SQL Server batch file, set the number of Buffers to the number full text catalogs defined for the database. The backup will complete successfully with Status 0. How to: On the SQL Server, in the NetBackup for SQL Server GUI, In File; NetBackup client properties; Tuning Parameters; Client buffers per DBMS stripe, increase the buffers based on the database configuration.

Note: This is less of a problem with NetBackup 6.5.x because the default number of buffers was increased to 2.