Tera-Tom on Teradata Utilities V12-V13

by Tom Coffing Coffing Data Warehousing. (c) 2011. Copying Prohibited.

Reprinted for Jaskaran Singh, Capgemini US LLC none@books24x7.com Reprinted with permission as a subscription benefit of Books24x7, http://www.books24x7.com/

All rights reserved. Reproduction and/or distribution in whole or in part in electronic,paper or other forms without written permission is prohibited.

TeraTomonTeradataUtilitiesV12V13

Chapter 10: Support Environment

Overview Only he who attempts the ridiculous may achieve the impossible. - Don Quixote As seen in the many of the Teradata Utilities, the introduction of the capabilities of the Support Environment (SE) is a valuable asset. It is an inherit part of the utilities and acts as a front-end to FastExport, MultiLoad, and TPump. The purpose of the SE is to provide a feature rich scripting tool. As the newer load and extract functionalities were being proposed for use with the Teradata RDBMS, it became obvious that certain capabilities were going to be needed by all the utilities. Rather than writing these capabilities over and over again into multiple programs, it was written once into a single module/environment called the SE. This environment/module is included with the newer utilities. The Support Environment Commands Alphabetic Command List
Command .ACCEPT .BEGIN .DATEFORM .DISPLAY .END .ENDIF .ELSE .IF .LOGTABLE .LOGON .LOGOFF .ROUTE .RUN .SET .SYSTEM Functionality Read an input record that provides one or more parameter values for variables Invoke one of the utilities Define the acceptable or desired format for a date in this execution as either (YY/MM/DD) or (YYYY-MM-DD) Write an output message to a specified file Exit the utility Define the scope of a .IF command, allows multiple operations based on a conditional comparison Optionally, perform an operation when a condition is not true Compare variables and values to conditionally perform one or more operations Specify the restart log Establish a Teradata session Terminate a Teradata session Write output to a specified file Read and run commands stored in an external script file Establish or change a value stored in a variable Allows for the execution of a command at the computers operating system level from within the script

The SE allows the writer of the script to perform housekeeping chores prior to calling the desired utility with a .BEGIN. At a minimum, these chores include the specification of the restart log table and logging onto Teradata. Yet, it brings to the party the ability to perform any Data Definition Language (DDL) and Data Control Language (DCL) command available to the user as defined in the Data Dictionary. In addition, all Data Manipulation Language (DML) commands except a SELECT are allowed within the SE. Required Operational Command List A few of the SE commands are mandatory. The rest of the commands are optional and only used when they satisfy a need. The following section in this chapter elaborates on the required commands. The optional commands are covered in later sections. Once the explanation and syntax is shown, an example of their use is shown in a script at the end of this chapter. Creating a Restart Log Table The Restart Log table is a mandatory requirement to run a utility that may need to perform a restart. It is used by the utility to monitor its own progress and provide the basis for restart from the point of a failure. This restart facility becomes critical
Page 2 / 7 ReprintedforQ4OGY\237048,CapgeminiUSLLC CoffingDataWarehousing,CoffingPublishing(c)2011,CopyingProhibited

TeraTomonTeradataUtilitiesV12V13

when processing millions of data rows. It is normally better to restart where the error occurred rather than rerunning the job from the beginning (like BTEQ). The utilities use the restart log table to ascertain what type of restart, if any, is required as a result of the type of failure. Failures can occur at a Teradata, network or client system level. The Restart log makes the process of restarting the utility very much automatic once the problem causing the failure has been corrected. The syntax for creating a log table: .LOGTABLE [<database-name>.]<table-name> ; When the utility completes successfully with a return code of zero, the restart log table is automatically dropped. Creating Teradata Sessions Teradata will not perform any operation for a user who has not logged onto the system. It needs the user information to determine whether or not the proper privileges exist before allowing the operation requested. Therefore, it is necessary to require the user to provide authentication via a LOGON request. As a matter of performance, the utilities that use the SE look at the number of AMP tasks to determine the number of sessions to establish. However, the number of sessions is configurable, but not as a part of the .LOGON. Instead, setting the number of sessions to establish that is covered in the .BEGIN paragraph (next). The syntax for logging onto Teradata:
.LOGON [<tdpid>/]<user-name>,<user-password> [,acct-id] ;

Notice that we are discussing the .LOGON after the .LOGTABLE command. Although a log table cannot be created until after a session is established, the .LOGTABLE command is coded first. At the same time, the order isnt strictly enforced and the logon can come first. However, you will see a warning message displayed from the SE if the .LOGON command is issued first. So, it is best to make a habit of always specifying the .LOGTABLE command first. Once a session is established, based on privileges, the user can perform any of the following:
n

DDL DCL Any DML (with the exception of SELECT) Establish system variables Accept parameter values from a file Perform dynamic substitution of values including object names

Beginning a Utility Once the script has connected to Teradata and established all needed environmental conditions, it is time to run the desired utility. This is accomplished using the .BEGIN command. Beyond running the utility, it is used to define most of the options used within the execution of the utility. As an example, setting the number of sessions is requested here. See each of the individual utilities for the names, usage and any recommendations for the options specific to it. The syntax for writing a .BEGIN command:
.BEGIN <utility-task> [ <utility-options> ] ;

The utility task is defined as one of the following:

FastExport MultiLoad to load or modify rows MultiLoad to delete rows TPump .BEGIN EXPORT .BEGIN [ IMPORT] MLOAD .BEGIN DELETEMLOAD .BEGIN LOAD

Page 3 / 7 ReprintedforQ4OGY\237048,CapgeminiUSLLC CoffingDataWarehousing,CoffingPublishing(c)2011,CopyingProhibited

TeraTomonTeradataUtilitiesV12V13

Figure 9-2 Ending a Utility Once the utility finishes its task, it needs to be ended. To request the termination, use the .END command. The syntax for writing a .END command:
.END <utility-task> ;

When the utility ends, control is returned to the SE. It can then check the return code (see Figure 9-4) status and verify that the utility finished the task successfully. Based on the status value in the return code, the SE can be used to determine what processing should occur next. Terminating a Teradata Sessions Once the sessions are no longer needed, they also should to be ended. To request their termination, use the .LOGOFF. The syntax for logging onto Teradata:
.LOGOFF [<return-code>] ;

Optionally, the user may request a specific return code be sent to the host computer that was used to start the utility. The script might be executed from the job control language (JCL) on a mainframe, the shell script for a UNIX system, or bat file on DOS. This value can then be checked by that system to determine conditional processing as a result of the completion code specified. Optional Command List The following commands are available to add functionality to the SE. They allow for additional processing within the preparation for the utility instead of requiring the user to access or other external tools. As with the required commands above, an example of their use is shown in a script at the end of this chapter. Accepting a Parameter Value(s) Allowing the use of parameter values within the SE is a very powerful tool. A parameter can be substituted into the script much like the substitution of values within a Teradata macro. However, it is much more elaborate in that the substitution includes the object names used in the SQL, not just data. When accepting one or more parameter values, they must be in a single record. If multiple records are needed, they can be read using multiple .ACCEPT commands from different files. Each record may contain one or more values delimited by a space. Therefore, it is necessary to put character strings in single quotes. Once accepted by the script, these values are examined and are stored dynamically stored into parameters named within the script. The syntax for writing a .ACCEPT command: .ACCEPT <parameter-name> [, <parameter-name> ] { [ FROM ] FILE <file-name> | { ENVIRONMENT | ENV } VARIABLE | VAR} <sys-variable> } [ IGNORE <character-position> [ THRU <character-position> ] ] ; The format of the accepted record is comprised of either character or numeric data. Character data must be enclosed in single quotes () and numeric data does not need quotes. When multiple values are specified on a single record, a space is used to delimit them from one another. The assignment of a value to a parameter is done sequentially as the names appear in the .ACCEPT and the data appears on the record. The first value is assigned to the first parameter and so forth until there are no more parameter names in which to assign values. The system variables are defined later in this chapter. They are automatically set by the system to provide information regarding the execution of the utility. For example, they include the date, time and return code, to name a few. Here they can be used to establish the value for a user parameter instead of reading the data from a file. Example of using a .ACCEPT command:
Page 4 / 7 ReprintedforQ4OGY\237048,CapgeminiUSLLC CoffingDataWarehousing,CoffingPublishing(c)2011,CopyingProhibited

TeraTomonTeradataUtilitiesV12V13

.ACCEPT char_parm, int_num_parm, dec_num_parm FILE parm-record;

Contents of parm-record:
'This is some character data enclosed in quotes with spaces in it123 35.67

Once accepted, this data is available for use within the script. Optionally, an IGNORE can be used to skip one or more of the specified variables in the record. This makes it easy to provide one parameter record that is used by multiple job scripts and allowing the script to determine which and how many of the values it needs. To not use the integer data, the above .ACCEPT would be written as:
.ACCEPT char_parm, dec_num_parm FILE parm-record IGNORE 39 THRU 42;

Note: if the system is a mainframe, the FILE is used to name the DD statement in the Job Control Language (JCL). For example, for the above .ACCEPT, the following JCL would be required:
//PARM-RECORD DD DSN=<pds-member-name>, DISP=(old, keep)

Establishing the Default Date Format Depending on the mode (Teradata or ANSI) defined within the DBC Control Record, the dates are displayed and read according to that default format. When reading date data that does not match that format, it is rejected and stored in an error table. This rejection includes a valid ANSI date when it is looking for a Teradata date. To ease the writing of the code by eliminating the need to specifically define the format of incoming dates, the .DATEFORM is a useful command. It allows for the user to declare an incoming date with the ANSI format (YYYY-MMDD) or the Teradata format (YY/MM/DD). The syntax for writing a .DATEFORM command:
.DATEFORM { ANSIDATE | INTEGERDATE } /* INTEGERDATE is the default */ ;

Since these are the only two pre-defined formats, any other format must be defined in the INSERT of the utility, as in the following example for a MM/DD/YYYY date:
NSERT INTO my_date_table VALUES ( :incoming_date (DATE, FORMAT mm/dd/yyyy));

Displaying an Output Message The .DISPLAY command is used to write a text message to a file name specified in the command. Normally, this technique is used to provide operational or informational output to the user regarding one or more conditions encountered during the processing of the utility or SE. The default file is system printer (SYSPRINT) on a mainframe and standard out (STDOUT) on other platforms. The message is normally built using a literal character string. However, a user may request the output to consist of substituted variable or parameter data. This is accomplished using an ampersand (&) in front of the variables name. See the section below on using a variable in a script for more details. The syntax for writing a .DISPLAY command:
.DISPLAY '<message-text-here>[TO] FILE <file-name> ;

Note: If the system is a mainframe, the FILE portion of the command is used to name the DD statement in the JCL. The JCL must also contain any names, space requirements, record and block size, or disposition information needed by the system to create the file. Comparing Variable Data The .IF command is used to compare the contents of named variable data. Normally, a variable is compared to a known literal value for control purposes. However, anything can be compared where it makes sense to do so. The syntax for writing a .IF command: .IF {<variable-name> | <literal>} <comparison> {<literal> | <variable-name>} [ THEN ] <operation-to-perform> [,<operation-to-perform> ]

Page 5 / 7 ReprintedforQ4OGY\237048,CapgeminiUSLLC CoffingDataWarehousing,CoffingPublishing(c)2011,CopyingProhibited

TeraTomonTeradataUtilitiesV12V13

[ ELSE { {<variable-name> | <literal>} <comparison> {<literal> | <variable-name>} |<operation-to-perform> [,<operation-to-perform> ] } .ENDIF ; The comparison symbols are normally one of the following:
Equal = Less than < Greater than > Not equal <> Less than or equal <= Greater than or equal >=

Figure 9-3 Routing Messages The .ROUTE command is used to write messages to an output file. This is normally system information generated by the SE during the execution of a utility. The default file is SYSPRINT on a mainframe and STDOUT on other platforms. The syntax for writing a .ROUTE command:
.ROUTE <messages> [ TO ] FILE <file-name>

[ [WITH] ECHO { OFF | [ TO ] FILE <file-name> ] ;

Note: If the system is a mainframe, the FILE is used to name the DD statement in the JCL. The JCL must also contain any names, space requirements, record and block size, or disposition information needed by the system. Running Commands from a File The .RUN command is used to read and execute other commands from a file. This is a great tool for using pre-defined and stored command files. This is especially a good way to secure your user id and password from being written into the script. In other words, you save your .LOGON in a secured file that only you can see. Then, use the .RUN to access it for processing. In addition, more than one command can be put into the file. Therefore, it can add flexibility to the utility by building commands into the file instead of into the script. The syntax for writing a .RUN command:
.RUN FILE <file-name> [ IGNORE <character-position> [ THRU character-position> ]] ;

The IGNORE and the THRU options work here the same as they do as explained in the .ACCEPT above. If the system is a mainframe, the FILE is used to name the DD statement in the JCL. Setting Variables to Values The .SET command is used to assign a new value or change an existing value within a variable. This is done to make the execution of the script more flexible and provide user with more control of the processing. The syntax for writing a .SET command:
.SET <variable-name> [ TO ] <expression> ;

Note: The expression can be a literal value based on the data type of the variable or a mathematical operation for numeric data. The math can use one or more variables and one or more literals. Running a System Command The .SYSTEM command is used as a hook to the operating system on which the utility is running. This is done to communicate with the host computer and request an operation that the SE cannot do on its own. When using this command, it is important to know which operating system is being used. This information can be obtained from one of the system variables below. The syntax for writing a .SYSTEM command:
Page 6 / 7 ReprintedforQ4OGY\237048,CapgeminiUSLLC CoffingDataWarehousing,CoffingPublishing(c)2011,CopyingProhibited

TeraTomonTeradataUtilitiesV12V13

.SYSTEM <operating-system-specific-command>;

Note: There is a system variable that contains this data and can be found in the System Variable section of this chapter. Using a Variable in a Script The SE dynamically establishes a memory area definition for a variable at the point it is first referenced. The data used to initialize it also determines the data type it is to use. To distinguish the referenced name as a variable instead of being a database object name, a special character is needed. The character used to identify the substitution of variable data into the SQL, is the ampersand (&) in front of the variable name. However, the ampersand is not used when the value is being set. The Support Environment System Variables The following variables are available within the SE to help determine conditions and system data for processing of the script.
&SYSDATE &SYSDATE(4) &SYSTIME &SYSDAY &SYSOS &SYSUSER &SYSRC &SYSINSCNT[n] &SYSUPDCNT [n] &SYSDELCNT [n] Provides the system date in YY/MM/DD format Provides the system date in YYYY-MM-DD format Provides the system time in HH:MM:SS format Provides the system day as three capitalized characters; ex: MON Provides the system host operating system as a maximum of five characters; ex: VM/SP Provides the user ID Provides the return or status code from the previous operation Provides the count for rows inserted into the table where n=1-5, relative number to identify the table from the TABLES portion of the .BEGIN in MultiLoad Provides the count for rows updated in the table where n=1-5, relative number to identify the table from the TABLES portion of the .BEGIN in MultiLoad Provides the count for rows deleted from the table where n=1-5,relative number to identify the table from the TABLES portion of the .BEGIN in MultiLoad

Figure 9-4 Support Environment Example

/* build the restart log called MJL_Util_log in database WORKDB */ .LOGTABLE WORKDB.MJL_Util_log; .DATEFORM ansidate; /* get the logon from a file called logon-file*/ .RUN FILE logon-file; /* test the system day to see if it is Friday notice that the character string is used in single quotes so that it is compared as a character string. Contrast this below for the table name */.IF '&SYSDAY' = 'FRI' THEN; .DISPLAY &SYSDATE(4) is a &SYSDAYFILE outfl.txt; .ELSE; .DISPLAY '&SYSUSER, &SYSDATE(4) is not Fridayto FILE outfl.txt; .LOGOFF 16; /* notice that the endif allows for more than one operation after the comparison*/ .ENDIF; /* the table name and two values are obtained from a file*/ .ACCEPT tablename, parm_data1, parm_data2 FILE myparmfile; /* establish and store data into a variable */.SET variable1 TO &parm_data1 + 125; /* the table name is not in quotes here because it is not character data. But the value in parm_data2 is in quotes because it is character data. This is the power of it all ! */ INSERT INTO &tablename VALUES (&variable1, &parm_data2, &parm_data1); .LOGOFF; - - - - - - - - - - - - - - - - Contents of logon-file: .logon ProdSys/mikel,larkins1 Contents of myparmfile: My_test_table 125 Some character data

Page 7 / 7 ReprintedforQ4OGY\237048,CapgeminiUSLLC CoffingDataWarehousing,CoffingPublishing(c)2011,CopyingProhibited