Академический Документы
Профессиональный Документы
Культура Документы
Version 8 Release 5
SC18-9889-02
SC18-9889-02
Note Before using this information and the product that it supports, read the information in Notices and trademarks on page 77.
Copyright IBM Corporation 2006, 2010. US Government Users Restricted Rights Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.
Contents
Chapter 1. Introduction . . . . . . . . 1 Chapter 2. Tutorial project goals . . . . 3 Chapter 3. Module 1: Opening and running the sample job . . . . . . . . 5
Lesson 1.1: Opening the sample job. . . . . . The Designer client . . . . . . . . . . The sample job for the tutorial . . . . . . Starting the Designer client and opening the sample job . . . . . . . . . . . . . Lesson checkpoint . . . . . . . . . . Lesson 1.2: Viewing and compiling the sample job. Exploring the Sequential File stage . . . . . Exploring the Data Set stage . . . . . . . Compiling the sample job . . . . . . . . Lesson checkpoint . . . . . . . . . . Lesson 1.3: Running the sample job . . . . . Running the job . . . . . . . . . . . Viewing the data set . . . . . . . . . Lesson checkpoint . . . . . . . . . . Module 1: Summary . . . . . . . . . . . 5 . 5 . 6 . 6 . 8 . 8 . 8 . 9 . 9 . 9 . 10 . 10 . 14 . 14 . 14 Lesson 3.1: Designing the transformation job . The transformer job . . . . . . . . Creating the transformation job and adding stages and links . . . . . . . . . . Configuring the Data Set stages . . . . Configuring the Transformer stage. . . . Running the transformation job. . . . . Lesson checkpoint . . . . . . . . . Lesson 3.2: Combining data in a job . . . . Using a Lookup stage . . . . . . . . Creating a lookup job . . . . . . . . Configuring the Lookup File Set stage . . Configuring the Lookup stage . . . . . Lesson checkpoint . . . . . . . . . Lesson 3.3: Capturing rejected data . . . . Lesson checkpoint . . . . . . . . . Lesson 3.4: Performing multiple transformations single job . . . . . . . . . . . . . Adding new stages and links . . . . . Configuring the Business_Rules Transformer stage . . . . . . . . . . . . . Configuring the Lookup operation. . . . Lesson checkpoint . . . . . . . . . Module 3 summary. . . . . . . . . . . . . . . . . . . . . . . . . in . . . . . . . 29 . 29 . . . . . . . . . . . . . a . . . . . . 29 30 31 33 33 33 33 34 35 35 38 38 39 40 40 41 44 45 45
. 17 . . . . . . . . . . . . . . 20 20 20 21 21 23 23 23 23 24 25 25 25 25
. 26 . 27 . 27
iii
. . .
. . .
. . .
. 62 . 62 . 63
Creating a DSN for the tutorial table computer . . . . . . . . . Creating a DSN for the tutorial table Linux computer . . . . . . .
on a . . on a . .
Windows . . . . 69 UNIX or . . . . 69
Chapter 8. Tutorial summary . . . . . 65 Contacting IBM . . . . . . . . . . . 71 Appendix. Installing and setting up the tutorial . . . . . . . . . . . . . . . 67
Creating a folder for the tutorial files . . . . . Creating the tutorial project . . . . . . . . Copying the data files to the project folder or directory . . . . . . . . . . . . . . Importing the tutorial components into the tutorial project . . . . . . . . . . . . . . . Creating a target database table . . . . . . . 67 . 67 . 68 . 68 . 68
iv
Chapter 1. Introduction
In this tutorial, you will learn the basic skills that you need to design and run IBM InfoSphere DataStage parallel jobs.
Learning objectives
By completing this tutorial, you will achieve the following learning objectives: v Learn how to design parallel jobs that extract, transform, and load data. v Learn how to run the jobs that you have designed, and how to view the results. v Learn how to create reusable objects that can be included in other job designs.
Learning objectives
As you work through the job scenario, you will learn how to do the following tasks: v Design parallel jobs that extract, transform, and load data v Run the jobs that you design and view the results v Create reusable objects that can be included in other job designs This tutorial should take approximately four hours to finish. If you explore other concepts related to this tutorial, it can take longer to complete.
Skill level
You can do this tutorial with only a beginning level of understanding of InfoSphere DataStage concepts.
Audience
This tutorial is intended for InfoSphere DataStage designers who want to learn how to create parallel jobs.
System requirements
The tutorial requires the following hardware and software: v InfoSphere DataStage clients installed on a Windows XP platform. v Connection to a InfoSphere DataStage server on a Windows or UNIX platform (Windows servers can be on the same computer as the clients). v To run the parallel processing module (module 5), the InfoSphere DataStage server must be installed on a multi-processor system (SMP or MPP).
Prerequisites
You need to complete the following tasks before starting the tutorial: v Get DataStage developer privileges from the InfoSphere DataStage administrator v Check that the InfoSphere DataStage administrator has installed and set up the tutorial by following the procedures described in Appendix A v Obtain the name of the tutorial folder on the InfoSphere DataStage client computer and the tutorial project folder or directory on the InfoSphere DataStage server computer from the InfoSphere DataStage administrator.
Learning objectives
After you complete the lessons in this module, you will understand how to do the following tasks: v Start the InfoSphere DataStage and QualityStage Designer (Designer client) and attach a project. v Open an existing job. v Compile a job so that it ready to run. v Open the Director client and run a job. v View the results of the job. This module should take approximately 30 minutes to complete.
Prerequisites
Ensure that you have DataStage user authority.
In the design area of the Designer client, you work with the tools and objects to create your job designs. The sample job opens in a design window.
Lesson checkpoint
In this lesson, you opened your first job. You learned the following tasks: v How to start the Designer client v How to open a job v Where to find the tutorial objects in the repository tree
7. In the Value field of the Resolve Job Parameter window, specify the name of the directory in which the tutorial data was installed and click OK (you have to specify directory path whenever you view data or run the job). 8. In the Data Browser window, click OK. A window opens that shows the first 100 rows of the data that the GlobalCo_BillTo.csv file contains (100 rows is the default setting, but you can change it). 9. Click Close to close the Data Browser window. 10. Click OK to close the Sequential File stage editor.
Lesson checkpoint
In this lesson, you explored a simple data extraction job that reads data from a file and writes it to a staging area. You learned the following tasks: v How to open stage editors v How to view the data that a stage represents v How to compile a job so that it is ready to run
10
2. Select the sample job in the right pane of the Director client, and select Job Run Now.
11
3. In the Job Run Options window, specify the path of the project folder (for example, C:\IBM\InformationServer\Server\Projects\Tutorial and click Run. The job status changes to Running. 4. When the job status changes to Finished, select View Log. 5. Examine the job log to see the type of information that the Director client reports as it runs a job. The messages that you see are either control or information type. Jobs can also have Fatal and Warning messages. The following figure shows the log view of the job.
12
13
Lesson checkpoint
In this lesson you ran the sample job and looked at the results. You learned the following tasks: v How to start the Director client from the Designer client v How to run a job and look at the log file v How to view the data written by the job
Module 1: Summary
You have now opened, compiled, and run your first data extraction job. Now that you have run a data extraction job, you can start creating your own jobs. The next module guides you through the process of creating a simple job that does more data extraction.
Lessons learned
By completing this module, you learned about the following concepts and tasks: v Starting the Designer client. v v v v v Opening an existing job. Compiling the job. Starting the Director client from the Designer client. Running the sample job. Viewing the results of the sample job and seeing how the job extracts data from a comma-separated file and writes it to a staging area.
Additional resources
For more information about the features that you have learned about, see the following guides: v IBM InfoSphere DataStage Designer Client Guide v IBM InfoSphere DataStage Director Client Guide
14
Learning objectives
After completing the lessons in this module, you will understand how to do the following tasks: v Add stages and links to a job. v Specify the properties of the stages and links to determine what they will do when the job is run. v Learn how to specify column metadata. v Consolidate your knowledge of compiling and running jobs. This module should take approximately 90 minutes to complete.
15
Lesson checkpoint
In this lesson you created a job and saved it to a specified place in the repository. You learned the following tasks: v How to create a job in the Designer client. v How to name the job and save it to a folder in the repository tree.
16
a. Select each stage or link. b. Right-click and select Rename. c. Type the new name.
Stage or link Sequential File stage Lookup File Set stage Link New name country_codes country_code_lookup country_codes_data
Always use specific names for your stages and links rather than the default names assigned by the Designer client. Using specific names make your job designs easier to document and easier to maintain. 8. Select File Save to save the job. Your job design should now look something like the one shown in this figure:
Specifying properties and column metadata for the Sequential File stage
You will now edit the first of the stages that you added to specify what the stage does when you run the job. You will also specify the column metadata for the data that will flow down the link that joins the two stages. To edit the stages and add properties and metadata: 1. Double-click the country_codes Sequential File stage to open the stage editor. The editor opens in the Properties tab of the Output page. 2. Select the File property under the Source category. 3. In the File field, type the path name for your project folder (where the data files were copied when the tutorial was set up) and add the name CustomerCountry.csv (for example C:\IBM\InformationServer\Server\ Projects\Tutorial\CustomerCountry.csv), and then press enter. (You can
Chapter 4. Module 2: Designing your first job
17
browse for the path name if you prefer, click the browse button on the right of the File field.) You specified the name of the comma-separated file that the stage reads when the job runs. 4. Select the First Line is Column Name property under the Options category. 5. Click the down arrow next to the First Line is Columns Names field and select True from the list. The row that contains the column names is dropped when the job reads the file. 6. Click the Format tab. 7. In the record-level category, select the Record delimiter string property from the Available properties to add. 8. Select DOS format from the Record delimiter string list. This setting ensures that the file can be read when the engine tier is installed on a UNIX or Linux computer. 9. Click the Columns tab. Because the CustomerCountry.csv file contains only three columns, type the column definitions into the Columns tab. (If a file contains many columns, it is less time consuming and more accurate to import the column definitions directly from the data source.) Note that column names are case-sensitive, so use the case in the instructions. 10. Double-click the first line of the table. Fill in the fields as follows:
Table 1. Column definition Column Name CUSTOMER_ NUMBER Key Yes SQL Type Char Length 7 Description Key column for the look up - the customer identifier
You will use the default values for the remaining fields. 11. Add two more rows to the table to specify the remaining two columns and fill them in as follows:
Table 2. Additional column definitions Column Name COUNTRY Key No SQL Type Char Length 2 Description The code that identifies the customer's country The code that identifies the customer's language
LANGUAGE
No
Char
Your Columns tab should look like the one in the following figure (if you have National Language Support installed, there is an additional field named Extended):
18
12. Click the Save button to save the column definitions that you specified as a table definition object in the repository. The definitions can then be reused in other jobs. 13. In the Save Table Definition window, enter the following information:
Option Data source type Data source name Table/file name Short description Long description Description Saved CustomerCountry.csv country_codes_data date and time of saving Table definition for country codes source file
14. Click OK to specify the locator for the table definition. The locator identifies the table definition. 15. In the Save Table Definition As window, save the table definition in the Tutorial folder and name it country_codes_data. 16. Click the View Data button and click OK in the Data Browser window to use the default settings. The data browser shows you the data that the CustomerCountry.csv file contains. Since you specified the column definitions, the Designer client can read the file and show you the results. 17. Close the Data Browser window. 18. Click OK to close the stage editor. 19. Save the job. Notice that a small table icon has appeared on the Country_codes_data link. This icon shows that the link now has metadata. You have designed the first part of your job.
Chapter 4. Module 2: Designing your first job
19
Specifying properties for the Lookup File Set stage and running the job
In this part of the lesson, you configure the next stage in your job. You already specified the column metadata for data that will flow down the link between the two stages, so there are fewer properties to specify in this task. To configure the Lookup File Set stage: 1. Double-click the country_code_lookup Lookup File Set stage to open the stage editor. The editor opens in the Properties tab of the Input page. 2. Select the Lookup Keys category; then double-click the Key property in the Available Properties to add area. 3. In the Key field, click the down arrow and select CUSTOMER_NUMBER from the list and press enter. You specified that the CUSTOMER_NUMBER column will be the lookup key for the lookup table that you are creating. 4. Select the Lookup File Set property under the Target category. 5. In the Lookup File Set field, type the path name for the lookup file set that the stage will create, (for example, C:\IBM\InformationServer\Server\Projects\ Tutorial\countrylookup.fs) and press enter. 6. Click OK to save your property settings and close the Lookup File Set stage editor. 7. Save the job and then compile and run the job by using the techniques that you learned in Lesson 1. You have now written a lookup table that can be used by another job later on in the tutorial.
Lesson checkpoint
You have now designed and run your very first job. You learned the following tasks: v How to add stages and links to a job v How to set the stage properties that determine what the stage will do when you run the job v How to specify column metadata for the job and to save the column metadata to the repository for use in other jobs
20
Your job design should now look like the one shown in this figure:
Chapter 4. Module 2: Designing your first job
21
3. Open the stage editor for the special_handling Sequential File stage and specify that it will read the file SpecialHandling.csv and that the first line of this file contains column names. 4. Click the Format tab. 5. In the record-level category, select the Record delimiter string property from the Available properties to add. 6. Select DOS format from the Record delimiter string list. This setting ensures that the file can be read when the engine tier is installed on a UNIX or Linux computer. 7. Click the Columns tab. 8. Click Load. You load the column metadata from the table definition that you previously saved as an object in the repository. 9. In the Table Definitions window, browse the repository tree to the folder where you stored the SpecialHandling.csv column definitions. 10. Select the SpecialHandling.csv table definition and click OK. 11. In the Selected Columns window, ensure that all of the columns appear in the Selected columns list and click OK. The column definitions appear in the Columns tab of the stage editor. 12. Close the Sequential File stage editor. 13. Open the stage editor for the special_handling_lookup stage. 14. Specify a path name for the destination file set and specify that the lookup key is the SPECIAL_HANDLING_CODE column then close the stage editor. 15. Save, compile, and run the job.
22
Lesson checkpoint
You have now added to your job design and learned how to import the metadata that the job uses. You learned the following tasks: v How to import column metadata directly from a data source v How to load column metadata from a definition that you saved in the repository
Job parameters
Sometimes, you want to specify information when you run the job rather than when you design it. In your job design, you can specify a job parameter to represent this information. When you run the job, you are then prompted to supply a value for the job parameter. You specified the location of four files in the job that you designed in Lesson 2.3. In each part of the job, you specified a file that contains the source data and a file to write the lookup data set to. In this lesson, you will replace all four file names with job parameters. You will then supply the actual path names of the files when you run the job. You will save the definitions of these job parameters in a parameter set in the repository. When you want use the same job parameters in a job later on in this tutorial, you can load them into the job design from the parameter set. Parameter sets enable the same job parameters to be used by different jobs.
path name for the path name country codes lookup file set
23
Table 3. Job parameters (continued) Parameter Name special_handling_source Prompt Type Help text Enter the path name for the comma-separated file that contains the special handling code definitions Enter the path name for the file set for the special handling lookup table
path name for the path name special handling codes file
special_handling_lookup
path name for the path name special handling lookup file set
The Parameters tab of the Job Properties window should now look like the one in the following figure:
9. Click OK to close the Job Properties window. 10. Click File Save to save the job.
24
4. Select country_codes_source from the list and press enter. The text #country_codes_source# appears in the File field. This text specifies that the job will request the name of the file when you run the job. 5. Repeat these steps for each of the stages in the job, specifying job parameters for each of the File properties as follows:
Table 4. Job parameters to be added to job Stage country_codes_lookup stage special_handling stage special_handling_lookup stage Property Lookup file set File Lookup file set Parameter name country_codes_lookup special_handling_source special_handling_lookup
Lesson checkpoint
You defined job parameters to represent the file names in your job and specified values for these parameters when you ran the job. You learned the following tasks: v How to define job parameters v How to add job parameters in your job design v How to specify values for the job parameters when you run the job
Parameter sets
You use parameter sets to define job parameters that you are likely to reuse in other jobs. Whenever you need this set of parameters in a job design, you can insert them into the job properties from the parameter set. You can also define different sets of values for each parameter set. These parameter sets are stored as files in the InfoSphere DataStage server installation directory and are available to
Chapter 4. Module 2: Designing your first job
25
use in your job designs or when you run jobs that use these parameter sets. If you make any changes to a parameter set object, these changes are reflected in job designs that use this object until the job is compiled. The parameters that a job is compiled with are available when the job is run. However, if you change the design after the job is compiled, the job will link to the current version of the parameter set. You can create parameter sets from existing job parameters, or you can specify the job parameters as part of the task of creating a new parameter set.
26
10. Click OK, specify a repository folder in which to store the parameter set, and then click Save. 11. The Designer client asks if you want to replace the selected parameters with the parameter set that you have just created. Click No. 12. Click OK to close the Job Parameters window. 13. Save the job. You created a parameter set that is available for another job that you will create later in this tutorial. The current job continues to use the individual parameters rather than the parameter set.
Lesson checkpoint
You have now created a parameter set. You learned the following tasks: v How to create a parameter set from a set of existing job parameters v How to specify a set of default values for the parameters in the parameter set
Module 2 Summary
In this module, you designed and ran a data extraction job. You also learned how to create reusable objects such as table definitions and parameters sets that you can include in other jobs that you design.
Lessons learned
By completing this module, you learned about the following concepts and tasks: v Creating new jobs and saving them in the repository v Adding stages and links and specifying their properties v Specifying column metadata and saving it as a table definition to reuse later v Specifying job parameters to make your job design more flexible, and saving the parameters in the repository to reuse later
27
28
Learning objectives
After completing the lessons in this module, you will understand how to do the following tasks: v How to use a Transformer stage to transform data v How to handle rejected data v How to combine data by using a Lookup stage This module should take approximately 60 minutes to complete.
29
1. Create a parallel job, save it as TrimAndStrip, and store it in the tutorial folder in the repository tree. 2. Add two Data Set stages to the design area. 3. Name the Data Set stage on the left GlobalCoBillTo, and name the one on the right int_GlobalCoBillTo. 4. Click Palette Processing to locate and drag a Transformer stage to the design area. 5. Drop the Transformer stage between the two Data Set stages and name the Transformer stage Trim_and_Strip. 6. Right-click the GlobalCoBillTo Data Set stage and drag a link to the Transformer stage. This method of linking the stages is fast and easy. You do not need to go back to the palette and grab a link to connect each stage. 7. Use the same method to link the Transformer stage to the int_GlobalCoBillTo Data Set stage. 8. Name the first link full_bill_to and name the second link stripped_bill_to. Your job should look like the one in the following picture:
30
5. Open the stage editor for the int_GlobalCoBillTo Data Set stage. 6. Set the File property in the Source category to point to a new staging data set (for example, C:\IBM\InformationServer\Server\Projects\Tutorial\ int_GlobalCoBillTo.ds).
By specifying stricter data typing for your data, you will be able to better diagnose inconsistencies in your source data when you run the job. 5. Double-click the Derivation field for the CUSTOMER_NUMBER column in the stripped_bill_to link. The expression editor opens.
Chapter 5. Module 3: Designing a transformation job
31
6. In the expression editor, type the following text: trim(full_bill_to.CUSTOMER_NUMBER,' ','A'). The text specifies a function that deletes all the space characters from the CUSTOMER_NUMBER column on the full_bill_to link before writing it to the CUSTOMER_NUMBER column on the stripped_bill_to link. Your Transformer stage editor should look like the one in the following figure:
7. Click OK to close the Transformer stage editor. 8. Open the stage editor for the int_GlobalCoBillTo Data Set stage and go to the Columns tab of the Input page. Notice that the stage editor has acquired the metadata from the stripped_bill_to link. 9. Save and then compile your TrimAndStrip job.
32
Lesson checkpoint
In this lesson you learned how to design and configure a transformation job. You learned the following tasks: v How to configure a Transformer stage v How to link stages by using a different method for drawing links. v How to load column metadata into a link, by using a drag-and-drop operation. v How to run a job from within the Designer client and monitor the performance of the job.
33
tutorial. When you use lookup file sets, you must specify the lookup key column when you define the file set. You defined the key columns for the lookup tables that you used in this lesson when you created the file sets in Module 2.
34
35
1. Double-click the Lookup_Country Lookup stage to open the Lookup stage editor. The Lookup stage editor is similar in appearance to the Transformer stage editor. 2. Click the title bar of the stripped_bill_to link in the left pane and drag it over to the Column Name column of the country_code link in the right pane. When the cursor changes shape, release the mouse button. All of the columns from the stripped_bill_to link appear in the country_code link. 3. Select the Country column in the Country_Reference link and drag it to the country_code link. The result of copying the columns from the Country_Reference link to the country_code link is that whenever the value of the incoming CUSTOMER_NUMBER column matches the value of the CUSTOMER_NUMBER column of the lookup table, the corresponding Country column will be added to that row of data. The stage editor looks like the one in the following figure:
36
4. Double-click the Condition bar in the Country_Reference link. The Lookup Stage Conditions window opens. Select the Lookup Failure field and select Continue from the list. You are specifying that, if a CUSTOMER_NUMBER value from the stripped_bill_to link does not match any CUSTOMER_NUMBER column values in the reference table, the job continues to the next CUSTOMER_NUMBER column. 5. Close the Lookup stage editor. 6. Open the temp_dataset Data Set stage and specify a file name for the data set. 7. Save, compile and run the job. The Job Run Options window displays all the parameters in the parameter set.
37
8. In the Job Run Options window, select lookupvalues1 from the list next to the parameter set name. The parameters values are filled in with the path names that you specified when you created the parameter set. 9. Click Run to run the job and then click View Data in the temp_dataset stage to examine the results.
Lesson checkpoint
With this lesson, you started to design more complex and sophisticated jobs. You learned the following tasks: v How to copy stages, links, and associated configuration data between jobs. v How to combine data in a job by using a Lookup stage.
38
3. Double-click the Lookup_Country Lookup stage to open the Lookup stage editor. 4. Double-Click the Condition bar in the country_reference link to open the Lookup Stage Conditions window. 5. In the Lookup Stage Conditions window, select the Lookup Failure field and select Reject from the list. Close the Lookup stage editor. This step specifies that, whenever a row from the stripped_bill_to link has no matching entry in the country code lookup table, the row is rejected and written to the Rejected_Rows Sequential File stage. 6. Edit the Rejected_Rows Sequential File stage and specify a path name for the file that the stage will write to (for example, c:\tutorial\rejects.txt). This stage derives the column metadata from the Lookup stage, and you cannot alter it. 7. Save, compile the CleansePrepare job, and run the job. 8. Open the Rejected_Rows Sequential File stage editor and click View Data to look at the rows that were rejected.
Lesson checkpoint
You learned the following tasks: v How to add a reject link to your job v How to configure the Lookups stage so that it rejects data where a lookup fails
39
40
2. Select the following columns in the country_code input link and drag them to the with_business_rules output link: v v v v v v CUSTOMER_NUMBER CUST_NAME ADDR_1 ADDR_2 CITY REGION_CODE
v ZIP v TEL_NUM 3. In the metadata area for the with_business_rules output link, add the following new columns:
Table 6. Column definitions Column name SOURCE RECNUM SETUP_DATE SQL Type Char Char Char Length 10 10 10 Nullable No No Yes
41
Table 6. Column definitions (continued) Column name SQL Type Length 10 Nullable Yes
The new columns appear in the graphical representation of the link, but are highlighted in red because they do not yet have valid derivations. 4. In the graphical area, double-click the Derivation field of the SOURCE column. 5. In the expression editor, type 'GlobalCo':. Position your mouse pointer immediately to the right of this text, right-click and select Input Column from the menu. Then select the COUNTRY column from the list. When you run the job, the SOURCE column for each row will contain the two-letter country code prefixed with the text GlobalCo, for example, GlobalCoUS. 6. In the Transformer stage editor toolbar, click the Stage Properties tool on the far left. The Transformer Stage Properties window opens. 7. Click the Variables tab and, by using the techniques that you learned for defining table definitions, add the following stage variables to the grid:
Table 7. Stage variables Name xtractSpecialHandling TrimDate SQL Type Char VarChar Precision 1 10
When you close the Properties window, these stage variables appear in the Stage Variables area above the with_business_rules link. 8. Double-click the Derivation fields of each of the stage variables in turn and type the following expressions in the expression editor:
Table 8. Derivations Stage variable xtractSpecialHandling Expression if Len (country_code.SETUP_DATE) < 2 Then country_code.SETUP_DATE Else Field (country_code.SETUP_DATE,' ',2) Description This expression checks that the SETUP_DATE column contains a special handling code. If it does, the value of xtractSpecialHandling is set to that code. If the column contains a date and a code, the code is extracted and the value of xtractSpecialHandling is set to that code. This expression checks that the SETUP_DATE column contains a date. If the SETUP_DATE column does not contain a date, then the expression sets the value of the TrimDate variable to the string 01/01/0001. If the SETUP_DATE column contains a date, then the date is extracted and the value of the TrimDate variable is set to a string that contains the date.
TrimDate
42
9. Select the xtractSpecialHandling stage variable and drag it to the Derivation field of the SPECIAL_HANDLING_CODE column and drop it on the with_business_rules link. A line is drawn between the stage variable and the column, and the name xtractSpecialHandling appears in the Derivation field. For each row that is processed, the SPECIAL_HANDLING_CODE column writes the current value of the xtractSpecialHandling variable. 10. Select the TrimDate stage variable and drag it to the Derivation field of the SETUP_DATE column and drop it on the with_business_rules link. A line is drawn between the stage variable and the column, and the name TrimDate appears in the Derivation field. For each row processed, the SETUP_DATE column writes the current value of the TrimDate variable. 11. Double-click the Derivation field of the RECNUM column and type 'GC': in the expression editor. Right-click and select System Variable from the menu. Then select @OUTROWNUM. You added row numbers to your output. Your transformer editor should look like the one in the following picture:
43
44
6. Select the following columns in the with_business_rules input link and drag them to the finished_data output link. v CUSTOMER_NUMBER v CUST_NAME v ADDR_1 v ADDR_2 v CITY v REGION_CODE v ZIP v TEL_NUM v SOURCE v RECNUM v SETUP_DATE v SPECIAL_HANDLING_CODE 7. Select the DESCRIPTION column in the special_handling reference link and drag it to the finished_data output link (the LANGUAGE column is not used). 8. Double-click the Condition bar in the special_handling reference link to open the Lookup Stage Conditions window. Specify that the processing will continue if the lookup fails for a data row. You do not need to specify a reject link for this stage. Only a minority of the rows in the bill_to data contain a special handling code, so if the rows that do not contain a code are rejected, most of the data is rejected. 9. Specify a job parameter to represent the file that the Target Sequential File stage will write to, and add this job parameter to the stage. 10. Save, compile and run the CleansePrepare job.
Lesson checkpoint
In this lesson, you consolidated your existing skills in defining transformation jobs and added some new skills. You learned the following tasks: v How to define and use stage variables in a Transformer stage v How to use system variables to generate output column values
Module 3 summary
In this module you refined and added to your job design skills. You learned how to design more complex jobs that transform the data that your previous jobs extracted.
Lessons learned
By completing this module, you learned the following concepts and tasks: v How to drop data columns from your data flow v How to use the transform functions that are provided with the Designer client v How to combine data from two different sources v How to capture rejected data
45
46
Learning objectives
After completing the lessons in this module, you will understand how to do the following tasks: v How to define a data connection object that you use and reuse to connect to a database. v How to import column metadata from a database. v How to write data to a relational database target. This module should take approximately 60 minutes to complete.
Prerequisites
Ensure that your database administrator runs the relevant database scripts that are supplied with the tutorial and set up a DSN for you to use when connecting to an ODBC connector.
47
changes to the job design will once again link to the data connection object and pick up any changes that were made to that object. You can create data connection objects directly in the repository. Also, you can create data connection objects when you are using a connector stage to import metadata by saving the connection details. This lesson shows you how to create a data connection object directly in the repository.
7. Click OK. 8. In the Save Data Connection As window, select the tutorial folder and click Save.
Lesson checkpoint
You learned how to create a data connection object and store the object in the repository.
48
To import column metadata by using the ODBC connector: 1. Select Import Table Definitions Start Connector Import Wizard. 2. In the Data Source Location page of the Import Connector Metadata wizard, select the computer that hosts the database from the Host name where database resides list. 3. Click the New location link. 4. In the Shared Metadata Management window, select your host name and click Add new database. 5. In the Add new database window, type the name of the database that has been created on the relational database for this exercise (ask your database administrator if you do not know the name of the database) and click OK, then click Close to close the Shared Metadata Management window. 6. In the Data Source Location page, select the database from the Database name list and click Next. 7. In the Connector Selection page, select ODBC Connector from the list and click Next. 8. In the Connection details page, select your DSN from the Data source list and click the Load link. 9. In the Open window, open the tutorial folder, select the data connection object that you created in Lesson 4.1, and click Open. The Data source, Username, and Password fields are populated with the corresponding data in the data connection object. 10. Click the Test Connection link to ensure that you can connect to the database by using the connection details and then click Next. 11. In the Filter page, select the schema from the Schema list (ask your database administrator if you do not know the name of the schema) and click Next. 12. In the Selection page, select the tutorial table from the list and click Next. 13. In the Confirm import page, review the import details, and then click Import. 14. In the Select Folder window, select the tutorial folder and click OK. The table definition is imported and appears in the tutorial folder. The table definition has a different icon from the table definitions that you used previously. This icon identifies that the table definition was imported by using a connector and is available to other projects and to other suite components.
Lesson checkpoint
You learned how to import column metadata from a database using a connector.
Connectors
Connectors are stages that you use to connect to data sources and data targets to read or write data.
Chapter 6. Module 4: Loading a data target
49
In the Database section of the palette in the Designer are many types of stages that connect to the same types of data sources or targets. For example, if you click the down arrow next to the ODBC icon in the palette, you can choose to add either an ODBC connector stage or an ODBC Enterprise stage to your job. If your database type supports connector stages, use them because they provide the following advantages over other types of stages: v Creates job parameters from the connector stage (without first defining the job parameters in the job properties). v Saves any connection information that you specify in the stage as a data connection object. v Reconciles data types between source and target to avoid runtime errors. v Generates detailed error information if a connector encounters problems when the job runs.
50
3. In the Columns page, click Load. 4. In the Table Definitions window, open the tutorial folder, select the table definition that you created in Lesson 4.2, and click OK. The columns grid is populated with the column metadata. 5. Click OK to close the stage editor.
51
d. Click the SQL tab to view the SQL statement; then click OK to close the SQL builder. The SQL statement is displayed in the Insert statement field, and your ODBC connector should look like the one in the following figure:
52
9. Click OK to close the ODBC Connector. 10. Save, compile, and run the job. You wrote the BillTo data to the tutorial database table. This table forms the bill_to dimension of the star schema that is being implemented for the GlobalCo delivery data in the business scenario that the tutorial is based on.
Lesson checkpoint
You learned how to use a connector stage to connect to and write to a relational database table. You learned the following tasks: v How to configure a connector stage v How to use a data connection object to supply database connection details
Chapter 6. Module 4: Loading a data target
53
v How to use the SQL builder to define the SQL statement by accessing the database.
Module 4 summary
In this module, you designed a job that writes data to a table in a relational database. In lesson 4.1, you learned how to define a data connection object; in lesson 4.2, you imported column metadata from a database; and in lesson 4.3, you learned how to write data to a relational database target.
Lessons learned
By completing this module, you learned about the following concepts and tasks: v Load data into data targets v Use the Designer clients reusable components
54
Learning objectives
After completing the lessons in this module, you will know how to do the following: v How to use the configuration file to optimize parallel processing. v How to control parallel processing at the stage level in your job design. v How to control the partitioning of data so that it can be handled by multiple processors. This module should take approximately 60 minutes to complete.
Prerequisites
You must be working on a computer with multiple processors. You must have DataStage administrator privileges to create and use a new configuration file.
55
2. In the Configuration window, select default from the list. The contents of the default configuration file are displayed.
The default configuration file is created when InfoSphere DataStage is installed. Although the system has four processors, the configuration file specifies two processing nodes. Specify fewer processing nodes than there are physical processors to ensure that your computer has processing resources available for other tasks while it runs InfoSphere DataStage jobs. This file contains the following fields: node The name of the processing node that this entry defines.
fastname The name of the node as it is referred to on the fastest network in the system. For an SMP system, all processors share a single connection to the network, so the fastname node is the same for all the nodes that you are defining in the configuration file. pools Specifies that nodes belong to a particular pool of processing nodes. A pool of nodes typically has access to the same resource, for example, access to a high-speed network link or to a mainframe computer. The pools string is empty for both nodes, specifying that both nodes belong to the default pool.
resource disk Specifies the name of the directory where the processing node will write data set files. When you create a data set or file set, you specify where the controlling file is called and where it is stored, but the controlling file points to other files that store the data. These files are written to the directory that is specified by the resource disk field. resource scratchdisk Specifies the name of a directory where intermediate, temporary data is stored. Configuration files can be more complex and sophisticated than the example file and can be used to tune your system to get the best possible performance from the parallel jobs that you design.
56
Lesson checkpoint
In this lesson, you learned how the configuration file is used to control parallel processing. You learned the following concepts and tasks: v About configuration files v How to open the default configuration file v What the default configuration file contains
57
4. Click the disk icon in the toolbar to open the Data Set viewer
and click OK. 5. View the data in the data set to see its structure. 6. Close the window.
58
To 1. 2. 3. 4.
specify round-robin partitioning: Open the sample job that you used in Module 1. Open the GlobalCoBillTo_ds Data Set stage editor. Open the Partitioning tab of the Input page. In the Partition type field, select the round-robin partitioning method.
5. Compile and run the job. 6. Return to the data set management tool and open the GlobalCo_BillTo.ds data set. You can see that the data set now has multiple data partitions. The following figure shows the data set partitions on the system.
Lesson checkpoint
In this lesson, you learned some basics about data partitioning. You learned the following tasks: v How to use the data set management tool to view data sets v How to set a partitioning method for a stage
59
60
61
You deployed your new configuration file. Keep the Administrator client open, because you will use it to restore the default configuration file at the end of this lesson.
Lesson checkpoint
You learned how to create a configuration file and use it to alter the operation of parallel jobs. You learned the following tasks: v How to create a configuration file based on the default file. v How to edit the configuration file.
62
Module 5 summary
In this module, you learned how to use the configuration file to control how your parallel jobs are run. You also learned how to control the partitioning of data at the level of individual stages.
Lessons learned
By completing this module, you learned about the following concepts and tasks: v The configuration file v How to use the configuration editor to edit the configuration file v How to control data
63
64
Lessons learned
By completing this tutorial, you learned about the following concepts and tasks: v How to extract, transform, and load data by using InfoSphere DataStage. v Using the parallel processing power of InfoSphere DataStage. v How to reuse the job design elements
65
66
5. Creating a target database table on page 68 6. One of the following tasks: v Creating a DSN for the tutorial table on a Windows computer on page 69 v Creating a DSN for the tutorial table on a UNIX or Linux computer on page 69
67
68
1. Open the administrator client for your database (for example, the DB2 Control Center). 2. Create a new database named Tutorial. 3. Connect to the new database. 4. Run the appropriate DDL script to create the tutorial table in the new database. The scripts are in the tutorial folder and are named as follows: v DB2_table.ddl v Oracle_table.ddl v SQLserver_table.ddl 5. Close the database administrator client.
5. In the Create New Data Source window, select a driver for the database. A window opens that is specific to the driver that you selected. 6. Specify the details that the driver requires to connect to the tutorial database. If you specified a user name and password for the database, then the connection details include user name and password.
69
70
Contacting IBM
You can contact IBM for customer support, software services, product information, and general information. You also can provide feedback to IBM about products and documentation. The following table lists resources for customer support, software services, training, and product and solutions information.
Table 10. IBM resources Resource IBM Support Portal Description and location You can customize support information by choosing the products and the topics that interest you at www.ibm.com/support/ entry/portal/Software/ Information_Management/ InfoSphere_Information_Server You can find information about software, IT, and business consulting services, on the solutions site at www.ibm.com/ businesssolutions/ You can manage links to IBM Web sites and information that meet your specific technical support needs by creating an account on the My IBM site at www.ibm.com/account/ You can learn about technical training and education services designed for individuals, companies, and public organizations to acquire, maintain, and optimize their IT skills at http://www.ibm.com/software/swtraining/ You can contact an IBM representative to learn about solutions at www.ibm.com/connect/ibm/us/en/
Software services
My IBM
IBM representatives
Providing feedback
The following table describes how to provide feedback to IBM about products and product documentation.
Table 11. Providing feedback to IBM Type of feedback Product feedback Action You can provide general product feedback through the Consumability Survey at www.ibm.com/software/data/info/ consumability-survey
71
Table 11. Providing feedback to IBM (continued) Type of feedback Documentation feedback Action To comment on the information center, click the Feedback link on the top right side of any topic in the information center. You can also send comments about PDF file books, the information center, or any other documentation in the following ways: v Online reader comment form: www.ibm.com/software/data/rcf/ v E-mail: comments@us.ibm.com
72
73
74
Product accessibility
You can get information about the accessibility status of IBM products. The IBM InfoSphere Information Server product modules and user interfaces are not fully accessible. The installation program installs the following product modules and components: v IBM InfoSphere Business Glossary v IBM InfoSphere Business Glossary Anywhere v IBM InfoSphere DataStage v IBM InfoSphere FastTrack v v v v IBM IBM IBM IBM InfoSphere InfoSphere InfoSphere InfoSphere Information Analyzer Information Services Director Metadata Workbench QualityStage
For information about the accessibility status of IBM products, see the IBM product accessibility information at http://www.ibm.com/able/product_accessibility/ index.html.
Accessible documentation
Accessible documentation for InfoSphere Information Server products is provided in an information center. The information center presents the documentation in XHTML 1.0 format, which is viewable in most Web browsers. XHTML allows you to set display preferences in your browser. It also allows you to use screen readers and other assistive technologies to access the documentation.
75
76
Notices
IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to evaluate and verify the operation of any non-IBM product, program, or service. IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not grant you any license to these patents. You can send license inquiries, in writing, to: IBM Director of Licensing IBM Corporation North Castle Drive Armonk, NY 10504-1785 U.S.A. For license inquiries regarding double-byte character set (DBCS) information, contact the IBM Intellectual Property Department in your country or send inquiries, in writing, to: Intellectual Property Licensing Legal and Intellectual Property Law IBM Japan Ltd. 1623-14, Shimotsuruma, Yamato-shi Kanagawa 242-8502 Japan The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you. This information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice. Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web
77
sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk. IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you. Licensees of this program who wish to have information about it for the purpose of enabling: (i) the exchange of information between independently created programs and other programs (including this one) and (ii) the mutual use of the information which has been exchanged, should contact: IBM Corporation J46A/G4 555 Bailey Avenue San Jose, CA 95141-1003 U.S.A. Such information may be available, subject to appropriate terms and conditions, including in some cases, payment of a fee. The licensed program described in this document and all licensed material available for it are provided by IBM under terms of the IBM Customer Agreement, IBM International Program License Agreement or any equivalent agreement between us. Any performance data contained herein was determined in a controlled environment. Therefore, the results obtained in other operating environments may vary significantly. Some measurements may have been made on development-level systems and there is no guarantee that these measurements will be the same on generally available systems. Furthermore, some measurements may have been estimated through extrapolation. Actual results may vary. Users of this document should verify the applicable data for their specific environment. Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. All statements regarding IBM's future direction or intent are subject to change or withdrawal without notice, and represent goals and objectives only. This information is for planning purposes only. The information herein is subject to change before the products described become available. This information contains examples of data and reports used in daily business operations. To illustrate them as completely as possible, the examples include the names of individuals, companies, brands, and products. All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental. COPYRIGHT LICENSE: This information contains sample application programs in source language, which illustrate programming techniques on various operating platforms. You may copy, modify, and distribute these sample programs in any form without payment to
78
IBM, for the purposes of developing, using, marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore, cannot guarantee or imply reliability, serviceability, or function of these programs. The sample programs are provided "AS IS", without warranty of any kind. IBM shall not be liable for any damages arising out of your use of the sample programs. Each copy or any portion of these sample programs or any derivative work, must include a copyright notice as follows: (your company name) (year). Portions of this code are derived from IBM Corp. Sample Programs. Copyright IBM Corp. _enter the year or years_. All rights reserved. If you are viewing this information softcopy, the photographs and color illustrations may not appear.
Trademarks
IBM, the IBM logo, and ibm.com are trademarks of International Business Machines Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at www.ibm.com/legal/copytrade.shtml. The following terms are trademarks or registered trademarks of other companies: Adobe is a registered trademark of Adobe Systems Incorporated in the United States, and/or other countries. Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both. UNIX is a registered trademark of The Open Group in the United States and other countries. Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. The United States Postal Service owns the following trademarks: CASS, CASS Certified, DPV, LACSLink, ZIP, ZIP + 4, ZIP Code, Post Office, Postal Service, USPS and United States Postal Service. IBM Corporation is a non-exclusive DPV and LACSLink licensee of the United States Postal Service. Other company, product or service names may be trademarks or service marks of others.
79
80
Index A
adding job parameters 23 adding stages 16 Administrator client 67 starting 61
I
importing column metadata 20, 48 tutorial components 68 installing 67
repository 5 repository objects data connections 47 parameter set 25 table definition 20, 48 running jobs 10
C
column metadata importing 20, 48 loading 21 comma-separated files 68 compiling jobs 9 configuration file 55, 60 default 55 configurations viewer 55, 60 connector ODBC 49 configuring 51 creating job parameters 26 creating a job 15 customer support 71
J
job parameters adding 23 parameter sets 25 job properties 23 jobs compiling 9 creating 15 opening 5 running 10
S
sample job 5 Sequential File stage 8 setting up 67 software services 71 source data files 68 SQL Builder 51 stage Lookup File Set 20, 35 Transformer 31 stage properties 17 stages adding 16 starting the Designer client support customer 71
L
legal notices 77 loading column metadata 21 lookup 33 Lookup File Set stage 20, 35 Lookup stage 33
D
data combining 33 looking up 33 partitioning 57 reject 38 stage Lookup 33 data browser 14 data connection creating 47 data files 68 data set management tool Data Set stage 9 database 68 Designer client 5, 68 starting 6 Director client starting 10 DSN 69
T
table definition 20, 48 Transformer stage 31 tutorial folder 67
M
metadata importing 20, 48 loading 21
V
viewing data 14
O
ODBC 49, 51, 69 opening a job 5 57
P
parameter set creating 26 parameter sets 25 partitioning data 57 partitions creating 58 product accessibility accessibility 75 project creating 67 properties job 23 stage 17
E
environment variables 61
F
files data 68 folder 67
R
reject data 38 relational database 68
81
82
Printed in USA
SC18-9889-02