2/7/13

different formats of the flat file for the same target

Register | Login

Not endorsed by or affiliated w ith SAP

De live ring Be tte r Busine ss De cisions (O pe ns a ne w window)

Index | About

Sponsors | C ontact Us | C alendar | FAQ | Rules and Guidelines | Privacy Policy | Search

General Notice: We are proud to introduce the new Blogger Link Program for BOB. If you write a blog related to business intelligence and would like to register for this program, read the details here. Thanks. General Notice: Upcoming Events: ASUG Ontario: Feb 12. There are 5 ASUG sponsored events in the next 45 days, see calendar for details.

different formats of the flat file for the same target

Register or Login to Post Author bobj1008
Forum Me m be r

Search this topic... | Search DI: Designer and Job Design... | Search Box

Forum Index -> Data Integrator -> DI: Designer and Job Design Message

Poste d: Tue Aug 17, 2010 1:24 pm Post subje ct: diffe re nt form ats of the flat file for the sam e targe t

Joine d: 17 Aug 2010 Posts: 1

In our deployment, we use plugin code to extract the csv files in the required format. The customers are on same version of datamart, but they are on different versions of source database - from 3.x to 4.5 depending on which version of application they are using. In 4.0, we introduced a new column email in the user table in the source database. Accordingly, plugin will add the field in the csv file. But not all the customers will get the upgraded version of plugin at the same time. So ETL code needs to decide which data flow to process depending on the format of the csv file to load data to the same target table. I made the email field in the target table nullable but it still expects the same format of the csv file with delimiter for null value. Need help to achieve this. Can I read the structure of the flat file in DS or get the count of delimiters so that I can use a conditional to use different data flow based on the format of the flat files. Can I make the email column in the flat file optional? Thanks much in advance.

Back to top Cirno

Poste d: W e d Aug 25, 2010 9:26 am Post subje ct: R e : diffe re nt form ats of the flat file for the sam e targe t

Se nior Me m be r

For delimited file formats you can set the 'Adaptable schema' flag and it won't complain if you don't have all the columns. It also won't complain if you are missing a big piece of a line so keep that in mind.
Joine d: 25 Jul 2008 Posts: 53 Location: Unde fine d Fantastic O bje ct

Back to top CLS69

Poste d: Mon Aug 30, 2010 5:16 pm Post subje ct: R e : diffe re nt form ats of the flat file for the sam e targe t

Principal Me m be r

In order to understand which version of flat file you are reading, you might also do the following: 1. make an extra file format where you read the whole line in one field 2. read only the first line and perform a set of different "match_regex" of the line so to understand which fields it contains.
Joine d: 11 Jun 2009 Posts: 194 Location: Italy

Once you got the version, you know how to process it. Ciao Claudio

Back to top Shivbaba

Poste d: Fri Se p 10, 2010 1:34 am Post subje ct: R e : diffe re nt form ats of the flat file for the sam e targe t

Principal Me m be r

Hi,

www.forumtopics.com/busobj/viewtopic.php?t=161304&highlight=csv+files

1/3

2/7/13

Hi,

different formats of the flat file for the same target

Joine d: 01 Jun 2010 Posts: 132

I am also facing the same situation.... In my case, I need to compare the header (1st row of the file ) with the columns defined in the input format. The order and name of the columns defined in the file format must match with order and name of the columns in the input file. If it doesn't match, I should get some error message out and terminate the job immediately. All suggestions invited... Thanks in advance...

Back to top CLS69

Poste d: Fri Se p 10, 2010 7:38 am Post subje ct: R e : diffe re nt form ats of the flat file for the sam e targe t

Principal Me m be r

Considering that you the file format definition, you might check the first row against a regex like the following (assuming tab as column separator): <first col name>\t<second col name>\t....\t<last col name>
Joine d: 11 Jun 2009 Posts: 194 Location: Italy

and the file will match if and only if the outcome is 1.

Back to top Shivbaba

Poste d: Fri Se p 10, 2010 7:53 am Post subje ct: R e : diffe re nt form ats of the flat file for the sam e targe t

Principal Me m be r

Hi, Thanks for your reply... I am sorry but I don't know the syntax of match_regex function. Can you please elaborate on how to use this function..Can you please give the exact syntax...

Joine d: 01 Jun 2010 Posts: 132

Back to top CLS69

Poste d: Fri Se p 10, 2010 12:36 pm Post subje ct: R e : diffe re nt form ats of the flat file for the sam e targe t

Principal Me m be r

match_regex(string to match,regex, NULL) the third parameter set to NULL will fit mostly all the times
Joine d: 11 Jun 2009 Posts: 194 Location: Italy

of course, in order to match the header, you need an additional file format where the whole row is read in one field. hth Claudio

Back to top Shivbaba

Poste d: Sun Se p 12, 2010 11:31 pm Post subje ct: R e : diffe re nt form ats of the flat file for the sam e targe t

Principal Me m be r

OK... Thanks for the reply.. I will try it out and revert back...
Joine d: 01 Jun 2010 Posts: 132

Back to top FreekIM

Poste d: Tue Se p 21, 2010 2:52 am Post subje ct: R e : diffe re nt form ats of the flat file for the sam e targe t

Forum Me m be r

I am new to data integrator and i am facing a similar problem, I have got 5 different Excel versions that need to be loaded into 1 source. The files havent got unique

www.forumtopics.com/busobj/viewtopic.php?t=161304&highlight=csv+files

2/3

2/7/13
Joine d: 21 Se p 2010 Posts: 2

different formats of the flat file for the same target

characteristics in the filename. So the different versions must be determined based on the content of the files. I have loaded all files into a flat table, now based on some unique characteristics within the file i must determine what format to use for further handling. How can i make this work?

Back to top
Display posts from pre vious:

All Posts

Oldest First

Go

Register or Login to Post Page 1 of 1 Select a search

Forum Index -> Data Integrator -> DI: Designer and Job Design
All tim e s are GMT - 5 Hours

Go

Jum p to:

DI: Designer and Job Design

Go

Index | About | FAQ | RAG | Privacy | Search | Register | Login Get community updates via Twitter: Follow @BOB_Board 443 follow ers

Not endorsed by or affiliated w ith SAP Pow ered by phpBB phpBB Group Generated in 0.7976 seconds using 16 queries. (SQL 0.0123 Parse 0.7319 Other 0.0534) Mozilla/5.0 (Window s NT 6.1; WOW64) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.56 Safari/537.17

Hosted by ForumTopics.com | Terms of Service phpBB Customizations by the phpBBDoctor.com Shameless plug for MomentsOfLight.com

www.forumtopics.com/busobj/viewtopic.php?t=161304&highlight=csv+files

3/3