Key Stages to the Data Collections Process

Refresh Collection Snapshots

1.       This process is launched by the stage Planning Data Pull to refresh the
snapshots used to collect the data.
2.       The snapshots work in conjunction with the views to provide the Planning Data
Pull with the information that will be loaded into the staging tables.
3.       If the setup profile MSC: Source Setup Required = Yes (or Y in 11.5.10 and
below), then this process will also launch the Setup Requests that create all the
objects used by Data Collections in the EBS Source applications.
7.       Processes Launched by Refresh Collection Snapshots
a.       Standard processes
1.       Refresh Collection Snapshots
2.       Refresh Collections Snapshots Thread .

Data Pull
1.       When the Snapshot process is finished, then the Planning Data Pull will move
the data from the Snapshots and Views to the Staging Tables - all names starting
with MSC_ST%
4.       Processes Launched by Planning Data Pull
a.    This launches the Refresh Collection Snapshots request, which must
complete before any other workers planning data pull processes can
launch. The planning data pull with go to status – Pending while waiting for
this to complete.
b.    Note that the timeout parameter (default 180 minutes) does not count time
used for Refresh Collection Snapshots request – so if Refresh takes 45
minutes and Data Pull takes 150 minutes, it will not time out.
c.     When Snapshot completes, then it launches the Planning Data Pull
Worker requests, and it performs some those same tasks as well.
d.    When all workers are finished, then it finishes, it reports back to the main
request, which will launch the Planning ODS Load for the next stage of
collections.

Planning ODS Load

1.       This will launch and perform Key Transformations of critical data into unique
keys for the VCP applications. For instance, since we can collect from multiple
instances, the INVENTORY_ITEM_ID in MSC_SYSTEM_ITEMS must be given a
unique value. This happens for several entities. We store the Source instance
key values in columns with SR_ like SR_INVENTORY_ITEM_ID in
MSC_SYSTEM_ITEMS
a.       These Key transformations are split into separate requests starting in
12.1
 Generate Items and Category Sets Keys
Generate Trading Partner Keys
2.       Then ODS then launches Planning ODS Loader Worker requests to handle the
many different load tasks that move data from the MSC_ST staging tables to the
base MSC tables.
3.       During complete refresh we create TEMP tables to copy data from MSC_ST
staging tables into the TEMP tables, then use Exchange Partition technology to
flip this temp table into the partitioned table.
a.       For instance, if your instance code is TST, you can see in the log file
where we create temp table SYSTEM_ITEMS_TST,
b.      Then we move data into this table from MSC_ST_SYSTEM_ITEMS
5.       The ODS Load will launch Planning Data Collections - Purge Staging Tables to
remove data from the MSC_ST staging tables and if Data Collections fails at any
stage then you should run Planning Data Collections - Purge Staging Tables
with parameter Validation = No to reset the process and get ready to launch it
again. If you miss this step then it will error with messages like:
a.       Another planning data pull process is running.
b.      Staging tables data not collected. Launch ODS Load

Complete, Net Change and Targeted Refresh

1.       Complete Refresh replaces all data for the entity by deleting the base table data
and replacing with data from the staging table, with a few exceptions. These
exception exist for Global entities because we can collect data from multiple source
instances:
a.       Items are never deleted in background table MSC_ITEMS.
b.      Trading Partners and Trading Partner Sites are never deleted.
-- MSC_TRADING_PARTNER_SITES 
-- MSC_TRADING_PARTNERS  - For partner_type = 1.2, and 4  ( 1 = Supplier;
2 = Customer; 4= Carrier / For 3 = Organization we do delete and replace this
type)
c.       Unit of Measure related tables are another Global Entity that is not ever
deleted during Complete Refresh. 
-- MSC_UNITS_OF_MEASURE, MSC_UOM_CONVERSIONS,
MSC_UOM_CLASS_CONVERSIONS
2.       Sales orders are special entity. When you run with Complete Refresh, in most
releases we will default Sales Orders parameter – No, AND we will still collect sales
order – but in Net Change mode. This is done for performance reasons.  SO -- If you
need to refresh sales orders in Complete mode, then you will have to explicitly set
this parameter to Yes -- OR you can run standalone Targeted Refresh of only Sales
Orders.
3.       For Refresh Mode = Net Change or Targeted, you must set Planning Data Pull
parameter Purge Previously Collected Data = No.
4.       Net Change picks up only the changes to the entities.
a.       This can only be run for transactional entities, like WIP, PO, OM and some
setup entities like, Items, BOM, etc. The XLS attachment in Note 179522.1
explains which entities can be collected in Net Change mode.
b.      If you set parameter Yes for a setup entity - like ATP Rules - it is ignored
when running Net Change collections.
c.       By design, Sales Orders are always collected for data integrity and
performance reasons and you may not set Sales Order entity – No in Net
Change Collections
5.       Targeted Refresh is used to collect one of more entities in complete refresh mode.
This is very useful to synching data for certain business purposes, or collecting
certain setup data, or quick collection for setting up a test case.
Examples:
a.       Business Purpose - Customer runs an ASCP Plan intraday at 1 pm, after
running Net Change collections, they run a Targeted Refresh of Customers,
Suppliers, Orgs to get all new customers created that morning.
b.      We run Net Change Collections during the week, and Complete Collections
on weekends, but during the week, we need to collect new Sourcing rules
and Customers daily, so we run targeted refresh each night.
c.       I need to change Sourcing rules and setup a Supplier capacity calendar and
run a testcase. You can collect Approved Supplier List, Calendars, and
Sourcing Rules to get these changes quickly into the MSC tables for testing
without running a complete refresh of all entities.
6.       OPM customers cannot use Targeted Refresh or Net Change Refresh. They must
use Complete Refresh to collect OPM data.