Data Source Identification and Design

Developers spend a great deal of effort to ensure that the data from other systems is mapped correctly to TIBCO MDM. You can import such external data using data source.

Most data sources have already been identified for use in TIBCO MDM and new ones are built to fill in gaps or present the existing information better. The following best practices should help design data sources to gain maximum benefit:

  • Whenever possible identify which data sources present the most accurate collection of data for any repositories and use it to populate your primary key fields and any other data fields.
  • Use the same data source for any tables that store similar information. For example, if you have a data source that provides all customer ID information, use this for all tables (such as Business and Normal customers) where the ID is required. Splitting it over two or more data sources gives rise to inconsistencies and degrades the quality of the data within TIBCO MDM.
  • You can join more than one data source to merge data into a single repository. You can then map different data source data to different parts of the repository in just one action.
  • Do not transform the data while mapping data source to input map using input map expressions, which is slow and has limited functionality. Instead use rulebase during import to transform the data.
  • If a lot of data transformation and lookups are required, prepare the data before importing it into TIBCO MDM. While TIBCO MDM is able to complete lookups and change data, it may be computationally expensive and time consuming. For example, a simple data lookup where an ID is converted into a text value is acceptable within TIBCO MDM. However, if it has to look up a value and then execute a collection of rules based on this value, which then changes other attributes, TIBCO recommends performing this externally. TIBCO MDM executes these rules every time a record (in hundreds or thousands) is presented to it. There are a variety of ways to achieve this. For example, you can use the following:
    • TIBCO Clarity for data discovery and data transformation.
    • ActiveMatrix BusinessWorks (or similar) to access the data required from the source system. Process the data internally to produce an end result that adheres to the required business rules.
    • Other ETL tools such as Kettle.