How to Promote Data from DEV to PRODUCTION

User Story

Large enterprises are concerned with data accuracy. As such, they may have a “user test” environment that is separate from QA where they want users to test the application and author data. This process is critical to ensure that users are not authoring data for the first time in production in case there are high-visibility mistakes or critical errors.

After business users enter data in the “TEST” environment, they do not like to re-enter the same data into the production environment because it is tedious and risks human-prone data entry errors. Therefore, users want the ability to copy the data from the TEST environment to the Production environment so they don’t need to reenter the same information.

A separate user story is the reverse: how to refresh data from the Production data location to the Dev data location so that dev can have fresh data from production to test match rules and model rules again. This article covers how to promote data in the direction from Dev to Production.

This article is about how to promote data, not metadata and not the model. For information on how to promote your metadata or model, check out the Administration guide.

Solution

The main two ways we recommend promoting data from dev to prod is:

  1. If the data volumes are small: export the data from development and import into production.

  2. If the data volumes are large or if automation is preferred: use your ETL tool or scripts to load the data from development to production.

While we cannot provide more precise recommendations (every xDM user has specific requirements), there are multiple factors you should consider to be successful. Here are some factors to think about and the best practices around them.

Factors to Consider

Our recommendations in this article are intended for basic entities. Refreshing from dev to prod is not intended for fuzzy-matching (unless Semarchy support recommends it). 

Manual and SemQL IDs (Basic Entity) Are OK

For manual and SemQL IDs, the 2 solutions listed above should work. You don’t need to worry too much about ID considerations because your business users must specify the IDs correctly. Export the data into a spreadsheet in dev and import the data into Production for small volumes of data. Use ETL for big volumes of data. 

Sequence IDs (Basic Entity) Beware!

If your entity uses a sequence ID, you will get new IDs when you import the data into production because the sequence you use in development will likely be at a different value than the sequence in production. (v5.1 and previous version behaves this way – v5.2 offers new features to make this task easier).

The instability of the IDs because of the inconsistency of the next value in the development vs. production sequence means you must factor in the references/relationship IDs. Respect the sequence values to avoid sequence conflicts. Manually increment the sequence to a bigger number in Production (this means make sure the next value for the sequence in Production is set to a larger number than the range of ID values you are using in the dev data you’re about to import).