Semarchy xDM upgrade guide

Before starting the upgrade, you should review the following documents:

Version numbers in Semarchy xDM are expressed in the following format: <major_version>.<minor_version>.<patch_version>. For example, version 1.3.2.

The upgrade may be a major upgrade, a minor upgrade or a patch. The differences between the old version number and the new version number define the type of upgrade.

Although the process is the same for all types of upgrade, be aware that the repository and data location upgrade steps may be skipped for patches.

Unless specified otherwise, the upgrade path between two version of Semarchy xDM is direct. You do not need to install intermediate versions.

The upgrade path is as follows:

The upgrade path is as follows:

The following changes in the system requirements may impact your setup:

The following changes in the system requirements may impact your setup:

Semarchy xDM discontinues the support for old web browsers. The updated list of supported web browser is available in the Client Requirements section of Semarchy xDM Installation Guide.

SOAP web services available in previous versions of Semarchy xDM are decommissioned in this release. A new REST API is made available for new users, providing similar integration capabilities for querying data in the hub and publishing data into the MDM hub. Note that certain features available in the SOAP Web Services are not available in the REST API, such as workflow management and administrative services.

This version does not include the Pulse dashboards in the generated application.

The data edition feature is removed in this release. A data location now hosts one deployed model edition at a time and provide access to the current state of the data in the hub, based on the deployed model edition.

The upgrade process prunes the data in closed data editions from the data locations. Only data from the latest data edition is preserved in each data location.

Error storage has been changed in this release. The golden errors and source error tables no longer store the entire history of errors, but only the latest error instances.

The data location upgrade process prunes the history of errors on golden and source records, preserving only the latest error instances.

To install the new application version for in-place upgrade:

To perform an in-place upgrade for Apache Tomcat.

To install the new application version for out-of-place upgrade:

For in-place upgrade, the updated deployment reuses the configuration already in place. Check this configuration in the application server console or configuration files. The following points should be checked:

To configure the new application version in an out-of-place upgrade:

The Oracle database has introduced in version 11g and 12c a new feature called Automatic Column Group Detection that automatically creates extended statistics for column groups. These extended statistics prevent the upgrade process to rename certain columns in the repository, causing an ORA-54032: column to be renamed is used in a virtual column expressionerror to appear during the repository upgrade process.

Before upgrading a repository hosted in an Oracle 11g or 12c instance, it is required to drop these statistics. They will be automatically re-created by the database as needed.

To drop unwanted extended statistics for repositories hosted in Oracle 11g and 12c:

Once the unwanted extended statistics are dropped, proceed with the repository upgrade.

To upgrade the repository:

The repository upgrade starts. When it completes, the Overview window appears.

Before upgrading a data location hosted in an Oracle 11g or 12c instance, it is required to drop the extended statistics that may prevent column renaming. For more information, refer to Before Upgrading the Repository.

To drop unwanted extended statistics:

Once the unwanted extended statistics are dropped, proceed with the upgrade.

To upgrade the data locations:

The upgrade starts. When it completes, the Overview window is refreshed and and warning no longer appears.

In this version, business views support searching and filtering every collection in the business view. In previous releases, filtering was only supported on the root collection of the business view. This new feature implies the following changes:

In this version, a new Date datatype replaces Datetime to store dates with no time or timestamp. The Timestamp datatype remains to store date and time with timestamp information. The upgrade process automatically converts Datetime attributes to the new Date datatype in the model and data locations. As a consequence, data columns using the old Datetime type are automatically trimmed to store a Date.

The DATETIME type is deprecated for plug-in enrichers and validators, and the new DATE type is added..

In this version, the data purge is modified with the following changes:

In addition:

A new option is available for matching entities, on duplicates managers and form steps. Enable Master Value Picking enables users overriding the value of consolidated golden records to select values from the master records in addition to being able to enter their own values.

When upgrading to this version, this option is automatically activated for duplicate managers, but not for form steps.

This version introduces a new visualization for business views. Hierarchies views provide a user-friendly way for business users to navigates hierarchies (of products, organization, cost centers, employees) with an intuitive user interface.

Hierarchies are not configured by default during the upgrade.

In this version, it is possible to enable automated historization to keep track of all changes on master and golden data for each entity. Applications can be configured to allow browsing the records history.

The upgrade does not automatically enable historization for the entities.

This version introduces the capability to perform soft and hard deletes for records in all entity types.

This option is not automatically enabled by the upgrade.

Starting with version 4.3.0, ID and fuzzy matched entities support authoring (creating, editing, importing) source or erroneous records on behalf of publishers. This option is enabled when configuring authoring actions for ID and fuzzy matched entities.

Existing action sets are not modified by the upgrade process to support this feature.

In this release, the following changes take place in the integration interfaces and should be reviewed by the integration specialists:

After a model upgrade, the data entry workflows that existed in version prior to version 4.0.0 are not automatically upgraded. They are preserved in a hidden form in the model.

The data authoring experience in version 4 supports two methods:

Workflows and steppers are triggered using actions defined in action sets.

After a model upgrade, the duplicate management workflows that existed in version prior to version 4.0.0 are not automatically upgraded. They are preserved in a hidden form in the model.

The duplicate management experience in version 4 now uses a dedicated design-time artifact, the Duplicate Manager, duplicate management actions defined in action sets to review, confirm, merge or split duplicates and suggestions.

ID and fuzzy matched entities (also known as UDPK and SDPK entities) now support overriding data values on golden records. When data authoring operations are performed in these entities, the data authored (Authoring Data) by the users is clearly separated from the data loaded from external publishers (Source Data).

The consolidators have evolved into Survivorship rules, and support two phases:

The survivorship rules allow implementing simply the pattern into which users fix consolidated values and/or create directly golden records in matching entities, without having to rely on matching and consolidation rules. There as now used only for the data loaded from external publishers (Source Data)

In this version, a Documentation property is added to entity attributes, complex types' definition attributes, references, matching and survivorship rules. The documentation property supports the Markdown syntax for rich text. It provides detailed information to end-users using the generated application.

The Description property is not only used for describing/documenting model artifacts for model designers. Descriptions are no longer visible to end-users.

The documentation of an attribute/reference is used by default in all forms fields based on this attribute/reference. It is possible to override this inherited documentation in the form field.

The upgrade process automatically copies the content of the Description properties (which was visible to end-users before) to the Documentation property.

The data edition feature is removed in this release. A data location now hosts one deployed model edition at a time and provide access to the current state of the data in the hub, based on the deployed model edition.

Post-consolidation validations, running on consolidated data, are no longer removing golden records. They only flag these records as erroneous, yet keeping them as golden records.

The storage of errors has changed in this release:

The structure and the content of the applications and application components have considerably changed between versions 3.x and 4.x. This section details how the repository upgrade converts original artifacts into new ones.

Customers who have upgraded to version 3.2 prior to version 3.2.4 may experience an issue with singleton golden records (for fuzzy matching entities) already in the hub not being merged with incoming records that match them. This issue is caused by a defect in the upgrade mechanism.

New capabilities added to the SemQL language introduce the following reserved keywords:

The Integration Job changes from a "by entity" to a "by Phase" execution scheme. This means that instead of processing entirely each entity, for example, Customers then Contact, it now processes entirely each phase (Enrichment for all entities, then validation for all entities, etc). This change does not impact the outcome of the integration job.

In versions prior to 3.2, a matcher includes a single matching condition.
Starting with version 3.2, a matcher is composed of several matching rules and supports scoring for automated merge and confirmation. To reflect this new matcher and provide performance enhancements, the data certification job was re-designed and the data location structures have been modified.

In versions prior to 3.2, all the default built-in search methods (full text, advanced, by form and SemQL) were available in the business object views. Starting with 3.2, it is possible to create customized forms and define which of the search methods or customized forms are available in applications.

Starting with version 3.2, it is possible to declare user-defined PL/SQL functions. By declaring these functions, designers make them available in the SemQL editor and avoid SemQL warnings raised by these functions.

Prior to version 3.2, the attributes' physical column names could take any value. Starting with version 3.2.0, it is no longer possible to define physical column names starting with B_ as they may collide with reserved column names.

Models with physical column names violating this rule no longer pass model validation and each column will raise a validation error such as Invalid physical name: <column_name> matches the reserved pattern B_*.

In versions prior to 3.1, the semarchyAdmin role had a hardcoded full access to the data on all models.
Starting with version 3.1 the privilege grant for this role is no longer hardcoded and can be modified.

When a creating a new model, or upgrading an existing one, a model-level privilege grant is automatically created for the semarchyAdmin role with Grant full access to the model selected to keep the same level of privileges.

In versions prior to 3.1, accessing the integration services required a role having platform-level read/write privileges to the data location. Starting with version 3.1, a specific option called Grant access to integration services appears in the model privilege grants to define whether a role can access this model’s data via the services.

This privilege applies to the SOAP and REST Service APIs.

When upgrading, roles that could access the integration services thanks to their privileges on the data locations will automatically have this option checked for the models deployed in the data location.