Polyglot Data Model

Starting with version 5.2.0 of Hackolade Studio, users can now define structures once in a technology-agnostic polyglot data model, complete with denormalization and complex data types, then represent these structures in a variety of physical data models respecting the specific aspects of each target technology.

 

You may want to first read about the background for our polyglot data model approach, and as well as the definition of the polyglot data modeling concept.

 

The polyglot and target models are separate models.  But they are related to ensure consistency.  In other words, the behavior is similar to references to external definitions, but at the model level.  The polyglot data model is the master, and targets models are dependent on the polyglot model.  If changes are required to an object, they must be done in the polyglot model, then the target models must be updated to reflect the changes.

 

Exceptions are possible: some objects in a polyglot model can be flagged as polyglot-only so they do not appear in the target models.  Conversely, you can deactivate entities and attributes in the target model, or you can add objects in the target models that are not reflected in the polyglot model.  What is important, mostly, is that, if attributes or properties of an object change in the polyglot model, you may update dependent target models to reflect those changes.

 

Push and Pull from the target model

Many customers having accumulated physical data models in Hackolade prior to the introduction of the new functionality will want to convert their target model into a polyglot model.  This is done by opening the physical target model, selecting the menu Tools > Polyglot > Convert to Polyglot Model...

 

You must choose whether to make your target dependent of the Polyglot model being created.  This is the recommended approach as the Polyglot model should now become the master of changes, and the target model should be dependent of the Polyglot model.

 

 

You may then choose the location where you want to store the polyglot model.  You're now ready to maintain and update the polyglot data model.

 

You may also create a polyglot data model from scratch and build it, as you would with physical models.  Except that a polyglot data model cannot be forward-engineer to a database instance.

 

Once you're ready to create a physical data model for a given target, the process is that a new model must be created for the target of your choice.  then, from the target model, you derive your physical model from the polyglot data model by selecting the menu Tools > Polyglot > Derive from Polyglot Model:

 

 

You may then choose the polyglot data model file:

 

You are then presented with a dialog where you can narrow down the selection of entities to reference in the target model.  By default, all objects are selected.  Using multi-select (ctrl-click or shift+click) , you may change the selection of containers, entities, and definitions to derive in the target model.

 

 

Attributes of entities cannot be individually deselected in the above screen. But 2 options are available, depending on the use case:

1) in the polyglot model, each object, including attributes can be marked as "Polyglot only", meaning that they will not be included in any target models.

2) in a target model you can deselect the IsActivated property, which will cause the attribute to be filtered out (or commented if the target syntax allows it) in the forward-engineering script.

 

 

Data type mapping

On the polyglot data model side, a long list of physical data types has been compiled across the different targets support by different Hackolade plugins.  It was on purpose that this approach was decided instead of providing only a list of logical data types.  That is so the most relevant level of granularity could be chosen at the polyglot level to be automatically mapped to the appropriate physical data type without minimal user intervention.

 

A data type matrix mapping table is used to provide bi-directional transformation: when converting a target model to polyglot, and when deriving a target model from polyglot.

 

Naming conventions

Polyglot models being technology-agnostic, it is normal that only business names are created and maintained here.  
 
Then when a physical model is derived for any target, it is often the case that the rules to generate physical names from business names are different.  For example, in MongoDB, JSON, Avro, OpenAPI, developers tend to prefer camelCase, whereas some targets like Cassandra cannot have any upper case characters and therefore snake_case is a typical conversion.
 
To generate technical names in physical target models when deriving from a polyglot model, you must first enable Business-to-Technical coupling for the chosen target in Tools > Options > Naming Conventions.  Target-specific transformations rules are applied during the process of deriving from the polyglot model.

 

Maintain and update the target model

Once a target model has been created and derived from a polyglot model, it contains all the derived objects.  You may make some changes to the target model:

- make changes to the properties of entities and attributes without changing them in the polyglot model.  This is used if you need minor deviations;

- add new entities and views that are specific to the physical target model, for example based on access patterns;

- remove entities from the target model.  This action does not delete the entity in the polyglot model;

- add, delete or modify attributes, again without effect on the Polyglot model.

 

You may also join entities from multiple polyglot models by repeating the operation to derive from polyglot.

 

You may supplement the target model with metadata, index and partitioning information specific to the target technology, then forward-engineer artifacts according to your use case.

 

If the polyglot model changes during the time you're editing the target model, you may refresh the derived objects by clicking the update button which can be found in the model-level Properties Pane:

If you have multiple links, possibly to the same polyglot model so you can maintain different profiles, it may be useful to indicate a meaningful name for the link.

 

An impact analysis dialog is displayed to let you decide which objects to include as part of this refresh operation.  Maybe these objects were not selected originally, or they could have been added to the polyglot model since the previous refresh or derive operation, or they could have been deleted in the target model.  

 

 

When opening a target model with a reference to one or more polyglot models, you are presented with the option to update the objects.

 

You may also permanently break the connection to the polyglot model.  This operation should be done with care of course, as it cannot be re-established afterwards without restarting all the steps from scratch.  The operation replaces all references to the objects of the polyglot model by the objects themselves.  But further modifications in the polyglot model will no longer have any effect in the target model.

 

Advanced use cases

Attribute deviations

After deriving from a Polyglot model into any target, you may want to adjust the properties of some attributes, delete unneeded attributes, or add more attributes. From v5.4.0 it is possible to modify attribute properties, and from v6.6.4 it is possible to delete and insert attributes.

Doing deviations is simple, but be aware that some operations are restricted. For example, it is not allowed to change the data type of an attribute derived from a Polyglot model.  The software behaves this way to avoid possible conflicts in the properties because each type has its own set of properties. 

If you need such a change, you may either change the data type in the Polyglot model, then update the target model.  Or you may delete the derived attribute in the target model, and create/insert a new one.

Other operations are restricted for derived attributes in target models:
- convert a regular attribute to pattern attribute
- convert a pattern attribute to a regular attribute
- replace a reference by its attributes (if the attribute is a reference to a udt/definition)
- change the “JSON Type” of an attribute (if type supports this)

All deviations are stored in the target model and don’t affect the base polyglot model.  

 

Derive subsets from multiple polyglot models

The polyglot architecture provides tremendous power and flexibility.  It is possible to derive a target model from multiple polyglot models, or subsets thereof:

 

This is done in a target model by clicking the + sign in the model-level Properties Pane, and choosing an additional polyglot model from which to select the object or objects you wish to derive into your target model.

In some case, you may even have a use case to derive multiple subsets from the same polyglot model.  An example if why you may want to do this is if you want different subsets and/or deviations from a given polyglot model in the same target model.  In this OpenAPI model, different instantiations of the same User object are justified as you want a different response structure depending on whether a criteria was satisfied, or not.

 

Impact Analysis

The impact analysis window is an extended version of model comparison window. You may open this window by clicking the update button near the link to the Polyglot model on the properties pane, when the model is selected.

In the impact analysis window, all entities and attributes in the right-hand pane have checkboxes. The checked state means that this object will be included in the target model after the update.  If you want an object to not appear in the target model, you should unchecked the box.  

Color coding and badges help you understand additions, deletions, and modifications:
- turquoise: deviations in the target model, either:

- insertion of target-only attributes (plus sign + in blue badge)
- or modification of properties of existing polyglot attribute (exclamation mark ! in yellow badge)

- red: deletions in the polyglot model (strike through font and x in red badge) 

- blue: attributes in polyglot model and

- were removed in target model (strike through font and plus sign + in blue badge)
- were added in polyglot model (plus sign + in blue badge)

Added, deleted, and modified attributes in the Polyglot model have blue, red and yellow colors respectively (as in the regular compare models window)

If you are not sure what type of difference it is and whether it is changed in the Polyglot model or target model, you can hover on the icon and read the tooltip.

In the impact analysis window also possible to restore attributes in the target model if they were deleted in the Polyglot model. In this case, attributes will be preserved and will work in the same way as attributes that were added to the target model. So the next time you update references to your Polyglot model, you will see their torque.

To view what properties were changed in the target model you can select the attribute, and on properties panes below you can see properties with turquoise color.