A Guide to (Careful) Ontology Changes

The ontology is crucially connected to the entire Octostar platform. A modification to the ontology can have cascading consequences due to inheritance and dependencies, and MUST be reflected in:

• Local data tables in ClickHouse
• Mappings in Timbr
• Dashboards in the Octostar Platform
• Applications in the Octostar Platform
• Templates in the Octostar Platform
• The superset-octostar codebase, in the worst cases (and therefore potentially the streamlit-octostar-research and octostar-python-client wrapper libraries)

🚨 Any part of the Octostar platform can break after an ontology change, BE CAREFUL 🚨

General Tips

BEFORE ANY MODIFICATIONS

Before making any changes to the ontology, It is good practice to run the BACKUP ONTOLOGY timbr command

[

timbr - Produce Smarter Queries

truehttps://timbr.octostar-main.saas-dev.octostar.com/doc/docs/sql/ontology#backup-a-specific-ontology

](https://timbr.octostar-main.saas-dev.octostar.com/doc/docs/sql/ontology#backup-a-specific-ontology)

The ontology can be modified directly from the GUI of the Fusion Center (login credentials may be asked again). Some functionalities may not yet be supported by the GUI, in which case it is best to create the corresponding SQL statement and execute it e.g. via the SQL Lab.

Any modification which produced undesired effects can be undone via the Version History tool

AFTER ANY MODIFICATIONS

Immediately after making changes to the ontology, it is good practice to:

• run the REFRESH ONTOLOGY command in timbr
• clear the cache from your browser session in the Octostar platform
• test, test, test

Revert if any behaviors are unexpected.

Operations Guide

Following are the specific steps to follow after an ontology change, by type. Do note that some specific changes may require actions beyond what's described here, as we strive to make this guide more comprehensive.

Adding a New Business Concept

GENERALLY SAFE

Non-destructive and safe, operation, especially if the concept inherits from a concept lower in the hierarchy. No further actions are needed after this change.

If the concept directly inherits from os_business_workspace_record, keep in mind it will show as a high-level concept e.g. in the Explore tab.

Adding a New OS Concept

GENERALLY SAFE

Further actions may be required depending on the purpose of this new concept. Any change in the codebase using this new concept should be standardized, in particular:

• Add a constant as appropriate at the top of octostar-superset\superset-frontend\src\octostar\interface.ts and always use it to refer to that concept
• The same applies for apps, templates, etc.

Renaming an Existing (OS or Business) Concept

BE CAREFUL

A heavy-duty operation. The following operations must be performed:

• The local entities table in ClickHouse must be updated to reflect the new entity name. Run the following query with utmost care:

ALTER TABLE {ONTOLOGY_NAME}_os_local_etimbr_table [ON CLUSTER {CLUSTER_NAME}]
UPDATE COLUMN os_concept='{NEW_CONCEPT_NAME}' WHERE os_concept='{OLD_CONCEPT_NAME'}

• Both fields os_entity_type_from and os_entity_type_to in the local entities table must be altered to reflect the new concept name, like so:

ALTER TABLE {ONTOLOGY_NAME}_os_local_etimbr_table [ON CLUSTER {CLUSTER_NAME}]
UPDATE COLUMN os_entity_type_from = '{NEW_CONCEPT_NAME}' WHERE os_entity_type_from = '{OLD_CONCEPT_NAME}';
ALTER TABLE {ONTOLOGY_NAME}_os_local_etimbr_table [ON CLUSTER {CLUSTER_NAME}]
UPDATE COLUMN os_entity_type_to = '{NEW_CONCEPT_NAME}' WHERE os_entity_type_to = '{OLD_CONCEPT_NAME}'

• The concept name may need to be updated in file octostar-superset\superset-frontend\src\octostar\interface.ts
• The concept name may need to be updated in the rest of the octostar-superset codebase, run a grep for the old concept name
• Any application/template using that concept name as a literal should update to the new name. This generally includes manifest.yaml as well as scanning all codefiles
• Semantic bindings in dashboards and templates may need to be updated
• Any mappings containing the old concept name as a literal '{OLD_CONCEPT_NAME}' should be updated to the new concept name. Care should be taken to correctly interpret the meaning of the literal in the mapping
• Any mappings containing the old table name should be updated to the new table name (via the redo_mappings Octostar API)

Removing an Existing Business Concept

BE CAREFUL

A relatively safe operation. In all cases you should be careful that:

• Semantic sets and queries (including app and templates codefiles) which would have included that concept may not include it anymore, or may error if they relied on a property/relationship which is not present anymore
• These changes recursively impact all concepts inheriting from the modified concept

If the removed relationship is an OTM relationship, no further actions are needed. If it is an MTM relationship, you may want to:

• Drop any MTM tables used in precedence, if they are not needed anymore
• Clean up data from the local entities table for the entities of type os_workspace_relationship

Removing an Existing OS Concept

DON'T

The only reasons to remove an OS concept are if, and only if:

• The concept is completely unused, meaning it has no active data, no active mappings and nothing in the codebase/applications depends on it, OR
• The concept is being superseded by a new OS concept. In that case, follow the steps to add a new OS concept, the guide to changing a concept properties/parents, as well as the guide to rename a concept FIRST.

Then test, test, test the platform! Do NOT drop data yet! Make sure that:

1. The platform opens and workspaces are loaded correctly

2. Querying for the new data works, including in linkcharts, dashboards, etc.

3. Opening records with the record viewer works

4. Importing/Exporting files/workspaces work

5. Dashboards work

6. Applications work

and so on...

Then get someone to review the work, and ONLY THEN, you may drop any old data and the old concept in the ontology.

Adding a New/Existing Property to a Concept

GENERALLY SAFE

A safe operation, however the following operation MUST be performed:

• Any local mappings involving the property must be updated (via the refresh_local_tables Octostar API) for the new property to become writable
• Any global mapping involving the property must be manually updated

Removing a Business Property from a Concept

BE CAREFUL

A dangerous operation. If the property is part of the concept's label, you must:

• Remove the concept label first
• Then remove the property, and add back the label (without the removed field)

In all cases, you should be careful that:

• Semantic sets and queries (including codefiles from apps and templates) which involved that property will error out
• The change recursively impact all concepts inheriting from the modified concept

Removing a Business Property from the Ontology

BE CAREFUL

timbr provides a one-click solution to remove the property altogether from the GUI. There are similar considerations to be made as Removing a Business Property from a Concept.

Removing an OS Property from a Concept or the Ontology

DON'T

Similar considerations as Removing an existing OS Concept.

Renaming an Existing Property

CUMBERSOME AND ERROR PRONE

Currently, the timbr GUI does not allow renaming a property already in use by one or multiple concepts. Therefore, the process to rename an existing property would be to:

• Remove that property from all concepts, and remove all labels containing that property
• Rename the property
• Add back the property to all concepts, and add back all removed labels

This is currently a manual job (although a script could be created for this). Furthermore, one would need to:

• Manually move the property data in the local entities table from the old to the new column:

ALTER TABLE {ONTOLOGY_NAME}_os_local_etimbr_table [ON CLUSTER {CLUSTER_NAME}]
ADD COLUMN {NEW_PROPERTY_NAME} Nullable({NEW_PROPERTY_TYPE}) DEFAULT {OLD_PROPERTY_NAME}

where the new property type should be equal to the old property type.

• Any local mappings involving the property must be updated (via the refresh_local_tables Octostar API) for the new property to become visible
• Any global mappings must be manually adjusted to account for the new property name

Changing a Business Property's Type

BE CAREFUL

Although changing the timbr type can be done in one click, the consequences are severe:

• Queries involving that property will likely fail, as there is now a mismatch between the table type and the expected timbr type

Therefore, you must also:

• Adjust the table type for the property according to this table below:

Timbr Type	Table type
String	String
Integer	Int32
Float	Float32
Boolean	Bool
Double	Float64
Date	Date32
DateTime	DateTime
SmallInt	Int16
BigInt	Int64
Decimal	Float32

Of course, depending on the nature of the original data, an implicit (or manual) conversion may need to be performed:

  ALTER TABLE {ONTOLOGY_NAME}_os_local_etimbr_table [ON CLUSTER {CLUSTER_NAME}]
  MODIFY COLUMN {PROPERTY_NAME} Nullable({NEW_TYPE})

Or, in the worst case (incompatible types), the data should be entirely flushed:

ALTER TABLE {ONTOLOGY_NAME}_os_local_etimbr_table [ON CLUSTER {CLUSTER_NAME}]
UPDATE COLUMN {PROPERTY_NAME} = NULL;
ALTER TABLE {ONTOLOGY_NAME}_os_local_etimbr_table [ON CLUSTER {CLUSTER_NAME}]
MODIFY COLUMN {PROPERTY_NAME} Nullable({NEW_TYPE})

• Any local mappings involving the property must be updated (via the refresh_local_tables Octostar API) for the new property to become visible
• Any global mappings must be manually adjusted to account for the new property name

Changing/Removing/Adding a Concept's Label

GENERALLY SAFE

Generally safe, as the label is mostly cosmetic/UI-related. Be mindful that:

• Only the top-level concept with a defined label can be modified, and those changes will be propagated to all inheriting entities
• If a label is removed from an entity, then that entity is considered abstract and local instances of that entity cannot be added anymore by the user. Within the platform, those entities appear without any label and cannot be therefore easily distinguished. Similar considerations in the opposite direction when adding a label to an entity that didn't have one

Adding a Relationship to an Existing Concept

GENERALLY SAFE

Generally safe, non-destructive operation, and requires no further actions.

Removing a Business Relationship from an Existing Concept

BE CAREFUL

A relatively safe operation. In all cases you should be careful that:

• Semantic sets and queries (including codefiles from apps and templates) which involved that relationship will error out
• The change recursively impact all concepts inheriting from the modified concept (that is, the one which defined the relationship)

If the removed relationship is an OTM relationship, no further actions are needed. If it is an MTM relationship, you may want to:

• Drop any MTM tables used in precedence, if they are not needed anymore
• Clean up data from the local entities table for the entities of type os_workspace_relationship

Removing an OS Relationship from an Existing Concept

DON'T

Similar considerations as Removing an existing OS Concept.

Renaming a Relationship

BE CAREFUL

A relatively safe operation. The following operations must be performed:

• Field os_relationship_name in local table {ONTOLOGY_NAME}_os_local_etimbr_table must be altered to reflect the new relationship name, like so:

ALTER TABLE {ONTOLOGY_NAME}_os_local_etimbr_table [ON CLUSTER {CLUSTER_NAME}]
UPDATE COLUMN os_relationship_name = '{NEW_RELATIONSHIP_NAME}' WHERE os_concept='os_workspace_relationship' AND os_relationship_name = '{OLD_RELATIONSHIP_NAME}'

• The concept name may need to be updated in file octostar-superset\superset-frontend\src\octostar\interface.ts
• The concept name may need to be updated in the rest of the octostar-superset codebase, run a grep for the old concept name
• Any application/template using that concept name as a literal should update to the new name. This generally includes manifest.yaml as well as scanning all codefiles
• Any mappings containing the old relationship name as a literal '{OLD_RELATIONSHIP_NAME}' should be updated to the new relationship name. Care should be taken to correctly interpret the meaning of the literal in the mapping

Changing a Relationship's Type (OTM, MTM)

GENERALLY SAFE

This involves removing the relationship and then re-creating it, so follow the corresponding guides. Note that:

• When going from OTM ➝ MTM, you may want to create and map an intermediate table where each row contains tuples (entity_id_from, entity_id_to) corresponding to each OTM match
• When going from MTM ➝ OTM, the intermediate MTM table is not used anymore, therefore the relationships indicated by the MTM table will not be visible anymore. To fix this, the OTM foreign fields should correspond to the old MTM matches in the intermediate table. This can be achieved e.g. by adding a foreign key field to either of the two concepts

You may also want to clean left-over data from the local entities table.

Changing a Concept/Relationship/Property Tag or Description

GENERALLY SAFE

No further actions are needed, as tags are used mostly for appearance and documentation purposes.

Changing an Existing Concept's Parents

BE CAREFUL

Although this operation does not imply removal of the concept (and therefore loss of data), care must be taken regarding the following:

• Properties may be lost and added
• Relationships may be lost and added
• The label field may change or be lost
• Semantic sets and queries which would have included that concept may not include it anymore, or may error if they relied on a property/relationship which is not present anymore
• These changes recursively impact all concepts inheriting from the modified concept

For the local table entities in particular the old properties are generally not lost, such that undoing this modification should bring them back without further actions needed.

If you've made sure this change is behaving as expected, you may drop old data, particularly for unused relationships in the local entities table.