...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
...
Introduction
TIC 4.0 uses the JSON format as default because it allows to express an array of subjects (for several sub-subjects) or observed properties (for different combinations of timestamps, pom, pomt, names, etc). JSON is also the default file format for many protocols, such as REST and MQTT, which are extensively used to share data or publish IoT data. These arrays give us a high flexibility thanks to the hierarchical structure. Other hierarchical structures, such as XML, are possible in TIC 4.0 lbut we recommend using JSON as default.
...
This document contains the rules to convert the JSON structure into a Flat one (2022.005 will cover Flat to JSON).
JSON to Flat: the general concept
The main goal of the conversion is to ensure that all the JSON information is transferred in the flat messages after the conversion. As flat messages have limitations this would require creating several flat messages for each JSON message. As the JSON format can include arrays of data, the flat format must include a way to identify the array in the flat message. This will be done by including some information in the flat variable name transforming the name of the variable with hierarchical and array information.
...
The first option will reduce the number of messages but will create a specific variable name for each CHE. The second option will create a lot of messages, but all of them will have the same variable names, which is very important if you want to compare the CHE values.
Clarifications
It is important to understand that One message with a hierarchy structure (JSON or similar) would be transformed into one or more flat messages (and vice versa). The number of messages will depend on the number of different timestamps and the classification per subject (if require).
...
The semantic combines 6 basic elements: HEADER, SUBJECT, CONCEPT, OBSERVED PROPERTY, POINT OF MEASUREMENT and VALUE to represent a unique reality. TIC4.0 JSON format uses arrays for the SUBJECT and OBSERVED PROPERTY:
The SUBJECT never has timestamps
The OBSERVED PROPERTY always has a timestamp.
The CONCEPT never has arrays.
The POINT OF MEASUREMENT and VALUE (with their units) are always part of the observed properties so they cannot have an array (but can be part of one array).
...
In this JSON example, “msg” is an object and “id” is a value in JSON language. The objects contain values and the values have a payload with “data”. TIC4.0 distinguish between two types of JSON “values”: “metadata“ and “value”. The “metadata” doesn't change across the time and the “value” changes (it requires a timestamp).
Rule 1: One flat message per timestamp
A single message will be generated for each of the different timestamps present in the source JSON message.
...
| Bloque de código |
|---|
[
{
"msg|id": "001",
"msg|timestamp": "2021-11-18T08:27:28.609Z",
"msg|sample": 1,
"che|@|name": "STS01",
"che|@|on|totalcounter|ioutput|actual|timestamp": "2021-11-18T08:27:28.609Z",
"che|@|on|totalcounter|ioutput|actual|value": "0"
}
] |
Rule 2: Pathname includes objects and properties
To define the VALUE it is required to combine the SUBJECT, CONCEPT, OBSERVED PROPERTY, POINT OF MEASUREMENT in the path name. If the array contains values, it will be considered as a primitive element, but if it contains an object, the name of the final property will include the value of some of the attributes in the path.
...
| Bloque de código |
|---|
[
{
"msg|id": "001",
"msg|version": "2022.003",
"msg|timestamp": "2022-02-14T08:23:55.000Z",
"msg|sample": 1,
"che|@|name": "STS01",
"che|@|family": "STS",
"che|@|arrayid": "1",
"che|@|manufacter.name": "Example",
"che|@|manufacter.acronym": "E"
}
] |
Rule 3: Subjects include the extension “@”
The SUBJECT is always an array. To be able to convert automatically from Flat to JSON an identifier must be included in the pathname to identify that the subject is an array. Any array without a timestamp is a SUBJECT.
In consequence, if there is an array without any timestamp we will add to the pathname the character “@” with the “@arrayid” if exist. The “@” is always necessary and in case the array has an “arrayid” identification the value of it must be included after the ”@arrayid".
...
| Bloque de código |
|---|
[
{
"msg|id": "001",
"msg|sample": 1,
"che|@|name": "STS01",
"che|@|spreader|@spreader1@|id": "brm23423",
"che|@|spreader|@spreader1@|name": "spreader1",
"che|@|spreader|@spreader1@|number": "33",
"che|@|spreader|@spreader1@|arrayid": "spreader1",
"che|@|spreader|@spreader1@|location|technical|ioutput|actual|timestamp": "2021-11-18T08:27:28.613Z",
"che|@|spreader|@spreader1@|location|technical|ioutput|actual|value": "1",
"che|@|spreader|@spreader1@|location|technical|ioutput|actual|qualifier": "spreader"
}
] |
...
The previous example only contains 1 subject and 1 timestamp. If the JSON contains several subjects of the same type or several timestamps in the same message it is not possible to express it in a single flat plain message. In such a case, the JSON message must split into several FLAT messages and/or use the “@arrayid” extension to express the same information in flat format identifying the arrays.
Rule 4: Observed properties included in the pathname
The OBSERVED PROPERTY always has an array that includes at least the “timestamp”. Each array element is always a unique combination of the values of the object: timestamp, POINT OF MEASUREMENT, unit etc. Therefore the array identification is made by combining part of the content to make it unique. At the present release 2022.004 (in the future are expected to incorporate new values to this list) the values to include in the path name to identify the array are:
...
| Expandir | |||||||||
|---|---|---|---|---|---|---|---|---|---|
| |||||||||
Would result in 2 messages because of two different timestamps:
|
Rule 5: Split by subject
For many applications, it is necessary to create a message per subject (or sub-subject) as it is not possible to compare similar subjects. For example, compare CHE or spreaders or tires. So it is necessary to create a database based on independent registers per one specific subject.
...
| Expandir | ||
|---|---|---|
| ||
|
Message Validation
It is an independent process from the flattening procedure. Usually to be done before flattening procedure or just to check if the message complies with the TIC4.0 schema.
The validations are made based on a JSON Schema. The schema doesn't limit the content to TIC4.0, but if the attribute is a TIC4.0 attribute the Schema will check if the format is TIC4.0.
Open Source Code
The backend of these code is public available at
...