Open topic with navigation

TIBCO® Control Plane 1.2.0

# Preparing Logs and Traces Servers

Before you start using the Logs and Traces server configured for observability resource, you must create corresponding index templates on the Elasticsearch server.

You can create and manage index templates by using one of the following ways:

- Use the index template APIs to create and update index templates.

- You can also manage index templates from Index Management in Kibana.

Elasticsearch applies templates to new indices based on an wildcard pattern that matches the index name. Index templates are applied during index creation. Settings and mappings specified in a create index request override any settings or mappings specified in an index template.

Note: The index settings provided in the following steps are example configurations and might not be applicable to your organization. Check with your Elasticsearch server admin for the appropriate configurations.

## Index Templates Repository on GitHub

All index templates related to logs and traces server are available on GitHub repository.

Before you begin

If the Elasticsearch security features are enabled, you must have the `manage_index_templates` or manage cluster privilege to use this API. For more information, refer Elasticsearch documentation.

## Step 1: Creating App Logs Index Template

The user app log index template is used by BWCE and Flogo capability. Index of the user app logs is created when the index template is created and the configured index name of the observability resource for the app log matches the regex of the index template name.

User App Log Index Template

    #
    # Copyright © 2023. Cloud Software Group, Inc.
    # This file is subject to the license terms contained
    # in the license file that is distributed with this file.
    #
    {
      "index_patterns": "user-app-*",
      "settings": {
        "number_of_shards": 1,
        "number_of_replicas": 0,
        "refresh_interval": "5s",
        "index": {
          "codec": "best_compression",
          "lifecycle": {
            "name": "your-user-app-index-60d-lifecycle-policy",
            "rollover_alias": "user-apps"
          }
        }
      },
      "mappings": {
        "properties": {
          "@timestamp": {
            "type": "date"
          },
          "Body": {
            "properties": {
              "log_time": {
                "type": "date"
              },
              "log_level": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "message": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              }
            }
          },
          "Resource": {
            "properties": {
              "app_id": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "app_type": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "dataplane_id": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "workload_type": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "app_tags": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "pod_namespace": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "pod_name": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "pod_uid": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              }
            }
          }
        }
      }
    }

Lifecycle Policy for User App Log Index Template

    {
          "policy": {
            "phases": {
              "hot": {
                "min_age": "0ms",
                "actions": {
                  "rollover": {
                    "max_primary_shard_size": "50gb",
                    "max_age": "7d"
                  },
                  "set_priority": {
                    "priority": 100
                  }
                }
              },
              "delete": {
                "min_age": "60d",
                "actions": {
                  "delete": {}
                }
              }
            }
          }
        }

## Step 2: Creating User App Traces Index Template

The user app trace index template is used by BWCE and Flogo capability. Index of the user app traces is created when the index template is created and the configure index name of the observability resource for the app traces matches the regex of the index template name.

Jaeger Service Trace Index Template

    {
      "priority": 5,
      "index_patterns": ["*jaeger-service-*"],
      "template": {
        "aliases": {
          "jaeger-service-read": {}
        },
        "settings": {
          "index.number_of_shards": 6,
          "index.number_of_replicas": 1,
          "index.mapping.nested_fields.limit": 50,
          "index.requests.cache.enable": true,
          "lifecycle": {
            "name": "jaeger-policy",
            "rollover_alias": "*jaeger-service-write"
          }
        },
        "mappings": {
          "dynamic_templates": [
            {
              "span_tags_map": {
                "mapping": {
                  "type": "keyword",
                  "ignore_above": 256
                },
                "path_match": "tag.*"
              }
            },
            {
              "process_tags_map": {
                "mapping": {
                  "type": "keyword",
                  "ignore_above": 256
                },
                "path_match": "process.tag.*"
              }
            }
          ],
          "properties": {
            "serviceName": {
              "type": "keyword",
              "ignore_above": 256
            },
            "operationName": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        }
      }
    }

Jaeger Span Trace Index Template

    {
      "priority": 3,
      "index_patterns": ["*jaeger-span-*"],
      "template": {
        "aliases": {
          "jaeger-span-read": {}
        },
        "settings": {
          "index.number_of_shards": 6,
          "index.number_of_replicas": 1,
          "index.mapping.nested_fields.limit": 50,
          "index.requests.cache.enable": true,
          "lifecycle": {
            "name": "jaeger-policy",
            "rollover_alias": "*jaeger-span-write"
          }
        },
        "mappings": {
          "dynamic_templates": [
            {
              "span_tags_map": {
                "mapping": {
                  "type": "keyword",
                  "ignore_above": 256
                },
                "path_match": "tag.*"
              }
            },
            {
              "process_tags_map": {
                "mapping": {
                  "type": "keyword",
                  "ignore_above": 256
                },
                "path_match": "process.tag.*"
              }
            }
          ],
          "properties": {
            "traceID": {
              "type": "keyword",
              "ignore_above": 256
            },
            "parentSpanID": {
              "type": "keyword",
              "ignore_above": 256
            },
            "spanID": {
              "type": "keyword",
              "ignore_above": 256
            },
            "operationName": {
              "type": "keyword",
              "ignore_above": 256
            },
            "startTime": {
              "type": "long"
            },
            "startTimeMillis": {
              "type": "date",
              "format": "epoch_millis"
            },
            "duration": {
              "type": "long"
            },
            "flags": {
              "type": "integer"
            },
            "logs": {
              "type": "nested",
              "dynamic": false,
              "properties": {
                "timestamp": {
                  "type": "long"
                },
                "fields": {
                  "type": "nested",
                  "dynamic": false,
                  "properties": {
                    "key": {
                      "type": "keyword",
                      "ignore_above": 256
                    },
                    "value": {
                      "type": "keyword",
                      "ignore_above": 256
                    },
                    "tagType": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                }
              }
            },
            "process": {
              "properties": {
                "serviceName": {
                  "type": "keyword",
                  "ignore_above": 256
                },
                "tag": {
                  "type": "object"
                },
                "tags": {
                  "type": "nested",
                  "dynamic": false,
                  "properties": {
                    "key": {
                      "type": "keyword",
                      "ignore_above": 256
                    },
                    "value": {
                      "type": "keyword",
                      "ignore_above": 256
                    },
                    "tagType": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                }
              }
            },
            "references": {
              "type": "nested",
              "dynamic": false,
              "properties": {
                "refType": {
                  "type": "keyword",
                  "ignore_above": 256
                },
                "traceID": {
                  "type": "keyword",
                  "ignore_above": 256
                },
                "spanID": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "tag": {
              "type": "object"
            },
            "tags": {
              "type": "nested",
              "dynamic": false,
              "properties": {
                "key": {
                  "type": "keyword",
                  "ignore_above": 256
                },
                "value": {
                  "type": "keyword",
                  "ignore_above": 256
                },
                "tagType": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            }
          }
        }
      }
    }

Lifecycle Policy for Jaeger Index Template

    {
          "policy": {
            "_meta": {
              "description": "this will be used for traces",
              "project": {
                "name": "jaeger",
                "department": "platform infra"
              }
            },
            "phases": {
              "hot": {
                "actions": {
                  "rollover": {
                    "max_size": "2GB",
                    "max_age": "7d"
                  }
                }
              },
              "warm": {
                "min_age": "10d",
                "actions": {
                  "forcemerge": {
                    "max_num_segments": 1
                  }
                }
              },
              "delete": {
                "min_age": "30d",
                "actions": {
                  "delete": {}
                }
              }
            }
          }
        }

## Step 3: Creating TIBCO Service Logs Index Template

The TIBCO service app log index template is used by services provisioned by TIBCO Control Plane. Index of the TIBCO Service logs is created once the index template is created and the configured index name of the observability resource for the service log matches the regex of the index template name.

TIBCO Service Log Index Template

    #
    # Copyright © 2023. Cloud Software Group, Inc.
    # This file is subject to the license terms contained
    # in the license file that is distributed with this file.
    #
    {
      "index_patterns": [
        "tibco-service-*"
      ],
      "settings": {
        "number_of_shards": 1,
        "number_of_replicas": 0,
        "refresh_interval": "5s",
        "index": {
          "codec": "best_compression",
          "lifecycle": {
            "name": "your-tibco-service-index-60d-lifecycle-policy",
            "rollover_alias": "user-apps"
          }
        }
      },
      "mappings": {
        "properties": {
          "@timestamp": {
            "type": "date"
          },
          "Body": {
            "properties": {
              "log": {
                "properties": {
                  "caller": {
                    "type": "text",
                    "fields": {
                      "keyword": {
                        "type": "keyword",
                        "ignore_above": 256
                      }
                    }
                  },
                  "error": {
                    "type": "text",
                    "fields": {
                      "keyword": {
                        "type": "keyword",
                        "ignore_above": 256
                      }
                    }
                  },
                  "errorVerbose": {
                    "type": "text",
                    "fields": {
                      "keyword": {
                        "type": "keyword",
                        "ignore_above": 256
                      }
                    }
                  },
                  "level": {
                    "type": "text",
                    "fields": {
                      "keyword": {
                        "type": "keyword",
                        "ignore_above": 256
                      }
                    }
                  },
                  "msg": {
                    "type": "text",
                    "fields": {
                      "keyword": {
                        "type": "keyword",
                        "ignore_above": 256
                      }
                    }
                  },
                  "stacktrace": {
                    "type": "text",
                    "fields": {
                      "keyword": {
                        "type": "keyword",
                        "ignore_above": 256
                      }
                    }
                  }
                }
              },
              "time": {
                "type": "date"
              }
            }
          },
          "Resource": {
            "properties": {
              "dataplane_id": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "k8s": {
                "properties": {
                  "namespace": {
                    "properties": {
                      "name": {
                        "type": "text",
                        "fields": {
                          "keyword": {
                            "type": "keyword",
                            "ignore_above": 256
                          }
                        }
                      }
                    }
                  },
                  "pod": {
                    "properties": {
                      "ip": {
                        "type": "text",
                        "fields": {
                          "keyword": {
                            "type": "keyword",
                            "ignore_above": 256
                          }
                        }
                      },
                      "name": {
                        "type": "text",
                        "fields": {
                          "keyword": {
                            "type": "keyword",
                            "ignore_above": 256
                          }
                        }
                      },
                      "uid": {
                        "type": "text",
                        "fields": {
                          "keyword": {
                            "type": "keyword",
                            "ignore_above": 256
                          }
                        }
                      }
                    }
                  }
                }
              },
              "workload_type": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              }
            }
          },
          "SeverityNumber": {
            "type": "long"
          },
          "TraceFlags": {
            "type": "long"
          }
        }
      }
    }

Did you find this helpful?

Yes No
Great!
Thanks for taking the time to give us some feedback.
Sorry about that

Why wasn't this helpful? (check all that apply)

Could not find what I was looking for
Instructions confusing or unclear
Instructions did not work

Want to tell us more?

Please select at least one option or provide details about the issue!!

Error occurred while submitting feedback.

Error2 occurred while submitting feedback.

TIBCO® Control Plane

Copyright © 2023-2024. Cloud Software Group, Inc. All Rights Reserved.